Re: [Xen-devel] [Xen-users] xen domU stall on 4.12.1

Re: [Xen-devel] [Xen-users] xen domU stall on 4.12.1

[Glen]

On Sun, Feb 23, 2020 at 11:12 PM Tomas Mozes <hydrapolic@gmail.com> wrote:
> As reported in https://lists.xenproject.org/archives/html/xen-devel/2020-01/msg00361.html and https://lists.xenproject.org/archives/html/xen-users/2020-02/msg00042.html, switching back to credit1 scheduler seems to make it working again. I've migrated 6 machines to Xen 4.12 with sched=credit xen option and haven't observed a hang for more than a week now.

My experience is the same.  I have migrated all 16 of my physical
hosts back to OpenSuse 15.1 with Xen 4.12.1 with sched=credit .  All
guests are now running perfectly, without any issues at all.  Over
this past week I performed directed stress-testing against several of
my guests, and they all survived without any problems at all.  I've
now completely my migration to the new guests, and everyone is happy.

I'm now going to bring one of the previously-live guests on its own
host back to credit2 so I can crash it and try to capture debugging
output for xen-devel as requested.  But sched=credit is definitely
what we needed to solve this problem!  Thank you all for helping us
get there!

Glen

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] [Xen-users] xen domU stall on 4.12.1

[Tomas Mozes]

[-- Attachment #1.1: Type: text/plain, Size: 1338 bytes --]

On Mon, Feb 24, 2020 at 4:55 PM Glen <glenbarney@gmail.com> wrote:

> On Sun, Feb 23, 2020 at 11:12 PM Tomas Mozes <hydrapolic@gmail.com> wrote:
> > As reported in
> https://lists.xenproject.org/archives/html/xen-devel/2020-01/msg00361.html
> and
> https://lists.xenproject.org/archives/html/xen-users/2020-02/msg00042.html,
> switching back to credit1 scheduler seems to make it working again. I've
> migrated 6 machines to Xen 4.12 with sched=credit xen option and haven't
> observed a hang for more than a week now.
>
> My experience is the same.  I have migrated all 16 of my physical
> hosts back to OpenSuse 15.1 with Xen 4.12.1 with sched=credit .  All
> guests are now running perfectly, without any issues at all.  Over
> this past week I performed directed stress-testing against several of
> my guests, and they all survived without any problems at all.  I've
> now completely my migration to the new guests, and everyone is happy.
>
> I'm now going to bring one of the previously-live guests on its own
> host back to credit2 so I can crash it and try to capture debugging
> output for xen-devel as requested.  But sched=credit is definitely
> what we needed to solve this problem!  Thank you all for helping us
> get there!
>
> Glen
>

Thank you too for your report. Hope we'll find the reason why credit2
misbehaves.

Tomas

[-- Attachment #1.2: Type: text/html, Size: 2076 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-users] xen domU stall on 4.12.1

[Tomas Mozes]

[-- Attachment #1: Type: text/plain, Size: 1561 bytes --]

On Mon, Feb 24, 2020 at 6:02 PM Tomas Mozes <hydrapolic@gmail.com> wrote:

>
>
> On Mon, Feb 24, 2020 at 4:55 PM Glen <glenbarney@gmail.com> wrote:
>
>> On Sun, Feb 23, 2020 at 11:12 PM Tomas Mozes <hydrapolic@gmail.com>
>> wrote:
>> > As reported in
>> https://lists.xenproject.org/archives/html/xen-devel/2020-01/msg00361.html
>> and
>> https://lists.xenproject.org/archives/html/xen-users/2020-02/msg00042.html,
>> switching back to credit1 scheduler seems to make it working again. I've
>> migrated 6 machines to Xen 4.12 with sched=credit xen option and haven't
>> observed a hang for more than a week now.
>>
>> My experience is the same.  I have migrated all 16 of my physical
>> hosts back to OpenSuse 15.1 with Xen 4.12.1 with sched=credit .  All
>> guests are now running perfectly, without any issues at all.  Over
>> this past week I performed directed stress-testing against several of
>> my guests, and they all survived without any problems at all.  I've
>> now completely my migration to the new guests, and everyone is happy.
>>
>> I'm now going to bring one of the previously-live guests on its own
>> host back to credit2 so I can crash it and try to capture debugging
>> output for xen-devel as requested.  But sched=credit is definitely
>> what we needed to solve this problem!  Thank you all for helping us
>> get there!
>>
>> Glen
>>
>
> Thank you too for your report. Hope we'll find the reason why credit2
> misbehaves.
>
> Tomas
>

Just tested Xen 4.12.3, but a domU hanged again with credit2. It works rock
solid with credit1.

Tomas

[-- Attachment #2: Type: text/html, Size: 2635 bytes --]

Re: [Xen-users] xen domU stall on 4.12.1

[Glen]

Tomas -

On Tue, Jun 2, 2020 at 7:43 PM Tomas Mozes <hydrapolic@gmail.com> wrote:
>> On Mon, Feb 24, 2020 at 4:55 PM Glen <glenbarney@gmail.com> wrote:
>>> I'm now going to bring one of the previously-live guests on its own
>>> host back to credit2 so I can crash it and try to capture debugging
>>> output for xen-devel as requested.  But sched=credit is definitely
>>> what we needed to solve this problem!  Thank you all for helping us
>>> get there!
> Just tested Xen 4.12.3, but a domU hanged again with credit2. It works rock solid with credit1.

I have several hosts back on credit2, no problems so far.  But the
bulk of my production hosts are still on credit1, and they do seem to
run "better" (subjectively, looking at responsiveness and load
averages) but of course by subjectively I mean that I have no real
data to back this feeling.

I was hoping one of my domU's on credit2 would crash so I could grab
debugging info for the development team - I hope you are/were able to
grab and submit data on that crash???

Glen

Re: [Xen-users] xen domU stall on 4.12.1

[Tomas Mozes]

[-- Attachment #1: Type: text/plain, Size: 1261 bytes --]

On Wed, Jun 3, 2020 at 5:30 PM Glen <glenbarney@gmail.com> wrote:

> Tomas -
>
> On Tue, Jun 2, 2020 at 7:43 PM Tomas Mozes <hydrapolic@gmail.com> wrote:
> >> On Mon, Feb 24, 2020 at 4:55 PM Glen <glenbarney@gmail.com> wrote:
> >>> I'm now going to bring one of the previously-live guests on its own
> >>> host back to credit2 so I can crash it and try to capture debugging
> >>> output for xen-devel as requested.  But sched=credit is definitely
> >>> what we needed to solve this problem!  Thank you all for helping us
> >>> get there!
> > Just tested Xen 4.12.3, but a domU hanged again with credit2. It works
> rock solid with credit1.
>
> I have several hosts back on credit2, no problems so far.  But the
> bulk of my production hosts are still on credit1, and they do seem to
> run "better" (subjectively, looking at responsiveness and load
> averages) but of course by subjectively I mean that I have no real
> data to back this feeling.
>
> I was hoping one of my domU's on credit2 would crash so I could grab
> debugging info for the development team - I hope you are/were able to
> grab and submit data on that crash???
>
> Glen
>

Unfortunately no, it was one of my production hosts so I wanted to get it
back working as quickly as possible.

Tomas

[-- Attachment #2: Type: text/html, Size: 1898 bytes --]