xenbus and the message of doom

xenbus and the message of doom

[Stefan Bader]

I was investigating a bug report[1] about newer kernels (>3.1) not booting as
HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
it lead me at least close and with some crash dump data I think I figured the
problem.

commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05
Author: Olaf Hering <olaf@aepfle.de>
Date:   Thu Sep 22 16:14:49 2011 +0200

    xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old
    kernel

This change introduced a xs_reset_watches() call. The problem seems to be that
there is at least some version of Xen (I was able to reproduce with a 3.4.3
version which I admit to deliberately not having updated) for which xenstore
will not return any reply.

At least the backtraces in crash showed that xs_init had been calling
xs_reset_watches() and that was happily idling in read_reply(). Effectively
nothing was going on and the boot just hung.
By just not doing that xs_reset_watches() call, I was able to boot under the
same host. And for what it is worth there has not been an issue with Xen 4.1.1
and a 3.0 dom0 kernel. Just this "older" release is trouble.

Now the big question is, should this never happen and the host needs urgent
updating. Or, should xs_talkv() set up a time limit and assume failure when not
receiving a message after that? I could imagine the latter might lead at least
to a more helpful "there is something wrong here, dude" than just hanging around
without any response. ;)

-Stefan

Re: xenbus and the message of doom

[Konrad Rzeszutek Wilk]

On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote:
> I was investigating a bug report[1] about newer kernels (>3.1) not booting as
> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
> it lead me at least close and with some crash dump data I think I figured the
> problem.

Stefan, thanks for finding this.

Olaf, what are your thoughts? Should I prep a patch to revert the patch
below and then we can work on 3.3 and rethink this in 3.3? The clock is
ticking for 3.2 and there is not much runway to fix stuff.

> 
> commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05
> Author: Olaf Hering <olaf@aepfle.de>
> Date:   Thu Sep 22 16:14:49 2011 +0200
> 
>     xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old
>     kernel
> 
> This change introduced a xs_reset_watches() call. The problem seems to be that
> there is at least some version of Xen (I was able to reproduce with a 3.4.3
> version which I admit to deliberately not having updated) for which xenstore
> will not return any reply.

And oxenstore too, but Ian prepped a patch for this. Perhaps that is
what Amazon is running.
> 
> At least the backtraces in crash showed that xs_init had been calling
> xs_reset_watches() and that was happily idling in read_reply(). Effectively
> nothing was going on and the boot just hung.

So at least we should have a timeout read_reply. But I don't see
anything in the code that we could immediately use.

> By just not doing that xs_reset_watches() call, I was able to boot under the
> same host. And for what it is worth there has not been an issue with Xen 4.1.1
> and a 3.0 dom0 kernel. Just this "older" release is trouble.
> 
> Now the big question is, should this never happen and the host needs urgent
> updating. Or, should xs_talkv() set up a time limit and assume failure when not
> receiving a message after that? I could imagine the latter might lead at least
> to a more helpful "there is something wrong here, dude" than just hanging around
> without any response. ;)
> 
> -Stefan
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: xenbus and the message of doom

[Stefan Bader]

On 15.12.2011 20:39, Konrad Rzeszutek Wilk wrote:
> On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote:
>> I was investigating a bug report[1] about newer kernels (>3.1) not booting as
>> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
>> it lead me at least close and with some crash dump data I think I figured the
>> problem.
> 
> Stefan, thanks for finding this.
> 

I realize I wanted to add the reference to our bug report but completely forgot
to do so. So just for completeness:

http://bugs.launchpad.net/bugs/901305


> Olaf, what are your thoughts? Should I prep a patch to revert the patch
> below and then we can work on 3.3 and rethink this in 3.3? The clock is
> ticking for 3.2 and there is not much runway to fix stuff.
> 
>>
>> commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05
>> Author: Olaf Hering <olaf@aepfle.de>
>> Date:   Thu Sep 22 16:14:49 2011 +0200
>>
>>     xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old
>>     kernel
>>
>> This change introduced a xs_reset_watches() call. The problem seems to be that
>> there is at least some version of Xen (I was able to reproduce with a 3.4.3
>> version which I admit to deliberately not having updated) for which xenstore
>> will not return any reply.
> 
> And oxenstore too, but Ian prepped a patch for this. Perhaps that is
> what Amazon is running.
>>
>> At least the backtraces in crash showed that xs_init had been calling
>> xs_reset_watches() and that was happily idling in read_reply(). Effectively
>> nothing was going on and the boot just hung.
> 
> So at least we should have a timeout read_reply. But I don't see
> anything in the code that we could immediately use.
> 
>> By just not doing that xs_reset_watches() call, I was able to boot under the
>> same host. And for what it is worth there has not been an issue with Xen 4.1.1
>> and a 3.0 dom0 kernel. Just this "older" release is trouble.
>>
>> Now the big question is, should this never happen and the host needs urgent
>> updating. Or, should xs_talkv() set up a time limit and assume failure when not
>> receiving a message after that? I could imagine the latter might lead at least
>> to a more helpful "there is something wrong here, dude" than just hanging around
>> without any response. ;)
>>
>> -Stefan
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel

Re: xenbus and the message of doom

[Ian Campbell]

On Thu, 2011-12-15 at 19:20 +0000, Stefan Bader wrote:
> I was investigating a bug report[1] about newer kernels (>3.1) not booting as
> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
> it lead me at least close and with some crash dump data I think I figured the
> problem.
> 
> commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05
> Author: Olaf Hering <olaf@aepfle.de>
> Date:   Thu Sep 22 16:14:49 2011 +0200
> 
>     xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old
>     kernel
> 
> This change introduced a xs_reset_watches() call. The problem seems to be that
> there is at least some version of Xen (I was able to reproduce with a 3.4.3
> version which I admit to deliberately not having updated) for which xenstore
> will not return any reply.
> 
> At least the backtraces in crash showed that xs_init had been calling
> xs_reset_watches() and that was happily idling in read_reply(). Effectively
> nothing was going on and the boot just hung.
> By just not doing that xs_reset_watches() call, I was able to boot under the
> same host. And for what it is worth there has not been an issue with Xen 4.1.1
> and a 3.0 dom0 kernel. Just this "older" release is trouble.

I sent a patch to fix exactly this issue in oxenstored (the ocaml
xenstore) just this week. Is there any chance that you are running C
xenstored with Xen 4.1.1 and oxenstored with Xen 3.4.3?

> Now the big question is, should this never happen and the host needs urgent
> updating. Or, should xs_talkv() set up a time limit and assume failure when not
> receiving a message after that? I could imagine the latter might lead at least
> to a more helpful "there is something wrong here, dude" than just hanging around
> without any response. ;)
> 
> -Stefan
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: xenbus and the message of doom

[Stefan Bader]

On 15.12.2011 21:53, Ian Campbell wrote:
> On Thu, 2011-12-15 at 19:20 +0000, Stefan Bader wrote:
>> I was investigating a bug report[1] about newer kernels (>3.1) not booting as
>> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
>> it lead me at least close and with some crash dump data I think I figured the
>> problem.
>>
>> commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05
>> Author: Olaf Hering <olaf@aepfle.de>
>> Date:   Thu Sep 22 16:14:49 2011 +0200
>>
>>     xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old
>>     kernel
>>
>> This change introduced a xs_reset_watches() call. The problem seems to be that
>> there is at least some version of Xen (I was able to reproduce with a 3.4.3
>> version which I admit to deliberately not having updated) for which xenstore
>> will not return any reply.
>>
>> At least the backtraces in crash showed that xs_init had been calling
>> xs_reset_watches() and that was happily idling in read_reply(). Effectively
>> nothing was going on and the boot just hung.
>> By just not doing that xs_reset_watches() call, I was able to boot under the
>> same host. And for what it is worth there has not been an issue with Xen 4.1.1
>> and a 3.0 dom0 kernel. Just this "older" release is trouble.
> 
> I sent a patch to fix exactly this issue in oxenstored (the ocaml
> xenstore) just this week. Is there any chance that you are running C
> xenstored with Xen 4.1.1 and oxenstored with Xen 3.4.3?

Thanks for the pointer, I missed that thread. Now dumb question, would
oxenstored be named that way? Or iow, how do I quickly find out what is running?
The binary running in 3.4.3 is xenstored which is a linked executable (same in
4.1.1).

But I guess, whatever version is running, any oxenstored would not have the
bugfix because things take longer to reach any packaged versions. I rather would
suspect that in 4.1.1, the reset watches message probably is just known and thus
avoiding the problem. Unfortunately it is near impossible to tell for sure what
exactly EC2 is running.

The major point here probably is that when the upstream kernels are calling that
message and there are versions of xenstored in production that will just ignore
it while the kernel blocks waiting, this is a painful path. Production systems
tend to update slowly and the symptoms are not that obvious. Having a timeout
maybe could be useful not only for this case, but clearly it is nothing that
should be rushed.

So reverting the patch introducing that call (at least in the distro kernel) may
be the best thing to do (knowing that this will be bought by loosing the fix for
kexec boots fo crash kernels).

-Stefan

>> Now the big question is, should this never happen and the host needs urgent
>> updating. Or, should xs_talkv() set up a time limit and assume failure when not
>> receiving a message after that? I could imagine the latter might lead at least
>> to a more helpful "there is something wrong here, dude" than just hanging around
>> without any response. ;)
>>
>> -Stefan
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: xenbus and the message of doom

[Ian Campbell]

On Fri, 2011-12-16 at 09:18 +0000, Stefan Bader wrote:
> On 15.12.2011 21:53, Ian Campbell wrote:
> > On Thu, 2011-12-15 at 19:20 +0000, Stefan Bader wrote:
> >> I was investigating a bug report[1] about newer kernels (>3.1) not booting as
> >> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
> >> it lead me at least close and with some crash dump data I think I figured the
> >> problem.
> >>
> >> commit ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05
> >> Author: Olaf Hering <olaf@aepfle.de>
> >> Date:   Thu Sep 22 16:14:49 2011 +0200
> >>
> >>     xen/pv-on-hvm kexec: add xs_reset_watches to shutdown watches from old
> >>     kernel
> >>
> >> This change introduced a xs_reset_watches() call. The problem seems to be that
> >> there is at least some version of Xen (I was able to reproduce with a 3.4.3
> >> version which I admit to deliberately not having updated) for which xenstore
> >> will not return any reply.
> >>
> >> At least the backtraces in crash showed that xs_init had been calling
> >> xs_reset_watches() and that was happily idling in read_reply(). Effectively
> >> nothing was going on and the boot just hung.
> >> By just not doing that xs_reset_watches() call, I was able to boot under the
> >> same host. And for what it is worth there has not been an issue with Xen 4.1.1
> >> and a 3.0 dom0 kernel. Just this "older" release is trouble.
> > 
> > I sent a patch to fix exactly this issue in oxenstored (the ocaml
> > xenstore) just this week. Is there any chance that you are running C
> > xenstored with Xen 4.1.1 and oxenstored with Xen 3.4.3?
> 
> Thanks for the pointer, I missed that thread. Now dumb question, would
> oxenstored be named that way? Or iow, how do I quickly find out what is running?

The process name will be oxenstored instead of xenstored.

> The binary running in 3.4.3 is xenstored which is a linked executable (same in
> 4.1.1).

The sounds like you are running the C xenstored in both cases so this is
a red-herring.

> But I guess, whatever version is running, any oxenstored would not have the
> bugfix because things take longer to reach any packaged versions.

Correct.

> I rather would suspect that in 4.1.1, the reset watches message probably is just
> known and thus avoiding the problem. Unfortunately it is near impossible to tell
> for sure what exactly EC2 is running.
> 
> The major point here probably is that when the upstream kernels are calling that
> message and there are versions of xenstored in production that will just ignore
> it while the kernel blocks waiting, this is a painful path. Production systems
> tend to update slowly and the symptoms are not that obvious.

Yes. It is unfortunate that xenstored in the field has this bug but it
does mean that the approach taken here with this new message cannot
work.

FWIW there weren't all that many changes to C xenstored between 3.4.x
and current unstable so it wouldn't be hard to identify where the fix
is, but that doesn't really help people stuck with an older xenstored.

> Having a timeout
> maybe could be useful not only for this case, but clearly it is nothing that
> should be rushed.

A timeout may not help, it depends what the daemon does after the
invalid message.

It looks as if oxenstored just throws it away and will process the next
message fine so that is OK but I didn't check the C version.

I think (or hope!) that Olaf tested with the most recent version of C
xenstored which did not support this new message and so I presume that
it correct returns an error for an unknown message but that doesn't help
us with the older C xenstored which you have.

In any case someone really needs to check both the ocaml and the
version's behaviour.

> So reverting the patch introducing that call (at least in the distro kernel) may
> be the best thing to do (knowing that this will be bought by loosing the fix for
> kexec boots fo crash kernels).

Agreed. We should revert the kernel change for now and revisit it.

One potential solution, depending on the actual behaviour of the
daemons, would be to follow the potentially unsupported command with an
innocuous well established one and use the ID field to identify which we
get a response to.

Does the target kernel know that it has been kexec'd? Perhaps we should
only reset xenstore watches if we are booting after a kexec. Worst case
the kexec tool can add a command line argument to trigger this. Doing it
this way means there is no possibility of regressions for normal boot
and kexec wasn't supported on older xenstored anyway.

Ian.

> 
> -Stefan
> 
> >> Now the big question is, should this never happen and the host needs urgent
> >> updating. Or, should xs_talkv() set up a time limit and assume failure when not
> >> receiving a message after that? I could imagine the latter might lead at least
> >> to a more helpful "there is something wrong here, dude" than just hanging around
> >> without any response. ;)
> >>
> >> -Stefan
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> > 
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> 

Re: xenbus and the message of doom

[Olaf Hering]

On Thu, Dec 15, Konrad Rzeszutek Wilk wrote:

> On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote:
> > I was investigating a bug report[1] about newer kernels (>3.1) not booting as
> > HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
> > it lead me at least close and with some crash dump data I think I figured the
> > problem.
> 
> Stefan, thanks for finding this.
> 
> Olaf, what are your thoughts? Should I prep a patch to revert the patch
> below and then we can work on 3.3 and rethink this in 3.3? The clock is
> ticking for 3.2 and there is not much runway to fix stuff.

Sometimes guest changes expose bugs in the host. Its my understanding
that hosts should be kept uptodate so that it can serve both old and new
guests well.

In my testing with Xen4 based hosts their xenstored did properly ignore
the new command.

I proposed several ways to get rid of existing watches, but finally we
came to the conclusion that a new xenstored command would be the
cleanest way.

Wether adding a timeout is a good idea has to be decided. I can imagine
that a busy host may take some time to respond to guest commands.


Perhaps we should figure out what exactly EC2 is using as host and why
it only breaks with upstream kernels. So far I havent received reports
for SLES11 guests. SP1 got an update recently, so their HVM guests would
have seen the hang as well. The not yet released SP2 sends
XS_RESET_WATCHES as well since quite some time.


Olaf

Re: xenbus and the message of doom

[Konrad Rzeszutek Wilk]

On Fri, Dec 16, 2011 at 12:33:00PM +0100, Olaf Hering wrote:
> On Thu, Dec 15, Konrad Rzeszutek Wilk wrote:
> 
> > On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote:
> > > I was investigating a bug report[1] about newer kernels (>3.1) not booting as
> > > HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
> > > it lead me at least close and with some crash dump data I think I figured the
> > > problem.
> > 
> > Stefan, thanks for finding this.
> > 
> > Olaf, what are your thoughts? Should I prep a patch to revert the patch
> > below and then we can work on 3.3 and rethink this in 3.3? The clock is
> > ticking for 3.2 and there is not much runway to fix stuff.
> 
> Sometimes guest changes expose bugs in the host. Its my understanding
> that hosts should be kept uptodate so that it can serve both old and new
> guests well.
> 
> In my testing with Xen4 based hosts their xenstored did properly ignore
> the new command.
> 
> I proposed several ways to get rid of existing watches, but finally we
> came to the conclusion that a new xenstored command would be the
> cleanest way.
> 
> Wether adding a timeout is a good idea has to be decided. I can imagine
> that a busy host may take some time to respond to guest commands.
> 
> 
> Perhaps we should figure out what exactly EC2 is using as host and why
> it only breaks with upstream kernels. So far I havent received reports

Good point. Stefan were you able to provide to Scott a kernel without the
git commit mentioned to see if that fixed the issue?

CC-ing here Vincent in hopes of getting some hints..

> for SLES11 guests. SP1 got an update recently, so their HVM guests would
> have seen the hang as well. The not yet released SP2 sends
> XS_RESET_WATCHES as well since quite some time.
> 
> 
> Olaf

Re: xenbus and the message of doom

[Olaf Hering]

On Fri, Dec 16, Ian Campbell wrote:

> I think (or hope!) that Olaf tested with the most recent version of C
> xenstored which did not support this new message and so I presume that
> it correct returns an error for an unknown message but that doesn't help
> us with the older C xenstored which you have.

I just checked a SLES10SP3 host (xen 3.2.3) with a SLES11SP2 hvm guest
and it booted fine.

So I suspect EC2 is broken in this case.

Olaf

Re: xenbus and the message of doom

[Alessandro Salvatori]

by backing out  ddacf5ef684a655abe2bb50c4b2a5b72ae0d5e05, 3.2.0-rc5
boots just fine.

-Alessandro-
 Here i am, A young man,
 A crashing computer program,
 Here is a pen, write out my name...

(from: The Servant - Orchestra)



On Fri, Dec 16, 2011 at 09:01, Olaf Hering <olaf@aepfle.de> wrote:
> On Fri, Dec 16, Ian Campbell wrote:
>
>> I think (or hope!) that Olaf tested with the most recent version of C
>> xenstored which did not support this new message and so I presume that
>> it correct returns an error for an unknown message but that doesn't help
>> us with the older C xenstored which you have.
>
> I just checked a SLES10SP3 host (xen 3.2.3) with a SLES11SP2 hvm guest
> and it booted fine.
>
> So I suspect EC2 is broken in this case.
>
> Olaf
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: xenbus and the message of doom

[Ian Campbell]

On Fri, 2011-12-16 at 11:33 +0000, Olaf Hering wrote:
> On Thu, Dec 15, Konrad Rzeszutek Wilk wrote:
> 
> > On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote:
> > > I was investigating a bug report[1] about newer kernels (>3.1) not booting as
> > > HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
> > > it lead me at least close and with some crash dump data I think I figured the
> > > problem.
> > 
> > Stefan, thanks for finding this.
> > 
> > Olaf, what are your thoughts? Should I prep a patch to revert the patch
> > below and then we can work on 3.3 and rethink this in 3.3? The clock is
> > ticking for 3.2 and there is not much runway to fix stuff.
> 
> Sometimes guest changes expose bugs in the host. Its my understanding
> that hosts should be kept uptodate so that it can serve both old and new
> guests well.

In an ideal world yes but we need to balance this against breaking stuff
which is still widely used. It seems like in this case we may have
gotten the balance wrong because people are reporting bugs.

What's wrong with only doing this reset if we know we are kexec'd? If
that can't be automatically detected then e.g. using an explicit
reset_watches command line option. You could even make a tenuous
argument for hanging this off reset_devices?

[...]
> Perhaps we should figure out what exactly EC2 is using as host and why
> it only breaks with upstream kernels.

and in the meantime we leave upstream (and any distros which picks up a
new enough kernel) on EC2? I think at this stage in the rc cycle we'd be
better off reverting and trying again for 3.3.

>  So far I havent received reports
> for SLES11 guests. SP1 got an update recently, so their HVM guests would
> have seen the hang as well. The not yet released SP2 sends
> XS_RESET_WATCHES as well since quite some time.

This thread contains 3 reports of actual regressions. Stefan explicitly
reported an EC2 bug and a reproduction on 3.4.3 and Alessandro by
implication has reported breakage in his environment.

Ian.


> 
> 
> Olaf
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: xenbus and the message of doom

[Olaf Hering]

On Tue, Dec 20, Ian Campbell wrote:

> What's wrong with only doing this reset if we know we are kexec'd? If
> that can't be automatically detected then e.g. using an explicit
> reset_watches command line option. You could even make a tenuous
> argument for hanging this off reset_devices?

The kexec kernel does not know that it was loaded via kexec.
We could make the reset_devices option mandatory for kexec in PVonHVM
guests, so the change to drivers/xen/xenbus/xenbus_xs.c would be very
small, like "if (hvm && reset_devices) xs_reset_watches();"

> > Perhaps we should figure out what exactly EC2 is using as host and why
> > it only breaks with upstream kernels.
> 
> and in the meantime we leave upstream (and any distros which picks up a
> new enough kernel) on EC2? I think at this stage in the rc cycle we'd be
> better off reverting and trying again for 3.3.

If EC2 is unable to fix it in time (or provide info what exactly they
use), I'm ok with reverting/disabling the call to xs_reset_watches().
I can continue to work on this next year.

Olaf

Re: xenbus and the message of doom

[Konrad Rzeszutek Wilk]

On Tue, Dec 20, 2011 at 02:15:33PM +0100, Olaf Hering wrote:
> On Tue, Dec 20, Ian Campbell wrote:
> 
> > What's wrong with only doing this reset if we know we are kexec'd? If
> > that can't be automatically detected then e.g. using an explicit
> > reset_watches command line option. You could even make a tenuous
> > argument for hanging this off reset_devices?
> 
> The kexec kernel does not know that it was loaded via kexec.
> We could make the reset_devices option mandatory for kexec in PVonHVM
> guests, so the change to drivers/xen/xenbus/xenbus_xs.c would be very
> small, like "if (hvm && reset_devices) xs_reset_watches();"

<nods> OK that would be one way. Granted if one tried to kexec under
Amazon EC2 an PVonHVM domain we would hit this bug again. But then
I don't think kexecing without this patch works, so that scenario is
probably moot.

> 
> > > Perhaps we should figure out what exactly EC2 is using as host and why
> > > it only breaks with upstream kernels.
> > 
> > and in the meantime we leave upstream (and any distros which picks up a
> > new enough kernel) on EC2? I think at this stage in the rc cycle we'd be
> > better off reverting and trying again for 3.3.
> 
> If EC2 is unable to fix it in time (or provide info what exactly they
> use), I'm ok with reverting/disabling the call to xs_reset_watches().

By my reckoning the 3.2 is going to come out Dec 29th (60 days after 3.1
was released) or it might slip. With folks buying presents online (and
potentially using Amazon) they [Amazon] is not going to fix anything -
they are in "must work now to sell stuff mode" - which means fix only
those $1M bugs. With the craze of purchases stopping around January I
think they could start addressing this sometime in Janurary - which would
be past the 3.2 release date.

Sorry Olaf, have to revert that commit.

> I can continue to work on this next year.

Ok. I need to serioulsy get a free Amazon EC2 instance to run nightly tests.
This is the third breakage this year.

Re: xenbus and the message of doom

[Ian Jackson]

Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] xenbus and the message of doom"):
> Sorry Olaf, have to revert that commit.

I agree.  When we introduced it we weren't aware that most existing
implementations of xenstored simply ignore unknown commands rather
than replying with an error.  If we had known this we would not have
approved Olaf's patch.

That they ignore unknown commands is of course a bug but expecting
everyone to update is no good.  Really the best approach would be some
kind of discovery mechanism.

Maybe we should have a special path @xenstore/fail_unknown_commands
which you could read, or something.  But this time we should try it
against old implementations.

Ian.

Re: xenbus and the message of doom

[Ian Campbell]

On Tue, 2011-12-20 at 17:29 +0000, Ian Jackson wrote:
> Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] xenbus and the message of doom"):
> > Sorry Olaf, have to revert that commit.
> 
> I agree.  When we introduced it we weren't aware that most existing
> implementations of xenstored simply ignore unknown commands rather
> than replying with an error.  If we had known this we would not have
> approved Olaf's patch.
> 
> That they ignore unknown commands is of course a bug but expecting
> everyone to update is no good.  Really the best approach would be some
> kind of discovery mechanism.
> 
> Maybe we should have a special path @xenstore/fail_unknown_commands
> which you could read, or something.  But this time we should try it
> against old implementations.

I was sure I'd seen some precedent (and therefore an existing path) for
this sort of thing at some point but I can't for the life of me find it.

The closest I could find is
the /local/domain/<N>/control/platform-feature-multiprocessor-suspend
node which we write statically for every domain. I suppose putting it
under /local/domain/<N> is consistent with restricting the domain to
mostly it's own home area.

Ian.

Re: xenbus and the message of doom

[Stefan Bader]

On 16.12.2011 12:33, Olaf Hering wrote:
> On Thu, Dec 15, Konrad Rzeszutek Wilk wrote:
> 
>> On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote:
>>> I was investigating a bug report[1] about newer kernels (>3.1) not booting as
>>> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
>>> it lead me at least close and with some crash dump data I think I figured the
>>> problem.
>>
>> Stefan, thanks for finding this.
>>
>> Olaf, what are your thoughts? Should I prep a patch to revert the patch
>> below and then we can work on 3.3 and rethink this in 3.3? The clock is
>> ticking for 3.2 and there is not much runway to fix stuff.
> 
> Sometimes guest changes expose bugs in the host. Its my understanding
> that hosts should be kept uptodate so that it can serve both old and new
> guests well.

That would be the ideal world. Unfortunately, in reality, hosts stick to a
particular version and maybe get updated with what is provided as stable or
security updates.

> In my testing with Xen4 based hosts their xenstored did properly ignore
> the new command.

I can only take what evidence I see publicly on running EC2 instances. The
closest we can get there is something Redhat or CentOS based with a variation of
3.x based hypervisors (my test box is keeping CentOS 5.6 and Xen 3.4.3 to verify
bugs). Checking against our Xen host reintroduced with 11.10, there is no
problem either.

> I proposed several ways to get rid of existing watches, but finally we
> came to the conclusion that a new xenstored command would be the
> cleanest way.
> 
> Wether adding a timeout is a good idea has to be decided. I can imagine
> that a busy host may take some time to respond to guest commands.
> 
> 
> Perhaps we should figure out what exactly EC2 is using as host and why
> it only breaks with upstream kernels. So far I havent received reports
> for SLES11 guests. SP1 got an update recently, so their HVM guests would
> have seen the hang as well. The not yet released SP2 sends
> XS_RESET_WATCHES as well since quite some time.

As said before, it is hard to say exactly. We can even see variations of Xen 3.x
(I can at least remember having seen three versions ranging from 3.0 to 3.4.3).
So to a certain degree there are updates going on. But it is impossible to say
what and when. But definitely the problem was still there with 3.4.3 (I stopped
updating the test box because I was rather glad to have a state that exhibits
most of the weirdness that happens in the cloud).

-Stefan
> 
> 
> Olaf

Re: xenbus and the message of doom

[Stefan Bader]

On 16.12.2011 16:25, Konrad Rzeszutek Wilk wrote:
> On Fri, Dec 16, 2011 at 12:33:00PM +0100, Olaf Hering wrote:
>> On Thu, Dec 15, Konrad Rzeszutek Wilk wrote:
>>
>>> On Thu, Dec 15, 2011 at 08:20:23PM +0100, Stefan Bader wrote:
>>>> I was investigating a bug report[1] about newer kernels (>3.1) not booting as
>>>> HVM guests on Amazon EC2. For some reason git bisect did give the some pain, but
>>>> it lead me at least close and with some crash dump data I think I figured the
>>>> problem.
>>>
>>> Stefan, thanks for finding this.
>>>
>>> Olaf, what are your thoughts? Should I prep a patch to revert the patch
>>> below and then we can work on 3.3 and rethink this in 3.3? The clock is
>>> ticking for 3.2 and there is not much runway to fix stuff.
>>
>> Sometimes guest changes expose bugs in the host. Its my understanding
>> that hosts should be kept uptodate so that it can serve both old and new
>> guests well.
>>
>> In my testing with Xen4 based hosts their xenstored did properly ignore
>> the new command.
>>
>> I proposed several ways to get rid of existing watches, but finally we
>> came to the conclusion that a new xenstored command would be the
>> cleanest way.
>>
>> Wether adding a timeout is a good idea has to be decided. I can imagine
>> that a busy host may take some time to respond to guest commands.
>>
>>
>> Perhaps we should figure out what exactly EC2 is using as host and why
>> it only breaks with upstream kernels. So far I havent received reports
> 
> Good point. Stefan were you able to provide to Scott a kernel without the
> git commit mentioned to see if that fixed the issue?

Sorry have been off over the end of year and I try to be seriously off when I am
off. ;)
Am working my way through email now and maybe this is already obsolete as I see
some submissions which I have not read, yet. But doing that was on my list.

-Stefan

> CC-ing here Vincent in hopes of getting some hints..
> 
>> for SLES11 guests. SP1 got an update recently, so their HVM guests would
>> have seen the hang as well. The not yet released SP2 sends
>> XS_RESET_WATCHES as well since quite some time.
>>
>>
>> Olaf

Re: xenbus and the message of doom

[Olaf Hering]

On Tue, Dec 20, Ian Campbell wrote:

> On Tue, 2011-12-20 at 17:29 +0000, Ian Jackson wrote:
> > Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] xenbus and the message of doom"):
> > > Sorry Olaf, have to revert that commit.
> > 
> > I agree.  When we introduced it we weren't aware that most existing
> > implementations of xenstored simply ignore unknown commands rather
> > than replying with an error.  If we had known this we would not have
> > approved Olaf's patch.
> > 
> > That they ignore unknown commands is of course a bug but expecting
> > everyone to update is no good.  Really the best approach would be some
> > kind of discovery mechanism.
> > 
> > Maybe we should have a special path @xenstore/fail_unknown_commands
> > which you could read, or something.  But this time we should try it
> > against old implementations.
> 
> I was sure I'd seen some precedent (and therefore an existing path) for
> this sort of thing at some point but I can't for the life of me find it.


If you could come up with a patch to present xenstored (or other
dom0/toolstack) features, that would be great.

Olaf

Re: xenbus and the message of doom

[Ian Campbell]

On Mon, 2012-01-02 at 17:16 +0000, Olaf Hering wrote:
> On Tue, Dec 20, Ian Campbell wrote:
> 
> > On Tue, 2011-12-20 at 17:29 +0000, Ian Jackson wrote:
> > > Konrad Rzeszutek Wilk writes ("Re: [Xen-devel] xenbus and the message of doom"):
> > > > Sorry Olaf, have to revert that commit.
> > > 
> > > I agree.  When we introduced it we weren't aware that most existing
> > > implementations of xenstored simply ignore unknown commands rather
> > > than replying with an error.  If we had known this we would not have
> > > approved Olaf's patch.
> > > 
> > > That they ignore unknown commands is of course a bug but expecting
> > > everyone to update is no good.  Really the best approach would be some
> > > kind of discovery mechanism.
> > > 
> > > Maybe we should have a special path @xenstore/fail_unknown_commands
> > > which you could read, or something.  But this time we should try it
> > > against old implementations.
> > 
> > I was sure I'd seen some precedent (and therefore an existing path) for
> > this sort of thing at some point but I can't for the life of me find it.
> 
> 
> If you could come up with a patch to present xenstored (or other
> dom0/toolstack) features, that would be great.

As I said in the bit you trimmed the precedent seems to be to
write /local/domain/<N>/control/platform-feature-<X> for each guest that
you want to expose the feature to. I think a per-guest key makes sense
since even if something seems obviously global you never know when you
might need to hide it from a guest (e.g. to workaround an in-guest bug).

I'm afraid I don't have time to work on this myself but it seems pretty
simple?

Ian.

Re: xenbus and the message of doom

[Olaf Hering]

On Tue, Jan 03, Ian Campbell wrote:

> As I said in the bit you trimmed the precedent seems to be to
> write /local/domain/<N>/control/platform-feature-<X> for each guest that
> you want to expose the feature to. I think a per-guest key makes sense
> since even if something seems obviously global you never know when you
> might need to hide it from a guest (e.g. to workaround an in-guest bug).
> 
> I'm afraid I don't have time to work on this myself but it seems pretty
> simple?

After looking at this, the (compile tested) patch below should be enough:


diff -r 3a22ed3ec534 tools/xenstore/xs.c
--- a/tools/xenstore/xs.c
+++ b/tools/xenstore/xs.c
@@ -884,6 +884,10 @@ bool xs_introduce_domain(struct xs_handl
 	char mfn_str[MAX_STRLEN(mfn)];
 	char eventchn_str[MAX_STRLEN(eventchn)];
 	struct iovec iov[3];
+	static const char feat[] = "control/platform-feature-xs_reset_watches";
+	int len;
+	char *dom_path, *feat_path;
+	bool ret;
 
 	snprintf(domid_str, sizeof(domid_str), "%u", domid);
 	snprintf(mfn_str, sizeof(mfn_str), "%lu", mfn);
@@ -896,8 +900,21 @@ bool xs_introduce_domain(struct xs_handl
 	iov[2].iov_base = eventchn_str;
 	iov[2].iov_len = strlen(eventchn_str) + 1;
 
-	return xs_bool(xs_talkv(h, XBT_NULL, XS_INTRODUCE, iov,
+	ret = xs_bool(xs_talkv(h, XBT_NULL, XS_INTRODUCE, iov,
 				ARRAY_SIZE(iov), NULL));
+
+	dom_path = xs_get_domain_path(h, domid);
+	if (dom_path) {
+		len = strlen(dom_path) + 1 + strlen(feat);
+		feat_path = malloc(len + 1);
+		if (feat_path) {
+			snprintf(feat_path, len + 1, "%s/%s", dom_path, feat);
+			xs_write(h, XBT_NULL, feat_path, "1", 1);
+			free(feat_path);
+		}
+		free(dom_path);
+	}
+	return ret;
 }
 
 bool xs_set_target(struct xs_handle *h,

Re: xenbus and the message of doom

[Ian Campbell]

On Wed, 2012-01-04 at 15:57 +0000, Olaf Hering wrote:
> On Tue, Jan 03, Ian Campbell wrote:
> 
> > As I said in the bit you trimmed the precedent seems to be to
> > write /local/domain/<N>/control/platform-feature-<X> for each guest that
> > you want to expose the feature to. I think a per-guest key makes sense
> > since even if something seems obviously global you never know when you
> > might need to hide it from a guest (e.g. to workaround an in-guest bug).
> > 
> > I'm afraid I don't have time to work on this myself but it seems pretty
> > simple?
> 
> After looking at this, the (compile tested) patch below should be enough:
> 
> 
> diff -r 3a22ed3ec534 tools/xenstore/xs.c
> --- a/tools/xenstore/xs.c
> +++ b/tools/xenstore/xs.c

I think the toolstack needs to be involved with this. IOW this should be
done in libxl__domain_make() around the same place as
control/platform-feature-multiprocessor-suspend is written.

Ian.

> @@ -884,6 +884,10 @@ bool xs_introduce_domain(struct xs_handl
>  	char mfn_str[MAX_STRLEN(mfn)];
>  	char eventchn_str[MAX_STRLEN(eventchn)];
>  	struct iovec iov[3];
> +	static const char feat[] = "control/platform-feature-xs_reset_watches";
> +	int len;
> +	char *dom_path, *feat_path;
> +	bool ret;
>  
>  	snprintf(domid_str, sizeof(domid_str), "%u", domid);
>  	snprintf(mfn_str, sizeof(mfn_str), "%lu", mfn);
> @@ -896,8 +900,21 @@ bool xs_introduce_domain(struct xs_handl
>  	iov[2].iov_base = eventchn_str;
>  	iov[2].iov_len = strlen(eventchn_str) + 1;
>  
> -	return xs_bool(xs_talkv(h, XBT_NULL, XS_INTRODUCE, iov,
> +	ret = xs_bool(xs_talkv(h, XBT_NULL, XS_INTRODUCE, iov,
>  				ARRAY_SIZE(iov), NULL));
> +
> +	dom_path = xs_get_domain_path(h, domid);
> +	if (dom_path) {
> +		len = strlen(dom_path) + 1 + strlen(feat);
> +		feat_path = malloc(len + 1);
> +		if (feat_path) {
> +			snprintf(feat_path, len + 1, "%s/%s", dom_path, feat);
> +			xs_write(h, XBT_NULL, feat_path, "1", 1);
> +			free(feat_path);
> +		}
> +		free(dom_path);
> +	}
> +	return ret;
>  }
>  
>  bool xs_set_target(struct xs_handle *h,

Re: xenbus and the message of doom

[Olaf Hering]

On Wed, Jan 04, Ian Campbell wrote:

> I think the toolstack needs to be involved with this. IOW this should be
> done in libxl__domain_make() around the same place as
> control/platform-feature-multiprocessor-suspend is written.

Isnt XS_RESET_WATCHES a feature of xenstored, so it should add its own stuff itself?

Olaf

Re: xenbus and the message of doom

[Ian Campbell]

On Wed, 2012-01-04 at 16:27 +0000, Olaf Hering wrote:
> On Wed, Jan 04, Ian Campbell wrote:
> 
> > I think the toolstack needs to be involved with this. IOW this should be
> > done in libxl__domain_make() around the same place as
> > control/platform-feature-multiprocessor-suspend is written.
> 
> Isnt XS_RESET_WATCHES a feature of xenstored, so it should add its own stuff itself?

Ultimately I think it needs to be the toolstack so that it can control
if this gets written or not based on configuration should that become
necessary.

This is similar to how control/platform-feature-multiprocessor-suspend
is a feature of libxc but libxl still writes the node.

In general we tend to consider the toolstack as a matched set so in a
given release the toolstack is allowed to rely on features (and in
particular bugfixes) of xenstored in that release.

Ian.

Re: xenbus and the message of doom

[Olaf Hering]

On Thu, Jan 05, Ian Campbell wrote:

> On Wed, 2012-01-04 at 16:27 +0000, Olaf Hering wrote:
> > On Wed, Jan 04, Ian Campbell wrote:
> > 
> > > I think the toolstack needs to be involved with this. IOW this should be
> > > done in libxl__domain_make() around the same place as
> > > control/platform-feature-multiprocessor-suspend is written.
> > 
> > Isnt XS_RESET_WATCHES a feature of xenstored, so it should add its own stuff itself?
> 
> Ultimately I think it needs to be the toolstack so that it can control
> if this gets written or not based on configuration should that become
> necessary.

I have sent a patch to add control/platform-feature-xs_reset_watches
from libxl/xend. With this patch a guest can now decide wether it can
issue the XS_RESET_WATCHES command.

Olaf