Problems starting Xen domU after latest stable update

Problems starting Xen domU after latest stable update

[Michael D Labriola]

Hey, everyone.  I've run into problems starting up my Xen domUs as of
the latest batch of stable updates.  Whenever I try to create one, I
get a bunch of block device errors like this:

libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed

I'm using Xen 4.13.1 on the box I've been testing with.

I bisected down to this commit, and reverting it does indeed fix my
problem.  Well, this commit upstream and it's cherry-picked variants
on linux-5.4.y and linux-5.10.y.

commit 3499ba8198cad47b731792e5e56b9ec2a78a83a2
Author: David Woodhouse <dwmw@amazon.co.uk>
Date:   Wed Jan 13 13:26:02 2021 +0000

    xen: Fix event channel callback via INTX/GSI
    
    For a while, event channel notification via the PCI platform device
    has been broken, because we attempt to communicate with xenstore before
    we even have notifications working, with the xs_reset_watches() call
    in xs_init().
    
    We tend to get away with this on Xen versions below 4.0 because we avoid
    calling xs_reset_watches() anyway, because xenstore might not cope with
    reading a non-existent key. And newer Xen *does* have the vector
    callback support, so we rarely fall back to INTX/GSI delivery.
    
    To fix it, clean up a bit of the mess of xs_init() and xenbus_probe()
    startup. Call xs_init() directly from xenbus_init() only in the !XS_HVM
    case, deferring it to be called from xenbus_probe() in the XS_HVM case
    instead.
    
    Then fix up the invocation of xenbus_probe() to happen either from its
    device_initcall if the callback is available early enough, or when the
    callback is finally set up. This means that the hack of calling
    xenbus_probe() from a workqueue after the first interrupt, or directly
    from the PCI platform device setup, is no longer needed.
    
    Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
    Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
    Link: https://lore.kernel.org/r/20210113132606.422794-2-dwmw2@infradead.org
    Signed-off-by: Juergen Gross <jgross@suse.com>

Re: Problems starting Xen domU after latest stable update

[Boris Ostrovsky]

On 1/28/21 6:52 PM, Michael D Labriola wrote:
> Hey, everyone.  I've run into problems starting up my Xen domUs as of
> the latest batch of stable updates.  Whenever I try to create one, I
> get a bunch of block device errors like this:
>
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
> libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
> libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
> libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
> libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
> libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed
>
> I'm using Xen 4.13.1 on the box I've been testing with.
>
> I bisected down to this commit, and reverting it does indeed fix my
> problem.  Well, this commit upstream and it's cherry-picked variants
> on linux-5.4.y and linux-5.10.y.


You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit Linus tree a few hours ago.


-boris

Re: Problems starting Xen domU after latest stable update

[Michael Labriola]

On Thu, Jan 28, 2021 at 7:03 PM Boris Ostrovsky
<boris.ostrovsky@oracle.com> wrote:
>
>
> On 1/28/21 6:52 PM, Michael D Labriola wrote:
> > Hey, everyone.  I've run into problems starting up my Xen domUs as of
> > the latest batch of stable updates.  Whenever I try to create one, I
> > get a bunch of block device errors like this:
> >
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
> > libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
> > libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
> > libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
> > libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
> > libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed
> >
> > I'm using Xen 4.13.1 on the box I've been testing with.
> >
> > I bisected down to this commit, and reverting it does indeed fix my
> > problem.  Well, this commit upstream and it's cherry-picked variants
> > on linux-5.4.y and linux-5.10.y.
>
>
> You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit Linus tree a few hours ago.

Indeed!  That commit fixes my problem when cherry-picked onto master
and the 5.4 and 5.10 stable branches.

Is that commit already headed to the stable/longterm branches or do we
have to make a bit more noise?

Thanks!

-Mike

Re: Problems starting Xen domU after latest stable update

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2761 bytes --]

On Thu, Jan 28, 2021 at 07:03:00PM -0500, Boris Ostrovsky wrote:
> 
> On 1/28/21 6:52 PM, Michael D Labriola wrote:
> > Hey, everyone.  I've run into problems starting up my Xen domUs as of
> > the latest batch of stable updates.  Whenever I try to create one, I
> > get a bunch of block device errors like this:
> >
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
> > libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
> > libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
> > libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
> > libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
> > libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed
> >
> > I'm using Xen 4.13.1 on the box I've been testing with.
> >
> > I bisected down to this commit, and reverting it does indeed fix my
> > problem.  Well, this commit upstream and it's cherry-picked variants
> > on linux-5.4.y and linux-5.10.y.
> 
> 
> You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit Linus tree a few hours ago.

I can confirm this fixes the same issue for me (too?), thanks!
Shouldn't this patch have Cc: stable?


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Problems starting Xen domU after latest stable update

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 3067 bytes --]

On 29.01.21 01:51, Marek Marczykowski-Górecki wrote:
> On Thu, Jan 28, 2021 at 07:03:00PM -0500, Boris Ostrovsky wrote:
>>
>> On 1/28/21 6:52 PM, Michael D Labriola wrote:
>>> Hey, everyone.  I've run into problems starting up my Xen domUs as of
>>> the latest batch of stable updates.  Whenever I try to create one, I
>>> get a bunch of block device errors like this:
>>>
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
>>> libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
>>> libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
>>> libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
>>> libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
>>> libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed
>>>
>>> I'm using Xen 4.13.1 on the box I've been testing with.
>>>
>>> I bisected down to this commit, and reverting it does indeed fix my
>>> problem.  Well, this commit upstream and it's cherry-picked variants
>>> on linux-5.4.y and linux-5.10.y.
>>
>>
>> You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit Linus tree a few hours ago.
> 
> I can confirm this fixes the same issue for me (too?), thanks!
> Shouldn't this patch have Cc: stable?

No, I don't think so.

The issue being fixed by the patch has been introduced only in 5.11
and the fixing patch references the buggy patch via a Fixes: tag.

If the buggy patch has been put into stable this Fixes: tag should
result in the fix being put into the same stable branches as well.


Juergen


[-- Attachment #1.1.2: OpenPGP_0xB0DE9DD628BF132F.asc --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Problems starting Xen domU after latest stable update

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

On Fri, Jan 29, 2021 at 03:16:52PM +0100, Jürgen Groß wrote:
> On 29.01.21 15:13, Michael Labriola wrote:
> > On Fri, Jan 29, 2021 at 12:26 AM Jürgen Groß <jgross@suse.com> wrote:
> > > If the buggy patch has been put into stable this Fixes: tag should
> > > result in the fix being put into the same stable branches as well.
> > 
> > I've never done this before...  does this happen automatically?  Or is
> > there somebody we should ping to make sure it happens?
> 
> This happens automatically (I think).
> 
> I have seen mails for the patch been taken for 4.14, 4.19, 5.4 and 5.10.

Hmm, I can't find it in LKML archive, nor stable@ archive. And also it
isn't included in 5.10.12 released yesterday, nor included in
queue/5.10 [1]. Are you sure it wasn't lost somewhere in the meantime?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/log/?h=queue/5.10

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Problems starting Xen domU after latest stable update

[Jason Andryuk]

On Sat, Jan 30, 2021 at 6:25 PM Marek Marczykowski-Górecki
<marmarek@invisiblethingslab.com> wrote:
>
> On Fri, Jan 29, 2021 at 03:16:52PM +0100, Jürgen Groß wrote:
> > On 29.01.21 15:13, Michael Labriola wrote:
> > > On Fri, Jan 29, 2021 at 12:26 AM Jürgen Groß <jgross@suse.com> wrote:
> > > > If the buggy patch has been put into stable this Fixes: tag should
> > > > result in the fix being put into the same stable branches as well.
> > >
> > > I've never done this before...  does this happen automatically?  Or is
> > > there somebody we should ping to make sure it happens?
> >
> > This happens automatically (I think).
> >
> > I have seen mails for the patch been taken for 4.14, 4.19, 5.4 and 5.10.
>
> Hmm, I can't find it in LKML archive, nor stable@ archive. And also it
> isn't included in 5.10.12 released yesterday, nor included in
> queue/5.10 [1]. Are you sure it wasn't lost somewhere in the meantime?

It probably would have gotten in eventually, but I made the explicit
request to move things along.

-Jason