Problems starting Xen domU after latest stable update

Problems starting Xen domU after latest stable update

[Michael D Labriola]

Hey, everyone.  I've run into problems starting up my Xen domUs as of
the latest batch of stable updates.  Whenever I try to create one, I
get a bunch of block device errors like this:

libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed

I'm using Xen 4.13.1 on the box I've been testing with.

I bisected down to this commit, and reverting it does indeed fix my
problem.  Well, this commit upstream and it's cherry-picked variants
on linux-5.4.y and linux-5.10.y.

commit 3499ba8198cad47b731792e5e56b9ec2a78a83a2
Author: David Woodhouse <dwmw@amazon.co.uk>
Date:   Wed Jan 13 13:26:02 2021 +0000

    xen: Fix event channel callback via INTX/GSI
    
    For a while, event channel notification via the PCI platform device
    has been broken, because we attempt to communicate with xenstore before
    we even have notifications working, with the xs_reset_watches() call
    in xs_init().
    
    We tend to get away with this on Xen versions below 4.0 because we avoid
    calling xs_reset_watches() anyway, because xenstore might not cope with
    reading a non-existent key. And newer Xen *does* have the vector
    callback support, so we rarely fall back to INTX/GSI delivery.
    
    To fix it, clean up a bit of the mess of xs_init() and xenbus_probe()
    startup. Call xs_init() directly from xenbus_init() only in the !XS_HVM
    case, deferring it to be called from xenbus_probe() in the XS_HVM case
    instead.
    
    Then fix up the invocation of xenbus_probe() to happen either from its
    device_initcall if the callback is available early enough, or when the
    callback is finally set up. This means that the hack of calling
    xenbus_probe() from a workqueue after the first interrupt, or directly
    from the PCI platform device setup, is no longer needed.
    
    Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
    Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
    Link: https://lore.kernel.org/r/20210113132606.422794-2-dwmw2@infradead.org
    Signed-off-by: Juergen Gross <jgross@suse.com>

Problems starting Xen domU after latest stable update

[Michael D Labriola]

Hey, everyone.  I've run into problems starting up my Xen domUs as of
the latest batch of stable updates.  Whenever I try to create one, I
get a bunch of block device errors like this:

libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed

I'm using Xen 4.13.1 on the box I've been testing with.

I bisected down to this commit, and reverting it does indeed fix my
problem.  Well, this commit upstream and it's cherry-picked variants
on linux-5.4.y and linux-5.10.y.

commit 3499ba8198cad47b731792e5e56b9ec2a78a83a2
Author: David Woodhouse <dwmw@amazon.co.uk>
Date:   Wed Jan 13 13:26:02 2021 +0000

    xen: Fix event channel callback via INTX/GSI
    
    For a while, event channel notification via the PCI platform device
    has been broken, because we attempt to communicate with xenstore before
    we even have notifications working, with the xs_reset_watches() call
    in xs_init().
    
    We tend to get away with this on Xen versions below 4.0 because we avoid
    calling xs_reset_watches() anyway, because xenstore might not cope with
    reading a non-existent key. And newer Xen *does* have the vector
    callback support, so we rarely fall back to INTX/GSI delivery.
    
    To fix it, clean up a bit of the mess of xs_init() and xenbus_probe()
    startup. Call xs_init() directly from xenbus_init() only in the !XS_HVM
    case, deferring it to be called from xenbus_probe() in the XS_HVM case
    instead.
    
    Then fix up the invocation of xenbus_probe() to happen either from its
    device_initcall if the callback is available early enough, or when the
    callback is finally set up. This means that the hack of calling
    xenbus_probe() from a workqueue after the first interrupt, or directly
    from the PCI platform device setup, is no longer needed.
    
    Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
    Reviewed-by: Boris Ostrovsky <boris.ostrovsky@oracle.com>
    Link: https://lore.kernel.org/r/20210113132606.422794-2-dwmw2@infradead.org
    Signed-off-by: Juergen Gross <jgross@suse.com>

Re: Problems starting Xen domU after latest stable update

[Boris Ostrovsky]

On 1/28/21 6:52 PM, Michael D Labriola wrote:
> Hey, everyone.  I've run into problems starting up my Xen domUs as of
> the latest batch of stable updates.  Whenever I try to create one, I
> get a bunch of block device errors like this:
>
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
> libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
> libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
> libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
> libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
> libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed
>
> I'm using Xen 4.13.1 on the box I've been testing with.
>
> I bisected down to this commit, and reverting it does indeed fix my
> problem.  Well, this commit upstream and it's cherry-picked variants
> on linux-5.4.y and linux-5.10.y.


You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit Linus tree a few hours ago.


-boris

Re: Problems starting Xen domU after latest stable update

[Michael Labriola]

On Thu, Jan 28, 2021 at 7:03 PM Boris Ostrovsky
<boris.ostrovsky@oracle.com> wrote:
>
>
> On 1/28/21 6:52 PM, Michael D Labriola wrote:
> > Hey, everyone.  I've run into problems starting up my Xen domUs as of
> > the latest batch of stable updates.  Whenever I try to create one, I
> > get a bunch of block device errors like this:
> >
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
> > libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
> > libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
> > libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
> > libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
> > libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed
> >
> > I'm using Xen 4.13.1 on the box I've been testing with.
> >
> > I bisected down to this commit, and reverting it does indeed fix my
> > problem.  Well, this commit upstream and it's cherry-picked variants
> > on linux-5.4.y and linux-5.10.y.
>
>
> You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit Linus tree a few hours ago.

Indeed!  That commit fixes my problem when cherry-picked onto master
and the 5.4 and 5.10 stable branches.

Is that commit already headed to the stable/longterm branches or do we
have to make a bit more noise?

Thanks!

-Mike

Re: Problems starting Xen domU after latest stable update

[Michael Labriola]

On Thu, Jan 28, 2021 at 7:03 PM Boris Ostrovsky
<boris.ostrovsky@oracle.com> wrote:
>
>
> On 1/28/21 6:52 PM, Michael D Labriola wrote:
> > Hey, everyone.  I've run into problems starting up my Xen domUs as of
> > the latest batch of stable updates.  Whenever I try to create one, I
> > get a bunch of block device errors like this:
> >
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
> > libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
> > libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
> > libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
> > libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
> > libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed
> >
> > I'm using Xen 4.13.1 on the box I've been testing with.
> >
> > I bisected down to this commit, and reverting it does indeed fix my
> > problem.  Well, this commit upstream and it's cherry-picked variants
> > on linux-5.4.y and linux-5.10.y.
>
>
> You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit Linus tree a few hours ago.

Indeed!  That commit fixes my problem when cherry-picked onto master
and the 5.4 and 5.10 stable branches.

Is that commit already headed to the stable/longterm branches or do we
have to make a bit more noise?

Thanks!

-Mike

Re: Problems starting Xen domU after latest stable update

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 2761 bytes --]

On Thu, Jan 28, 2021 at 07:03:00PM -0500, Boris Ostrovsky wrote:
> 
> On 1/28/21 6:52 PM, Michael D Labriola wrote:
> > Hey, everyone.  I've run into problems starting up my Xen domUs as of
> > the latest batch of stable updates.  Whenever I try to create one, I
> > get a bunch of block device errors like this:
> >
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
> > libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
> > libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
> > libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
> > libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
> > libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
> > libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed
> >
> > I'm using Xen 4.13.1 on the box I've been testing with.
> >
> > I bisected down to this commit, and reverting it does indeed fix my
> > problem.  Well, this commit upstream and it's cherry-picked variants
> > on linux-5.4.y and linux-5.10.y.
> 
> 
> You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit Linus tree a few hours ago.

I can confirm this fixes the same issue for me (too?), thanks!
Shouldn't this patch have Cc: stable?


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Problems starting Xen domU after latest stable update

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 3067 bytes --]

On 29.01.21 01:51, Marek Marczykowski-Górecki wrote:
> On Thu, Jan 28, 2021 at 07:03:00PM -0500, Boris Ostrovsky wrote:
>>
>> On 1/28/21 6:52 PM, Michael D Labriola wrote:
>>> Hey, everyone.  I've run into problems starting up my Xen domUs as of
>>> the latest batch of stable updates.  Whenever I try to create one, I
>>> get a bunch of block device errors like this:
>>>
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
>>> libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
>>> libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
>>> libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
>>> libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
>>> libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed
>>>
>>> I'm using Xen 4.13.1 on the box I've been testing with.
>>>
>>> I bisected down to this commit, and reverting it does indeed fix my
>>> problem.  Well, this commit upstream and it's cherry-picked variants
>>> on linux-5.4.y and linux-5.10.y.
>>
>>
>> You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit Linus tree a few hours ago.
> 
> I can confirm this fixes the same issue for me (too?), thanks!
> Shouldn't this patch have Cc: stable?

No, I don't think so.

The issue being fixed by the patch has been introduced only in 5.11
and the fixing patch references the buggy patch via a Fixes: tag.

If the buggy patch has been put into stable this Fixes: tag should
result in the fix being put into the same stable branches as well.


Juergen


[-- Attachment #1.1.2: OpenPGP_0xB0DE9DD628BF132F.asc --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Problems starting Xen domU after latest stable update

[Michael Labriola]

On Fri, Jan 29, 2021 at 12:26 AM Jürgen Groß <jgross@suse.com> wrote:
>
> On 29.01.21 01:51, Marek Marczykowski-Górecki wrote:
> > On Thu, Jan 28, 2021 at 07:03:00PM -0500, Boris Ostrovsky wrote:
> >>
> >> On 1/28/21 6:52 PM, Michael D Labriola wrote:
> >>> Hey, everyone.  I've run into problems starting up my Xen domUs as of
> >>> the latest batch of stable updates.  Whenever I try to create one, I
> >>> get a bunch of block device errors like this:
> >>>
> >>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
> >>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
> >>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
> >>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
> >>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
> >>> libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
> >>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
> >>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
> >>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
> >>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
> >>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
> >>> libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
> >>> libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
> >>> libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
> >>> libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed
> >>>
> >>> I'm using Xen 4.13.1 on the box I've been testing with.
> >>>
> >>> I bisected down to this commit, and reverting it does indeed fix my
> >>> problem.  Well, this commit upstream and it's cherry-picked variants
> >>> on linux-5.4.y and linux-5.10.y.
> >>
> >>
> >> You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit Linus tree a few hours ago.
> >
> > I can confirm this fixes the same issue for me (too?), thanks!
> > Shouldn't this patch have Cc: stable?
>
> No, I don't think so.
>
> The issue being fixed by the patch has been introduced only in 5.11

For the record, the issue is also in the latest stable 5.4.y and
5.10.y releases (I assume older ones as well, but those are the only 2
I tested).  That's where I ran into it initially.

> and the fixing patch references the buggy patch via a Fixes: tag.
>
> If the buggy patch has been put into stable this Fixes: tag should
> result in the fix being put into the same stable branches as well.

I've never done this before...  does this happen automatically?  Or is
there somebody we should ping to make sure it happens?

Thanks!

-Mike

Re: Problems starting Xen domU after latest stable update

[Jürgen Groß]

[-- Attachment #1.1.1: Type: text/plain, Size: 3737 bytes --]

On 29.01.21 15:13, Michael Labriola wrote:
> On Fri, Jan 29, 2021 at 12:26 AM Jürgen Groß <jgross@suse.com> wrote:
>>
>> On 29.01.21 01:51, Marek Marczykowski-Górecki wrote:
>>> On Thu, Jan 28, 2021 at 07:03:00PM -0500, Boris Ostrovsky wrote:
>>>>
>>>> On 1/28/21 6:52 PM, Michael D Labriola wrote:
>>>>> Hey, everyone.  I've run into problems starting up my Xen domUs as of
>>>>> the latest batch of stable updates.  Whenever I try to create one, I
>>>>> get a bunch of block device errors like this:
>>>>>
>>>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51712
>>>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51728
>>>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51744
>>>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51760
>>>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to add device with path /local/domain/0/backend/vbd/4/51776
>>>>> libxl: error: libxl_create.c:1452:domcreate_launch_dm: Domain 4:unable to add disk devices
>>>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51712
>>>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51728
>>>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51744
>>>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51760
>>>>> libxl: error: libxl_device.c:1105:device_backend_callback: Domain 4:unable to remove device with path /local/domain/0/backend/vbd/4/51776
>>>>> libxl: error: libxl_domain.c:1290:devices_destroy_cb: Domain 4:libxl__devices_destroy failed
>>>>> libxl: error: libxl_domain.c:1177:libxl__destroy_domid: Domain 4:Non-existant domain
>>>>> libxl: error: libxl_domain.c:1131:domain_destroy_callback: Domain 4:Unable to destroy guest
>>>>> libxl: error: libxl_domain.c:1058:domain_destroy_cb: Domain 4:Destruction of domain failed
>>>>>
>>>>> I'm using Xen 4.13.1 on the box I've been testing with.
>>>>>
>>>>> I bisected down to this commit, and reverting it does indeed fix my
>>>>> problem.  Well, this commit upstream and it's cherry-picked variants
>>>>> on linux-5.4.y and linux-5.10.y.
>>>>
>>>>
>>>> You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit Linus tree a few hours ago.
>>>
>>> I can confirm this fixes the same issue for me (too?), thanks!
>>> Shouldn't this patch have Cc: stable?
>>
>> No, I don't think so.
>>
>> The issue being fixed by the patch has been introduced only in 5.11
> 
> For the record, the issue is also in the latest stable 5.4.y and
> 5.10.y releases (I assume older ones as well, but those are the only 2
> I tested).  That's where I ran into it initially.
> 
>> and the fixing patch references the buggy patch via a Fixes: tag.
>>
>> If the buggy patch has been put into stable this Fixes: tag should
>> result in the fix being put into the same stable branches as well.
> 
> I've never done this before...  does this happen automatically?  Or is
> there somebody we should ping to make sure it happens?

This happens automatically (I think).

I have seen mails for the patch been taken for 4.14, 4.19, 5.4 and 5.10.


Juergen


[-- Attachment #1.1.2: OpenPGP_0xB0DE9DD628BF132F.asc --]
[-- Type: application/pgp-keys, Size: 3135 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: [EXTERNAL] Problems starting Xen domU after latest stable update

[David Woodhouse]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

On Thu, 2021-01-28 at 19:03 -0500, Boris Ostrovsky wrote:
> > I'm using Xen 4.13.1 on the box I've been testing with.
> > 
> > I bisected down to this commit, and reverting it does indeed fix my
> > problem.  Well, this commit upstream and it's cherry-picked variants
> > on linux-5.4.y and linux-5.10.y.
> 
> 
> You most likely need 5f46400f7a6a4fad635d5a79e2aa5a04a30ffea1. It hit
> Linus tree a few hours ago.

Yeah, apologies for that. I didn't think I was changing anything for PV
and Dom0, so I had only tested as DomU.

I didn't mark the original series for stable@ but they made their way
in anyway; I'm not quite sure by what means.

For the *fix*, since it has a Fixes: tag which identifies a commit
which had already been backported to the various stable trees, that
should make it in automatically now that it's hit Linus's tree.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 5174 bytes --]

Re: Problems starting Xen domU after latest stable update

[Marek Marczykowski-Górecki]

[-- Attachment #1: Type: text/plain, Size: 984 bytes --]

On Fri, Jan 29, 2021 at 03:16:52PM +0100, Jürgen Groß wrote:
> On 29.01.21 15:13, Michael Labriola wrote:
> > On Fri, Jan 29, 2021 at 12:26 AM Jürgen Groß <jgross@suse.com> wrote:
> > > If the buggy patch has been put into stable this Fixes: tag should
> > > result in the fix being put into the same stable branches as well.
> > 
> > I've never done this before...  does this happen automatically?  Or is
> > there somebody we should ping to make sure it happens?
> 
> This happens automatically (I think).
> 
> I have seen mails for the patch been taken for 4.14, 4.19, 5.4 and 5.10.

Hmm, I can't find it in LKML archive, nor stable@ archive. And also it
isn't included in 5.10.12 released yesterday, nor included in
queue/5.10 [1]. Are you sure it wasn't lost somewhere in the meantime?

[1] https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/log/?h=queue/5.10

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: Problems starting Xen domU after latest stable update

[Jason Andryuk]

On Sat, Jan 30, 2021 at 6:25 PM Marek Marczykowski-Górecki
<marmarek@invisiblethingslab.com> wrote:
>
> On Fri, Jan 29, 2021 at 03:16:52PM +0100, Jürgen Groß wrote:
> > On 29.01.21 15:13, Michael Labriola wrote:
> > > On Fri, Jan 29, 2021 at 12:26 AM Jürgen Groß <jgross@suse.com> wrote:
> > > > If the buggy patch has been put into stable this Fixes: tag should
> > > > result in the fix being put into the same stable branches as well.
> > >
> > > I've never done this before...  does this happen automatically?  Or is
> > > there somebody we should ping to make sure it happens?
> >
> > This happens automatically (I think).
> >
> > I have seen mails for the patch been taken for 4.14, 4.19, 5.4 and 5.10.
>
> Hmm, I can't find it in LKML archive, nor stable@ archive. And also it
> isn't included in 5.10.12 released yesterday, nor included in
> queue/5.10 [1]. Are you sure it wasn't lost somewhere in the meantime?

It probably would have gotten in eventually, but I made the explicit
request to move things along.

-Jason