RE: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?

From: Paul Durrant <xadimgnik@gmail.com>
To: "'Brian Marcotte'" <marcotte@panix.com>
Cc: 'Jules' <jules@ispire.me>,
	xen-devel@lists.xenproject.org, oleksandr_grytsov@epam.com,
	wl@xen.org
Subject: RE: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?
Date: Thu, 16 Jul 2020 08:13:55 +0100	[thread overview]
Message-ID: <000b01d65b40$ab7fead0$027fc070$@xen.org> (raw)
In-Reply-To: <20200715191705.GA20643@panix.com>

> -----Original Message-----
> From: Brian Marcotte <marcotte@panix.com>
> Sent: 15 July 2020 20:17
> To: Durrant, Paul <pdurrant@amazon.co.uk>
> Cc: Jules <jules@ispire.me>; xen-devel@lists.xenproject.org; oleksandr_grytsov@epam.com; wl@xen.org
> Subject: RE: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?
> 
> CAUTION: This email originated from outside of the organization. Do not click links or open
> attachments unless you can confirm the sender and know the content is safe.
> 
> 
> 
> This issue with Xen 4.13 and Ceph/RBD was last discussed back in February.
> 
> > Remote network Ceph image works fine with Xen 4.12.x ...
> >
> > In Xen 4.13.0 which I have tested recently it blames with the error
> > message "no such file or directory" as it would try accessing the image
> > over filesystem instead of remote network image.
> > ---
> >
> > I doubt the issue is in xl/libxl; sounds more likely to be in QEMU. The
> > PV block backend infrastructure in QEMU was changed between the 4.12
> > and 4.13 releases. Have you tried using an older QEMU with 4.13?
> 
> I'm also encountering the problem:
> 
>     failed to create drive: Could not open 'rbd:rbd/machine.disk0': No such file or directory
> 
> Xenstore has "params" like this:
> 
>     aio:rbd:rbd/machine.disk0
> 
> If I set it to "rbd:rbd/machine.disk0", I get a different message:
> 
>   failed to create drive: Parameter 'pool' is missing
> 
> Using upstream QEMU versions 2 or 3 works fine.
> 
> The interesting thing is that access by the virtual BIOS works fine. So,
> for a PVHVM domain, GRUB loads which loads a kernel, but the kernel can't
> access the disks.

Brian,

  That's not entirely surprising as the BIOS is likely to be using an emulated device rather than a PV interface. Your issue stems
from the auto-creation code in xen-block:

https://git.qemu.org/?p=qemu.git;a=blob;f=hw/block/xen-block.c;hb=HEAD#l723

  The "aio:rbd:rbd/machine.disk0" string is generated by libxl and does look a little odd and will fool the parser there, but the
error you see after modifying the string appears to be because QEMU's QMP block device instantiation code is objecting to a missing
parameter. Older QEMUs circumvented that code which is almost certainly why you don't see the issue with versions 2 or 3.

  Paul

> 
> --
> - Brian