[Xen-devel] XEN Qdisk Ceph rbd support broken?

[Xen-devel] XEN Qdisk Ceph rbd support broken?

[Jules]

[-- Attachment #1.1: Type: text/plain, Size: 750 bytes --]

Hey,

I don’t know if it was this or a previous change in qdisk driver, but can it be that remote Ceph RBD support is broken?
https://github.com/xen-project/xen/commit/8f486344a00652ed202ade43c02c96771812bf8c <https://github.com/xen-project/xen/commit/8f486344a00652ed202ade43c02c96771812bf8c>

Remote network Ceph image works fine with Xen 4.12.x with a config syntax like this:
disk = [ 'format=raw, vdev=xvda1, access=rw,backendtype=qdisk, target=rbd:<pool-Name>/<Image-Name>:id=<cephx-Id>‘ ]

In Xen 4.13.0 which I have tested recently it blames with the error message „no such file or directory“ as it would try accessing the image over filesystem instead of remote network image.

Please help confirm.


Kind Regards
Jules

[-- Attachment #1.2: Type: text/html, Size: 1322 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] XEN Qdisk Ceph rbd support broken?

[Durrant, Paul]

De-htmling...

---
From: Xen-devel <xen-devel-bounces@lists.xenproject.org> On Behalf Of Jules
Sent: 03 February 2020 17:35
To: xen-devel@lists.xenproject.org
Cc: oleksandr_grytsov@epam.com; wl@xen.org
Subject: [Xen-devel] XEN Qdisk Ceph rbd support broken?

Hey,

I don’t know if it was this or a previous change in qdisk driver, but can it be that remote Ceph RBD support is broken?
https://github.com/xen-project/xen/commit/8f486344a00652ed202ade43c02c96771812bf8c

Remote network Ceph image works fine with Xen 4.12.x with a config syntax like this:
disk = [ 'format=raw, vdev=xvda1, access=rw,backendtype=qdisk, target=rbd:<pool-Name>/<Image-Name>:id=<cephx-Id>‘ ]

In Xen 4.13.0 which I have tested recently it blames with the error message „no such file or directory“ as it would try accessing the image over filesystem instead of remote network image.
---

I doubt the issue is in xl/libxl; sounds more likely to be in QEMU. The PV block backend infrastructure in QEMU was changed between the 4.12 and 4.13 releases. Have you tried using an older QEMU with 4.13?

  Paul
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [Xen-devel] XEN Qdisk Ceph rbd support broken?

[Brian Marcotte]

This issue with Xen 4.13 and Ceph/RBD was last discussed back in February.

> Remote network Ceph image works fine with Xen 4.12.x ...
> 
> In Xen 4.13.0 which I have tested recently it blames with the error
> message "no such file or directory" as it would try accessing the image
> over filesystem instead of remote network image.
> ---
> 
> I doubt the issue is in xl/libxl; sounds more likely to be in QEMU. The
> PV block backend infrastructure in QEMU was changed between the 4.12
> and 4.13 releases. Have you tried using an older QEMU with 4.13?

I'm also encountering the problem:

    failed to create drive: Could not open 'rbd:rbd/machine.disk0': No such file or directory

Xenstore has "params" like this:

    aio:rbd:rbd/machine.disk0

If I set it to "rbd:rbd/machine.disk0", I get a different message:

  failed to create drive: Parameter 'pool' is missing

Using upstream QEMU versions 2 or 3 works fine.

The interesting thing is that access by the virtual BIOS works fine. So,
for a PVHVM domain, GRUB loads which loads a kernel, but the kernel can't
access the disks.

--
- Brian

RE: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?

[Paul Durrant]

> -----Original Message-----
> From: Brian Marcotte <marcotte@panix.com>
> Sent: 15 July 2020 20:17
> To: Durrant, Paul <pdurrant@amazon.co.uk>
> Cc: Jules <jules@ispire.me>; xen-devel@lists.xenproject.org; oleksandr_grytsov@epam.com; wl@xen.org
> Subject: RE: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?
> 
> CAUTION: This email originated from outside of the organization. Do not click links or open
> attachments unless you can confirm the sender and know the content is safe.
> 
> 
> 
> This issue with Xen 4.13 and Ceph/RBD was last discussed back in February.
> 
> > Remote network Ceph image works fine with Xen 4.12.x ...
> >
> > In Xen 4.13.0 which I have tested recently it blames with the error
> > message "no such file or directory" as it would try accessing the image
> > over filesystem instead of remote network image.
> > ---
> >
> > I doubt the issue is in xl/libxl; sounds more likely to be in QEMU. The
> > PV block backend infrastructure in QEMU was changed between the 4.12
> > and 4.13 releases. Have you tried using an older QEMU with 4.13?
> 
> I'm also encountering the problem:
> 
>     failed to create drive: Could not open 'rbd:rbd/machine.disk0': No such file or directory
> 
> Xenstore has "params" like this:
> 
>     aio:rbd:rbd/machine.disk0
> 
> If I set it to "rbd:rbd/machine.disk0", I get a different message:
> 
>   failed to create drive: Parameter 'pool' is missing
> 
> Using upstream QEMU versions 2 or 3 works fine.
> 
> The interesting thing is that access by the virtual BIOS works fine. So,
> for a PVHVM domain, GRUB loads which loads a kernel, but the kernel can't
> access the disks.

Brian,

  That's not entirely surprising as the BIOS is likely to be using an emulated device rather than a PV interface. Your issue stems
from the auto-creation code in xen-block:

https://git.qemu.org/?p=qemu.git;a=blob;f=hw/block/xen-block.c;hb=HEAD#l723

  The "aio:rbd:rbd/machine.disk0" string is generated by libxl and does look a little odd and will fool the parser there, but the
error you see after modifying the string appears to be because QEMU's QMP block device instantiation code is objecting to a missing
parameter. Older QEMUs circumvented that code which is almost certainly why you don't see the issue with versions 2 or 3.

  Paul





> 
> --
> - Brian

Re: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?

[Brian Marcotte]

> Your issue stems from the auto-creation code in xen-block:
> 
> The "aio:rbd:rbd/machine.disk0" string is generated by libxl and does
> look a little odd and will fool the parser there, but the error you see
> after modifying the string appears to be because QEMU's QMP block
> device instantiation code is objecting to a missing parameter. Older
> QEMUs circumvented that code which is almost certainly why you don't
> see the issue with versions 2 or 3.

Xen 4.13 and 4.14 includes QEMU 4 and 5. They don't work with Ceph/RBD.

Are you saying that xl/libxl is doing the right thing and the problem
needs to be fixed in QEMU?

--
- Brian

RE: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?

[Paul Durrant]

> -----Original Message-----
> From: Brian Marcotte <marcotte@panix.com>
> Sent: 16 July 2020 21:24
> To: Paul Durrant <xadimgnik@gmail.com>
> Cc: paul@xen.org; 'Jules' <jules@ispire.me>; xen-devel@lists.xenproject.org;
> oleksandr_grytsov@epam.com; wl@xen.org
> Subject: Re: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?
> 
> > Your issue stems from the auto-creation code in xen-block:
> >
> > The "aio:rbd:rbd/machine.disk0" string is generated by libxl and does
> > look a little odd and will fool the parser there, but the error you see
> > after modifying the string appears to be because QEMU's QMP block
> > device instantiation code is objecting to a missing parameter. Older
> > QEMUs circumvented that code which is almost certainly why you don't
> > see the issue with versions 2 or 3.
> 
> Xen 4.13 and 4.14 includes QEMU 4 and 5. They don't work with Ceph/RBD.
> 
> Are you saying that xl/libxl is doing the right thing and the problem
> needs to be fixed in QEMU?

Unfortunately, from what you describe, it sounds like there is a problem in both. To get something going, you could bring a domain
up paused and then try manually adding your rbd device using the QMP shell.

It would be useful if a toolstack maintainer could take a look at this issue in the near future.

  Paul

> 
> --
> - Brian

Re: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?

[Anthony PERARD]

[-- Attachment #1: Type: text/plain, Size: 2493 bytes --]

On Fri, Jul 17, 2020 at 08:48:01AM +0100, Paul Durrant wrote:
> > -----Original Message-----
> > From: Brian Marcotte <marcotte@panix.com>
> > Sent: 16 July 2020 21:24
> > To: Paul Durrant <xadimgnik@gmail.com>
> > Cc: paul@xen.org; 'Jules' <jules@ispire.me>; xen-devel@lists.xenproject.org;
> > oleksandr_grytsov@epam.com; wl@xen.org
> > Subject: Re: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?
> > 
> > > Your issue stems from the auto-creation code in xen-block:
> > >
> > > The "aio:rbd:rbd/machine.disk0" string is generated by libxl and does
> > > look a little odd and will fool the parser there, but the error you see
> > > after modifying the string appears to be because QEMU's QMP block
> > > device instantiation code is objecting to a missing parameter. Older
> > > QEMUs circumvented that code which is almost certainly why you don't
> > > see the issue with versions 2 or 3.
> > 
> > Xen 4.13 and 4.14 includes QEMU 4 and 5. They don't work with Ceph/RBD.
> > 
> > Are you saying that xl/libxl is doing the right thing and the problem
> > needs to be fixed in QEMU?
> 
> Unfortunately, from what you describe, it sounds like there is a problem in both. To get something going, you could bring a domain
> up paused and then try manually adding your rbd device using the QMP shell.
> 
> It would be useful if a toolstack maintainer could take a look at this issue in the near future.
> 

Hi,

I did start working on a solution some time ago and produced a patch for
QEMU (attached) which would allow QEMU to parse the aio:rdb:... string
from xenstore.

But I ran into an other issue when I tried with nbd (Network Block
Device), QEMU would connect twice to the NBD server and the server I had
didn't like it. Maybe Ceph would allow two connections to the same disk?

The two connections issue is less likely to happen on older QEMU because
it would delay the second connection until the guest connect to the PV
backend, so after the emulated disk has been unplugged (and thus the
first connection disconnected).

Anyway, it would be better to upgrade libxl to be able to create a QEMU
PV backend via QMP or qemu's command line rather than via xenstore, but
I don't think I have time to work on it just yet. But I feel like we are
going to have the same issue that QEMU will try to connect twice to the
Ceph server where this wasn't likely to happen before.

Jules, Brian, could you maybe try the attach QEMU patch and see if that
works?

Cheers,

-- 
Anthony PERARD

[-- Attachment #2: 0001-xen-block-Fix-parsing-of-legacy-options.patch --]
[-- Type: text/plain, Size: 4031 bytes --]

From 1b8d700007f1f8709a6ef1960111ea022cfb6d74 Mon Sep 17 00:00:00 2001
From: Anthony PERARD <anthony.perard@citrix.com>
Date: Fri, 17 Jan 2020 12:05:09 +0000
Subject: [PATCH] xen-block: Fix parsing of legacy options

Even though the xen-disk PV backend can be instantiated via QMP, we
still need to handle the case where the backend is created via
xenstore. This means that we need to be able to parse legacy disk
options such as "aio:nbd://host:1234/disk".

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
 block.c                        |  6 ++++++
 hw/block/xen-block.c           | 25 +++++++++++++++++++++----
 include/sysemu/block-backend.h |  3 +++
 3 files changed, 30 insertions(+), 4 deletions(-)

diff --git a/block.c b/block.c
index ecd09dbbfd89..13b8690e5006 100644
--- a/block.c
+++ b/block.c
@@ -1705,6 +1705,12 @@ static int bdrv_fill_options(QDict **options, const char *filename,
 
     return 0;
 }
+int bdrv_fill_options_legacy(QDict **options, const char *filename,
+                             int *flags, Error **errp)
+{
+    return bdrv_fill_options(options, filename, flags, errp);
+}
+
 
 static int bdrv_child_check_perm(BdrvChild *c, BlockReopenQueue *q,
                                  uint64_t perm, uint64_t shared,
diff --git a/hw/block/xen-block.c b/hw/block/xen-block.c
index 879fc310a4c5..1cc97a001e1f 100644
--- a/hw/block/xen-block.c
+++ b/hw/block/xen-block.c
@@ -28,6 +28,7 @@
 #include "sysemu/iothread.h"
 #include "dataplane/xen-block.h"
 #include "trace.h"
+#include "include/block/qdict.h"
 
 static char *xen_block_get_name(XenDevice *xendev, Error **errp)
 {
@@ -687,7 +688,12 @@ static char *xen_block_blockdev_add(const char *id, QDict *qdict,
 
     trace_xen_block_blockdev_add(node_name);
 
-    v = qobject_input_visitor_new(QOBJECT(qdict));
+    qdict_flatten(qdict);
+    v = qobject_input_visitor_new_flat_confused(qdict, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        goto fail;
+    }
     visit_type_BlockdevOptions(v, NULL, &options, &local_err);
     visit_free(v);
 
@@ -782,8 +788,14 @@ static XenBlockDrive *xen_block_drive_create(const char *id,
     file_layer = qdict_new();
     driver_layer = qdict_new();
 
-    qdict_put_str(file_layer, "driver", "file");
-    qdict_put_str(file_layer, "filename", filename);
+    int flags = BDRV_O_PROTOCOL | BDRV_O_RDWR;
+    if (mode && *mode != 'w') {
+        flags &= ~BDRV_O_RDWR;
+    }
+    bdrv_fill_options_legacy(&file_layer, filename, &flags, &local_err);
+    if (local_err)
+        goto done;
+
     g_free(filename);
 
     if (mode && *mode != 'w') {
@@ -816,7 +828,12 @@ static XenBlockDrive *xen_block_drive_create(const char *id,
      * It is necessary to turn file locking off as an emulated device
      * may have already opened the same image file.
      */
-    qdict_put_str(file_layer, "locking", "off");
+    const char *file_driver = qdict_get_str(file_layer, "driver");
+    if (!strcmp("file", file_driver) ||
+        !strcmp("host_device", file_driver) ||
+        !strcmp("host_cdrom", file_driver)
+        )
+        qdict_put_str(file_layer, "locking", "off");
 
     qdict_put_str(driver_layer, "driver", driver);
     g_free(driver);
diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index b198deca0b24..93efded0ab61 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -98,6 +98,9 @@ void blk_remove_bs(BlockBackend *blk);
 int blk_insert_bs(BlockBackend *blk, BlockDriverState *bs, Error **errp);
 bool bdrv_has_blk(BlockDriverState *bs);
 bool bdrv_is_root_node(BlockDriverState *bs);
+/* deprecated, not to be used for new backends */
+int bdrv_fill_options_legacy(QDict **options, const char *filename,
+                             int *flags, Error **errp);
 int blk_set_perm(BlockBackend *blk, uint64_t perm, uint64_t shared_perm,
                  Error **errp);
 void blk_get_perm(BlockBackend *blk, uint64_t *perm, uint64_t *shared_perm);
-- 
Anthony PERARD

Re: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?

[Brian Marcotte]

> Jules, Brian, could you maybe try the attach QEMU patch and see if that
> works?

I applied the patch to the latest release QEMU (5.1.0), and successfully
ran these types of guests:

   PVHVM Linux
   PVHVM FreeBSD
   PV NetBSD
   PVH NetBSD

--
- Brian