Re: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?

[Anthony PERARD <anthony.perard@citrix.com>]

[-- Attachment #1: Type: text/plain, Size: 2493 bytes --]

On Fri, Jul 17, 2020 at 08:48:01AM +0100, Paul Durrant wrote:
> > -----Original Message-----
> > From: Brian Marcotte <marcotte@panix.com>
> > Sent: 16 July 2020 21:24
> > To: Paul Durrant <xadimgnik@gmail.com>
> > Cc: paul@xen.org; 'Jules' <jules@ispire.me>; xen-devel@lists.xenproject.org;
> > oleksandr_grytsov@epam.com; wl@xen.org
> > Subject: Re: [EXTERNAL] [Xen-devel] XEN Qdisk Ceph rbd support broken?
> > 
> > > Your issue stems from the auto-creation code in xen-block:
> > >
> > > The "aio:rbd:rbd/machine.disk0" string is generated by libxl and does
> > > look a little odd and will fool the parser there, but the error you see
> > > after modifying the string appears to be because QEMU's QMP block
> > > device instantiation code is objecting to a missing parameter. Older
> > > QEMUs circumvented that code which is almost certainly why you don't
> > > see the issue with versions 2 or 3.
> > 
> > Xen 4.13 and 4.14 includes QEMU 4 and 5. They don't work with Ceph/RBD.
> > 
> > Are you saying that xl/libxl is doing the right thing and the problem
> > needs to be fixed in QEMU?
> 
> Unfortunately, from what you describe, it sounds like there is a problem in both. To get something going, you could bring a domain
> up paused and then try manually adding your rbd device using the QMP shell.
> 
> It would be useful if a toolstack maintainer could take a look at this issue in the near future.
> 

Hi,

I did start working on a solution some time ago and produced a patch for
QEMU (attached) which would allow QEMU to parse the aio:rdb:... string
from xenstore.

But I ran into an other issue when I tried with nbd (Network Block
Device), QEMU would connect twice to the NBD server and the server I had
didn't like it. Maybe Ceph would allow two connections to the same disk?

The two connections issue is less likely to happen on older QEMU because
it would delay the second connection until the guest connect to the PV
backend, so after the emulated disk has been unplugged (and thus the
first connection disconnected).

Anyway, it would be better to upgrade libxl to be able to create a QEMU
PV backend via QMP or qemu's command line rather than via xenstore, but
I don't think I have time to work on it just yet. But I feel like we are
going to have the same issue that QEMU will try to connect twice to the
Ceph server where this wasn't likely to happen before.

Jules, Brian, could you maybe try the attach QEMU patch and see if that
works?

Cheers,

-- 
Anthony PERARD

[-- Attachment #2: 0001-xen-block-Fix-parsing-of-legacy-options.patch --]
[-- Type: text/plain, Size: 4031 bytes --]

From 1b8d700007f1f8709a6ef1960111ea022cfb6d74 Mon Sep 17 00:00:00 2001
From: Anthony PERARD <anthony.perard@citrix.com>
Date: Fri, 17 Jan 2020 12:05:09 +0000
Subject: [PATCH] xen-block: Fix parsing of legacy options

Even though the xen-disk PV backend can be instantiated via QMP, we
still need to handle the case where the backend is created via
xenstore. This means that we need to be able to parse legacy disk
options such as "aio:nbd://host:1234/disk".

Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
---
 block.c                        |  6 ++++++
 hw/block/xen-block.c           | 25 +++++++++++++++++++++----
 include/sysemu/block-backend.h |  3 +++
 3 files changed, 30 insertions(+), 4 deletions(-)

diff --git a/block.c b/block.c
index ecd09dbbfd89..13b8690e5006 100644
--- a/block.c
+++ b/block.c
@@ -1705,6 +1705,12 @@ static int bdrv_fill_options(QDict **options, const char *filename,
 
     return 0;
 }
+int bdrv_fill_options_legacy(QDict **options, const char *filename,
+                             int *flags, Error **errp)
+{
+    return bdrv_fill_options(options, filename, flags, errp);
+}
+
 
 static int bdrv_child_check_perm(BdrvChild *c, BlockReopenQueue *q,
                                  uint64_t perm, uint64_t shared,
diff --git a/hw/block/xen-block.c b/hw/block/xen-block.c
index 879fc310a4c5..1cc97a001e1f 100644
--- a/hw/block/xen-block.c
+++ b/hw/block/xen-block.c
@@ -28,6 +28,7 @@
 #include "sysemu/iothread.h"
 #include "dataplane/xen-block.h"
 #include "trace.h"
+#include "include/block/qdict.h"
 
 static char *xen_block_get_name(XenDevice *xendev, Error **errp)
 {
@@ -687,7 +688,12 @@ static char *xen_block_blockdev_add(const char *id, QDict *qdict,
 
     trace_xen_block_blockdev_add(node_name);
 
-    v = qobject_input_visitor_new(QOBJECT(qdict));
+    qdict_flatten(qdict);
+    v = qobject_input_visitor_new_flat_confused(qdict, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        goto fail;
+    }
     visit_type_BlockdevOptions(v, NULL, &options, &local_err);
     visit_free(v);
 
@@ -782,8 +788,14 @@ static XenBlockDrive *xen_block_drive_create(const char *id,
     file_layer = qdict_new();
     driver_layer = qdict_new();
 
-    qdict_put_str(file_layer, "driver", "file");
-    qdict_put_str(file_layer, "filename", filename);
+    int flags = BDRV_O_PROTOCOL | BDRV_O_RDWR;
+    if (mode && *mode != 'w') {
+        flags &= ~BDRV_O_RDWR;
+    }
+    bdrv_fill_options_legacy(&file_layer, filename, &flags, &local_err);
+    if (local_err)
+        goto done;
+
     g_free(filename);
 
     if (mode && *mode != 'w') {
@@ -816,7 +828,12 @@ static XenBlockDrive *xen_block_drive_create(const char *id,
      * It is necessary to turn file locking off as an emulated device
      * may have already opened the same image file.
      */
-    qdict_put_str(file_layer, "locking", "off");
+    const char *file_driver = qdict_get_str(file_layer, "driver");
+    if (!strcmp("file", file_driver) ||
+        !strcmp("host_device", file_driver) ||
+        !strcmp("host_cdrom", file_driver)
+        )
+        qdict_put_str(file_layer, "locking", "off");
 
     qdict_put_str(driver_layer, "driver", driver);
     g_free(driver);
diff --git a/include/sysemu/block-backend.h b/include/sysemu/block-backend.h
index b198deca0b24..93efded0ab61 100644
--- a/include/sysemu/block-backend.h
+++ b/include/sysemu/block-backend.h
@@ -98,6 +98,9 @@ void blk_remove_bs(BlockBackend *blk);
 int blk_insert_bs(BlockBackend *blk, BlockDriverState *bs, Error **errp);
 bool bdrv_has_blk(BlockDriverState *bs);
 bool bdrv_is_root_node(BlockDriverState *bs);
+/* deprecated, not to be used for new backends */
+int bdrv_fill_options_legacy(QDict **options, const char *filename,
+                             int *flags, Error **errp);
 int blk_set_perm(BlockBackend *blk, uint64_t perm, uint64_t shared_perm,
                  Error **errp);
 void blk_get_perm(BlockBackend *blk, uint64_t *perm, uint64_t *shared_perm);
-- 
Anthony PERARD