qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [RFC PATCH 00/18] Add qemu-storage-daemon
@ 2019-10-17 13:01 Kevin Wolf
  2019-10-17 13:01 ` [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool Kevin Wolf
                   ` (22 more replies)
  0 siblings, 23 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

This series adds a new tool 'qemu-storage-daemon', which can be used to
export and perform operations on block devices. There is some overlap
between qemu-img/qemu-nbd and the new qemu-storage-daemon, but there are
a few important differences:

* The qemu-storage-daemon has QMP support. The command set is obviously
  restricted compared to the system emulator because there is no guest,
  but all of the block operations are present.

  This means that it can access advanced options or operations that the
  qemu-img command line doesn't expose. For example, blockdev-create is
  a lot more powerful than 'qemu-img create', and qemu-storage-daemon
  allows to execute it without starting a guest.

  Compared to qemu-nbd it means that, for example, block jobs can now be
  executed on the server side, and backing chains shared by multiple VMs
  can be modified this way.

* The existing tools all have a separately invented one-off syntax for
  the job at hand, which usually comes with restrictions compared to the
  system emulator. qemu-storage-daemon shares the same syntax with the
  system emulator for most options and prefers QAPI based interfaces
  where possible (such as --blockdev), so it should be easy to make use
  of in libvirt.

* While this series implements only NBD exports, the storage daemon is
  intended to serve multiple protocols and its syntax reflects this. In
  the past, we had proposals to add new one-off tools for exporting over
  new protocols like FUSE or TCMU.

  With a generic storage daemon, additional export methods have a home
  without adding a new tool for each of them.

I'm posting this as an RFC mainly for two reasons:

1. The monitor integration, which could be argued to be a little hackish
   (some generated QAPI source files are built from a separate QAPI
   schema, but the per-module ones are taken from the system emulator)
   and Markus will want to have a closer look there. But from the IRC
   discussions we had, we seem to agree on the general approach here.

2. I'm not completely sure if the command line syntax is the final
   version that we want to support long-term. Many options directly use
   QAPI visitors (--blockdev, --export, --nbd-server) and should be
   fine. However, others use QemuOpts (--chardev, --monitor, --object).

   This is the same as in the system emulator, so we wouldn't be adding
   a new problem, but as there was talk about QAPIfying the command
   line, and I wouldn't want later syntax changes or adding lots of
   compatibility code to a new tool, I thought we should probably
   discuss whether QAPIfying from the start would be an option.

I would like to have something merged for 4.2, but I'm considering
marking the whole tool as experimental and unstable ABI by requiring a
command line option like --experimental for the tool to even start if we
know that we want to change some things later.

This series is based on:
* [PATCH] blockdev: Use error_report() in hmp_commit()
* [PATCH 0/7] qapi: Cleanups and test speedup

Based-on: <20191015123932.12214-1-kwolf@redhat.com>
Based-on: <20191001191514.11208-1-armbru@redhat.com>

Kevin Wolf (18):
  qemu-storage-daemon: Add barebone tool
  qemu-storage-daemon: Add --object option
  stubs: Add arch_type
  stubs: Add blk_by_qdev_id()
  qemu-storage-daemon: Add --blockdev option
  qemu-storage-daemon: Add --nbd-server option
  blockdev-nbd: Boxed argument type for nbd-server-add
  qemu-storage-daemon: Add --export option
  qemu-storage-daemon: Add main loop
  qemu-storage-daemon: Add --chardev option
  monitor: Move monitor option parsing to monitor/monitor.c
  stubs: Update monitor stubs for qemu-storage-daemon
  qapi: Create module 'monitor'
  monitor: Create monitor/qmp-cmds-monitor.c
  qapi: Support empty modules
  qapi: Create 'pragma' module
  monitor: Move qmp_query_qmp_schema to qmp-cmds-monitor.c
  qemu-storage-daemon: Add --monitor option

 qapi/block.json                          |  65 ++++-
 qapi/misc.json                           | 212 ---------------
 qapi/monitor.json                        | 218 +++++++++++++++
 qapi/pragma.json                         |  24 ++
 qapi/qapi-schema.json                    |  26 +-
 storage-daemon/qapi/qapi-schema.json     |  15 ++
 configure                                |   2 +-
 include/block/nbd.h                      |   1 +
 include/monitor/monitor.h                |   4 +
 include/sysemu/arch_init.h               |   2 +
 include/sysemu/sysemu.h                  |   1 -
 monitor/monitor-internal.h               |   4 +
 blockdev-nbd.c                           |  30 ++-
 monitor/hmp-cmds.c                       |  22 +-
 monitor/misc.c                           | 126 ---------
 monitor/monitor.c                        |  52 ++++
 monitor/qmp-cmds-monitor.c               | 169 ++++++++++++
 monitor/qmp-cmds.c                       |  15 +-
 monitor/qmp.c                            |   2 +-
 qemu-storage-daemon.c                    | 326 +++++++++++++++++++++++
 stubs/arch_type.c                        |   4 +
 stubs/blk-by-qdev-id.c                   |   9 +
 stubs/monitor-core.c                     |  21 ++
 stubs/monitor.c                          |  15 +-
 tests/qmp-test.c                         |   2 +-
 ui/gtk.c                                 |   1 +
 vl.c                                     |  45 +---
 Makefile                                 |  34 +++
 Makefile.objs                            |   9 +
 block/Makefile.objs                      |   2 +-
 monitor/Makefile.objs                    |   5 +-
 qapi/Makefile.objs                       |   9 +-
 qom/Makefile.objs                        |   1 +
 scripts/qapi/gen.py                      |   5 +
 scripts/qapi/schema.py                   |   9 +
 storage-daemon/Makefile.objs             |   1 +
 storage-daemon/qapi/Makefile.objs        |   1 +
 stubs/Makefile.objs                      |   3 +
 tests/qapi-schema/comments.out           |   2 +
 tests/qapi-schema/doc-bad-section.out    |   2 +
 tests/qapi-schema/doc-good.out           |   2 +
 tests/qapi-schema/empty.out              |   2 +
 tests/qapi-schema/event-case.out         |   2 +
 tests/qapi-schema/include-repetition.out |   4 +
 tests/qapi-schema/include-simple.out     |   3 +
 tests/qapi-schema/indented-expr.out      |   2 +
 tests/qapi-schema/qapi-schema-test.out   |   4 +
 47 files changed, 1052 insertions(+), 463 deletions(-)
 create mode 100644 qapi/monitor.json
 create mode 100644 qapi/pragma.json
 create mode 100644 storage-daemon/qapi/qapi-schema.json
 create mode 100644 monitor/qmp-cmds-monitor.c
 create mode 100644 qemu-storage-daemon.c
 create mode 100644 stubs/arch_type.c
 create mode 100644 stubs/blk-by-qdev-id.c
 create mode 100644 stubs/monitor-core.c
 create mode 100644 storage-daemon/Makefile.objs
 create mode 100644 storage-daemon/qapi/Makefile.objs

-- 
2.20.1



^ permalink raw reply	[flat|nested] 65+ messages in thread

* [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-10-24 13:50   ` Eric Blake
                     ` (2 more replies)
  2019-10-17 13:01 ` [RFC PATCH 02/18] qemu-storage-daemon: Add --object option Kevin Wolf
                   ` (21 subsequent siblings)
  22 siblings, 3 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

This adds a new binary qemu-storage-daemon that doesn't yet do more than
some typical initialisation for tools and parsing the basic command
options --version, --help and --trace.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 configure             |   2 +-
 qemu-storage-daemon.c | 141 ++++++++++++++++++++++++++++++++++++++++++
 Makefile              |   1 +
 3 files changed, 143 insertions(+), 1 deletion(-)
 create mode 100644 qemu-storage-daemon.c

diff --git a/configure b/configure
index 08ca4bcb46..bb3d55fb25 100755
--- a/configure
+++ b/configure
@@ -6034,7 +6034,7 @@ tools=""
 if test "$want_tools" = "yes" ; then
   tools="qemu-img\$(EXESUF) qemu-io\$(EXESUF) qemu-edid\$(EXESUF) $tools"
   if [ "$linux" = "yes" -o "$bsd" = "yes" -o "$solaris" = "yes" ] ; then
-    tools="qemu-nbd\$(EXESUF) $tools"
+    tools="qemu-nbd\$(EXESUF) qemu-storage-daemon\$(EXESUF) $tools"
   fi
   if [ "$ivshmem" = "yes" ]; then
     tools="ivshmem-client\$(EXESUF) ivshmem-server\$(EXESUF) $tools"
diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
new file mode 100644
index 0000000000..a251dc255c
--- /dev/null
+++ b/qemu-storage-daemon.c
@@ -0,0 +1,141 @@
+/*
+ * QEMU storage daemon
+ *
+ * Copyright (c) 2019 Kevin Wolf <kwolf@redhat.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+
+#include "block/block.h"
+#include "crypto/init.h"
+
+#include "qapi/error.h"
+#include "qemu-common.h"
+#include "qemu-version.h"
+#include "qemu/config-file.h"
+#include "qemu/error-report.h"
+#include "qemu/log.h"
+#include "qemu/main-loop.h"
+#include "qemu/module.h"
+
+#include "trace/control.h"
+
+#include <getopt.h>
+
+static void help(void)
+{
+    printf(
+"Usage: %s [options]\n"
+"QEMU storage daemon\n"
+"\n"
+"  -h, --help             display this help and exit\n"
+"  -T, --trace [[enable=]<pattern>][,events=<file>][,file=<file>]\n"
+"                         specify tracing options\n"
+"  -V, --version          output version information and exit\n"
+"\n"
+QEMU_HELP_BOTTOM "\n",
+    error_get_progname());
+}
+
+static int process_options(int argc, char *argv[], Error **errp)
+{
+    int c;
+    char *trace_file = NULL;
+    int ret = -EINVAL;
+
+    static const struct option long_options[] = {
+        {"help", no_argument, 0, 'h'},
+        {"version", no_argument, 0, 'V'},
+        {"trace", required_argument, NULL, 'T'},
+        {0, 0, 0, 0}
+    };
+
+    while ((c = getopt_long(argc, argv, ":hT:V", long_options, NULL)) != -1) {
+        switch (c) {
+        case '?':
+            error_setg(errp, "Unknown option '%s'", argv[optind - 1]);
+            goto out;
+        case ':':
+            error_setg(errp, "Missing option argument for '%s'",
+                       argv[optind - 1]);
+            goto out;
+        case 'h':
+            help();
+            exit(EXIT_SUCCESS);
+        case 'V':
+            printf("qemu-storage-daemon version "
+                   QEMU_FULL_VERSION "\n" QEMU_COPYRIGHT "\n");
+            exit(EXIT_SUCCESS);
+        case 'T':
+            g_free(trace_file);
+            trace_file = trace_opt_parse(optarg);
+            break;
+        }
+    }
+    if (optind != argc) {
+        error_setg(errp, "Unexpected argument: %s", argv[optind]);
+        goto out;
+    }
+
+    if (!trace_init_backends()) {
+        error_setg(errp, "Could not initialize trace backends");
+        goto out;
+    }
+    trace_init_file(trace_file);
+    qemu_set_log(LOG_TRACE);
+
+    ret = 0;
+out:
+    g_free(trace_file);
+    return ret;
+}
+
+int main(int argc, char *argv[])
+{
+    Error *local_err = NULL;
+    int ret;
+
+#ifdef CONFIG_POSIX
+    signal(SIGPIPE, SIG_IGN);
+#endif
+
+    error_init(argv[0]);
+    qemu_init_exec_dir(argv[0]);
+
+    module_call_init(MODULE_INIT_QOM);
+    module_call_init(MODULE_INIT_TRACE);
+    qemu_add_opts(&qemu_trace_opts);
+    qcrypto_init(&error_fatal);
+    bdrv_init();
+
+    if (qemu_init_main_loop(&local_err)) {
+        error_report_err(local_err);
+        return EXIT_FAILURE;
+    }
+
+    ret = process_options(argc, argv, &local_err);
+    if (ret < 0) {
+        error_report_err(local_err);
+        return EXIT_FAILURE;
+    }
+
+    return EXIT_SUCCESS;
+}
diff --git a/Makefile b/Makefile
index 30f0abfb42..76338d0ab4 100644
--- a/Makefile
+++ b/Makefile
@@ -558,6 +558,7 @@ qemu-img.o: qemu-img-cmds.h
 qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
+qemu-storage-daemon$(EXESUF): qemu-storage-daemon.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 
 qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COMMON_LDADDS)
 
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 02/18] qemu-storage-daemon: Add --object option
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
  2019-10-17 13:01 ` [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-11-07 20:36   ` Markus Armbruster
  2019-10-17 13:01 ` [RFC PATCH 03/18] stubs: Add arch_type Kevin Wolf
                   ` (20 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

Add a command line option to create user-creatable QOM objects.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-storage-daemon.c | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
index a251dc255c..48d6af43a6 100644
--- a/qemu-storage-daemon.c
+++ b/qemu-storage-daemon.c
@@ -35,6 +35,8 @@
 #include "qemu/log.h"
 #include "qemu/main-loop.h"
 #include "qemu/module.h"
+#include "qemu/option.h"
+#include "qom/object_interfaces.h"
 
 #include "trace/control.h"
 
@@ -51,10 +53,26 @@ static void help(void)
 "                         specify tracing options\n"
 "  -V, --version          output version information and exit\n"
 "\n"
+"  --object <properties>  define a QOM object such as 'secret' for\n"
+"                         passwords and/or encryption keys\n"
+"\n"
 QEMU_HELP_BOTTOM "\n",
     error_get_progname());
 }
 
+enum {
+    OPTION_OBJECT = 256,
+};
+
+static QemuOptsList qemu_object_opts = {
+    .name = "object",
+    .implied_opt_name = "qom-type",
+    .head = QTAILQ_HEAD_INITIALIZER(qemu_object_opts.head),
+    .desc = {
+        { }
+    },
+};
+
 static int process_options(int argc, char *argv[], Error **errp)
 {
     int c;
@@ -63,6 +81,7 @@ static int process_options(int argc, char *argv[], Error **errp)
 
     static const struct option long_options[] = {
         {"help", no_argument, 0, 'h'},
+        {"object", required_argument, 0, OPTION_OBJECT},
         {"version", no_argument, 0, 'V'},
         {"trace", required_argument, NULL, 'T'},
         {0, 0, 0, 0}
@@ -88,6 +107,22 @@ static int process_options(int argc, char *argv[], Error **errp)
             g_free(trace_file);
             trace_file = trace_opt_parse(optarg);
             break;
+        case OPTION_OBJECT:
+            {
+                QemuOpts *opts;
+                const char *type;
+
+                opts = qemu_opts_parse(&qemu_object_opts,
+                                       optarg, true, &error_fatal);
+                type = qemu_opt_get(opts, "qom-type");
+
+                if (user_creatable_print_help(type, opts)) {
+                    exit(EXIT_SUCCESS);
+                }
+                user_creatable_add_opts(opts, &error_fatal);
+                qemu_opts_del(opts);
+                break;
+            }
         }
     }
     if (optind != argc) {
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 03/18] stubs: Add arch_type
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
  2019-10-17 13:01 ` [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool Kevin Wolf
  2019-10-17 13:01 ` [RFC PATCH 02/18] qemu-storage-daemon: Add --object option Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-10-17 13:01 ` [RFC PATCH 04/18] stubs: Add blk_by_qdev_id() Kevin Wolf
                   ` (19 subsequent siblings)
  22 siblings, 0 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

blockdev.c uses the arch_type constant, so before we can use the file in
tools (i.e. outside of the system emulator), we need to add a stub for
it. A new QEMU_ARCH_NONE is introduced for this case.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 include/sysemu/arch_init.h | 2 ++
 stubs/arch_type.c          | 4 ++++
 stubs/Makefile.objs        | 1 +
 3 files changed, 7 insertions(+)
 create mode 100644 stubs/arch_type.c

diff --git a/include/sysemu/arch_init.h b/include/sysemu/arch_init.h
index 62c6fe4cf1..01392dc945 100644
--- a/include/sysemu/arch_init.h
+++ b/include/sysemu/arch_init.h
@@ -24,6 +24,8 @@ enum {
     QEMU_ARCH_NIOS2 = (1 << 17),
     QEMU_ARCH_HPPA = (1 << 18),
     QEMU_ARCH_RISCV = (1 << 19),
+
+    QEMU_ARCH_NONE = (1 << 31),
 };
 
 extern const uint32_t arch_type;
diff --git a/stubs/arch_type.c b/stubs/arch_type.c
new file mode 100644
index 0000000000..fc5423bc98
--- /dev/null
+++ b/stubs/arch_type.c
@@ -0,0 +1,4 @@
+#include "qemu/osdep.h"
+#include "sysemu/arch_init.h"
+
+const uint32_t arch_type = QEMU_ARCH_NONE;
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 4a50e95ec3..9f4eb25e02 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -1,3 +1,4 @@
+stub-obj-y += arch_type.o
 stub-obj-y += bdrv-next-monitor-owned.o
 stub-obj-y += blk-commit-all.o
 stub-obj-y += blockdev-close-all-bdrv-states.o
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 04/18] stubs: Add blk_by_qdev_id()
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (2 preceding siblings ...)
  2019-10-17 13:01 ` [RFC PATCH 03/18] stubs: Add arch_type Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-11-08  9:03   ` Markus Armbruster
  2019-10-17 13:01 ` [RFC PATCH 05/18] qemu-storage-daemon: Add --blockdev option Kevin Wolf
                   ` (18 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

blockdev.c uses the blk_by_qdev_id() function, so before we can use the
file in tools (i.e. outside of the system emulator), we need to add a
stub for it. The function always returns an error.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 stubs/blk-by-qdev-id.c | 9 +++++++++
 stubs/Makefile.objs    | 1 +
 2 files changed, 10 insertions(+)
 create mode 100644 stubs/blk-by-qdev-id.c

diff --git a/stubs/blk-by-qdev-id.c b/stubs/blk-by-qdev-id.c
new file mode 100644
index 0000000000..0b6160fefa
--- /dev/null
+++ b/stubs/blk-by-qdev-id.c
@@ -0,0 +1,9 @@
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "sysemu/block-backend.h"
+
+BlockBackend *blk_by_qdev_id(const char *id, Error **errp)
+{
+    error_setg(errp, "qdev IDs/QOM paths exist only in the system emulator");
+    return NULL;
+}
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 9f4eb25e02..77fbf72576 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -1,5 +1,6 @@
 stub-obj-y += arch_type.o
 stub-obj-y += bdrv-next-monitor-owned.o
+stub-obj-y += blk-by-qdev-id.o
 stub-obj-y += blk-commit-all.o
 stub-obj-y += blockdev-close-all-bdrv-states.o
 stub-obj-y += clock-warp.o
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 05/18] qemu-storage-daemon: Add --blockdev option
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (3 preceding siblings ...)
  2019-10-17 13:01 ` [RFC PATCH 04/18] stubs: Add blk_by_qdev_id() Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-11-08 13:29   ` Markus Armbruster
  2019-10-17 13:01 ` [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option Kevin Wolf
                   ` (17 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

This adds a --blockdev option to the storage daemon that works the same
as the -blockdev option of the system emulator.

In order to be able to link with blockdev.o, we also need to change
stream.o from common-obj to block-obj, which is where all other block
jobs already are.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-storage-daemon.c | 29 +++++++++++++++++++++++++++++
 Makefile              |  5 ++++-
 Makefile.objs         |  7 +++++++
 block/Makefile.objs   |  2 +-
 4 files changed, 41 insertions(+), 2 deletions(-)

diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
index 48d6af43a6..904e3c3a46 100644
--- a/qemu-storage-daemon.c
+++ b/qemu-storage-daemon.c
@@ -28,6 +28,10 @@
 #include "crypto/init.h"
 
 #include "qapi/error.h"
+#include "qapi/qapi-visit-block-core.h"
+#include "qapi/qapi-commands-block-core.h"
+#include "qapi/qobject-input-visitor.h"
+
 #include "qemu-common.h"
 #include "qemu-version.h"
 #include "qemu/config-file.h"
@@ -53,6 +57,13 @@ static void help(void)
 "                         specify tracing options\n"
 "  -V, --version          output version information and exit\n"
 "\n"
+"  --blockdev [driver=]<driver>[,node-name=<N>][,discard=ignore|unmap]\n"
+"             [,cache.direct=on|off][,cache.no-flush=on|off]\n"
+"             [,read-only=on|off][,auto-read-only=on|off]\n"
+"             [,force-share=on|off][,detect-zeroes=on|off|unmap]\n"
+"             [,driver specific parameters...]\n"
+"                         configure a block backend\n"
+"\n"
 "  --object <properties>  define a QOM object such as 'secret' for\n"
 "                         passwords and/or encryption keys\n"
 "\n"
@@ -62,6 +73,7 @@ QEMU_HELP_BOTTOM "\n",
 
 enum {
     OPTION_OBJECT = 256,
+    OPTION_BLOCKDEV,
 };
 
 static QemuOptsList qemu_object_opts = {
@@ -82,6 +94,7 @@ static int process_options(int argc, char *argv[], Error **errp)
     static const struct option long_options[] = {
         {"help", no_argument, 0, 'h'},
         {"object", required_argument, 0, OPTION_OBJECT},
+        {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
         {"version", no_argument, 0, 'V'},
         {"trace", required_argument, NULL, 'T'},
         {0, 0, 0, 0}
@@ -123,6 +136,22 @@ static int process_options(int argc, char *argv[], Error **errp)
                 qemu_opts_del(opts);
                 break;
             }
+            break;
+        case OPTION_BLOCKDEV:
+            {
+                Visitor *v;
+                BlockdevOptions *options;
+
+                v = qobject_input_visitor_new_str(optarg, "driver",
+                                                  &error_fatal);
+
+                visit_type_BlockdevOptions(v, NULL, &options, &error_fatal);
+                visit_free(v);
+
+                qmp_blockdev_add(options, &error_fatal);
+                qapi_free_BlockdevOptions(options);
+                break;
+            }
         }
     }
     if (optind != argc) {
diff --git a/Makefile b/Makefile
index 76338d0ab4..b913d4d736 100644
--- a/Makefile
+++ b/Makefile
@@ -432,6 +432,8 @@ dummy := $(call unnest-vars,, \
                 qga-vss-dll-obj-y \
                 block-obj-y \
                 block-obj-m \
+                storage-daemon-obj-y \
+                storage-daemon-obj-m \
                 crypto-obj-y \
                 crypto-user-obj-y \
                 qom-obj-y \
@@ -469,6 +471,7 @@ TARGET_DIRS_RULES := $(foreach t, all clean install, $(addsuffix /$(t), $(TARGET
 SOFTMMU_ALL_RULES=$(filter %-softmmu/all, $(TARGET_DIRS_RULES))
 $(SOFTMMU_ALL_RULES): $(authz-obj-y)
 $(SOFTMMU_ALL_RULES): $(block-obj-y)
+$(SOFTMMU_ALL_RULES): $(storage-daemon-obj-y)
 $(SOFTMMU_ALL_RULES): $(chardev-obj-y)
 $(SOFTMMU_ALL_RULES): $(crypto-obj-y)
 $(SOFTMMU_ALL_RULES): $(io-obj-y)
@@ -558,7 +561,7 @@ qemu-img.o: qemu-img-cmds.h
 qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
-qemu-storage-daemon$(EXESUF): qemu-storage-daemon.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
+qemu-storage-daemon$(EXESUF): qemu-storage-daemon.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(storage-daemon-obj-y) $(COMMON_LDADDS)
 
 qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COMMON_LDADDS)
 
diff --git a/Makefile.objs b/Makefile.objs
index abcbd89654..00fdf54500 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -37,6 +37,13 @@ qom-obj-y = qom/
 
 io-obj-y = io/
 
+#######################################################################
+# storage-daemon-obj-y is code used by qemu-storage-daemon (these objects are
+# used for system emulation, too, but specified separately there)
+
+storage-daemon-obj-y = block/
+storage-daemon-obj-y += blockdev.o iothread.o
+
 ######################################################################
 # Target independent part of system emulation. The long term path is to
 # suppress *all* target specific code in case of system emulation, i.e. a
diff --git a/block/Makefile.objs b/block/Makefile.objs
index e394fe0b6c..d24fc93b7f 100644
--- a/block/Makefile.objs
+++ b/block/Makefile.objs
@@ -44,7 +44,7 @@ block-obj-y += crypto.o
 block-obj-y += aio_task.o
 block-obj-y += backup-top.o
 
-common-obj-y += stream.o
+block-obj-y += stream.o
 
 nfs.o-libs         := $(LIBNFS_LIBS)
 iscsi.o-cflags     := $(LIBISCSI_CFLAGS)
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (4 preceding siblings ...)
  2019-10-17 13:01 ` [RFC PATCH 05/18] qemu-storage-daemon: Add --blockdev option Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-11-06 12:51   ` Max Reitz
  2019-10-17 13:01 ` [RFC PATCH 07/18] blockdev-nbd: Boxed argument type for nbd-server-add Kevin Wolf
                   ` (16 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

Add a --nbd-server option to qemu-storage-daemon to start the built-in
NBD server right away. It maps the arguments for nbd-server-start to the
command line. Example (only with required options):

    --nbd-server addr.type=inet,addr.host=localhost,addr.port=10809

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qapi/block.json       | 18 ++++++++++++++++++
 include/block/nbd.h   |  1 +
 blockdev-nbd.c        |  5 +++++
 qemu-storage-daemon.c | 26 +++++++++++++++++++++++++-
 Makefile.objs         |  2 +-
 5 files changed, 50 insertions(+), 2 deletions(-)

diff --git a/qapi/block.json b/qapi/block.json
index 145c268bb6..7fe0cf6538 100644
--- a/qapi/block.json
+++ b/qapi/block.json
@@ -215,6 +215,24 @@
             '*id': 'str',
             '*force': 'bool' } }
 
+##
+# @NbdServerOptions:
+#
+# @addr: Address on which to listen.
+# @tls-creds: ID of the TLS credentials object (since 2.6).
+# @tls-authz: ID of the QAuthZ authorization object used to validate
+#             the client's x509 distinguished name. This object is
+#             is only resolved at time of use, so can be deleted and
+#             recreated on the fly while the NBD server is active.
+#             If missing, it will default to denying access (since 4.0).
+#
+# Since: 4.2
+##
+{ 'struct': 'NbdServerOptions',
+  'data': { 'addr': 'SocketAddress',
+            '*tls-creds': 'str',
+            '*tls-authz': 'str'} }
+
 ##
 # @nbd-server-start:
 #
diff --git a/include/block/nbd.h b/include/block/nbd.h
index 316fd705a9..2a7441491a 100644
--- a/include/block/nbd.h
+++ b/include/block/nbd.h
@@ -353,6 +353,7 @@ void nbd_client_put(NBDClient *client);
 
 void nbd_server_start(SocketAddress *addr, const char *tls_creds,
                       const char *tls_authz, Error **errp);
+void nbd_server_start_options(NbdServerOptions *arg, Error **errp);
 
 /* nbd_read
  * Reads @size bytes from @ioc. Returns 0 on success.
diff --git a/blockdev-nbd.c b/blockdev-nbd.c
index 6a8b206e1d..d4c1fd4166 100644
--- a/blockdev-nbd.c
+++ b/blockdev-nbd.c
@@ -132,6 +132,11 @@ void nbd_server_start(SocketAddress *addr, const char *tls_creds,
     nbd_server = NULL;
 }
 
+void nbd_server_start_options(NbdServerOptions *arg, Error **errp)
+{
+    nbd_server_start(arg->addr, arg->tls_creds, arg->tls_authz, errp);
+}
+
 void qmp_nbd_server_start(SocketAddressLegacy *addr,
                           bool has_tls_creds, const char *tls_creds,
                           bool has_tls_authz, const char *tls_authz,
diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
index 904e3c3a46..51882452f3 100644
--- a/qemu-storage-daemon.c
+++ b/qemu-storage-daemon.c
@@ -25,11 +25,14 @@
 #include "qemu/osdep.h"
 
 #include "block/block.h"
+#include "block/nbd.h"
 #include "crypto/init.h"
 
 #include "qapi/error.h"
-#include "qapi/qapi-visit-block-core.h"
+#include "qapi/qapi-commands-block.h"
 #include "qapi/qapi-commands-block-core.h"
+#include "qapi/qapi-visit-block.h"
+#include "qapi/qapi-visit-block-core.h"
 #include "qapi/qobject-input-visitor.h"
 
 #include "qemu-common.h"
@@ -64,6 +67,12 @@ static void help(void)
 "             [,driver specific parameters...]\n"
 "                         configure a block backend\n"
 "\n"
+"  --nbd-server addr.type=inet,addr.host=<host>,addr.port=<port>\n"
+"               [,tls-creds=<id>][,tls-authz=<id>]\n"
+"  --nbd-server addr.type=unix,addr.path=<path>\n"
+"               [,tls-creds=<id>][,tls-authz=<id>]\n"
+"                         start an NBD server for exporting block nodes\n"
+"\n"
 "  --object <properties>  define a QOM object such as 'secret' for\n"
 "                         passwords and/or encryption keys\n"
 "\n"
@@ -74,6 +83,7 @@ QEMU_HELP_BOTTOM "\n",
 enum {
     OPTION_OBJECT = 256,
     OPTION_BLOCKDEV,
+    OPTION_NBD_SERVER,
 };
 
 static QemuOptsList qemu_object_opts = {
@@ -95,6 +105,7 @@ static int process_options(int argc, char *argv[], Error **errp)
         {"help", no_argument, 0, 'h'},
         {"object", required_argument, 0, OPTION_OBJECT},
         {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
+        {"nbd-server", required_argument, 0, OPTION_NBD_SERVER},
         {"version", no_argument, 0, 'V'},
         {"trace", required_argument, NULL, 'T'},
         {0, 0, 0, 0}
@@ -152,6 +163,19 @@ static int process_options(int argc, char *argv[], Error **errp)
                 qapi_free_BlockdevOptions(options);
                 break;
             }
+        case OPTION_NBD_SERVER:
+            {
+                Visitor *v;
+                NbdServerOptions *options;
+
+                v = qobject_input_visitor_new_str(optarg, NULL, &error_fatal);
+                visit_type_NbdServerOptions(v, NULL, &options, &error_fatal);
+                visit_free(v);
+
+                nbd_server_start_options(options, &error_fatal);
+                qapi_free_NbdServerOptions(options);
+                break;
+            }
         }
     }
     if (optind != argc) {
diff --git a/Makefile.objs b/Makefile.objs
index 00fdf54500..cc262e445f 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -42,7 +42,7 @@ io-obj-y = io/
 # used for system emulation, too, but specified separately there)
 
 storage-daemon-obj-y = block/
-storage-daemon-obj-y += blockdev.o iothread.o
+storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o
 
 ######################################################################
 # Target independent part of system emulation. The long term path is to
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 07/18] blockdev-nbd: Boxed argument type for nbd-server-add
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (5 preceding siblings ...)
  2019-10-17 13:01 ` [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-10-17 13:01 ` [RFC PATCH 08/18] qemu-storage-daemon: Add --export option Kevin Wolf
                   ` (15 subsequent siblings)
  22 siblings, 0 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

Move the arguments of nbd-server-add to a new struct BlockExportNbd and
convert the command to 'boxed': true. This makes it easier to share code
with the storage daemon.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qapi/block.json    | 20 +++++++++++++++-----
 blockdev-nbd.c     | 25 ++++++++++++-------------
 monitor/hmp-cmds.c | 21 +++++++++++++++++----
 3 files changed, 44 insertions(+), 22 deletions(-)

diff --git a/qapi/block.json b/qapi/block.json
index 7fe0cf6538..567f116875 100644
--- a/qapi/block.json
+++ b/qapi/block.json
@@ -259,9 +259,9 @@
             '*tls-authz': 'str'} }
 
 ##
-# @nbd-server-add:
+# @BlockExportNbd:
 #
-# Export a block node to QEMU's embedded NBD server.
+# An NBD block export.
 #
 # @device: The device name or node name of the node to be exported
 #
@@ -270,19 +270,29 @@
 #
 # @writable: Whether clients should be able to write to the device via the
 #     NBD connection (default false).
-
+#
 # @bitmap: Also export the dirty bitmap reachable from @device, so the
 #          NBD client can use NBD_OPT_SET_META_CONTEXT with
 #          "qemu:dirty-bitmap:NAME" to inspect the bitmap. (since 4.0)
 #
+# Since: 4.2
+##
+{ 'struct': 'BlockExportNbd',
+  'data': {'device': 'str', '*name': 'str', '*writable': 'bool',
+           '*bitmap': 'str' } }
+
+##
+# @nbd-server-add:
+#
+# Export a block node to QEMU's embedded NBD server.
+#
 # Returns: error if the server is not running, or export with the same name
 #          already exists.
 #
 # Since: 1.3.0
 ##
 { 'command': 'nbd-server-add',
-  'data': {'device': 'str', '*name': 'str', '*writable': 'bool',
-           '*bitmap': 'str' } }
+  'data': 'BlockExportNbd', 'boxed': true }
 
 ##
 # @NbdServerRemoveMode:
diff --git a/blockdev-nbd.c b/blockdev-nbd.c
index d4c1fd4166..ee8262cd4c 100644
--- a/blockdev-nbd.c
+++ b/blockdev-nbd.c
@@ -148,9 +148,7 @@ void qmp_nbd_server_start(SocketAddressLegacy *addr,
     qapi_free_SocketAddress(addr_flat);
 }
 
-void qmp_nbd_server_add(const char *device, bool has_name, const char *name,
-                        bool has_writable, bool writable,
-                        bool has_bitmap, const char *bitmap, Error **errp)
+void qmp_nbd_server_add(BlockExportNbd *arg, Error **errp)
 {
     BlockDriverState *bs = NULL;
     BlockBackend *on_eject_blk;
@@ -163,18 +161,18 @@ void qmp_nbd_server_add(const char *device, bool has_name, const char *name,
         return;
     }
 
-    if (!has_name) {
-        name = device;
+    if (!arg->has_name) {
+        arg->name = arg->device;
     }
 
-    if (nbd_export_find(name)) {
-        error_setg(errp, "NBD server already has export named '%s'", name);
+    if (nbd_export_find(arg->name)) {
+        error_setg(errp, "NBD server already has export named '%s'", arg->name);
         return;
     }
 
-    on_eject_blk = blk_by_name(device);
+    on_eject_blk = blk_by_name(arg->device);
 
-    bs = bdrv_lookup_bs(device, device, errp);
+    bs = bdrv_lookup_bs(arg->device, arg->device, errp);
     if (!bs) {
         return;
     }
@@ -188,14 +186,15 @@ void qmp_nbd_server_add(const char *device, bool has_name, const char *name,
         goto out;
     }
 
-    if (!has_writable) {
-        writable = false;
+    if (!arg->has_writable) {
+        arg->writable = false;
     }
     if (bdrv_is_read_only(bs)) {
-        writable = false;
+        arg->writable = false;
     }
 
-    exp = nbd_export_new(bs, 0, len, name, NULL, bitmap, !writable, !writable,
+    exp = nbd_export_new(bs, 0, len, arg->name, NULL, arg->bitmap,
+                         !arg->writable, !arg->writable,
                          NULL, false, on_eject_blk, errp);
     if (!exp) {
         goto out;
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index b2551c16d1..22d7e6e222 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -2320,6 +2320,7 @@ void hmp_nbd_server_start(Monitor *mon, const QDict *qdict)
     Error *local_err = NULL;
     BlockInfoList *block_list, *info;
     SocketAddress *addr;
+    BlockExportNbd export;
 
     if (writable && !all) {
         error_setg(&local_err, "-w only valid together with -a");
@@ -2352,8 +2353,13 @@ void hmp_nbd_server_start(Monitor *mon, const QDict *qdict)
             continue;
         }
 
-        qmp_nbd_server_add(info->value->device, false, NULL,
-                           true, writable, false, NULL, &local_err);
+        export = (BlockExportNbd) {
+            .device         = info->value->device,
+            .has_writable   = true,
+            .writable       = writable,
+        };
+
+        qmp_nbd_server_add(&export, &local_err);
 
         if (local_err != NULL) {
             qmp_nbd_server_stop(NULL);
@@ -2374,8 +2380,15 @@ void hmp_nbd_server_add(Monitor *mon, const QDict *qdict)
     bool writable = qdict_get_try_bool(qdict, "writable", false);
     Error *local_err = NULL;
 
-    qmp_nbd_server_add(device, !!name, name, true, writable,
-                       false, NULL, &local_err);
+    BlockExportNbd export = {
+        .device         = (char *) device,
+        .has_name       = !!name,
+        .name           = (char *) name,
+        .has_writable   = true,
+        .writable       = writable,
+    };
+
+    qmp_nbd_server_add(&export, &local_err);
     hmp_handle_error(mon, &local_err);
 }
 
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 08/18] qemu-storage-daemon: Add --export option
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (6 preceding siblings ...)
  2019-10-17 13:01 ` [RFC PATCH 07/18] blockdev-nbd: Boxed argument type for nbd-server-add Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-11-06 13:11   ` Max Reitz
  2019-10-17 13:01 ` [RFC PATCH 09/18] qemu-storage-daemon: Add main loop Kevin Wolf
                   ` (14 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

Add a --export option to qemu-storage-daemon to export a block node. For
now, only NBD exports are implemented. Apart from the 'type' option
(which is the implied key), it maps the arguments for nbd-server-add to
the command line. Example:

    --export nbd,device=disk,name=test-export,writable=on

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qapi/block.json       | 27 +++++++++++++++++++++++++++
 qemu-storage-daemon.c | 31 +++++++++++++++++++++++++++++++
 2 files changed, 58 insertions(+)

diff --git a/qapi/block.json b/qapi/block.json
index 567f116875..d9b1f16fbf 100644
--- a/qapi/block.json
+++ b/qapi/block.json
@@ -481,3 +481,30 @@
 { 'event': 'QUORUM_REPORT_BAD',
   'data': { 'type': 'QuorumOpType', '*error': 'str', 'node-name': 'str',
             'sector-num': 'int', 'sectors-count': 'int' } }
+
+##
+# @BlockExportType:
+#
+# An enumeration of block export types
+#
+# @nbd: NBD export
+#
+# Since: 4.2
+##
+{ 'enum': 'BlockExportType',
+  'data': [ 'nbd' ] }
+
+##
+# @BlockExport:
+#
+# Describes a block export, i.e. how single node should be exported on an
+# external interface.
+#
+# Since: 4.2
+##
+{ 'union': 'BlockExport',
+  'base': { 'type': 'BlockExportType' },
+  'discriminator': 'type',
+  'data': {
+      'nbd': 'BlockExportNbd'
+   } }
diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
index 51882452f3..9e5f474fd0 100644
--- a/qemu-storage-daemon.c
+++ b/qemu-storage-daemon.c
@@ -67,6 +67,11 @@ static void help(void)
 "             [,driver specific parameters...]\n"
 "                         configure a block backend\n"
 "\n"
+"  --export [type=]nbd,device=<node-name>[,name=<export-name>]\n"
+"           [,writable=on|off][,bitmap=<name>]\n"
+"                         export the specified block node over NBD\n"
+"                         (requires --nbd-server)\n"
+"\n"
 "  --nbd-server addr.type=inet,addr.host=<host>,addr.port=<port>\n"
 "               [,tls-creds=<id>][,tls-authz=<id>]\n"
 "  --nbd-server addr.type=unix,addr.path=<path>\n"
@@ -84,6 +89,7 @@ enum {
     OPTION_OBJECT = 256,
     OPTION_BLOCKDEV,
     OPTION_NBD_SERVER,
+    OPTION_EXPORT,
 };
 
 static QemuOptsList qemu_object_opts = {
@@ -95,6 +101,17 @@ static QemuOptsList qemu_object_opts = {
     },
 };
 
+static void init_export(BlockExport *export, Error **errp)
+{
+    switch (export->type) {
+    case BLOCK_EXPORT_TYPE_NBD:
+        qmp_nbd_server_add(&export->u.nbd, errp);
+        break;
+    default:
+        g_assert_not_reached();
+    }
+}
+
 static int process_options(int argc, char *argv[], Error **errp)
 {
     int c;
@@ -106,6 +123,7 @@ static int process_options(int argc, char *argv[], Error **errp)
         {"object", required_argument, 0, OPTION_OBJECT},
         {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
         {"nbd-server", required_argument, 0, OPTION_NBD_SERVER},
+        {"export", required_argument, 0, OPTION_EXPORT},
         {"version", no_argument, 0, 'V'},
         {"trace", required_argument, NULL, 'T'},
         {0, 0, 0, 0}
@@ -176,6 +194,19 @@ static int process_options(int argc, char *argv[], Error **errp)
                 qapi_free_NbdServerOptions(options);
                 break;
             }
+        case OPTION_EXPORT:
+            {
+                Visitor *v;
+                BlockExport *export;
+
+                v = qobject_input_visitor_new_str(optarg, "type", &error_fatal);
+                visit_type_BlockExport(v, NULL, &export, &error_fatal);
+                visit_free(v);
+
+                init_export(export, &error_fatal);
+                qapi_free_BlockExport(export);
+                break;
+            }
         }
     }
     if (optind != argc) {
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 09/18] qemu-storage-daemon: Add main loop
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (7 preceding siblings ...)
  2019-10-17 13:01 ` [RFC PATCH 08/18] qemu-storage-daemon: Add --export option Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-11-08 16:02   ` Markus Armbruster
  2019-10-17 13:01 ` [RFC PATCH 10/18] qemu-storage-daemon: Add --chardev option Kevin Wolf
                   ` (13 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

Instead of exiting after processing all command line options, start a
main loop and keep processing events until exit is requested with a
signal (e.g. SIGINT).

Now qemu-storage-daemon can be used as an alternative for qemu-nbd that
provides a few features that were previously only available from QMP,
such as access to options only available with -blockdev and the socket
types 'vsock' and 'fd'.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-storage-daemon.c | 13 +++++++++++++
 Makefile.objs         |  2 ++
 2 files changed, 15 insertions(+)

diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
index 9e5f474fd0..099388f645 100644
--- a/qemu-storage-daemon.c
+++ b/qemu-storage-daemon.c
@@ -45,10 +45,18 @@
 #include "qemu/option.h"
 #include "qom/object_interfaces.h"
 
+#include "sysemu/runstate.h"
 #include "trace/control.h"
 
 #include <getopt.h>
 
+static bool exit_requested = false;
+
+void qemu_system_killed(int signal, pid_t pid)
+{
+    exit_requested = true;
+}
+
 static void help(void)
 {
     printf(
@@ -238,6 +246,7 @@ int main(int argc, char *argv[])
 
     error_init(argv[0]);
     qemu_init_exec_dir(argv[0]);
+    os_setup_signal_handling();
 
     module_call_init(MODULE_INIT_QOM);
     module_call_init(MODULE_INIT_TRACE);
@@ -256,5 +265,9 @@ int main(int argc, char *argv[])
         return EXIT_FAILURE;
     }
 
+    while (!exit_requested) {
+        main_loop_wait(false);
+    }
+
     return EXIT_SUCCESS;
 }
diff --git a/Makefile.objs b/Makefile.objs
index cc262e445f..b667d3f07b 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -43,6 +43,8 @@ io-obj-y = io/
 
 storage-daemon-obj-y = block/
 storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o
+storage-daemon-obj-$(CONFIG_WIN32) += os-win32.o
+storage-daemon-obj-$(CONFIG_POSIX) += os-posix.o
 
 ######################################################################
 # Target independent part of system emulation. The long term path is to
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 10/18] qemu-storage-daemon: Add --chardev option
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (8 preceding siblings ...)
  2019-10-17 13:01 ` [RFC PATCH 09/18] qemu-storage-daemon: Add main loop Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-11-08 16:27   ` Markus Armbruster
  2019-10-17 13:01 ` [RFC PATCH 11/18] monitor: Move monitor option parsing to monitor/monitor.c Kevin Wolf
                   ` (12 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

This adds a --chardev option to the storage daemon that works the same
as the -chardev option of the system emulator.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qemu-storage-daemon.c | 19 +++++++++++++++++++
 Makefile              |  2 +-
 2 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
index 099388f645..46e0a6ea56 100644
--- a/qemu-storage-daemon.c
+++ b/qemu-storage-daemon.c
@@ -26,6 +26,7 @@
 
 #include "block/block.h"
 #include "block/nbd.h"
+#include "chardev/char.h"
 #include "crypto/init.h"
 
 #include "qapi/error.h"
@@ -75,6 +76,9 @@ static void help(void)
 "             [,driver specific parameters...]\n"
 "                         configure a block backend\n"
 "\n"
+"  --chardev <options>    configure a character device backend\n"
+"                         (see the qemu(1) man page for possible options)\n"
+"\n"
 "  --export [type=]nbd,device=<node-name>[,name=<export-name>]\n"
 "           [,writable=on|off][,bitmap=<name>]\n"
 "                         export the specified block node over NBD\n"
@@ -96,10 +100,13 @@ QEMU_HELP_BOTTOM "\n",
 enum {
     OPTION_OBJECT = 256,
     OPTION_BLOCKDEV,
+    OPTION_CHARDEV,
     OPTION_NBD_SERVER,
     OPTION_EXPORT,
 };
 
+extern QemuOptsList qemu_chardev_opts;
+
 static QemuOptsList qemu_object_opts = {
     .name = "object",
     .implied_opt_name = "qom-type",
@@ -130,6 +137,7 @@ static int process_options(int argc, char *argv[], Error **errp)
         {"help", no_argument, 0, 'h'},
         {"object", required_argument, 0, OPTION_OBJECT},
         {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
+        {"chardev", required_argument, 0, OPTION_CHARDEV},
         {"nbd-server", required_argument, 0, OPTION_NBD_SERVER},
         {"export", required_argument, 0, OPTION_EXPORT},
         {"version", no_argument, 0, 'V'},
@@ -189,6 +197,17 @@ static int process_options(int argc, char *argv[], Error **errp)
                 qapi_free_BlockdevOptions(options);
                 break;
             }
+        case OPTION_CHARDEV:
+            {
+                QemuOpts *opts = qemu_opts_parse(&qemu_chardev_opts,
+                                                 optarg, true, &error_fatal);
+                if (!qemu_chr_new_from_opts(opts, NULL, &error_fatal)) {
+                    /* No error, but NULL returned means help was printed */
+                    exit(EXIT_SUCCESS);
+                }
+                qemu_opts_del(opts);
+                break;
+            }
         case OPTION_NBD_SERVER:
             {
                 Visitor *v;
diff --git a/Makefile b/Makefile
index b913d4d736..0e3e98582d 100644
--- a/Makefile
+++ b/Makefile
@@ -561,7 +561,7 @@ qemu-img.o: qemu-img-cmds.h
 qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
 qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
-qemu-storage-daemon$(EXESUF): qemu-storage-daemon.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(storage-daemon-obj-y) $(COMMON_LDADDS)
+qemu-storage-daemon$(EXESUF): qemu-storage-daemon.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(chardev-obj-y) $(io-obj-y) $(qom-obj-y) $(storage-daemon-obj-y) $(COMMON_LDADDS)
 
 qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COMMON_LDADDS)
 
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 11/18] monitor: Move monitor option parsing to monitor/monitor.c
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (9 preceding siblings ...)
  2019-10-17 13:01 ` [RFC PATCH 10/18] qemu-storage-daemon: Add --chardev option Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-10-17 13:01 ` [RFC PATCH 12/18] stubs: Update monitor stubs for qemu-storage-daemon Kevin Wolf
                   ` (11 subsequent siblings)
  22 siblings, 0 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

Both the system emulators and the storage daemon will need to parse
monitor options, so move that code to monitor/monitor.c, which can be
linked into binaries that aren't a system emulator.

This patch moves the monitor option parsing from vl.c and adds an
allow_hmp parameter so that callers can support QMP without HMP.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 include/monitor/monitor.h |  4 +++
 include/sysemu/sysemu.h   |  1 -
 monitor/monitor.c         | 52 +++++++++++++++++++++++++++++++++++++++
 vl.c                      | 45 +--------------------------------
 4 files changed, 57 insertions(+), 45 deletions(-)

diff --git a/include/monitor/monitor.h b/include/monitor/monitor.h
index a81eeff5f8..d3e8da36a5 100644
--- a/include/monitor/monitor.h
+++ b/include/monitor/monitor.h
@@ -3,6 +3,7 @@
 
 #include "block/block.h"
 #include "qapi/qapi-types-misc.h"
+#include "qemu/option.h"
 #include "qemu/readline.h"
 
 extern __thread Monitor *cur_mon;
@@ -10,12 +11,15 @@ typedef struct MonitorHMP MonitorHMP;
 
 #define QMP_REQ_QUEUE_LEN_MAX 8
 
+extern QemuOptsList qemu_mon_opts;
+
 bool monitor_cur_is_qmp(void);
 
 void monitor_init_globals(void);
 void monitor_init_globals_core(void);
 void monitor_init_qmp(Chardev *chr, bool pretty);
 void monitor_init_hmp(Chardev *chr, bool use_readline);
+int monitor_init_opts(QemuOpts *opts, bool allow_hmp, Error **errp);
 void monitor_cleanup(void);
 
 int monitor_suspend(Monitor *mon);
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 44f18eb739..e54934c311 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -124,7 +124,6 @@ extern QemuOptsList qemu_netdev_opts;
 extern QemuOptsList qemu_nic_opts;
 extern QemuOptsList qemu_net_opts;
 extern QemuOptsList qemu_global_opts;
-extern QemuOptsList qemu_mon_opts;
 extern QemuOptsList qemu_semihosting_config_opts;
 
 #endif
diff --git a/monitor/monitor.c b/monitor/monitor.c
index 12898b6448..316b71b928 100644
--- a/monitor/monitor.c
+++ b/monitor/monitor.c
@@ -609,6 +609,58 @@ void monitor_init_globals_core(void)
                                    NULL);
 }
 
+int monitor_init_opts(QemuOpts *opts, bool allow_hmp, Error **errp)
+{
+    Chardev *chr;
+    bool qmp;
+    bool pretty = false;
+    const char *chardev;
+    const char *mode;
+
+    mode = qemu_opt_get(opts, "mode");
+    if (mode == NULL) {
+        mode = allow_hmp ? "readline" : "control";
+    }
+    if (strcmp(mode, "readline") == 0) {
+        qmp = false;
+    } else if (strcmp(mode, "control") == 0) {
+        qmp = true;
+    } else {
+        error_setg(errp, "unknown monitor mode \"%s\"", mode);
+        return -1;
+    }
+    if (!allow_hmp && !qmp) {
+        error_setg(errp, "Only QMP is supported");
+        return -1;
+    }
+
+    if (!qmp && qemu_opt_get(opts, "pretty")) {
+        warn_report("'pretty' is deprecated for HMP monitors, it has no effect "
+                    "and will be removed in future versions");
+    }
+    if (qemu_opt_get_bool(opts, "pretty", 0)) {
+        pretty = true;
+    }
+
+    chardev = qemu_opt_get(opts, "chardev");
+    if (!chardev) {
+        error_report("chardev is required");
+        exit(1);
+    }
+    chr = qemu_chr_find(chardev);
+    if (chr == NULL) {
+        error_setg(errp, "chardev \"%s\" not found", chardev);
+        return -1;
+    }
+
+    if (qmp) {
+        monitor_init_qmp(chr, pretty);
+    } else {
+        monitor_init_hmp(chr, true);
+    }
+    return 0;
+}
+
 QemuOptsList qemu_mon_opts = {
     .name = "mon",
     .implied_opt_name = "chardev",
diff --git a/vl.c b/vl.c
index 4489cfb2bb..e9d9a44acf 100644
--- a/vl.c
+++ b/vl.c
@@ -2234,50 +2234,7 @@ static int fsdev_init_func(void *opaque, QemuOpts *opts, Error **errp)
 
 static int mon_init_func(void *opaque, QemuOpts *opts, Error **errp)
 {
-    Chardev *chr;
-    bool qmp;
-    bool pretty = false;
-    const char *chardev;
-    const char *mode;
-
-    mode = qemu_opt_get(opts, "mode");
-    if (mode == NULL) {
-        mode = "readline";
-    }
-    if (strcmp(mode, "readline") == 0) {
-        qmp = false;
-    } else if (strcmp(mode, "control") == 0) {
-        qmp = true;
-    } else {
-        error_setg(errp, "unknown monitor mode \"%s\"", mode);
-        return -1;
-    }
-
-    if (!qmp && qemu_opt_get(opts, "pretty")) {
-        warn_report("'pretty' is deprecated for HMP monitors, it has no effect "
-                    "and will be removed in future versions");
-    }
-    if (qemu_opt_get_bool(opts, "pretty", 0)) {
-        pretty = true;
-    }
-
-    chardev = qemu_opt_get(opts, "chardev");
-    if (!chardev) {
-        error_report("chardev is required");
-        exit(1);
-    }
-    chr = qemu_chr_find(chardev);
-    if (chr == NULL) {
-        error_setg(errp, "chardev \"%s\" not found", chardev);
-        return -1;
-    }
-
-    if (qmp) {
-        monitor_init_qmp(chr, pretty);
-    } else {
-        monitor_init_hmp(chr, true);
-    }
-    return 0;
+    return monitor_init_opts(opts, true, errp);
 }
 
 static void monitor_parse(const char *optarg, const char *mode, bool pretty)
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 12/18] stubs: Update monitor stubs for qemu-storage-daemon
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (10 preceding siblings ...)
  2019-10-17 13:01 ` [RFC PATCH 11/18] monitor: Move monitor option parsing to monitor/monitor.c Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-11-08 16:45   ` Markus Armbruster
  2019-10-17 13:01 ` [RFC PATCH 13/18] qapi: Create module 'monitor' Kevin Wolf
                   ` (10 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

Before we can add the monitor to qemu-storage-daemon, we need to add a
few monitor stubs, and we need to make sure that stubs that are actually
implemented in the monitor core aren't linked so that we don't get
linker errors because of duplicate symbols.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 stubs/monitor-core.c | 21 +++++++++++++++++++++
 stubs/monitor.c      | 15 ++-------------
 stubs/Makefile.objs  |  1 +
 3 files changed, 24 insertions(+), 13 deletions(-)
 create mode 100644 stubs/monitor-core.c

diff --git a/stubs/monitor-core.c b/stubs/monitor-core.c
new file mode 100644
index 0000000000..403c00a6d0
--- /dev/null
+++ b/stubs/monitor-core.c
@@ -0,0 +1,21 @@
+#include "qemu/osdep.h"
+#include "monitor/monitor.h"
+#include "qemu-common.h"
+#include "qapi/qapi-emit-events.h"
+
+__thread Monitor *cur_mon;
+
+void monitor_init_qmp(Chardev *chr, bool pretty)
+{
+}
+
+void qapi_event_emit(QAPIEvent event, QDict *qdict)
+{
+}
+
+int monitor_vprintf(Monitor *mon, const char *fmt, va_list ap)
+{
+    abort();
+}
+
+
diff --git a/stubs/monitor.c b/stubs/monitor.c
index c3e9a2e4dc..9403f8e72c 100644
--- a/stubs/monitor.c
+++ b/stubs/monitor.c
@@ -1,14 +1,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
-#include "qapi/qapi-emit-events.h"
 #include "monitor/monitor.h"
-
-__thread Monitor *cur_mon;
-
-int monitor_vprintf(Monitor *mon, const char *fmt, va_list ap)
-{
-    abort();
-}
+#include "../monitor/monitor-internal.h"
 
 int monitor_get_fd(Monitor *mon, const char *name, Error **errp)
 {
@@ -16,14 +9,10 @@ int monitor_get_fd(Monitor *mon, const char *name, Error **errp)
     return -1;
 }
 
-void monitor_init_qmp(Chardev *chr, bool pretty)
-{
-}
-
 void monitor_init_hmp(Chardev *chr, bool use_readline)
 {
 }
 
-void qapi_event_emit(QAPIEvent event, QDict *qdict)
+void monitor_fdsets_cleanup(void)
 {
 }
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 77fbf72576..ad4515ac70 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -19,6 +19,7 @@ stub-obj-y += machine-init-done.o
 stub-obj-y += migr-blocker.o
 stub-obj-y += change-state-handler.o
 stub-obj-y += monitor.o
+stub-obj-y += monitor-core.o
 stub-obj-y += notify-event.o
 stub-obj-y += qtest.o
 stub-obj-y += replay.o
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 13/18] qapi: Create module 'monitor'
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (11 preceding siblings ...)
  2019-10-17 13:01 ` [RFC PATCH 12/18] stubs: Update monitor stubs for qemu-storage-daemon Kevin Wolf
@ 2019-10-17 13:01 ` Kevin Wolf
  2019-11-11  9:36   ` Markus Armbruster
  2019-10-17 13:02 ` [RFC PATCH 14/18] monitor: Create monitor/qmp-cmds-monitor.c Kevin Wolf
                   ` (9 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:01 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

misc.json contains definitions that are related to the system emulator,
so it can't be used for the storage daemon. This patch moves basic
functionality that is related to the monitor itself into a new
monitor.json, which could be used in tools as well.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qapi/misc.json             | 212 ------------------------------------
 qapi/monitor.json          | 218 +++++++++++++++++++++++++++++++++++++
 qapi/qapi-schema.json      |   1 +
 monitor/monitor-internal.h |   1 +
 monitor/hmp-cmds.c         |   1 +
 monitor/qmp-cmds.c         |   1 +
 monitor/qmp.c              |   2 +-
 tests/qmp-test.c           |   2 +-
 ui/gtk.c                   |   1 +
 qapi/Makefile.objs         |   4 +-
 10 files changed, 227 insertions(+), 216 deletions(-)
 create mode 100644 qapi/monitor.json

diff --git a/qapi/misc.json b/qapi/misc.json
index 6bd11f50e6..7a05c286d5 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -7,159 +7,6 @@
 
 { 'include': 'common.json' }
 
-##
-# @qmp_capabilities:
-#
-# Enable QMP capabilities.
-#
-# Arguments:
-#
-# @enable:   An optional list of QMPCapability values to enable.  The
-#            client must not enable any capability that is not
-#            mentioned in the QMP greeting message.  If the field is not
-#            provided, it means no QMP capabilities will be enabled.
-#            (since 2.12)
-#
-# Example:
-#
-# -> { "execute": "qmp_capabilities",
-#      "arguments": { "enable": [ "oob" ] } }
-# <- { "return": {} }
-#
-# Notes: This command is valid exactly when first connecting: it must be
-# issued before any other command will be accepted, and will fail once the
-# monitor is accepting other commands. (see qemu docs/interop/qmp-spec.txt)
-#
-# The QMP client needs to explicitly enable QMP capabilities, otherwise
-# all the QMP capabilities will be turned off by default.
-#
-# Since: 0.13
-#
-##
-{ 'command': 'qmp_capabilities',
-  'data': { '*enable': [ 'QMPCapability' ] },
-  'allow-preconfig': true }
-
-##
-# @QMPCapability:
-#
-# Enumeration of capabilities to be advertised during initial client
-# connection, used for agreeing on particular QMP extension behaviors.
-#
-# @oob:   QMP ability to support out-of-band requests.
-#         (Please refer to qmp-spec.txt for more information on OOB)
-#
-# Since: 2.12
-#
-##
-{ 'enum': 'QMPCapability',
-  'data': [ 'oob' ] }
-
-##
-# @VersionTriple:
-#
-# A three-part version number.
-#
-# @major:  The major version number.
-#
-# @minor:  The minor version number.
-#
-# @micro:  The micro version number.
-#
-# Since: 2.4
-##
-{ 'struct': 'VersionTriple',
-  'data': {'major': 'int', 'minor': 'int', 'micro': 'int'} }
-
-
-##
-# @VersionInfo:
-#
-# A description of QEMU's version.
-#
-# @qemu:        The version of QEMU.  By current convention, a micro
-#               version of 50 signifies a development branch.  A micro version
-#               greater than or equal to 90 signifies a release candidate for
-#               the next minor version.  A micro version of less than 50
-#               signifies a stable release.
-#
-# @package:     QEMU will always set this field to an empty string.  Downstream
-#               versions of QEMU should set this to a non-empty string.  The
-#               exact format depends on the downstream however it highly
-#               recommended that a unique name is used.
-#
-# Since: 0.14.0
-##
-{ 'struct': 'VersionInfo',
-  'data': {'qemu': 'VersionTriple', 'package': 'str'} }
-
-##
-# @query-version:
-#
-# Returns the current version of QEMU.
-#
-# Returns:  A @VersionInfo object describing the current version of QEMU.
-#
-# Since: 0.14.0
-#
-# Example:
-#
-# -> { "execute": "query-version" }
-# <- {
-#       "return":{
-#          "qemu":{
-#             "major":0,
-#             "minor":11,
-#             "micro":5
-#          },
-#          "package":""
-#       }
-#    }
-#
-##
-{ 'command': 'query-version', 'returns': 'VersionInfo',
-  'allow-preconfig': true }
-
-##
-# @CommandInfo:
-#
-# Information about a QMP command
-#
-# @name: The command name
-#
-# Since: 0.14.0
-##
-{ 'struct': 'CommandInfo', 'data': {'name': 'str'} }
-
-##
-# @query-commands:
-#
-# Return a list of supported QMP commands by this server
-#
-# Returns: A list of @CommandInfo for all supported commands
-#
-# Since: 0.14.0
-#
-# Example:
-#
-# -> { "execute": "query-commands" }
-# <- {
-#      "return":[
-#         {
-#            "name":"query-balloon"
-#         },
-#         {
-#            "name":"system_powerdown"
-#         }
-#      ]
-#    }
-#
-# Note: This example has been shortened as the real response is too long.
-#
-##
-{ 'command': 'query-commands', 'returns': ['CommandInfo'],
-  'allow-preconfig': true }
-
 ##
 # @LostTickPolicy:
 #
@@ -300,48 +147,6 @@
 ##
 { 'command': 'query-uuid', 'returns': 'UuidInfo', 'allow-preconfig': true }
 
-##
-# @EventInfo:
-#
-# Information about a QMP event
-#
-# @name: The event name
-#
-# Since: 1.2.0
-##
-{ 'struct': 'EventInfo', 'data': {'name': 'str'} }
-
-##
-# @query-events:
-#
-# Return information on QMP events.
-#
-# Returns: A list of @EventInfo.
-#
-# Since: 1.2.0
-#
-# Note: This command is deprecated, because its output doesn't reflect
-# compile-time configuration.  Use query-qmp-schema instead.
-#
-# Example:
-#
-# -> { "execute": "query-events" }
-# <- {
-#      "return": [
-#          {
-#             "name":"SHUTDOWN"
-#          },
-#          {
-#             "name":"RESET"
-#          }
-#       ]
-#    }
-#
-# Note: This example has been shortened as the real response is too long.
-#
-##
-{ 'command': 'query-events', 'returns': ['EventInfo'] }
-
 ##
 # @IOThreadInfo:
 #
@@ -764,23 +569,6 @@
 ##
 { 'command': 'query-pci', 'returns': ['PciInfo'] }
 
-##
-# @quit:
-#
-# This command will cause the QEMU process to exit gracefully.  While every
-# attempt is made to send the QMP response before terminating, this is not
-# guaranteed.  When using this interface, a premature EOF would not be
-# unexpected.
-#
-# Since: 0.14.0
-#
-# Example:
-#
-# -> { "execute": "quit" }
-# <- { "return": {} }
-##
-{ 'command': 'quit' }
-
 ##
 # @stop:
 #
diff --git a/qapi/monitor.json b/qapi/monitor.json
new file mode 100644
index 0000000000..a82a18da1a
--- /dev/null
+++ b/qapi/monitor.json
@@ -0,0 +1,218 @@
+# -*- Mode: Python -*-
+#
+
+##
+# = Monitor definitions (shared between system emulator and tools)
+##
+
+##
+# @qmp_capabilities:
+#
+# Enable QMP capabilities.
+#
+# Arguments:
+#
+# @enable:   An optional list of QMPCapability values to enable.  The
+#            client must not enable any capability that is not
+#            mentioned in the QMP greeting message.  If the field is not
+#            provided, it means no QMP capabilities will be enabled.
+#            (since 2.12)
+#
+# Example:
+#
+# -> { "execute": "qmp_capabilities",
+#      "arguments": { "enable": [ "oob" ] } }
+# <- { "return": {} }
+#
+# Notes: This command is valid exactly when first connecting: it must be
+# issued before any other command will be accepted, and will fail once the
+# monitor is accepting other commands. (see qemu docs/interop/qmp-spec.txt)
+#
+# The QMP client needs to explicitly enable QMP capabilities, otherwise
+# all the QMP capabilities will be turned off by default.
+#
+# Since: 0.13
+#
+##
+{ 'command': 'qmp_capabilities',
+  'data': { '*enable': [ 'QMPCapability' ] },
+  'allow-preconfig': true }
+
+##
+# @QMPCapability:
+#
+# Enumeration of capabilities to be advertised during initial client
+# connection, used for agreeing on particular QMP extension behaviors.
+#
+# @oob:   QMP ability to support out-of-band requests.
+#         (Please refer to qmp-spec.txt for more information on OOB)
+#
+# Since: 2.12
+#
+##
+{ 'enum': 'QMPCapability',
+  'data': [ 'oob' ] }
+
+##
+# @VersionTriple:
+#
+# A three-part version number.
+#
+# @major:  The major version number.
+#
+# @minor:  The minor version number.
+#
+# @micro:  The micro version number.
+#
+# Since: 2.4
+##
+{ 'struct': 'VersionTriple',
+  'data': {'major': 'int', 'minor': 'int', 'micro': 'int'} }
+
+
+##
+# @VersionInfo:
+#
+# A description of QEMU's version.
+#
+# @qemu:        The version of QEMU.  By current convention, a micro
+#               version of 50 signifies a development branch.  A micro version
+#               greater than or equal to 90 signifies a release candidate for
+#               the next minor version.  A micro version of less than 50
+#               signifies a stable release.
+#
+# @package:     QEMU will always set this field to an empty string.  Downstream
+#               versions of QEMU should set this to a non-empty string.  The
+#               exact format depends on the downstream however it highly
+#               recommended that a unique name is used.
+#
+# Since: 0.14.0
+##
+{ 'struct': 'VersionInfo',
+  'data': {'qemu': 'VersionTriple', 'package': 'str'} }
+
+##
+# @query-version:
+#
+# Returns the current version of QEMU.
+#
+# Returns:  A @VersionInfo object describing the current version of QEMU.
+#
+# Since: 0.14.0
+#
+# Example:
+#
+# -> { "execute": "query-version" }
+# <- {
+#       "return":{
+#          "qemu":{
+#             "major":0,
+#             "minor":11,
+#             "micro":5
+#          },
+#          "package":""
+#       }
+#    }
+#
+##
+{ 'command': 'query-version', 'returns': 'VersionInfo',
+  'allow-preconfig': true }
+
+##
+# @CommandInfo:
+#
+# Information about a QMP command
+#
+# @name: The command name
+#
+# Since: 0.14.0
+##
+{ 'struct': 'CommandInfo', 'data': {'name': 'str'} }
+
+##
+# @query-commands:
+#
+# Return a list of supported QMP commands by this server
+#
+# Returns: A list of @CommandInfo for all supported commands
+#
+# Since: 0.14.0
+#
+# Example:
+#
+# -> { "execute": "query-commands" }
+# <- {
+#      "return":[
+#         {
+#            "name":"query-balloon"
+#         },
+#         {
+#            "name":"system_powerdown"
+#         }
+#      ]
+#    }
+#
+# Note: This example has been shortened as the real response is too long.
+#
+##
+{ 'command': 'query-commands', 'returns': ['CommandInfo'],
+  'allow-preconfig': true }
+
+##
+# @EventInfo:
+#
+# Information about a QMP event
+#
+# @name: The event name
+#
+# Since: 1.2.0
+##
+{ 'struct': 'EventInfo', 'data': {'name': 'str'} }
+
+##
+# @query-events:
+#
+# Return information on QMP events.
+#
+# Returns: A list of @EventInfo.
+#
+# Since: 1.2.0
+#
+# Note: This command is deprecated, because its output doesn't reflect
+# compile-time configuration.  Use query-qmp-schema instead.
+#
+# Example:
+#
+# -> { "execute": "query-events" }
+# <- {
+#      "return": [
+#          {
+#             "name":"SHUTDOWN"
+#          },
+#          {
+#             "name":"RESET"
+#          }
+#       ]
+#    }
+#
+# Note: This example has been shortened as the real response is too long.
+#
+##
+{ 'command': 'query-events', 'returns': ['EventInfo'] }
+
+##
+# @quit:
+#
+# This command will cause the QEMU process to exit gracefully.  While every
+# attempt is made to send the QMP response before terminating, this is not
+# guaranteed.  When using this interface, a premature EOF would not be
+# unexpected.
+#
+# Since: 0.14.0
+#
+# Example:
+#
+# -> { "execute": "quit" }
+# <- { "return": {} }
+##
+{ 'command': 'quit' }
diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json
index 9751b11f8f..be90422ffe 100644
--- a/qapi/qapi-schema.json
+++ b/qapi/qapi-schema.json
@@ -103,6 +103,7 @@
 { 'include': 'qdev.json' }
 { 'include': 'machine.json' }
 { 'include': 'machine-target.json' }
+{ 'include': 'monitor.json' }
 { 'include': 'misc.json' }
 { 'include': 'misc-target.json' }
 { 'include': 'audio.json' }
diff --git a/monitor/monitor-internal.h b/monitor/monitor-internal.h
index d78f5ca190..451aa64c1a 100644
--- a/monitor/monitor-internal.h
+++ b/monitor/monitor-internal.h
@@ -27,6 +27,7 @@
 
 #include "chardev/char-fe.h"
 #include "monitor/monitor.h"
+#include "qapi/qapi-types-monitor.h"
 #include "qapi/qmp/dispatch.h"
 #include "qapi/qmp/json-parser.h"
 #include "qemu/readline.h"
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index 22d7e6e222..ccdfb547d6 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -33,6 +33,7 @@
 #include "qapi/qapi-commands-char.h"
 #include "qapi/qapi-commands-migration.h"
 #include "qapi/qapi-commands-misc.h"
+#include "qapi/qapi-commands-monitor.h"
 #include "qapi/qapi-commands-net.h"
 #include "qapi/qapi-commands-rocker.h"
 #include "qapi/qapi-commands-run-state.h"
diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
index 0880341a2d..b40a9b74de 100644
--- a/monitor/qmp-cmds.c
+++ b/monitor/qmp-cmds.c
@@ -34,6 +34,7 @@
 #include "qapi/qapi-commands-block-core.h"
 #include "qapi/qapi-commands-machine.h"
 #include "qapi/qapi-commands-misc.h"
+#include "qapi/qapi-commands-monitor.h"
 #include "qapi/qapi-commands-ui.h"
 #include "qapi/qmp/qerror.h"
 #include "hw/mem/memory-device.h"
diff --git a/monitor/qmp.c b/monitor/qmp.c
index 9d9e5d8b27..6605f35048 100644
--- a/monitor/qmp.c
+++ b/monitor/qmp.c
@@ -27,7 +27,7 @@
 #include "chardev/char-io.h"
 #include "monitor-internal.h"
 #include "qapi/error.h"
-#include "qapi/qapi-commands-misc.h"
+#include "qapi/qapi-commands-monitor.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qjson.h"
 #include "qapi/qmp/qlist.h"
diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index 1b0eb69832..7fc646ffc2 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -13,7 +13,7 @@
 #include "qemu/osdep.h"
 #include "libqtest.h"
 #include "qapi/error.h"
-#include "qapi/qapi-visit-misc.h"
+#include "qapi/qapi-visit-monitor.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qlist.h"
 #include "qapi/qobject-input-visitor.h"
diff --git a/ui/gtk.c b/ui/gtk.c
index 2f23922afc..d96a11deea 100644
--- a/ui/gtk.c
+++ b/ui/gtk.c
@@ -33,6 +33,7 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qapi/qapi-commands-misc.h"
+#include "qapi/qapi-commands-monitor.h"
 #include "qemu/cutils.h"
 
 #include "ui/console.h"
diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
index dd3f5e6f94..519b6f1a8e 100644
--- a/qapi/Makefile.objs
+++ b/qapi/Makefile.objs
@@ -6,8 +6,8 @@ util-obj-y += qmp-event.o
 util-obj-y += qapi-util.o
 
 QAPI_COMMON_MODULES = audio authz block-core block char common crypto
-QAPI_COMMON_MODULES += dump error introspect job machine migration misc net
-QAPI_COMMON_MODULES += qdev qom rdma rocker run-state sockets tpm
+QAPI_COMMON_MODULES += dump error introspect job machine migration misc monitor
+QAPI_COMMON_MODULES += net qdev qom rdma rocker run-state sockets tpm
 QAPI_COMMON_MODULES += trace transaction ui
 QAPI_TARGET_MODULES = machine-target misc-target
 QAPI_MODULES = $(QAPI_COMMON_MODULES) $(QAPI_TARGET_MODULES)
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 14/18] monitor: Create monitor/qmp-cmds-monitor.c
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (12 preceding siblings ...)
  2019-10-17 13:01 ` [RFC PATCH 13/18] qapi: Create module 'monitor' Kevin Wolf
@ 2019-10-17 13:02 ` Kevin Wolf
  2019-10-17 13:02 ` [RFC PATCH 15/18] qapi: Support empty modules Kevin Wolf
                   ` (8 subsequent siblings)
  22 siblings, 0 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:02 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

Move all of the QMP commands handlers to implement the 'monitor' module
(qapi/monitor.json) that can be shared between the system emulator and
the storage daemon to a new file monitor/qmp-cmds-monitor.c.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 monitor/misc.c             | 110 --------------------------
 monitor/qmp-cmds-monitor.c | 153 +++++++++++++++++++++++++++++++++++++
 monitor/qmp-cmds.c         |  14 ----
 monitor/Makefile.objs      |   3 +-
 4 files changed, 155 insertions(+), 125 deletions(-)
 create mode 100644 monitor/qmp-cmds-monitor.c

diff --git a/monitor/misc.c b/monitor/misc.c
index aef16f6cfb..95c87df09b 100644
--- a/monitor/misc.c
+++ b/monitor/misc.c
@@ -67,7 +67,6 @@
 #include "qemu/thread.h"
 #include "block/qapi.h"
 #include "qapi/qapi-commands.h"
-#include "qapi/qapi-emit-events.h"
 #include "qapi/error.h"
 #include "qapi/qmp-event.h"
 #include "qapi/qapi-introspect.h"
@@ -230,58 +229,6 @@ static void hmp_info_help(Monitor *mon, const QDict *qdict)
     help_cmd(mon, "info");
 }
 
-static void query_commands_cb(QmpCommand *cmd, void *opaque)
-{
-    CommandInfoList *info, **list = opaque;
-
-    if (!cmd->enabled) {
-        return;
-    }
-
-    info = g_malloc0(sizeof(*info));
-    info->value = g_malloc0(sizeof(*info->value));
-    info->value->name = g_strdup(cmd->name);
-    info->next = *list;
-    *list = info;
-}
-
-CommandInfoList *qmp_query_commands(Error **errp)
-{
-    CommandInfoList *list = NULL;
-    MonitorQMP *mon;
-
-    assert(monitor_is_qmp(cur_mon));
-    mon = container_of(cur_mon, MonitorQMP, common);
-
-    qmp_for_each_command(mon->commands, query_commands_cb, &list);
-
-    return list;
-}
-
-EventInfoList *qmp_query_events(Error **errp)
-{
-    /*
-     * TODO This deprecated command is the only user of
-     * QAPIEvent_str() and QAPIEvent_lookup[].  When the command goes,
-     * they should go, too.
-     */
-    EventInfoList *info, *ev_list = NULL;
-    QAPIEvent e;
-
-    for (e = 0 ; e < QAPI_EVENT__MAX ; e++) {
-        const char *event_name = QAPIEvent_str(e);
-        assert(event_name != NULL);
-        info = g_malloc0(sizeof(*info));
-        info->value = g_malloc0(sizeof(*info->value));
-        info->value->name = g_strdup(event_name);
-
-        info->next = ev_list;
-        ev_list = info;
-    }
-
-    return ev_list;
-}
-
 /*
  * Minor hack: generated marshalling suppressed for this command
  * ('gen': false in the schema) so we can parse the JSON string
@@ -320,63 +267,6 @@ static void monitor_init_qmp_commands(void)
                          qmp_marshal_qmp_capabilities, QCO_ALLOW_PRECONFIG);
 }
 
-/*
- * Accept QMP capabilities in @list for @mon.
- * On success, set mon->qmp.capab[], and return true.
- * On error, set @errp, and return false.
- */
-static bool qmp_caps_accept(MonitorQMP *mon, QMPCapabilityList *list,
-                            Error **errp)
-{
-    GString *unavailable = NULL;
-    bool capab[QMP_CAPABILITY__MAX];
-
-    memset(capab, 0, sizeof(capab));
-
-    for (; list; list = list->next) {
-        if (!mon->capab_offered[list->value]) {
-            if (!unavailable) {
-                unavailable = g_string_new(QMPCapability_str(list->value));
-            } else {
-                g_string_append_printf(unavailable, ", %s",
-                                      QMPCapability_str(list->value));
-            }
-        }
-        capab[list->value] = true;
-    }
-
-    if (unavailable) {
-        error_setg(errp, "Capability %s not available", unavailable->str);
-        g_string_free(unavailable, true);
-        return false;
-    }
-
-    memcpy(mon->capab, capab, sizeof(capab));
-    return true;
-}
-
-void qmp_qmp_capabilities(bool has_enable, QMPCapabilityList *enable,
-                          Error **errp)
-{
-    MonitorQMP *mon;
-
-    assert(monitor_is_qmp(cur_mon));
-    mon = container_of(cur_mon, MonitorQMP, common);
-
-    if (mon->commands == &qmp_commands) {
-        error_set(errp, ERROR_CLASS_COMMAND_NOT_FOUND,
-                  "Capabilities negotiation is already complete, command "
-                  "ignored");
-        return;
-    }
-
-    if (!qmp_caps_accept(mon, enable, errp)) {
-        return;
-    }
-
-    mon->commands = &qmp_commands;
-}
-
 /* Set the current CPU defined by the user. Callers must hold BQL. */
 int monitor_set_cpu(int cpu_index)
 {
diff --git a/monitor/qmp-cmds-monitor.c b/monitor/qmp-cmds-monitor.c
new file mode 100644
index 0000000000..acebfd3716
--- /dev/null
+++ b/monitor/qmp-cmds-monitor.c
@@ -0,0 +1,153 @@
+/*
+ * QMP commands related to the monitor (common functions for sysemu and tools)
+ *
+ * Copyright (c) 2003-2004 Fabrice Bellard
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+ * THE SOFTWARE.
+ */
+
+#include "qemu/osdep.h"
+
+#include "monitor-internal.h"
+#include "qemu-version.h"
+#include "qapi/error.h"
+#include "qapi/qapi-commands-monitor.h"
+#include "qapi/qapi-emit-events.h"
+
+/*
+ * Accept QMP capabilities in @list for @mon.
+ * On success, set mon->qmp.capab[], and return true.
+ * On error, set @errp, and return false.
+ */
+static bool qmp_caps_accept(MonitorQMP *mon, QMPCapabilityList *list,
+                            Error **errp)
+{
+    GString *unavailable = NULL;
+    bool capab[QMP_CAPABILITY__MAX];
+
+    memset(capab, 0, sizeof(capab));
+
+    for (; list; list = list->next) {
+        if (!mon->capab_offered[list->value]) {
+            if (!unavailable) {
+                unavailable = g_string_new(QMPCapability_str(list->value));
+            } else {
+                g_string_append_printf(unavailable, ", %s",
+                                      QMPCapability_str(list->value));
+            }
+        }
+        capab[list->value] = true;
+    }
+
+    if (unavailable) {
+        error_setg(errp, "Capability %s not available", unavailable->str);
+        g_string_free(unavailable, true);
+        return false;
+    }
+
+    memcpy(mon->capab, capab, sizeof(capab));
+    return true;
+}
+
+void qmp_qmp_capabilities(bool has_enable, QMPCapabilityList *enable,
+                          Error **errp)
+{
+    MonitorQMP *mon;
+
+    assert(monitor_is_qmp(cur_mon));
+    mon = container_of(cur_mon, MonitorQMP, common);
+
+    if (mon->commands == &qmp_commands) {
+        error_set(errp, ERROR_CLASS_COMMAND_NOT_FOUND,
+                  "Capabilities negotiation is already complete, command "
+                  "ignored");
+        return;
+    }
+
+    if (!qmp_caps_accept(mon, enable, errp)) {
+        return;
+    }
+
+    mon->commands = &qmp_commands;
+}
+
+VersionInfo *qmp_query_version(Error **errp)
+{
+    VersionInfo *info = g_new0(VersionInfo, 1);
+
+    info->qemu = g_new0(VersionTriple, 1);
+    info->qemu->major = QEMU_VERSION_MAJOR;
+    info->qemu->minor = QEMU_VERSION_MINOR;
+    info->qemu->micro = QEMU_VERSION_MICRO;
+    info->package = g_strdup(QEMU_PKGVERSION);
+
+    return info;
+}
+
+static void query_commands_cb(QmpCommand *cmd, void *opaque)
+{
+    CommandInfoList *info, **list = opaque;
+
+    if (!cmd->enabled) {
+        return;
+    }
+
+    info = g_malloc0(sizeof(*info));
+    info->value = g_malloc0(sizeof(*info->value));
+    info->value->name = g_strdup(cmd->name);
+    info->next = *list;
+    *list = info;
+}
+
+CommandInfoList *qmp_query_commands(Error **errp)
+{
+    CommandInfoList *list = NULL;
+    MonitorQMP *mon;
+
+    assert(monitor_is_qmp(cur_mon));
+    mon = container_of(cur_mon, MonitorQMP, common);
+
+    qmp_for_each_command(mon->commands, query_commands_cb, &list);
+
+    return list;
+}
+
+EventInfoList *qmp_query_events(Error **errp)
+{
+    /*
+     * TODO This deprecated command is the only user of
+     * QAPIEvent_str() and QAPIEvent_lookup[].  When the command goes,
+     * they should go, too.
+     */
+    EventInfoList *info, *ev_list = NULL;
+    QAPIEvent e;
+
+    for (e = 0 ; e < QAPI_EVENT__MAX ; e++) {
+        const char *event_name = QAPIEvent_str(e);
+        assert(event_name != NULL);
+        info = g_malloc0(sizeof(*info));
+        info->value = g_malloc0(sizeof(*info->value));
+        info->value->name = g_strdup(event_name);
+
+        info->next = ev_list;
+        ev_list = info;
+    }
+
+    return ev_list;
+}
diff --git a/monitor/qmp-cmds.c b/monitor/qmp-cmds.c
index b40a9b74de..22a3375194 100644
--- a/monitor/qmp-cmds.c
+++ b/monitor/qmp-cmds.c
@@ -15,7 +15,6 @@
 
 #include "qemu/osdep.h"
 #include "qemu-common.h"
-#include "qemu-version.h"
 #include "qemu/cutils.h"
 #include "qemu/option.h"
 #include "monitor/monitor.h"
@@ -52,19 +51,6 @@ NameInfo *qmp_query_name(Error **errp)
     return info;
 }
 
-VersionInfo *qmp_query_version(Error **errp)
-{
-    VersionInfo *info = g_new0(VersionInfo, 1);
-
-    info->qemu = g_new0(VersionTriple, 1);
-    info->qemu->major = QEMU_VERSION_MAJOR;
-    info->qemu->minor = QEMU_VERSION_MINOR;
-    info->qemu->micro = QEMU_VERSION_MICRO;
-    info->package = g_strdup(QEMU_PKGVERSION);
-
-    return info;
-}
-
 KvmInfo *qmp_query_kvm(Error **errp)
 {
     KvmInfo *info = g_malloc0(sizeof(*info));
diff --git a/monitor/Makefile.objs b/monitor/Makefile.objs
index e91a8581cd..15eb6380c5 100644
--- a/monitor/Makefile.objs
+++ b/monitor/Makefile.objs
@@ -1,3 +1,4 @@
 obj-y += misc.o
 common-obj-y += monitor.o qmp.o hmp.o
-common-obj-y += qmp-cmds.o hmp-cmds.o
+common-obj-y += qmp-cmds.o qmp-cmds-monitor.o
+common-obj-y += hmp-cmds.o
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 15/18] qapi: Support empty modules
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (13 preceding siblings ...)
  2019-10-17 13:02 ` [RFC PATCH 14/18] monitor: Create monitor/qmp-cmds-monitor.c Kevin Wolf
@ 2019-10-17 13:02 ` Kevin Wolf
  2019-11-12  8:29   ` Markus Armbruster
  2019-10-17 13:02 ` [RFC PATCH 16/18] qapi: Create 'pragma' module Kevin Wolf
                   ` (7 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:02 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

If you added an include file that doesn't contain any definitions, no
source files would be generated for it. However, in other source files,
you would still get an #include for the header files of the empty
module.

The intended behaviour is that empty source files are created for empty
modules. This patch makes QAPISchema keep a list of all modules
(including empty ones) and modifies visit() to first visit all modules
in that list.

Some test reference outputs need to be updated due to the additional
visitor calls.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 scripts/qapi/schema.py                   | 9 +++++++++
 tests/qapi-schema/comments.out           | 2 ++
 tests/qapi-schema/doc-bad-section.out    | 2 ++
 tests/qapi-schema/doc-good.out           | 2 ++
 tests/qapi-schema/empty.out              | 2 ++
 tests/qapi-schema/event-case.out         | 2 ++
 tests/qapi-schema/include-repetition.out | 4 ++++
 tests/qapi-schema/include-simple.out     | 3 +++
 tests/qapi-schema/indented-expr.out      | 2 ++
 tests/qapi-schema/qapi-schema-test.out   | 4 ++++
 10 files changed, 32 insertions(+)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 38041098bd..e1b034d67d 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -749,6 +749,7 @@ class QAPISchema(object):
         self.docs = parser.docs
         self._entity_list = []
         self._entity_dict = {}
+        self._modules = [os.path.basename(fname)]
         self._predefining = True
         self._def_predefineds()
         self._predefining = False
@@ -800,6 +801,8 @@ class QAPISchema(object):
             main_info = main_info.parent
         fname = os.path.relpath(include, os.path.dirname(main_info.fname))
         self._def_entity(QAPISchemaInclude(fname, info))
+        if fname not in self._modules:
+            self._modules.append(fname)
 
     def _def_builtin_type(self, name, json_type, c_type):
         self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
@@ -1033,6 +1036,12 @@ class QAPISchema(object):
         visitor.visit_begin(self)
         module = None
         visitor.visit_module(module)
+
+        # Make sure that all modules are visited, even if they contain no
+        # entities
+        for module in self._modules:
+            visitor.visit_module(module)
+
         for entity in self._entity_list:
             if visitor.visit_needed(entity):
                 if entity.module != module:
diff --git a/tests/qapi-schema/comments.out b/tests/qapi-schema/comments.out
index 273f0f54e1..fa7e95d1cc 100644
--- a/tests/qapi-schema/comments.out
+++ b/tests/qapi-schema/comments.out
@@ -1,4 +1,6 @@
 module None
+module comments.json
+module None
 object q_empty
 enum QType
     prefix QTYPE
diff --git a/tests/qapi-schema/doc-bad-section.out b/tests/qapi-schema/doc-bad-section.out
index 367e2a1c3e..331237cfbe 100644
--- a/tests/qapi-schema/doc-bad-section.out
+++ b/tests/qapi-schema/doc-bad-section.out
@@ -1,4 +1,6 @@
 module None
+module doc-bad-section.json
+module None
 object q_empty
 enum QType
     prefix QTYPE
diff --git a/tests/qapi-schema/doc-good.out b/tests/qapi-schema/doc-good.out
index d3bca343eb..8f3577bb21 100644
--- a/tests/qapi-schema/doc-good.out
+++ b/tests/qapi-schema/doc-good.out
@@ -1,4 +1,6 @@
 module None
+module doc-good.json
+module None
 object q_empty
 enum QType
     prefix QTYPE
diff --git a/tests/qapi-schema/empty.out b/tests/qapi-schema/empty.out
index 5b53d00702..3671cbbe59 100644
--- a/tests/qapi-schema/empty.out
+++ b/tests/qapi-schema/empty.out
@@ -1,4 +1,6 @@
 module None
+module empty.json
+module None
 object q_empty
 enum QType
     prefix QTYPE
diff --git a/tests/qapi-schema/event-case.out b/tests/qapi-schema/event-case.out
index ec8a1406e4..2b2d8548e9 100644
--- a/tests/qapi-schema/event-case.out
+++ b/tests/qapi-schema/event-case.out
@@ -1,4 +1,6 @@
 module None
+module event-case.json
+module None
 object q_empty
 enum QType
     prefix QTYPE
diff --git a/tests/qapi-schema/include-repetition.out b/tests/qapi-schema/include-repetition.out
index 5423983239..ebaac1813d 100644
--- a/tests/qapi-schema/include-repetition.out
+++ b/tests/qapi-schema/include-repetition.out
@@ -1,4 +1,8 @@
 module None
+module include-repetition.json
+module comments.json
+module include-repetition-sub.json
+module None
 object q_empty
 enum QType
     prefix QTYPE
diff --git a/tests/qapi-schema/include-simple.out b/tests/qapi-schema/include-simple.out
index 061f81e509..dea51f9738 100644
--- a/tests/qapi-schema/include-simple.out
+++ b/tests/qapi-schema/include-simple.out
@@ -1,4 +1,7 @@
 module None
+module include-simple.json
+module include-simple-sub.json
+module None
 object q_empty
 enum QType
     prefix QTYPE
diff --git a/tests/qapi-schema/indented-expr.out b/tests/qapi-schema/indented-expr.out
index bffdf6756d..d4cffb9c1b 100644
--- a/tests/qapi-schema/indented-expr.out
+++ b/tests/qapi-schema/indented-expr.out
@@ -1,4 +1,6 @@
 module None
+module indented-expr.json
+module None
 object q_empty
 enum QType
     prefix QTYPE
diff --git a/tests/qapi-schema/qapi-schema-test.out b/tests/qapi-schema/qapi-schema-test.out
index 98031da96f..93c944a2fb 100644
--- a/tests/qapi-schema/qapi-schema-test.out
+++ b/tests/qapi-schema/qapi-schema-test.out
@@ -1,4 +1,8 @@
 module None
+module qapi-schema-test.json
+module include/sub-module.json
+module sub-sub-module.json
+module None
 object q_empty
 enum QType
     prefix QTYPE
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 16/18] qapi: Create 'pragma' module
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (14 preceding siblings ...)
  2019-10-17 13:02 ` [RFC PATCH 15/18] qapi: Support empty modules Kevin Wolf
@ 2019-10-17 13:02 ` Kevin Wolf
  2019-11-12  9:41   ` Markus Armbruster
  2019-10-17 13:02 ` [RFC PATCH 17/18] monitor: Move qmp_query_qmp_schema to qmp-cmds-monitor.c Kevin Wolf
                   ` (6 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:02 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

We want to share the whitelists between the system emulator schema and
the storage daemon schema, so move all the pragmas from the main schema
file into a separate file that can be included from both.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 qapi/pragma.json      | 24 ++++++++++++++++++++++++
 qapi/qapi-schema.json | 25 +------------------------
 qapi/Makefile.objs    |  2 +-
 3 files changed, 26 insertions(+), 25 deletions(-)
 create mode 100644 qapi/pragma.json

diff --git a/qapi/pragma.json b/qapi/pragma.json
new file mode 100644
index 0000000000..cffae27666
--- /dev/null
+++ b/qapi/pragma.json
@@ -0,0 +1,24 @@
+{ 'pragma': { 'doc-required': true } }
+
+# Whitelists to permit QAPI rule violations; think twice before you
+# add to them!
+{ 'pragma': {
+    # Commands allowed to return a non-dictionary:
+    'returns-whitelist': [
+        'human-monitor-command',
+        'qom-get',
+        'query-migrate-cache-size',
+        'query-tpm-models',
+        'query-tpm-types',
+        'ringbuf-read' ],
+    'name-case-whitelist': [
+        'ACPISlotType',             # DIMM, visible through query-acpi-ospm-status
+        'CpuInfoMIPS',              # PC, visible through query-cpu
+        'CpuInfoTricore',           # PC, visible through query-cpu
+        'BlockdevVmdkSubformat',    # all members, to match VMDK spec spellings
+        'BlockdevVmdkAdapterType',  # legacyESX, to match VMDK spec spellings
+        'QapiErrorClass',           # all members, visible through errors
+        'UuidInfo',                 # UUID, visible through query-uuid
+        'X86CPURegister32',         # all members, visible indirectly through qom-get
+        'CpuInfo'                   # CPU, visible through query-cpu
+    ] } }
diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json
index be90422ffe..85b4048535 100644
--- a/qapi/qapi-schema.json
+++ b/qapi/qapi-schema.json
@@ -49,30 +49,7 @@
 #
 ##
 
-{ 'pragma': { 'doc-required': true } }
-
-# Whitelists to permit QAPI rule violations; think twice before you
-# add to them!
-{ 'pragma': {
-    # Commands allowed to return a non-dictionary:
-    'returns-whitelist': [
-        'human-monitor-command',
-        'qom-get',
-        'query-migrate-cache-size',
-        'query-tpm-models',
-        'query-tpm-types',
-        'ringbuf-read' ],
-    'name-case-whitelist': [
-        'ACPISlotType',             # DIMM, visible through query-acpi-ospm-status
-        'CpuInfoMIPS',              # PC, visible through query-cpu
-        'CpuInfoTricore',           # PC, visible through query-cpu
-        'BlockdevVmdkSubformat',    # all members, to match VMDK spec spellings
-        'BlockdevVmdkAdapterType',  # legacyESX, to match VMDK spec spellings
-        'QapiErrorClass',           # all members, visible through errors
-        'UuidInfo',                 # UUID, visible through query-uuid
-        'X86CPURegister32',         # all members, visible indirectly through qom-get
-        'CpuInfo'                   # CPU, visible through query-cpu
-    ] } }
+{ 'include': 'pragma.json' }
 
 # Documentation generated with qapi-gen.py is in source order, with
 # included sub-schemas inserted at the first include directive
diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
index 519b6f1a8e..3e04e299ed 100644
--- a/qapi/Makefile.objs
+++ b/qapi/Makefile.objs
@@ -7,7 +7,7 @@ util-obj-y += qapi-util.o
 
 QAPI_COMMON_MODULES = audio authz block-core block char common crypto
 QAPI_COMMON_MODULES += dump error introspect job machine migration misc monitor
-QAPI_COMMON_MODULES += net qdev qom rdma rocker run-state sockets tpm
+QAPI_COMMON_MODULES += net pragma qdev qom rdma rocker run-state sockets tpm
 QAPI_COMMON_MODULES += trace transaction ui
 QAPI_TARGET_MODULES = machine-target misc-target
 QAPI_MODULES = $(QAPI_COMMON_MODULES) $(QAPI_TARGET_MODULES)
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 17/18] monitor: Move qmp_query_qmp_schema to qmp-cmds-monitor.c
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (15 preceding siblings ...)
  2019-10-17 13:02 ` [RFC PATCH 16/18] qapi: Create 'pragma' module Kevin Wolf
@ 2019-10-17 13:02 ` Kevin Wolf
  2019-10-17 13:02 ` [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option Kevin Wolf
                   ` (5 subsequent siblings)
  22 siblings, 0 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:02 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

monitor/misc.c contains code that works only in the system emulator, so
it can't be linked to the storage daemon. In order to make schema
introspection available for the storage daemon, move the function to
monitor/qmp-cmds-monitor.c, which can be linked into the storage daemon.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 monitor/monitor-internal.h |  3 +++
 monitor/misc.c             | 16 ----------------
 monitor/qmp-cmds-monitor.c | 16 ++++++++++++++++
 3 files changed, 19 insertions(+), 16 deletions(-)

diff --git a/monitor/monitor-internal.h b/monitor/monitor-internal.h
index 451aa64c1a..b91cb09ee5 100644
--- a/monitor/monitor-internal.h
+++ b/monitor/monitor-internal.h
@@ -180,4 +180,7 @@ void help_cmd(Monitor *mon, const char *name);
 void handle_hmp_command(MonitorHMP *mon, const char *cmdline);
 int hmp_compare_cmd(const char *name, const char *list);
 
+void qmp_query_qmp_schema(QDict *qdict, QObject **ret_data,
+                                 Error **errp);
+
 #endif
diff --git a/monitor/misc.c b/monitor/misc.c
index 95c87df09b..4c90c6813e 100644
--- a/monitor/misc.c
+++ b/monitor/misc.c
@@ -69,7 +69,6 @@
 #include "qapi/qapi-commands.h"
 #include "qapi/error.h"
 #include "qapi/qmp-event.h"
-#include "qapi/qapi-introspect.h"
 #include "sysemu/cpus.h"
 #include "qemu/cutils.h"
 #include "tcg/tcg.h"
@@ -229,21 +228,6 @@ static void hmp_info_help(Monitor *mon, const QDict *qdict)
     help_cmd(mon, "info");
 }
 
-/*
- * Minor hack: generated marshalling suppressed for this command
- * ('gen': false in the schema) so we can parse the JSON string
- * directly into QObject instead of first parsing it with
- * visit_type_SchemaInfoList() into a SchemaInfoList, then marshal it
- * to QObject with generated output marshallers, every time.  Instead,
- * we do it in test-qobject-input-visitor.c, just to make sure
- * qapi-gen.py's output actually conforms to the schema.
- */
-static void qmp_query_qmp_schema(QDict *qdict, QObject **ret_data,
-                                 Error **errp)
-{
-    *ret_data = qobject_from_qlit(&qmp_schema_qlit);
-}
-
 static void monitor_init_qmp_commands(void)
 {
     /*
diff --git a/monitor/qmp-cmds-monitor.c b/monitor/qmp-cmds-monitor.c
index acebfd3716..7215392f3e 100644
--- a/monitor/qmp-cmds-monitor.c
+++ b/monitor/qmp-cmds-monitor.c
@@ -29,6 +29,7 @@
 #include "qapi/error.h"
 #include "qapi/qapi-commands-monitor.h"
 #include "qapi/qapi-emit-events.h"
+#include "qapi/qapi-introspect.h"
 
 /*
  * Accept QMP capabilities in @list for @mon.
@@ -151,3 +152,18 @@ EventInfoList *qmp_query_events(Error **errp)
 
     return ev_list;
 }
+
+/*
+ * Minor hack: generated marshalling suppressed for this command
+ * ('gen': false in the schema) so we can parse the JSON string
+ * directly into QObject instead of first parsing it with
+ * visit_type_SchemaInfoList() into a SchemaInfoList, then marshal it
+ * to QObject with generated output marshallers, every time.  Instead,
+ * we do it in test-qobject-input-visitor.c, just to make sure
+ * qapi-gen.py's output actually conforms to the schema.
+ */
+void qmp_query_qmp_schema(QDict *qdict, QObject **ret_data,
+                                 Error **errp)
+{
+    *ret_data = qobject_from_qlit(&qmp_schema_qlit);
+}
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (16 preceding siblings ...)
  2019-10-17 13:02 ` [RFC PATCH 17/18] monitor: Move qmp_query_qmp_schema to qmp-cmds-monitor.c Kevin Wolf
@ 2019-10-17 13:02 ` Kevin Wolf
  2019-11-06 14:32   ` Max Reitz
                     ` (2 more replies)
  2019-10-24 11:33 ` [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (4 subsequent siblings)
  22 siblings, 3 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-10-17 13:02 UTC (permalink / raw)
  To: qemu-block; +Cc: kwolf, pkrempa, qemu-devel, armbru, mreitz

This adds and parses the --monitor option, so that a QMP monitor can be
used in the storage daemon. The monitor offers commands defined in the
QAPI schema at storage-daemon/qapi/qapi-schema.json.

Signed-off-by: Kevin Wolf <kwolf@redhat.com>
---
 storage-daemon/qapi/qapi-schema.json | 15 ++++++++++++
 qemu-storage-daemon.c                | 34 ++++++++++++++++++++++++++++
 Makefile                             | 30 ++++++++++++++++++++++++
 Makefile.objs                        |  4 ++--
 monitor/Makefile.objs                |  2 ++
 qapi/Makefile.objs                   |  5 ++++
 qom/Makefile.objs                    |  1 +
 scripts/qapi/gen.py                  |  5 ++++
 storage-daemon/Makefile.objs         |  1 +
 storage-daemon/qapi/Makefile.objs    |  1 +
 10 files changed, 96 insertions(+), 2 deletions(-)
 create mode 100644 storage-daemon/qapi/qapi-schema.json
 create mode 100644 storage-daemon/Makefile.objs
 create mode 100644 storage-daemon/qapi/Makefile.objs

diff --git a/storage-daemon/qapi/qapi-schema.json b/storage-daemon/qapi/qapi-schema.json
new file mode 100644
index 0000000000..58c561ebea
--- /dev/null
+++ b/storage-daemon/qapi/qapi-schema.json
@@ -0,0 +1,15 @@
+# -*- Mode: Python -*-
+
+{ 'include': '../../qapi/pragma.json' }
+
+{ 'include': '../../qapi/block.json' }
+{ 'include': '../../qapi/block-core.json' }
+{ 'include': '../../qapi/char.json' }
+{ 'include': '../../qapi/common.json' }
+{ 'include': '../../qapi/crypto.json' }
+{ 'include': '../../qapi/introspect.json' }
+{ 'include': '../../qapi/job.json' }
+{ 'include': '../../qapi/monitor.json' }
+{ 'include': '../../qapi/qom.json' }
+{ 'include': '../../qapi/sockets.json' }
+{ 'include': '../../qapi/transaction.json' }
diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
index 46e0a6ea56..4939e6b41f 100644
--- a/qemu-storage-daemon.c
+++ b/qemu-storage-daemon.c
@@ -28,12 +28,16 @@
 #include "block/nbd.h"
 #include "chardev/char.h"
 #include "crypto/init.h"
+#include "monitor/monitor.h"
+#include "monitor/monitor-internal.h"
 
 #include "qapi/error.h"
 #include "qapi/qapi-commands-block.h"
 #include "qapi/qapi-commands-block-core.h"
+#include "qapi/qapi-commands-monitor.h"
 #include "qapi/qapi-visit-block.h"
 #include "qapi/qapi-visit-block-core.h"
+#include "qapi/qmp/qstring.h"
 #include "qapi/qobject-input-visitor.h"
 
 #include "qemu-common.h"
@@ -46,6 +50,8 @@
 #include "qemu/option.h"
 #include "qom/object_interfaces.h"
 
+#include "storage-daemon/qapi/qapi-commands.h"
+
 #include "sysemu/runstate.h"
 #include "trace/control.h"
 
@@ -58,6 +64,11 @@ void qemu_system_killed(int signal, pid_t pid)
     exit_requested = true;
 }
 
+void qmp_quit(Error **errp)
+{
+    exit_requested = true;
+}
+
 static void help(void)
 {
     printf(
@@ -101,6 +112,7 @@ enum {
     OPTION_OBJECT = 256,
     OPTION_BLOCKDEV,
     OPTION_CHARDEV,
+    OPTION_MONITOR,
     OPTION_NBD_SERVER,
     OPTION_EXPORT,
 };
@@ -116,6 +128,17 @@ static QemuOptsList qemu_object_opts = {
     },
 };
 
+static void init_qmp_commands(void)
+{
+    qmp_init_marshal(&qmp_commands);
+    qmp_register_command(&qmp_commands, "query-qmp-schema",
+                         qmp_query_qmp_schema, QCO_ALLOW_PRECONFIG);
+
+    QTAILQ_INIT(&qmp_cap_negotiation_commands);
+    qmp_register_command(&qmp_cap_negotiation_commands, "qmp_capabilities",
+                         qmp_marshal_qmp_capabilities, QCO_ALLOW_PRECONFIG);
+}
+
 static void init_export(BlockExport *export, Error **errp)
 {
     switch (export->type) {
@@ -138,6 +161,7 @@ static int process_options(int argc, char *argv[], Error **errp)
         {"object", required_argument, 0, OPTION_OBJECT},
         {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
         {"chardev", required_argument, 0, OPTION_CHARDEV},
+        {"monitor", required_argument, 0, OPTION_MONITOR},
         {"nbd-server", required_argument, 0, OPTION_NBD_SERVER},
         {"export", required_argument, 0, OPTION_EXPORT},
         {"version", no_argument, 0, 'V'},
@@ -208,6 +232,14 @@ static int process_options(int argc, char *argv[], Error **errp)
                 qemu_opts_del(opts);
                 break;
             }
+        case OPTION_MONITOR:
+            {
+                QemuOpts *opts = qemu_opts_parse(&qemu_mon_opts,
+                                                 optarg, true, &error_fatal);
+                monitor_init_opts(opts, false, &error_fatal);
+                qemu_opts_del(opts);
+                break;
+            }
         case OPTION_NBD_SERVER:
             {
                 Visitor *v;
@@ -272,6 +304,8 @@ int main(int argc, char *argv[])
     qemu_add_opts(&qemu_trace_opts);
     qcrypto_init(&error_fatal);
     bdrv_init();
+    monitor_init_globals_core();
+    init_qmp_commands();
 
     if (qemu_init_main_loop(&local_err)) {
         error_report_err(local_err);
diff --git a/Makefile b/Makefile
index 0e3e98582d..e367d2b28a 100644
--- a/Makefile
+++ b/Makefile
@@ -121,7 +121,26 @@ GENERATED_QAPI_FILES += $(QAPI_MODULES:%=qapi/qapi-events-%.c)
 GENERATED_QAPI_FILES += qapi/qapi-introspect.c qapi/qapi-introspect.h
 GENERATED_QAPI_FILES += qapi/qapi-doc.texi
 
+GENERATED_STORAGE_DAEMON_QAPI_FILES = storage-daemon/qapi/qapi-builtin-types.h
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-types.c
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-visit.h
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-visit.c
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-commands.h
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-commands.c
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-emit-events.h
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-emit-events.c
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-events.h
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-events.c
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-introspect.h
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-introspect.c
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-types.h
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-types.c
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-visit.h
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-visit.c
+GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-doc.texi
+
 generated-files-y += $(GENERATED_QAPI_FILES)
+generated-files-y += $(GENERATED_STORAGE_DAEMON_QAPI_FILES)
 
 generated-files-y += trace/generated-tcg-tracers.h
 
@@ -616,6 +635,17 @@ qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
 		"GEN","$(@:%-timestamp=%)")
 	@>$@
 
+qapi-modules-storage-daemon = \
+	$(SRC_PATH)/storage-daemon/qapi/qapi-schema.json \
+    $(QAPI_MODULES_STORAGE_DAEMON:%=$(SRC_PATH)/qapi/%.json)
+
+$(GENERATED_STORAGE_DAEMON_QAPI_FILES): storage-daemon/qapi/qapi-gen-timestamp ;
+storage-daemon/qapi/qapi-gen-timestamp: $(qapi-modules-storage-daemon) $(qapi-py)
+	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
+		-o "storage-daemon/qapi" -b $<, \
+		"GEN","$(@:%-timestamp=%)")
+	@>$@
+
 QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qapi-commands.h)
 $(qga-obj-y): $(QGALIB_GEN)
 
diff --git a/Makefile.objs b/Makefile.objs
index b667d3f07b..d4e0daddee 100644
--- a/Makefile.objs
+++ b/Makefile.objs
@@ -41,8 +41,8 @@ io-obj-y = io/
 # storage-daemon-obj-y is code used by qemu-storage-daemon (these objects are
 # used for system emulation, too, but specified separately there)
 
-storage-daemon-obj-y = block/
-storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o
+storage-daemon-obj-y = block/ monitor/ qapi/ qom/ storage-daemon/
+storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o job-qmp.o
 storage-daemon-obj-$(CONFIG_WIN32) += os-win32.o
 storage-daemon-obj-$(CONFIG_POSIX) += os-posix.o
 
diff --git a/monitor/Makefile.objs b/monitor/Makefile.objs
index 15eb6380c5..6e4ef60601 100644
--- a/monitor/Makefile.objs
+++ b/monitor/Makefile.objs
@@ -2,3 +2,5 @@ obj-y += misc.o
 common-obj-y += monitor.o qmp.o hmp.o
 common-obj-y += qmp-cmds.o qmp-cmds-monitor.o
 common-obj-y += hmp-cmds.o
+
+storage-daemon-obj-y += monitor.o qmp.o qmp-cmds-monitor.o
diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
index 3e04e299ed..03d256f0a4 100644
--- a/qapi/Makefile.objs
+++ b/qapi/Makefile.objs
@@ -30,3 +30,8 @@ obj-y += $(QAPI_TARGET_MODULES:%=qapi-events-%.o)
 obj-y += qapi-events.o
 obj-y += $(QAPI_TARGET_MODULES:%=qapi-commands-%.o)
 obj-y += qapi-commands.o
+
+QAPI_MODULES_STORAGE_DAEMON = block block-core char common crypto introspect
+QAPI_MODULES_STORAGE_DAEMON += job monitor qom sockets pragma transaction
+
+storage-daemon-obj-y += $(QAPI_MODULES_STORAGE_DAEMON:%=qapi-commands-%.o)
diff --git a/qom/Makefile.objs b/qom/Makefile.objs
index f9d77350ac..1b45d104ba 100644
--- a/qom/Makefile.objs
+++ b/qom/Makefile.objs
@@ -2,3 +2,4 @@ qom-obj-y = object.o container.o qom-qobject.o
 qom-obj-y += object_interfaces.o
 
 common-obj-$(CONFIG_SOFTMMU) += qom-hmp-cmds.o qom-qmp-cmds.o
+storage-daemon-obj-y += qom-qmp-cmds.o
diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
index 796c17c38a..c25634f673 100644
--- a/scripts/qapi/gen.py
+++ b/scripts/qapi/gen.py
@@ -44,6 +44,11 @@ class QAPIGen(object):
         return ''
 
     def write(self, output_dir):
+        # Include paths starting with ../ are used to reuse modules of the main
+        # schema in specialised schemas. Don't overwrite the files that are
+        # already generated for the main schema.
+        if self.fname.startswith('../'):
+            return
         pathname = os.path.join(output_dir, self.fname)
         dir = os.path.dirname(pathname)
         if dir:
diff --git a/storage-daemon/Makefile.objs b/storage-daemon/Makefile.objs
new file mode 100644
index 0000000000..cfe6beee52
--- /dev/null
+++ b/storage-daemon/Makefile.objs
@@ -0,0 +1 @@
+storage-daemon-obj-y += qapi/
diff --git a/storage-daemon/qapi/Makefile.objs b/storage-daemon/qapi/Makefile.objs
new file mode 100644
index 0000000000..df8946bdae
--- /dev/null
+++ b/storage-daemon/qapi/Makefile.objs
@@ -0,0 +1 @@
+storage-daemon-obj-y += qapi-commands.o qapi-introspect.o
-- 
2.20.1



^ permalink raw reply related	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (17 preceding siblings ...)
  2019-10-17 13:02 ` [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option Kevin Wolf
@ 2019-10-24 11:33 ` Kevin Wolf
  2019-10-24 13:55 ` Vladimir Sementsov-Ogievskiy
                   ` (3 subsequent siblings)
  22 siblings, 0 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-10-24 11:33 UTC (permalink / raw)
  To: qemu-block; +Cc: qemu-devel, pkrempa, armbru, mreitz

Am 17.10.2019 um 15:01 hat Kevin Wolf geschrieben:
> This series adds a new tool 'qemu-storage-daemon', which can be used to
> export and perform operations on block devices. There is some overlap
> between qemu-img/qemu-nbd and the new qemu-storage-daemon, but there are
> a few important differences:
> [...]

Ping?



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool
  2019-10-17 13:01 ` [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool Kevin Wolf
@ 2019-10-24 13:50   ` Eric Blake
  2019-11-13 14:12     ` Kevin Wolf
  2019-11-06 12:11   ` Max Reitz
  2019-11-07 16:21   ` Markus Armbruster
  2 siblings, 1 reply; 65+ messages in thread
From: Eric Blake @ 2019-10-24 13:50 UTC (permalink / raw)
  To: Kevin Wolf, qemu-block; +Cc: qemu-devel, pkrempa, armbru, mreitz

On 10/17/19 8:01 AM, Kevin Wolf wrote:
> This adds a new binary qemu-storage-daemon that doesn't yet do more than
> some typical initialisation for tools and parsing the basic command
> options --version, --help and --trace.
> 
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>   configure             |   2 +-
>   qemu-storage-daemon.c | 141 ++++++++++++++++++++++++++++++++++++++++++
>   Makefile              |   1 +
>   3 files changed, 143 insertions(+), 1 deletion(-)
>   create mode 100644 qemu-storage-daemon.c
> 
> diff --git a/configure b/configure

> +++ b/qemu-storage-daemon.c
> @@ -0,0 +1,141 @@
> +/*
> + * QEMU storage daemon
> + *
> + * Copyright (c) 2019 Kevin Wolf <kwolf@redhat.com>
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a copy
> + * of this software and associated documentation files (the "Software"), to deal
> + * in the Software without restriction, including without limitation the rights
> + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> + * copies of the Software, and to permit persons to whom the Software is
> + * furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> + * THE SOFTWARE.

Is there an intent to license this binary as BSD (by restricting sources 
that can be linked in), or is it going to end up as GPLv2+ for other 
reasons? If the latter, would it be better to license this file GPLv2+?


> + */
> +
> +#include "qemu/osdep.h"
> +
> +#include "block/block.h"
> +#include "crypto/init.h"
> +
> +#include "qapi/error.h"
> +#include "qemu-common.h"
> +#include "qemu-version.h"
> +#include "qemu/config-file.h"
> +#include "qemu/error-report.h"
> +#include "qemu/log.h"
> +#include "qemu/main-loop.h"
> +#include "qemu/module.h"
> +
> +#include "trace/control.h"
> +
> +#include <getopt.h>

Shouldn't system headers appear right after qemu/osdep.h?

> +
> +static void help(void)
> +{
> +    printf(
> +"Usage: %s [options]\n"
> +"QEMU storage daemon\n"
> +"\n"
> +"  -h, --help             display this help and exit\n"
> +"  -T, --trace [[enable=]<pattern>][,events=<file>][,file=<file>]\n"
> +"                         specify tracing options\n"
> +"  -V, --version          output version information and exit\n"
> +"\n"
> +QEMU_HELP_BOTTOM "\n",
> +    error_get_progname());
> +}
> +
> +static int process_options(int argc, char *argv[], Error **errp)
> +{
> +    int c;
> +    char *trace_file = NULL;
> +    int ret = -EINVAL;
> +
> +    static const struct option long_options[] = {
> +        {"help", no_argument, 0, 'h'},
> +        {"version", no_argument, 0, 'V'},
> +        {"trace", required_argument, NULL, 'T'},

I find it harder to maintain lists of options (which will get longer 
over time) when the order of the struct...

> +        {0, 0, 0, 0}
> +    };
> +
> +    while ((c = getopt_long(argc, argv, ":hT:V", long_options, NULL)) != -1) {

...the order of the short options...

> +        switch (c) {
> +        case '?':
> +            error_setg(errp, "Unknown option '%s'", argv[optind - 1]);
> +            goto out;
> +        case ':':
> +            error_setg(errp, "Missing option argument for '%s'",
> +                       argv[optind - 1]);
> +            goto out;
> +        case 'h':
> +            help();
> +            exit(EXIT_SUCCESS);
> +        case 'V':
> +            printf("qemu-storage-daemon version "
> +                   QEMU_FULL_VERSION "\n" QEMU_COPYRIGHT "\n");
> +            exit(EXIT_SUCCESS);
> +        case 'T':
> +            g_free(trace_file);
> +            trace_file = trace_opt_parse(optarg);
> +            break;

...and the order of the case statements are not identical. 
Case-insensitive alphabetical may be easiest (matching your shortopt 
ordering of ":hT:V").

> +        }
> +    }
> +    if (optind != argc) {
> +        error_setg(errp, "Unexpected argument: %s", argv[optind]);
> +        goto out;
> +    }
> +
> +    if (!trace_init_backends()) {
> +        error_setg(errp, "Could not initialize trace backends");
> +        goto out;
> +    }
> +    trace_init_file(trace_file);
> +    qemu_set_log(LOG_TRACE);
> +
> +    ret = 0;
> +out:
> +    g_free(trace_file);

Mark trace_file as g_auto, and you can avoid the out: label altogether.

> +    return ret;
> +}
> +
> +int main(int argc, char *argv[])
> +{
> +    Error *local_err = NULL;
> +    int ret;
> +
> +#ifdef CONFIG_POSIX
> +    signal(SIGPIPE, SIG_IGN);
> +#endif
> +
> +    error_init(argv[0]);
> +    qemu_init_exec_dir(argv[0]);
> +
> +    module_call_init(MODULE_INIT_QOM);
> +    module_call_init(MODULE_INIT_TRACE);
> +    qemu_add_opts(&qemu_trace_opts);
> +    qcrypto_init(&error_fatal);
> +    bdrv_init();
> +
> +    if (qemu_init_main_loop(&local_err)) {
> +        error_report_err(local_err);
> +        return EXIT_FAILURE;
> +    }
> +
> +    ret = process_options(argc, argv, &local_err);
> +    if (ret < 0) {
> +        error_report_err(local_err);
> +        return EXIT_FAILURE;
> +    }
> +
> +    return EXIT_SUCCESS;
> +}

Quite a trivial shell for now, but looks interesting.  Sadly, I don't 
have much time to review the rest of the series until after KVM Forum, 
which means getting this in (even as experimental) for 4.2 is at risk.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (18 preceding siblings ...)
  2019-10-24 11:33 ` [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
@ 2019-10-24 13:55 ` Vladimir Sementsov-Ogievskiy
  2019-11-14 10:44   ` Kevin Wolf
  2019-11-05 15:52 ` Stefan Hajnoczi
                   ` (2 subsequent siblings)
  22 siblings, 1 reply; 65+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2019-10-24 13:55 UTC (permalink / raw)
  To: Kevin Wolf, qemu-block; +Cc: pkrempa, qemu-devel, mreitz, armbru

Hi!

This reflects our idea of using qemu binary instead of qemu-img for doing
block-layer operations offline.

What is the practical difference between qemu-storage-daemon and starting
qemu binary in stopped state?

17.10.2019 16:01, Kevin Wolf wrote:
> This series adds a new tool 'qemu-storage-daemon', which can be used to
> export and perform operations on block devices. There is some overlap
> between qemu-img/qemu-nbd and the new qemu-storage-daemon, but there are
> a few important differences:
> 
> * The qemu-storage-daemon has QMP support. The command set is obviously
>    restricted compared to the system emulator because there is no guest,
>    but all of the block operations are present.
> 
>    This means that it can access advanced options or operations that the
>    qemu-img command line doesn't expose. For example, blockdev-create is
>    a lot more powerful than 'qemu-img create', and qemu-storage-daemon
>    allows to execute it without starting a guest.

qemu binary can do that too, with -S option..

> 
>    Compared to qemu-nbd it means that, for example, block jobs can now be
>    executed on the server side, and backing chains shared by multiple VMs
>    can be modified this way.
> 
> * The existing tools all have a separately invented one-off syntax for
>    the job at hand, which usually comes with restrictions compared to the
>    system emulator. qemu-storage-daemon shares the same syntax with the
>    system emulator for most options and prefers QAPI based interfaces
>    where possible (such as --blockdev), so it should be easy to make use
>    of in libvirt.
> 
> * While this series implements only NBD exports, the storage daemon is
>    intended to serve multiple protocols and its syntax reflects this. In
>    the past, we had proposals to add new one-off tools for exporting over
>    new protocols like FUSE or TCMU.
> 
>    With a generic storage daemon, additional export methods have a home
>    without adding a new tool for each of them.
> 
> I'm posting this as an RFC mainly for two reasons:
> 
> 1. The monitor integration, which could be argued to be a little hackish
>     (some generated QAPI source files are built from a separate QAPI
>     schema, but the per-module ones are taken from the system emulator)
>     and Markus will want to have a closer look there. But from the IRC
>     discussions we had, we seem to agree on the general approach here.
> 
> 2. I'm not completely sure if the command line syntax is the final
>     version that we want to support long-term. Many options directly use
>     QAPI visitors (--blockdev, --export, --nbd-server) and should be
>     fine. However, others use QemuOpts (--chardev, --monitor, --object).
> 
>     This is the same as in the system emulator, so we wouldn't be adding
>     a new problem, but as there was talk about QAPIfying the command
>     line, and I wouldn't want later syntax changes or adding lots of
>     compatibility code to a new tool, I thought we should probably
>     discuss whether QAPIfying from the start would be an option.
> 
> I would like to have something merged for 4.2, but I'm considering
> marking the whole tool as experimental and unstable ABI by requiring a
> command line option like --experimental for the tool to even start if we
> know that we want to change some things later.
> 
> This series is based on:
> * [PATCH] blockdev: Use error_report() in hmp_commit()
> * [PATCH 0/7] qapi: Cleanups and test speedup
> 
> Based-on: <20191015123932.12214-1-kwolf@redhat.com>
> Based-on: <20191001191514.11208-1-armbru@redhat.com>
> 
> Kevin Wolf (18):
>    qemu-storage-daemon: Add barebone tool
>    qemu-storage-daemon: Add --object option
>    stubs: Add arch_type
>    stubs: Add blk_by_qdev_id()
>    qemu-storage-daemon: Add --blockdev option
>    qemu-storage-daemon: Add --nbd-server option
>    blockdev-nbd: Boxed argument type for nbd-server-add
>    qemu-storage-daemon: Add --export option
>    qemu-storage-daemon: Add main loop
>    qemu-storage-daemon: Add --chardev option
>    monitor: Move monitor option parsing to monitor/monitor.c
>    stubs: Update monitor stubs for qemu-storage-daemon
>    qapi: Create module 'monitor'
>    monitor: Create monitor/qmp-cmds-monitor.c
>    qapi: Support empty modules
>    qapi: Create 'pragma' module
>    monitor: Move qmp_query_qmp_schema to qmp-cmds-monitor.c
>    qemu-storage-daemon: Add --monitor option
> 
>   qapi/block.json                          |  65 ++++-
>   qapi/misc.json                           | 212 ---------------
>   qapi/monitor.json                        | 218 +++++++++++++++
>   qapi/pragma.json                         |  24 ++
>   qapi/qapi-schema.json                    |  26 +-
>   storage-daemon/qapi/qapi-schema.json     |  15 ++
>   configure                                |   2 +-
>   include/block/nbd.h                      |   1 +
>   include/monitor/monitor.h                |   4 +
>   include/sysemu/arch_init.h               |   2 +
>   include/sysemu/sysemu.h                  |   1 -
>   monitor/monitor-internal.h               |   4 +
>   blockdev-nbd.c                           |  30 ++-
>   monitor/hmp-cmds.c                       |  22 +-
>   monitor/misc.c                           | 126 ---------
>   monitor/monitor.c                        |  52 ++++
>   monitor/qmp-cmds-monitor.c               | 169 ++++++++++++
>   monitor/qmp-cmds.c                       |  15 +-
>   monitor/qmp.c                            |   2 +-
>   qemu-storage-daemon.c                    | 326 +++++++++++++++++++++++
>   stubs/arch_type.c                        |   4 +
>   stubs/blk-by-qdev-id.c                   |   9 +
>   stubs/monitor-core.c                     |  21 ++
>   stubs/monitor.c                          |  15 +-
>   tests/qmp-test.c                         |   2 +-
>   ui/gtk.c                                 |   1 +
>   vl.c                                     |  45 +---
>   Makefile                                 |  34 +++
>   Makefile.objs                            |   9 +
>   block/Makefile.objs                      |   2 +-
>   monitor/Makefile.objs                    |   5 +-
>   qapi/Makefile.objs                       |   9 +-
>   qom/Makefile.objs                        |   1 +
>   scripts/qapi/gen.py                      |   5 +
>   scripts/qapi/schema.py                   |   9 +
>   storage-daemon/Makefile.objs             |   1 +
>   storage-daemon/qapi/Makefile.objs        |   1 +
>   stubs/Makefile.objs                      |   3 +
>   tests/qapi-schema/comments.out           |   2 +
>   tests/qapi-schema/doc-bad-section.out    |   2 +
>   tests/qapi-schema/doc-good.out           |   2 +
>   tests/qapi-schema/empty.out              |   2 +
>   tests/qapi-schema/event-case.out         |   2 +
>   tests/qapi-schema/include-repetition.out |   4 +
>   tests/qapi-schema/include-simple.out     |   3 +
>   tests/qapi-schema/indented-expr.out      |   2 +
>   tests/qapi-schema/qapi-schema-test.out   |   4 +
>   47 files changed, 1052 insertions(+), 463 deletions(-)
>   create mode 100644 qapi/monitor.json
>   create mode 100644 qapi/pragma.json
>   create mode 100644 storage-daemon/qapi/qapi-schema.json
>   create mode 100644 monitor/qmp-cmds-monitor.c
>   create mode 100644 qemu-storage-daemon.c
>   create mode 100644 stubs/arch_type.c
>   create mode 100644 stubs/blk-by-qdev-id.c
>   create mode 100644 stubs/monitor-core.c
>   create mode 100644 storage-daemon/Makefile.objs
>   create mode 100644 storage-daemon/qapi/Makefile.objs
> 


-- 
Best regards,
Vladimir

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (19 preceding siblings ...)
  2019-10-24 13:55 ` Vladimir Sementsov-Ogievskiy
@ 2019-11-05 15:52 ` Stefan Hajnoczi
  2019-11-06 14:37 ` Max Reitz
  2019-11-07 10:33 ` Daniel P. Berrangé
  22 siblings, 0 replies; 65+ messages in thread
From: Stefan Hajnoczi @ 2019-11-05 15:52 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: mreitz, pkrempa, qemu-devel, qemu-block, armbru

[-- Attachment #1: Type: text/plain, Size: 1967 bytes --]

On Thu, Oct 17, 2019 at 03:01:46PM +0200, Kevin Wolf wrote:
> This series adds a new tool 'qemu-storage-daemon', which can be used to
> export and perform operations on block devices. There is some overlap
> between qemu-img/qemu-nbd and the new qemu-storage-daemon, but there are
> a few important differences:
> 
> * The qemu-storage-daemon has QMP support. The command set is obviously
>   restricted compared to the system emulator because there is no guest,
>   but all of the block operations are present.
> 
>   This means that it can access advanced options or operations that the
>   qemu-img command line doesn't expose. For example, blockdev-create is
>   a lot more powerful than 'qemu-img create', and qemu-storage-daemon
>   allows to execute it without starting a guest.
> 
>   Compared to qemu-nbd it means that, for example, block jobs can now be
>   executed on the server side, and backing chains shared by multiple VMs
>   can be modified this way.
> 
> * The existing tools all have a separately invented one-off syntax for
>   the job at hand, which usually comes with restrictions compared to the
>   system emulator. qemu-storage-daemon shares the same syntax with the
>   system emulator for most options and prefers QAPI based interfaces
>   where possible (such as --blockdev), so it should be easy to make use
>   of in libvirt.
> 
> * While this series implements only NBD exports, the storage daemon is
>   intended to serve multiple protocols and its syntax reflects this. In
>   the past, we had proposals to add new one-off tools for exporting over
>   new protocols like FUSE or TCMU.
> 
>   With a generic storage daemon, additional export methods have a home
>   without adding a new tool for each of them.

No comments on the command-line, QAPI, or monitor aspects, but the
general idea of having a qemu-storage-daemon is likely to be useful.

Acked-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool
  2019-10-17 13:01 ` [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool Kevin Wolf
  2019-10-24 13:50   ` Eric Blake
@ 2019-11-06 12:11   ` Max Reitz
  2019-11-07 16:21   ` Markus Armbruster
  2 siblings, 0 replies; 65+ messages in thread
From: Max Reitz @ 2019-11-06 12:11 UTC (permalink / raw)
  To: Kevin Wolf, qemu-block; +Cc: pkrempa, armbru, qemu-devel


[-- Attachment #1.1: Type: text/plain, Size: 4408 bytes --]

On 17.10.19 15:01, Kevin Wolf wrote:
> This adds a new binary qemu-storage-daemon that doesn't yet do more than
> some typical initialisation for tools and parsing the basic command
> options --version, --help and --trace.
> 
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  configure             |   2 +-
>  qemu-storage-daemon.c | 141 ++++++++++++++++++++++++++++++++++++++++++
>  Makefile              |   1 +
>  3 files changed, 143 insertions(+), 1 deletion(-)
>  create mode 100644 qemu-storage-daemon.c
> 
> diff --git a/configure b/configure
> index 08ca4bcb46..bb3d55fb25 100755
> --- a/configure
> +++ b/configure
> @@ -6034,7 +6034,7 @@ tools=""
>  if test "$want_tools" = "yes" ; then
>    tools="qemu-img\$(EXESUF) qemu-io\$(EXESUF) qemu-edid\$(EXESUF) $tools"
>    if [ "$linux" = "yes" -o "$bsd" = "yes" -o "$solaris" = "yes" ] ; then
> -    tools="qemu-nbd\$(EXESUF) $tools"
> +    tools="qemu-nbd\$(EXESUF) qemu-storage-daemon\$(EXESUF) $tools"
>    fi
>    if [ "$ivshmem" = "yes" ]; then
>      tools="ivshmem-client\$(EXESUF) ivshmem-server\$(EXESUF) $tools"
> diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
> new file mode 100644
> index 0000000000..a251dc255c
> --- /dev/null
> +++ b/qemu-storage-daemon.c
> @@ -0,0 +1,141 @@
> +/*
> + * QEMU storage daemon
> + *
> + * Copyright (c) 2019 Kevin Wolf <kwolf@redhat.com>
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a copy
> + * of this software and associated documentation files (the "Software"), to deal
> + * in the Software without restriction, including without limitation the rights
> + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> + * copies of the Software, and to permit persons to whom the Software is
> + * furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> + * THE SOFTWARE.
> + */
> +
> +#include "qemu/osdep.h"
> +
> +#include "block/block.h"
> +#include "crypto/init.h"
> +
> +#include "qapi/error.h"
> +#include "qemu-common.h"
> +#include "qemu-version.h"
> +#include "qemu/config-file.h"
> +#include "qemu/error-report.h"
> +#include "qemu/log.h"
> +#include "qemu/main-loop.h"
> +#include "qemu/module.h"
> +
> +#include "trace/control.h"
> +
> +#include <getopt.h>
> +
> +static void help(void)
> +{
> +    printf(
> +"Usage: %s [options]\n"
> +"QEMU storage daemon\n"
> +"\n"
> +"  -h, --help             display this help and exit\n"
> +"  -T, --trace [[enable=]<pattern>][,events=<file>][,file=<file>]\n"
> +"                         specify tracing options\n"
> +"  -V, --version          output version information and exit\n"
> +"\n"
> +QEMU_HELP_BOTTOM "\n",
> +    error_get_progname());
> +}
> +
> +static int process_options(int argc, char *argv[], Error **errp)
> +{
> +    int c;
> +    char *trace_file = NULL;
> +    int ret = -EINVAL;
> +
> +    static const struct option long_options[] = {
> +        {"help", no_argument, 0, 'h'},
> +        {"version", no_argument, 0, 'V'},
> +        {"trace", required_argument, NULL, 'T'},
> +        {0, 0, 0, 0}
> +    };
> +
> +    while ((c = getopt_long(argc, argv, ":hT:V", long_options, NULL)) != -1) {
> +        switch (c) {
> +        case '?':
> +            error_setg(errp, "Unknown option '%s'", argv[optind - 1]);
> +            goto out;

Am I doing something wrong or is optind really not updated when '?' is
returned?

Because I’m getting this:

$ ./qemu-storage-daemon -42
qemu-storage-daemon: Unknown option './qemu-storage-daemon'

But OTOH I also get:

$ ./qemu-img create -42
qemu-img: unrecognized option 'create'
Try 'qemu-img --help' for more information

So, uh, well.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option
  2019-10-17 13:01 ` [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option Kevin Wolf
@ 2019-11-06 12:51   ` Max Reitz
  2019-11-06 19:25     ` Eric Blake
  2019-11-08 15:36     ` Markus Armbruster
  0 siblings, 2 replies; 65+ messages in thread
From: Max Reitz @ 2019-11-06 12:51 UTC (permalink / raw)
  To: Kevin Wolf, qemu-block; +Cc: pkrempa, armbru, qemu-devel


[-- Attachment #1.1: Type: text/plain, Size: 7174 bytes --]

On 17.10.19 15:01, Kevin Wolf wrote:
> Add a --nbd-server option to qemu-storage-daemon to start the built-in
> NBD server right away. It maps the arguments for nbd-server-start to the
> command line.

Well, it doesn’t quite, because nbd-server-start takes a
SocketAddressLegacy, and this takes a SocketAddress.

On one hand I can understand why you would do it differently (especially
for command-line options), but on the other I find it a bit problematic
to have --nbd-server be slightly different from nbd-server-start when
both are intended to be the same.

My biggest problem though lies in the duplication in the QAPI schema.
If NbdServerOptions.addr were a SocketAddressLegacy, we could let
nbd-server-start’s options just be of type NbdServerOptions and thus get
rid of the duplication.

I suspect in practice it’s all not that big of a problem.  I can’t call
it bad if --nbd-server is just nicer to use.  And the biggest problem
with duplication in the QAPI schema is that nbd-server-start and
--nbd-server might get out of sync.  But realistically, I don’t see that
happen, because if nbd-server-start changes, nbd_server_start() will
change, too, so we’ll get compile errors in nbd_server_start_options().

*shrug*

But I do think the commit message should explain why we can’t just use
NbdServerOptions for nbd-server-start.

Max

> Example (only with required options):
> 
>     --nbd-server addr.type=inet,addr.host=localhost,addr.port=10809
> 
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  qapi/block.json       | 18 ++++++++++++++++++
>  include/block/nbd.h   |  1 +
>  blockdev-nbd.c        |  5 +++++
>  qemu-storage-daemon.c | 26 +++++++++++++++++++++++++-
>  Makefile.objs         |  2 +-
>  5 files changed, 50 insertions(+), 2 deletions(-)
> 
> diff --git a/qapi/block.json b/qapi/block.json
> index 145c268bb6..7fe0cf6538 100644
> --- a/qapi/block.json
> +++ b/qapi/block.json
> @@ -215,6 +215,24 @@
>              '*id': 'str',
>              '*force': 'bool' } }
>  
> +##
> +# @NbdServerOptions:
> +#
> +# @addr: Address on which to listen.
> +# @tls-creds: ID of the TLS credentials object (since 2.6).
> +# @tls-authz: ID of the QAuthZ authorization object used to validate
> +#             the client's x509 distinguished name. This object is
> +#             is only resolved at time of use, so can be deleted and
> +#             recreated on the fly while the NBD server is active.
> +#             If missing, it will default to denying access (since 4.0).
> +#
> +# Since: 4.2
> +##
> +{ 'struct': 'NbdServerOptions',
> +  'data': { 'addr': 'SocketAddress',
> +            '*tls-creds': 'str',
> +            '*tls-authz': 'str'} }
> +
>  ##
>  # @nbd-server-start:
>  #
> diff --git a/include/block/nbd.h b/include/block/nbd.h
> index 316fd705a9..2a7441491a 100644
> --- a/include/block/nbd.h
> +++ b/include/block/nbd.h
> @@ -353,6 +353,7 @@ void nbd_client_put(NBDClient *client);
>  
>  void nbd_server_start(SocketAddress *addr, const char *tls_creds,
>                        const char *tls_authz, Error **errp);
> +void nbd_server_start_options(NbdServerOptions *arg, Error **errp);
>  
>  /* nbd_read
>   * Reads @size bytes from @ioc. Returns 0 on success.
> diff --git a/blockdev-nbd.c b/blockdev-nbd.c
> index 6a8b206e1d..d4c1fd4166 100644
> --- a/blockdev-nbd.c
> +++ b/blockdev-nbd.c
> @@ -132,6 +132,11 @@ void nbd_server_start(SocketAddress *addr, const char *tls_creds,
>      nbd_server = NULL;
>  }
>  
> +void nbd_server_start_options(NbdServerOptions *arg, Error **errp)
> +{
> +    nbd_server_start(arg->addr, arg->tls_creds, arg->tls_authz, errp);
> +}
> +
>  void qmp_nbd_server_start(SocketAddressLegacy *addr,
>                            bool has_tls_creds, const char *tls_creds,
>                            bool has_tls_authz, const char *tls_authz,
> diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
> index 904e3c3a46..51882452f3 100644
> --- a/qemu-storage-daemon.c
> +++ b/qemu-storage-daemon.c
> @@ -25,11 +25,14 @@
>  #include "qemu/osdep.h"
>  
>  #include "block/block.h"
> +#include "block/nbd.h"
>  #include "crypto/init.h"
>  
>  #include "qapi/error.h"
> -#include "qapi/qapi-visit-block-core.h"
> +#include "qapi/qapi-commands-block.h"
>  #include "qapi/qapi-commands-block-core.h"
> +#include "qapi/qapi-visit-block.h"
> +#include "qapi/qapi-visit-block-core.h"
>  #include "qapi/qobject-input-visitor.h"
>  
>  #include "qemu-common.h"
> @@ -64,6 +67,12 @@ static void help(void)
>  "             [,driver specific parameters...]\n"
>  "                         configure a block backend\n"
>  "\n"
> +"  --nbd-server addr.type=inet,addr.host=<host>,addr.port=<port>\n"
> +"               [,tls-creds=<id>][,tls-authz=<id>]\n"
> +"  --nbd-server addr.type=unix,addr.path=<path>\n"
> +"               [,tls-creds=<id>][,tls-authz=<id>]\n"
> +"                         start an NBD server for exporting block nodes\n"
> +"\n"
>  "  --object <properties>  define a QOM object such as 'secret' for\n"
>  "                         passwords and/or encryption keys\n"
>  "\n"
> @@ -74,6 +83,7 @@ QEMU_HELP_BOTTOM "\n",
>  enum {
>      OPTION_OBJECT = 256,
>      OPTION_BLOCKDEV,
> +    OPTION_NBD_SERVER,
>  };
>  
>  static QemuOptsList qemu_object_opts = {
> @@ -95,6 +105,7 @@ static int process_options(int argc, char *argv[], Error **errp)
>          {"help", no_argument, 0, 'h'},
>          {"object", required_argument, 0, OPTION_OBJECT},
>          {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
> +        {"nbd-server", required_argument, 0, OPTION_NBD_SERVER},
>          {"version", no_argument, 0, 'V'},
>          {"trace", required_argument, NULL, 'T'},
>          {0, 0, 0, 0}
> @@ -152,6 +163,19 @@ static int process_options(int argc, char *argv[], Error **errp)
>                  qapi_free_BlockdevOptions(options);
>                  break;
>              }
> +        case OPTION_NBD_SERVER:
> +            {
> +                Visitor *v;
> +                NbdServerOptions *options;
> +
> +                v = qobject_input_visitor_new_str(optarg, NULL, &error_fatal);
> +                visit_type_NbdServerOptions(v, NULL, &options, &error_fatal);
> +                visit_free(v);
> +
> +                nbd_server_start_options(options, &error_fatal);
> +                qapi_free_NbdServerOptions(options);
> +                break;
> +            }
>          }
>      }
>      if (optind != argc) {
> diff --git a/Makefile.objs b/Makefile.objs
> index 00fdf54500..cc262e445f 100644
> --- a/Makefile.objs
> +++ b/Makefile.objs
> @@ -42,7 +42,7 @@ io-obj-y = io/
>  # used for system emulation, too, but specified separately there)
>  
>  storage-daemon-obj-y = block/
> -storage-daemon-obj-y += blockdev.o iothread.o
> +storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o
>  
>  ######################################################################
>  # Target independent part of system emulation. The long term path is to
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 08/18] qemu-storage-daemon: Add --export option
  2019-10-17 13:01 ` [RFC PATCH 08/18] qemu-storage-daemon: Add --export option Kevin Wolf
@ 2019-11-06 13:11   ` Max Reitz
  2019-11-06 13:34     ` Kevin Wolf
  0 siblings, 1 reply; 65+ messages in thread
From: Max Reitz @ 2019-11-06 13:11 UTC (permalink / raw)
  To: Kevin Wolf, qemu-block; +Cc: pkrempa, armbru, qemu-devel


[-- Attachment #1.1: Type: text/plain, Size: 893 bytes --]

On 17.10.19 15:01, Kevin Wolf wrote:
> Add a --export option to qemu-storage-daemon to export a block node. For
> now, only NBD exports are implemented. Apart from the 'type' option
> (which is the implied key), it maps the arguments for nbd-server-add to
> the command line. Example:
> 
>     --export nbd,device=disk,name=test-export,writable=on
> 
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  qapi/block.json       | 27 +++++++++++++++++++++++++++
>  qemu-storage-daemon.c | 31 +++++++++++++++++++++++++++++++
>  2 files changed, 58 insertions(+)

Would it be better to collect the BlockExports in a list and work on it
after all arguments have been parsed?  As it is, it’s important that
users define block devices and things like NBD servers before --export.
 Yes, I know, that’s exactly how it works with qemu, but is that really
the best way?

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 08/18] qemu-storage-daemon: Add --export option
  2019-11-06 13:11   ` Max Reitz
@ 2019-11-06 13:34     ` Kevin Wolf
  2019-11-06 13:39       ` Max Reitz
  2019-11-08 15:57       ` Markus Armbruster
  0 siblings, 2 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-11-06 13:34 UTC (permalink / raw)
  To: Max Reitz; +Cc: pkrempa, armbru, qemu-block, qemu-devel

[-- Attachment #1: Type: text/plain, Size: 1886 bytes --]

Am 06.11.2019 um 14:11 hat Max Reitz geschrieben:
> On 17.10.19 15:01, Kevin Wolf wrote:
> > Add a --export option to qemu-storage-daemon to export a block node. For
> > now, only NBD exports are implemented. Apart from the 'type' option
> > (which is the implied key), it maps the arguments for nbd-server-add to
> > the command line. Example:
> > 
> >     --export nbd,device=disk,name=test-export,writable=on
> > 
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> > ---
> >  qapi/block.json       | 27 +++++++++++++++++++++++++++
> >  qemu-storage-daemon.c | 31 +++++++++++++++++++++++++++++++
> >  2 files changed, 58 insertions(+)
> 
> Would it be better to collect the BlockExports in a list and work on it
> after all arguments have been parsed?  As it is, it’s important that
> users define block devices and things like NBD servers before --export.
>  Yes, I know, that’s exactly how it works with qemu, but is that really
> the best way?

It's actually not how QEMU works generally. QEMU collects things in
QemuOptsLists and then tries to interpret them in the right order. Of
course, we never get the order actually right, which results in constant
reshuffling of the order of initialisations in vl.c.

It also means that vl.c (!) has a list of -object types that need to be
created early so that other backends can make use of them, and of those
types that actually depend on a backend already being present (see
object_create_initial() for details).

I think it's much cleaner to simply use the order in the command line
instead of adding magic that tries to resolve (and fails at actually
resolving) all the dependencies. I seem to remember that this was in
fact one of the things Markus keeps mentioning he would change if he
were to rewrite the QEMU command line parser from scratch without
compatibility requirements.

Kevin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 08/18] qemu-storage-daemon: Add --export option
  2019-11-06 13:34     ` Kevin Wolf
@ 2019-11-06 13:39       ` Max Reitz
  2019-11-08 15:57       ` Markus Armbruster
  1 sibling, 0 replies; 65+ messages in thread
From: Max Reitz @ 2019-11-06 13:39 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, armbru, qemu-block, qemu-devel


[-- Attachment #1.1: Type: text/plain, Size: 1971 bytes --]

On 06.11.19 14:34, Kevin Wolf wrote:
> Am 06.11.2019 um 14:11 hat Max Reitz geschrieben:
>> On 17.10.19 15:01, Kevin Wolf wrote:
>>> Add a --export option to qemu-storage-daemon to export a block node. For
>>> now, only NBD exports are implemented. Apart from the 'type' option
>>> (which is the implied key), it maps the arguments for nbd-server-add to
>>> the command line. Example:
>>>
>>>     --export nbd,device=disk,name=test-export,writable=on
>>>
>>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>>> ---
>>>  qapi/block.json       | 27 +++++++++++++++++++++++++++
>>>  qemu-storage-daemon.c | 31 +++++++++++++++++++++++++++++++
>>>  2 files changed, 58 insertions(+)
>>
>> Would it be better to collect the BlockExports in a list and work on it
>> after all arguments have been parsed?  As it is, it’s important that
>> users define block devices and things like NBD servers before --export.
>>  Yes, I know, that’s exactly how it works with qemu, but is that really
>> the best way?
> 
> It's actually not how QEMU works generally. QEMU collects things in
> QemuOptsLists and then tries to interpret them in the right order. Of
> course, we never get the order actually right, which results in constant
> reshuffling of the order of initialisations in vl.c.
> 
> It also means that vl.c (!) has a list of -object types that need to be
> created early so that other backends can make use of them, and of those
> types that actually depend on a backend already being present (see
> object_create_initial() for details).
> 
> I think it's much cleaner to simply use the order in the command line
> instead of adding magic that tries to resolve (and fails at actually
> resolving) all the dependencies. I seem to remember that this was in
> fact one of the things Markus keeps mentioning he would change if he
> were to rewrite the QEMU command line parser from scratch without
> compatibility requirements.

OK.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option
  2019-10-17 13:02 ` [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option Kevin Wolf
@ 2019-11-06 14:32   ` Max Reitz
  2019-11-07 10:12     ` Kevin Wolf
  2019-11-08  8:59   ` Markus Armbruster
  2019-11-12 14:25   ` Markus Armbruster
  2 siblings, 1 reply; 65+ messages in thread
From: Max Reitz @ 2019-11-06 14:32 UTC (permalink / raw)
  To: Kevin Wolf, qemu-block; +Cc: pkrempa, armbru, qemu-devel


[-- Attachment #1.1: Type: text/plain, Size: 2552 bytes --]

On 17.10.19 15:02, Kevin Wolf wrote:
> This adds and parses the --monitor option, so that a QMP monitor can be
> used in the storage daemon. The monitor offers commands defined in the
> QAPI schema at storage-daemon/qapi/qapi-schema.json.
> 
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  storage-daemon/qapi/qapi-schema.json | 15 ++++++++++++
>  qemu-storage-daemon.c                | 34 ++++++++++++++++++++++++++++
>  Makefile                             | 30 ++++++++++++++++++++++++
>  Makefile.objs                        |  4 ++--
>  monitor/Makefile.objs                |  2 ++
>  qapi/Makefile.objs                   |  5 ++++
>  qom/Makefile.objs                    |  1 +
>  scripts/qapi/gen.py                  |  5 ++++
>  storage-daemon/Makefile.objs         |  1 +
>  storage-daemon/qapi/Makefile.objs    |  1 +
>  10 files changed, 96 insertions(+), 2 deletions(-)
>  create mode 100644 storage-daemon/qapi/qapi-schema.json
>  create mode 100644 storage-daemon/Makefile.objs
>  create mode 100644 storage-daemon/qapi/Makefile.objs

[...]

> diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
> index 3e04e299ed..03d256f0a4 100644
> --- a/qapi/Makefile.objs
> +++ b/qapi/Makefile.objs
> @@ -30,3 +30,8 @@ obj-y += $(QAPI_TARGET_MODULES:%=qapi-events-%.o)
>  obj-y += qapi-events.o
>  obj-y += $(QAPI_TARGET_MODULES:%=qapi-commands-%.o)
>  obj-y += qapi-commands.o
> +
> +QAPI_MODULES_STORAGE_DAEMON = block block-core char common crypto introspect
> +QAPI_MODULES_STORAGE_DAEMON += job monitor qom sockets pragma transaction

I took a look into the rest, and I wonder whether query-iothreads from
misc.json would be useful, too.

> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> index 796c17c38a..c25634f673 100644
> --- a/scripts/qapi/gen.py
> +++ b/scripts/qapi/gen.py
> @@ -44,6 +44,11 @@ class QAPIGen(object):
>          return ''
>  
>      def write(self, output_dir):
> +        # Include paths starting with ../ are used to reuse modules of the main
> +        # schema in specialised schemas. Don't overwrite the files that are
> +        # already generated for the main schema.
> +        if self.fname.startswith('../'):
> +            return

Sorry, but I’m about to ask a clueless question: How do we ensure that
the main schema is generated before something else could make sure of
the specialized schemas?

Max

>          pathname = os.path.join(output_dir, self.fname)
>          dir = os.path.dirname(pathname)
>          if dir:


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (20 preceding siblings ...)
  2019-11-05 15:52 ` Stefan Hajnoczi
@ 2019-11-06 14:37 ` Max Reitz
  2019-11-06 14:58   ` Kevin Wolf
  2019-11-07 10:33 ` Daniel P. Berrangé
  22 siblings, 1 reply; 65+ messages in thread
From: Max Reitz @ 2019-11-06 14:37 UTC (permalink / raw)
  To: Kevin Wolf, qemu-block; +Cc: pkrempa, armbru, qemu-devel


[-- Attachment #1.1: Type: text/plain, Size: 895 bytes --]

On 17.10.19 15:01, Kevin Wolf wrote:
> This series adds a new tool 'qemu-storage-daemon', which can be used to
> export and perform operations on block devices.

Looks good to me.

I remember a discussion at some KVM Forum a couple of years ago where
someone (Berto?) was asking about adding QMP to qemu-nbd.  I found it a
pragmatic solution, but I remember that Markus was against it, based on
the fact that we wanted qemu -M none.

Well, but anyway.  Just as I didn’t have anything against adding QMP to
qemu-nbd, I don’t have anything against adding a new application that
kind of fulfills the same purpose.  And I think introducing a new
application instead of reusing qemu-nbd that focuses on all-around QAPI
compatibility (which qemu-nbd decidedly does not have) makes sense.


The only thing I don’t like is the name, but that’s what <Tab> is for. :-)

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-11-06 14:37 ` Max Reitz
@ 2019-11-06 14:58   ` Kevin Wolf
  2019-11-06 15:35     ` Max Reitz
                       ` (2 more replies)
  0 siblings, 3 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-11-06 14:58 UTC (permalink / raw)
  To: Max Reitz; +Cc: pkrempa, armbru, qemu-block, qemu-devel

[-- Attachment #1: Type: text/plain, Size: 1942 bytes --]

Am 06.11.2019 um 15:37 hat Max Reitz geschrieben:
> On 17.10.19 15:01, Kevin Wolf wrote:
> > This series adds a new tool 'qemu-storage-daemon', which can be used to
> > export and perform operations on block devices.
> 
> Looks good to me.
> 
> I remember a discussion at some KVM Forum a couple of years ago where
> someone (Berto?) was asking about adding QMP to qemu-nbd.  I found it a
> pragmatic solution, but I remember that Markus was against it, based on
> the fact that we wanted qemu -M none.

Yes, but it turned out that qemu -M none is a bit too heavyweight in
practice and fixing that would involve a lot of work. As I understand it
(mostly what I took from discussions on the list), even if someone were
interested in doing that and started now, it's the kind of thing that
would take multiple years.

As long as we keep the code simple and the interesting parts are just
reused and shared with the system emulator and other tools, it shouldn't
be hard to maintain.

> Well, but anyway.  Just as I didn’t have anything against adding QMP to
> qemu-nbd, I don’t have anything against adding a new application that
> kind of fulfills the same purpose.  And I think introducing a new
> application instead of reusing qemu-nbd that focuses on all-around QAPI
> compatibility (which qemu-nbd decidedly does not have) makes sense.

Yes, QAPI is one big reason for creating a new tool that doesn't need to
support the old qemu-nbd command line. Another is that we can add other
types of exports that are not NBD.

> The only thing I don’t like is the name, but that’s what <Tab> is for.
> :-)

I'm open for suggestions, but I thought 'qsd' was a bit too terse. :-)

(Actually, maybe we could even pick something that doesn't mention
storage or block? After all, it can do all kinds of QEMU backends in
theory. Not sure if there's any standalone use for them, but who
knows...)

Kevin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-11-06 14:58   ` Kevin Wolf
@ 2019-11-06 15:35     ` Max Reitz
  2019-11-06 17:13     ` Eric Blake
  2019-11-21 10:32     ` Stefan Hajnoczi
  2 siblings, 0 replies; 65+ messages in thread
From: Max Reitz @ 2019-11-06 15:35 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, armbru, qemu-block, qemu-devel


[-- Attachment #1.1: Type: text/plain, Size: 2314 bytes --]

On 06.11.19 15:58, Kevin Wolf wrote:
> Am 06.11.2019 um 15:37 hat Max Reitz geschrieben:
>> On 17.10.19 15:01, Kevin Wolf wrote:
>>> This series adds a new tool 'qemu-storage-daemon', which can be used to
>>> export and perform operations on block devices.
>>
>> Looks good to me.
>>
>> I remember a discussion at some KVM Forum a couple of years ago where
>> someone (Berto?) was asking about adding QMP to qemu-nbd.  I found it a
>> pragmatic solution, but I remember that Markus was against it, based on
>> the fact that we wanted qemu -M none.
> 
> Yes, but it turned out that qemu -M none is a bit too heavyweight in
> practice and fixing that would involve a lot of work. As I understand it
> (mostly what I took from discussions on the list), even if someone were
> interested in doing that and started now, it's the kind of thing that
> would take multiple years.

I didn’t want to give the impression I wouldn’t agree. O:-)

(I agree completely, and basically that was my
understanding/opinion/feeling back when we discussed it, too.)

> As long as we keep the code simple and the interesting parts are just
> reused and shared with the system emulator and other tools, it shouldn't
> be hard to maintain.
> 
>> Well, but anyway.  Just as I didn’t have anything against adding QMP to
>> qemu-nbd, I don’t have anything against adding a new application that
>> kind of fulfills the same purpose.  And I think introducing a new
>> application instead of reusing qemu-nbd that focuses on all-around QAPI
>> compatibility (which qemu-nbd decidedly does not have) makes sense.
> 
> Yes, QAPI is one big reason for creating a new tool that doesn't need to
> support the old qemu-nbd command line. Another is that we can add other
> types of exports that are not NBD.

Sure.

>> The only thing I don’t like is the name, but that’s what <Tab> is for.
>> :-)
> 
> I'm open for suggestions, but I thought 'qsd' was a bit too terse. :-)
> 
> (Actually, maybe we could even pick something that doesn't mention
> storage or block? After all, it can do all kinds of QEMU backends in
> theory. Not sure if there's any standalone use for them, but who
> knows...)

Be careful, if we stuff too much into it, we’ll end up with just qemu
again. :-)

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-11-06 14:58   ` Kevin Wolf
  2019-11-06 15:35     ` Max Reitz
@ 2019-11-06 17:13     ` Eric Blake
  2019-11-21 10:32     ` Stefan Hajnoczi
  2 siblings, 0 replies; 65+ messages in thread
From: Eric Blake @ 2019-11-06 17:13 UTC (permalink / raw)
  To: Kevin Wolf, Max Reitz; +Cc: pkrempa, armbru, qemu-block, qemu-devel

On 11/6/19 8:58 AM, Kevin Wolf wrote:

>> Well, but anyway.  Just as I didn’t have anything against adding QMP to
>> qemu-nbd, I don’t have anything against adding a new application that
>> kind of fulfills the same purpose.  And I think introducing a new
>> application instead of reusing qemu-nbd that focuses on all-around QAPI
>> compatibility (which qemu-nbd decidedly does not have) makes sense.
> 
> Yes, QAPI is one big reason for creating a new tool that doesn't need to
> support the old qemu-nbd command line. Another is that we can add other
> types of exports that are not NBD.

If we could make qemu-nbd a thin wrapper around the new tool (even if 
the two are not necessarily command-line compatible), that might be 
worthwhile.

> 
>> The only thing I don’t like is the name, but that’s what <Tab> is for.
>> :-)
> 
> I'm open for suggestions, but I thought 'qsd' was a bit too terse. :-)
> 
> (Actually, maybe we could even pick something that doesn't mention
> storage or block? After all, it can do all kinds of QEMU backends in
> theory. Not sure if there's any standalone use for them, but who
> knows...)

Maybe 'qback', for qemu-backend?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option
  2019-11-06 12:51   ` Max Reitz
@ 2019-11-06 19:25     ` Eric Blake
  2019-11-07  8:33       ` Kevin Wolf
  2019-11-08 15:36     ` Markus Armbruster
  1 sibling, 1 reply; 65+ messages in thread
From: Eric Blake @ 2019-11-06 19:25 UTC (permalink / raw)
  To: Max Reitz, Kevin Wolf, qemu-block; +Cc: pkrempa, armbru, qemu-devel

On 11/6/19 6:51 AM, Max Reitz wrote:
> On 17.10.19 15:01, Kevin Wolf wrote:
>> Add a --nbd-server option to qemu-storage-daemon to start the built-in
>> NBD server right away. It maps the arguments for nbd-server-start to the
>> command line.
> 
> Well, it doesn’t quite, because nbd-server-start takes a
> SocketAddressLegacy, and this takes a SocketAddress.
> 
> On one hand I can understand why you would do it differently (especially
> for command-line options), but on the other I find it a bit problematic
> to have --nbd-server be slightly different from nbd-server-start when
> both are intended to be the same.
> 
> My biggest problem though lies in the duplication in the QAPI schema.
> If NbdServerOptions.addr were a SocketAddressLegacy, we could let
> nbd-server-start’s options just be of type NbdServerOptions and thus get
> rid of the duplication.

I would love to somehow deprecate the use of SocketAddressLegacy and get 
QMP nbd-server-start to accept SocketAddress instead.  Maybe it could be 
done by adding a new nbd-server-begin command in 5.0 with a saner wire 
layout, and deprecating nbd-server-start at that time; by the 5.2 
release, we could then drop nbd-server-start.  But we're too late for 4.2.

> 
> I suspect in practice it’s all not that big of a problem.  I can’t call
> it bad if --nbd-server is just nicer to use.  And the biggest problem
> with duplication in the QAPI schema is that nbd-server-start and
> --nbd-server might get out of sync.  But realistically, I don’t see that
> happen, because if nbd-server-start changes, nbd_server_start() will
> change, too, so we’ll get compile errors in nbd_server_start_options().
> 
> *shrug*
> 
> But I do think the commit message should explain why we can’t just use
> NbdServerOptions for nbd-server-start.
> 
> Max
> 
>> Example (only with required options):
>>
>>      --nbd-server addr.type=inet,addr.host=localhost,addr.port=10809
>>
>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>> ---
>>   qapi/block.json       | 18 ++++++++++++++++++
>>   include/block/nbd.h   |  1 +
>>   blockdev-nbd.c        |  5 +++++
>>   qemu-storage-daemon.c | 26 +++++++++++++++++++++++++-
>>   Makefile.objs         |  2 +-
>>   5 files changed, 50 insertions(+), 2 deletions(-)
>>
>> diff --git a/qapi/block.json b/qapi/block.json
>> index 145c268bb6..7fe0cf6538 100644
>> --- a/qapi/block.json
>> +++ b/qapi/block.json
>> @@ -215,6 +215,24 @@
>>               '*id': 'str',
>>               '*force': 'bool' } }
>>   
>> +##
>> +# @NbdServerOptions:
>> +#
>> +# @addr: Address on which to listen.
>> +# @tls-creds: ID of the TLS credentials object (since 2.6).
>> +# @tls-authz: ID of the QAuthZ authorization object used to validate
>> +#             the client's x509 distinguished name. This object is
>> +#             is only resolved at time of use, so can be deleted and
>> +#             recreated on the fly while the NBD server is active.
>> +#             If missing, it will default to denying access (since 4.0).
>> +#
>> +# Since: 4.2
>> +##
>> +{ 'struct': 'NbdServerOptions',
>> +  'data': { 'addr': 'SocketAddress',
>> +            '*tls-creds': 'str',
>> +            '*tls-authz': 'str'} }
>> +
>>   ##
>>   # @nbd-server-start:
>>   #
>> diff --git a/include/block/nbd.h b/include/block/nbd.h
>> index 316fd705a9..2a7441491a 100644
>> --- a/include/block/nbd.h
>> +++ b/include/block/nbd.h
>> @@ -353,6 +353,7 @@ void nbd_client_put(NBDClient *client);
>>   
>>   void nbd_server_start(SocketAddress *addr, const char *tls_creds,
>>                         const char *tls_authz, Error **errp);
>> +void nbd_server_start_options(NbdServerOptions *arg, Error **errp);
>>   
>>   /* nbd_read
>>    * Reads @size bytes from @ioc. Returns 0 on success.
>> diff --git a/blockdev-nbd.c b/blockdev-nbd.c
>> index 6a8b206e1d..d4c1fd4166 100644
>> --- a/blockdev-nbd.c
>> +++ b/blockdev-nbd.c
>> @@ -132,6 +132,11 @@ void nbd_server_start(SocketAddress *addr, const char *tls_creds,
>>       nbd_server = NULL;
>>   }
>>   
>> +void nbd_server_start_options(NbdServerOptions *arg, Error **errp)
>> +{
>> +    nbd_server_start(arg->addr, arg->tls_creds, arg->tls_authz, errp);
>> +}
>> +
>>   void qmp_nbd_server_start(SocketAddressLegacy *addr,
>>                             bool has_tls_creds, const char *tls_creds,
>>                             bool has_tls_authz, const char *tls_authz,
>> diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
>> index 904e3c3a46..51882452f3 100644
>> --- a/qemu-storage-daemon.c
>> +++ b/qemu-storage-daemon.c
>> @@ -25,11 +25,14 @@
>>   #include "qemu/osdep.h"
>>   
>>   #include "block/block.h"
>> +#include "block/nbd.h"
>>   #include "crypto/init.h"
>>   
>>   #include "qapi/error.h"
>> -#include "qapi/qapi-visit-block-core.h"
>> +#include "qapi/qapi-commands-block.h"
>>   #include "qapi/qapi-commands-block-core.h"
>> +#include "qapi/qapi-visit-block.h"
>> +#include "qapi/qapi-visit-block-core.h"
>>   #include "qapi/qobject-input-visitor.h"
>>   
>>   #include "qemu-common.h"
>> @@ -64,6 +67,12 @@ static void help(void)
>>   "             [,driver specific parameters...]\n"
>>   "                         configure a block backend\n"
>>   "\n"
>> +"  --nbd-server addr.type=inet,addr.host=<host>,addr.port=<port>\n"
>> +"               [,tls-creds=<id>][,tls-authz=<id>]\n"
>> +"  --nbd-server addr.type=unix,addr.path=<path>\n"
>> +"               [,tls-creds=<id>][,tls-authz=<id>]\n"
>> +"                         start an NBD server for exporting block nodes\n"
>> +"\n"
>>   "  --object <properties>  define a QOM object such as 'secret' for\n"
>>   "                         passwords and/or encryption keys\n"
>>   "\n"
>> @@ -74,6 +83,7 @@ QEMU_HELP_BOTTOM "\n",
>>   enum {
>>       OPTION_OBJECT = 256,
>>       OPTION_BLOCKDEV,
>> +    OPTION_NBD_SERVER,
>>   };
>>   
>>   static QemuOptsList qemu_object_opts = {
>> @@ -95,6 +105,7 @@ static int process_options(int argc, char *argv[], Error **errp)
>>           {"help", no_argument, 0, 'h'},
>>           {"object", required_argument, 0, OPTION_OBJECT},
>>           {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
>> +        {"nbd-server", required_argument, 0, OPTION_NBD_SERVER},
>>           {"version", no_argument, 0, 'V'},
>>           {"trace", required_argument, NULL, 'T'},
>>           {0, 0, 0, 0}
>> @@ -152,6 +163,19 @@ static int process_options(int argc, char *argv[], Error **errp)
>>                   qapi_free_BlockdevOptions(options);
>>                   break;
>>               }
>> +        case OPTION_NBD_SERVER:
>> +            {
>> +                Visitor *v;
>> +                NbdServerOptions *options;
>> +
>> +                v = qobject_input_visitor_new_str(optarg, NULL, &error_fatal);
>> +                visit_type_NbdServerOptions(v, NULL, &options, &error_fatal);
>> +                visit_free(v);
>> +
>> +                nbd_server_start_options(options, &error_fatal);
>> +                qapi_free_NbdServerOptions(options);
>> +                break;
>> +            }
>>           }
>>       }
>>       if (optind != argc) {
>> diff --git a/Makefile.objs b/Makefile.objs
>> index 00fdf54500..cc262e445f 100644
>> --- a/Makefile.objs
>> +++ b/Makefile.objs
>> @@ -42,7 +42,7 @@ io-obj-y = io/
>>   # used for system emulation, too, but specified separately there)
>>   
>>   storage-daemon-obj-y = block/
>> -storage-daemon-obj-y += blockdev.o iothread.o
>> +storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o
>>   
>>   ######################################################################
>>   # Target independent part of system emulation. The long term path is to
>>
> 
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option
  2019-11-06 19:25     ` Eric Blake
@ 2019-11-07  8:33       ` Kevin Wolf
  2019-11-07 13:45         ` Eric Blake
  0 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-11-07  8:33 UTC (permalink / raw)
  To: Eric Blake; +Cc: qemu-devel, pkrempa, armbru, qemu-block, Max Reitz

Am 06.11.2019 um 20:25 hat Eric Blake geschrieben:
> On 11/6/19 6:51 AM, Max Reitz wrote:
> > On 17.10.19 15:01, Kevin Wolf wrote:
> > > Add a --nbd-server option to qemu-storage-daemon to start the built-in
> > > NBD server right away. It maps the arguments for nbd-server-start to the
> > > command line.
> > 
> > Well, it doesn’t quite, because nbd-server-start takes a
> > SocketAddressLegacy, and this takes a SocketAddress.
> > 
> > On one hand I can understand why you would do it differently (especially
> > for command-line options), but on the other I find it a bit problematic
> > to have --nbd-server be slightly different from nbd-server-start when
> > both are intended to be the same.
> > 
> > My biggest problem though lies in the duplication in the QAPI schema.
> > If NbdServerOptions.addr were a SocketAddressLegacy, we could let
> > nbd-server-start’s options just be of type NbdServerOptions and thus get
> > rid of the duplication.
> 
> I would love to somehow deprecate the use of SocketAddressLegacy and get QMP
> nbd-server-start to accept SocketAddress instead.  Maybe it could be done by
> adding a new nbd-server-begin command in 5.0 with a saner wire layout, and
> deprecating nbd-server-start at that time; by the 5.2 release, we could then
> drop nbd-server-start.  But we're too late for 4.2.

As a replacement nbd-server-add, I envisioned adding something like a
block-export-add, which would work the way that --export already does.
It would also come with query-block-exports and block-export-del, and it
wouldn't contain only NBD devices, but also vhost-user, FUSE, etc.
exports.

Now I'm wondering if the same would make sense for nbd-server-start.
Maybe an API change would even allow us to start multiple NBD servers
(e.g. listening on different IP addresses or using different tls-creds).

Kevin



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option
  2019-11-06 14:32   ` Max Reitz
@ 2019-11-07 10:12     ` Kevin Wolf
  2019-11-07 10:44       ` Max Reitz
  0 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-11-07 10:12 UTC (permalink / raw)
  To: Max Reitz; +Cc: pkrempa, armbru, qemu-block, qemu-devel

[-- Attachment #1: Type: text/plain, Size: 3448 bytes --]

Am 06.11.2019 um 15:32 hat Max Reitz geschrieben:
> On 17.10.19 15:02, Kevin Wolf wrote:
> > This adds and parses the --monitor option, so that a QMP monitor can be
> > used in the storage daemon. The monitor offers commands defined in the
> > QAPI schema at storage-daemon/qapi/qapi-schema.json.
> > 
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> > ---
> >  storage-daemon/qapi/qapi-schema.json | 15 ++++++++++++
> >  qemu-storage-daemon.c                | 34 ++++++++++++++++++++++++++++
> >  Makefile                             | 30 ++++++++++++++++++++++++
> >  Makefile.objs                        |  4 ++--
> >  monitor/Makefile.objs                |  2 ++
> >  qapi/Makefile.objs                   |  5 ++++
> >  qom/Makefile.objs                    |  1 +
> >  scripts/qapi/gen.py                  |  5 ++++
> >  storage-daemon/Makefile.objs         |  1 +
> >  storage-daemon/qapi/Makefile.objs    |  1 +
> >  10 files changed, 96 insertions(+), 2 deletions(-)
> >  create mode 100644 storage-daemon/qapi/qapi-schema.json
> >  create mode 100644 storage-daemon/Makefile.objs
> >  create mode 100644 storage-daemon/qapi/Makefile.objs
> 
> [...]
> 
> > diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
> > index 3e04e299ed..03d256f0a4 100644
> > --- a/qapi/Makefile.objs
> > +++ b/qapi/Makefile.objs
> > @@ -30,3 +30,8 @@ obj-y += $(QAPI_TARGET_MODULES:%=qapi-events-%.o)
> >  obj-y += qapi-events.o
> >  obj-y += $(QAPI_TARGET_MODULES:%=qapi-commands-%.o)
> >  obj-y += qapi-commands.o
> > +
> > +QAPI_MODULES_STORAGE_DAEMON = block block-core char common crypto introspect
> > +QAPI_MODULES_STORAGE_DAEMON += job monitor qom sockets pragma transaction
> 
> I took a look into the rest, and I wonder whether query-iothreads from
> misc.json would be useful, too.

Possibly. It would be a separate patch, but I can add it.

The question is just where to move query-iothreads. Do we have a good
place, or do I need to separate misc.json and a new misc-sysemu.json?

> > diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> > index 796c17c38a..c25634f673 100644
> > --- a/scripts/qapi/gen.py
> > +++ b/scripts/qapi/gen.py
> > @@ -44,6 +44,11 @@ class QAPIGen(object):
> >          return ''
> >  
> >      def write(self, output_dir):
> > +        # Include paths starting with ../ are used to reuse modules of the main
> > +        # schema in specialised schemas. Don't overwrite the files that are
> > +        # already generated for the main schema.
> > +        if self.fname.startswith('../'):
> > +            return
> 
> Sorry, but I’m about to ask a clueless question: How do we ensure that
> the main schema is generated before something else could make sure of
> the specialized schemas?

"Make sure"?

I think the order of the generation doesn't matter because generating
the storage daemon files doesn't actually access the main ones.
Generated C files shouldn't be a problem either because if we link an
object file into a binary, we have a make dependency for it.

Maybe the only a bit trickier question is whether we have the
dependencies right so that qemu-storage-daemon.c is only built after the
header files of both the main schema and the specific one have been
generated. If I understand the Makefile correctly, generated-files-y
takes care of this, and this patch adds all new header files to it if I
didn't miss any.

Kevin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
                   ` (21 preceding siblings ...)
  2019-11-06 14:37 ` Max Reitz
@ 2019-11-07 10:33 ` Daniel P. Berrangé
  2019-11-07 12:03   ` Kevin Wolf
  22 siblings, 1 reply; 65+ messages in thread
From: Daniel P. Berrangé @ 2019-11-07 10:33 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: mreitz, pkrempa, qemu-devel, qemu-block, armbru

On Thu, Oct 17, 2019 at 03:01:46PM +0200, Kevin Wolf wrote:
> This series adds a new tool 'qemu-storage-daemon', which can be used to
> export and perform operations on block devices. There is some overlap
> between qemu-img/qemu-nbd and the new qemu-storage-daemon, but there are
> a few important differences:
> 
> * The qemu-storage-daemon has QMP support. The command set is obviously
>   restricted compared to the system emulator because there is no guest,
>   but all of the block operations are present.
> 
>   This means that it can access advanced options or operations that the
>   qemu-img command line doesn't expose. For example, blockdev-create is
>   a lot more powerful than 'qemu-img create', and qemu-storage-daemon
>   allows to execute it without starting a guest.
> 
>   Compared to qemu-nbd it means that, for example, block jobs can now be
>   executed on the server side, and backing chains shared by multiple VMs
>   can be modified this way.
> 
> * The existing tools all have a separately invented one-off syntax for
>   the job at hand, which usually comes with restrictions compared to the
>   system emulator. qemu-storage-daemon shares the same syntax with the
>   system emulator for most options and prefers QAPI based interfaces
>   where possible (such as --blockdev), so it should be easy to make use
>   of in libvirt.
> 
> * While this series implements only NBD exports, the storage daemon is
>   intended to serve multiple protocols and its syntax reflects this. In
>   the past, we had proposals to add new one-off tools for exporting over
>   new protocols like FUSE or TCMU.
> 
>   With a generic storage daemon, additional export methods have a home
>   without adding a new tool for each of them.
> 
> I'm posting this as an RFC mainly for two reasons:
> 
> 1. The monitor integration, which could be argued to be a little hackish
>    (some generated QAPI source files are built from a separate QAPI
>    schema, but the per-module ones are taken from the system emulator)
>    and Markus will want to have a closer look there. But from the IRC
>    discussions we had, we seem to agree on the general approach here.
> 
> 2. I'm not completely sure if the command line syntax is the final
>    version that we want to support long-term. Many options directly use
>    QAPI visitors (--blockdev, --export, --nbd-server) and should be
>    fine. However, others use QemuOpts (--chardev, --monitor, --object).
> 
>    This is the same as in the system emulator, so we wouldn't be adding
>    a new problem, but as there was talk about QAPIfying the command
>    line, and I wouldn't want later syntax changes or adding lots of
>    compatibility code to a new tool, I thought we should probably
>    discuss whether QAPIfying from the start would be an option.

I think that following what the QEMU emulators currently do for
CLI args should be an explicit anti-goal, because we know that it is
a long standing source of pain.  Fixing it in the emulator binaries
is hard due to backward compatibility, but for this new binary we
have a clean slate.

This feels like a good opportunity to implement & demonstrate what
we think QEMU configuration ought to look like. Work done for this
in the qemu-storage-daemon may well help us understand how we'll
be able to move the QEMU emulators into a new scheme later.

My personal wish would be to have no use of QemuOpts at all.

Use GOptionContext *only* for parsing command line arguments
related to execution of the daemon - ie things like --help,
--version, --daemon, --pid-file.

The use a "--config /path/to/json/file" arg to point to the config
file for everything else using QAPI schema to describe it fully.

When loading the config file, things should be created in order
in which they are specified. ie don't try and group things,
otherwise we end up back with the horrific hacks for objects
where some are created early & some late.



For an ambitious stretch goal, I think we should seriously consider
whether our use of chardevs is appropriate in all cases that exist,
and whether we can avoid the trap of over-using chardev in the new
storage daemon since it is a clean slate in terms of user facing
CLI config.

chardevs are designed for & reasonably well suited to attaching to
devices like serial ports, parallel ports, etc. You have a 1:1
remote:local peer relationship. The transport is a dumb byte
stream, nothing special needed on top & the user can cope with
any type of chardev.

Many cases where we've used chardevs as a backend in QEMU are a
poor fit. We just used chardevs as an "easy" way to configure a
UNIX or TCP socket from the CLI, and don't care about, nor work
with, any othuer chardev backends. As a result of this misuse
we've had to put in an increasing number of hacks in the chardev
code to deal with fact that callers want to know about  & use
socket semantics. eg FD passing, the chardev reconnection polling
code.

The monitor is a prime example of a bad fit - it would be better
suited by simply referencing a SocketAddress QAPI type, instead
of having the chardev indirection. It would then directly use
the QIOChannelSocket APIs and avoid the inconvenient chardev
abstractions which are a source of complexity & instability for
no net gain.  vhostuser is another prime example, responsible
for much of the complexity & bugs recently added to chardevs
to expose socket semantics


This is a long winded way of saying that we should consider what
syntax we expose for the monitor socket configuration with the
new daemon. Even if the internal code still uses a chardev for
the forseeable future, we have the option to hide this from the
user facing configuration. Let the user specify a SocketAddress,
which we use to secretly instantiate a chardev. Eventually we can
convert the monitor code to stop using a chardev internally too,
with a suitable deprecation period for main QEMU binarijes.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option
  2019-11-07 10:12     ` Kevin Wolf
@ 2019-11-07 10:44       ` Max Reitz
  0 siblings, 0 replies; 65+ messages in thread
From: Max Reitz @ 2019-11-07 10:44 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, armbru, qemu-block, qemu-devel


[-- Attachment #1.1: Type: text/plain, Size: 3851 bytes --]

On 07.11.19 11:12, Kevin Wolf wrote:
> Am 06.11.2019 um 15:32 hat Max Reitz geschrieben:
>> On 17.10.19 15:02, Kevin Wolf wrote:
>>> This adds and parses the --monitor option, so that a QMP monitor can be
>>> used in the storage daemon. The monitor offers commands defined in the
>>> QAPI schema at storage-daemon/qapi/qapi-schema.json.
>>>
>>> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>>> ---
>>>  storage-daemon/qapi/qapi-schema.json | 15 ++++++++++++
>>>  qemu-storage-daemon.c                | 34 ++++++++++++++++++++++++++++
>>>  Makefile                             | 30 ++++++++++++++++++++++++
>>>  Makefile.objs                        |  4 ++--
>>>  monitor/Makefile.objs                |  2 ++
>>>  qapi/Makefile.objs                   |  5 ++++
>>>  qom/Makefile.objs                    |  1 +
>>>  scripts/qapi/gen.py                  |  5 ++++
>>>  storage-daemon/Makefile.objs         |  1 +
>>>  storage-daemon/qapi/Makefile.objs    |  1 +
>>>  10 files changed, 96 insertions(+), 2 deletions(-)
>>>  create mode 100644 storage-daemon/qapi/qapi-schema.json
>>>  create mode 100644 storage-daemon/Makefile.objs
>>>  create mode 100644 storage-daemon/qapi/Makefile.objs
>>
>> [...]
>>
>>> diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
>>> index 3e04e299ed..03d256f0a4 100644
>>> --- a/qapi/Makefile.objs
>>> +++ b/qapi/Makefile.objs
>>> @@ -30,3 +30,8 @@ obj-y += $(QAPI_TARGET_MODULES:%=qapi-events-%.o)
>>>  obj-y += qapi-events.o
>>>  obj-y += $(QAPI_TARGET_MODULES:%=qapi-commands-%.o)
>>>  obj-y += qapi-commands.o
>>> +
>>> +QAPI_MODULES_STORAGE_DAEMON = block block-core char common crypto introspect
>>> +QAPI_MODULES_STORAGE_DAEMON += job monitor qom sockets pragma transaction
>>
>> I took a look into the rest, and I wonder whether query-iothreads from
>> misc.json would be useful, too.
> 
> Possibly. It would be a separate patch, but I can add it.
> 
> The question is just where to move query-iothreads. Do we have a good
> place, or do I need to separate misc.json and a new misc-sysemu.json?

I’d just put it in block.json because of the “io”... O:-)

>>> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>>> index 796c17c38a..c25634f673 100644
>>> --- a/scripts/qapi/gen.py
>>> +++ b/scripts/qapi/gen.py
>>> @@ -44,6 +44,11 @@ class QAPIGen(object):
>>>          return ''
>>>  
>>>      def write(self, output_dir):
>>> +        # Include paths starting with ../ are used to reuse modules of the main
>>> +        # schema in specialised schemas. Don't overwrite the files that are
>>> +        # already generated for the main schema.
>>> +        if self.fname.startswith('../'):
>>> +            return
>>
>> Sorry, but I’m about to ask a clueless question: How do we ensure that
>> the main schema is generated before something else could make sure of
>> the specialized schemas?
> 
> "Make sure"?

Oops, s/ sure/ use/.

> I think the order of the generation doesn't matter because generating
> the storage daemon files doesn't actually access the main ones.
> Generated C files shouldn't be a problem either because if we link an
> object file into a binary, we have a make dependency for it.

I was mostly wondering about the fact that make mustn’t try to compile
the “generated files” (which aren’t really generated here) before they
are actually generated when the main schema is processed.

Max

> Maybe the only a bit trickier question is whether we have the
> dependencies right so that qemu-storage-daemon.c is only built after the
> header files of both the main schema and the specific one have been
> generated. If I understand the Makefile correctly, generated-files-y
> takes care of this, and this patch adds all new header files to it if I
> didn't miss any.
> 
> Kevin
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-11-07 10:33 ` Daniel P. Berrangé
@ 2019-11-07 12:03   ` Kevin Wolf
  0 siblings, 0 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-11-07 12:03 UTC (permalink / raw)
  To: Daniel P. Berrangé; +Cc: mreitz, pkrempa, qemu-devel, qemu-block, armbru

Am 07.11.2019 um 11:33 hat Daniel P. Berrangé geschrieben:
> On Thu, Oct 17, 2019 at 03:01:46PM +0200, Kevin Wolf wrote:
> > 2. I'm not completely sure if the command line syntax is the final
> >    version that we want to support long-term. Many options directly use
> >    QAPI visitors (--blockdev, --export, --nbd-server) and should be
> >    fine. However, others use QemuOpts (--chardev, --monitor, --object).
> > 
> >    This is the same as in the system emulator, so we wouldn't be adding
> >    a new problem, but as there was talk about QAPIfying the command
> >    line, and I wouldn't want later syntax changes or adding lots of
> >    compatibility code to a new tool, I thought we should probably
> >    discuss whether QAPIfying from the start would be an option.
> 
> I think that following what the QEMU emulators currently do for
> CLI args should be an explicit anti-goal, because we know that it is
> a long standing source of pain.  Fixing it in the emulator binaries
> is hard due to backward compatibility, but for this new binary we
> have a clean slate.
> 
> This feels like a good opportunity to implement & demonstrate what
> we think QEMU configuration ought to look like. Work done for this
> in the qemu-storage-daemon may well help us understand how we'll
> be able to move the QEMU emulators into a new scheme later.

It might be, which is why I'm asking. Now that the storage daemon has
missed 4.2, we have a little more time to decide what the command line
should look like in detail.

However, I don't think this is something that should delay the storage
daemon until after 5.0.

> My personal wish would be to have no use of QemuOpts at all.
> 
> Use GOptionContext *only* for parsing command line arguments
> related to execution of the daemon - ie things like --help,
> --version, --daemon, --pid-file.

I really don't believe that the solution for having too much variety in
option parsing is adding in yet another type. GOptionContext is not
something I'm considering at the moment.

But it's a getopt() replacement, not something that could actually parse
the more complex options, so it's a separate question anyway. If we ever
want to use it, we can replace getopt() in all binaries at once.

> The use a "--config /path/to/json/file" arg to point to the config
> file for everything else using QAPI schema to describe it fully.

If this implies that the storage daemon can only do useful things if you
specify a config file, I disagree.

I agree that it's not really nice if you can't use a config file to
specify a lengthy configuration and that supporting one would be good.

But it is at least equally unfriendly to require a config file for
simple configurations where using command line arguments is easily
possible.

> When loading the config file, things should be created in order
> in which they are specified. ie don't try and group things,
> otherwise we end up back with the horrific hacks for objects
> where some are created early & some late.

Yes. This is how the storage daemon command line works, too.

I think Markus already had some patches for command line QAPIfication
that were incomplete at least for the system emulator. It might be
easier to make it feature complete for the storage daemon because it
supports much less problematic options. Maybe he can post a useful
subset (if it's too much work to clean up the full thing right now) and
we can work from there.

The one that I expect to be a bit tricky to be QAPIfied is --object.

> For an ambitious stretch goal, I think we should seriously consider
> whether our use of chardevs is appropriate in all cases that exist,
> and whether we can avoid the trap of over-using chardev in the new
> storage daemon since it is a clean slate in terms of user facing
> CLI config.
> 
> chardevs are designed for & reasonably well suited to attaching to
> devices like serial ports, parallel ports, etc. You have a 1:1
> remote:local peer relationship. The transport is a dumb byte
> stream, nothing special needed on top & the user can cope with
> any type of chardev.
> 
> Many cases where we've used chardevs as a backend in QEMU are a
> poor fit. We just used chardevs as an "easy" way to configure a
> UNIX or TCP socket from the CLI, and don't care about, nor work
> with, any othuer chardev backends. As a result of this misuse
> we've had to put in an increasing number of hacks in the chardev
> code to deal with fact that callers want to know about  & use
> socket semantics. eg FD passing, the chardev reconnection polling
> code.
> 
> The monitor is a prime example of a bad fit - it would be better
> suited by simply referencing a SocketAddress QAPI type, instead
> of having the chardev indirection. It would then directly use
> the QIOChannelSocket APIs and avoid the inconvenient chardev
> abstractions which are a source of complexity & instability for
> no net gain.  vhostuser is another prime example, responsible
> for much of the complexity & bugs recently added to chardevs
> to expose socket semantics
> 
> 
> This is a long winded way of saying that we should consider what
> syntax we expose for the monitor socket configuration with the
> new daemon. Even if the internal code still uses a chardev for
> the forseeable future, we have the option to hide this from the
> user facing configuration. Let the user specify a SocketAddress,
> which we use to secretly instantiate a chardev. Eventually we can
> convert the monitor code to stop using a chardev internally too,
> with a suitable deprecation period for main QEMU binarijes.

The monitor is actually the prime example for chardev backends like
stdio or vc. I'm using them all the time and they aren't covered by
SocketAddress, so I'm afraid SocketAddress alone is not an option.

In any case, the goal of the storage daemon code is to be as boring as
possible. It should just be a thin wrapper around existing code that
glues the command line, monitor and various backends together.

So any change to the usage of chardevs (that is more than a few lines of
wrapper code) would probably have to be made in the system emulator
first, otherwise we'd end up duplicating a lot of code for the storage
daemon instead of just reusing it. I guess using SocketAddress for the
storage daemon and internall creating a chardev-socket would be possible
with a few lines if that covered stdio - but it doesn't.

Kevin



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option
  2019-11-07  8:33       ` Kevin Wolf
@ 2019-11-07 13:45         ` Eric Blake
  2019-11-07 15:27           ` Kevin Wolf
  0 siblings, 1 reply; 65+ messages in thread
From: Eric Blake @ 2019-11-07 13:45 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: qemu-devel, pkrempa, armbru, qemu-block, Max Reitz

On 11/7/19 2:33 AM, Kevin Wolf wrote:

> 
> As a replacement nbd-server-add, I envisioned adding something like a
> block-export-add, which would work the way that --export already does.
> It would also come with query-block-exports and block-export-del, and it
> wouldn't contain only NBD devices, but also vhost-user, FUSE, etc.
> exports.
> 
> Now I'm wondering if the same would make sense for nbd-server-start.
> Maybe an API change would even allow us to start multiple NBD servers
> (e.g. listening on different IP addresses or using different tls-creds).

We want that (the ability to run multiple parallel NBD servers) anyway, 
to allow parallel incremental backups from different points in time to 
different clients.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option
  2019-11-07 13:45         ` Eric Blake
@ 2019-11-07 15:27           ` Kevin Wolf
  2019-11-07 15:36             ` Eric Blake
  0 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-11-07 15:27 UTC (permalink / raw)
  To: Eric Blake; +Cc: qemu-devel, pkrempa, armbru, qemu-block, Max Reitz

Am 07.11.2019 um 14:45 hat Eric Blake geschrieben:
> On 11/7/19 2:33 AM, Kevin Wolf wrote:
> > As a replacement nbd-server-add, I envisioned adding something like a
> > block-export-add, which would work the way that --export already does.
> > It would also come with query-block-exports and block-export-del, and it
> > wouldn't contain only NBD devices, but also vhost-user, FUSE, etc.
> > exports.
> > 
> > Now I'm wondering if the same would make sense for nbd-server-start.
> > Maybe an API change would even allow us to start multiple NBD servers
> > (e.g. listening on different IP addresses or using different tls-creds).
> 
> We want that (the ability to run multiple parallel NBD servers) anyway, to
> allow parallel incremental backups from different points in time to
> different clients.

Can't you already have multiple exports on a single NBD server for
multiple clients today? Or do you need a different server configuration
for each client?

Kevin



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option
  2019-11-07 15:27           ` Kevin Wolf
@ 2019-11-07 15:36             ` Eric Blake
  0 siblings, 0 replies; 65+ messages in thread
From: Eric Blake @ 2019-11-07 15:36 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: qemu-devel, pkrempa, armbru, qemu-block, Max Reitz

On 11/7/19 9:27 AM, Kevin Wolf wrote:
> Am 07.11.2019 um 14:45 hat Eric Blake geschrieben:
>> On 11/7/19 2:33 AM, Kevin Wolf wrote:
>>> As a replacement nbd-server-add, I envisioned adding something like a
>>> block-export-add, which would work the way that --export already does.
>>> It would also come with query-block-exports and block-export-del, and it
>>> wouldn't contain only NBD devices, but also vhost-user, FUSE, etc.
>>> exports.
>>>
>>> Now I'm wondering if the same would make sense for nbd-server-start.
>>> Maybe an API change would even allow us to start multiple NBD servers
>>> (e.g. listening on different IP addresses or using different tls-creds).
>>
>> We want that (the ability to run multiple parallel NBD servers) anyway, to
>> allow parallel incremental backups from different points in time to
>> different clients.
> 
> Can't you already have multiple exports on a single NBD server for
> multiple clients today? Or do you need a different server configuration
> for each client?

With our current code base, you can only run a single NBD server, with 
multiple exports, but the TLS creds are shared among all exports.  It is 
indeed technically possible to tweak things where the single server 
changes _which_ exports are exposed based on _which_ creds were used by 
the client (but only when TLS is used, and note that qemu-nbd currently 
refuses to mix TLS and Unix sockets, although I need to post a v2 of a 
patch I proposed a while back to improve that).  But it is easier still 
to run two separate servers on different ports with two different creds, 
and where there is no magic on which exports to show merely based on 
which creds were presented (and this includes a plaintext connection 
over Unix).  Either way, it requires code changes, and most likely for 5.0.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool
  2019-10-17 13:01 ` [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool Kevin Wolf
  2019-10-24 13:50   ` Eric Blake
  2019-11-06 12:11   ` Max Reitz
@ 2019-11-07 16:21   ` Markus Armbruster
  2 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-07 16:21 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

In addition to Eric's review:

Kevin Wolf <kwolf@redhat.com> writes:

> This adds a new binary qemu-storage-daemon that doesn't yet do more than
> some typical initialisation for tools and parsing the basic command
> options --version, --help and --trace.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  configure             |   2 +-
>  qemu-storage-daemon.c | 141 ++++++++++++++++++++++++++++++++++++++++++
>  Makefile              |   1 +
>  3 files changed, 143 insertions(+), 1 deletion(-)
>  create mode 100644 qemu-storage-daemon.c
>
> diff --git a/configure b/configure
> index 08ca4bcb46..bb3d55fb25 100755
> --- a/configure
> +++ b/configure
> @@ -6034,7 +6034,7 @@ tools=""
>  if test "$want_tools" = "yes" ; then
>    tools="qemu-img\$(EXESUF) qemu-io\$(EXESUF) qemu-edid\$(EXESUF) $tools"
>    if [ "$linux" = "yes" -o "$bsd" = "yes" -o "$solaris" = "yes" ] ; then
> -    tools="qemu-nbd\$(EXESUF) $tools"
> +    tools="qemu-nbd\$(EXESUF) qemu-storage-daemon\$(EXESUF) $tools"
>    fi
>    if [ "$ivshmem" = "yes" ]; then
>      tools="ivshmem-client\$(EXESUF) ivshmem-server\$(EXESUF) $tools"
> diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
> new file mode 100644
> index 0000000000..a251dc255c
> --- /dev/null
> +++ b/qemu-storage-daemon.c
> @@ -0,0 +1,141 @@
> +/*
> + * QEMU storage daemon
> + *
> + * Copyright (c) 2019 Kevin Wolf <kwolf@redhat.com>
> + *
> + * Permission is hereby granted, free of charge, to any person obtaining a copy
> + * of this software and associated documentation files (the "Software"), to deal
> + * in the Software without restriction, including without limitation the rights
> + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> + * copies of the Software, and to permit persons to whom the Software is
> + * furnished to do so, subject to the following conditions:
> + *
> + * The above copyright notice and this permission notice shall be included in
> + * all copies or substantial portions of the Software.
> + *
> + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> + * THE SOFTWARE.
> + */

Standard request for new code: please make this GPLv2+, or tell us why
it has to be something else.

> +
> +#include "qemu/osdep.h"
> +
> +#include "block/block.h"
> +#include "crypto/init.h"
> +
> +#include "qapi/error.h"
> +#include "qemu-common.h"
> +#include "qemu-version.h"
> +#include "qemu/config-file.h"
> +#include "qemu/error-report.h"
> +#include "qemu/log.h"
> +#include "qemu/main-loop.h"
> +#include "qemu/module.h"
> +
> +#include "trace/control.h"
> +
> +#include <getopt.h>
> +
> +static void help(void)
> +{
> +    printf(
> +"Usage: %s [options]\n"
> +"QEMU storage daemon\n"
> +"\n"
> +"  -h, --help             display this help and exit\n"
> +"  -T, --trace [[enable=]<pattern>][,events=<file>][,file=<file>]\n"
> +"                         specify tracing options\n"
> +"  -V, --version          output version information and exit\n"
> +"\n"
> +QEMU_HELP_BOTTOM "\n",
> +    error_get_progname());
> +}
> +
> +static int process_options(int argc, char *argv[], Error **errp)
> +{
> +    int c;
> +    char *trace_file = NULL;
> +    int ret = -EINVAL;
> +
> +    static const struct option long_options[] = {
> +        {"help", no_argument, 0, 'h'},

You initialize member int *flag with 0 here, ....

> +        {"version", no_argument, 0, 'V'},
> +        {"trace", required_argument, NULL, 'T'},

... and with NULL here.  Recommend to pick one and stick to it.

> +        {0, 0, 0, 0}

{0} or {} would do.

> +    };
> +
> +    while ((c = getopt_long(argc, argv, ":hT:V", long_options, NULL)) != -1) {
> +        switch (c) {
> +        case '?':
> +            error_setg(errp, "Unknown option '%s'", argv[optind - 1]);
> +            goto out;
> +        case ':':
> +            error_setg(errp, "Missing option argument for '%s'",
> +                       argv[optind - 1]);
> +            goto out;
> +        case 'h':
> +            help();
> +            exit(EXIT_SUCCESS);
> +        case 'V':
> +            printf("qemu-storage-daemon version "
> +                   QEMU_FULL_VERSION "\n" QEMU_COPYRIGHT "\n");
> +            exit(EXIT_SUCCESS);
> +        case 'T':
> +            g_free(trace_file);
> +            trace_file = trace_opt_parse(optarg);

This is QemuOpts below the hood.  Fact, not criticism :)

> +            break;
> +        }

Suggest (your preferred variation of) default: assert(0) to catch
omissions.

> +    }
> +    if (optind != argc) {
> +        error_setg(errp, "Unexpected argument: %s", argv[optind]);
> +        goto out;
> +    }
> +
> +    if (!trace_init_backends()) {
> +        error_setg(errp, "Could not initialize trace backends");
> +        goto out;
> +    }
> +    trace_init_file(trace_file);
> +    qemu_set_log(LOG_TRACE);

I suspect the only reason for hiding trace initialization within
process_options() is avoiding a global variable @trace_file.  I'd prefer
the global variable over the hiding.

> +
> +    ret = 0;
> +out:
> +    g_free(trace_file);
> +    return ret;
> +}

Since the function exit(0)s on -h and -V anyway, let's exit(1) on error
instead of mucking around with error_setg().  You can then leave
reporting unknown options and missing option arguments to getopt_long().
Saves you the trouble of fixing the bug Max pointed out.

> +
> +int main(int argc, char *argv[])
> +{
> +    Error *local_err = NULL;
> +    int ret;
> +
> +#ifdef CONFIG_POSIX
> +    signal(SIGPIPE, SIG_IGN);
> +#endif
> +
> +    error_init(argv[0]);
> +    qemu_init_exec_dir(argv[0]);
> +
> +    module_call_init(MODULE_INIT_QOM);
> +    module_call_init(MODULE_INIT_TRACE);
> +    qemu_add_opts(&qemu_trace_opts);
> +    qcrypto_init(&error_fatal);
> +    bdrv_init();

Out of curiosity: how did you come up with this set of initializations?

> +
> +    if (qemu_init_main_loop(&local_err)) {
> +        error_report_err(local_err);
> +        return EXIT_FAILURE;

What's wrong with

       qemu_init_main_loop(&error_fatal)

?

> +    }
> +
> +    ret = process_options(argc, argv, &local_err);
> +    if (ret < 0) {
> +        error_report_err(local_err);
> +        return EXIT_FAILURE;
> +    }
> +
> +    return EXIT_SUCCESS;
> +}
> diff --git a/Makefile b/Makefile
> index 30f0abfb42..76338d0ab4 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -558,6 +558,7 @@ qemu-img.o: qemu-img-cmds.h
>  qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
>  qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
>  qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
> +qemu-storage-daemon$(EXESUF): qemu-storage-daemon.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
>  
>  qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COMMON_LDADDS)



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 02/18] qemu-storage-daemon: Add --object option
  2019-10-17 13:01 ` [RFC PATCH 02/18] qemu-storage-daemon: Add --object option Kevin Wolf
@ 2019-11-07 20:36   ` Markus Armbruster
  2019-11-14 12:05     ` Kevin Wolf
  0 siblings, 1 reply; 65+ messages in thread
From: Markus Armbruster @ 2019-11-07 20:36 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Kevin Wolf <kwolf@redhat.com> writes:

> Add a command line option to create user-creatable QOM objects.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  qemu-storage-daemon.c | 35 +++++++++++++++++++++++++++++++++++
>  1 file changed, 35 insertions(+)
>
> diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
> index a251dc255c..48d6af43a6 100644
> --- a/qemu-storage-daemon.c
> +++ b/qemu-storage-daemon.c
> @@ -35,6 +35,8 @@
>  #include "qemu/log.h"
>  #include "qemu/main-loop.h"
>  #include "qemu/module.h"
> +#include "qemu/option.h"
> +#include "qom/object_interfaces.h"
>  
>  #include "trace/control.h"
>  
> @@ -51,10 +53,26 @@ static void help(void)
>  "                         specify tracing options\n"
>  "  -V, --version          output version information and exit\n"
>  "\n"
> +"  --object <properties>  define a QOM object such as 'secret' for\n"
> +"                         passwords and/or encryption keys\n"

This is less helpful than qemu-system-FOO's help:

-object TYPENAME[,PROP1=VALUE1,...]
                create a new object of type TYPENAME setting properties
                in the order they are specified.  Note that the 'id'
                property must be set.  These objects are placed in the
                '/objects' path.

> +"\n"
>  QEMU_HELP_BOTTOM "\n",
>      error_get_progname());
>  }
>  
> +enum {
> +    OPTION_OBJECT = 256,
> +};
> +
> +static QemuOptsList qemu_object_opts = {
> +    .name = "object",
> +    .implied_opt_name = "qom-type",
> +    .head = QTAILQ_HEAD_INITIALIZER(qemu_object_opts.head),
> +    .desc = {
> +        { }
> +    },
> +};
> +

Note for later: copied from vl.c.

>  static int process_options(int argc, char *argv[], Error **errp)
>  {
>      int c;
> @@ -63,6 +81,7 @@ static int process_options(int argc, char *argv[], Error **errp)
>  
>      static const struct option long_options[] = {
>          {"help", no_argument, 0, 'h'},
> +        {"object", required_argument, 0, OPTION_OBJECT},
>          {"version", no_argument, 0, 'V'},
>          {"trace", required_argument, NULL, 'T'},
>          {0, 0, 0, 0}
> @@ -88,6 +107,22 @@ static int process_options(int argc, char *argv[], Error **errp)
>              g_free(trace_file);
>              trace_file = trace_opt_parse(optarg);
>              break;
> +        case OPTION_OBJECT:
> +            {
> +                QemuOpts *opts;
> +                const char *type;
> +
> +                opts = qemu_opts_parse(&qemu_object_opts,
> +                                       optarg, true, &error_fatal);
> +                type = qemu_opt_get(opts, "qom-type");
> +
> +                if (user_creatable_print_help(type, opts)) {
> +                    exit(EXIT_SUCCESS);
> +                }
> +                user_creatable_add_opts(opts, &error_fatal);
> +                qemu_opts_del(opts);
> +                break;
> +            }
>          }
>      }
>      if (optind != argc) {

PATCH 01 duplicates case QEMU_OPTION_trace pretty much verbatim.  Makes
sense, as qemu-storage-daemon is basically qemu-system-FOO with "FOO"
and most "system" cut away.

This patch adds vl.c's case QEMU_OPTION_object in a much simpler form.
This is one of my least favourite options, and I'll tell you why below.
Let's compare the two versions.

vl.c:

            case QEMU_OPTION_object:
                opts = qemu_opts_parse_noisily(qemu_find_opts("object"),
                                               optarg, true);
                if (!opts) {
                    exit(1);
                }
                break;

Further down:

    qemu_opts_foreach(qemu_find_opts("object"),
                      user_creatable_add_opts_foreach,
                      object_create_initial, &error_fatal);

Still further down:

    qemu_opts_foreach(qemu_find_opts("object"),
                      user_creatable_add_opts_foreach,
                      object_create_delayed, &error_fatal);

These are basically

    for opts in qemu_object_opts {
        type = qemu_opt_get(opts, "qom-type");
        if (type) {
            if (user_creatable_print_help(type, opts)) {
                exit(0);
            }
            if (!predicate(type)) {
                continue;
            }
        }
        obj = user_creatable_add_opts(opts, &error_fatal);
        object_unref(obj);
    }

where predicate(type) is true in exactly one of the two places for each
QOM type.

The reason for these gymnastics is to create objects at the right time
during startup, except there is no right time, but two.

Differences:

* Options are processed left to right without gymnastics.  Getting their
  order right is the user's problem.  I consider this an improvement.

* You use &qemu_object_opts instead of qemu_find_opts("object").  Also
  an improvement.

* You use qemu_opts_parse() instead of qemu_opts_parse_noisily().
  The latter can print help.  I failed to find a case where we lose help
  compared to qemu-system-FOO.  I didn't try very hard.

* You neglect to guard user_creatable_print_help():

    $ qemu-storage-daemon --object wrong=1,help
    Segmentation fault (core dumped)

* You neglect to object_unref().  I just double-checked the final
  reference count: it's 2.

These bugs shouldn't be hard to fix.


At this point you might wonder why I dislike this option so much.
vl.c's gymnastics are ugly, but not unusually ugly, and they're gone
here.  To explain my distaste, I have to go back a little bit.

Like quite a few options, --object is paired with QMP command, namely
object-add.  Both have the same parameters: QOM type, object ID, and
additional type-specific object properties.  There's a difference,
though: object-add wraps the latter in a 'props' object, while --object
does not.

QAPI schema:

    { 'command': 'object-add',
      'data': {'qom-type': 'str', 'id': 'str', '*props': 'any'} }

QAPIfying this part of the CLI isn't easy.

The obvious QAPIfied CLI buddy of object-add is incompatible to current
--object.  That's not a concern for the storage daemon.  But it's also
ugly, because object-add's nesting of the type-specific properties
within @props is.  In QMP, it's merely yet another pair of curlies.  In
the CLI, we get to prefix props. to each type-specific property.

If we want to give the storage daemon a QAPIfied command line from the
start (and I think we do), we'll have to decide how to address this
issue, and possibly more (I'm only at PATCH 02/18).

We have a long history of rather careless interface design, and now some
of these chickens come home to roost.



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option
  2019-10-17 13:02 ` [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option Kevin Wolf
  2019-11-06 14:32   ` Max Reitz
@ 2019-11-08  8:59   ` Markus Armbruster
  2019-11-12 14:25   ` Markus Armbruster
  2 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-08  8:59 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Quick observation: --help fails to mention --monitor.



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 04/18] stubs: Add blk_by_qdev_id()
  2019-10-17 13:01 ` [RFC PATCH 04/18] stubs: Add blk_by_qdev_id() Kevin Wolf
@ 2019-11-08  9:03   ` Markus Armbruster
  0 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-08  9:03 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Kevin Wolf <kwolf@redhat.com> writes:

> blockdev.c uses the blk_by_qdev_id() function, so before we can use the
> file in tools (i.e. outside of the system emulator), we need to add a
> stub for it. The function always returns an error.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  stubs/blk-by-qdev-id.c | 9 +++++++++
>  stubs/Makefile.objs    | 1 +
>  2 files changed, 10 insertions(+)
>  create mode 100644 stubs/blk-by-qdev-id.c
>
> diff --git a/stubs/blk-by-qdev-id.c b/stubs/blk-by-qdev-id.c
> new file mode 100644
> index 0000000000..0b6160fefa
> --- /dev/null
> +++ b/stubs/blk-by-qdev-id.c
> @@ -0,0 +1,9 @@
> +#include "qemu/osdep.h"
> +#include "qapi/error.h"
> +#include "sysemu/block-backend.h"
> +
> +BlockBackend *blk_by_qdev_id(const char *id, Error **errp)
> +{
> +    error_setg(errp, "qdev IDs/QOM paths exist only in the system emulator");
> +    return NULL;
> +}

Is this function meant to be called?  Let me check...  it seems to be
used just for QMP commands that accept either a qdev ID or QOM path (by
convention 'id') or a block backend name (by convention 'device').
Evidence:

   static BlockBackend *qmp_get_blk(const char *blk_name, const char *qdev_id,
                                    Error **errp)
   {
       BlockBackend *blk;

       if (!blk_name == !qdev_id) {
           error_setg(errp, "Need exactly one of 'device' and 'id'");
           return NULL;
       }

       if (qdev_id) {
           blk = blk_by_qdev_id(qdev_id, errp);
       } else {
           blk = blk_by_name(blk_name);
           if (blk == NULL) {
               error_set(errp, ERROR_CLASS_DEVICE_NOT_FOUND,
                         "Device '%s' not found", blk_name);
           }
       }

       return blk;
   }

Users:

* eject, blockdev-open-tray, blockdev-change-medium via do_open_tray()

* blockdev-close-tray

* eject, blockdev-remove-medium via blockdev_remove_medium()

* blockdev-insert-medium via blockdev_insert_medium()

  Aside: blockdev_insert_medium() could be inlined.

* blockdev-change-medium

* block_set_io_throttle

* block-latency-histogram-set

The newer ones them don't provide 'device', and pass a null @blk_name to
qmp_get_blk().  Using blk_by_qdev_id() directly would be clearer.

Where 'device' is provided, it's been deprecated since v2.8.0.  Not
documented in qemu-deprecated.texi, though.

As far as I understand the storage daemon's intended purpose, none of
these commands make sense there.  Testing.... aha: it provides them all
anyway.  Is that a good idea?

> diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
> index 9f4eb25e02..77fbf72576 100644
> --- a/stubs/Makefile.objs
> +++ b/stubs/Makefile.objs
> @@ -1,5 +1,6 @@
>  stub-obj-y += arch_type.o
>  stub-obj-y += bdrv-next-monitor-owned.o
> +stub-obj-y += blk-by-qdev-id.o
>  stub-obj-y += blk-commit-all.o
>  stub-obj-y += blockdev-close-all-bdrv-states.o
>  stub-obj-y += clock-warp.o



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 05/18] qemu-storage-daemon: Add --blockdev option
  2019-10-17 13:01 ` [RFC PATCH 05/18] qemu-storage-daemon: Add --blockdev option Kevin Wolf
@ 2019-11-08 13:29   ` Markus Armbruster
  0 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-08 13:29 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: mreitz, pkrempa, qemu-devel, qemu-block, armbru

Kevin Wolf <kwolf@redhat.com> writes:

> This adds a --blockdev option to the storage daemon that works the same
> as the -blockdev option of the system emulator.
>
> In order to be able to link with blockdev.o, we also need to change
> stream.o from common-obj to block-obj, which is where all other block
> jobs already are.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  qemu-storage-daemon.c | 29 +++++++++++++++++++++++++++++
>  Makefile              |  5 ++++-
>  Makefile.objs         |  7 +++++++
>  block/Makefile.objs   |  2 +-
>  4 files changed, 41 insertions(+), 2 deletions(-)
>
> diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
> index 48d6af43a6..904e3c3a46 100644
> --- a/qemu-storage-daemon.c
> +++ b/qemu-storage-daemon.c
> @@ -28,6 +28,10 @@
>  #include "crypto/init.h"
>  
>  #include "qapi/error.h"
> +#include "qapi/qapi-visit-block-core.h"
> +#include "qapi/qapi-commands-block-core.h"
> +#include "qapi/qobject-input-visitor.h"
> +
>  #include "qemu-common.h"
>  #include "qemu-version.h"
>  #include "qemu/config-file.h"
> @@ -53,6 +57,13 @@ static void help(void)
>  "                         specify tracing options\n"
>  "  -V, --version          output version information and exit\n"
>  "\n"
> +"  --blockdev [driver=]<driver>[,node-name=<N>][,discard=ignore|unmap]\n"
> +"             [,cache.direct=on|off][,cache.no-flush=on|off]\n"
> +"             [,read-only=on|off][,auto-read-only=on|off]\n"
> +"             [,force-share=on|off][,detect-zeroes=on|off|unmap]\n"
> +"             [,driver specific parameters...]\n"
> +"                         configure a block backend\n"
> +"\n"
>  "  --object <properties>  define a QOM object such as 'secret' for\n"
>  "                         passwords and/or encryption keys\n"
>  "\n"
> @@ -62,6 +73,7 @@ QEMU_HELP_BOTTOM "\n",
>  
>  enum {
>      OPTION_OBJECT = 256,
> +    OPTION_BLOCKDEV,
>  };
>  
>  static QemuOptsList qemu_object_opts = {
> @@ -82,6 +94,7 @@ static int process_options(int argc, char *argv[], Error **errp)
>      static const struct option long_options[] = {
>          {"help", no_argument, 0, 'h'},
>          {"object", required_argument, 0, OPTION_OBJECT},
> +        {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
>          {"version", no_argument, 0, 'V'},
>          {"trace", required_argument, NULL, 'T'},
>          {0, 0, 0, 0}
> @@ -123,6 +136,22 @@ static int process_options(int argc, char *argv[], Error **errp)
>                  qemu_opts_del(opts);
>                  break;
>              }
> +            break;
> +        case OPTION_BLOCKDEV:
> +            {
> +                Visitor *v;
> +                BlockdevOptions *options;
> +
> +                v = qobject_input_visitor_new_str(optarg, "driver",
> +                                                  &error_fatal);
> +
> +                visit_type_BlockdevOptions(v, NULL, &options, &error_fatal);
> +                visit_free(v);
> +
> +                qmp_blockdev_add(options, &error_fatal);
> +                qapi_free_BlockdevOptions(options);
> +                break;
> +            }
>          }
>      }
>      if (optind != argc) {

Unlike --object, --blockdev is already QAPIfied, albeit crudely.

Similar difference to vl.c as for --object: vl.c records the option for
later processing, while here you process it right away.  Simpler and
saner.

I figure you intend to do all options this way.

Should you explain the difference in a commit message?  A comment?

[...]



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option
  2019-11-06 12:51   ` Max Reitz
  2019-11-06 19:25     ` Eric Blake
@ 2019-11-08 15:36     ` Markus Armbruster
  1 sibling, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-08 15:36 UTC (permalink / raw)
  To: Max Reitz; +Cc: Kevin Wolf, pkrempa, qemu-devel, qemu-block

Max Reitz <mreitz@redhat.com> writes:

> On 17.10.19 15:01, Kevin Wolf wrote:
>> Add a --nbd-server option to qemu-storage-daemon to start the built-in
>> NBD server right away. It maps the arguments for nbd-server-start to the
>> command line.
>
> Well, it doesn’t quite, because nbd-server-start takes a
> SocketAddressLegacy, and this takes a SocketAddress.
>
> On one hand I can understand why you would do it differently (especially
> for command-line options), but on the other I find it a bit problematic
> to have --nbd-server be slightly different from nbd-server-start when
> both are intended to be the same.
>
> My biggest problem though lies in the duplication in the QAPI schema.
> If NbdServerOptions.addr were a SocketAddressLegacy, we could let
> nbd-server-start’s options just be of type NbdServerOptions and thus get
> rid of the duplication.
>
> I suspect in practice it’s all not that big of a problem.  I can’t call
> it bad if --nbd-server is just nicer to use.  And the biggest problem
> with duplication in the QAPI schema is that nbd-server-start and
> --nbd-server might get out of sync.  But realistically, I don’t see that
> happen, because if nbd-server-start changes, nbd_server_start() will
> change, too, so we’ll get compile errors in nbd_server_start_options().
>
> *shrug*

Two good reasons for making new --nbd-server differ from existing
nbd-server-start:

1. The nesting sucks.  The CLI's dotted key syntax makes it suck harder.

2. New interfaces should not use SocketAddressLegacy (or any other
   "simple" union) if we can help it.

The duplication is the price we pay for getting it right om the second
try.

> But I do think the commit message should explain why we can’t just use
> NbdServerOptions for nbd-server-start.

Yes, and throw in a comment.



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 08/18] qemu-storage-daemon: Add --export option
  2019-11-06 13:34     ` Kevin Wolf
  2019-11-06 13:39       ` Max Reitz
@ 2019-11-08 15:57       ` Markus Armbruster
  1 sibling, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-08 15:57 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, Max Reitz

Kevin Wolf <kwolf@redhat.com> writes:

> Am 06.11.2019 um 14:11 hat Max Reitz geschrieben:
>> On 17.10.19 15:01, Kevin Wolf wrote:
>> > Add a --export option to qemu-storage-daemon to export a block node. For
>> > now, only NBD exports are implemented. Apart from the 'type' option
>> > (which is the implied key), it maps the arguments for nbd-server-add to
>> > the command line. Example:
>> > 
>> >     --export nbd,device=disk,name=test-export,writable=on
>> > 
>> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>> > ---
>> >  qapi/block.json       | 27 +++++++++++++++++++++++++++
>> >  qemu-storage-daemon.c | 31 +++++++++++++++++++++++++++++++
>> >  2 files changed, 58 insertions(+)
>> 
>> Would it be better to collect the BlockExports in a list and work on it
>> after all arguments have been parsed?  As it is, it’s important that
>> users define block devices and things like NBD servers before --export.
>>  Yes, I know, that’s exactly how it works with qemu, but is that really
>> the best way?
>
> It's actually not how QEMU works generally. QEMU collects things in
> QemuOptsLists and then tries to interpret them in the right order. Of
> course, we never get the order actually right, which results in constant
> reshuffling of the order of initialisations in vl.c.
>
> It also means that vl.c (!) has a list of -object types that need to be
> created early so that other backends can make use of them, and of those
> types that actually depend on a backend already being present (see
> object_create_initial() for details).
>
> I think it's much cleaner to simply use the order in the command line
> instead of adding magic that tries to resolve (and fails at actually
> resolving) all the dependencies.

Seconded.

The "process arguments strictly left to right" strategy is also visible
in PATCH 02 and 05.

>                                  I seem to remember that this was in
> fact one of the things Markus keeps mentioning he would change if he
> were to rewrite the QEMU command line parser from scratch without
> compatibility requirements.

E.g.
Message-ID: <87poomg77m.fsf@dusky.pond.sub.org>
https://lists.nongnu.org/archive/html/qemu-devel/2016-09/msg00163.html



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 09/18] qemu-storage-daemon: Add main loop
  2019-10-17 13:01 ` [RFC PATCH 09/18] qemu-storage-daemon: Add main loop Kevin Wolf
@ 2019-11-08 16:02   ` Markus Armbruster
  0 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-08 16:02 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Kevin Wolf <kwolf@redhat.com> writes:

> Instead of exiting after processing all command line options, start a
> main loop and keep processing events until exit is requested with a
> signal (e.g. SIGINT).
>
> Now qemu-storage-daemon can be used as an alternative for qemu-nbd that
> provides a few features that were previously only available from QMP,
> such as access to options only available with -blockdev and the socket
> types 'vsock' and 'fd'.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  qemu-storage-daemon.c | 13 +++++++++++++
>  Makefile.objs         |  2 ++
>  2 files changed, 15 insertions(+)
>
> diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
> index 9e5f474fd0..099388f645 100644
> --- a/qemu-storage-daemon.c
> +++ b/qemu-storage-daemon.c
> @@ -45,10 +45,18 @@
>  #include "qemu/option.h"
>  #include "qom/object_interfaces.h"
>  
> +#include "sysemu/runstate.h"
>  #include "trace/control.h"
>  
>  #include <getopt.h>
>  
> +static bool exit_requested = false;
> +
> +void qemu_system_killed(int signal, pid_t pid)
> +{
> +    exit_requested = true;
> +}
> +

This runs within a signal handler, so better make @exit_requested
volatile.  Hmm, vl.c gets it wrong, too.

>  static void help(void)
>  {
>      printf(
> @@ -238,6 +246,7 @@ int main(int argc, char *argv[])
>  
>      error_init(argv[0]);
>      qemu_init_exec_dir(argv[0]);
> +    os_setup_signal_handling();
>  
>      module_call_init(MODULE_INIT_QOM);
>      module_call_init(MODULE_INIT_TRACE);
> @@ -256,5 +265,9 @@ int main(int argc, char *argv[])
>          return EXIT_FAILURE;
>      }
>  
> +    while (!exit_requested) {
> +        main_loop_wait(false);
> +    }
> +
>      return EXIT_SUCCESS;
>  }
> diff --git a/Makefile.objs b/Makefile.objs
> index cc262e445f..b667d3f07b 100644
> --- a/Makefile.objs
> +++ b/Makefile.objs
> @@ -43,6 +43,8 @@ io-obj-y = io/
>  
>  storage-daemon-obj-y = block/
>  storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o
> +storage-daemon-obj-$(CONFIG_WIN32) += os-win32.o
> +storage-daemon-obj-$(CONFIG_POSIX) += os-posix.o
>  
>  ######################################################################
>  # Target independent part of system emulation. The long term path is to



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 10/18] qemu-storage-daemon: Add --chardev option
  2019-10-17 13:01 ` [RFC PATCH 10/18] qemu-storage-daemon: Add --chardev option Kevin Wolf
@ 2019-11-08 16:27   ` Markus Armbruster
  0 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-08 16:27 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Kevin Wolf <kwolf@redhat.com> writes:

> This adds a --chardev option to the storage daemon that works the same
> as the -chardev option of the system emulator.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  qemu-storage-daemon.c | 19 +++++++++++++++++++
>  Makefile              |  2 +-
>  2 files changed, 20 insertions(+), 1 deletion(-)
>
> diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
> index 099388f645..46e0a6ea56 100644
> --- a/qemu-storage-daemon.c
> +++ b/qemu-storage-daemon.c
> @@ -26,6 +26,7 @@
>  
>  #include "block/block.h"
>  #include "block/nbd.h"
> +#include "chardev/char.h"
>  #include "crypto/init.h"
>  
>  #include "qapi/error.h"
> @@ -75,6 +76,9 @@ static void help(void)
>  "             [,driver specific parameters...]\n"
>  "                         configure a block backend\n"
>  "\n"
> +"  --chardev <options>    configure a character device backend\n"
> +"                         (see the qemu(1) man page for possible options)\n"

If pointing to the manual page was good enough for --help, we could save
ourselves a ton of trouble :)

> +"\n"
>  "  --export [type=]nbd,device=<node-name>[,name=<export-name>]\n"
>  "           [,writable=on|off][,bitmap=<name>]\n"
>  "                         export the specified block node over NBD\n"
> @@ -96,10 +100,13 @@ QEMU_HELP_BOTTOM "\n",
>  enum {
>      OPTION_OBJECT = 256,
>      OPTION_BLOCKDEV,
> +    OPTION_CHARDEV,
>      OPTION_NBD_SERVER,
>      OPTION_EXPORT,
>  };
>  
> +extern QemuOptsList qemu_chardev_opts;
> +
>  static QemuOptsList qemu_object_opts = {
>      .name = "object",
>      .implied_opt_name = "qom-type",
> @@ -130,6 +137,7 @@ static int process_options(int argc, char *argv[], Error **errp)
>          {"help", no_argument, 0, 'h'},
>          {"object", required_argument, 0, OPTION_OBJECT},
>          {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
> +        {"chardev", required_argument, 0, OPTION_CHARDEV},
>          {"nbd-server", required_argument, 0, OPTION_NBD_SERVER},
>          {"export", required_argument, 0, OPTION_EXPORT},
>          {"version", no_argument, 0, 'V'},
> @@ -189,6 +197,17 @@ static int process_options(int argc, char *argv[], Error **errp)
>                  qapi_free_BlockdevOptions(options);
>                  break;
>              }
> +        case OPTION_CHARDEV:
> +            {
> +                QemuOpts *opts = qemu_opts_parse(&qemu_chardev_opts,
> +                                                 optarg, true, &error_fatal);
> +                if (!qemu_chr_new_from_opts(opts, NULL, &error_fatal)) {
> +                    /* No error, but NULL returned means help was printed */
> +                    exit(EXIT_SUCCESS);
> +                }
> +                qemu_opts_del(opts);
> +                break;
> +            }

Differs from vl.c similar to --object [PATCH 02]:

* Options are processed left to right.  Good.

* You use &qemu_chardev_opts instead of qemu_opts_find().  Good.

* You use qemu_opts_parse() instead of qemu_opts_parse_noisily().  Only
  here I can actually point to a loss of help.

  For options where the argument is essentially a tagged union, an
  option argument "help" usually lists the tags, an argument "T,help"
  shows help on the parameters that go with tag T.

  --chardev is such an option.  Consider:

    $ qemu-system-x86_64 --chardev help
    Available chardev backend types: 
      ringbuf
    [...]
      tty
    $ qemu-storage-daemon --chardev help
    Available chardev backend types: 
      pty
    [...]
      tty

  Works the same, only the available backend types differ.  Intentional?

  Next:

    $ qemu-system-x86_64 --chardev tty,help
    chardev options:
      append=<bool (on/off)>
      backend=<str>
      chardev=<str>
      cols=<num>
    [...]
      width=<num>
    [Exit 1 ]

  The second help isn't very helpful, and the exit code is wrong.  Not
  this patch's problem.

    $ qemu-storage-daemon --chardev tty,help
    qemu-storage-daemon: Invalid parameter 'help'
    [Exit 1 ]

  This patch's problem :)

Also like --object, --chardev is paired with a QMP command, namely
chardev-add, and the two differ for historical reasons.  If we want to
give the storage daemon a QAPIfied command line from the start, we'll
have to decide how to address this issue.


>          case OPTION_NBD_SERVER:
>              {
>                  Visitor *v;
> diff --git a/Makefile b/Makefile
> index b913d4d736..0e3e98582d 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -561,7 +561,7 @@ qemu-img.o: qemu-img-cmds.h
>  qemu-img$(EXESUF): qemu-img.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
>  qemu-nbd$(EXESUF): qemu-nbd.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
>  qemu-io$(EXESUF): qemu-io.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(COMMON_LDADDS)
> -qemu-storage-daemon$(EXESUF): qemu-storage-daemon.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(io-obj-y) $(qom-obj-y) $(storage-daemon-obj-y) $(COMMON_LDADDS)
> +qemu-storage-daemon$(EXESUF): qemu-storage-daemon.o $(authz-obj-y) $(block-obj-y) $(crypto-obj-y) $(chardev-obj-y) $(io-obj-y) $(qom-obj-y) $(storage-daemon-obj-y) $(COMMON_LDADDS)
>  
>  qemu-bridge-helper$(EXESUF): qemu-bridge-helper.o $(COMMON_LDADDS)



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 12/18] stubs: Update monitor stubs for qemu-storage-daemon
  2019-10-17 13:01 ` [RFC PATCH 12/18] stubs: Update monitor stubs for qemu-storage-daemon Kevin Wolf
@ 2019-11-08 16:45   ` Markus Armbruster
  0 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-08 16:45 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Kevin Wolf <kwolf@redhat.com> writes:

> Before we can add the monitor to qemu-storage-daemon, we need to add a
> few monitor stubs,

I can see just one: monitor_fdsets_cleanup().

>                    and we need to make sure that stubs that are actually
> implemented in the monitor core aren't linked so that we don't get
> linker errors because of duplicate symbols.

, by moving them from stubs/monitor.c to new stubs/monitor-core.c

>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  stubs/monitor-core.c | 21 +++++++++++++++++++++
>  stubs/monitor.c      | 15 ++-------------
>  stubs/Makefile.objs  |  1 +
>  3 files changed, 24 insertions(+), 13 deletions(-)
>  create mode 100644 stubs/monitor-core.c
>
> diff --git a/stubs/monitor-core.c b/stubs/monitor-core.c
> new file mode 100644
> index 0000000000..403c00a6d0
> --- /dev/null
> +++ b/stubs/monitor-core.c
> @@ -0,0 +1,21 @@
> +#include "qemu/osdep.h"
> +#include "monitor/monitor.h"
> +#include "qemu-common.h"
> +#include "qapi/qapi-emit-events.h"
> +
> +__thread Monitor *cur_mon;
> +
> +void monitor_init_qmp(Chardev *chr, bool pretty)
> +{
> +}
> +
> +void qapi_event_emit(QAPIEvent event, QDict *qdict)
> +{
> +}
> +
> +int monitor_vprintf(Monitor *mon, const char *fmt, va_list ap)
> +{
> +    abort();
> +}
> +
> +
> diff --git a/stubs/monitor.c b/stubs/monitor.c
> index c3e9a2e4dc..9403f8e72c 100644
> --- a/stubs/monitor.c
> +++ b/stubs/monitor.c
> @@ -1,14 +1,7 @@
>  #include "qemu/osdep.h"
>  #include "qapi/error.h"
> -#include "qapi/qapi-emit-events.h"
>  #include "monitor/monitor.h"
> -
> -__thread Monitor *cur_mon;
> -
> -int monitor_vprintf(Monitor *mon, const char *fmt, va_list ap)
> -{
> -    abort();
> -}
> +#include "../monitor/monitor-internal.h"

../ is ugly.  I don't have better ideas.

>  
>  int monitor_get_fd(Monitor *mon, const char *name, Error **errp)
>  {
> @@ -16,14 +9,10 @@ int monitor_get_fd(Monitor *mon, const char *name, Error **errp)
>      return -1;
>  }
>  
> -void monitor_init_qmp(Chardev *chr, bool pretty)
> -{
> -}
> -
>  void monitor_init_hmp(Chardev *chr, bool use_readline)
>  {
>  }
>  
> -void qapi_event_emit(QAPIEvent event, QDict *qdict)
> +void monitor_fdsets_cleanup(void)
>  {
>  }
> diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
> index 77fbf72576..ad4515ac70 100644
> --- a/stubs/Makefile.objs
> +++ b/stubs/Makefile.objs
> @@ -19,6 +19,7 @@ stub-obj-y += machine-init-done.o
>  stub-obj-y += migr-blocker.o
>  stub-obj-y += change-state-handler.o
>  stub-obj-y += monitor.o
> +stub-obj-y += monitor-core.o
>  stub-obj-y += notify-event.o
>  stub-obj-y += qtest.o
>  stub-obj-y += replay.o



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 13/18] qapi: Create module 'monitor'
  2019-10-17 13:01 ` [RFC PATCH 13/18] qapi: Create module 'monitor' Kevin Wolf
@ 2019-11-11  9:36   ` Markus Armbruster
  0 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-11  9:36 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Kevin Wolf <kwolf@redhat.com> writes:

> misc.json contains definitions that are related to the system emulator,
> so it can't be used for the storage daemon. This patch moves basic
> functionality that is related to the monitor itself into a new
> monitor.json, which could be used in tools as well.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>

I'm afraid this commit message is hard to understand unless the reader
already knows you're working towards having
storage-daemon/qapi/qapi-schema.json and qapi/qapi-schema.json share
modules.  Becomes clear only in PATCH 18.

Would qmp.json be more fitting than monitor.json?



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 15/18] qapi: Support empty modules
  2019-10-17 13:02 ` [RFC PATCH 15/18] qapi: Support empty modules Kevin Wolf
@ 2019-11-12  8:29   ` Markus Armbruster
  0 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-12  8:29 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Kevin Wolf <kwolf@redhat.com> writes:

> If you added an include file that doesn't contain any definitions, no
> source files would be generated for it. However, in other source files,
> you would still get an #include for the header files of the empty
> module.

Bug.

Cause: we generate #include module.h always, and the module.h when
visiting its first definition.  If there are no definitions, we don't.

> The intended behaviour is that empty source files are created for empty
> modules.

Yes.

>          This patch makes QAPISchema keep a list of all modules
> (including empty ones) and modifies visit() to first visit all modules
> in that list.

Minimally invasive fix.  Backends still initialize module output on
first visit_module(), but now all modules are visited upfront.

Separating "initialize module" from "switch to module" might be easier
to understand.  Idea, not demand.

> Some test reference outputs need to be updated due to the additional
> visitor calls.
>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  scripts/qapi/schema.py                   | 9 +++++++++
>  tests/qapi-schema/comments.out           | 2 ++
>  tests/qapi-schema/doc-bad-section.out    | 2 ++
>  tests/qapi-schema/doc-good.out           | 2 ++
>  tests/qapi-schema/empty.out              | 2 ++
>  tests/qapi-schema/event-case.out         | 2 ++
>  tests/qapi-schema/include-repetition.out | 4 ++++
>  tests/qapi-schema/include-simple.out     | 3 +++
>  tests/qapi-schema/indented-expr.out      | 2 ++
>  tests/qapi-schema/qapi-schema-test.out   | 4 ++++
>  10 files changed, 32 insertions(+)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 38041098bd..e1b034d67d 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -749,6 +749,7 @@ class QAPISchema(object):
>          self.docs = parser.docs
>          self._entity_list = []
>          self._entity_dict = {}
> +        self._modules = [os.path.basename(fname)]
>          self._predefining = True
>          self._def_predefineds()
>          self._predefining = False
> @@ -800,6 +801,8 @@ class QAPISchema(object):
>              main_info = main_info.parent
>          fname = os.path.relpath(include, os.path.dirname(main_info.fname))
>          self._def_entity(QAPISchemaInclude(fname, info))
> +        if fname not in self._modules:
> +            self._modules.append(fname)
>  
>      def _def_builtin_type(self, name, json_type, c_type):
>          self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
> @@ -1033,6 +1036,12 @@ class QAPISchema(object):
>          visitor.visit_begin(self)
>          module = None
>          visitor.visit_module(module)
> +
> +        # Make sure that all modules are visited, even if they contain no
> +        # entities
> +        for module in self._modules:
> +            visitor.visit_module(module)
> +

Slightly neater, I think:

           visitor.visit_begin(self)
  +
  +        # Visit all modules, to ensure @visitor sees them
  +        for module in self._modules:
  +            visitor.visit_module(module)
  +
           module = None
           visitor.visit_module(module)

This way, we keep starting with module None rather than whatever user
module comes last.  The .out diffs below then don't add a nother "module
None" line.

>          for entity in self._entity_list:
>              if visitor.visit_needed(entity):
>                  if entity.module != module:
> diff --git a/tests/qapi-schema/comments.out b/tests/qapi-schema/comments.out
> index 273f0f54e1..fa7e95d1cc 100644
> --- a/tests/qapi-schema/comments.out
> +++ b/tests/qapi-schema/comments.out
> @@ -1,4 +1,6 @@
>  module None
> +module comments.json
> +module None
>  object q_empty
>  enum QType
>      prefix QTYPE
[...]



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 16/18] qapi: Create 'pragma' module
  2019-10-17 13:02 ` [RFC PATCH 16/18] qapi: Create 'pragma' module Kevin Wolf
@ 2019-11-12  9:41   ` Markus Armbruster
  0 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-12  9:41 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Kevin Wolf <kwolf@redhat.com> writes:

> We want to share the whitelists between the system emulator schema and
> the storage daemon schema, so move all the pragmas from the main schema
> file into a separate file that can be included from both.

Confusing because the storage daemon schema doesn't exist at this point.
PATCH 13's commit message has the same issue.

I'll revisit this when I review PATCH 18.

>
> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  qapi/pragma.json      | 24 ++++++++++++++++++++++++
>  qapi/qapi-schema.json | 25 +------------------------
>  qapi/Makefile.objs    |  2 +-
>  3 files changed, 26 insertions(+), 25 deletions(-)
>  create mode 100644 qapi/pragma.json
>
> diff --git a/qapi/pragma.json b/qapi/pragma.json
> new file mode 100644
> index 0000000000..cffae27666
> --- /dev/null
> +++ b/qapi/pragma.json
> @@ -0,0 +1,24 @@
> +{ 'pragma': { 'doc-required': true } }
> +
> +# Whitelists to permit QAPI rule violations; think twice before you
> +# add to them!
> +{ 'pragma': {
> +    # Commands allowed to return a non-dictionary:
> +    'returns-whitelist': [
> +        'human-monitor-command',
> +        'qom-get',
> +        'query-migrate-cache-size',
> +        'query-tpm-models',
> +        'query-tpm-types',
> +        'ringbuf-read' ],
> +    'name-case-whitelist': [
> +        'ACPISlotType',             # DIMM, visible through query-acpi-ospm-status
> +        'CpuInfoMIPS',              # PC, visible through query-cpu
> +        'CpuInfoTricore',           # PC, visible through query-cpu
> +        'BlockdevVmdkSubformat',    # all members, to match VMDK spec spellings
> +        'BlockdevVmdkAdapterType',  # legacyESX, to match VMDK spec spellings
> +        'QapiErrorClass',           # all members, visible through errors
> +        'UuidInfo',                 # UUID, visible through query-uuid
> +        'X86CPURegister32',         # all members, visible indirectly through qom-get
> +        'CpuInfo'                   # CPU, visible through query-cpu
> +    ] } }
> diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json
> index be90422ffe..85b4048535 100644
> --- a/qapi/qapi-schema.json
> +++ b/qapi/qapi-schema.json
> @@ -49,30 +49,7 @@
>  #
>  ##
>  
> -{ 'pragma': { 'doc-required': true } }
> -
> -# Whitelists to permit QAPI rule violations; think twice before you
> -# add to them!
> -{ 'pragma': {
> -    # Commands allowed to return a non-dictionary:
> -    'returns-whitelist': [
> -        'human-monitor-command',
> -        'qom-get',
> -        'query-migrate-cache-size',
> -        'query-tpm-models',
> -        'query-tpm-types',
> -        'ringbuf-read' ],
> -    'name-case-whitelist': [
> -        'ACPISlotType',             # DIMM, visible through query-acpi-ospm-status
> -        'CpuInfoMIPS',              # PC, visible through query-cpu
> -        'CpuInfoTricore',           # PC, visible through query-cpu
> -        'BlockdevVmdkSubformat',    # all members, to match VMDK spec spellings
> -        'BlockdevVmdkAdapterType',  # legacyESX, to match VMDK spec spellings
> -        'QapiErrorClass',           # all members, visible through errors
> -        'UuidInfo',                 # UUID, visible through query-uuid
> -        'X86CPURegister32',         # all members, visible indirectly through qom-get
> -        'CpuInfo'                   # CPU, visible through query-cpu
> -    ] } }
> +{ 'include': 'pragma.json' }
>  
>  # Documentation generated with qapi-gen.py is in source order, with
>  # included sub-schemas inserted at the first include directive
> diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
> index 519b6f1a8e..3e04e299ed 100644
> --- a/qapi/Makefile.objs
> +++ b/qapi/Makefile.objs
> @@ -7,7 +7,7 @@ util-obj-y += qapi-util.o
>  
>  QAPI_COMMON_MODULES = audio authz block-core block char common crypto
>  QAPI_COMMON_MODULES += dump error introspect job machine migration misc monitor
> -QAPI_COMMON_MODULES += net qdev qom rdma rocker run-state sockets tpm
> +QAPI_COMMON_MODULES += net pragma qdev qom rdma rocker run-state sockets tpm
>  QAPI_COMMON_MODULES += trace transaction ui
>  QAPI_TARGET_MODULES = machine-target misc-target
>  QAPI_MODULES = $(QAPI_COMMON_MODULES) $(QAPI_TARGET_MODULES)

Only works because we accept and ignore names in these whitelists that
don't exist in the schema.

Which parts of the whitelists are actually needed in the storage daemon?



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option
  2019-10-17 13:02 ` [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option Kevin Wolf
  2019-11-06 14:32   ` Max Reitz
  2019-11-08  8:59   ` Markus Armbruster
@ 2019-11-12 14:25   ` Markus Armbruster
  2019-11-13 10:58     ` Kevin Wolf
  2 siblings, 1 reply; 65+ messages in thread
From: Markus Armbruster @ 2019-11-12 14:25 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Kevin Wolf <kwolf@redhat.com> writes:

> This adds and parses the --monitor option, so that a QMP monitor can be
> used in the storage daemon. The monitor offers commands defined in the
> QAPI schema at storage-daemon/qapi/qapi-schema.json.

I feel we should explain the module sharing between the two QAPI
schemata here.  It's not exactly trivial, as we'll see below.

> Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> ---
>  storage-daemon/qapi/qapi-schema.json | 15 ++++++++++++
>  qemu-storage-daemon.c                | 34 ++++++++++++++++++++++++++++
>  Makefile                             | 30 ++++++++++++++++++++++++
>  Makefile.objs                        |  4 ++--
>  monitor/Makefile.objs                |  2 ++
>  qapi/Makefile.objs                   |  5 ++++
>  qom/Makefile.objs                    |  1 +
>  scripts/qapi/gen.py                  |  5 ++++
>  storage-daemon/Makefile.objs         |  1 +
>  storage-daemon/qapi/Makefile.objs    |  1 +
>  10 files changed, 96 insertions(+), 2 deletions(-)
>  create mode 100644 storage-daemon/qapi/qapi-schema.json
>  create mode 100644 storage-daemon/Makefile.objs
>  create mode 100644 storage-daemon/qapi/Makefile.objs
>
> diff --git a/storage-daemon/qapi/qapi-schema.json b/storage-daemon/qapi/qapi-schema.json
> new file mode 100644
> index 0000000000..58c561ebea
> --- /dev/null
> +++ b/storage-daemon/qapi/qapi-schema.json
> @@ -0,0 +1,15 @@
> +# -*- Mode: Python -*-
> +
> +{ 'include': '../../qapi/pragma.json' }
> +
> +{ 'include': '../../qapi/block.json' }
> +{ 'include': '../../qapi/block-core.json' }
> +{ 'include': '../../qapi/char.json' }
> +{ 'include': '../../qapi/common.json' }
> +{ 'include': '../../qapi/crypto.json' }
> +{ 'include': '../../qapi/introspect.json' }
> +{ 'include': '../../qapi/job.json' }
> +{ 'include': '../../qapi/monitor.json' }
> +{ 'include': '../../qapi/qom.json' }
> +{ 'include': '../../qapi/sockets.json' }
> +{ 'include': '../../qapi/transaction.json' }
> diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
> index 46e0a6ea56..4939e6b41f 100644
> --- a/qemu-storage-daemon.c
> +++ b/qemu-storage-daemon.c
> @@ -28,12 +28,16 @@
>  #include "block/nbd.h"
>  #include "chardev/char.h"
>  #include "crypto/init.h"
> +#include "monitor/monitor.h"
> +#include "monitor/monitor-internal.h"
>  
>  #include "qapi/error.h"
>  #include "qapi/qapi-commands-block.h"
>  #include "qapi/qapi-commands-block-core.h"
> +#include "qapi/qapi-commands-monitor.h"

Aren't these three redundant with ...

>  #include "qapi/qapi-visit-block.h"
>  #include "qapi/qapi-visit-block-core.h"
> +#include "qapi/qmp/qstring.h"
>  #include "qapi/qobject-input-visitor.h"
>  
>  #include "qemu-common.h"
> @@ -46,6 +50,8 @@
>  #include "qemu/option.h"
>  #include "qom/object_interfaces.h"
>  
> +#include "storage-daemon/qapi/qapi-commands.h"
> +

... this one now?

>  #include "sysemu/runstate.h"
>  #include "trace/control.h"
>  
> @@ -58,6 +64,11 @@ void qemu_system_killed(int signal, pid_t pid)
>      exit_requested = true;
>  }
>  
> +void qmp_quit(Error **errp)
> +{
> +    exit_requested = true;
> +}
> +
>  static void help(void)
>  {
>      printf(
> @@ -101,6 +112,7 @@ enum {
>      OPTION_OBJECT = 256,
>      OPTION_BLOCKDEV,
>      OPTION_CHARDEV,
> +    OPTION_MONITOR,
>      OPTION_NBD_SERVER,
>      OPTION_EXPORT,
>  };

Recommend to keep these sorted alphabetically.

> @@ -116,6 +128,17 @@ static QemuOptsList qemu_object_opts = {
>      },
>  };
>  
> +static void init_qmp_commands(void)
> +{
> +    qmp_init_marshal(&qmp_commands);
> +    qmp_register_command(&qmp_commands, "query-qmp-schema",
> +                         qmp_query_qmp_schema, QCO_ALLOW_PRECONFIG);
> +
> +    QTAILQ_INIT(&qmp_cap_negotiation_commands);
> +    qmp_register_command(&qmp_cap_negotiation_commands, "qmp_capabilities",
> +                         qmp_marshal_qmp_capabilities, QCO_ALLOW_PRECONFIG);
> +}

Duplicates monitor_init_qmp_commands() less two 'gen': false commands
that don't exist in the storage daemon.

Hmm, could we let commands.py generate command registration even for
'gen': false commands?

> +
>  static void init_export(BlockExport *export, Error **errp)
>  {
>      switch (export->type) {
> @@ -138,6 +161,7 @@ static int process_options(int argc, char *argv[], Error **errp)
>          {"object", required_argument, 0, OPTION_OBJECT},
>          {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
>          {"chardev", required_argument, 0, OPTION_CHARDEV},
> +        {"monitor", required_argument, 0, OPTION_MONITOR},
>          {"nbd-server", required_argument, 0, OPTION_NBD_SERVER},
>          {"export", required_argument, 0, OPTION_EXPORT},
>          {"version", no_argument, 0, 'V'},
> @@ -208,6 +232,14 @@ static int process_options(int argc, char *argv[], Error **errp)
>                  qemu_opts_del(opts);
>                  break;
>              }
> +        case OPTION_MONITOR:
> +            {
> +                QemuOpts *opts = qemu_opts_parse(&qemu_mon_opts,
> +                                                 optarg, true, &error_fatal);
> +                monitor_init_opts(opts, false, &error_fatal);
> +                qemu_opts_del(opts);
> +                break;
> +            }
>          case OPTION_NBD_SERVER:
>              {
>                  Visitor *v;

Recommend to keep the switch cases sorted.  Perhaps put help first.
Order the short options string and the long_options[] array to match.

> @@ -272,6 +304,8 @@ int main(int argc, char *argv[])
>      qemu_add_opts(&qemu_trace_opts);
>      qcrypto_init(&error_fatal);
>      bdrv_init();
> +    monitor_init_globals_core();
> +    init_qmp_commands();
>  
>      if (qemu_init_main_loop(&local_err)) {
>          error_report_err(local_err);
> diff --git a/Makefile b/Makefile
> index 0e3e98582d..e367d2b28a 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -121,7 +121,26 @@ GENERATED_QAPI_FILES += $(QAPI_MODULES:%=qapi/qapi-events-%.c)
>  GENERATED_QAPI_FILES += qapi/qapi-introspect.c qapi/qapi-introspect.h
>  GENERATED_QAPI_FILES += qapi/qapi-doc.texi
>  
> +GENERATED_STORAGE_DAEMON_QAPI_FILES = storage-daemon/qapi/qapi-builtin-types.h
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-types.c
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-visit.h
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-visit.c

Any particular reason for not reusing qapi/qapi-builtin-*?  See also the
recipe for storage-daemon/qapi/qapi-gen-timestamp below.

> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-commands.h
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-commands.c
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-emit-events.h
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-emit-events.c
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-events.h
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-events.c
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-introspect.h
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-introspect.c
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-types.h
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-types.c
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-visit.h
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-visit.c
> +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-doc.texi

These are the files generated for the storage daemon's main module.  The
files generated for its sub-modules (all shared right now) are absent.
I think this could use a comment.

See also scripts/qapi/gen.py below.

> +
>  generated-files-y += $(GENERATED_QAPI_FILES)
> +generated-files-y += $(GENERATED_STORAGE_DAEMON_QAPI_FILES)
>  
>  generated-files-y += trace/generated-tcg-tracers.h
>  
> @@ -616,6 +635,17 @@ qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
>  		"GEN","$(@:%-timestamp=%)")
>  	@>$@
>  
> +qapi-modules-storage-daemon = \
> +	$(SRC_PATH)/storage-daemon/qapi/qapi-schema.json \
> +    $(QAPI_MODULES_STORAGE_DAEMON:%=$(SRC_PATH)/qapi/%.json)
> +
> +$(GENERATED_STORAGE_DAEMON_QAPI_FILES): storage-daemon/qapi/qapi-gen-timestamp ;
> +storage-daemon/qapi/qapi-gen-timestamp: $(qapi-modules-storage-daemon) $(qapi-py)
> +	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
> +		-o "storage-daemon/qapi" -b $<, \

If we can reuse qapi/qapi-builtin-*, then omit -b to suppress generating
redundant copies.

> +		"GEN","$(@:%-timestamp=%)")
> +	@>$@
> +
>  QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qapi-commands.h)
>  $(qga-obj-y): $(QGALIB_GEN)
>  
> diff --git a/Makefile.objs b/Makefile.objs
> index b667d3f07b..d4e0daddee 100644
> --- a/Makefile.objs
> +++ b/Makefile.objs
> @@ -41,8 +41,8 @@ io-obj-y = io/
>  # storage-daemon-obj-y is code used by qemu-storage-daemon (these objects are
>  # used for system emulation, too, but specified separately there)
>  
> -storage-daemon-obj-y = block/
> -storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o
> +storage-daemon-obj-y = block/ monitor/ qapi/ qom/ storage-daemon/
> +storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o job-qmp.o
>  storage-daemon-obj-$(CONFIG_WIN32) += os-win32.o
>  storage-daemon-obj-$(CONFIG_POSIX) += os-posix.o
>  
> diff --git a/monitor/Makefile.objs b/monitor/Makefile.objs
> index 15eb6380c5..6e4ef60601 100644
> --- a/monitor/Makefile.objs
> +++ b/monitor/Makefile.objs
> @@ -2,3 +2,5 @@ obj-y += misc.o
>  common-obj-y += monitor.o qmp.o hmp.o
>  common-obj-y += qmp-cmds.o qmp-cmds-monitor.o
>  common-obj-y += hmp-cmds.o
> +
> +storage-daemon-obj-y += monitor.o qmp.o qmp-cmds-monitor.o
> diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
> index 3e04e299ed..03d256f0a4 100644
> --- a/qapi/Makefile.objs
> +++ b/qapi/Makefile.objs
> @@ -30,3 +30,8 @@ obj-y += $(QAPI_TARGET_MODULES:%=qapi-events-%.o)
>  obj-y += qapi-events.o
>  obj-y += $(QAPI_TARGET_MODULES:%=qapi-commands-%.o)
>  obj-y += qapi-commands.o
> +
> +QAPI_MODULES_STORAGE_DAEMON = block block-core char common crypto introspect
> +QAPI_MODULES_STORAGE_DAEMON += job monitor qom sockets pragma transaction
> +
> +storage-daemon-obj-y += $(QAPI_MODULES_STORAGE_DAEMON:%=qapi-commands-%.o)
> diff --git a/qom/Makefile.objs b/qom/Makefile.objs
> index f9d77350ac..1b45d104ba 100644
> --- a/qom/Makefile.objs
> +++ b/qom/Makefile.objs
> @@ -2,3 +2,4 @@ qom-obj-y = object.o container.o qom-qobject.o
>  qom-obj-y += object_interfaces.o
>  
>  common-obj-$(CONFIG_SOFTMMU) += qom-hmp-cmds.o qom-qmp-cmds.o
> +storage-daemon-obj-y += qom-qmp-cmds.o
> diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> index 796c17c38a..c25634f673 100644
> --- a/scripts/qapi/gen.py
> +++ b/scripts/qapi/gen.py
> @@ -44,6 +44,11 @@ class QAPIGen(object):
>          return ''
>  
>      def write(self, output_dir):
> +        # Include paths starting with ../ are used to reuse modules of the main
> +        # schema in specialised schemas. Don't overwrite the files that are
> +        # already generated for the main schema.
> +        if self.fname.startswith('../'):
> +            return
>          pathname = os.path.join(output_dir, self.fname)
>          dir = os.path.dirname(pathname)
>          if dir:

Ugh.

tests/qapi-schema/include/sub-module.json violates the stated assumption
"Include paths starting with ../ are used to reuse modules of the main
schema":

    { 'include': '../sub-sub-module.json' }

Works anyway because it's only used via

    { 'include': 'include/sub-module.json' }

where the ../ is folded with the include/ and vanishes.  self.fname is
relative to the main schema's directory.

Still, baking the assumption into gen.py makes it a part of the QAPI
schema language.  It needs to be documented as such.

Complication: because we generate array types and supporting code code
only when they're actually used, the code generated for a shared module
can depend on how it is used outside the shared module.

Right now, storage-daemon/qapi/qapi-schema.json contains nothing but
modules shared with qapi/qapi-schema.json.  Works.

If storage-daemon/qapi/qapi-schema.json ever acquires an array reference
that is not also in qapi/qapi-schema.json, the generated C won't
compile.

This also needs to be documented because it's anything but obvious.

Instead of suppressing code generated for reused modules outright, we
could somehow avoid clobbering the main schema's output.  I guess we
can't use -p for that, because it makes interferes with reusing the
hand-written QMP code.  Can we make do with -o?

By doing that, we sidestep making ../ special in the QAPI schema
language.  We then have two options for the storage daemon.

One, use the code generated for the storage daemon schema.  Simple,
robust, but we compile more.

Two, we use the code generated for the main schema (same as now).  We
compile less, but expose ourselves to the array problem described
above.

> diff --git a/storage-daemon/Makefile.objs b/storage-daemon/Makefile.objs
> new file mode 100644
> index 0000000000..cfe6beee52
> --- /dev/null
> +++ b/storage-daemon/Makefile.objs
> @@ -0,0 +1 @@
> +storage-daemon-obj-y += qapi/
> diff --git a/storage-daemon/qapi/Makefile.objs b/storage-daemon/qapi/Makefile.objs
> new file mode 100644
> index 0000000000..df8946bdae
> --- /dev/null
> +++ b/storage-daemon/qapi/Makefile.objs
> @@ -0,0 +1 @@
> +storage-daemon-obj-y += qapi-commands.o qapi-introspect.o

Now let's take a step back and examine the problem this patch solves.

The storage daemon needs a control interface.  Instead of inventing a
new one, we want to reuse QMP, both the protocol and the actual
commands, as far as they make sense in the storage daemon.  Makes sense.

To get the QMP protocol, the storage daemon needs a QAPI schema.

To reuse existing QMP commands, the storage daemon's QAPI schema needs
to include the parts of the existing QAPI schema that define them.

The stupidest possible way to do that is copying.  Unworkable; too much
code, too hard to keep it in sync.

Instead, this patch reuses schema modules.  Summary of issues so far:

* The include directive lets us avoid duplicating schema code, but to
  avoid duplicating the generated code, we resort to a gross hack.

  The hack needs a certain amount of rather awkward documentation (still
  to be written).

  Putting non-shared bits into the storage daemon's schema risks array
  trouble.

  Accepting duplicated generated code might be the lesser evil.

* The discussion of how this module reuse thing works, assumptions,
  shortcomings need to go on the permanent record, be it code comments,
  something in docs/, or at least commit messages.

  Commit messages of PATCH 13 and 16 need to hint at where we're going
  to make sense.  I figure that's easier to do once we got this patch
  into shape.

* We get a bunch of commands that make no sense in the storage daemon,
  see my review of PATCH 04 for examples.

  To avoid that, we need to split modules.  May well be a good idea
  anyway; qapi/block-core.json is by far the fattest module: 850
  non-blank, non-comment lines, 4.5 times fatter than the runner-up
  ui.json.

  Keeping unwanted commands out of the storage daemon will be an ongoing
  process: we need to keep unwanted commands from creeping into modules
  shared with the storage daemon.  Hopefully, properly focused modules
  can make such creep unlikely.

A less hacky solution is to extend the QAPI schema language from one set
of QMP commands and events to many.  Hard to justify when we have just
two sets.

I'm afraid my review isn't a straight "do A, B and C to get my
blessings".  I hope it's useful anyway.



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option
  2019-11-12 14:25   ` Markus Armbruster
@ 2019-11-13 10:58     ` Kevin Wolf
  2019-11-13 13:53       ` Markus Armbruster
  0 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-11-13 10:58 UTC (permalink / raw)
  To: Markus Armbruster; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Am 12.11.2019 um 15:25 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
> 
> > This adds and parses the --monitor option, so that a QMP monitor can be
> > used in the storage daemon. The monitor offers commands defined in the
> > QAPI schema at storage-daemon/qapi/qapi-schema.json.
> 
> I feel we should explain the module sharing between the two QAPI
> schemata here.  It's not exactly trivial, as we'll see below.
> 
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> > ---
> >  storage-daemon/qapi/qapi-schema.json | 15 ++++++++++++
> >  qemu-storage-daemon.c                | 34 ++++++++++++++++++++++++++++
> >  Makefile                             | 30 ++++++++++++++++++++++++
> >  Makefile.objs                        |  4 ++--
> >  monitor/Makefile.objs                |  2 ++
> >  qapi/Makefile.objs                   |  5 ++++
> >  qom/Makefile.objs                    |  1 +
> >  scripts/qapi/gen.py                  |  5 ++++
> >  storage-daemon/Makefile.objs         |  1 +
> >  storage-daemon/qapi/Makefile.objs    |  1 +
> >  10 files changed, 96 insertions(+), 2 deletions(-)
> >  create mode 100644 storage-daemon/qapi/qapi-schema.json
> >  create mode 100644 storage-daemon/Makefile.objs
> >  create mode 100644 storage-daemon/qapi/Makefile.objs
> >
> > diff --git a/storage-daemon/qapi/qapi-schema.json b/storage-daemon/qapi/qapi-schema.json
> > new file mode 100644
> > index 0000000000..58c561ebea
> > --- /dev/null
> > +++ b/storage-daemon/qapi/qapi-schema.json
> > @@ -0,0 +1,15 @@
> > +# -*- Mode: Python -*-
> > +
> > +{ 'include': '../../qapi/pragma.json' }
> > +
> > +{ 'include': '../../qapi/block.json' }
> > +{ 'include': '../../qapi/block-core.json' }
> > +{ 'include': '../../qapi/char.json' }
> > +{ 'include': '../../qapi/common.json' }
> > +{ 'include': '../../qapi/crypto.json' }
> > +{ 'include': '../../qapi/introspect.json' }
> > +{ 'include': '../../qapi/job.json' }
> > +{ 'include': '../../qapi/monitor.json' }
> > +{ 'include': '../../qapi/qom.json' }
> > +{ 'include': '../../qapi/sockets.json' }
> > +{ 'include': '../../qapi/transaction.json' }
> > diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
> > index 46e0a6ea56..4939e6b41f 100644
> > --- a/qemu-storage-daemon.c
> > +++ b/qemu-storage-daemon.c
> > @@ -28,12 +28,16 @@
> >  #include "block/nbd.h"
> >  #include "chardev/char.h"
> >  #include "crypto/init.h"
> > +#include "monitor/monitor.h"
> > +#include "monitor/monitor-internal.h"
> >  
> >  #include "qapi/error.h"
> >  #include "qapi/qapi-commands-block.h"
> >  #include "qapi/qapi-commands-block-core.h"
> > +#include "qapi/qapi-commands-monitor.h"
> 
> Aren't these three redundant with ...
> 
> >  #include "qapi/qapi-visit-block.h"
> >  #include "qapi/qapi-visit-block-core.h"
> > +#include "qapi/qmp/qstring.h"
> >  #include "qapi/qobject-input-visitor.h"
> >  
> >  #include "qemu-common.h"
> > @@ -46,6 +50,8 @@
> >  #include "qemu/option.h"
> >  #include "qom/object_interfaces.h"
> >  
> > +#include "storage-daemon/qapi/qapi-commands.h"
> > +
> 
> ... this one now?

Looks like it.

> >  #include "sysemu/runstate.h"
> >  #include "trace/control.h"
> >  
> > @@ -58,6 +64,11 @@ void qemu_system_killed(int signal, pid_t pid)
> >      exit_requested = true;
> >  }
> >  
> > +void qmp_quit(Error **errp)
> > +{
> > +    exit_requested = true;
> > +}
> > +
> >  static void help(void)
> >  {
> >      printf(
> > @@ -101,6 +112,7 @@ enum {
> >      OPTION_OBJECT = 256,
> >      OPTION_BLOCKDEV,
> >      OPTION_CHARDEV,
> > +    OPTION_MONITOR,
> >      OPTION_NBD_SERVER,
> >      OPTION_EXPORT,
> >  };
> 
> Recommend to keep these sorted alphabetically.
> 
> > @@ -116,6 +128,17 @@ static QemuOptsList qemu_object_opts = {
> >      },
> >  };
> >  
> > +static void init_qmp_commands(void)
> > +{
> > +    qmp_init_marshal(&qmp_commands);
> > +    qmp_register_command(&qmp_commands, "query-qmp-schema",
> > +                         qmp_query_qmp_schema, QCO_ALLOW_PRECONFIG);
> > +
> > +    QTAILQ_INIT(&qmp_cap_negotiation_commands);
> > +    qmp_register_command(&qmp_cap_negotiation_commands, "qmp_capabilities",
> > +                         qmp_marshal_qmp_capabilities, QCO_ALLOW_PRECONFIG);
> > +}
> 
> Duplicates monitor_init_qmp_commands() less two 'gen': false commands
> that don't exist in the storage daemon.
> 
> Hmm, could we let commands.py generate command registration even for
> 'gen': false commands?

Possibly. Are you telling me to do it or is it just an idea for a later
cleanup?

> > +
> >  static void init_export(BlockExport *export, Error **errp)
> >  {
> >      switch (export->type) {
> > @@ -138,6 +161,7 @@ static int process_options(int argc, char *argv[], Error **errp)
> >          {"object", required_argument, 0, OPTION_OBJECT},
> >          {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
> >          {"chardev", required_argument, 0, OPTION_CHARDEV},
> > +        {"monitor", required_argument, 0, OPTION_MONITOR},
> >          {"nbd-server", required_argument, 0, OPTION_NBD_SERVER},
> >          {"export", required_argument, 0, OPTION_EXPORT},
> >          {"version", no_argument, 0, 'V'},
> > @@ -208,6 +232,14 @@ static int process_options(int argc, char *argv[], Error **errp)
> >                  qemu_opts_del(opts);
> >                  break;
> >              }
> > +        case OPTION_MONITOR:
> > +            {
> > +                QemuOpts *opts = qemu_opts_parse(&qemu_mon_opts,
> > +                                                 optarg, true, &error_fatal);
> > +                monitor_init_opts(opts, false, &error_fatal);
> > +                qemu_opts_del(opts);
> > +                break;
> > +            }
> >          case OPTION_NBD_SERVER:
> >              {
> >                  Visitor *v;
> 
> Recommend to keep the switch cases sorted.  Perhaps put help first.
> Order the short options string and the long_options[] array to match.
> 
> > @@ -272,6 +304,8 @@ int main(int argc, char *argv[])
> >      qemu_add_opts(&qemu_trace_opts);
> >      qcrypto_init(&error_fatal);
> >      bdrv_init();
> > +    monitor_init_globals_core();
> > +    init_qmp_commands();
> >  
> >      if (qemu_init_main_loop(&local_err)) {
> >          error_report_err(local_err);
> > diff --git a/Makefile b/Makefile
> > index 0e3e98582d..e367d2b28a 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -121,7 +121,26 @@ GENERATED_QAPI_FILES += $(QAPI_MODULES:%=qapi/qapi-events-%.c)
> >  GENERATED_QAPI_FILES += qapi/qapi-introspect.c qapi/qapi-introspect.h
> >  GENERATED_QAPI_FILES += qapi/qapi-doc.texi
> >  
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES = storage-daemon/qapi/qapi-builtin-types.h
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-types.c
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-visit.h
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-visit.c
> 
> Any particular reason for not reusing qapi/qapi-builtin-*?  See also the
> recipe for storage-daemon/qapi/qapi-gen-timestamp below.

Hm, I guess not. I'm just tweaking stuff until it seems to work, and
this seemed to work, so...

But as far as I can see, the generated file are never actually used, so
I'll get rid of them.

> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-commands.h
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-commands.c
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-emit-events.h
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-emit-events.c
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-events.h
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-events.c
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-introspect.h
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-introspect.c
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-types.h
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-types.c
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-visit.h
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-visit.c
> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-doc.texi
> 
> These are the files generated for the storage daemon's main module.  The
> files generated for its sub-modules (all shared right now) are absent.
> I think this could use a comment.

Ok.

> See also scripts/qapi/gen.py below.
> 
> > +
> >  generated-files-y += $(GENERATED_QAPI_FILES)
> > +generated-files-y += $(GENERATED_STORAGE_DAEMON_QAPI_FILES)
> >  
> >  generated-files-y += trace/generated-tcg-tracers.h
> >  
> > @@ -616,6 +635,17 @@ qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
> >  		"GEN","$(@:%-timestamp=%)")
> >  	@>$@
> >  
> > +qapi-modules-storage-daemon = \
> > +	$(SRC_PATH)/storage-daemon/qapi/qapi-schema.json \
> > +    $(QAPI_MODULES_STORAGE_DAEMON:%=$(SRC_PATH)/qapi/%.json)
> > +
> > +$(GENERATED_STORAGE_DAEMON_QAPI_FILES): storage-daemon/qapi/qapi-gen-timestamp ;
> > +storage-daemon/qapi/qapi-gen-timestamp: $(qapi-modules-storage-daemon) $(qapi-py)
> > +	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
> > +		-o "storage-daemon/qapi" -b $<, \
> 
> If we can reuse qapi/qapi-builtin-*, then omit -b to suppress generating
> redundant copies.
> 
> > +		"GEN","$(@:%-timestamp=%)")
> > +	@>$@
> > +
> >  QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qapi-commands.h)
> >  $(qga-obj-y): $(QGALIB_GEN)
> >  
> > diff --git a/Makefile.objs b/Makefile.objs
> > index b667d3f07b..d4e0daddee 100644
> > --- a/Makefile.objs
> > +++ b/Makefile.objs
> > @@ -41,8 +41,8 @@ io-obj-y = io/
> >  # storage-daemon-obj-y is code used by qemu-storage-daemon (these objects are
> >  # used for system emulation, too, but specified separately there)
> >  
> > -storage-daemon-obj-y = block/
> > -storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o
> > +storage-daemon-obj-y = block/ monitor/ qapi/ qom/ storage-daemon/
> > +storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o job-qmp.o
> >  storage-daemon-obj-$(CONFIG_WIN32) += os-win32.o
> >  storage-daemon-obj-$(CONFIG_POSIX) += os-posix.o
> >  
> > diff --git a/monitor/Makefile.objs b/monitor/Makefile.objs
> > index 15eb6380c5..6e4ef60601 100644
> > --- a/monitor/Makefile.objs
> > +++ b/monitor/Makefile.objs
> > @@ -2,3 +2,5 @@ obj-y += misc.o
> >  common-obj-y += monitor.o qmp.o hmp.o
> >  common-obj-y += qmp-cmds.o qmp-cmds-monitor.o
> >  common-obj-y += hmp-cmds.o
> > +
> > +storage-daemon-obj-y += monitor.o qmp.o qmp-cmds-monitor.o
> > diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
> > index 3e04e299ed..03d256f0a4 100644
> > --- a/qapi/Makefile.objs
> > +++ b/qapi/Makefile.objs
> > @@ -30,3 +30,8 @@ obj-y += $(QAPI_TARGET_MODULES:%=qapi-events-%.o)
> >  obj-y += qapi-events.o
> >  obj-y += $(QAPI_TARGET_MODULES:%=qapi-commands-%.o)
> >  obj-y += qapi-commands.o
> > +
> > +QAPI_MODULES_STORAGE_DAEMON = block block-core char common crypto introspect
> > +QAPI_MODULES_STORAGE_DAEMON += job monitor qom sockets pragma transaction
> > +
> > +storage-daemon-obj-y += $(QAPI_MODULES_STORAGE_DAEMON:%=qapi-commands-%.o)
> > diff --git a/qom/Makefile.objs b/qom/Makefile.objs
> > index f9d77350ac..1b45d104ba 100644
> > --- a/qom/Makefile.objs
> > +++ b/qom/Makefile.objs
> > @@ -2,3 +2,4 @@ qom-obj-y = object.o container.o qom-qobject.o
> >  qom-obj-y += object_interfaces.o
> >  
> >  common-obj-$(CONFIG_SOFTMMU) += qom-hmp-cmds.o qom-qmp-cmds.o
> > +storage-daemon-obj-y += qom-qmp-cmds.o
> > diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
> > index 796c17c38a..c25634f673 100644
> > --- a/scripts/qapi/gen.py
> > +++ b/scripts/qapi/gen.py
> > @@ -44,6 +44,11 @@ class QAPIGen(object):
> >          return ''
> >  
> >      def write(self, output_dir):
> > +        # Include paths starting with ../ are used to reuse modules of the main
> > +        # schema in specialised schemas. Don't overwrite the files that are
> > +        # already generated for the main schema.
> > +        if self.fname.startswith('../'):
> > +            return
> >          pathname = os.path.join(output_dir, self.fname)
> >          dir = os.path.dirname(pathname)
> >          if dir:
> 
> Ugh.

That's why I asked you before writing it like this. :-)

> tests/qapi-schema/include/sub-module.json violates the stated assumption
> "Include paths starting with ../ are used to reuse modules of the main
> schema":
> 
>     { 'include': '../sub-sub-module.json' }
> 
> Works anyway because it's only used via
> 
>     { 'include': 'include/sub-module.json' }
> 
> where the ../ is folded with the include/ and vanishes.  self.fname is
> relative to the main schema's directory.
> 
> Still, baking the assumption into gen.py makes it a part of the QAPI
> schema language.  It needs to be documented as such.

Maybe you'd like it better if we made it explicit?

    { 'include': '../../qapi/block.json', 'gen': false }

But how do I actually get this information down to QAPIGen? This brings
us back to the problem that the QAPI generator doesn't really have a
notion of modules, but only entities have a module name and the first
entity to use a module name causes it to be generated. We don't have
access to the QAPISchemaInclude that created the module.

Even the QAPISchema._modules that I introduced only contains filenames.
Should it contain QAPISchemaInclude objects instead?

> Complication: because we generate array types and supporting code code
> only when they're actually used, the code generated for a shared module
> can depend on how it is used outside the shared module.
> 
> Right now, storage-daemon/qapi/qapi-schema.json contains nothing but
> modules shared with qapi/qapi-schema.json.  Works.
> 
> If storage-daemon/qapi/qapi-schema.json ever acquires an array reference
> that is not also in qapi/qapi-schema.json, the generated C won't
> compile.
> 
> This also needs to be documented because it's anything but obvious.

A comment in the storage daemon main module?

> Instead of suppressing code generated for reused modules outright, we
> could somehow avoid clobbering the main schema's output.  I guess we
> can't use -p for that, because it makes interferes with reusing the
> hand-written QMP code.  Can we make do with -o?

I guess we could by artificially generating to a sub-sub-directory, so
that ../../ relative to the output directory specified with -o is still
a directory specific to the storage daemon. Not very nice, though.

It also results in (hopefully!) unused files with duplicate file names
that are supposed to be mostly the same, modulo bugs.

> By doing that, we sidestep making ../ special in the QAPI schema
> language.  We then have two options for the storage daemon.
> 
> One, use the code generated for the storage daemon schema.  Simple,
> robust, but we compile more.

Not simple at all: What do you mean by "use"? For linking? For compiling
qemu-storage-daemon.c? For compiling blockdev.c?

The latter would mean that we compile everything that uses QAPI types
twice. Considering that we're trying to get rid of per-target
compilation as much as we can, it doesn't appear desirable to start
compiling shared files multiple times for the storage daemon. (I think
you only meant the generated files, not everything that uses QAPI types,
when you said "we compile more"?)

Only compiling a single file with the headers generated from the storage
daemon schema doesn't sound like a great idea either. The storage daemon
headers _should_ be a subset of the main schema ones, but if there is
ever a bug and data structures differ, I don't want to be the poor guy
to debug this. And anyway - you can compile against different versions
of the header, but you can link in only one version of the generated .c
files. I'm not sure what to call this, but "robust" is probably not in
the selection.

> Two, we use the code generated for the main schema (same as now).  We
> compile less, but expose ourselves to the array problem described
> above.

I think we can live with this restriction. If we ever need an array
type only in the storage daemon, we can still define the type that uses
it in the main schema, but leave it unused there.

> > diff --git a/storage-daemon/Makefile.objs b/storage-daemon/Makefile.objs
> > new file mode 100644
> > index 0000000000..cfe6beee52
> > --- /dev/null
> > +++ b/storage-daemon/Makefile.objs
> > @@ -0,0 +1 @@
> > +storage-daemon-obj-y += qapi/
> > diff --git a/storage-daemon/qapi/Makefile.objs b/storage-daemon/qapi/Makefile.objs
> > new file mode 100644
> > index 0000000000..df8946bdae
> > --- /dev/null
> > +++ b/storage-daemon/qapi/Makefile.objs
> > @@ -0,0 +1 @@
> > +storage-daemon-obj-y += qapi-commands.o qapi-introspect.o
> 
> Now let's take a step back and examine the problem this patch solves.
> 
> The storage daemon needs a control interface.  Instead of inventing a
> new one, we want to reuse QMP, both the protocol and the actual
> commands, as far as they make sense in the storage daemon.  Makes sense.
> 
> To get the QMP protocol, the storage daemon needs a QAPI schema.
> 
> To reuse existing QMP commands, the storage daemon's QAPI schema needs
> to include the parts of the existing QAPI schema that define them.
> 
> The stupidest possible way to do that is copying.  Unworkable; too much
> code, too hard to keep it in sync.
> 
> Instead, this patch reuses schema modules.  Summary of issues so far:
> 
> * The include directive lets us avoid duplicating schema code, but to
>   avoid duplicating the generated code, we resort to a gross hack.
> 
>   The hack needs a certain amount of rather awkward documentation (still
>   to be written).

Can possibly be solved by making it explicit.

>   Putting non-shared bits into the storage daemon's schema risks array
>   trouble.
> 
>   Accepting duplicated generated code might be the lesser evil.

I disagree on this one.

> * The discussion of how this module reuse thing works, assumptions,
>   shortcomings need to go on the permanent record, be it code comments,
>   something in docs/, or at least commit messages.
> 
>   Commit messages of PATCH 13 and 16 need to hint at where we're going
>   to make sense.  I figure that's easier to do once we got this patch
>   into shape.
> 
> * We get a bunch of commands that make no sense in the storage daemon,
>   see my review of PATCH 04 for examples.
> 
>   To avoid that, we need to split modules.  May well be a good idea
>   anyway; qapi/block-core.json is by far the fattest module: 850
>   non-blank, non-comment lines, 4.5 times fatter than the runner-up
>   ui.json.
> 
>   Keeping unwanted commands out of the storage daemon will be an ongoing
>   process: we need to keep unwanted commands from creeping into modules
>   shared with the storage daemon.  Hopefully, properly focused modules
>   can make such creep unlikely.

Maybe we can have a bit of a protection against this by moving
qmp_get_blk() and friends into a separate file that isn't linked in the
storage daemon and removing blk_by_qdev_id() from the stubs so that the
build fails rather than just getting an error message back from the
QMP command that makes no sense in the storage daemon.

> A less hacky solution is to extend the QAPI schema language from one set
> of QMP commands and events to many.  Hard to justify when we have just
> two sets.

Right, and when the only actual problem is a hypothetical corner case
that can be worked around relatively easily (array types).

Kevin



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option
  2019-11-13 10:58     ` Kevin Wolf
@ 2019-11-13 13:53       ` Markus Armbruster
  0 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-13 13:53 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Kevin Wolf <kwolf@redhat.com> writes:

> Am 12.11.2019 um 15:25 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>> 
>> > This adds and parses the --monitor option, so that a QMP monitor can be
>> > used in the storage daemon. The monitor offers commands defined in the
>> > QAPI schema at storage-daemon/qapi/qapi-schema.json.
>> 
>> I feel we should explain the module sharing between the two QAPI
>> schemata here.  It's not exactly trivial, as we'll see below.
>> 
>> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>> > ---
>> >  storage-daemon/qapi/qapi-schema.json | 15 ++++++++++++
>> >  qemu-storage-daemon.c                | 34 ++++++++++++++++++++++++++++
>> >  Makefile                             | 30 ++++++++++++++++++++++++
>> >  Makefile.objs                        |  4 ++--
>> >  monitor/Makefile.objs                |  2 ++
>> >  qapi/Makefile.objs                   |  5 ++++
>> >  qom/Makefile.objs                    |  1 +
>> >  scripts/qapi/gen.py                  |  5 ++++
>> >  storage-daemon/Makefile.objs         |  1 +
>> >  storage-daemon/qapi/Makefile.objs    |  1 +
>> >  10 files changed, 96 insertions(+), 2 deletions(-)
>> >  create mode 100644 storage-daemon/qapi/qapi-schema.json
>> >  create mode 100644 storage-daemon/Makefile.objs
>> >  create mode 100644 storage-daemon/qapi/Makefile.objs
>> >
>> > diff --git a/storage-daemon/qapi/qapi-schema.json b/storage-daemon/qapi/qapi-schema.json
>> > new file mode 100644
>> > index 0000000000..58c561ebea
>> > --- /dev/null
>> > +++ b/storage-daemon/qapi/qapi-schema.json
>> > @@ -0,0 +1,15 @@
>> > +# -*- Mode: Python -*-
>> > +
>> > +{ 'include': '../../qapi/pragma.json' }
>> > +
>> > +{ 'include': '../../qapi/block.json' }
>> > +{ 'include': '../../qapi/block-core.json' }
>> > +{ 'include': '../../qapi/char.json' }
>> > +{ 'include': '../../qapi/common.json' }
>> > +{ 'include': '../../qapi/crypto.json' }
>> > +{ 'include': '../../qapi/introspect.json' }
>> > +{ 'include': '../../qapi/job.json' }
>> > +{ 'include': '../../qapi/monitor.json' }
>> > +{ 'include': '../../qapi/qom.json' }
>> > +{ 'include': '../../qapi/sockets.json' }
>> > +{ 'include': '../../qapi/transaction.json' }
>> > diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
>> > index 46e0a6ea56..4939e6b41f 100644
>> > --- a/qemu-storage-daemon.c
>> > +++ b/qemu-storage-daemon.c
>> > @@ -28,12 +28,16 @@
>> >  #include "block/nbd.h"
>> >  #include "chardev/char.h"
>> >  #include "crypto/init.h"
>> > +#include "monitor/monitor.h"
>> > +#include "monitor/monitor-internal.h"
>> >  
>> >  #include "qapi/error.h"
>> >  #include "qapi/qapi-commands-block.h"
>> >  #include "qapi/qapi-commands-block-core.h"
>> > +#include "qapi/qapi-commands-monitor.h"
>> 
>> Aren't these three redundant with ...
>> 
>> >  #include "qapi/qapi-visit-block.h"
>> >  #include "qapi/qapi-visit-block-core.h"
>> > +#include "qapi/qmp/qstring.h"
>> >  #include "qapi/qobject-input-visitor.h"
>> >  
>> >  #include "qemu-common.h"
>> > @@ -46,6 +50,8 @@
>> >  #include "qemu/option.h"
>> >  #include "qom/object_interfaces.h"
>> >  
>> > +#include "storage-daemon/qapi/qapi-commands.h"
>> > +
>> 
>> ... this one now?
>
> Looks like it.
>
>> >  #include "sysemu/runstate.h"
>> >  #include "trace/control.h"
>> >  
>> > @@ -58,6 +64,11 @@ void qemu_system_killed(int signal, pid_t pid)
>> >      exit_requested = true;
>> >  }
>> >  
>> > +void qmp_quit(Error **errp)
>> > +{
>> > +    exit_requested = true;
>> > +}
>> > +
>> >  static void help(void)
>> >  {
>> >      printf(
>> > @@ -101,6 +112,7 @@ enum {
>> >      OPTION_OBJECT = 256,
>> >      OPTION_BLOCKDEV,
>> >      OPTION_CHARDEV,
>> > +    OPTION_MONITOR,
>> >      OPTION_NBD_SERVER,
>> >      OPTION_EXPORT,
>> >  };
>> 
>> Recommend to keep these sorted alphabetically.
>> 
>> > @@ -116,6 +128,17 @@ static QemuOptsList qemu_object_opts = {
>> >      },
>> >  };
>> >  
>> > +static void init_qmp_commands(void)
>> > +{
>> > +    qmp_init_marshal(&qmp_commands);
>> > +    qmp_register_command(&qmp_commands, "query-qmp-schema",
>> > +                         qmp_query_qmp_schema, QCO_ALLOW_PRECONFIG);
>> > +
>> > +    QTAILQ_INIT(&qmp_cap_negotiation_commands);
>> > +    qmp_register_command(&qmp_cap_negotiation_commands, "qmp_capabilities",
>> > +                         qmp_marshal_qmp_capabilities, QCO_ALLOW_PRECONFIG);
>> > +}
>> 
>> Duplicates monitor_init_qmp_commands() less two 'gen': false commands
>> that don't exist in the storage daemon.
>> 
>> Hmm, could we let commands.py generate command registration even for
>> 'gen': false commands?
>
> Possibly. Are you telling me to do it or is it just an idea for a later
> cleanup?

Exploring the idea would be nice, but it's definitely not required.

>> > +
>> >  static void init_export(BlockExport *export, Error **errp)
>> >  {
>> >      switch (export->type) {
>> > @@ -138,6 +161,7 @@ static int process_options(int argc, char *argv[], Error **errp)
>> >          {"object", required_argument, 0, OPTION_OBJECT},
>> >          {"blockdev", required_argument, 0, OPTION_BLOCKDEV},
>> >          {"chardev", required_argument, 0, OPTION_CHARDEV},
>> > +        {"monitor", required_argument, 0, OPTION_MONITOR},
>> >          {"nbd-server", required_argument, 0, OPTION_NBD_SERVER},
>> >          {"export", required_argument, 0, OPTION_EXPORT},
>> >          {"version", no_argument, 0, 'V'},
>> > @@ -208,6 +232,14 @@ static int process_options(int argc, char *argv[], Error **errp)
>> >                  qemu_opts_del(opts);
>> >                  break;
>> >              }
>> > +        case OPTION_MONITOR:
>> > +            {
>> > +                QemuOpts *opts = qemu_opts_parse(&qemu_mon_opts,
>> > +                                                 optarg, true, &error_fatal);
>> > +                monitor_init_opts(opts, false, &error_fatal);
>> > +                qemu_opts_del(opts);
>> > +                break;
>> > +            }
>> >          case OPTION_NBD_SERVER:
>> >              {
>> >                  Visitor *v;
>> 
>> Recommend to keep the switch cases sorted.  Perhaps put help first.
>> Order the short options string and the long_options[] array to match.
>> 
>> > @@ -272,6 +304,8 @@ int main(int argc, char *argv[])
>> >      qemu_add_opts(&qemu_trace_opts);
>> >      qcrypto_init(&error_fatal);
>> >      bdrv_init();
>> > +    monitor_init_globals_core();
>> > +    init_qmp_commands();
>> >  
>> >      if (qemu_init_main_loop(&local_err)) {
>> >          error_report_err(local_err);
>> > diff --git a/Makefile b/Makefile
>> > index 0e3e98582d..e367d2b28a 100644
>> > --- a/Makefile
>> > +++ b/Makefile
>> > @@ -121,7 +121,26 @@ GENERATED_QAPI_FILES += $(QAPI_MODULES:%=qapi/qapi-events-%.c)
>> >  GENERATED_QAPI_FILES += qapi/qapi-introspect.c qapi/qapi-introspect.h
>> >  GENERATED_QAPI_FILES += qapi/qapi-doc.texi
>> >  
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES = storage-daemon/qapi/qapi-builtin-types.h
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-types.c
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-visit.h
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-builtin-visit.c
>> 
>> Any particular reason for not reusing qapi/qapi-builtin-*?  See also the
>> recipe for storage-daemon/qapi/qapi-gen-timestamp below.
>
> Hm, I guess not. I'm just tweaking stuff until it seems to work, and
> this seemed to work, so...
>
> But as far as I can see, the generated file are never actually used, so
> I'll get rid of them.
>
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-commands.h
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-commands.c
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-emit-events.h
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-emit-events.c
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-events.h
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-events.c
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-introspect.h
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-introspect.c
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-types.h
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-types.c
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-visit.h
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-visit.c
>> > +GENERATED_STORAGE_DAEMON_QAPI_FILES += storage-daemon/qapi/qapi-doc.texi
>> 
>> These are the files generated for the storage daemon's main module.  The
>> files generated for its sub-modules (all shared right now) are absent.
>> I think this could use a comment.
>
> Ok.
>
>> See also scripts/qapi/gen.py below.
>> 
>> > +
>> >  generated-files-y += $(GENERATED_QAPI_FILES)
>> > +generated-files-y += $(GENERATED_STORAGE_DAEMON_QAPI_FILES)
>> >  
>> >  generated-files-y += trace/generated-tcg-tracers.h
>> >  
>> > @@ -616,6 +635,17 @@ qapi-gen-timestamp: $(qapi-modules) $(qapi-py)
>> >  		"GEN","$(@:%-timestamp=%)")
>> >  	@>$@
>> >  
>> > +qapi-modules-storage-daemon = \
>> > +	$(SRC_PATH)/storage-daemon/qapi/qapi-schema.json \
>> > +    $(QAPI_MODULES_STORAGE_DAEMON:%=$(SRC_PATH)/qapi/%.json)
>> > +
>> > +$(GENERATED_STORAGE_DAEMON_QAPI_FILES): storage-daemon/qapi/qapi-gen-timestamp ;
>> > +storage-daemon/qapi/qapi-gen-timestamp: $(qapi-modules-storage-daemon) $(qapi-py)
>> > +	$(call quiet-command,$(PYTHON) $(SRC_PATH)/scripts/qapi-gen.py \
>> > +		-o "storage-daemon/qapi" -b $<, \
>> 
>> If we can reuse qapi/qapi-builtin-*, then omit -b to suppress generating
>> redundant copies.
>> 
>> > +		"GEN","$(@:%-timestamp=%)")
>> > +	@>$@
>> > +
>> >  QGALIB_GEN=$(addprefix qga/qapi-generated/, qga-qapi-types.h qga-qapi-visit.h qga-qapi-commands.h)
>> >  $(qga-obj-y): $(QGALIB_GEN)
>> >  
>> > diff --git a/Makefile.objs b/Makefile.objs
>> > index b667d3f07b..d4e0daddee 100644
>> > --- a/Makefile.objs
>> > +++ b/Makefile.objs
>> > @@ -41,8 +41,8 @@ io-obj-y = io/
>> >  # storage-daemon-obj-y is code used by qemu-storage-daemon (these objects are
>> >  # used for system emulation, too, but specified separately there)
>> >  
>> > -storage-daemon-obj-y = block/
>> > -storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o
>> > +storage-daemon-obj-y = block/ monitor/ qapi/ qom/ storage-daemon/
>> > +storage-daemon-obj-y += blockdev.o blockdev-nbd.o iothread.o job-qmp.o
>> >  storage-daemon-obj-$(CONFIG_WIN32) += os-win32.o
>> >  storage-daemon-obj-$(CONFIG_POSIX) += os-posix.o
>> >  
>> > diff --git a/monitor/Makefile.objs b/monitor/Makefile.objs
>> > index 15eb6380c5..6e4ef60601 100644
>> > --- a/monitor/Makefile.objs
>> > +++ b/monitor/Makefile.objs
>> > @@ -2,3 +2,5 @@ obj-y += misc.o
>> >  common-obj-y += monitor.o qmp.o hmp.o
>> >  common-obj-y += qmp-cmds.o qmp-cmds-monitor.o
>> >  common-obj-y += hmp-cmds.o
>> > +
>> > +storage-daemon-obj-y += monitor.o qmp.o qmp-cmds-monitor.o
>> > diff --git a/qapi/Makefile.objs b/qapi/Makefile.objs
>> > index 3e04e299ed..03d256f0a4 100644
>> > --- a/qapi/Makefile.objs
>> > +++ b/qapi/Makefile.objs
>> > @@ -30,3 +30,8 @@ obj-y += $(QAPI_TARGET_MODULES:%=qapi-events-%.o)
>> >  obj-y += qapi-events.o
>> >  obj-y += $(QAPI_TARGET_MODULES:%=qapi-commands-%.o)
>> >  obj-y += qapi-commands.o
>> > +
>> > +QAPI_MODULES_STORAGE_DAEMON = block block-core char common crypto introspect
>> > +QAPI_MODULES_STORAGE_DAEMON += job monitor qom sockets pragma transaction
>> > +
>> > +storage-daemon-obj-y += $(QAPI_MODULES_STORAGE_DAEMON:%=qapi-commands-%.o)
>> > diff --git a/qom/Makefile.objs b/qom/Makefile.objs
>> > index f9d77350ac..1b45d104ba 100644
>> > --- a/qom/Makefile.objs
>> > +++ b/qom/Makefile.objs
>> > @@ -2,3 +2,4 @@ qom-obj-y = object.o container.o qom-qobject.o
>> >  qom-obj-y += object_interfaces.o
>> >  
>> >  common-obj-$(CONFIG_SOFTMMU) += qom-hmp-cmds.o qom-qmp-cmds.o
>> > +storage-daemon-obj-y += qom-qmp-cmds.o
>> > diff --git a/scripts/qapi/gen.py b/scripts/qapi/gen.py
>> > index 796c17c38a..c25634f673 100644
>> > --- a/scripts/qapi/gen.py
>> > +++ b/scripts/qapi/gen.py
>> > @@ -44,6 +44,11 @@ class QAPIGen(object):
>> >          return ''
>> >  
>> >      def write(self, output_dir):
>> > +        # Include paths starting with ../ are used to reuse modules of the main
>> > +        # schema in specialised schemas. Don't overwrite the files that are
>> > +        # already generated for the main schema.
>> > +        if self.fname.startswith('../'):
>> > +            return
>> >          pathname = os.path.join(output_dir, self.fname)
>> >          dir = os.path.dirname(pathname)
>> >          if dir:
>> 
>> Ugh.
>
> That's why I asked you before writing it like this. :-)
>
>> tests/qapi-schema/include/sub-module.json violates the stated assumption
>> "Include paths starting with ../ are used to reuse modules of the main
>> schema":
>> 
>>     { 'include': '../sub-sub-module.json' }
>> 
>> Works anyway because it's only used via
>> 
>>     { 'include': 'include/sub-module.json' }
>> 
>> where the ../ is folded with the include/ and vanishes.  self.fname is
>> relative to the main schema's directory.
>> 
>> Still, baking the assumption into gen.py makes it a part of the QAPI
>> schema language.  It needs to be documented as such.
>
> Maybe you'd like it better if we made it explicit?
>
>     { 'include': '../../qapi/block.json', 'gen': false }

Maybe.

Another possibility is tightening the stated assumption: "sub-modules
from outside the main module's directory are assumed to be shared, and
no code will be generated for them".  Needs to be documented in
docs/devel/qapi-code-gen.txt.  Which so far neglects to explain modules
properly.  Perhaps I can fix that in parallel, and then we figure out
how to proceed.

> But how do I actually get this information down to QAPIGen? This brings
> us back to the problem that the QAPI generator doesn't really have a
> notion of modules, but only entities have a module name and the first
> entity to use a module name causes it to be generated. We don't have
> access to the QAPISchemaInclude that created the module.
>
> Even the QAPISchema._modules that I introduced only contains filenames.
> Should it contain QAPISchemaInclude objects instead?

The old Python story: it's just a string; oh, it's two, so make it
tuple; oh, this is confusing, so make it a dict; oh, want methods, so
make it an object.

Time for a QAPISchemaModule object, I guess.  I'll have a try and get
back to you.

>> Complication: because we generate array types and supporting code code
>> only when they're actually used, the code generated for a shared module
>> can depend on how it is used outside the shared module.
>> 
>> Right now, storage-daemon/qapi/qapi-schema.json contains nothing but
>> modules shared with qapi/qapi-schema.json.  Works.
>> 
>> If storage-daemon/qapi/qapi-schema.json ever acquires an array reference
>> that is not also in qapi/qapi-schema.json, the generated C won't
>> compile.
>> 
>> This also needs to be documented because it's anything but obvious.
>
> A comment in the storage daemon main module?

Suffices as long as module sharing is a hack we use in the storage
daemon's QAPI schema.  If it spreads elsewhere, we better cover it in
docs/devel/qapi-code-gen.txt.

However, it looks like module sharing needs a bit of help from the QAPI
generator, which amounts to a schema language tweak, which means we need
to update qapi-code-gen.txt for it.  Depending on the nature of the
tweak, we may want to cover the array restriction in its doc update.

No need to worry about the doc details until we've figured out what to
do in the QAPI generator.

A comment in the storage daemon's main module won't hurt regardless.

>> Instead of suppressing code generated for reused modules outright, we
>> could somehow avoid clobbering the main schema's output.  I guess we
>> can't use -p for that, because it makes interferes with reusing the
>> hand-written QMP code.  Can we make do with -o?
>
> I guess we could by artificially generating to a sub-sub-directory, so
> that ../../ relative to the output directory specified with -o is still
> a directory specific to the storage daemon. Not very nice, though.
>
> It also results in (hopefully!) unused files with duplicate file names
> that are supposed to be mostly the same, modulo bugs.
>
>> By doing that, we sidestep making ../ special in the QAPI schema
>> language.  We then have two options for the storage daemon.
>> 
>> One, use the code generated for the storage daemon schema.  Simple,
>> robust, but we compile more.
>
> Not simple at all: What do you mean by "use"? For linking? For compiling
> qemu-storage-daemon.c? For compiling blockdev.c?
>
> The latter would mean that we compile everything that uses QAPI types
> twice. Considering that we're trying to get rid of per-target
> compilation as much as we can, it doesn't appear desirable to start
> compiling shared files multiple times for the storage daemon. (I think
> you only meant the generated files, not everything that uses QAPI types,
> when you said "we compile more"?)
>
> Only compiling a single file with the headers generated from the storage
> daemon schema doesn't sound like a great idea either. The storage daemon
> headers _should_ be a subset of the main schema ones, but if there is
> ever a bug and data structures differ, I don't want to be the poor guy
> to debug this. And anyway - you can compile against different versions
> of the header, but you can link in only one version of the generated .c
> files. I'm not sure what to call this, but "robust" is probably not in
> the selection.

You're right.

>> Two, we use the code generated for the main schema (same as now).  We
>> compile less, but expose ourselves to the array problem described
>> above.
>
> I think we can live with this restriction. If we ever need an array
> type only in the storage daemon,

The guy staring at the gcc error won't be nearly as poor as the one
debugging diverged generated QAPI code, but I'd prefer not to be him
all the same.  Anyway:

>                                  we can still define the type that uses
> it in the main schema, but leave it unused there.

Yes, that's workable.

>> > diff --git a/storage-daemon/Makefile.objs b/storage-daemon/Makefile.objs
>> > new file mode 100644
>> > index 0000000000..cfe6beee52
>> > --- /dev/null
>> > +++ b/storage-daemon/Makefile.objs
>> > @@ -0,0 +1 @@
>> > +storage-daemon-obj-y += qapi/
>> > diff --git a/storage-daemon/qapi/Makefile.objs b/storage-daemon/qapi/Makefile.objs
>> > new file mode 100644
>> > index 0000000000..df8946bdae
>> > --- /dev/null
>> > +++ b/storage-daemon/qapi/Makefile.objs
>> > @@ -0,0 +1 @@
>> > +storage-daemon-obj-y += qapi-commands.o qapi-introspect.o
>> 
>> Now let's take a step back and examine the problem this patch solves.
>> 
>> The storage daemon needs a control interface.  Instead of inventing a
>> new one, we want to reuse QMP, both the protocol and the actual
>> commands, as far as they make sense in the storage daemon.  Makes sense.
>> 
>> To get the QMP protocol, the storage daemon needs a QAPI schema.
>> 
>> To reuse existing QMP commands, the storage daemon's QAPI schema needs
>> to include the parts of the existing QAPI schema that define them.
>> 
>> The stupidest possible way to do that is copying.  Unworkable; too much
>> code, too hard to keep it in sync.
>> 
>> Instead, this patch reuses schema modules.  Summary of issues so far:
>> 
>> * The include directive lets us avoid duplicating schema code, but to
>>   avoid duplicating the generated code, we resort to a gross hack.
>> 
>>   The hack needs a certain amount of rather awkward documentation (still
>>   to be written).
>
> Can possibly be solved by making it explicit.

Needs a hopefully smaller amount of less awkward documentation (still to
be written).

>>   Putting non-shared bits into the storage daemon's schema risks array
>>   trouble.
>> 
>>   Accepting duplicated generated code might be the lesser evil.
>
> I disagree on this one.

I wasn't sure when I wrote it (thus "might"), and you've now convinced
me it's not workable.

>> * The discussion of how this module reuse thing works, assumptions,
>>   shortcomings need to go on the permanent record, be it code comments,
>>   something in docs/, or at least commit messages.
>> 
>>   Commit messages of PATCH 13 and 16 need to hint at where we're going
>>   to make sense.  I figure that's easier to do once we got this patch
>>   into shape.
>> 
>> * We get a bunch of commands that make no sense in the storage daemon,
>>   see my review of PATCH 04 for examples.
>> 
>>   To avoid that, we need to split modules.  May well be a good idea
>>   anyway; qapi/block-core.json is by far the fattest module: 850
>>   non-blank, non-comment lines, 4.5 times fatter than the runner-up
>>   ui.json.
>> 
>>   Keeping unwanted commands out of the storage daemon will be an ongoing
>>   process: we need to keep unwanted commands from creeping into modules
>>   shared with the storage daemon.  Hopefully, properly focused modules
>>   can make such creep unlikely.
>
> Maybe we can have a bit of a protection against this by moving
> qmp_get_blk() and friends into a separate file that isn't linked in the
> storage daemon and removing blk_by_qdev_id() from the stubs so that the
> build fails rather than just getting an error message back from the
> QMP command that makes no sense in the storage daemon.

Yes, please.

>> A less hacky solution is to extend the QAPI schema language from one set
>> of QMP commands and events to many.  Hard to justify when we have just
>> two sets.

I figure the code change wouldn't be so bad, but the documentation
update to go with it puts me off.

> Right, and when the only actual problem is a hypothetical corner case
> that can be worked around relatively easily (array types).

I consider documenting the hack also an actual problem, albeit a
solvable one.



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool
  2019-10-24 13:50   ` Eric Blake
@ 2019-11-13 14:12     ` Kevin Wolf
  0 siblings, 0 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-11-13 14:12 UTC (permalink / raw)
  To: Eric Blake; +Cc: qemu-devel, pkrempa, armbru, qemu-block, mreitz

Am 24.10.2019 um 15:50 hat Eric Blake geschrieben:
> On 10/17/19 8:01 AM, Kevin Wolf wrote:
> > This adds a new binary qemu-storage-daemon that doesn't yet do more than
> > some typical initialisation for tools and parsing the basic command
> > options --version, --help and --trace.
> > 
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> > ---
> >   configure             |   2 +-
> >   qemu-storage-daemon.c | 141 ++++++++++++++++++++++++++++++++++++++++++
> >   Makefile              |   1 +
> >   3 files changed, 143 insertions(+), 1 deletion(-)
> >   create mode 100644 qemu-storage-daemon.c
> > 
> > diff --git a/configure b/configure
> 
> > +++ b/qemu-storage-daemon.c
> > @@ -0,0 +1,141 @@
> > +/*
> > + * QEMU storage daemon
> > + *
> > + * Copyright (c) 2019 Kevin Wolf <kwolf@redhat.com>
> > + *
> > + * Permission is hereby granted, free of charge, to any person obtaining a copy
> > + * of this software and associated documentation files (the "Software"), to deal
> > + * in the Software without restriction, including without limitation the rights
> > + * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
> > + * copies of the Software, and to permit persons to whom the Software is
> > + * furnished to do so, subject to the following conditions:
> > + *
> > + * The above copyright notice and this permission notice shall be included in
> > + * all copies or substantial portions of the Software.
> > + *
> > + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
> > + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
> > + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
> > + * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
> > + * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
> > + * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
> > + * THE SOFTWARE.
> 
> Is there an intent to license this binary as BSD (by restricting sources
> that can be linked in), or is it going to end up as GPLv2+ for other
> reasons? If the latter, would it be better to license this file GPLv2+?

The binary will be GPLv2 either way. Maybe even GPLv2+, not sure about
that.

This file copies quite a bit from qemu-img.c and vl.c, so I'm using the
same license as there. Of course, the "bug" in my patch is that I also
need to keep not only the license text, but also the actual copyright
line from there. Will fix.

> 
> > + */
> > +
> > +#include "qemu/osdep.h"
> > +
> > +#include "block/block.h"
> > +#include "crypto/init.h"
> > +
> > +#include "qapi/error.h"
> > +#include "qemu-common.h"
> > +#include "qemu-version.h"
> > +#include "qemu/config-file.h"
> > +#include "qemu/error-report.h"
> > +#include "qemu/log.h"
> > +#include "qemu/main-loop.h"
> > +#include "qemu/module.h"
> > +
> > +#include "trace/control.h"
> > +
> > +#include <getopt.h>
> 
> Shouldn't system headers appear right after qemu/osdep.h?

I wasn't aware of this, but CODING_STYLE.rst agrees with you.

> > +
> > +static void help(void)
> > +{
> > +    printf(
> > +"Usage: %s [options]\n"
> > +"QEMU storage daemon\n"
> > +"\n"
> > +"  -h, --help             display this help and exit\n"
> > +"  -T, --trace [[enable=]<pattern>][,events=<file>][,file=<file>]\n"
> > +"                         specify tracing options\n"
> > +"  -V, --version          output version information and exit\n"
> > +"\n"
> > +QEMU_HELP_BOTTOM "\n",
> > +    error_get_progname());
> > +}
> > +
> > +static int process_options(int argc, char *argv[], Error **errp)
> > +{
> > +    int c;
> > +    char *trace_file = NULL;
> > +    int ret = -EINVAL;
> > +
> > +    static const struct option long_options[] = {
> > +        {"help", no_argument, 0, 'h'},
> > +        {"version", no_argument, 0, 'V'},
> > +        {"trace", required_argument, NULL, 'T'},
> 
> I find it harder to maintain lists of options (which will get longer over
> time) when the order of the struct...
> 
> > +        {0, 0, 0, 0}
> > +    };
> > +
> > +    while ((c = getopt_long(argc, argv, ":hT:V", long_options, NULL)) != -1) {
> 
> ...the order of the short options...
> 
> > +        switch (c) {
> > +        case '?':
> > +            error_setg(errp, "Unknown option '%s'", argv[optind - 1]);
> > +            goto out;
> > +        case ':':
> > +            error_setg(errp, "Missing option argument for '%s'",
> > +                       argv[optind - 1]);
> > +            goto out;
> > +        case 'h':
> > +            help();
> > +            exit(EXIT_SUCCESS);
> > +        case 'V':
> > +            printf("qemu-storage-daemon version "
> > +                   QEMU_FULL_VERSION "\n" QEMU_COPYRIGHT "\n");
> > +            exit(EXIT_SUCCESS);
> > +        case 'T':
> > +            g_free(trace_file);
> > +            trace_file = trace_opt_parse(optarg);
> > +            break;
> 
> ...and the order of the case statements are not identical. Case-insensitive
> alphabetical may be easiest (matching your shortopt ordering of ":hT:V").

Makes sense. (I think I tried to remember to keep things in alphabetical
order at least sometimes, but apparently I didn't try hard enough.)

> > +        }
> > +    }
> > +    if (optind != argc) {
> > +        error_setg(errp, "Unexpected argument: %s", argv[optind]);
> > +        goto out;
> > +    }
> > +
> > +    if (!trace_init_backends()) {
> > +        error_setg(errp, "Could not initialize trace backends");
> > +        goto out;
> > +    }
> > +    trace_init_file(trace_file);
> > +    qemu_set_log(LOG_TRACE);
> > +
> > +    ret = 0;
> > +out:
> > +    g_free(trace_file);
> 
> Mark trace_file as g_auto, and you can avoid the out: label altogether.

Oooh, magic! :-)

Kevin



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-10-24 13:55 ` Vladimir Sementsov-Ogievskiy
@ 2019-11-14 10:44   ` Kevin Wolf
  0 siblings, 0 replies; 65+ messages in thread
From: Kevin Wolf @ 2019-11-14 10:44 UTC (permalink / raw)
  To: Vladimir Sementsov-Ogievskiy
  Cc: mreitz, pkrempa, qemu-devel, qemu-block, armbru

Am 24.10.2019 um 15:55 hat Vladimir Sementsov-Ogievskiy geschrieben:
> This reflects our idea of using qemu binary instead of qemu-img for doing
> block-layer operations offline.
> 
> What is the practical difference between qemu-storage-daemon and starting
> qemu binary in stopped state?

If I'm doing things right, QEMU should be able to do the exact same
things as the storage daemon (and more). So functionality isn't what
makes the daemon desirable.

One point to consider is that the QEMU binary with a full system
emulator is (and has to be) relatively heavyweight compared to the
daemon.

I think libvirt once said they didn't want to use the qemu binary on the
grounds that it takes too long to start, though I'm not sure if that's
really an argument. I just did a quick test (qemu with -M none
-nodefaults -display none) and it's 30 vs. 60 ms on my laptop. I get the
same factor 2 for RSS.

More interesting is maybe the overhead in terms of binary size and
dependencies, especially if you need only either the storage daemon _or_
the QEMU binary (e.g. consider a case where the storage daemon runs in
one container and the VM in another).

Having two binaries allows to cut down the dependencies, and to some
extent also the binary size, for both binaries: The storage daemon
doesn't need anything related to system emulation, UI, network etc., and
in QEMU we can probably compile out most of the block layer then, which
gets rid of dependencies like libiscsi, librbd, libgfapi, etc.

As a bonus, this might even reduce the attack surface a little.

So yes, I agree that the storage daemon doesn't offer any functionality
that QEMU can't offer and we don't have a clear-cut requirement that
unambiguously calls for a separate storage daemon, but I still see some
advantage in having two separate binaries.

Another thing to mention is that on IRC, Stefan suggested the other day
to export block devices from the storage daemon not as vhost-user, but
using muser instead (i.e. providing a whole PCI device) and exporting
the existing virtio-blk-pci implementation. This would pull qdev and
device emulations into the storage daemon. I think that would be the
point where using the QEMU binary instead might make more sense (and
maybe compile it twice with different options if need be).

Kevin



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 02/18] qemu-storage-daemon: Add --object option
  2019-11-07 20:36   ` Markus Armbruster
@ 2019-11-14 12:05     ` Kevin Wolf
  2019-11-18  9:10       ` Markus Armbruster
  0 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-11-14 12:05 UTC (permalink / raw)
  To: Markus Armbruster; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Am 07.11.2019 um 21:36 hat Markus Armbruster geschrieben:
> Kevin Wolf <kwolf@redhat.com> writes:
> 
> > Add a command line option to create user-creatable QOM objects.
> >
> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
> > ---
> >  qemu-storage-daemon.c | 35 +++++++++++++++++++++++++++++++++++
> >  1 file changed, 35 insertions(+)
> >
> > diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
> > index a251dc255c..48d6af43a6 100644
> > --- a/qemu-storage-daemon.c
> > +++ b/qemu-storage-daemon.c
> > @@ -35,6 +35,8 @@
> >  #include "qemu/log.h"
> >  #include "qemu/main-loop.h"
> >  #include "qemu/module.h"
> > +#include "qemu/option.h"
> > +#include "qom/object_interfaces.h"
> >  
> >  #include "trace/control.h"
> >  
> > @@ -51,10 +53,26 @@ static void help(void)
> >  "                         specify tracing options\n"
> >  "  -V, --version          output version information and exit\n"
> >  "\n"
> > +"  --object <properties>  define a QOM object such as 'secret' for\n"
> > +"                         passwords and/or encryption keys\n"
> 
> This is less helpful than qemu-system-FOO's help:
> 
> -object TYPENAME[,PROP1=VALUE1,...]
>                 create a new object of type TYPENAME setting properties
>                 in the order they are specified.  Note that the 'id'
>                 property must be set.  These objects are placed in the
>                 '/objects' path.

Hm, yes. I took the description from the tools. I can switch to the vl.c
one (should the tools, too?).

But honestly, neither of the two is enough to tell anyone how to
actually use this... Considering how many different objects there are,
maybe the best we can do is referring to the man page for details.

> > +"\n"
> >  QEMU_HELP_BOTTOM "\n",
> >      error_get_progname());
> >  }
> >  
> > +enum {
> > +    OPTION_OBJECT = 256,
> > +};
> > +
> > +static QemuOptsList qemu_object_opts = {
> > +    .name = "object",
> > +    .implied_opt_name = "qom-type",
> > +    .head = QTAILQ_HEAD_INITIALIZER(qemu_object_opts.head),
> > +    .desc = {
> > +        { }
> > +    },
> > +};
> > +
> 
> Note for later: copied from vl.c.
> 
> >  static int process_options(int argc, char *argv[], Error **errp)
> >  {
> >      int c;
> > @@ -63,6 +81,7 @@ static int process_options(int argc, char *argv[], Error **errp)
> >  
> >      static const struct option long_options[] = {
> >          {"help", no_argument, 0, 'h'},
> > +        {"object", required_argument, 0, OPTION_OBJECT},
> >          {"version", no_argument, 0, 'V'},
> >          {"trace", required_argument, NULL, 'T'},
> >          {0, 0, 0, 0}
> > @@ -88,6 +107,22 @@ static int process_options(int argc, char *argv[], Error **errp)
> >              g_free(trace_file);
> >              trace_file = trace_opt_parse(optarg);
> >              break;
> > +        case OPTION_OBJECT:
> > +            {
> > +                QemuOpts *opts;
> > +                const char *type;
> > +
> > +                opts = qemu_opts_parse(&qemu_object_opts,
> > +                                       optarg, true, &error_fatal);
> > +                type = qemu_opt_get(opts, "qom-type");
> > +
> > +                if (user_creatable_print_help(type, opts)) {
> > +                    exit(EXIT_SUCCESS);
> > +                }
> > +                user_creatable_add_opts(opts, &error_fatal);
> > +                qemu_opts_del(opts);
> > +                break;
> > +            }
> >          }
> >      }
> >      if (optind != argc) {
> 
> PATCH 01 duplicates case QEMU_OPTION_trace pretty much verbatim.  Makes
> sense, as qemu-storage-daemon is basically qemu-system-FOO with "FOO"
> and most "system" cut away.
> 
> This patch adds vl.c's case QEMU_OPTION_object in a much simpler form.
> This is one of my least favourite options, and I'll tell you why below.
> Let's compare the two versions.
> 
> vl.c:
> 
>             case QEMU_OPTION_object:
>                 opts = qemu_opts_parse_noisily(qemu_find_opts("object"),
>                                                optarg, true);
>                 if (!opts) {
>                     exit(1);
>                 }
>                 break;
> 
> Further down:
> 
>     qemu_opts_foreach(qemu_find_opts("object"),
>                       user_creatable_add_opts_foreach,
>                       object_create_initial, &error_fatal);
> 
> Still further down:
> 
>     qemu_opts_foreach(qemu_find_opts("object"),
>                       user_creatable_add_opts_foreach,
>                       object_create_delayed, &error_fatal);
> 
> These are basically
> 
>     for opts in qemu_object_opts {
>         type = qemu_opt_get(opts, "qom-type");
>         if (type) {
>             if (user_creatable_print_help(type, opts)) {
>                 exit(0);
>             }
>             if (!predicate(type)) {
>                 continue;
>             }
>         }
>         obj = user_creatable_add_opts(opts, &error_fatal);
>         object_unref(obj);
>     }
> 
> where predicate(type) is true in exactly one of the two places for each
> QOM type.
> 
> The reason for these gymnastics is to create objects at the right time
> during startup, except there is no right time, but two.
> 
> Differences:
> 
> * Options are processed left to right without gymnastics.  Getting their
>   order right is the user's problem.  I consider this an improvement.
> 
> * You use &qemu_object_opts instead of qemu_find_opts("object").  Also
>   an improvement.
> 
> * You use qemu_opts_parse() instead of qemu_opts_parse_noisily().
>   The latter can print help.  I failed to find a case where we lose help
>   compared to qemu-system-FOO.  I didn't try very hard.

I tried to reuse that code from qemu_opts_parse_noisily(), until I
realised that it wasn't even used for -object.

I don't remember the details why qemu_opts_print_help() wasn't even
called, but it's obvious that we can't lose anything from it:
qemu_object_opts has an empty list of properties, it accepts anything.
QemuOpts can't print any useful help when this is all the information it
has.

> * You neglect to guard user_creatable_print_help():
> 
>     $ qemu-storage-daemon --object wrong=1,help
>     Segmentation fault (core dumped)

Thanks for catching this. (You don't even need the ",help" part, just
--object wrong=1 is enough.)

> * You neglect to object_unref().  I just double-checked the final
>   reference count: it's 2.

Hm, yes. Weird interface, no caller actually needs this reference.

> These bugs shouldn't be hard to fix.
> 
> 
> At this point you might wonder why I dislike this option so much.
> vl.c's gymnastics are ugly, but not unusually ugly, and they're gone
> here.  To explain my distaste, I have to go back a little bit.
> 
> Like quite a few options, --object is paired with QMP command, namely
> object-add.  Both have the same parameters: QOM type, object ID, and
> additional type-specific object properties.  There's a difference,
> though: object-add wraps the latter in a 'props' object, while --object
> does not.
> 
> QAPI schema:
> 
>     { 'command': 'object-add',
>       'data': {'qom-type': 'str', 'id': 'str', '*props': 'any'} }
> 
> QAPIfying this part of the CLI isn't easy.
> 
> The obvious QAPIfied CLI buddy of object-add is incompatible to current
> --object.  That's not a concern for the storage daemon.  But it's also
> ugly, because object-add's nesting of the type-specific properties
> within @props is.  In QMP, it's merely yet another pair of curlies.  In
> the CLI, we get to prefix props. to each type-specific property.
> 
> If we want to give the storage daemon a QAPIfied command line from the
> start (and I think we do), we'll have to decide how to address this
> issue, and possibly more (I'm only at PATCH 02/18).
> 
> We have a long history of rather careless interface design, and now some
> of these chickens come home to roost.

On IRC, we discussed last week that we could turn object-add into a
'gen': false command and accept things both as options in props and as
flat options on the top level.

However, looking at this again, I'm afraid we forgot the context while
discussing specifics: How would this be used in a command line parser?

We don't start with a QDict here, but with a string. Getting a QDict
that could serve as an input to a modified qmp_object_add() would still
involve going through QemuOpts for parsing the option string, and then
converting it to a QDict. Using a visitor isn't possible with '*props':
'any' and even less so with 'gen': false.

So would this really improve things? Or do we have to wait until we have
an actual schema for object-add before calling qmp_object_add() actually
makes sense?

Kevin



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 02/18] qemu-storage-daemon: Add --object option
  2019-11-14 12:05     ` Kevin Wolf
@ 2019-11-18  9:10       ` Markus Armbruster
  0 siblings, 0 replies; 65+ messages in thread
From: Markus Armbruster @ 2019-11-18  9:10 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: pkrempa, qemu-devel, qemu-block, mreitz

Kevin Wolf <kwolf@redhat.com> writes:

> Am 07.11.2019 um 21:36 hat Markus Armbruster geschrieben:
>> Kevin Wolf <kwolf@redhat.com> writes:
>> 
>> > Add a command line option to create user-creatable QOM objects.
>> >
>> > Signed-off-by: Kevin Wolf <kwolf@redhat.com>
>> > ---
>> >  qemu-storage-daemon.c | 35 +++++++++++++++++++++++++++++++++++
>> >  1 file changed, 35 insertions(+)
>> >
>> > diff --git a/qemu-storage-daemon.c b/qemu-storage-daemon.c
>> > index a251dc255c..48d6af43a6 100644
>> > --- a/qemu-storage-daemon.c
>> > +++ b/qemu-storage-daemon.c
>> > @@ -35,6 +35,8 @@
>> >  #include "qemu/log.h"
>> >  #include "qemu/main-loop.h"
>> >  #include "qemu/module.h"
>> > +#include "qemu/option.h"
>> > +#include "qom/object_interfaces.h"
>> >  
>> >  #include "trace/control.h"
>> >  
>> > @@ -51,10 +53,26 @@ static void help(void)
>> >  "                         specify tracing options\n"
>> >  "  -V, --version          output version information and exit\n"
>> >  "\n"
>> > +"  --object <properties>  define a QOM object such as 'secret' for\n"
>> > +"                         passwords and/or encryption keys\n"
>> 
>> This is less helpful than qemu-system-FOO's help:
>> 
>> -object TYPENAME[,PROP1=VALUE1,...]
>>                 create a new object of type TYPENAME setting properties
>>                 in the order they are specified.  Note that the 'id'
>>                 property must be set.  These objects are placed in the
>>                 '/objects' path.
>
> Hm, yes. I took the description from the tools. I can switch to the vl.c
> one (should the tools, too?).
>
> But honestly, neither of the two is enough to tell anyone how to
> actually use this... Considering how many different objects there are,
> maybe the best we can do is referring to the man page for details.

For a simple program, --help can provide pretty much the same usage
information as the manual page.

For a beast like QEMU, that's hopeless.  But --help can still serve as a
quick reminder of syntax and such.

Another argument is consistency: as long as --help shows syntax for the
other options, it should probably show syntax for --object, too.

We can certainly discuss level of detail.  For instance,

    --blockdev [driver=]<driver>[,node-name=<N>][,discard=ignore|unmap]
               [,cache.direct=on|off][,cache.no-flush=on|off]
               [,read-only=on|off][,auto-read-only=on|off]
               [,force-share=on|off][,detect-zeroes=on|off|unmap]
               [,driver specific parameters...]
                           configure a block backend

is arguably hardly more useful than

    --blockdev [driver=]<driver>[,node-name=<node-name>][,<prop>=<value>,...]
                           configure a block backend

The screen space would arguably be better spend on explaining <driver>
and <node-name>.

By the way, we should pick *one* way to mark up variable parts, and
stick to it.  As it is, we have

    -machine [type=]name[,prop[=value][,...]]
    -object TYPENAME[,PROP1=VALUE1,...]
    -blockdev [driver=]driver[,node-name=N][,discard=ignore|unmap]

Frankly, this sucks.  Let's not mindlessly duplicate the suckage into
the storage daemon's help.

>> > +"\n"
>> >  QEMU_HELP_BOTTOM "\n",
>> >      error_get_progname());
>> >  }
>> >  
>> > +enum {
>> > +    OPTION_OBJECT = 256,
>> > +};
>> > +
>> > +static QemuOptsList qemu_object_opts = {
>> > +    .name = "object",
>> > +    .implied_opt_name = "qom-type",
>> > +    .head = QTAILQ_HEAD_INITIALIZER(qemu_object_opts.head),
>> > +    .desc = {
>> > +        { }
>> > +    },
>> > +};
>> > +
>> 
>> Note for later: copied from vl.c.
>> 
>> >  static int process_options(int argc, char *argv[], Error **errp)
>> >  {
>> >      int c;
>> > @@ -63,6 +81,7 @@ static int process_options(int argc, char *argv[], Error **errp)
>> >  
>> >      static const struct option long_options[] = {
>> >          {"help", no_argument, 0, 'h'},
>> > +        {"object", required_argument, 0, OPTION_OBJECT},
>> >          {"version", no_argument, 0, 'V'},
>> >          {"trace", required_argument, NULL, 'T'},
>> >          {0, 0, 0, 0}
>> > @@ -88,6 +107,22 @@ static int process_options(int argc, char *argv[], Error **errp)
>> >              g_free(trace_file);
>> >              trace_file = trace_opt_parse(optarg);
>> >              break;
>> > +        case OPTION_OBJECT:
>> > +            {
>> > +                QemuOpts *opts;
>> > +                const char *type;
>> > +
>> > +                opts = qemu_opts_parse(&qemu_object_opts,
>> > +                                       optarg, true, &error_fatal);
>> > +                type = qemu_opt_get(opts, "qom-type");
>> > +
>> > +                if (user_creatable_print_help(type, opts)) {
>> > +                    exit(EXIT_SUCCESS);
>> > +                }
>> > +                user_creatable_add_opts(opts, &error_fatal);
>> > +                qemu_opts_del(opts);
>> > +                break;
>> > +            }
>> >          }
>> >      }
>> >      if (optind != argc) {
>> 
>> PATCH 01 duplicates case QEMU_OPTION_trace pretty much verbatim.  Makes
>> sense, as qemu-storage-daemon is basically qemu-system-FOO with "FOO"
>> and most "system" cut away.
>> 
>> This patch adds vl.c's case QEMU_OPTION_object in a much simpler form.
>> This is one of my least favourite options, and I'll tell you why below.
>> Let's compare the two versions.
>> 
>> vl.c:
>> 
>>             case QEMU_OPTION_object:
>>                 opts = qemu_opts_parse_noisily(qemu_find_opts("object"),
>>                                                optarg, true);
>>                 if (!opts) {
>>                     exit(1);
>>                 }
>>                 break;
>> 
>> Further down:
>> 
>>     qemu_opts_foreach(qemu_find_opts("object"),
>>                       user_creatable_add_opts_foreach,
>>                       object_create_initial, &error_fatal);
>> 
>> Still further down:
>> 
>>     qemu_opts_foreach(qemu_find_opts("object"),
>>                       user_creatable_add_opts_foreach,
>>                       object_create_delayed, &error_fatal);
>> 
>> These are basically
>> 
>>     for opts in qemu_object_opts {
>>         type = qemu_opt_get(opts, "qom-type");
>>         if (type) {
>>             if (user_creatable_print_help(type, opts)) {
>>                 exit(0);
>>             }
>>             if (!predicate(type)) {
>>                 continue;
>>             }
>>         }
>>         obj = user_creatable_add_opts(opts, &error_fatal);
>>         object_unref(obj);
>>     }
>> 
>> where predicate(type) is true in exactly one of the two places for each
>> QOM type.
>> 
>> The reason for these gymnastics is to create objects at the right time
>> during startup, except there is no right time, but two.
>> 
>> Differences:
>> 
>> * Options are processed left to right without gymnastics.  Getting their
>>   order right is the user's problem.  I consider this an improvement.
>> 
>> * You use &qemu_object_opts instead of qemu_find_opts("object").  Also
>>   an improvement.
>> 
>> * You use qemu_opts_parse() instead of qemu_opts_parse_noisily().
>>   The latter can print help.  I failed to find a case where we lose help
>>   compared to qemu-system-FOO.  I didn't try very hard.
>
> I tried to reuse that code from qemu_opts_parse_noisily(), until I
> realised that it wasn't even used for -object.

What do you mean by "not used"?  vl.c:

            case QEMU_OPTION_object:
                opts = qemu_opts_parse_noisily(qemu_find_opts("object"),
                                               optarg, true);
                if (!opts) {
                    exit(1);
                }
                break;

> I don't remember the details why qemu_opts_print_help() wasn't even
> called, but it's obvious that we can't lose anything from it:
> qemu_object_opts has an empty list of properties, it accepts anything.
> QemuOpts can't print any useful help when this is all the information it
> has.

I see.

Do we want to bake that knowledge into main()?  I guess your answer
would be "we already do, we call user_creatable_print_help()".

Shouldn't we do the same in both main()s then?

>> * You neglect to guard user_creatable_print_help():
>> 
>>     $ qemu-storage-daemon --object wrong=1,help
>>     Segmentation fault (core dumped)
>
> Thanks for catching this. (You don't even need the ",help" part, just
> --object wrong=1 is enough.)
>
>> * You neglect to object_unref().  I just double-checked the final
>>   reference count: it's 2.
>
> Hm, yes. Weird interface, no caller actually needs this reference.

Feel free to simplify.

>> These bugs shouldn't be hard to fix.
>> 
>> 
>> At this point you might wonder why I dislike this option so much.
>> vl.c's gymnastics are ugly, but not unusually ugly, and they're gone
>> here.  To explain my distaste, I have to go back a little bit.
>> 
>> Like quite a few options, --object is paired with QMP command, namely
>> object-add.  Both have the same parameters: QOM type, object ID, and
>> additional type-specific object properties.  There's a difference,
>> though: object-add wraps the latter in a 'props' object, while --object
>> does not.
>> 
>> QAPI schema:
>> 
>>     { 'command': 'object-add',
>>       'data': {'qom-type': 'str', 'id': 'str', '*props': 'any'} }
>> 
>> QAPIfying this part of the CLI isn't easy.
>> 
>> The obvious QAPIfied CLI buddy of object-add is incompatible to current
>> --object.  That's not a concern for the storage daemon.  But it's also
>> ugly, because object-add's nesting of the type-specific properties
>> within @props is.  In QMP, it's merely yet another pair of curlies.  In
>> the CLI, we get to prefix props. to each type-specific property.
>> 
>> If we want to give the storage daemon a QAPIfied command line from the
>> start (and I think we do), we'll have to decide how to address this
>> issue, and possibly more (I'm only at PATCH 02/18).
>> 
>> We have a long history of rather careless interface design, and now some
>> of these chickens come home to roost.
>
> On IRC, we discussed last week that we could turn object-add into a
> 'gen': false command and accept things both as options in props and as
> flat options on the top level.

... to get rid of the nesting both for qemu-system-FOO (where we need
backward compatibility) and qemu-storage-daemon (where we don't).

We can also discuss getting rid of it only for qemu-storage-daemon.

> However, looking at this again, I'm afraid we forgot the context while
> discussing specifics: How would this be used in a command line parser?
>
> We don't start with a QDict here, but with a string. Getting a QDict
> that could serve as an input to a modified qmp_object_add() would still
> involve going through QemuOpts for parsing the option string, and then
> converting it to a QDict. Using a visitor isn't possible with '*props':
> 'any' and even less so with 'gen': false.
>
> So would this really improve things? Or do we have to wait until we have
> an actual schema for object-add before calling qmp_object_add() actually
> makes sense?

Good points.

qmp_object_add(), hmp_object_add() and vl.c's main() each use their own
interface to a common core.

QMP command handlers accept arguments specified in the schema as
separate C arguments.  For qmp_object_add(), that's @qom-type, @id (both
strings) and @props (a QDict).  It uses user_creatable_add_type() to
create an instance of QOM class @qom-type, set QOM properties for @props
with a QObject input visitor, insert into the QOM composition tree as
directed by @id.

This is simply how QMP commands work.  They parse JSON text into a
QDict, then feed it to the QObject input visitor to convert to native C
types.  The only slightly unusual thing is that when we feed the QDict
to the visitor in the generated qmp_marshal_object_add(), the visitor
passes through @props verbatim, because it's of type 'any', leaving the
conversion to native C types to the command handler.

With 'gen': false, we bypass qmp_marshal_object_add().  qmp_object_add()
gets the QDict, and does all the conversion to C types work.  Same as
now, plus two more visit_type_str() for @qom-type and @id.

main() parses the option argument as QemuOpts "monitor".  It uses
user_creatable_add_opts(), which splits it into @qom-type, @id and
@props for user_creatable_add_type(), then calls it with the options
visitor.

This is how the options visitor wants to be used.  We parse a
key=value,... string into a QemuOpts, then feed it to the options
visitor.

When I QAPIfied -blockdev (crudely!), I didn't use the options visitor,
because it is by design too limited for the job.  I created a new flow
instead: parse a key=value,... string into a QDict with keyval_parse(),
then feed it to the QObject input visitor.

This leads me to the flow I'd like us to try in the storage daemon:
parse into a QDict with keyval_parse(), feed to the 'gen': false
qmp_object_add().

Does that make some sense?

I've glossed over hmp_object_add() so far, because it's tangential to
the problem at hand.  Just for completeness (and laughs, possibly
bitter): HMP command handlers accept arguments specified in .hx as a
single QDict.  For hmp_object_add(), that's a string parsed as QemuOpts
"monitor" converted to QDict.  hmp_object_add() converts it right back
to QemuOpts "monitor", then uses user_creatable_add_opts().

Ugh!  Need I say more?



^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-11-06 14:58   ` Kevin Wolf
  2019-11-06 15:35     ` Max Reitz
  2019-11-06 17:13     ` Eric Blake
@ 2019-11-21 10:32     ` Stefan Hajnoczi
  2019-11-21 11:08       ` Kevin Wolf
  2 siblings, 1 reply; 65+ messages in thread
From: Stefan Hajnoczi @ 2019-11-21 10:32 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: qemu-devel, armbru, qemu-block, Max Reitz

[-- Attachment #1: Type: text/plain, Size: 655 bytes --]

On Wed, Nov 06, 2019 at 03:58:00PM +0100, Kevin Wolf wrote:
> Am 06.11.2019 um 15:37 hat Max Reitz geschrieben:
> > On 17.10.19 15:01, Kevin Wolf wrote:
> > The only thing I don’t like is the name, but that’s what <Tab> is for.
> > :-)
> 
> I'm open for suggestions, but I thought 'qsd' was a bit too terse. :-)
> 
> (Actually, maybe we could even pick something that doesn't mention
> storage or block? After all, it can do all kinds of QEMU backends in
> theory. Not sure if there's any standalone use for them, but who
> knows...)

It's likely that non-storage use cases will want to a daemon too.  Maybe
just qemu-daemon?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-11-21 10:32     ` Stefan Hajnoczi
@ 2019-11-21 11:08       ` Kevin Wolf
  2019-12-12 11:16         ` Stefan Hajnoczi
  0 siblings, 1 reply; 65+ messages in thread
From: Kevin Wolf @ 2019-11-21 11:08 UTC (permalink / raw)
  To: Stefan Hajnoczi; +Cc: qemu-devel, armbru, qemu-block, Max Reitz

[-- Attachment #1: Type: text/plain, Size: 1022 bytes --]

Am 21.11.2019 um 11:32 hat Stefan Hajnoczi geschrieben:
> On Wed, Nov 06, 2019 at 03:58:00PM +0100, Kevin Wolf wrote:
> > Am 06.11.2019 um 15:37 hat Max Reitz geschrieben:
> > > On 17.10.19 15:01, Kevin Wolf wrote:
> > > The only thing I don’t like is the name, but that’s what <Tab> is for.
> > > :-)
> > 
> > I'm open for suggestions, but I thought 'qsd' was a bit too terse. :-)
> > 
> > (Actually, maybe we could even pick something that doesn't mention
> > storage or block? After all, it can do all kinds of QEMU backends in
> > theory. Not sure if there's any standalone use for them, but who
> > knows...)
> 
> It's likely that non-storage use cases will want to a daemon too.  Maybe
> just qemu-daemon?

Do you have specific use cases in mind? Most backends seem rather
useless without a device to attach them to. In older QEMU versions,
maybe you could put multiple network backends into a common vlan to
forward between them, but even that vlan concept doesn't exist any more.

Kevin

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

* Re: [RFC PATCH 00/18] Add qemu-storage-daemon
  2019-11-21 11:08       ` Kevin Wolf
@ 2019-12-12 11:16         ` Stefan Hajnoczi
  0 siblings, 0 replies; 65+ messages in thread
From: Stefan Hajnoczi @ 2019-12-12 11:16 UTC (permalink / raw)
  To: Kevin Wolf; +Cc: qemu-devel, armbru, qemu-block, Max Reitz

[-- Attachment #1: Type: text/plain, Size: 1368 bytes --]

On Thu, Nov 21, 2019 at 12:08:16PM +0100, Kevin Wolf wrote:
> Am 21.11.2019 um 11:32 hat Stefan Hajnoczi geschrieben:
> > On Wed, Nov 06, 2019 at 03:58:00PM +0100, Kevin Wolf wrote:
> > > Am 06.11.2019 um 15:37 hat Max Reitz geschrieben:
> > > > On 17.10.19 15:01, Kevin Wolf wrote:
> > > > The only thing I don’t like is the name, but that’s what <Tab> is for.
> > > > :-)
> > > 
> > > I'm open for suggestions, but I thought 'qsd' was a bit too terse. :-)
> > > 
> > > (Actually, maybe we could even pick something that doesn't mention
> > > storage or block? After all, it can do all kinds of QEMU backends in
> > > theory. Not sure if there's any standalone use for them, but who
> > > knows...)
> > 
> > It's likely that non-storage use cases will want to a daemon too.  Maybe
> > just qemu-daemon?
> 
> Do you have specific use cases in mind? Most backends seem rather
> useless without a device to attach them to. In older QEMU versions,
> maybe you could put multiple network backends into a common vlan to
> forward between them, but even that vlan concept doesn't exist any more.

I was thinking about the model where device emulation happens in the
daemon.  We've since agreed that device emulation will continue to
happen in a more traditional QEMU process, so I can't think of
non-storage use cases at the moment.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

^ permalink raw reply	[flat|nested] 65+ messages in thread

end of thread, other threads:[~2019-12-12 11:18 UTC | newest]

Thread overview: 65+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-17 13:01 [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
2019-10-17 13:01 ` [RFC PATCH 01/18] qemu-storage-daemon: Add barebone tool Kevin Wolf
2019-10-24 13:50   ` Eric Blake
2019-11-13 14:12     ` Kevin Wolf
2019-11-06 12:11   ` Max Reitz
2019-11-07 16:21   ` Markus Armbruster
2019-10-17 13:01 ` [RFC PATCH 02/18] qemu-storage-daemon: Add --object option Kevin Wolf
2019-11-07 20:36   ` Markus Armbruster
2019-11-14 12:05     ` Kevin Wolf
2019-11-18  9:10       ` Markus Armbruster
2019-10-17 13:01 ` [RFC PATCH 03/18] stubs: Add arch_type Kevin Wolf
2019-10-17 13:01 ` [RFC PATCH 04/18] stubs: Add blk_by_qdev_id() Kevin Wolf
2019-11-08  9:03   ` Markus Armbruster
2019-10-17 13:01 ` [RFC PATCH 05/18] qemu-storage-daemon: Add --blockdev option Kevin Wolf
2019-11-08 13:29   ` Markus Armbruster
2019-10-17 13:01 ` [RFC PATCH 06/18] qemu-storage-daemon: Add --nbd-server option Kevin Wolf
2019-11-06 12:51   ` Max Reitz
2019-11-06 19:25     ` Eric Blake
2019-11-07  8:33       ` Kevin Wolf
2019-11-07 13:45         ` Eric Blake
2019-11-07 15:27           ` Kevin Wolf
2019-11-07 15:36             ` Eric Blake
2019-11-08 15:36     ` Markus Armbruster
2019-10-17 13:01 ` [RFC PATCH 07/18] blockdev-nbd: Boxed argument type for nbd-server-add Kevin Wolf
2019-10-17 13:01 ` [RFC PATCH 08/18] qemu-storage-daemon: Add --export option Kevin Wolf
2019-11-06 13:11   ` Max Reitz
2019-11-06 13:34     ` Kevin Wolf
2019-11-06 13:39       ` Max Reitz
2019-11-08 15:57       ` Markus Armbruster
2019-10-17 13:01 ` [RFC PATCH 09/18] qemu-storage-daemon: Add main loop Kevin Wolf
2019-11-08 16:02   ` Markus Armbruster
2019-10-17 13:01 ` [RFC PATCH 10/18] qemu-storage-daemon: Add --chardev option Kevin Wolf
2019-11-08 16:27   ` Markus Armbruster
2019-10-17 13:01 ` [RFC PATCH 11/18] monitor: Move monitor option parsing to monitor/monitor.c Kevin Wolf
2019-10-17 13:01 ` [RFC PATCH 12/18] stubs: Update monitor stubs for qemu-storage-daemon Kevin Wolf
2019-11-08 16:45   ` Markus Armbruster
2019-10-17 13:01 ` [RFC PATCH 13/18] qapi: Create module 'monitor' Kevin Wolf
2019-11-11  9:36   ` Markus Armbruster
2019-10-17 13:02 ` [RFC PATCH 14/18] monitor: Create monitor/qmp-cmds-monitor.c Kevin Wolf
2019-10-17 13:02 ` [RFC PATCH 15/18] qapi: Support empty modules Kevin Wolf
2019-11-12  8:29   ` Markus Armbruster
2019-10-17 13:02 ` [RFC PATCH 16/18] qapi: Create 'pragma' module Kevin Wolf
2019-11-12  9:41   ` Markus Armbruster
2019-10-17 13:02 ` [RFC PATCH 17/18] monitor: Move qmp_query_qmp_schema to qmp-cmds-monitor.c Kevin Wolf
2019-10-17 13:02 ` [RFC PATCH 18/18] qemu-storage-daemon: Add --monitor option Kevin Wolf
2019-11-06 14:32   ` Max Reitz
2019-11-07 10:12     ` Kevin Wolf
2019-11-07 10:44       ` Max Reitz
2019-11-08  8:59   ` Markus Armbruster
2019-11-12 14:25   ` Markus Armbruster
2019-11-13 10:58     ` Kevin Wolf
2019-11-13 13:53       ` Markus Armbruster
2019-10-24 11:33 ` [RFC PATCH 00/18] Add qemu-storage-daemon Kevin Wolf
2019-10-24 13:55 ` Vladimir Sementsov-Ogievskiy
2019-11-14 10:44   ` Kevin Wolf
2019-11-05 15:52 ` Stefan Hajnoczi
2019-11-06 14:37 ` Max Reitz
2019-11-06 14:58   ` Kevin Wolf
2019-11-06 15:35     ` Max Reitz
2019-11-06 17:13     ` Eric Blake
2019-11-21 10:32     ` Stefan Hajnoczi
2019-11-21 11:08       ` Kevin Wolf
2019-12-12 11:16         ` Stefan Hajnoczi
2019-11-07 10:33 ` Daniel P. Berrangé
2019-11-07 12:03   ` Kevin Wolf

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).