[Qemu-devel] [PATCH v8 00/23] QMP: out-of-band (OOB) execution support

[Qemu-devel] [PATCH v8 00/23] QMP: out-of-band (OOB) execution support

[Peter Xu]

Based-on: <20180306053320.15401-1-peterx@redhat.com>

The series is based on the following series:

  [PATCH v2 0/9] chardev: qio related non-default context support
  (actually patch 9 is based on version 2.1, Dan's version)

This version only contains some trivial changes, most of the work is
posted separately in either the QIO or CHARDEV series.

Four days until soft freeze.  Hmm...

Please review.  Thanks,

v8:
- fix up doc patch as suggested [Eric]
- rename functions in form of X_get_Y() or X_bh() [Stefan]
- English fixes [Stefan]
- make sure to kick iothread after suspend count update [Stefan]
- remove the hack to remove fd in monitor_init(), since now we have
  the QIO/CHARDEV series to solve the root problem.

v7:
- add some r-bs, and remove some.
- remove the chardev fix since already queued by Paolo
- use local var in qemu_chr_fe_add_watch [Stefan]
- move doc patch to front, mention it in some patches [Eric]
- Quite a few of English fixes [Eric]
- fix unlock missing in handle_qmp_command [Stefan]
- squash some patches according to the review comments
- don't break gdbserver usage on HMP non-interactive mode by fixing up
  the suspend/resume logic [Fam, Stefan]
- move the qemu_chr_fe_set_handlers() call in monitor_init() into a
  bottom half to avoid race between the call itself and
  iothread. [Stefan]
- spent quite a lot of time debugging another assertion failure in
  io_watch_poll_finalize() after above change is made (ouch! I really
  hoped we always have the latest glib): when QEMU inits chardevs in
  chardev_init_func() it's possible that QEMU registers the chardev
  handlers there, even before CharBackend is connected to that chardev
  in monitor_init().  Then, when we reach monitor_init() we must make
  sure we unregister that old one first, or there can have one orphan
  GSource still in default gcontext (note that this can really happen
  when we start to use QEMUBH to setup chardev frontends, which is
  above change).

v6:
- add r-bs
- s/negociate/negotiate/ [Dave]
- let mon_global be anonymous struct [Stefan]
- in monitor_init(): call qemu_chr_fe_set_handlers() later than init
  parser, then it's safe [Stefan]
- drop patch "qjson: add "opaque" field to JSONMessageParser",
  re-write the following one to use container_of(). [Stefan]
- keep get_qmp_greeting() the old way, add cap list [Stefan, Fam]
- fix the iothread_stop() comment since that's not really related to
  the glib bug [Stefan]
- when do qmp greeting, don't expose oob capability if iothread not
  used [Stefan, Fam]
- squash "qmp: introduce some capability helpers" into patch "monitor:
  separate QMP parser and dispatcher" [Fam]
- tune comments for monitor_qmp_respond(). [Stefan]
- use atomic_mb_{read|set}() where proper instead of no-mb ones
  [Stefan]
- add patch to let monitor suspend/resume really support QMP.  Notify
  iothread when needed, in both suspend/resume. [Stefan]
- add comment for qmp_queue_lock on lock taking ordering. [Stefan]
- refactor monitor_qmp_bh_dispatcher() to not loop, instead queue
  itself after each request. [Stefan]
- rename "request" into "command" in the new qmp event [Stefan]
- in handle_qmp_command() protect the queue length fetch with
  qmp_queue_lock [Stefan]
- one more s/2.11/2.12/.  [Fam]
- when isolating response queue, not only flush the queue, but also
  flush the monitor out buffer [Stefan]
- document rewrite [Stefan]
- some other touch-ups that I forgot to mention and some tiny new
  patches... anyway, rbs will be gone if any of the patch is touched
  up.  Please have a look!

v5
- rename "monitor_iothread" to "mon_iothread" [Dave]
- add comment in monitor_cleanup(), note that when the hacks can be
  removed. [Dan]
- add a note section in qmp-spec.txt, mentioning about how to migrate
  existing QMP command to oob-capable command. [Dave]
- drop patch "qmp: let migrate-incoming allow out-of-band".  All
  migration related changes will all be put into postcopy-recovery
  series.

v4:
- drop first patch to fix IOWatchPool [Stefan, Dan]
- add s-o-b where missing, and newly got r-bs
- fix English error in commit msg [Fam]
- some tunes on request-dropped event:  [Stefan]
  - firstly let 'id' be any type, meanwhile make sure "id" is there as
    long as OOB is enabled for the monitor.
  - some comments fix
- add new command "x-oob-test" for testing oob commands [Stefan]
- simplify the test codes to use new x-oob-test
- flush response queue before cleanup monitors

This series was born from this one:

  https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html

The idea comes from Markus Armbruster and the discussion we had in the
thread.  It's not a super ideal solution (I believe Markus had been
thinking hard to keep everything in order meanwhile trying to satisfy
the migration requirement), but AFAIU it's currently the best.

What is OOB?
============

It's the shortcut of Out-Of-Band execution, its name is given by
Markus.  It's a way to quickly execute a QMP request.  Say, originally
QMP is going throw these steps:

      JSON Parser --> QMP Dispatcher --> Respond
          /|\    (2)                (3)     |
       (1) |                               \|/ (4)
           +---------  main thread  --------+

The requests are executed by the so-called QMP-dispatcher after the
JSON is parsed.  If OOB is on, we run the command directly in the
parser and quickly returns.

This series changed the QMP handling logic by moving the command
parsing and responding phases into IOThreads, so to be more accurate,
after the series the above graph would change into this:

               queue/kick              queue/kick
     JSON Parser ======> QMP Dispatcher =====> Responder
         /|\ |     (3)       /|\  |      (4)      | /|\
      (1) |  | (2)            |   |               |  |
          |  |                |  \|/           (6)|  |(5)
          |  |            main thread             |  |
          |  |                                    |  |
          |  +--------> monitor IO thread <-------+  |
          +-----------/                   \----------+

New Interfaces
==============

QMP Introspection Changes
-------------------------

When do query-qmp-schema, we were getting something like:

  {"name": "migrate-incoming", "ret-type": "17",
   "meta-type": "command", "arg-type": "89"}

Now we will get a new key named "allow-oob":

  {"name": "migrate-incoming", "ret-type": "17", "allow-oob": true,
   "meta-type": "command", "arg-type": "89"}

Which shows whether a command supports OOB.

QMP Negociation Changes
-----------------------

We were running "qmp_capabilities" always without parameters like:

  {"execute": "qmp_capabilities"}

Now we can enable capabilities (we don't have any capability before
this series) like OOB using:

  {"execute": "qmp_capabilities", "arguments": {"enable": ["oob"]}}

Only after we explicitly enable OOB capability can we send requests in
OOB manner.  Otherwise we'll have exactly the same QMP session as
before, just like when OOB is not there.

When OOB is enabled, it's possible that OOB reply reaches faster than
previous command, so clients should be prepared.

Trigger OOB execution
---------------------

Let's take migrate-incoming as example.  The old command looks like:

  {"execute": "migrate-incoming", "arguments" : {"uri": "xxxxxx"}}

To execute a command with OOB execution, we need to specify it in the
QMP request in the extra "control" key:

  {"execute": "migrate-incoming", "arguments" : {"uri": "xxxxxx"},
   "control" { "run-oob": true } }

Then the command will be run in parser, and it can preempt other
commands.

Others
=================

The last patch of OOB test may need some attention.  I used
dump-guest-memory as a time-consuming command to test whether OOB is
working, and the only command I can test now is "migrate-incoming".  I
hope that is a "okay" solution for unit tests.  Any other suggestions
on that would be welcomed.

Please review.  Thanks.

Peter Xu (23):
  docs: update QMP documents for OOB commands
  qobject: introduce qstring_get_try_str()
  qobject: introduce qobject_get_try_str()
  qobject: let object_property_get_str() use new API
  monitor: move skip_flush into monitor_data_init
  monitor: move the cur_mon hack deeper for QMP
  monitor: unify global init
  monitor: let mon_list be tail queue
  monitor: allow using IO thread for parsing
  qmp: introduce QMPCapability
  monitor: introduce monitor_qmp_respond()
  monitor: let suspend_cnt be thread safe
  monitor: let suspend/resume work even with QMPs
  monitor: separate QMP parser and dispatcher
  qmp: add new event "command-dropped"
  monitor: send event when command queue full
  qapi: introduce new cmd option "allow-oob"
  qmp: support out-of-band (oob) execution
  qmp: isolate responses into io thread
  monitor: enable IO thread for (qmp & !mux) typed
  qmp: add command "x-oob-test"
  tests: qmp-test: verify command batching
  tests: qmp-test: add oob test

 docs/devel/qapi-code-gen.txt   |  65 +++-
 docs/interop/qmp-spec.txt      |  30 +-
 include/monitor/monitor.h      |   2 +-
 include/qapi/qmp/dispatch.h    |   7 +-
 include/qapi/qmp/qstring.h     |   2 +
 monitor.c                      | 695 ++++++++++++++++++++++++++++++++++++-----
 qapi/introspect.json           |   6 +-
 qapi/misc.json                 |  87 +++++-
 qapi/qmp-dispatch.c            |  33 +-
 qmp.c                          |  16 +
 qobject/qstring.c              |  21 ++
 qom/object.c                   |   9 +-
 scripts/qapi/commands.py       |  18 +-
 scripts/qapi/common.py         |  15 +-
 scripts/qapi/doc.py            |   2 +-
 scripts/qapi/introspect.py     |  10 +-
 tests/qapi-schema/test-qapi.py |   2 +-
 tests/qmp-test.c               |  97 +++++-
 trace-events                   |   3 +
 vl.c                           |   7 +-
 20 files changed, 1010 insertions(+), 117 deletions(-)

-- 
2.14.3

[Qemu-devel] [PATCH v8 01/23] docs: update QMP documents for OOB commands

[Peter Xu]

Update both the developer and spec for the new QMP OOB (Out-Of-Band)
command.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 docs/devel/qapi-code-gen.txt | 65 ++++++++++++++++++++++++++++++++++++++++----
 docs/interop/qmp-spec.txt    | 30 +++++++++++++++++---
 2 files changed, 86 insertions(+), 9 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index 25b7180a18..05fe8676af 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -554,9 +554,12 @@ following example objects:
 
 === Commands ===
 
+--- General Command Layout ---
+
 Usage: { 'command': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT,
          '*returns': TYPE-NAME, '*boxed': true,
-         '*gen': false, '*success-response': false }
+         '*gen': false, '*success-response': false,
+         '*allow-oob': true }
 
 Commands are defined by using a dictionary containing several members,
 where three members are most common.  The 'command' member is a
@@ -636,6 +639,56 @@ possible, the command expression should include the optional key
 'success-response' with boolean value false.  So far, only QGA makes
 use of this member.
 
+A command can be declared to support Out-Of-Band (OOB) execution.  By
+default, commands do not support OOB.  To declare a command that
+supports it, we need an extra 'allow-oob' field.  For example:
+
+ { 'command': 'migrate_recover',
+   'data': { 'uri': 'str' }, 'allow-oob': true }
+
+To execute a command in Out-Of-Band way, we need to specify the
+"control" field in the request, with "run-oob" set to true. Example:
+
+ => { "execute": "command-support-oob",
+      "arguments": { ... },
+      "control": { "run-oob": true } }
+ <= { "return": { } }
+
+Without it, even the commands that support out-of-band execution will
+still be run In-Band.
+
+--- About Out-Of-Band (OOB) Command Execution ---
+
+Out-Of-Band does not mean a special kind of command. Instead, it's a
+special way to execute the command.  One normal command can be
+declared to support Out-Of-Band execution when 'allow-oob' field is
+set to true when defining the command.  With that, it can be run in an
+Out-Of-Band way if 'run-oob' is specified in 'control' field of
+command request.
+
+Under normal QMP command execution, the following apply to each
+command:
+
+- They are executed in order,
+- They run only in main thread of QEMU,
+- They have the BQL taken during execution.
+
+When a command is executed with OOB, the following changes occur:
+
+- They can be completed before a pending in-band command,
+- They run in a monitor dedicated thread,
+- They do not take the BQL during execution.
+
+OOB command handlers must satisfy the following conditions:
+
+- It executes extremely fast,
+- It does not take any lock, or, it can take very small locks if all
+  critical regions also follow the rules for OOB command handler code,
+- It does not invoke system calls that may block,
+- It does not access guest RAM that may block when userfaultfd is
+  enabled for postcopy live migration.
+
+If in doubt, do not implement OOB execution support.
 
 === Events ===
 
@@ -739,10 +792,12 @@ references by name.
 QAPI schema definitions not reachable that way are omitted.
 
 The SchemaInfo for a command has meta-type "command", and variant
-members "arg-type" and "ret-type".  On the wire, the "arguments"
-member of a client's "execute" command must conform to the object type
-named by "arg-type".  The "return" member that the server passes in a
-success response conforms to the type named by "ret-type".
+members "arg-type", "ret-type" and "allow-oob".  On the wire, the
+"arguments" member of a client's "execute" command must conform to the
+object type named by "arg-type".  The "return" member that the server
+passes in a success response conforms to the type named by
+"ret-type".  When "allow-oob" is set, it means the command supports
+out-of-band execution.
 
 If the command takes no arguments, "arg-type" names an object type
 without members.  Likewise, if the command returns nothing, "ret-type"
diff --git a/docs/interop/qmp-spec.txt b/docs/interop/qmp-spec.txt
index f8b5356015..9a208589f6 100644
--- a/docs/interop/qmp-spec.txt
+++ b/docs/interop/qmp-spec.txt
@@ -83,16 +83,27 @@ The greeting message format is:
 2.2.1 Capabilities
 ------------------
 
-As of the date this document was last revised, no server or client
-capability strings have been defined.
+Currently supported capabilities are:
 
+- "oob": it means the QMP server supports "Out-Of-Band" command
+  execution.  For more details, please see "run-oob" parameter in
+  "Issuing Commands" section below.  Not all commands allow this "oob"
+  execution.  The "query-qmp-schema" command can be used to inspect
+  which commands support "oob" execution.
+
+QMP clients can get a list of supported QMP capabilities of the QMP
+server in the greeting message mentioned above.  By default, all the
+capabilities are off.  To enable any QMP capabilities, the QMP client
+needs to send the "qmp_capabilities" command with an extra parameter
+for the requested capabilities.
 
 2.3 Issuing Commands
 --------------------
 
 The format for command execution is:
 
-{ "execute": json-string, "arguments": json-object, "id": json-value }
+{ "execute": json-string, "arguments": json-object, "id": json-value,
+  "control": json-dict }
 
  Where,
 
@@ -102,10 +113,16 @@ The format for command execution is:
   required. Each command documents what contents will be considered
   valid when handling the json-argument
 - The "id" member is a transaction identification associated with the
-  command execution, it is optional and will be part of the response if
+  command execution.  It is required if OOB is enabled, and optional
+  if not.  The same "id" field will be part of the response if
   provided. The "id" member can be any json-value, although most
   clients merely use a json-number incremented for each successive
   command
+- The "control" member is optional, and currently only used for
+  "out-of-band" execution ("oob" as shortcut). The handling or
+  response of an "oob" command can overtake prior in-band commands.
+  To enable "oob" handling of a particular command, just provide a
+  control field with: { "control": { "run-oob": true } }
 
 2.4 Commands Responses
 ----------------------
@@ -113,6 +130,11 @@ The format for command execution is:
 There are two possible responses which the Server will issue as the result
 of a command execution: success or error.
 
+As long as the commands were issued with a proper "id" field, then the
+same "id" field will be attached in the corresponding response message
+so that requests and responses can match.  Clients should drop all the
+responses that are with unknown "id" field.
+
 2.4.1 success
 -------------
 
-- 
2.14.3

[Qemu-devel] [PATCH v8 02/23] qobject: introduce qstring_get_try_str()

[Peter Xu]

The only difference from qstring_get_str() is that it allows the qstring
to be NULL.  If so, NULL is returned.

CC: Eric Blake <eblake@redhat.com>
CC: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/qstring.h |  1 +
 qobject/qstring.c          | 10 ++++++++++
 2 files changed, 11 insertions(+)

diff --git a/include/qapi/qmp/qstring.h b/include/qapi/qmp/qstring.h
index 98070ef3d6..27a691fca1 100644
--- a/include/qapi/qmp/qstring.h
+++ b/include/qapi/qmp/qstring.h
@@ -27,6 +27,7 @@ QString *qstring_from_str(const char *str);
 QString *qstring_from_substr(const char *str, int start, int end);
 size_t qstring_get_length(const QString *qstring);
 const char *qstring_get_str(const QString *qstring);
+const char *qstring_get_try_str(const QString *qstring);
 void qstring_append_int(QString *qstring, int64_t value);
 void qstring_append(QString *qstring, const char *str);
 void qstring_append_chr(QString *qstring, int c);
diff --git a/qobject/qstring.c b/qobject/qstring.c
index 05b4bbc2d6..5564df5586 100644
--- a/qobject/qstring.c
+++ b/qobject/qstring.c
@@ -127,6 +127,16 @@ const char *qstring_get_str(const QString *qstring)
     return qstring->string;
 }
 
+/**
+ * qstring_get_try_str(): Return a pointer to the stored string
+ *
+ * NOTE: will return NULL if qstring is not provided.
+ */
+const char *qstring_get_try_str(const QString *qstring)
+{
+    return qstring ? qstring_get_str(qstring) : NULL;
+}
+
 /**
  * qstring_is_equal(): Test whether the two QStrings are equal
  */
-- 
2.14.3

[Qemu-devel] [PATCH v8 03/23] qobject: introduce qobject_get_try_str()

[Peter Xu]

A quick way to fetch string from qobject when it's a QString.

Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/qstring.h |  1 +
 qobject/qstring.c          | 11 +++++++++++
 2 files changed, 12 insertions(+)

diff --git a/include/qapi/qmp/qstring.h b/include/qapi/qmp/qstring.h
index 27a691fca1..2932dc3591 100644
--- a/include/qapi/qmp/qstring.h
+++ b/include/qapi/qmp/qstring.h
@@ -28,6 +28,7 @@ QString *qstring_from_substr(const char *str, int start, int end);
 size_t qstring_get_length(const QString *qstring);
 const char *qstring_get_str(const QString *qstring);
 const char *qstring_get_try_str(const QString *qstring);
+const char *qobject_get_try_str(const QObject *qstring);
 void qstring_append_int(QString *qstring, int64_t value);
 void qstring_append(QString *qstring, const char *str);
 void qstring_append_chr(QString *qstring, int c);
diff --git a/qobject/qstring.c b/qobject/qstring.c
index 5564df5586..9b3ee392ec 100644
--- a/qobject/qstring.c
+++ b/qobject/qstring.c
@@ -137,6 +137,17 @@ const char *qstring_get_try_str(const QString *qstring)
     return qstring ? qstring_get_str(qstring) : NULL;
 }
 
+/**
+ * qobject_get_try_str(): Return a pointer to the corresponding string
+ *
+ * NOTE: the string will only be returned if the object is valid, and
+ * its type is QString, otherwise NULL is returned.
+ */
+const char *qobject_get_try_str(const QObject *qstring)
+{
+    return qstring_get_try_str(qobject_to_qstring(qstring));
+}
+
 /**
  * qstring_is_equal(): Test whether the two QStrings are equal
  */
-- 
2.14.3

[Qemu-devel] [PATCH v8 04/23] qobject: let object_property_get_str() use new API

[Peter Xu]

We can simplify object_property_get_str() using the new
qobject_get_try_str().

Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qom/object.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/qom/object.c b/qom/object.c
index 755ad03819..00152fe3c1 100644
--- a/qom/object.c
+++ b/qom/object.c
@@ -1123,18 +1123,15 @@ char *object_property_get_str(Object *obj, const char *name,
                               Error **errp)
 {
     QObject *ret = object_property_get_qobject(obj, name, errp);
-    QString *qstring;
     char *retval;
 
     if (!ret) {
         return NULL;
     }
-    qstring = qobject_to_qstring(ret);
-    if (!qstring) {
+
+    retval = g_strdup(qobject_get_try_str(ret));
+    if (!retval) {
         error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name, "string");
-        retval = NULL;
-    } else {
-        retval = g_strdup(qstring_get_str(qstring));
     }
 
     qobject_decref(ret);
-- 
2.14.3

[Qemu-devel] [PATCH v8 05/23] monitor: move skip_flush into monitor_data_init

[Peter Xu]

It's part of the data init.  Collect it.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/monitor.c b/monitor.c
index a4417f26cd..48645e0e91 100644
--- a/monitor.c
+++ b/monitor.c
@@ -570,13 +570,14 @@ static void monitor_qapi_event_init(void)
 
 static void handle_hmp_command(Monitor *mon, const char *cmdline);
 
-static void monitor_data_init(Monitor *mon)
+static void monitor_data_init(Monitor *mon, bool skip_flush)
 {
     memset(mon, 0, sizeof(Monitor));
     qemu_mutex_init(&mon->out_lock);
     mon->outbuf = qstring_new();
     /* Use *mon_cmds by default. */
     mon->cmd_table = mon_cmds;
+    mon->skip_flush = skip_flush;
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -597,8 +598,7 @@ char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
     char *output = NULL;
     Monitor *old_mon, hmp;
 
-    monitor_data_init(&hmp);
-    hmp.skip_flush = true;
+    monitor_data_init(&hmp, true);
 
     old_mon = cur_mon;
     cur_mon = &hmp;
@@ -4039,7 +4039,7 @@ void monitor_init(Chardev *chr, int flags)
     }
 
     mon = g_malloc(sizeof(*mon));
-    monitor_data_init(mon);
+    monitor_data_init(mon, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
-- 
2.14.3

[Qemu-devel] [PATCH v8 06/23] monitor: move the cur_mon hack deeper for QMP

[Peter Xu]

In monitor_qmp_read(), we have the hack to temporarily replace the
cur_mon pointer.  Now we move this hack deeper inside the QMP dispatcher
routine since the Monitor pointer can be actually obtained using
container_of() upon the parser object, just like most of the other JSON
parser users do.

This does not make much sense as a single patch.  However, this will be
a big step for the next patch, when the QMP dispatcher routine will be
split from the QMP parser.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/monitor.c b/monitor.c
index 48645e0e91..2b70294772 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3762,7 +3762,9 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
 {
     QObject *req, *rsp = NULL, *id = NULL;
     QDict *qdict = NULL;
-    Monitor *mon = cur_mon;
+    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
+    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
+
     Error *err = NULL;
 
     req = json_parser_parse_err(tokens, NULL, &err);
@@ -3787,8 +3789,13 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
         QDECREF(req_json);
     }
 
+    old_mon = cur_mon;
+    cur_mon = mon;
+
     rsp = qmp_dispatch(cur_mon->qmp.commands, req);
 
+    cur_mon = old_mon;
+
     if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
         qdict = qdict_get_qdict(qobject_to_qdict(rsp), "error");
         if (qdict
@@ -3825,13 +3832,9 @@ err_out:
 
 static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
 {
-    Monitor *old_mon = cur_mon;
-
-    cur_mon = opaque;
-
-    json_message_parser_feed(&cur_mon->qmp.parser, (const char *) buf, size);
+    Monitor *mon = opaque;
 
-    cur_mon = old_mon;
+    json_message_parser_feed(&mon->qmp.parser, (const char *) buf, size);
 }
 
 static void monitor_read(void *opaque, const uint8_t *buf, int size)
-- 
2.14.3

[Qemu-devel] [PATCH v8 07/23] monitor: unify global init

[Peter Xu]

There are many places where the monitor initializes its globals:

- monitor_init_qmp_commands() at the very beginning
- single function to init monitor_lock
- in the first entry of monitor_init() using "is_first_init"

Unify them a bit.

monitor_lock is not used before monitor_init() (as confirmed by code
analysis and gdb watchpoints); so we are safe delaying what was a
constructor-time initialization of the mutex into the later first call
to monitor_init().

Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/monitor/monitor.h |  2 +-
 monitor.c                 | 25 ++++++++++---------------
 vl.c                      |  7 ++++++-
 3 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/include/monitor/monitor.h b/include/monitor/monitor.h
index d1024d4bdc..0cb0538a31 100644
--- a/include/monitor/monitor.h
+++ b/include/monitor/monitor.h
@@ -16,7 +16,7 @@ extern Monitor *cur_mon;
 
 bool monitor_cur_is_qmp(void);
 
-void monitor_init_qmp_commands(void);
+void monitor_init_globals(void);
 void monitor_init(Chardev *chr, int flags);
 void monitor_cleanup(void);
 
diff --git a/monitor.c b/monitor.c
index 2b70294772..16713b7c6d 100644
--- a/monitor.c
+++ b/monitor.c
@@ -1003,7 +1003,7 @@ static void qmp_unregister_commands_hack(void)
 #endif
 }
 
-void monitor_init_qmp_commands(void)
+static void monitor_init_qmp_commands(void)
 {
     /*
      * Two command lists:
@@ -3985,6 +3985,14 @@ static void sortcmdlist(void)
     qsort((void *)info_cmds, array_num, elem_size, compare_mon_cmd);
 }
 
+void monitor_init_globals(void)
+{
+    monitor_init_qmp_commands();
+    monitor_qapi_event_init();
+    sortcmdlist();
+    qemu_mutex_init(&monitor_lock);
+}
+
 /* These functions just adapt the readline interface in a typesafe way.  We
  * could cast function pointers but that discards compiler checks.
  */
@@ -4025,23 +4033,10 @@ void error_vprintf_unless_qmp(const char *fmt, va_list ap)
     }
 }
 
-static void __attribute__((constructor)) monitor_lock_init(void)
-{
-    qemu_mutex_init(&monitor_lock);
-}
-
 void monitor_init(Chardev *chr, int flags)
 {
-    static int is_first_init = 1;
-    Monitor *mon;
-
-    if (is_first_init) {
-        monitor_qapi_event_init();
-        sortcmdlist();
-        is_first_init = 0;
-    }
+    Monitor *mon = g_malloc(sizeof(*mon));
 
-    mon = g_malloc(sizeof(*mon));
     monitor_data_init(mon, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
diff --git a/vl.c b/vl.c
index 452fbf5043..4d6b36688f 100644
--- a/vl.c
+++ b/vl.c
@@ -3065,7 +3065,6 @@ int main(int argc, char **argv, char **envp)
     qemu_init_exec_dir(argv[0]);
 
     module_call_init(MODULE_INIT_QOM);
-    monitor_init_qmp_commands();
 
     qemu_add_opts(&qemu_drive_opts);
     qemu_add_drive_opts(&qemu_legacy_drive_opts);
@@ -4535,6 +4534,12 @@ int main(int argc, char **argv, char **envp)
     default_drive(default_floppy, snapshot, IF_FLOPPY, 0, FD_OPTS);
     default_drive(default_sdcard, snapshot, IF_SD, 0, SD_OPTS);
 
+    /*
+     * Note: qtest_enabled() (which is used in monitor_qapi_event_init())
+     * depends on configure_accelerator() above.
+     */
+    monitor_init_globals();
+
     if (qemu_opts_foreach(qemu_find_opts("mon"),
                           mon_init_func, NULL, NULL)) {
         exit(1);
-- 
2.14.3

[Qemu-devel] [PATCH v8 08/23] monitor: let mon_list be tail queue

[Peter Xu]

It was QLIST.  I want to use this list to do monitor priority job later,
which need tail insertion ability.  So switching to a tail queue.

Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/monitor.c b/monitor.c
index 16713b7c6d..5d10642b75 100644
--- a/monitor.c
+++ b/monitor.c
@@ -207,7 +207,7 @@ struct Monitor {
     void *password_opaque;
     mon_cmd_t *cmd_table;
     QLIST_HEAD(,mon_fd_t) fds;
-    QLIST_ENTRY(Monitor) entry;
+    QTAILQ_ENTRY(Monitor) entry;
 };
 
 /* QMP checker flags */
@@ -216,7 +216,7 @@ struct Monitor {
 /* Protects mon_list, monitor_event_state.  */
 static QemuMutex monitor_lock;
 
-static QLIST_HEAD(mon_list, Monitor) mon_list;
+static QTAILQ_HEAD(mon_list, Monitor) mon_list;
 static QLIST_HEAD(mon_fdsets, MonFdset) mon_fdsets;
 static int mon_refcount;
 
@@ -417,7 +417,7 @@ static void monitor_qapi_event_emit(QAPIEvent event, QDict *qdict)
     Monitor *mon;
 
     trace_monitor_protocol_event_emit(event, qdict);
-    QLIST_FOREACH(mon, &mon_list, entry) {
+    QTAILQ_FOREACH(mon, &mon_list, entry) {
         if (monitor_is_qmp(mon)
             && mon->qmp.commands != &qmp_cap_negotiation_commands) {
             monitor_json_emitter(mon, QOBJECT(qdict));
@@ -4060,7 +4060,7 @@ void monitor_init(Chardev *chr, int flags)
     }
 
     qemu_mutex_lock(&monitor_lock);
-    QLIST_INSERT_HEAD(&mon_list, mon, entry);
+    QTAILQ_INSERT_HEAD(&mon_list, mon, entry);
     qemu_mutex_unlock(&monitor_lock);
 }
 
@@ -4069,8 +4069,8 @@ void monitor_cleanup(void)
     Monitor *mon, *next;
 
     qemu_mutex_lock(&monitor_lock);
-    QLIST_FOREACH_SAFE(mon, &mon_list, entry, next) {
-        QLIST_REMOVE(mon, entry);
+    QTAILQ_FOREACH_SAFE(mon, &mon_list, entry, next) {
+        QTAILQ_REMOVE(&mon_list, mon, entry);
         monitor_data_destroy(mon);
         g_free(mon);
     }
-- 
2.14.3

[Qemu-devel] [PATCH v8 09/23] monitor: allow using IO thread for parsing

[Peter Xu]

For each Monitor, add one field "use_io_thr" to show whether it will be
using the dedicated monitor IO thread to handle input/output.  When set,
monitor IO parsing work will be offloaded to the dedicated monitor IO
thread, rather than the original main loop thread.

This only works for QMP.  HMP will always be run on the main loop
thread.

Currently we're still keeping use_io_thr off always.  Will turn it on
later at some point.

One thing to mention is that we cannot set use_io_thr for every QMP
monitor.  The problem is that MUXed typed chardevs may not work well
with it now. When MUX is used, frontend of chardev can be the monitor
plus something else.  The only thing we know would be safe to be run
outside main thread so far is the monitor frontend. All the rest of the
frontends should still be run in main thread only.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 90 insertions(+), 8 deletions(-)

diff --git a/monitor.c b/monitor.c
index 5d10642b75..6a5270dcd8 100644
--- a/monitor.c
+++ b/monitor.c
@@ -35,6 +35,7 @@
 #include "net/net.h"
 #include "net/slirp.h"
 #include "chardev/char-fe.h"
+#include "chardev/char-io.h"
 #include "ui/qemu-spice.h"
 #include "sysemu/numa.h"
 #include "monitor/monitor.h"
@@ -79,6 +80,7 @@
 #include "qapi/qapi-introspect.h"
 #include "sysemu/qtest.h"
 #include "sysemu/cpus.h"
+#include "sysemu/iothread.h"
 #include "qemu/cutils.h"
 
 #if defined(TARGET_S390X)
@@ -192,6 +194,7 @@ struct Monitor {
     int flags;
     int suspend_cnt;
     bool skip_flush;
+    bool use_io_thr;
 
     QemuMutex out_lock;
     QString *outbuf;
@@ -210,6 +213,11 @@ struct Monitor {
     QTAILQ_ENTRY(Monitor) entry;
 };
 
+/* Let's add monitor global variables to this struct. */
+static struct {
+    IOThread *mon_iothread;
+} mon_global;
+
 /* QMP checker flags */
 #define QMP_ACCEPT_UNKNOWNS 1
 
@@ -570,7 +578,8 @@ static void monitor_qapi_event_init(void)
 
 static void handle_hmp_command(Monitor *mon, const char *cmdline);
 
-static void monitor_data_init(Monitor *mon, bool skip_flush)
+static void monitor_data_init(Monitor *mon, bool skip_flush,
+                              bool use_io_thr)
 {
     memset(mon, 0, sizeof(Monitor));
     qemu_mutex_init(&mon->out_lock);
@@ -578,6 +587,7 @@ static void monitor_data_init(Monitor *mon, bool skip_flush)
     /* Use *mon_cmds by default. */
     mon->cmd_table = mon_cmds;
     mon->skip_flush = skip_flush;
+    mon->use_io_thr = use_io_thr;
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -598,7 +608,7 @@ char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
     char *output = NULL;
     Monitor *old_mon, hmp;
 
-    monitor_data_init(&hmp, true);
+    monitor_data_init(&hmp, true, false);
 
     old_mon = cur_mon;
     cur_mon = &hmp;
@@ -3985,12 +3995,29 @@ static void sortcmdlist(void)
     qsort((void *)info_cmds, array_num, elem_size, compare_mon_cmd);
 }
 
+static GMainContext *monitor_get_io_context(void)
+{
+    return iothread_get_g_main_context(mon_global.mon_iothread);
+}
+
+static AioContext *monitor_get_aio_context(void)
+{
+    return iothread_get_aio_context(mon_global.mon_iothread);
+}
+
+static void monitor_iothread_init(void)
+{
+    mon_global.mon_iothread = iothread_create("mon_iothread",
+                                              &error_abort);
+}
+
 void monitor_init_globals(void)
 {
     monitor_init_qmp_commands();
     monitor_qapi_event_init();
     sortcmdlist();
     qemu_mutex_init(&monitor_lock);
+    monitor_iothread_init();
 }
 
 /* These functions just adapt the readline interface in a typesafe way.  We
@@ -4033,11 +4060,41 @@ void error_vprintf_unless_qmp(const char *fmt, va_list ap)
     }
 }
 
+static void monitor_list_append(Monitor *mon)
+{
+    qemu_mutex_lock(&monitor_lock);
+    QTAILQ_INSERT_HEAD(&mon_list, mon, entry);
+    qemu_mutex_unlock(&monitor_lock);
+}
+
+static void monitor_qmp_setup_handlers_bh(void *opaque)
+{
+    Monitor *mon = opaque;
+    GMainContext *context;
+
+    if (mon->use_io_thr) {
+        /*
+         * When use_io_thr is set, we use the global shared dedicated
+         * IO thread for this monitor to handle input/output.
+         */
+        context = monitor_get_io_context();
+        /* We should have inited globals before reaching here. */
+        assert(context);
+    } else {
+        /* The default main loop, which is the main thread */
+        context = NULL;
+    }
+
+    qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
+                             monitor_qmp_event, NULL, mon, context, true);
+    monitor_list_append(mon);
+}
+
 void monitor_init(Chardev *chr, int flags)
 {
     Monitor *mon = g_malloc(sizeof(*mon));
 
-    monitor_data_init(mon, false);
+    monitor_data_init(mon, false, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
@@ -4050,24 +4107,46 @@ void monitor_init(Chardev *chr, int flags)
     }
 
     if (monitor_is_qmp(mon)) {
-        qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
-                                 monitor_qmp_event, NULL, mon, NULL, true);
         qemu_chr_fe_set_echo(&mon->chr, true);
         json_message_parser_init(&mon->qmp.parser, handle_qmp_command);
+        if (mon->use_io_thr) {
+            /*
+             * We can't call qemu_chr_fe_set_handlers() directly here
+             * since during the procedure the chardev will be active
+             * and running in monitor iothread, while we'll still do
+             * something before returning from it, which is a possible
+             * race too.  To avoid that, we just create a BH to setup
+             * the handlers.
+             */
+            aio_bh_schedule_oneshot(monitor_get_aio_context(),
+                                    monitor_qmp_setup_handlers_bh, mon);
+            /* We'll add this to mon_list in the BH when setup done */
+            return;
+        } else {
+            qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read,
+                                     monitor_qmp_read, monitor_qmp_event,
+                                     NULL, mon, NULL, true);
+        }
     } else {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_read,
                                  monitor_event, NULL, mon, NULL, true);
     }
 
-    qemu_mutex_lock(&monitor_lock);
-    QTAILQ_INSERT_HEAD(&mon_list, mon, entry);
-    qemu_mutex_unlock(&monitor_lock);
+    monitor_list_append(mon);
 }
 
 void monitor_cleanup(void)
 {
     Monitor *mon, *next;
 
+    /*
+     * We need to explicitly stop the iothread (but not destroy it),
+     * cleanup the monitor resources, then destroy the iothread since
+     * we need to unregister from chardev below in
+     * monitor_data_destroy(), and chardev is not thread-safe yet
+     */
+    iothread_stop(mon_global.mon_iothread);
+
     qemu_mutex_lock(&monitor_lock);
     QTAILQ_FOREACH_SAFE(mon, &mon_list, entry, next) {
         QTAILQ_REMOVE(&mon_list, mon, entry);
@@ -4075,6 +4154,9 @@ void monitor_cleanup(void)
         g_free(mon);
     }
     qemu_mutex_unlock(&monitor_lock);
+
+    iothread_destroy(mon_global.mon_iothread);
+    mon_global.mon_iothread = NULL;
 }
 
 QemuOptsList qemu_mon_opts = {
-- 
2.14.3

[Qemu-devel] [PATCH v8 10/23] qmp: introduce QMPCapability

[Peter Xu]

There were no QMP capabilities defined.  Define the first "oob" as
capability to allow out-of-band messages.

After this patch, we will allow QMP clients to enable QMP capabilities
when sending the first "qmp_capabilities" command.  Originally we are
starting QMP session with no arguments like:

  { "execute": "qmp_capabilities" }

Now we can enable some QMP capabilities using (take OOB as example,
which is the only one capability that we support):

  { "execute": "qmp_capabilities",
    "argument": { "enable": [ "oob" ] } }

When the "argument" key is not provided, no capability is enabled.

For capability "oob", the monitor needs to be run on dedicated IO
thread, otherwise the command will fail.  For example, trying to enable
OOB on a MUXed typed QMP monitor will fail.

One thing to mention is that, QMP capabilities are per-monitor, and also
when the connection is closed due to some reason, the capabilities will
be reset.

Also, touch up qmp-test.c to test the new bits.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c        | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++----
 qapi/misc.json   | 32 ++++++++++++++++++++---
 tests/qmp-test.c | 10 +++++++-
 3 files changed, 110 insertions(+), 9 deletions(-)

diff --git a/monitor.c b/monitor.c
index 6a5270dcd8..74eb1d8b8b 100644
--- a/monitor.c
+++ b/monitor.c
@@ -59,6 +59,7 @@
 #include "qapi/qmp/qjson.h"
 #include "qapi/qmp/json-streamer.h"
 #include "qapi/qmp/json-parser.h"
+#include "qapi/qmp/qlist.h"
 #include "qom/object_interfaces.h"
 #include "trace-root.h"
 #include "trace/control.h"
@@ -170,6 +171,7 @@ typedef struct {
      * mode.
      */
     QmpCommandList *commands;
+    bool qmp_caps[QMP_CAPABILITY__MAX];
 } MonitorQMP;
 
 /*
@@ -1039,8 +1041,42 @@ static void monitor_init_qmp_commands(void)
                          qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
 }
 
-void qmp_qmp_capabilities(Error **errp)
+static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
+                           Error **errp)
+{
+    for (; list; list = list->next) {
+        assert(list->value < QMP_CAPABILITY__MAX);
+        switch (list->value) {
+        case QMP_CAPABILITY_OOB:
+            if (!mon->use_io_thr) {
+                /*
+                 * Out-Of-Band only works with monitors that are
+                 * running on dedicated IOThread.
+                 */
+                error_setg(errp, "This monitor does not support "
+                           "Out-Of-Band (OOB)");
+                return;
+            }
+            break;
+        default:
+            break;
+        }
+    }
+}
+
+/* This function should only be called after capabilities are checked. */
+static void qmp_caps_apply(Monitor *mon, QMPCapabilityList *list)
 {
+    for (; list; list = list->next) {
+        mon->qmp.qmp_caps[list->value] = true;
+    }
+}
+
+void qmp_qmp_capabilities(bool has_enable, QMPCapabilityList *enable,
+                          Error **errp)
+{
+    Error *local_err = NULL;
+
     if (cur_mon->qmp.commands == &qmp_commands) {
         error_set(errp, ERROR_CLASS_COMMAND_NOT_FOUND,
                   "Capabilities negotiation is already complete, command "
@@ -1048,6 +1084,21 @@ void qmp_qmp_capabilities(Error **errp)
         return;
     }
 
+    /* Enable QMP capabilities provided by the client if applicable. */
+    if (has_enable) {
+        qmp_caps_check(cur_mon, enable, &local_err);
+        if (local_err) {
+            /*
+             * Failed check on any of the capabilities will fail the
+             * entire command (and thus not apply any of the other
+             * capabilities that were also requested).
+             */
+            error_propagate(errp, local_err);
+            return;
+        }
+        qmp_caps_apply(cur_mon, enable);
+    }
+
     cur_mon->qmp.commands = &qmp_commands;
 }
 
@@ -3893,14 +3944,29 @@ void monitor_resume(Monitor *mon)
         readline_show_prompt(mon->rs);
 }
 
-static QObject *get_qmp_greeting(void)
+static QObject *get_qmp_greeting(Monitor *mon)
 {
+    QList *cap_list = qlist_new();
     QObject *ver = NULL;
+    QMPCapability cap;
 
     qmp_marshal_query_version(NULL, &ver, NULL);
 
-    return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': []}}",
-                              ver);
+    for (cap = 0; cap < QMP_CAPABILITY__MAX; cap++) {
+        if (!mon->use_io_thr && cap == QMP_CAPABILITY_OOB) {
+            /* Monitors that are not using IOThread won't support OOB */
+            continue;
+        }
+        qlist_append(cap_list, qstring_from_str(QMPCapability_str(cap)));
+    }
+
+    return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': %p}}",
+                              ver, cap_list);
+}
+
+static void monitor_qmp_caps_reset(Monitor *mon)
+{
+    memset(mon->qmp.qmp_caps, 0, sizeof(mon->qmp.qmp_caps));
 }
 
 static void monitor_qmp_event(void *opaque, int event)
@@ -3911,7 +3977,8 @@ static void monitor_qmp_event(void *opaque, int event)
     switch (event) {
     case CHR_EVENT_OPENED:
         mon->qmp.commands = &qmp_cap_negotiation_commands;
-        data = get_qmp_greeting();
+        monitor_qmp_caps_reset(mon);
+        data = get_qmp_greeting(mon);
         monitor_json_emitter(mon, data);
         qobject_decref(data);
         mon_refcount++;
diff --git a/qapi/misc.json b/qapi/misc.json
index bd04469a4b..0123143d4b 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -10,21 +10,47 @@
 #
 # Enable QMP capabilities.
 #
-# Arguments: None.
+# Arguments:
+#
+# @enable:   An optional list of QMPCapability values to enable.  The
+#            client must not enable any capability that is not
+#            mentioned in the QMP greeting message.  If the field is not
+#            provided, it means no QMP capabilities will be enabled.
+#            (since 2.12)
 #
 # Example:
 #
-# -> { "execute": "qmp_capabilities" }
+# -> { "execute": "qmp_capabilities",
+#      "arguments": { "enable": [ "oob" ] } }
 # <- { "return": {} }
 #
 # Notes: This command is valid exactly when first connecting: it must be
 # issued before any other command will be accepted, and will fail once the
 # monitor is accepting other commands. (see qemu docs/interop/qmp-spec.txt)
 #
+# The QMP client needs to explicitly enable QMP capabilities, otherwise
+# all the QMP capabilities will be turned off by default.
+#
 # Since: 0.13
 #
 ##
-{ 'command': 'qmp_capabilities' }
+{ 'command': 'qmp_capabilities',
+  'data': { '*enable': [ 'QMPCapability' ] } }
+
+##
+# @QMPCapability:
+#
+# Enumeration of capabilities to be advertised during initial client
+# connection, used for agreeing on particular QMP extension behaviors.
+#
+# @oob:   QMP ability to support Out-Of-Band requests.
+#         (Please refer to qmp-spec.txt for more information on OOB)
+#
+# Since: 2.12
+#
+##
+{ 'enum': 'QMPCapability',
+  'data': [ 'oob' ] }
 
 ##
 # @VersionTriple:
diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index 22445d9ec2..70d1152d32 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -20,6 +20,7 @@
 #include "qapi/qobject-input-visitor.h"
 #include "qapi/util.h"
 #include "qapi/visitor.h"
+#include "qapi/qmp/qstring.h"
 
 const char common_args[] = "-nodefaults -machine none";
 
@@ -79,6 +80,8 @@ static void test_qmp_protocol(void)
     QDict *resp, *q, *ret;
     QList *capabilities;
     QTestState *qts;
+    const QListEntry *entry;
+    QString *qstr;
 
     qts = qtest_init_without_qmp_handshake(common_args);
 
@@ -88,7 +91,12 @@ static void test_qmp_protocol(void)
     g_assert(q);
     test_version(qdict_get(q, "version"));
     capabilities = qdict_get_qlist(q, "capabilities");
-    g_assert(capabilities && qlist_empty(capabilities));
+    g_assert(capabilities);
+    entry = qlist_first(capabilities);
+    g_assert(entry);
+    qstr = qobject_to_qstring(entry->value);
+    g_assert(qstr);
+    g_assert_cmpstr(qstring_get_str(qstr), ==, "oob");
     QDECREF(resp);
 
     /* Test valid command before handshake */
-- 
2.14.3

[Qemu-devel] [PATCH v8 11/23] monitor: introduce monitor_qmp_respond()

[Peter Xu]

A tiny refactoring, preparing to split the QMP dispatcher away.

Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 50 +++++++++++++++++++++++++++++++++-----------------
 1 file changed, 33 insertions(+), 17 deletions(-)

diff --git a/monitor.c b/monitor.c
index 74eb1d8b8b..6959e963e2 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3819,6 +3819,38 @@ static int monitor_can_read(void *opaque)
     return (mon->suspend_cnt == 0) ? 1 : 0;
 }
 
+/*
+ * 1. This function takes ownership of rsp, err, and id.
+ * 2. rsp, err, and id may be NULL.
+ * 3. If err != NULL then rsp must be NULL.
+ */
+static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
+                                Error *err, QObject *id)
+{
+    QDict *qdict = NULL;
+
+    if (err) {
+        assert(!rsp);
+        qdict = qdict_new();
+        qdict_put_obj(qdict, "error", qmp_build_error_object(err));
+        error_free(err);
+        rsp = QOBJECT(qdict);
+    }
+
+    if (rsp) {
+        if (id) {
+            /* This is for the qdict below. */
+            qobject_incref(id);
+            qdict_put_obj(qobject_to_qdict(rsp), "id", id);
+        }
+
+        monitor_json_emitter(mon, rsp);
+    }
+
+    qobject_decref(id);
+    qobject_decref(rsp);
+}
+
 static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
 {
     QObject *req, *rsp = NULL, *id = NULL;
@@ -3870,24 +3902,8 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
     }
 
 err_out:
-    if (err) {
-        qdict = qdict_new();
-        qdict_put_obj(qdict, "error", qmp_build_error_object(err));
-        error_free(err);
-        rsp = QOBJECT(qdict);
-    }
+    monitor_qmp_respond(mon, rsp, err, id);
 
-    if (rsp) {
-        if (id) {
-            qdict_put_obj(qobject_to_qdict(rsp), "id", id);
-            id = NULL;
-        }
-
-        monitor_json_emitter(mon, rsp);
-    }
-
-    qobject_decref(id);
-    qobject_decref(rsp);
     qobject_decref(req);
 }
 
-- 
2.14.3

[Qemu-devel] [PATCH v8 12/23] monitor: let suspend_cnt be thread safe

[Peter Xu]

Monitor code now can be run in more than one thread.  Let it be thread
safe when accessing suspend_cnt counter.

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/monitor.c b/monitor.c
index 6959e963e2..71b696c8f3 100644
--- a/monitor.c
+++ b/monitor.c
@@ -194,7 +194,7 @@ struct Monitor {
     CharBackend chr;
     int reset_seen;
     int flags;
-    int suspend_cnt;
+    int suspend_cnt;            /* Needs to be accessed atomically */
     bool skip_flush;
     bool use_io_thr;
 
@@ -3816,7 +3816,7 @@ static int monitor_can_read(void *opaque)
 {
     Monitor *mon = opaque;
 
-    return (mon->suspend_cnt == 0) ? 1 : 0;
+    return !atomic_mb_read(&mon->suspend_cnt);
 }
 
 /*
@@ -3948,7 +3948,7 @@ int monitor_suspend(Monitor *mon)
 {
     if (!mon->rs)
         return -ENOTTY;
-    mon->suspend_cnt++;
+    atomic_inc(&mon->suspend_cnt);
     return 0;
 }
 
@@ -3956,8 +3956,9 @@ void monitor_resume(Monitor *mon)
 {
     if (!mon->rs)
         return;
-    if (--mon->suspend_cnt == 0)
+    if (atomic_dec_fetch(&mon->suspend_cnt) == 0) {
         readline_show_prompt(mon->rs);
+    }
 }
 
 static QObject *get_qmp_greeting(Monitor *mon)
@@ -4022,19 +4023,19 @@ static void monitor_event(void *opaque, int event)
             monitor_resume(mon);
             monitor_flush(mon);
         } else {
-            mon->suspend_cnt = 0;
+            atomic_mb_set(&mon->suspend_cnt, 0);
         }
         break;
 
     case CHR_EVENT_MUX_OUT:
         if (mon->reset_seen) {
-            if (mon->suspend_cnt == 0) {
+            if (atomic_mb_read(&mon->suspend_cnt) == 0) {
                 monitor_printf(mon, "\n");
             }
             monitor_flush(mon);
             monitor_suspend(mon);
         } else {
-            mon->suspend_cnt++;
+            atomic_inc(&mon->suspend_cnt);
         }
         qemu_mutex_lock(&mon->out_lock);
         mon->mux_out = 1;
-- 
2.14.3

[Qemu-devel] [PATCH v8 13/23] monitor: let suspend/resume work even with QMPs

[Peter Xu]

This patches allows QMP monitors to be suspended/resumed.

One thing to mention is that for QMPs that are using IOThreads, we need
an explicit kick for the IOThread in case it is sleeping.

Meanwhile, we need to take special care on non-interactive HMPs.
Currently only gdbserver is using that.  For these monitors, we still
don't allow suspend/resume operations.

Since at it, add traces for the operations.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c    | 47 ++++++++++++++++++++++++++++++++++++++++++++---
 trace-events |  1 +
 2 files changed, 45 insertions(+), 3 deletions(-)

diff --git a/monitor.c b/monitor.c
index 71b696c8f3..de9343be87 100644
--- a/monitor.c
+++ b/monitor.c
@@ -250,6 +250,21 @@ static inline bool monitor_is_qmp(const Monitor *mon)
     return (mon->flags & MONITOR_USE_CONTROL);
 }
 
+/**
+ * Whether @mon is using readline?  Note: not all HMP monitors use
+ * readline, e.g., gdbserver has a non-interactive HMP monitor, so
+ * readline is not used there.
+ */
+static inline bool monitor_uses_readline(const Monitor *mon)
+{
+    return mon->flags & MONITOR_USE_READLINE;
+}
+
+static inline bool monitor_is_hmp_non_interactive(const Monitor *mon)
+{
+    return !monitor_is_qmp(mon) && !monitor_uses_readline(mon);
+}
+
 /**
  * Is the current monitor, if any, a QMP monitor?
  */
@@ -3946,19 +3961,45 @@ static void monitor_command_cb(void *opaque, const char *cmdline,
 
 int monitor_suspend(Monitor *mon)
 {
-    if (!mon->rs)
+    if (monitor_is_hmp_non_interactive(mon)) {
         return -ENOTTY;
+    }
+
     atomic_inc(&mon->suspend_cnt);
+
+    if (monitor_is_qmp(mon)) {
+        /*
+         * Kick iothread to make sure this takes effect.  It'll be
+         * evaluated again in prepare() of the watch object.
+         */
+        aio_notify(iothread_get_aio_context(mon_global.mon_iothread));
+    }
+
+    trace_monitor_suspend(mon, 1);
     return 0;
 }
 
 void monitor_resume(Monitor *mon)
 {
-    if (!mon->rs)
+    if (monitor_is_hmp_non_interactive(mon)) {
         return;
+    }
+
     if (atomic_dec_fetch(&mon->suspend_cnt) == 0) {
-        readline_show_prompt(mon->rs);
+        if (monitor_is_qmp(mon)) {
+            /*
+             * For QMP monitors that are running in IOThread, let's
+             * kick the thread in case it's sleeping.
+             */
+            if (mon->use_io_thr) {
+                aio_notify(iothread_get_aio_context(mon_global.mon_iothread));
+            }
+        } else {
+            assert(mon->rs);
+            readline_show_prompt(mon->rs);
+        }
     }
+    trace_monitor_suspend(mon, -1);
 }
 
 static QObject *get_qmp_greeting(Monitor *mon)
diff --git a/trace-events b/trace-events
index 89fcad0fd1..fe10c3b487 100644
--- a/trace-events
+++ b/trace-events
@@ -47,6 +47,7 @@ monitor_protocol_event_emit(uint32_t event, void *data) "event=%d data=%p"
 monitor_protocol_event_queue(uint32_t event, void *qdict, uint64_t rate) "event=%d data=%p rate=%" PRId64
 handle_hmp_command(void *mon, const char *cmdline) "mon %p cmdline: %s"
 handle_qmp_command(void *mon, const char *req) "mon %p req: %s"
+monitor_suspend(void *ptr, int cnt) "mon %p: %d"
 
 # dma-helpers.c
 dma_blk_io(void *dbs, void *bs, int64_t offset, bool to_dev) "dbs=%p bs=%p offset=%" PRId64 " to_dev=%d"
-- 
2.14.3

[Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Peter Xu]

Originally QMP goes through these steps:

  JSON Parser --> QMP Dispatcher --> Respond
      /|\    (2)                (3)     |
   (1) |                               \|/ (4)
       +---------  main thread  --------+

This patch does this:

  JSON Parser     QMP Dispatcher --> Respond
      /|\ |           /|\       (4)     |
       |  | (2)        | (3)            |  (5)
   (1) |  +----->      |               \|/
       +---------  main thread  <-------+

So the parsing job and the dispatching job is isolated now.  It gives us
a chance in following up patches to totally move the parser outside.

The isolation is done using one QEMUBH. Only one dispatcher QEMUBH is
used for all the monitors.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-------
 1 file changed, 178 insertions(+), 23 deletions(-)

diff --git a/monitor.c b/monitor.c
index de9343be87..5104e5db07 100644
--- a/monitor.c
+++ b/monitor.c
@@ -172,6 +172,13 @@ typedef struct {
      */
     QmpCommandList *commands;
     bool qmp_caps[QMP_CAPABILITY__MAX];
+    /*
+     * Protects qmp request/response queue.  Please take monitor_lock
+     * first when used together.
+     */
+    QemuMutex qmp_queue_lock;
+    /* Input queue that holds all the parsed QMP requests */
+    GQueue *qmp_requests;
 } MonitorQMP;
 
 /*
@@ -218,6 +225,8 @@ struct Monitor {
 /* Let's add monitor global variables to this struct. */
 static struct {
     IOThread *mon_iothread;
+    /* Bottom half to dispatch the requests received from IO thread */
+    QEMUBH *qmp_dispatcher_bh;
 } mon_global;
 
 /* QMP checker flags */
@@ -600,11 +609,13 @@ static void monitor_data_init(Monitor *mon, bool skip_flush,
 {
     memset(mon, 0, sizeof(Monitor));
     qemu_mutex_init(&mon->out_lock);
+    qemu_mutex_init(&mon->qmp.qmp_queue_lock);
     mon->outbuf = qstring_new();
     /* Use *mon_cmds by default. */
     mon->cmd_table = mon_cmds;
     mon->skip_flush = skip_flush;
     mon->use_io_thr = use_io_thr;
+    mon->qmp.qmp_requests = g_queue_new();
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -617,6 +628,8 @@ static void monitor_data_destroy(Monitor *mon)
     readline_free(mon->rs);
     QDECREF(mon->outbuf);
     qemu_mutex_destroy(&mon->out_lock);
+    qemu_mutex_destroy(&mon->qmp.qmp_queue_lock);
+    g_queue_free(mon->qmp.qmp_requests);
 }
 
 char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
@@ -1056,6 +1069,16 @@ static void monitor_init_qmp_commands(void)
                          qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
 }
 
+static bool qmp_cap_enabled(Monitor *mon, QMPCapability cap)
+{
+    return mon->qmp.qmp_caps[cap];
+}
+
+static bool qmp_oob_enabled(Monitor *mon)
+{
+    return qmp_cap_enabled(mon, QMP_CAPABILITY_OOB);
+}
+
 static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
                            Error **errp)
 {
@@ -3866,30 +3889,39 @@ static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
     qobject_decref(rsp);
 }
 
-static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
+struct QMPRequest {
+    /* Owner of the request */
+    Monitor *mon;
+    /* "id" field of the request */
+    QObject *id;
+    /* Request object to be handled */
+    QObject *req;
+    /*
+     * Whether we need to resume the monitor afterward.  This flag is
+     * used to emulate the old QMP server behavior that the current
+     * command must be completed before execution of the next one.
+     */
+    bool need_resume;
+};
+typedef struct QMPRequest QMPRequest;
+
+/*
+ * Dispatch one single QMP request. The function will free the req_obj
+ * and objects inside it before return.
+ */
+static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
 {
-    QObject *req, *rsp = NULL, *id = NULL;
+    Monitor *mon, *old_mon;
+    QObject *req, *rsp = NULL, *id;
     QDict *qdict = NULL;
-    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
-    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
-
-    Error *err = NULL;
+    bool need_resume;
 
-    req = json_parser_parse_err(tokens, NULL, &err);
-    if (!req && !err) {
-        /* json_parser_parse_err() sucks: can fail without setting @err */
-        error_setg(&err, QERR_JSON_PARSING);
-    }
-    if (err) {
-        goto err_out;
-    }
+    req = req_obj->req;
+    mon = req_obj->mon;
+    id = req_obj->id;
+    need_resume = req_obj->need_resume;
 
-    qdict = qobject_to_qdict(req);
-    if (qdict) {
-        id = qdict_get(qdict, "id");
-        qobject_incref(id);
-        qdict_del(qdict, "id");
-    } /* else will fail qmp_dispatch() */
+    g_free(req_obj);
 
     if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
         QString *req_json = qobject_to_json(req);
@@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
     old_mon = cur_mon;
     cur_mon = mon;
 
-    rsp = qmp_dispatch(cur_mon->qmp.commands, req);
+    rsp = qmp_dispatch(mon->qmp.commands, req);
 
     cur_mon = old_mon;
 
@@ -3916,12 +3948,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
         }
     }
 
-err_out:
-    monitor_qmp_respond(mon, rsp, err, id);
+    /* Respond if necessary */
+    monitor_qmp_respond(mon, rsp, NULL, id);
+
+    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
+    if (need_resume) {
+        monitor_resume(mon);
+    }
 
     qobject_decref(req);
 }
 
+/*
+ * Pop one QMP request from monitor queues, return NULL if not found.
+ * We are using round-robin fashion to pop the request, to avoid
+ * processing command only on a very busy monitor.  To achieve that,
+ * when we processed one request on specific monitor, we put that
+ * monitor to the end of mon_list queue.
+ */
+static QMPRequest *monitor_qmp_requests_pop_one(void)
+{
+    QMPRequest *req_obj = NULL;
+    Monitor *mon;
+
+    qemu_mutex_lock(&monitor_lock);
+
+    QTAILQ_FOREACH(mon, &mon_list, entry) {
+        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+        req_obj = g_queue_pop_head(mon->qmp.qmp_requests);
+        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+        if (req_obj) {
+            break;
+        }
+    }
+
+    if (req_obj) {
+        /*
+         * We found one request on the monitor. Degrade this monitor's
+         * priority to lowest by re-inserting it to end of queue.
+         */
+        QTAILQ_REMOVE(&mon_list, mon, entry);
+        QTAILQ_INSERT_TAIL(&mon_list, mon, entry);
+    }
+
+    qemu_mutex_unlock(&monitor_lock);
+
+    return req_obj;
+}
+
+static void monitor_qmp_bh_dispatcher(void *data)
+{
+    QMPRequest *req_obj = monitor_qmp_requests_pop_one();
+
+    if (req_obj) {
+        monitor_qmp_dispatch_one(req_obj);
+        /* Reschedule instead of looping so the main loop stays responsive */
+        qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
+    }
+}
+
+static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
+{
+    QObject *req, *id = NULL;
+    QDict *qdict = NULL;
+    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
+    Monitor *mon = container_of(mon_qmp, Monitor, qmp);
+    Error *err = NULL;
+    QMPRequest *req_obj;
+
+    req = json_parser_parse_err(tokens, NULL, &err);
+    if (!req && !err) {
+        /* json_parser_parse_err() sucks: can fail without setting @err */
+        error_setg(&err, QERR_JSON_PARSING);
+    }
+    if (err) {
+        monitor_qmp_respond(mon, NULL, err, NULL);
+        qobject_decref(req);
+        return;
+    }
+
+    qdict = qobject_to_qdict(req);
+    if (qdict) {
+        id = qdict_get(qdict, "id");
+        qobject_incref(id);
+        qdict_del(qdict, "id");
+    } /* else will fail qmp_dispatch() */
+
+    req_obj = g_new0(QMPRequest, 1);
+    req_obj->mon = mon;
+    req_obj->id = id;
+    req_obj->req = req;
+    req_obj->need_resume = false;
+
+    /*
+     * If OOB is not enabled on current monitor, we'll emulate the old
+     * behavior that we won't process current monitor any more until
+     * it is responded.  This helps make sure that as long as OOB is
+     * not enabled, the server will never drop any command.
+     */
+    if (!qmp_oob_enabled(mon)) {
+        monitor_suspend(mon);
+        req_obj->need_resume = true;
+    }
+
+    /*
+     * Put the request to the end of queue so that requests will be
+     * handled in time order.  Ownership for req_obj, req, id,
+     * etc. will be delivered to the handler side.
+     */
+    qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+    g_queue_push_tail(mon->qmp.qmp_requests, req_obj);
+    qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+
+    /* Kick the dispatcher routine */
+    qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
+}
+
 static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
 {
     Monitor *mon = opaque;
@@ -4134,6 +4276,15 @@ static void monitor_iothread_init(void)
 {
     mon_global.mon_iothread = iothread_create("mon_iothread",
                                               &error_abort);
+
+    /*
+     * This MUST be on main loop thread since we have commands that
+     * have assumption to be run on main loop thread.  It would be
+     * nice that one day we can remove this assumption in the future.
+     */
+    mon_global.qmp_dispatcher_bh = aio_bh_new(qemu_get_aio_context(),
+                                              monitor_qmp_bh_dispatcher,
+                                              NULL);
 }
 
 void monitor_init_globals(void)
@@ -4280,6 +4431,10 @@ void monitor_cleanup(void)
     }
     qemu_mutex_unlock(&monitor_lock);
 
+    /* QEMUBHs needs to be deleted before destroying the IOThread. */
+    qemu_bh_delete(mon_global.qmp_dispatcher_bh);
+    mon_global.qmp_dispatcher_bh = NULL;
+
     iothread_destroy(mon_global.mon_iothread);
     mon_global.mon_iothread = NULL;
 }
-- 
2.14.3

[Qemu-devel] [PATCH v8 15/23] qmp: add new event "command-dropped"

[Peter Xu]

This event will be emitted if one QMP command is dropped.  Along,
declare an enum for the reasons.

Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qapi/misc.json | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/qapi/misc.json b/qapi/misc.json
index 0123143d4b..8c7e736681 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -3227,3 +3227,40 @@
 # Since: 2.9
 ##
 { 'command': 'query-vm-generation-id', 'returns': 'GuidInfo' }
+
+##
+# @CommandDropReason:
+#
+# Reasons that caused one command to be dropped.
+#
+# @queue-full: the command queue is full. This can only occur when
+#              the client sends a new non-oob command before the
+#              response to the previous non-oob command has been
+#              received.
+#
+# Since: 2.12
+##
+{ 'enum': 'CommandDropReason',
+  'data': [ 'queue-full' ] }
+
+##
+# @COMMAND_DROPPED:
+#
+# Emitted when a command is dropped due to some reason.  Commands can
+# only be dropped when the oob capability is enabled.
+#
+# @id: The dropped command's "id" field.
+#
+# @reason: The reason why the command is dropped.
+#
+# Since: 2.12
+#
+# Example:
+#
+# { "event": "COMMAND_DROPPED",
+#   "data": {"result": {"id": "libvirt-102",
+#                       "reason": "queue-full" } } }
+#
+##
+{ 'event': 'COMMAND_DROPPED' ,
+  'data': { 'id': 'any', 'reason': 'CommandDropReason' } }
-- 
2.14.3

[Qemu-devel] [PATCH v8 16/23] monitor: send event when command queue full

[Peter Xu]

Set maximum QMP command queue length to 8.  If queue full, instead of
queue the command, we directly return a "command-dropped" event, telling
client that specific command is dropped.

Note that this flow control mechanism is only valid if OOB is enabled.
If it's not, the effective queue length will always be 1, which strictly
follows original behavior of QMP command handling (which never drop
messages).

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 18 +++++++++++++++++-
 1 file changed, 17 insertions(+), 1 deletion(-)

diff --git a/monitor.c b/monitor.c
index 5104e5db07..4d57a8d341 100644
--- a/monitor.c
+++ b/monitor.c
@@ -4007,6 +4007,8 @@ static void monitor_qmp_bh_dispatcher(void *data)
     }
 }
 
+#define  QMP_REQ_QUEUE_LEN_MAX  (8)
+
 static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
 {
     QObject *req, *id = NULL;
@@ -4040,6 +4042,9 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
     req_obj->req = req;
     req_obj->need_resume = false;
 
+    /* Protect qmp_requests and fetching its length. */
+    qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+
     /*
      * If OOB is not enabled on current monitor, we'll emulate the old
      * behavior that we won't process current monitor any more until
@@ -4049,6 +4054,18 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
     if (!qmp_oob_enabled(mon)) {
         monitor_suspend(mon);
         req_obj->need_resume = true;
+    } else {
+        /* Drop the request if queue is full. */
+        if (mon->qmp.qmp_requests->length >= QMP_REQ_QUEUE_LEN_MAX) {
+            qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+            qapi_event_send_command_dropped(id,
+                                            COMMAND_DROP_REASON_QUEUE_FULL,
+                                            NULL);
+            qobject_decref(id);
+            qobject_decref(req);
+            g_free(req_obj);
+            return;
+        }
     }
 
     /*
@@ -4056,7 +4073,6 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
      * handled in time order.  Ownership for req_obj, req, id,
      * etc. will be delivered to the handler side.
      */
-    qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
     g_queue_push_tail(mon->qmp.qmp_requests, req_obj);
     qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
 
-- 
2.14.3

[Qemu-devel] [PATCH v8 17/23] qapi: introduce new cmd option "allow-oob"

[Peter Xu]

Here "oob" stands for "Out-Of-Band".  When "allow-oob" is set, it means
the command allows out-of-band execution.

The "oob" idea is proposed by Markus Armbruster in following thread:

  https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg02057.html

This new "allow-oob" boolean will be exposed by "query-qmp-schema" as
well for command entries, so that QMP clients can know which command can
be used as out-of-band calls. For example the command "migrate"
originally looks like:

  {"name": "migrate", "ret-type": "17", "meta-type": "command",
   "arg-type": "86"}

And it'll be changed into:

  {"name": "migrate", "ret-type": "17", "allow-oob": false,
   "meta-type": "command", "arg-type": "86"}

This patch only provides the QMP interface level changes.  It does not
contains the real out-of-band execution implementation yet.

Suggested-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/dispatch.h    |  5 +++--
 qapi/introspect.json           |  6 +++++-
 scripts/qapi/commands.py       | 18 +++++++++++++-----
 scripts/qapi/common.py         | 15 ++++++++++-----
 scripts/qapi/doc.py            |  2 +-
 scripts/qapi/introspect.py     | 10 ++++++++--
 tests/qapi-schema/test-qapi.py |  2 +-
 7 files changed, 41 insertions(+), 17 deletions(-)

diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
index 1e694b5ecf..26eb13ff41 100644
--- a/include/qapi/qmp/dispatch.h
+++ b/include/qapi/qmp/dispatch.h
@@ -20,8 +20,9 @@ typedef void (QmpCommandFunc)(QDict *, QObject **, Error **);
 
 typedef enum QmpCommandOptions
 {
-    QCO_NO_OPTIONS = 0x0,
-    QCO_NO_SUCCESS_RESP = 0x1,
+    QCO_NO_OPTIONS            =  0x0,
+    QCO_NO_SUCCESS_RESP       =  (1U << 0),
+    QCO_ALLOW_OOB             =  (1U << 1),
 } QmpCommandOptions;
 
 typedef struct QmpCommand
diff --git a/qapi/introspect.json b/qapi/introspect.json
index 5b3e6e9d78..c7f67b7d78 100644
--- a/qapi/introspect.json
+++ b/qapi/introspect.json
@@ -259,12 +259,16 @@
 #
 # @ret-type: the name of the command's result type.
 #
+# @allow-oob: whether the command allows out-of-band execution.
+#             (Since: 2.12)
+#
 # TODO: @success-response (currently irrelevant, because it's QGA, not QMP)
 #
 # Since: 2.5
 ##
 { 'struct': 'SchemaInfoCommand',
-  'data': { 'arg-type': 'str', 'ret-type': 'str' } }
+  'data': { 'arg-type': 'str', 'ret-type': 'str',
+            'allow-oob': 'bool' } }
 
 ##
 # @SchemaInfoEvent:
diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
index 21a7e0dbe6..0c5da3a54d 100644
--- a/scripts/qapi/commands.py
+++ b/scripts/qapi/commands.py
@@ -193,10 +193,18 @@ out:
     return ret
 
 
-def gen_register_command(name, success_response):
-    options = 'QCO_NO_OPTIONS'
+def gen_register_command(name, success_response, allow_oob):
+    options = []
+
     if not success_response:
-        options = 'QCO_NO_SUCCESS_RESP'
+        options += ['QCO_NO_SUCCESS_RESP']
+    if allow_oob:
+        options += ['QCO_ALLOW_OOB']
+
+    if not options:
+        options = ['QCO_NO_OPTIONS']
+
+    options = " | ".join(options)
 
     ret = mcgen('''
     qmp_register_command(cmds, "%(name)s",
@@ -268,7 +276,7 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
         genc.add(gen_registry(self._regy, self._prefix))
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         if not gen:
             return
         self._genh.add(gen_command_decl(name, arg_type, boxed, ret_type))
@@ -277,7 +285,7 @@ void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds);
             self._genc.add(gen_marshal_output(ret_type))
         self._genh.add(gen_marshal_decl(name))
         self._genc.add(gen_marshal(name, arg_type, boxed, ret_type))
-        self._regy += gen_register_command(name, success_response)
+        self._regy += gen_register_command(name, success_response, allow_oob)
 
 
 def gen_commands(schema, output_dir, prefix):
diff --git a/scripts/qapi/common.py b/scripts/qapi/common.py
index 97e9060b67..2c05e3c284 100644
--- a/scripts/qapi/common.py
+++ b/scripts/qapi/common.py
@@ -921,7 +921,8 @@ def check_exprs(exprs):
         elif 'command' in expr:
             meta = 'command'
             check_keys(expr_elem, 'command', [],
-                       ['data', 'returns', 'gen', 'success-response', 'boxed'])
+                       ['data', 'returns', 'gen', 'success-response',
+                        'boxed', 'allow-oob'])
         elif 'event' in expr:
             meta = 'event'
             check_keys(expr_elem, 'event', [], ['data', 'boxed'])
@@ -1044,7 +1045,7 @@ class QAPISchemaVisitor(object):
         pass
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         pass
 
     def visit_event(self, name, info, arg_type, boxed):
@@ -1421,7 +1422,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
 class QAPISchemaCommand(QAPISchemaEntity):
     def __init__(self, name, info, doc, arg_type, ret_type,
-                 gen, success_response, boxed):
+                 gen, success_response, boxed, allow_oob):
         QAPISchemaEntity.__init__(self, name, info, doc)
         assert not arg_type or isinstance(arg_type, str)
         assert not ret_type or isinstance(ret_type, str)
@@ -1432,6 +1433,7 @@ class QAPISchemaCommand(QAPISchemaEntity):
         self.gen = gen
         self.success_response = success_response
         self.boxed = boxed
+        self.allow_oob = allow_oob
 
     def check(self, schema):
         if self._arg_type_name:
@@ -1455,7 +1457,8 @@ class QAPISchemaCommand(QAPISchemaEntity):
     def visit(self, visitor):
         visitor.visit_command(self.name, self.info,
                               self.arg_type, self.ret_type,
-                              self.gen, self.success_response, self.boxed)
+                              self.gen, self.success_response,
+                              self.boxed, self.allow_oob)
 
 
 class QAPISchemaEvent(QAPISchemaEntity):
@@ -1674,6 +1677,7 @@ class QAPISchema(object):
         gen = expr.get('gen', True)
         success_response = expr.get('success-response', True)
         boxed = expr.get('boxed', False)
+        allow_oob = expr.get('allow-oob', False)
         if isinstance(data, OrderedDict):
             data = self._make_implicit_object_type(
                 name, info, doc, 'arg', self._make_members(data, info))
@@ -1681,7 +1685,8 @@ class QAPISchema(object):
             assert len(rets) == 1
             rets = self._make_array_type(rets[0], info)
         self._def_entity(QAPISchemaCommand(name, info, doc, data, rets,
-                                           gen, success_response, boxed))
+                                           gen, success_response,
+                                           boxed, allow_oob))
 
     def _def_event(self, expr, info, doc):
         name = expr['event']
diff --git a/scripts/qapi/doc.py b/scripts/qapi/doc.py
index 0ea68bf813..73d286f808 100644
--- a/scripts/qapi/doc.py
+++ b/scripts/qapi/doc.py
@@ -228,7 +228,7 @@ class QAPISchemaGenDocVisitor(qapi.common.QAPISchemaVisitor):
                                body=texi_entity(doc, 'Members')))
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         doc = self.cur_doc
         if boxed:
             body = texi_body(doc)
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index f66c397fb0..4d6588c5fb 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -29,6 +29,11 @@ def to_json(obj, level=0):
                               to_json(obj[key], level + 1))
                 for key in sorted(obj.keys())]
         ret = '{' + ', '.join(elts) + '}'
+    elif isinstance(obj, bool):
+        if obj:
+            ret = 'true'
+        else:
+            ret = 'false'
     else:
         assert False                # not implemented
     if level == 1:
@@ -160,12 +165,13 @@ const char %(c_name)s[] = %(c_string)s;
                                     for m in variants.variants]})
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         arg_type = arg_type or self._schema.the_empty_object_type
         ret_type = ret_type or self._schema.the_empty_object_type
         self._gen_json(name, 'command',
                        {'arg-type': self._use_type(arg_type),
-                        'ret-type': self._use_type(ret_type)})
+                        'ret-type': self._use_type(ret_type),
+                        'allow-oob': allow_oob})
 
     def visit_event(self, name, info, arg_type, boxed):
         arg_type = arg_type or self._schema.the_empty_object_type
diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index 67e417e298..10e68b01d9 100644
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -42,7 +42,7 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
         self._print_variants(variants)
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         print('command %s %s -> %s' % \
               (name, arg_type and arg_type.name, ret_type and ret_type.name))
         print('   gen=%s success_response=%s boxed=%s' % \
-- 
2.14.3

[Qemu-devel] [PATCH v8 18/23] qmp: support out-of-band (oob) execution

[Peter Xu]

Having "allow-oob" to true for a command does not mean that this command
will always be run in out-of-band mode.  The out-of-band quick path will
only be executed if we specify the extra "run-oob" flag when sending the
QMP request:

    { "execute":   "command-that-allows-oob",
      "arguments": { ... },
      "control":   { "run-oob": true } }

The "control" key is introduced to store this extra flag.  "control"
field is used to store arguments that are shared by all the commands,
rather than command specific arguments.  Let "run-oob" be the first.

Note that in the patch I exported qmp_dispatch_check_obj() to be used to
check the request earlier, and at the same time allowed "id" field to be
there since actually we always allow that.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/dispatch.h |  2 ++
 monitor.c                   | 84 ++++++++++++++++++++++++++++++++++++++++-----
 qapi/qmp-dispatch.c         | 33 +++++++++++++++++-
 trace-events                |  2 ++
 4 files changed, 111 insertions(+), 10 deletions(-)

diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
index 26eb13ff41..ffb4652f71 100644
--- a/include/qapi/qmp/dispatch.h
+++ b/include/qapi/qmp/dispatch.h
@@ -48,6 +48,8 @@ bool qmp_command_is_enabled(const QmpCommand *cmd);
 const char *qmp_command_name(const QmpCommand *cmd);
 bool qmp_has_success_response(const QmpCommand *cmd);
 QObject *qmp_build_error_object(Error *err);
+QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp);
+bool qmp_is_oob(QDict *dict);
 
 typedef void (*qmp_cmd_callback_fn)(QmpCommand *cmd, void *opaque);
 
diff --git a/monitor.c b/monitor.c
index 4d57a8d341..5c8afe9f50 100644
--- a/monitor.c
+++ b/monitor.c
@@ -1110,6 +1110,44 @@ static void qmp_caps_apply(Monitor *mon, QMPCapabilityList *list)
     }
 }
 
+/*
+ * Return true if check successful, or false otherwise.  When false is
+ * returned, detailed error will be in errp if provided.
+ */
+static bool qmp_cmd_oob_check(Monitor *mon, QDict *req, Error **errp)
+{
+    const char *command;
+    QmpCommand *cmd;
+
+    command = qdict_get_try_str(req, "execute");
+    if (!command) {
+        error_setg(errp, "Command field 'execute' missing");
+        return false;
+    }
+
+    cmd = qmp_find_command(mon->qmp.commands, command);
+    if (!cmd) {
+        error_set(errp, ERROR_CLASS_COMMAND_NOT_FOUND,
+                  "The command %s has not been found", command);
+        return false;
+    }
+
+    if (qmp_is_oob(req)) {
+        if (!qmp_oob_enabled(mon)) {
+            error_setg(errp, "Please enable Out-Of-Band first "
+                       "for the session during capabilities negociation");
+            return false;
+        }
+        if (!(cmd->options & QCO_ALLOW_OOB)) {
+            error_setg(errp, "The command %s does not support OOB",
+                       command);
+            return false;
+        }
+    }
+
+    return true;
+}
+
 void qmp_qmp_capabilities(bool has_enable, QMPCapabilityList *enable,
                           Error **errp)
 {
@@ -4001,6 +4039,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
     QMPRequest *req_obj = monitor_qmp_requests_pop_one();
 
     if (req_obj) {
+        trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
         monitor_qmp_dispatch_one(req_obj);
         /* Reschedule instead of looping so the main loop stays responsive */
         qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
@@ -4024,17 +4063,31 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
         error_setg(&err, QERR_JSON_PARSING);
     }
     if (err) {
-        monitor_qmp_respond(mon, NULL, err, NULL);
-        qobject_decref(req);
-        return;
+        goto err;
+    }
+
+    /* Check against the request in general layout */
+    qdict = qmp_dispatch_check_obj(req, &err);
+    if (!qdict) {
+        goto err;
     }
 
-    qdict = qobject_to_qdict(req);
-    if (qdict) {
-        id = qdict_get(qdict, "id");
-        qobject_incref(id);
-        qdict_del(qdict, "id");
-    } /* else will fail qmp_dispatch() */
+    /* Check against OOB specific */
+    if (!qmp_cmd_oob_check(mon, qdict, &err)) {
+        goto err;
+    }
+
+    id = qdict_get(qdict, "id");
+
+    /* When OOB is enabled, the "id" field is mandatory. */
+    if (qmp_oob_enabled(mon) && !id) {
+        error_setg(&err, "Out-Of-Band capability requires that "
+                   "every command contains an 'id' field");
+        goto err;
+    }
+
+    qobject_incref(id);
+    qdict_del(qdict, "id");
 
     req_obj = g_new0(QMPRequest, 1);
     req_obj->mon = mon;
@@ -4042,6 +4095,14 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
     req_obj->req = req;
     req_obj->need_resume = false;
 
+    if (qmp_is_oob(qdict)) {
+        /* Out-Of-Band (OOB) requests are executed directly in parser. */
+        trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
+                                          ?: "");
+        monitor_qmp_dispatch_one(req_obj);
+        return;
+    }
+
     /* Protect qmp_requests and fetching its length. */
     qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
 
@@ -4078,6 +4139,11 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
 
     /* Kick the dispatcher routine */
     qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
+    return;
+
+err:
+    monitor_qmp_respond(mon, NULL, err, NULL);
+    qobject_decref(req);
 }
 
 static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
diff --git a/qapi/qmp-dispatch.c b/qapi/qmp-dispatch.c
index e31ac4be1f..9a2e18c768 100644
--- a/qapi/qmp-dispatch.c
+++ b/qapi/qmp-dispatch.c
@@ -17,8 +17,9 @@
 #include "qapi/qmp/json-parser.h"
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qjson.h"
+#include "qapi/qmp/qbool.h"
 
-static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
+QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
 {
     const QDictEntry *ent;
     const char *arg_name;
@@ -50,6 +51,14 @@ static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
                            "QMP input member 'arguments' must be an object");
                 return NULL;
             }
+        } else if (!strcmp(arg_name, "id")) {
+            continue;
+        } else if (!strcmp(arg_name, "control")) {
+            if (qobject_type(arg_obj) != QTYPE_QDICT) {
+                error_setg(errp,
+                           "QMP input member 'control' must be a dict");
+                return NULL;
+            }
         } else {
             error_setg(errp, "QMP input member '%s' is unexpected",
                        arg_name);
@@ -120,6 +129,28 @@ QObject *qmp_build_error_object(Error *err)
                               error_get_pretty(err));
 }
 
+/*
+ * Detect whether a request should be run out-of-band, by quickly
+ * peeking at whether we have: { "control": { "run-oob": True } }. By
+ * default commands are run in-band.
+ */
+bool qmp_is_oob(QDict *dict)
+{
+    QBool *bool_obj;
+
+    dict = qdict_get_qdict(dict, "control");
+    if (!dict) {
+        return false;
+    }
+
+    bool_obj = qobject_to_qbool(qdict_get(dict, "run-oob"));
+    if (!bool_obj) {
+        return false;
+    }
+
+    return qbool_get_bool(bool_obj);
+}
+
 QObject *qmp_dispatch(QmpCommandList *cmds, QObject *request)
 {
     Error *err = NULL;
diff --git a/trace-events b/trace-events
index fe10c3b487..bd036da278 100644
--- a/trace-events
+++ b/trace-events
@@ -48,6 +48,8 @@ monitor_protocol_event_queue(uint32_t event, void *qdict, uint64_t rate) "event=
 handle_hmp_command(void *mon, const char *cmdline) "mon %p cmdline: %s"
 handle_qmp_command(void *mon, const char *req) "mon %p req: %s"
 monitor_suspend(void *ptr, int cnt) "mon %p: %d"
+monitor_qmp_cmd_in_band(const char *id) "%s"
+monitor_qmp_cmd_out_of_band(const char *id) "%s"
 
 # dma-helpers.c
 dma_blk_io(void *dbs, void *bs, int64_t offset, bool to_dev) "dbs=%p bs=%p offset=%" PRId64 " to_dev=%d"
-- 
2.14.3

[Qemu-devel] [PATCH v8 19/23] qmp: isolate responses into io thread

[Peter Xu]

For those monitors who have enabled IO thread, we'll offload the
responding procedure into IO thread.  The main reason is that chardev is
not thread safe, and we need to do all the read/write IOs in the same
thread.  For use_io_thr=true monitors, that thread is the IO thread.

We do this isolation in similar pattern as what we have done to the
request queue: we first create one response queue for each monitor, then
instead of replying directly in the main thread, we queue the responses
and kick the IO thread to do the rest of the job for us.

A funny thing after doing this is that, when the QMP clients send "quit"
to QEMU, it's possible that we close the IOThread even earlier than
replying to that "quit".  So another thing we need to do before cleaning
up the monitors is that we need to flush the response queue (we don't
need to do that for command queue; after all we are quitting) to make
sure replies for handled commands are always flushed back to clients.

Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 95 insertions(+), 1 deletion(-)

diff --git a/monitor.c b/monitor.c
index 5c8afe9f50..c6de5f123e 100644
--- a/monitor.c
+++ b/monitor.c
@@ -179,6 +179,8 @@ typedef struct {
     QemuMutex qmp_queue_lock;
     /* Input queue that holds all the parsed QMP requests */
     GQueue *qmp_requests;
+    /* Output queue contains all the QMP responses in order */
+    GQueue *qmp_responses;
 } MonitorQMP;
 
 /*
@@ -205,6 +207,7 @@ struct Monitor {
     bool skip_flush;
     bool use_io_thr;
 
+    /* We can't access guest memory when holding the lock */
     QemuMutex out_lock;
     QString *outbuf;
     guint out_watch;
@@ -227,6 +230,8 @@ static struct {
     IOThread *mon_iothread;
     /* Bottom half to dispatch the requests received from IO thread */
     QEMUBH *qmp_dispatcher_bh;
+    /* Bottom half to deliver the responses back to clients */
+    QEMUBH *qmp_respond_bh;
 } mon_global;
 
 /* QMP checker flags */
@@ -416,7 +421,8 @@ int monitor_fprintf(FILE *stream, const char *fmt, ...)
     return 0;
 }
 
-static void monitor_json_emitter(Monitor *mon, const QObject *data)
+static void monitor_json_emitter_raw(Monitor *mon,
+                                     QObject *data)
 {
     QString *json;
 
@@ -430,6 +436,71 @@ static void monitor_json_emitter(Monitor *mon, const QObject *data)
     QDECREF(json);
 }
 
+static void monitor_json_emitter(Monitor *mon, QObject *data)
+{
+    if (mon->use_io_thr) {
+        /*
+         * If using IO thread, we need to queue the item so that IO
+         * thread will do the rest for us.  Take refcount so that
+         * caller won't free the data (which will be finally freed in
+         * responder thread).
+         */
+        qobject_incref(data);
+        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+        g_queue_push_tail(mon->qmp.qmp_responses, (void *)data);
+        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+        qemu_bh_schedule(mon_global.qmp_respond_bh);
+    } else {
+        /*
+         * If not using monitor IO thread, then we are in main thread.
+         * Do the emission right away.
+         */
+        monitor_json_emitter_raw(mon, data);
+    }
+}
+
+struct QMPResponse {
+    Monitor *mon;
+    QObject *data;
+};
+typedef struct QMPResponse QMPResponse;
+
+/*
+ * Return one QMPResponse.  The response is only valid if
+ * response.data is not NULL.
+ */
+static QMPResponse monitor_qmp_response_pop_one(void)
+{
+    Monitor *mon;
+    QObject *data = NULL;
+
+    qemu_mutex_lock(&monitor_lock);
+    QTAILQ_FOREACH(mon, &mon_list, entry) {
+        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+        data = g_queue_pop_head(mon->qmp.qmp_responses);
+        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+        if (data) {
+            break;
+        }
+    }
+    qemu_mutex_unlock(&monitor_lock);
+    return (QMPResponse) { .mon = mon, .data = data };
+}
+
+static void monitor_qmp_bh_responder(void *opaque)
+{
+    QMPResponse response;
+
+    while (true) {
+        response = monitor_qmp_response_pop_one();
+        if (!response.data) {
+            break;
+        }
+        monitor_json_emitter_raw(response.mon, response.data);
+        qobject_decref(response.data);
+    }
+}
+
 static MonitorQAPIEventConf monitor_qapi_event_conf[QAPI_EVENT__MAX] = {
     /* Limit guest-triggerable events to 1 per second */
     [QAPI_EVENT_RTC_CHANGE]        = { 1000 * SCALE_MS },
@@ -616,6 +687,7 @@ static void monitor_data_init(Monitor *mon, bool skip_flush,
     mon->skip_flush = skip_flush;
     mon->use_io_thr = use_io_thr;
     mon->qmp.qmp_requests = g_queue_new();
+    mon->qmp.qmp_responses = g_queue_new();
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -630,6 +702,7 @@ static void monitor_data_destroy(Monitor *mon)
     qemu_mutex_destroy(&mon->out_lock);
     qemu_mutex_destroy(&mon->qmp.qmp_queue_lock);
     g_queue_free(mon->qmp.qmp_requests);
+    g_queue_free(mon->qmp.qmp_responses);
 }
 
 char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
@@ -4367,6 +4440,15 @@ static void monitor_iothread_init(void)
     mon_global.qmp_dispatcher_bh = aio_bh_new(qemu_get_aio_context(),
                                               monitor_qmp_bh_dispatcher,
                                               NULL);
+
+    /*
+     * Unlike the dispatcher BH, this must be run on the monitor IO
+     * thread, so that monitors that are using IO thread will make
+     * sure read/write operations are all done on the IO thread.
+     */
+    mon_global.qmp_respond_bh = aio_bh_new(monitor_get_aio_context(),
+                                           monitor_qmp_bh_responder,
+                                           NULL);
 }
 
 void monitor_init_globals(void)
@@ -4505,9 +4587,19 @@ void monitor_cleanup(void)
      */
     iothread_stop(mon_global.mon_iothread);
 
+    /*
+     * After we have IOThread to send responses, it's possible that
+     * when we stop the IOThread there are still replies queued in the
+     * responder queue.  Flush all of them.  Note that even after this
+     * flush it's still possible that out buffer is not flushed.
+     * It'll be done in below monitor_flush() as the last resort.
+     */
+    monitor_qmp_bh_responder(NULL);
+
     qemu_mutex_lock(&monitor_lock);
     QTAILQ_FOREACH_SAFE(mon, &mon_list, entry, next) {
         QTAILQ_REMOVE(&mon_list, mon, entry);
+        monitor_flush(mon);
         monitor_data_destroy(mon);
         g_free(mon);
     }
@@ -4516,6 +4608,8 @@ void monitor_cleanup(void)
     /* QEMUBHs needs to be deleted before destroying the IOThread. */
     qemu_bh_delete(mon_global.qmp_dispatcher_bh);
     mon_global.qmp_dispatcher_bh = NULL;
+    qemu_bh_delete(mon_global.qmp_respond_bh);
+    mon_global.qmp_respond_bh = NULL;
 
     iothread_destroy(mon_global.mon_iothread);
     mon_global.mon_iothread = NULL;
-- 
2.14.3

[Qemu-devel] [PATCH v8 20/23] monitor: enable IO thread for (qmp & !mux) typed

[Peter Xu]

Start to use dedicate IO thread for QMP monitors that are not using
MUXed chardev.

Reviewed-by: Fam Zheng <famz@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/monitor.c b/monitor.c
index c6de5f123e..9e8ee2ed14 100644
--- a/monitor.c
+++ b/monitor.c
@@ -36,6 +36,7 @@
 #include "net/slirp.h"
 #include "chardev/char-fe.h"
 #include "chardev/char-io.h"
+#include "chardev/char-mux.h"
 #include "ui/qemu-spice.h"
 #include "sysemu/numa.h"
 #include "monitor/monitor.h"
@@ -4533,8 +4534,10 @@ static void monitor_qmp_setup_handlers_bh(void *opaque)
 void monitor_init(Chardev *chr, int flags)
 {
     Monitor *mon = g_malloc(sizeof(*mon));
+    /* Enable IOThread for QMPs that are not using MUX chardev backends. */
+    bool use_io_thr = (!CHARDEV_IS_MUX(chr)) && (flags & MONITOR_USE_CONTROL);
 
-    monitor_data_init(mon, false, false);
+    monitor_data_init(mon, false, use_io_thr);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
-- 
2.14.3

[Qemu-devel] [PATCH v8 21/23] qmp: add command "x-oob-test"

[Peter Xu]

This command is only used to test OOB functionality.  It should not be
used for any other purposes.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qapi/misc.json | 18 ++++++++++++++++++
 qmp.c          | 16 ++++++++++++++++
 2 files changed, 34 insertions(+)

diff --git a/qapi/misc.json b/qapi/misc.json
index 8c7e736681..07932bdacf 100644
--- a/qapi/misc.json
+++ b/qapi/misc.json
@@ -3264,3 +3264,21 @@
 ##
 { 'event': 'COMMAND_DROPPED' ,
   'data': { 'id': 'any', 'reason': 'CommandDropReason' } }
+
+##
+# @x-oob-test:
+#
+# Test OOB functionality.  When send this command with lock=true,
+# it'll try to hang the dispatcher.  When send it with lock=false,
+# it'll try to notify the locked thread to continue.  Note: it should
+# only be used by QMP test program rather than anything else.
+#
+# Since: 2.12
+#
+# Example:
+#
+# { "execute": "x-oob-test",
+#   "arguments": { "lock": true } }
+##
+{ 'command': 'x-oob-test', 'data' : { 'lock': 'bool' },
+  'allow-oob': true }
diff --git a/qmp.c b/qmp.c
index 8c7d1cc479..d95d132448 100644
--- a/qmp.c
+++ b/qmp.c
@@ -770,3 +770,19 @@ MemoryInfo *qmp_query_memory_size_summary(Error **errp)
 
     return mem_info;
 }
+
+static QemuSemaphore x_oob_test_sem;
+
+static void __attribute__((constructor)) x_oob_test_init(void)
+{
+    qemu_sem_init(&x_oob_test_sem, 0);
+}
+
+void qmp_x_oob_test(bool lock, Error **errp)
+{
+    if (lock) {
+        qemu_sem_wait(&x_oob_test_sem);
+    } else {
+        qemu_sem_post(&x_oob_test_sem);
+    }
+}
-- 
2.14.3

[Qemu-devel] [PATCH v8 22/23] tests: qmp-test: verify command batching

[Peter Xu]

OOB introduced DROP event for flow control.  This should not affect old
QMP clients.  Add a command batching check to make sure of it.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 tests/qmp-test.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index 70d1152d32..caf3ec4a78 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -82,6 +82,7 @@ static void test_qmp_protocol(void)
     QTestState *qts;
     const QListEntry *entry;
     QString *qstr;
+    int i;
 
     qts = qtest_init_without_qmp_handshake(common_args);
 
@@ -139,6 +140,27 @@ static void test_qmp_protocol(void)
     g_assert_cmpint(qdict_get_int(resp, "id"), ==, 2);
     QDECREF(resp);
 
+    /*
+     * Test command batching.  In current test OOB is not enabled, we
+     * should be able to run as many commands in batch as we like.
+     * Using 16 (>8, which is OOB queue length) to make sure OOB won't
+     * break existing clients.  Note: this test does not control the
+     * scheduling of QEMU's QMP command processing threads so it may
+     * not really trigger batching inside QEMU.  This is just a
+     * best-effort test.
+     */
+    for (i = 0; i < 16; i++) {
+        qtest_async_qmp(qts, "{ 'execute': 'query-version' }");
+    }
+    /* Verify the replies to make sure no command is dropped. */
+    for (i = 0; i < 16; i++) {
+        resp = qtest_qmp_receive(qts);
+        /* It should never be dropped.  Each of them should be a reply. */
+        g_assert(qdict_haskey(resp, "return"));
+        g_assert(!qdict_haskey(resp, "event"));
+        QDECREF(resp);
+    }
+
     qtest_quit(qts);
 }
 
-- 
2.14.3

[Qemu-devel] [PATCH v8 23/23] tests: qmp-test: add oob test

[Peter Xu]

Test the new OOB capability.  Here we used the new "x-oob-test" command.
Firstly, we send a lock=true and oob=false command to hang the main
thread.  Then send another lock=false and oob=true command (which will
be run inside parser this time) to free that hanged command.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 tests/qmp-test.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 65 insertions(+)

diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index caf3ec4a78..4c591cbc80 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -164,6 +164,70 @@ static void test_qmp_protocol(void)
     qtest_quit(qts);
 }
 
+/* Tests for Out-Of-Band support. */
+static void test_qmp_oob(void)
+{
+    QDict *resp;
+    int acks = 0;
+    const char *cmd_id;
+
+    global_qtest = qtest_init_without_qmp_handshake(common_args);
+
+    /* Ignore the greeting message. */
+    resp = qmp_receive();
+    g_assert(qdict_get_qdict(resp, "QMP"));
+    QDECREF(resp);
+
+    /* Try a fake capability, it should fail. */
+    resp = qmp("{ 'execute': 'qmp_capabilities', "
+               "  'arguments': { 'enable': [ 'cap-does-not-exist' ] } }");
+    g_assert(qdict_haskey(resp, "error"));
+    QDECREF(resp);
+
+    /* Now, enable OOB in current QMP session, it should success. */
+    resp = qmp("{ 'execute': 'qmp_capabilities', "
+               "  'arguments': { 'enable': [ 'oob' ] } }");
+    g_assert(qdict_haskey(resp, "return"));
+    QDECREF(resp);
+
+    /*
+     * Try any command that does not support OOB but with OOB flag. We
+     * should get failure.
+     */
+    resp = qmp("{ 'execute': 'query-cpus',"
+               "  'control': { 'run-oob': true } }");
+    g_assert(qdict_haskey(resp, "error"));
+    QDECREF(resp);
+
+    /*
+     * Firstly send the "x-oob-test" command with lock=true and
+     * oob=false, it should hang the dispatcher and main thread;
+     * later, we send another lock=false with oob=true to continue
+     * that thread processing.  Finally we should receive replies from
+     * both commands.
+     */
+    qmp_async("{ 'execute': 'x-oob-test',"
+              "  'arguments': { 'lock': true }, "
+              "  'id': 'lock-cmd'}");
+    qmp_async("{ 'execute': 'x-oob-test', "
+              "  'arguments': { 'lock': false }, "
+              "  'control': { 'run-oob': true }, "
+              "  'id': 'unlock-cmd' }");
+
+    /* Ignore all events.  Wait for 2 acks */
+    while (acks < 2) {
+        resp = qmp_receive();
+        cmd_id = qdict_get_str(resp, "id");
+        if (!g_strcmp0(cmd_id, "lock-cmd") ||
+            !g_strcmp0(cmd_id, "unlock-cmd")) {
+            acks++;
+        }
+        QDECREF(resp);
+    }
+
+    qtest_end();
+}
+
 static int query_error_class(const char *cmd)
 {
     static struct {
@@ -343,6 +407,7 @@ int main(int argc, char *argv[])
     g_test_init(&argc, &argv, NULL);
 
     qtest_add_func("qmp/protocol", test_qmp_protocol);
+    qtest_add_func("qmp/oob", test_qmp_oob);
     qmp_schema_init(&schema);
     add_query_tests(&schema);
 
-- 
2.14.3

Re: [Qemu-devel] [PATCH v8 01/23] docs: update QMP documents for OOB commands

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> Update both the developer and spec for the new QMP OOB (Out-Of-Band)
> command.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   docs/devel/qapi-code-gen.txt | 65 ++++++++++++++++++++++++++++++++++++++++----
>   docs/interop/qmp-spec.txt    | 30 +++++++++++++++++---
>   2 files changed, 86 insertions(+), 9 deletions(-)

If all goes well, I'm planning on taking this series through my QAPI 
tree, with a pull request either late today or early Monday in order to 
make the soft freeze deadline.  We'll see what my review turns up, but 
hopefully at this point it's either clean or minor enough tweaks that I 
can polish it without a v9.

> +
> + { 'command': 'migrate_recover',
> +   'data': { 'uri': 'str' }, 'allow-oob': true }
> +
> +To execute a command in Out-Of-Band way, we need to specify the

Either 'execute a command in an Out-Of-Band way,' or 'execute a command 
in Out-Of-Band order,'.

> +"control" field in the request, with "run-oob" set to true. Example:
> +
> + => { "execute": "command-support-oob",
> +      "arguments": { ... },
> +      "control": { "run-oob": true } }
> + <= { "return": { } }
> +
> +Without it, even the commands that support out-of-band execution will
> +still be run In-Band.
> +
> +--- About Out-Of-Band (OOB) Command Execution ---
> +
> +Out-Of-Band does not mean a special kind of command. Instead, it's a
> +special way to execute the command.  One normal command can be

Perhaps s/One normal command can be/A normal command is/

> +declared to support Out-Of-Band execution when 'allow-oob' field is

s/when/when the/

> +set to true when defining the command.  With that, it can be run in an

s/when defining the command/in the command definition/

> +Out-Of-Band way if 'run-oob' is specified in 'control' field of
> +command request.
> +
> +Under normal QMP command execution, the following apply to each
> +command:
> +
> +- They are executed in order,
> +- They run only in main thread of QEMU,
> +- They have the BQL taken during execution.
> +
> +When a command is executed with OOB, the following changes occur:
> +
> +- They can be completed before a pending in-band command,
> +- They run in a monitor dedicated thread,

s/monitor dedicated/dedicated monitor/

> +- They do not take the BQL during execution.
> +
> +OOB command handlers must satisfy the following conditions:
> +
> +- It executes extremely fast,
> +- It does not take any lock, or, it can take very small locks if all
> +  critical regions also follow the rules for OOB command handler code,
> +- It does not invoke system calls that may block,
> +- It does not access guest RAM that may block when userfaultfd is
> +  enabled for postcopy live migration.
> +
> +If in doubt, do not implement OOB execution support.
>   
>   === Events ===
>   
> @@ -739,10 +792,12 @@ references by name.
>   QAPI schema definitions not reachable that way are omitted.
>   
>   The SchemaInfo for a command has meta-type "command", and variant
> -members "arg-type" and "ret-type".  On the wire, the "arguments"
> -member of a client's "execute" command must conform to the object type
> -named by "arg-type".  The "return" member that the server passes in a
> -success response conforms to the type named by "ret-type".
> +members "arg-type", "ret-type" and "allow-oob".  On the wire, the
> +"arguments" member of a client's "execute" command must conform to the
> +object type named by "arg-type".  The "return" member that the server
> +passes in a success response conforms to the type named by
> +"ret-type".  When "allow-oob" is set, it means the command supports
> +out-of-band execution.

We're inconsistent on whether it is capitalized 'Out-Of-Band'.  I'm 
probably okay with the first mention being capitalized (next to the 
first use of the OOB acronym), and all subsequent uses being lower case.

>   
>   If the command takes no arguments, "arg-type" names an object type
>   without members.  Likewise, if the command returns nothing, "ret-type"
> diff --git a/docs/interop/qmp-spec.txt b/docs/interop/qmp-spec.txt
> index f8b5356015..9a208589f6 100644
> --- a/docs/interop/qmp-spec.txt
> +++ b/docs/interop/qmp-spec.txt
> @@ -83,16 +83,27 @@ The greeting message format is:
>   2.2.1 Capabilities
>   ------------------
>   
> -As of the date this document was last revised, no server or client
> -capability strings have been defined.
> +Currently supported capabilities are:
>   
> +- "oob": it means the QMP server supports "Out-Of-Band" command

s/it means //

> +  execution.  For more details, please see "run-oob" parameter in

s/see/see the/

> +  "Issuing Commands" section below.  Not all commands allow this "oob"
> +  execution.  The "query-qmp-schema" command can be used to inspect
> +  which commands support "oob" execution.
> +
> +QMP clients can get a list of supported QMP capabilities of the QMP
> +server in the greeting message mentioned above.  By default, all the
> +capabilities are off.  To enable any QMP capabilities, the QMP client
> +needs to send the "qmp_capabilities" command with an extra parameter
> +for the requested capabilities.
>   
>   2.3 Issuing Commands
>   --------------------
>   
>   The format for command execution is:
>   
> -{ "execute": json-string, "arguments": json-object, "id": json-value }
> +{ "execute": json-string, "arguments": json-object, "id": json-value,
> +  "control": json-dict }

s/json-dict/json-object/

(The JSON term is Object, even though dictionary is a common term for 
the typical data representation of an object)

>   
>    Where,
>   
> @@ -102,10 +113,16 @@ The format for command execution is:
>     required. Each command documents what contents will be considered
>     valid when handling the json-argument
>   - The "id" member is a transaction identification associated with the
> -  command execution, it is optional and will be part of the response if
> +  command execution.  It is required if OOB is enabled, and optional
> +  if not.  The same "id" field will be part of the response if

Ambiguous on whether this is the per-command 'run-oob', or whether this 
is the generic QMP capability requested during qmp_capabilities.  I 
think the intent is that if you enabled the QMP capability up front, ALL 
commands must have an "id", even if they are run in-band without 
'run-oob', because the 'id' in the response is what will distinguish 
whether a later OOB request overtook a pending in-band reply.

>     provided. The "id" member can be any json-value, although most
>     clients merely use a json-number incremented for each successive
>     command
> +- The "control" member is optional, and currently only used for
> +  "out-of-band" execution ("oob" as shortcut). The handling or

A bit late to be introducing "oob" as shortcut, given that you've 
already used the abbreviation oob earlier in the document.

> +  response of an "oob" command can overtake prior in-band commands.
> +  To enable "oob" handling of a particular command, just provide a
> +  control field with: { "control": { "run-oob": true } }
>   
>   2.4 Commands Responses
>   ----------------------
> @@ -113,6 +130,11 @@ The format for command execution is:
>   There are two possible responses which the Server will issue as the result
>   of a command execution: success or error.
>   
> +As long as the commands were issued with a proper "id" field, then the
> +same "id" field will be attached in the corresponding response message
> +so that requests and responses can match.  Clients should drop all the
> +responses that are with unknown "id" field.

s/are with/have an/

I've got quite a few wording tweaks, but the general concept is good. 
If you need to respin for other reasons, feel free to make those 
improvements, but I also don't mind making tweaks myself as part of 
queuing for a pull request, so:

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 03/23] qobject: introduce qobject_get_try_str()

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> A quick way to fetch string from qobject when it's a QString.
> 
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   include/qapi/qmp/qstring.h |  1 +
>   qobject/qstring.c          | 11 +++++++++++
>   2 files changed, 12 insertions(+)
> 

> +++ b/qobject/qstring.c
> @@ -137,6 +137,17 @@ const char *qstring_get_try_str(const QString *qstring)
>       return qstring ? qstring_get_str(qstring) : NULL;
>   }
>   
> +/**
> + * qobject_get_try_str(): Return a pointer to the corresponding string
> + *
> + * NOTE: the string will only be returned if the object is valid, and
> + * its type is QString, otherwise NULL is returned.
> + */
> +const char *qobject_get_try_str(const QObject *qstring)
> +{
> +    return qstring_get_try_str(qobject_to_qstring(qstring));

Conflicts with Max's refactoring to a qobject_to() macro.
https://lists.gnu.org/archive/html/qemu-devel/2018-02/msg06230.html

Whoever lands first gets to watch the other (or the maintainer) rebase ;)

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 06/23] monitor: move the cur_mon hack deeper for QMP

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> In monitor_qmp_read(), we have the hack to temporarily replace the
> cur_mon pointer.  Now we move this hack deeper inside the QMP dispatcher
> routine since the Monitor pointer can be actually obtained using
> container_of() upon the parser object, just like most of the other JSON
> parser users do.
> 
> This does not make much sense as a single patch.  However, this will be
> a big step for the next patch, when the QMP dispatcher routine will be
> split from the QMP parser.
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Reviewed-by: Eric Blake <eblake@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   monitor.c | 17 ++++++++++-------
>   1 file changed, 10 insertions(+), 7 deletions(-)
> 

>   
> +    cur_mon = old_mon;
> +
>       if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
>           qdict = qdict_get_qdict(qobject_to_qdict(rsp), "error");

More conflict fun with the qobject_to() series.  Sadly, 'git am -3' had 
a tough time, even though the conflict was only in the context, but I 
managed to get it all sorted out.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 09/23] monitor: allow using IO thread for parsing

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> For each Monitor, add one field "use_io_thr" to show whether it will be
> using the dedicated monitor IO thread to handle input/output.  When set,
> monitor IO parsing work will be offloaded to the dedicated monitor IO
> thread, rather than the original main loop thread.
> 
> This only works for QMP.  HMP will always be run on the main loop
> thread.
> 
> Currently we're still keeping use_io_thr off always.  Will turn it on
> later at some point.
> 
> One thing to mention is that we cannot set use_io_thr for every QMP
> monitor.  The problem is that MUXed typed chardevs may not work well
> with it now. When MUX is used, frontend of chardev can be the monitor
> plus something else.  The only thing we know would be safe to be run
> outside main thread so far is the monitor frontend. All the rest of the
> frontends should still be run in main thread only.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   monitor.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++------
>   1 file changed, 90 insertions(+), 8 deletions(-)
> 

> @@ -210,6 +213,11 @@ struct Monitor {
>       QTAILQ_ENTRY(Monitor) entry;
>   };
>   
> +/* Let's add monitor global variables to this struct. */
> +static struct {
> +    IOThread *mon_iothread;
> +} mon_global;

Anonymous struct type, but not the first time someone has done this, so 
it's okay.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 10/23] qmp: introduce QMPCapability

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> There were no QMP capabilities defined.  Define the first "oob" as
> capability to allow out-of-band messages.

Reads better as "Define the first capability, "oob", to allow..."

> 
> After this patch, we will allow QMP clients to enable QMP capabilities
> when sending the first "qmp_capabilities" command.  Originally we are
> starting QMP session with no arguments like:
> 
>    { "execute": "qmp_capabilities" }
> 
> Now we can enable some QMP capabilities using (take OOB as example,
> which is the only one capability that we support):
> 
>    { "execute": "qmp_capabilities",
>      "argument": { "enable": [ "oob" ] } }

s/argument/arguments/

> 
> When the "argument" key is not provided, no capability is enabled.

again.  Also when "arguments":{} is provided with an empty dict, or even 
with "arguments":{"enable":[]} an empty enable array.  But no need to 
mention them here.

> 
> For capability "oob", the monitor needs to be run on dedicated IO

s/on/on a/

> thread, otherwise the command will fail.  For example, trying to enable
> OOB on a MUXed typed QMP monitor will fail.
> 
> One thing to mention is that, QMP capabilities are per-monitor, and also

s/that,/that/

> when the connection is closed due to some reason, the capabilities will
> be reset.
> 
> Also, touch up qmp-test.c to test the new bits.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   monitor.c        | 77 ++++++++++++++++++++++++++++++++++++++++++++++++++++----
>   qapi/misc.json   | 32 ++++++++++++++++++++---
>   tests/qmp-test.c | 10 +++++++-
>   3 files changed, 110 insertions(+), 9 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 11/23] monitor: introduce monitor_qmp_respond()

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> A tiny refactoring, preparing to split the QMP dispatcher away.
> 
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   monitor.c | 50 +++++++++++++++++++++++++++++++++-----------------
>   1 file changed, 33 insertions(+), 17 deletions(-)
> 

> +    if (rsp) {
> +        if (id) {
> +            /* This is for the qdict below. */
> +            qobject_incref(id);
> +            qdict_put_obj(qobject_to_qdict(rsp), "id", id);

Another victim of qobject_to() refactoring.

Fixing that is minor, so
Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 13/23] monitor: let suspend/resume work even with QMPs

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> This patches allows QMP monitors to be suspended/resumed.
> 
> One thing to mention is that for QMPs that are using IOThreads, we need
> an explicit kick for the IOThread in case it is sleeping.
> 
> Meanwhile, we need to take special care on non-interactive HMPs.
> Currently only gdbserver is using that.  For these monitors, we still
> don't allow suspend/resume operations.
> 
> Since at it, add traces for the operations.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   monitor.c    | 47 ++++++++++++++++++++++++++++++++++++++++++++---
>   trace-events |  1 +
>   2 files changed, 45 insertions(+), 3 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> Originally QMP goes through these steps:
> 
>    JSON Parser --> QMP Dispatcher --> Respond
>        /|\    (2)                (3)     |
>     (1) |                               \|/ (4)
>         +---------  main thread  --------+
> 
> This patch does this:
> 
>    JSON Parser     QMP Dispatcher --> Respond
>        /|\ |           /|\       (4)     |
>         |  | (2)        | (3)            |  (5)
>     (1) |  +----->      |               \|/
>         +---------  main thread  <-------+
> 
> So the parsing job and the dispatching job is isolated now.  It gives us
> a chance in following up patches to totally move the parser outside.

s/following/follow/

> 
> The isolation is done using one QEMUBH. Only one dispatcher QEMUBH is
> used for all the monitors.
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   monitor.c | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-------
>   1 file changed, 178 insertions(+), 23 deletions(-)
> 

>   
>       if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
>           QString *req_json = qobject_to_json(req);

More context conflicts.


> +/*
> + * Pop one QMP request from monitor queues, return NULL if not found.
> + * We are using round-robin fashion to pop the request, to avoid
> + * processing command only on a very busy monitor.  To achieve that,

s/command/commands/

> + * when we processed one request on specific monitor, we put that

s/processed/process/
s/on/on a/

> + * monitor to the end of mon_list queue.
> + */
> +static QMPRequest *monitor_qmp_requests_pop_one(void)
> +
> +    /*
> +     * If OOB is not enabled on current monitor, we'll emulate the old
> +     * behavior that we won't process current monitor any more until

s/current/the current/g

> +     * it is responded.  This helps make sure that as long as OOB is

/is responded/has responded/

> +     * not enabled, the server will never drop any command.
> +     */

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 15/23] qmp: add new event "command-dropped"

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> This event will be emitted if one QMP command is dropped.  Along,

s/Along,/Also,/

> declare an enum for the reasons.
> 
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   qapi/misc.json | 37 +++++++++++++++++++++++++++++++++++++
>   1 file changed, 37 insertions(+)

Reviewed-by: Eric Blake <eblake@redhat.com>

> 
> diff --git a/qapi/misc.json b/qapi/misc.json
> index 0123143d4b..8c7e736681 100644
> --- a/qapi/misc.json
> +++ b/qapi/misc.json
-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 16/23] monitor: send event when command queue full

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> Set maximum QMP command queue length to 8.  If queue full, instead of

If the queue is full

> queue the command, we directly return a "command-dropped" event, telling

s/queue/queuing/

> client that specific command is dropped.

s/client that/the client that a/

> 
> Note that this flow control mechanism is only valid if OOB is enabled.
> If it's not, the effective queue length will always be 1, which strictly
> follows original behavior of QMP command handling (which never drop

s/drop/drops/

> messages).
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   monitor.c | 18 +++++++++++++++++-
>   1 file changed, 17 insertions(+), 1 deletion(-)

> 
> diff --git a/monitor.c b/monitor.c
> index 5104e5db07..4d57a8d341 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -4007,6 +4007,8 @@ static void monitor_qmp_bh_dispatcher(void *data)
>       }
>   }
>   
> +#define  QMP_REQ_QUEUE_LEN_MAX  (8)

Parens are redundant on a single int, but it doesn't hurt to leave them 
in a macro definition.

> @@ -4049,6 +4054,18 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>       if (!qmp_oob_enabled(mon)) {
>           monitor_suspend(mon);
>           req_obj->need_resume = true;
> +    } else {
> +        /* Drop the request if queue is full. */
> +        if (mon->qmp.qmp_requests->length >= QMP_REQ_QUEUE_LEN_MAX) {
> +            qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> +            qapi_event_send_command_dropped(id,
> +                                            COMMAND_DROP_REASON_QUEUE_FULL,
> +                                            NULL);

Most qapi_event_send_* callers pass &error_abort (migration/ram.c being 
the only exception I could find); why not this one?  I'm changing it for 
consistency; we can revert back to NULL if testing finds a problem with it.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 17/23] qapi: introduce new cmd option "allow-oob"

[Eric Blake]

On 03/09/2018 03:00 AM, Peter Xu wrote:
> Here "oob" stands for "Out-Of-Band".  When "allow-oob" is set, it means
> the command allows out-of-band execution.
> 
> The "oob" idea is proposed by Markus Armbruster in following thread:
> 
>    https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg02057.html
> 
> This new "allow-oob" boolean will be exposed by "query-qmp-schema" as
> well for command entries, so that QMP clients can know which command can

s/command can/commands/can/

> be used as out-of-band calls. For example the command "migrate"
> originally looks like:
> 
>    {"name": "migrate", "ret-type": "17", "meta-type": "command",
>     "arg-type": "86"}
> 
> And it'll be changed into:
> 
>    {"name": "migrate", "ret-type": "17", "allow-oob": false,
>     "meta-type": "command", "arg-type": "86"}
> 
> This patch only provides the QMP interface level changes.  It does not
> contains the real out-of-band execution implementation yet.

s/contains/contain/

> 
> Suggested-by: Markus Armbruster <armbru@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  include/qapi/qmp/dispatch.h    |  5 +++--
>  qapi/introspect.json           |  6 +++++-
>  scripts/qapi/commands.py       | 18 +++++++++++++-----
>  scripts/qapi/common.py         | 15 ++++++++++-----
>  scripts/qapi/doc.py            |  2 +-
>  scripts/qapi/introspect.py     | 10 ++++++++--
>  tests/qapi-schema/test-qapi.py |  2 +-
>  7 files changed, 41 insertions(+), 17 deletions(-)

I'm a bit disappointed that tests/qapi-schema/qapi-schema-test.json 
doesn't have any "allow-oob":true commands at any point in the series. 
I consider that to be a bug, so it can be fixed by followup patch even 
after soft freeze; but won't make the lack of good testing hold up this 
series from making the release.

> +++ b/scripts/qapi/introspect.py
> @@ -29,6 +29,11 @@ def to_json(obj, level=0):
>                                 to_json(obj[key], level + 1))
>                   for key in sorted(obj.keys())]
>           ret = '{' + ', '.join(elts) + '}'
> +    elif isinstance(obj, bool):
> +        if obj:
> +            ret = 'true'
> +        else:
> +            ret = 'false'

Conflicts with Marc-Andre's work to perform introspection by QLIT_*; but 
it's easy enough to adjust:

+    elif isinstance(obj, bool):
+        ret += 'QLIT_QBOOL(%s)' % ('true' if obj else 'false')

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 18/23] qmp: support out-of-band (oob) execution

[Eric Blake]

On 03/09/2018 03:00 AM, Peter Xu wrote:
> Having "allow-oob" to true for a command does not mean that this command

s/to/:/

> will always be run in out-of-band mode.  The out-of-band quick path will
> only be executed if we specify the extra "run-oob" flag when sending the
> QMP request:
> 
>      { "execute":   "command-that-allows-oob",
>        "arguments": { ... },
>        "control":   { "run-oob": true } }
> 
> The "control" key is introduced to store this extra flag.  "control"
> field is used to store arguments that are shared by all the commands,
> rather than command specific arguments.  Let "run-oob" be the first.
> 
> Note that in the patch I exported qmp_dispatch_check_obj() to be used to
> check the request earlier, and at the same time allowed "id" field to be
> there since actually we always allow that.
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   include/qapi/qmp/dispatch.h |  2 ++
>   monitor.c                   | 84 ++++++++++++++++++++++++++++++++++++++++-----
>   qapi/qmp-dispatch.c         | 33 +++++++++++++++++-
>   trace-events                |  2 ++
>   4 files changed, 111 insertions(+), 10 deletions(-)
> 

> +
> +    if (qmp_is_oob(req)) {
> +        if (!qmp_oob_enabled(mon)) {
> +            error_setg(errp, "Please enable Out-Of-Band first "
> +                       "for the session during capabilities negociation");

s/negociation/negotiation/

> +++ b/qapi/qmp-dispatch.c
> @@ -17,8 +17,9 @@

> +/*
> + * Detect whether a request should be run out-of-band, by quickly
> + * peeking at whether we have: { "control": { "run-oob": True } }. By

s/True/true/

> + * default commands are run in-band.
> + */
> +bool qmp_is_oob(QDict *dict)
> +{
> +    QBool *bool_obj;
> +
> +    dict = qdict_get_qdict(dict, "control");
> +    if (!dict) {
> +        return false;
> +    }
> +
> +    bool_obj = qobject_to_qbool(qdict_get(dict, "run-oob"));

Another qobject_to() rebase victim.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 21/23] qmp: add command "x-oob-test"

[Eric Blake]

On 03/09/2018 03:00 AM, Peter Xu wrote:
> This command is only used to test OOB functionality.  It should not be
> used for any other purposes.
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   qapi/misc.json | 18 ++++++++++++++++++
>   qmp.c          | 16 ++++++++++++++++
>   2 files changed, 34 insertions(+)
> 
> diff --git a/qapi/misc.json b/qapi/misc.json
> index 8c7e736681..07932bdacf 100644
> --- a/qapi/misc.json
> +++ b/qapi/misc.json
> @@ -3264,3 +3264,21 @@
>   ##
>   { 'event': 'COMMAND_DROPPED' ,
>     'data': { 'id': 'any', 'reason': 'CommandDropReason' } }
> +
> +##
> +# @x-oob-test:
> +#
> +# Test OOB functionality.  When send this command with lock=true,

s/send/sending/

> +# it'll try to hang the dispatcher.  When send it with lock=false,

s/send/sending/

> +# it'll try to notify the locked thread to continue.  Note: it should
> +# only be used by QMP test program rather than anything else.
> +#
> +# Since: 2.12
> +#
> +# Example:
> +#
> +# { "execute": "x-oob-test",
> +#   "arguments": { "lock": true } }
> +##
> +{ 'command': 'x-oob-test', 'data' : { 'lock': 'bool' },
> +  'allow-oob': true }
Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 22/23] tests: qmp-test: verify command batching

[Eric Blake]

On 03/09/2018 03:00 AM, Peter Xu wrote:
> OOB introduced DROP event for flow control.  This should not affect old
> QMP clients.  Add a command batching check to make sure of it.
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   tests/qmp-test.c | 22 ++++++++++++++++++++++
>   1 file changed, 22 insertions(+)
> 

>   
> +    /*
> +     * Test command batching.  In current test OOB is not enabled, we
> +     * should be able to run as many commands in batch as we like.
> +     * Using 16 (>8, which is OOB queue length) to make sure OOB won't
> +     * break existing clients.  Note: this test does not control the
> +     * scheduling of QEMU's QMP command processing threads so it may
> +     * not really trigger batching inside QEMU.  This is just a
> +     * best-effort test.
> +     */
> +    for (i = 0; i < 16; i++) {
> +        qtest_async_qmp(qts, "{ 'execute': 'query-version' }");

Would the test be any more robust if we could generate a single string 
to send all at once, rather than multiple separate calls to 
qtest_async_qmp() (the overhead in generating separate strings means the 
monitor may have made progress before we send the next string).  But 
that can be a followup, if you want to pursue the idea.

> +    }
> +    /* Verify the replies to make sure no command is dropped. */
> +    for (i = 0; i < 16; i++) {
> +        resp = qtest_qmp_receive(qts);
> +        /* It should never be dropped.  Each of them should be a reply. */
> +        g_assert(qdict_haskey(resp, "return"));
> +        g_assert(!qdict_haskey(resp, "event"));
> +        QDECREF(resp);

Should we also be sending unique ids, and ensure that the responses 
arrive with ids in the same order?  Again, idea for a followup.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 23/23] tests: qmp-test: add oob test

[Eric Blake]

On 03/09/2018 03:00 AM, Peter Xu wrote:
> Test the new OOB capability.  Here we used the new "x-oob-test" command.
> Firstly, we send a lock=true and oob=false command to hang the main

s/Firstly/First/

> thread.  Then send another lock=false and oob=true command (which will
> be run inside parser this time) to free that hanged command.
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>   tests/qmp-test.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 65 insertions(+)
> 

> +    /* Now, enable OOB in current QMP session, it should success. */

s/success/succeed/

> +
> +    /*
> +     * Firstly send the "x-oob-test" command with lock=true and

s/Firstly/First/

> +     * oob=false, it should hang the dispatcher and main thread;
> +     * later, we send another lock=false with oob=true to continue
> +     * that thread processing.  Finally we should receive replies from
> +     * both commands.
> +     */
> +    qmp_async("{ 'execute': 'x-oob-test',"
> +              "  'arguments': { 'lock': true }, "
> +              "  'id': 'lock-cmd'}");
> +    qmp_async("{ 'execute': 'x-oob-test', "
> +              "  'arguments': { 'lock': false }, "
> +              "  'control': { 'run-oob': true }, "
> +              "  'id': 'unlock-cmd' }");
> +
> +    /* Ignore all events.  Wait for 2 acks */
> +    while (acks < 2) {
> +        resp = qmp_receive();
> +        cmd_id = qdict_get_str(resp, "id");
> +        if (!g_strcmp0(cmd_id, "lock-cmd") ||
> +            !g_strcmp0(cmd_id, "unlock-cmd")) {
> +            acks++;
> +        }
> +        QDECREF(resp);
> +    }

Can you make the reply order deterministic?  Perhaps by having the lock 
command insert a sleep after locking but before replying, so that the 
unlock always gets to reply first?  But that can be a followup.

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 00/23] QMP: out-of-band (OOB) execution support

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> Based-on: <20180306053320.15401-1-peterx@redhat.com>
> 
> The series is based on the following series:
> 
>    [PATCH v2 0/9] chardev: qio related non-default context support
>    (actually patch 9 is based on version 2.1, Dan's version)
> 
> This version only contains some trivial changes, most of the work is
> posted separately in either the QIO or CHARDEV series.
> 
> Four days until soft freeze.  Hmm...
> 
> Please review.  Thanks,
> 
> v8:
> - fix up doc patch as suggested [Eric]
> - rename functions in form of X_get_Y() or X_bh() [Stefan]
> - English fixes [Stefan]
> - make sure to kick iothread after suspend count update [Stefan]
> - remove the hack to remove fd in monitor_init(), since now we have
>    the QIO/CHARDEV series to solve the root problem.

Still a lot that I had to clean up (in part due to conflicts with other 
pending patches that also touch qapi), but given the utility of this 
feature, and the impending deadline for freeze, I've gone ahead and done 
the cleanup work.  Your patches are now stages on my QAPI tree.

git://repo.or.cz/qemu/ericb.git qapi
http://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/qapi

Note that I had some ideas for followup patches (mainly strengthening 
the testsuite, which is typically safe early on in freeze because it is 
not introducing features but strengthening the quality of the release).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 01/23] docs: update QMP documents for OOB commands

[Peter Xu]

On Fri, Mar 09, 2018 at 11:13:23AM -0600, Eric Blake wrote:
> On 03/09/2018 02:59 AM, Peter Xu wrote:
> > Update both the developer and spec for the new QMP OOB (Out-Of-Band)
> > command.
> > 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >   docs/devel/qapi-code-gen.txt | 65 ++++++++++++++++++++++++++++++++++++++++----
> >   docs/interop/qmp-spec.txt    | 30 +++++++++++++++++---
> >   2 files changed, 86 insertions(+), 9 deletions(-)
> 
> If all goes well, I'm planning on taking this series through my QAPI tree,
> with a pull request either late today or early Monday in order to make the
> soft freeze deadline.  We'll see what my review turns up, but hopefully at
> this point it's either clean or minor enough tweaks that I can polish it
> without a v9.

That'll be great.  Thanks Eric!

[...]

> > @@ -102,10 +113,16 @@ The format for command execution is:
> >     required. Each command documents what contents will be considered
> >     valid when handling the json-argument
> >   - The "id" member is a transaction identification associated with the
> > -  command execution, it is optional and will be part of the response if
> > +  command execution.  It is required if OOB is enabled, and optional
> > +  if not.  The same "id" field will be part of the response if
> 
> Ambiguous on whether this is the per-command 'run-oob', or whether this is
> the generic QMP capability requested during qmp_capabilities.  I think the
> intent is that if you enabled the QMP capability up front, ALL commands must
> have an "id", even if they are run in-band without 'run-oob', because the
> 'id' in the response is what will distinguish whether a later OOB request
> overtook a pending in-band reply.

Yes, maybe I should use "OOB capability" to be explicit - exactly as
you explained.

> 
> >     provided. The "id" member can be any json-value, although most
> >     clients merely use a json-number incremented for each successive
> >     command
> > +- The "control" member is optional, and currently only used for
> > +  "out-of-band" execution ("oob" as shortcut). The handling or
> 
> A bit late to be introducing "oob" as shortcut, given that you've already
> used the abbreviation oob earlier in the document.
> 
> > +  response of an "oob" command can overtake prior in-band commands.
> > +  To enable "oob" handling of a particular command, just provide a
> > +  control field with: { "control": { "run-oob": true } }
> >   2.4 Commands Responses
> >   ----------------------
> > @@ -113,6 +130,11 @@ The format for command execution is:
> >   There are two possible responses which the Server will issue as the result
> >   of a command execution: success or error.
> > +As long as the commands were issued with a proper "id" field, then the
> > +same "id" field will be attached in the corresponding response message
> > +so that requests and responses can match.  Clients should drop all the
> > +responses that are with unknown "id" field.
> 
> s/are with/have an/
> 
> I've got quite a few wording tweaks, but the general concept is good. If you
> need to respin for other reasons, feel free to make those improvements, but
> I also don't mind making tweaks myself as part of queuing for a pull
> request, so:
> 
> Reviewed-by: Eric Blake <eblake@redhat.com>

(I agree on all the rest of the review comments too)

Thanks!

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 17/23] qapi: introduce new cmd option "allow-oob"

[Peter Xu]

On Sat, Mar 10, 2018 at 08:27:22PM -0600, Eric Blake wrote:

[...]

> I'm a bit disappointed that tests/qapi-schema/qapi-schema-test.json doesn't
> have any "allow-oob":true commands at any point in the series. I consider
> that to be a bug, so it can be fixed by followup patch even after soft
> freeze; but won't make the lack of good testing hold up this series from
> making the release.

Sorry, obviously I missed that one.

If this series will reach master soon, I'll post separate patch for
that before release as bugfix.

If this series can't make it due to any reason, I'll append a patch to
the series when repost.

> 
> > +++ b/scripts/qapi/introspect.py
> > @@ -29,6 +29,11 @@ def to_json(obj, level=0):
> >                                 to_json(obj[key], level + 1))
> >                   for key in sorted(obj.keys())]
> >           ret = '{' + ', '.join(elts) + '}'
> > +    elif isinstance(obj, bool):
> > +        if obj:
> > +            ret = 'true'
> > +        else:
> > +            ret = 'false'
> 
> Conflicts with Marc-Andre's work to perform introspection by QLIT_*; but
> it's easy enough to adjust:
> 
> +    elif isinstance(obj, bool):
> +        ret += 'QLIT_QBOOL(%s)' % ('true' if obj else 'false')
> 
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 22/23] tests: qmp-test: verify command batching

[Peter Xu]

On Sat, Mar 10, 2018 at 08:45:46PM -0600, Eric Blake wrote:
> On 03/09/2018 03:00 AM, Peter Xu wrote:
> > OOB introduced DROP event for flow control.  This should not affect old
> > QMP clients.  Add a command batching check to make sure of it.
> > 
> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >   tests/qmp-test.c | 22 ++++++++++++++++++++++
> >   1 file changed, 22 insertions(+)
> > 
> 
> > +    /*
> > +     * Test command batching.  In current test OOB is not enabled, we
> > +     * should be able to run as many commands in batch as we like.
> > +     * Using 16 (>8, which is OOB queue length) to make sure OOB won't
> > +     * break existing clients.  Note: this test does not control the
> > +     * scheduling of QEMU's QMP command processing threads so it may
> > +     * not really trigger batching inside QEMU.  This is just a
> > +     * best-effort test.
> > +     */
> > +    for (i = 0; i < 16; i++) {
> > +        qtest_async_qmp(qts, "{ 'execute': 'query-version' }");
> 
> Would the test be any more robust if we could generate a single string to
> send all at once, rather than multiple separate calls to qtest_async_qmp()
> (the overhead in generating separate strings means the monitor may have made
> progress before we send the next string).  But that can be a followup, if
> you want to pursue the idea.

Yes, a single string would be nicer.

As the comment said (which was actually pointed out by Stefan), the
test is only a best effort test considering that we cannot really
control how the QEMU QMP internal handles the requests.  E.g., even if
we send the strings in one write() call, it's still only buffered on
the receiver's side only, and it'll be QEMU who decide how to fetch
the buffer content.

I can fix this up if there is a new post, or do it separately
otherwise along with the other suggestion below.

> 
> > +    }
> > +    /* Verify the replies to make sure no command is dropped. */
> > +    for (i = 0; i < 16; i++) {
> > +        resp = qtest_qmp_receive(qts);
> > +        /* It should never be dropped.  Each of them should be a reply. */
> > +        g_assert(qdict_haskey(resp, "return"));
> > +        g_assert(!qdict_haskey(resp, "event"));
> > +        QDECREF(resp);
> 
> Should we also be sending unique ids, and ensure that the responses arrive
> with ids in the same order?  Again, idea for a followup.

Yes, it can be better.  Will apply the same idea as above, depending
on the fate of current series.

> 
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 23/23] tests: qmp-test: add oob test

[Peter Xu]

On Sat, Mar 10, 2018 at 08:49:42PM -0600, Eric Blake wrote:
> On 03/09/2018 03:00 AM, Peter Xu wrote:
> > Test the new OOB capability.  Here we used the new "x-oob-test" command.
> > Firstly, we send a lock=true and oob=false command to hang the main
> 
> s/Firstly/First/
> 
> > thread.  Then send another lock=false and oob=true command (which will
> > be run inside parser this time) to free that hanged command.
> > 
> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >   tests/qmp-test.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> >   1 file changed, 65 insertions(+)
> > 
> 
> > +    /* Now, enable OOB in current QMP session, it should success. */
> 
> s/success/succeed/
> 
> > +
> > +    /*
> > +     * Firstly send the "x-oob-test" command with lock=true and
> 
> s/Firstly/First/
> 
> > +     * oob=false, it should hang the dispatcher and main thread;
> > +     * later, we send another lock=false with oob=true to continue
> > +     * that thread processing.  Finally we should receive replies from
> > +     * both commands.
> > +     */
> > +    qmp_async("{ 'execute': 'x-oob-test',"
> > +              "  'arguments': { 'lock': true }, "
> > +              "  'id': 'lock-cmd'}");
> > +    qmp_async("{ 'execute': 'x-oob-test', "
> > +              "  'arguments': { 'lock': false }, "
> > +              "  'control': { 'run-oob': true }, "
> > +              "  'id': 'unlock-cmd' }");
> > +
> > +    /* Ignore all events.  Wait for 2 acks */
> > +    while (acks < 2) {
> > +        resp = qmp_receive();
> > +        cmd_id = qdict_get_str(resp, "id");
> > +        if (!g_strcmp0(cmd_id, "lock-cmd") ||
> > +            !g_strcmp0(cmd_id, "unlock-cmd")) {
> > +            acks++;
> > +        }
> > +        QDECREF(resp);
> > +    }
> 
> Can you make the reply order deterministic?  Perhaps by having the lock
> command insert a sleep after locking but before replying, so that the unlock
> always gets to reply first?  But that can be a followup.

Yes, I could.  I'm just afraid that sleep might be undeterministic too
in some extreme cases, since IMHO it'll still depend on other things
(e.g., the scheduler of host, resources/workload on the host, etc.) to
make sure the order of the two commands will be exactly what we want.

So, even if current test seems to be undetermistic on the order of
received responses, IMHO it's very good on the other hand that it is
very determinstic on the test result...

> 
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 00/23] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Sat, Mar 10, 2018 at 08:59:38PM -0600, Eric Blake wrote:
> On 03/09/2018 02:59 AM, Peter Xu wrote:
> > Based-on: <20180306053320.15401-1-peterx@redhat.com>
> > 
> > The series is based on the following series:
> > 
> >    [PATCH v2 0/9] chardev: qio related non-default context support
> >    (actually patch 9 is based on version 2.1, Dan's version)

[1]

> > 
> > This version only contains some trivial changes, most of the work is
> > posted separately in either the QIO or CHARDEV series.
> > 
> > Four days until soft freeze.  Hmm...
> > 
> > Please review.  Thanks,
> > 
> > v8:
> > - fix up doc patch as suggested [Eric]
> > - rename functions in form of X_get_Y() or X_bh() [Stefan]
> > - English fixes [Stefan]
> > - make sure to kick iothread after suspend count update [Stefan]
> > - remove the hack to remove fd in monitor_init(), since now we have
> >    the QIO/CHARDEV series to solve the root problem.
> 
> Still a lot that I had to clean up (in part due to conflicts with other
> pending patches that also touch qapi), but given the utility of this
> feature, and the impending deadline for freeze, I've gone ahead and done the
> cleanup work.  Your patches are now stages on my QAPI tree.
> 
> git://repo.or.cz/qemu/ericb.git qapi
> http://repo.or.cz/qemu/ericb.git/shortlog/refs/heads/qapi
> 
> Note that I had some ideas for followup patches (mainly strengthening the
> testsuite, which is typically safe early on in freeze because it is not
> introducing features but strengthening the quality of the release).

I really appreciate very much on your work and time on the series.

I've noted down all the review comments, including the test
enhancements (which I'll do in a proper way soon).  I'll avoid
replying to every review comments since all of them are sane.

Note that this series will depend on the chardev fixes ([1] above) in
Paolo's tree. They'll possibly not be needed for compilation, but
might be needed for further tests.  Hope that won't be a big problem.

Thanks again!

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 00/23] QMP: out-of-band (OOB) execution support

[Eric Blake]

On 03/09/2018 02:59 AM, Peter Xu wrote:
> Based-on: <20180306053320.15401-1-peterx@redhat.com>
> 
> The series is based on the following series:
> 
>    [PATCH v2 0/9] chardev: qio related non-default context support
>    (actually patch 9 is based on version 2.1, Dan's version)

I see Dan already submitted a pull request for (some of) your qio work; 
see commit 3ef91576.  Are there any more outstanding qio patches that 
have to land first?

> 
> This series was born from this one:
> 
>    https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> 
> The idea comes from Markus Armbruster and the discussion we had in the
> thread.  It's not a super ideal solution (I believe Markus had been
> thinking hard to keep everything in order meanwhile trying to satisfy
> the migration requirement), but AFAIU it's currently the best.
> 
> What is OOB?
> ============
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 00/23] QMP: out-of-band (OOB) execution support

[Daniel P. Berrangé]

On Mon, Mar 12, 2018 at 07:01:50AM -0500, Eric Blake wrote:
> On 03/09/2018 02:59 AM, Peter Xu wrote:
> > Based-on: <20180306053320.15401-1-peterx@redhat.com>
> > 
> > The series is based on the following series:
> > 
> >    [PATCH v2 0/9] chardev: qio related non-default context support
> >    (actually patch 9 is based on version 2.1, Dan's version)
> 
> I see Dan already submitted a pull request for (some of) your qio work; see
> commit 3ef91576.  Are there any more outstanding qio patches that have to
> land first?

AFAIK, there isn't anymore qio stuff pending merge - just the chardev
series Paolo has queued.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [PATCH v8 00/23] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Mon, Mar 12, 2018 at 12:55:28PM +0000, Daniel P. Berrangé wrote:
> On Mon, Mar 12, 2018 at 07:01:50AM -0500, Eric Blake wrote:
> > On 03/09/2018 02:59 AM, Peter Xu wrote:
> > > Based-on: <20180306053320.15401-1-peterx@redhat.com>
> > > 
> > > The series is based on the following series:
> > > 
> > >    [PATCH v2 0/9] chardev: qio related non-default context support
> > >    (actually patch 9 is based on version 2.1, Dan's version)
> > 
> > I see Dan already submitted a pull request for (some of) your qio work; see
> > commit 3ef91576.  Are there any more outstanding qio patches that have to
> > land first?
> 
> AFAIK, there isn't anymore qio stuff pending merge - just the chardev
> series Paolo has queued.

Yes, there are still chardev patches that used the new QIO code queued
in Paolo's tree.  There can be either 8/9 patches there, dpending on
whether Paolo has queued the 9th patch from Dan.

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Marc-André Lureau]

Hi

On Fri, Mar 9, 2018 at 9:59 AM, Peter Xu <peterx@redhat.com> wrote:
> Originally QMP goes through these steps:
>
>   JSON Parser --> QMP Dispatcher --> Respond
>       /|\    (2)                (3)     |
>    (1) |                               \|/ (4)
>        +---------  main thread  --------+
>
> This patch does this:
>
>   JSON Parser     QMP Dispatcher --> Respond
>       /|\ |           /|\       (4)     |
>        |  | (2)        | (3)            |  (5)
>    (1) |  +----->      |               \|/
>        +---------  main thread  <-------+
>
> So the parsing job and the dispatching job is isolated now.  It gives us
> a chance in following up patches to totally move the parser outside.
>
> The isolation is done using one QEMUBH. Only one dispatcher QEMUBH is
> used for all the monitors.
>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-------
>  1 file changed, 178 insertions(+), 23 deletions(-)
>
> diff --git a/monitor.c b/monitor.c
> index de9343be87..5104e5db07 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -172,6 +172,13 @@ typedef struct {
>       */
>      QmpCommandList *commands;
>      bool qmp_caps[QMP_CAPABILITY__MAX];
> +    /*
> +     * Protects qmp request/response queue.  Please take monitor_lock
> +     * first when used together.
> +     */
> +    QemuMutex qmp_queue_lock;
> +    /* Input queue that holds all the parsed QMP requests */
> +    GQueue *qmp_requests;
>  } MonitorQMP;
>
>  /*
> @@ -218,6 +225,8 @@ struct Monitor {
>  /* Let's add monitor global variables to this struct. */
>  static struct {
>      IOThread *mon_iothread;
> +    /* Bottom half to dispatch the requests received from IO thread */
> +    QEMUBH *qmp_dispatcher_bh;
>  } mon_global;
>
>  /* QMP checker flags */
> @@ -600,11 +609,13 @@ static void monitor_data_init(Monitor *mon, bool skip_flush,
>  {
>      memset(mon, 0, sizeof(Monitor));
>      qemu_mutex_init(&mon->out_lock);
> +    qemu_mutex_init(&mon->qmp.qmp_queue_lock);
>      mon->outbuf = qstring_new();
>      /* Use *mon_cmds by default. */
>      mon->cmd_table = mon_cmds;
>      mon->skip_flush = skip_flush;
>      mon->use_io_thr = use_io_thr;
> +    mon->qmp.qmp_requests = g_queue_new();
>  }
>
>  static void monitor_data_destroy(Monitor *mon)
> @@ -617,6 +628,8 @@ static void monitor_data_destroy(Monitor *mon)
>      readline_free(mon->rs);
>      QDECREF(mon->outbuf);
>      qemu_mutex_destroy(&mon->out_lock);
> +    qemu_mutex_destroy(&mon->qmp.qmp_queue_lock);
> +    g_queue_free(mon->qmp.qmp_requests);
>  }
>
>  char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
> @@ -1056,6 +1069,16 @@ static void monitor_init_qmp_commands(void)
>                           qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
>  }
>
> +static bool qmp_cap_enabled(Monitor *mon, QMPCapability cap)
> +{
> +    return mon->qmp.qmp_caps[cap];
> +}
> +
> +static bool qmp_oob_enabled(Monitor *mon)
> +{
> +    return qmp_cap_enabled(mon, QMP_CAPABILITY_OOB);
> +}
> +
>  static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
>                             Error **errp)
>  {
> @@ -3866,30 +3889,39 @@ static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
>      qobject_decref(rsp);
>  }
>
> -static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> +struct QMPRequest {
> +    /* Owner of the request */
> +    Monitor *mon;
> +    /* "id" field of the request */
> +    QObject *id;
> +    /* Request object to be handled */
> +    QObject *req;
> +    /*
> +     * Whether we need to resume the monitor afterward.  This flag is
> +     * used to emulate the old QMP server behavior that the current
> +     * command must be completed before execution of the next one.
> +     */
> +    bool need_resume;
> +};
> +typedef struct QMPRequest QMPRequest;
> +
> +/*
> + * Dispatch one single QMP request. The function will free the req_obj
> + * and objects inside it before return.
> + */
> +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
>  {
> -    QObject *req, *rsp = NULL, *id = NULL;
> +    Monitor *mon, *old_mon;
> +    QObject *req, *rsp = NULL, *id;
>      QDict *qdict = NULL;
> -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
> -
> -    Error *err = NULL;
> +    bool need_resume;
>
> -    req = json_parser_parse_err(tokens, NULL, &err);
> -    if (!req && !err) {
> -        /* json_parser_parse_err() sucks: can fail without setting @err */
> -        error_setg(&err, QERR_JSON_PARSING);
> -    }
> -    if (err) {
> -        goto err_out;
> -    }
> +    req = req_obj->req;
> +    mon = req_obj->mon;
> +    id = req_obj->id;
> +    need_resume = req_obj->need_resume;
>
> -    qdict = qobject_to_qdict(req);
> -    if (qdict) {
> -        id = qdict_get(qdict, "id");
> -        qobject_incref(id);
> -        qdict_del(qdict, "id");
> -    } /* else will fail qmp_dispatch() */
> +    g_free(req_obj);
>
>      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
>          QString *req_json = qobject_to_json(req);
> @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>      old_mon = cur_mon;
>      cur_mon = mon;
>
> -    rsp = qmp_dispatch(cur_mon->qmp.commands, req);
> +    rsp = qmp_dispatch(mon->qmp.commands, req);
>
>      cur_mon = old_mon;
>
> @@ -3916,12 +3948,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>          }
>      }
>
> -err_out:
> -    monitor_qmp_respond(mon, rsp, err, id);
> +    /* Respond if necessary */
> +    monitor_qmp_respond(mon, rsp, NULL, id);
> +
> +    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
> +    if (need_resume) {
> +        monitor_resume(mon);
> +    }
>
>      qobject_decref(req);
>  }
>
> +/*
> + * Pop one QMP request from monitor queues, return NULL if not found.
> + * We are using round-robin fashion to pop the request, to avoid
> + * processing command only on a very busy monitor.  To achieve that,
> + * when we processed one request on specific monitor, we put that
> + * monitor to the end of mon_list queue.
> + */
> +static QMPRequest *monitor_qmp_requests_pop_one(void)
> +{
> +    QMPRequest *req_obj = NULL;
> +    Monitor *mon;
> +
> +    qemu_mutex_lock(&monitor_lock);
> +
> +    QTAILQ_FOREACH(mon, &mon_list, entry) {
> +        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> +        req_obj = g_queue_pop_head(mon->qmp.qmp_requests);
> +        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> +        if (req_obj) {
> +            break;
> +        }
> +    }
> +
> +    if (req_obj) {
> +        /*
> +         * We found one request on the monitor. Degrade this monitor's
> +         * priority to lowest by re-inserting it to end of queue.
> +         */
> +        QTAILQ_REMOVE(&mon_list, mon, entry);
> +        QTAILQ_INSERT_TAIL(&mon_list, mon, entry);
> +    }
> +
> +    qemu_mutex_unlock(&monitor_lock);
> +
> +    return req_obj;
> +}
> +
> +static void monitor_qmp_bh_dispatcher(void *data)
> +{
> +    QMPRequest *req_obj = monitor_qmp_requests_pop_one();
> +
> +    if (req_obj) {
> +        monitor_qmp_dispatch_one(req_obj);
> +        /* Reschedule instead of looping so the main loop stays responsive */
> +        qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> +    }
> +}
> +
> +static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> +{
> +    QObject *req, *id = NULL;
> +    QDict *qdict = NULL;
> +    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> +    Monitor *mon = container_of(mon_qmp, Monitor, qmp);
> +    Error *err = NULL;
> +    QMPRequest *req_obj;
> +
> +    req = json_parser_parse_err(tokens, NULL, &err);
> +    if (!req && !err) {
> +        /* json_parser_parse_err() sucks: can fail without setting @err */
> +        error_setg(&err, QERR_JSON_PARSING);
> +    }
> +    if (err) {
> +        monitor_qmp_respond(mon, NULL, err, NULL);
> +        qobject_decref(req);
> +        return;
> +    }
> +
> +    qdict = qobject_to_qdict(req);
> +    if (qdict) {
> +        id = qdict_get(qdict, "id");
> +        qobject_incref(id);
> +        qdict_del(qdict, "id");
> +    } /* else will fail qmp_dispatch() */
> +
> +    req_obj = g_new0(QMPRequest, 1);
> +    req_obj->mon = mon;
> +    req_obj->id = id;
> +    req_obj->req = req;
> +    req_obj->need_resume = false;
> +
> +    /*
> +     * If OOB is not enabled on current monitor, we'll emulate the old
> +     * behavior that we won't process current monitor any more until
> +     * it is responded.  This helps make sure that as long as OOB is
> +     * not enabled, the server will never drop any command.
> +     */
> +    if (!qmp_oob_enabled(mon)) {
> +        monitor_suspend(mon);
> +        req_obj->need_resume = true;
> +    }
> +
> +    /*
> +     * Put the request to the end of queue so that requests will be
> +     * handled in time order.  Ownership for req_obj, req, id,

I think the order is not respected if subsequent messages have errors
(in either json parsing, dispatch_check_obj, oob_check). So if I
enable oob, and queue a few command, then send a bad command/message,
I won't be able to tell for which command.

> +     * etc. will be delivered to the handler side.
> +     */
> +    qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> +    g_queue_push_tail(mon->qmp.qmp_requests, req_obj);
> +    qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> +
> +    /* Kick the dispatcher routine */
> +    qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> +}
> +
>  static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
>  {
>      Monitor *mon = opaque;
> @@ -4134,6 +4276,15 @@ static void monitor_iothread_init(void)
>  {
>      mon_global.mon_iothread = iothread_create("mon_iothread",
>                                                &error_abort);
> +
> +    /*
> +     * This MUST be on main loop thread since we have commands that
> +     * have assumption to be run on main loop thread.  It would be
> +     * nice that one day we can remove this assumption in the future.
> +     */
> +    mon_global.qmp_dispatcher_bh = aio_bh_new(qemu_get_aio_context(),
> +                                              monitor_qmp_bh_dispatcher,
> +                                              NULL);
>  }
>
>  void monitor_init_globals(void)
> @@ -4280,6 +4431,10 @@ void monitor_cleanup(void)
>      }
>      qemu_mutex_unlock(&monitor_lock);
>
> +    /* QEMUBHs needs to be deleted before destroying the IOThread. */
> +    qemu_bh_delete(mon_global.qmp_dispatcher_bh);
> +    mon_global.qmp_dispatcher_bh = NULL;
> +
>      iothread_destroy(mon_global.mon_iothread);
>      mon_global.mon_iothread = NULL;
>  }
> --
> 2.14.3
>
>



-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Dr. David Alan Gilbert]

* Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> Hi
> 
> On Fri, Mar 9, 2018 at 9:59 AM, Peter Xu <peterx@redhat.com> wrote:
> > Originally QMP goes through these steps:
> >
> >   JSON Parser --> QMP Dispatcher --> Respond
> >       /|\    (2)                (3)     |
> >    (1) |                               \|/ (4)
> >        +---------  main thread  --------+
> >
> > This patch does this:
> >
> >   JSON Parser     QMP Dispatcher --> Respond
> >       /|\ |           /|\       (4)     |
> >        |  | (2)        | (3)            |  (5)
> >    (1) |  +----->      |               \|/
> >        +---------  main thread  <-------+
> >
> > So the parsing job and the dispatching job is isolated now.  It gives us
> > a chance in following up patches to totally move the parser outside.
> >
> > The isolation is done using one QEMUBH. Only one dispatcher QEMUBH is
> > used for all the monitors.
> >
> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  monitor.c | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-------
> >  1 file changed, 178 insertions(+), 23 deletions(-)
> >
> > diff --git a/monitor.c b/monitor.c
> > index de9343be87..5104e5db07 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -172,6 +172,13 @@ typedef struct {
> >       */
> >      QmpCommandList *commands;
> >      bool qmp_caps[QMP_CAPABILITY__MAX];
> > +    /*
> > +     * Protects qmp request/response queue.  Please take monitor_lock
> > +     * first when used together.
> > +     */
> > +    QemuMutex qmp_queue_lock;
> > +    /* Input queue that holds all the parsed QMP requests */
> > +    GQueue *qmp_requests;
> >  } MonitorQMP;
> >
> >  /*
> > @@ -218,6 +225,8 @@ struct Monitor {
> >  /* Let's add monitor global variables to this struct. */
> >  static struct {
> >      IOThread *mon_iothread;
> > +    /* Bottom half to dispatch the requests received from IO thread */
> > +    QEMUBH *qmp_dispatcher_bh;
> >  } mon_global;
> >
> >  /* QMP checker flags */
> > @@ -600,11 +609,13 @@ static void monitor_data_init(Monitor *mon, bool skip_flush,
> >  {
> >      memset(mon, 0, sizeof(Monitor));
> >      qemu_mutex_init(&mon->out_lock);
> > +    qemu_mutex_init(&mon->qmp.qmp_queue_lock);
> >      mon->outbuf = qstring_new();
> >      /* Use *mon_cmds by default. */
> >      mon->cmd_table = mon_cmds;
> >      mon->skip_flush = skip_flush;
> >      mon->use_io_thr = use_io_thr;
> > +    mon->qmp.qmp_requests = g_queue_new();
> >  }
> >
> >  static void monitor_data_destroy(Monitor *mon)
> > @@ -617,6 +628,8 @@ static void monitor_data_destroy(Monitor *mon)
> >      readline_free(mon->rs);
> >      QDECREF(mon->outbuf);
> >      qemu_mutex_destroy(&mon->out_lock);
> > +    qemu_mutex_destroy(&mon->qmp.qmp_queue_lock);
> > +    g_queue_free(mon->qmp.qmp_requests);
> >  }
> >
> >  char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
> > @@ -1056,6 +1069,16 @@ static void monitor_init_qmp_commands(void)
> >                           qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
> >  }
> >
> > +static bool qmp_cap_enabled(Monitor *mon, QMPCapability cap)
> > +{
> > +    return mon->qmp.qmp_caps[cap];
> > +}
> > +
> > +static bool qmp_oob_enabled(Monitor *mon)
> > +{
> > +    return qmp_cap_enabled(mon, QMP_CAPABILITY_OOB);
> > +}
> > +
> >  static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
> >                             Error **errp)
> >  {
> > @@ -3866,30 +3889,39 @@ static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
> >      qobject_decref(rsp);
> >  }
> >
> > -static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> > +struct QMPRequest {
> > +    /* Owner of the request */
> > +    Monitor *mon;
> > +    /* "id" field of the request */
> > +    QObject *id;
> > +    /* Request object to be handled */
> > +    QObject *req;
> > +    /*
> > +     * Whether we need to resume the monitor afterward.  This flag is
> > +     * used to emulate the old QMP server behavior that the current
> > +     * command must be completed before execution of the next one.
> > +     */
> > +    bool need_resume;
> > +};
> > +typedef struct QMPRequest QMPRequest;
> > +
> > +/*
> > + * Dispatch one single QMP request. The function will free the req_obj
> > + * and objects inside it before return.
> > + */
> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> >  {
> > -    QObject *req, *rsp = NULL, *id = NULL;
> > +    Monitor *mon, *old_mon;
> > +    QObject *req, *rsp = NULL, *id;
> >      QDict *qdict = NULL;
> > -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> > -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
> > -
> > -    Error *err = NULL;
> > +    bool need_resume;
> >
> > -    req = json_parser_parse_err(tokens, NULL, &err);
> > -    if (!req && !err) {
> > -        /* json_parser_parse_err() sucks: can fail without setting @err */
> > -        error_setg(&err, QERR_JSON_PARSING);
> > -    }
> > -    if (err) {
> > -        goto err_out;
> > -    }
> > +    req = req_obj->req;
> > +    mon = req_obj->mon;
> > +    id = req_obj->id;
> > +    need_resume = req_obj->need_resume;
> >
> > -    qdict = qobject_to_qdict(req);
> > -    if (qdict) {
> > -        id = qdict_get(qdict, "id");
> > -        qobject_incref(id);
> > -        qdict_del(qdict, "id");
> > -    } /* else will fail qmp_dispatch() */
> > +    g_free(req_obj);
> >
> >      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
> >          QString *req_json = qobject_to_json(req);
> > @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >      old_mon = cur_mon;
> >      cur_mon = mon;
> >
> > -    rsp = qmp_dispatch(cur_mon->qmp.commands, req);
> > +    rsp = qmp_dispatch(mon->qmp.commands, req);
> >
> >      cur_mon = old_mon;
> >
> > @@ -3916,12 +3948,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >          }
> >      }
> >
> > -err_out:
> > -    monitor_qmp_respond(mon, rsp, err, id);
> > +    /* Respond if necessary */
> > +    monitor_qmp_respond(mon, rsp, NULL, id);
> > +
> > +    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
> > +    if (need_resume) {
> > +        monitor_resume(mon);
> > +    }
> >
> >      qobject_decref(req);
> >  }
> >
> > +/*
> > + * Pop one QMP request from monitor queues, return NULL if not found.
> > + * We are using round-robin fashion to pop the request, to avoid
> > + * processing command only on a very busy monitor.  To achieve that,
> > + * when we processed one request on specific monitor, we put that
> > + * monitor to the end of mon_list queue.
> > + */
> > +static QMPRequest *monitor_qmp_requests_pop_one(void)
> > +{
> > +    QMPRequest *req_obj = NULL;
> > +    Monitor *mon;
> > +
> > +    qemu_mutex_lock(&monitor_lock);
> > +
> > +    QTAILQ_FOREACH(mon, &mon_list, entry) {
> > +        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> > +        req_obj = g_queue_pop_head(mon->qmp.qmp_requests);
> > +        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> > +        if (req_obj) {
> > +            break;
> > +        }
> > +    }
> > +
> > +    if (req_obj) {
> > +        /*
> > +         * We found one request on the monitor. Degrade this monitor's
> > +         * priority to lowest by re-inserting it to end of queue.
> > +         */
> > +        QTAILQ_REMOVE(&mon_list, mon, entry);
> > +        QTAILQ_INSERT_TAIL(&mon_list, mon, entry);
> > +    }
> > +
> > +    qemu_mutex_unlock(&monitor_lock);
> > +
> > +    return req_obj;
> > +}
> > +
> > +static void monitor_qmp_bh_dispatcher(void *data)
> > +{
> > +    QMPRequest *req_obj = monitor_qmp_requests_pop_one();
> > +
> > +    if (req_obj) {
> > +        monitor_qmp_dispatch_one(req_obj);
> > +        /* Reschedule instead of looping so the main loop stays responsive */
> > +        qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> > +    }
> > +}
> > +
> > +static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> > +{
> > +    QObject *req, *id = NULL;
> > +    QDict *qdict = NULL;
> > +    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> > +    Monitor *mon = container_of(mon_qmp, Monitor, qmp);
> > +    Error *err = NULL;
> > +    QMPRequest *req_obj;
> > +
> > +    req = json_parser_parse_err(tokens, NULL, &err);
> > +    if (!req && !err) {
> > +        /* json_parser_parse_err() sucks: can fail without setting @err */
> > +        error_setg(&err, QERR_JSON_PARSING);
> > +    }
> > +    if (err) {
> > +        monitor_qmp_respond(mon, NULL, err, NULL);
> > +        qobject_decref(req);
> > +        return;
> > +    }
> > +
> > +    qdict = qobject_to_qdict(req);
> > +    if (qdict) {
> > +        id = qdict_get(qdict, "id");
> > +        qobject_incref(id);
> > +        qdict_del(qdict, "id");
> > +    } /* else will fail qmp_dispatch() */
> > +
> > +    req_obj = g_new0(QMPRequest, 1);
> > +    req_obj->mon = mon;
> > +    req_obj->id = id;
> > +    req_obj->req = req;
> > +    req_obj->need_resume = false;
> > +
> > +    /*
> > +     * If OOB is not enabled on current monitor, we'll emulate the old
> > +     * behavior that we won't process current monitor any more until
> > +     * it is responded.  This helps make sure that as long as OOB is
> > +     * not enabled, the server will never drop any command.
> > +     */
> > +    if (!qmp_oob_enabled(mon)) {
> > +        monitor_suspend(mon);
> > +        req_obj->need_resume = true;
> > +    }
> > +
> > +    /*
> > +     * Put the request to the end of queue so that requests will be
> > +     * handled in time order.  Ownership for req_obj, req, id,
> 
> I think the order is not respected if subsequent messages have errors
> (in either json parsing, dispatch_check_obj, oob_check). So if I
> enable oob, and queue a few command, then send a bad command/message,
> I won't be able to tell for which command.

Doesn't OOB insist on having an ID field with the command?

Dave

> > +     * etc. will be delivered to the handler side.
> > +     */
> > +    qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> > +    g_queue_push_tail(mon->qmp.qmp_requests, req_obj);
> > +    qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> > +
> > +    /* Kick the dispatcher routine */
> > +    qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> > +}
> > +
> >  static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
> >  {
> >      Monitor *mon = opaque;
> > @@ -4134,6 +4276,15 @@ static void monitor_iothread_init(void)
> >  {
> >      mon_global.mon_iothread = iothread_create("mon_iothread",
> >                                                &error_abort);
> > +
> > +    /*
> > +     * This MUST be on main loop thread since we have commands that
> > +     * have assumption to be run on main loop thread.  It would be
> > +     * nice that one day we can remove this assumption in the future.
> > +     */
> > +    mon_global.qmp_dispatcher_bh = aio_bh_new(qemu_get_aio_context(),
> > +                                              monitor_qmp_bh_dispatcher,
> > +                                              NULL);
> >  }
> >
> >  void monitor_init_globals(void)
> > @@ -4280,6 +4431,10 @@ void monitor_cleanup(void)
> >      }
> >      qemu_mutex_unlock(&monitor_lock);
> >
> > +    /* QEMUBHs needs to be deleted before destroying the IOThread. */
> > +    qemu_bh_delete(mon_global.qmp_dispatcher_bh);
> > +    mon_global.qmp_dispatcher_bh = NULL;
> > +
> >      iothread_destroy(mon_global.mon_iothread);
> >      mon_global.mon_iothread = NULL;
> >  }
> > --
> > 2.14.3
> >
> >
> 
> 
> 
> -- 
> Marc-André Lureau
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Eric Blake]

On 03/21/2018 03:09 PM, Dr. David Alan Gilbert wrote:

>>>
>>> So the parsing job and the dispatching job is isolated now.  It gives us
>>> a chance in following up patches to totally move the parser outside.
>>>
>>> The isolation is done using one QEMUBH. Only one dispatcher QEMUBH is
>>> used for all the monitors.
>>>

>>> +
>>> +    /*
>>> +     * If OOB is not enabled on current monitor, we'll emulate the old
>>> +     * behavior that we won't process current monitor any more until
>>> +     * it is responded.  This helps make sure that as long as OOB is
>>> +     * not enabled, the server will never drop any command.
>>> +     */
>>> +    if (!qmp_oob_enabled(mon)) {
>>> +        monitor_suspend(mon);
>>> +        req_obj->need_resume = true;
>>> +    }
>>> +
>>> +    /*
>>> +     * Put the request to the end of queue so that requests will be
>>> +     * handled in time order.  Ownership for req_obj, req, id,
>>
>> I think the order is not respected if subsequent messages have errors
>> (in either json parsing, dispatch_check_obj, oob_check). So if I
>> enable oob, and queue a few command, then send a bad command/message,
>> I won't be able to tell for which command.
> 
> Doesn't OOB insist on having an ID field with the command?

OOB insists on an id field - but there is the situation that SOME errors 
occur even before the id field has been encountered (for example, if you 
send non-JSON, the parser gets all confused - it has to emit SOME error, 
but that error can't refer to an id because it wasn't able to parse one 
yet).  A well-formed client will never do this, but we MUST be robust 
even in the face of a malicious client (or even a well-intentioned 
client but a noisy communication medium that manages to corrupt bytes). 
I'm not sure if the testsuite adequately covers what happens in this 
scenario.  Peter?

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Marc-André Lureau]

Hi

On Wed, Mar 21, 2018 at 9:33 PM, Eric Blake <eblake@redhat.com> wrote:
> On 03/21/2018 03:09 PM, Dr. David Alan Gilbert wrote:
>
>>>>
>>>> So the parsing job and the dispatching job is isolated now.  It gives us
>>>> a chance in following up patches to totally move the parser outside.
>>>>
>>>> The isolation is done using one QEMUBH. Only one dispatcher QEMUBH is
>>>> used for all the monitors.
>>>>
>
>>>> +
>>>> +    /*
>>>> +     * If OOB is not enabled on current monitor, we'll emulate the old
>>>> +     * behavior that we won't process current monitor any more until
>>>> +     * it is responded.  This helps make sure that as long as OOB is
>>>> +     * not enabled, the server will never drop any command.
>>>> +     */
>>>> +    if (!qmp_oob_enabled(mon)) {
>>>> +        monitor_suspend(mon);
>>>> +        req_obj->need_resume = true;
>>>> +    }
>>>> +
>>>> +    /*
>>>> +     * Put the request to the end of queue so that requests will be
>>>> +     * handled in time order.  Ownership for req_obj, req, id,
>>>
>>>
>>> I think the order is not respected if subsequent messages have errors
>>> (in either json parsing, dispatch_check_obj, oob_check). So if I
>>> enable oob, and queue a few command, then send a bad command/message,
>>> I won't be able to tell for which command.
>>
>>
>> Doesn't OOB insist on having an ID field with the command?
>
>
> OOB insists on an id field - but there is the situation that SOME errors
> occur even before the id field has been encountered (for example, if you
> send non-JSON, the parser gets all confused - it has to emit SOME error, but
> that error can't refer to an id because it wasn't able to parse one yet).  A
> well-formed client will never do this, but we MUST be robust even in the
> face of a malicious client (or even a well-intentioned client but a noisy
> communication medium that manages to corrupt bytes). I'm not sure if the
> testsuite adequately covers what happens in this scenario.  Peter?

I think a solution would be to queue the error reply (the reply only)
instead of replying directly.

Another problem I see with the current solution is that pending
commands are not discarded when a new client connects. So I suppose a
new client can receive replies for commands it didn't make, with id
namespace that may conflict with its own. breaking ordering etc. A
possible solution is to mark the pending request to not send the reply
somehow?


-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Thu, Mar 22, 2018 at 12:32:36AM +0100, Marc-André Lureau wrote:
> Hi
> 
> On Wed, Mar 21, 2018 at 9:33 PM, Eric Blake <eblake@redhat.com> wrote:
> > On 03/21/2018 03:09 PM, Dr. David Alan Gilbert wrote:
> >
> >>>>
> >>>> So the parsing job and the dispatching job is isolated now.  It gives us
> >>>> a chance in following up patches to totally move the parser outside.
> >>>>
> >>>> The isolation is done using one QEMUBH. Only one dispatcher QEMUBH is
> >>>> used for all the monitors.
> >>>>
> >
> >>>> +
> >>>> +    /*
> >>>> +     * If OOB is not enabled on current monitor, we'll emulate the old
> >>>> +     * behavior that we won't process current monitor any more until
> >>>> +     * it is responded.  This helps make sure that as long as OOB is
> >>>> +     * not enabled, the server will never drop any command.
> >>>> +     */
> >>>> +    if (!qmp_oob_enabled(mon)) {
> >>>> +        monitor_suspend(mon);
> >>>> +        req_obj->need_resume = true;
> >>>> +    }
> >>>> +
> >>>> +    /*
> >>>> +     * Put the request to the end of queue so that requests will be
> >>>> +     * handled in time order.  Ownership for req_obj, req, id,
> >>>
> >>>
> >>> I think the order is not respected if subsequent messages have errors
> >>> (in either json parsing, dispatch_check_obj, oob_check). So if I
> >>> enable oob, and queue a few command, then send a bad command/message,
> >>> I won't be able to tell for which command.
> >>
> >>
> >> Doesn't OOB insist on having an ID field with the command?
> >
> >
> > OOB insists on an id field - but there is the situation that SOME errors
> > occur even before the id field has been encountered (for example, if you
> > send non-JSON, the parser gets all confused - it has to emit SOME error, but
> > that error can't refer to an id because it wasn't able to parse one yet).  A
> > well-formed client will never do this, but we MUST be robust even in the
> > face of a malicious client (or even a well-intentioned client but a noisy
> > communication medium that manages to corrupt bytes). I'm not sure if the
> > testsuite adequately covers what happens in this scenario.  Peter?
> 
> I think a solution would be to queue the error reply (the reply only)
> instead of replying directly.

IMHO this should be fine.

Note that to be compatible with existing QMP we'll suspend the monitor
if OOB is not enabled in the session on receiving the first command.
It means even if another faulty command is sent after the good one,
it'll not be read by QMP parser until the first one is fully complete.

Since I'm working on some test patches after all, I'll try to add a
test case for this to see whether Eric would like them too.

> 
> Another problem I see with the current solution is that pending
> commands are not discarded when a new client connects. So I suppose a
> new client can receive replies for commands it didn't make, with id
> namespace that may conflict with its own. breaking ordering etc. A
> possible solution is to mark the pending request to not send the reply
> somehow?

Yeah, to be simpler - maybe we can even drop the commands that have
not yet been dispatched?

I'll treat a command as "complete" only until the client receives a
response, otherwise if a client disconnects before receiving a reply
then I think it's fine the behavior is undefined - IMHO it's fine
either the QMP server executes the command or not (and no matter what,
we drop the responses).  Would that work?

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 18/23] qmp: support out-of-band (oob) execution

[Marc-André Lureau]

Hi

On Fri, Mar 9, 2018 at 10:00 AM, Peter Xu <peterx@redhat.com> wrote:
> Having "allow-oob" to true for a command does not mean that this command
> will always be run in out-of-band mode.  The out-of-band quick path will
> only be executed if we specify the extra "run-oob" flag when sending the
> QMP request:
>
>     { "execute":   "command-that-allows-oob",
>       "arguments": { ... },
>       "control":   { "run-oob": true } }
>
> The "control" key is introduced to store this extra flag.  "control"
> field is used to store arguments that are shared by all the commands,
> rather than command specific arguments.  Let "run-oob" be the first.
>
> Note that in the patch I exported qmp_dispatch_check_obj() to be used to
> check the request earlier, and at the same time allowed "id" field to be
> there since actually we always allow that.
>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  include/qapi/qmp/dispatch.h |  2 ++
>  monitor.c                   | 84 ++++++++++++++++++++++++++++++++++++++++-----
>  qapi/qmp-dispatch.c         | 33 +++++++++++++++++-
>  trace-events                |  2 ++
>  4 files changed, 111 insertions(+), 10 deletions(-)
>
> diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
> index 26eb13ff41..ffb4652f71 100644
> --- a/include/qapi/qmp/dispatch.h
> +++ b/include/qapi/qmp/dispatch.h
> @@ -48,6 +48,8 @@ bool qmp_command_is_enabled(const QmpCommand *cmd);
>  const char *qmp_command_name(const QmpCommand *cmd);
>  bool qmp_has_success_response(const QmpCommand *cmd);
>  QObject *qmp_build_error_object(Error *err);
> +QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp);
> +bool qmp_is_oob(QDict *dict);
>
>  typedef void (*qmp_cmd_callback_fn)(QmpCommand *cmd, void *opaque);
>
> diff --git a/monitor.c b/monitor.c
> index 4d57a8d341..5c8afe9f50 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -1110,6 +1110,44 @@ static void qmp_caps_apply(Monitor *mon, QMPCapabilityList *list)
>      }
>  }
>
> +/*
> + * Return true if check successful, or false otherwise.  When false is
> + * returned, detailed error will be in errp if provided.
> + */
> +static bool qmp_cmd_oob_check(Monitor *mon, QDict *req, Error **errp)
> +{
> +    const char *command;
> +    QmpCommand *cmd;
> +
> +    command = qdict_get_try_str(req, "execute");
> +    if (!command) {
> +        error_setg(errp, "Command field 'execute' missing");
> +        return false;
> +    }
> +
> +    cmd = qmp_find_command(mon->qmp.commands, command);
> +    if (!cmd) {
> +        error_set(errp, ERROR_CLASS_COMMAND_NOT_FOUND,
> +                  "The command %s has not been found", command);
> +        return false;
> +    }
> +
> +    if (qmp_is_oob(req)) {
> +        if (!qmp_oob_enabled(mon)) {
> +            error_setg(errp, "Please enable Out-Of-Band first "
> +                       "for the session during capabilities negociation");
> +            return false;
> +        }
> +        if (!(cmd->options & QCO_ALLOW_OOB)) {
> +            error_setg(errp, "The command %s does not support OOB",
> +                       command);
> +            return false;
> +        }
> +    }
> +
> +    return true;
> +}
> +
>  void qmp_qmp_capabilities(bool has_enable, QMPCapabilityList *enable,
>                            Error **errp)
>  {
> @@ -4001,6 +4039,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
>      QMPRequest *req_obj = monitor_qmp_requests_pop_one();
>
>      if (req_obj) {
> +        trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
>          monitor_qmp_dispatch_one(req_obj);
>          /* Reschedule instead of looping so the main loop stays responsive */
>          qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> @@ -4024,17 +4063,31 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>          error_setg(&err, QERR_JSON_PARSING);
>      }
>      if (err) {
> -        monitor_qmp_respond(mon, NULL, err, NULL);
> -        qobject_decref(req);
> -        return;
> +        goto err;
> +    }
> +
> +    /* Check against the request in general layout */
> +    qdict = qmp_dispatch_check_obj(req, &err);
> +    if (!qdict) {
> +        goto err;
>      }
>
> -    qdict = qobject_to_qdict(req);
> -    if (qdict) {
> -        id = qdict_get(qdict, "id");
> -        qobject_incref(id);
> -        qdict_del(qdict, "id");
> -    } /* else will fail qmp_dispatch() */
> +    /* Check against OOB specific */
> +    if (!qmp_cmd_oob_check(mon, qdict, &err)) {
> +        goto err;
> +    }

This check happens before monitor_qmp_dispatch_one(), where there is
now dead code for COMMAND_NOT_FOUND error handling (changing the error
description), since it will error here.

A solution is to move the error manipulation code to monitor_qmp_respond():


diff --git a/monitor.c b/monitor.c
index 52b4c77ac3..df09a1657d 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3997,6 +3997,18 @@ static void monitor_qmp_respond(Monitor *mon,
QObject *rsp,
             qdict_put_obj(qobject_to(QDict, rsp), "id", id);
         }

+        if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
+            qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
+            if (qdict
+                && !g_strcmp0(qdict_get_try_str(qdict, "class"),
+
QapiErrorClass_str(ERROR_CLASS_COMMAND_NOT_FOUND))) {
+                /* Provide a more useful error message */
+                qdict_del(qdict, "desc");
+                qdict_put_str(qdict, "desc", "Expecting capabilities
negotiation"
+                              " with 'qmp_capabilities'");
+            }
+        }
+
         monitor_json_emitter(mon, rsp);
     }

@@ -4022,7 +4034,6 @@ static void monitor_qmp_dispatch_one(QMPRequest
*req_obj, bool oob_cmd)
 {
     Monitor *mon, *old_mon;
     QObject *req, *rsp = NULL, *id;
-    QDict *qdict = NULL;
     bool need_resume;

     req = req_obj->req;
@@ -4045,18 +4056,6 @@ static void monitor_qmp_dispatch_one(QMPRequest
*req_obj, bool oob_cmd)

     cur_mon = old_mon;

-    if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
-        qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
-        if (qdict
-            && !g_strcmp0(qdict_get_try_str(qdict, "class"),
-                    QapiErrorClass_str(ERROR_CLASS_COMMAND_NOT_FOUND))) {
-            /* Provide a more useful error message */
-            qdict_del(qdict, "desc");
-            qdict_put_str(qdict, "desc", "Expecting capabilities negotiation"
-                          " with 'qmp_capabilities'");
-        }
-    }
-
     /* Respond if necessary */
     monitor_qmp_respond(mon, rsp, NULL, id);


> +
> +    id = qdict_get(qdict, "id");
> +
> +    /* When OOB is enabled, the "id" field is mandatory. */
> +    if (qmp_oob_enabled(mon) && !id) {
> +        error_setg(&err, "Out-Of-Band capability requires that "
> +                   "every command contains an 'id' field");
> +        goto err;
> +    }
> +
> +    qobject_incref(id);
> +    qdict_del(qdict, "id");
>
>      req_obj = g_new0(QMPRequest, 1);
>      req_obj->mon = mon;
> @@ -4042,6 +4095,14 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>      req_obj->req = req;
>      req_obj->need_resume = false;
>
> +    if (qmp_is_oob(qdict)) {
> +        /* Out-Of-Band (OOB) requests are executed directly in parser. */
> +        trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
> +                                          ?: "");
> +        monitor_qmp_dispatch_one(req_obj);
> +        return;
> +    }
> +
>      /* Protect qmp_requests and fetching its length. */
>      qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
>
> @@ -4078,6 +4139,11 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>
>      /* Kick the dispatcher routine */
>      qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> +    return;
> +
> +err:
> +    monitor_qmp_respond(mon, NULL, err, NULL);
> +    qobject_decref(req);
>  }
>
>  static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
> diff --git a/qapi/qmp-dispatch.c b/qapi/qmp-dispatch.c
> index e31ac4be1f..9a2e18c768 100644
> --- a/qapi/qmp-dispatch.c
> +++ b/qapi/qmp-dispatch.c
> @@ -17,8 +17,9 @@
>  #include "qapi/qmp/json-parser.h"
>  #include "qapi/qmp/qdict.h"
>  #include "qapi/qmp/qjson.h"
> +#include "qapi/qmp/qbool.h"
>
> -static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
> +QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
>  {
>      const QDictEntry *ent;
>      const char *arg_name;
> @@ -50,6 +51,14 @@ static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
>                             "QMP input member 'arguments' must be an object");
>                  return NULL;
>              }
> +        } else if (!strcmp(arg_name, "id")) {
> +            continue;
> +        } else if (!strcmp(arg_name, "control")) {
> +            if (qobject_type(arg_obj) != QTYPE_QDICT) {
> +                error_setg(errp,
> +                           "QMP input member 'control' must be a dict");
> +                return NULL;
> +            }
>          } else {
>              error_setg(errp, "QMP input member '%s' is unexpected",
>                         arg_name);
> @@ -120,6 +129,28 @@ QObject *qmp_build_error_object(Error *err)
>                                error_get_pretty(err));
>  }
>
> +/*
> + * Detect whether a request should be run out-of-band, by quickly
> + * peeking at whether we have: { "control": { "run-oob": True } }. By
> + * default commands are run in-band.
> + */
> +bool qmp_is_oob(QDict *dict)
> +{
> +    QBool *bool_obj;
> +
> +    dict = qdict_get_qdict(dict, "control");
> +    if (!dict) {
> +        return false;
> +    }
> +
> +    bool_obj = qobject_to_qbool(qdict_get(dict, "run-oob"));
> +    if (!bool_obj) {
> +        return false;
> +    }
> +
> +    return qbool_get_bool(bool_obj);
> +}
> +
>  QObject *qmp_dispatch(QmpCommandList *cmds, QObject *request)
>  {
>      Error *err = NULL;
> diff --git a/trace-events b/trace-events
> index fe10c3b487..bd036da278 100644
> --- a/trace-events
> +++ b/trace-events
> @@ -48,6 +48,8 @@ monitor_protocol_event_queue(uint32_t event, void *qdict, uint64_t rate) "event=
>  handle_hmp_command(void *mon, const char *cmdline) "mon %p cmdline: %s"
>  handle_qmp_command(void *mon, const char *req) "mon %p req: %s"
>  monitor_suspend(void *ptr, int cnt) "mon %p: %d"
> +monitor_qmp_cmd_in_band(const char *id) "%s"
> +monitor_qmp_cmd_out_of_band(const char *id) "%s"
>
>  # dma-helpers.c
>  dma_blk_io(void *dbs, void *bs, int64_t offset, bool to_dev) "dbs=%p bs=%p offset=%" PRId64 " to_dev=%d"
> --
> 2.14.3
>
>



-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v8 19/23] qmp: isolate responses into io thread

[Marc-André Lureau]

Hi

On Fri, Mar 9, 2018 at 10:00 AM, Peter Xu <peterx@redhat.com> wrote:
> For those monitors who have enabled IO thread, we'll offload the
> responding procedure into IO thread.  The main reason is that chardev is
> not thread safe, and we need to do all the read/write IOs in the same
> thread.  For use_io_thr=true monitors, that thread is the IO thread.

Actually, the chr write path is suppose to be somewhat thread safe
(chr_write_lock).

Secondly, the function responsible to write monitor data has some
thread-safety, it's called monitor_flush_locked(), because you need
the mon->out_lock.

I think that patch is making things more complicated than they need to
be. You should be able to call monitor_json_emitter/monitor_puts()
directly from any thread, it will queue the data, start writing, and
add a watch if necessary in the appropriate context.

Am I missing something?

> We do this isolation in similar pattern as what we have done to the
> request queue: we first create one response queue for each monitor, then
> instead of replying directly in the main thread, we queue the responses
> and kick the IO thread to do the rest of the job for us.
>
> A funny thing after doing this is that, when the QMP clients send "quit"
> to QEMU, it's possible that we close the IOThread even earlier than
> replying to that "quit".  So another thing we need to do before cleaning
> up the monitors is that we need to flush the response queue (we don't
> need to do that for command queue; after all we are quitting) to make
> sure replies for handled commands are always flushed back to clients.
>
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 96 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 95 insertions(+), 1 deletion(-)
>
> diff --git a/monitor.c b/monitor.c
> index 5c8afe9f50..c6de5f123e 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -179,6 +179,8 @@ typedef struct {
>      QemuMutex qmp_queue_lock;
>      /* Input queue that holds all the parsed QMP requests */
>      GQueue *qmp_requests;
> +    /* Output queue contains all the QMP responses in order */
> +    GQueue *qmp_responses;
>  } MonitorQMP;
>
>  /*
> @@ -205,6 +207,7 @@ struct Monitor {
>      bool skip_flush;
>      bool use_io_thr;
>
> +    /* We can't access guest memory when holding the lock */
>      QemuMutex out_lock;
>      QString *outbuf;
>      guint out_watch;
> @@ -227,6 +230,8 @@ static struct {
>      IOThread *mon_iothread;
>      /* Bottom half to dispatch the requests received from IO thread */
>      QEMUBH *qmp_dispatcher_bh;
> +    /* Bottom half to deliver the responses back to clients */
> +    QEMUBH *qmp_respond_bh;
>  } mon_global;
>
>  /* QMP checker flags */
> @@ -416,7 +421,8 @@ int monitor_fprintf(FILE *stream, const char *fmt, ...)
>      return 0;
>  }
>
> -static void monitor_json_emitter(Monitor *mon, const QObject *data)
> +static void monitor_json_emitter_raw(Monitor *mon,
> +                                     QObject *data)
>  {
>      QString *json;
>
> @@ -430,6 +436,71 @@ static void monitor_json_emitter(Monitor *mon, const QObject *data)
>      QDECREF(json);
>  }
>
> +static void monitor_json_emitter(Monitor *mon, QObject *data)
> +{
> +    if (mon->use_io_thr) {
> +        /*
> +         * If using IO thread, we need to queue the item so that IO
> +         * thread will do the rest for us.  Take refcount so that
> +         * caller won't free the data (which will be finally freed in
> +         * responder thread).
> +         */
> +        qobject_incref(data);
> +        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> +        g_queue_push_tail(mon->qmp.qmp_responses, (void *)data);
> +        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> +        qemu_bh_schedule(mon_global.qmp_respond_bh);
> +    } else {
> +        /*
> +         * If not using monitor IO thread, then we are in main thread.
> +         * Do the emission right away.
> +         */
> +        monitor_json_emitter_raw(mon, data);
> +    }
> +}
> +
> +struct QMPResponse {
> +    Monitor *mon;
> +    QObject *data;
> +};
> +typedef struct QMPResponse QMPResponse;
> +
> +/*
> + * Return one QMPResponse.  The response is only valid if
> + * response.data is not NULL.
> + */
> +static QMPResponse monitor_qmp_response_pop_one(void)
> +{
> +    Monitor *mon;
> +    QObject *data = NULL;
> +
> +    qemu_mutex_lock(&monitor_lock);
> +    QTAILQ_FOREACH(mon, &mon_list, entry) {
> +        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> +        data = g_queue_pop_head(mon->qmp.qmp_responses);
> +        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> +        if (data) {
> +            break;
> +        }
> +    }
> +    qemu_mutex_unlock(&monitor_lock);
> +    return (QMPResponse) { .mon = mon, .data = data };
> +}
> +
> +static void monitor_qmp_bh_responder(void *opaque)
> +{
> +    QMPResponse response;
> +
> +    while (true) {
> +        response = monitor_qmp_response_pop_one();
> +        if (!response.data) {
> +            break;
> +        }
> +        monitor_json_emitter_raw(response.mon, response.data);
> +        qobject_decref(response.data);
> +    }
> +}
> +
>  static MonitorQAPIEventConf monitor_qapi_event_conf[QAPI_EVENT__MAX] = {
>      /* Limit guest-triggerable events to 1 per second */
>      [QAPI_EVENT_RTC_CHANGE]        = { 1000 * SCALE_MS },
> @@ -616,6 +687,7 @@ static void monitor_data_init(Monitor *mon, bool skip_flush,
>      mon->skip_flush = skip_flush;
>      mon->use_io_thr = use_io_thr;
>      mon->qmp.qmp_requests = g_queue_new();
> +    mon->qmp.qmp_responses = g_queue_new();
>  }
>
>  static void monitor_data_destroy(Monitor *mon)
> @@ -630,6 +702,7 @@ static void monitor_data_destroy(Monitor *mon)
>      qemu_mutex_destroy(&mon->out_lock);
>      qemu_mutex_destroy(&mon->qmp.qmp_queue_lock);
>      g_queue_free(mon->qmp.qmp_requests);
> +    g_queue_free(mon->qmp.qmp_responses);
>  }
>
>  char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
> @@ -4367,6 +4440,15 @@ static void monitor_iothread_init(void)
>      mon_global.qmp_dispatcher_bh = aio_bh_new(qemu_get_aio_context(),
>                                                monitor_qmp_bh_dispatcher,
>                                                NULL);
> +
> +    /*
> +     * Unlike the dispatcher BH, this must be run on the monitor IO
> +     * thread, so that monitors that are using IO thread will make
> +     * sure read/write operations are all done on the IO thread.
> +     */
> +    mon_global.qmp_respond_bh = aio_bh_new(monitor_get_aio_context(),
> +                                           monitor_qmp_bh_responder,
> +                                           NULL);
>  }
>
>  void monitor_init_globals(void)
> @@ -4505,9 +4587,19 @@ void monitor_cleanup(void)
>       */
>      iothread_stop(mon_global.mon_iothread);
>
> +    /*
> +     * After we have IOThread to send responses, it's possible that
> +     * when we stop the IOThread there are still replies queued in the
> +     * responder queue.  Flush all of them.  Note that even after this
> +     * flush it's still possible that out buffer is not flushed.
> +     * It'll be done in below monitor_flush() as the last resort.
> +     */
> +    monitor_qmp_bh_responder(NULL);
> +
>      qemu_mutex_lock(&monitor_lock);
>      QTAILQ_FOREACH_SAFE(mon, &mon_list, entry, next) {
>          QTAILQ_REMOVE(&mon_list, mon, entry);
> +        monitor_flush(mon);
>          monitor_data_destroy(mon);
>          g_free(mon);
>      }
> @@ -4516,6 +4608,8 @@ void monitor_cleanup(void)
>      /* QEMUBHs needs to be deleted before destroying the IOThread. */
>      qemu_bh_delete(mon_global.qmp_dispatcher_bh);
>      mon_global.qmp_dispatcher_bh = NULL;
> +    qemu_bh_delete(mon_global.qmp_respond_bh);
> +    mon_global.qmp_respond_bh = NULL;
>
>      iothread_destroy(mon_global.mon_iothread);
>      mon_global.mon_iothread = NULL;
> --
> 2.14.3
>
>



-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Eric Blake]

On 03/22/2018 12:00 AM, Peter Xu wrote:

>>>> Doesn't OOB insist on having an ID field with the command?
>>>
>>>
>>> OOB insists on an id field - but there is the situation that SOME errors
>>> occur even before the id field has been encountered (for example, if you
>>> send non-JSON, the parser gets all confused - it has to emit SOME error, but
>>> that error can't refer to an id because it wasn't able to parse one yet).  A
>>> well-formed client will never do this, but we MUST be robust even in the
>>> face of a malicious client (or even a well-intentioned client but a noisy
>>> communication medium that manages to corrupt bytes). I'm not sure if the
>>> testsuite adequately covers what happens in this scenario.  Peter?
>>
>> I think a solution would be to queue the error reply (the reply only)
>> instead of replying directly.
> 
> IMHO this should be fine.

Seems reasonable - conceptually, the error reply comes no sooner than 
pre-patch (the client wouldn't see a parse error message until after all 
earlier successful parsed messages completed), making a parse error a 
sort of synchronization barrier (if the client gets one, then all 
outstanding message prior to the parse error have finished sending 
replies to the client, and nothing after the parse error has been 
processed at the time the error message was sent).

> 
> Note that to be compatible with existing QMP we'll suspend the monitor
> if OOB is not enabled in the session on receiving the first command.
> It means even if another faulty command is sent after the good one,
> it'll not be read by QMP parser until the first one is fully complete.
> 
> Since I'm working on some test patches after all, I'll try to add a
> test case for this to see whether Eric would like them too.

Yes, these are the sorts of bug fixes and test suite coverage that we 
want to include during freeze, for better robustness.

> 
>>
>> Another problem I see with the current solution is that pending
>> commands are not discarded when a new client connects. So I suppose a
>> new client can receive replies for commands it didn't make, with id
>> namespace that may conflict with its own. breaking ordering etc. A
>> possible solution is to mark the pending request to not send the reply
>> somehow?
> 
> Yeah, to be simpler - maybe we can even drop the commands that have
> not yet been dispatched?
> 
> I'll treat a command as "complete" only until the client receives a
> response, otherwise if a client disconnects before receiving a reply
> then I think it's fine the behavior is undefined - IMHO it's fine
> either the QMP server executes the command or not (and no matter what,
> we drop the responses).  Would that work?

If a client disconnects before receiving a reply, I'm fine with either 
approach (the command gets discarded from the queue and never run - 
which matches if the client disconnect raced with the server actually 
getting the command, or the command ran but the response is dropped - 
which matches pre-patch behavior), while agreeing that sending the 
response to the next client is probably not a good idea (clients are 
supposed to ignore unknown id's, but we can't guarantee an 'id' 
collision will not confuse the client).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH v8 18/23] qmp: support out-of-band (oob) execution

[Peter Xu]

On Thu, Mar 22, 2018 at 11:22:14AM +0100, Marc-André Lureau wrote:
> Hi
> 
> On Fri, Mar 9, 2018 at 10:00 AM, Peter Xu <peterx@redhat.com> wrote:
> > Having "allow-oob" to true for a command does not mean that this command
> > will always be run in out-of-band mode.  The out-of-band quick path will
> > only be executed if we specify the extra "run-oob" flag when sending the
> > QMP request:
> >
> >     { "execute":   "command-that-allows-oob",
> >       "arguments": { ... },
> >       "control":   { "run-oob": true } }
> >
> > The "control" key is introduced to store this extra flag.  "control"
> > field is used to store arguments that are shared by all the commands,
> > rather than command specific arguments.  Let "run-oob" be the first.
> >
> > Note that in the patch I exported qmp_dispatch_check_obj() to be used to
> > check the request earlier, and at the same time allowed "id" field to be
> > there since actually we always allow that.
> >
> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  include/qapi/qmp/dispatch.h |  2 ++
> >  monitor.c                   | 84 ++++++++++++++++++++++++++++++++++++++++-----
> >  qapi/qmp-dispatch.c         | 33 +++++++++++++++++-
> >  trace-events                |  2 ++
> >  4 files changed, 111 insertions(+), 10 deletions(-)
> >
> > diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
> > index 26eb13ff41..ffb4652f71 100644
> > --- a/include/qapi/qmp/dispatch.h
> > +++ b/include/qapi/qmp/dispatch.h
> > @@ -48,6 +48,8 @@ bool qmp_command_is_enabled(const QmpCommand *cmd);
> >  const char *qmp_command_name(const QmpCommand *cmd);
> >  bool qmp_has_success_response(const QmpCommand *cmd);
> >  QObject *qmp_build_error_object(Error *err);
> > +QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp);
> > +bool qmp_is_oob(QDict *dict);
> >
> >  typedef void (*qmp_cmd_callback_fn)(QmpCommand *cmd, void *opaque);
> >
> > diff --git a/monitor.c b/monitor.c
> > index 4d57a8d341..5c8afe9f50 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -1110,6 +1110,44 @@ static void qmp_caps_apply(Monitor *mon, QMPCapabilityList *list)
> >      }
> >  }
> >
> > +/*
> > + * Return true if check successful, or false otherwise.  When false is
> > + * returned, detailed error will be in errp if provided.
> > + */
> > +static bool qmp_cmd_oob_check(Monitor *mon, QDict *req, Error **errp)
> > +{
> > +    const char *command;
> > +    QmpCommand *cmd;
> > +
> > +    command = qdict_get_try_str(req, "execute");
> > +    if (!command) {
> > +        error_setg(errp, "Command field 'execute' missing");
> > +        return false;
> > +    }
> > +
> > +    cmd = qmp_find_command(mon->qmp.commands, command);
> > +    if (!cmd) {
> > +        error_set(errp, ERROR_CLASS_COMMAND_NOT_FOUND,
> > +                  "The command %s has not been found", command);
> > +        return false;
> > +    }
> > +
> > +    if (qmp_is_oob(req)) {
> > +        if (!qmp_oob_enabled(mon)) {
> > +            error_setg(errp, "Please enable Out-Of-Band first "
> > +                       "for the session during capabilities negociation");
> > +            return false;
> > +        }
> > +        if (!(cmd->options & QCO_ALLOW_OOB)) {
> > +            error_setg(errp, "The command %s does not support OOB",
> > +                       command);
> > +            return false;
> > +        }
> > +    }
> > +
> > +    return true;
> > +}
> > +
> >  void qmp_qmp_capabilities(bool has_enable, QMPCapabilityList *enable,
> >                            Error **errp)
> >  {
> > @@ -4001,6 +4039,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
> >      QMPRequest *req_obj = monitor_qmp_requests_pop_one();
> >
> >      if (req_obj) {
> > +        trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
> >          monitor_qmp_dispatch_one(req_obj);
> >          /* Reschedule instead of looping so the main loop stays responsive */
> >          qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> > @@ -4024,17 +4063,31 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >          error_setg(&err, QERR_JSON_PARSING);
> >      }
> >      if (err) {
> > -        monitor_qmp_respond(mon, NULL, err, NULL);
> > -        qobject_decref(req);
> > -        return;
> > +        goto err;
> > +    }
> > +
> > +    /* Check against the request in general layout */
> > +    qdict = qmp_dispatch_check_obj(req, &err);
> > +    if (!qdict) {
> > +        goto err;
> >      }
> >
> > -    qdict = qobject_to_qdict(req);
> > -    if (qdict) {
> > -        id = qdict_get(qdict, "id");
> > -        qobject_incref(id);
> > -        qdict_del(qdict, "id");
> > -    } /* else will fail qmp_dispatch() */
> > +    /* Check against OOB specific */
> > +    if (!qmp_cmd_oob_check(mon, qdict, &err)) {
> > +        goto err;
> > +    }
> 
> This check happens before monitor_qmp_dispatch_one(), where there is
> now dead code for COMMAND_NOT_FOUND error handling (changing the error
> description), since it will error here.
> 
> A solution is to move the error manipulation code to monitor_qmp_respond():

So yes, now it'll be checked earlier now since the allow-oob is bound
to per command, then we'll need that info earlier here.  While I kept
the other check untouched since there'll be other places that calls
qmp_dispatch() too.

I'm just wondering would there be any problem even without below change?

Thanks,

> 
> 
> diff --git a/monitor.c b/monitor.c
> index 52b4c77ac3..df09a1657d 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -3997,6 +3997,18 @@ static void monitor_qmp_respond(Monitor *mon,
> QObject *rsp,
>              qdict_put_obj(qobject_to(QDict, rsp), "id", id);
>          }
> 
> +        if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
> +            qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
> +            if (qdict
> +                && !g_strcmp0(qdict_get_try_str(qdict, "class"),
> +
> QapiErrorClass_str(ERROR_CLASS_COMMAND_NOT_FOUND))) {
> +                /* Provide a more useful error message */
> +                qdict_del(qdict, "desc");
> +                qdict_put_str(qdict, "desc", "Expecting capabilities
> negotiation"
> +                              " with 'qmp_capabilities'");
> +            }
> +        }
> +
>          monitor_json_emitter(mon, rsp);
>      }
> 
> @@ -4022,7 +4034,6 @@ static void monitor_qmp_dispatch_one(QMPRequest
> *req_obj, bool oob_cmd)
>  {
>      Monitor *mon, *old_mon;
>      QObject *req, *rsp = NULL, *id;
> -    QDict *qdict = NULL;
>      bool need_resume;
> 
>      req = req_obj->req;
> @@ -4045,18 +4056,6 @@ static void monitor_qmp_dispatch_one(QMPRequest
> *req_obj, bool oob_cmd)
> 
>      cur_mon = old_mon;
> 
> -    if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
> -        qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
> -        if (qdict
> -            && !g_strcmp0(qdict_get_try_str(qdict, "class"),
> -                    QapiErrorClass_str(ERROR_CLASS_COMMAND_NOT_FOUND))) {
> -            /* Provide a more useful error message */
> -            qdict_del(qdict, "desc");
> -            qdict_put_str(qdict, "desc", "Expecting capabilities negotiation"
> -                          " with 'qmp_capabilities'");
> -        }
> -    }
> -
>      /* Respond if necessary */
>      monitor_qmp_respond(mon, rsp, NULL, id);
> 
> 
> > +
> > +    id = qdict_get(qdict, "id");
> > +
> > +    /* When OOB is enabled, the "id" field is mandatory. */
> > +    if (qmp_oob_enabled(mon) && !id) {
> > +        error_setg(&err, "Out-Of-Band capability requires that "
> > +                   "every command contains an 'id' field");
> > +        goto err;
> > +    }
> > +
> > +    qobject_incref(id);
> > +    qdict_del(qdict, "id");
> >
> >      req_obj = g_new0(QMPRequest, 1);
> >      req_obj->mon = mon;
> > @@ -4042,6 +4095,14 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >      req_obj->req = req;
> >      req_obj->need_resume = false;
> >
> > +    if (qmp_is_oob(qdict)) {
> > +        /* Out-Of-Band (OOB) requests are executed directly in parser. */
> > +        trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
> > +                                          ?: "");
> > +        monitor_qmp_dispatch_one(req_obj);
> > +        return;
> > +    }
> > +
> >      /* Protect qmp_requests and fetching its length. */
> >      qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> >
> > @@ -4078,6 +4139,11 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >
> >      /* Kick the dispatcher routine */
> >      qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> > +    return;
> > +
> > +err:
> > +    monitor_qmp_respond(mon, NULL, err, NULL);
> > +    qobject_decref(req);
> >  }
> >
> >  static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
> > diff --git a/qapi/qmp-dispatch.c b/qapi/qmp-dispatch.c
> > index e31ac4be1f..9a2e18c768 100644
> > --- a/qapi/qmp-dispatch.c
> > +++ b/qapi/qmp-dispatch.c
> > @@ -17,8 +17,9 @@
> >  #include "qapi/qmp/json-parser.h"
> >  #include "qapi/qmp/qdict.h"
> >  #include "qapi/qmp/qjson.h"
> > +#include "qapi/qmp/qbool.h"
> >
> > -static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
> > +QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
> >  {
> >      const QDictEntry *ent;
> >      const char *arg_name;
> > @@ -50,6 +51,14 @@ static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
> >                             "QMP input member 'arguments' must be an object");
> >                  return NULL;
> >              }
> > +        } else if (!strcmp(arg_name, "id")) {
> > +            continue;
> > +        } else if (!strcmp(arg_name, "control")) {
> > +            if (qobject_type(arg_obj) != QTYPE_QDICT) {
> > +                error_setg(errp,
> > +                           "QMP input member 'control' must be a dict");
> > +                return NULL;
> > +            }
> >          } else {
> >              error_setg(errp, "QMP input member '%s' is unexpected",
> >                         arg_name);
> > @@ -120,6 +129,28 @@ QObject *qmp_build_error_object(Error *err)
> >                                error_get_pretty(err));
> >  }
> >
> > +/*
> > + * Detect whether a request should be run out-of-band, by quickly
> > + * peeking at whether we have: { "control": { "run-oob": True } }. By
> > + * default commands are run in-band.
> > + */
> > +bool qmp_is_oob(QDict *dict)
> > +{
> > +    QBool *bool_obj;
> > +
> > +    dict = qdict_get_qdict(dict, "control");
> > +    if (!dict) {
> > +        return false;
> > +    }
> > +
> > +    bool_obj = qobject_to_qbool(qdict_get(dict, "run-oob"));
> > +    if (!bool_obj) {
> > +        return false;
> > +    }
> > +
> > +    return qbool_get_bool(bool_obj);
> > +}
> > +
> >  QObject *qmp_dispatch(QmpCommandList *cmds, QObject *request)
> >  {
> >      Error *err = NULL;
> > diff --git a/trace-events b/trace-events
> > index fe10c3b487..bd036da278 100644
> > --- a/trace-events
> > +++ b/trace-events
> > @@ -48,6 +48,8 @@ monitor_protocol_event_queue(uint32_t event, void *qdict, uint64_t rate) "event=
> >  handle_hmp_command(void *mon, const char *cmdline) "mon %p cmdline: %s"
> >  handle_qmp_command(void *mon, const char *req) "mon %p req: %s"
> >  monitor_suspend(void *ptr, int cnt) "mon %p: %d"
> > +monitor_qmp_cmd_in_band(const char *id) "%s"
> > +monitor_qmp_cmd_out_of_band(const char *id) "%s"
> >
> >  # dma-helpers.c
> >  dma_blk_io(void *dbs, void *bs, int64_t offset, bool to_dev) "dbs=%p bs=%p offset=%" PRId64 " to_dev=%d"
> > --
> > 2.14.3
> >
> >
> 
> 
> 
> -- 
> Marc-André Lureau

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 19/23] qmp: isolate responses into io thread

[Peter Xu]

On Thu, Mar 22, 2018 at 01:00:19PM +0100, Marc-André Lureau wrote:
> Hi
> 
> On Fri, Mar 9, 2018 at 10:00 AM, Peter Xu <peterx@redhat.com> wrote:
> > For those monitors who have enabled IO thread, we'll offload the
> > responding procedure into IO thread.  The main reason is that chardev is
> > not thread safe, and we need to do all the read/write IOs in the same
> > thread.  For use_io_thr=true monitors, that thread is the IO thread.
> 
> Actually, the chr write path is suppose to be somewhat thread safe
> (chr_write_lock).
> 
> Secondly, the function responsible to write monitor data has some
> thread-safety, it's called monitor_flush_locked(), because you need
> the mon->out_lock.
> 
> I think that patch is making things more complicated than they need to
> be. You should be able to call monitor_json_emitter/monitor_puts()
> directly from any thread, it will queue the data, start writing, and
> add a watch if necessary in the appropriate context.
> 
> Am I missing something?

Yes there are the locks either in monitor code and chardev code to
protect write operations.  However I don't know enough to be sure of
its safety.  Considering that, having the whole chardev operations
separated into the other IOThread seems to be the only safe approach
to me for now.

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 19/23] qmp: isolate responses into io thread

[Marc-André Lureau]

Hi

On Fri, Mar 23, 2018 at 6:50 AM, Peter Xu <peterx@redhat.com> wrote:
> On Thu, Mar 22, 2018 at 01:00:19PM +0100, Marc-André Lureau wrote:
>> Hi
>>
>> On Fri, Mar 9, 2018 at 10:00 AM, Peter Xu <peterx@redhat.com> wrote:
>> > For those monitors who have enabled IO thread, we'll offload the
>> > responding procedure into IO thread.  The main reason is that chardev is
>> > not thread safe, and we need to do all the read/write IOs in the same
>> > thread.  For use_io_thr=true monitors, that thread is the IO thread.
>>
>> Actually, the chr write path is suppose to be somewhat thread safe
>> (chr_write_lock).
>>
>> Secondly, the function responsible to write monitor data has some
>> thread-safety, it's called monitor_flush_locked(), because you need
>> the mon->out_lock.
>>
>> I think that patch is making things more complicated than they need to
>> be. You should be able to call monitor_json_emitter/monitor_puts()
>> directly from any thread, it will queue the data, start writing, and
>> add a watch if necessary in the appropriate context.
>>
>> Am I missing something?
>
> Yes there are the locks either in monitor code and chardev code to
> protect write operations.  However I don't know enough to be sure of
> its safety.  Considering that, having the whole chardev operations
> separated into the other IOThread seems to be the only safe approach
> to me for now.

I have studied the code in more details, and I'll propose to revert
this patch early in 2.13. I'll send a 2.13 RFC qmp series shortly.



-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v8 18/23] qmp: support out-of-band (oob) execution

[Marc-André Lureau]

Hi

On Fri, Mar 23, 2018 at 6:18 AM, Peter Xu <peterx@redhat.com> wrote:
> On Thu, Mar 22, 2018 at 11:22:14AM +0100, Marc-André Lureau wrote:
>> Hi
>>
>> On Fri, Mar 9, 2018 at 10:00 AM, Peter Xu <peterx@redhat.com> wrote:
>> > Having "allow-oob" to true for a command does not mean that this command
>> > will always be run in out-of-band mode.  The out-of-band quick path will
>> > only be executed if we specify the extra "run-oob" flag when sending the
>> > QMP request:
>> >
>> >     { "execute":   "command-that-allows-oob",
>> >       "arguments": { ... },
>> >       "control":   { "run-oob": true } }
>> >
>> > The "control" key is introduced to store this extra flag.  "control"
>> > field is used to store arguments that are shared by all the commands,
>> > rather than command specific arguments.  Let "run-oob" be the first.
>> >
>> > Note that in the patch I exported qmp_dispatch_check_obj() to be used to
>> > check the request earlier, and at the same time allowed "id" field to be
>> > there since actually we always allow that.
>> >
>> > Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
>> > Signed-off-by: Peter Xu <peterx@redhat.com>
>> > ---
>> >  include/qapi/qmp/dispatch.h |  2 ++
>> >  monitor.c                   | 84 ++++++++++++++++++++++++++++++++++++++++-----
>> >  qapi/qmp-dispatch.c         | 33 +++++++++++++++++-
>> >  trace-events                |  2 ++
>> >  4 files changed, 111 insertions(+), 10 deletions(-)
>> >
>> > diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
>> > index 26eb13ff41..ffb4652f71 100644
>> > --- a/include/qapi/qmp/dispatch.h
>> > +++ b/include/qapi/qmp/dispatch.h
>> > @@ -48,6 +48,8 @@ bool qmp_command_is_enabled(const QmpCommand *cmd);
>> >  const char *qmp_command_name(const QmpCommand *cmd);
>> >  bool qmp_has_success_response(const QmpCommand *cmd);
>> >  QObject *qmp_build_error_object(Error *err);
>> > +QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp);
>> > +bool qmp_is_oob(QDict *dict);
>> >
>> >  typedef void (*qmp_cmd_callback_fn)(QmpCommand *cmd, void *opaque);
>> >
>> > diff --git a/monitor.c b/monitor.c
>> > index 4d57a8d341..5c8afe9f50 100644
>> > --- a/monitor.c
>> > +++ b/monitor.c
>> > @@ -1110,6 +1110,44 @@ static void qmp_caps_apply(Monitor *mon, QMPCapabilityList *list)
>> >      }
>> >  }
>> >
>> > +/*
>> > + * Return true if check successful, or false otherwise.  When false is
>> > + * returned, detailed error will be in errp if provided.
>> > + */
>> > +static bool qmp_cmd_oob_check(Monitor *mon, QDict *req, Error **errp)
>> > +{
>> > +    const char *command;
>> > +    QmpCommand *cmd;
>> > +
>> > +    command = qdict_get_try_str(req, "execute");
>> > +    if (!command) {
>> > +        error_setg(errp, "Command field 'execute' missing");
>> > +        return false;
>> > +    }
>> > +
>> > +    cmd = qmp_find_command(mon->qmp.commands, command);
>> > +    if (!cmd) {
>> > +        error_set(errp, ERROR_CLASS_COMMAND_NOT_FOUND,
>> > +                  "The command %s has not been found", command);
>> > +        return false;
>> > +    }
>> > +
>> > +    if (qmp_is_oob(req)) {
>> > +        if (!qmp_oob_enabled(mon)) {
>> > +            error_setg(errp, "Please enable Out-Of-Band first "
>> > +                       "for the session during capabilities negociation");
>> > +            return false;
>> > +        }
>> > +        if (!(cmd->options & QCO_ALLOW_OOB)) {
>> > +            error_setg(errp, "The command %s does not support OOB",
>> > +                       command);
>> > +            return false;
>> > +        }
>> > +    }
>> > +
>> > +    return true;
>> > +}
>> > +
>> >  void qmp_qmp_capabilities(bool has_enable, QMPCapabilityList *enable,
>> >                            Error **errp)
>> >  {
>> > @@ -4001,6 +4039,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
>> >      QMPRequest *req_obj = monitor_qmp_requests_pop_one();
>> >
>> >      if (req_obj) {
>> > +        trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
>> >          monitor_qmp_dispatch_one(req_obj);
>> >          /* Reschedule instead of looping so the main loop stays responsive */
>> >          qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
>> > @@ -4024,17 +4063,31 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>> >          error_setg(&err, QERR_JSON_PARSING);
>> >      }
>> >      if (err) {
>> > -        monitor_qmp_respond(mon, NULL, err, NULL);
>> > -        qobject_decref(req);
>> > -        return;
>> > +        goto err;
>> > +    }
>> > +
>> > +    /* Check against the request in general layout */
>> > +    qdict = qmp_dispatch_check_obj(req, &err);
>> > +    if (!qdict) {
>> > +        goto err;
>> >      }
>> >
>> > -    qdict = qobject_to_qdict(req);
>> > -    if (qdict) {
>> > -        id = qdict_get(qdict, "id");
>> > -        qobject_incref(id);
>> > -        qdict_del(qdict, "id");
>> > -    } /* else will fail qmp_dispatch() */
>> > +    /* Check against OOB specific */
>> > +    if (!qmp_cmd_oob_check(mon, qdict, &err)) {
>> > +        goto err;
>> > +    }
>>
>> This check happens before monitor_qmp_dispatch_one(), where there is
>> now dead code for COMMAND_NOT_FOUND error handling (changing the error
>> description), since it will error here.
>>
>> A solution is to move the error manipulation code to monitor_qmp_respond():
>
> So yes, now it'll be checked earlier now since the allow-oob is bound
> to per command, then we'll need that info earlier here.  While I kept
> the other check untouched since there'll be other places that calls
> qmp_dispatch() too.
>
> I'm just wondering would there be any problem even without below change?

Not a problem, just less descriptive error. And we should remove the dead code.

It's best imho to apply the below change. I will send a patch.

> Thanks,
>
>>
>>
>> diff --git a/monitor.c b/monitor.c
>> index 52b4c77ac3..df09a1657d 100644
>> --- a/monitor.c
>> +++ b/monitor.c
>> @@ -3997,6 +3997,18 @@ static void monitor_qmp_respond(Monitor *mon,
>> QObject *rsp,
>>              qdict_put_obj(qobject_to(QDict, rsp), "id", id);
>>          }
>>
>> +        if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
>> +            qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
>> +            if (qdict
>> +                && !g_strcmp0(qdict_get_try_str(qdict, "class"),
>> +
>> QapiErrorClass_str(ERROR_CLASS_COMMAND_NOT_FOUND))) {
>> +                /* Provide a more useful error message */
>> +                qdict_del(qdict, "desc");
>> +                qdict_put_str(qdict, "desc", "Expecting capabilities
>> negotiation"
>> +                              " with 'qmp_capabilities'");
>> +            }
>> +        }
>> +
>>          monitor_json_emitter(mon, rsp);
>>      }
>>
>> @@ -4022,7 +4034,6 @@ static void monitor_qmp_dispatch_one(QMPRequest
>> *req_obj, bool oob_cmd)
>>  {
>>      Monitor *mon, *old_mon;
>>      QObject *req, *rsp = NULL, *id;
>> -    QDict *qdict = NULL;
>>      bool need_resume;
>>
>>      req = req_obj->req;
>> @@ -4045,18 +4056,6 @@ static void monitor_qmp_dispatch_one(QMPRequest
>> *req_obj, bool oob_cmd)
>>
>>      cur_mon = old_mon;
>>
>> -    if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
>> -        qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
>> -        if (qdict
>> -            && !g_strcmp0(qdict_get_try_str(qdict, "class"),
>> -                    QapiErrorClass_str(ERROR_CLASS_COMMAND_NOT_FOUND))) {
>> -            /* Provide a more useful error message */
>> -            qdict_del(qdict, "desc");
>> -            qdict_put_str(qdict, "desc", "Expecting capabilities negotiation"
>> -                          " with 'qmp_capabilities'");
>> -        }
>> -    }
>> -
>>      /* Respond if necessary */
>>      monitor_qmp_respond(mon, rsp, NULL, id);
>>
>>
>> > +
>> > +    id = qdict_get(qdict, "id");
>> > +
>> > +    /* When OOB is enabled, the "id" field is mandatory. */
>> > +    if (qmp_oob_enabled(mon) && !id) {
>> > +        error_setg(&err, "Out-Of-Band capability requires that "
>> > +                   "every command contains an 'id' field");
>> > +        goto err;
>> > +    }
>> > +
>> > +    qobject_incref(id);
>> > +    qdict_del(qdict, "id");
>> >
>> >      req_obj = g_new0(QMPRequest, 1);
>> >      req_obj->mon = mon;
>> > @@ -4042,6 +4095,14 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>> >      req_obj->req = req;
>> >      req_obj->need_resume = false;
>> >
>> > +    if (qmp_is_oob(qdict)) {
>> > +        /* Out-Of-Band (OOB) requests are executed directly in parser. */
>> > +        trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
>> > +                                          ?: "");
>> > +        monitor_qmp_dispatch_one(req_obj);
>> > +        return;
>> > +    }
>> > +
>> >      /* Protect qmp_requests and fetching its length. */
>> >      qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
>> >
>> > @@ -4078,6 +4139,11 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>> >
>> >      /* Kick the dispatcher routine */
>> >      qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
>> > +    return;
>> > +
>> > +err:
>> > +    monitor_qmp_respond(mon, NULL, err, NULL);
>> > +    qobject_decref(req);
>> >  }
>> >
>> >  static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
>> > diff --git a/qapi/qmp-dispatch.c b/qapi/qmp-dispatch.c
>> > index e31ac4be1f..9a2e18c768 100644
>> > --- a/qapi/qmp-dispatch.c
>> > +++ b/qapi/qmp-dispatch.c
>> > @@ -17,8 +17,9 @@
>> >  #include "qapi/qmp/json-parser.h"
>> >  #include "qapi/qmp/qdict.h"
>> >  #include "qapi/qmp/qjson.h"
>> > +#include "qapi/qmp/qbool.h"
>> >
>> > -static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
>> > +QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
>> >  {
>> >      const QDictEntry *ent;
>> >      const char *arg_name;
>> > @@ -50,6 +51,14 @@ static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
>> >                             "QMP input member 'arguments' must be an object");
>> >                  return NULL;
>> >              }
>> > +        } else if (!strcmp(arg_name, "id")) {
>> > +            continue;
>> > +        } else if (!strcmp(arg_name, "control")) {
>> > +            if (qobject_type(arg_obj) != QTYPE_QDICT) {
>> > +                error_setg(errp,
>> > +                           "QMP input member 'control' must be a dict");
>> > +                return NULL;
>> > +            }
>> >          } else {
>> >              error_setg(errp, "QMP input member '%s' is unexpected",
>> >                         arg_name);
>> > @@ -120,6 +129,28 @@ QObject *qmp_build_error_object(Error *err)
>> >                                error_get_pretty(err));
>> >  }
>> >
>> > +/*
>> > + * Detect whether a request should be run out-of-band, by quickly
>> > + * peeking at whether we have: { "control": { "run-oob": True } }. By
>> > + * default commands are run in-band.
>> > + */
>> > +bool qmp_is_oob(QDict *dict)
>> > +{
>> > +    QBool *bool_obj;
>> > +
>> > +    dict = qdict_get_qdict(dict, "control");
>> > +    if (!dict) {
>> > +        return false;
>> > +    }
>> > +
>> > +    bool_obj = qobject_to_qbool(qdict_get(dict, "run-oob"));
>> > +    if (!bool_obj) {
>> > +        return false;
>> > +    }
>> > +
>> > +    return qbool_get_bool(bool_obj);
>> > +}
>> > +
>> >  QObject *qmp_dispatch(QmpCommandList *cmds, QObject *request)
>> >  {
>> >      Error *err = NULL;
>> > diff --git a/trace-events b/trace-events
>> > index fe10c3b487..bd036da278 100644
>> > --- a/trace-events
>> > +++ b/trace-events
>> > @@ -48,6 +48,8 @@ monitor_protocol_event_queue(uint32_t event, void *qdict, uint64_t rate) "event=
>> >  handle_hmp_command(void *mon, const char *cmdline) "mon %p cmdline: %s"
>> >  handle_qmp_command(void *mon, const char *req) "mon %p req: %s"
>> >  monitor_suspend(void *ptr, int cnt) "mon %p: %d"
>> > +monitor_qmp_cmd_in_band(const char *id) "%s"
>> > +monitor_qmp_cmd_out_of_band(const char *id) "%s"
>> >
>> >  # dma-helpers.c
>> >  dma_blk_io(void *dbs, void *bs, int64_t offset, bool to_dev) "dbs=%p bs=%p offset=%" PRId64 " to_dev=%d"
>> > --
>> > 2.14.3
>> >
>> >
>>
>>
>>
>> --
>> Marc-André Lureau
>
> --
> Peter Xu



-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v8 20/23] monitor: enable IO thread for (qmp & !mux) typed

[Christian Borntraeger]

As Max Reitz said, this breaks several qemu iotests. I can reproduce that on s390
e.g. with ./check -qcow2 030 in the qemu-iotest.

Why was this merged?


On 03/09/2018 10:00 AM, Peter Xu wrote:
> Start to use dedicate IO thread for QMP monitors that are not using
> MUXed chardev.
> 
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 5 ++++-
>  1 file changed, 4 insertions(+), 1 deletion(-)
> 
> diff --git a/monitor.c b/monitor.c
> index c6de5f123e..9e8ee2ed14 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -36,6 +36,7 @@
>  #include "net/slirp.h"
>  #include "chardev/char-fe.h"
>  #include "chardev/char-io.h"
> +#include "chardev/char-mux.h"
>  #include "ui/qemu-spice.h"
>  #include "sysemu/numa.h"
>  #include "monitor/monitor.h"
> @@ -4533,8 +4534,10 @@ static void monitor_qmp_setup_handlers_bh(void *opaque)
>  void monitor_init(Chardev *chr, int flags)
>  {
>      Monitor *mon = g_malloc(sizeof(*mon));
> +    /* Enable IOThread for QMPs that are not using MUX chardev backends. */
> +    bool use_io_thr = (!CHARDEV_IS_MUX(chr)) && (flags & MONITOR_USE_CONTROL);
> 
> -    monitor_data_init(mon, false, false);
> +    monitor_data_init(mon, false, use_io_thr);
> 
>      qemu_chr_fe_init(&mon->chr, chr, &error_abort);
>      mon->flags = flags;
> 

Re: [Qemu-devel] [PATCH v8 20/23] monitor: enable IO thread for (qmp & !mux) typed

[Peter Xu]

On Fri, Mar 23, 2018 at 01:10:51PM +0100, Christian Borntraeger wrote:
> As Max Reitz said, this breaks several qemu iotests. I can reproduce that on s390
> e.g. with ./check -qcow2 030 in the qemu-iotest.
> 
> Why was this merged?

My fault.  Sorry.  I should have done iotests before submission.

Now I'm trying to fix those iotest issues.

I'm not extremely familiar with block, so the progress is a bit slow,
but I'll do it asap (though I'll possibly need some help from the
block team, since there are some not-easy ones for me).

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 20/23] monitor: enable IO thread for (qmp & !mux) typed

[Christian Borntraeger]

On 03/23/2018 01:25 PM, Peter Xu wrote:
> On Fri, Mar 23, 2018 at 01:10:51PM +0100, Christian Borntraeger wrote:
>> As Max Reitz said, this breaks several qemu iotests. I can reproduce that on s390
>> e.g. with ./check -qcow2 030 in the qemu-iotest.
>>
>> Why was this merged?
> 
> My fault.  Sorry.  I should have done iotests before submission.
> 
> Now I'm trying to fix those iotest issues.
> 
> I'm not extremely familiar with block, so the progress is a bit slow,
> but I'll do it asap (though I'll possibly need some help from the
> block team, since there are some not-easy ones for me).

As this patch also seems to trigger something else for Eric, can we revert this
patch for the time being and re-apply as soon as we have solutions for the issues?

After all a hard hang in the iotest can break CI environments (as you run into timeouts).

Re: [Qemu-devel] [PATCH v8 20/23] monitor: enable IO thread for (qmp & !mux) typed

[Peter Xu]

On Fri, Mar 23, 2018 at 01:44:54PM +0100, Christian Borntraeger wrote:
> 
> 
> On 03/23/2018 01:25 PM, Peter Xu wrote:
> > On Fri, Mar 23, 2018 at 01:10:51PM +0100, Christian Borntraeger wrote:
> >> As Max Reitz said, this breaks several qemu iotests. I can reproduce that on s390
> >> e.g. with ./check -qcow2 030 in the qemu-iotest.
> >>
> >> Why was this merged?
> > 
> > My fault.  Sorry.  I should have done iotests before submission.
> > 
> > Now I'm trying to fix those iotest issues.
> > 
> > I'm not extremely familiar with block, so the progress is a bit slow,
> > but I'll do it asap (though I'll possibly need some help from the
> > block team, since there are some not-easy ones for me).
> 
> As this patch also seems to trigger something else for Eric, can we revert this
> patch for the time being and re-apply as soon as we have solutions for the issues?
> 
> After all a hard hang in the iotest can break CI environments (as you run into timeouts).

I'm fine with it if it helps.

Then I can work in background to solved existing problems, then we
turn it on again in 2.13 when ready (but still it might break things -
at least it can be during dev cycle rather than softfreeze).

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 20/23] monitor: enable IO thread for (qmp & !mux) typed

[Peter Maydell]

On 23 March 2018 at 13:01, Peter Xu <peterx@redhat.com> wrote:
> On Fri, Mar 23, 2018 at 01:44:54PM +0100, Christian Borntraeger wrote:
>>
>>
>> On 03/23/2018 01:25 PM, Peter Xu wrote:
>> > On Fri, Mar 23, 2018 at 01:10:51PM +0100, Christian Borntraeger wrote:
>> >> As Max Reitz said, this breaks several qemu iotests. I can reproduce that on s390
>> >> e.g. with ./check -qcow2 030 in the qemu-iotest.
>> >>
>> >> Why was this merged?
>> >
>> > My fault.  Sorry.  I should have done iotests before submission.
>> >
>> > Now I'm trying to fix those iotest issues.
>> >
>> > I'm not extremely familiar with block, so the progress is a bit slow,
>> > but I'll do it asap (though I'll possibly need some help from the
>> > block team, since there are some not-easy ones for me).
>>
>> As this patch also seems to trigger something else for Eric, can we revert this
>> patch for the time being and re-apply as soon as we have solutions for the issues?
>>
>> After all a hard hang in the iotest can break CI environments (as you run into timeouts).
>
> I'm fine with it if it helps.

OK. I'm running a revert of commit 3fd2457d18edf5736f713dfe1ada9c87a9badab1
through my buildtests and will push it to master assuming it passes.

thanks
-- PMM

Re: [Qemu-devel] [PATCH v8 20/23] monitor: enable IO thread for (qmp & !mux) typed

[Christian Borntraeger]

On 03/23/2018 02:01 PM, Peter Xu wrote:
> On Fri, Mar 23, 2018 at 01:44:54PM +0100, Christian Borntraeger wrote:
>>
>>
>> On 03/23/2018 01:25 PM, Peter Xu wrote:
>>> On Fri, Mar 23, 2018 at 01:10:51PM +0100, Christian Borntraeger wrote:
>>>> As Max Reitz said, this breaks several qemu iotests. I can reproduce that on s390
>>>> e.g. with ./check -qcow2 030 in the qemu-iotest.
>>>>
>>>> Why was this merged?
>>>
>>> My fault.  Sorry.  I should have done iotests before submission.
>>>
>>> Now I'm trying to fix those iotest issues.
>>>
>>> I'm not extremely familiar with block, so the progress is a bit slow,
>>> but I'll do it asap (though I'll possibly need some help from the
>>> block team, since there are some not-easy ones for me).
>>
>> As this patch also seems to trigger something else for Eric, can we revert this
>> patch for the time being and re-apply as soon as we have solutions for the issues?
>>
>> After all a hard hang in the iotest can break CI environments (as you run into timeouts).
> 
> I'm fine with it if it helps.
> 
> Then I can work in background to solved existing problems, then we
> turn it on again in 2.13 when ready (but still it might break things -
> at least it can be during dev cycle rather than softfreeze).

Strangely enough, reverting 3fd2457d18edf5736f713dfe1
now causes make check errors... :-/


  GTESTER check-qtest-s390x
  GTESTER check-qtest-i386
**
ERROR:/home/cborntra/REPOS/qemu/tests/qmp-test.c:97:test_qmp_protocol: assertion failed: (entry)
GTester: last random seed: R02S1cbb7e297cc349b33ae39f10dd92764a
**
ERROR:/home/cborntra/REPOS/qemu/tests/qmp-test.c:190:test_qmp_oob: assertion failed: (qdict_haskey(resp, "return"))
GTester: last random seed: R02S7afc5ab63f6f29401d702dceb1f8030b
make: *** [/home/cborntra/REPOS/qemu/tests/Makefile.include:880: check-qtest-s390x] Error 1
make: *** Waiting for unfinished jobs....
^Cmake: *** [/home/cborntra/REPOS/qemu/tests/Makefile.include:880: check-qtest-i386] Interrupt


Do we have patches depending on this change?

Re: [Qemu-devel] [PATCH v8 20/23] monitor: enable IO thread for (qmp & !mux) typed

[Peter Xu]

On Fri, Mar 23, 2018 at 02:23:27PM +0100, Christian Borntraeger wrote:
> 
> 
> On 03/23/2018 02:01 PM, Peter Xu wrote:
> > On Fri, Mar 23, 2018 at 01:44:54PM +0100, Christian Borntraeger wrote:
> >>
> >>
> >> On 03/23/2018 01:25 PM, Peter Xu wrote:
> >>> On Fri, Mar 23, 2018 at 01:10:51PM +0100, Christian Borntraeger wrote:
> >>>> As Max Reitz said, this breaks several qemu iotests. I can reproduce that on s390
> >>>> e.g. with ./check -qcow2 030 in the qemu-iotest.
> >>>>
> >>>> Why was this merged?
> >>>
> >>> My fault.  Sorry.  I should have done iotests before submission.
> >>>
> >>> Now I'm trying to fix those iotest issues.
> >>>
> >>> I'm not extremely familiar with block, so the progress is a bit slow,
> >>> but I'll do it asap (though I'll possibly need some help from the
> >>> block team, since there are some not-easy ones for me).
> >>
> >> As this patch also seems to trigger something else for Eric, can we revert this
> >> patch for the time being and re-apply as soon as we have solutions for the issues?
> >>
> >> After all a hard hang in the iotest can break CI environments (as you run into timeouts).
> > 
> > I'm fine with it if it helps.
> > 
> > Then I can work in background to solved existing problems, then we
> > turn it on again in 2.13 when ready (but still it might break things -
> > at least it can be during dev cycle rather than softfreeze).
> 
> Strangely enough, reverting 3fd2457d18edf5736f713dfe1
> now causes make check errors... :-/
> 
> 
>   GTESTER check-qtest-s390x
>   GTESTER check-qtest-i386
> **
> ERROR:/home/cborntra/REPOS/qemu/tests/qmp-test.c:97:test_qmp_protocol: assertion failed: (entry)
> GTester: last random seed: R02S1cbb7e297cc349b33ae39f10dd92764a
> **
> ERROR:/home/cborntra/REPOS/qemu/tests/qmp-test.c:190:test_qmp_oob: assertion failed: (qdict_haskey(resp, "return"))
> GTester: last random seed: R02S7afc5ab63f6f29401d702dceb1f8030b
> make: *** [/home/cborntra/REPOS/qemu/tests/Makefile.include:880: check-qtest-s390x] Error 1
> make: *** Waiting for unfinished jobs....
> ^Cmake: *** [/home/cborntra/REPOS/qemu/tests/Makefile.include:880: check-qtest-i386] Interrupt
> 
> 
> Do we have patches depending on this change?

It's the new OOB tests that's failing.  I'm preparing a patchset that
reverts the enablement patch, along with those tests.

Now I'm testing that branch with more platforms and also all
raw+qcow2 iotests.  After that's solid, I'll post that out soon if it
goes smoothly.

Hopefully that would fix all problems so far.

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Marc-André Lureau]

On Fri, Mar 9, 2018 at 9:59 AM, Peter Xu <peterx@redhat.com> wrote:
> Originally QMP goes through these steps:
>
>   JSON Parser --> QMP Dispatcher --> Respond
>       /|\    (2)                (3)     |
>    (1) |                               \|/ (4)
>        +---------  main thread  --------+
>
> This patch does this:
>
>   JSON Parser     QMP Dispatcher --> Respond
>       /|\ |           /|\       (4)     |
>        |  | (2)        | (3)            |  (5)
>    (1) |  +----->      |               \|/
>        +---------  main thread  <-------+
>
> So the parsing job and the dispatching job is isolated now.  It gives us
> a chance in following up patches to totally move the parser outside.
>
> The isolation is done using one QEMUBH. Only one dispatcher QEMUBH is
> used for all the monitors.
>
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 201 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-------
>  1 file changed, 178 insertions(+), 23 deletions(-)
>
> diff --git a/monitor.c b/monitor.c
> index de9343be87..5104e5db07 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -172,6 +172,13 @@ typedef struct {
>       */
>      QmpCommandList *commands;
>      bool qmp_caps[QMP_CAPABILITY__MAX];
> +    /*
> +     * Protects qmp request/response queue.  Please take monitor_lock
> +     * first when used together.
> +     */
> +    QemuMutex qmp_queue_lock;
> +    /* Input queue that holds all the parsed QMP requests */
> +    GQueue *qmp_requests;
>  } MonitorQMP;
>
>  /*
> @@ -218,6 +225,8 @@ struct Monitor {
>  /* Let's add monitor global variables to this struct. */
>  static struct {
>      IOThread *mon_iothread;
> +    /* Bottom half to dispatch the requests received from IO thread */
> +    QEMUBH *qmp_dispatcher_bh;
>  } mon_global;
>
>  /* QMP checker flags */
> @@ -600,11 +609,13 @@ static void monitor_data_init(Monitor *mon, bool skip_flush,
>  {
>      memset(mon, 0, sizeof(Monitor));
>      qemu_mutex_init(&mon->out_lock);
> +    qemu_mutex_init(&mon->qmp.qmp_queue_lock);
>      mon->outbuf = qstring_new();
>      /* Use *mon_cmds by default. */
>      mon->cmd_table = mon_cmds;
>      mon->skip_flush = skip_flush;
>      mon->use_io_thr = use_io_thr;
> +    mon->qmp.qmp_requests = g_queue_new();
>  }
>
>  static void monitor_data_destroy(Monitor *mon)
> @@ -617,6 +628,8 @@ static void monitor_data_destroy(Monitor *mon)
>      readline_free(mon->rs);
>      QDECREF(mon->outbuf);
>      qemu_mutex_destroy(&mon->out_lock);
> +    qemu_mutex_destroy(&mon->qmp.qmp_queue_lock);
> +    g_queue_free(mon->qmp.qmp_requests);
>  }
>
>  char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
> @@ -1056,6 +1069,16 @@ static void monitor_init_qmp_commands(void)
>                           qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
>  }
>
> +static bool qmp_cap_enabled(Monitor *mon, QMPCapability cap)
> +{
> +    return mon->qmp.qmp_caps[cap];
> +}
> +
> +static bool qmp_oob_enabled(Monitor *mon)
> +{
> +    return qmp_cap_enabled(mon, QMP_CAPABILITY_OOB);
> +}
> +
>  static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
>                             Error **errp)
>  {
> @@ -3866,30 +3889,39 @@ static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
>      qobject_decref(rsp);
>  }
>
> -static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> +struct QMPRequest {
> +    /* Owner of the request */
> +    Monitor *mon;
> +    /* "id" field of the request */
> +    QObject *id;
> +    /* Request object to be handled */
> +    QObject *req;
> +    /*
> +     * Whether we need to resume the monitor afterward.  This flag is
> +     * used to emulate the old QMP server behavior that the current
> +     * command must be completed before execution of the next one.
> +     */
> +    bool need_resume;
> +};
> +typedef struct QMPRequest QMPRequest;
> +
> +/*
> + * Dispatch one single QMP request. The function will free the req_obj
> + * and objects inside it before return.
> + */
> +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
>  {
> -    QObject *req, *rsp = NULL, *id = NULL;
> +    Monitor *mon, *old_mon;
> +    QObject *req, *rsp = NULL, *id;
>      QDict *qdict = NULL;
> -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
> -
> -    Error *err = NULL;
> +    bool need_resume;
>
> -    req = json_parser_parse_err(tokens, NULL, &err);
> -    if (!req && !err) {
> -        /* json_parser_parse_err() sucks: can fail without setting @err */
> -        error_setg(&err, QERR_JSON_PARSING);
> -    }
> -    if (err) {
> -        goto err_out;
> -    }
> +    req = req_obj->req;
> +    mon = req_obj->mon;
> +    id = req_obj->id;
> +    need_resume = req_obj->need_resume;
>
> -    qdict = qobject_to_qdict(req);
> -    if (qdict) {
> -        id = qdict_get(qdict, "id");
> -        qobject_incref(id);
> -        qdict_del(qdict, "id");
> -    } /* else will fail qmp_dispatch() */
> +    g_free(req_obj);
>
>      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
>          QString *req_json = qobject_to_json(req);
> @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>      old_mon = cur_mon;
>      cur_mon = mon;

There is another issue with this series, since cur_mon is global (and
not protected), an oob command may change the cur_mon while another
command is running in the main thread with unexpected consequences. I
don't have a clear idea what is the best way to solve it. Making the
variable per-thread, or going all the way to get rid of cur_mon (my
preference, but much harder)

>
> -    rsp = qmp_dispatch(cur_mon->qmp.commands, req);
> +    rsp = qmp_dispatch(mon->qmp.commands, req);
>
>      cur_mon = old_mon;
>
> @@ -3916,12 +3948,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>          }
>      }
>
> -err_out:
> -    monitor_qmp_respond(mon, rsp, err, id);
> +    /* Respond if necessary */
> +    monitor_qmp_respond(mon, rsp, NULL, id);
> +
> +    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
> +    if (need_resume) {
> +        monitor_resume(mon);
> +    }
>
>      qobject_decref(req);
>  }
>
> +/*
> + * Pop one QMP request from monitor queues, return NULL if not found.
> + * We are using round-robin fashion to pop the request, to avoid
> + * processing command only on a very busy monitor.  To achieve that,
> + * when we processed one request on specific monitor, we put that
> + * monitor to the end of mon_list queue.
> + */
> +static QMPRequest *monitor_qmp_requests_pop_one(void)
> +{
> +    QMPRequest *req_obj = NULL;
> +    Monitor *mon;
> +
> +    qemu_mutex_lock(&monitor_lock);
> +
> +    QTAILQ_FOREACH(mon, &mon_list, entry) {
> +        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> +        req_obj = g_queue_pop_head(mon->qmp.qmp_requests);
> +        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> +        if (req_obj) {
> +            break;
> +        }
> +    }
> +
> +    if (req_obj) {
> +        /*
> +         * We found one request on the monitor. Degrade this monitor's
> +         * priority to lowest by re-inserting it to end of queue.
> +         */
> +        QTAILQ_REMOVE(&mon_list, mon, entry);
> +        QTAILQ_INSERT_TAIL(&mon_list, mon, entry);
> +    }
> +
> +    qemu_mutex_unlock(&monitor_lock);
> +
> +    return req_obj;
> +}
> +
> +static void monitor_qmp_bh_dispatcher(void *data)
> +{
> +    QMPRequest *req_obj = monitor_qmp_requests_pop_one();
> +
> +    if (req_obj) {
> +        monitor_qmp_dispatch_one(req_obj);
> +        /* Reschedule instead of looping so the main loop stays responsive */
> +        qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> +    }
> +}
> +
> +static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> +{
> +    QObject *req, *id = NULL;
> +    QDict *qdict = NULL;
> +    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> +    Monitor *mon = container_of(mon_qmp, Monitor, qmp);
> +    Error *err = NULL;
> +    QMPRequest *req_obj;
> +
> +    req = json_parser_parse_err(tokens, NULL, &err);
> +    if (!req && !err) {
> +        /* json_parser_parse_err() sucks: can fail without setting @err */
> +        error_setg(&err, QERR_JSON_PARSING);
> +    }
> +    if (err) {
> +        monitor_qmp_respond(mon, NULL, err, NULL);
> +        qobject_decref(req);
> +        return;
> +    }
> +
> +    qdict = qobject_to_qdict(req);
> +    if (qdict) {
> +        id = qdict_get(qdict, "id");
> +        qobject_incref(id);
> +        qdict_del(qdict, "id");
> +    } /* else will fail qmp_dispatch() */
> +
> +    req_obj = g_new0(QMPRequest, 1);
> +    req_obj->mon = mon;
> +    req_obj->id = id;
> +    req_obj->req = req;
> +    req_obj->need_resume = false;
> +
> +    /*
> +     * If OOB is not enabled on current monitor, we'll emulate the old
> +     * behavior that we won't process current monitor any more until
> +     * it is responded.  This helps make sure that as long as OOB is
> +     * not enabled, the server will never drop any command.
> +     */
> +    if (!qmp_oob_enabled(mon)) {
> +        monitor_suspend(mon);
> +        req_obj->need_resume = true;
> +    }
> +
> +    /*
> +     * Put the request to the end of queue so that requests will be
> +     * handled in time order.  Ownership for req_obj, req, id,
> +     * etc. will be delivered to the handler side.
> +     */
> +    qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> +    g_queue_push_tail(mon->qmp.qmp_requests, req_obj);
> +    qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> +
> +    /* Kick the dispatcher routine */
> +    qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> +}
> +
>  static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
>  {
>      Monitor *mon = opaque;
> @@ -4134,6 +4276,15 @@ static void monitor_iothread_init(void)
>  {
>      mon_global.mon_iothread = iothread_create("mon_iothread",
>                                                &error_abort);
> +
> +    /*
> +     * This MUST be on main loop thread since we have commands that
> +     * have assumption to be run on main loop thread.  It would be
> +     * nice that one day we can remove this assumption in the future.
> +     */
> +    mon_global.qmp_dispatcher_bh = aio_bh_new(qemu_get_aio_context(),
> +                                              monitor_qmp_bh_dispatcher,
> +                                              NULL);
>  }
>
>  void monitor_init_globals(void)
> @@ -4280,6 +4431,10 @@ void monitor_cleanup(void)
>      }
>      qemu_mutex_unlock(&monitor_lock);
>
> +    /* QEMUBHs needs to be deleted before destroying the IOThread. */
> +    qemu_bh_delete(mon_global.qmp_dispatcher_bh);
> +    mon_global.qmp_dispatcher_bh = NULL;
> +
>      iothread_destroy(mon_global.mon_iothread);
>      mon_global.mon_iothread = NULL;
>  }
> --
> 2.14.3
>
>



-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Fri, Mar 23, 2018 at 05:18:53PM +0100, Marc-André Lureau wrote:

[...]

> > +/*
> > + * Dispatch one single QMP request. The function will free the req_obj
> > + * and objects inside it before return.
> > + */
> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> >  {
> > -    QObject *req, *rsp = NULL, *id = NULL;
> > +    Monitor *mon, *old_mon;
> > +    QObject *req, *rsp = NULL, *id;
> >      QDict *qdict = NULL;
> > -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> > -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
> > -
> > -    Error *err = NULL;
> > +    bool need_resume;
> >
> > -    req = json_parser_parse_err(tokens, NULL, &err);
> > -    if (!req && !err) {
> > -        /* json_parser_parse_err() sucks: can fail without setting @err */
> > -        error_setg(&err, QERR_JSON_PARSING);
> > -    }
> > -    if (err) {
> > -        goto err_out;
> > -    }
> > +    req = req_obj->req;
> > +    mon = req_obj->mon;
> > +    id = req_obj->id;
> > +    need_resume = req_obj->need_resume;
> >
> > -    qdict = qobject_to_qdict(req);
> > -    if (qdict) {
> > -        id = qdict_get(qdict, "id");
> > -        qobject_incref(id);
> > -        qdict_del(qdict, "id");
> > -    } /* else will fail qmp_dispatch() */
> > +    g_free(req_obj);
> >
> >      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
> >          QString *req_json = qobject_to_json(req);
> > @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >      old_mon = cur_mon;
> >      cur_mon = mon;
> 
> There is another issue with this series, since cur_mon is global (and
> not protected), an oob command may change the cur_mon while another
> command is running in the main thread with unexpected consequences. I
> don't have a clear idea what is the best way to solve it. Making the
> variable per-thread, or going all the way to get rid of cur_mon (my
> preference, but much harder)

IMHO it is fine too.

Note that this cur_mon operation is in monitor_qmp_dispatch_one() now,
which is still running in main thread.  So AFAICT all the cur_mon
references are in main thread, and monitor IOThread does not modify
that variable at all.  Then we should probably be safe.

I would be far more than glad to see cur_mon go away one day.

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Marc-André Lureau]

Hi

On Mon, Mar 26, 2018 at 10:07 AM, Peter Xu <peterx@redhat.com> wrote:
> On Fri, Mar 23, 2018 at 05:18:53PM +0100, Marc-André Lureau wrote:
>
> [...]
>
>> > +/*
>> > + * Dispatch one single QMP request. The function will free the req_obj
>> > + * and objects inside it before return.
>> > + */
>> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
>> >  {
>> > -    QObject *req, *rsp = NULL, *id = NULL;
>> > +    Monitor *mon, *old_mon;
>> > +    QObject *req, *rsp = NULL, *id;
>> >      QDict *qdict = NULL;
>> > -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
>> > -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
>> > -
>> > -    Error *err = NULL;
>> > +    bool need_resume;
>> >
>> > -    req = json_parser_parse_err(tokens, NULL, &err);
>> > -    if (!req && !err) {
>> > -        /* json_parser_parse_err() sucks: can fail without setting @err */
>> > -        error_setg(&err, QERR_JSON_PARSING);
>> > -    }
>> > -    if (err) {
>> > -        goto err_out;
>> > -    }
>> > +    req = req_obj->req;
>> > +    mon = req_obj->mon;
>> > +    id = req_obj->id;
>> > +    need_resume = req_obj->need_resume;
>> >
>> > -    qdict = qobject_to_qdict(req);
>> > -    if (qdict) {
>> > -        id = qdict_get(qdict, "id");
>> > -        qobject_incref(id);
>> > -        qdict_del(qdict, "id");
>> > -    } /* else will fail qmp_dispatch() */
>> > +    g_free(req_obj);
>> >
>> >      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
>> >          QString *req_json = qobject_to_json(req);
>> > @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>> >      old_mon = cur_mon;
>> >      cur_mon = mon;
>>
>> There is another issue with this series, since cur_mon is global (and
>> not protected), an oob command may change the cur_mon while another
>> command is running in the main thread with unexpected consequences. I
>> don't have a clear idea what is the best way to solve it. Making the
>> variable per-thread, or going all the way to get rid of cur_mon (my
>> preference, but much harder)
>
> IMHO it is fine too.
>
> Note that this cur_mon operation is in monitor_qmp_dispatch_one() now,
> which is still running in main thread.  So AFAICT all the cur_mon
> references are in main thread, and monitor IOThread does not modify
> that variable at all.  Then we should probably be safe.

But monitor_qmp_dispatch_one() is called from iothread if the command
is oob, so cur_mon may be updated while another command is running in
main thread, or am I wrong?

>
> I would be far more than glad to see cur_mon go away one day.
>
> Thanks,
>
> --
> Peter Xu



-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Mon, Mar 26, 2018 at 10:33:27AM +0200, Marc-André Lureau wrote:
> Hi
> 
> On Mon, Mar 26, 2018 at 10:07 AM, Peter Xu <peterx@redhat.com> wrote:
> > On Fri, Mar 23, 2018 at 05:18:53PM +0100, Marc-André Lureau wrote:
> >
> > [...]
> >
> >> > +/*
> >> > + * Dispatch one single QMP request. The function will free the req_obj
> >> > + * and objects inside it before return.
> >> > + */
> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> >> >  {
> >> > -    QObject *req, *rsp = NULL, *id = NULL;
> >> > +    Monitor *mon, *old_mon;
> >> > +    QObject *req, *rsp = NULL, *id;
> >> >      QDict *qdict = NULL;
> >> > -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> >> > -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
> >> > -
> >> > -    Error *err = NULL;
> >> > +    bool need_resume;
> >> >
> >> > -    req = json_parser_parse_err(tokens, NULL, &err);
> >> > -    if (!req && !err) {
> >> > -        /* json_parser_parse_err() sucks: can fail without setting @err */
> >> > -        error_setg(&err, QERR_JSON_PARSING);
> >> > -    }
> >> > -    if (err) {
> >> > -        goto err_out;
> >> > -    }
> >> > +    req = req_obj->req;
> >> > +    mon = req_obj->mon;
> >> > +    id = req_obj->id;
> >> > +    need_resume = req_obj->need_resume;
> >> >
> >> > -    qdict = qobject_to_qdict(req);
> >> > -    if (qdict) {
> >> > -        id = qdict_get(qdict, "id");
> >> > -        qobject_incref(id);
> >> > -        qdict_del(qdict, "id");
> >> > -    } /* else will fail qmp_dispatch() */
> >> > +    g_free(req_obj);
> >> >
> >> >      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
> >> >          QString *req_json = qobject_to_json(req);
> >> > @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >> >      old_mon = cur_mon;
> >> >      cur_mon = mon;
> >>
> >> There is another issue with this series, since cur_mon is global (and
> >> not protected), an oob command may change the cur_mon while another
> >> command is running in the main thread with unexpected consequences. I
> >> don't have a clear idea what is the best way to solve it. Making the
> >> variable per-thread, or going all the way to get rid of cur_mon (my
> >> preference, but much harder)
> >
> > IMHO it is fine too.
> >
> > Note that this cur_mon operation is in monitor_qmp_dispatch_one() now,
> > which is still running in main thread.  So AFAICT all the cur_mon
> > references are in main thread, and monitor IOThread does not modify
> > that variable at all.  Then we should probably be safe.
> 
> But monitor_qmp_dispatch_one() is called from iothread if the command
> is oob, so cur_mon may be updated while another command is running in
> main thread, or am I wrong?

You are right. I missed that, sorry...

Would this be a simple workaround (but hopefully efficient) solution?

diff --git a/monitor.c b/monitor.c
index 77f4c41cfa..99641c0c6d 100644
--- a/monitor.c
+++ b/monitor.c
@@ -4023,7 +4023,7 @@ typedef struct QMPRequest QMPRequest;
  * Dispatch one single QMP request. The function will free the req_obj
  * and objects inside it before return.
  */
-static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
+static void monitor_qmp_dispatch_one(QMPRequest *req_obj, bool hack_curmon)
 {
     Monitor *mon, *old_mon;
     QObject *req, *rsp = NULL, *id;
@@ -4043,12 +4043,16 @@ static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
         QDECREF(req_json);
     }

-    old_mon = cur_mon;
-    cur_mon = mon;
+    if (hack_curmon) {
+        old_mon = cur_mon;
+        cur_mon = mon;
+    }

     rsp = qmp_dispatch(mon->qmp.commands, req);

-    cur_mon = old_mon;
+    if (hack_curmon) {
+        cur_mon = old_mon;
+    }

     if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
         qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
@@ -4116,7 +4120,7 @@ static void monitor_qmp_bh_dispatcher(void *data)

     if (req_obj) {
         trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
-        monitor_qmp_dispatch_one(req_obj);
+        monitor_qmp_dispatch_one(req_obj, true);
         /* Reschedule instead of looping so the main loop stays responsive */
         qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
     }
@@ -4175,7 +4179,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
         /* Out-Of-Band (OOB) requests are executed directly in parser. */
         trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
                                           ?: "");
-        monitor_qmp_dispatch_one(req_obj);
+        monitor_qmp_dispatch_one(req_obj, false);
         return;
     }

Then we forbit touching that evil cur_mon in OOB-capable command
handlers.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Marc-André Lureau]

Hi

On Mon, Mar 26, 2018 at 11:08 AM, Peter Xu <peterx@redhat.com> wrote:
> On Mon, Mar 26, 2018 at 10:33:27AM +0200, Marc-André Lureau wrote:
>> Hi
>>
>> On Mon, Mar 26, 2018 at 10:07 AM, Peter Xu <peterx@redhat.com> wrote:
>> > On Fri, Mar 23, 2018 at 05:18:53PM +0100, Marc-André Lureau wrote:
>> >
>> > [...]
>> >
>> >> > +/*
>> >> > + * Dispatch one single QMP request. The function will free the req_obj
>> >> > + * and objects inside it before return.
>> >> > + */
>> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
>> >> >  {
>> >> > -    QObject *req, *rsp = NULL, *id = NULL;
>> >> > +    Monitor *mon, *old_mon;
>> >> > +    QObject *req, *rsp = NULL, *id;
>> >> >      QDict *qdict = NULL;
>> >> > -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
>> >> > -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
>> >> > -
>> >> > -    Error *err = NULL;
>> >> > +    bool need_resume;
>> >> >
>> >> > -    req = json_parser_parse_err(tokens, NULL, &err);
>> >> > -    if (!req && !err) {
>> >> > -        /* json_parser_parse_err() sucks: can fail without setting @err */
>> >> > -        error_setg(&err, QERR_JSON_PARSING);
>> >> > -    }
>> >> > -    if (err) {
>> >> > -        goto err_out;
>> >> > -    }
>> >> > +    req = req_obj->req;
>> >> > +    mon = req_obj->mon;
>> >> > +    id = req_obj->id;
>> >> > +    need_resume = req_obj->need_resume;
>> >> >
>> >> > -    qdict = qobject_to_qdict(req);
>> >> > -    if (qdict) {
>> >> > -        id = qdict_get(qdict, "id");
>> >> > -        qobject_incref(id);
>> >> > -        qdict_del(qdict, "id");
>> >> > -    } /* else will fail qmp_dispatch() */
>> >> > +    g_free(req_obj);
>> >> >
>> >> >      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
>> >> >          QString *req_json = qobject_to_json(req);
>> >> > @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>> >> >      old_mon = cur_mon;
>> >> >      cur_mon = mon;
>> >>
>> >> There is another issue with this series, since cur_mon is global (and
>> >> not protected), an oob command may change the cur_mon while another
>> >> command is running in the main thread with unexpected consequences. I
>> >> don't have a clear idea what is the best way to solve it. Making the
>> >> variable per-thread, or going all the way to get rid of cur_mon (my
>> >> preference, but much harder)
>> >
>> > IMHO it is fine too.
>> >
>> > Note that this cur_mon operation is in monitor_qmp_dispatch_one() now,
>> > which is still running in main thread.  So AFAICT all the cur_mon
>> > references are in main thread, and monitor IOThread does not modify
>> > that variable at all.  Then we should probably be safe.
>>
>> But monitor_qmp_dispatch_one() is called from iothread if the command
>> is oob, so cur_mon may be updated while another command is running in
>> main thread, or am I wrong?
>
> You are right. I missed that, sorry...
>
> Would this be a simple workaround (but hopefully efficient) solution?
>
> diff --git a/monitor.c b/monitor.c
> index 77f4c41cfa..99641c0c6d 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -4023,7 +4023,7 @@ typedef struct QMPRequest QMPRequest;
>   * Dispatch one single QMP request. The function will free the req_obj
>   * and objects inside it before return.
>   */
> -static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> +static void monitor_qmp_dispatch_one(QMPRequest *req_obj, bool hack_curmon)
>  {
>      Monitor *mon, *old_mon;
>      QObject *req, *rsp = NULL, *id;
> @@ -4043,12 +4043,16 @@ static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
>          QDECREF(req_json);
>      }
>
> -    old_mon = cur_mon;
> -    cur_mon = mon;
> +    if (hack_curmon) {
> +        old_mon = cur_mon;
> +        cur_mon = mon;
> +    }
>
>      rsp = qmp_dispatch(mon->qmp.commands, req);
>
> -    cur_mon = old_mon;
> +    if (hack_curmon) {
> +        cur_mon = old_mon;
> +    }
>
>      if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
>          qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
> @@ -4116,7 +4120,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
>
>      if (req_obj) {
>          trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
> -        monitor_qmp_dispatch_one(req_obj);
> +        monitor_qmp_dispatch_one(req_obj, true);
>          /* Reschedule instead of looping so the main loop stays responsive */
>          qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
>      }
> @@ -4175,7 +4179,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>          /* Out-Of-Band (OOB) requests are executed directly in parser. */
>          trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
>                                            ?: "");
> -        monitor_qmp_dispatch_one(req_obj);
> +        monitor_qmp_dispatch_one(req_obj, false);
>          return;
>      }
>
> Then we forbit touching that evil cur_mon in OOB-capable command
> handlers.  Thanks,

That's not easy to enforce though, afaict it is being used for:
- error reporting decision
- file & socket lookup (fd: & /dev/fdset etc)
- the current state of the monitor / list of commands, cpu_path, capabilities..

Wouldn't it be simpler to make it per-thread? I think it could also
use helpers to push/pop the current monitor.

-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Mon, Mar 26, 2018 at 11:46:13AM +0200, Marc-André Lureau wrote:
> Hi
> 
> On Mon, Mar 26, 2018 at 11:08 AM, Peter Xu <peterx@redhat.com> wrote:
> > On Mon, Mar 26, 2018 at 10:33:27AM +0200, Marc-André Lureau wrote:
> >> Hi
> >>
> >> On Mon, Mar 26, 2018 at 10:07 AM, Peter Xu <peterx@redhat.com> wrote:
> >> > On Fri, Mar 23, 2018 at 05:18:53PM +0100, Marc-André Lureau wrote:
> >> >
> >> > [...]
> >> >
> >> >> > +/*
> >> >> > + * Dispatch one single QMP request. The function will free the req_obj
> >> >> > + * and objects inside it before return.
> >> >> > + */
> >> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> >> >> >  {
> >> >> > -    QObject *req, *rsp = NULL, *id = NULL;
> >> >> > +    Monitor *mon, *old_mon;
> >> >> > +    QObject *req, *rsp = NULL, *id;
> >> >> >      QDict *qdict = NULL;
> >> >> > -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> >> >> > -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
> >> >> > -
> >> >> > -    Error *err = NULL;
> >> >> > +    bool need_resume;
> >> >> >
> >> >> > -    req = json_parser_parse_err(tokens, NULL, &err);
> >> >> > -    if (!req && !err) {
> >> >> > -        /* json_parser_parse_err() sucks: can fail without setting @err */
> >> >> > -        error_setg(&err, QERR_JSON_PARSING);
> >> >> > -    }
> >> >> > -    if (err) {
> >> >> > -        goto err_out;
> >> >> > -    }
> >> >> > +    req = req_obj->req;
> >> >> > +    mon = req_obj->mon;
> >> >> > +    id = req_obj->id;
> >> >> > +    need_resume = req_obj->need_resume;
> >> >> >
> >> >> > -    qdict = qobject_to_qdict(req);
> >> >> > -    if (qdict) {
> >> >> > -        id = qdict_get(qdict, "id");
> >> >> > -        qobject_incref(id);
> >> >> > -        qdict_del(qdict, "id");
> >> >> > -    } /* else will fail qmp_dispatch() */
> >> >> > +    g_free(req_obj);
> >> >> >
> >> >> >      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
> >> >> >          QString *req_json = qobject_to_json(req);
> >> >> > @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >> >> >      old_mon = cur_mon;
> >> >> >      cur_mon = mon;
> >> >>
> >> >> There is another issue with this series, since cur_mon is global (and
> >> >> not protected), an oob command may change the cur_mon while another
> >> >> command is running in the main thread with unexpected consequences. I
> >> >> don't have a clear idea what is the best way to solve it. Making the
> >> >> variable per-thread, or going all the way to get rid of cur_mon (my
> >> >> preference, but much harder)
> >> >
> >> > IMHO it is fine too.
> >> >
> >> > Note that this cur_mon operation is in monitor_qmp_dispatch_one() now,
> >> > which is still running in main thread.  So AFAICT all the cur_mon
> >> > references are in main thread, and monitor IOThread does not modify
> >> > that variable at all.  Then we should probably be safe.
> >>
> >> But monitor_qmp_dispatch_one() is called from iothread if the command
> >> is oob, so cur_mon may be updated while another command is running in
> >> main thread, or am I wrong?
> >
> > You are right. I missed that, sorry...
> >
> > Would this be a simple workaround (but hopefully efficient) solution?
> >
> > diff --git a/monitor.c b/monitor.c
> > index 77f4c41cfa..99641c0c6d 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -4023,7 +4023,7 @@ typedef struct QMPRequest QMPRequest;
> >   * Dispatch one single QMP request. The function will free the req_obj
> >   * and objects inside it before return.
> >   */
> > -static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj, bool hack_curmon)
> >  {
> >      Monitor *mon, *old_mon;
> >      QObject *req, *rsp = NULL, *id;
> > @@ -4043,12 +4043,16 @@ static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> >          QDECREF(req_json);
> >      }
> >
> > -    old_mon = cur_mon;
> > -    cur_mon = mon;
> > +    if (hack_curmon) {
> > +        old_mon = cur_mon;
> > +        cur_mon = mon;
> > +    }
> >
> >      rsp = qmp_dispatch(mon->qmp.commands, req);
> >
> > -    cur_mon = old_mon;
> > +    if (hack_curmon) {
> > +        cur_mon = old_mon;
> > +    }
> >
> >      if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
> >          qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
> > @@ -4116,7 +4120,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
> >
> >      if (req_obj) {
> >          trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
> > -        monitor_qmp_dispatch_one(req_obj);
> > +        monitor_qmp_dispatch_one(req_obj, true);
> >          /* Reschedule instead of looping so the main loop stays responsive */
> >          qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> >      }
> > @@ -4175,7 +4179,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >          /* Out-Of-Band (OOB) requests are executed directly in parser. */
> >          trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
> >                                            ?: "");
> > -        monitor_qmp_dispatch_one(req_obj);
> > +        monitor_qmp_dispatch_one(req_obj, false);
> >          return;
> >      }
> >
> > Then we forbit touching that evil cur_mon in OOB-capable command
> > handlers.  Thanks,
> 
> That's not easy to enforce though, afaict it is being used for:
> - error reporting decision

IMO this should not be a problem, since any QMP handler (including
OOB-capable ones) will be with an Error** there, so logically speaking
people should never call things like error_report() in that.

> - file & socket lookup (fd: & /dev/fdset etc)

I suppose only very rare commands will use it?  It'll be a big problem
to solve when we want to completely remove cur_mon though.

> - the current state of the monitor / list of commands, cpu_path, capabilities..

This is very rare to be used too?  Most commands should not use them AFAIU.

> 
> Wouldn't it be simpler to make it per-thread? I think it could also
> use helpers to push/pop the current monitor.

Anyway I think yes this is still a good option (though the cur_mon
logic will be a bit more complicated).

Do you plan to post some patch about this, or do you want me to do
this?  I suppose we'll change the qemu_thread_create() a bit to pass
the cur_mon inside, and I suppose this might be better material after
2.12 release if OOB is off now.

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Marc-André Lureau]

Hi Peter

On Wed, Mar 28, 2018 at 6:02 AM, Peter Xu <peterx@redhat.com> wrote:
> On Mon, Mar 26, 2018 at 11:46:13AM +0200, Marc-André Lureau wrote:
>> Hi
>>
>> On Mon, Mar 26, 2018 at 11:08 AM, Peter Xu <peterx@redhat.com> wrote:
>> > On Mon, Mar 26, 2018 at 10:33:27AM +0200, Marc-André Lureau wrote:
>> >> Hi
>> >>
>> >> On Mon, Mar 26, 2018 at 10:07 AM, Peter Xu <peterx@redhat.com> wrote:
>> >> > On Fri, Mar 23, 2018 at 05:18:53PM +0100, Marc-André Lureau wrote:
>> >> >
>> >> > [...]
>> >> >
>> >> >> > +/*
>> >> >> > + * Dispatch one single QMP request. The function will free the req_obj
>> >> >> > + * and objects inside it before return.
>> >> >> > + */
>> >> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
>> >> >> >  {
>> >> >> > -    QObject *req, *rsp = NULL, *id = NULL;
>> >> >> > +    Monitor *mon, *old_mon;
>> >> >> > +    QObject *req, *rsp = NULL, *id;
>> >> >> >      QDict *qdict = NULL;
>> >> >> > -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
>> >> >> > -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
>> >> >> > -
>> >> >> > -    Error *err = NULL;
>> >> >> > +    bool need_resume;
>> >> >> >
>> >> >> > -    req = json_parser_parse_err(tokens, NULL, &err);
>> >> >> > -    if (!req && !err) {
>> >> >> > -        /* json_parser_parse_err() sucks: can fail without setting @err */
>> >> >> > -        error_setg(&err, QERR_JSON_PARSING);
>> >> >> > -    }
>> >> >> > -    if (err) {
>> >> >> > -        goto err_out;
>> >> >> > -    }
>> >> >> > +    req = req_obj->req;
>> >> >> > +    mon = req_obj->mon;
>> >> >> > +    id = req_obj->id;
>> >> >> > +    need_resume = req_obj->need_resume;
>> >> >> >
>> >> >> > -    qdict = qobject_to_qdict(req);
>> >> >> > -    if (qdict) {
>> >> >> > -        id = qdict_get(qdict, "id");
>> >> >> > -        qobject_incref(id);
>> >> >> > -        qdict_del(qdict, "id");
>> >> >> > -    } /* else will fail qmp_dispatch() */
>> >> >> > +    g_free(req_obj);
>> >> >> >
>> >> >> >      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
>> >> >> >          QString *req_json = qobject_to_json(req);
>> >> >> > @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>> >> >> >      old_mon = cur_mon;
>> >> >> >      cur_mon = mon;
>> >> >>
>> >> >> There is another issue with this series, since cur_mon is global (and
>> >> >> not protected), an oob command may change the cur_mon while another
>> >> >> command is running in the main thread with unexpected consequences. I
>> >> >> don't have a clear idea what is the best way to solve it. Making the
>> >> >> variable per-thread, or going all the way to get rid of cur_mon (my
>> >> >> preference, but much harder)
>> >> >
>> >> > IMHO it is fine too.
>> >> >
>> >> > Note that this cur_mon operation is in monitor_qmp_dispatch_one() now,
>> >> > which is still running in main thread.  So AFAICT all the cur_mon
>> >> > references are in main thread, and monitor IOThread does not modify
>> >> > that variable at all.  Then we should probably be safe.
>> >>
>> >> But monitor_qmp_dispatch_one() is called from iothread if the command
>> >> is oob, so cur_mon may be updated while another command is running in
>> >> main thread, or am I wrong?
>> >
>> > You are right. I missed that, sorry...
>> >
>> > Would this be a simple workaround (but hopefully efficient) solution?
>> >
>> > diff --git a/monitor.c b/monitor.c
>> > index 77f4c41cfa..99641c0c6d 100644
>> > --- a/monitor.c
>> > +++ b/monitor.c
>> > @@ -4023,7 +4023,7 @@ typedef struct QMPRequest QMPRequest;
>> >   * Dispatch one single QMP request. The function will free the req_obj
>> >   * and objects inside it before return.
>> >   */
>> > -static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
>> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj, bool hack_curmon)
>> >  {
>> >      Monitor *mon, *old_mon;
>> >      QObject *req, *rsp = NULL, *id;
>> > @@ -4043,12 +4043,16 @@ static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
>> >          QDECREF(req_json);
>> >      }
>> >
>> > -    old_mon = cur_mon;
>> > -    cur_mon = mon;
>> > +    if (hack_curmon) {
>> > +        old_mon = cur_mon;
>> > +        cur_mon = mon;
>> > +    }
>> >
>> >      rsp = qmp_dispatch(mon->qmp.commands, req);
>> >
>> > -    cur_mon = old_mon;
>> > +    if (hack_curmon) {
>> > +        cur_mon = old_mon;
>> > +    }
>> >
>> >      if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
>> >          qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
>> > @@ -4116,7 +4120,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
>> >
>> >      if (req_obj) {
>> >          trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
>> > -        monitor_qmp_dispatch_one(req_obj);
>> > +        monitor_qmp_dispatch_one(req_obj, true);
>> >          /* Reschedule instead of looping so the main loop stays responsive */
>> >          qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
>> >      }
>> > @@ -4175,7 +4179,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>> >          /* Out-Of-Band (OOB) requests are executed directly in parser. */
>> >          trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
>> >                                            ?: "");
>> > -        monitor_qmp_dispatch_one(req_obj);
>> > +        monitor_qmp_dispatch_one(req_obj, false);
>> >          return;
>> >      }
>> >
>> > Then we forbit touching that evil cur_mon in OOB-capable command
>> > handlers.  Thanks,
>>
>> That's not easy to enforce though, afaict it is being used for:
>> - error reporting decision
>
> IMO this should not be a problem, since any QMP handler (including
> OOB-capable ones) will be with an Error** there, so logically speaking
> people should never call things like error_report() in that.
>
>> - file & socket lookup (fd: & /dev/fdset etc)
>
> I suppose only very rare commands will use it?  It'll be a big problem
> to solve when we want to completely remove cur_mon though.
>
>> - the current state of the monitor / list of commands, cpu_path, capabilities..
>
> This is very rare to be used too?  Most commands should not use them AFAIU.
>
>>
>> Wouldn't it be simpler to make it per-thread? I think it could also
>> use helpers to push/pop the current monitor.
>
> Anyway I think yes this is still a good option (though the cur_mon
> logic will be a bit more complicated).
>
> Do you plan to post some patch about this, or do you want me to do
> this?  I suppose we'll change the qemu_thread_create() a bit to pass
> the cur_mon inside, and I suppose this might be better material after
> 2.12 release if OOB is off now.

Have you looked at making cur_mon per-thread?

thanks


-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Wed, Apr 04, 2018 at 03:58:56PM +0200, Marc-André Lureau wrote:
> Hi Peter
> 
> On Wed, Mar 28, 2018 at 6:02 AM, Peter Xu <peterx@redhat.com> wrote:
> > On Mon, Mar 26, 2018 at 11:46:13AM +0200, Marc-André Lureau wrote:
> >> Hi
> >>
> >> On Mon, Mar 26, 2018 at 11:08 AM, Peter Xu <peterx@redhat.com> wrote:
> >> > On Mon, Mar 26, 2018 at 10:33:27AM +0200, Marc-André Lureau wrote:
> >> >> Hi
> >> >>
> >> >> On Mon, Mar 26, 2018 at 10:07 AM, Peter Xu <peterx@redhat.com> wrote:
> >> >> > On Fri, Mar 23, 2018 at 05:18:53PM +0100, Marc-André Lureau wrote:
> >> >> >
> >> >> > [...]
> >> >> >
> >> >> >> > +/*
> >> >> >> > + * Dispatch one single QMP request. The function will free the req_obj
> >> >> >> > + * and objects inside it before return.
> >> >> >> > + */
> >> >> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> >> >> >> >  {
> >> >> >> > -    QObject *req, *rsp = NULL, *id = NULL;
> >> >> >> > +    Monitor *mon, *old_mon;
> >> >> >> > +    QObject *req, *rsp = NULL, *id;
> >> >> >> >      QDict *qdict = NULL;
> >> >> >> > -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> >> >> >> > -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
> >> >> >> > -
> >> >> >> > -    Error *err = NULL;
> >> >> >> > +    bool need_resume;
> >> >> >> >
> >> >> >> > -    req = json_parser_parse_err(tokens, NULL, &err);
> >> >> >> > -    if (!req && !err) {
> >> >> >> > -        /* json_parser_parse_err() sucks: can fail without setting @err */
> >> >> >> > -        error_setg(&err, QERR_JSON_PARSING);
> >> >> >> > -    }
> >> >> >> > -    if (err) {
> >> >> >> > -        goto err_out;
> >> >> >> > -    }
> >> >> >> > +    req = req_obj->req;
> >> >> >> > +    mon = req_obj->mon;
> >> >> >> > +    id = req_obj->id;
> >> >> >> > +    need_resume = req_obj->need_resume;
> >> >> >> >
> >> >> >> > -    qdict = qobject_to_qdict(req);
> >> >> >> > -    if (qdict) {
> >> >> >> > -        id = qdict_get(qdict, "id");
> >> >> >> > -        qobject_incref(id);
> >> >> >> > -        qdict_del(qdict, "id");
> >> >> >> > -    } /* else will fail qmp_dispatch() */
> >> >> >> > +    g_free(req_obj);
> >> >> >> >
> >> >> >> >      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
> >> >> >> >          QString *req_json = qobject_to_json(req);
> >> >> >> > @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >> >> >> >      old_mon = cur_mon;
> >> >> >> >      cur_mon = mon;
> >> >> >>
> >> >> >> There is another issue with this series, since cur_mon is global (and
> >> >> >> not protected), an oob command may change the cur_mon while another
> >> >> >> command is running in the main thread with unexpected consequences. I
> >> >> >> don't have a clear idea what is the best way to solve it. Making the
> >> >> >> variable per-thread, or going all the way to get rid of cur_mon (my
> >> >> >> preference, but much harder)
> >> >> >
> >> >> > IMHO it is fine too.
> >> >> >
> >> >> > Note that this cur_mon operation is in monitor_qmp_dispatch_one() now,
> >> >> > which is still running in main thread.  So AFAICT all the cur_mon
> >> >> > references are in main thread, and monitor IOThread does not modify
> >> >> > that variable at all.  Then we should probably be safe.
> >> >>
> >> >> But monitor_qmp_dispatch_one() is called from iothread if the command
> >> >> is oob, so cur_mon may be updated while another command is running in
> >> >> main thread, or am I wrong?
> >> >
> >> > You are right. I missed that, sorry...
> >> >
> >> > Would this be a simple workaround (but hopefully efficient) solution?
> >> >
> >> > diff --git a/monitor.c b/monitor.c
> >> > index 77f4c41cfa..99641c0c6d 100644
> >> > --- a/monitor.c
> >> > +++ b/monitor.c
> >> > @@ -4023,7 +4023,7 @@ typedef struct QMPRequest QMPRequest;
> >> >   * Dispatch one single QMP request. The function will free the req_obj
> >> >   * and objects inside it before return.
> >> >   */
> >> > -static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj, bool hack_curmon)
> >> >  {
> >> >      Monitor *mon, *old_mon;
> >> >      QObject *req, *rsp = NULL, *id;
> >> > @@ -4043,12 +4043,16 @@ static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> >> >          QDECREF(req_json);
> >> >      }
> >> >
> >> > -    old_mon = cur_mon;
> >> > -    cur_mon = mon;
> >> > +    if (hack_curmon) {
> >> > +        old_mon = cur_mon;
> >> > +        cur_mon = mon;
> >> > +    }
> >> >
> >> >      rsp = qmp_dispatch(mon->qmp.commands, req);
> >> >
> >> > -    cur_mon = old_mon;
> >> > +    if (hack_curmon) {
> >> > +        cur_mon = old_mon;
> >> > +    }
> >> >
> >> >      if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
> >> >          qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
> >> > @@ -4116,7 +4120,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
> >> >
> >> >      if (req_obj) {
> >> >          trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
> >> > -        monitor_qmp_dispatch_one(req_obj);
> >> > +        monitor_qmp_dispatch_one(req_obj, true);
> >> >          /* Reschedule instead of looping so the main loop stays responsive */
> >> >          qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> >> >      }
> >> > @@ -4175,7 +4179,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >> >          /* Out-Of-Band (OOB) requests are executed directly in parser. */
> >> >          trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
> >> >                                            ?: "");
> >> > -        monitor_qmp_dispatch_one(req_obj);
> >> > +        monitor_qmp_dispatch_one(req_obj, false);
> >> >          return;
> >> >      }
> >> >
> >> > Then we forbit touching that evil cur_mon in OOB-capable command
> >> > handlers.  Thanks,
> >>
> >> That's not easy to enforce though, afaict it is being used for:
> >> - error reporting decision
> >
> > IMO this should not be a problem, since any QMP handler (including
> > OOB-capable ones) will be with an Error** there, so logically speaking
> > people should never call things like error_report() in that.
> >
> >> - file & socket lookup (fd: & /dev/fdset etc)
> >
> > I suppose only very rare commands will use it?  It'll be a big problem
> > to solve when we want to completely remove cur_mon though.
> >
> >> - the current state of the monitor / list of commands, cpu_path, capabilities..
> >
> > This is very rare to be used too?  Most commands should not use them AFAIU.
> >
> >>
> >> Wouldn't it be simpler to make it per-thread? I think it could also
> >> use helpers to push/pop the current monitor.
> >
> > Anyway I think yes this is still a good option (though the cur_mon
> > logic will be a bit more complicated).
> >
> > Do you plan to post some patch about this, or do you want me to do
> > this?  I suppose we'll change the qemu_thread_create() a bit to pass
> > the cur_mon inside, and I suppose this might be better material after
> > 2.12 release if OOB is off now.
> 
> Have you looked at making cur_mon per-thread?

Above was my idea, nothing else has been done.

Please feel free to post a patch for this, or I'll do this after 2.12
release.

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Marc-André Lureau]

Hi

On Sun, Apr 8, 2018 at 5:02 AM, Peter Xu <peterx@redhat.com> wrote:
> On Wed, Apr 04, 2018 at 03:58:56PM +0200, Marc-André Lureau wrote:
>> Hi Peter
>>
>> On Wed, Mar 28, 2018 at 6:02 AM, Peter Xu <peterx@redhat.com> wrote:
>> > On Mon, Mar 26, 2018 at 11:46:13AM +0200, Marc-André Lureau wrote:
>> >> Hi
>> >>
>> >> On Mon, Mar 26, 2018 at 11:08 AM, Peter Xu <peterx@redhat.com> wrote:
>> >> > On Mon, Mar 26, 2018 at 10:33:27AM +0200, Marc-André Lureau wrote:
>> >> >> Hi
>> >> >>
>> >> >> On Mon, Mar 26, 2018 at 10:07 AM, Peter Xu <peterx@redhat.com> wrote:
>> >> >> > On Fri, Mar 23, 2018 at 05:18:53PM +0100, Marc-André Lureau wrote:
>> >> >> >
>> >> >> > [...]
>> >> >> >
>> >> >> >> > +/*
>> >> >> >> > + * Dispatch one single QMP request. The function will free the req_obj
>> >> >> >> > + * and objects inside it before return.
>> >> >> >> > + */
>> >> >> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
>> >> >> >> >  {
>> >> >> >> > -    QObject *req, *rsp = NULL, *id = NULL;
>> >> >> >> > +    Monitor *mon, *old_mon;
>> >> >> >> > +    QObject *req, *rsp = NULL, *id;
>> >> >> >> >      QDict *qdict = NULL;
>> >> >> >> > -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
>> >> >> >> > -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
>> >> >> >> > -
>> >> >> >> > -    Error *err = NULL;
>> >> >> >> > +    bool need_resume;
>> >> >> >> >
>> >> >> >> > -    req = json_parser_parse_err(tokens, NULL, &err);
>> >> >> >> > -    if (!req && !err) {
>> >> >> >> > -        /* json_parser_parse_err() sucks: can fail without setting @err */
>> >> >> >> > -        error_setg(&err, QERR_JSON_PARSING);
>> >> >> >> > -    }
>> >> >> >> > -    if (err) {
>> >> >> >> > -        goto err_out;
>> >> >> >> > -    }
>> >> >> >> > +    req = req_obj->req;
>> >> >> >> > +    mon = req_obj->mon;
>> >> >> >> > +    id = req_obj->id;
>> >> >> >> > +    need_resume = req_obj->need_resume;
>> >> >> >> >
>> >> >> >> > -    qdict = qobject_to_qdict(req);
>> >> >> >> > -    if (qdict) {
>> >> >> >> > -        id = qdict_get(qdict, "id");
>> >> >> >> > -        qobject_incref(id);
>> >> >> >> > -        qdict_del(qdict, "id");
>> >> >> >> > -    } /* else will fail qmp_dispatch() */
>> >> >> >> > +    g_free(req_obj);
>> >> >> >> >
>> >> >> >> >      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
>> >> >> >> >          QString *req_json = qobject_to_json(req);
>> >> >> >> > @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>> >> >> >> >      old_mon = cur_mon;
>> >> >> >> >      cur_mon = mon;
>> >> >> >>
>> >> >> >> There is another issue with this series, since cur_mon is global (and
>> >> >> >> not protected), an oob command may change the cur_mon while another
>> >> >> >> command is running in the main thread with unexpected consequences. I
>> >> >> >> don't have a clear idea what is the best way to solve it. Making the
>> >> >> >> variable per-thread, or going all the way to get rid of cur_mon (my
>> >> >> >> preference, but much harder)
>> >> >> >
>> >> >> > IMHO it is fine too.
>> >> >> >
>> >> >> > Note that this cur_mon operation is in monitor_qmp_dispatch_one() now,
>> >> >> > which is still running in main thread.  So AFAICT all the cur_mon
>> >> >> > references are in main thread, and monitor IOThread does not modify
>> >> >> > that variable at all.  Then we should probably be safe.
>> >> >>
>> >> >> But monitor_qmp_dispatch_one() is called from iothread if the command
>> >> >> is oob, so cur_mon may be updated while another command is running in
>> >> >> main thread, or am I wrong?
>> >> >
>> >> > You are right. I missed that, sorry...
>> >> >
>> >> > Would this be a simple workaround (but hopefully efficient) solution?
>> >> >
>> >> > diff --git a/monitor.c b/monitor.c
>> >> > index 77f4c41cfa..99641c0c6d 100644
>> >> > --- a/monitor.c
>> >> > +++ b/monitor.c
>> >> > @@ -4023,7 +4023,7 @@ typedef struct QMPRequest QMPRequest;
>> >> >   * Dispatch one single QMP request. The function will free the req_obj
>> >> >   * and objects inside it before return.
>> >> >   */
>> >> > -static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
>> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj, bool hack_curmon)
>> >> >  {
>> >> >      Monitor *mon, *old_mon;
>> >> >      QObject *req, *rsp = NULL, *id;
>> >> > @@ -4043,12 +4043,16 @@ static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
>> >> >          QDECREF(req_json);
>> >> >      }
>> >> >
>> >> > -    old_mon = cur_mon;
>> >> > -    cur_mon = mon;
>> >> > +    if (hack_curmon) {
>> >> > +        old_mon = cur_mon;
>> >> > +        cur_mon = mon;
>> >> > +    }
>> >> >
>> >> >      rsp = qmp_dispatch(mon->qmp.commands, req);
>> >> >
>> >> > -    cur_mon = old_mon;
>> >> > +    if (hack_curmon) {
>> >> > +        cur_mon = old_mon;
>> >> > +    }
>> >> >
>> >> >      if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
>> >> >          qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
>> >> > @@ -4116,7 +4120,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
>> >> >
>> >> >      if (req_obj) {
>> >> >          trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
>> >> > -        monitor_qmp_dispatch_one(req_obj);
>> >> > +        monitor_qmp_dispatch_one(req_obj, true);
>> >> >          /* Reschedule instead of looping so the main loop stays responsive */
>> >> >          qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
>> >> >      }
>> >> > @@ -4175,7 +4179,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
>> >> >          /* Out-Of-Band (OOB) requests are executed directly in parser. */
>> >> >          trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
>> >> >                                            ?: "");
>> >> > -        monitor_qmp_dispatch_one(req_obj);
>> >> > +        monitor_qmp_dispatch_one(req_obj, false);
>> >> >          return;
>> >> >      }
>> >> >
>> >> > Then we forbit touching that evil cur_mon in OOB-capable command
>> >> > handlers.  Thanks,
>> >>
>> >> That's not easy to enforce though, afaict it is being used for:
>> >> - error reporting decision
>> >
>> > IMO this should not be a problem, since any QMP handler (including
>> > OOB-capable ones) will be with an Error** there, so logically speaking
>> > people should never call things like error_report() in that.
>> >
>> >> - file & socket lookup (fd: & /dev/fdset etc)
>> >
>> > I suppose only very rare commands will use it?  It'll be a big problem
>> > to solve when we want to completely remove cur_mon though.
>> >
>> >> - the current state of the monitor / list of commands, cpu_path, capabilities..
>> >
>> > This is very rare to be used too?  Most commands should not use them AFAIU.
>> >
>> >>
>> >> Wouldn't it be simpler to make it per-thread? I think it could also
>> >> use helpers to push/pop the current monitor.
>> >
>> > Anyway I think yes this is still a good option (though the cur_mon
>> > logic will be a bit more complicated).
>> >
>> > Do you plan to post some patch about this, or do you want me to do
>> > this?  I suppose we'll change the qemu_thread_create() a bit to pass
>> > the cur_mon inside, and I suppose this might be better material after
>> > 2.12 release if OOB is off now.
>>
>> Have you looked at making cur_mon per-thread?
>
> Above was my idea, nothing else has been done.
>
> Please feel free to post a patch for this, or I'll do this after 2.12
> release.

If it's fixed after 2.12, I think we should document the race as a known issue.




-- 
Marc-André Lureau

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Mon, Apr 09, 2018 at 11:19:43AM +0200, Marc-André Lureau wrote:
> Hi
> 
> On Sun, Apr 8, 2018 at 5:02 AM, Peter Xu <peterx@redhat.com> wrote:
> > On Wed, Apr 04, 2018 at 03:58:56PM +0200, Marc-André Lureau wrote:
> >> Hi Peter
> >>
> >> On Wed, Mar 28, 2018 at 6:02 AM, Peter Xu <peterx@redhat.com> wrote:
> >> > On Mon, Mar 26, 2018 at 11:46:13AM +0200, Marc-André Lureau wrote:
> >> >> Hi
> >> >>
> >> >> On Mon, Mar 26, 2018 at 11:08 AM, Peter Xu <peterx@redhat.com> wrote:
> >> >> > On Mon, Mar 26, 2018 at 10:33:27AM +0200, Marc-André Lureau wrote:
> >> >> >> Hi
> >> >> >>
> >> >> >> On Mon, Mar 26, 2018 at 10:07 AM, Peter Xu <peterx@redhat.com> wrote:
> >> >> >> > On Fri, Mar 23, 2018 at 05:18:53PM +0100, Marc-André Lureau wrote:
> >> >> >> >
> >> >> >> > [...]
> >> >> >> >
> >> >> >> >> > +/*
> >> >> >> >> > + * Dispatch one single QMP request. The function will free the req_obj
> >> >> >> >> > + * and objects inside it before return.
> >> >> >> >> > + */
> >> >> >> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> >> >> >> >> >  {
> >> >> >> >> > -    QObject *req, *rsp = NULL, *id = NULL;
> >> >> >> >> > +    Monitor *mon, *old_mon;
> >> >> >> >> > +    QObject *req, *rsp = NULL, *id;
> >> >> >> >> >      QDict *qdict = NULL;
> >> >> >> >> > -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> >> >> >> >> > -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
> >> >> >> >> > -
> >> >> >> >> > -    Error *err = NULL;
> >> >> >> >> > +    bool need_resume;
> >> >> >> >> >
> >> >> >> >> > -    req = json_parser_parse_err(tokens, NULL, &err);
> >> >> >> >> > -    if (!req && !err) {
> >> >> >> >> > -        /* json_parser_parse_err() sucks: can fail without setting @err */
> >> >> >> >> > -        error_setg(&err, QERR_JSON_PARSING);
> >> >> >> >> > -    }
> >> >> >> >> > -    if (err) {
> >> >> >> >> > -        goto err_out;
> >> >> >> >> > -    }
> >> >> >> >> > +    req = req_obj->req;
> >> >> >> >> > +    mon = req_obj->mon;
> >> >> >> >> > +    id = req_obj->id;
> >> >> >> >> > +    need_resume = req_obj->need_resume;
> >> >> >> >> >
> >> >> >> >> > -    qdict = qobject_to_qdict(req);
> >> >> >> >> > -    if (qdict) {
> >> >> >> >> > -        id = qdict_get(qdict, "id");
> >> >> >> >> > -        qobject_incref(id);
> >> >> >> >> > -        qdict_del(qdict, "id");
> >> >> >> >> > -    } /* else will fail qmp_dispatch() */
> >> >> >> >> > +    g_free(req_obj);
> >> >> >> >> >
> >> >> >> >> >      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
> >> >> >> >> >          QString *req_json = qobject_to_json(req);
> >> >> >> >> > @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >> >> >> >> >      old_mon = cur_mon;
> >> >> >> >> >      cur_mon = mon;
> >> >> >> >>
> >> >> >> >> There is another issue with this series, since cur_mon is global (and
> >> >> >> >> not protected), an oob command may change the cur_mon while another
> >> >> >> >> command is running in the main thread with unexpected consequences. I
> >> >> >> >> don't have a clear idea what is the best way to solve it. Making the
> >> >> >> >> variable per-thread, or going all the way to get rid of cur_mon (my
> >> >> >> >> preference, but much harder)
> >> >> >> >
> >> >> >> > IMHO it is fine too.
> >> >> >> >
> >> >> >> > Note that this cur_mon operation is in monitor_qmp_dispatch_one() now,
> >> >> >> > which is still running in main thread.  So AFAICT all the cur_mon
> >> >> >> > references are in main thread, and monitor IOThread does not modify
> >> >> >> > that variable at all.  Then we should probably be safe.
> >> >> >>
> >> >> >> But monitor_qmp_dispatch_one() is called from iothread if the command
> >> >> >> is oob, so cur_mon may be updated while another command is running in
> >> >> >> main thread, or am I wrong?
> >> >> >
> >> >> > You are right. I missed that, sorry...
> >> >> >
> >> >> > Would this be a simple workaround (but hopefully efficient) solution?
> >> >> >
> >> >> > diff --git a/monitor.c b/monitor.c
> >> >> > index 77f4c41cfa..99641c0c6d 100644
> >> >> > --- a/monitor.c
> >> >> > +++ b/monitor.c
> >> >> > @@ -4023,7 +4023,7 @@ typedef struct QMPRequest QMPRequest;
> >> >> >   * Dispatch one single QMP request. The function will free the req_obj
> >> >> >   * and objects inside it before return.
> >> >> >   */
> >> >> > -static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> >> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj, bool hack_curmon)
> >> >> >  {
> >> >> >      Monitor *mon, *old_mon;
> >> >> >      QObject *req, *rsp = NULL, *id;
> >> >> > @@ -4043,12 +4043,16 @@ static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> >> >> >          QDECREF(req_json);
> >> >> >      }
> >> >> >
> >> >> > -    old_mon = cur_mon;
> >> >> > -    cur_mon = mon;
> >> >> > +    if (hack_curmon) {
> >> >> > +        old_mon = cur_mon;
> >> >> > +        cur_mon = mon;
> >> >> > +    }
> >> >> >
> >> >> >      rsp = qmp_dispatch(mon->qmp.commands, req);
> >> >> >
> >> >> > -    cur_mon = old_mon;
> >> >> > +    if (hack_curmon) {
> >> >> > +        cur_mon = old_mon;
> >> >> > +    }
> >> >> >
> >> >> >      if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
> >> >> >          qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
> >> >> > @@ -4116,7 +4120,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
> >> >> >
> >> >> >      if (req_obj) {
> >> >> >          trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
> >> >> > -        monitor_qmp_dispatch_one(req_obj);
> >> >> > +        monitor_qmp_dispatch_one(req_obj, true);
> >> >> >          /* Reschedule instead of looping so the main loop stays responsive */
> >> >> >          qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> >> >> >      }
> >> >> > @@ -4175,7 +4179,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> >> >> >          /* Out-Of-Band (OOB) requests are executed directly in parser. */
> >> >> >          trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
> >> >> >                                            ?: "");
> >> >> > -        monitor_qmp_dispatch_one(req_obj);
> >> >> > +        monitor_qmp_dispatch_one(req_obj, false);
> >> >> >          return;
> >> >> >      }
> >> >> >
> >> >> > Then we forbit touching that evil cur_mon in OOB-capable command
> >> >> > handlers.  Thanks,
> >> >>
> >> >> That's not easy to enforce though, afaict it is being used for:
> >> >> - error reporting decision
> >> >
> >> > IMO this should not be a problem, since any QMP handler (including
> >> > OOB-capable ones) will be with an Error** there, so logically speaking
> >> > people should never call things like error_report() in that.
> >> >
> >> >> - file & socket lookup (fd: & /dev/fdset etc)
> >> >
> >> > I suppose only very rare commands will use it?  It'll be a big problem
> >> > to solve when we want to completely remove cur_mon though.
> >> >
> >> >> - the current state of the monitor / list of commands, cpu_path, capabilities..
> >> >
> >> > This is very rare to be used too?  Most commands should not use them AFAIU.
> >> >
> >> >>
> >> >> Wouldn't it be simpler to make it per-thread? I think it could also
> >> >> use helpers to push/pop the current monitor.
> >> >
> >> > Anyway I think yes this is still a good option (though the cur_mon
> >> > logic will be a bit more complicated).
> >> >
> >> > Do you plan to post some patch about this, or do you want me to do
> >> > this?  I suppose we'll change the qemu_thread_create() a bit to pass
> >> > the cur_mon inside, and I suppose this might be better material after
> >> > 2.12 release if OOB is off now.
> >>
> >> Have you looked at making cur_mon per-thread?
> >
> > Above was my idea, nothing else has been done.
> >
> > Please feel free to post a patch for this, or I'll do this after 2.12
> > release.
> 
> If it's fixed after 2.12, I think we should document the race as a known issue.

Do you mean this page?

https://wiki.qemu.org/Planning/2.12

To be simpler, I'll see whether I can post the patches soon, and
whether that can be accepted as 2.12 material.

-- 
Peter Xu

Re: [Qemu-devel] [PATCH v8 14/23] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Tue, Apr 10, 2018 at 03:15:57PM +0800, Peter Xu wrote:
> On Mon, Apr 09, 2018 at 11:19:43AM +0200, Marc-André Lureau wrote:
> > Hi
> > 
> > On Sun, Apr 8, 2018 at 5:02 AM, Peter Xu <peterx@redhat.com> wrote:
> > > On Wed, Apr 04, 2018 at 03:58:56PM +0200, Marc-André Lureau wrote:
> > >> Hi Peter
> > >>
> > >> On Wed, Mar 28, 2018 at 6:02 AM, Peter Xu <peterx@redhat.com> wrote:
> > >> > On Mon, Mar 26, 2018 at 11:46:13AM +0200, Marc-André Lureau wrote:
> > >> >> Hi
> > >> >>
> > >> >> On Mon, Mar 26, 2018 at 11:08 AM, Peter Xu <peterx@redhat.com> wrote:
> > >> >> > On Mon, Mar 26, 2018 at 10:33:27AM +0200, Marc-André Lureau wrote:
> > >> >> >> Hi
> > >> >> >>
> > >> >> >> On Mon, Mar 26, 2018 at 10:07 AM, Peter Xu <peterx@redhat.com> wrote:
> > >> >> >> > On Fri, Mar 23, 2018 at 05:18:53PM +0100, Marc-André Lureau wrote:
> > >> >> >> >
> > >> >> >> > [...]
> > >> >> >> >
> > >> >> >> >> > +/*
> > >> >> >> >> > + * Dispatch one single QMP request. The function will free the req_obj
> > >> >> >> >> > + * and objects inside it before return.
> > >> >> >> >> > + */
> > >> >> >> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> > >> >> >> >> >  {
> > >> >> >> >> > -    QObject *req, *rsp = NULL, *id = NULL;
> > >> >> >> >> > +    Monitor *mon, *old_mon;
> > >> >> >> >> > +    QObject *req, *rsp = NULL, *id;
> > >> >> >> >> >      QDict *qdict = NULL;
> > >> >> >> >> > -    MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> > >> >> >> >> > -    Monitor *old_mon, *mon = container_of(mon_qmp, Monitor, qmp);
> > >> >> >> >> > -
> > >> >> >> >> > -    Error *err = NULL;
> > >> >> >> >> > +    bool need_resume;
> > >> >> >> >> >
> > >> >> >> >> > -    req = json_parser_parse_err(tokens, NULL, &err);
> > >> >> >> >> > -    if (!req && !err) {
> > >> >> >> >> > -        /* json_parser_parse_err() sucks: can fail without setting @err */
> > >> >> >> >> > -        error_setg(&err, QERR_JSON_PARSING);
> > >> >> >> >> > -    }
> > >> >> >> >> > -    if (err) {
> > >> >> >> >> > -        goto err_out;
> > >> >> >> >> > -    }
> > >> >> >> >> > +    req = req_obj->req;
> > >> >> >> >> > +    mon = req_obj->mon;
> > >> >> >> >> > +    id = req_obj->id;
> > >> >> >> >> > +    need_resume = req_obj->need_resume;
> > >> >> >> >> >
> > >> >> >> >> > -    qdict = qobject_to_qdict(req);
> > >> >> >> >> > -    if (qdict) {
> > >> >> >> >> > -        id = qdict_get(qdict, "id");
> > >> >> >> >> > -        qobject_incref(id);
> > >> >> >> >> > -        qdict_del(qdict, "id");
> > >> >> >> >> > -    } /* else will fail qmp_dispatch() */
> > >> >> >> >> > +    g_free(req_obj);
> > >> >> >> >> >
> > >> >> >> >> >      if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
> > >> >> >> >> >          QString *req_json = qobject_to_json(req);
> > >> >> >> >> > @@ -3900,7 +3932,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> > >> >> >> >> >      old_mon = cur_mon;
> > >> >> >> >> >      cur_mon = mon;
> > >> >> >> >>
> > >> >> >> >> There is another issue with this series, since cur_mon is global (and
> > >> >> >> >> not protected), an oob command may change the cur_mon while another
> > >> >> >> >> command is running in the main thread with unexpected consequences. I
> > >> >> >> >> don't have a clear idea what is the best way to solve it. Making the
> > >> >> >> >> variable per-thread, or going all the way to get rid of cur_mon (my
> > >> >> >> >> preference, but much harder)
> > >> >> >> >
> > >> >> >> > IMHO it is fine too.
> > >> >> >> >
> > >> >> >> > Note that this cur_mon operation is in monitor_qmp_dispatch_one() now,
> > >> >> >> > which is still running in main thread.  So AFAICT all the cur_mon
> > >> >> >> > references are in main thread, and monitor IOThread does not modify
> > >> >> >> > that variable at all.  Then we should probably be safe.
> > >> >> >>
> > >> >> >> But monitor_qmp_dispatch_one() is called from iothread if the command
> > >> >> >> is oob, so cur_mon may be updated while another command is running in
> > >> >> >> main thread, or am I wrong?
> > >> >> >
> > >> >> > You are right. I missed that, sorry...
> > >> >> >
> > >> >> > Would this be a simple workaround (but hopefully efficient) solution?
> > >> >> >
> > >> >> > diff --git a/monitor.c b/monitor.c
> > >> >> > index 77f4c41cfa..99641c0c6d 100644
> > >> >> > --- a/monitor.c
> > >> >> > +++ b/monitor.c
> > >> >> > @@ -4023,7 +4023,7 @@ typedef struct QMPRequest QMPRequest;
> > >> >> >   * Dispatch one single QMP request. The function will free the req_obj
> > >> >> >   * and objects inside it before return.
> > >> >> >   */
> > >> >> > -static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> > >> >> > +static void monitor_qmp_dispatch_one(QMPRequest *req_obj, bool hack_curmon)
> > >> >> >  {
> > >> >> >      Monitor *mon, *old_mon;
> > >> >> >      QObject *req, *rsp = NULL, *id;
> > >> >> > @@ -4043,12 +4043,16 @@ static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
> > >> >> >          QDECREF(req_json);
> > >> >> >      }
> > >> >> >
> > >> >> > -    old_mon = cur_mon;
> > >> >> > -    cur_mon = mon;
> > >> >> > +    if (hack_curmon) {
> > >> >> > +        old_mon = cur_mon;
> > >> >> > +        cur_mon = mon;
> > >> >> > +    }
> > >> >> >
> > >> >> >      rsp = qmp_dispatch(mon->qmp.commands, req);
> > >> >> >
> > >> >> > -    cur_mon = old_mon;
> > >> >> > +    if (hack_curmon) {
> > >> >> > +        cur_mon = old_mon;
> > >> >> > +    }
> > >> >> >
> > >> >> >      if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
> > >> >> >          qdict = qdict_get_qdict(qobject_to(QDict, rsp), "error");
> > >> >> > @@ -4116,7 +4120,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
> > >> >> >
> > >> >> >      if (req_obj) {
> > >> >> >          trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id) ?: "");
> > >> >> > -        monitor_qmp_dispatch_one(req_obj);
> > >> >> > +        monitor_qmp_dispatch_one(req_obj, true);
> > >> >> >          /* Reschedule instead of looping so the main loop stays responsive */
> > >> >> >          qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> > >> >> >      }
> > >> >> > @@ -4175,7 +4179,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
> > >> >> >          /* Out-Of-Band (OOB) requests are executed directly in parser. */
> > >> >> >          trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id)
> > >> >> >                                            ?: "");
> > >> >> > -        monitor_qmp_dispatch_one(req_obj);
> > >> >> > +        monitor_qmp_dispatch_one(req_obj, false);
> > >> >> >          return;
> > >> >> >      }
> > >> >> >
> > >> >> > Then we forbit touching that evil cur_mon in OOB-capable command
> > >> >> > handlers.  Thanks,
> > >> >>
> > >> >> That's not easy to enforce though, afaict it is being used for:
> > >> >> - error reporting decision
> > >> >
> > >> > IMO this should not be a problem, since any QMP handler (including
> > >> > OOB-capable ones) will be with an Error** there, so logically speaking
> > >> > people should never call things like error_report() in that.
> > >> >
> > >> >> - file & socket lookup (fd: & /dev/fdset etc)
> > >> >
> > >> > I suppose only very rare commands will use it?  It'll be a big problem
> > >> > to solve when we want to completely remove cur_mon though.
> > >> >
> > >> >> - the current state of the monitor / list of commands, cpu_path, capabilities..
> > >> >
> > >> > This is very rare to be used too?  Most commands should not use them AFAIU.
> > >> >
> > >> >>
> > >> >> Wouldn't it be simpler to make it per-thread? I think it could also
> > >> >> use helpers to push/pop the current monitor.
> > >> >
> > >> > Anyway I think yes this is still a good option (though the cur_mon
> > >> > logic will be a bit more complicated).
> > >> >
> > >> > Do you plan to post some patch about this, or do you want me to do
> > >> > this?  I suppose we'll change the qemu_thread_create() a bit to pass
> > >> > the cur_mon inside, and I suppose this might be better material after
> > >> > 2.12 release if OOB is off now.
> > >>
> > >> Have you looked at making cur_mon per-thread?
> > >
> > > Above was my idea, nothing else has been done.
> > >
> > > Please feel free to post a patch for this, or I'll do this after 2.12
> > > release.
> > 
> > If it's fixed after 2.12, I think we should document the race as a known issue.
> 
> Do you mean this page?
> 
> https://wiki.qemu.org/Planning/2.12
> 
> To be simpler, I'll see whether I can post the patches soon, and
> whether that can be accepted as 2.12 material.

Btw I think it's not a 2.12 "known issue" - IMHO now it's not an issue
at all.  Because we don't have any real command support OOB (let's
ignore the x-oob-test command since it never touches cur_mon).  So
cur_mon will still only be accessed by the main thread but never
anything else.

I'll post the patch as usual in case further OOB commands will touch
cur_mon, but I suppose that'll be for after the release.

-- 
Peter Xu