[Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support

[Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support

[Peter Xu]

This is RFC v3 of Monitor Out-Of-Band series.  It's getting longer and
changes happens between versions, so I decided to re-write the cover
letter.

This series was born from this one:

  https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html

The idea comes from Markus Armbruster and the discussion we had in the
thread.  It's not a super ideal solution (I believe Markus had been
thinking hard to keep everything in order meanwhile trying to satisfy
the migration requirement), but AFAIU it's currently the best.

What is OOB?
============

It's the shortcut of Out-Of-Band execution, its name is given by
Markus.  It's a way to quickly execute a QMP request.  Say, originally
QMP is going throw these steps:

      JSON Parser --> QMP Dispatcher --> Respond
          /|\    (2)                (3)     |
       (1) |                               \|/ (4)
           +---------  main thread  --------+

The requests are executed by the so-called QMP-dispatcher after the
JSON is parsed.  If OOB is on, we run the command directly in the
parser and quickly returns.

This series changed the QMP handling logic by moving the command
parsing and responding phases into IOThreads, so to be more accurate,
after the series the above graph would change into this:

               queue/kick              queue/kick
     JSON Parser ======> QMP Dispatcher =====> Responder
         /|\ |     (3)       /|\  |      (4)      | /|\
      (1) |  | (2)            |   |               |  |
          |  |                |  \|/           (6)|  |(5)
          |  |            main thread             |  |
          |  |                                    |  |
          |  +--------> monitor IO thread <-------+  |
          +-----------/                   \----------+

New Interfaces
==============

QMP Introspection Changes
-------------------------

When do query-qmp-schema, we were getting something like:

  {"name": "migrate-incoming", "ret-type": "17",
   "meta-type": "command", "arg-type": "89"}

Now we will get a new key named "allow-oob":

  {"name": "migrate-incoming", "ret-type": "17", "allow-oob": true,
   "meta-type": "command", "arg-type": "89"}

Which shows whether a command supports OOB.

QMP Negociation Changes
-----------------------

We were running "qmp_capabilities" always without parameters like:

  {"execute": "qmp_capabilities"}

Now we can enable capabilities (we don't have any capability before
this series) like OOB using:

  {"execute": "qmp_capabilities", "arguments": {"enable": ["oob"]}}

Only after we explicitly enable OOB capability can we send requests in
OOB manner.  Otherwise we'll have exactly the same QMP session as
before, just like when OOB is not there.

When OOB is enabled, it's possible that OOB reply reaches faster than
previous command, so clients should be prepared.

Trigger OOB execution
---------------------

Let's take migrate-incoming as example.  The old command looks like:

  {"execute": "migrate-incoming", "arguments" : {"uri": "xxxxxx"}}

To execute a command with OOB execution, we need to specify it in the
QMP request in the extra "control" key:

  {"execute": "migrate-incoming", "arguments" : {"uri": "xxxxxx"},
   "control" { "run-oob": true } }

Then the command will be run in parser, and it can preempt other
commands.

Others
=================

The last patch of OOB test may need some attention.  I used
dump-guest-memory as a time-consuming command to test whether OOB is
working, and the only command I can test now is "migrate-incoming".  I
hope that is a "okay" solution for unit tests.  Any other suggestions
on that would be welcomed.

v3 changelog:
- inline monitor_io_thread_destroy(). [Stefan]
- rename usage of "*io_thread*" into "*iothread*" [Stefan]
- fixed quite a few English errors [Stefan]
- add the missing "return" (oops!) in handle_qmp_command(). [Stefan]
- rename QMP_ASYNC_QUEUE_LEN_MAX to QMP_REQ_QUEUE_LEN_MAX. [Stefan]
- add OOB cap negociationg phase, conditionally enable OOB.  Make sure
  that old QMP client requests will never be dropped by using
  monitor_suspend() when OOB not enabled. [Stefan]
- add new lock Monitor.qmp.qmp_queue_lock, then take it when queuing
  for requests/responses. [Stefan]
- move Monitor fields "qmp_requests" & "qmp_responses" into MonitorQMP
  struct for cleaness.
- write more tests for oob, and some smoke test with libvirt, hoping
  that it won't break too many things

v2 changelog:
- use 10-char hash for git commit ID [Eric]
- instead of changing qstring_get_str(), add new
  qstring_get_try_str(), and rename qobject_get_str() to
  qobject_get_try_str() to follow the naming rule [Eric]
- add one more patch to let object_property_get_str() leverage the new
  qobject_get_try_str(). [Eric]
- introduce JSONMessageEmitFunc in the JSON parser patch [Eric]
- use per-monitor request queue, rather than a global queue
- add queue flow control, when full (current queue depth: 8), send
  event to notify client (two new patches added for this)
- stop monitor thread first before destroying monitor objects [Dan]
- document that client should drop responses with unknown IDs.
- move all the documents into a single standalone patch.
- use iothread for the monitor io thread.
- respond in IO thread, rather than the dispatcher
- new patch: in QMP greeting message, add 'oob' field in
  'capabilities', so that client can explicitly know whether server
  supports oob

Please review.  Thanks.

Peter Xu (27):
  char-io: fix possible race on IOWatchPoll
  qobject: introduce qstring_get_try_str()
  qobject: introduce qobject_get_try_str()
  qobject: let object_property_get_str() use new API
  monitor: move skip_flush into monitor_data_init
  qjson: add "opaque" field to JSONMessageParser
  monitor: move the cur_mon hack deeper for QMP
  monitor: unify global init
  monitor: let mon_list be tail queue
  monitor: create monitor dedicate iothread
  monitor: allow to use IO thread for parsing
  qmp: introduce QMPCapability
  qmp: negociate QMP capabilities
  qmp: introduce some capability helpers
  monitor: introduce monitor_qmp_respond()
  monitor: let monitor_{suspend|resume} thread safe
  monitor: separate QMP parser and dispatcher
  qmp: add new event "request-dropped"
  monitor: send event when request queue full
  qapi: introduce new cmd option "allow-oob"
  qmp: support out-of-band (oob) execution
  qmp: let migrate-incoming allow out-of-band
  qmp: isolate responses into io thread
  monitor: enable IO thread for (qmp & !mux) typed
  docs: update QMP documents for OOB commands
  tests: qmp-test: verify command batching
  tests: qmp-test: add oob test

 chardev/char-io.c                |  16 +-
 docs/devel/qapi-code-gen.txt     |  51 +++-
 docs/interop/qmp-spec.txt        |  32 ++-
 include/monitor/monitor.h        |   2 +-
 include/qapi/qmp/dispatch.h      |   2 +
 include/qapi/qmp/json-streamer.h |  10 +-
 include/qapi/qmp/qstring.h       |   2 +
 monitor.c                        | 546 +++++++++++++++++++++++++++++++++------
 qapi-schema.json                 |  63 ++++-
 qapi/introspect.json             |   6 +-
 qapi/migration.json              |   3 +-
 qapi/qmp-dispatch.c              |  39 +++
 qga/main.c                       |   5 +-
 qobject/json-streamer.c          |   6 +-
 qobject/qjson.c                  |   5 +-
 qobject/qstring.c                |  21 ++
 qom/object.c                     |   9 +-
 scripts/qapi-commands.py         |  19 +-
 scripts/qapi-introspect.py       |  10 +-
 scripts/qapi.py                  |  15 +-
 scripts/qapi2texi.py             |   2 +-
 tests/libqtest.c                 |   5 +-
 tests/qapi-schema/test-qapi.py   |   2 +-
 tests/qmp-test.c                 | 122 ++++++++-
 trace-events                     |   2 +
 vl.c                             |   3 +-
 26 files changed, 876 insertions(+), 122 deletions(-)

-- 
2.13.5

[Qemu-devel] [RFC v3 01/27] char-io: fix possible race on IOWatchPoll

[Peter Xu]

This is not a problem if we are only having one single loop thread like
before.  However, after per-monitor thread is introduced, this is not
true any more, and the race can happen.

The race can be triggered with "make check -j8" sometimes:

  qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
  io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.

This patch keeps the reference for the watch object when creating in
io_add_watch_poll(), so that the object will never be released in the
context main loop, especially when the context loop is running in
another standalone thread.  Meanwhile, when we want to remove the watch
object, we always first detach the watch object from its owner context,
then we continue with the cleanup.

Without this patch, calling io_remove_watch_poll() in main loop thread
is not thread-safe, since the other per-monitor thread may be modifying
the watch object at the same time.

Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 chardev/char-io.c | 16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

diff --git a/chardev/char-io.c b/chardev/char-io.c
index f81052481a..50b5bac704 100644
--- a/chardev/char-io.c
+++ b/chardev/char-io.c
@@ -122,7 +122,6 @@ GSource *io_add_watch_poll(Chardev *chr,
     g_free(name);
 
     g_source_attach(&iwp->parent, context);
-    g_source_unref(&iwp->parent);
     return (GSource *)iwp;
 }
 
@@ -131,12 +130,25 @@ static void io_remove_watch_poll(GSource *source)
     IOWatchPoll *iwp;
 
     iwp = io_watch_poll_from_source(source);
+
+    /*
+     * Here the order of destruction really matters.  We need to first
+     * detach the IOWatchPoll object from the context (which may still
+     * be running in another loop thread), only after that could we
+     * continue to operate on iwp->src, or there may be race condition
+     * between current thread and the context loop thread.
+     *
+     * Let's blame the glib bug mentioned in commit 2b316774f6
+     * ("qemu-char: do not operate on sources from finalize
+     * callbacks") for this extra complexity.
+     */
+    g_source_destroy(&iwp->parent);
     if (iwp->src) {
         g_source_destroy(iwp->src);
         g_source_unref(iwp->src);
         iwp->src = NULL;
     }
-    g_source_destroy(&iwp->parent);
+    g_source_unref(&iwp->parent);
 }
 
 void remove_fd_in_watch(Chardev *chr)
-- 
2.13.5

[Qemu-devel] [RFC v3 02/27] qobject: introduce qstring_get_try_str()

[Peter Xu]

The only difference from qstring_get_str() is that it allows the qstring
to be NULL.  If so, NULL is returned.

CC: Eric Blake <eblake@redhat.com>
CC: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/qstring.h |  1 +
 qobject/qstring.c          | 10 ++++++++++
 2 files changed, 11 insertions(+)

diff --git a/include/qapi/qmp/qstring.h b/include/qapi/qmp/qstring.h
index 10076b7c8c..34278bd639 100644
--- a/include/qapi/qmp/qstring.h
+++ b/include/qapi/qmp/qstring.h
@@ -27,6 +27,7 @@ QString *qstring_from_str(const char *str);
 QString *qstring_from_substr(const char *str, int start, int end);
 size_t qstring_get_length(const QString *qstring);
 const char *qstring_get_str(const QString *qstring);
+const char *qstring_get_try_str(const QString *qstring);
 void qstring_append_int(QString *qstring, int64_t value);
 void qstring_append(QString *qstring, const char *str);
 void qstring_append_chr(QString *qstring, int c);
diff --git a/qobject/qstring.c b/qobject/qstring.c
index 5da7b5f37c..9df04e3c7b 100644
--- a/qobject/qstring.c
+++ b/qobject/qstring.c
@@ -129,6 +129,16 @@ const char *qstring_get_str(const QString *qstring)
 }
 
 /**
+ * qstring_get_try_str(): Return a pointer to the stored string
+ *
+ * NOTE: will return NULL if qstring is not provided.
+ */
+const char *qstring_get_try_str(const QString *qstring)
+{
+    return qstring ? qstring_get_str(qstring) : NULL;
+}
+
+/**
  * qstring_destroy_obj(): Free all memory allocated by a QString
  * object
  */
-- 
2.13.5

[Qemu-devel] [RFC v3 03/27] qobject: introduce qobject_get_try_str()

[Peter Xu]

A quick way to fetch string from qobject when it's a QString.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/qstring.h |  1 +
 qobject/qstring.c          | 11 +++++++++++
 2 files changed, 12 insertions(+)

diff --git a/include/qapi/qmp/qstring.h b/include/qapi/qmp/qstring.h
index 34278bd639..12ae4e1c29 100644
--- a/include/qapi/qmp/qstring.h
+++ b/include/qapi/qmp/qstring.h
@@ -28,6 +28,7 @@ QString *qstring_from_substr(const char *str, int start, int end);
 size_t qstring_get_length(const QString *qstring);
 const char *qstring_get_str(const QString *qstring);
 const char *qstring_get_try_str(const QString *qstring);
+const char *qobject_get_try_str(const QObject *qstring);
 void qstring_append_int(QString *qstring, int64_t value);
 void qstring_append(QString *qstring, const char *str);
 void qstring_append_chr(QString *qstring, int c);
diff --git a/qobject/qstring.c b/qobject/qstring.c
index 9df04e3c7b..bbe689a9e3 100644
--- a/qobject/qstring.c
+++ b/qobject/qstring.c
@@ -139,6 +139,17 @@ const char *qstring_get_try_str(const QString *qstring)
 }
 
 /**
+ * qobject_get_try_str(): Return a pointer of the backstore string
+ *
+ * NOTE: the string will only be returned if the object is valid, and
+ * its type is QString, otherwise NULL is returned.
+ */
+const char *qobject_get_try_str(const QObject *qstring)
+{
+    return qstring_get_try_str(qobject_to_qstring(qstring));
+}
+
+/**
  * qstring_destroy_obj(): Free all memory allocated by a QString
  * object
  */
-- 
2.13.5

[Qemu-devel] [RFC v3 04/27] qobject: let object_property_get_str() use new API

[Peter Xu]

We can simplify object_property_get_str() using the new
qobject_get_try_str().

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qom/object.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/qom/object.c b/qom/object.c
index c58c52d518..9cbeb51f0b 100644
--- a/qom/object.c
+++ b/qom/object.c
@@ -1116,18 +1116,15 @@ char *object_property_get_str(Object *obj, const char *name,
                               Error **errp)
 {
     QObject *ret = object_property_get_qobject(obj, name, errp);
-    QString *qstring;
     char *retval;
 
     if (!ret) {
         return NULL;
     }
-    qstring = qobject_to_qstring(ret);
-    if (!qstring) {
+
+    retval = g_strdup(qobject_get_try_str(ret));
+    if (!retval) {
         error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name, "string");
-        retval = NULL;
-    } else {
-        retval = g_strdup(qstring_get_str(qstring));
     }
 
     qobject_decref(ret);
-- 
2.13.5

[Qemu-devel] [RFC v3 05/27] monitor: move skip_flush into monitor_data_init

[Peter Xu]

It's part of the data init.  Collect it.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/monitor.c b/monitor.c
index e36fb5308d..3940737c1c 100644
--- a/monitor.c
+++ b/monitor.c
@@ -568,13 +568,14 @@ static void monitor_qapi_event_init(void)
 
 static void handle_hmp_command(Monitor *mon, const char *cmdline);
 
-static void monitor_data_init(Monitor *mon)
+static void monitor_data_init(Monitor *mon, bool skip_flush)
 {
     memset(mon, 0, sizeof(Monitor));
     qemu_mutex_init(&mon->out_lock);
     mon->outbuf = qstring_new();
     /* Use *mon_cmds by default. */
     mon->cmd_table = mon_cmds;
+    mon->skip_flush = skip_flush;
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -595,8 +596,7 @@ char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
     char *output = NULL;
     Monitor *old_mon, hmp;
 
-    monitor_data_init(&hmp);
-    hmp.skip_flush = true;
+    monitor_data_init(&hmp, true);
 
     old_mon = cur_mon;
     cur_mon = &hmp;
@@ -4089,7 +4089,7 @@ void monitor_init(Chardev *chr, int flags)
     }
 
     mon = g_malloc(sizeof(*mon));
-    monitor_data_init(mon);
+    monitor_data_init(mon, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
-- 
2.13.5

[Qemu-devel] [RFC v3 06/27] qjson: add "opaque" field to JSONMessageParser

[Peter Xu]

It'll be passed to emit() as well when it happens.  Since at it, add a
typedef for the emitter function.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/json-streamer.h | 10 ++++++++--
 monitor.c                        |  7 ++++---
 qga/main.c                       |  5 +++--
 qobject/json-streamer.c          |  6 ++++--
 qobject/qjson.c                  |  5 +++--
 tests/libqtest.c                 |  5 +++--
 6 files changed, 25 insertions(+), 13 deletions(-)

diff --git a/include/qapi/qmp/json-streamer.h b/include/qapi/qmp/json-streamer.h
index 00d8a23af8..9198b67342 100644
--- a/include/qapi/qmp/json-streamer.h
+++ b/include/qapi/qmp/json-streamer.h
@@ -23,9 +23,14 @@ typedef struct JSONToken {
     char str[];
 } JSONToken;
 
+struct JSONMessageParser;
+typedef void (*JSONMessageEmitFunc)(struct JSONMessageParser *parser,
+                                    GQueue *tokens, void *opaque);
+
 typedef struct JSONMessageParser
 {
-    void (*emit)(struct JSONMessageParser *parser, GQueue *tokens);
+    JSONMessageEmitFunc emit;
+    void *opaque;
     JSONLexer lexer;
     int brace_count;
     int bracket_count;
@@ -34,7 +39,8 @@ typedef struct JSONMessageParser
 } JSONMessageParser;
 
 void json_message_parser_init(JSONMessageParser *parser,
-                              void (*func)(JSONMessageParser *, GQueue *));
+                              JSONMessageEmitFunc func,
+                              void *opaque);
 
 int json_message_parser_feed(JSONMessageParser *parser,
                              const char *buffer, size_t size);
diff --git a/monitor.c b/monitor.c
index 3940737c1c..ab80d32c70 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3808,7 +3808,8 @@ static int monitor_can_read(void *opaque)
     return (mon->suspend_cnt == 0) ? 1 : 0;
 }
 
-static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
+static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
+                               void *opaque)
 {
     QObject *req, *rsp = NULL, *id = NULL;
     QDict *qdict = NULL;
@@ -3955,7 +3956,7 @@ static void monitor_qmp_event(void *opaque, int event)
         break;
     case CHR_EVENT_CLOSED:
         json_message_parser_destroy(&mon->qmp.parser);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
         mon_refcount--;
         monitor_fdsets_cleanup();
         break;
@@ -4105,7 +4106,7 @@ void monitor_init(Chardev *chr, int flags)
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
                                  monitor_qmp_event, NULL, mon, NULL, true);
         qemu_chr_fe_set_echo(&mon->chr, true);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
     } else {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_read,
                                  monitor_event, NULL, mon, NULL, true);
diff --git a/qga/main.c b/qga/main.c
index 62a62755bd..3b5ebbc1ee 100644
--- a/qga/main.c
+++ b/qga/main.c
@@ -593,7 +593,8 @@ static void process_command(GAState *s, QDict *req)
 }
 
 /* handle requests/control events coming in over the channel */
-static void process_event(JSONMessageParser *parser, GQueue *tokens)
+static void process_event(JSONMessageParser *parser, GQueue *tokens,
+                          void *opaque)
 {
     GAState *s = container_of(parser, GAState, parser);
     QDict *qdict;
@@ -1320,7 +1321,7 @@ static int run_agent(GAState *s, GAConfig *config, int socket_activation)
     s->command_state = ga_command_state_new();
     ga_command_state_init(s, s->command_state);
     ga_command_state_init_all(s->command_state);
-    json_message_parser_init(&s->parser, process_event);
+    json_message_parser_init(&s->parser, process_event, NULL);
 
 #ifndef _WIN32
     if (!register_signal_handlers()) {
diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c
index c51c2021f9..8fc1b15321 100644
--- a/qobject/json-streamer.c
+++ b/qobject/json-streamer.c
@@ -102,18 +102,20 @@ out_emit:
      */
     tokens = parser->tokens;
     parser->tokens = g_queue_new();
-    parser->emit(parser, tokens);
+    parser->emit(parser, tokens, parser->opaque);
     parser->token_size = 0;
 }
 
 void json_message_parser_init(JSONMessageParser *parser,
-                              void (*func)(JSONMessageParser *, GQueue *))
+                              JSONMessageEmitFunc func,
+                              void *opaque)
 {
     parser->emit = func;
     parser->brace_count = 0;
     parser->bracket_count = 0;
     parser->tokens = g_queue_new();
     parser->token_size = 0;
+    parser->opaque = opaque;
 
     json_lexer_init(&parser->lexer, json_message_process_token);
 }
diff --git a/qobject/qjson.c b/qobject/qjson.c
index 2e0930884e..f9766febe3 100644
--- a/qobject/qjson.c
+++ b/qobject/qjson.c
@@ -28,7 +28,8 @@ typedef struct JSONParsingState
     Error *err;
 } JSONParsingState;
 
-static void parse_json(JSONMessageParser *parser, GQueue *tokens)
+static void parse_json(JSONMessageParser *parser, GQueue *tokens,
+                       void *opaque)
 {
     JSONParsingState *s = container_of(parser, JSONParsingState, parser);
 
@@ -41,7 +42,7 @@ QObject *qobject_from_jsonv(const char *string, va_list *ap, Error **errp)
 
     state.ap = ap;
 
-    json_message_parser_init(&state.parser, parse_json);
+    json_message_parser_init(&state.parser, parse_json, NULL);
     json_message_parser_feed(&state.parser, string, strlen(string));
     json_message_parser_flush(&state.parser);
     json_message_parser_destroy(&state.parser);
diff --git a/tests/libqtest.c b/tests/libqtest.c
index 0ec8af2923..6e5ddc2ffd 100644
--- a/tests/libqtest.c
+++ b/tests/libqtest.c
@@ -412,7 +412,8 @@ typedef struct {
     QDict *response;
 } QMPResponseParser;
 
-static void qmp_response(JSONMessageParser *parser, GQueue *tokens)
+static void qmp_response(JSONMessageParser *parser, GQueue *tokens,
+                         void *opaque)
 {
     QMPResponseParser *qmp = container_of(parser, QMPResponseParser, parser);
     QObject *obj;
@@ -434,7 +435,7 @@ QDict *qmp_fd_receive(int fd)
     bool log = getenv("QTEST_LOG") != NULL;
 
     qmp.response = NULL;
-    json_message_parser_init(&qmp.parser, qmp_response);
+    json_message_parser_init(&qmp.parser, qmp_response, NULL);
     while (!qmp.response) {
         ssize_t len;
         char c;
-- 
2.13.5

[Qemu-devel] [RFC v3 07/27] monitor: move the cur_mon hack deeper for QMP

[Peter Xu]

In monitor_qmp_read(), we have the hack to temporarily replace the
cur_mon pointer.  Now we move this hack deeper inside the QMP dispatcher
routine since the Monitor pointer can be passed in to that using the new
JSON Parser opaque field now.

This does not make much sense as a single patch.  However, this will be
a big step for the next patch, when the QMP dispatcher routine will be
split from the QMP parser.

Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/monitor.c b/monitor.c
index ab80d32c70..322dfb5f31 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3813,7 +3813,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
 {
     QObject *req, *rsp = NULL, *id = NULL;
     QDict *qdict = NULL;
-    Monitor *mon = cur_mon;
+    Monitor *mon = opaque, *old_mon;
     Error *err = NULL;
 
     req = json_parser_parse_err(tokens, NULL, &err);
@@ -3838,8 +3838,13 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
         QDECREF(req_json);
     }
 
+    old_mon = cur_mon;
+    cur_mon = mon;
+
     rsp = qmp_dispatch(cur_mon->qmp.commands, req);
 
+    cur_mon = old_mon;
+
     if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
         qdict = qdict_get_qdict(qobject_to_qdict(rsp), "error");
         if (qdict
@@ -3876,13 +3881,9 @@ err_out:
 
 static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
 {
-    Monitor *old_mon = cur_mon;
-
-    cur_mon = opaque;
-
-    json_message_parser_feed(&cur_mon->qmp.parser, (const char *) buf, size);
+    Monitor *mon = opaque;
 
-    cur_mon = old_mon;
+    json_message_parser_feed(&mon->qmp.parser, (const char *) buf, size);
 }
 
 static void monitor_read(void *opaque, const uint8_t *buf, int size)
@@ -3956,7 +3957,7 @@ static void monitor_qmp_event(void *opaque, int event)
         break;
     case CHR_EVENT_CLOSED:
         json_message_parser_destroy(&mon->qmp.parser);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
         mon_refcount--;
         monitor_fdsets_cleanup();
         break;
@@ -4106,7 +4107,7 @@ void monitor_init(Chardev *chr, int flags)
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
                                  monitor_qmp_event, NULL, mon, NULL, true);
         qemu_chr_fe_set_echo(&mon->chr, true);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
     } else {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_read,
                                  monitor_event, NULL, mon, NULL, true);
-- 
2.13.5

[Qemu-devel] [RFC v3 08/27] monitor: unify global init

[Peter Xu]

There are many places for monitor init its globals, at least:

- monitor_init_qmp_commands() at the very beginning
- single function to init monitor_lock
- in the first entry of monitor_init() using "is_first_init"

Unify them a bit.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/monitor/monitor.h |  2 +-
 monitor.c                 | 25 ++++++++++---------------
 vl.c                      |  3 ++-
 3 files changed, 13 insertions(+), 17 deletions(-)

diff --git a/include/monitor/monitor.h b/include/monitor/monitor.h
index 83ea4a1aaf..3a5128ab1b 100644
--- a/include/monitor/monitor.h
+++ b/include/monitor/monitor.h
@@ -16,7 +16,7 @@ extern Monitor *cur_mon;
 
 bool monitor_cur_is_qmp(void);
 
-void monitor_init_qmp_commands(void);
+void monitor_init_globals(void);
 void monitor_init(Chardev *chr, int flags);
 void monitor_cleanup(void);
 
diff --git a/monitor.c b/monitor.c
index 322dfb5f31..ac5313023b 100644
--- a/monitor.c
+++ b/monitor.c
@@ -1001,7 +1001,7 @@ static void qmp_unregister_commands_hack(void)
 #endif
 }
 
-void monitor_init_qmp_commands(void)
+static void monitor_init_qmp_commands(void)
 {
     /*
      * Two command lists:
@@ -4034,6 +4034,14 @@ static void sortcmdlist(void)
     qsort((void *)info_cmds, array_num, elem_size, compare_mon_cmd);
 }
 
+void monitor_init_globals(void)
+{
+    monitor_init_qmp_commands();
+    monitor_qapi_event_init();
+    sortcmdlist();
+    qemu_mutex_init(&monitor_lock);
+}
+
 /* These functions just adapt the readline interface in a typesafe way.  We
  * could cast function pointers but that discards compiler checks.
  */
@@ -4074,23 +4082,10 @@ void error_vprintf_unless_qmp(const char *fmt, va_list ap)
     }
 }
 
-static void __attribute__((constructor)) monitor_lock_init(void)
-{
-    qemu_mutex_init(&monitor_lock);
-}
-
 void monitor_init(Chardev *chr, int flags)
 {
-    static int is_first_init = 1;
-    Monitor *mon;
-
-    if (is_first_init) {
-        monitor_qapi_event_init();
-        sortcmdlist();
-        is_first_init = 0;
-    }
+    Monitor *mon = g_malloc(sizeof(*mon));
 
-    mon = g_malloc(sizeof(*mon));
     monitor_data_init(mon, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
diff --git a/vl.c b/vl.c
index ec299099ff..2118750a9c 100644
--- a/vl.c
+++ b/vl.c
@@ -3144,7 +3144,6 @@ int main(int argc, char **argv, char **envp)
     qemu_init_exec_dir(argv[0]);
 
     module_call_init(MODULE_INIT_QOM);
-    monitor_init_qmp_commands();
 
     qemu_add_opts(&qemu_drive_opts);
     qemu_add_drive_opts(&qemu_legacy_drive_opts);
@@ -4677,6 +4676,8 @@ int main(int argc, char **argv, char **envp)
 
     parse_numa_opts(current_machine);
 
+    monitor_init_globals();
+
     if (qemu_opts_foreach(qemu_find_opts("mon"),
                           mon_init_func, NULL, NULL)) {
         exit(1);
-- 
2.13.5

[Qemu-devel] [RFC v3 09/27] monitor: let mon_list be tail queue

[Peter Xu]

It was QLIST.  I want to use this list to do monitor priority job later,
which need tail insertion ability.  So switching to a tail queue.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/monitor.c b/monitor.c
index ac5313023b..a70ab5606b 100644
--- a/monitor.c
+++ b/monitor.c
@@ -205,7 +205,7 @@ struct Monitor {
     void *password_opaque;
     mon_cmd_t *cmd_table;
     QLIST_HEAD(,mon_fd_t) fds;
-    QLIST_ENTRY(Monitor) entry;
+    QTAILQ_ENTRY(Monitor) entry;
 };
 
 /* QMP checker flags */
@@ -214,7 +214,7 @@ struct Monitor {
 /* Protects mon_list, monitor_event_state.  */
 static QemuMutex monitor_lock;
 
-static QLIST_HEAD(mon_list, Monitor) mon_list;
+static QTAILQ_HEAD(mon_list, Monitor) mon_list;
 static QLIST_HEAD(mon_fdsets, MonFdset) mon_fdsets;
 static int mon_refcount;
 
@@ -415,7 +415,7 @@ static void monitor_qapi_event_emit(QAPIEvent event, QDict *qdict)
     Monitor *mon;
 
     trace_monitor_protocol_event_emit(event, qdict);
-    QLIST_FOREACH(mon, &mon_list, entry) {
+    QTAILQ_FOREACH(mon, &mon_list, entry) {
         if (monitor_is_qmp(mon)
             && mon->qmp.commands != &qmp_cap_negotiation_commands) {
             monitor_json_emitter(mon, QOBJECT(qdict));
@@ -4118,8 +4118,8 @@ void monitor_cleanup(void)
     Monitor *mon, *next;
 
     qemu_mutex_lock(&monitor_lock);
-    QLIST_FOREACH_SAFE(mon, &mon_list, entry, next) {
-        QLIST_REMOVE(mon, entry);
+    QTAILQ_FOREACH_SAFE(mon, &mon_list, entry, next) {
+        QTAILQ_REMOVE(&mon_list, mon, entry);
         monitor_data_destroy(mon);
         g_free(mon);
     }
-- 
2.13.5

[Qemu-devel] [RFC v3 10/27] monitor: create monitor dedicate iothread

[Peter Xu]

Create one IOThread for the monitors, prepared to handle all the
input/output IOs using existing iothread framework.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/monitor.c b/monitor.c
index a70ab5606b..df1ec8d037 100644
--- a/monitor.c
+++ b/monitor.c
@@ -76,6 +76,7 @@
 #include "qmp-introspect.h"
 #include "sysemu/qtest.h"
 #include "sysemu/cpus.h"
+#include "sysemu/iothread.h"
 #include "qemu/cutils.h"
 #include "qapi/qmp/dispatch.h"
 
@@ -208,6 +209,12 @@ struct Monitor {
     QTAILQ_ENTRY(Monitor) entry;
 };
 
+struct MonitorGlobal {
+    IOThread *mon_iothread;
+};
+
+static struct MonitorGlobal mon_global;
+
 /* QMP checker flags */
 #define QMP_ACCEPT_UNKNOWNS 1
 
@@ -4034,12 +4041,29 @@ static void sortcmdlist(void)
     qsort((void *)info_cmds, array_num, elem_size, compare_mon_cmd);
 }
 
+static GMainContext *monitor_io_context_get(void)
+{
+    return iothread_get_g_main_context(mon_global.mon_iothread);
+}
+
+static void monitor_iothread_init(void)
+{
+    mon_global.mon_iothread = iothread_create("monitor_iothread",
+                                              &error_abort);
+    /*
+     * GContext in iothread is using lazy init - the first time we
+     * fetch the context we'll have that initialized.
+     */
+    monitor_io_context_get();
+}
+
 void monitor_init_globals(void)
 {
     monitor_init_qmp_commands();
     monitor_qapi_event_init();
     sortcmdlist();
     qemu_mutex_init(&monitor_lock);
+    monitor_iothread_init();
 }
 
 /* These functions just adapt the readline interface in a typesafe way.  We
@@ -4117,6 +4141,13 @@ void monitor_cleanup(void)
 {
     Monitor *mon, *next;
 
+    /*
+     * We need to explicitly stop the iothread (but not destroy it),
+     * cleanup the monitor resources, then destroy the iothread.  See
+     * again on the glib bug mentioned in 2b316774f6 for a reason.
+     */
+    iothread_stop(mon_global.mon_iothread);
+
     qemu_mutex_lock(&monitor_lock);
     QTAILQ_FOREACH_SAFE(mon, &mon_list, entry, next) {
         QTAILQ_REMOVE(&mon_list, mon, entry);
@@ -4124,6 +4155,9 @@ void monitor_cleanup(void)
         g_free(mon);
     }
     qemu_mutex_unlock(&monitor_lock);
+
+    iothread_destroy(mon_global.mon_iothread);
+    mon_global.mon_iothread = NULL;
 }
 
 QemuOptsList qemu_mon_opts = {
-- 
2.13.5

[Qemu-devel] [RFC v3 11/27] monitor: allow to use IO thread for parsing

[Peter Xu]

For each Monitor, add one field "use_io_thr" to show whether it will be
using the dedicated monitor IO thread to handle input/output.  When set,
monitor IO parsing work will be offloaded to dedicated monitor IO
thread, rather than the original main loop thread.

This only works for QMP.  HMP will always be run on main loop thread.

Currently we're still keeping use_io_thr to off always.  Will turn it on
later at some point.

One thing to mention is that we cannot set use_io_thr for every QMP
monitors.  The problem is that MUXed typed chardevs may not work well
with it now. When MUX is used, frontend of chardev can be the monitor
plus something else.  The only thing we know would be safe to be run
outside main thread so far is the monitor frontend. All the rest of the
frontends should still be run in main thread only.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 38 ++++++++++++++++++++++++++++++--------
 1 file changed, 30 insertions(+), 8 deletions(-)

diff --git a/monitor.c b/monitor.c
index df1ec8d037..7255c110f9 100644
--- a/monitor.c
+++ b/monitor.c
@@ -191,6 +191,7 @@ struct Monitor {
     int flags;
     int suspend_cnt;
     bool skip_flush;
+    bool use_io_thr;
 
     QemuMutex out_lock;
     QString *outbuf;
@@ -575,7 +576,8 @@ static void monitor_qapi_event_init(void)
 
 static void handle_hmp_command(Monitor *mon, const char *cmdline);
 
-static void monitor_data_init(Monitor *mon, bool skip_flush)
+static void monitor_data_init(Monitor *mon, bool skip_flush,
+                              bool use_io_thr)
 {
     memset(mon, 0, sizeof(Monitor));
     qemu_mutex_init(&mon->out_lock);
@@ -583,6 +585,7 @@ static void monitor_data_init(Monitor *mon, bool skip_flush)
     /* Use *mon_cmds by default. */
     mon->cmd_table = mon_cmds;
     mon->skip_flush = skip_flush;
+    mon->use_io_thr = use_io_thr;
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -603,7 +606,7 @@ char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
     char *output = NULL;
     Monitor *old_mon, hmp;
 
-    monitor_data_init(&hmp, true);
+    monitor_data_init(&hmp, true, false);
 
     old_mon = cur_mon;
     cur_mon = &hmp;
@@ -4109,8 +4112,9 @@ void error_vprintf_unless_qmp(const char *fmt, va_list ap)
 void monitor_init(Chardev *chr, int flags)
 {
     Monitor *mon = g_malloc(sizeof(*mon));
+    GMainContext *context;
 
-    monitor_data_init(mon, false);
+    monitor_data_init(mon, false, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
@@ -4122,19 +4126,37 @@ void monitor_init(Chardev *chr, int flags)
         monitor_read_command(mon, 0);
     }
 
+    if (mon->use_io_thr) {
+        /*
+         * When use_io_thr is set, we use the global shared dedicated
+         * IO thread for this monitor to handle input/output.
+         */
+        context = monitor_io_context_get();
+        /* We should have inited globals before reaching here. */
+        assert(context);
+    } else {
+        /* The default main loop, which is the main thread */
+        context = NULL;
+    }
+
+    /*
+     * Add the monitor before running it (which is triggered by
+     * qemu_chr_fe_set_handlers), otherwise one monitor may find
+     * itself not on the mon_list when running.
+     */
+    qemu_mutex_lock(&monitor_lock);
+    QTAILQ_INSERT_HEAD(&mon_list, mon, entry);
+    qemu_mutex_unlock(&monitor_lock);
+
     if (monitor_is_qmp(mon)) {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
-                                 monitor_qmp_event, NULL, mon, NULL, true);
+                                 monitor_qmp_event, NULL, mon, context, true);
         qemu_chr_fe_set_echo(&mon->chr, true);
         json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
     } else {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_read,
                                  monitor_event, NULL, mon, NULL, true);
     }
-
-    qemu_mutex_lock(&monitor_lock);
-    QLIST_INSERT_HEAD(&mon_list, mon, entry);
-    qemu_mutex_unlock(&monitor_lock);
 }
 
 void monitor_cleanup(void)
-- 
2.13.5

[Qemu-devel] [RFC v3 12/27] qmp: introduce QMPCapability

[Peter Xu]

There was no QMP capabilities defined.  Define the first "oob" as
capability to allow out-of-band messages.

Also, touch up qmp-test.c to test the new bits.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c        | 15 +++++++++++++--
 qapi-schema.json | 13 +++++++++++++
 tests/qmp-test.c | 10 +++++++++-
 3 files changed, 35 insertions(+), 3 deletions(-)

diff --git a/monitor.c b/monitor.c
index 7255c110f9..dd192b6b91 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3944,12 +3944,23 @@ void monitor_resume(Monitor *mon)
 
 static QObject *get_qmp_greeting(void)
 {
+    QDict *result = qdict_new(), *qmp = qdict_new();
+    QList *cap_list = qlist_new();
     QObject *ver = NULL;
+    QMPCapability cap;
+
+    qdict_put(result, "QMP", qmp);
 
     qmp_marshal_query_version(NULL, &ver, NULL);
+    qdict_put_obj(qmp, "version", ver);
+
+    for (cap = 0; cap < QMP_CAPABILITY__MAX; cap++) {
+        qlist_append(cap_list, qstring_from_str(
+                         QMPCapability_str(cap)));
+    }
+    qdict_put(qmp, "capabilities", cap_list);
 
-    return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': []}}",
-                              ver);
+    return QOBJECT(result);
 }
 
 static void monitor_qmp_event(void *opaque, int event)
diff --git a/qapi-schema.json b/qapi-schema.json
index 18457954a8..03201578b4 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -119,6 +119,19 @@
 { 'command': 'qmp_capabilities' }
 
 ##
+# @QMPCapability:
+#
+# QMP supported capabilities to be broadcasted to the clients.
+#
+# @oob:   QMP ability to support Out-Of-Band requests.
+#
+# Since: 2.12
+#
+##
+{ 'enum': 'QMPCapability',
+  'data': [ 'oob' ] }
+
+##
 # @VersionTriple:
 #
 # A three-part version number.
diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index c5a5c10b41..292c5f135a 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -17,6 +17,7 @@
 #include "qapi/qobject-input-visitor.h"
 #include "qapi/util.h"
 #include "qapi/visitor.h"
+#include "qapi/qmp/qstring.h"
 
 const char common_args[] = "-nodefaults -machine none";
 
@@ -75,6 +76,8 @@ static void test_qmp_protocol(void)
 {
     QDict *resp, *q, *ret;
     QList *capabilities;
+    const QListEntry *entry;
+    QString *qstr;
 
     global_qtest = qtest_init_without_qmp_handshake(common_args);
 
@@ -84,7 +87,12 @@ static void test_qmp_protocol(void)
     g_assert(q);
     test_version(qdict_get(q, "version"));
     capabilities = qdict_get_qlist(q, "capabilities");
-    g_assert(capabilities && qlist_empty(capabilities));
+    g_assert(capabilities);
+    entry = qlist_first(capabilities);
+    g_assert(entry);
+    qstr = qobject_to_qstring(entry->value);
+    g_assert(qstr);
+    g_assert_cmpstr(qstring_get_str(qstr), ==, "oob");
     QDECREF(resp);
 
     /* Test valid command before handshake */
-- 
2.13.5

[Qemu-devel] [RFC v3 13/27] qmp: negociate QMP capabilities

[Peter Xu]

After this patch, we will allow QMP clients to enable QMP capabilities
when sending the first "qmp_capabilities" command.  Originally we are
starting QMP session with no arguments like:

  { "execute": "qmp_capabilities" }

Now we can enable some QMP capabilities using (take OOB as example,
which is the only one capability that we support):

  { "execute": "qmp_capabilities",
    "argument": { "enable": [ "oob" ] } }

When the "argument" key is not provided, no capability is enabled.

For capability "oob", the monitor needs to be run on dedicated IO
thread, otherwise the command will fail.  For example, trying to enable
OOB on a MUXed typed QMP monitor will fail.

One thing to mention is that, QMP capabilities are per-monitor, and also
when the connection is closed due to some reason, the capabilities will
be reset.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c        | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 qapi-schema.json | 15 ++++++++++++---
 2 files changed, 68 insertions(+), 4 deletions(-)

diff --git a/monitor.c b/monitor.c
index dd192b6b91..dba56fbcdf 100644
--- a/monitor.c
+++ b/monitor.c
@@ -167,6 +167,7 @@ typedef struct {
      * mode.
      */
     QmpCommandList *commands;
+    bool qmp_caps[QMP_CAPABILITY__MAX];
 } MonitorQMP;
 
 /*
@@ -1037,8 +1038,42 @@ static void monitor_init_qmp_commands(void)
                          qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
 }
 
-void qmp_qmp_capabilities(Error **errp)
+static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
+                           Error **errp)
+{
+    for (; list; list = list->next) {
+        assert(list->value < QMP_CAPABILITY__MAX);
+        switch (list->value) {
+        case QMP_CAPABILITY_OOB:
+            if (!mon->use_io_thr) {
+                /*
+                 * Out-Of-Band only works with monitors that are
+                 * running on dedicated IOThread.
+                 */
+                error_setg(errp, "This monitor does not support "
+                           "Out-Of-Band (OOB)");
+                return;
+            }
+            break;
+        default:
+            break;
+        }
+    }
+}
+
+/* This function should only be called after capabilities are checked. */
+static void qmp_caps_apply(Monitor *mon, QMPCapabilityList *list)
+{
+    for (; list; list = list->next) {
+        mon->qmp.qmp_caps[list->value] = true;
+    }
+}
+
+void qmp_qmp_capabilities(bool has_enable, QMPCapabilityList *enable,
+                          Error **errp)
 {
+    Error *local_err = NULL;
+
     if (cur_mon->qmp.commands == &qmp_commands) {
         error_set(errp, ERROR_CLASS_COMMAND_NOT_FOUND,
                   "Capabilities negotiation is already complete, command "
@@ -1046,6 +1081,20 @@ void qmp_qmp_capabilities(Error **errp)
         return;
     }
 
+    /* Enable QMP capabilities provided by the guest if applicable. */
+    if (has_enable) {
+        qmp_caps_check(cur_mon, enable, &local_err);
+        if (local_err) {
+            /*
+             * Failed check on either of the capabilities will fail
+             * the apply of all.
+             */
+            error_propagate(errp, local_err);
+            return;
+        }
+        qmp_caps_apply(cur_mon, enable);
+    }
+
     cur_mon->qmp.commands = &qmp_commands;
 }
 
@@ -3963,6 +4012,11 @@ static QObject *get_qmp_greeting(void)
     return QOBJECT(result);
 }
 
+static void monitor_qmp_caps_reset(Monitor *mon)
+{
+    memset(mon->qmp.qmp_caps, 0, sizeof(mon->qmp.qmp_caps));
+}
+
 static void monitor_qmp_event(void *opaque, int event)
 {
     QObject *data;
@@ -3971,6 +4025,7 @@ static void monitor_qmp_event(void *opaque, int event)
     switch (event) {
     case CHR_EVENT_OPENED:
         mon->qmp.commands = &qmp_cap_negotiation_commands;
+        monitor_qmp_caps_reset(mon);
         data = get_qmp_greeting();
         monitor_json_emitter(mon, data);
         qobject_decref(data);
diff --git a/qapi-schema.json b/qapi-schema.json
index 03201578b4..531fd4c0db 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -102,21 +102,30 @@
 #
 # Enable QMP capabilities.
 #
-# Arguments: None.
+# Arguments:
+#
+# @enable:    List of QMPCapabilities to enable, which is optional.
+#             If not provided, it means no QMP capabilities will be
+#             enabled.  (since 2.12)
 #
 # Example:
 #
-# -> { "execute": "qmp_capabilities" }
+# -> { "execute": "qmp_capabilities",
+#      "arguments": { "enable": [ "oob" ] } }
 # <- { "return": {} }
 #
 # Notes: This command is valid exactly when first connecting: it must be
 # issued before any other command will be accepted, and will fail once the
 # monitor is accepting other commands. (see qemu docs/interop/qmp-spec.txt)
 #
+# QMP client needs to explicitly enable QMP capabilities, otherwise
+# all the QMP capabilities will be turned off by default.
+#
 # Since: 0.13
 #
 ##
-{ 'command': 'qmp_capabilities' }
+{ 'command': 'qmp_capabilities',
+  'data': { '*enable': [ 'QMPCapability' ] } }
 
 ##
 # @QMPCapability:
-- 
2.13.5

[Qemu-devel] [RFC v3 14/27] qmp: introduce some capability helpers

[Peter Xu]

Introduce qmp_cap_enabled() and qmp_oob_enabled() helpers.
---
 monitor.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/monitor.c b/monitor.c
index dba56fbcdf..442d711d5d 100644
--- a/monitor.c
+++ b/monitor.c
@@ -1038,6 +1038,16 @@ static void monitor_init_qmp_commands(void)
                          qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
 }
 
+static bool qmp_cap_enabled(Monitor *mon, QMPCapability cap)
+{
+    return mon->qmp.qmp_caps[cap];
+}
+
+static bool qmp_oob_enabled(Monitor *mon)
+{
+    return qmp_cap_enabled(mon, QMP_CAPABILITY_OOB);
+}
+
 static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
                            Error **errp)
 {
-- 
2.13.5

[Qemu-devel] [RFC v3 15/27] monitor: introduce monitor_qmp_respond()

[Peter Xu]

A tiny refactoring, preparing to split the QMP dispatcher away.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 48 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 31 insertions(+), 17 deletions(-)

diff --git a/monitor.c b/monitor.c
index 442d711d5d..1e87de87f8 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3877,6 +3877,36 @@ static int monitor_can_read(void *opaque)
     return (mon->suspend_cnt == 0) ? 1 : 0;
 }
 
+/*
+ * When rsp/err/id is passed in, the function will be responsible for
+ * the cleanup.
+ */
+static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
+                                Error *err, QObject *id)
+{
+    QDict *qdict = NULL;
+
+    if (err) {
+        qdict = qdict_new();
+        qdict_put_obj(qdict, "error", qmp_build_error_object(err));
+        error_free(err);
+        rsp = QOBJECT(qdict);
+    }
+
+    if (rsp) {
+        if (id) {
+            /* This is for the qdict below. */
+            qobject_incref(id);
+            qdict_put_obj(qobject_to_qdict(rsp), "id", id);
+        }
+
+        monitor_json_emitter(mon, rsp);
+    }
+
+    qobject_decref(id);
+    qobject_decref(rsp);
+}
+
 static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
                                void *opaque)
 {
@@ -3927,24 +3957,8 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
     }
 
 err_out:
-    if (err) {
-        qdict = qdict_new();
-        qdict_put_obj(qdict, "error", qmp_build_error_object(err));
-        error_free(err);
-        rsp = QOBJECT(qdict);
-    }
+    monitor_qmp_respond(mon, rsp, err, id);
 
-    if (rsp) {
-        if (id) {
-            qdict_put_obj(qobject_to_qdict(rsp), "id", id);
-            id = NULL;
-        }
-
-        monitor_json_emitter(mon, rsp);
-    }
-
-    qobject_decref(id);
-    qobject_decref(rsp);
     qobject_decref(req);
 }
 
-- 
2.13.5

[Qemu-devel] [RFC v3 16/27] monitor: let monitor_{suspend|resume} thread safe

[Peter Xu]

Monitor code now can be run in more than one thread.  Let the suspend
and resume code for thread safety.
---
 monitor.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/monitor.c b/monitor.c
index 1e87de87f8..47e969244d 100644
--- a/monitor.c
+++ b/monitor.c
@@ -4003,7 +4003,7 @@ int monitor_suspend(Monitor *mon)
 {
     if (!mon->rs)
         return -ENOTTY;
-    mon->suspend_cnt++;
+    atomic_inc(&mon->suspend_cnt);
     return 0;
 }
 
@@ -4011,7 +4011,7 @@ void monitor_resume(Monitor *mon)
 {
     if (!mon->rs)
         return;
-    if (--mon->suspend_cnt == 0)
+    if (atomic_dec_fetch(&mon->suspend_cnt) == 0)
         readline_show_prompt(mon->rs);
 }
 
-- 
2.13.5

[Qemu-devel] [RFC v3 17/27] monitor: separate QMP parser and dispatcher

[Peter Xu]

Originally QMP goes throw these steps:

  JSON Parser --> QMP Dispatcher --> Respond
      /|\    (2)                (3)     |
   (1) |                               \|/ (4)
       +---------  main thread  --------+

This patch does this:

  JSON Parser     QMP Dispatcher --> Respond
      /|\ |           /|\       (4)     |
       |  | (2)        | (3)            |  (5)
   (1) |  +----->      |               \|/
       +---------  main thread  <-------+

So the parsing job and the dispatching job is isolated now.  It gives us
a chance in following up patches to totally move the parser outside.

The isloation is done using one QEMUBH. Only one dispatcher QEMUBH is
used for all the monitors.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 179 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 157 insertions(+), 22 deletions(-)

diff --git a/monitor.c b/monitor.c
index 47e969244d..e327ae115b 100644
--- a/monitor.c
+++ b/monitor.c
@@ -168,6 +168,10 @@ typedef struct {
      */
     QmpCommandList *commands;
     bool qmp_caps[QMP_CAPABILITY__MAX];
+    /* Protects qmp request/response queue. */
+    QemuMutex qmp_queue_lock;
+    /* Input queue that holds all the parsed QMP requests */
+    GQueue *qmp_requests;
 } MonitorQMP;
 
 /*
@@ -213,6 +217,8 @@ struct Monitor {
 
 struct MonitorGlobal {
     IOThread *mon_iothread;
+    /* Bottom half to dispatch the requests received from IO thread */
+    QEMUBH *qmp_dispatcher_bh;
 };
 
 static struct MonitorGlobal mon_global;
@@ -582,11 +588,13 @@ static void monitor_data_init(Monitor *mon, bool skip_flush,
 {
     memset(mon, 0, sizeof(Monitor));
     qemu_mutex_init(&mon->out_lock);
+    qemu_mutex_init(&mon->qmp.qmp_queue_lock);
     mon->outbuf = qstring_new();
     /* Use *mon_cmds by default. */
     mon->cmd_table = mon_cmds;
     mon->skip_flush = skip_flush;
     mon->use_io_thr = use_io_thr;
+    mon->qmp.qmp_requests = g_queue_new();
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -599,6 +607,8 @@ static void monitor_data_destroy(Monitor *mon)
     g_free(mon->rs);
     QDECREF(mon->outbuf);
     qemu_mutex_destroy(&mon->out_lock);
+    qemu_mutex_destroy(&mon->qmp.qmp_queue_lock);
+    g_queue_free(mon->qmp.qmp_requests);
 }
 
 char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
@@ -3907,29 +3917,31 @@ static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
     qobject_decref(rsp);
 }
 
-static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
-                               void *opaque)
+struct QMPRequest {
+    /* Owner of the request */
+    Monitor *mon;
+    /* "id" field of the request */
+    QObject *id;
+    /* Request object to be handled */
+    QObject *req;
+};
+typedef struct QMPRequest QMPRequest;
+
+/*
+ * Dispatch one single QMP request. The function will free the req_obj
+ * and objects inside it before return.
+ */
+static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
 {
-    QObject *req, *rsp = NULL, *id = NULL;
+    Monitor *mon, *old_mon;
+    QObject *req, *rsp = NULL, *id;
     QDict *qdict = NULL;
-    Monitor *mon = opaque, *old_mon;
-    Error *err = NULL;
 
-    req = json_parser_parse_err(tokens, NULL, &err);
-    if (!req && !err) {
-        /* json_parser_parse_err() sucks: can fail without setting @err */
-        error_setg(&err, QERR_JSON_PARSING);
-    }
-    if (err) {
-        goto err_out;
-    }
+    req = req_obj->req;
+    mon = req_obj->mon;
+    id = req_obj->id;
 
-    qdict = qobject_to_qdict(req);
-    if (qdict) {
-        id = qdict_get(qdict, "id");
-        qobject_incref(id);
-        qdict_del(qdict, "id");
-    } /* else will fail qmp_dispatch() */
+    g_free(req_obj);
 
     if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
         QString *req_json = qobject_to_json(req);
@@ -3940,7 +3952,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
     old_mon = cur_mon;
     cur_mon = mon;
 
-    rsp = qmp_dispatch(cur_mon->qmp.commands, req);
+    rsp = qmp_dispatch(mon->qmp.commands, req);
 
     cur_mon = old_mon;
 
@@ -3956,12 +3968,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
         }
     }
 
-err_out:
-    monitor_qmp_respond(mon, rsp, err, id);
+    /* Respond if necessary */
+    monitor_qmp_respond(mon, rsp, NULL, id);
+
+    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
+    if (!qmp_oob_enabled(mon)) {
+        monitor_resume(mon);
+    }
 
     qobject_decref(req);
 }
 
+/*
+ * Pop one QMP request from monitor queues, return NULL if not found.
+ * We are using round-robin fasion to pop the request, to avoid
+ * processing command only on a very busy monitor.  To achieve that,
+ * when we processed one request on specific monitor, we put that
+ * monitor to the end of mon_list queue.
+ */
+static QMPRequest *monitor_qmp_requests_pop_one(void)
+{
+    QMPRequest *req_obj = NULL;
+    Monitor *mon;
+
+    qemu_mutex_lock(&monitor_lock);
+
+    QTAILQ_FOREACH(mon, &mon_list, entry) {
+        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+        req_obj = g_queue_pop_head(mon->qmp.qmp_requests);
+        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+        if (req_obj) {
+            break;
+        }
+    }
+
+    if (req_obj) {
+        /*
+         * We found one request on the monitor. Degrade this monitor's
+         * priority to lowest by re-inserting it to end of queue.
+         */
+        QTAILQ_REMOVE(&mon_list, mon, entry);
+        QTAILQ_INSERT_TAIL(&mon_list, mon, entry);
+    }
+
+    qemu_mutex_unlock(&monitor_lock);
+
+    return req_obj;
+}
+
+static void monitor_qmp_bh_dispatcher(void *data)
+{
+    QMPRequest *req_obj;
+
+    while (true) {
+        req_obj = monitor_qmp_requests_pop_one();
+        if (!req_obj) {
+            break;
+        }
+        monitor_qmp_dispatch_one(req_obj);
+    }
+}
+
+static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
+                               void *opaque)
+{
+    QObject *req, *id = NULL;
+    QDict *qdict = NULL;
+    Monitor *mon = opaque;
+    Error *err = NULL;
+    QMPRequest *req_obj;
+
+    req = json_parser_parse_err(tokens, NULL, &err);
+    if (!req && !err) {
+        /* json_parser_parse_err() sucks: can fail without setting @err */
+        error_setg(&err, QERR_JSON_PARSING);
+    }
+    if (err) {
+        monitor_qmp_respond(mon, NULL, err, NULL);
+        qobject_decref(req);
+        return;
+    }
+
+    qdict = qobject_to_qdict(req);
+    if (qdict) {
+        id = qdict_get(qdict, "id");
+        qobject_incref(id);
+        qdict_del(qdict, "id");
+    } /* else will fail qmp_dispatch() */
+
+    req_obj = g_new0(QMPRequest, 1);
+    req_obj->mon = mon;
+    req_obj->id = id;
+    req_obj->req = req;
+
+    /*
+     * Put the request to the end of queue so that requests will be
+     * handled in time order.  Ownership for req_obj, req, id,
+     * etc. will be delivered to the handler side.
+     */
+    qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+    g_queue_push_tail(mon->qmp.qmp_requests, req_obj);
+    qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+
+    /* Kick the dispatcher routine */
+    qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
+
+    /*
+     * If OOB is not enabled on current monitor, we'll emulate the old
+     * behavior that we won't process current monitor any more until
+     * it is responded.  This helps make sure that as long as OOB is
+     * not enabled, the server will never drop any command.
+     */
+    if (!qmp_oob_enabled(mon)) {
+        monitor_suspend(mon);
+    }
+}
+
 static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
 {
     Monitor *mon = opaque;
@@ -4148,6 +4270,15 @@ static void monitor_iothread_init(void)
      * fetch the context we'll have that initialized.
      */
     monitor_io_context_get();
+
+    /*
+     * This MUST be on main loop thread since we have commands that
+     * have assumption to be run on main loop thread (Yeah, we'd
+     * better remove this assumption in the future).
+     */
+    mon_global.qmp_dispatcher_bh = aio_bh_new(qemu_get_aio_context(),
+                                              monitor_qmp_bh_dispatcher,
+                                              NULL);
 }
 
 void monitor_init_globals(void)
@@ -4268,6 +4399,10 @@ void monitor_cleanup(void)
     }
     qemu_mutex_unlock(&monitor_lock);
 
+    /* QEMUBHs needs to be deleted before destroying the IOThread. */
+    qemu_bh_delete(mon_global.qmp_dispatcher_bh);
+    mon_global.qmp_dispatcher_bh = NULL;
+
     iothread_destroy(mon_global.mon_iothread);
     mon_global.mon_iothread = NULL;
 }
-- 
2.13.5

[Qemu-devel] [RFC v3 18/27] qmp: add new event "request-dropped"

[Peter Xu]

This event will be emitted if one QMP request is dropped.  Along,
declare an enum for the reasons.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qapi-schema.json | 35 +++++++++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)

diff --git a/qapi-schema.json b/qapi-schema.json
index 531fd4c0db..650714da06 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -3222,3 +3222,38 @@
 # Since: 2.11
 ##
 { 'command': 'watchdog-set-action', 'data' : {'action': 'WatchdogAction'} }
+
+##
+# @RequestDropReason:
+#
+# Reasons that caused one request to be dropped.
+#
+# @queue-full: the queue of request is full.
+#
+# Since: 2.12
+##
+{ 'enum': 'RequestDropReason',
+  'data': ['queue-full' ] }
+
+##
+# @REQUEST_DROPPED:
+#
+# Emitted when one QMP request is dropped due to some reason.
+#
+# @id:    If the original request contains an string-typed "id" field,
+#         it'll be put into this field.  Otherwise it'll be an empty
+#         string.
+#
+# @reason: The reason why the request is dropped.
+#
+# Since: 2.12
+#
+# Example:
+#
+# { "event": "REQUEST_DROPPED",
+#   "data": {"result": {"id": "libvirt-102",
+#                       "reason": "queue-full" } } }
+#
+##
+{ 'event': 'REQUEST_DROPPED' ,
+  'data': { 'id': 'str', 'reason': 'RequestDropReason' } }
-- 
2.13.5

[Qemu-devel] [RFC v3 19/27] monitor: send event when request queue full

[Peter Xu]

Set maximum QMP request queue length to 8.  If queue full, instead of
queue the command, we directly return a "request-dropped" event, telling
client that specific command is dropped.

Note that this flow control mechanism is only valid if OOB is enabled.
If it's not, the effective queue length will always be 1, which strictly
follows original behavior of QMP command handling (which never drop
messages).

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/monitor.c b/monitor.c
index e327ae115b..f1f850f5c6 100644
--- a/monitor.c
+++ b/monitor.c
@@ -4029,6 +4029,8 @@ static void monitor_qmp_bh_dispatcher(void *data)
     }
 }
 
+#define  QMP_REQ_QUEUE_LEN_MAX  (8)
+
 static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
                                void *opaque)
 {
@@ -4061,6 +4063,21 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
     req_obj->id = id;
     req_obj->req = req;
 
+    if (qmp_oob_enabled(mon)) {
+        /* Drop the request if queue is full. */
+        if (mon->qmp.qmp_requests->length >= QMP_REQ_QUEUE_LEN_MAX) {
+            const char *id_str = qobject_get_try_str(id);
+
+            qapi_event_send_request_dropped(id_str ? id_str : "",
+                                            REQUEST_DROP_REASON_QUEUE_FULL,
+                                            NULL);
+            qobject_decref(id);
+            qobject_decref(req);
+            g_free(req_obj);
+            return;
+        }
+    }
+
     /*
      * Put the request to the end of queue so that requests will be
      * handled in time order.  Ownership for req_obj, req, id,
-- 
2.13.5

[Qemu-devel] [RFC v3 20/27] qapi: introduce new cmd option "allow-oob"

[Peter Xu]

Here "oob" stands for "Out-Of-Band".  When "allow-oob" is set, it means
the command allows out-of-band execution.

The "oob" idea is proposed by Markus Armbruster in following thread:

  https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg02057.html

This new "allow-oob" boolean will be exposed by "query-qmp-schema" as
well for command entries, so that QMP clients can know which command can
be used as out-of-band calls. For example the command "migrate"
originally looks like:

  {"name": "migrate", "ret-type": "17", "meta-type": "command",
   "arg-type": "86"}

And it'll be changed into:

  {"name": "migrate", "ret-type": "17", "allow-oob": false,
   "meta-type": "command", "arg-type": "86"}

This patch only provides the QMP interface level changes.  It does not
contains the real out-of-band execution implementation yet.

Suggested-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/dispatch.h    |  1 +
 qapi/introspect.json           |  6 +++++-
 scripts/qapi-commands.py       | 19 ++++++++++++++-----
 scripts/qapi-introspect.py     | 10 ++++++++--
 scripts/qapi.py                | 15 ++++++++++-----
 scripts/qapi2texi.py           |  2 +-
 tests/qapi-schema/test-qapi.py |  2 +-
 7 files changed, 40 insertions(+), 15 deletions(-)

diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
index 20578dcd48..b76798800c 100644
--- a/include/qapi/qmp/dispatch.h
+++ b/include/qapi/qmp/dispatch.h
@@ -23,6 +23,7 @@ typedef enum QmpCommandOptions
 {
     QCO_NO_OPTIONS = 0x0,
     QCO_NO_SUCCESS_RESP = 0x1,
+    QCO_ALLOW_OOB = 0x2,
 } QmpCommandOptions;
 
 typedef struct QmpCommand
diff --git a/qapi/introspect.json b/qapi/introspect.json
index 5b3e6e9d78..57cc137627 100644
--- a/qapi/introspect.json
+++ b/qapi/introspect.json
@@ -259,12 +259,16 @@
 #
 # @ret-type: the name of the command's result type.
 #
+# @allow-oob: whether the command allows out-of-band execution.
+#             (Since: 2.11)
+#
 # TODO: @success-response (currently irrelevant, because it's QGA, not QMP)
 #
 # Since: 2.5
 ##
 { 'struct': 'SchemaInfoCommand',
-  'data': { 'arg-type': 'str', 'ret-type': 'str' } }
+  'data': { 'arg-type': 'str', 'ret-type': 'str',
+            'allow-oob': 'bool' } }
 
 ##
 # @SchemaInfoEvent:
diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 974d0a4a80..b2b0bc0510 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -192,10 +192,18 @@ out:
     return ret
 
 
-def gen_register_command(name, success_response):
-    options = 'QCO_NO_OPTIONS'
+def gen_register_command(name, success_response, allow_oob):
+    options = []
+
     if not success_response:
-        options = 'QCO_NO_SUCCESS_RESP'
+        options += ['QCO_NO_SUCCESS_RESP']
+    if allow_oob:
+        options += ['QCO_ALLOW_OOB']
+
+    if not options:
+        options = ['QCO_NO_OPTIONS']
+
+    options = " | ".join(options)
 
     ret = mcgen('''
     qmp_register_command(cmds, "%(name)s",
@@ -241,7 +249,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._visited_ret_types = None
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         if not gen:
             return
         self.decl += gen_command_decl(name, arg_type, boxed, ret_type)
@@ -250,7 +258,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
             self.defn += gen_marshal_output(ret_type)
         self.decl += gen_marshal_decl(name)
         self.defn += gen_marshal(name, arg_type, boxed, ret_type)
-        self._regy += gen_register_command(name, success_response)
+        self._regy += gen_register_command(name, success_response,
+                                           allow_oob)
 
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index 032bcea491..9fbf88b644 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -28,6 +28,11 @@ def to_json(obj, level=0):
                               to_json(obj[key], level + 1))
                 for key in sorted(obj.keys())]
         ret = '{' + ', '.join(elts) + '}'
+    elif isinstance(obj, bool):
+        if obj:
+            ret = 'true'
+        else:
+            ret = 'false'
     else:
         assert False                # not implemented
     if level == 1:
@@ -154,12 +159,13 @@ const char %(c_name)s[] = %(c_string)s;
                                     for m in variants.variants]})
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         arg_type = arg_type or self._schema.the_empty_object_type
         ret_type = ret_type or self._schema.the_empty_object_type
         self._gen_json(name, 'command',
                        {'arg-type': self._use_type(arg_type),
-                        'ret-type': self._use_type(ret_type)})
+                        'ret-type': self._use_type(ret_type),
+                        'allow-oob': allow_oob})
 
     def visit_event(self, name, info, arg_type, boxed):
         arg_type = arg_type or self._schema.the_empty_object_type
diff --git a/scripts/qapi.py b/scripts/qapi.py
index 62dc52ed6e..f411b8fc91 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -920,7 +920,8 @@ def check_exprs(exprs):
         elif 'command' in expr:
             meta = 'command'
             check_keys(expr_elem, 'command', [],
-                       ['data', 'returns', 'gen', 'success-response', 'boxed'])
+                       ['data', 'returns', 'gen', 'success-response',
+                        'boxed', 'allow-oob'])
         elif 'event' in expr:
             meta = 'event'
             check_keys(expr_elem, 'event', [], ['data', 'boxed'])
@@ -1031,7 +1032,7 @@ class QAPISchemaVisitor(object):
         pass
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         pass
 
     def visit_event(self, name, info, arg_type, boxed):
@@ -1398,7 +1399,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
 class QAPISchemaCommand(QAPISchemaEntity):
     def __init__(self, name, info, doc, arg_type, ret_type,
-                 gen, success_response, boxed):
+                 gen, success_response, boxed, allow_oob):
         QAPISchemaEntity.__init__(self, name, info, doc)
         assert not arg_type or isinstance(arg_type, str)
         assert not ret_type or isinstance(ret_type, str)
@@ -1409,6 +1410,7 @@ class QAPISchemaCommand(QAPISchemaEntity):
         self.gen = gen
         self.success_response = success_response
         self.boxed = boxed
+        self.allow_oob = allow_oob
 
     def check(self, schema):
         if self._arg_type_name:
@@ -1432,7 +1434,8 @@ class QAPISchemaCommand(QAPISchemaEntity):
     def visit(self, visitor):
         visitor.visit_command(self.name, self.info,
                               self.arg_type, self.ret_type,
-                              self.gen, self.success_response, self.boxed)
+                              self.gen, self.success_response,
+                              self.boxed, self.allow_oob)
 
 
 class QAPISchemaEvent(QAPISchemaEntity):
@@ -1640,6 +1643,7 @@ class QAPISchema(object):
         gen = expr.get('gen', True)
         success_response = expr.get('success-response', True)
         boxed = expr.get('boxed', False)
+        allow_oob = expr.get('allow-oob', False)
         if isinstance(data, OrderedDict):
             data = self._make_implicit_object_type(
                 name, info, doc, 'arg', self._make_members(data, info))
@@ -1647,7 +1651,8 @@ class QAPISchema(object):
             assert len(rets) == 1
             rets = self._make_array_type(rets[0], info)
         self._def_entity(QAPISchemaCommand(name, info, doc, data, rets,
-                                           gen, success_response, boxed))
+                                           gen, success_response,
+                                           boxed, allow_oob))
 
     def _def_event(self, expr, info, doc):
         name = expr['event']
diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
index a317526e51..0ac0df517a 100755
--- a/scripts/qapi2texi.py
+++ b/scripts/qapi2texi.py
@@ -236,7 +236,7 @@ class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
                              body=texi_entity(doc, 'Members'))
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         doc = self.cur_doc
         if self.out:
             self.out += '\n'
diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index c7724d3437..6749e9e397 100644
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -36,7 +36,7 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
         self._print_variants(variants)
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         print 'command %s %s -> %s' % \
             (name, arg_type and arg_type.name, ret_type and ret_type.name)
         print '   gen=%s success_response=%s boxed=%s' % \
-- 
2.13.5

[Qemu-devel] [RFC v3 21/27] qmp: support out-of-band (oob) execution

[Peter Xu]

Having "allow-oob" to true for a command does not mean that this command
will always be run in out-of-band mode.  The out-of-band quick path will
only be executed if we specify the extra "run-oob" flag when sending the
QMP request:

    { "execute":   "command-that-allows-oob",
      "arguments": { ... },
      "control":   { "run-oob": true } }

The "control" key is introduced to store this extra flag.  "control"
field is used to store arguments that are shared by all the commands,
rather than command specific arguments.  Let "run-oob" be the first.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/dispatch.h |  1 +
 monitor.c                   | 16 ++++++++++++++++
 qapi/qmp-dispatch.c         | 39 +++++++++++++++++++++++++++++++++++++++
 trace-events                |  2 ++
 4 files changed, 58 insertions(+)

diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
index b76798800c..ee2b8ce576 100644
--- a/include/qapi/qmp/dispatch.h
+++ b/include/qapi/qmp/dispatch.h
@@ -49,6 +49,7 @@ bool qmp_command_is_enabled(const QmpCommand *cmd);
 const char *qmp_command_name(const QmpCommand *cmd);
 bool qmp_has_success_response(const QmpCommand *cmd);
 QObject *qmp_build_error_object(Error *err);
+bool qmp_is_oob(const QObject *request);
 
 typedef void (*qmp_cmd_callback_fn)(QmpCommand *cmd, void *opaque);
 
diff --git a/monitor.c b/monitor.c
index f1f850f5c6..45d293469a 100644
--- a/monitor.c
+++ b/monitor.c
@@ -4025,6 +4025,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
         if (!req_obj) {
             break;
         }
+        trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id));
         monitor_qmp_dispatch_one(req_obj);
     }
 }
@@ -4051,6 +4052,14 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
         return;
     }
 
+    if (!qmp_oob_enabled(mon) && qmp_is_oob(req)) {
+        error_setg(&err, "Out-Of-Band is only allowed "
+                   "when OOB is enabled.");
+        monitor_qmp_respond(mon, NULL, err, NULL);
+        qobject_decref(req);
+        return;
+    }
+
     qdict = qobject_to_qdict(req);
     if (qdict) {
         id = qdict_get(qdict, "id");
@@ -4064,6 +4073,13 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
     req_obj->req = req;
 
     if (qmp_oob_enabled(mon)) {
+        if (qmp_is_oob(req)) {
+            /* Out-Of-Band (OOB) requests are executed directly in parser. */
+            trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id));
+            monitor_qmp_dispatch_one(req_obj);
+            return;
+        }
+
         /* Drop the request if queue is full. */
         if (mon->qmp.qmp_requests->length >= QMP_REQ_QUEUE_LEN_MAX) {
             const char *id_str = qobject_get_try_str(id);
diff --git a/qapi/qmp-dispatch.c b/qapi/qmp-dispatch.c
index b41fa174fe..ed7e5d5a75 100644
--- a/qapi/qmp-dispatch.c
+++ b/qapi/qmp-dispatch.c
@@ -52,6 +52,12 @@ static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
                            "QMP input member 'arguments' must be an object");
                 return NULL;
             }
+        } else if (!strcmp(arg_name, "control")) {
+            if (qobject_type(arg_obj) != QTYPE_QDICT) {
+                error_setg(errp,
+                           "QMP input member 'control' must be a dict");
+                return NULL;
+            }
         } else {
             error_setg(errp, "QMP input member '%s' is unexpected",
                        arg_name);
@@ -94,6 +100,11 @@ static QObject *do_qmp_dispatch(QmpCommandList *cmds, QObject *request,
         return NULL;
     }
 
+    if (qmp_is_oob(request) && !(cmd->options & QCO_ALLOW_OOB)) {
+        error_setg(errp, "The command %s does not support OOB", command);
+        return NULL;
+    }
+
     if (!qdict_haskey(dict, "arguments")) {
         args = qdict_new();
     } else {
@@ -122,6 +133,34 @@ QObject *qmp_build_error_object(Error *err)
                               error_get_pretty(err));
 }
 
+/*
+ * Detect whether a request should be run out-of-band, by quickly
+ * peeking at whether we have: { "control": { "run-oob": True } }. By
+ * default commands are run in-band.
+ */
+bool qmp_is_oob(const QObject *request)
+{
+    QDict *dict;
+    QBool *bool_obj;
+
+    dict = qobject_to_qdict(request);
+    if (!dict) {
+        return false;
+    }
+
+    dict = qdict_get_qdict(dict, "control");
+    if (!dict) {
+        return false;
+    }
+
+    bool_obj = qobject_to_qbool(qdict_get(dict, "run-oob"));
+    if (!bool_obj) {
+        return false;
+    }
+
+    return qbool_get_bool(bool_obj);
+}
+
 QObject *qmp_dispatch(QmpCommandList *cmds, QObject *request)
 {
     Error *err = NULL;
diff --git a/trace-events b/trace-events
index 1d2eb5d3e4..3148e590c9 100644
--- a/trace-events
+++ b/trace-events
@@ -47,6 +47,8 @@ monitor_protocol_event_emit(uint32_t event, void *data) "event=%d data=%p"
 monitor_protocol_event_queue(uint32_t event, void *qdict, uint64_t rate) "event=%d data=%p rate=%" PRId64
 handle_hmp_command(void *mon, const char *cmdline) "mon %p cmdline: %s"
 handle_qmp_command(void *mon, const char *req) "mon %p req: %s"
+monitor_qmp_cmd_in_band(const char *id) "%s"
+monitor_qmp_cmd_out_of_band(const char *id) "%s"
 
 # dma-helpers.c
 dma_blk_io(void *dbs, void *bs, int64_t offset, bool to_dev) "dbs=%p bs=%p offset=%" PRId64 " to_dev=%d"
-- 
2.13.5

[Qemu-devel] [RFC v3 22/27] qmp: let migrate-incoming allow out-of-band

[Peter Xu]

So it can get rid of being run on main thread.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qapi/migration.json | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/qapi/migration.json b/qapi/migration.json
index bbc4671ded..95098072dd 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -1063,7 +1063,8 @@
 # <- { "return": {} }
 #
 ##
-{ 'command': 'migrate-incoming', 'data': {'uri': 'str' } }
+{ 'command': 'migrate-incoming', 'data': {'uri': 'str' },
+  'allow-oob': true }
 
 ##
 # @xen-save-devices-state:
-- 
2.13.5

[Qemu-devel] [RFC v3 23/27] qmp: isolate responses into io thread

[Peter Xu]

For those monitors who has enabled IO thread, we'll offload the
responding procedure into IO thread.  The main reason is that chardev is
not thread safe, and we need to do all the read/write IOs in the same
thread.  For use_io_thr=true monitors, that thread is the IO thread.

We do this isolation in similar pattern as what we have done to the
request queue: we first create one response queue for each monitor, then
instead of reply directly in main thread, we queue the responses and
kick the IO thread to do the rest of the job for us.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 89 insertions(+), 1 deletion(-)

diff --git a/monitor.c b/monitor.c
index 45d293469a..a2550d79a8 100644
--- a/monitor.c
+++ b/monitor.c
@@ -172,6 +172,8 @@ typedef struct {
     QemuMutex qmp_queue_lock;
     /* Input queue that holds all the parsed QMP requests */
     GQueue *qmp_requests;
+    /* Output queue contains all the QMP responses in order */
+    GQueue *qmp_responses;
 } MonitorQMP;
 
 /*
@@ -219,6 +221,8 @@ struct MonitorGlobal {
     IOThread *mon_iothread;
     /* Bottom half to dispatch the requests received from IO thread */
     QEMUBH *qmp_dispatcher_bh;
+    /* Bottom half to deliver the responses back to clients */
+    QEMUBH *qmp_respond_bh;
 };
 
 static struct MonitorGlobal mon_global;
@@ -395,7 +399,8 @@ int monitor_fprintf(FILE *stream, const char *fmt, ...)
     return 0;
 }
 
-static void monitor_json_emitter(Monitor *mon, const QObject *data)
+static void monitor_json_emitter_raw(Monitor *mon,
+                                     QObject *data)
 {
     QString *json;
 
@@ -409,6 +414,71 @@ static void monitor_json_emitter(Monitor *mon, const QObject *data)
     QDECREF(json);
 }
 
+static void monitor_json_emitter(Monitor *mon, QObject *data)
+{
+    if (mon->use_io_thr) {
+        /*
+         * If using IO thread, we need to queue the item so that IO
+         * thread will do the rest for us.  Take refcount so that
+         * caller won't free the data (which will be finally freed in
+         * responder thread).
+         */
+        qobject_incref(data);
+        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+        g_queue_push_tail(mon->qmp.qmp_responses, (void *)data);
+        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+        qemu_bh_schedule(mon_global.qmp_respond_bh);
+    } else {
+        /*
+         * If not using monitor IO thread, then we are in main thread.
+         * Do the emission right away.
+         */
+        monitor_json_emitter_raw(mon, data);
+    }
+}
+
+struct QMPResponse {
+    Monitor *mon;
+    QObject *data;
+};
+typedef struct QMPResponse QMPResponse;
+
+/*
+ * Return one QMPResponse.  The response is only valid if
+ * response.data is not NULL.
+ */
+static QMPResponse monitor_qmp_response_pop_one(void)
+{
+    Monitor *mon;
+    QObject *data = NULL;
+
+    qemu_mutex_lock(&monitor_lock);
+    QTAILQ_FOREACH(mon, &mon_list, entry) {
+        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+        data = g_queue_pop_head(mon->qmp.qmp_responses);
+        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+        if (data) {
+            break;
+        }
+    }
+    qemu_mutex_unlock(&monitor_lock);
+    return (QMPResponse) { .mon = mon, .data = data };
+}
+
+static void monitor_qmp_bh_responder(void *opaque)
+{
+    QMPResponse response;
+
+    while (true) {
+        response = monitor_qmp_response_pop_one();
+        if (!response.data) {
+            break;
+        }
+        monitor_json_emitter_raw(response.mon, response.data);
+        qobject_decref(response.data);
+    }
+}
+
 static MonitorQAPIEventConf monitor_qapi_event_conf[QAPI_EVENT__MAX] = {
     /* Limit guest-triggerable events to 1 per second */
     [QAPI_EVENT_RTC_CHANGE]        = { 1000 * SCALE_MS },
@@ -595,6 +665,7 @@ static void monitor_data_init(Monitor *mon, bool skip_flush,
     mon->skip_flush = skip_flush;
     mon->use_io_thr = use_io_thr;
     mon->qmp.qmp_requests = g_queue_new();
+    mon->qmp.qmp_responses = g_queue_new();
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -609,6 +680,7 @@ static void monitor_data_destroy(Monitor *mon)
     qemu_mutex_destroy(&mon->out_lock);
     qemu_mutex_destroy(&mon->qmp.qmp_queue_lock);
     g_queue_free(mon->qmp.qmp_requests);
+    g_queue_free(mon->qmp.qmp_responses);
 }
 
 char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
@@ -4294,6 +4366,11 @@ static GMainContext *monitor_io_context_get(void)
     return iothread_get_g_main_context(mon_global.mon_iothread);
 }
 
+static AioContext *monitor_aio_context_get(void)
+{
+    return iothread_get_aio_context(mon_global.mon_iothread);
+}
+
 static void monitor_iothread_init(void)
 {
     mon_global.mon_iothread = iothread_create("monitor_iothread",
@@ -4312,6 +4389,15 @@ static void monitor_iothread_init(void)
     mon_global.qmp_dispatcher_bh = aio_bh_new(qemu_get_aio_context(),
                                               monitor_qmp_bh_dispatcher,
                                               NULL);
+
+    /*
+     * Unlike the dispatcher BH, this must be run on the monitor IO
+     * thread, so that monitors that are using IO thread will make
+     * sure read/write operations are all done on the IO thread.
+     */
+    mon_global.qmp_respond_bh = aio_bh_new(monitor_aio_context_get(),
+                                           monitor_qmp_bh_responder,
+                                           NULL);
 }
 
 void monitor_init_globals(void)
@@ -4435,6 +4521,8 @@ void monitor_cleanup(void)
     /* QEMUBHs needs to be deleted before destroying the IOThread. */
     qemu_bh_delete(mon_global.qmp_dispatcher_bh);
     mon_global.qmp_dispatcher_bh = NULL;
+    qemu_bh_delete(mon_global.qmp_respond_bh);
+    mon_global.qmp_respond_bh = NULL;
 
     iothread_destroy(mon_global.mon_iothread);
     mon_global.mon_iothread = NULL;
-- 
2.13.5

[Qemu-devel] [RFC v3 24/27] monitor: enable IO thread for (qmp & !mux) typed

[Peter Xu]

Start to use dedicate IO thread for QMP monitors that are not using
MUXed chardev.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/monitor.c b/monitor.c
index a2550d79a8..be6b3a914f 100644
--- a/monitor.c
+++ b/monitor.c
@@ -36,6 +36,7 @@
 #include "net/net.h"
 #include "net/slirp.h"
 #include "chardev/char-fe.h"
+#include "chardev/char-mux.h"
 #include "ui/qemu-spice.h"
 #include "sysemu/numa.h"
 #include "monitor/monitor.h"
@@ -4454,7 +4455,7 @@ void monitor_init(Chardev *chr, int flags)
     Monitor *mon = g_malloc(sizeof(*mon));
     GMainContext *context;
 
-    monitor_data_init(mon, false, false);
+    monitor_data_init(mon, false, !CHARDEV_IS_MUX(chr));
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
-- 
2.13.5

[Qemu-devel] [RFC v3 25/27] docs: update QMP documents for OOB commands

[Peter Xu]

Update both the developer and spec for the new QMP OOB (Out-Of-Band)
command.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 docs/devel/qapi-code-gen.txt | 51 +++++++++++++++++++++++++++++++++++++++-----
 docs/interop/qmp-spec.txt    | 32 +++++++++++++++++++++++----
 2 files changed, 74 insertions(+), 9 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index f04c63fe82..8597fdb087 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -556,7 +556,8 @@ following example objects:
 
 Usage: { 'command': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT,
          '*returns': TYPE-NAME, '*boxed': true,
-         '*gen': false, '*success-response': false }
+         '*gen': false, '*success-response': false,
+         '*allow-oob': false }
 
 Commands are defined by using a dictionary containing several members,
 where three members are most common.  The 'command' member is a
@@ -636,6 +637,44 @@ possible, the command expression should include the optional key
 'success-response' with boolean value false.  So far, only QGA makes
 use of this member.
 
+Most of the QMP commands are handled sequentially in such a order:
+Firstly, the JSON Parser parses the command request into some internal
+message, delivers the message to QMP dispatchers. Secondly, the QMP
+dispatchers will handle the commands one by one in time order, respond
+when necessary.  For some commands that always complete "quickly" can
+instead be executed directly during parsing, at the QMP client's
+request.  This kind of commands that allow direct execution is called
+"out-of-band" ("oob" as shortcut) commands. The response can overtake
+prior in-band commands' responses.  By default, commands are always
+in-band.  We need to explicitly specify "allow-oob" to "True" to show
+that one command can be run out-of-band.
+
+One thing to mention for developers is that, although out-of-band
+execution of commands benefit from quick and asynchronous execution,
+it need to satisfy at least the following:
+
+(1) It is extremely quick and never blocks, so that its execution will
+    not block parsing routine of any other monitors.
+
+(2) It does not need BQL, since the parser can be run without BQL,
+    while the dispatcher is always with BQL held.
+
+If not, the command is not suitable to be allowed to run out-of-band,
+and it should set its "allow-oob" to "False".  Whether a command is
+allowed to run out-of-band can also be introspected using
+query-qmp-schema command.  Please see the section "Client JSON
+Protocol introspection" for more information.
+
+To execute a command in out-of-band way, we need to specify the
+"control" field in the request, with "run-oob" set to true. Example:
+
+ => { "execute": "command-support-oob",
+      "arguments": { ... },
+      "control": { "run-oob": true } }
+ <= { "return": { } }
+
+Without it, even the commands that supports out-of-band execution will
+still be run in-band.
 
 === Events ===
 
@@ -739,10 +778,12 @@ references by name.
 QAPI schema definitions not reachable that way are omitted.
 
 The SchemaInfo for a command has meta-type "command", and variant
-members "arg-type" and "ret-type".  On the wire, the "arguments"
-member of a client's "execute" command must conform to the object type
-named by "arg-type".  The "return" member that the server passes in a
-success response conforms to the type named by "ret-type".
+members "arg-type", "ret-type" and "allow-oob".  On the wire, the
+"arguments" member of a client's "execute" command must conform to the
+object type named by "arg-type".  The "return" member that the server
+passes in a success response conforms to the type named by
+"ret-type".  When "allow-oob" is set, it means the command supports
+out-of-band execution.
 
 If the command takes no arguments, "arg-type" names an object type
 without members.  Likewise, if the command returns nothing, "ret-type"
diff --git a/docs/interop/qmp-spec.txt b/docs/interop/qmp-spec.txt
index f8b5356015..b289e66924 100644
--- a/docs/interop/qmp-spec.txt
+++ b/docs/interop/qmp-spec.txt
@@ -78,21 +78,32 @@ The greeting message format is:
 - The "capabilities" member specify the availability of features beyond the
   baseline specification; the order of elements in this array has no
   particular significance, so a client must search the entire array
-  when looking for a particular capability
+  when looking for a particular capability.
 
 2.2.1 Capabilities
 ------------------
 
-As of the date this document was last revised, no server or client
-capability strings have been defined.
+Currently supported capabilities are:
 
+- "oob": it means the QMP server supports "Out-Of-Band" command
+  execution.  For more detail, please see "run-oob" parameter in
+  "Issuing Commands" section below.  Not all commands allow this "oob"
+  execution.  One can know whether one command supports "oob" by
+  "query-qmp-schema" command.
+
+QMP clients can get a list of supported QMP capabilities of the QMP
+server in the greeting message mentioned above.  By default, all the
+capabilities are off.  To enable a specific or multiple of QMP
+capabilities, QMP client needs to send "qmp_capabilities" command with
+extra parameter for the capabilities.
 
 2.3 Issuing Commands
 --------------------
 
 The format for command execution is:
 
-{ "execute": json-string, "arguments": json-object, "id": json-value }
+{ "execute": json-string, "arguments": json-object, "id": json-value,
+  "control": json-dict }
 
  Where,
 
@@ -106,6 +117,14 @@ The format for command execution is:
   provided. The "id" member can be any json-value, although most
   clients merely use a json-number incremented for each successive
   command
+- The "control" member is optionally, and currently only used for
+  "out-of-band" execution.  For some commands that always complete
+  "quickly" can be executed directly during parsing at the QMP
+  client's request.  This kind of commands that allow direct execution
+  is called "out-of-band" ("oob" as shortcut) commands. The response
+  of "oob" commands can overtake prior in-band commands' responses.
+  To enable "oob" feature, just provide a control field with:
+  { "control": { "run-oob": true } }
 
 2.4 Commands Responses
 ----------------------
@@ -113,6 +132,11 @@ The format for command execution is:
 There are two possible responses which the Server will issue as the result
 of a command execution: success or error.
 
+As long as the commands were issued with a proper "id" field, then the
+same "id" field will be attached in the corresponding response message
+so that requests and responses can match.  Clients should drop all the
+responses that are with unknown "id" field.
+
 2.4.1 success
 -------------
 
-- 
2.13.5

[Qemu-devel] [RFC v3 26/27] tests: qmp-test: verify command batching

[Peter Xu]

OOB introduced DROP event for flow control.  This should not affect old
QMP clients.  Add a command batching check to make sure of it.
---
 tests/qmp-test.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index 292c5f135a..729ec59b0a 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -78,6 +78,7 @@ static void test_qmp_protocol(void)
     QList *capabilities;
     const QListEntry *entry;
     QString *qstr;
+    int i;
 
     global_qtest = qtest_init_without_qmp_handshake(common_args);
 
@@ -135,6 +136,24 @@ static void test_qmp_protocol(void)
     g_assert_cmpint(qdict_get_int(resp, "id"), ==, 2);
     QDECREF(resp);
 
+    /*
+     * Test command batching.  In current test OOB is not enabled, we
+     * should be able to run as many commands in batch as we like.
+     * Using 16 (>8, which is OOB queue length) to make sure OOB
+     * won't break existing clients.
+     */
+    for (i = 0; i < 16; i++) {
+        qmp_async("{ 'execute': 'query-version' }");
+    }
+    /* Verify the replies to make sure no command is dropped. */
+    for (i = 0; i < 16; i++) {
+        resp = qmp_receive();
+        /* It should never be dropped.  Each of them should be a reply. */
+        g_assert(qdict_haskey(resp, "return"));
+        g_assert(!qdict_haskey(resp, "event"));
+        QDECREF(resp);
+    }
+
     qtest_end();
 }
 
-- 
2.13.5

[Qemu-devel] [RFC v3 27/27] tests: qmp-test: add oob test

[Peter Xu]

Test the new OOB capability.  Here we used the "dump-guest-memory"
command to hang the thread a bit to test working of OOB preemption.
Note that currently it is only running for x86/arm/ppc/s390x.

Note that for some platforms we may need to specify "-cpu" to make sure
the dump-guest-memory commands can really work.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 tests/qmp-test.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 93 insertions(+)

diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index 729ec59b0a..7c1b9f9b9e 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -157,6 +157,98 @@ static void test_qmp_protocol(void)
     qtest_end();
 }
 
+/* Tests for Out-Of-Band support. */
+static void test_qmp_oob(void)
+{
+    QDict *resp;
+    int acks = 0;
+    char *qtest_params;
+    const char *cmd_id, *extra_params;
+    const char *arch = qtest_get_arch();
+
+    /*
+     * Some archs need to specify cpu to make sure dump-guest-memory
+     * can work.  I chose CPU type randomly.
+     */
+    if (g_strcmp0(arch, "aarch64") == 0) {
+        extra_params = "-cpu cortex-a57";
+    } else if (g_strcmp0(arch, "ppc64") == 0) {
+        extra_params = "-cpu power8";
+    } else {
+        extra_params = "";
+    }
+
+    /*
+     * Let's have some memory to make sure dump-guest-memory will be
+     * time consuming.  That is required to test OOB functionaility.
+     */
+    qtest_params = g_strdup_printf("-nodefaults -machine none -m 1G %s",
+                                   extra_params);
+    global_qtest = qtest_init_without_qmp_handshake(qtest_params);
+    g_free(qtest_params);
+
+    /* Ignore the greeting message. */
+    resp = qmp_receive();
+    g_assert(qdict_get_qdict(resp, "QMP"));
+    QDECREF(resp);
+
+    /* Try a fake capability, it should fail. */
+    resp = qmp("{ 'execute': 'qmp_capabilities', "
+               "  'arguments': { 'enable': [ 'cap-does-not-exist' ] } }");
+    g_assert(qdict_haskey(resp, "error"));
+
+    /* Now, enable OOB in current QMP session, it should success. */
+    resp = qmp("{ 'execute': 'qmp_capabilities', "
+               "  'arguments': { 'enable': [ 'oob' ] } }");
+    g_assert(qdict_haskey(resp, "return"));
+
+    /*
+     * Try any command that does not support OOB but with OOB flag. We
+     * should get failure.
+     */
+    resp = qmp("{ 'execute': 'query-cpus',"
+               "  'control': { 'run-oob': true } }");
+    g_assert(qdict_haskey(resp, "error"));
+
+    /*
+     * Try a time-consuming command, following by a OOB command, make
+     * sure we get OOB command before the time-consuming one (which is
+     * run in the parser).
+     *
+     * When writting up this test script, the only command that
+     * support OOB is migrate-incoming.  It's not the best command to
+     * test OOB but we don't really have a choice here.  We will check
+     * arriving order but not command errors, which does not really
+     * matter to us.
+     */
+    qmp_async("{ 'execute': 'dump-guest-memory',"
+              "  'arguments': { 'paging': true, "
+              "                 'protocol': 'file:/dev/null' }, "
+              "  'id': 'time-consuming-cmd'}");
+    qmp_async("{ 'execute': 'migrate-incoming', "
+              "  'control': { 'run-oob': true }, "
+              "  'id': 'oob-cmd' }");
+
+    /* Ignore all events.  Wait for 2 acks */
+    while (acks < 2) {
+        resp = qmp_receive();
+        if (qdict_haskey(resp, "event")) {
+            /* Skip possible events */
+            continue;
+        }
+        cmd_id = qdict_get_str(resp, "id");
+        if (acks == 0) {
+            /* Need to receive OOB response first */
+            g_assert_cmpstr(cmd_id, ==, "oob-cmd");
+        } else if (acks == 1) {
+            g_assert_cmpstr(cmd_id, ==, "time-consuming-cmd");
+        }
+        acks++;
+    }
+
+    qtest_end();
+}
+
 static int query_error_class(const char *cmd)
 {
     static struct {
@@ -335,6 +427,7 @@ int main(int argc, char *argv[])
     g_test_init(&argc, &argv, NULL);
 
     qtest_add_func("qmp/protocol", test_qmp_protocol);
+    qtest_add_func("qmp/oob", test_qmp_oob);
     qmp_schema_init(&schema);
     add_query_tests(&schema);
 
-- 
2.13.5

Re: [Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support

[no-reply]

Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support
Type: series
Message-id: 20171106094643.14881-1-peterx@redhat.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
From https://github.com/patchew-project/qemu
 * [new tag]               patchew/20171106094643.14881-1-peterx@redhat.com -> patchew/20171106094643.14881-1-peterx@redhat.com
Switched to a new branch 'test'
5525c4e791 tests: qmp-test: add oob test
ccc9e4c399 tests: qmp-test: verify command batching
7f45b4c6c0 docs: update QMP documents for OOB commands
58cfe877d2 monitor: enable IO thread for (qmp & !mux) typed
5e1d56ce74 qmp: isolate responses into io thread
aef4275536 qmp: let migrate-incoming allow out-of-band
5e68beacf3 qmp: support out-of-band (oob) execution
43c7215a30 qapi: introduce new cmd option "allow-oob"
e11127ba4b monitor: send event when request queue full
45cef4f7e4 qmp: add new event "request-dropped"
485da28be1 monitor: separate QMP parser and dispatcher
4892fe9ca2 monitor: let monitor_{suspend|resume} thread safe
1b86166d9c monitor: introduce monitor_qmp_respond()
0f48093add qmp: introduce some capability helpers
8d3f33043d qmp: negociate QMP capabilities
023b386d0e qmp: introduce QMPCapability
2bde5ca8ce monitor: allow to use IO thread for parsing
f4cc112f80 monitor: create monitor dedicate iothread
3590fdc1d4 monitor: let mon_list be tail queue
11c818a9ac monitor: unify global init
36d3efb87d monitor: move the cur_mon hack deeper for QMP
bf3e493a86 qjson: add "opaque" field to JSONMessageParser
17367fe7a1 monitor: move skip_flush into monitor_data_init
0c98d4baa4 qobject: let object_property_get_str() use new API
aa4b973dd5 qobject: introduce qobject_get_try_str()
981ccebc1e qobject: introduce qstring_get_try_str()
d40ba38085 char-io: fix possible race on IOWatchPoll

=== OUTPUT BEGIN ===
Checking PATCH 1/27: char-io: fix possible race on IOWatchPoll...
Checking PATCH 2/27: qobject: introduce qstring_get_try_str()...
Checking PATCH 3/27: qobject: introduce qobject_get_try_str()...
Checking PATCH 4/27: qobject: let object_property_get_str() use new API...
Checking PATCH 5/27: monitor: move skip_flush into monitor_data_init...
Checking PATCH 6/27: qjson: add "opaque" field to JSONMessageParser...
Checking PATCH 7/27: monitor: move the cur_mon hack deeper for QMP...
Checking PATCH 8/27: monitor: unify global init...
Checking PATCH 9/27: monitor: let mon_list be tail queue...
Checking PATCH 10/27: monitor: create monitor dedicate iothread...
Checking PATCH 11/27: monitor: allow to use IO thread for parsing...
Checking PATCH 12/27: qmp: introduce QMPCapability...
Checking PATCH 13/27: qmp: negociate QMP capabilities...
Checking PATCH 14/27: qmp: introduce some capability helpers...
Checking PATCH 15/27: monitor: introduce monitor_qmp_respond()...
Checking PATCH 16/27: monitor: let monitor_{suspend|resume} thread safe...
ERROR: braces {} are necessary for all arms of this statement
#28: FILE: monitor.c:4014:
+    if (atomic_dec_fetch(&mon->suspend_cnt) == 0)
[...]

ERROR: Missing Signed-off-by: line(s)

total: 2 errors, 0 warnings, 16 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 17/27: monitor: separate QMP parser and dispatcher...
Checking PATCH 18/27: qmp: add new event "request-dropped"...
Checking PATCH 19/27: monitor: send event when request queue full...
Checking PATCH 20/27: qapi: introduce new cmd option "allow-oob"...
Checking PATCH 21/27: qmp: support out-of-band (oob) execution...
Checking PATCH 22/27: qmp: let migrate-incoming allow out-of-band...
Checking PATCH 23/27: qmp: isolate responses into io thread...
Checking PATCH 24/27: monitor: enable IO thread for (qmp & !mux) typed...
Checking PATCH 25/27: docs: update QMP documents for OOB commands...
Checking PATCH 26/27: tests: qmp-test: verify command batching...
Checking PATCH 27/27: tests: qmp-test: add oob test...
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@freelists.org

Re: [Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Mon, Nov 06, 2017 at 02:12:17AM -0800, no-reply@patchew.org wrote:
> Hi,
> 
> This series seems to have some coding style problems. See output below for
> more information:
> 
> Subject: [Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support
> Type: series
> Message-id: 20171106094643.14881-1-peterx@redhat.com
> 
> === TEST SCRIPT BEGIN ===
> #!/bin/bash
> 
> BASE=base
> n=1
> total=$(git log --oneline $BASE.. | wc -l)
> failed=0
> 
> git config --local diff.renamelimit 0
> git config --local diff.renames True
> 
> commits="$(git log --format=%H --reverse $BASE..)"
> for c in $commits; do
>     echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
>     if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
>         failed=1
>         echo
>     fi
>     n=$((n+1))
> done
> 
> exit $failed
> === TEST SCRIPT END ===
> 
> Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
> From https://github.com/patchew-project/qemu
>  * [new tag]               patchew/20171106094643.14881-1-peterx@redhat.com -> patchew/20171106094643.14881-1-peterx@redhat.com
> Switched to a new branch 'test'
> 5525c4e791 tests: qmp-test: add oob test
> ccc9e4c399 tests: qmp-test: verify command batching
> 7f45b4c6c0 docs: update QMP documents for OOB commands
> 58cfe877d2 monitor: enable IO thread for (qmp & !mux) typed
> 5e1d56ce74 qmp: isolate responses into io thread
> aef4275536 qmp: let migrate-incoming allow out-of-band
> 5e68beacf3 qmp: support out-of-band (oob) execution
> 43c7215a30 qapi: introduce new cmd option "allow-oob"
> e11127ba4b monitor: send event when request queue full
> 45cef4f7e4 qmp: add new event "request-dropped"
> 485da28be1 monitor: separate QMP parser and dispatcher
> 4892fe9ca2 monitor: let monitor_{suspend|resume} thread safe
> 1b86166d9c monitor: introduce monitor_qmp_respond()
> 0f48093add qmp: introduce some capability helpers
> 8d3f33043d qmp: negociate QMP capabilities
> 023b386d0e qmp: introduce QMPCapability
> 2bde5ca8ce monitor: allow to use IO thread for parsing
> f4cc112f80 monitor: create monitor dedicate iothread
> 3590fdc1d4 monitor: let mon_list be tail queue
> 11c818a9ac monitor: unify global init
> 36d3efb87d monitor: move the cur_mon hack deeper for QMP
> bf3e493a86 qjson: add "opaque" field to JSONMessageParser
> 17367fe7a1 monitor: move skip_flush into monitor_data_init
> 0c98d4baa4 qobject: let object_property_get_str() use new API
> aa4b973dd5 qobject: introduce qobject_get_try_str()
> 981ccebc1e qobject: introduce qstring_get_try_str()
> d40ba38085 char-io: fix possible race on IOWatchPoll
> 
> === OUTPUT BEGIN ===
> Checking PATCH 1/27: char-io: fix possible race on IOWatchPoll...
> Checking PATCH 2/27: qobject: introduce qstring_get_try_str()...
> Checking PATCH 3/27: qobject: introduce qobject_get_try_str()...
> Checking PATCH 4/27: qobject: let object_property_get_str() use new API...
> Checking PATCH 5/27: monitor: move skip_flush into monitor_data_init...
> Checking PATCH 6/27: qjson: add "opaque" field to JSONMessageParser...
> Checking PATCH 7/27: monitor: move the cur_mon hack deeper for QMP...
> Checking PATCH 8/27: monitor: unify global init...
> Checking PATCH 9/27: monitor: let mon_list be tail queue...
> Checking PATCH 10/27: monitor: create monitor dedicate iothread...
> Checking PATCH 11/27: monitor: allow to use IO thread for parsing...
> Checking PATCH 12/27: qmp: introduce QMPCapability...
> Checking PATCH 13/27: qmp: negociate QMP capabilities...
> Checking PATCH 14/27: qmp: introduce some capability helpers...
> Checking PATCH 15/27: monitor: introduce monitor_qmp_respond()...
> Checking PATCH 16/27: monitor: let monitor_{suspend|resume} thread safe...
> ERROR: braces {} are necessary for all arms of this statement
> #28: FILE: monitor.c:4014:
> +    if (atomic_dec_fetch(&mon->suspend_cnt) == 0)
> [...]
> 
> ERROR: Missing Signed-off-by: line(s)

Will add it in next post.  I still haven't found a good way to let
magit add this line for me every time automatically.

Please review the rest of things, thanks.

-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 01/27] char-io: fix possible race on IOWatchPoll

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> This is not a problem if we are only having one single loop thread like
> before.  However, after per-monitor thread is introduced, this is not
> true any more, and the race can happen.
> 
> The race can be triggered with "make check -j8" sometimes:
> 
>   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
>   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> 
> This patch keeps the reference for the watch object when creating in
> io_add_watch_poll(), so that the object will never be released in the
> context main loop, especially when the context loop is running in
> another standalone thread.  Meanwhile, when we want to remove the watch
> object, we always first detach the watch object from its owner context,
> then we continue with the cleanup.
> 
> Without this patch, calling io_remove_watch_poll() in main loop thread
> is not thread-safe, since the other per-monitor thread may be modifying
> the watch object at the same time.

Looks sane,

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [RFC v3 02/27] qobject: introduce qstring_get_try_str()

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> The only difference from qstring_get_str() is that it allows the qstring
> to be NULL.  If so, NULL is returned.
> 
> CC: Eric Blake <eblake@redhat.com>
> CC: Markus Armbruster <armbru@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [RFC v3 03/27] qobject: introduce qobject_get_try_str()

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> A quick way to fetch string from qobject when it's a QString.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [RFC v3 04/27] qobject: let object_property_get_str() use new API

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> We can simplify object_property_get_str() using the new
> qobject_get_try_str().
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [RFC v3 05/27] monitor: move skip_flush into monitor_data_init

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> It's part of the data init.  Collect it.
> 
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [RFC v3 06/27] qjson: add "opaque" field to JSONMessageParser

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> It'll be passed to emit() as well when it happens.  Since at it, add a
> typedef for the emitter function.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [RFC v3 07/27] monitor: move the cur_mon hack deeper for QMP

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> In monitor_qmp_read(), we have the hack to temporarily replace the
> cur_mon pointer.  Now we move this hack deeper inside the QMP dispatcher
> routine since the Monitor pointer can be passed in to that using the new
> JSON Parser opaque field now.
> 
> This does not make much sense as a single patch.  However, this will be
> a big step for the next patch, when the QMP dispatcher routine will be
> split from the QMP parser.
> 
> Reviewed-by: Eric Blake <eblake@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [RFC v3 08/27] monitor: unify global init

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> There are many places for monitor init its globals, at least:
> 
> - monitor_init_qmp_commands() at the very beginning
> - single function to init monitor_lock
> - in the first entry of monitor_init() using "is_first_init"
> 
> Unify them a bit.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [RFC v3 09/27] monitor: let mon_list be tail queue

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> It was QLIST.  I want to use this list to do monitor priority job later,
> which need tail insertion ability.  So switching to a tail queue.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [RFC v3 10/27] monitor: create monitor dedicate iothread

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> +static GMainContext *monitor_io_context_get(void)
> +{
> +    return iothread_get_g_main_context(mon_global.mon_iothread);
> +}
> +
> +static void monitor_iothread_init(void)
> +{
> +    mon_global.mon_iothread = iothread_create("monitor_iothread",
> +                                              &error_abort);
> +    /*
> +     * GContext in iothread is using lazy init - the first time we
> +     * fetch the context we'll have that initialized.
> +     */
> +    monitor_io_context_get();

Why do you need an eager init here?

Fam

Re: [Qemu-devel] [RFC v3 11/27] monitor: allow to use IO thread for parsing

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> For each Monitor, add one field "use_io_thr" to show whether it will be
> using the dedicated monitor IO thread to handle input/output.  When set,
> monitor IO parsing work will be offloaded to dedicated monitor IO
> thread, rather than the original main loop thread.
> 
> This only works for QMP.  HMP will always be run on main loop thread.
> 
> Currently we're still keeping use_io_thr to off always.  Will turn it on
> later at some point.
> 
> One thing to mention is that we cannot set use_io_thr for every QMP
> monitors.  The problem is that MUXed typed chardevs may not work well
> with it now. When MUX is used, frontend of chardev can be the monitor
> plus something else.  The only thing we know would be safe to be run
> outside main thread so far is the monitor frontend. All the rest of the
> frontends should still be run in main thread only.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [RFC v3 15/27] monitor: introduce monitor_qmp_respond()

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> A tiny refactoring, preparing to split the QMP dispatcher away.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 48 +++++++++++++++++++++++++++++++-----------------
>  1 file changed, 31 insertions(+), 17 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index 442d711d5d..1e87de87f8 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -3877,6 +3877,36 @@ static int monitor_can_read(void *opaque)
>      return (mon->suspend_cnt == 0) ? 1 : 0;
>  }
>  
> +/*
> + * When rsp/err/id is passed in, the function will be responsible for
> + * the cleanup.
> + */
> +static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
> +                                Error *err, QObject *id)
> +{
> +    QDict *qdict = NULL;
> +
> +    if (err) {
> +        qdict = qdict_new();
> +        qdict_put_obj(qdict, "error", qmp_build_error_object(err));
> +        error_free(err);
> +        rsp = QOBJECT(qdict);
> +    }
> +
> +    if (rsp) {
> +        if (id) {
> +            /* This is for the qdict below. */
> +            qobject_incref(id);

A small change from the old code (that does no incref, and sets id to NULL
afterwards to the final qobject_decref(id) is no-op), but looks correct.

Reviewed-by: Fam Zheng <famz@redhat.com>

> +            qdict_put_obj(qobject_to_qdict(rsp), "id", id);
> +        }
> +
> +        monitor_json_emitter(mon, rsp);
> +    }
> +
> +    qobject_decref(id);
> +    qobject_decref(rsp);
> +}
> +
>  static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
>                                 void *opaque)
>  {
> @@ -3927,24 +3957,8 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
>      }
>  
>  err_out:
> -    if (err) {
> -        qdict = qdict_new();
> -        qdict_put_obj(qdict, "error", qmp_build_error_object(err));
> -        error_free(err);
> -        rsp = QOBJECT(qdict);
> -    }
> +    monitor_qmp_respond(mon, rsp, err, id);
>  
> -    if (rsp) {
> -        if (id) {
> -            qdict_put_obj(qobject_to_qdict(rsp), "id", id);
> -            id = NULL;
> -        }
> -
> -        monitor_json_emitter(mon, rsp);
> -    }
> -
> -    qobject_decref(id);
> -    qobject_decref(rsp);
>      qobject_decref(req);
>  }
>  
> -- 
> 2.13.5
> 

Re: [Qemu-devel] [RFC v3 16/27] monitor: let monitor_{suspend|resume} thread safe

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> Monitor code now can be run in more than one thread.  Let the suspend
> and resume code for thread safety.

s/for thread safety/be thread safe/

Apart from that,

Reviewed-by: Fam Zheng <famz@redhat.com>

Re: [Qemu-devel] [RFC v3 23/27] qmp: isolate responses into io thread

[Fam Zheng]

On Mon, 11/06 17:46, Peter Xu wrote:
> @@ -4294,6 +4366,11 @@ static GMainContext *monitor_io_context_get(void)
>      return iothread_get_g_main_context(mon_global.mon_iothread);
>  }
>  
> +static AioContext *monitor_aio_context_get(void)
> +{
> +    return iothread_get_aio_context(mon_global.mon_iothread);
> +}
> +

This hunk fits better in patch 10, I think?

>  static void monitor_iothread_init(void)
>  {
>      mon_global.mon_iothread = iothread_create("monitor_iothread",

Fam

Re: [Qemu-devel] [RFC v3 10/27] monitor: create monitor dedicate iothread

[Peter Xu]

On Tue, Nov 07, 2017 at 03:11:31PM +0800, Fam Zheng wrote:
> On Mon, 11/06 17:46, Peter Xu wrote:
> > +static GMainContext *monitor_io_context_get(void)
> > +{
> > +    return iothread_get_g_main_context(mon_global.mon_iothread);
> > +}
> > +
> > +static void monitor_iothread_init(void)
> > +{
> > +    mon_global.mon_iothread = iothread_create("monitor_iothread",
> > +                                              &error_abort);
> > +    /*
> > +     * GContext in iothread is using lazy init - the first time we
> > +     * fetch the context we'll have that initialized.
> > +     */
> > +    monitor_io_context_get();
> 
> Why do you need an eager init here?

I thought putting this line with IOThread creation will make it more
clear that the IOThread is using GMainContext polling.  I am not that
strong to keep it, but I would prefer.

-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 23/27] qmp: isolate responses into io thread

[Peter Xu]

On Tue, Nov 07, 2017 at 03:57:08PM +0800, Fam Zheng wrote:
> On Mon, 11/06 17:46, Peter Xu wrote:
> > @@ -4294,6 +4366,11 @@ static GMainContext *monitor_io_context_get(void)
> >      return iothread_get_g_main_context(mon_global.mon_iothread);
> >  }
> >  
> > +static AioContext *monitor_aio_context_get(void)
> > +{
> > +    return iothread_get_aio_context(mon_global.mon_iothread);
> > +}
> > +
> 
> This hunk fits better in patch 10, I think?

This function is only used in current patch, so I think it's fine to
put it here.  But sure I can move it into patch 10 as well.

Thanks,

> 
> >  static void monitor_iothread_init(void)
> >  {
> >      mon_global.mon_iothread = iothread_create("monitor_iothread",
> 
> Fam

-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 23/27] qmp: isolate responses into io thread

[Fam Zheng]

On Wed, 11/08 15:31, Peter Xu wrote:
> On Tue, Nov 07, 2017 at 03:57:08PM +0800, Fam Zheng wrote:
> > On Mon, 11/06 17:46, Peter Xu wrote:
> > > @@ -4294,6 +4366,11 @@ static GMainContext *monitor_io_context_get(void)
> > >      return iothread_get_g_main_context(mon_global.mon_iothread);
> > >  }
> > >  
> > > +static AioContext *monitor_aio_context_get(void)
> > > +{
> > > +    return iothread_get_aio_context(mon_global.mon_iothread);
> > > +}
> > > +
> > 
> > This hunk fits better in patch 10, I think?
> 
> This function is only used in current patch, so I think it's fine to
> put it here.  But sure I can move it into patch 10 as well.

OK, so it's static and you want to keep the compiler happy. That's fine then.

Fam

Re: [Qemu-devel] [RFC v3 10/27] monitor: create monitor dedicate iothread

[Fam Zheng]

On Wed, 11/08 15:25, Peter Xu wrote:
> On Tue, Nov 07, 2017 at 03:11:31PM +0800, Fam Zheng wrote:
> > On Mon, 11/06 17:46, Peter Xu wrote:
> > > +static GMainContext *monitor_io_context_get(void)
> > > +{
> > > +    return iothread_get_g_main_context(mon_global.mon_iothread);
> > > +}
> > > +
> > > +static void monitor_iothread_init(void)
> > > +{
> > > +    mon_global.mon_iothread = iothread_create("monitor_iothread",
> > > +                                              &error_abort);
> > > +    /*
> > > +     * GContext in iothread is using lazy init - the first time we
> > > +     * fetch the context we'll have that initialized.
> > > +     */
> > > +    monitor_io_context_get();
> > 
> > Why do you need an eager init here?
> 
> I thought putting this line with IOThread creation will make it more
> clear that the IOThread is using GMainContext polling.  I am not that
> strong to keep it, but I would prefer.

The "iothread only starts using GMainContext polling after the lazy init
happens" is completely opaque to its user, so I don't think there is anything
but confusion to have this.

Fam

Re: [Qemu-devel] [RFC v3 10/27] monitor: create monitor dedicate iothread

[Peter Xu]

On Wed, Nov 08, 2017 at 07:18:44PM +0800, Fam Zheng wrote:
> On Wed, 11/08 15:25, Peter Xu wrote:
> > On Tue, Nov 07, 2017 at 03:11:31PM +0800, Fam Zheng wrote:
> > > On Mon, 11/06 17:46, Peter Xu wrote:
> > > > +static GMainContext *monitor_io_context_get(void)
> > > > +{
> > > > +    return iothread_get_g_main_context(mon_global.mon_iothread);
> > > > +}
> > > > +
> > > > +static void monitor_iothread_init(void)
> > > > +{
> > > > +    mon_global.mon_iothread = iothread_create("monitor_iothread",
> > > > +                                              &error_abort);
> > > > +    /*
> > > > +     * GContext in iothread is using lazy init - the first time we
> > > > +     * fetch the context we'll have that initialized.
> > > > +     */
> > > > +    monitor_io_context_get();
> > > 
> > > Why do you need an eager init here?
> > 
> > I thought putting this line with IOThread creation will make it more
> > clear that the IOThread is using GMainContext polling.  I am not that
> > strong to keep it, but I would prefer.
> 
> The "iothread only starts using GMainContext polling after the lazy init
> happens" is completely opaque to its user, so I don't think there is anything
> but confusion to have this.

Makes sense.  Let me remove it.

-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 01/27] char-io: fix possible race on IOWatchPoll

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2934 bytes --]

On Mon, Nov 06, 2017 at 05:46:17PM +0800, Peter Xu wrote:
> This is not a problem if we are only having one single loop thread like
> before.  However, after per-monitor thread is introduced, this is not
> true any more, and the race can happen.
> 
> The race can be triggered with "make check -j8" sometimes:

Please mention a specific test case that fails.

> 
>   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
>   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> 
> This patch keeps the reference for the watch object when creating in
> io_add_watch_poll(), so that the object will never be released in the
> context main loop, especially when the context loop is running in
> another standalone thread.  Meanwhile, when we want to remove the watch
> object, we always first detach the watch object from its owner context,
> then we continue with the cleanup.
> 
> Without this patch, calling io_remove_watch_poll() in main loop thread
> is not thread-safe, since the other per-monitor thread may be modifying
> the watch object at the same time.
> 
> Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  chardev/char-io.c | 16 ++++++++++++++--
>  1 file changed, 14 insertions(+), 2 deletions(-)
> 
> diff --git a/chardev/char-io.c b/chardev/char-io.c
> index f81052481a..50b5bac704 100644
> --- a/chardev/char-io.c
> +++ b/chardev/char-io.c
> @@ -122,7 +122,6 @@ GSource *io_add_watch_poll(Chardev *chr,
>      g_free(name);
>  
>      g_source_attach(&iwp->parent, context);
> -    g_source_unref(&iwp->parent);
>      return (GSource *)iwp;
>  }
>  
> @@ -131,12 +130,25 @@ static void io_remove_watch_poll(GSource *source)
>      IOWatchPoll *iwp;
>  
>      iwp = io_watch_poll_from_source(source);
> +
> +    /*
> +     * Here the order of destruction really matters.  We need to first
> +     * detach the IOWatchPoll object from the context (which may still
> +     * be running in another loop thread), only after that could we
> +     * continue to operate on iwp->src, or there may be race condition
> +     * between current thread and the context loop thread.
> +     *
> +     * Let's blame the glib bug mentioned in commit 2b316774f6
> +     * ("qemu-char: do not operate on sources from finalize
> +     * callbacks") for this extra complexity.

I don't understand how this bug is to blame.  Isn't the problem here a
race condition between two QEMU threads?

Why are two threads accessing the watch at the same time?

> +     */
> +    g_source_destroy(&iwp->parent);
>      if (iwp->src) {
>          g_source_destroy(iwp->src);
>          g_source_unref(iwp->src);
>          iwp->src = NULL;
>      }
> -    g_source_destroy(&iwp->parent);
> +    g_source_unref(&iwp->parent);
>  }
>  
>  void remove_fd_in_watch(Chardev *chr)
> -- 
> 2.13.5
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v3 01/27] char-io: fix possible race on IOWatchPoll

[Peter Xu]

On Mon, Nov 13, 2017 at 04:52:11PM +0000, Stefan Hajnoczi wrote:
> On Mon, Nov 06, 2017 at 05:46:17PM +0800, Peter Xu wrote:
> > This is not a problem if we are only having one single loop thread like
> > before.  However, after per-monitor thread is introduced, this is not
> > true any more, and the race can happen.
> > 
> > The race can be triggered with "make check -j8" sometimes:
> 
> Please mention a specific test case that fails.

It was any of the check-qtest-$(TARGET)s that failed.  I'll mention
that in next post.

> 
> > 
> >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > 
> > This patch keeps the reference for the watch object when creating in
> > io_add_watch_poll(), so that the object will never be released in the
> > context main loop, especially when the context loop is running in
> > another standalone thread.  Meanwhile, when we want to remove the watch
> > object, we always first detach the watch object from its owner context,
> > then we continue with the cleanup.
> > 
> > Without this patch, calling io_remove_watch_poll() in main loop thread
> > is not thread-safe, since the other per-monitor thread may be modifying
> > the watch object at the same time.
> > 
> > Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  chardev/char-io.c | 16 ++++++++++++++--
> >  1 file changed, 14 insertions(+), 2 deletions(-)
> > 
> > diff --git a/chardev/char-io.c b/chardev/char-io.c
> > index f81052481a..50b5bac704 100644
> > --- a/chardev/char-io.c
> > +++ b/chardev/char-io.c
> > @@ -122,7 +122,6 @@ GSource *io_add_watch_poll(Chardev *chr,
> >      g_free(name);
> >  
> >      g_source_attach(&iwp->parent, context);
> > -    g_source_unref(&iwp->parent);
> >      return (GSource *)iwp;
> >  }
> >  
> > @@ -131,12 +130,25 @@ static void io_remove_watch_poll(GSource *source)
> >      IOWatchPoll *iwp;
> >  
> >      iwp = io_watch_poll_from_source(source);
> > +
> > +    /*
> > +     * Here the order of destruction really matters.  We need to first
> > +     * detach the IOWatchPoll object from the context (which may still
> > +     * be running in another loop thread), only after that could we
> > +     * continue to operate on iwp->src, or there may be race condition
> > +     * between current thread and the context loop thread.
> > +     *
> > +     * Let's blame the glib bug mentioned in commit 2b316774f6
> > +     * ("qemu-char: do not operate on sources from finalize
> > +     * callbacks") for this extra complexity.
> 
> I don't understand how this bug is to blame.  Isn't the problem here a
> race condition between two QEMU threads?

Yes, it is.

The problem is, we won't have the race condition if glib does not have
that bug mentioned.  Then the thread running GMainContext will have
full control of iwp->src destruction, and destruction of it would be
fairly straightforward (unref iwp->src in IOWatchPoll destructor).
Now IIUC we are doing this in a hacky way, say, we destroy iwp->src
explicitly from main thread before quitting (see [1] below, the whole
if clause).

> 
> Why are two threads accessing the watch at the same time?

Here is how I understand:

Firstly we need to tackle with that bug, by an explicit destruction of
iwp->src below; meanwhile when we are destroying it, the GMainContext
can still be running somewhere (it's not happening in current series
since I stopped iothread earlier than this point, however it can still
happen if in the future we don't do that), then we possibly want this
patch.

Again, without this patch, current series should work; however I do
hope this patch can be in, in case someday we want to provide complete
thread safety for Chardevs (now it is not really thread-safe).

> 
> > +     */
> > +    g_source_destroy(&iwp->parent);
> >      if (iwp->src) {
> >          g_source_destroy(iwp->src);
> >          g_source_unref(iwp->src);
> >          iwp->src = NULL;
> >      }

[1]

> > -    g_source_destroy(&iwp->parent);
> > +    g_source_unref(&iwp->parent);
> >  }
> >  
> >  void remove_fd_in_watch(Chardev *chr)
> > -- 
> > 2.13.5
> > 

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 01/27] char-io: fix possible race on IOWatchPoll

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5175 bytes --]

On Tue, Nov 14, 2017 at 02:09:39PM +0800, Peter Xu wrote:
> On Mon, Nov 13, 2017 at 04:52:11PM +0000, Stefan Hajnoczi wrote:
> > On Mon, Nov 06, 2017 at 05:46:17PM +0800, Peter Xu wrote:
> > > This is not a problem if we are only having one single loop thread like
> > > before.  However, after per-monitor thread is introduced, this is not
> > > true any more, and the race can happen.
> > > 
> > > The race can be triggered with "make check -j8" sometimes:
> > 
> > Please mention a specific test case that fails.
> 
> It was any of the check-qtest-$(TARGET)s that failed.  I'll mention
> that in next post.
> 
> > 
> > > 
> > >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> > >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > > 
> > > This patch keeps the reference for the watch object when creating in
> > > io_add_watch_poll(), so that the object will never be released in the
> > > context main loop, especially when the context loop is running in
> > > another standalone thread.  Meanwhile, when we want to remove the watch
> > > object, we always first detach the watch object from its owner context,
> > > then we continue with the cleanup.
> > > 
> > > Without this patch, calling io_remove_watch_poll() in main loop thread
> > > is not thread-safe, since the other per-monitor thread may be modifying
> > > the watch object at the same time.
> > > 
> > > Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > > Signed-off-by: Peter Xu <peterx@redhat.com>
> > > ---
> > >  chardev/char-io.c | 16 ++++++++++++++--
> > >  1 file changed, 14 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/chardev/char-io.c b/chardev/char-io.c
> > > index f81052481a..50b5bac704 100644
> > > --- a/chardev/char-io.c
> > > +++ b/chardev/char-io.c
> > > @@ -122,7 +122,6 @@ GSource *io_add_watch_poll(Chardev *chr,
> > >      g_free(name);
> > >  
> > >      g_source_attach(&iwp->parent, context);
> > > -    g_source_unref(&iwp->parent);
> > >      return (GSource *)iwp;
> > >  }
> > >  
> > > @@ -131,12 +130,25 @@ static void io_remove_watch_poll(GSource *source)
> > >      IOWatchPoll *iwp;
> > >  
> > >      iwp = io_watch_poll_from_source(source);
> > > +
> > > +    /*
> > > +     * Here the order of destruction really matters.  We need to first
> > > +     * detach the IOWatchPoll object from the context (which may still
> > > +     * be running in another loop thread), only after that could we
> > > +     * continue to operate on iwp->src, or there may be race condition
> > > +     * between current thread and the context loop thread.
> > > +     *
> > > +     * Let's blame the glib bug mentioned in commit 2b316774f6
> > > +     * ("qemu-char: do not operate on sources from finalize
> > > +     * callbacks") for this extra complexity.
> > 
> > I don't understand how this bug is to blame.  Isn't the problem here a
> > race condition between two QEMU threads?
> 
> Yes, it is.
> 
> The problem is, we won't have the race condition if glib does not have
> that bug mentioned.  Then the thread running GMainContext will have
> full control of iwp->src destruction, and destruction of it would be
> fairly straightforward (unref iwp->src in IOWatchPoll destructor).
> Now IIUC we are doing this in a hacky way, say, we destroy iwp->src
> explicitly from main thread before quitting (see [1] below, the whole
> if clause).
> 
> > 
> > Why are two threads accessing the watch at the same time?
> 
> Here is how I understand:
> 
> Firstly we need to tackle with that bug, by an explicit destruction of
> iwp->src below; meanwhile when we are destroying it, the GMainContext
> can still be running somewhere (it's not happening in current series
> since I stopped iothread earlier than this point, however it can still
> happen if in the future we don't do that), then we possibly want this
> patch.
> 
> Again, without this patch, current series should work; however I do
> hope this patch can be in, in case someday we want to provide complete
> thread safety for Chardevs (now it is not really thread-safe).

You said qtests fail with "Assertion `iwp->src == NULL' failed" but then
you said "without this patch, current series should work".  How do you
reproduce the failure if it doesn't occur?

It looks like remove_fd_in_watch() -> io_remove_watch_poll() callers
fall into two categories: called from within the event loop and called
when a chardev is destroyed.  Do the thread-safety issues occur when the
chardev is destroyed by the QEMU main loop thread?  Or did I miss cases
where remove_fd_in_watch() is called from other threads?

> 
> > 
> > > +     */
> > > +    g_source_destroy(&iwp->parent);
> > >      if (iwp->src) {
> > >          g_source_destroy(iwp->src);
> > >          g_source_unref(iwp->src);
> > >          iwp->src = NULL;
> > >      }
> 
> [1]
> 
> > > -    g_source_destroy(&iwp->parent);
> > > +    g_source_unref(&iwp->parent);
> > >  }
> > >  
> > >  void remove_fd_in_watch(Chardev *chr)
> > > -- 
> > > 2.13.5
> > > 
> 
> Thanks,
> 
> -- 
> Peter Xu

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v3 01/27] char-io: fix possible race on IOWatchPoll

[Peter Xu]

On Tue, Nov 14, 2017 at 10:32:19AM +0000, Stefan Hajnoczi wrote:
> On Tue, Nov 14, 2017 at 02:09:39PM +0800, Peter Xu wrote:
> > On Mon, Nov 13, 2017 at 04:52:11PM +0000, Stefan Hajnoczi wrote:
> > > On Mon, Nov 06, 2017 at 05:46:17PM +0800, Peter Xu wrote:
> > > > This is not a problem if we are only having one single loop thread like
> > > > before.  However, after per-monitor thread is introduced, this is not
> > > > true any more, and the race can happen.
> > > > 
> > > > The race can be triggered with "make check -j8" sometimes:
> > > 
> > > Please mention a specific test case that fails.
> > 
> > It was any of the check-qtest-$(TARGET)s that failed.  I'll mention
> > that in next post.
> > 
> > > 
> > > > 
> > > >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> > > >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > > > 
> > > > This patch keeps the reference for the watch object when creating in
> > > > io_add_watch_poll(), so that the object will never be released in the
> > > > context main loop, especially when the context loop is running in
> > > > another standalone thread.  Meanwhile, when we want to remove the watch
> > > > object, we always first detach the watch object from its owner context,
> > > > then we continue with the cleanup.
> > > > 
> > > > Without this patch, calling io_remove_watch_poll() in main loop thread
> > > > is not thread-safe, since the other per-monitor thread may be modifying
> > > > the watch object at the same time.
> > > > 
> > > > Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > > > Signed-off-by: Peter Xu <peterx@redhat.com>
> > > > ---
> > > >  chardev/char-io.c | 16 ++++++++++++++--
> > > >  1 file changed, 14 insertions(+), 2 deletions(-)
> > > > 
> > > > diff --git a/chardev/char-io.c b/chardev/char-io.c
> > > > index f81052481a..50b5bac704 100644
> > > > --- a/chardev/char-io.c
> > > > +++ b/chardev/char-io.c
> > > > @@ -122,7 +122,6 @@ GSource *io_add_watch_poll(Chardev *chr,
> > > >      g_free(name);
> > > >  
> > > >      g_source_attach(&iwp->parent, context);
> > > > -    g_source_unref(&iwp->parent);
> > > >      return (GSource *)iwp;
> > > >  }
> > > >  
> > > > @@ -131,12 +130,25 @@ static void io_remove_watch_poll(GSource *source)
> > > >      IOWatchPoll *iwp;
> > > >  
> > > >      iwp = io_watch_poll_from_source(source);
> > > > +
> > > > +    /*
> > > > +     * Here the order of destruction really matters.  We need to first
> > > > +     * detach the IOWatchPoll object from the context (which may still
> > > > +     * be running in another loop thread), only after that could we
> > > > +     * continue to operate on iwp->src, or there may be race condition
> > > > +     * between current thread and the context loop thread.
> > > > +     *
> > > > +     * Let's blame the glib bug mentioned in commit 2b316774f6
> > > > +     * ("qemu-char: do not operate on sources from finalize
> > > > +     * callbacks") for this extra complexity.
> > > 
> > > I don't understand how this bug is to blame.  Isn't the problem here a
> > > race condition between two QEMU threads?
> > 
> > Yes, it is.
> > 
> > The problem is, we won't have the race condition if glib does not have
> > that bug mentioned.  Then the thread running GMainContext will have
> > full control of iwp->src destruction, and destruction of it would be
> > fairly straightforward (unref iwp->src in IOWatchPoll destructor).
> > Now IIUC we are doing this in a hacky way, say, we destroy iwp->src
> > explicitly from main thread before quitting (see [1] below, the whole
> > if clause).
> > 
> > > 
> > > Why are two threads accessing the watch at the same time?
> > 
> > Here is how I understand:
> > 
> > Firstly we need to tackle with that bug, by an explicit destruction of
> > iwp->src below; meanwhile when we are destroying it, the GMainContext
> > can still be running somewhere (it's not happening in current series
> > since I stopped iothread earlier than this point, however it can still
> > happen if in the future we don't do that), then we possibly want this
> > patch.
> > 
> > Again, without this patch, current series should work; however I do
> > hope this patch can be in, in case someday we want to provide complete
> > thread safety for Chardevs (now it is not really thread-safe).
> 
> You said qtests fail with "Assertion `iwp->src == NULL' failed" but then
> you said "without this patch, current series should work".  How do you
> reproduce the failure if it doesn't occur?

Actually it occurs in some old versions, but not in current version.
Current version destroys the iothread earlier (as Dan suggested), so
it can avoid the issue.  Sorry for not being clear.

> 
> It looks like remove_fd_in_watch() -> io_remove_watch_poll() callers
> fall into two categories: called from within the event loop and called
> when a chardev is destroyed.  Do the thread-safety issues occur when the
> chardev is destroyed by the QEMU main loop thread?  Or did I miss cases
> where remove_fd_in_watch() is called from other threads?

I think this can also be called in monitor iothread?  Even if so, it's
pretty safe since if the monitor iothread is calling
remove_fd_in_watch() then it must not be using it after all.  The race
can happen when we are destroying the IOWatchPoll while the other
event loop thread (which may not be the main thread) is still running,
just like what I did in my old series.

> 
> > 
> > > 
> > > > +     */
> > > > +    g_source_destroy(&iwp->parent);
> > > >      if (iwp->src) {
> > > >          g_source_destroy(iwp->src);
> > > >          g_source_unref(iwp->src);
> > > >          iwp->src = NULL;
> > > >      }
> > 
> > [1]
> > 
> > > > -    g_source_destroy(&iwp->parent);
> > > > +    g_source_unref(&iwp->parent);
> > > >  }
> > > >  
> > > >  void remove_fd_in_watch(Chardev *chr)
> > > > -- 
> > > > 2.13.5
> > > > 
> > 
> > Thanks,
> > 
> > -- 
> > Peter Xu



-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 01/27] char-io: fix possible race on IOWatchPoll

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 6518 bytes --]

On Tue, Nov 14, 2017 at 07:31:10PM +0800, Peter Xu wrote:
> On Tue, Nov 14, 2017 at 10:32:19AM +0000, Stefan Hajnoczi wrote:
> > On Tue, Nov 14, 2017 at 02:09:39PM +0800, Peter Xu wrote:
> > > On Mon, Nov 13, 2017 at 04:52:11PM +0000, Stefan Hajnoczi wrote:
> > > > On Mon, Nov 06, 2017 at 05:46:17PM +0800, Peter Xu wrote:
> > > > > This is not a problem if we are only having one single loop thread like
> > > > > before.  However, after per-monitor thread is introduced, this is not
> > > > > true any more, and the race can happen.
> > > > > 
> > > > > The race can be triggered with "make check -j8" sometimes:
> > > > 
> > > > Please mention a specific test case that fails.
> > > 
> > > It was any of the check-qtest-$(TARGET)s that failed.  I'll mention
> > > that in next post.
> > > 
> > > > 
> > > > > 
> > > > >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> > > > >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > > > > 
> > > > > This patch keeps the reference for the watch object when creating in
> > > > > io_add_watch_poll(), so that the object will never be released in the
> > > > > context main loop, especially when the context loop is running in
> > > > > another standalone thread.  Meanwhile, when we want to remove the watch
> > > > > object, we always first detach the watch object from its owner context,
> > > > > then we continue with the cleanup.
> > > > > 
> > > > > Without this patch, calling io_remove_watch_poll() in main loop thread
> > > > > is not thread-safe, since the other per-monitor thread may be modifying
> > > > > the watch object at the same time.
> > > > > 
> > > > > Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > > > > Signed-off-by: Peter Xu <peterx@redhat.com>
> > > > > ---
> > > > >  chardev/char-io.c | 16 ++++++++++++++--
> > > > >  1 file changed, 14 insertions(+), 2 deletions(-)
> > > > > 
> > > > > diff --git a/chardev/char-io.c b/chardev/char-io.c
> > > > > index f81052481a..50b5bac704 100644
> > > > > --- a/chardev/char-io.c
> > > > > +++ b/chardev/char-io.c
> > > > > @@ -122,7 +122,6 @@ GSource *io_add_watch_poll(Chardev *chr,
> > > > >      g_free(name);
> > > > >  
> > > > >      g_source_attach(&iwp->parent, context);
> > > > > -    g_source_unref(&iwp->parent);
> > > > >      return (GSource *)iwp;
> > > > >  }
> > > > >  
> > > > > @@ -131,12 +130,25 @@ static void io_remove_watch_poll(GSource *source)
> > > > >      IOWatchPoll *iwp;
> > > > >  
> > > > >      iwp = io_watch_poll_from_source(source);
> > > > > +
> > > > > +    /*
> > > > > +     * Here the order of destruction really matters.  We need to first
> > > > > +     * detach the IOWatchPoll object from the context (which may still
> > > > > +     * be running in another loop thread), only after that could we
> > > > > +     * continue to operate on iwp->src, or there may be race condition
> > > > > +     * between current thread and the context loop thread.
> > > > > +     *
> > > > > +     * Let's blame the glib bug mentioned in commit 2b316774f6
> > > > > +     * ("qemu-char: do not operate on sources from finalize
> > > > > +     * callbacks") for this extra complexity.
> > > > 
> > > > I don't understand how this bug is to blame.  Isn't the problem here a
> > > > race condition between two QEMU threads?
> > > 
> > > Yes, it is.
> > > 
> > > The problem is, we won't have the race condition if glib does not have
> > > that bug mentioned.  Then the thread running GMainContext will have
> > > full control of iwp->src destruction, and destruction of it would be
> > > fairly straightforward (unref iwp->src in IOWatchPoll destructor).
> > > Now IIUC we are doing this in a hacky way, say, we destroy iwp->src
> > > explicitly from main thread before quitting (see [1] below, the whole
> > > if clause).
> > > 
> > > > 
> > > > Why are two threads accessing the watch at the same time?
> > > 
> > > Here is how I understand:
> > > 
> > > Firstly we need to tackle with that bug, by an explicit destruction of
> > > iwp->src below; meanwhile when we are destroying it, the GMainContext
> > > can still be running somewhere (it's not happening in current series
> > > since I stopped iothread earlier than this point, however it can still
> > > happen if in the future we don't do that), then we possibly want this
> > > patch.
> > > 
> > > Again, without this patch, current series should work; however I do
> > > hope this patch can be in, in case someday we want to provide complete
> > > thread safety for Chardevs (now it is not really thread-safe).
> > 
> > You said qtests fail with "Assertion `iwp->src == NULL' failed" but then
> > you said "without this patch, current series should work".  How do you
> > reproduce the failure if it doesn't occur?
> 
> Actually it occurs in some old versions, but not in current version.
> Current version destroys the iothread earlier (as Dan suggested), so
> it can avoid the issue.  Sorry for not being clear.
> 
> > 
> > It looks like remove_fd_in_watch() -> io_remove_watch_poll() callers
> > fall into two categories: called from within the event loop and called
> > when a chardev is destroyed.  Do the thread-safety issues occur when the
> > chardev is destroyed by the QEMU main loop thread?  Or did I miss cases
> > where remove_fd_in_watch() is called from other threads?
> 
> I think this can also be called in monitor iothread?

When I say "event loop", I mean any thread that is running an event loop
including IOThreads and the main loop thread.

What do you mean by "monitor iothread"?

> Even if so, it's
> pretty safe since if the monitor iothread is calling
> remove_fd_in_watch() then it must not be using it after all.  The race
> can happen when we are destroying the IOWatchPoll while the other
> event loop thread (which may not be the main thread) is still running,
> just like what I did in my old series.

The scenario this patch is trying to address doesn't make a lot of sense
since there will be further thread-safety problems if two threads are
modifying a Chardev at the same time.  A lock will probably be required
to protect the state and this patch might not be necessary then.

This patch seems very speculative and it's unclear what concrete
scenario it addresses.  I suggest dropping the patch from this series so
it is not a distraction from what you're actually trying to achieve.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4489 bytes --]

On Mon, Nov 06, 2017 at 09:08:00PM +0800, Peter Xu wrote:
> On Mon, Nov 06, 2017 at 02:12:17AM -0800, no-reply@patchew.org wrote:
> > Hi,
> > 
> > This series seems to have some coding style problems. See output below for
> > more information:
> > 
> > Subject: [Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support
> > Type: series
> > Message-id: 20171106094643.14881-1-peterx@redhat.com
> > 
> > === TEST SCRIPT BEGIN ===
> > #!/bin/bash
> > 
> > BASE=base
> > n=1
> > total=$(git log --oneline $BASE.. | wc -l)
> > failed=0
> > 
> > git config --local diff.renamelimit 0
> > git config --local diff.renames True
> > 
> > commits="$(git log --format=%H --reverse $BASE..)"
> > for c in $commits; do
> >     echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
> >     if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
> >         failed=1
> >         echo
> >     fi
> >     n=$((n+1))
> > done
> > 
> > exit $failed
> > === TEST SCRIPT END ===
> > 
> > Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
> > From https://github.com/patchew-project/qemu
> >  * [new tag]               patchew/20171106094643.14881-1-peterx@redhat.com -> patchew/20171106094643.14881-1-peterx@redhat.com
> > Switched to a new branch 'test'
> > 5525c4e791 tests: qmp-test: add oob test
> > ccc9e4c399 tests: qmp-test: verify command batching
> > 7f45b4c6c0 docs: update QMP documents for OOB commands
> > 58cfe877d2 monitor: enable IO thread for (qmp & !mux) typed
> > 5e1d56ce74 qmp: isolate responses into io thread
> > aef4275536 qmp: let migrate-incoming allow out-of-band
> > 5e68beacf3 qmp: support out-of-band (oob) execution
> > 43c7215a30 qapi: introduce new cmd option "allow-oob"
> > e11127ba4b monitor: send event when request queue full
> > 45cef4f7e4 qmp: add new event "request-dropped"
> > 485da28be1 monitor: separate QMP parser and dispatcher
> > 4892fe9ca2 monitor: let monitor_{suspend|resume} thread safe
> > 1b86166d9c monitor: introduce monitor_qmp_respond()
> > 0f48093add qmp: introduce some capability helpers
> > 8d3f33043d qmp: negociate QMP capabilities
> > 023b386d0e qmp: introduce QMPCapability
> > 2bde5ca8ce monitor: allow to use IO thread for parsing
> > f4cc112f80 monitor: create monitor dedicate iothread
> > 3590fdc1d4 monitor: let mon_list be tail queue
> > 11c818a9ac monitor: unify global init
> > 36d3efb87d monitor: move the cur_mon hack deeper for QMP
> > bf3e493a86 qjson: add "opaque" field to JSONMessageParser
> > 17367fe7a1 monitor: move skip_flush into monitor_data_init
> > 0c98d4baa4 qobject: let object_property_get_str() use new API
> > aa4b973dd5 qobject: introduce qobject_get_try_str()
> > 981ccebc1e qobject: introduce qstring_get_try_str()
> > d40ba38085 char-io: fix possible race on IOWatchPoll
> > 
> > === OUTPUT BEGIN ===
> > Checking PATCH 1/27: char-io: fix possible race on IOWatchPoll...
> > Checking PATCH 2/27: qobject: introduce qstring_get_try_str()...
> > Checking PATCH 3/27: qobject: introduce qobject_get_try_str()...
> > Checking PATCH 4/27: qobject: let object_property_get_str() use new API...
> > Checking PATCH 5/27: monitor: move skip_flush into monitor_data_init...
> > Checking PATCH 6/27: qjson: add "opaque" field to JSONMessageParser...
> > Checking PATCH 7/27: monitor: move the cur_mon hack deeper for QMP...
> > Checking PATCH 8/27: monitor: unify global init...
> > Checking PATCH 9/27: monitor: let mon_list be tail queue...
> > Checking PATCH 10/27: monitor: create monitor dedicate iothread...
> > Checking PATCH 11/27: monitor: allow to use IO thread for parsing...
> > Checking PATCH 12/27: qmp: introduce QMPCapability...
> > Checking PATCH 13/27: qmp: negociate QMP capabilities...
> > Checking PATCH 14/27: qmp: introduce some capability helpers...
> > Checking PATCH 15/27: monitor: introduce monitor_qmp_respond()...
> > Checking PATCH 16/27: monitor: let monitor_{suspend|resume} thread safe...
> > ERROR: braces {} are necessary for all arms of this statement
> > #28: FILE: monitor.c:4014:
> > +    if (atomic_dec_fetch(&mon->suspend_cnt) == 0)
> > [...]
> > 
> > ERROR: Missing Signed-off-by: line(s)
> 
> Will add it in next post.  I still haven't found a good way to let
> magit add this line for me every time automatically.

If you don't want to type "git commit -s" all the time you could use the
"format.signOff = on" git-config(1) variable.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v3 01/27] char-io: fix possible race on IOWatchPoll

[Peter Xu]

On Wed, Nov 15, 2017 at 09:37:40AM +0000, Stefan Hajnoczi wrote:
> On Tue, Nov 14, 2017 at 07:31:10PM +0800, Peter Xu wrote:
> > On Tue, Nov 14, 2017 at 10:32:19AM +0000, Stefan Hajnoczi wrote:
> > > On Tue, Nov 14, 2017 at 02:09:39PM +0800, Peter Xu wrote:
> > > > On Mon, Nov 13, 2017 at 04:52:11PM +0000, Stefan Hajnoczi wrote:
> > > > > On Mon, Nov 06, 2017 at 05:46:17PM +0800, Peter Xu wrote:
> > > > > > This is not a problem if we are only having one single loop thread like
> > > > > > before.  However, after per-monitor thread is introduced, this is not
> > > > > > true any more, and the race can happen.
> > > > > > 
> > > > > > The race can be triggered with "make check -j8" sometimes:
> > > > > 
> > > > > Please mention a specific test case that fails.
> > > > 
> > > > It was any of the check-qtest-$(TARGET)s that failed.  I'll mention
> > > > that in next post.
> > > > 
> > > > > 
> > > > > > 
> > > > > >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> > > > > >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > > > > > 
> > > > > > This patch keeps the reference for the watch object when creating in
> > > > > > io_add_watch_poll(), so that the object will never be released in the
> > > > > > context main loop, especially when the context loop is running in
> > > > > > another standalone thread.  Meanwhile, when we want to remove the watch
> > > > > > object, we always first detach the watch object from its owner context,
> > > > > > then we continue with the cleanup.
> > > > > > 
> > > > > > Without this patch, calling io_remove_watch_poll() in main loop thread
> > > > > > is not thread-safe, since the other per-monitor thread may be modifying
> > > > > > the watch object at the same time.
> > > > > > 
> > > > > > Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > > > > > Signed-off-by: Peter Xu <peterx@redhat.com>
> > > > > > ---
> > > > > >  chardev/char-io.c | 16 ++++++++++++++--
> > > > > >  1 file changed, 14 insertions(+), 2 deletions(-)
> > > > > > 
> > > > > > diff --git a/chardev/char-io.c b/chardev/char-io.c
> > > > > > index f81052481a..50b5bac704 100644
> > > > > > --- a/chardev/char-io.c
> > > > > > +++ b/chardev/char-io.c
> > > > > > @@ -122,7 +122,6 @@ GSource *io_add_watch_poll(Chardev *chr,
> > > > > >      g_free(name);
> > > > > >  
> > > > > >      g_source_attach(&iwp->parent, context);
> > > > > > -    g_source_unref(&iwp->parent);
> > > > > >      return (GSource *)iwp;
> > > > > >  }
> > > > > >  
> > > > > > @@ -131,12 +130,25 @@ static void io_remove_watch_poll(GSource *source)
> > > > > >      IOWatchPoll *iwp;
> > > > > >  
> > > > > >      iwp = io_watch_poll_from_source(source);
> > > > > > +
> > > > > > +    /*
> > > > > > +     * Here the order of destruction really matters.  We need to first
> > > > > > +     * detach the IOWatchPoll object from the context (which may still
> > > > > > +     * be running in another loop thread), only after that could we
> > > > > > +     * continue to operate on iwp->src, or there may be race condition
> > > > > > +     * between current thread and the context loop thread.
> > > > > > +     *
> > > > > > +     * Let's blame the glib bug mentioned in commit 2b316774f6
> > > > > > +     * ("qemu-char: do not operate on sources from finalize
> > > > > > +     * callbacks") for this extra complexity.
> > > > > 
> > > > > I don't understand how this bug is to blame.  Isn't the problem here a
> > > > > race condition between two QEMU threads?
> > > > 
> > > > Yes, it is.
> > > > 
> > > > The problem is, we won't have the race condition if glib does not have
> > > > that bug mentioned.  Then the thread running GMainContext will have
> > > > full control of iwp->src destruction, and destruction of it would be
> > > > fairly straightforward (unref iwp->src in IOWatchPoll destructor).
> > > > Now IIUC we are doing this in a hacky way, say, we destroy iwp->src
> > > > explicitly from main thread before quitting (see [1] below, the whole
> > > > if clause).
> > > > 
> > > > > 
> > > > > Why are two threads accessing the watch at the same time?
> > > > 
> > > > Here is how I understand:
> > > > 
> > > > Firstly we need to tackle with that bug, by an explicit destruction of
> > > > iwp->src below; meanwhile when we are destroying it, the GMainContext
> > > > can still be running somewhere (it's not happening in current series
> > > > since I stopped iothread earlier than this point, however it can still
> > > > happen if in the future we don't do that), then we possibly want this
> > > > patch.
> > > > 
> > > > Again, without this patch, current series should work; however I do
> > > > hope this patch can be in, in case someday we want to provide complete
> > > > thread safety for Chardevs (now it is not really thread-safe).
> > > 
> > > You said qtests fail with "Assertion `iwp->src == NULL' failed" but then
> > > you said "without this patch, current series should work".  How do you
> > > reproduce the failure if it doesn't occur?
> > 
> > Actually it occurs in some old versions, but not in current version.
> > Current version destroys the iothread earlier (as Dan suggested), so
> > it can avoid the issue.  Sorry for not being clear.
> > 
> > > 
> > > It looks like remove_fd_in_watch() -> io_remove_watch_poll() callers
> > > fall into two categories: called from within the event loop and called
> > > when a chardev is destroyed.  Do the thread-safety issues occur when the
> > > chardev is destroyed by the QEMU main loop thread?  Or did I miss cases
> > > where remove_fd_in_watch() is called from other threads?
> > 
> > I think this can also be called in monitor iothread?
> 
> When I say "event loop", I mean any thread that is running an event loop
> including IOThreads and the main loop thread.
> 
> What do you mean by "monitor iothread"?

Ah, I see.  Yes, then I think it's true - the failure only happens
when remove_fd_in_watch() is called during destruction in main loop
thread.

> 
> > Even if so, it's
> > pretty safe since if the monitor iothread is calling
> > remove_fd_in_watch() then it must not be using it after all.  The race
> > can happen when we are destroying the IOWatchPoll while the other
> > event loop thread (which may not be the main thread) is still running,
> > just like what I did in my old series.
> 
> The scenario this patch is trying to address doesn't make a lot of sense
> since there will be further thread-safety problems if two threads are
> modifying a Chardev at the same time.  A lock will probably be required
> to protect the state and this patch might not be necessary then.
> 
> This patch seems very speculative and it's unclear what concrete
> scenario it addresses.  I suggest dropping the patch from this series so
> it is not a distraction from what you're actually trying to achieve.

Ok, then let me drop it.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 27/27] tests: qmp-test: add oob test

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2049 bytes --]

On Mon, Nov 06, 2017 at 05:46:43PM +0800, Peter Xu wrote:
> +    /*
> +     * Try a time-consuming command, following by a OOB command, make
> +     * sure we get OOB command before the time-consuming one (which is
> +     * run in the parser).
> +     *
> +     * When writting up this test script, the only command that
> +     * support OOB is migrate-incoming.  It's not the best command to
> +     * test OOB but we don't really have a choice here.  We will check
> +     * arriving order but not command errors, which does not really
> +     * matter to us.
> +     */
> +    qmp_async("{ 'execute': 'dump-guest-memory',"
> +              "  'arguments': { 'paging': true, "
> +              "                 'protocol': 'file:/dev/null' }, "
> +              "  'id': 'time-consuming-cmd'}");
> +    qmp_async("{ 'execute': 'migrate-incoming', "
> +              "  'control': { 'run-oob': true }, "
> +              "  'id': 'oob-cmd' }");
> +
> +    /* Ignore all events.  Wait for 2 acks */
> +    while (acks < 2) {
> +        resp = qmp_receive();
> +        if (qdict_haskey(resp, "event")) {
> +            /* Skip possible events */
> +            continue;
> +        }
> +        cmd_id = qdict_get_str(resp, "id");
> +        if (acks == 0) {
> +            /* Need to receive OOB response first */
> +            g_assert_cmpstr(cmd_id, ==, "oob-cmd");
> +        } else if (acks == 1) {
> +            g_assert_cmpstr(cmd_id, ==, "time-consuming-cmd");
> +        }
> +        acks++;
> +    }

This test is non-deterministic.  The dump-guest-memory command could
complete first on a fast machine.

On a slow machine this test might take a long time...

Please introduce a test command that is deterministic.  For example,
when 'x-oob-test' is invoked without 'run-oob': true it waits until
invoked again, this time with 'run-oob': true.

We have similar interfaces in the block layer for controlling the order
in which parallel I/O requests are processed.  This allows test cases to
deterministically take specific code paths.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v3 18/27] qmp: add new event "request-dropped"

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1584 bytes --]

On Mon, Nov 06, 2017 at 05:46:34PM +0800, Peter Xu wrote:
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 531fd4c0db..650714da06 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -3222,3 +3222,38 @@
>  # Since: 2.11
>  ##
>  { 'command': 'watchdog-set-action', 'data' : {'action': 'WatchdogAction'} }
> +
> +##
> +# @RequestDropReason:
> +#
> +# Reasons that caused one request to be dropped.

Please use "command" consistently.  QMP does not call it not "request".

> +#
> +# @queue-full: the queue of request is full.
> +#
> +# Since: 2.12
> +##
> +{ 'enum': 'RequestDropReason',
> +  'data': ['queue-full' ] }
> +
> +##
> +# @REQUEST_DROPPED:
> +#
> +# Emitted when one QMP request is dropped due to some reason.

Please add:

  REQUEST_DROPPED is only emitted when the oob capability is enabled.

Rationale: old clients don't know about this event so they cannot be
expected to handle it!

> +#
> +# @id:    If the original request contains an string-typed "id" field,
> +#         it'll be put into this field.  Otherwise it'll be an empty
> +#         string.

Please change:

  @id: The dropped command's string-typed "id" field.

Sending commands without the id field is likely to cause confusion since
there are cases where the client is unable to determine which command
was meant.  Since client code needs to be updated to enable the oob
capability anyway, we might as well require that clients always include
the id field with every command when the oob capability is enabled.
Please mention this requirement where the oob capability is documented.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Wed, Nov 15, 2017 at 09:42:46AM +0000, Stefan Hajnoczi wrote:
> On Mon, Nov 06, 2017 at 09:08:00PM +0800, Peter Xu wrote:
> > On Mon, Nov 06, 2017 at 02:12:17AM -0800, no-reply@patchew.org wrote:
> > > Hi,
> > > 
> > > This series seems to have some coding style problems. See output below for
> > > more information:
> > > 
> > > Subject: [Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support
> > > Type: series
> > > Message-id: 20171106094643.14881-1-peterx@redhat.com
> > > 
> > > === TEST SCRIPT BEGIN ===
> > > #!/bin/bash
> > > 
> > > BASE=base
> > > n=1
> > > total=$(git log --oneline $BASE.. | wc -l)
> > > failed=0
> > > 
> > > git config --local diff.renamelimit 0
> > > git config --local diff.renames True
> > > 
> > > commits="$(git log --format=%H --reverse $BASE..)"
> > > for c in $commits; do
> > >     echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
> > >     if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
> > >         failed=1
> > >         echo
> > >     fi
> > >     n=$((n+1))
> > > done
> > > 
> > > exit $failed
> > > === TEST SCRIPT END ===
> > > 
> > > Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
> > > From https://github.com/patchew-project/qemu
> > >  * [new tag]               patchew/20171106094643.14881-1-peterx@redhat.com -> patchew/20171106094643.14881-1-peterx@redhat.com
> > > Switched to a new branch 'test'
> > > 5525c4e791 tests: qmp-test: add oob test
> > > ccc9e4c399 tests: qmp-test: verify command batching
> > > 7f45b4c6c0 docs: update QMP documents for OOB commands
> > > 58cfe877d2 monitor: enable IO thread for (qmp & !mux) typed
> > > 5e1d56ce74 qmp: isolate responses into io thread
> > > aef4275536 qmp: let migrate-incoming allow out-of-band
> > > 5e68beacf3 qmp: support out-of-band (oob) execution
> > > 43c7215a30 qapi: introduce new cmd option "allow-oob"
> > > e11127ba4b monitor: send event when request queue full
> > > 45cef4f7e4 qmp: add new event "request-dropped"
> > > 485da28be1 monitor: separate QMP parser and dispatcher
> > > 4892fe9ca2 monitor: let monitor_{suspend|resume} thread safe
> > > 1b86166d9c monitor: introduce monitor_qmp_respond()
> > > 0f48093add qmp: introduce some capability helpers
> > > 8d3f33043d qmp: negociate QMP capabilities
> > > 023b386d0e qmp: introduce QMPCapability
> > > 2bde5ca8ce monitor: allow to use IO thread for parsing
> > > f4cc112f80 monitor: create monitor dedicate iothread
> > > 3590fdc1d4 monitor: let mon_list be tail queue
> > > 11c818a9ac monitor: unify global init
> > > 36d3efb87d monitor: move the cur_mon hack deeper for QMP
> > > bf3e493a86 qjson: add "opaque" field to JSONMessageParser
> > > 17367fe7a1 monitor: move skip_flush into monitor_data_init
> > > 0c98d4baa4 qobject: let object_property_get_str() use new API
> > > aa4b973dd5 qobject: introduce qobject_get_try_str()
> > > 981ccebc1e qobject: introduce qstring_get_try_str()
> > > d40ba38085 char-io: fix possible race on IOWatchPoll
> > > 
> > > === OUTPUT BEGIN ===
> > > Checking PATCH 1/27: char-io: fix possible race on IOWatchPoll...
> > > Checking PATCH 2/27: qobject: introduce qstring_get_try_str()...
> > > Checking PATCH 3/27: qobject: introduce qobject_get_try_str()...
> > > Checking PATCH 4/27: qobject: let object_property_get_str() use new API...
> > > Checking PATCH 5/27: monitor: move skip_flush into monitor_data_init...
> > > Checking PATCH 6/27: qjson: add "opaque" field to JSONMessageParser...
> > > Checking PATCH 7/27: monitor: move the cur_mon hack deeper for QMP...
> > > Checking PATCH 8/27: monitor: unify global init...
> > > Checking PATCH 9/27: monitor: let mon_list be tail queue...
> > > Checking PATCH 10/27: monitor: create monitor dedicate iothread...
> > > Checking PATCH 11/27: monitor: allow to use IO thread for parsing...
> > > Checking PATCH 12/27: qmp: introduce QMPCapability...
> > > Checking PATCH 13/27: qmp: negociate QMP capabilities...
> > > Checking PATCH 14/27: qmp: introduce some capability helpers...
> > > Checking PATCH 15/27: monitor: introduce monitor_qmp_respond()...
> > > Checking PATCH 16/27: monitor: let monitor_{suspend|resume} thread safe...
> > > ERROR: braces {} are necessary for all arms of this statement
> > > #28: FILE: monitor.c:4014:
> > > +    if (atomic_dec_fetch(&mon->suspend_cnt) == 0)
> > > [...]
> > > 
> > > ERROR: Missing Signed-off-by: line(s)
> > 
> > Will add it in next post.  I still haven't found a good way to let
> > magit add this line for me every time automatically.
> 
> If you don't want to type "git commit -s" all the time you could use the
> "format.signOff = on" git-config(1) variable.

I tried it, but it seems to be for when formatting patches rather than
putting that s-o-b line into commit messages.  I think what I wanted
was something like commit.signOff but after some searches I found that
idea was proposed but rejected somehow:

http://git.661346.n2.nabble.com/PATCH-Add-a-commit-signoff-configuration-variable-to-always-use-signoff-td1886619.html#a2093996

So I think I'll just live with my current approach.  Anway, thanks for
the hint!

-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support

[Fam Zheng]

On Thu, 11/16 13:32, Peter Xu wrote:
> On Wed, Nov 15, 2017 at 09:42:46AM +0000, Stefan Hajnoczi wrote:
> > On Mon, Nov 06, 2017 at 09:08:00PM +0800, Peter Xu wrote:
> > > On Mon, Nov 06, 2017 at 02:12:17AM -0800, no-reply@patchew.org wrote:
> > > > Hi,
> > > > 
> > > > This series seems to have some coding style problems. See output below for
> > > > more information:
> > > > 
> > > > Subject: [Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support
> > > > Type: series
> > > > Message-id: 20171106094643.14881-1-peterx@redhat.com
> > > > 
> > > > === TEST SCRIPT BEGIN ===
> > > > #!/bin/bash
> > > > 
> > > > BASE=base
> > > > n=1
> > > > total=$(git log --oneline $BASE.. | wc -l)
> > > > failed=0
> > > > 
> > > > git config --local diff.renamelimit 0
> > > > git config --local diff.renames True
> > > > 
> > > > commits="$(git log --format=%H --reverse $BASE..)"
> > > > for c in $commits; do
> > > >     echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
> > > >     if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
> > > >         failed=1
> > > >         echo
> > > >     fi
> > > >     n=$((n+1))
> > > > done
> > > > 
> > > > exit $failed
> > > > === TEST SCRIPT END ===
> > > > 
> > > > Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
> > > > From https://github.com/patchew-project/qemu
> > > >  * [new tag]               patchew/20171106094643.14881-1-peterx@redhat.com -> patchew/20171106094643.14881-1-peterx@redhat.com
> > > > Switched to a new branch 'test'
> > > > 5525c4e791 tests: qmp-test: add oob test
> > > > ccc9e4c399 tests: qmp-test: verify command batching
> > > > 7f45b4c6c0 docs: update QMP documents for OOB commands
> > > > 58cfe877d2 monitor: enable IO thread for (qmp & !mux) typed
> > > > 5e1d56ce74 qmp: isolate responses into io thread
> > > > aef4275536 qmp: let migrate-incoming allow out-of-band
> > > > 5e68beacf3 qmp: support out-of-band (oob) execution
> > > > 43c7215a30 qapi: introduce new cmd option "allow-oob"
> > > > e11127ba4b monitor: send event when request queue full
> > > > 45cef4f7e4 qmp: add new event "request-dropped"
> > > > 485da28be1 monitor: separate QMP parser and dispatcher
> > > > 4892fe9ca2 monitor: let monitor_{suspend|resume} thread safe
> > > > 1b86166d9c monitor: introduce monitor_qmp_respond()
> > > > 0f48093add qmp: introduce some capability helpers
> > > > 8d3f33043d qmp: negociate QMP capabilities
> > > > 023b386d0e qmp: introduce QMPCapability
> > > > 2bde5ca8ce monitor: allow to use IO thread for parsing
> > > > f4cc112f80 monitor: create monitor dedicate iothread
> > > > 3590fdc1d4 monitor: let mon_list be tail queue
> > > > 11c818a9ac monitor: unify global init
> > > > 36d3efb87d monitor: move the cur_mon hack deeper for QMP
> > > > bf3e493a86 qjson: add "opaque" field to JSONMessageParser
> > > > 17367fe7a1 monitor: move skip_flush into monitor_data_init
> > > > 0c98d4baa4 qobject: let object_property_get_str() use new API
> > > > aa4b973dd5 qobject: introduce qobject_get_try_str()
> > > > 981ccebc1e qobject: introduce qstring_get_try_str()
> > > > d40ba38085 char-io: fix possible race on IOWatchPoll
> > > > 
> > > > === OUTPUT BEGIN ===
> > > > Checking PATCH 1/27: char-io: fix possible race on IOWatchPoll...
> > > > Checking PATCH 2/27: qobject: introduce qstring_get_try_str()...
> > > > Checking PATCH 3/27: qobject: introduce qobject_get_try_str()...
> > > > Checking PATCH 4/27: qobject: let object_property_get_str() use new API...
> > > > Checking PATCH 5/27: monitor: move skip_flush into monitor_data_init...
> > > > Checking PATCH 6/27: qjson: add "opaque" field to JSONMessageParser...
> > > > Checking PATCH 7/27: monitor: move the cur_mon hack deeper for QMP...
> > > > Checking PATCH 8/27: monitor: unify global init...
> > > > Checking PATCH 9/27: monitor: let mon_list be tail queue...
> > > > Checking PATCH 10/27: monitor: create monitor dedicate iothread...
> > > > Checking PATCH 11/27: monitor: allow to use IO thread for parsing...
> > > > Checking PATCH 12/27: qmp: introduce QMPCapability...
> > > > Checking PATCH 13/27: qmp: negociate QMP capabilities...
> > > > Checking PATCH 14/27: qmp: introduce some capability helpers...
> > > > Checking PATCH 15/27: monitor: introduce monitor_qmp_respond()...
> > > > Checking PATCH 16/27: monitor: let monitor_{suspend|resume} thread safe...
> > > > ERROR: braces {} are necessary for all arms of this statement
> > > > #28: FILE: monitor.c:4014:
> > > > +    if (atomic_dec_fetch(&mon->suspend_cnt) == 0)
> > > > [...]
> > > > 
> > > > ERROR: Missing Signed-off-by: line(s)
> > > 
> > > Will add it in next post.  I still haven't found a good way to let
> > > magit add this line for me every time automatically.
> > 
> > If you don't want to type "git commit -s" all the time you could use the
> > "format.signOff = on" git-config(1) variable.
> 
> I tried it, but it seems to be for when formatting patches rather than
> putting that s-o-b line into commit messages.  I think what I wanted
> was something like commit.signOff but after some searches I found that
> idea was proposed but rejected somehow:
> 
> http://git.661346.n2.nabble.com/PATCH-Add-a-commit-signoff-configuration-variable-to-always-use-signoff-td1886619.html#a2093996
> 
> So I think I'll just live with my current approach.  Anway, thanks for
> the hint!

I think what you could better use here is a checkpatch.pl hook, either before
committing or before posting. I personally prefer the checkpatch.pl check in the
pre-publish-tag hook of git-publish so I can freely do dirty commits that
doesn't need s-o-b lines when I'm working on the branch, but still remember to
add it before posting.

Fam

Re: [Qemu-devel] [RFC v3 18/27] qmp: add new event "request-dropped"

[Peter Xu]

On Wed, Nov 15, 2017 at 10:50:15AM +0000, Stefan Hajnoczi wrote:
> On Mon, Nov 06, 2017 at 05:46:34PM +0800, Peter Xu wrote:
> > diff --git a/qapi-schema.json b/qapi-schema.json
> > index 531fd4c0db..650714da06 100644
> > --- a/qapi-schema.json
> > +++ b/qapi-schema.json
> > @@ -3222,3 +3222,38 @@
> >  # Since: 2.11
> >  ##
> >  { 'command': 'watchdog-set-action', 'data' : {'action': 'WatchdogAction'} }
> > +
> > +##
> > +# @RequestDropReason:
> > +#
> > +# Reasons that caused one request to be dropped.
> 
> Please use "command" consistently.  QMP does not call it not "request".

Sure.

> 
> > +#
> > +# @queue-full: the queue of request is full.
> > +#
> > +# Since: 2.12
> > +##
> > +{ 'enum': 'RequestDropReason',
> > +  'data': ['queue-full' ] }
> > +
> > +##
> > +# @REQUEST_DROPPED:
> > +#
> > +# Emitted when one QMP request is dropped due to some reason.
> 
> Please add:
> 
>   REQUEST_DROPPED is only emitted when the oob capability is enabled.
> 
> Rationale: old clients don't know about this event so they cannot be
> expected to handle it!

Added.

> 
> > +#
> > +# @id:    If the original request contains an string-typed "id" field,
> > +#         it'll be put into this field.  Otherwise it'll be an empty
> > +#         string.
> 
> Please change:
> 
>   @id: The dropped command's string-typed "id" field.

Ok.

> 
> Sending commands without the id field is likely to cause confusion since
> there are cases where the client is unable to determine which command
> was meant.  Since client code needs to be updated to enable the oob
> capability anyway, we might as well require that clients always include
> the id field with every command when the oob capability is enabled.
> Please mention this requirement where the oob capability is documented.

Will do.  Thanks!

-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 18/27] qmp: add new event "request-dropped"

[Peter Xu]

On Thu, Nov 16, 2017 at 01:56:54PM +0800, Peter Xu wrote:
> On Wed, Nov 15, 2017 at 10:50:15AM +0000, Stefan Hajnoczi wrote:
> > On Mon, Nov 06, 2017 at 05:46:34PM +0800, Peter Xu wrote:
> > > diff --git a/qapi-schema.json b/qapi-schema.json
> > > index 531fd4c0db..650714da06 100644
> > > --- a/qapi-schema.json
> > > +++ b/qapi-schema.json
> > > @@ -3222,3 +3222,38 @@
> > >  # Since: 2.11
> > >  ##
> > >  { 'command': 'watchdog-set-action', 'data' : {'action': 'WatchdogAction'} }
> > > +
> > > +##
> > > +# @RequestDropReason:
> > > +#
> > > +# Reasons that caused one request to be dropped.
> > 
> > Please use "command" consistently.  QMP does not call it not "request".
> 
> Sure.
> 
> > 
> > > +#
> > > +# @queue-full: the queue of request is full.
> > > +#
> > > +# Since: 2.12
> > > +##
> > > +{ 'enum': 'RequestDropReason',
> > > +  'data': ['queue-full' ] }
> > > +
> > > +##
> > > +# @REQUEST_DROPPED:
> > > +#
> > > +# Emitted when one QMP request is dropped due to some reason.
> > 
> > Please add:
> > 
> >   REQUEST_DROPPED is only emitted when the oob capability is enabled.
> > 
> > Rationale: old clients don't know about this event so they cannot be
> > expected to handle it!
> 
> Added.
> 
> > 
> > > +#
> > > +# @id:    If the original request contains an string-typed "id" field,
> > > +#         it'll be put into this field.  Otherwise it'll be an empty
> > > +#         string.
> > 
> > Please change:
> > 
> >   @id: The dropped command's string-typed "id" field.
> 
> Ok.

I just found this one tricky: currently we allow all kinds of "id"
fields in QMP commands, which means that here I should allow all kinds
of "id" fields as well rather than restrict it as "string" typed.

But I don't really know how to do that in QMP, say, what I want is
something like:

{ 'event': 'REQUEST_DROPPED' ,
  'data': { 'id': 'object', 'reason': 'RequestDropReason' } }
                  ^^^^^^^^

Any thoughts on how to do that the simple way?

-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 00/27] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Thu, Nov 16, 2017 at 01:39:51PM +0800, Fam Zheng wrote:

[...]

> I think what you could better use here is a checkpatch.pl hook, either before
> committing or before posting. I personally prefer the checkpatch.pl check in the
> pre-publish-tag hook of git-publish so I can freely do dirty commits that
> doesn't need s-o-b lines when I'm working on the branch, but still remember to
> add it before posting.

Noted. :)

-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 18/27] qmp: add new event "request-dropped"

[Peter Xu]

On Thu, Nov 16, 2017 at 02:29:05PM +0800, Peter Xu wrote:

[...]

> > > > +#
> > > > +# @id:    If the original request contains an string-typed "id" field,
> > > > +#         it'll be put into this field.  Otherwise it'll be an empty
> > > > +#         string.
> > > 
> > > Please change:
> > > 
> > >   @id: The dropped command's string-typed "id" field.
> > 
> > Ok.
> 
> I just found this one tricky: currently we allow all kinds of "id"
> fields in QMP commands, which means that here I should allow all kinds
> of "id" fields as well rather than restrict it as "string" typed.
> 
> But I don't really know how to do that in QMP, say, what I want is
> something like:
> 
> { 'event': 'REQUEST_DROPPED' ,
>   'data': { 'id': 'object', 'reason': 'RequestDropReason' } }
>                   ^^^^^^^^
> 
> Any thoughts on how to do that the simple way?

Please ignore it.  I just noticed that we have type 'any' which I can
use.  Sorry for the noise.

-- 
Peter Xu

Re: [Qemu-devel] [RFC v3 27/27] tests: qmp-test: add oob test

[Peter Xu]

On Wed, Nov 15, 2017 at 10:21:16AM +0000, Stefan Hajnoczi wrote:
> On Mon, Nov 06, 2017 at 05:46:43PM +0800, Peter Xu wrote:
> > +    /*
> > +     * Try a time-consuming command, following by a OOB command, make
> > +     * sure we get OOB command before the time-consuming one (which is
> > +     * run in the parser).
> > +     *
> > +     * When writting up this test script, the only command that
> > +     * support OOB is migrate-incoming.  It's not the best command to
> > +     * test OOB but we don't really have a choice here.  We will check
> > +     * arriving order but not command errors, which does not really
> > +     * matter to us.
> > +     */
> > +    qmp_async("{ 'execute': 'dump-guest-memory',"
> > +              "  'arguments': { 'paging': true, "
> > +              "                 'protocol': 'file:/dev/null' }, "
> > +              "  'id': 'time-consuming-cmd'}");
> > +    qmp_async("{ 'execute': 'migrate-incoming', "
> > +              "  'control': { 'run-oob': true }, "
> > +              "  'id': 'oob-cmd' }");
> > +
> > +    /* Ignore all events.  Wait for 2 acks */
> > +    while (acks < 2) {
> > +        resp = qmp_receive();
> > +        if (qdict_haskey(resp, "event")) {
> > +            /* Skip possible events */
> > +            continue;
> > +        }
> > +        cmd_id = qdict_get_str(resp, "id");
> > +        if (acks == 0) {
> > +            /* Need to receive OOB response first */
> > +            g_assert_cmpstr(cmd_id, ==, "oob-cmd");
> > +        } else if (acks == 1) {
> > +            g_assert_cmpstr(cmd_id, ==, "time-consuming-cmd");
> > +        }
> > +        acks++;
> > +    }
> 
> This test is non-deterministic.  The dump-guest-memory command could
> complete first on a fast machine.
> 
> On a slow machine this test might take a long time...
> 
> Please introduce a test command that is deterministic.  For example,
> when 'x-oob-test' is invoked without 'run-oob': true it waits until
> invoked again, this time with 'run-oob': true.

Yes this sounds good.

> 
> We have similar interfaces in the block layer for controlling the order
> in which parallel I/O requests are processed.  This allows test cases to
> deterministically take specific code paths.

It's great to know that I can create a command to test it.  That
should be much easier.  Thanks,

-- 
Peter Xu