[Qemu-devel] [RFC v5 00/26] QMP: out-of-band (OOB) execution support

[Qemu-devel] [RFC v5 00/26] QMP: out-of-band (OOB) execution support

[Peter Xu]

This version is mostly document update, and dropped the single patch
that is migration related (will be put into postcopy recovery
series).

v5
- rename "monitor_iothread" to "mon_iothread" [Dave]
- add comment in monitor_cleanup(), note that when the hacks can be
  removed. [Dan]
- add a note section in qmp-spec.txt, mentioning about how to migrate
  existing QMP command to oob-capable command. [Dave]
- drop patch "qmp: let migrate-incoming allow out-of-band".  All
  migration related changes will all be put into postcopy-recovery
  series.

v4:
- drop first patch to fix IOWatchPool [Stefan, Dan]
- add s-o-b where missing, and newly got r-bs
- fix English error in commit msg [Fam]
- some tunes on request-dropped event:  [Stefan]
  - firstly let 'id' be any type, meanwhile make sure "id" is there as
    long as OOB is enabled for the monitor.
  - some comments fix
- add new command "x-oob-test" for testing oob commands [Stefan]
- simplify the test codes to use new x-oob-test
- flush response queue before cleanup monitors

This series was born from this one:

  https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html

The idea comes from Markus Armbruster and the discussion we had in the
thread.  It's not a super ideal solution (I believe Markus had been
thinking hard to keep everything in order meanwhile trying to satisfy
the migration requirement), but AFAIU it's currently the best.

What is OOB?
============

It's the shortcut of Out-Of-Band execution, its name is given by
Markus.  It's a way to quickly execute a QMP request.  Say, originally
QMP is going throw these steps:

      JSON Parser --> QMP Dispatcher --> Respond
          /|\    (2)                (3)     |
       (1) |                               \|/ (4)
           +---------  main thread  --------+

The requests are executed by the so-called QMP-dispatcher after the
JSON is parsed.  If OOB is on, we run the command directly in the
parser and quickly returns.

This series changed the QMP handling logic by moving the command
parsing and responding phases into IOThreads, so to be more accurate,
after the series the above graph would change into this:

               queue/kick              queue/kick
     JSON Parser ======> QMP Dispatcher =====> Responder
         /|\ |     (3)       /|\  |      (4)      | /|\
      (1) |  | (2)            |   |               |  |
          |  |                |  \|/           (6)|  |(5)
          |  |            main thread             |  |
          |  |                                    |  |
          |  +--------> monitor IO thread <-------+  |
          +-----------/                   \----------+

New Interfaces
==============

QMP Introspection Changes
-------------------------

When do query-qmp-schema, we were getting something like:

  {"name": "migrate-incoming", "ret-type": "17",
   "meta-type": "command", "arg-type": "89"}

Now we will get a new key named "allow-oob":

  {"name": "migrate-incoming", "ret-type": "17", "allow-oob": true,
   "meta-type": "command", "arg-type": "89"}

Which shows whether a command supports OOB.

QMP Negociation Changes
-----------------------

We were running "qmp_capabilities" always without parameters like:

  {"execute": "qmp_capabilities"}

Now we can enable capabilities (we don't have any capability before
this series) like OOB using:

  {"execute": "qmp_capabilities", "arguments": {"enable": ["oob"]}}

Only after we explicitly enable OOB capability can we send requests in
OOB manner.  Otherwise we'll have exactly the same QMP session as
before, just like when OOB is not there.

When OOB is enabled, it's possible that OOB reply reaches faster than
previous command, so clients should be prepared.

Trigger OOB execution
---------------------

Let's take migrate-incoming as example.  The old command looks like:

  {"execute": "migrate-incoming", "arguments" : {"uri": "xxxxxx"}}

To execute a command with OOB execution, we need to specify it in the
QMP request in the extra "control" key:

  {"execute": "migrate-incoming", "arguments" : {"uri": "xxxxxx"},
   "control" { "run-oob": true } }

Then the command will be run in parser, and it can preempt other
commands.

Others
=================

The last patch of OOB test may need some attention.  I used
dump-guest-memory as a time-consuming command to test whether OOB is
working, and the only command I can test now is "migrate-incoming".  I
hope that is a "okay" solution for unit tests.  Any other suggestions
on that would be welcomed.

Please review.  Thanks.

Peter Xu (26):
  qobject: introduce qstring_get_try_str()
  qobject: introduce qobject_get_try_str()
  qobject: let object_property_get_str() use new API
  monitor: move skip_flush into monitor_data_init
  qjson: add "opaque" field to JSONMessageParser
  monitor: move the cur_mon hack deeper for QMP
  monitor: unify global init
  monitor: let mon_list be tail queue
  monitor: create monitor dedicate iothread
  monitor: allow to use IO thread for parsing
  qmp: introduce QMPCapability
  qmp: negociate QMP capabilities
  qmp: introduce some capability helpers
  monitor: introduce monitor_qmp_respond()
  monitor: let suspend_cnt be thread safe
  monitor: separate QMP parser and dispatcher
  qmp: add new event "request-dropped"
  monitor: send event when request queue full
  qapi: introduce new cmd option "allow-oob"
  qmp: support out-of-band (oob) execution
  qmp: isolate responses into io thread
  monitor: enable IO thread for (qmp & !mux) typed
  qmp: add command "x-oob-test"
  docs: update QMP documents for OOB commands
  tests: qmp-test: verify command batching
  tests: qmp-test: add oob test

 docs/devel/qapi-code-gen.txt     |  51 +++-
 docs/interop/qmp-spec.txt        |  49 +++-
 include/monitor/monitor.h        |   2 +-
 include/qapi/qmp/dispatch.h      |   2 +
 include/qapi/qmp/json-streamer.h |  10 +-
 include/qapi/qmp/qstring.h       |   2 +
 monitor.c                        | 566 +++++++++++++++++++++++++++++++++------
 qapi-schema.json                 |  80 +++++-
 qapi/introspect.json             |   6 +-
 qapi/qmp-dispatch.c              |  39 +++
 qga/main.c                       |   5 +-
 qmp.c                            |  16 ++
 qobject/json-streamer.c          |   6 +-
 qobject/qjson.c                  |   5 +-
 qobject/qstring.c                |  21 ++
 qom/object.c                     |   9 +-
 scripts/qapi-commands.py         |  19 +-
 scripts/qapi-introspect.py       |  10 +-
 scripts/qapi.py                  |  15 +-
 scripts/qapi2texi.py             |   2 +-
 tests/libqtest.c                 |   5 +-
 tests/qapi-schema/test-qapi.py   |   2 +-
 tests/qmp-test.c                 |  90 ++++++-
 trace-events                     |   2 +
 vl.c                             |   3 +-
 25 files changed, 892 insertions(+), 125 deletions(-)

-- 
2.14.3

[Qemu-devel] [RFC v5 01/26] qobject: introduce qstring_get_try_str()

[Peter Xu]

The only difference from qstring_get_str() is that it allows the qstring
to be NULL.  If so, NULL is returned.

CC: Eric Blake <eblake@redhat.com>
CC: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/qstring.h |  1 +
 qobject/qstring.c          | 10 ++++++++++
 2 files changed, 11 insertions(+)

diff --git a/include/qapi/qmp/qstring.h b/include/qapi/qmp/qstring.h
index 65c05a9be5..a145c8ca00 100644
--- a/include/qapi/qmp/qstring.h
+++ b/include/qapi/qmp/qstring.h
@@ -27,6 +27,7 @@ QString *qstring_from_str(const char *str);
 QString *qstring_from_substr(const char *str, int start, int end);
 size_t qstring_get_length(const QString *qstring);
 const char *qstring_get_str(const QString *qstring);
+const char *qstring_get_try_str(const QString *qstring);
 void qstring_append_int(QString *qstring, int64_t value);
 void qstring_append(QString *qstring, const char *str);
 void qstring_append_chr(QString *qstring, int c);
diff --git a/qobject/qstring.c b/qobject/qstring.c
index 74182a1c02..7638ab990d 100644
--- a/qobject/qstring.c
+++ b/qobject/qstring.c
@@ -128,6 +128,16 @@ const char *qstring_get_str(const QString *qstring)
     return qstring->string;
 }
 
+/**
+ * qstring_get_try_str(): Return a pointer to the stored string
+ *
+ * NOTE: will return NULL if qstring is not provided.
+ */
+const char *qstring_get_try_str(const QString *qstring)
+{
+    return qstring ? qstring_get_str(qstring) : NULL;
+}
+
 /**
  * qstring_is_equal(): Test whether the two QStrings are equal
  */
-- 
2.14.3

[Qemu-devel] [RFC v5 02/26] qobject: introduce qobject_get_try_str()

[Peter Xu]

A quick way to fetch string from qobject when it's a QString.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/qstring.h |  1 +
 qobject/qstring.c          | 11 +++++++++++
 2 files changed, 12 insertions(+)

diff --git a/include/qapi/qmp/qstring.h b/include/qapi/qmp/qstring.h
index a145c8ca00..6517d8e377 100644
--- a/include/qapi/qmp/qstring.h
+++ b/include/qapi/qmp/qstring.h
@@ -28,6 +28,7 @@ QString *qstring_from_substr(const char *str, int start, int end);
 size_t qstring_get_length(const QString *qstring);
 const char *qstring_get_str(const QString *qstring);
 const char *qstring_get_try_str(const QString *qstring);
+const char *qobject_get_try_str(const QObject *qstring);
 void qstring_append_int(QString *qstring, int64_t value);
 void qstring_append(QString *qstring, const char *str);
 void qstring_append_chr(QString *qstring, int c);
diff --git a/qobject/qstring.c b/qobject/qstring.c
index 7638ab990d..fc6834baa6 100644
--- a/qobject/qstring.c
+++ b/qobject/qstring.c
@@ -138,6 +138,17 @@ const char *qstring_get_try_str(const QString *qstring)
     return qstring ? qstring_get_str(qstring) : NULL;
 }
 
+/**
+ * qobject_get_try_str(): Return a pointer of the backstore string
+ *
+ * NOTE: the string will only be returned if the object is valid, and
+ * its type is QString, otherwise NULL is returned.
+ */
+const char *qobject_get_try_str(const QObject *qstring)
+{
+    return qstring_get_try_str(qobject_to_qstring(qstring));
+}
+
 /**
  * qstring_is_equal(): Test whether the two QStrings are equal
  */
-- 
2.14.3

[Qemu-devel] [RFC v5 03/26] qobject: let object_property_get_str() use new API

[Peter Xu]

We can simplify object_property_get_str() using the new
qobject_get_try_str().

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qom/object.c | 9 +++------
 1 file changed, 3 insertions(+), 6 deletions(-)

diff --git a/qom/object.c b/qom/object.c
index c58c52d518..9cbeb51f0b 100644
--- a/qom/object.c
+++ b/qom/object.c
@@ -1116,18 +1116,15 @@ char *object_property_get_str(Object *obj, const char *name,
                               Error **errp)
 {
     QObject *ret = object_property_get_qobject(obj, name, errp);
-    QString *qstring;
     char *retval;
 
     if (!ret) {
         return NULL;
     }
-    qstring = qobject_to_qstring(ret);
-    if (!qstring) {
+
+    retval = g_strdup(qobject_get_try_str(ret));
+    if (!retval) {
         error_setg(errp, QERR_INVALID_PARAMETER_TYPE, name, "string");
-        retval = NULL;
-    } else {
-        retval = g_strdup(qstring_get_str(qstring));
     }
 
     qobject_decref(ret);
-- 
2.14.3

[Qemu-devel] [RFC v5 04/26] monitor: move skip_flush into monitor_data_init

[Peter Xu]

It's part of the data init.  Collect it.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/monitor.c b/monitor.c
index e36fb5308d..3940737c1c 100644
--- a/monitor.c
+++ b/monitor.c
@@ -568,13 +568,14 @@ static void monitor_qapi_event_init(void)
 
 static void handle_hmp_command(Monitor *mon, const char *cmdline);
 
-static void monitor_data_init(Monitor *mon)
+static void monitor_data_init(Monitor *mon, bool skip_flush)
 {
     memset(mon, 0, sizeof(Monitor));
     qemu_mutex_init(&mon->out_lock);
     mon->outbuf = qstring_new();
     /* Use *mon_cmds by default. */
     mon->cmd_table = mon_cmds;
+    mon->skip_flush = skip_flush;
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -595,8 +596,7 @@ char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
     char *output = NULL;
     Monitor *old_mon, hmp;
 
-    monitor_data_init(&hmp);
-    hmp.skip_flush = true;
+    monitor_data_init(&hmp, true);
 
     old_mon = cur_mon;
     cur_mon = &hmp;
@@ -4089,7 +4089,7 @@ void monitor_init(Chardev *chr, int flags)
     }
 
     mon = g_malloc(sizeof(*mon));
-    monitor_data_init(mon);
+    monitor_data_init(mon, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
-- 
2.14.3

[Qemu-devel] [RFC v5 05/26] qjson: add "opaque" field to JSONMessageParser

[Peter Xu]

It'll be passed to emit() as well when it happens.  Since at it, add a
typedef for the emitter function.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/json-streamer.h | 10 ++++++++--
 monitor.c                        |  7 ++++---
 qga/main.c                       |  5 +++--
 qobject/json-streamer.c          |  6 ++++--
 qobject/qjson.c                  |  5 +++--
 tests/libqtest.c                 |  5 +++--
 6 files changed, 25 insertions(+), 13 deletions(-)

diff --git a/include/qapi/qmp/json-streamer.h b/include/qapi/qmp/json-streamer.h
index 00d8a23af8..9198b67342 100644
--- a/include/qapi/qmp/json-streamer.h
+++ b/include/qapi/qmp/json-streamer.h
@@ -23,9 +23,14 @@ typedef struct JSONToken {
     char str[];
 } JSONToken;
 
+struct JSONMessageParser;
+typedef void (*JSONMessageEmitFunc)(struct JSONMessageParser *parser,
+                                    GQueue *tokens, void *opaque);
+
 typedef struct JSONMessageParser
 {
-    void (*emit)(struct JSONMessageParser *parser, GQueue *tokens);
+    JSONMessageEmitFunc emit;
+    void *opaque;
     JSONLexer lexer;
     int brace_count;
     int bracket_count;
@@ -34,7 +39,8 @@ typedef struct JSONMessageParser
 } JSONMessageParser;
 
 void json_message_parser_init(JSONMessageParser *parser,
-                              void (*func)(JSONMessageParser *, GQueue *));
+                              JSONMessageEmitFunc func,
+                              void *opaque);
 
 int json_message_parser_feed(JSONMessageParser *parser,
                              const char *buffer, size_t size);
diff --git a/monitor.c b/monitor.c
index 3940737c1c..ab80d32c70 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3808,7 +3808,8 @@ static int monitor_can_read(void *opaque)
     return (mon->suspend_cnt == 0) ? 1 : 0;
 }
 
-static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
+static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
+                               void *opaque)
 {
     QObject *req, *rsp = NULL, *id = NULL;
     QDict *qdict = NULL;
@@ -3955,7 +3956,7 @@ static void monitor_qmp_event(void *opaque, int event)
         break;
     case CHR_EVENT_CLOSED:
         json_message_parser_destroy(&mon->qmp.parser);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
         mon_refcount--;
         monitor_fdsets_cleanup();
         break;
@@ -4105,7 +4106,7 @@ void monitor_init(Chardev *chr, int flags)
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
                                  monitor_qmp_event, NULL, mon, NULL, true);
         qemu_chr_fe_set_echo(&mon->chr, true);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
     } else {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_read,
                                  monitor_event, NULL, mon, NULL, true);
diff --git a/qga/main.c b/qga/main.c
index 62a62755bd..3b5ebbc1ee 100644
--- a/qga/main.c
+++ b/qga/main.c
@@ -593,7 +593,8 @@ static void process_command(GAState *s, QDict *req)
 }
 
 /* handle requests/control events coming in over the channel */
-static void process_event(JSONMessageParser *parser, GQueue *tokens)
+static void process_event(JSONMessageParser *parser, GQueue *tokens,
+                          void *opaque)
 {
     GAState *s = container_of(parser, GAState, parser);
     QDict *qdict;
@@ -1320,7 +1321,7 @@ static int run_agent(GAState *s, GAConfig *config, int socket_activation)
     s->command_state = ga_command_state_new();
     ga_command_state_init(s, s->command_state);
     ga_command_state_init_all(s->command_state);
-    json_message_parser_init(&s->parser, process_event);
+    json_message_parser_init(&s->parser, process_event, NULL);
 
 #ifndef _WIN32
     if (!register_signal_handlers()) {
diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c
index c51c2021f9..8fc1b15321 100644
--- a/qobject/json-streamer.c
+++ b/qobject/json-streamer.c
@@ -102,18 +102,20 @@ out_emit:
      */
     tokens = parser->tokens;
     parser->tokens = g_queue_new();
-    parser->emit(parser, tokens);
+    parser->emit(parser, tokens, parser->opaque);
     parser->token_size = 0;
 }
 
 void json_message_parser_init(JSONMessageParser *parser,
-                              void (*func)(JSONMessageParser *, GQueue *))
+                              JSONMessageEmitFunc func,
+                              void *opaque)
 {
     parser->emit = func;
     parser->brace_count = 0;
     parser->bracket_count = 0;
     parser->tokens = g_queue_new();
     parser->token_size = 0;
+    parser->opaque = opaque;
 
     json_lexer_init(&parser->lexer, json_message_process_token);
 }
diff --git a/qobject/qjson.c b/qobject/qjson.c
index 2e0930884e..f9766febe3 100644
--- a/qobject/qjson.c
+++ b/qobject/qjson.c
@@ -28,7 +28,8 @@ typedef struct JSONParsingState
     Error *err;
 } JSONParsingState;
 
-static void parse_json(JSONMessageParser *parser, GQueue *tokens)
+static void parse_json(JSONMessageParser *parser, GQueue *tokens,
+                       void *opaque)
 {
     JSONParsingState *s = container_of(parser, JSONParsingState, parser);
 
@@ -41,7 +42,7 @@ QObject *qobject_from_jsonv(const char *string, va_list *ap, Error **errp)
 
     state.ap = ap;
 
-    json_message_parser_init(&state.parser, parse_json);
+    json_message_parser_init(&state.parser, parse_json, NULL);
     json_message_parser_feed(&state.parser, string, strlen(string));
     json_message_parser_flush(&state.parser);
     json_message_parser_destroy(&state.parser);
diff --git a/tests/libqtest.c b/tests/libqtest.c
index 0ec8af2923..6e5ddc2ffd 100644
--- a/tests/libqtest.c
+++ b/tests/libqtest.c
@@ -412,7 +412,8 @@ typedef struct {
     QDict *response;
 } QMPResponseParser;
 
-static void qmp_response(JSONMessageParser *parser, GQueue *tokens)
+static void qmp_response(JSONMessageParser *parser, GQueue *tokens,
+                         void *opaque)
 {
     QMPResponseParser *qmp = container_of(parser, QMPResponseParser, parser);
     QObject *obj;
@@ -434,7 +435,7 @@ QDict *qmp_fd_receive(int fd)
     bool log = getenv("QTEST_LOG") != NULL;
 
     qmp.response = NULL;
-    json_message_parser_init(&qmp.parser, qmp_response);
+    json_message_parser_init(&qmp.parser, qmp_response, NULL);
     while (!qmp.response) {
         ssize_t len;
         char c;
-- 
2.14.3

[Qemu-devel] [RFC v5 06/26] monitor: move the cur_mon hack deeper for QMP

[Peter Xu]

In monitor_qmp_read(), we have the hack to temporarily replace the
cur_mon pointer.  Now we move this hack deeper inside the QMP dispatcher
routine since the Monitor pointer can be passed in to that using the new
JSON Parser opaque field now.

This does not make much sense as a single patch.  However, this will be
a big step for the next patch, when the QMP dispatcher routine will be
split from the QMP parser.

Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/monitor.c b/monitor.c
index ab80d32c70..322dfb5f31 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3813,7 +3813,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
 {
     QObject *req, *rsp = NULL, *id = NULL;
     QDict *qdict = NULL;
-    Monitor *mon = cur_mon;
+    Monitor *mon = opaque, *old_mon;
     Error *err = NULL;
 
     req = json_parser_parse_err(tokens, NULL, &err);
@@ -3838,8 +3838,13 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
         QDECREF(req_json);
     }
 
+    old_mon = cur_mon;
+    cur_mon = mon;
+
     rsp = qmp_dispatch(cur_mon->qmp.commands, req);
 
+    cur_mon = old_mon;
+
     if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
         qdict = qdict_get_qdict(qobject_to_qdict(rsp), "error");
         if (qdict
@@ -3876,13 +3881,9 @@ err_out:
 
 static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
 {
-    Monitor *old_mon = cur_mon;
-
-    cur_mon = opaque;
-
-    json_message_parser_feed(&cur_mon->qmp.parser, (const char *) buf, size);
+    Monitor *mon = opaque;
 
-    cur_mon = old_mon;
+    json_message_parser_feed(&mon->qmp.parser, (const char *) buf, size);
 }
 
 static void monitor_read(void *opaque, const uint8_t *buf, int size)
@@ -3956,7 +3957,7 @@ static void monitor_qmp_event(void *opaque, int event)
         break;
     case CHR_EVENT_CLOSED:
         json_message_parser_destroy(&mon->qmp.parser);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
         mon_refcount--;
         monitor_fdsets_cleanup();
         break;
@@ -4106,7 +4107,7 @@ void monitor_init(Chardev *chr, int flags)
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
                                  monitor_qmp_event, NULL, mon, NULL, true);
         qemu_chr_fe_set_echo(&mon->chr, true);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
     } else {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_read,
                                  monitor_event, NULL, mon, NULL, true);
-- 
2.14.3

[Qemu-devel] [RFC v5 07/26] monitor: unify global init

[Peter Xu]

There are many places for monitor init its globals, at least:

- monitor_init_qmp_commands() at the very beginning
- single function to init monitor_lock
- in the first entry of monitor_init() using "is_first_init"

Unify them a bit.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/monitor/monitor.h |  2 +-
 monitor.c                 | 25 ++++++++++---------------
 vl.c                      |  3 ++-
 3 files changed, 13 insertions(+), 17 deletions(-)

diff --git a/include/monitor/monitor.h b/include/monitor/monitor.h
index 83ea4a1aaf..3a5128ab1b 100644
--- a/include/monitor/monitor.h
+++ b/include/monitor/monitor.h
@@ -16,7 +16,7 @@ extern Monitor *cur_mon;
 
 bool monitor_cur_is_qmp(void);
 
-void monitor_init_qmp_commands(void);
+void monitor_init_globals(void);
 void monitor_init(Chardev *chr, int flags);
 void monitor_cleanup(void);
 
diff --git a/monitor.c b/monitor.c
index 322dfb5f31..ac5313023b 100644
--- a/monitor.c
+++ b/monitor.c
@@ -1001,7 +1001,7 @@ static void qmp_unregister_commands_hack(void)
 #endif
 }
 
-void monitor_init_qmp_commands(void)
+static void monitor_init_qmp_commands(void)
 {
     /*
      * Two command lists:
@@ -4034,6 +4034,14 @@ static void sortcmdlist(void)
     qsort((void *)info_cmds, array_num, elem_size, compare_mon_cmd);
 }
 
+void monitor_init_globals(void)
+{
+    monitor_init_qmp_commands();
+    monitor_qapi_event_init();
+    sortcmdlist();
+    qemu_mutex_init(&monitor_lock);
+}
+
 /* These functions just adapt the readline interface in a typesafe way.  We
  * could cast function pointers but that discards compiler checks.
  */
@@ -4074,23 +4082,10 @@ void error_vprintf_unless_qmp(const char *fmt, va_list ap)
     }
 }
 
-static void __attribute__((constructor)) monitor_lock_init(void)
-{
-    qemu_mutex_init(&monitor_lock);
-}
-
 void monitor_init(Chardev *chr, int flags)
 {
-    static int is_first_init = 1;
-    Monitor *mon;
-
-    if (is_first_init) {
-        monitor_qapi_event_init();
-        sortcmdlist();
-        is_first_init = 0;
-    }
+    Monitor *mon = g_malloc(sizeof(*mon));
 
-    mon = g_malloc(sizeof(*mon));
     monitor_data_init(mon, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
diff --git a/vl.c b/vl.c
index 1ad1c04637..1ec995a6ae 100644
--- a/vl.c
+++ b/vl.c
@@ -3144,7 +3144,6 @@ int main(int argc, char **argv, char **envp)
     qemu_init_exec_dir(argv[0]);
 
     module_call_init(MODULE_INIT_QOM);
-    monitor_init_qmp_commands();
 
     qemu_add_opts(&qemu_drive_opts);
     qemu_add_drive_opts(&qemu_legacy_drive_opts);
@@ -4690,6 +4689,8 @@ int main(int argc, char **argv, char **envp)
     default_drive(default_floppy, snapshot, IF_FLOPPY, 0, FD_OPTS);
     default_drive(default_sdcard, snapshot, IF_SD, 0, SD_OPTS);
 
+    monitor_init_globals();
+
     if (qemu_opts_foreach(qemu_find_opts("mon"),
                           mon_init_func, NULL, NULL)) {
         exit(1);
-- 
2.14.3

[Qemu-devel] [RFC v5 08/26] monitor: let mon_list be tail queue

[Peter Xu]

It was QLIST.  I want to use this list to do monitor priority job later,
which need tail insertion ability.  So switching to a tail queue.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/monitor.c b/monitor.c
index ac5313023b..a70ab5606b 100644
--- a/monitor.c
+++ b/monitor.c
@@ -205,7 +205,7 @@ struct Monitor {
     void *password_opaque;
     mon_cmd_t *cmd_table;
     QLIST_HEAD(,mon_fd_t) fds;
-    QLIST_ENTRY(Monitor) entry;
+    QTAILQ_ENTRY(Monitor) entry;
 };
 
 /* QMP checker flags */
@@ -214,7 +214,7 @@ struct Monitor {
 /* Protects mon_list, monitor_event_state.  */
 static QemuMutex monitor_lock;
 
-static QLIST_HEAD(mon_list, Monitor) mon_list;
+static QTAILQ_HEAD(mon_list, Monitor) mon_list;
 static QLIST_HEAD(mon_fdsets, MonFdset) mon_fdsets;
 static int mon_refcount;
 
@@ -415,7 +415,7 @@ static void monitor_qapi_event_emit(QAPIEvent event, QDict *qdict)
     Monitor *mon;
 
     trace_monitor_protocol_event_emit(event, qdict);
-    QLIST_FOREACH(mon, &mon_list, entry) {
+    QTAILQ_FOREACH(mon, &mon_list, entry) {
         if (monitor_is_qmp(mon)
             && mon->qmp.commands != &qmp_cap_negotiation_commands) {
             monitor_json_emitter(mon, QOBJECT(qdict));
@@ -4118,8 +4118,8 @@ void monitor_cleanup(void)
     Monitor *mon, *next;
 
     qemu_mutex_lock(&monitor_lock);
-    QLIST_FOREACH_SAFE(mon, &mon_list, entry, next) {
-        QLIST_REMOVE(mon, entry);
+    QTAILQ_FOREACH_SAFE(mon, &mon_list, entry, next) {
+        QTAILQ_REMOVE(&mon_list, mon, entry);
         monitor_data_destroy(mon);
         g_free(mon);
     }
-- 
2.14.3

[Qemu-devel] [RFC v5 09/26] monitor: create monitor dedicate iothread

[Peter Xu]

Create one IOThread for the monitors, prepared to handle all the
input/output IOs using existing iothread framework.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/monitor.c b/monitor.c
index a70ab5606b..6b60f6d91b 100644
--- a/monitor.c
+++ b/monitor.c
@@ -76,6 +76,7 @@
 #include "qmp-introspect.h"
 #include "sysemu/qtest.h"
 #include "sysemu/cpus.h"
+#include "sysemu/iothread.h"
 #include "qemu/cutils.h"
 #include "qapi/qmp/dispatch.h"
 
@@ -208,6 +209,12 @@ struct Monitor {
     QTAILQ_ENTRY(Monitor) entry;
 };
 
+struct MonitorGlobal {
+    IOThread *mon_iothread;
+};
+
+static struct MonitorGlobal mon_global;
+
 /* QMP checker flags */
 #define QMP_ACCEPT_UNKNOWNS 1
 
@@ -4034,12 +4041,24 @@ static void sortcmdlist(void)
     qsort((void *)info_cmds, array_num, elem_size, compare_mon_cmd);
 }
 
+static GMainContext *monitor_io_context_get(void)
+{
+    return iothread_get_g_main_context(mon_global.mon_iothread);
+}
+
+static void monitor_iothread_init(void)
+{
+    mon_global.mon_iothread = iothread_create("mon_iothread",
+                                              &error_abort);
+}
+
 void monitor_init_globals(void)
 {
     monitor_init_qmp_commands();
     monitor_qapi_event_init();
     sortcmdlist();
     qemu_mutex_init(&monitor_lock);
+    monitor_iothread_init();
 }
 
 /* These functions just adapt the readline interface in a typesafe way.  We
@@ -4117,6 +4136,16 @@ void monitor_cleanup(void)
 {
     Monitor *mon, *next;
 
+    /*
+     * We need to explicitly stop the iothread (but not destroy it),
+     * cleanup the monitor resources, then destroy the iothread.  See
+     * again on the glib bug mentioned in 2b316774f6 for a reason.
+     *
+     * TODO: the bug is fixed in glib 2.28, so we can remove this hack
+     * as long as we won't support glib versions older than it.
+     */
+    iothread_stop(mon_global.mon_iothread);
+
     qemu_mutex_lock(&monitor_lock);
     QTAILQ_FOREACH_SAFE(mon, &mon_list, entry, next) {
         QTAILQ_REMOVE(&mon_list, mon, entry);
@@ -4124,6 +4153,9 @@ void monitor_cleanup(void)
         g_free(mon);
     }
     qemu_mutex_unlock(&monitor_lock);
+
+    iothread_destroy(mon_global.mon_iothread);
+    mon_global.mon_iothread = NULL;
 }
 
 QemuOptsList qemu_mon_opts = {
-- 
2.14.3

[Qemu-devel] [RFC v5 10/26] monitor: allow to use IO thread for parsing

[Peter Xu]

For each Monitor, add one field "use_io_thr" to show whether it will be
using the dedicated monitor IO thread to handle input/output.  When set,
monitor IO parsing work will be offloaded to dedicated monitor IO
thread, rather than the original main loop thread.

This only works for QMP.  HMP will always be run on main loop thread.

Currently we're still keeping use_io_thr to off always.  Will turn it on
later at some point.

One thing to mention is that we cannot set use_io_thr for every QMP
monitors.  The problem is that MUXed typed chardevs may not work well
with it now. When MUX is used, frontend of chardev can be the monitor
plus something else.  The only thing we know would be safe to be run
outside main thread so far is the monitor frontend. All the rest of the
frontends should still be run in main thread only.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 38 ++++++++++++++++++++++++++++++--------
 1 file changed, 30 insertions(+), 8 deletions(-)

diff --git a/monitor.c b/monitor.c
index 6b60f6d91b..e8f5a586e4 100644
--- a/monitor.c
+++ b/monitor.c
@@ -191,6 +191,7 @@ struct Monitor {
     int flags;
     int suspend_cnt;
     bool skip_flush;
+    bool use_io_thr;
 
     QemuMutex out_lock;
     QString *outbuf;
@@ -575,7 +576,8 @@ static void monitor_qapi_event_init(void)
 
 static void handle_hmp_command(Monitor *mon, const char *cmdline);
 
-static void monitor_data_init(Monitor *mon, bool skip_flush)
+static void monitor_data_init(Monitor *mon, bool skip_flush,
+                              bool use_io_thr)
 {
     memset(mon, 0, sizeof(Monitor));
     qemu_mutex_init(&mon->out_lock);
@@ -583,6 +585,7 @@ static void monitor_data_init(Monitor *mon, bool skip_flush)
     /* Use *mon_cmds by default. */
     mon->cmd_table = mon_cmds;
     mon->skip_flush = skip_flush;
+    mon->use_io_thr = use_io_thr;
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -603,7 +606,7 @@ char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
     char *output = NULL;
     Monitor *old_mon, hmp;
 
-    monitor_data_init(&hmp, true);
+    monitor_data_init(&hmp, true, false);
 
     old_mon = cur_mon;
     cur_mon = &hmp;
@@ -4104,8 +4107,9 @@ void error_vprintf_unless_qmp(const char *fmt, va_list ap)
 void monitor_init(Chardev *chr, int flags)
 {
     Monitor *mon = g_malloc(sizeof(*mon));
+    GMainContext *context;
 
-    monitor_data_init(mon, false);
+    monitor_data_init(mon, false, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
@@ -4117,19 +4121,37 @@ void monitor_init(Chardev *chr, int flags)
         monitor_read_command(mon, 0);
     }
 
+    if (mon->use_io_thr) {
+        /*
+         * When use_io_thr is set, we use the global shared dedicated
+         * IO thread for this monitor to handle input/output.
+         */
+        context = monitor_io_context_get();
+        /* We should have inited globals before reaching here. */
+        assert(context);
+    } else {
+        /* The default main loop, which is the main thread */
+        context = NULL;
+    }
+
+    /*
+     * Add the monitor before running it (which is triggered by
+     * qemu_chr_fe_set_handlers), otherwise one monitor may find
+     * itself not on the mon_list when running.
+     */
+    qemu_mutex_lock(&monitor_lock);
+    QTAILQ_INSERT_HEAD(&mon_list, mon, entry);
+    qemu_mutex_unlock(&monitor_lock);
+
     if (monitor_is_qmp(mon)) {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
-                                 monitor_qmp_event, NULL, mon, NULL, true);
+                                 monitor_qmp_event, NULL, mon, context, true);
         qemu_chr_fe_set_echo(&mon->chr, true);
         json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
     } else {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_read,
                                  monitor_event, NULL, mon, NULL, true);
     }
-
-    qemu_mutex_lock(&monitor_lock);
-    QLIST_INSERT_HEAD(&mon_list, mon, entry);
-    qemu_mutex_unlock(&monitor_lock);
 }
 
 void monitor_cleanup(void)
-- 
2.14.3

[Qemu-devel] [RFC v5 11/26] qmp: introduce QMPCapability

[Peter Xu]

There was no QMP capabilities defined.  Define the first "oob" as
capability to allow out-of-band messages.

Also, touch up qmp-test.c to test the new bits.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c        | 15 +++++++++++++--
 qapi-schema.json | 13 +++++++++++++
 tests/qmp-test.c | 10 +++++++++-
 3 files changed, 35 insertions(+), 3 deletions(-)

diff --git a/monitor.c b/monitor.c
index e8f5a586e4..bad6ee8dd1 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3944,12 +3944,23 @@ void monitor_resume(Monitor *mon)
 
 static QObject *get_qmp_greeting(void)
 {
+    QDict *result = qdict_new(), *qmp = qdict_new();
+    QList *cap_list = qlist_new();
     QObject *ver = NULL;
+    QMPCapability cap;
+
+    qdict_put(result, "QMP", qmp);
 
     qmp_marshal_query_version(NULL, &ver, NULL);
+    qdict_put_obj(qmp, "version", ver);
+
+    for (cap = 0; cap < QMP_CAPABILITY__MAX; cap++) {
+        qlist_append(cap_list, qstring_from_str(
+                         QMPCapability_str(cap)));
+    }
+    qdict_put(qmp, "capabilities", cap_list);
 
-    return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': []}}",
-                              ver);
+    return QOBJECT(result);
 }
 
 static void monitor_qmp_event(void *opaque, int event)
diff --git a/qapi-schema.json b/qapi-schema.json
index 18457954a8..03201578b4 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -118,6 +118,19 @@
 ##
 { 'command': 'qmp_capabilities' }
 
+##
+# @QMPCapability:
+#
+# QMP supported capabilities to be broadcasted to the clients.
+#
+# @oob:   QMP ability to support Out-Of-Band requests.
+#
+# Since: 2.12
+#
+##
+{ 'enum': 'QMPCapability',
+  'data': [ 'oob' ] }
+
 ##
 # @VersionTriple:
 #
diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index c5a5c10b41..292c5f135a 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -17,6 +17,7 @@
 #include "qapi/qobject-input-visitor.h"
 #include "qapi/util.h"
 #include "qapi/visitor.h"
+#include "qapi/qmp/qstring.h"
 
 const char common_args[] = "-nodefaults -machine none";
 
@@ -75,6 +76,8 @@ static void test_qmp_protocol(void)
 {
     QDict *resp, *q, *ret;
     QList *capabilities;
+    const QListEntry *entry;
+    QString *qstr;
 
     global_qtest = qtest_init_without_qmp_handshake(common_args);
 
@@ -84,7 +87,12 @@ static void test_qmp_protocol(void)
     g_assert(q);
     test_version(qdict_get(q, "version"));
     capabilities = qdict_get_qlist(q, "capabilities");
-    g_assert(capabilities && qlist_empty(capabilities));
+    g_assert(capabilities);
+    entry = qlist_first(capabilities);
+    g_assert(entry);
+    qstr = qobject_to_qstring(entry->value);
+    g_assert(qstr);
+    g_assert_cmpstr(qstring_get_str(qstr), ==, "oob");
     QDECREF(resp);
 
     /* Test valid command before handshake */
-- 
2.14.3

[Qemu-devel] [RFC v5 12/26] qmp: negociate QMP capabilities

[Peter Xu]

After this patch, we will allow QMP clients to enable QMP capabilities
when sending the first "qmp_capabilities" command.  Originally we are
starting QMP session with no arguments like:

  { "execute": "qmp_capabilities" }

Now we can enable some QMP capabilities using (take OOB as example,
which is the only one capability that we support):

  { "execute": "qmp_capabilities",
    "argument": { "enable": [ "oob" ] } }

When the "argument" key is not provided, no capability is enabled.

For capability "oob", the monitor needs to be run on dedicated IO
thread, otherwise the command will fail.  For example, trying to enable
OOB on a MUXed typed QMP monitor will fail.

One thing to mention is that, QMP capabilities are per-monitor, and also
when the connection is closed due to some reason, the capabilities will
be reset.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c        | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 qapi-schema.json | 15 ++++++++++++---
 2 files changed, 68 insertions(+), 4 deletions(-)

diff --git a/monitor.c b/monitor.c
index bad6ee8dd1..097312e65f 100644
--- a/monitor.c
+++ b/monitor.c
@@ -167,6 +167,7 @@ typedef struct {
      * mode.
      */
     QmpCommandList *commands;
+    bool qmp_caps[QMP_CAPABILITY__MAX];
 } MonitorQMP;
 
 /*
@@ -1037,8 +1038,42 @@ static void monitor_init_qmp_commands(void)
                          qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
 }
 
-void qmp_qmp_capabilities(Error **errp)
+static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
+                           Error **errp)
+{
+    for (; list; list = list->next) {
+        assert(list->value < QMP_CAPABILITY__MAX);
+        switch (list->value) {
+        case QMP_CAPABILITY_OOB:
+            if (!mon->use_io_thr) {
+                /*
+                 * Out-Of-Band only works with monitors that are
+                 * running on dedicated IOThread.
+                 */
+                error_setg(errp, "This monitor does not support "
+                           "Out-Of-Band (OOB)");
+                return;
+            }
+            break;
+        default:
+            break;
+        }
+    }
+}
+
+/* This function should only be called after capabilities are checked. */
+static void qmp_caps_apply(Monitor *mon, QMPCapabilityList *list)
+{
+    for (; list; list = list->next) {
+        mon->qmp.qmp_caps[list->value] = true;
+    }
+}
+
+void qmp_qmp_capabilities(bool has_enable, QMPCapabilityList *enable,
+                          Error **errp)
 {
+    Error *local_err = NULL;
+
     if (cur_mon->qmp.commands == &qmp_commands) {
         error_set(errp, ERROR_CLASS_COMMAND_NOT_FOUND,
                   "Capabilities negotiation is already complete, command "
@@ -1046,6 +1081,20 @@ void qmp_qmp_capabilities(Error **errp)
         return;
     }
 
+    /* Enable QMP capabilities provided by the guest if applicable. */
+    if (has_enable) {
+        qmp_caps_check(cur_mon, enable, &local_err);
+        if (local_err) {
+            /*
+             * Failed check on either of the capabilities will fail
+             * the apply of all.
+             */
+            error_propagate(errp, local_err);
+            return;
+        }
+        qmp_caps_apply(cur_mon, enable);
+    }
+
     cur_mon->qmp.commands = &qmp_commands;
 }
 
@@ -3963,6 +4012,11 @@ static QObject *get_qmp_greeting(void)
     return QOBJECT(result);
 }
 
+static void monitor_qmp_caps_reset(Monitor *mon)
+{
+    memset(mon->qmp.qmp_caps, 0, sizeof(mon->qmp.qmp_caps));
+}
+
 static void monitor_qmp_event(void *opaque, int event)
 {
     QObject *data;
@@ -3971,6 +4025,7 @@ static void monitor_qmp_event(void *opaque, int event)
     switch (event) {
     case CHR_EVENT_OPENED:
         mon->qmp.commands = &qmp_cap_negotiation_commands;
+        monitor_qmp_caps_reset(mon);
         data = get_qmp_greeting();
         monitor_json_emitter(mon, data);
         qobject_decref(data);
diff --git a/qapi-schema.json b/qapi-schema.json
index 03201578b4..531fd4c0db 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -102,21 +102,30 @@
 #
 # Enable QMP capabilities.
 #
-# Arguments: None.
+# Arguments:
+#
+# @enable:    List of QMPCapabilities to enable, which is optional.
+#             If not provided, it means no QMP capabilities will be
+#             enabled.  (since 2.12)
 #
 # Example:
 #
-# -> { "execute": "qmp_capabilities" }
+# -> { "execute": "qmp_capabilities",
+#      "arguments": { "enable": [ "oob" ] } }
 # <- { "return": {} }
 #
 # Notes: This command is valid exactly when first connecting: it must be
 # issued before any other command will be accepted, and will fail once the
 # monitor is accepting other commands. (see qemu docs/interop/qmp-spec.txt)
 #
+# QMP client needs to explicitly enable QMP capabilities, otherwise
+# all the QMP capabilities will be turned off by default.
+#
 # Since: 0.13
 #
 ##
-{ 'command': 'qmp_capabilities' }
+{ 'command': 'qmp_capabilities',
+  'data': { '*enable': [ 'QMPCapability' ] } }
 
 ##
 # @QMPCapability:
-- 
2.14.3

[Qemu-devel] [RFC v5 13/26] qmp: introduce some capability helpers

[Peter Xu]

Introduce qmp_cap_enabled() and qmp_oob_enabled() helpers.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/monitor.c b/monitor.c
index 097312e65f..9666115259 100644
--- a/monitor.c
+++ b/monitor.c
@@ -1038,6 +1038,16 @@ static void monitor_init_qmp_commands(void)
                          qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
 }
 
+static bool qmp_cap_enabled(Monitor *mon, QMPCapability cap)
+{
+    return mon->qmp.qmp_caps[cap];
+}
+
+static bool qmp_oob_enabled(Monitor *mon)
+{
+    return qmp_cap_enabled(mon, QMP_CAPABILITY_OOB);
+}
+
 static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
                            Error **errp)
 {
-- 
2.14.3

[Qemu-devel] [RFC v5 14/26] monitor: introduce monitor_qmp_respond()

[Peter Xu]

A tiny refactoring, preparing to split the QMP dispatcher away.

Reviewed-by: Fam Zheng <famz@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 48 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 31 insertions(+), 17 deletions(-)

diff --git a/monitor.c b/monitor.c
index 9666115259..35d8925636 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3877,6 +3877,36 @@ static int monitor_can_read(void *opaque)
     return (mon->suspend_cnt == 0) ? 1 : 0;
 }
 
+/*
+ * When rsp/err/id is passed in, the function will be responsible for
+ * the cleanup.
+ */
+static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
+                                Error *err, QObject *id)
+{
+    QDict *qdict = NULL;
+
+    if (err) {
+        qdict = qdict_new();
+        qdict_put_obj(qdict, "error", qmp_build_error_object(err));
+        error_free(err);
+        rsp = QOBJECT(qdict);
+    }
+
+    if (rsp) {
+        if (id) {
+            /* This is for the qdict below. */
+            qobject_incref(id);
+            qdict_put_obj(qobject_to_qdict(rsp), "id", id);
+        }
+
+        monitor_json_emitter(mon, rsp);
+    }
+
+    qobject_decref(id);
+    qobject_decref(rsp);
+}
+
 static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
                                void *opaque)
 {
@@ -3927,24 +3957,8 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
     }
 
 err_out:
-    if (err) {
-        qdict = qdict_new();
-        qdict_put_obj(qdict, "error", qmp_build_error_object(err));
-        error_free(err);
-        rsp = QOBJECT(qdict);
-    }
+    monitor_qmp_respond(mon, rsp, err, id);
 
-    if (rsp) {
-        if (id) {
-            qdict_put_obj(qobject_to_qdict(rsp), "id", id);
-            id = NULL;
-        }
-
-        monitor_json_emitter(mon, rsp);
-    }
-
-    qobject_decref(id);
-    qobject_decref(rsp);
     qobject_decref(req);
 }
 
-- 
2.14.3

[Qemu-devel] [RFC v5 15/26] monitor: let suspend_cnt be thread safe

[Peter Xu]

Monitor code now can be run in more than one thread.  Let it be thread
safe when accessing suspend_cnt counter.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/monitor.c b/monitor.c
index 35d8925636..6ac1b2065d 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3874,7 +3874,7 @@ static int monitor_can_read(void *opaque)
 {
     Monitor *mon = opaque;
 
-    return (mon->suspend_cnt == 0) ? 1 : 0;
+    return (atomic_read(&mon->suspend_cnt) == 0) ? 1 : 0;
 }
 
 /*
@@ -4003,7 +4003,7 @@ int monitor_suspend(Monitor *mon)
 {
     if (!mon->rs)
         return -ENOTTY;
-    mon->suspend_cnt++;
+    atomic_inc(&mon->suspend_cnt);
     return 0;
 }
 
@@ -4011,8 +4011,9 @@ void monitor_resume(Monitor *mon)
 {
     if (!mon->rs)
         return;
-    if (--mon->suspend_cnt == 0)
+    if (atomic_dec_fetch(&mon->suspend_cnt) == 0) {
         readline_show_prompt(mon->rs);
+    }
 }
 
 static QObject *get_qmp_greeting(void)
@@ -4078,19 +4079,19 @@ static void monitor_event(void *opaque, int event)
             monitor_resume(mon);
             monitor_flush(mon);
         } else {
-            mon->suspend_cnt = 0;
+            atomic_set(&mon->suspend_cnt, 0);
         }
         break;
 
     case CHR_EVENT_MUX_OUT:
         if (mon->reset_seen) {
-            if (mon->suspend_cnt == 0) {
+            if (atomic_read(&mon->suspend_cnt) == 0) {
                 monitor_printf(mon, "\n");
             }
             monitor_flush(mon);
             monitor_suspend(mon);
         } else {
-            mon->suspend_cnt++;
+            atomic_inc(&mon->suspend_cnt);
         }
         qemu_mutex_lock(&mon->out_lock);
         mon->mux_out = 1;
-- 
2.14.3

[Qemu-devel] [RFC v5 16/26] monitor: separate QMP parser and dispatcher

[Peter Xu]

Originally QMP goes throw these steps:

  JSON Parser --> QMP Dispatcher --> Respond
      /|\    (2)                (3)     |
   (1) |                               \|/ (4)
       +---------  main thread  --------+

This patch does this:

  JSON Parser     QMP Dispatcher --> Respond
      /|\ |           /|\       (4)     |
       |  | (2)        | (3)            |  (5)
   (1) |  +----->      |               \|/
       +---------  main thread  <-------+

So the parsing job and the dispatching job is isolated now.  It gives us
a chance in following up patches to totally move the parser outside.

The isloation is done using one QEMUBH. Only one dispatcher QEMUBH is
used for all the monitors.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 179 ++++++++++++++++++++++++++++++++++++++++++++++++++++++--------
 1 file changed, 157 insertions(+), 22 deletions(-)

diff --git a/monitor.c b/monitor.c
index 6ac1b2065d..c20e659740 100644
--- a/monitor.c
+++ b/monitor.c
@@ -168,6 +168,10 @@ typedef struct {
      */
     QmpCommandList *commands;
     bool qmp_caps[QMP_CAPABILITY__MAX];
+    /* Protects qmp request/response queue. */
+    QemuMutex qmp_queue_lock;
+    /* Input queue that holds all the parsed QMP requests */
+    GQueue *qmp_requests;
 } MonitorQMP;
 
 /*
@@ -213,6 +217,8 @@ struct Monitor {
 
 struct MonitorGlobal {
     IOThread *mon_iothread;
+    /* Bottom half to dispatch the requests received from IO thread */
+    QEMUBH *qmp_dispatcher_bh;
 };
 
 static struct MonitorGlobal mon_global;
@@ -582,11 +588,13 @@ static void monitor_data_init(Monitor *mon, bool skip_flush,
 {
     memset(mon, 0, sizeof(Monitor));
     qemu_mutex_init(&mon->out_lock);
+    qemu_mutex_init(&mon->qmp.qmp_queue_lock);
     mon->outbuf = qstring_new();
     /* Use *mon_cmds by default. */
     mon->cmd_table = mon_cmds;
     mon->skip_flush = skip_flush;
     mon->use_io_thr = use_io_thr;
+    mon->qmp.qmp_requests = g_queue_new();
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -599,6 +607,8 @@ static void monitor_data_destroy(Monitor *mon)
     g_free(mon->rs);
     QDECREF(mon->outbuf);
     qemu_mutex_destroy(&mon->out_lock);
+    qemu_mutex_destroy(&mon->qmp.qmp_queue_lock);
+    g_queue_free(mon->qmp.qmp_requests);
 }
 
 char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
@@ -3907,29 +3917,31 @@ static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
     qobject_decref(rsp);
 }
 
-static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
-                               void *opaque)
+struct QMPRequest {
+    /* Owner of the request */
+    Monitor *mon;
+    /* "id" field of the request */
+    QObject *id;
+    /* Request object to be handled */
+    QObject *req;
+};
+typedef struct QMPRequest QMPRequest;
+
+/*
+ * Dispatch one single QMP request. The function will free the req_obj
+ * and objects inside it before return.
+ */
+static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
 {
-    QObject *req, *rsp = NULL, *id = NULL;
+    Monitor *mon, *old_mon;
+    QObject *req, *rsp = NULL, *id;
     QDict *qdict = NULL;
-    Monitor *mon = opaque, *old_mon;
-    Error *err = NULL;
 
-    req = json_parser_parse_err(tokens, NULL, &err);
-    if (!req && !err) {
-        /* json_parser_parse_err() sucks: can fail without setting @err */
-        error_setg(&err, QERR_JSON_PARSING);
-    }
-    if (err) {
-        goto err_out;
-    }
+    req = req_obj->req;
+    mon = req_obj->mon;
+    id = req_obj->id;
 
-    qdict = qobject_to_qdict(req);
-    if (qdict) {
-        id = qdict_get(qdict, "id");
-        qobject_incref(id);
-        qdict_del(qdict, "id");
-    } /* else will fail qmp_dispatch() */
+    g_free(req_obj);
 
     if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
         QString *req_json = qobject_to_json(req);
@@ -3940,7 +3952,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
     old_mon = cur_mon;
     cur_mon = mon;
 
-    rsp = qmp_dispatch(cur_mon->qmp.commands, req);
+    rsp = qmp_dispatch(mon->qmp.commands, req);
 
     cur_mon = old_mon;
 
@@ -3956,12 +3968,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
         }
     }
 
-err_out:
-    monitor_qmp_respond(mon, rsp, err, id);
+    /* Respond if necessary */
+    monitor_qmp_respond(mon, rsp, NULL, id);
+
+    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
+    if (!qmp_oob_enabled(mon)) {
+        monitor_resume(mon);
+    }
 
     qobject_decref(req);
 }
 
+/*
+ * Pop one QMP request from monitor queues, return NULL if not found.
+ * We are using round-robin fasion to pop the request, to avoid
+ * processing command only on a very busy monitor.  To achieve that,
+ * when we processed one request on specific monitor, we put that
+ * monitor to the end of mon_list queue.
+ */
+static QMPRequest *monitor_qmp_requests_pop_one(void)
+{
+    QMPRequest *req_obj = NULL;
+    Monitor *mon;
+
+    qemu_mutex_lock(&monitor_lock);
+
+    QTAILQ_FOREACH(mon, &mon_list, entry) {
+        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+        req_obj = g_queue_pop_head(mon->qmp.qmp_requests);
+        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+        if (req_obj) {
+            break;
+        }
+    }
+
+    if (req_obj) {
+        /*
+         * We found one request on the monitor. Degrade this monitor's
+         * priority to lowest by re-inserting it to end of queue.
+         */
+        QTAILQ_REMOVE(&mon_list, mon, entry);
+        QTAILQ_INSERT_TAIL(&mon_list, mon, entry);
+    }
+
+    qemu_mutex_unlock(&monitor_lock);
+
+    return req_obj;
+}
+
+static void monitor_qmp_bh_dispatcher(void *data)
+{
+    QMPRequest *req_obj;
+
+    while (true) {
+        req_obj = monitor_qmp_requests_pop_one();
+        if (!req_obj) {
+            break;
+        }
+        monitor_qmp_dispatch_one(req_obj);
+    }
+}
+
+static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
+                               void *opaque)
+{
+    QObject *req, *id = NULL;
+    QDict *qdict = NULL;
+    Monitor *mon = opaque;
+    Error *err = NULL;
+    QMPRequest *req_obj;
+
+    req = json_parser_parse_err(tokens, NULL, &err);
+    if (!req && !err) {
+        /* json_parser_parse_err() sucks: can fail without setting @err */
+        error_setg(&err, QERR_JSON_PARSING);
+    }
+    if (err) {
+        monitor_qmp_respond(mon, NULL, err, NULL);
+        qobject_decref(req);
+        return;
+    }
+
+    qdict = qobject_to_qdict(req);
+    if (qdict) {
+        id = qdict_get(qdict, "id");
+        qobject_incref(id);
+        qdict_del(qdict, "id");
+    } /* else will fail qmp_dispatch() */
+
+    req_obj = g_new0(QMPRequest, 1);
+    req_obj->mon = mon;
+    req_obj->id = id;
+    req_obj->req = req;
+
+    /*
+     * Put the request to the end of queue so that requests will be
+     * handled in time order.  Ownership for req_obj, req, id,
+     * etc. will be delivered to the handler side.
+     */
+    qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+    g_queue_push_tail(mon->qmp.qmp_requests, req_obj);
+    qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+
+    /* Kick the dispatcher routine */
+    qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
+
+    /*
+     * If OOB is not enabled on current monitor, we'll emulate the old
+     * behavior that we won't process current monitor any more until
+     * it is responded.  This helps make sure that as long as OOB is
+     * not enabled, the server will never drop any command.
+     */
+    if (!qmp_oob_enabled(mon)) {
+        monitor_suspend(mon);
+    }
+}
+
 static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
 {
     Monitor *mon = opaque;
@@ -4144,6 +4266,15 @@ static void monitor_iothread_init(void)
 {
     mon_global.mon_iothread = iothread_create("mon_iothread",
                                               &error_abort);
+
+    /*
+     * This MUST be on main loop thread since we have commands that
+     * have assumption to be run on main loop thread (Yeah, we'd
+     * better remove this assumption in the future).
+     */
+    mon_global.qmp_dispatcher_bh = aio_bh_new(qemu_get_aio_context(),
+                                              monitor_qmp_bh_dispatcher,
+                                              NULL);
 }
 
 void monitor_init_globals(void)
@@ -4267,6 +4398,10 @@ void monitor_cleanup(void)
     }
     qemu_mutex_unlock(&monitor_lock);
 
+    /* QEMUBHs needs to be deleted before destroying the IOThread. */
+    qemu_bh_delete(mon_global.qmp_dispatcher_bh);
+    mon_global.qmp_dispatcher_bh = NULL;
+
     iothread_destroy(mon_global.mon_iothread);
     mon_global.mon_iothread = NULL;
 }
-- 
2.14.3

[Qemu-devel] [RFC v5 17/26] qmp: add new event "request-dropped"

[Peter Xu]

This event will be emitted if one QMP request is dropped.  Along,
declare an enum for the reasons.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qapi-schema.json | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/qapi-schema.json b/qapi-schema.json
index 531fd4c0db..9d2625b6b3 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -3222,3 +3222,37 @@
 # Since: 2.11
 ##
 { 'command': 'watchdog-set-action', 'data' : {'action': 'WatchdogAction'} }
+
+##
+# @RequestDropReason:
+#
+# Reasons that caused one command to be dropped.
+#
+# @queue-full: the queue of request is full.
+#
+# Since: 2.12
+##
+{ 'enum': 'RequestDropReason',
+  'data': [ 'queue-full' ] }
+
+##
+# @REQUEST_DROPPED:
+#
+# Emitted when one QMP request is dropped due to some reason.
+# REQUEST_DROPPED is only emitted when the oob capability is enabled.
+#
+# @id: The dropped command's "id" field.
+#
+# @reason: The reason why the request is dropped.
+#
+# Since: 2.12
+#
+# Example:
+#
+# { "event": "REQUEST_DROPPED",
+#   "data": {"result": {"id": "libvirt-102",
+#                       "reason": "queue-full" } } }
+#
+##
+{ 'event': 'REQUEST_DROPPED' ,
+  'data': { 'id': 'any', 'reason': 'RequestDropReason' } }
-- 
2.14.3

[Qemu-devel] [RFC v5 18/26] monitor: send event when request queue full

[Peter Xu]

Set maximum QMP request queue length to 8.  If queue full, instead of
queue the command, we directly return a "request-dropped" event, telling
client that specific command is dropped.

Note that this flow control mechanism is only valid if OOB is enabled.
If it's not, the effective queue length will always be 1, which strictly
follows original behavior of QMP command handling (which never drop
messages).

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/monitor.c b/monitor.c
index c20e659740..f7923c4590 100644
--- a/monitor.c
+++ b/monitor.c
@@ -4029,6 +4029,8 @@ static void monitor_qmp_bh_dispatcher(void *data)
     }
 }
 
+#define  QMP_REQ_QUEUE_LEN_MAX  (8)
+
 static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
                                void *opaque)
 {
@@ -4061,6 +4063,19 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
     req_obj->id = id;
     req_obj->req = req;
 
+    if (qmp_oob_enabled(mon)) {
+        /* Drop the request if queue is full. */
+        if (mon->qmp.qmp_requests->length >= QMP_REQ_QUEUE_LEN_MAX) {
+            qapi_event_send_request_dropped(id,
+                                            REQUEST_DROP_REASON_QUEUE_FULL,
+                                            NULL);
+            qobject_decref(id);
+            qobject_decref(req);
+            g_free(req_obj);
+            return;
+        }
+    }
+
     /*
      * Put the request to the end of queue so that requests will be
      * handled in time order.  Ownership for req_obj, req, id,
-- 
2.14.3

[Qemu-devel] [RFC v5 19/26] qapi: introduce new cmd option "allow-oob"

[Peter Xu]

Here "oob" stands for "Out-Of-Band".  When "allow-oob" is set, it means
the command allows out-of-band execution.

The "oob" idea is proposed by Markus Armbruster in following thread:

  https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg02057.html

This new "allow-oob" boolean will be exposed by "query-qmp-schema" as
well for command entries, so that QMP clients can know which command can
be used as out-of-band calls. For example the command "migrate"
originally looks like:

  {"name": "migrate", "ret-type": "17", "meta-type": "command",
   "arg-type": "86"}

And it'll be changed into:

  {"name": "migrate", "ret-type": "17", "allow-oob": false,
   "meta-type": "command", "arg-type": "86"}

This patch only provides the QMP interface level changes.  It does not
contains the real out-of-band execution implementation yet.

Suggested-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/dispatch.h    |  1 +
 qapi/introspect.json           |  6 +++++-
 scripts/qapi-commands.py       | 19 ++++++++++++++-----
 scripts/qapi-introspect.py     | 10 ++++++++--
 scripts/qapi.py                | 15 ++++++++++-----
 scripts/qapi2texi.py           |  2 +-
 tests/qapi-schema/test-qapi.py |  2 +-
 7 files changed, 40 insertions(+), 15 deletions(-)

diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
index 20578dcd48..b76798800c 100644
--- a/include/qapi/qmp/dispatch.h
+++ b/include/qapi/qmp/dispatch.h
@@ -23,6 +23,7 @@ typedef enum QmpCommandOptions
 {
     QCO_NO_OPTIONS = 0x0,
     QCO_NO_SUCCESS_RESP = 0x1,
+    QCO_ALLOW_OOB = 0x2,
 } QmpCommandOptions;
 
 typedef struct QmpCommand
diff --git a/qapi/introspect.json b/qapi/introspect.json
index 5b3e6e9d78..57cc137627 100644
--- a/qapi/introspect.json
+++ b/qapi/introspect.json
@@ -259,12 +259,16 @@
 #
 # @ret-type: the name of the command's result type.
 #
+# @allow-oob: whether the command allows out-of-band execution.
+#             (Since: 2.11)
+#
 # TODO: @success-response (currently irrelevant, because it's QGA, not QMP)
 #
 # Since: 2.5
 ##
 { 'struct': 'SchemaInfoCommand',
-  'data': { 'arg-type': 'str', 'ret-type': 'str' } }
+  'data': { 'arg-type': 'str', 'ret-type': 'str',
+            'allow-oob': 'bool' } }
 
 ##
 # @SchemaInfoEvent:
diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 974d0a4a80..b2b0bc0510 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -192,10 +192,18 @@ out:
     return ret
 
 
-def gen_register_command(name, success_response):
-    options = 'QCO_NO_OPTIONS'
+def gen_register_command(name, success_response, allow_oob):
+    options = []
+
     if not success_response:
-        options = 'QCO_NO_SUCCESS_RESP'
+        options += ['QCO_NO_SUCCESS_RESP']
+    if allow_oob:
+        options += ['QCO_ALLOW_OOB']
+
+    if not options:
+        options = ['QCO_NO_OPTIONS']
+
+    options = " | ".join(options)
 
     ret = mcgen('''
     qmp_register_command(cmds, "%(name)s",
@@ -241,7 +249,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._visited_ret_types = None
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         if not gen:
             return
         self.decl += gen_command_decl(name, arg_type, boxed, ret_type)
@@ -250,7 +258,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
             self.defn += gen_marshal_output(ret_type)
         self.decl += gen_marshal_decl(name)
         self.defn += gen_marshal(name, arg_type, boxed, ret_type)
-        self._regy += gen_register_command(name, success_response)
+        self._regy += gen_register_command(name, success_response,
+                                           allow_oob)
 
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index 032bcea491..9fbf88b644 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -28,6 +28,11 @@ def to_json(obj, level=0):
                               to_json(obj[key], level + 1))
                 for key in sorted(obj.keys())]
         ret = '{' + ', '.join(elts) + '}'
+    elif isinstance(obj, bool):
+        if obj:
+            ret = 'true'
+        else:
+            ret = 'false'
     else:
         assert False                # not implemented
     if level == 1:
@@ -154,12 +159,13 @@ const char %(c_name)s[] = %(c_string)s;
                                     for m in variants.variants]})
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         arg_type = arg_type or self._schema.the_empty_object_type
         ret_type = ret_type or self._schema.the_empty_object_type
         self._gen_json(name, 'command',
                        {'arg-type': self._use_type(arg_type),
-                        'ret-type': self._use_type(ret_type)})
+                        'ret-type': self._use_type(ret_type),
+                        'allow-oob': allow_oob})
 
     def visit_event(self, name, info, arg_type, boxed):
         arg_type = arg_type or self._schema.the_empty_object_type
diff --git a/scripts/qapi.py b/scripts/qapi.py
index 62dc52ed6e..f411b8fc91 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -920,7 +920,8 @@ def check_exprs(exprs):
         elif 'command' in expr:
             meta = 'command'
             check_keys(expr_elem, 'command', [],
-                       ['data', 'returns', 'gen', 'success-response', 'boxed'])
+                       ['data', 'returns', 'gen', 'success-response',
+                        'boxed', 'allow-oob'])
         elif 'event' in expr:
             meta = 'event'
             check_keys(expr_elem, 'event', [], ['data', 'boxed'])
@@ -1031,7 +1032,7 @@ class QAPISchemaVisitor(object):
         pass
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         pass
 
     def visit_event(self, name, info, arg_type, boxed):
@@ -1398,7 +1399,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
 class QAPISchemaCommand(QAPISchemaEntity):
     def __init__(self, name, info, doc, arg_type, ret_type,
-                 gen, success_response, boxed):
+                 gen, success_response, boxed, allow_oob):
         QAPISchemaEntity.__init__(self, name, info, doc)
         assert not arg_type or isinstance(arg_type, str)
         assert not ret_type or isinstance(ret_type, str)
@@ -1409,6 +1410,7 @@ class QAPISchemaCommand(QAPISchemaEntity):
         self.gen = gen
         self.success_response = success_response
         self.boxed = boxed
+        self.allow_oob = allow_oob
 
     def check(self, schema):
         if self._arg_type_name:
@@ -1432,7 +1434,8 @@ class QAPISchemaCommand(QAPISchemaEntity):
     def visit(self, visitor):
         visitor.visit_command(self.name, self.info,
                               self.arg_type, self.ret_type,
-                              self.gen, self.success_response, self.boxed)
+                              self.gen, self.success_response,
+                              self.boxed, self.allow_oob)
 
 
 class QAPISchemaEvent(QAPISchemaEntity):
@@ -1640,6 +1643,7 @@ class QAPISchema(object):
         gen = expr.get('gen', True)
         success_response = expr.get('success-response', True)
         boxed = expr.get('boxed', False)
+        allow_oob = expr.get('allow-oob', False)
         if isinstance(data, OrderedDict):
             data = self._make_implicit_object_type(
                 name, info, doc, 'arg', self._make_members(data, info))
@@ -1647,7 +1651,8 @@ class QAPISchema(object):
             assert len(rets) == 1
             rets = self._make_array_type(rets[0], info)
         self._def_entity(QAPISchemaCommand(name, info, doc, data, rets,
-                                           gen, success_response, boxed))
+                                           gen, success_response,
+                                           boxed, allow_oob))
 
     def _def_event(self, expr, info, doc):
         name = expr['event']
diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
index a317526e51..0ac0df517a 100755
--- a/scripts/qapi2texi.py
+++ b/scripts/qapi2texi.py
@@ -236,7 +236,7 @@ class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
                              body=texi_entity(doc, 'Members'))
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         doc = self.cur_doc
         if self.out:
             self.out += '\n'
diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index c7724d3437..6749e9e397 100644
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -36,7 +36,7 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
         self._print_variants(variants)
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         print 'command %s %s -> %s' % \
             (name, arg_type and arg_type.name, ret_type and ret_type.name)
         print '   gen=%s success_response=%s boxed=%s' % \
-- 
2.14.3

[Qemu-devel] [RFC v5 20/26] qmp: support out-of-band (oob) execution

[Peter Xu]

Having "allow-oob" to true for a command does not mean that this command
will always be run in out-of-band mode.  The out-of-band quick path will
only be executed if we specify the extra "run-oob" flag when sending the
QMP request:

    { "execute":   "command-that-allows-oob",
      "arguments": { ... },
      "control":   { "run-oob": true } }

The "control" key is introduced to store this extra flag.  "control"
field is used to store arguments that are shared by all the commands,
rather than command specific arguments.  Let "run-oob" be the first.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/dispatch.h |  1 +
 monitor.c                   | 24 ++++++++++++++++++++++++
 qapi/qmp-dispatch.c         | 39 +++++++++++++++++++++++++++++++++++++++
 trace-events                |  2 ++
 4 files changed, 66 insertions(+)

diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
index b76798800c..ee2b8ce576 100644
--- a/include/qapi/qmp/dispatch.h
+++ b/include/qapi/qmp/dispatch.h
@@ -49,6 +49,7 @@ bool qmp_command_is_enabled(const QmpCommand *cmd);
 const char *qmp_command_name(const QmpCommand *cmd);
 bool qmp_has_success_response(const QmpCommand *cmd);
 QObject *qmp_build_error_object(Error *err);
+bool qmp_is_oob(const QObject *request);
 
 typedef void (*qmp_cmd_callback_fn)(QmpCommand *cmd, void *opaque);
 
diff --git a/monitor.c b/monitor.c
index f7923c4590..aa3049bbca 100644
--- a/monitor.c
+++ b/monitor.c
@@ -4025,6 +4025,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
         if (!req_obj) {
             break;
         }
+        trace_monitor_qmp_cmd_in_band(qobject_get_try_str(req_obj->id));
         monitor_qmp_dispatch_one(req_obj);
     }
 }
@@ -4051,9 +4052,25 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
         return;
     }
 
+    if (!qmp_oob_enabled(mon) && qmp_is_oob(req)) {
+        error_setg(&err, "Out-Of-Band is only allowed "
+                   "when OOB is enabled.");
+        monitor_qmp_respond(mon, NULL, err, NULL);
+        qobject_decref(req);
+        return;
+    }
+
     qdict = qobject_to_qdict(req);
     if (qdict) {
         id = qdict_get(qdict, "id");
+        /* When OOB is enabled, the "id" field is mandatory. */
+        if (qmp_oob_enabled(mon) && !id) {
+            error_setg(&err, "Out-Of-Band capability requires that "
+                       "every command contains an 'id' field.");
+            monitor_qmp_respond(mon, NULL, err, NULL);
+            qobject_decref(req);
+            return;
+        }
         qobject_incref(id);
         qdict_del(qdict, "id");
     } /* else will fail qmp_dispatch() */
@@ -4064,6 +4081,13 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
     req_obj->req = req;
 
     if (qmp_oob_enabled(mon)) {
+        if (qmp_is_oob(req)) {
+            /* Out-Of-Band (OOB) requests are executed directly in parser. */
+            trace_monitor_qmp_cmd_out_of_band(qobject_get_try_str(req_obj->id));
+            monitor_qmp_dispatch_one(req_obj);
+            return;
+        }
+
         /* Drop the request if queue is full. */
         if (mon->qmp.qmp_requests->length >= QMP_REQ_QUEUE_LEN_MAX) {
             qapi_event_send_request_dropped(id,
diff --git a/qapi/qmp-dispatch.c b/qapi/qmp-dispatch.c
index b41fa174fe..ed7e5d5a75 100644
--- a/qapi/qmp-dispatch.c
+++ b/qapi/qmp-dispatch.c
@@ -52,6 +52,12 @@ static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
                            "QMP input member 'arguments' must be an object");
                 return NULL;
             }
+        } else if (!strcmp(arg_name, "control")) {
+            if (qobject_type(arg_obj) != QTYPE_QDICT) {
+                error_setg(errp,
+                           "QMP input member 'control' must be a dict");
+                return NULL;
+            }
         } else {
             error_setg(errp, "QMP input member '%s' is unexpected",
                        arg_name);
@@ -94,6 +100,11 @@ static QObject *do_qmp_dispatch(QmpCommandList *cmds, QObject *request,
         return NULL;
     }
 
+    if (qmp_is_oob(request) && !(cmd->options & QCO_ALLOW_OOB)) {
+        error_setg(errp, "The command %s does not support OOB", command);
+        return NULL;
+    }
+
     if (!qdict_haskey(dict, "arguments")) {
         args = qdict_new();
     } else {
@@ -122,6 +133,34 @@ QObject *qmp_build_error_object(Error *err)
                               error_get_pretty(err));
 }
 
+/*
+ * Detect whether a request should be run out-of-band, by quickly
+ * peeking at whether we have: { "control": { "run-oob": True } }. By
+ * default commands are run in-band.
+ */
+bool qmp_is_oob(const QObject *request)
+{
+    QDict *dict;
+    QBool *bool_obj;
+
+    dict = qobject_to_qdict(request);
+    if (!dict) {
+        return false;
+    }
+
+    dict = qdict_get_qdict(dict, "control");
+    if (!dict) {
+        return false;
+    }
+
+    bool_obj = qobject_to_qbool(qdict_get(dict, "run-oob"));
+    if (!bool_obj) {
+        return false;
+    }
+
+    return qbool_get_bool(bool_obj);
+}
+
 QObject *qmp_dispatch(QmpCommandList *cmds, QObject *request)
 {
     Error *err = NULL;
diff --git a/trace-events b/trace-events
index 1d2eb5d3e4..3148e590c9 100644
--- a/trace-events
+++ b/trace-events
@@ -47,6 +47,8 @@ monitor_protocol_event_emit(uint32_t event, void *data) "event=%d data=%p"
 monitor_protocol_event_queue(uint32_t event, void *qdict, uint64_t rate) "event=%d data=%p rate=%" PRId64
 handle_hmp_command(void *mon, const char *cmdline) "mon %p cmdline: %s"
 handle_qmp_command(void *mon, const char *req) "mon %p req: %s"
+monitor_qmp_cmd_in_band(const char *id) "%s"
+monitor_qmp_cmd_out_of_band(const char *id) "%s"
 
 # dma-helpers.c
 dma_blk_io(void *dbs, void *bs, int64_t offset, bool to_dev) "dbs=%p bs=%p offset=%" PRId64 " to_dev=%d"
-- 
2.14.3

[Qemu-devel] [RFC v5 21/26] qmp: isolate responses into io thread

[Peter Xu]

For those monitors who has enabled IO thread, we'll offload the
responding procedure into IO thread.  The main reason is that chardev is
not thread safe, and we need to do all the read/write IOs in the same
thread.  For use_io_thr=true monitors, that thread is the IO thread.

We do this isolation in similar pattern as what we have done to the
request queue: we first create one response queue for each monitor, then
instead of reply directly in main thread, we queue the responses and
kick the IO thread to do the rest of the job for us.

A funny thing after doing this is that, when the QMP clients send "quit"
to QEMU, it's possible that we close the IOThread even earlier than
replying to that "quit".  So another thing we need to do before cleaning
up the monitors is that we need to flush the response queue (we don't
need to do that for command queue; after all we are quitting) to make
sure replies for handled commands are always flushed back to clients.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 96 insertions(+), 1 deletion(-)

diff --git a/monitor.c b/monitor.c
index aa3049bbca..298703d90f 100644
--- a/monitor.c
+++ b/monitor.c
@@ -172,6 +172,8 @@ typedef struct {
     QemuMutex qmp_queue_lock;
     /* Input queue that holds all the parsed QMP requests */
     GQueue *qmp_requests;
+    /* Output queue contains all the QMP responses in order */
+    GQueue *qmp_responses;
 } MonitorQMP;
 
 /*
@@ -219,6 +221,8 @@ struct MonitorGlobal {
     IOThread *mon_iothread;
     /* Bottom half to dispatch the requests received from IO thread */
     QEMUBH *qmp_dispatcher_bh;
+    /* Bottom half to deliver the responses back to clients */
+    QEMUBH *qmp_respond_bh;
 };
 
 static struct MonitorGlobal mon_global;
@@ -395,7 +399,8 @@ int monitor_fprintf(FILE *stream, const char *fmt, ...)
     return 0;
 }
 
-static void monitor_json_emitter(Monitor *mon, const QObject *data)
+static void monitor_json_emitter_raw(Monitor *mon,
+                                     QObject *data)
 {
     QString *json;
 
@@ -409,6 +414,71 @@ static void monitor_json_emitter(Monitor *mon, const QObject *data)
     QDECREF(json);
 }
 
+static void monitor_json_emitter(Monitor *mon, QObject *data)
+{
+    if (mon->use_io_thr) {
+        /*
+         * If using IO thread, we need to queue the item so that IO
+         * thread will do the rest for us.  Take refcount so that
+         * caller won't free the data (which will be finally freed in
+         * responder thread).
+         */
+        qobject_incref(data);
+        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+        g_queue_push_tail(mon->qmp.qmp_responses, (void *)data);
+        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+        qemu_bh_schedule(mon_global.qmp_respond_bh);
+    } else {
+        /*
+         * If not using monitor IO thread, then we are in main thread.
+         * Do the emission right away.
+         */
+        monitor_json_emitter_raw(mon, data);
+    }
+}
+
+struct QMPResponse {
+    Monitor *mon;
+    QObject *data;
+};
+typedef struct QMPResponse QMPResponse;
+
+/*
+ * Return one QMPResponse.  The response is only valid if
+ * response.data is not NULL.
+ */
+static QMPResponse monitor_qmp_response_pop_one(void)
+{
+    Monitor *mon;
+    QObject *data = NULL;
+
+    qemu_mutex_lock(&monitor_lock);
+    QTAILQ_FOREACH(mon, &mon_list, entry) {
+        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
+        data = g_queue_pop_head(mon->qmp.qmp_responses);
+        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
+        if (data) {
+            break;
+        }
+    }
+    qemu_mutex_unlock(&monitor_lock);
+    return (QMPResponse) { .mon = mon, .data = data };
+}
+
+static void monitor_qmp_bh_responder(void *opaque)
+{
+    QMPResponse response;
+
+    while (true) {
+        response = monitor_qmp_response_pop_one();
+        if (!response.data) {
+            break;
+        }
+        monitor_json_emitter_raw(response.mon, response.data);
+        qobject_decref(response.data);
+    }
+}
+
 static MonitorQAPIEventConf monitor_qapi_event_conf[QAPI_EVENT__MAX] = {
     /* Limit guest-triggerable events to 1 per second */
     [QAPI_EVENT_RTC_CHANGE]        = { 1000 * SCALE_MS },
@@ -595,6 +665,7 @@ static void monitor_data_init(Monitor *mon, bool skip_flush,
     mon->skip_flush = skip_flush;
     mon->use_io_thr = use_io_thr;
     mon->qmp.qmp_requests = g_queue_new();
+    mon->qmp.qmp_responses = g_queue_new();
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -609,6 +680,7 @@ static void monitor_data_destroy(Monitor *mon)
     qemu_mutex_destroy(&mon->out_lock);
     qemu_mutex_destroy(&mon->qmp.qmp_queue_lock);
     g_queue_free(mon->qmp.qmp_requests);
+    g_queue_free(mon->qmp.qmp_responses);
 }
 
 char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
@@ -4301,6 +4373,11 @@ static GMainContext *monitor_io_context_get(void)
     return iothread_get_g_main_context(mon_global.mon_iothread);
 }
 
+static AioContext *monitor_aio_context_get(void)
+{
+    return iothread_get_aio_context(mon_global.mon_iothread);
+}
+
 static void monitor_iothread_init(void)
 {
     mon_global.mon_iothread = iothread_create("mon_iothread",
@@ -4314,6 +4391,15 @@ static void monitor_iothread_init(void)
     mon_global.qmp_dispatcher_bh = aio_bh_new(qemu_get_aio_context(),
                                               monitor_qmp_bh_dispatcher,
                                               NULL);
+
+    /*
+     * Unlike the dispatcher BH, this must be run on the monitor IO
+     * thread, so that monitors that are using IO thread will make
+     * sure read/write operations are all done on the IO thread.
+     */
+    mon_global.qmp_respond_bh = aio_bh_new(monitor_aio_context_get(),
+                                           monitor_qmp_bh_responder,
+                                           NULL);
 }
 
 void monitor_init_globals(void)
@@ -4429,6 +4515,13 @@ void monitor_cleanup(void)
      */
     iothread_stop(mon_global.mon_iothread);
 
+    /*
+     * After we have IOThread to send responses, it's possible that
+     * when we stop the IOThread there are still replies queued in the
+     * responder queue.  Flush all of them.
+     */
+    monitor_qmp_bh_responder(NULL);
+
     qemu_mutex_lock(&monitor_lock);
     QTAILQ_FOREACH_SAFE(mon, &mon_list, entry, next) {
         QTAILQ_REMOVE(&mon_list, mon, entry);
@@ -4440,6 +4533,8 @@ void monitor_cleanup(void)
     /* QEMUBHs needs to be deleted before destroying the IOThread. */
     qemu_bh_delete(mon_global.qmp_dispatcher_bh);
     mon_global.qmp_dispatcher_bh = NULL;
+    qemu_bh_delete(mon_global.qmp_respond_bh);
+    mon_global.qmp_respond_bh = NULL;
 
     iothread_destroy(mon_global.mon_iothread);
     mon_global.mon_iothread = NULL;
-- 
2.14.3

[Qemu-devel] [RFC v5 22/26] monitor: enable IO thread for (qmp & !mux) typed

[Peter Xu]

Start to use dedicate IO thread for QMP monitors that are not using
MUXed chardev.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/monitor.c b/monitor.c
index 298703d90f..e864b17ed9 100644
--- a/monitor.c
+++ b/monitor.c
@@ -36,6 +36,7 @@
 #include "net/net.h"
 #include "net/slirp.h"
 #include "chardev/char-fe.h"
+#include "chardev/char-mux.h"
 #include "ui/qemu-spice.h"
 #include "sysemu/numa.h"
 #include "monitor/monitor.h"
@@ -4456,7 +4457,7 @@ void monitor_init(Chardev *chr, int flags)
     Monitor *mon = g_malloc(sizeof(*mon));
     GMainContext *context;
 
-    monitor_data_init(mon, false, false);
+    monitor_data_init(mon, false, !CHARDEV_IS_MUX(chr));
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
-- 
2.14.3

[Qemu-devel] [RFC v5 23/26] qmp: add command "x-oob-test"

[Peter Xu]

This command is only used to test OOB functionality.  It should not be
used for any other purposes.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qapi-schema.json | 18 ++++++++++++++++++
 qmp.c            | 16 ++++++++++++++++
 2 files changed, 34 insertions(+)

diff --git a/qapi-schema.json b/qapi-schema.json
index 9d2625b6b3..5bd59bd266 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -3256,3 +3256,21 @@
 ##
 { 'event': 'REQUEST_DROPPED' ,
   'data': { 'id': 'any', 'reason': 'RequestDropReason' } }
+
+##
+# @x-oob-test:
+#
+# Test OOB functionality.  When send this command with lock=true,
+# it'll try to hang the dispatcher.  When send it with lock=false,
+# it'll try to notify the locked thread to continue.  Note: it should
+# only be used by QMP test program rather than anything else.
+#
+# Since: 2.12
+#
+# Example:
+#
+# { "execute": "x-oob-test",
+#   "arguments": { "lock": true } }
+##
+{ 'command': 'x-oob-test', 'data' : { 'lock': 'bool' },
+  'allow-oob': true }
diff --git a/qmp.c b/qmp.c
index e8c303116a..ca2968aad2 100644
--- a/qmp.c
+++ b/qmp.c
@@ -722,3 +722,19 @@ MemoryInfo *qmp_query_memory_size_summary(Error **errp)
 
     return mem_info;
 }
+
+static QemuSemaphore x_oob_test_sem;
+
+static void __attribute__((constructor)) x_oob_test_init(void)
+{
+    qemu_sem_init(&x_oob_test_sem, 0);
+}
+
+void qmp_x_oob_test(bool lock, Error **errp)
+{
+    if (lock) {
+        qemu_sem_wait(&x_oob_test_sem);
+    } else {
+        qemu_sem_post(&x_oob_test_sem);
+    }
+}
-- 
2.14.3

[Qemu-devel] [RFC v5 24/26] docs: update QMP documents for OOB commands

[Peter Xu]

Update both the developer and spec for the new QMP OOB (Out-Of-Band)
command.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 docs/devel/qapi-code-gen.txt | 51 +++++++++++++++++++++++++++++++++++++++-----
 docs/interop/qmp-spec.txt    | 49 ++++++++++++++++++++++++++++++++++++------
 2 files changed, 89 insertions(+), 11 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index f04c63fe82..8597fdb087 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -556,7 +556,8 @@ following example objects:
 
 Usage: { 'command': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT,
          '*returns': TYPE-NAME, '*boxed': true,
-         '*gen': false, '*success-response': false }
+         '*gen': false, '*success-response': false,
+         '*allow-oob': false }
 
 Commands are defined by using a dictionary containing several members,
 where three members are most common.  The 'command' member is a
@@ -636,6 +637,44 @@ possible, the command expression should include the optional key
 'success-response' with boolean value false.  So far, only QGA makes
 use of this member.
 
+Most of the QMP commands are handled sequentially in such a order:
+Firstly, the JSON Parser parses the command request into some internal
+message, delivers the message to QMP dispatchers. Secondly, the QMP
+dispatchers will handle the commands one by one in time order, respond
+when necessary.  For some commands that always complete "quickly" can
+instead be executed directly during parsing, at the QMP client's
+request.  This kind of commands that allow direct execution is called
+"out-of-band" ("oob" as shortcut) commands. The response can overtake
+prior in-band commands' responses.  By default, commands are always
+in-band.  We need to explicitly specify "allow-oob" to "True" to show
+that one command can be run out-of-band.
+
+One thing to mention for developers is that, although out-of-band
+execution of commands benefit from quick and asynchronous execution,
+it need to satisfy at least the following:
+
+(1) It is extremely quick and never blocks, so that its execution will
+    not block parsing routine of any other monitors.
+
+(2) It does not need BQL, since the parser can be run without BQL,
+    while the dispatcher is always with BQL held.
+
+If not, the command is not suitable to be allowed to run out-of-band,
+and it should set its "allow-oob" to "False".  Whether a command is
+allowed to run out-of-band can also be introspected using
+query-qmp-schema command.  Please see the section "Client JSON
+Protocol introspection" for more information.
+
+To execute a command in out-of-band way, we need to specify the
+"control" field in the request, with "run-oob" set to true. Example:
+
+ => { "execute": "command-support-oob",
+      "arguments": { ... },
+      "control": { "run-oob": true } }
+ <= { "return": { } }
+
+Without it, even the commands that supports out-of-band execution will
+still be run in-band.
 
 === Events ===
 
@@ -739,10 +778,12 @@ references by name.
 QAPI schema definitions not reachable that way are omitted.
 
 The SchemaInfo for a command has meta-type "command", and variant
-members "arg-type" and "ret-type".  On the wire, the "arguments"
-member of a client's "execute" command must conform to the object type
-named by "arg-type".  The "return" member that the server passes in a
-success response conforms to the type named by "ret-type".
+members "arg-type", "ret-type" and "allow-oob".  On the wire, the
+"arguments" member of a client's "execute" command must conform to the
+object type named by "arg-type".  The "return" member that the server
+passes in a success response conforms to the type named by
+"ret-type".  When "allow-oob" is set, it means the command supports
+out-of-band execution.
 
 If the command takes no arguments, "arg-type" names an object type
 without members.  Likewise, if the command returns nothing, "ret-type"
diff --git a/docs/interop/qmp-spec.txt b/docs/interop/qmp-spec.txt
index f8b5356015..01a2df0c79 100644
--- a/docs/interop/qmp-spec.txt
+++ b/docs/interop/qmp-spec.txt
@@ -78,21 +78,44 @@ The greeting message format is:
 - The "capabilities" member specify the availability of features beyond the
   baseline specification; the order of elements in this array has no
   particular significance, so a client must search the entire array
-  when looking for a particular capability
+  when looking for a particular capability.
 
 2.2.1 Capabilities
 ------------------
 
-As of the date this document was last revised, no server or client
-capability strings have been defined.
-
+Currently supported capabilities are:
+
+- "oob": it means the QMP server supports "Out-Of-Band" command
+  execution.  For more detail, please see "run-oob" parameter in
+  "Issuing Commands" section below.  Not all commands allow this "oob"
+  execution.  One can know whether one command supports "oob" by
+  "query-qmp-schema" command.
+
+  NOTE: Converting an existing QMP command to be OOB-capable can be
+  either very easy or extremely hard.  The most important thing is
+  that OOB-capable command should never be blocked for a long time.
+  Some bad examples: (1) doing IOs, especially when the backend can be
+  an NFS storage; or (2) accessing guest memories, which can be simply
+  blocked for a very long time when it triggers a page fault, which
+  may not be handled immediately.  It's still legal to take a mutex in
+  an OOB-capable command handler, however, we need to make sure that
+  all the other code paths that are protected by the same lock won't
+  be blocked very long as well, otherwise the OOB handler might be
+  blocked too when it tries to take the lock.
+
+QMP clients can get a list of supported QMP capabilities of the QMP
+server in the greeting message mentioned above.  By default, all the
+capabilities are off.  To enable a specific or multiple of QMP
+capabilities, QMP client needs to send "qmp_capabilities" command with
+extra parameter for the capabilities.
 
 2.3 Issuing Commands
 --------------------
 
 The format for command execution is:
 
-{ "execute": json-string, "arguments": json-object, "id": json-value }
+{ "execute": json-string, "arguments": json-object, "id": json-value,
+  "control": json-dict }
 
  Where,
 
@@ -102,10 +125,19 @@ The format for command execution is:
   required. Each command documents what contents will be considered
   valid when handling the json-argument
 - The "id" member is a transaction identification associated with the
-  command execution, it is optional and will be part of the response if
+  command execution.  It is required if OOB is enabled, and optional
+  if not.  The same "id" field will be part of the response if
   provided. The "id" member can be any json-value, although most
   clients merely use a json-number incremented for each successive
   command
+- The "control" member is optionally, and currently only used for
+  "out-of-band" execution.  For some commands that always complete
+  "quickly" can be executed directly during parsing at the QMP
+  client's request.  This kind of commands that allow direct execution
+  is called "out-of-band" ("oob" as shortcut) commands. The response
+  of "oob" commands can overtake prior in-band commands' responses.
+  To enable "oob" feature, just provide a control field with:
+  { "control": { "run-oob": true } }
 
 2.4 Commands Responses
 ----------------------
@@ -113,6 +145,11 @@ The format for command execution is:
 There are two possible responses which the Server will issue as the result
 of a command execution: success or error.
 
+As long as the commands were issued with a proper "id" field, then the
+same "id" field will be attached in the corresponding response message
+so that requests and responses can match.  Clients should drop all the
+responses that are with unknown "id" field.
+
 2.4.1 success
 -------------
 
-- 
2.14.3

[Qemu-devel] [RFC v5 25/26] tests: qmp-test: verify command batching

[Peter Xu]

OOB introduced DROP event for flow control.  This should not affect old
QMP clients.  Add a command batching check to make sure of it.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 tests/qmp-test.c | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index 292c5f135a..729ec59b0a 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -78,6 +78,7 @@ static void test_qmp_protocol(void)
     QList *capabilities;
     const QListEntry *entry;
     QString *qstr;
+    int i;
 
     global_qtest = qtest_init_without_qmp_handshake(common_args);
 
@@ -135,6 +136,24 @@ static void test_qmp_protocol(void)
     g_assert_cmpint(qdict_get_int(resp, "id"), ==, 2);
     QDECREF(resp);
 
+    /*
+     * Test command batching.  In current test OOB is not enabled, we
+     * should be able to run as many commands in batch as we like.
+     * Using 16 (>8, which is OOB queue length) to make sure OOB
+     * won't break existing clients.
+     */
+    for (i = 0; i < 16; i++) {
+        qmp_async("{ 'execute': 'query-version' }");
+    }
+    /* Verify the replies to make sure no command is dropped. */
+    for (i = 0; i < 16; i++) {
+        resp = qmp_receive();
+        /* It should never be dropped.  Each of them should be a reply. */
+        g_assert(qdict_haskey(resp, "return"));
+        g_assert(!qdict_haskey(resp, "event"));
+        QDECREF(resp);
+    }
+
     qtest_end();
 }
 
-- 
2.14.3

[Qemu-devel] [RFC v5 26/26] tests: qmp-test: add oob test

[Peter Xu]

Test the new OOB capability.  Here we used the new "x-oob-test" command.
Firstly, we send a lock=true and oob=false command to hang the main
thread.  Then send another lock=false and oob=true command (which will
be run inside parser this time) to free that hanged command.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 tests/qmp-test.c | 61 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 61 insertions(+)

diff --git a/tests/qmp-test.c b/tests/qmp-test.c
index 729ec59b0a..bb12122a4e 100644
--- a/tests/qmp-test.c
+++ b/tests/qmp-test.c
@@ -157,6 +157,66 @@ static void test_qmp_protocol(void)
     qtest_end();
 }
 
+/* Tests for Out-Of-Band support. */
+static void test_qmp_oob(void)
+{
+    QDict *resp;
+    int acks = 0;
+    const char *cmd_id;
+
+    global_qtest = qtest_init_without_qmp_handshake(common_args);
+
+    /* Ignore the greeting message. */
+    resp = qmp_receive();
+    g_assert(qdict_get_qdict(resp, "QMP"));
+    QDECREF(resp);
+
+    /* Try a fake capability, it should fail. */
+    resp = qmp("{ 'execute': 'qmp_capabilities', "
+               "  'arguments': { 'enable': [ 'cap-does-not-exist' ] } }");
+    g_assert(qdict_haskey(resp, "error"));
+
+    /* Now, enable OOB in current QMP session, it should success. */
+    resp = qmp("{ 'execute': 'qmp_capabilities', "
+               "  'arguments': { 'enable': [ 'oob' ] } }");
+    g_assert(qdict_haskey(resp, "return"));
+
+    /*
+     * Try any command that does not support OOB but with OOB flag. We
+     * should get failure.
+     */
+    resp = qmp("{ 'execute': 'query-cpus',"
+               "  'control': { 'run-oob': true } }");
+    g_assert(qdict_haskey(resp, "error"));
+
+    /*
+     * Firstly send the "x-oob-test" command with lock=true and
+     * oob=false, it should hang the dispatcher and main thread;
+     * later, we send another lock=false with oob=true to continue
+     * that thread processing.  Finally we should receive replies from
+     * both commands.
+     */
+    qmp_async("{ 'execute': 'x-oob-test',"
+              "  'arguments': { 'lock': true }, "
+              "  'id': 'lock-cmd'}");
+    qmp_async("{ 'execute': 'x-oob-test', "
+              "  'arguments': { 'lock': false }, "
+              "  'control': { 'run-oob': true }, "
+              "  'id': 'unlock-cmd' }");
+
+    /* Ignore all events.  Wait for 2 acks */
+    while (acks < 2) {
+        resp = qmp_receive();
+        cmd_id = qdict_get_str(resp, "id");
+        if (!g_strcmp0(cmd_id, "lock-cmd") ||
+            !g_strcmp0(cmd_id, "unlock-cmd")) {
+            acks++;
+        }
+    }
+
+    qtest_end();
+}
+
 static int query_error_class(const char *cmd)
 {
     static struct {
@@ -335,6 +395,7 @@ int main(int argc, char *argv[])
     g_test_init(&argc, &argv, NULL);
 
     qtest_add_func("qmp/protocol", test_qmp_protocol);
+    qtest_add_func("qmp/oob", test_qmp_oob);
     qmp_schema_init(&schema);
     add_query_tests(&schema);
 
-- 
2.14.3

Re: [Qemu-devel] [RFC v5 12/26] qmp: negociate QMP capabilities

[Dr. David Alan Gilbert]

* Peter Xu (peterx@redhat.com) wrote:
> After this patch, we will allow QMP clients to enable QMP capabilities
> when sending the first "qmp_capabilities" command.  Originally we are
> starting QMP session with no arguments like:
> 
>   { "execute": "qmp_capabilities" }
> 
> Now we can enable some QMP capabilities using (take OOB as example,
> which is the only one capability that we support):
> 
>   { "execute": "qmp_capabilities",
>     "argument": { "enable": [ "oob" ] } }
> 
> When the "argument" key is not provided, no capability is enabled.
> 
> For capability "oob", the monitor needs to be run on dedicated IO
> thread, otherwise the command will fail.  For example, trying to enable
> OOB on a MUXed typed QMP monitor will fail.
> 
> One thing to mention is that, QMP capabilities are per-monitor, and also
> when the connection is closed due to some reason, the capabilities will
> be reset.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>

Typo in the title nego*t*iate, other than that.


Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

> ---
>  monitor.c        | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>  qapi-schema.json | 15 ++++++++++++---
>  2 files changed, 68 insertions(+), 4 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index bad6ee8dd1..097312e65f 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -167,6 +167,7 @@ typedef struct {
>       * mode.
>       */
>      QmpCommandList *commands;
> +    bool qmp_caps[QMP_CAPABILITY__MAX];
>  } MonitorQMP;
>  
>  /*
> @@ -1037,8 +1038,42 @@ static void monitor_init_qmp_commands(void)
>                           qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
>  }
>  
> -void qmp_qmp_capabilities(Error **errp)
> +static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
> +                           Error **errp)
> +{
> +    for (; list; list = list->next) {
> +        assert(list->value < QMP_CAPABILITY__MAX);
> +        switch (list->value) {
> +        case QMP_CAPABILITY_OOB:
> +            if (!mon->use_io_thr) {
> +                /*
> +                 * Out-Of-Band only works with monitors that are
> +                 * running on dedicated IOThread.
> +                 */
> +                error_setg(errp, "This monitor does not support "
> +                           "Out-Of-Band (OOB)");
> +                return;
> +            }
> +            break;
> +        default:
> +            break;
> +        }
> +    }
> +}
> +
> +/* This function should only be called after capabilities are checked. */
> +static void qmp_caps_apply(Monitor *mon, QMPCapabilityList *list)
> +{
> +    for (; list; list = list->next) {
> +        mon->qmp.qmp_caps[list->value] = true;
> +    }
> +}
> +
> +void qmp_qmp_capabilities(bool has_enable, QMPCapabilityList *enable,
> +                          Error **errp)
>  {
> +    Error *local_err = NULL;
> +
>      if (cur_mon->qmp.commands == &qmp_commands) {
>          error_set(errp, ERROR_CLASS_COMMAND_NOT_FOUND,
>                    "Capabilities negotiation is already complete, command "
> @@ -1046,6 +1081,20 @@ void qmp_qmp_capabilities(Error **errp)
>          return;
>      }
>  
> +    /* Enable QMP capabilities provided by the guest if applicable. */
> +    if (has_enable) {
> +        qmp_caps_check(cur_mon, enable, &local_err);
> +        if (local_err) {
> +            /*
> +             * Failed check on either of the capabilities will fail
> +             * the apply of all.
> +             */
> +            error_propagate(errp, local_err);
> +            return;
> +        }
> +        qmp_caps_apply(cur_mon, enable);
> +    }
> +
>      cur_mon->qmp.commands = &qmp_commands;
>  }
>  
> @@ -3963,6 +4012,11 @@ static QObject *get_qmp_greeting(void)
>      return QOBJECT(result);
>  }
>  
> +static void monitor_qmp_caps_reset(Monitor *mon)
> +{
> +    memset(mon->qmp.qmp_caps, 0, sizeof(mon->qmp.qmp_caps));
> +}
> +
>  static void monitor_qmp_event(void *opaque, int event)
>  {
>      QObject *data;
> @@ -3971,6 +4025,7 @@ static void monitor_qmp_event(void *opaque, int event)
>      switch (event) {
>      case CHR_EVENT_OPENED:
>          mon->qmp.commands = &qmp_cap_negotiation_commands;
> +        monitor_qmp_caps_reset(mon);
>          data = get_qmp_greeting();
>          monitor_json_emitter(mon, data);
>          qobject_decref(data);
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 03201578b4..531fd4c0db 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -102,21 +102,30 @@
>  #
>  # Enable QMP capabilities.
>  #
> -# Arguments: None.
> +# Arguments:
> +#
> +# @enable:    List of QMPCapabilities to enable, which is optional.
> +#             If not provided, it means no QMP capabilities will be
> +#             enabled.  (since 2.12)
>  #
>  # Example:
>  #
> -# -> { "execute": "qmp_capabilities" }
> +# -> { "execute": "qmp_capabilities",
> +#      "arguments": { "enable": [ "oob" ] } }
>  # <- { "return": {} }
>  #
>  # Notes: This command is valid exactly when first connecting: it must be
>  # issued before any other command will be accepted, and will fail once the
>  # monitor is accepting other commands. (see qemu docs/interop/qmp-spec.txt)
>  #
> +# QMP client needs to explicitly enable QMP capabilities, otherwise
> +# all the QMP capabilities will be turned off by default.
> +#
>  # Since: 0.13
>  #
>  ##
> -{ 'command': 'qmp_capabilities' }
> +{ 'command': 'qmp_capabilities',
> +  'data': { '*enable': [ 'QMPCapability' ] } }
>  
>  ##
>  # @QMPCapability:
> -- 
> 2.14.3
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC v5 13/26] qmp: introduce some capability helpers

[Dr. David Alan Gilbert]

* Peter Xu (peterx@redhat.com) wrote:
> Introduce qmp_cap_enabled() and qmp_oob_enabled() helpers.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

> ---
>  monitor.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/monitor.c b/monitor.c
> index 097312e65f..9666115259 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -1038,6 +1038,16 @@ static void monitor_init_qmp_commands(void)
>                           qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
>  }
>  
> +static bool qmp_cap_enabled(Monitor *mon, QMPCapability cap)
> +{
> +    return mon->qmp.qmp_caps[cap];
> +}
> +
> +static bool qmp_oob_enabled(Monitor *mon)
> +{
> +    return qmp_cap_enabled(mon, QMP_CAPABILITY_OOB);
> +}
> +
>  static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
>                             Error **errp)
>  {
> -- 
> 2.14.3
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC v5 01/26] qobject: introduce qstring_get_try_str()

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 538 bytes --]

On Tue, Dec 05, 2017 at 01:51:35PM +0800, Peter Xu wrote:
> The only difference from qstring_get_str() is that it allows the qstring
> to be NULL.  If so, NULL is returned.
> 
> CC: Eric Blake <eblake@redhat.com>
> CC: Markus Armbruster <armbru@redhat.com>
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  include/qapi/qmp/qstring.h |  1 +
>  qobject/qstring.c          | 10 ++++++++++
>  2 files changed, 11 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 02/26] qobject: introduce qobject_get_try_str()

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 404 bytes --]

On Tue, Dec 05, 2017 at 01:51:36PM +0800, Peter Xu wrote:
> A quick way to fetch string from qobject when it's a QString.
> 
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  include/qapi/qmp/qstring.h |  1 +
>  qobject/qstring.c          | 11 +++++++++++
>  2 files changed, 12 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 03/26] qobject: let object_property_get_str() use new API

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 383 bytes --]

On Tue, Dec 05, 2017 at 01:51:37PM +0800, Peter Xu wrote:
> We can simplify object_property_get_str() using the new
> qobject_get_try_str().
> 
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  qom/object.c | 9 +++------
>  1 file changed, 3 insertions(+), 6 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 04/26] monitor: move skip_flush into monitor_data_init

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 399 bytes --]

On Tue, Dec 05, 2017 at 01:51:38PM +0800, Peter Xu wrote:
> It's part of the data init.  Collect it.
> 
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 05/26] qjson: add "opaque" field to JSONMessageParser

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]

On Tue, Dec 05, 2017 at 01:51:39PM +0800, Peter Xu wrote:
> diff --git a/qga/main.c b/qga/main.c
> index 62a62755bd..3b5ebbc1ee 100644
> --- a/qga/main.c
> +++ b/qga/main.c
> @@ -593,7 +593,8 @@ static void process_command(GAState *s, QDict *req)
>  }
>  
>  /* handle requests/control events coming in over the channel */
> -static void process_event(JSONMessageParser *parser, GQueue *tokens)
> +static void process_event(JSONMessageParser *parser, GQueue *tokens,
> +                          void *opaque)
>  {
>      GAState *s = container_of(parser, GAState, parser);
>      QDict *qdict;
> @@ -1320,7 +1321,7 @@ static int run_agent(GAState *s, GAConfig *config, int socket_activation)
>      s->command_state = ga_command_state_new();
>      ga_command_state_init(s, s->command_state);
>      ga_command_state_init_all(s->command_state);
> -    json_message_parser_init(&s->parser, process_event);
> +    json_message_parser_init(&s->parser, process_event, NULL);

This patch leaves the code with 2 ways of getting at state from the
parser pointer:
1. Use container_of() like existing users.
2. Use the new (unused) opaque argument.

Given that #1 exists, is this patch really necessary?

Please explain in the commit description, don't make me look ahead in
the patch series.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 06/26] monitor: move the cur_mon hack deeper for QMP

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1270 bytes --]

On Tue, Dec 05, 2017 at 01:51:40PM +0800, Peter Xu wrote:
> In monitor_qmp_read(), we have the hack to temporarily replace the
> cur_mon pointer.  Now we move this hack deeper inside the QMP dispatcher
> routine since the Monitor pointer can be passed in to that using the new
> JSON Parser opaque field now.
> 
> This does not make much sense as a single patch.  However, this will be
> a big step for the next patch, when the QMP dispatcher routine will be
> split from the QMP parser.
> 
> Reviewed-by: Eric Blake <eblake@redhat.com>
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 19 ++++++++++---------
>  1 file changed, 10 insertions(+), 9 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index ab80d32c70..322dfb5f31 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -3813,7 +3813,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
>  {
>      QObject *req, *rsp = NULL, *id = NULL;
>      QDict *qdict = NULL;
> -    Monitor *mon = cur_mon;
> +    Monitor *mon = opaque, *old_mon;

Or without the new opaque argument:

MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
Monitor *mon = container_of(mon_qmp, Monitor, qmp);

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 07/26] monitor: unify global init

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

On Tue, Dec 05, 2017 at 01:51:41PM +0800, Peter Xu wrote:
> diff --git a/vl.c b/vl.c
> index 1ad1c04637..1ec995a6ae 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -3144,7 +3144,6 @@ int main(int argc, char **argv, char **envp)
>      qemu_init_exec_dir(argv[0]);
>  
>      module_call_init(MODULE_INIT_QOM);
> -    monitor_init_qmp_commands();
>  
>      qemu_add_opts(&qemu_drive_opts);
>      qemu_add_drive_opts(&qemu_legacy_drive_opts);
> @@ -4690,6 +4689,8 @@ int main(int argc, char **argv, char **envp)
>      default_drive(default_floppy, snapshot, IF_FLOPPY, 0, FD_OPTS);
>      default_drive(default_sdcard, snapshot, IF_SD, 0, SD_OPTS);
>  
> +    monitor_init_globals();

Why did you move monitor_init_qmp_commands() down here?

There are many function calls between the old position and the new
position.  Did you check all of them to make sure they don't touch the
monitor which is now totally uninitialized?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 08/26] monitor: let mon_list be tail queue

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 442 bytes --]

On Tue, Dec 05, 2017 at 01:51:42PM +0800, Peter Xu wrote:
> It was QLIST.  I want to use this list to do monitor priority job later,
> which need tail insertion ability.  So switching to a tail queue.
> 
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 09/26] monitor: create monitor dedicate iothread

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1817 bytes --]

On Tue, Dec 05, 2017 at 01:51:43PM +0800, Peter Xu wrote:
> @@ -208,6 +209,12 @@ struct Monitor {
>      QTAILQ_ENTRY(Monitor) entry;
>  };
>  
> +struct MonitorGlobal {
> +    IOThread *mon_iothread;
> +};
> +
> +static struct MonitorGlobal mon_global;

structs can be anonymous.  That avoids the QEMU coding style violation
(structs must be typedefed):

  static struct {
      IOThread *mon_iothread;
  } mon_global;

In general global variables are usually top-level variables in QEMU.
I'm not sure why wrapping globals in a struct is useful.

> @@ -4117,6 +4136,16 @@ void monitor_cleanup(void)
>  {
>      Monitor *mon, *next;
>  
> +    /*
> +     * We need to explicitly stop the iothread (but not destroy it),
> +     * cleanup the monitor resources, then destroy the iothread.  See
> +     * again on the glib bug mentioned in 2b316774f6 for a reason.
> +     *
> +     * TODO: the bug is fixed in glib 2.28, so we can remove this hack
> +     * as long as we won't support glib versions older than it.
> +     */

I find this comment confusing.  There is no GSource .finalize() in
monitor.c so why does monitor_cleanup() need to work around the bug?

I see that monitor_data_destroy() is not thread-safe so the IOThread
must be stopped first.  That is unrelated to glib.

> +    iothread_stop(mon_global.mon_iothread);
> +
>      qemu_mutex_lock(&monitor_lock);
>      QTAILQ_FOREACH_SAFE(mon, &mon_list, entry, next) {
>          QTAILQ_REMOVE(&mon_list, mon, entry);
> @@ -4124,6 +4153,9 @@ void monitor_cleanup(void)
>          g_free(mon);
>      }
>      qemu_mutex_unlock(&monitor_lock);
> +
> +    iothread_destroy(mon_global.mon_iothread);
> +    mon_global.mon_iothread = NULL;
>  }
>  
>  QemuOptsList qemu_mon_opts = {
> -- 
> 2.14.3
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 10/26] monitor: allow to use IO thread for parsing

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2587 bytes --]

On Tue, Dec 05, 2017 at 01:51:44PM +0800, Peter Xu wrote:
> @@ -583,6 +585,7 @@ static void monitor_data_init(Monitor *mon, bool skip_flush)
>      /* Use *mon_cmds by default. */
>      mon->cmd_table = mon_cmds;
>      mon->skip_flush = skip_flush;
> +    mon->use_io_thr = use_io_thr;

Why is mon->use_io_thr is a field instead of a monitor_init() argument.

> @@ -4117,19 +4121,37 @@ void monitor_init(Chardev *chr, int flags)
>          monitor_read_command(mon, 0);
>      }
>  
> +    if (mon->use_io_thr) {
> +        /*
> +         * When use_io_thr is set, we use the global shared dedicated
> +         * IO thread for this monitor to handle input/output.
> +         */
> +        context = monitor_io_context_get();
> +        /* We should have inited globals before reaching here. */
> +        assert(context);
> +    } else {
> +        /* The default main loop, which is the main thread */
> +        context = NULL;
> +    }
> +
> +    /*
> +     * Add the monitor before running it (which is triggered by
> +     * qemu_chr_fe_set_handlers), otherwise one monitor may find
> +     * itself not on the mon_list when running.
> +     */
> +    qemu_mutex_lock(&monitor_lock);
> +    QTAILQ_INSERT_HEAD(&mon_list, mon, entry);
> +    qemu_mutex_unlock(&monitor_lock);
> +
>      if (monitor_is_qmp(mon)) {
>          qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
> -                                 monitor_qmp_event, NULL, mon, NULL, true);
> +                                 monitor_qmp_event, NULL, mon, context, true);
>          qemu_chr_fe_set_echo(&mon->chr, true);
>          json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);

The comment above mentions the race condition between
qemu_chr_fe_set_handlers() and mon_list.  I think that means the order
must be:

  json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
  qemu_chr_fe_set_echo(&mon->chr, true);
  qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
                           monitor_qmp_event, NULL, mon, context, true);

>      } else {

assert(!context);  /* HMP does not support IOThreads */

>          qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_read,
>                                   monitor_event, NULL, mon, NULL, true);
>      }
> -
> -    qemu_mutex_lock(&monitor_lock);
> -    QLIST_INSERT_HEAD(&mon_list, mon, entry);
> -    qemu_mutex_unlock(&monitor_lock);
>  }
>  
>  void monitor_cleanup(void)
> -- 
> 2.14.3
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 11/26] qmp: introduce QMPCapability

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2020 bytes --]

On Tue, Dec 05, 2017 at 01:51:45PM +0800, Peter Xu wrote:
> There was no QMP capabilities defined.  Define the first "oob" as
> capability to allow out-of-band messages.
> 
> Also, touch up qmp-test.c to test the new bits.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c        | 15 +++++++++++++--
>  qapi-schema.json | 13 +++++++++++++
>  tests/qmp-test.c | 10 +++++++++-
>  3 files changed, 35 insertions(+), 3 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index e8f5a586e4..bad6ee8dd1 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -3944,12 +3944,23 @@ void monitor_resume(Monitor *mon)
>  
>  static QObject *get_qmp_greeting(void)
>  {
> +    QDict *result = qdict_new(), *qmp = qdict_new();
> +    QList *cap_list = qlist_new();
>      QObject *ver = NULL;
> +    QMPCapability cap;
> +
> +    qdict_put(result, "QMP", qmp);
>  
>      qmp_marshal_query_version(NULL, &ver, NULL);
> +    qdict_put_obj(qmp, "version", ver);
> +
> +    for (cap = 0; cap < QMP_CAPABILITY__MAX; cap++) {
> +        qlist_append(cap_list, qstring_from_str(
> +                         QMPCapability_str(cap)));
> +    }
> +    qdict_put(qmp, "capabilities", cap_list);
>  
> -    return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': []}}",
> -                              ver);
> +    return QOBJECT(result);
>  }

Why did you replace qobject_from_jsonf() with manual qdict_*() calls?

I was expecting this (it's shorter and easier to read):

  static QObject *get_qmp_greeting(void)
  {
      QList *cap_list = qlist_new();
      QObject *ver = NULL;
      QMPCapability cap;

      qmp_marshal_query_version(NULL, &ver, NULL);

      for (cap = 0; cap < QMP_CAPABILITY__MAX; cap++) {
          qlist_append(cap_list, qstring_from_str(
                           QMPCapability_str(cap)));
      }

      return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': %p}}",
                                ver, cap);
  }

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 12/26] qmp: negociate QMP capabilities

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1122 bytes --]

On Tue, Dec 05, 2017 at 01:51:46PM +0800, Peter Xu wrote:
> @@ -1037,8 +1038,42 @@ static void monitor_init_qmp_commands(void)
>                           qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
>  }
>  
> -void qmp_qmp_capabilities(Error **errp)
> +static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
> +                           Error **errp)
> +{
> +    for (; list; list = list->next) {
> +        assert(list->value < QMP_CAPABILITY__MAX);
> +        switch (list->value) {
> +        case QMP_CAPABILITY_OOB:
> +            if (!mon->use_io_thr) {
> +                /*
> +                 * Out-Of-Band only works with monitors that are
> +                 * running on dedicated IOThread.
> +                 */
> +                error_setg(errp, "This monitor does not support "
> +                           "Out-Of-Band (OOB)");
> +                return;
> +            }
> +            break;

QEMU always offers the 'oob' capability, even if the monitor does not
support it.  Should it send 'oob' only when mon->use_io_thr to make
things easier for clients?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 13/26] qmp: introduce some capability helpers

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 300 bytes --]

On Tue, Dec 05, 2017 at 01:51:47PM +0800, Peter Xu wrote:
> Introduce qmp_cap_enabled() and qmp_oob_enabled() helpers.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 14/26] monitor: introduce monitor_qmp_respond()

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1055 bytes --]

On Tue, Dec 05, 2017 at 01:51:48PM +0800, Peter Xu wrote:
> A tiny refactoring, preparing to split the QMP dispatcher away.
> 
> Reviewed-by: Fam Zheng <famz@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 48 +++++++++++++++++++++++++++++++-----------------
>  1 file changed, 31 insertions(+), 17 deletions(-)
> 
> diff --git a/monitor.c b/monitor.c
> index 9666115259..35d8925636 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -3877,6 +3877,36 @@ static int monitor_can_read(void *opaque)
>      return (mon->suspend_cnt == 0) ? 1 : 0;
>  }
>  
> +/*
> + * When rsp/err/id is passed in, the function will be responsible for
> + * the cleanup.
> + */

Please document this function fully.  I had to look at the
implementation to learn how the arguments work:

1. This function takes ownership of rsp, err, and id.  (Please use this
exact wording because the function does not perform the "cleanup" when
refcount > 1.)
2. rsp, err, and id may be NULL.
3. If err != NULL then rsp must be NULL.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 15/26] monitor: let suspend_cnt be thread safe

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1069 bytes --]

On Tue, Dec 05, 2017 at 01:51:49PM +0800, Peter Xu wrote:
> Monitor code now can be run in more than one thread.  Let it be thread
> safe when accessing suspend_cnt counter.

Please add doc comments to the functions explaining which thread they
may be called from (especially public functions).  Without this
information other people will have a hard time keeping code thread-safe.

> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 13 +++++++------
>  1 file changed, 7 insertions(+), 6 deletions(-)

From docs/devel/atomics.txt:

  Macros defined by qemu/atomic.h fall in three camps:

  - compiler barriers: barrier();

  - weak atomic access and manual memory barriers: atomic_read(),
    atomic_set(), smp_rmb(), smp_wmb(), smp_mb(), smp_mb_acquire(),
    smp_mb_release(), smp_read_barrier_depends();

  - sequentially consistent atomic access: everything else.

The atomic_read() and atomic_set() calls in this patch aren't effective
without memory barriers.  Please use atomic_mb_read() and
atomic_mb_set() instead.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 16/26] monitor: separate QMP parser and dispatcher

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4161 bytes --]

On Tue, Dec 05, 2017 at 01:51:50PM +0800, Peter Xu wrote:
> @@ -3956,12 +3968,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
>          }
>      }
>  
> -err_out:
> -    monitor_qmp_respond(mon, rsp, err, id);
> +    /* Respond if necessary */
> +    monitor_qmp_respond(mon, rsp, NULL, id);
> +
> +    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
> +    if (!qmp_oob_enabled(mon)) {
> +        monitor_resume(mon);

monitor_resume() does not work between threads: if the event loop is
currently blocked in poll() it won't notice that the monitor fd should
be watched again.

Please add aio_notify() to monitor_resume() and monitor_suspend().  That
way the event loop is forced to check can_read() again.

> +    }
>  
>      qobject_decref(req);
>  }
>  
> +/*
> + * Pop one QMP request from monitor queues, return NULL if not found.
> + * We are using round-robin fasion to pop the request, to avoid
> + * processing command only on a very busy monitor.  To achieve that,
> + * when we processed one request on specific monitor, we put that
> + * monitor to the end of mon_list queue.
> + */
> +static QMPRequest *monitor_qmp_requests_pop_one(void)
> +{
> +    QMPRequest *req_obj = NULL;
> +    Monitor *mon;
> +
> +    qemu_mutex_lock(&monitor_lock);
> +
> +    QTAILQ_FOREACH(mon, &mon_list, entry) {
> +        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);

Please add a doc comment about the monitor_lock < qmp_queue_lock lock
ordering in the qmp_queue_lock field declaration so that deadlocks can
be prevented.

> +        req_obj = g_queue_pop_head(mon->qmp.qmp_requests);
> +        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> +        if (req_obj) {
> +            break;
> +        }
> +    }
> +
> +    if (req_obj) {
> +        /*
> +         * We found one request on the monitor. Degrade this monitor's
> +         * priority to lowest by re-inserting it to end of queue.
> +         */
> +        QTAILQ_REMOVE(&mon_list, mon, entry);
> +        QTAILQ_INSERT_TAIL(&mon_list, mon, entry);
> +    }
> +
> +    qemu_mutex_unlock(&monitor_lock);
> +
> +    return req_obj;
> +}
> +
> +static void monitor_qmp_bh_dispatcher(void *data)
> +{
> +    QMPRequest *req_obj;
> +
> +    while (true) {

Previously QEMU could only dispatch 1 QMP command per main loop
iteration.  Now multiple requests can be processed in a single main loop
iteration.

If a producer enqueues requests faster than the dispatcher can dequeue
them then this is infinite loop will prevent the caller (i.e.  QEMU main
loop) from making progress.

The following keeps 1 QMP command per main loop iteration and avoids the
infinite loop:

  static void monitor_qmp_bh_dispatcher(void *data)
  {
      QMPRequest *req_obj = monitor_qmp_requests_pop_one();

      if (req_obj) {
          monitor_qmp_dispatch_one(req_obj);

	  /* Reschedule instead of looping so the main loop stays responsive */
          qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
      }
  }

> +    /*
> +     * Put the request to the end of queue so that requests will be
> +     * handled in time order.  Ownership for req_obj, req, id,
> +     * etc. will be delivered to the handler side.
> +     */
> +    qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> +    g_queue_push_tail(mon->qmp.qmp_requests, req_obj);
> +    qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> +
> +    /* Kick the dispatcher routine */
> +    qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> +
> +    /*
> +     * If OOB is not enabled on current monitor, we'll emulate the old
> +     * behavior that we won't process current monitor any more until
> +     * it is responded.  This helps make sure that as long as OOB is
> +     * not enabled, the server will never drop any command.
> +     */
> +    if (!qmp_oob_enabled(mon)) {
> +        monitor_suspend(mon);
> +    }

monitor_suspend() must be called before g_queue_push_tail().  Otherwise
the other thread might complete the request and call monitor_resume()
before we call monitor_suspend().

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 17/26] qmp: add new event "request-dropped"

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 826 bytes --]

On Tue, Dec 05, 2017 at 01:51:51PM +0800, Peter Xu wrote:
> +##
> +# @RequestDropReason:
> +#
> +# Reasons that caused one command to be dropped.
> +#
> +# @queue-full: the queue of request is full.
> +#
> +# Since: 2.12
> +##
> +{ 'enum': 'RequestDropReason',

qapi-schema.json calls them "commands", not "requests".  Please rename
it CommandDropReason for consistency and update the documentation.

> +  'data': [ 'queue-full' ] }
> +
> +##
> +# @REQUEST_DROPPED:
> +#
> +# Emitted when one QMP request is dropped due to some reason.
> +# REQUEST_DROPPED is only emitted when the oob capability is enabled.

Please make it clear that requests can only be dropped when the oob
capability is enabled.

  Emitted when a command is dropped due to some reason.  Commands can
  only be dropped when the oob capability is enabled.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 17/26] qmp: add new event "request-dropped"

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

On Tue, Dec 05, 2017 at 01:51:51PM +0800, Peter Xu wrote:
> +##
> +# @RequestDropReason:
> +#
> +# Reasons that caused one command to be dropped.
> +#
> +# @queue-full: the queue of request is full.

Please add:

  This can only occur when the client sends a new non-oob command before
  the response to the previous non-oob command has been received.

This way simple clients that only send 1 command at a time don't need to
worry about queue-full.  I think this documents the behavior implemented
by this patch series.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 18/26] monitor: send event when request queue full

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2038 bytes --]

On Tue, Dec 05, 2017 at 01:51:52PM +0800, Peter Xu wrote:
> Set maximum QMP request queue length to 8.  If queue full, instead of
> queue the command, we directly return a "request-dropped" event, telling
> client that specific command is dropped.
> 
> Note that this flow control mechanism is only valid if OOB is enabled.
> If it's not, the effective queue length will always be 1, which strictly
> follows original behavior of QMP command handling (which never drop
> messages).
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 15 +++++++++++++++
>  1 file changed, 15 insertions(+)
> 
> diff --git a/monitor.c b/monitor.c
> index c20e659740..f7923c4590 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -4029,6 +4029,8 @@ static void monitor_qmp_bh_dispatcher(void *data)
>      }
>  }
>  
> +#define  QMP_REQ_QUEUE_LEN_MAX  (8)
> +
>  static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
>                                 void *opaque)
>  {
> @@ -4061,6 +4063,19 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
>      req_obj->id = id;
>      req_obj->req = req;
>  
> +    if (qmp_oob_enabled(mon)) {
> +        /* Drop the request if queue is full. */
> +        if (mon->qmp.qmp_requests->length >= QMP_REQ_QUEUE_LEN_MAX) {

qmp_queue_lock must be held.  Perhaps it's simplest to move this check
into the qmp_queue_lock critical section below and unlock before calling
qapi_event_send_request_dropped()...

> +            qapi_event_send_request_dropped(id,
> +                                            REQUEST_DROP_REASON_QUEUE_FULL,
> +                                            NULL);
> +            qobject_decref(id);
> +            qobject_decref(req);
> +            g_free(req_obj);
> +            return;
> +        }
> +    }
> +
>      /*
>       * Put the request to the end of queue so that requests will be
>       * handled in time order.  Ownership for req_obj, req, id,

... down here.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 19/26] qapi: introduce new cmd option "allow-oob"

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1527 bytes --]

On Tue, Dec 05, 2017 at 01:51:53PM +0800, Peter Xu wrote:
> Here "oob" stands for "Out-Of-Band".  When "allow-oob" is set, it means
> the command allows out-of-band execution.
> 
> The "oob" idea is proposed by Markus Armbruster in following thread:
> 
>   https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg02057.html
> 
> This new "allow-oob" boolean will be exposed by "query-qmp-schema" as
> well for command entries, so that QMP clients can know which command can
> be used as out-of-band calls. For example the command "migrate"
> originally looks like:
> 
>   {"name": "migrate", "ret-type": "17", "meta-type": "command",
>    "arg-type": "86"}
> 
> And it'll be changed into:
> 
>   {"name": "migrate", "ret-type": "17", "allow-oob": false,
>    "meta-type": "command", "arg-type": "86"}
> 
> This patch only provides the QMP interface level changes.  It does not
> contains the real out-of-band execution implementation yet.
> 
> Suggested-by: Markus Armbruster <armbru@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  include/qapi/qmp/dispatch.h    |  1 +
>  qapi/introspect.json           |  6 +++++-
>  scripts/qapi-commands.py       | 19 ++++++++++++++-----
>  scripts/qapi-introspect.py     | 10 ++++++++--
>  scripts/qapi.py                | 15 ++++++++++-----
>  scripts/qapi2texi.py           |  2 +-
>  tests/qapi-schema/test-qapi.py |  2 +-
>  7 files changed, 40 insertions(+), 15 deletions(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 20/26] qmp: support out-of-band (oob) execution

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 421 bytes --]

On Tue, Dec 05, 2017 at 01:51:54PM +0800, Peter Xu wrote:
>      if (qdict) {
>          id = qdict_get(qdict, "id");
> +        /* When OOB is enabled, the "id" field is mandatory. */
> +        if (qmp_oob_enabled(mon) && !id) {
> +            error_setg(&err, "Out-Of-Band capability requires that "
> +                       "every command contains an 'id' field.");

Is this documented in docs/interop/qmp-spec.txt?

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 21/26] qmp: isolate responses into io thread

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1195 bytes --]

On Tue, Dec 05, 2017 at 01:51:55PM +0800, Peter Xu wrote:
> @@ -4429,6 +4515,13 @@ void monitor_cleanup(void)
>       */
>      iothread_stop(mon_global.mon_iothread);
>  
> +    /*
> +     * After we have IOThread to send responses, it's possible that
> +     * when we stop the IOThread there are still replies queued in the
> +     * responder queue.  Flush all of them.
> +     */
> +    monitor_qmp_bh_responder(NULL);

This doesn't work because monitor_qmp_bh_responder() does not guarantee
that the full response has been written when it returns.

When qemu_chr_fe_write() returns EAGAIN then qemu_chr_fe_add_watch() is
used to register an event loop callback when the chardev becomes
writable again.  But you stopped the event loop using iothread_stop() so
we will never complete the write.

I suggest draining the monitor while the IOThread is still running
(that way the AioContext and GMainContext are still operational).  You
can:
1. Suspend the monitor so new commands will not be read.
2. Wait until all responses and outbuf are empty.

Another option is moving the chardev back to the main loop but I'm not
sure if the chardev subsystem supports that.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 22/26] monitor: enable IO thread for (qmp & !mux) typed

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 394 bytes --]

On Tue, Dec 05, 2017 at 01:51:56PM +0800, Peter Xu wrote:
> Start to use dedicate IO thread for QMP monitors that are not using
> MUXed chardev.
> 
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 23/26] qmp: add command "x-oob-test"

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 403 bytes --]

On Tue, Dec 05, 2017 at 01:51:57PM +0800, Peter Xu wrote:
> This command is only used to test OOB functionality.  It should not be
> used for any other purposes.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  qapi-schema.json | 18 ++++++++++++++++++
>  qmp.c            | 16 ++++++++++++++++
>  2 files changed, 34 insertions(+)

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 24/26] docs: update QMP documents for OOB commands

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 6245 bytes --]

On Tue, Dec 05, 2017 at 01:51:58PM +0800, Peter Xu wrote:
> diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
> index f04c63fe82..8597fdb087 100644
> --- a/docs/devel/qapi-code-gen.txt
> +++ b/docs/devel/qapi-code-gen.txt
> @@ -556,7 +556,8 @@ following example objects:
>  
>  Usage: { 'command': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT,
>           '*returns': TYPE-NAME, '*boxed': true,
> -         '*gen': false, '*success-response': false }
> +         '*gen': false, '*success-response': false,
> +         '*allow-oob': false }
>  
>  Commands are defined by using a dictionary containing several members,
>  where three members are most common.  The 'command' member is a
> @@ -636,6 +637,44 @@ possible, the command expression should include the optional key
>  'success-response' with boolean value false.  So far, only QGA makes
>  use of this member.
>  
> +Most of the QMP commands are handled sequentially in such a order:
> +Firstly, the JSON Parser parses the command request into some internal
> +message, delivers the message to QMP dispatchers. Secondly, the QMP
> +dispatchers will handle the commands one by one in time order, respond
> +when necessary.

The important points to cover about normal commands:
1. They execute in order
2. They run the main loop
3. They run under the BQL

The other stuff about parsing requests into internal messages,
dispatchers, etc is not relevant.  It's better not to include it in
documentation because it can change and could also confuse readers
(since they don't need this info).

> For some commands that always complete "quickly" can

I've mentioned before that "quickly" is misleading and not what oob
commands are about.  I suggest changing this to:

  Certain urgent commands can

I've made similar comments further down where I think the text focusses
on words like "quickly" or "asynchronous" too much.

> +instead be executed directly during parsing, at the QMP client's
> +request.  This kind of commands that allow direct execution is called
> +"out-of-band" ("oob" as shortcut) commands. The response can overtake
> +prior in-band commands' responses.  By default, commands are always
> +in-band.  We need to explicitly specify "allow-oob" to "True" to show

s/"True"/true/  (JSON is case-sensitive)

> +that one command can be run out-of-band.
> +
> +One thing to mention for developers is that, although out-of-band
> +execution of commands benefit from quick and asynchronous execution,
> +it need to satisfy at least the following:
> +
> +(1) It is extremely quick and never blocks, so that its execution will
> +    not block parsing routine of any other monitors.
> +
> +(2) It does not need BQL, since the parser can be run without BQL,
> +    while the dispatcher is always with BQL held.

These conditions are not sufficient for postcopy recovery.  I suggest
rephrasing this section as follows:

  An out-of-band command must:

  1. Complete extremely quickly
  2. Not take locks
  3. Not invoke blocking system calls
  4. Not access guest RAM or memory that blocks when userfaultfd is
     enabled for postcopy live migration

We could make #2 less strict by saying "Not take locks except for
spinlocks (or mutexes that could be spinlocks because the critical
regions are tiny) or indirectly via g_malloc()".

> Whether a command is
> +allowed to run out-of-band can also be introspected using
> +query-qmp-schema command.  Please see the section "Client JSON
> +Protocol introspection" for more information.

This is relevant to client authors, not QEMU developers learning about
qapi.  I suggest dropping it.

> +
> +To execute a command in out-of-band way, we need to specify the
> +"control" field in the request, with "run-oob" set to true. Example:
> +
> + => { "execute": "command-support-oob",
> +      "arguments": { ... },
> +      "control": { "run-oob": true } }
> + <= { "return": { } }
> +
> +Without it, even the commands that supports out-of-band execution will
> +still be run in-band.

This is relevant to client authors, not QEMU developers learning about
qapi.  I suggest instead explaining how qmp functions need to test for
qmp_is_oob() so that they know which mode they are executing in.

>  2.2.1 Capabilities
>  ------------------
>  
> -As of the date this document was last revised, no server or client
> -capability strings have been defined.
> -
> +Currently supported capabilities are:
> +
> +- "oob": it means the QMP server supports "Out-Of-Band" command
> +  execution.  For more detail, please see "run-oob" parameter in
> +  "Issuing Commands" section below.  Not all commands allow this "oob"
> +  execution.  One can know whether one command supports "oob" by
> +  "query-qmp-schema" command.
> +
> +  NOTE: Converting an existing QMP command to be OOB-capable can be
> +  either very easy or extremely hard.  The most important thing is
> +  that OOB-capable command should never be blocked for a long time.
> +  Some bad examples: (1) doing IOs, especially when the backend can be
> +  an NFS storage; or (2) accessing guest memories, which can be simply
> +  blocked for a very long time when it triggers a page fault, which
> +  may not be handled immediately.  It's still legal to take a mutex in
> +  an OOB-capable command handler, however, we need to make sure that
> +  all the other code paths that are protected by the same lock won't
> +  be blocked very long as well, otherwise the OOB handler might be
> +  blocked too when it tries to take the lock.

Please drop this paragraph, this is the qmp-spec.txt document so the
implementation of QEMU's QMP commands shouldn't be discussed here.

> For some commands that always complete
> +  "quickly" can be executed directly during parsing at the QMP
> +  client's request.

Please drop this sentence.  "quickly" isn't the important quality, it's
urgency.  Also the description of execution in the QMP parser isn't
relevant for qmp-spec.txt, what matters is that oob commands can execute
while a normal monitor command is still running (this is described next
so this whole sentence can be dropped).

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 25/26] tests: qmp-test: verify command batching

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1507 bytes --]

On Tue, Dec 05, 2017 at 01:51:59PM +0800, Peter Xu wrote:
> OOB introduced DROP event for flow control.  This should not affect old
> QMP clients.  Add a command batching check to make sure of it.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  tests/qmp-test.c | 19 +++++++++++++++++++
>  1 file changed, 19 insertions(+)
> 
> diff --git a/tests/qmp-test.c b/tests/qmp-test.c
> index 292c5f135a..729ec59b0a 100644
> --- a/tests/qmp-test.c
> +++ b/tests/qmp-test.c
> @@ -78,6 +78,7 @@ static void test_qmp_protocol(void)
>      QList *capabilities;
>      const QListEntry *entry;
>      QString *qstr;
> +    int i;
>  
>      global_qtest = qtest_init_without_qmp_handshake(common_args);
>  
> @@ -135,6 +136,24 @@ static void test_qmp_protocol(void)
>      g_assert_cmpint(qdict_get_int(resp, "id"), ==, 2);
>      QDECREF(resp);
>  
> +    /*
> +     * Test command batching.  In current test OOB is not enabled, we
> +     * should be able to run as many commands in batch as we like.
> +     * Using 16 (>8, which is OOB queue length) to make sure OOB
> +     * won't break existing clients.
> +     */

It's possible that QEMU processes the command and responds before the
test case submits the next one.  Please add:

  Note: this test does not control the scheduling of QEMU's QMP command
  processing threads so it may not really trigger batching inside QEMU.
  This is just a best-effort test.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 26/26] tests: qmp-test: add oob test

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

On Tue, Dec 05, 2017 at 01:52:00PM +0800, Peter Xu wrote:
> +    /* Try a fake capability, it should fail. */
> +    resp = qmp("{ 'execute': 'qmp_capabilities', "
> +               "  'arguments': { 'enable': [ 'cap-does-not-exist' ] } }");
> +    g_assert(qdict_haskey(resp, "error"));

Missing QDECREF(resp) here and throughout the rest of the function.

Besides that:

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 00/26] QMP: out-of-band (OOB) execution support

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 286 bytes --]

On Tue, Dec 05, 2017 at 01:51:34PM +0800, Peter Xu wrote:
> This version is mostly document update, and dropped the single patch
> that is migration related (will be put into postcopy recovery
> series).

I've finished reviewing this revision.  The overall approach looks good.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 05/26] qjson: add "opaque" field to JSONMessageParser

[Peter Xu]

On Wed, Dec 13, 2017 at 03:37:02PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:39PM +0800, Peter Xu wrote:
> > diff --git a/qga/main.c b/qga/main.c
> > index 62a62755bd..3b5ebbc1ee 100644
> > --- a/qga/main.c
> > +++ b/qga/main.c
> > @@ -593,7 +593,8 @@ static void process_command(GAState *s, QDict *req)
> >  }
> >  
> >  /* handle requests/control events coming in over the channel */
> > -static void process_event(JSONMessageParser *parser, GQueue *tokens)
> > +static void process_event(JSONMessageParser *parser, GQueue *tokens,
> > +                          void *opaque)
> >  {
> >      GAState *s = container_of(parser, GAState, parser);
> >      QDict *qdict;
> > @@ -1320,7 +1321,7 @@ static int run_agent(GAState *s, GAConfig *config, int socket_activation)
> >      s->command_state = ga_command_state_new();
> >      ga_command_state_init(s, s->command_state);
> >      ga_command_state_init_all(s->command_state);
> > -    json_message_parser_init(&s->parser, process_event);
> > +    json_message_parser_init(&s->parser, process_event, NULL);
> 
> This patch leaves the code with 2 ways of getting at state from the
> parser pointer:
> 1. Use container_of() like existing users.
> 2. Use the new (unused) opaque argument.
> 
> Given that #1 exists, is this patch really necessary?

I didn't really notice that.  Thanks for pointing out.

However even if so I would still prefer the opaque way to do it if
asked.  Existing #1 of course works but IMHO is less flexible and has
dependency between structure layouts.

How about I append another patch to convert existing users (or, I can
post as separate patches after this series)?  It's not really a lot,
and the conversion would be obvious:

*** qga/main.c:
run_agent[1324]                json_message_parser_init(&s->parser, process_event, NULL);
*** qobject/qjson.c:
qobject_from_jsonv[45]         json_message_parser_init(&state.parser, parse_json, NULL);
*** tests/libqtest.c:
qmp_fd_receive[438]            json_message_parser_init(&qmp.parser, qmp_response, NULL);

Though, if you still insist, I can drop it too.

> 
> Please explain in the commit description, don't make me look ahead in
> the patch series.

I will add some comments if this patch is still alive in next
version.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 06/26] monitor: move the cur_mon hack deeper for QMP

[Peter Xu]

On Wed, Dec 13, 2017 at 03:41:49PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:40PM +0800, Peter Xu wrote:
> > In monitor_qmp_read(), we have the hack to temporarily replace the
> > cur_mon pointer.  Now we move this hack deeper inside the QMP dispatcher
> > routine since the Monitor pointer can be passed in to that using the new
> > JSON Parser opaque field now.
> > 
> > This does not make much sense as a single patch.  However, this will be
> > a big step for the next patch, when the QMP dispatcher routine will be
> > split from the QMP parser.
> > 
> > Reviewed-by: Eric Blake <eblake@redhat.com>
> > Reviewed-by: Fam Zheng <famz@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  monitor.c | 19 ++++++++++---------
> >  1 file changed, 10 insertions(+), 9 deletions(-)
> > 
> > diff --git a/monitor.c b/monitor.c
> > index ab80d32c70..322dfb5f31 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -3813,7 +3813,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> >  {
> >      QObject *req, *rsp = NULL, *id = NULL;
> >      QDict *qdict = NULL;
> > -    Monitor *mon = cur_mon;
> > +    Monitor *mon = opaque, *old_mon;
> 
> Or without the new opaque argument:
> 
> MonitorQMP *mon_qmp = container_of(parser, MonitorQMP, parser);
> Monitor *mon = container_of(mon_qmp, Monitor, qmp);

Please see previous reply.  Please just let me know if you insist on
keeping method #1 then I can drop the other one and rewrite this
patch.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 07/26] monitor: unify global init

[Peter Xu]

On Wed, Dec 13, 2017 at 03:48:06PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:41PM +0800, Peter Xu wrote:
> > diff --git a/vl.c b/vl.c
> > index 1ad1c04637..1ec995a6ae 100644
> > --- a/vl.c
> > +++ b/vl.c
> > @@ -3144,7 +3144,6 @@ int main(int argc, char **argv, char **envp)
> >      qemu_init_exec_dir(argv[0]);
> >  
> >      module_call_init(MODULE_INIT_QOM);
> > -    monitor_init_qmp_commands();
> >  
> >      qemu_add_opts(&qemu_drive_opts);
> >      qemu_add_drive_opts(&qemu_legacy_drive_opts);
> > @@ -4690,6 +4689,8 @@ int main(int argc, char **argv, char **envp)
> >      default_drive(default_floppy, snapshot, IF_FLOPPY, 0, FD_OPTS);
> >      default_drive(default_sdcard, snapshot, IF_SD, 0, SD_OPTS);
> >  
> > +    monitor_init_globals();
> 
> Why did you move monitor_init_qmp_commands() down here?
> 
> There are many function calls between the old position and the new
> position.  Did you check all of them to make sure they don't touch the
> monitor which is now totally uninitialized?

Yeh, this patch is a bit hairy, but I really think we should do it.
Because there are too many places (as you have seen) that we inited
monitor stuff in different places.

IMHO monitor_init_qmp_commands() should be safe to be called here
since it only do registration of commands, and AFAICT those commands
won't be touched until we init any of the monitors below
monitor_init_globals():

    if (qemu_opts_foreach(qemu_find_opts("mon"),
                          mon_init_func, NULL, NULL)) {
        exit(1);
    }

Similar things to other stuff in monitor_init_globals().  They just
won't be touched up (correct me if not) until this mon_init_func().

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 09/26] monitor: create monitor dedicate iothread

[Peter Xu]

On Wed, Dec 13, 2017 at 04:20:22PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:43PM +0800, Peter Xu wrote:
> > @@ -208,6 +209,12 @@ struct Monitor {
> >      QTAILQ_ENTRY(Monitor) entry;
> >  };
> >  
> > +struct MonitorGlobal {
> > +    IOThread *mon_iothread;
> > +};
> > +
> > +static struct MonitorGlobal mon_global;
> 
> structs can be anonymous.  That avoids the QEMU coding style violation
> (structs must be typedefed):
> 
>   static struct {
>       IOThread *mon_iothread;
>   } mon_global;

Will fix this up.  Thanks.

> 
> In general global variables are usually top-level variables in QEMU.
> I'm not sure why wrapping globals in a struct is useful.

Because I see too many global variables for monitor code, and from
this patch I wanted to start moving them altogether into this global
struct.  I didn't really do that in current series because it's more
like a clean up, but if you see future patches, it at least grows with
new monitor global variables introduced with current series.

I can add a comment in the commit message, like: "Let's start to
create a struct to keep monitor global variables together".  Would
that help?

> 
> > @@ -4117,6 +4136,16 @@ void monitor_cleanup(void)
> >  {
> >      Monitor *mon, *next;
> >  
> > +    /*
> > +     * We need to explicitly stop the iothread (but not destroy it),
> > +     * cleanup the monitor resources, then destroy the iothread.  See
> > +     * again on the glib bug mentioned in 2b316774f6 for a reason.
> > +     *
> > +     * TODO: the bug is fixed in glib 2.28, so we can remove this hack
> > +     * as long as we won't support glib versions older than it.
> > +     */
> 
> I find this comment confusing.  There is no GSource .finalize() in
> monitor.c so why does monitor_cleanup() need to work around the bug?
> 
> I see that monitor_data_destroy() is not thread-safe so the IOThread
> must be stopped first.  That is unrelated to glib.

Yeah actually that's a suggestion by Dave and Dan in previous review
comments which makes sense to me:

  http://lists.gnu.org/archive/html/qemu-devel/2017-11/msg04344.html

I'm fine with either way: keep it as it is, or instead saying
"monitor_data_destroy() is not thread-safe" (which finally will still
root cause to that glib bug).  But how about we just keep it in case
it may be helpful some day?

Thanks,

> 
> > +    iothread_stop(mon_global.mon_iothread);
> > +
> >      qemu_mutex_lock(&monitor_lock);
> >      QTAILQ_FOREACH_SAFE(mon, &mon_list, entry, next) {
> >          QTAILQ_REMOVE(&mon_list, mon, entry);
> > @@ -4124,6 +4153,9 @@ void monitor_cleanup(void)
> >          g_free(mon);
> >      }
> >      qemu_mutex_unlock(&monitor_lock);
> > +
> > +    iothread_destroy(mon_global.mon_iothread);
> > +    mon_global.mon_iothread = NULL;
> >  }
> >  
> >  QemuOptsList qemu_mon_opts = {
> > -- 
> > 2.14.3
> > 

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 10/26] monitor: allow to use IO thread for parsing

[Peter Xu]

On Wed, Dec 13, 2017 at 04:35:00PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:44PM +0800, Peter Xu wrote:
> > @@ -583,6 +585,7 @@ static void monitor_data_init(Monitor *mon, bool skip_flush)
> >      /* Use *mon_cmds by default. */
> >      mon->cmd_table = mon_cmds;
> >      mon->skip_flush = skip_flush;
> > +    mon->use_io_thr = use_io_thr;
> 
> Why is mon->use_io_thr is a field instead of a monitor_init() argument.

It's used in other part of code when we want to know whether IOThread
is enabled for a monitor.

> 
> > @@ -4117,19 +4121,37 @@ void monitor_init(Chardev *chr, int flags)
> >          monitor_read_command(mon, 0);
> >      }
> >  
> > +    if (mon->use_io_thr) {
> > +        /*
> > +         * When use_io_thr is set, we use the global shared dedicated
> > +         * IO thread for this monitor to handle input/output.
> > +         */
> > +        context = monitor_io_context_get();
> > +        /* We should have inited globals before reaching here. */
> > +        assert(context);
> > +    } else {
> > +        /* The default main loop, which is the main thread */
> > +        context = NULL;
> > +    }
> > +
> > +    /*
> > +     * Add the monitor before running it (which is triggered by
> > +     * qemu_chr_fe_set_handlers), otherwise one monitor may find
> > +     * itself not on the mon_list when running.
> > +     */
> > +    qemu_mutex_lock(&monitor_lock);
> > +    QTAILQ_INSERT_HEAD(&mon_list, mon, entry);
> > +    qemu_mutex_unlock(&monitor_lock);
> > +
> >      if (monitor_is_qmp(mon)) {
> >          qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
> > -                                 monitor_qmp_event, NULL, mon, NULL, true);
> > +                                 monitor_qmp_event, NULL, mon, context, true);
> >          qemu_chr_fe_set_echo(&mon->chr, true);
> >          json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
> 
> The comment above mentions the race condition between
> qemu_chr_fe_set_handlers() and mon_list.  I think that means the order
> must be:
> 
>   json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
>   qemu_chr_fe_set_echo(&mon->chr, true);
>   qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
>                            monitor_qmp_event, NULL, mon, context, true);

Yeh this looks safer!

> 
> >      } else {
> 
> assert(!context);  /* HMP does not support IOThreads */

I can add this.  Thanks,

> 
> >          qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_read,
> >                                   monitor_event, NULL, mon, NULL, true);
> >      }
> > -
> > -    qemu_mutex_lock(&monitor_lock);
> > -    QLIST_INSERT_HEAD(&mon_list, mon, entry);
> > -    qemu_mutex_unlock(&monitor_lock);
> >  }
> >  
> >  void monitor_cleanup(void)
> > -- 
> > 2.14.3
> > 



-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 11/26] qmp: introduce QMPCapability

[Peter Xu]

On Wed, Dec 13, 2017 at 04:56:51PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:45PM +0800, Peter Xu wrote:
> > There was no QMP capabilities defined.  Define the first "oob" as
> > capability to allow out-of-band messages.
> > 
> > Also, touch up qmp-test.c to test the new bits.
> > 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  monitor.c        | 15 +++++++++++++--
> >  qapi-schema.json | 13 +++++++++++++
> >  tests/qmp-test.c | 10 +++++++++-
> >  3 files changed, 35 insertions(+), 3 deletions(-)
> > 
> > diff --git a/monitor.c b/monitor.c
> > index e8f5a586e4..bad6ee8dd1 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -3944,12 +3944,23 @@ void monitor_resume(Monitor *mon)
> >  
> >  static QObject *get_qmp_greeting(void)
> >  {
> > +    QDict *result = qdict_new(), *qmp = qdict_new();
> > +    QList *cap_list = qlist_new();
> >      QObject *ver = NULL;
> > +    QMPCapability cap;
> > +
> > +    qdict_put(result, "QMP", qmp);
> >  
> >      qmp_marshal_query_version(NULL, &ver, NULL);
> > +    qdict_put_obj(qmp, "version", ver);
> > +
> > +    for (cap = 0; cap < QMP_CAPABILITY__MAX; cap++) {
> > +        qlist_append(cap_list, qstring_from_str(
> > +                         QMPCapability_str(cap)));
> > +    }
> > +    qdict_put(qmp, "capabilities", cap_list);
> >  
> > -    return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': []}}",
> > -                              ver);
> > +    return QOBJECT(result);
> >  }
> 
> Why did you replace qobject_from_jsonf() with manual qdict_*() calls?
> 
> I was expecting this (it's shorter and easier to read):
> 
>   static QObject *get_qmp_greeting(void)
>   {
>       QList *cap_list = qlist_new();
>       QObject *ver = NULL;
>       QMPCapability cap;
> 
>       qmp_marshal_query_version(NULL, &ver, NULL);
> 
>       for (cap = 0; cap < QMP_CAPABILITY__MAX; cap++) {
>           qlist_append(cap_list, qstring_from_str(
>                            QMPCapability_str(cap)));
>       }
> 
>       return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': %p}}",
>                                 ver, cap);

(I believe you mean s/cap/cap_list/ here?)

>   }

Oh I just didn't notice that "%p" magic at all... :(

I think for me it's fine in either way.  Frankly speaking creating the
objects explicitly would be even easier to understand for me instead
of using a mixture of two ways... But just let me know if you want me
to do it your way.  I can switch.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 11/26] qmp: introduce QMPCapability

[Fam Zheng]

On Fri, 12/15 17:14, Peter Xu wrote:
> On Wed, Dec 13, 2017 at 04:56:51PM +0000, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:51:45PM +0800, Peter Xu wrote:
> > > There was no QMP capabilities defined.  Define the first "oob" as
> > > capability to allow out-of-band messages.
> > > 
> > > Also, touch up qmp-test.c to test the new bits.
> > > 
> > > Signed-off-by: Peter Xu <peterx@redhat.com>
> > > ---
> > >  monitor.c        | 15 +++++++++++++--
> > >  qapi-schema.json | 13 +++++++++++++
> > >  tests/qmp-test.c | 10 +++++++++-
> > >  3 files changed, 35 insertions(+), 3 deletions(-)
> > > 
> > > diff --git a/monitor.c b/monitor.c
> > > index e8f5a586e4..bad6ee8dd1 100644
> > > --- a/monitor.c
> > > +++ b/monitor.c
> > > @@ -3944,12 +3944,23 @@ void monitor_resume(Monitor *mon)
> > >  
> > >  static QObject *get_qmp_greeting(void)
> > >  {
> > > +    QDict *result = qdict_new(), *qmp = qdict_new();
> > > +    QList *cap_list = qlist_new();
> > >      QObject *ver = NULL;
> > > +    QMPCapability cap;
> > > +
> > > +    qdict_put(result, "QMP", qmp);
> > >  
> > >      qmp_marshal_query_version(NULL, &ver, NULL);
> > > +    qdict_put_obj(qmp, "version", ver);
> > > +
> > > +    for (cap = 0; cap < QMP_CAPABILITY__MAX; cap++) {
> > > +        qlist_append(cap_list, qstring_from_str(
> > > +                         QMPCapability_str(cap)));
> > > +    }
> > > +    qdict_put(qmp, "capabilities", cap_list);
> > >  
> > > -    return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': []}}",
> > > -                              ver);
> > > +    return QOBJECT(result);
> > >  }
> > 
> > Why did you replace qobject_from_jsonf() with manual qdict_*() calls?
> > 
> > I was expecting this (it's shorter and easier to read):
> > 
> >   static QObject *get_qmp_greeting(void)
> >   {
> >       QList *cap_list = qlist_new();
> >       QObject *ver = NULL;
> >       QMPCapability cap;
> > 
> >       qmp_marshal_query_version(NULL, &ver, NULL);
> > 
> >       for (cap = 0; cap < QMP_CAPABILITY__MAX; cap++) {
> >           qlist_append(cap_list, qstring_from_str(
> >                            QMPCapability_str(cap)));

And aligning the parameters would be even nicer.

> >       }
> > 
> >       return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': %p}}",
> >                                 ver, cap);
> 
> (I believe you mean s/cap/cap_list/ here?)
> 
> >   }
> 
> Oh I just didn't notice that "%p" magic at all... :(
> 
> I think for me it's fine in either way.  Frankly speaking creating the
> objects explicitly would be even easier to understand for me instead
> of using a mixture of two ways... But just let me know if you want me
> to do it your way.  I can switch.  Thanks,

I agree with Stefan here. (Readability is not judged based on how low level the
code goes when there is a higher level interface available, it's exactly the
opposite, and this doesn't change even when you happen to not know it.)

Fam

Re: [Qemu-devel] [RFC v5 12/26] qmp: negociate QMP capabilities

[Fam Zheng]

On Wed, 12/13 17:19, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:46PM +0800, Peter Xu wrote:
> > @@ -1037,8 +1038,42 @@ static void monitor_init_qmp_commands(void)
> >                           qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
> >  }
> >  
> > -void qmp_qmp_capabilities(Error **errp)
> > +static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
> > +                           Error **errp)
> > +{
> > +    for (; list; list = list->next) {
> > +        assert(list->value < QMP_CAPABILITY__MAX);
> > +        switch (list->value) {
> > +        case QMP_CAPABILITY_OOB:
> > +            if (!mon->use_io_thr) {
> > +                /*
> > +                 * Out-Of-Band only works with monitors that are
> > +                 * running on dedicated IOThread.
> > +                 */
> > +                error_setg(errp, "This monitor does not support "
> > +                           "Out-Of-Band (OOB)");
> > +                return;
> > +            }
> > +            break;
> 
> QEMU always offers the 'oob' capability, even if the monitor does not
> support it.  Should it send 'oob' only when mon->use_io_thr to make
> things easier for clients?

So should we firstly agree on whether the capabilities is on the current monitor
connection or QEMU as a whole?

Fam

Re: [Qemu-devel] [RFC v5 13/26] qmp: introduce some capability helpers

[Fam Zheng]

On Tue, 12/05 13:51, Peter Xu wrote:
> Introduce qmp_cap_enabled() and qmp_oob_enabled() helpers.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/monitor.c b/monitor.c
> index 097312e65f..9666115259 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -1038,6 +1038,16 @@ static void monitor_init_qmp_commands(void)
>                           qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
>  }
>  
> +static bool qmp_cap_enabled(Monitor *mon, QMPCapability cap)
> +{
> +    return mon->qmp.qmp_caps[cap];
> +}
> +
> +static bool qmp_oob_enabled(Monitor *mon)
> +{
> +    return qmp_cap_enabled(mon, QMP_CAPABILITY_OOB);
> +}
> +
>  static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
>                             Error **errp)
>  {
> -- 
> 2.14.3
> 

NACK, this commit doesn't commit for me:

/home/fam/work/qemu/monitor.c:1046:13: error: ‘qmp_oob_enabled’ defined but not used [-Werror=unused-function]
 static bool qmp_oob_enabled(Monitor *mon)
             ^~~~~~~~~~~~~~~
cc1: all warnings being treated as errors
  CC      x86_64-softmmu/accel/stubs/hax-stub.o

Re: [Qemu-devel] [RFC v5 19/26] qapi: introduce new cmd option "allow-oob"

[Fam Zheng]

On Tue, 12/05 13:51, Peter Xu wrote:
> --- a/qapi/introspect.json
> +++ b/qapi/introspect.json
> @@ -259,12 +259,16 @@
>  #
>  # @ret-type: the name of the command's result type.
>  #
> +# @allow-oob: whether the command allows out-of-band execution.
> +#             (Since: 2.11)

(Since: 2.12)

Fam

Re: [Qemu-devel] [RFC v5 00/26] QMP: out-of-band (OOB) execution support

[Fam Zheng]

On Tue, 12/05 13:51, Peter Xu wrote:
> This version is mostly document update, and dropped the single patch
> that is migration related (will be put into postcopy recovery
> series).

Sorry if I'm asking an already answered question, if so please remind me with
some pointers..

Can we completely hide the "OOB" concept from QMP to avoid the interface
complication? In other words, the new "migrate-recover" command can be
implemented in a way that it always try to run out of band as if it has
"run-oob=true", without user specifying it or even negotiating it.

Because it seems "migrate-recover" is only useful with "run-oob=true".

Of course without introducing the whole OOB concept in QMP, what this command
does will sound very hacky, but the bahavior can be documented just well on the
single command. I don't see a big advantage in generalizing OOB at the QMP level
so far. Do we have a list of other commands to be made OOB?

Fam

Re: [Qemu-devel] [RFC v5 00/26] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Fam Zheng (famz@redhat.com) wrote:
> On Tue, 12/05 13:51, Peter Xu wrote:
> > This version is mostly document update, and dropped the single patch
> > that is migration related (will be put into postcopy recovery
> > series).
> 
> Sorry if I'm asking an already answered question, if so please remind me with
> some pointers..
> 
> Can we completely hide the "OOB" concept from QMP to avoid the interface
> complication? In other words, the new "migrate-recover" command can be
> implemented in a way that it always try to run out of band as if it has
> "run-oob=true", without user specifying it or even negotiating it.
> 
> Because it seems "migrate-recover" is only useful with "run-oob=true".
> 
> Of course without introducing the whole OOB concept in QMP, what this command
> does will sound very hacky, but the bahavior can be documented just well on the
> single command. I don't see a big advantage in generalizing OOB at the QMP level
> so far. Do we have a list of other commands to be made OOB?

Many of the status commands can be converted to OOB, and the important
thing is that the management layer can tell whether OOB can tell which
commands are OOBable.

Dave

> Fam
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC v5 05/26] qjson: add "opaque" field to JSONMessageParser

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2438 bytes --]

On Fri, Dec 15, 2017 at 03:55:06PM +0800, Peter Xu wrote:
> On Wed, Dec 13, 2017 at 03:37:02PM +0000, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:51:39PM +0800, Peter Xu wrote:
> > > diff --git a/qga/main.c b/qga/main.c
> > > index 62a62755bd..3b5ebbc1ee 100644
> > > --- a/qga/main.c
> > > +++ b/qga/main.c
> > > @@ -593,7 +593,8 @@ static void process_command(GAState *s, QDict *req)
> > >  }
> > >  
> > >  /* handle requests/control events coming in over the channel */
> > > -static void process_event(JSONMessageParser *parser, GQueue *tokens)
> > > +static void process_event(JSONMessageParser *parser, GQueue *tokens,
> > > +                          void *opaque)
> > >  {
> > >      GAState *s = container_of(parser, GAState, parser);
> > >      QDict *qdict;
> > > @@ -1320,7 +1321,7 @@ static int run_agent(GAState *s, GAConfig *config, int socket_activation)
> > >      s->command_state = ga_command_state_new();
> > >      ga_command_state_init(s, s->command_state);
> > >      ga_command_state_init_all(s->command_state);
> > > -    json_message_parser_init(&s->parser, process_event);
> > > +    json_message_parser_init(&s->parser, process_event, NULL);
> > 
> > This patch leaves the code with 2 ways of getting at state from the
> > parser pointer:
> > 1. Use container_of() like existing users.
> > 2. Use the new (unused) opaque argument.
> > 
> > Given that #1 exists, is this patch really necessary?
> 
> I didn't really notice that.  Thanks for pointing out.
> 
> However even if so I would still prefer the opaque way to do it if
> asked.  Existing #1 of course works but IMHO is less flexible and has
> dependency between structure layouts.
> 
> How about I append another patch to convert existing users (or, I can
> post as separate patches after this series)?  It's not really a lot,
> and the conversion would be obvious:
> 
> *** qga/main.c:
> run_agent[1324]                json_message_parser_init(&s->parser, process_event, NULL);
> *** qobject/qjson.c:
> qobject_from_jsonv[45]         json_message_parser_init(&state.parser, parse_json, NULL);
> *** tests/libqtest.c:
> qmp_fd_receive[438]            json_message_parser_init(&qmp.parser, qmp_response, NULL);
> 
> Though, if you still insist, I can drop it too.

I prefer dropping it.  Smaller patch series get reviewed quicker, cause
fewer merge conflicts, are lower risk, etc.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 07/26] monitor: unify global init

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1458 bytes --]

On Fri, Dec 15, 2017 at 04:11:41PM +0800, Peter Xu wrote:
> On Wed, Dec 13, 2017 at 03:48:06PM +0000, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:51:41PM +0800, Peter Xu wrote:
> > > diff --git a/vl.c b/vl.c
> > > index 1ad1c04637..1ec995a6ae 100644
> > > --- a/vl.c
> > > +++ b/vl.c
> > > @@ -3144,7 +3144,6 @@ int main(int argc, char **argv, char **envp)
> > >      qemu_init_exec_dir(argv[0]);
> > >  
> > >      module_call_init(MODULE_INIT_QOM);
> > > -    monitor_init_qmp_commands();
> > >  
> > >      qemu_add_opts(&qemu_drive_opts);
> > >      qemu_add_drive_opts(&qemu_legacy_drive_opts);
> > > @@ -4690,6 +4689,8 @@ int main(int argc, char **argv, char **envp)
> > >      default_drive(default_floppy, snapshot, IF_FLOPPY, 0, FD_OPTS);
> > >      default_drive(default_sdcard, snapshot, IF_SD, 0, SD_OPTS);
> > >  
> > > +    monitor_init_globals();
> > 
> > Why did you move monitor_init_qmp_commands() down here?
> > 
> > There are many function calls between the old position and the new
> > position.  Did you check all of them to make sure they don't touch the
> > monitor which is now totally uninitialized?
> 
> Yeh, this patch is a bit hairy, but I really think we should do it.
> Because there are too many places (as you have seen) that we inited
> monitor stuff in different places.

But why did you move it down here?  Can you replace the
monitor_init_qmp_commands() call instead?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 09/26] monitor: create monitor dedicate iothread

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3791 bytes --]

On Fri, Dec 15, 2017 at 04:31:08PM +0800, Peter Xu wrote:
> On Wed, Dec 13, 2017 at 04:20:22PM +0000, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:51:43PM +0800, Peter Xu wrote:
> > > @@ -208,6 +209,12 @@ struct Monitor {
> > >      QTAILQ_ENTRY(Monitor) entry;
> > >  };
> > >  
> > > +struct MonitorGlobal {
> > > +    IOThread *mon_iothread;
> > > +};
> > > +
> > > +static struct MonitorGlobal mon_global;
> > 
> > structs can be anonymous.  That avoids the QEMU coding style violation
> > (structs must be typedefed):
> > 
> >   static struct {
> >       IOThread *mon_iothread;
> >   } mon_global;
> 
> Will fix this up.  Thanks.
> 
> > 
> > In general global variables are usually top-level variables in QEMU.
> > I'm not sure why wrapping globals in a struct is useful.
> 
> Because I see too many global variables for monitor code, and from
> this patch I wanted to start moving them altogether into this global
> struct.  I didn't really do that in current series because it's more
> like a clean up, but if you see future patches,

You cannot expect reviewers to jump around a 26 patch series to check
for possible future changes.  Each patch must be self-contained and the
changes need to be justified.

> I can add a comment in the commit message, like: "Let's start to
> create a struct to keep monitor global variables together".  Would
> that help?

It's better to add a comment in the code:

/* Add monitor global variables to this struct */

so that other people modifying the code know what this is about and can
participate.

Other people might not want to do it since it leads to repetitive and
long names like mon_global.mon_iothread.  Or they might just not see the
struct when defining a global.

The chance of the struct being used consistently is low and therefore I
wouldn't do it.

> > 
> > > @@ -4117,6 +4136,16 @@ void monitor_cleanup(void)
> > >  {
> > >      Monitor *mon, *next;
> > >  
> > > +    /*
> > > +     * We need to explicitly stop the iothread (but not destroy it),
> > > +     * cleanup the monitor resources, then destroy the iothread.  See
> > > +     * again on the glib bug mentioned in 2b316774f6 for a reason.
> > > +     *
> > > +     * TODO: the bug is fixed in glib 2.28, so we can remove this hack
> > > +     * as long as we won't support glib versions older than it.
> > > +     */
> > 
> > I find this comment confusing.  There is no GSource .finalize() in
> > monitor.c so why does monitor_cleanup() need to work around the bug?
> > 
> > I see that monitor_data_destroy() is not thread-safe so the IOThread
> > must be stopped first.  That is unrelated to glib.
> 
> Yeah actually that's a suggestion by Dave and Dan in previous review
> comments which makes sense to me:
> 
>   http://lists.gnu.org/archive/html/qemu-devel/2017-11/msg04344.html
> 
> I'm fine with either way: keep it as it is,

The meaning of the comment is unclear to me and you haven't been able to
explain it.  Therefore, merging this comment isn't justified.

> or instead saying
> "monitor_data_destroy() is not thread-safe" (which finally will still
> root cause to that glib bug).

This is incorrect.  The problem is that the IOThread may run chardev
handler functions while the main loop thread invokes
monitor_data_destroy().  There is nothing protecting the chardev itself
(it's not thread-safe!) nor the monitor state that is being freed, so a
running chardev handler function could crash.

This means that even without the glib bug, it's not safe to call
monitor_data_destroy() while the chardev is still attached to the
IOThread event loop.

> But how about we just keep it in case
> it may be helpful some day?

Please see above.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 12/26] qmp: negociate QMP capabilities

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1561 bytes --]

On Fri, Dec 15, 2017 at 05:40:10PM +0800, Fam Zheng wrote:
> On Wed, 12/13 17:19, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:51:46PM +0800, Peter Xu wrote:
> > > @@ -1037,8 +1038,42 @@ static void monitor_init_qmp_commands(void)
> > >                           qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
> > >  }
> > >  
> > > -void qmp_qmp_capabilities(Error **errp)
> > > +static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
> > > +                           Error **errp)
> > > +{
> > > +    for (; list; list = list->next) {
> > > +        assert(list->value < QMP_CAPABILITY__MAX);
> > > +        switch (list->value) {
> > > +        case QMP_CAPABILITY_OOB:
> > > +            if (!mon->use_io_thr) {
> > > +                /*
> > > +                 * Out-Of-Band only works with monitors that are
> > > +                 * running on dedicated IOThread.
> > > +                 */
> > > +                error_setg(errp, "This monitor does not support "
> > > +                           "Out-Of-Band (OOB)");
> > > +                return;
> > > +            }
> > > +            break;
> > 
> > QEMU always offers the 'oob' capability, even if the monitor does not
> > support it.  Should it send 'oob' only when mon->use_io_thr to make
> > things easier for clients?
> 
> So should we firstly agree on whether the capabilities is on the current monitor
> connection or QEMU as a whole?

It's more flexible to allow per-connection capabilities.  Is there a
reason against it?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 00/26] QMP: out-of-band (OOB) execution support

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1553 bytes --]

On Fri, Dec 15, 2017 at 06:41:01PM +0800, Fam Zheng wrote:
> On Tue, 12/05 13:51, Peter Xu wrote:
> > This version is mostly document update, and dropped the single patch
> > that is migration related (will be put into postcopy recovery
> > series).
> 
> Sorry if I'm asking an already answered question, if so please remind me with
> some pointers..
> 
> Can we completely hide the "OOB" concept from QMP to avoid the interface
> complication? In other words, the new "migrate-recover" command can be
> implemented in a way that it always try to run out of band as if it has
> "run-oob=true", without user specifying it or even negotiating it.
> 
> Because it seems "migrate-recover" is only useful with "run-oob=true".
> 
> Of course without introducing the whole OOB concept in QMP, what this command
> does will sound very hacky, but the bahavior can be documented just well on the
> single command. I don't see a big advantage in generalizing OOB at the QMP level
> so far. Do we have a list of other commands to be made OOB?

No, that's not possible because the batching behavior changes between
oob enabled vs disabled:

Clients that don't enable oob can send several commands at once.  They
expect them to be reliably executed in sequence.

When oob is enabled batched commands can be dropped because the queue is
full.  This is necessary because otherwise the client can fill the
monitors socket receive buffer with normal commands and then it's
impossible to get an oob command through to the monitor.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 12/26] qmp: negociate QMP capabilities

[Fam Zheng]

On Fri, 12/15 13:26, Stefan Hajnoczi wrote:
> > > QEMU always offers the 'oob' capability, even if the monitor does not
> > > support it.  Should it send 'oob' only when mon->use_io_thr to make
> > > things easier for clients?
> > 
> > So should we firstly agree on whether the capabilities is on the current monitor
> > connection or QEMU as a whole?
> 
> It's more flexible to allow per-connection capabilities.  Is there a
> reason against it?

No, I just think either way we should document it. So if we define it
per-connection, like you said, "oob" shouldn't be sent in the greeting message,
and patch 11 need to be updated.

Fam

Re: [Qemu-devel] [RFC v5 05/26] qjson: add "opaque" field to JSONMessageParser

[Peter Xu]

On Fri, Dec 15, 2017 at 12:45:24PM +0000, Stefan Hajnoczi wrote:
> On Fri, Dec 15, 2017 at 03:55:06PM +0800, Peter Xu wrote:
> > On Wed, Dec 13, 2017 at 03:37:02PM +0000, Stefan Hajnoczi wrote:
> > > On Tue, Dec 05, 2017 at 01:51:39PM +0800, Peter Xu wrote:
> > > > diff --git a/qga/main.c b/qga/main.c
> > > > index 62a62755bd..3b5ebbc1ee 100644
> > > > --- a/qga/main.c
> > > > +++ b/qga/main.c
> > > > @@ -593,7 +593,8 @@ static void process_command(GAState *s, QDict *req)
> > > >  }
> > > >  
> > > >  /* handle requests/control events coming in over the channel */
> > > > -static void process_event(JSONMessageParser *parser, GQueue *tokens)
> > > > +static void process_event(JSONMessageParser *parser, GQueue *tokens,
> > > > +                          void *opaque)
> > > >  {
> > > >      GAState *s = container_of(parser, GAState, parser);
> > > >      QDict *qdict;
> > > > @@ -1320,7 +1321,7 @@ static int run_agent(GAState *s, GAConfig *config, int socket_activation)
> > > >      s->command_state = ga_command_state_new();
> > > >      ga_command_state_init(s, s->command_state);
> > > >      ga_command_state_init_all(s->command_state);
> > > > -    json_message_parser_init(&s->parser, process_event);
> > > > +    json_message_parser_init(&s->parser, process_event, NULL);
> > > 
> > > This patch leaves the code with 2 ways of getting at state from the
> > > parser pointer:
> > > 1. Use container_of() like existing users.
> > > 2. Use the new (unused) opaque argument.
> > > 
> > > Given that #1 exists, is this patch really necessary?
> > 
> > I didn't really notice that.  Thanks for pointing out.
> > 
> > However even if so I would still prefer the opaque way to do it if
> > asked.  Existing #1 of course works but IMHO is less flexible and has
> > dependency between structure layouts.
> > 
> > How about I append another patch to convert existing users (or, I can
> > post as separate patches after this series)?  It's not really a lot,
> > and the conversion would be obvious:
> > 
> > *** qga/main.c:
> > run_agent[1324]                json_message_parser_init(&s->parser, process_event, NULL);
> > *** qobject/qjson.c:
> > qobject_from_jsonv[45]         json_message_parser_init(&state.parser, parse_json, NULL);
> > *** tests/libqtest.c:
> > qmp_fd_receive[438]            json_message_parser_init(&qmp.parser, qmp_response, NULL);
> > 
> > Though, if you still insist, I can drop it too.
> 
> I prefer dropping it.  Smaller patch series get reviewed quicker, cause
> fewer merge conflicts, are lower risk, etc.

Will do.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 07/26] monitor: unify global init

[Peter Xu]

On Fri, Dec 15, 2017 at 12:47:11PM +0000, Stefan Hajnoczi wrote:
> On Fri, Dec 15, 2017 at 04:11:41PM +0800, Peter Xu wrote:
> > On Wed, Dec 13, 2017 at 03:48:06PM +0000, Stefan Hajnoczi wrote:
> > > On Tue, Dec 05, 2017 at 01:51:41PM +0800, Peter Xu wrote:
> > > > diff --git a/vl.c b/vl.c
> > > > index 1ad1c04637..1ec995a6ae 100644
> > > > --- a/vl.c
> > > > +++ b/vl.c
> > > > @@ -3144,7 +3144,6 @@ int main(int argc, char **argv, char **envp)
> > > >      qemu_init_exec_dir(argv[0]);
> > > >  
> > > >      module_call_init(MODULE_INIT_QOM);
> > > > -    monitor_init_qmp_commands();
> > > >  
> > > >      qemu_add_opts(&qemu_drive_opts);
> > > >      qemu_add_drive_opts(&qemu_legacy_drive_opts);
> > > > @@ -4690,6 +4689,8 @@ int main(int argc, char **argv, char **envp)
> > > >      default_drive(default_floppy, snapshot, IF_FLOPPY, 0, FD_OPTS);
> > > >      default_drive(default_sdcard, snapshot, IF_SD, 0, SD_OPTS);
> > > >  
> > > > +    monitor_init_globals();
> > > 
> > > Why did you move monitor_init_qmp_commands() down here?
> > > 
> > > There are many function calls between the old position and the new
> > > position.  Did you check all of them to make sure they don't touch the
> > > monitor which is now totally uninitialized?
> > 
> > Yeh, this patch is a bit hairy, but I really think we should do it.
> > Because there are too many places (as you have seen) that we inited
> > monitor stuff in different places.
> 
> But why did you move it down here?  Can you replace the
> monitor_init_qmp_commands() call instead?

I just moved it closer to (actually, right above) initializations of
monitors to be clear that these globals will never be touched until
this point.  Or do you want me to move this call exactly back to the
place where monitor_init_qmp_commands() was called before?  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 11/26] qmp: introduce QMPCapability

[Peter Xu]

On Fri, Dec 15, 2017 at 05:38:03PM +0800, Fam Zheng wrote:
> On Fri, 12/15 17:14, Peter Xu wrote:
> > On Wed, Dec 13, 2017 at 04:56:51PM +0000, Stefan Hajnoczi wrote:
> > > On Tue, Dec 05, 2017 at 01:51:45PM +0800, Peter Xu wrote:
> > > > There was no QMP capabilities defined.  Define the first "oob" as
> > > > capability to allow out-of-band messages.
> > > > 
> > > > Also, touch up qmp-test.c to test the new bits.
> > > > 
> > > > Signed-off-by: Peter Xu <peterx@redhat.com>
> > > > ---
> > > >  monitor.c        | 15 +++++++++++++--
> > > >  qapi-schema.json | 13 +++++++++++++
> > > >  tests/qmp-test.c | 10 +++++++++-
> > > >  3 files changed, 35 insertions(+), 3 deletions(-)
> > > > 
> > > > diff --git a/monitor.c b/monitor.c
> > > > index e8f5a586e4..bad6ee8dd1 100644
> > > > --- a/monitor.c
> > > > +++ b/monitor.c
> > > > @@ -3944,12 +3944,23 @@ void monitor_resume(Monitor *mon)
> > > >  
> > > >  static QObject *get_qmp_greeting(void)
> > > >  {
> > > > +    QDict *result = qdict_new(), *qmp = qdict_new();
> > > > +    QList *cap_list = qlist_new();
> > > >      QObject *ver = NULL;
> > > > +    QMPCapability cap;
> > > > +
> > > > +    qdict_put(result, "QMP", qmp);
> > > >  
> > > >      qmp_marshal_query_version(NULL, &ver, NULL);
> > > > +    qdict_put_obj(qmp, "version", ver);
> > > > +
> > > > +    for (cap = 0; cap < QMP_CAPABILITY__MAX; cap++) {
> > > > +        qlist_append(cap_list, qstring_from_str(
> > > > +                         QMPCapability_str(cap)));
> > > > +    }
> > > > +    qdict_put(qmp, "capabilities", cap_list);
> > > >  
> > > > -    return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': []}}",
> > > > -                              ver);
> > > > +    return QOBJECT(result);
> > > >  }
> > > 
> > > Why did you replace qobject_from_jsonf() with manual qdict_*() calls?
> > > 
> > > I was expecting this (it's shorter and easier to read):
> > > 
> > >   static QObject *get_qmp_greeting(void)
> > >   {
> > >       QList *cap_list = qlist_new();
> > >       QObject *ver = NULL;
> > >       QMPCapability cap;
> > > 
> > >       qmp_marshal_query_version(NULL, &ver, NULL);
> > > 
> > >       for (cap = 0; cap < QMP_CAPABILITY__MAX; cap++) {
> > >           qlist_append(cap_list, qstring_from_str(
> > >                            QMPCapability_str(cap)));
> 
> And aligning the parameters would be even nicer.
> 
> > >       }
> > > 
> > >       return qobject_from_jsonf("{'QMP': {'version': %p, 'capabilities': %p}}",
> > >                                 ver, cap);
> > 
> > (I believe you mean s/cap/cap_list/ here?)
> > 
> > >   }
> > 
> > Oh I just didn't notice that "%p" magic at all... :(
> > 
> > I think for me it's fine in either way.  Frankly speaking creating the
> > objects explicitly would be even easier to understand for me instead
> > of using a mixture of two ways... But just let me know if you want me
> > to do it your way.  I can switch.  Thanks,
> 
> I agree with Stefan here. (Readability is not judged based on how low level the
> code goes when there is a higher level interface available, it's exactly the
> opposite, and this doesn't change even when you happen to not know it.)

I was talking mostly about using two ways or using only one way to do
this (especially that's quite simple even using raw objects...),
rather than which one I know. :)

Again, since I'm happy with either way actually, I'm giving in.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 09/26] monitor: create monitor dedicate iothread

[Peter Xu]

On Fri, Dec 15, 2017 at 01:21:42PM +0000, Stefan Hajnoczi wrote:
> On Fri, Dec 15, 2017 at 04:31:08PM +0800, Peter Xu wrote:
> > On Wed, Dec 13, 2017 at 04:20:22PM +0000, Stefan Hajnoczi wrote:
> > > On Tue, Dec 05, 2017 at 01:51:43PM +0800, Peter Xu wrote:
> > > > @@ -208,6 +209,12 @@ struct Monitor {
> > > >      QTAILQ_ENTRY(Monitor) entry;
> > > >  };
> > > >  
> > > > +struct MonitorGlobal {
> > > > +    IOThread *mon_iothread;
> > > > +};
> > > > +
> > > > +static struct MonitorGlobal mon_global;
> > > 
> > > structs can be anonymous.  That avoids the QEMU coding style violation
> > > (structs must be typedefed):
> > > 
> > >   static struct {
> > >       IOThread *mon_iothread;
> > >   } mon_global;
> > 
> > Will fix this up.  Thanks.
> > 
> > > 
> > > In general global variables are usually top-level variables in QEMU.
> > > I'm not sure why wrapping globals in a struct is useful.
> > 
> > Because I see too many global variables for monitor code, and from
> > this patch I wanted to start moving them altogether into this global
> > struct.  I didn't really do that in current series because it's more
> > like a clean up, but if you see future patches,
> 
> You cannot expect reviewers to jump around a 26 patch series to check
> for possible future changes.  Each patch must be self-contained and the
> changes need to be justified.

Noted.

> 
> > I can add a comment in the commit message, like: "Let's start to
> > create a struct to keep monitor global variables together".  Would
> > that help?
> 
> It's better to add a comment in the code:
> 
> /* Add monitor global variables to this struct */
> 
> so that other people modifying the code know what this is about and can
> participate.

Will do.

> 
> Other people might not want to do it since it leads to repetitive and
> long names like mon_global.mon_iothread.  Or they might just not see the
> struct when defining a global.
> 
> The chance of the struct being used consistently is low and therefore I
> wouldn't do it.
> 
> > > 
> > > > @@ -4117,6 +4136,16 @@ void monitor_cleanup(void)
> > > >  {
> > > >      Monitor *mon, *next;
> > > >  
> > > > +    /*
> > > > +     * We need to explicitly stop the iothread (but not destroy it),
> > > > +     * cleanup the monitor resources, then destroy the iothread.  See
> > > > +     * again on the glib bug mentioned in 2b316774f6 for a reason.
> > > > +     *
> > > > +     * TODO: the bug is fixed in glib 2.28, so we can remove this hack
> > > > +     * as long as we won't support glib versions older than it.
> > > > +     */
> > > 
> > > I find this comment confusing.  There is no GSource .finalize() in
> > > monitor.c so why does monitor_cleanup() need to work around the bug?
> > > 
> > > I see that monitor_data_destroy() is not thread-safe so the IOThread
> > > must be stopped first.  That is unrelated to glib.
> > 
> > Yeah actually that's a suggestion by Dave and Dan in previous review
> > comments which makes sense to me:
> > 
> >   http://lists.gnu.org/archive/html/qemu-devel/2017-11/msg04344.html
> > 
> > I'm fine with either way: keep it as it is,
> 
> The meaning of the comment is unclear to me and you haven't been able to
> explain it.  Therefore, merging this comment isn't justified.

(Please see below)

> 
> > or instead saying
> > "monitor_data_destroy() is not thread-safe" (which finally will still
> > root cause to that glib bug).
> 
> This is incorrect.  The problem is that the IOThread may run chardev
> handler functions while the main loop thread invokes
> monitor_data_destroy().  There is nothing protecting the chardev itself
> (it's not thread-safe!) nor the monitor state that is being freed, so a
> running chardev handler function could crash.

It's only about a single line of comment, but since we are at this, I
think it would still be good to discuss it in case I was wrong.

Firstly, I agree that chardevs are not thread-safe.  But IMHO monitors
are thread-safe.  There is the big monitor_lock to protect.  There can
be bug though, but generally speaking that lock should be doing the
thread safety work.

Next, basically when destroying the monitors logically we should never
touch the chardev if without that glib bug.  Or say, if without the
bug we should not call qemu_chr_fe_deinit() in monitor_data_destroy().
And if so, monitor does not really touch chardev at all.  Then, I do
think that we can destroy the monitor even without stopping the
iothread, and it is pretty safe.  Note that again that should be
ensured by the monitor_lock.

Now the problem is that to support the old glib versions we must call
qemu_chr_fe_deinit().  So the comment may help if one day that is not
true any more, and I do think we can remove the iothread_stop()
without any crashes.

(but I agree even keeping it forever it's not a big matter... same
 thing to the comment itself...)

As I said, I may be wrong.  Please correct me if necessary.  Thanks,

> 
> This means that even without the glib bug, it's not safe to call
> monitor_data_destroy() while the chardev is still attached to the
> IOThread event loop.
> 
> > But how about we just keep it in case
> > it may be helpful some day?
> 
> Please see above.
> 
> Stefan

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 09/26] monitor: create monitor dedicate iothread

[Peter Xu]

On Sat, Dec 16, 2017 at 12:42:00PM +0800, Peter Xu wrote:
> On Fri, Dec 15, 2017 at 01:21:42PM +0000, Stefan Hajnoczi wrote:
> > On Fri, Dec 15, 2017 at 04:31:08PM +0800, Peter Xu wrote:
> > > On Wed, Dec 13, 2017 at 04:20:22PM +0000, Stefan Hajnoczi wrote:
> > > > On Tue, Dec 05, 2017 at 01:51:43PM +0800, Peter Xu wrote:
> > > > > @@ -208,6 +209,12 @@ struct Monitor {
> > > > >      QTAILQ_ENTRY(Monitor) entry;
> > > > >  };
> > > > >  
> > > > > +struct MonitorGlobal {
> > > > > +    IOThread *mon_iothread;
> > > > > +};
> > > > > +
> > > > > +static struct MonitorGlobal mon_global;
> > > > 
> > > > structs can be anonymous.  That avoids the QEMU coding style violation
> > > > (structs must be typedefed):
> > > > 
> > > >   static struct {
> > > >       IOThread *mon_iothread;
> > > >   } mon_global;
> > > 
> > > Will fix this up.  Thanks.
> > > 
> > > > 
> > > > In general global variables are usually top-level variables in QEMU.
> > > > I'm not sure why wrapping globals in a struct is useful.
> > > 
> > > Because I see too many global variables for monitor code, and from
> > > this patch I wanted to start moving them altogether into this global
> > > struct.  I didn't really do that in current series because it's more
> > > like a clean up, but if you see future patches,
> > 
> > You cannot expect reviewers to jump around a 26 patch series to check
> > for possible future changes.  Each patch must be self-contained and the
> > changes need to be justified.
> 
> Noted.
> 
> > 
> > > I can add a comment in the commit message, like: "Let's start to
> > > create a struct to keep monitor global variables together".  Would
> > > that help?
> > 
> > It's better to add a comment in the code:
> > 
> > /* Add monitor global variables to this struct */
> > 
> > so that other people modifying the code know what this is about and can
> > participate.
> 
> Will do.
> 
> > 
> > Other people might not want to do it since it leads to repetitive and
> > long names like mon_global.mon_iothread.  Or they might just not see the
> > struct when defining a global.
> > 
> > The chance of the struct being used consistently is low and therefore I
> > wouldn't do it.
> > 
> > > > 
> > > > > @@ -4117,6 +4136,16 @@ void monitor_cleanup(void)
> > > > >  {
> > > > >      Monitor *mon, *next;
> > > > >  
> > > > > +    /*
> > > > > +     * We need to explicitly stop the iothread (but not destroy it),
> > > > > +     * cleanup the monitor resources, then destroy the iothread.  See
> > > > > +     * again on the glib bug mentioned in 2b316774f6 for a reason.
> > > > > +     *
> > > > > +     * TODO: the bug is fixed in glib 2.28, so we can remove this hack
> > > > > +     * as long as we won't support glib versions older than it.
> > > > > +     */
> > > > 
> > > > I find this comment confusing.  There is no GSource .finalize() in
> > > > monitor.c so why does monitor_cleanup() need to work around the bug?
> > > > 
> > > > I see that monitor_data_destroy() is not thread-safe so the IOThread
> > > > must be stopped first.  That is unrelated to glib.
> > > 
> > > Yeah actually that's a suggestion by Dave and Dan in previous review
> > > comments which makes sense to me:
> > > 
> > >   http://lists.gnu.org/archive/html/qemu-devel/2017-11/msg04344.html
> > > 
> > > I'm fine with either way: keep it as it is,
> > 
> > The meaning of the comment is unclear to me and you haven't been able to
> > explain it.  Therefore, merging this comment isn't justified.
> 
> (Please see below)
> 
> > 
> > > or instead saying
> > > "monitor_data_destroy() is not thread-safe" (which finally will still
> > > root cause to that glib bug).
> > 
> > This is incorrect.  The problem is that the IOThread may run chardev
> > handler functions while the main loop thread invokes
> > monitor_data_destroy().  There is nothing protecting the chardev itself
> > (it's not thread-safe!) nor the monitor state that is being freed, so a
> > running chardev handler function could crash.
> 
> It's only about a single line of comment, but since we are at this, I
> think it would still be good to discuss it in case I was wrong.
> 
> Firstly, I agree that chardevs are not thread-safe.  But IMHO monitors
> are thread-safe.  There is the big monitor_lock to protect.  There can
> be bug though, but generally speaking that lock should be doing the
> thread safety work.
> 
> Next, basically when destroying the monitors logically we should never
> touch the chardev if without that glib bug.  Or say, if without the
> bug we should not call qemu_chr_fe_deinit() in monitor_data_destroy().

Ouch. :-/

I was meaning remove_fd_in_watch() in qemu_chr_fe_set_handlers().  Of
course it needs to touch chardev to unregister the stuff. :-)

And I think you are right.  Let me remove the comment.  The
iothread_stop() needs to be there always, even without that bug.

Sorry for the noise.

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 12/26] qmp: negociate QMP capabilities

[Peter Xu]

On Fri, Dec 15, 2017 at 09:53:53PM +0800, Fam Zheng wrote:
> On Fri, 12/15 13:26, Stefan Hajnoczi wrote:
> > > > QEMU always offers the 'oob' capability, even if the monitor does not
> > > > support it.  Should it send 'oob' only when mon->use_io_thr to make
> > > > things easier for clients?

Agree.

> > > 
> > > So should we firstly agree on whether the capabilities is on the current monitor
> > > connection or QEMU as a whole?
> > 
> > It's more flexible to allow per-connection capabilities.  Is there a
> > reason against it?
> 
> No, I just think either way we should document it. So if we define it
> per-connection, like you said, "oob" shouldn't be sent in the greeting message,
> and patch 11 need to be updated.

Yeah, that's where I'll do the update.  Thanks!

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 13/26] qmp: introduce some capability helpers

[Peter Xu]

On Fri, Dec 15, 2017 at 05:42:55PM +0800, Fam Zheng wrote:
> On Tue, 12/05 13:51, Peter Xu wrote:
> > Introduce qmp_cap_enabled() and qmp_oob_enabled() helpers.
> > 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  monitor.c | 10 ++++++++++
> >  1 file changed, 10 insertions(+)
> > 
> > diff --git a/monitor.c b/monitor.c
> > index 097312e65f..9666115259 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -1038,6 +1038,16 @@ static void monitor_init_qmp_commands(void)
> >                           qmp_marshal_qmp_capabilities, QCO_NO_OPTIONS);
> >  }
> >  
> > +static bool qmp_cap_enabled(Monitor *mon, QMPCapability cap)
> > +{
> > +    return mon->qmp.qmp_caps[cap];
> > +}
> > +
> > +static bool qmp_oob_enabled(Monitor *mon)
> > +{
> > +    return qmp_cap_enabled(mon, QMP_CAPABILITY_OOB);
> > +}
> > +
> >  static void qmp_caps_check(Monitor *mon, QMPCapabilityList *list,
> >                             Error **errp)
> >  {
> > -- 
> > 2.14.3
> > 
> 
> NACK, this commit doesn't commit for me:
> 
> /home/fam/work/qemu/monitor.c:1046:13: error: ‘qmp_oob_enabled’ defined but not used [-Werror=unused-function]
>  static bool qmp_oob_enabled(Monitor *mon)
>              ^~~~~~~~~~~~~~~
> cc1: all warnings being treated as errors
>   CC      x86_64-softmmu/accel/stubs/hax-stub.o

I'll squash this patch to patch 16, where it started to be used.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 14/26] monitor: introduce monitor_qmp_respond()

[Peter Xu]

On Wed, Dec 13, 2017 at 05:35:33PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:48PM +0800, Peter Xu wrote:
> > A tiny refactoring, preparing to split the QMP dispatcher away.
> > 
> > Reviewed-by: Fam Zheng <famz@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  monitor.c | 48 +++++++++++++++++++++++++++++++-----------------
> >  1 file changed, 31 insertions(+), 17 deletions(-)
> > 
> > diff --git a/monitor.c b/monitor.c
> > index 9666115259..35d8925636 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -3877,6 +3877,36 @@ static int monitor_can_read(void *opaque)
> >      return (mon->suspend_cnt == 0) ? 1 : 0;
> >  }
> >  
> > +/*
> > + * When rsp/err/id is passed in, the function will be responsible for
> > + * the cleanup.
> > + */
> 
> Please document this function fully.  I had to look at the
> implementation to learn how the arguments work:
> 
> 1. This function takes ownership of rsp, err, and id.  (Please use this
> exact wording because the function does not perform the "cleanup" when
> refcount > 1.)
> 2. rsp, err, and id may be NULL.
> 3. If err != NULL then rsp must be NULL.

Will steal the lines.  Thanks!

Also, I'll add an assert() to make sure (3) is true, in case any rsp
refcount is leaked when accidentally provided with err.

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 15/26] monitor: let suspend_cnt be thread safe

[Peter Xu]

On Wed, Dec 13, 2017 at 06:43:58PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:49PM +0800, Peter Xu wrote:
> > Monitor code now can be run in more than one thread.  Let it be thread
> > safe when accessing suspend_cnt counter.
> 
> Please add doc comments to the functions explaining which thread they
> may be called from (especially public functions).  Without this
> information other people will have a hard time keeping code thread-safe.

The two public functions would be monitor_resume() and
monitor_suspend().  IIUC as long as these functions are using atomic
operations correctly they should be able to be called anywhere?

Could you help point out what I missed?

> 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  monitor.c | 13 +++++++------
> >  1 file changed, 7 insertions(+), 6 deletions(-)
> 
> From docs/devel/atomics.txt:
> 
>   Macros defined by qemu/atomic.h fall in three camps:
> 
>   - compiler barriers: barrier();
> 
>   - weak atomic access and manual memory barriers: atomic_read(),
>     atomic_set(), smp_rmb(), smp_wmb(), smp_mb(), smp_mb_acquire(),
>     smp_mb_release(), smp_read_barrier_depends();
> 
>   - sequentially consistent atomic access: everything else.
> 
> The atomic_read() and atomic_set() calls in this patch aren't effective
> without memory barriers.  Please use atomic_mb_read() and
> atomic_mb_set() instead.

Will do.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 16/26] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Wed, Dec 13, 2017 at 08:09:38PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:50PM +0800, Peter Xu wrote:
> > @@ -3956,12 +3968,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> >          }
> >      }
> >  
> > -err_out:
> > -    monitor_qmp_respond(mon, rsp, err, id);
> > +    /* Respond if necessary */
> > +    monitor_qmp_respond(mon, rsp, NULL, id);
> > +
> > +    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
> > +    if (!qmp_oob_enabled(mon)) {
> > +        monitor_resume(mon);
> 
> monitor_resume() does not work between threads: if the event loop is
> currently blocked in poll() it won't notice that the monitor fd should
> be watched again.
> 
> Please add aio_notify() to monitor_resume() and monitor_suspend().  That
> way the event loop is forced to check can_read() again.

Ah, yes.  I think monitor_suspend() does not need the notify?  Since
if it's sleeping it won't miss the next check in can_read() after all?

Regarding to monitor_resume(), I noticed that I missed the fact that
it's only tailored for HMP before, which seems to be a bigger problem.
Do you agree with a change like this?

diff --git a/monitor.c b/monitor.c
index 9970418d6f..8f96880ad7 100644
--- a/monitor.c
+++ b/monitor.c
@@ -4244,10 +4244,12 @@ int monitor_suspend(Monitor *mon)
 
 void monitor_resume(Monitor *mon)
 {
-    if (!mon->rs)
-        return;
     if (atomic_dec_fetch(&mon->suspend_cnt) == 0) {
-        readline_show_prompt(mon->rs);
+        if (monitor_is_qmp(mon)) {
+            aio_notify(mon_global.mon_iothread->ctx);
+        } else {
+            assert(mon->rs);
+            readline_show_prompt(mon->rs);
+        }
     }
 }

Even, I'm thinking about whether I should start to introduce
iothread_notify() now to mask out the IOThread.ctx details.

> 
> > +    }
> >  
> >      qobject_decref(req);
> >  }
> >  
> > +/*
> > + * Pop one QMP request from monitor queues, return NULL if not found.
> > + * We are using round-robin fasion to pop the request, to avoid
> > + * processing command only on a very busy monitor.  To achieve that,
> > + * when we processed one request on specific monitor, we put that
> > + * monitor to the end of mon_list queue.
> > + */
> > +static QMPRequest *monitor_qmp_requests_pop_one(void)
> > +{
> > +    QMPRequest *req_obj = NULL;
> > +    Monitor *mon;
> > +
> > +    qemu_mutex_lock(&monitor_lock);
> > +
> > +    QTAILQ_FOREACH(mon, &mon_list, entry) {
> > +        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> 
> Please add a doc comment about the monitor_lock < qmp_queue_lock lock
> ordering in the qmp_queue_lock field declaration so that deadlocks can
> be prevented.

Will do.

> 
> > +        req_obj = g_queue_pop_head(mon->qmp.qmp_requests);
> > +        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> > +        if (req_obj) {
> > +            break;
> > +        }
> > +    }
> > +
> > +    if (req_obj) {
> > +        /*
> > +         * We found one request on the monitor. Degrade this monitor's
> > +         * priority to lowest by re-inserting it to end of queue.
> > +         */
> > +        QTAILQ_REMOVE(&mon_list, mon, entry);
> > +        QTAILQ_INSERT_TAIL(&mon_list, mon, entry);
> > +    }
> > +
> > +    qemu_mutex_unlock(&monitor_lock);
> > +
> > +    return req_obj;
> > +}
> > +
> > +static void monitor_qmp_bh_dispatcher(void *data)
> > +{
> > +    QMPRequest *req_obj;
> > +
> > +    while (true) {
> 
> Previously QEMU could only dispatch 1 QMP command per main loop
> iteration.  Now multiple requests can be processed in a single main loop
> iteration.
> 
> If a producer enqueues requests faster than the dispatcher can dequeue
> them then this is infinite loop will prevent the caller (i.e.  QEMU main
> loop) from making progress.
> 
> The following keeps 1 QMP command per main loop iteration and avoids the
> infinite loop:
> 
>   static void monitor_qmp_bh_dispatcher(void *data)
>   {
>       QMPRequest *req_obj = monitor_qmp_requests_pop_one();
> 
>       if (req_obj) {
>           monitor_qmp_dispatch_one(req_obj);
> 
> 	  /* Reschedule instead of looping so the main loop stays responsive */
>           qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
>       }
>   }

Yes, this sounds better!

> 
> > +    /*
> > +     * Put the request to the end of queue so that requests will be
> > +     * handled in time order.  Ownership for req_obj, req, id,
> > +     * etc. will be delivered to the handler side.
> > +     */
> > +    qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> > +    g_queue_push_tail(mon->qmp.qmp_requests, req_obj);
> > +    qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> > +
> > +    /* Kick the dispatcher routine */
> > +    qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> > +
> > +    /*
> > +     * If OOB is not enabled on current monitor, we'll emulate the old
> > +     * behavior that we won't process current monitor any more until
> > +     * it is responded.  This helps make sure that as long as OOB is
> > +     * not enabled, the server will never drop any command.
> > +     */
> > +    if (!qmp_oob_enabled(mon)) {
> > +        monitor_suspend(mon);
> > +    }
> 
> monitor_suspend() must be called before g_queue_push_tail().  Otherwise
> the other thread might complete the request and call monitor_resume()
> before we call monitor_suspend().

Reasonable.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 16/26] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Sat, Dec 16, 2017 at 02:37:03PM +0800, Peter Xu wrote:
> On Wed, Dec 13, 2017 at 08:09:38PM +0000, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:51:50PM +0800, Peter Xu wrote:
> > > @@ -3956,12 +3968,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> > >          }
> > >      }
> > >  
> > > -err_out:
> > > -    monitor_qmp_respond(mon, rsp, err, id);
> > > +    /* Respond if necessary */
> > > +    monitor_qmp_respond(mon, rsp, NULL, id);
> > > +
> > > +    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
> > > +    if (!qmp_oob_enabled(mon)) {
> > > +        monitor_resume(mon);
> > 
> > monitor_resume() does not work between threads: if the event loop is
> > currently blocked in poll() it won't notice that the monitor fd should
> > be watched again.
> > 
> > Please add aio_notify() to monitor_resume() and monitor_suspend().  That
> > way the event loop is forced to check can_read() again.
> 
> Ah, yes.  I think monitor_suspend() does not need the notify?  Since
> if it's sleeping it won't miss the next check in can_read() after all?
> 
> Regarding to monitor_resume(), I noticed that I missed the fact that
> it's only tailored for HMP before, which seems to be a bigger problem.
> Do you agree with a change like this?
> 
> diff --git a/monitor.c b/monitor.c
> index 9970418d6f..8f96880ad7 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -4244,10 +4244,12 @@ int monitor_suspend(Monitor *mon)
>  
>  void monitor_resume(Monitor *mon)
>  {
> -    if (!mon->rs)
> -        return;
>      if (atomic_dec_fetch(&mon->suspend_cnt) == 0) {
> -        readline_show_prompt(mon->rs);
> +        if (monitor_is_qmp(mon)) {
> +            aio_notify(mon_global.mon_iothread->ctx);
> +        } else {
> +            assert(mon->rs);
> +            readline_show_prompt(mon->rs);
> +        }
>      }
>  }

And monitor_suspend() needs to be touched too to support QMP, which I
missed for sure... I plan to add a new patch for this:

>From d020db4529deccf6777f6b5a0f7ff9330d4de42c Mon Sep 17 00:00:00 2001
From: Peter Xu <peterx@redhat.com>
Date: Sat, 16 Dec 2017 14:42:04 +0800
Subject: [PATCH] monitor: let suspend/resume work even with QMPs

One thing to mention is that for QMPs that are using IOThreads, we need
an explicit kick for the IOThread in case it is sleeping.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/monitor.c b/monitor.c
index 9970418d6f..085146e84c 100644
--- a/monitor.c
+++ b/monitor.c
@@ -4236,18 +4236,24 @@ static void monitor_command_cb(void *opaque, const char *cmdline,

 int monitor_suspend(Monitor *mon)
 {
-    if (!mon->rs)
-        return -ENOTTY;
+    /* This will be effective in the next monitor_can_read() check */
     atomic_inc(&mon->suspend_cnt);
     return 0;
 }

 void monitor_resume(Monitor *mon)
 {
-    if (!mon->rs)
-        return;
     if (atomic_dec_fetch(&mon->suspend_cnt) == 0) {
-        readline_show_prompt(mon->rs);
+        if (monitor_is_qmp(mon) && mon->use_io_thr) {
+            /*
+             * For QMP monitors that are running in IOThread, let's
+             * kick the thread in case it's sleeping.
+             */
+            aio_notify(mon_global.mon_iothread->ctx);
+        } else {
+            assert(mon->rs);
+            readline_show_prompt(mon->rs);
+        }
     }
 }

Thanks,

> 
> Even, I'm thinking about whether I should start to introduce
> iothread_notify() now to mask out the IOThread.ctx details.
> 
> > 
> > > +    }
> > >  
> > >      qobject_decref(req);
> > >  }
> > >  
> > > +/*
> > > + * Pop one QMP request from monitor queues, return NULL if not found.
> > > + * We are using round-robin fasion to pop the request, to avoid
> > > + * processing command only on a very busy monitor.  To achieve that,
> > > + * when we processed one request on specific monitor, we put that
> > > + * monitor to the end of mon_list queue.
> > > + */
> > > +static QMPRequest *monitor_qmp_requests_pop_one(void)
> > > +{
> > > +    QMPRequest *req_obj = NULL;
> > > +    Monitor *mon;
> > > +
> > > +    qemu_mutex_lock(&monitor_lock);
> > > +
> > > +    QTAILQ_FOREACH(mon, &mon_list, entry) {
> > > +        qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> > 
> > Please add a doc comment about the monitor_lock < qmp_queue_lock lock
> > ordering in the qmp_queue_lock field declaration so that deadlocks can
> > be prevented.
> 
> Will do.
> 
> > 
> > > +        req_obj = g_queue_pop_head(mon->qmp.qmp_requests);
> > > +        qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> > > +        if (req_obj) {
> > > +            break;
> > > +        }
> > > +    }
> > > +
> > > +    if (req_obj) {
> > > +        /*
> > > +         * We found one request on the monitor. Degrade this monitor's
> > > +         * priority to lowest by re-inserting it to end of queue.
> > > +         */
> > > +        QTAILQ_REMOVE(&mon_list, mon, entry);
> > > +        QTAILQ_INSERT_TAIL(&mon_list, mon, entry);
> > > +    }
> > > +
> > > +    qemu_mutex_unlock(&monitor_lock);
> > > +
> > > +    return req_obj;
> > > +}
> > > +
> > > +static void monitor_qmp_bh_dispatcher(void *data)
> > > +{
> > > +    QMPRequest *req_obj;
> > > +
> > > +    while (true) {
> > 
> > Previously QEMU could only dispatch 1 QMP command per main loop
> > iteration.  Now multiple requests can be processed in a single main loop
> > iteration.
> > 
> > If a producer enqueues requests faster than the dispatcher can dequeue
> > them then this is infinite loop will prevent the caller (i.e.  QEMU main
> > loop) from making progress.
> > 
> > The following keeps 1 QMP command per main loop iteration and avoids the
> > infinite loop:
> > 
> >   static void monitor_qmp_bh_dispatcher(void *data)
> >   {
> >       QMPRequest *req_obj = monitor_qmp_requests_pop_one();
> > 
> >       if (req_obj) {
> >           monitor_qmp_dispatch_one(req_obj);
> > 
> > 	  /* Reschedule instead of looping so the main loop stays responsive */
> >           qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> >       }
> >   }
> 
> Yes, this sounds better!
> 
> > 
> > > +    /*
> > > +     * Put the request to the end of queue so that requests will be
> > > +     * handled in time order.  Ownership for req_obj, req, id,
> > > +     * etc. will be delivered to the handler side.
> > > +     */
> > > +    qemu_mutex_lock(&mon->qmp.qmp_queue_lock);
> > > +    g_queue_push_tail(mon->qmp.qmp_requests, req_obj);
> > > +    qemu_mutex_unlock(&mon->qmp.qmp_queue_lock);
> > > +
> > > +    /* Kick the dispatcher routine */
> > > +    qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
> > > +
> > > +    /*
> > > +     * If OOB is not enabled on current monitor, we'll emulate the old
> > > +     * behavior that we won't process current monitor any more until
> > > +     * it is responded.  This helps make sure that as long as OOB is
> > > +     * not enabled, the server will never drop any command.
> > > +     */
> > > +    if (!qmp_oob_enabled(mon)) {
> > > +        monitor_suspend(mon);
> > > +    }
> > 
> > monitor_suspend() must be called before g_queue_push_tail().  Otherwise
> > the other thread might complete the request and call monitor_resume()
> > before we call monitor_suspend().
> 
> Reasonable.  Thanks,
> 
> -- 
> Peter Xu

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 17/26] qmp: add new event "request-dropped"

[Peter Xu]

On Thu, Dec 14, 2017 at 11:16:02AM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:51PM +0800, Peter Xu wrote:
> > +##
> > +# @RequestDropReason:
> > +#
> > +# Reasons that caused one command to be dropped.
> > +#
> > +# @queue-full: the queue of request is full.
> > +#
> > +# Since: 2.12
> > +##
> > +{ 'enum': 'RequestDropReason',
> 
> qapi-schema.json calls them "commands", not "requests".  Please rename
> it CommandDropReason for consistency and update the documentation.
> 
> > +  'data': [ 'queue-full' ] }
> > +
> > +##
> > +# @REQUEST_DROPPED:
> > +#
> > +# Emitted when one QMP request is dropped due to some reason.
> > +# REQUEST_DROPPED is only emitted when the oob capability is enabled.
> 
> Please make it clear that requests can only be dropped when the oob
> capability is enabled.
> 
>   Emitted when a command is dropped due to some reason.  Commands can
>   only be dropped when the oob capability is enabled.

Will address all comments, including the other reply.

Meanwhile I'll rename REQUEST_DROPPED to COMMAND_DROPPED so no
"request" but all "command" then.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 18/26] monitor: send event when request queue full

[Peter Xu]

On Thu, Dec 14, 2017 at 11:41:36AM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:52PM +0800, Peter Xu wrote:
> > Set maximum QMP request queue length to 8.  If queue full, instead of
> > queue the command, we directly return a "request-dropped" event, telling
> > client that specific command is dropped.
> > 
> > Note that this flow control mechanism is only valid if OOB is enabled.
> > If it's not, the effective queue length will always be 1, which strictly
> > follows original behavior of QMP command handling (which never drop
> > messages).
> > 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  monitor.c | 15 +++++++++++++++
> >  1 file changed, 15 insertions(+)
> > 
> > diff --git a/monitor.c b/monitor.c
> > index c20e659740..f7923c4590 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -4029,6 +4029,8 @@ static void monitor_qmp_bh_dispatcher(void *data)
> >      }
> >  }
> >  
> > +#define  QMP_REQ_QUEUE_LEN_MAX  (8)
> > +
> >  static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> >                                 void *opaque)
> >  {
> > @@ -4061,6 +4063,19 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> >      req_obj->id = id;
> >      req_obj->req = req;
> >  
> > +    if (qmp_oob_enabled(mon)) {
> > +        /* Drop the request if queue is full. */
> > +        if (mon->qmp.qmp_requests->length >= QMP_REQ_QUEUE_LEN_MAX) {
> 
> qmp_queue_lock must be held.  Perhaps it's simplest to move this check
> into the qmp_queue_lock critical section below and unlock before calling
> qapi_event_send_request_dropped()...
> 
> > +            qapi_event_send_request_dropped(id,
> > +                                            REQUEST_DROP_REASON_QUEUE_FULL,
> > +                                            NULL);
> > +            qobject_decref(id);
> > +            qobject_decref(req);
> > +            g_free(req_obj);
> > +            return;
> > +        }
> > +    }
> > +
> >      /*
> >       * Put the request to the end of queue so that requests will be
> >       * handled in time order.  Ownership for req_obj, req, id,
> 
> ... down here.

Yeah can do.  I think it's not really a must (IMHO the worst case
won't be that worse if current queue length is 8)? but it's obviously
nicer.

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 19/26] qapi: introduce new cmd option "allow-oob"

[Peter Xu]

On Fri, Dec 15, 2017 at 05:51:04PM +0800, Fam Zheng wrote:
> On Tue, 12/05 13:51, Peter Xu wrote:
> > --- a/qapi/introspect.json
> > +++ b/qapi/introspect.json
> > @@ -259,12 +259,16 @@
> >  #
> >  # @ret-type: the name of the command's result type.
> >  #
> > +# @allow-oob: whether the command allows out-of-band execution.
> > +#             (Since: 2.11)
> 
> (Since: 2.12)

Ouch... So at last I still missed one.

Fixed.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 07/26] monitor: unify global init

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3136 bytes --]

On Sat, Dec 16, 2017 at 11:52:28AM +0800, Peter Xu wrote:
> On Fri, Dec 15, 2017 at 12:47:11PM +0000, Stefan Hajnoczi wrote:
> > On Fri, Dec 15, 2017 at 04:11:41PM +0800, Peter Xu wrote:
> > > On Wed, Dec 13, 2017 at 03:48:06PM +0000, Stefan Hajnoczi wrote:
> > > > On Tue, Dec 05, 2017 at 01:51:41PM +0800, Peter Xu wrote:
> > > > > diff --git a/vl.c b/vl.c
> > > > > index 1ad1c04637..1ec995a6ae 100644
> > > > > --- a/vl.c
> > > > > +++ b/vl.c
> > > > > @@ -3144,7 +3144,6 @@ int main(int argc, char **argv, char **envp)
> > > > >      qemu_init_exec_dir(argv[0]);
> > > > >  
> > > > >      module_call_init(MODULE_INIT_QOM);
> > > > > -    monitor_init_qmp_commands();
> > > > >  
> > > > >      qemu_add_opts(&qemu_drive_opts);
> > > > >      qemu_add_drive_opts(&qemu_legacy_drive_opts);
> > > > > @@ -4690,6 +4689,8 @@ int main(int argc, char **argv, char **envp)
> > > > >      default_drive(default_floppy, snapshot, IF_FLOPPY, 0, FD_OPTS);
> > > > >      default_drive(default_sdcard, snapshot, IF_SD, 0, SD_OPTS);
> > > > >  
> > > > > +    monitor_init_globals();
> > > > 
> > > > Why did you move monitor_init_qmp_commands() down here?
> > > > 
> > > > There are many function calls between the old position and the new
> > > > position.  Did you check all of them to make sure they don't touch the
> > > > monitor which is now totally uninitialized?
> > > 
> > > Yeh, this patch is a bit hairy, but I really think we should do it.
> > > Because there are too many places (as you have seen) that we inited
> > > monitor stuff in different places.
> > 
> > But why did you move it down here?  Can you replace the
> > monitor_init_qmp_commands() call instead?
> 
> I just moved it closer to (actually, right above) initializations of
> monitors to be clear that these globals will never be touched until
> this point.  Or do you want me to move this call exactly back to the
> place where monitor_init_qmp_commands() was called before?  Thanks,

Hmm...now that we're discussing this patch in detail I found an ordering
dependency:

qtest_enabled() only works after configure_accelerator(current_machine),
so the monitor_qapi_event_init() cannot be moved to the
monitor_init_qmp_commands() location.

Everything else has no dependencies.  I would be best to remove the
self-contained init from vl.c:main() so we don't have to worry about
dependencies again.  How about:

-static void __attribute__((constructor)) monitor_lock_init(void)
+static void __attribute__((constructor)) monitor_init_ctr(void)
 {
     qemu_mutex_init(&monitor_lock);
+    monitor_init_qmp_commands();
+    monitor_qapi_event_init();
+    sortcmdlist();
 }

Now vl.c:main() doesn't need to initialize the monitor.

We still need to handle the qtest_enabled() dependency:

-static QEMUClockType event_clock_type = QEMU_CLOCK_REALTIME;
+static inline QEMUClockType event_clock_type(void)
+{
+    return qtest_enabled() ? QEMU_CLOCK_VIRTUAL : QEMU_CLOCK_REALTIME;
+}

This way the qtest_enabled() call is deferred until later when the
accelerators have been initialized.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 15/26] monitor: let suspend_cnt be thread safe

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1370 bytes --]

On Sat, Dec 16, 2017 at 02:12:11PM +0800, Peter Xu wrote:
> On Wed, Dec 13, 2017 at 06:43:58PM +0000, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:51:49PM +0800, Peter Xu wrote:
> > > Monitor code now can be run in more than one thread.  Let it be thread
> > > safe when accessing suspend_cnt counter.
> > 
> > Please add doc comments to the functions explaining which thread they
> > may be called from (especially public functions).  Without this
> > information other people will have a hard time keeping code thread-safe.
> 
> The two public functions would be monitor_resume() and
> monitor_suspend().  IIUC as long as these functions are using atomic
> operations correctly they should be able to be called anywhere?
> 
> Could you help point out what I missed?

I don't mean just the public functions.  How are other people supposed
to know which monitor function executes in which thread without first
studying all the monitor code?  This is where comments, naming schemes,
or grouping functions helps.

Not every function needs to be documented in detail, but there needs to
be some structure to help people understand what happens in the IOThread
and what happens in the main loop.

Just adding threading to the existing code without leaving clues for
people who come after you is likely to result in future bugs.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 16/26] monitor: separate QMP parser and dispatcher

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2729 bytes --]

On Sat, Dec 16, 2017 at 02:37:03PM +0800, Peter Xu wrote:
> On Wed, Dec 13, 2017 at 08:09:38PM +0000, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:51:50PM +0800, Peter Xu wrote:
> > > @@ -3956,12 +3968,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> > >          }
> > >      }
> > >  
> > > -err_out:
> > > -    monitor_qmp_respond(mon, rsp, err, id);
> > > +    /* Respond if necessary */
> > > +    monitor_qmp_respond(mon, rsp, NULL, id);
> > > +
> > > +    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
> > > +    if (!qmp_oob_enabled(mon)) {
> > > +        monitor_resume(mon);
> > 
> > monitor_resume() does not work between threads: if the event loop is
> > currently blocked in poll() it won't notice that the monitor fd should
> > be watched again.
> > 
> > Please add aio_notify() to monitor_resume() and monitor_suspend().  That
> > way the event loop is forced to check can_read() again.
> 
> Ah, yes.  I think monitor_suspend() does not need the notify?  Since
> if it's sleeping it won't miss the next check in can_read() after all?

No, that would be a bug.  Imagine the IOThread is blocked in poll
monitoring the chardev file descriptor when the main loop calls
monitor_suspend().  If the file descriptors becomes readable then the
handler function executes even though the monitor is supposed to be
suspended!

> Regarding to monitor_resume(), I noticed that I missed the fact that
> it's only tailored for HMP before, which seems to be a bigger problem.
> Do you agree with a change like this?
> 
> diff --git a/monitor.c b/monitor.c
> index 9970418d6f..8f96880ad7 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -4244,10 +4244,12 @@ int monitor_suspend(Monitor *mon)
>  
>  void monitor_resume(Monitor *mon)
>  {
> -    if (!mon->rs)
> -        return;
>      if (atomic_dec_fetch(&mon->suspend_cnt) == 0) {
> -        readline_show_prompt(mon->rs);
> +        if (monitor_is_qmp(mon)) {
> +            aio_notify(mon_global.mon_iothread->ctx);

Please use iothread_get_aio_context() instead of accessing struct
members.

> +        } else {
> +            assert(mon->rs);
> +            readline_show_prompt(mon->rs);

I haven't studied the HMP and ->rs code.  I'll do that when reviewing
the next revision of this series.

> +        }
>      }
>  }
> 
> Even, I'm thinking about whether I should start to introduce
> iothread_notify() now to mask out the IOThread.ctx details.

aio_notify() is a low-level AioContext operation that is somewhat
bug-prone.  I think it's better to leave it directly exposed so callers
have to think about what they are doing.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 18/26] monitor: send event when request queue full

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2879 bytes --]

On Sat, Dec 16, 2017 at 03:17:06PM +0800, Peter Xu wrote:
> On Thu, Dec 14, 2017 at 11:41:36AM +0000, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:51:52PM +0800, Peter Xu wrote:
> > > Set maximum QMP request queue length to 8.  If queue full, instead of
> > > queue the command, we directly return a "request-dropped" event, telling
> > > client that specific command is dropped.
> > > 
> > > Note that this flow control mechanism is only valid if OOB is enabled.
> > > If it's not, the effective queue length will always be 1, which strictly
> > > follows original behavior of QMP command handling (which never drop
> > > messages).
> > > 
> > > Signed-off-by: Peter Xu <peterx@redhat.com>
> > > ---
> > >  monitor.c | 15 +++++++++++++++
> > >  1 file changed, 15 insertions(+)
> > > 
> > > diff --git a/monitor.c b/monitor.c
> > > index c20e659740..f7923c4590 100644
> > > --- a/monitor.c
> > > +++ b/monitor.c
> > > @@ -4029,6 +4029,8 @@ static void monitor_qmp_bh_dispatcher(void *data)
> > >      }
> > >  }
> > >  
> > > +#define  QMP_REQ_QUEUE_LEN_MAX  (8)
> > > +
> > >  static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> > >                                 void *opaque)
> > >  {
> > > @@ -4061,6 +4063,19 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> > >      req_obj->id = id;
> > >      req_obj->req = req;
> > >  
> > > +    if (qmp_oob_enabled(mon)) {
> > > +        /* Drop the request if queue is full. */
> > > +        if (mon->qmp.qmp_requests->length >= QMP_REQ_QUEUE_LEN_MAX) {
> > 
> > qmp_queue_lock must be held.  Perhaps it's simplest to move this check
> > into the qmp_queue_lock critical section below and unlock before calling
> > qapi_event_send_request_dropped()...
> > 
> > > +            qapi_event_send_request_dropped(id,
> > > +                                            REQUEST_DROP_REASON_QUEUE_FULL,
> > > +                                            NULL);
> > > +            qobject_decref(id);
> > > +            qobject_decref(req);
> > > +            g_free(req_obj);
> > > +            return;
> > > +        }
> > > +    }
> > > +
> > >      /*
> > >       * Put the request to the end of queue so that requests will be
> > >       * handled in time order.  Ownership for req_obj, req, id,
> > 
> > ... down here.
> 
> Yeah can do.  I think it's not really a must (IMHO the worst case
> won't be that worse if current queue length is 8)? but it's obviously
> nicer.

That style of coding is risky.  People reading the code won't know if
the lack of the lock was a mistake or intentional.  If not taking the
lock is necessary for performance (it isn't in this case and we're going
to take the lock anyway), then it should have a comment or nop
function/macro that documents the intention.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 07/26] monitor: unify global init

[Peter Xu]

On Sat, Dec 16, 2017 at 09:01:12AM +0000, Stefan Hajnoczi wrote:
> On Sat, Dec 16, 2017 at 11:52:28AM +0800, Peter Xu wrote:
> > On Fri, Dec 15, 2017 at 12:47:11PM +0000, Stefan Hajnoczi wrote:
> > > On Fri, Dec 15, 2017 at 04:11:41PM +0800, Peter Xu wrote:
> > > > On Wed, Dec 13, 2017 at 03:48:06PM +0000, Stefan Hajnoczi wrote:
> > > > > On Tue, Dec 05, 2017 at 01:51:41PM +0800, Peter Xu wrote:
> > > > > > diff --git a/vl.c b/vl.c
> > > > > > index 1ad1c04637..1ec995a6ae 100644
> > > > > > --- a/vl.c
> > > > > > +++ b/vl.c
> > > > > > @@ -3144,7 +3144,6 @@ int main(int argc, char **argv, char **envp)
> > > > > >      qemu_init_exec_dir(argv[0]);
> > > > > >  
> > > > > >      module_call_init(MODULE_INIT_QOM);
> > > > > > -    monitor_init_qmp_commands();
> > > > > >  
> > > > > >      qemu_add_opts(&qemu_drive_opts);
> > > > > >      qemu_add_drive_opts(&qemu_legacy_drive_opts);
> > > > > > @@ -4690,6 +4689,8 @@ int main(int argc, char **argv, char **envp)
> > > > > >      default_drive(default_floppy, snapshot, IF_FLOPPY, 0, FD_OPTS);
> > > > > >      default_drive(default_sdcard, snapshot, IF_SD, 0, SD_OPTS);
> > > > > >  
> > > > > > +    monitor_init_globals();
> > > > > 
> > > > > Why did you move monitor_init_qmp_commands() down here?
> > > > > 
> > > > > There are many function calls between the old position and the new
> > > > > position.  Did you check all of them to make sure they don't touch the
> > > > > monitor which is now totally uninitialized?
> > > > 
> > > > Yeh, this patch is a bit hairy, but I really think we should do it.
> > > > Because there are too many places (as you have seen) that we inited
> > > > monitor stuff in different places.
> > > 
> > > But why did you move it down here?  Can you replace the
> > > monitor_init_qmp_commands() call instead?
> > 
> > I just moved it closer to (actually, right above) initializations of
> > monitors to be clear that these globals will never be touched until
> > this point.  Or do you want me to move this call exactly back to the
> > place where monitor_init_qmp_commands() was called before?  Thanks,
> 
> Hmm...now that we're discussing this patch in detail I found an ordering
> dependency:
> 
> qtest_enabled() only works after configure_accelerator(current_machine),
> so the monitor_qapi_event_init() cannot be moved to the
> monitor_init_qmp_commands() location.
> 
> Everything else has no dependencies.  I would be best to remove the
> self-contained init from vl.c:main() so we don't have to worry about
> dependencies again.  How about:
> 
> -static void __attribute__((constructor)) monitor_lock_init(void)
> +static void __attribute__((constructor)) monitor_init_ctr(void)

(Curious: why name it "ctr"?)

>  {
>      qemu_mutex_init(&monitor_lock);
> +    monitor_init_qmp_commands();
> +    monitor_qapi_event_init();
> +    sortcmdlist();

Though I still need to init monitor iothreads, I'm not sure whether it
can be put into a constructor function since I think IOThead has not
yet been initialized now (it's done until module_call_init() in
main()).

>  }
> 
> Now vl.c:main() doesn't need to initialize the monitor.
> 
> We still need to handle the qtest_enabled() dependency:
> 
> -static QEMUClockType event_clock_type = QEMU_CLOCK_REALTIME;
> +static inline QEMUClockType event_clock_type(void)
> +{
> +    return qtest_enabled() ? QEMU_CLOCK_VIRTUAL : QEMU_CLOCK_REALTIME;
> +}
> 
> This way the qtest_enabled() call is deferred until later when the
> accelerators have been initialized.

Or... to be much simpler... How about we just use my patch?  IMHO it
calls monitor_init_globals() after configure_accelerator() so
monitor_qapi_event_init() and everything else would just work as
expected?

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 15/26] monitor: let suspend_cnt be thread safe

[Peter Xu]

On Sat, Dec 16, 2017 at 09:11:08AM +0000, Stefan Hajnoczi wrote:
> On Sat, Dec 16, 2017 at 02:12:11PM +0800, Peter Xu wrote:
> > On Wed, Dec 13, 2017 at 06:43:58PM +0000, Stefan Hajnoczi wrote:
> > > On Tue, Dec 05, 2017 at 01:51:49PM +0800, Peter Xu wrote:
> > > > Monitor code now can be run in more than one thread.  Let it be thread
> > > > safe when accessing suspend_cnt counter.
> > > 
> > > Please add doc comments to the functions explaining which thread they
> > > may be called from (especially public functions).  Without this
> > > information other people will have a hard time keeping code thread-safe.
> > 
> > The two public functions would be monitor_resume() and
> > monitor_suspend().  IIUC as long as these functions are using atomic
> > operations correctly they should be able to be called anywhere?
> > 
> > Could you help point out what I missed?
> 
> I don't mean just the public functions.  How are other people supposed
> to know which monitor function executes in which thread without first
> studying all the monitor code?  This is where comments, naming schemes,
> or grouping functions helps.
> 
> Not every function needs to be documented in detail, but there needs to
> be some structure to help people understand what happens in the IOThread
> and what happens in the main loop.
> 
> Just adding threading to the existing code without leaving clues for
> people who come after you is likely to result in future bugs.

Yes, I fully agree with you that we need to do better commenting when
doing things like this.

I am just not sure what you would prefer me to add since AFAICT these
functions (after with the atomic protections) are really free to be
called anywhere, and also in any thread (even newly created threads
rather than main thread or monitor IO thread), and they are free of
locks too, like monitor_lock.  Here I only mean monitor_suspend() and
monitor_resume() for sure.  For the rest of functions, IMHO since they
are internal to monitor code so people who will call them do need to
know something about monitors.

I would be more appreciated if you can provide some examples here to
show what you mean.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 16/26] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Sat, Dec 16, 2017 at 09:23:22AM +0000, Stefan Hajnoczi wrote:
> On Sat, Dec 16, 2017 at 02:37:03PM +0800, Peter Xu wrote:
> > On Wed, Dec 13, 2017 at 08:09:38PM +0000, Stefan Hajnoczi wrote:
> > > On Tue, Dec 05, 2017 at 01:51:50PM +0800, Peter Xu wrote:
> > > > @@ -3956,12 +3968,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> > > >          }
> > > >      }
> > > >  
> > > > -err_out:
> > > > -    monitor_qmp_respond(mon, rsp, err, id);
> > > > +    /* Respond if necessary */
> > > > +    monitor_qmp_respond(mon, rsp, NULL, id);
> > > > +
> > > > +    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
> > > > +    if (!qmp_oob_enabled(mon)) {
> > > > +        monitor_resume(mon);
> > > 
> > > monitor_resume() does not work between threads: if the event loop is
> > > currently blocked in poll() it won't notice that the monitor fd should
> > > be watched again.
> > > 
> > > Please add aio_notify() to monitor_resume() and monitor_suspend().  That
> > > way the event loop is forced to check can_read() again.
> > 
> > Ah, yes.  I think monitor_suspend() does not need the notify?  Since
> > if it's sleeping it won't miss the next check in can_read() after all?
> 
> No, that would be a bug.  Imagine the IOThread is blocked in poll
> monitoring the chardev file descriptor when the main loop calls
> monitor_suspend().  If the file descriptors becomes readable then the
> handler function executes even though the monitor is supposed to be
> suspended!

When you say "the handler function executes", do you mean the handler
that has already added to the qmp request queue, or the one that
hasn't yet parsed by the parser?

For the previous case (the handler that has queued already): IMHO
that's what we expect it to behave, say, when we call
monitor_suspend(), we only stop accepting and parsing new inputs from
the user, but the requests on the queue should still be processed.

For the latter (the handler of a newly typed command):
monitor_suspend() should suspend the parser already, so
monitor_can_read() check should fail, then that command should never
be queued until we call another monitor_resume().

> 
> > Regarding to monitor_resume(), I noticed that I missed the fact that
> > it's only tailored for HMP before, which seems to be a bigger problem.
> > Do you agree with a change like this?
> > 
> > diff --git a/monitor.c b/monitor.c
> > index 9970418d6f..8f96880ad7 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -4244,10 +4244,12 @@ int monitor_suspend(Monitor *mon)
> >  
> >  void monitor_resume(Monitor *mon)
> >  {
> > -    if (!mon->rs)
> > -        return;
> >      if (atomic_dec_fetch(&mon->suspend_cnt) == 0) {
> > -        readline_show_prompt(mon->rs);
> > +        if (monitor_is_qmp(mon)) {
> > +            aio_notify(mon_global.mon_iothread->ctx);
> 
> Please use iothread_get_aio_context() instead of accessing struct
> members.

Ah, yes!

> 
> > +        } else {
> > +            assert(mon->rs);
> > +            readline_show_prompt(mon->rs);
> 
> I haven't studied the HMP and ->rs code.  I'll do that when reviewing
> the next revision of this series.

Sure.

> 
> > +        }
> >      }
> >  }
> > 
> > Even, I'm thinking about whether I should start to introduce
> > iothread_notify() now to mask out the IOThread.ctx details.
> 
> aio_notify() is a low-level AioContext operation that is somewhat
> bug-prone.  I think it's better to leave it directly exposed so callers
> have to think about what they are doing.

Sure.  Thanks!

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 18/26] monitor: send event when request queue full

[Peter Xu]

On Sat, Dec 16, 2017 at 09:28:36AM +0000, Stefan Hajnoczi wrote:
> On Sat, Dec 16, 2017 at 03:17:06PM +0800, Peter Xu wrote:
> > On Thu, Dec 14, 2017 at 11:41:36AM +0000, Stefan Hajnoczi wrote:
> > > On Tue, Dec 05, 2017 at 01:51:52PM +0800, Peter Xu wrote:
> > > > Set maximum QMP request queue length to 8.  If queue full, instead of
> > > > queue the command, we directly return a "request-dropped" event, telling
> > > > client that specific command is dropped.
> > > > 
> > > > Note that this flow control mechanism is only valid if OOB is enabled.
> > > > If it's not, the effective queue length will always be 1, which strictly
> > > > follows original behavior of QMP command handling (which never drop
> > > > messages).
> > > > 
> > > > Signed-off-by: Peter Xu <peterx@redhat.com>
> > > > ---
> > > >  monitor.c | 15 +++++++++++++++
> > > >  1 file changed, 15 insertions(+)
> > > > 
> > > > diff --git a/monitor.c b/monitor.c
> > > > index c20e659740..f7923c4590 100644
> > > > --- a/monitor.c
> > > > +++ b/monitor.c
> > > > @@ -4029,6 +4029,8 @@ static void monitor_qmp_bh_dispatcher(void *data)
> > > >      }
> > > >  }
> > > >  
> > > > +#define  QMP_REQ_QUEUE_LEN_MAX  (8)
> > > > +
> > > >  static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> > > >                                 void *opaque)
> > > >  {
> > > > @@ -4061,6 +4063,19 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> > > >      req_obj->id = id;
> > > >      req_obj->req = req;
> > > >  
> > > > +    if (qmp_oob_enabled(mon)) {
> > > > +        /* Drop the request if queue is full. */
> > > > +        if (mon->qmp.qmp_requests->length >= QMP_REQ_QUEUE_LEN_MAX) {
> > > 
> > > qmp_queue_lock must be held.  Perhaps it's simplest to move this check
> > > into the qmp_queue_lock critical section below and unlock before calling
> > > qapi_event_send_request_dropped()...
> > > 
> > > > +            qapi_event_send_request_dropped(id,
> > > > +                                            REQUEST_DROP_REASON_QUEUE_FULL,
> > > > +                                            NULL);
> > > > +            qobject_decref(id);
> > > > +            qobject_decref(req);
> > > > +            g_free(req_obj);
> > > > +            return;
> > > > +        }
> > > > +    }
> > > > +
> > > >      /*
> > > >       * Put the request to the end of queue so that requests will be
> > > >       * handled in time order.  Ownership for req_obj, req, id,
> > > 
> > > ... down here.
> > 
> > Yeah can do.  I think it's not really a must (IMHO the worst case
> > won't be that worse if current queue length is 8)? but it's obviously
> > nicer.
> 
> That style of coding is risky.  People reading the code won't know if
> the lack of the lock was a mistake or intentional.  If not taking the
> lock is necessary for performance (it isn't in this case and we're going
> to take the lock anyway), then it should have a comment or nop
> function/macro that documents the intention.

I have seen no reason to care about performance for QMP yet especially
for this, so I fully agree with you.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 20/26] qmp: support out-of-band (oob) execution

[Peter Xu]

On Thu, Dec 14, 2017 at 01:16:32PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:54PM +0800, Peter Xu wrote:
> >      if (qdict) {
> >          id = qdict_get(qdict, "id");
> > +        /* When OOB is enabled, the "id" field is mandatory. */
> > +        if (qmp_oob_enabled(mon) && !id) {
> > +            error_setg(&err, "Out-Of-Band capability requires that "
> > +                       "every command contains an 'id' field.");
> 
> Is this documented in docs/interop/qmp-spec.txt?

Yes it is:

@@ -102,10 +125,19 @@ The format for command execution is:
   required. Each command documents what contents will be considered
   valid when handling the json-argument
 - The "id" member is a transaction identification associated with the
-  command execution, it is optional and will be part of the response if
+  command execution.  It is required if OOB is enabled, and optional
+  if not.  The same "id" field will be part of the response if
   provided. The "id" member can be any json-value, although most
   clients merely use a json-number incremented for each successive
   command

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 21/26] qmp: isolate responses into io thread

[Peter Xu]

On Thu, Dec 14, 2017 at 01:43:59PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:55PM +0800, Peter Xu wrote:
> > @@ -4429,6 +4515,13 @@ void monitor_cleanup(void)
> >       */
> >      iothread_stop(mon_global.mon_iothread);
> >  
> > +    /*
> > +     * After we have IOThread to send responses, it's possible that
> > +     * when we stop the IOThread there are still replies queued in the
> > +     * responder queue.  Flush all of them.
> > +     */
> > +    monitor_qmp_bh_responder(NULL);
> 
> This doesn't work because monitor_qmp_bh_responder() does not guarantee
> that the full response has been written when it returns.
> 
> When qemu_chr_fe_write() returns EAGAIN then qemu_chr_fe_add_watch() is
> used to register an event loop callback when the chardev becomes
> writable again.  But you stopped the event loop using iothread_stop() so
> we will never complete the write.

Good catch...

Actually I just noticed that for char frontend I missed a place to use
the chardev context for polling.  So before the flushing I possibly
need this:

diff --git a/chardev/char-fe.c b/chardev/char-fe.c                             
index ee6d596100..462c529f19 100644    
--- a/chardev/char-fe.c                
+++ b/chardev/char-fe.c                
@@ -356,7 +356,7 @@ guint qemu_chr_fe_add_watch(CharBackend *be, GIOCondition cond,                                                                           
     }                                 
                                       
     g_source_set_callback(src, (GSourceFunc)func, user_data, NULL);           
-    tag = g_source_attach(src, NULL); 
+    tag = g_source_attach(src, be->chr->gcontext);                            
     g_source_unref(src);              
                                       
     return tag;                       

Otherwise it'll be still be run in main thread always.

(I guess I haven't yet encountered an EAGAIN for it so far)

> 
> I suggest draining the monitor while the IOThread is still running
> (that way the AioContext and GMainContext are still operational).  You
> can:
> 1. Suspend the monitor so new commands will not be read.
> 2. Wait until all responses and outbuf are empty.
> 
> Another option is moving the chardev back to the main loop but I'm not
> sure if the chardev subsystem supports that.

Your suggestion is good to me.  I'll do that in IOThread before it
stops.  Thanks!

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 21/26] qmp: isolate responses into io thread

[Peter Xu]

On Mon, Dec 18, 2017 at 01:52:17PM +0800, Peter Xu wrote:
> On Thu, Dec 14, 2017 at 01:43:59PM +0000, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:51:55PM +0800, Peter Xu wrote:
> > > @@ -4429,6 +4515,13 @@ void monitor_cleanup(void)
> > >       */
> > >      iothread_stop(mon_global.mon_iothread);
> > >  
> > > +    /*
> > > +     * After we have IOThread to send responses, it's possible that
> > > +     * when we stop the IOThread there are still replies queued in the
> > > +     * responder queue.  Flush all of them.
> > > +     */
> > > +    monitor_qmp_bh_responder(NULL);
> > 
> > This doesn't work because monitor_qmp_bh_responder() does not guarantee
> > that the full response has been written when it returns.
> > 
> > When qemu_chr_fe_write() returns EAGAIN then qemu_chr_fe_add_watch() is
> > used to register an event loop callback when the chardev becomes
> > writable again.  But you stopped the event loop using iothread_stop() so
> > we will never complete the write.
> 
> Good catch...
> 
> Actually I just noticed that for char frontend I missed a place to use
> the chardev context for polling.  So before the flushing I possibly
> need this:
> 
> diff --git a/chardev/char-fe.c b/chardev/char-fe.c                             
> index ee6d596100..462c529f19 100644    
> --- a/chardev/char-fe.c                
> +++ b/chardev/char-fe.c                
> @@ -356,7 +356,7 @@ guint qemu_chr_fe_add_watch(CharBackend *be, GIOCondition cond,                                                                           
>      }                                 
>                                        
>      g_source_set_callback(src, (GSourceFunc)func, user_data, NULL);           
> -    tag = g_source_attach(src, NULL); 
> +    tag = g_source_attach(src, be->chr->gcontext);                            
>      g_source_unref(src);              
>                                        
>      return tag;                       
> 
> Otherwise it'll be still be run in main thread always.
> 
> (I guess I haven't yet encountered an EAGAIN for it so far)
> 
> > 
> > I suggest draining the monitor while the IOThread is still running
> > (that way the AioContext and GMainContext are still operational).  You
> > can:
> > 1. Suspend the monitor so new commands will not be read.
> > 2. Wait until all responses and outbuf are empty.
> > 
> > Another option is moving the chardev back to the main loop but I'm not
> > sure if the chardev subsystem supports that.
> 
> Your suggestion is good to me.  I'll do that in IOThread before it
> stops.  Thanks!

Hmm, after a second thought, I think maybe it would be nicer to just
call monitor_flush() after the monitor iothread is stopped.

Firstly, that is perfectly legal - since AFAICT commit 6cff3e8594
("monitor: protect outbuf and mux_out with mutex", 2014-06-23) is
tailored for doing that, though that was for block iothread, not
monitor.

Then, it's still possible that the write buffer is full and we got
EAGAIN during flushing.  Then we'll plug another task into the monitor
gcontext which will never run since the iothread is stopped already.
IMHO that's totally fine - I think it's already our best effort to
flush the out buffer once at this point.  If the buffer is still full
after one attemp of monitor_flush(), we should just give up the 2nd
flushing instead of waiting here.  After all, we are quitting QEMU.

As a summary, my planned change of this issue would be quite
straightforward: call monitor_flush() when looping over monitors,
before destroying them.  And when destroying, if there is dangling
Monitor.out_watch, detach them.  Hope that works.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 21/26] qmp: isolate responses into io thread

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4151 bytes --]

On Mon, Dec 18, 2017 at 03:32:29PM +0800, Peter Xu wrote:
> On Mon, Dec 18, 2017 at 01:52:17PM +0800, Peter Xu wrote:
> > On Thu, Dec 14, 2017 at 01:43:59PM +0000, Stefan Hajnoczi wrote:
> > > On Tue, Dec 05, 2017 at 01:51:55PM +0800, Peter Xu wrote:
> > > > @@ -4429,6 +4515,13 @@ void monitor_cleanup(void)
> > > >       */
> > > >      iothread_stop(mon_global.mon_iothread);
> > > >  
> > > > +    /*
> > > > +     * After we have IOThread to send responses, it's possible that
> > > > +     * when we stop the IOThread there are still replies queued in the
> > > > +     * responder queue.  Flush all of them.
> > > > +     */
> > > > +    monitor_qmp_bh_responder(NULL);
> > > 
> > > This doesn't work because monitor_qmp_bh_responder() does not guarantee
> > > that the full response has been written when it returns.
> > > 
> > > When qemu_chr_fe_write() returns EAGAIN then qemu_chr_fe_add_watch() is
> > > used to register an event loop callback when the chardev becomes
> > > writable again.  But you stopped the event loop using iothread_stop() so
> > > we will never complete the write.
> > 
> > Good catch...
> > 
> > Actually I just noticed that for char frontend I missed a place to use
> > the chardev context for polling.  So before the flushing I possibly
> > need this:
> > 
> > diff --git a/chardev/char-fe.c b/chardev/char-fe.c                             
> > index ee6d596100..462c529f19 100644    
> > --- a/chardev/char-fe.c                
> > +++ b/chardev/char-fe.c                
> > @@ -356,7 +356,7 @@ guint qemu_chr_fe_add_watch(CharBackend *be, GIOCondition cond,                                                                           
> >      }                                 
> >                                        
> >      g_source_set_callback(src, (GSourceFunc)func, user_data, NULL);           
> > -    tag = g_source_attach(src, NULL); 
> > +    tag = g_source_attach(src, be->chr->gcontext);                            
> >      g_source_unref(src);              
> >                                        
> >      return tag;                       
> > 
> > Otherwise it'll be still be run in main thread always.
> > 
> > (I guess I haven't yet encountered an EAGAIN for it so far)
> > 
> > > 
> > > I suggest draining the monitor while the IOThread is still running
> > > (that way the AioContext and GMainContext are still operational).  You
> > > can:
> > > 1. Suspend the monitor so new commands will not be read.
> > > 2. Wait until all responses and outbuf are empty.
> > > 
> > > Another option is moving the chardev back to the main loop but I'm not
> > > sure if the chardev subsystem supports that.
> > 
> > Your suggestion is good to me.  I'll do that in IOThread before it
> > stops.  Thanks!
> 
> Hmm, after a second thought, I think maybe it would be nicer to just
> call monitor_flush() after the monitor iothread is stopped.
> 
> Firstly, that is perfectly legal - since AFAICT commit 6cff3e8594
> ("monitor: protect outbuf and mux_out with mutex", 2014-06-23) is
> tailored for doing that, though that was for block iothread, not
> monitor.
> 
> Then, it's still possible that the write buffer is full and we got
> EAGAIN during flushing.  Then we'll plug another task into the monitor
> gcontext which will never run since the iothread is stopped already.
> IMHO that's totally fine - I think it's already our best effort to
> flush the out buffer once at this point.  If the buffer is still full
> after one attemp of monitor_flush(), we should just give up the 2nd
> flushing instead of waiting here.  After all, we are quitting QEMU.
> 
> As a summary, my planned change of this issue would be quite
> straightforward: call monitor_flush() when looping over monitors,
> before destroying them.  And when destroying, if there is dangling
> Monitor.out_watch, detach them.  Hope that works.  Thanks,

I think QEMU currently doesn't drain the monitor output buffer either.
Please make sure to write the code and commit description so it is clear
this is just a best-effort attempt.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 16/26] monitor: separate QMP parser and dispatcher

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3108 bytes --]

On Mon, Dec 18, 2017 at 01:26:49PM +0800, Peter Xu wrote:
> On Sat, Dec 16, 2017 at 09:23:22AM +0000, Stefan Hajnoczi wrote:
> > On Sat, Dec 16, 2017 at 02:37:03PM +0800, Peter Xu wrote:
> > > On Wed, Dec 13, 2017 at 08:09:38PM +0000, Stefan Hajnoczi wrote:
> > > > On Tue, Dec 05, 2017 at 01:51:50PM +0800, Peter Xu wrote:
> > > > > @@ -3956,12 +3968,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> > > > >          }
> > > > >      }
> > > > >  
> > > > > -err_out:
> > > > > -    monitor_qmp_respond(mon, rsp, err, id);
> > > > > +    /* Respond if necessary */
> > > > > +    monitor_qmp_respond(mon, rsp, NULL, id);
> > > > > +
> > > > > +    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
> > > > > +    if (!qmp_oob_enabled(mon)) {
> > > > > +        monitor_resume(mon);
> > > > 
> > > > monitor_resume() does not work between threads: if the event loop is
> > > > currently blocked in poll() it won't notice that the monitor fd should
> > > > be watched again.
> > > > 
> > > > Please add aio_notify() to monitor_resume() and monitor_suspend().  That
> > > > way the event loop is forced to check can_read() again.
> > > 
> > > Ah, yes.  I think monitor_suspend() does not need the notify?  Since
> > > if it's sleeping it won't miss the next check in can_read() after all?
> > 
> > No, that would be a bug.  Imagine the IOThread is blocked in poll
> > monitoring the chardev file descriptor when the main loop calls
> > monitor_suspend().  If the file descriptors becomes readable then the
> > handler function executes even though the monitor is supposed to be
> > suspended!
> 
> When you say "the handler function executes", do you mean the handler
> that has already added to the qmp request queue, or the one that
> hasn't yet parsed by the parser?

The chardev file descriptor handler function (the QMP parser).

> For the previous case (the handler that has queued already): IMHO
> that's what we expect it to behave, say, when we call
> monitor_suspend(), we only stop accepting and parsing new inputs from
> the user, but the requests on the queue should still be processed.
> 
> For the latter (the handler of a newly typed command):
> monitor_suspend() should suspend the parser already, so
> monitor_can_read() check should fail, then that command should never
> be queued until we call another monitor_resume().

You are assuming that monitor_can_read() is called *after* poll()
returns.  This is what I tried to explain in the previous reply.

The the monitor_can_read() function is called *before* the blocking
poll() syscall.  If something changes the monitor_can_read() return
value, you must* kick the event loop to ensure that the event loop
reflects this change.

If you want to check how this works, see chardev/char-io.c for how
fd_can_read() is used.

* Currently monitor.c doesn't need to kick the event loop explicitly
because it runs within the main loop thread.  Therefore the event loop
always calls monitor_can_read() again before entering poll().

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 07/26] monitor: unify global init

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 1459 bytes --]

On Mon, Dec 18, 2017 at 11:27:02AM +0800, Peter Xu wrote:
> On Sat, Dec 16, 2017 at 09:01:12AM +0000, Stefan Hajnoczi wrote:
> > On Sat, Dec 16, 2017 at 11:52:28AM +0800, Peter Xu wrote:
> > > On Fri, Dec 15, 2017 at 12:47:11PM +0000, Stefan Hajnoczi wrote:
> > > > On Fri, Dec 15, 2017 at 04:11:41PM +0800, Peter Xu wrote:
> > > > > On Wed, Dec 13, 2017 at 03:48:06PM +0000, Stefan Hajnoczi wrote:
> > > > > > On Tue, Dec 05, 2017 at 01:51:41PM +0800, Peter Xu wrote:
> >  }
> > 
> > Now vl.c:main() doesn't need to initialize the monitor.
> > 
> > We still need to handle the qtest_enabled() dependency:
> > 
> > -static QEMUClockType event_clock_type = QEMU_CLOCK_REALTIME;
> > +static inline QEMUClockType event_clock_type(void)
> > +{
> > +    return qtest_enabled() ? QEMU_CLOCK_VIRTUAL : QEMU_CLOCK_REALTIME;
> > +}
> > 
> > This way the qtest_enabled() call is deferred until later when the
> > accelerators have been initialized.
> 
> Or... to be much simpler... How about we just use my patch?  IMHO it
> calls monitor_init_globals() after configure_accelerator() so
> monitor_qapi_event_init() and everything else would just work as
> expected?

Ordering is complex in vl.c:main() so I think it's more maintainable to
put initialization in a monitor.c constructor function and/or
monitor_init().

If you want to keep your approach please add a comment about the
qtest_enabled() dependency in vl.c:main().

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 24/26] docs: update QMP documents for OOB commands

[Peter Xu]

On Thu, Dec 14, 2017 at 02:30:19PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:58PM +0800, Peter Xu wrote:
> > diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
> > index f04c63fe82..8597fdb087 100644
> > --- a/docs/devel/qapi-code-gen.txt
> > +++ b/docs/devel/qapi-code-gen.txt
> > @@ -556,7 +556,8 @@ following example objects:
> >  
> >  Usage: { 'command': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT,
> >           '*returns': TYPE-NAME, '*boxed': true,
> > -         '*gen': false, '*success-response': false }
> > +         '*gen': false, '*success-response': false,
> > +         '*allow-oob': false }
> >  
> >  Commands are defined by using a dictionary containing several members,
> >  where three members are most common.  The 'command' member is a
> > @@ -636,6 +637,44 @@ possible, the command expression should include the optional key
> >  'success-response' with boolean value false.  So far, only QGA makes
> >  use of this member.
> >  
> > +Most of the QMP commands are handled sequentially in such a order:
> > +Firstly, the JSON Parser parses the command request into some internal
> > +message, delivers the message to QMP dispatchers. Secondly, the QMP
> > +dispatchers will handle the commands one by one in time order, respond
> > +when necessary.
> 
> The important points to cover about normal commands:
> 1. They execute in order
> 2. They run the main loop
> 3. They run under the BQL
> 
> The other stuff about parsing requests into internal messages,
> dispatchers, etc is not relevant.  It's better not to include it in
> documentation because it can change and could also confuse readers
> (since they don't need this info).

Ah yes.  I'm thinking whether I should just remove most of the changes
to this file (qapi-code-gen.txt) since most of them are really not
related to code gen at all...  Maybe somewhere in qmp-spec.txt as
well?

> 
> > For some commands that always complete "quickly" can
> 
> I've mentioned before that "quickly" is misleading and not what oob
> commands are about.  I suggest changing this to:
> 
>   Certain urgent commands can
> 
> I've made similar comments further down where I think the text focusses
> on words like "quickly" or "asynchronous" too much.

I'll be more careful on the wording in next version on this.

> 
> > +instead be executed directly during parsing, at the QMP client's
> > +request.  This kind of commands that allow direct execution is called
> > +"out-of-band" ("oob" as shortcut) commands. The response can overtake
> > +prior in-band commands' responses.  By default, commands are always
> > +in-band.  We need to explicitly specify "allow-oob" to "True" to show
> 
> s/"True"/true/  (JSON is case-sensitive)

Ok.

> 
> > +that one command can be run out-of-band.
> > +
> > +One thing to mention for developers is that, although out-of-band
> > +execution of commands benefit from quick and asynchronous execution,
> > +it need to satisfy at least the following:
> > +
> > +(1) It is extremely quick and never blocks, so that its execution will
> > +    not block parsing routine of any other monitors.
> > +
> > +(2) It does not need BQL, since the parser can be run without BQL,
> > +    while the dispatcher is always with BQL held.
> 
> These conditions are not sufficient for postcopy recovery.  I suggest
> rephrasing this section as follows:
> 
>   An out-of-band command must:
> 
>   1. Complete extremely quickly
>   2. Not take locks
>   3. Not invoke blocking system calls
>   4. Not access guest RAM or memory that blocks when userfaultfd is
>      enabled for postcopy live migration
> 
> We could make #2 less strict by saying "Not take locks except for
> spinlocks (or mutexes that could be spinlocks because the critical
> regions are tiny) or indirectly via g_malloc()".

Ok.  I'm thinking whether I should also move these lines out too.

> 
> > Whether a command is
> > +allowed to run out-of-band can also be introspected using
> > +query-qmp-schema command.  Please see the section "Client JSON
> > +Protocol introspection" for more information.
> 
> This is relevant to client authors, not QEMU developers learning about
> qapi.  I suggest dropping it.

Ok.

> 
> > +
> > +To execute a command in out-of-band way, we need to specify the
> > +"control" field in the request, with "run-oob" set to true. Example:
> > +
> > + => { "execute": "command-support-oob",
> > +      "arguments": { ... },
> > +      "control": { "run-oob": true } }
> > + <= { "return": { } }
> > +
> > +Without it, even the commands that supports out-of-band execution will
> > +still be run in-band.
> 
> This is relevant to client authors, not QEMU developers learning about
> qapi.  I suggest instead explaining how qmp functions need to test for
> qmp_is_oob() so that they know which mode they are executing in.
> 
> >  2.2.1 Capabilities
> >  ------------------
> >  
> > -As of the date this document was last revised, no server or client
> > -capability strings have been defined.
> > -
> > +Currently supported capabilities are:
> > +
> > +- "oob": it means the QMP server supports "Out-Of-Band" command
> > +  execution.  For more detail, please see "run-oob" parameter in
> > +  "Issuing Commands" section below.  Not all commands allow this "oob"
> > +  execution.  One can know whether one command supports "oob" by
> > +  "query-qmp-schema" command.
> > +
> > +  NOTE: Converting an existing QMP command to be OOB-capable can be
> > +  either very easy or extremely hard.  The most important thing is
> > +  that OOB-capable command should never be blocked for a long time.
> > +  Some bad examples: (1) doing IOs, especially when the backend can be
> > +  an NFS storage; or (2) accessing guest memories, which can be simply
> > +  blocked for a very long time when it triggers a page fault, which
> > +  may not be handled immediately.  It's still legal to take a mutex in
> > +  an OOB-capable command handler, however, we need to make sure that
> > +  all the other code paths that are protected by the same lock won't
> > +  be blocked very long as well, otherwise the OOB handler might be
> > +  blocked too when it tries to take the lock.
> 
> Please drop this paragraph, this is the qmp-spec.txt document so the
> implementation of QEMU's QMP commands shouldn't be discussed here.

Ok.

> 
> > For some commands that always complete
> > +  "quickly" can be executed directly during parsing at the QMP
> > +  client's request.
> 
> Please drop this sentence.  "quickly" isn't the important quality, it's
> urgency.  Also the description of execution in the QMP parser isn't
> relevant for qmp-spec.txt, what matters is that oob commands can execute
> while a normal monitor command is still running (this is described next
> so this whole sentence can be dropped).

Ok.

I'll try to re-arrange the sentences better in my next post.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 25/26] tests: qmp-test: verify command batching

[Peter Xu]

On Thu, Dec 14, 2017 at 02:39:34PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:59PM +0800, Peter Xu wrote:
> > OOB introduced DROP event for flow control.  This should not affect old
> > QMP clients.  Add a command batching check to make sure of it.
> > 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  tests/qmp-test.c | 19 +++++++++++++++++++
> >  1 file changed, 19 insertions(+)
> > 
> > diff --git a/tests/qmp-test.c b/tests/qmp-test.c
> > index 292c5f135a..729ec59b0a 100644
> > --- a/tests/qmp-test.c
> > +++ b/tests/qmp-test.c
> > @@ -78,6 +78,7 @@ static void test_qmp_protocol(void)
> >      QList *capabilities;
> >      const QListEntry *entry;
> >      QString *qstr;
> > +    int i;
> >  
> >      global_qtest = qtest_init_without_qmp_handshake(common_args);
> >  
> > @@ -135,6 +136,24 @@ static void test_qmp_protocol(void)
> >      g_assert_cmpint(qdict_get_int(resp, "id"), ==, 2);
> >      QDECREF(resp);
> >  
> > +    /*
> > +     * Test command batching.  In current test OOB is not enabled, we
> > +     * should be able to run as many commands in batch as we like.
> > +     * Using 16 (>8, which is OOB queue length) to make sure OOB
> > +     * won't break existing clients.
> > +     */
> 
> It's possible that QEMU processes the command and responds before the
> test case submits the next one.  Please add:
> 
>   Note: this test does not control the scheduling of QEMU's QMP command
>   processing threads so it may not really trigger batching inside QEMU.
>   This is just a best-effort test.

True.

> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 26/26] tests: qmp-test: add oob test

[Peter Xu]

On Thu, Dec 14, 2017 at 02:47:12PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:52:00PM +0800, Peter Xu wrote:
> > +    /* Try a fake capability, it should fail. */
> > +    resp = qmp("{ 'execute': 'qmp_capabilities', "
> > +               "  'arguments': { 'enable': [ 'cap-does-not-exist' ] } }");
> > +    g_assert(qdict_haskey(resp, "error"));
> 
> Missing QDECREF(resp) here and throughout the rest of the function.

I believe I missed four QDECREF()s.

> 
> Besides that:
> 
> Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

Hope my counting is correct, then I'll add the r-b.  Thanks!

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 16/26] monitor: separate QMP parser and dispatcher

[Peter Xu]

On Mon, Dec 18, 2017 at 09:10:53AM +0000, Stefan Hajnoczi wrote:
> On Mon, Dec 18, 2017 at 01:26:49PM +0800, Peter Xu wrote:
> > On Sat, Dec 16, 2017 at 09:23:22AM +0000, Stefan Hajnoczi wrote:
> > > On Sat, Dec 16, 2017 at 02:37:03PM +0800, Peter Xu wrote:
> > > > On Wed, Dec 13, 2017 at 08:09:38PM +0000, Stefan Hajnoczi wrote:
> > > > > On Tue, Dec 05, 2017 at 01:51:50PM +0800, Peter Xu wrote:
> > > > > > @@ -3956,12 +3968,122 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> > > > > >          }
> > > > > >      }
> > > > > >  
> > > > > > -err_out:
> > > > > > -    monitor_qmp_respond(mon, rsp, err, id);
> > > > > > +    /* Respond if necessary */
> > > > > > +    monitor_qmp_respond(mon, rsp, NULL, id);
> > > > > > +
> > > > > > +    /* This pairs with the monitor_suspend() in handle_qmp_command(). */
> > > > > > +    if (!qmp_oob_enabled(mon)) {
> > > > > > +        monitor_resume(mon);
> > > > > 
> > > > > monitor_resume() does not work between threads: if the event loop is
> > > > > currently blocked in poll() it won't notice that the monitor fd should
> > > > > be watched again.
> > > > > 
> > > > > Please add aio_notify() to monitor_resume() and monitor_suspend().  That
> > > > > way the event loop is forced to check can_read() again.
> > > > 
> > > > Ah, yes.  I think monitor_suspend() does not need the notify?  Since
> > > > if it's sleeping it won't miss the next check in can_read() after all?
> > > 
> > > No, that would be a bug.  Imagine the IOThread is blocked in poll
> > > monitoring the chardev file descriptor when the main loop calls
> > > monitor_suspend().  If the file descriptors becomes readable then the
> > > handler function executes even though the monitor is supposed to be
> > > suspended!
> > 
> > When you say "the handler function executes", do you mean the handler
> > that has already added to the qmp request queue, or the one that
> > hasn't yet parsed by the parser?
> 
> The chardev file descriptor handler function (the QMP parser).
> 
> > For the previous case (the handler that has queued already): IMHO
> > that's what we expect it to behave, say, when we call
> > monitor_suspend(), we only stop accepting and parsing new inputs from
> > the user, but the requests on the queue should still be processed.
> > 
> > For the latter (the handler of a newly typed command):
> > monitor_suspend() should suspend the parser already, so
> > monitor_can_read() check should fail, then that command should never
> > be queued until we call another monitor_resume().
> 
> You are assuming that monitor_can_read() is called *after* poll()
> returns.  This is what I tried to explain in the previous reply.
> 
> The the monitor_can_read() function is called *before* the blocking
> poll() syscall.  If something changes the monitor_can_read() return
> value, you must* kick the event loop to ensure that the event loop
> reflects this change.
> 
> If you want to check how this works, see chardev/char-io.c for how
> fd_can_read() is used.
> 
> * Currently monitor.c doesn't need to kick the event loop explicitly
> because it runs within the main loop thread.  Therefore the event loop
> always calls monitor_can_read() again before entering poll().

Yeah I got it.  I was assuming the check is done in check() of the
watcher, but obviously I was wrong. :(

Thanks for clarifying this.  I'll do proper kicking.

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 07/26] monitor: unify global init

[Peter Xu]

On Mon, Dec 18, 2017 at 09:24:57AM +0000, Stefan Hajnoczi wrote:
> On Mon, Dec 18, 2017 at 11:27:02AM +0800, Peter Xu wrote:
> > On Sat, Dec 16, 2017 at 09:01:12AM +0000, Stefan Hajnoczi wrote:
> > > On Sat, Dec 16, 2017 at 11:52:28AM +0800, Peter Xu wrote:
> > > > On Fri, Dec 15, 2017 at 12:47:11PM +0000, Stefan Hajnoczi wrote:
> > > > > On Fri, Dec 15, 2017 at 04:11:41PM +0800, Peter Xu wrote:
> > > > > > On Wed, Dec 13, 2017 at 03:48:06PM +0000, Stefan Hajnoczi wrote:
> > > > > > > On Tue, Dec 05, 2017 at 01:51:41PM +0800, Peter Xu wrote:
> > >  }
> > > 
> > > Now vl.c:main() doesn't need to initialize the monitor.
> > > 
> > > We still need to handle the qtest_enabled() dependency:
> > > 
> > > -static QEMUClockType event_clock_type = QEMU_CLOCK_REALTIME;
> > > +static inline QEMUClockType event_clock_type(void)
> > > +{
> > > +    return qtest_enabled() ? QEMU_CLOCK_VIRTUAL : QEMU_CLOCK_REALTIME;
> > > +}
> > > 
> > > This way the qtest_enabled() call is deferred until later when the
> > > accelerators have been initialized.
> > 
> > Or... to be much simpler... How about we just use my patch?  IMHO it
> > calls monitor_init_globals() after configure_accelerator() so
> > monitor_qapi_event_init() and everything else would just work as
> > expected?
> 
> Ordering is complex in vl.c:main() so I think it's more maintainable to
> put initialization in a monitor.c constructor function and/or
> monitor_init().
> 
> If you want to keep your approach please add a comment about the
> qtest_enabled() dependency in vl.c:main().

Actually I still think this patch is the cleanest solution considering
all the above limitations. So I'll keep this one.  But for sure I can
add a comment mentioning about the dependency to accelerator codes.

Thanks!

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 21/26] qmp: isolate responses into io thread

[Peter Xu]

On Mon, Dec 18, 2017 at 08:40:47AM +0000, Stefan Hajnoczi wrote:
> On Mon, Dec 18, 2017 at 03:32:29PM +0800, Peter Xu wrote:
> > On Mon, Dec 18, 2017 at 01:52:17PM +0800, Peter Xu wrote:
> > > On Thu, Dec 14, 2017 at 01:43:59PM +0000, Stefan Hajnoczi wrote:
> > > > On Tue, Dec 05, 2017 at 01:51:55PM +0800, Peter Xu wrote:
> > > > > @@ -4429,6 +4515,13 @@ void monitor_cleanup(void)
> > > > >       */
> > > > >      iothread_stop(mon_global.mon_iothread);
> > > > >  
> > > > > +    /*
> > > > > +     * After we have IOThread to send responses, it's possible that
> > > > > +     * when we stop the IOThread there are still replies queued in the
> > > > > +     * responder queue.  Flush all of them.
> > > > > +     */
> > > > > +    monitor_qmp_bh_responder(NULL);
> > > > 
> > > > This doesn't work because monitor_qmp_bh_responder() does not guarantee
> > > > that the full response has been written when it returns.
> > > > 
> > > > When qemu_chr_fe_write() returns EAGAIN then qemu_chr_fe_add_watch() is
> > > > used to register an event loop callback when the chardev becomes
> > > > writable again.  But you stopped the event loop using iothread_stop() so
> > > > we will never complete the write.
> > > 
> > > Good catch...
> > > 
> > > Actually I just noticed that for char frontend I missed a place to use
> > > the chardev context for polling.  So before the flushing I possibly
> > > need this:
> > > 
> > > diff --git a/chardev/char-fe.c b/chardev/char-fe.c                             
> > > index ee6d596100..462c529f19 100644    
> > > --- a/chardev/char-fe.c                
> > > +++ b/chardev/char-fe.c                
> > > @@ -356,7 +356,7 @@ guint qemu_chr_fe_add_watch(CharBackend *be, GIOCondition cond,                                                                           
> > >      }                                 
> > >                                        
> > >      g_source_set_callback(src, (GSourceFunc)func, user_data, NULL);           
> > > -    tag = g_source_attach(src, NULL); 
> > > +    tag = g_source_attach(src, be->chr->gcontext);                            
> > >      g_source_unref(src);              
> > >                                        
> > >      return tag;                       
> > > 
> > > Otherwise it'll be still be run in main thread always.
> > > 
> > > (I guess I haven't yet encountered an EAGAIN for it so far)
> > > 
> > > > 
> > > > I suggest draining the monitor while the IOThread is still running
> > > > (that way the AioContext and GMainContext are still operational).  You
> > > > can:
> > > > 1. Suspend the monitor so new commands will not be read.
> > > > 2. Wait until all responses and outbuf are empty.
> > > > 
> > > > Another option is moving the chardev back to the main loop but I'm not
> > > > sure if the chardev subsystem supports that.
> > > 
> > > Your suggestion is good to me.  I'll do that in IOThread before it
> > > stops.  Thanks!
> > 
> > Hmm, after a second thought, I think maybe it would be nicer to just
> > call monitor_flush() after the monitor iothread is stopped.
> > 
> > Firstly, that is perfectly legal - since AFAICT commit 6cff3e8594
> > ("monitor: protect outbuf and mux_out with mutex", 2014-06-23) is
> > tailored for doing that, though that was for block iothread, not
> > monitor.
> > 
> > Then, it's still possible that the write buffer is full and we got
> > EAGAIN during flushing.  Then we'll plug another task into the monitor
> > gcontext which will never run since the iothread is stopped already.
> > IMHO that's totally fine - I think it's already our best effort to
> > flush the out buffer once at this point.  If the buffer is still full
> > after one attemp of monitor_flush(), we should just give up the 2nd
> > flushing instead of waiting here.  After all, we are quitting QEMU.
> > 
> > As a summary, my planned change of this issue would be quite
> > straightforward: call monitor_flush() when looping over monitors,
> > before destroying them.  And when destroying, if there is dangling
> > Monitor.out_watch, detach them.  Hope that works.  Thanks,
> 
> I think QEMU currently doesn't drain the monitor output buffer either.
> Please make sure to write the code and commit description so it is clear
> this is just a best-effort attempt.

Will do.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 26/26] tests: qmp-test: add oob test

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

On Mon, Dec 18, 2017 at 05:51:45PM +0800, Peter Xu wrote:
> On Thu, Dec 14, 2017 at 02:47:12PM +0000, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:52:00PM +0800, Peter Xu wrote:
> > > +    /* Try a fake capability, it should fail. */
> > > +    resp = qmp("{ 'execute': 'qmp_capabilities', "
> > > +               "  'arguments': { 'enable': [ 'cap-does-not-exist' ] } }");
> > > +    g_assert(qdict_haskey(resp, "error"));
> > 
> > Missing QDECREF(resp) here and throughout the rest of the function.
> 
> I believe I missed four QDECREF()s.

Yes.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 24/26] docs: update QMP documents for OOB commands

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 2510 bytes --]

On Mon, Dec 18, 2017 at 05:44:19PM +0800, Peter Xu wrote:
> On Thu, Dec 14, 2017 at 02:30:19PM +0000, Stefan Hajnoczi wrote:
> > On Tue, Dec 05, 2017 at 01:51:58PM +0800, Peter Xu wrote:
> > > diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
> > > index f04c63fe82..8597fdb087 100644
> > > --- a/docs/devel/qapi-code-gen.txt
> > > +++ b/docs/devel/qapi-code-gen.txt
> > > @@ -556,7 +556,8 @@ following example objects:
> > >  
> > >  Usage: { 'command': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT,
> > >           '*returns': TYPE-NAME, '*boxed': true,
> > > -         '*gen': false, '*success-response': false }
> > > +         '*gen': false, '*success-response': false,
> > > +         '*allow-oob': false }
> > >  
> > >  Commands are defined by using a dictionary containing several members,
> > >  where three members are most common.  The 'command' member is a
> > > @@ -636,6 +637,44 @@ possible, the command expression should include the optional key
> > >  'success-response' with boolean value false.  So far, only QGA makes
> > >  use of this member.
> > >  
> > > +Most of the QMP commands are handled sequentially in such a order:
> > > +Firstly, the JSON Parser parses the command request into some internal
> > > +message, delivers the message to QMP dispatchers. Secondly, the QMP
> > > +dispatchers will handle the commands one by one in time order, respond
> > > +when necessary.
> > 
> > The important points to cover about normal commands:
> > 1. They execute in order
> > 2. They run the main loop
> > 3. They run under the BQL
> > 
> > The other stuff about parsing requests into internal messages,
> > dispatchers, etc is not relevant.  It's better not to include it in
> > documentation because it can change and could also confuse readers
> > (since they don't need this info).
> 
> Ah yes.  I'm thinking whether I should just remove most of the changes
> to this file (qapi-code-gen.txt) since most of them are really not
> related to code gen at all...  Maybe somewhere in qmp-spec.txt as
> well?

qmp-spec.txt is the wire protocol that QMP clients and servers follow.
QEMU details like the BQL do not belong in qmp-spec.txt.

qapi-code-gen.txt is not a bad place to document the QEMU details that
are relevant to people writing QMP commands with allow-oob: true.  What
I was getting at is that this document should focus on things that QMP
command authors need to know, not on monitor implementation details.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] [RFC v5 24/26] docs: update QMP documents for OOB commands

[Peter Xu]

On Mon, Dec 18, 2017 at 02:09:36PM +0000, Stefan Hajnoczi wrote:
> On Mon, Dec 18, 2017 at 05:44:19PM +0800, Peter Xu wrote:
> > On Thu, Dec 14, 2017 at 02:30:19PM +0000, Stefan Hajnoczi wrote:
> > > On Tue, Dec 05, 2017 at 01:51:58PM +0800, Peter Xu wrote:
> > > > diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
> > > > index f04c63fe82..8597fdb087 100644
> > > > --- a/docs/devel/qapi-code-gen.txt
> > > > +++ b/docs/devel/qapi-code-gen.txt
> > > > @@ -556,7 +556,8 @@ following example objects:
> > > >  
> > > >  Usage: { 'command': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT,
> > > >           '*returns': TYPE-NAME, '*boxed': true,
> > > > -         '*gen': false, '*success-response': false }
> > > > +         '*gen': false, '*success-response': false,
> > > > +         '*allow-oob': false }
> > > >  
> > > >  Commands are defined by using a dictionary containing several members,
> > > >  where three members are most common.  The 'command' member is a
> > > > @@ -636,6 +637,44 @@ possible, the command expression should include the optional key
> > > >  'success-response' with boolean value false.  So far, only QGA makes
> > > >  use of this member.
> > > >  
> > > > +Most of the QMP commands are handled sequentially in such a order:
> > > > +Firstly, the JSON Parser parses the command request into some internal
> > > > +message, delivers the message to QMP dispatchers. Secondly, the QMP
> > > > +dispatchers will handle the commands one by one in time order, respond
> > > > +when necessary.
> > > 
> > > The important points to cover about normal commands:
> > > 1. They execute in order
> > > 2. They run the main loop
> > > 3. They run under the BQL
> > > 
> > > The other stuff about parsing requests into internal messages,
> > > dispatchers, etc is not relevant.  It's better not to include it in
> > > documentation because it can change and could also confuse readers
> > > (since they don't need this info).
> > 
> > Ah yes.  I'm thinking whether I should just remove most of the changes
> > to this file (qapi-code-gen.txt) since most of them are really not
> > related to code gen at all...  Maybe somewhere in qmp-spec.txt as
> > well?
> 
> qmp-spec.txt is the wire protocol that QMP clients and servers follow.
> QEMU details like the BQL do not belong in qmp-spec.txt.
> 
> qapi-code-gen.txt is not a bad place to document the QEMU details that
> are relevant to people writing QMP commands with allow-oob: true.  What
> I was getting at is that this document should focus on things that QMP
> command authors need to know, not on monitor implementation details.

Fair enough.  I'll try to avoid talking about QMP internals in either
of the documents.

Let me cook a better one.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC v5 00/26] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Thu, Dec 14, 2017 at 02:52:47PM +0000, Stefan Hajnoczi wrote:
> On Tue, Dec 05, 2017 at 01:51:34PM +0800, Peter Xu wrote:
> > This version is mostly document update, and dropped the single patch
> > that is migration related (will be put into postcopy recovery
> > series).
> 
> I've finished reviewing this revision.  The overall approach looks good.

Thanks for the in-depth review!  I'm posting another version soon
after the smoke test.

-- 
Peter Xu