[Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Peter Xu]

This series was born from this one:

  https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html

The design comes from Markus, and also the whole-bunch-of discussions
in previous thread.  My heartful thanks to Markus, Daniel, Dave,
Stefan, etc. on discussing the topic (...again!), providing shiny
ideas and suggestions.  Finally we got such a solution that seems to
satisfy everyone.

I re-started the versioning since this series is totally different
from previous one.  Now it's version 1.

In case new reviewers come along the way without reading previous
discussions, I will try to do a summary on what this is all about.

What is OOB execution?
======================

It's the shortcut of Out-Of-Band execution, its name is given by
Markus.  It's a way to quickly execute a QMP request.  Say, originally
QMP is going throw these steps:

      JSON Parser --> QMP Dispatcher --> Respond
          /|\    (2)                (3)     |
       (1) |                               \|/ (4)
           +---------  main thread  --------+

The requests are executed by the so-called QMP-dispatcher after the
JSON is parsed.  If OOB is on, we run the command directly in the
parser and quickly returns.

Yeah I know in current code the parser calls dispatcher directly
(please see handle_qmp_command()).  However it's not true again after
this series (parser will has its own IO thread, and dispatcher will
still be run in main thread).  So this OOB does brings something
different.

There are more details on why OOB and the difference/relationship
between OOB, async QMP, block/general jobs, etc.. but IMHO that's
slightly out of topic (and believe me, it's not easy for me to
summarize that).  For more information, please refers to [1].

Summary ends here.

Some Implementation Details
===========================

Again, I mentioned that the old QMP workflow is this:

      JSON Parser --> QMP Dispatcher --> Respond
          /|\    (2)                (3)     |
       (1) |                               \|/ (4)
           +---------  main thread  --------+

What this series does is, firstly:

      JSON Parser     QMP Dispatcher --> Respond
          /|\ |           /|\       (4)     |
           |  | (2)        | (3)            |  (5)
       (1) |  +----->      |               \|/
           +---------  main thread  <-------+

And further:

               queue/kick
     JSON Parser ======> QMP Dispatcher --> Respond
         /|\ |     (3)       /|\        (4)    |
      (1) |  | (2)            |                |  (5)
          | \|/               |               \|/
        IO thread         main thread  <-------+

Then it introduced the "allow-oob" parameter in QAPI schema to define
commands, and "run-oob" flag to let oob-allowed command to run in the
parser.

The last patch enables this for "migrate-incoming" command.

Please review.  Thanks.

[1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html

Peter Xu (15):
  char-io: fix possible race on IOWatchPoll
  qobject: allow NULL for qstring_get_str()
  qobject: introduce qobject_to_str()
  monitor: move skip_flush into monitor_data_init
  qjson: add "opaque" field to JSONMessageParser
  monitor: move the cur_mon hack deeper for QMP
  monitor: unify global init
  monitor: create IO thread
  monitor: allow to use IO thread for parsing
  monitor: introduce monitor_qmp_respond()
  monitor: separate QMP parser and dispatcher
  monitor: enable IO thread for (qmp & !mux) typed
  qapi: introduce new cmd option "allow-oob"
  qmp: support out-of-band (oob) execution
  qmp: let migrate-incoming allow out-of-band

 chardev/char-io.c                |  15 ++-
 docs/devel/qapi-code-gen.txt     |  51 ++++++-
 include/monitor/monitor.h        |   2 +-
 include/qapi/qmp/dispatch.h      |   2 +
 include/qapi/qmp/json-streamer.h |   8 +-
 include/qapi/qmp/qstring.h       |   1 +
 monitor.c                        | 283 +++++++++++++++++++++++++++++++--------
 qapi/introspect.json             |   6 +-
 qapi/migration.json              |   3 +-
 qapi/qmp-dispatch.c              |  34 +++++
 qga/main.c                       |   5 +-
 qobject/json-streamer.c          |   7 +-
 qobject/qjson.c                  |   5 +-
 qobject/qstring.c                |  13 +-
 scripts/qapi-commands.py         |  19 ++-
 scripts/qapi-introspect.py       |  10 +-
 scripts/qapi.py                  |  15 ++-
 scripts/qapi2texi.py             |   2 +-
 tests/libqtest.c                 |   5 +-
 tests/qapi-schema/test-qapi.py   |   2 +-
 trace-events                     |   2 +
 vl.c                             |   3 +-
 22 files changed, 398 insertions(+), 95 deletions(-)

-- 
2.7.4

[Qemu-devel] [RFC 01/15] char-io: fix possible race on IOWatchPoll

[Peter Xu]

This is not a problem if we are only having one single loop thread like
before.  However, after per-monitor thread is introduced, this is not
true any more, and the race can happen.

The race can be triggered with "make check -j8" sometimes:

  qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
  io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.

This patch keeps the reference for the watch object when creating in
io_add_watch_poll(), so that the object will never be released in the
context main loop, especially when the context loop is running in
another standalone thread.  Meanwhile, when we want to remove the watch
object, we always first detach the watch object from its owner context,
then we continue with the cleanup.

Without this patch, calling io_remove_watch_poll() in main loop thread
is not thread-safe, since the other per-monitor thread may be modifying
the watch object at the same time.

Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 chardev/char-io.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/chardev/char-io.c b/chardev/char-io.c
index f810524..3828c20 100644
--- a/chardev/char-io.c
+++ b/chardev/char-io.c
@@ -122,7 +122,6 @@ GSource *io_add_watch_poll(Chardev *chr,
     g_free(name);
 
     g_source_attach(&iwp->parent, context);
-    g_source_unref(&iwp->parent);
     return (GSource *)iwp;
 }
 
@@ -131,12 +130,24 @@ static void io_remove_watch_poll(GSource *source)
     IOWatchPoll *iwp;
 
     iwp = io_watch_poll_from_source(source);
+
+    /*
+     * Here the order of destruction really matters.  We need to first
+     * detach the IOWatchPoll object from the context (which may still
+     * be running in another loop thread), only after that could we
+     * continue to operate on iwp->src, or there may be race condition
+     * between current thread and the context loop thread.
+     *
+     * Let's blame the glib bug mentioned in commit 2b3167 (again) for
+     * this extra complexity.
+     */
+    g_source_destroy(&iwp->parent);
     if (iwp->src) {
         g_source_destroy(iwp->src);
         g_source_unref(iwp->src);
         iwp->src = NULL;
     }
-    g_source_destroy(&iwp->parent);
+    g_source_unref(&iwp->parent);
 }
 
 void remove_fd_in_watch(Chardev *chr)
-- 
2.7.4

[Qemu-devel] [RFC 02/15] qobject: allow NULL for qstring_get_str()

[Peter Xu]

Then I can get NULL rather than crash when calling things like:

  qstring_get_str(qobject_to_qstring(object));

when key does not exist.

CC: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qobject/qstring.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/qobject/qstring.c b/qobject/qstring.c
index 5da7b5f..c499fec 100644
--- a/qobject/qstring.c
+++ b/qobject/qstring.c
@@ -125,7 +125,7 @@ QString *qobject_to_qstring(const QObject *obj)
  */
 const char *qstring_get_str(const QString *qstring)
 {
-    return qstring->string;
+    return qstring ? qstring->string : NULL;
 }
 
 /**
-- 
2.7.4

[Qemu-devel] [RFC 03/15] qobject: introduce qobject_to_str()

[Peter Xu]

A quick way to fetch string from qobject when it's a QString.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/qstring.h |  1 +
 qobject/qstring.c          | 11 +++++++++++
 2 files changed, 12 insertions(+)

diff --git a/include/qapi/qmp/qstring.h b/include/qapi/qmp/qstring.h
index 10076b7..bfdfa46 100644
--- a/include/qapi/qmp/qstring.h
+++ b/include/qapi/qmp/qstring.h
@@ -27,6 +27,7 @@ QString *qstring_from_str(const char *str);
 QString *qstring_from_substr(const char *str, int start, int end);
 size_t qstring_get_length(const QString *qstring);
 const char *qstring_get_str(const QString *qstring);
+const char *qobject_get_str(const QObject *qstring);
 void qstring_append_int(QString *qstring, int64_t value);
 void qstring_append(QString *qstring, const char *str);
 void qstring_append_chr(QString *qstring, int c);
diff --git a/qobject/qstring.c b/qobject/qstring.c
index c499fec..103efaf 100644
--- a/qobject/qstring.c
+++ b/qobject/qstring.c
@@ -129,6 +129,17 @@ const char *qstring_get_str(const QString *qstring)
 }
 
 /**
+ * qstring_get_str(): Return a pointer of the backstore string
+ *
+ * NOTE: the string will only be returned if the object is QString,
+ * otherwise NULL is returned.
+ */
+const char *qobject_get_str(const QObject *qstring)
+{
+    return qstring_get_str(qobject_to_qstring(qstring));
+}
+
+/**
  * qstring_destroy_obj(): Free all memory allocated by a QString
  * object
  */
-- 
2.7.4

[Qemu-devel] [RFC 04/15] monitor: move skip_flush into monitor_data_init

[Peter Xu]

It's part of the data init.  Collect it.

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/monitor.c b/monitor.c
index 9239f7a..8b32519 100644
--- a/monitor.c
+++ b/monitor.c
@@ -568,13 +568,14 @@ static void monitor_qapi_event_init(void)
 
 static void handle_hmp_command(Monitor *mon, const char *cmdline);
 
-static void monitor_data_init(Monitor *mon)
+static void monitor_data_init(Monitor *mon, bool skip_flush)
 {
     memset(mon, 0, sizeof(Monitor));
     qemu_mutex_init(&mon->out_lock);
     mon->outbuf = qstring_new();
     /* Use *mon_cmds by default. */
     mon->cmd_table = mon_cmds;
+    mon->skip_flush = skip_flush;
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -594,8 +595,7 @@ char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
     char *output = NULL;
     Monitor *old_mon, hmp;
 
-    monitor_data_init(&hmp);
-    hmp.skip_flush = true;
+    monitor_data_init(&hmp, true);
 
     old_mon = cur_mon;
     cur_mon = &hmp;
@@ -4098,7 +4098,7 @@ void monitor_init(Chardev *chr, int flags)
     }
 
     mon = g_malloc(sizeof(*mon));
-    monitor_data_init(mon);
+    monitor_data_init(mon, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
-- 
2.7.4

[Qemu-devel] [RFC 05/15] qjson: add "opaque" field to JSONMessageParser

[Peter Xu]

It'll be passed to emit() as well when it happens.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/qapi/qmp/json-streamer.h | 8 ++++++--
 monitor.c                        | 7 ++++---
 qga/main.c                       | 5 +++--
 qobject/json-streamer.c          | 7 +++++--
 qobject/qjson.c                  | 5 +++--
 tests/libqtest.c                 | 5 +++--
 6 files changed, 24 insertions(+), 13 deletions(-)

diff --git a/include/qapi/qmp/json-streamer.h b/include/qapi/qmp/json-streamer.h
index 00d8a23..b83270c 100644
--- a/include/qapi/qmp/json-streamer.h
+++ b/include/qapi/qmp/json-streamer.h
@@ -25,16 +25,20 @@ typedef struct JSONToken {
 
 typedef struct JSONMessageParser
 {
-    void (*emit)(struct JSONMessageParser *parser, GQueue *tokens);
+    void (*emit)(struct JSONMessageParser *parser, GQueue *tokens, void *opaque);
     JSONLexer lexer;
     int brace_count;
     int bracket_count;
     GQueue *tokens;
     uint64_t token_size;
+    /* To be passed in when emit(). */
+    void *opaque;
 } JSONMessageParser;
 
 void json_message_parser_init(JSONMessageParser *parser,
-                              void (*func)(JSONMessageParser *, GQueue *));
+                              void (*func)(JSONMessageParser *, GQueue *,
+                                           void *opaque),
+                              void *opaque);
 
 int json_message_parser_feed(JSONMessageParser *parser,
                              const char *buffer, size_t size);
diff --git a/monitor.c b/monitor.c
index 8b32519..9096b64 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3817,7 +3817,8 @@ static int monitor_can_read(void *opaque)
     return (mon->suspend_cnt == 0) ? 1 : 0;
 }
 
-static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens)
+static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
+                               void *opaque)
 {
     QObject *req, *rsp = NULL, *id = NULL;
     QDict *qdict = NULL;
@@ -3964,7 +3965,7 @@ static void monitor_qmp_event(void *opaque, int event)
         break;
     case CHR_EVENT_CLOSED:
         json_message_parser_destroy(&mon->qmp.parser);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
         mon_refcount--;
         monitor_fdsets_cleanup();
         break;
@@ -4114,7 +4115,7 @@ void monitor_init(Chardev *chr, int flags)
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
                                  monitor_qmp_event, NULL, mon, NULL, true);
         qemu_chr_fe_set_echo(&mon->chr, true);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
     } else {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_read,
                                  monitor_event, NULL, mon, NULL, true);
diff --git a/qga/main.c b/qga/main.c
index 62a6275..3b5ebbc 100644
--- a/qga/main.c
+++ b/qga/main.c
@@ -593,7 +593,8 @@ static void process_command(GAState *s, QDict *req)
 }
 
 /* handle requests/control events coming in over the channel */
-static void process_event(JSONMessageParser *parser, GQueue *tokens)
+static void process_event(JSONMessageParser *parser, GQueue *tokens,
+                          void *opaque)
 {
     GAState *s = container_of(parser, GAState, parser);
     QDict *qdict;
@@ -1320,7 +1321,7 @@ static int run_agent(GAState *s, GAConfig *config, int socket_activation)
     s->command_state = ga_command_state_new();
     ga_command_state_init(s, s->command_state);
     ga_command_state_init_all(s->command_state);
-    json_message_parser_init(&s->parser, process_event);
+    json_message_parser_init(&s->parser, process_event, NULL);
 
 #ifndef _WIN32
     if (!register_signal_handlers()) {
diff --git a/qobject/json-streamer.c b/qobject/json-streamer.c
index c51c202..12865d5 100644
--- a/qobject/json-streamer.c
+++ b/qobject/json-streamer.c
@@ -102,18 +102,21 @@ out_emit:
      */
     tokens = parser->tokens;
     parser->tokens = g_queue_new();
-    parser->emit(parser, tokens);
+    parser->emit(parser, tokens, parser->opaque);
     parser->token_size = 0;
 }
 
 void json_message_parser_init(JSONMessageParser *parser,
-                              void (*func)(JSONMessageParser *, GQueue *))
+                              void (*func)(JSONMessageParser *,
+                                           GQueue *, void *opaque),
+                              void *opaque)
 {
     parser->emit = func;
     parser->brace_count = 0;
     parser->bracket_count = 0;
     parser->tokens = g_queue_new();
     parser->token_size = 0;
+    parser->opaque = opaque;
 
     json_lexer_init(&parser->lexer, json_message_process_token);
 }
diff --git a/qobject/qjson.c b/qobject/qjson.c
index 2e09308..f9766fe 100644
--- a/qobject/qjson.c
+++ b/qobject/qjson.c
@@ -28,7 +28,8 @@ typedef struct JSONParsingState
     Error *err;
 } JSONParsingState;
 
-static void parse_json(JSONMessageParser *parser, GQueue *tokens)
+static void parse_json(JSONMessageParser *parser, GQueue *tokens,
+                       void *opaque)
 {
     JSONParsingState *s = container_of(parser, JSONParsingState, parser);
 
@@ -41,7 +42,7 @@ QObject *qobject_from_jsonv(const char *string, va_list *ap, Error **errp)
 
     state.ap = ap;
 
-    json_message_parser_init(&state.parser, parse_json);
+    json_message_parser_init(&state.parser, parse_json, NULL);
     json_message_parser_feed(&state.parser, string, strlen(string));
     json_message_parser_flush(&state.parser);
     json_message_parser_destroy(&state.parser);
diff --git a/tests/libqtest.c b/tests/libqtest.c
index 0c12b38..fbd8e12 100644
--- a/tests/libqtest.c
+++ b/tests/libqtest.c
@@ -387,7 +387,8 @@ typedef struct {
     QDict *response;
 } QMPResponseParser;
 
-static void qmp_response(JSONMessageParser *parser, GQueue *tokens)
+static void qmp_response(JSONMessageParser *parser, GQueue *tokens,
+                         void *opaque)
 {
     QMPResponseParser *qmp = container_of(parser, QMPResponseParser, parser);
     QObject *obj;
@@ -409,7 +410,7 @@ QDict *qmp_fd_receive(int fd)
     bool log = getenv("QTEST_LOG") != NULL;
 
     qmp.response = NULL;
-    json_message_parser_init(&qmp.parser, qmp_response);
+    json_message_parser_init(&qmp.parser, qmp_response, NULL);
     while (!qmp.response) {
         ssize_t len;
         char c;
-- 
2.7.4

[Qemu-devel] [RFC 06/15] monitor: move the cur_mon hack deeper for QMP

[Peter Xu]

In monitor_qmp_read(), we have the hack to temporarily replace the
cur_mon pointer.  Now we move this hack deeper inside the QMP dispatcher
routine since the Monitor pointer can be passed in to that using the new
JSON Parser opaque field now.

This does not make much sense as a single patch.  However, this will be
a big step for the next patch, when the QMP dispatcher routine will be
splitted from the QMP parser.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/monitor.c b/monitor.c
index 9096b64..d7eb3c2 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3822,7 +3822,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
 {
     QObject *req, *rsp = NULL, *id = NULL;
     QDict *qdict = NULL;
-    Monitor *mon = cur_mon;
+    Monitor *mon = opaque, *old_mon;
     Error *err = NULL;
 
     req = json_parser_parse_err(tokens, NULL, &err);
@@ -3847,8 +3847,13 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
         QDECREF(req_json);
     }
 
+    old_mon = cur_mon;
+    cur_mon = mon;
+
     rsp = qmp_dispatch(cur_mon->qmp.commands, req);
 
+    cur_mon = old_mon;
+
     if (mon->qmp.commands == &qmp_cap_negotiation_commands) {
         qdict = qdict_get_qdict(qobject_to_qdict(rsp), "error");
         if (qdict
@@ -3885,13 +3890,9 @@ err_out:
 
 static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
 {
-    Monitor *old_mon = cur_mon;
-
-    cur_mon = opaque;
-
-    json_message_parser_feed(&cur_mon->qmp.parser, (const char *) buf, size);
+    Monitor *mon = opaque;
 
-    cur_mon = old_mon;
+    json_message_parser_feed(&mon->qmp.parser, (const char *) buf, size);
 }
 
 static void monitor_read(void *opaque, const uint8_t *buf, int size)
@@ -3965,7 +3966,7 @@ static void monitor_qmp_event(void *opaque, int event)
         break;
     case CHR_EVENT_CLOSED:
         json_message_parser_destroy(&mon->qmp.parser);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
         mon_refcount--;
         monitor_fdsets_cleanup();
         break;
@@ -4115,7 +4116,7 @@ void monitor_init(Chardev *chr, int flags)
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
                                  monitor_qmp_event, NULL, mon, NULL, true);
         qemu_chr_fe_set_echo(&mon->chr, true);
-        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, NULL);
+        json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
     } else {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_read,
                                  monitor_event, NULL, mon, NULL, true);
-- 
2.7.4

[Qemu-devel] [RFC 07/15] monitor: unify global init

[Peter Xu]

There are many places for monitor init its globals, at least:

- monitor_init_qmp_commands() at the very beginning
- single function to init monitor_lock
- in the first entry of monitor_init() using "is_first_init"

Unify them a bit.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 include/monitor/monitor.h |  2 +-
 monitor.c                 | 25 ++++++++++---------------
 vl.c                      |  3 ++-
 3 files changed, 13 insertions(+), 17 deletions(-)

diff --git a/include/monitor/monitor.h b/include/monitor/monitor.h
index 83ea4a1..3a5128a 100644
--- a/include/monitor/monitor.h
+++ b/include/monitor/monitor.h
@@ -16,7 +16,7 @@ extern Monitor *cur_mon;
 
 bool monitor_cur_is_qmp(void);
 
-void monitor_init_qmp_commands(void);
+void monitor_init_globals(void);
 void monitor_init(Chardev *chr, int flags);
 void monitor_cleanup(void);
 
diff --git a/monitor.c b/monitor.c
index d7eb3c2..7bd2e90 100644
--- a/monitor.c
+++ b/monitor.c
@@ -1000,7 +1000,7 @@ static void qmp_unregister_commands_hack(void)
 #endif
 }
 
-void monitor_init_qmp_commands(void)
+static void monitor_init_qmp_commands(void)
 {
     /*
      * Two command lists:
@@ -4043,6 +4043,14 @@ static void sortcmdlist(void)
     qsort((void *)info_cmds, array_num, elem_size, compare_mon_cmd);
 }
 
+void monitor_init_globals(void)
+{
+    monitor_init_qmp_commands();
+    monitor_qapi_event_init();
+    sortcmdlist();
+    qemu_mutex_init(&monitor_lock);
+}
+
 /* These functions just adapt the readline interface in a typesafe way.  We
  * could cast function pointers but that discards compiler checks.
  */
@@ -4083,23 +4091,10 @@ void error_vprintf_unless_qmp(const char *fmt, va_list ap)
     }
 }
 
-static void __attribute__((constructor)) monitor_lock_init(void)
-{
-    qemu_mutex_init(&monitor_lock);
-}
-
 void monitor_init(Chardev *chr, int flags)
 {
-    static int is_first_init = 1;
-    Monitor *mon;
-
-    if (is_first_init) {
-        monitor_qapi_event_init();
-        sortcmdlist();
-        is_first_init = 0;
-    }
+    Monitor *mon = g_malloc(sizeof(*mon));
 
-    mon = g_malloc(sizeof(*mon));
     monitor_data_init(mon, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
diff --git a/vl.c b/vl.c
index fb1f05b..850cf55 100644
--- a/vl.c
+++ b/vl.c
@@ -3049,7 +3049,6 @@ int main(int argc, char **argv, char **envp)
     qemu_init_exec_dir(argv[0]);
 
     module_call_init(MODULE_INIT_QOM);
-    monitor_init_qmp_commands();
 
     qemu_add_opts(&qemu_drive_opts);
     qemu_add_drive_opts(&qemu_legacy_drive_opts);
@@ -4587,6 +4586,8 @@ int main(int argc, char **argv, char **envp)
 
     parse_numa_opts(current_machine);
 
+    monitor_init_globals();
+
     if (qemu_opts_foreach(qemu_find_opts("mon"),
                           mon_init_func, NULL, NULL)) {
         exit(1);
-- 
2.7.4

[Qemu-devel] [RFC 08/15] monitor: create IO thread

[Peter Xu]

Create one IO thread for the monitors, prepared to handle all the
input/output IOs.  Only adding the thread, loop and context, but doing
nothing else yet.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/monitor.c b/monitor.c
index 7bd2e90..9e9a32e 100644
--- a/monitor.c
+++ b/monitor.c
@@ -208,6 +208,14 @@ struct Monitor {
     QLIST_ENTRY(Monitor) entry;
 };
 
+struct MonitorGlobal {
+    QemuThread mon_io_thread;
+    GMainContext *mon_context;
+    GMainLoop *mon_loop;
+};
+
+static struct MonitorGlobal mon_global;
+
 /* QMP checker flags */
 #define QMP_ACCEPT_UNKNOWNS 1
 
@@ -4043,12 +4051,32 @@ static void sortcmdlist(void)
     qsort((void *)info_cmds, array_num, elem_size, compare_mon_cmd);
 }
 
+static void *monitor_io_thread(void *data)
+{
+    rcu_register_thread();
+
+    g_main_loop_run(mon_global.mon_loop);
+
+    rcu_unregister_thread();
+
+    return NULL;
+}
+
+static void monitor_io_thread_init(void)
+{
+    mon_global.mon_context = g_main_context_new();
+    mon_global.mon_loop = g_main_loop_new(mon_global.mon_context, TRUE);
+    qemu_thread_create(&mon_global.mon_io_thread, "mon-io-thr",
+                       monitor_io_thread, NULL, QEMU_THREAD_JOINABLE);
+}
+
 void monitor_init_globals(void)
 {
     monitor_init_qmp_commands();
     monitor_qapi_event_init();
     sortcmdlist();
     qemu_mutex_init(&monitor_lock);
+    monitor_io_thread_init();
 }
 
 /* These functions just adapt the readline interface in a typesafe way.  We
@@ -4122,6 +4150,25 @@ void monitor_init(Chardev *chr, int flags)
     qemu_mutex_unlock(&monitor_lock);
 }
 
+static void monitor_io_thread_destroy(void)
+{
+    /* Notify the monitor IO thread to quit. */
+    g_main_loop_quit(mon_global.mon_loop);
+    /*
+     * Make sure the context will get the quit message since it's in
+     * another thread.  Without this, it may not be able to respond to
+     * the quit message immediately.
+     */
+    g_main_context_wakeup(mon_global.mon_context);
+    qemu_thread_join(&mon_global.mon_io_thread);
+
+    g_main_loop_unref(mon_global.mon_loop);
+    mon_global.mon_loop = NULL;
+
+    g_main_context_unref(mon_global.mon_context);
+    mon_global.mon_context = NULL;
+}
+
 void monitor_cleanup(void)
 {
     Monitor *mon, *next;
@@ -4133,6 +4180,8 @@ void monitor_cleanup(void)
         g_free(mon);
     }
     qemu_mutex_unlock(&monitor_lock);
+
+    monitor_io_thread_destroy();
 }
 
 QemuOptsList qemu_mon_opts = {
-- 
2.7.4

[Qemu-devel] [RFC 09/15] monitor: allow to use IO thread for parsing

[Peter Xu]

For each Monitor, add one field "use_io_thr" to show whether it will be
using the dedicated monitor IO thread to handle input/output.  When set,
monitor IO parsing work will be offloaded to dedicated monitor IO
thread, rather than the original main loop thread.

This only works for QMP.  HMP will always be run on main loop thread.

Currently we're still keeping use_io_thr to off always.  Will turn it on
later at some point.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/monitor.c b/monitor.c
index 9e9a32e..83f5e87 100644
--- a/monitor.c
+++ b/monitor.c
@@ -190,6 +190,7 @@ struct Monitor {
     int flags;
     int suspend_cnt;
     bool skip_flush;
+    bool use_io_thr;
 
     QemuMutex out_lock;
     QString *outbuf;
@@ -576,7 +577,8 @@ static void monitor_qapi_event_init(void)
 
 static void handle_hmp_command(Monitor *mon, const char *cmdline);
 
-static void monitor_data_init(Monitor *mon, bool skip_flush)
+static void monitor_data_init(Monitor *mon, bool skip_flush,
+                              bool use_io_thr)
 {
     memset(mon, 0, sizeof(Monitor));
     qemu_mutex_init(&mon->out_lock);
@@ -584,6 +586,7 @@ static void monitor_data_init(Monitor *mon, bool skip_flush)
     /* Use *mon_cmds by default. */
     mon->cmd_table = mon_cmds;
     mon->skip_flush = skip_flush;
+    mon->use_io_thr = use_io_thr;
 }
 
 static void monitor_data_destroy(Monitor *mon)
@@ -603,7 +606,7 @@ char *qmp_human_monitor_command(const char *command_line, bool has_cpu_index,
     char *output = NULL;
     Monitor *old_mon, hmp;
 
-    monitor_data_init(&hmp, true);
+    monitor_data_init(&hmp, true, false);
 
     old_mon = cur_mon;
     cur_mon = &hmp;
@@ -4122,8 +4125,9 @@ void error_vprintf_unless_qmp(const char *fmt, va_list ap)
 void monitor_init(Chardev *chr, int flags)
 {
     Monitor *mon = g_malloc(sizeof(*mon));
+    GMainContext *context;
 
-    monitor_data_init(mon, false);
+    monitor_data_init(mon, false, false);
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
@@ -4135,9 +4139,22 @@ void monitor_init(Chardev *chr, int flags)
         monitor_read_command(mon, 0);
     }
 
+    if (mon->use_io_thr) {
+        /*
+         * When use_io_thr is set, we use the global shared dedicated
+         * IO thread for this monitor to handle input/output.
+         */
+        context = mon_global.mon_context;
+        /* We should have inited globals before reaching here. */
+        assert(context);
+    } else {
+        /* The default main loop, which is the main thread */
+        context = NULL;
+    }
+
     if (monitor_is_qmp(mon)) {
         qemu_chr_fe_set_handlers(&mon->chr, monitor_can_read, monitor_qmp_read,
-                                 monitor_qmp_event, NULL, mon, NULL, true);
+                                 monitor_qmp_event, NULL, mon, context, true);
         qemu_chr_fe_set_echo(&mon->chr, true);
         json_message_parser_init(&mon->qmp.parser, handle_qmp_command, mon);
     } else {
-- 
2.7.4

[Qemu-devel] [RFC 10/15] monitor: introduce monitor_qmp_respond()

[Peter Xu]

A tiny refactoring, preparing to split the QMP dispatcher away.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 48 +++++++++++++++++++++++++++++++-----------------
 1 file changed, 31 insertions(+), 17 deletions(-)

diff --git a/monitor.c b/monitor.c
index 83f5e87..aa0c384 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3828,6 +3828,36 @@ static int monitor_can_read(void *opaque)
     return (mon->suspend_cnt == 0) ? 1 : 0;
 }
 
+/*
+ * When rsp/err/id is passed in, the function will be responsible for
+ * the cleanup.
+ */
+static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
+                                Error *err, QObject *id)
+{
+    QDict *qdict = NULL;
+
+    if (err) {
+        qdict = qdict_new();
+        qdict_put_obj(qdict, "error", qmp_build_error_object(err));
+        error_free(err);
+        rsp = QOBJECT(qdict);
+    }
+
+    if (rsp) {
+        if (id) {
+            /* This is for the qdict below. */
+            qobject_incref(id);
+            qdict_put_obj(qobject_to_qdict(rsp), "id", id);
+        }
+
+        monitor_json_emitter(mon, rsp);
+    }
+
+    qobject_decref(id);
+    qobject_decref(rsp);
+}
+
 static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
                                void *opaque)
 {
@@ -3878,24 +3908,8 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
     }
 
 err_out:
-    if (err) {
-        qdict = qdict_new();
-        qdict_put_obj(qdict, "error", qmp_build_error_object(err));
-        error_free(err);
-        rsp = QOBJECT(qdict);
-    }
+    monitor_qmp_respond(mon, rsp, err, id);
 
-    if (rsp) {
-        if (id) {
-            qdict_put_obj(qobject_to_qdict(rsp), "id", id);
-            id = NULL;
-        }
-
-        monitor_json_emitter(mon, rsp);
-    }
-
-    qobject_decref(id);
-    qobject_decref(rsp);
     qobject_decref(req);
 }
 
-- 
2.7.4

[Qemu-devel] [RFC 11/15] monitor: separate QMP parser and dispatcher

[Peter Xu]

Originally QMP is going throw these steps:

  JSON Parser --> QMP Dispatcher --> Respond
      /|\    (2)                (3)     |
   (1) |                               \|/ (4)
       +---------  main thread  --------+

This patch does this:

  JSON Parser     QMP Dispatcher --> Respond
      /|\ |           /|\       (4)     |
       |  | (2)        | (3)            |  (5)
   (1) |  +----->      |               \|/
       +---------  main thread  <-------+

So the parsing job and the dispatching job is isolated now.  It gives us
a chance in following up patches to totally move the parser outside.

The isloation is done using one QEMUBH. Only one dispatcher BH for all
the monitors.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 122 ++++++++++++++++++++++++++++++++++++++++++++++++++------------
 1 file changed, 99 insertions(+), 23 deletions(-)

diff --git a/monitor.c b/monitor.c
index aa0c384..f649d6a 100644
--- a/monitor.c
+++ b/monitor.c
@@ -213,6 +213,10 @@ struct MonitorGlobal {
     QemuThread mon_io_thread;
     GMainContext *mon_context;
     GMainLoop *mon_loop;
+    /* Bottom half to dispatch the requests received from IO thread */
+    QEMUBH *qmp_dispatcher_bh;
+    /* Input queue that hangs all the parsed QMP requests */
+    GQueue *qmp_requests;
 };
 
 static struct MonitorGlobal mon_global;
@@ -3858,29 +3862,31 @@ static void monitor_qmp_respond(Monitor *mon, QObject *rsp,
     qobject_decref(rsp);
 }
 
-static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
-                               void *opaque)
+struct QMPRequest {
+    /* Owner of the request */
+    Monitor *mon;
+    /* "id" field of the request */
+    QObject *id;
+    /* Request object to be handled */
+    QObject *req;
+};
+typedef struct QMPRequest QMPRequest;
+
+/*
+ * Dispatch one single QMP request. The function will free the req_obj
+ * and objects inside it before return.
+ */
+static void monitor_qmp_dispatch_one(QMPRequest *req_obj)
 {
-    QObject *req, *rsp = NULL, *id = NULL;
+    Monitor *mon, *old_mon;
+    QObject *req, *rsp = NULL, *id;
     QDict *qdict = NULL;
-    Monitor *mon = opaque, *old_mon;
-    Error *err = NULL;
 
-    req = json_parser_parse_err(tokens, NULL, &err);
-    if (!req && !err) {
-        /* json_parser_parse_err() sucks: can fail without setting @err */
-        error_setg(&err, QERR_JSON_PARSING);
-    }
-    if (err) {
-        goto err_out;
-    }
+    req = req_obj->req;
+    mon = req_obj->mon;
+    id = req_obj->id;
 
-    qdict = qobject_to_qdict(req);
-    if (qdict) {
-        id = qdict_get(qdict, "id");
-        qobject_incref(id);
-        qdict_del(qdict, "id");
-    } /* else will fail qmp_dispatch() */
+    g_free(req_obj);
 
     if (trace_event_get_state_backends(TRACE_HANDLE_QMP_COMMAND)) {
         QString *req_json = qobject_to_json(req);
@@ -3891,7 +3897,7 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
     old_mon = cur_mon;
     cur_mon = mon;
 
-    rsp = qmp_dispatch(cur_mon->qmp.commands, req);
+    rsp = qmp_dispatch(mon->qmp.commands, req);
 
     cur_mon = old_mon;
 
@@ -3907,12 +3913,66 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
         }
     }
 
-err_out:
-    monitor_qmp_respond(mon, rsp, err, id);
-
+    /* Respond if necessary */
+    monitor_qmp_respond(mon, rsp, NULL, id);
     qobject_decref(req);
 }
 
+static void monitor_qmp_bh_dispatcher(void *data)
+{
+    QMPRequest *req_obj;
+
+    while (true) {
+        req_obj = g_queue_pop_head(mon_global.qmp_requests);
+        if (!req_obj) {
+            break;
+        }
+        monitor_qmp_dispatch_one(req_obj);
+    }
+}
+
+static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
+                               void *opaque)
+{
+    QObject *req, *id = NULL;
+    QDict *qdict = NULL;
+    Monitor *mon = opaque;
+    Error *err = NULL;
+    QMPRequest *req_obj;
+
+    req = json_parser_parse_err(tokens, NULL, &err);
+    if (!req && !err) {
+        /* json_parser_parse_err() sucks: can fail without setting @err */
+        error_setg(&err, QERR_JSON_PARSING);
+    }
+    if (err) {
+        monitor_qmp_respond(mon, NULL, err, NULL);
+        qobject_decref(req);
+    }
+
+    qdict = qobject_to_qdict(req);
+    if (qdict) {
+        id = qdict_get(qdict, "id");
+        qobject_incref(id);
+        qdict_del(qdict, "id");
+    } /* else will fail qmp_dispatch() */
+
+    req_obj = g_new0(QMPRequest, 1);
+    req_obj->mon = mon;
+    req_obj->id = id;
+    req_obj->req = req;
+
+    /*
+     * Put the request to the end of queue so that requests will be
+     * handled in time order.  Ownership for req_obj, req, id,
+     * etc. will be delivered to the handler side.
+     */
+    g_queue_push_tail(mon_global.qmp_requests, req_obj);
+
+    /* Kick the dispatcher routine */
+    qemu_bh_schedule(mon_global.qmp_dispatcher_bh);
+}
+
 static void monitor_qmp_read(void *opaque, const uint8_t *buf, int size)
 {
     Monitor *mon = opaque;
@@ -4085,6 +4145,16 @@ static void monitor_io_thread_init(void)
     mon_global.mon_loop = g_main_loop_new(mon_global.mon_context, TRUE);
     qemu_thread_create(&mon_global.mon_io_thread, "mon-io-thr",
                        monitor_io_thread, NULL, QEMU_THREAD_JOINABLE);
+
+    /*
+     * This MUST be on main loop thread since we have commands that
+     * have assumption to be run on main loop thread (Yeah, we'd
+     * better remove this assumption in the future).
+     */
+    mon_global.qmp_dispatcher_bh = aio_bh_new(qemu_get_aio_context(),
+                                              monitor_qmp_bh_dispatcher,
+                                              NULL);
+    mon_global.qmp_requests = g_queue_new();
 }
 
 void monitor_init_globals(void)
@@ -4213,6 +4283,12 @@ void monitor_cleanup(void)
     qemu_mutex_unlock(&monitor_lock);
 
     monitor_io_thread_destroy();
+
+    qemu_bh_delete(mon_global.qmp_dispatcher_bh);
+    mon_global.qmp_dispatcher_bh = NULL;
+
+    g_queue_free(mon_global.qmp_requests);
+    mon_global.qmp_requests = NULL;
 }
 
 QemuOptsList qemu_mon_opts = {
-- 
2.7.4

[Qemu-devel] [RFC 12/15] monitor: enable IO thread for (qmp & !mux) typed

[Peter Xu]

Start to use dedicate IO thread for QMP monitors that are not using
MUXed chardev.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 monitor.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/monitor.c b/monitor.c
index f649d6a..599ea36 100644
--- a/monitor.c
+++ b/monitor.c
@@ -36,6 +36,7 @@
 #include "net/net.h"
 #include "net/slirp.h"
 #include "chardev/char-fe.h"
+#include "chardev/char-mux.h"
 #include "ui/qemu-spice.h"
 #include "sysemu/numa.h"
 #include "monitor/monitor.h"
@@ -4211,7 +4212,7 @@ void monitor_init(Chardev *chr, int flags)
     Monitor *mon = g_malloc(sizeof(*mon));
     GMainContext *context;
 
-    monitor_data_init(mon, false, false);
+    monitor_data_init(mon, false, !CHARDEV_IS_MUX(chr));
 
     qemu_chr_fe_init(&mon->chr, chr, &error_abort);
     mon->flags = flags;
-- 
2.7.4

[Qemu-devel] [RFC 13/15] qapi: introduce new cmd option "allow-oob"

[Peter Xu]

Here "oob" stands for "Out-Of-Band".  When "allow-oob" is set, it means
the command allows out-of-band execution.  Please see the spec update
for more details.

The "oob" idea is proposed by Markus Armbruster in following thread:

  https://lists.gnu.org/archive/html/qemu-devel/2017-09/msg02057.html

This new "allow-oob" boolean will be exposed by "query-qmp-schema" as
well for command entries, so that QMP clients can know which command can
be used as out-of-band calls. For example the command "migrate"
originally looks like:

  {"name": "migrate", "ret-type": "17", "meta-type": "command",
   "arg-type": "86"}

And it'll be changed into:

  {"name": "migrate", "ret-type": "17", "allow-oob": false,
   "meta-type": "command", "arg-type": "86"}

This patch only provides the QMP interface level changes.  It does not
contains the real out-of-band execution implementation yet.

Suggested-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
---
 docs/devel/qapi-code-gen.txt   | 41 ++++++++++++++++++++++++++++++++++++-----
 include/qapi/qmp/dispatch.h    |  1 +
 qapi/introspect.json           |  6 +++++-
 scripts/qapi-commands.py       | 19 ++++++++++++++-----
 scripts/qapi-introspect.py     | 10 ++++++++--
 scripts/qapi.py                | 15 ++++++++++-----
 scripts/qapi2texi.py           |  2 +-
 tests/qapi-schema/test-qapi.py |  2 +-
 8 files changed, 76 insertions(+), 20 deletions(-)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index f04c63f..61fa167 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -556,7 +556,8 @@ following example objects:
 
 Usage: { 'command': STRING, '*data': COMPLEX-TYPE-NAME-OR-DICT,
          '*returns': TYPE-NAME, '*boxed': true,
-         '*gen': false, '*success-response': false }
+         '*gen': false, '*success-response': false,
+         '*allow-oob': false }
 
 Commands are defined by using a dictionary containing several members,
 where three members are most common.  The 'command' member is a
@@ -636,6 +637,34 @@ possible, the command expression should include the optional key
 'success-response' with boolean value false.  So far, only QGA makes
 use of this member.
 
+Most of the QMP commands are handled sequentially in such a order.
+Firstly, the JSON Parser parses the command request into some internal
+message, then it delivers the message to QMP dispatchers; secondly,
+the QMP dispatchers will handle the commands one by one in time order,
+respond when necessary.  For some commands that always complete
+"quickly" can instead be executed directly during parsing, at the QMP
+client's request.  This kind of commands that allow direct execution
+is called "out-of-band" ("oob" as shortcut) commands. the response can
+overtake prior in-band commands' responses.  By default, commands are
+always in-band.  We need to explicitly specify "allow-oob" to "True"
+to show that one command can be run out-of-band.
+
+One thing to mention is that, although out-of-band execution of
+commands benefit from quick and asynchronous execution, it need to
+satisfy at least the following:
+
+(1) It is extremely quick and never blocks, so that its execution will
+    not block parsing routine of any other monitors.
+
+(2) It does not need BQL, since the parser can be run without BQL,
+    while the dispatcher is always with BQL held.
+
+If not, the command is not suitable to be allowed to run out-of-band,
+and it should set its "allow-oob" to "False".  Whether a command is
+allowed to run out-of-band can also be introspected using
+query-qmp-schema command.  Please see the section "Client JSON
+Protocol introspection" for more information.
+
 
 === Events ===
 
@@ -739,10 +768,12 @@ references by name.
 QAPI schema definitions not reachable that way are omitted.
 
 The SchemaInfo for a command has meta-type "command", and variant
-members "arg-type" and "ret-type".  On the wire, the "arguments"
-member of a client's "execute" command must conform to the object type
-named by "arg-type".  The "return" member that the server passes in a
-success response conforms to the type named by "ret-type".
+members "arg-type", "ret-type" and "allow-oob".  On the wire, the
+"arguments" member of a client's "execute" command must conform to the
+object type named by "arg-type".  The "return" member that the server
+passes in a success response conforms to the type named by
+"ret-type".  When "allow-oob" is set, it means the command supports
+out-of-band execution.
 
 If the command takes no arguments, "arg-type" names an object type
 without members.  Likewise, if the command returns nothing, "ret-type"
diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
index 20578dc..b767988 100644
--- a/include/qapi/qmp/dispatch.h
+++ b/include/qapi/qmp/dispatch.h
@@ -23,6 +23,7 @@ typedef enum QmpCommandOptions
 {
     QCO_NO_OPTIONS = 0x0,
     QCO_NO_SUCCESS_RESP = 0x1,
+    QCO_ALLOW_OOB = 0x2,
 } QmpCommandOptions;
 
 typedef struct QmpCommand
diff --git a/qapi/introspect.json b/qapi/introspect.json
index 5b3e6e9..57cc137 100644
--- a/qapi/introspect.json
+++ b/qapi/introspect.json
@@ -259,12 +259,16 @@
 #
 # @ret-type: the name of the command's result type.
 #
+# @allow-oob: whether the command allows out-of-band execution.
+#             (Since: 2.11)
+#
 # TODO: @success-response (currently irrelevant, because it's QGA, not QMP)
 #
 # Since: 2.5
 ##
 { 'struct': 'SchemaInfoCommand',
-  'data': { 'arg-type': 'str', 'ret-type': 'str' } }
+  'data': { 'arg-type': 'str', 'ret-type': 'str',
+            'allow-oob': 'bool' } }
 
 ##
 # @SchemaInfoEvent:
diff --git a/scripts/qapi-commands.py b/scripts/qapi-commands.py
index 974d0a4..b2b0bc0 100644
--- a/scripts/qapi-commands.py
+++ b/scripts/qapi-commands.py
@@ -192,10 +192,18 @@ out:
     return ret
 
 
-def gen_register_command(name, success_response):
-    options = 'QCO_NO_OPTIONS'
+def gen_register_command(name, success_response, allow_oob):
+    options = []
+
     if not success_response:
-        options = 'QCO_NO_SUCCESS_RESP'
+        options += ['QCO_NO_SUCCESS_RESP']
+    if allow_oob:
+        options += ['QCO_ALLOW_OOB']
+
+    if not options:
+        options = ['QCO_NO_OPTIONS']
+
+    options = " | ".join(options)
 
     ret = mcgen('''
     qmp_register_command(cmds, "%(name)s",
@@ -241,7 +249,7 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
         self._visited_ret_types = None
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         if not gen:
             return
         self.decl += gen_command_decl(name, arg_type, boxed, ret_type)
@@ -250,7 +258,8 @@ class QAPISchemaGenCommandVisitor(QAPISchemaVisitor):
             self.defn += gen_marshal_output(ret_type)
         self.decl += gen_marshal_decl(name)
         self.defn += gen_marshal(name, arg_type, boxed, ret_type)
-        self._regy += gen_register_command(name, success_response)
+        self._regy += gen_register_command(name, success_response,
+                                           allow_oob)
 
 
 (input_file, output_dir, do_c, do_h, prefix, opts) = parse_command_line()
diff --git a/scripts/qapi-introspect.py b/scripts/qapi-introspect.py
index 032bcea..9fbf88b 100644
--- a/scripts/qapi-introspect.py
+++ b/scripts/qapi-introspect.py
@@ -28,6 +28,11 @@ def to_json(obj, level=0):
                               to_json(obj[key], level + 1))
                 for key in sorted(obj.keys())]
         ret = '{' + ', '.join(elts) + '}'
+    elif isinstance(obj, bool):
+        if obj:
+            ret = 'true'
+        else:
+            ret = 'false'
     else:
         assert False                # not implemented
     if level == 1:
@@ -154,12 +159,13 @@ const char %(c_name)s[] = %(c_string)s;
                                     for m in variants.variants]})
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         arg_type = arg_type or self._schema.the_empty_object_type
         ret_type = ret_type or self._schema.the_empty_object_type
         self._gen_json(name, 'command',
                        {'arg-type': self._use_type(arg_type),
-                        'ret-type': self._use_type(ret_type)})
+                        'ret-type': self._use_type(ret_type),
+                        'allow-oob': allow_oob})
 
     def visit_event(self, name, info, arg_type, boxed):
         arg_type = arg_type or self._schema.the_empty_object_type
diff --git a/scripts/qapi.py b/scripts/qapi.py
index 62dc52e..f411b8f 100644
--- a/scripts/qapi.py
+++ b/scripts/qapi.py
@@ -920,7 +920,8 @@ def check_exprs(exprs):
         elif 'command' in expr:
             meta = 'command'
             check_keys(expr_elem, 'command', [],
-                       ['data', 'returns', 'gen', 'success-response', 'boxed'])
+                       ['data', 'returns', 'gen', 'success-response',
+                        'boxed', 'allow-oob'])
         elif 'event' in expr:
             meta = 'event'
             check_keys(expr_elem, 'event', [], ['data', 'boxed'])
@@ -1031,7 +1032,7 @@ class QAPISchemaVisitor(object):
         pass
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         pass
 
     def visit_event(self, name, info, arg_type, boxed):
@@ -1398,7 +1399,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
 class QAPISchemaCommand(QAPISchemaEntity):
     def __init__(self, name, info, doc, arg_type, ret_type,
-                 gen, success_response, boxed):
+                 gen, success_response, boxed, allow_oob):
         QAPISchemaEntity.__init__(self, name, info, doc)
         assert not arg_type or isinstance(arg_type, str)
         assert not ret_type or isinstance(ret_type, str)
@@ -1409,6 +1410,7 @@ class QAPISchemaCommand(QAPISchemaEntity):
         self.gen = gen
         self.success_response = success_response
         self.boxed = boxed
+        self.allow_oob = allow_oob
 
     def check(self, schema):
         if self._arg_type_name:
@@ -1432,7 +1434,8 @@ class QAPISchemaCommand(QAPISchemaEntity):
     def visit(self, visitor):
         visitor.visit_command(self.name, self.info,
                               self.arg_type, self.ret_type,
-                              self.gen, self.success_response, self.boxed)
+                              self.gen, self.success_response,
+                              self.boxed, self.allow_oob)
 
 
 class QAPISchemaEvent(QAPISchemaEntity):
@@ -1640,6 +1643,7 @@ class QAPISchema(object):
         gen = expr.get('gen', True)
         success_response = expr.get('success-response', True)
         boxed = expr.get('boxed', False)
+        allow_oob = expr.get('allow-oob', False)
         if isinstance(data, OrderedDict):
             data = self._make_implicit_object_type(
                 name, info, doc, 'arg', self._make_members(data, info))
@@ -1647,7 +1651,8 @@ class QAPISchema(object):
             assert len(rets) == 1
             rets = self._make_array_type(rets[0], info)
         self._def_entity(QAPISchemaCommand(name, info, doc, data, rets,
-                                           gen, success_response, boxed))
+                                           gen, success_response,
+                                           boxed, allow_oob))
 
     def _def_event(self, expr, info, doc):
         name = expr['event']
diff --git a/scripts/qapi2texi.py b/scripts/qapi2texi.py
index a317526..0ac0df5 100755
--- a/scripts/qapi2texi.py
+++ b/scripts/qapi2texi.py
@@ -236,7 +236,7 @@ class QAPISchemaGenDocVisitor(qapi.QAPISchemaVisitor):
                              body=texi_entity(doc, 'Members'))
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         doc = self.cur_doc
         if self.out:
             self.out += '\n'
diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index c7724d3..6749e9e 100644
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -36,7 +36,7 @@ class QAPISchemaTestVisitor(QAPISchemaVisitor):
         self._print_variants(variants)
 
     def visit_command(self, name, info, arg_type, ret_type,
-                      gen, success_response, boxed):
+                      gen, success_response, boxed, allow_oob):
         print 'command %s %s -> %s' % \
             (name, arg_type and arg_type.name, ret_type and ret_type.name)
         print '   gen=%s success_response=%s boxed=%s' % \
-- 
2.7.4

[Qemu-devel] [RFC 14/15] qmp: support out-of-band (oob) execution

[Peter Xu]

Having "allow-oob" to true for a command does not mean that this command
will always be run in out-of-band mode.  The out-of-band quick path will
only be executed if we specify the extra "run-oob" flag when sending the
QMP request:

  { "execute": "command-that-allows-oob",
    "arguments": { ... },
    "control": { "run-oob": true } }

The "control" key is introduced to store this extra flag.  "control"
field is used to store arguments that are shared by all the commands,
rather than command specific arguments.  Let "run-oob" be the first.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 docs/devel/qapi-code-gen.txt | 10 ++++++++++
 include/qapi/qmp/dispatch.h  |  1 +
 monitor.c                    | 11 +++++++++++
 qapi/qmp-dispatch.c          | 34 ++++++++++++++++++++++++++++++++++
 trace-events                 |  2 ++
 5 files changed, 58 insertions(+)

diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
index 61fa167..47d16bb 100644
--- a/docs/devel/qapi-code-gen.txt
+++ b/docs/devel/qapi-code-gen.txt
@@ -665,6 +665,16 @@ allowed to run out-of-band can also be introspected using
 query-qmp-schema command.  Please see the section "Client JSON
 Protocol introspection" for more information.
 
+To execute a command in out-of-band way, we need to specify the
+"control" field in the request, with "run-oob" set to true. Example:
+
+ => { "execute": "command-support-oob",
+      "arguments": { ... },
+      "control": { "run-oob": true } }
+ <= { "return": { } }
+
+Without it, even the commands that supports out-of-band execution will
+still be run in-band.
 
 === Events ===
 
diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
index b767988..ee2b8ce 100644
--- a/include/qapi/qmp/dispatch.h
+++ b/include/qapi/qmp/dispatch.h
@@ -49,6 +49,7 @@ bool qmp_command_is_enabled(const QmpCommand *cmd);
 const char *qmp_command_name(const QmpCommand *cmd);
 bool qmp_has_success_response(const QmpCommand *cmd);
 QObject *qmp_build_error_object(Error *err);
+bool qmp_is_oob(const QObject *request);
 
 typedef void (*qmp_cmd_callback_fn)(QmpCommand *cmd, void *opaque);
 
diff --git a/monitor.c b/monitor.c
index 599ea36..cb96204 100644
--- a/monitor.c
+++ b/monitor.c
@@ -3928,6 +3928,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
         if (!req_obj) {
             break;
         }
+        trace_monitor_qmp_cmd_in_band(qobject_get_str(req_obj->id));
         monitor_qmp_dispatch_one(req_obj);
     }
 }
@@ -3963,6 +3964,16 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
     req_obj->id = id;
     req_obj->req = req;
 
+    if (qmp_is_oob(req)) {
+        /*
+         * Trigger fast-path to handle the out-of-band request, by
+         * executing the command directly in parser.
+         */
+        trace_monitor_qmp_cmd_out_of_band(qobject_get_str(req_obj->id));
+        monitor_qmp_dispatch_one(req_obj);
+        return;
+    }
+
     /*
      * Put the request to the end of queue so that requests will be
      * handled in time order.  Ownership for req_obj, req, id,
diff --git a/qapi/qmp-dispatch.c b/qapi/qmp-dispatch.c
index b41fa17..9a05dfa 100644
--- a/qapi/qmp-dispatch.c
+++ b/qapi/qmp-dispatch.c
@@ -52,6 +52,12 @@ static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
                            "QMP input member 'arguments' must be an object");
                 return NULL;
             }
+        } else if (!strcmp(arg_name, "control")) {
+            if (qobject_type(arg_obj) != QTYPE_QDICT) {
+                error_setg(errp,
+                           "QMP input member 'control' must be an object");
+                return NULL;
+            }
         } else {
             error_setg(errp, "QMP input member '%s' is unexpected",
                        arg_name);
@@ -122,6 +128,34 @@ QObject *qmp_build_error_object(Error *err)
                               error_get_pretty(err));
 }
 
+/*
+ * Detect whether a request should be run out-of-band, by quickly
+ * peeking at whether we have: { "control": { "run-oob": True } }. By
+ * default commands are run in-band.
+ */
+bool qmp_is_oob(const QObject *request)
+{
+    QDict *dict;
+    QBool *bool_obj;
+
+    dict = qobject_to_qdict(request);
+    if (!dict) {
+        return false;
+    }
+
+    dict = qdict_get_qdict(dict, "control");
+    if (!dict) {
+        return false;
+    }
+
+    bool_obj = qobject_to_qbool(qdict_get(dict, "run-oob"));
+    if (!bool_obj) {
+        return false;
+    }
+
+    return qbool_get_bool(bool_obj);
+}
+
 QObject *qmp_dispatch(QmpCommandList *cmds, QObject *request)
 {
     Error *err = NULL;
diff --git a/trace-events b/trace-events
index 1f50f56..f7900a6 100644
--- a/trace-events
+++ b/trace-events
@@ -47,6 +47,8 @@ monitor_protocol_event_emit(uint32_t event, void *data) "event=%d data=%p"
 monitor_protocol_event_queue(uint32_t event, void *qdict, uint64_t rate) "event=%d data=%p rate=%" PRId64
 handle_hmp_command(void *mon, const char *cmdline) "mon %p cmdline: %s"
 handle_qmp_command(void *mon, const char *req) "mon %p req: %s"
+monitor_qmp_cmd_in_band(const char *id) "%s"
+monitor_qmp_cmd_out_of_band(const char *id) "%s"
 
 # dma-helpers.c
 dma_blk_io(void *dbs, void *bs, int64_t offset, bool to_dev) "dbs=%p bs=%p offset=%" PRId64 " to_dev=%d"
-- 
2.7.4

[Qemu-devel] [RFC 15/15] qmp: let migrate-incoming allow out-of-band

[Peter Xu]

So it can get rid of being run on main thread.

Signed-off-by: Peter Xu <peterx@redhat.com>
---
 qapi/migration.json | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/qapi/migration.json b/qapi/migration.json
index ee2b3b8..dedc4f8 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -986,7 +986,8 @@
 # <- { "return": {} }
 #
 ##
-{ 'command': 'migrate-incoming', 'data': {'uri': 'str' } }
+{ 'command': 'migrate-incoming', 'data': {'uri': 'str' },
+  'allow-oob': true }
 
 ##
 # @xen-save-devices-state:
-- 
2.7.4

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Marc-André Lureau]

Hi

On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
> This series was born from this one:
>
>   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
>
> The design comes from Markus, and also the whole-bunch-of discussions
> in previous thread.  My heartful thanks to Markus, Daniel, Dave,
> Stefan, etc. on discussing the topic (...again!), providing shiny
> ideas and suggestions.  Finally we got such a solution that seems to
> satisfy everyone.
>
> I re-started the versioning since this series is totally different
> from previous one.  Now it's version 1.
>
> In case new reviewers come along the way without reading previous
> discussions, I will try to do a summary on what this is all about.
>
> What is OOB execution?
> ======================
>
> It's the shortcut of Out-Of-Band execution, its name is given by
> Markus.  It's a way to quickly execute a QMP request.  Say, originally
> QMP is going throw these steps:
>
>       JSON Parser --> QMP Dispatcher --> Respond
>           /|\    (2)                (3)     |
>        (1) |                               \|/ (4)
>            +---------  main thread  --------+
>
> The requests are executed by the so-called QMP-dispatcher after the
> JSON is parsed.  If OOB is on, we run the command directly in the
> parser and quickly returns.

All commands should have the "id" field mandatory in this case, else
the client will not distinguish the replies coming from the last/oob
and the previous commands.

This should probably be enforced upfront by client capability checks,
more below.

> Yeah I know in current code the parser calls dispatcher directly
> (please see handle_qmp_command()).  However it's not true again after
> this series (parser will has its own IO thread, and dispatcher will
> still be run in main thread).  So this OOB does brings something
> different.
>
> There are more details on why OOB and the difference/relationship
> between OOB, async QMP, block/general jobs, etc.. but IMHO that's
> slightly out of topic (and believe me, it's not easy for me to
> summarize that).  For more information, please refers to [1].
>
> Summary ends here.
>
> Some Implementation Details
> ===========================
>
> Again, I mentioned that the old QMP workflow is this:
>
>       JSON Parser --> QMP Dispatcher --> Respond
>           /|\    (2)                (3)     |
>        (1) |                               \|/ (4)
>            +---------  main thread  --------+
>
> What this series does is, firstly:
>
>       JSON Parser     QMP Dispatcher --> Respond
>           /|\ |           /|\       (4)     |
>            |  | (2)        | (3)            |  (5)
>        (1) |  +----->      |               \|/
>            +---------  main thread  <-------+
>
> And further:
>
>                queue/kick
>      JSON Parser ======> QMP Dispatcher --> Respond
>          /|\ |     (3)       /|\        (4)    |
>       (1) |  | (2)            |                |  (5)
>           | \|/               |               \|/
>         IO thread         main thread  <-------+

Is the queue per monitor or per client? And is the dispatching going
to be processed even if the client is disconnected, and are new
clients going to receive the replies from previous clients commands? I
believe there should be a per-client context, so there won't be "id"
request conflicts.

>
> Then it introduced the "allow-oob" parameter in QAPI schema to define
> commands, and "run-oob" flag to let oob-allowed command to run in the
> parser.

>From a protocol point of view, I find that "run-oob" distinction per
command a bit pointless. It helps with legacy client that wouldn't
expect out-of-order replies if qemu were to run oob commands oob by
default though. Clients shouldn't care about how/where a command is
being queued or not. If they send a command, they want it processed as
quickly as possible. However, it can be interesting to know if the
implementation of the command will be able to deliver oob, so that
data in the introspection could be useful.

I would rather propose a client/server capability in qmp_capabilities,
call it "oob":

This capability indicates oob commands support.

An oob command is a regular client message request with the "id"
member mandatory, but the reply may be delivered
out of order by the server if the client supports
it too.

If both the server and the client have the "oob" capability, the
server can handle new client requests while previous requests are being
processed.

If the client doesn't have the "oob" capability, it may still call
an oob command, and make multiple outstanding calls. In this case,
the commands are processed in order, so the replies will also be in
order. The "id" member isn't mandatory in this case.

The client should match the replies with the "id" member associated
with the requests.

When a client is disconnected, the pending commands are not
necessarily cancelled. But the future clients will not get replies from
commands they didn't make (they might, however, receive side-effects
events).

Note that without "oob" support, a client may still receive
 messages (or events) from the server between the time a
request is handled by the server and the reply is received. It must
thus be prepared to handle dispatching both events and reply after
sending a request.


(see also https://lists.gnu.org/archive/html/qemu-devel/2017-01/msg03641.html)


> The last patch enables this for "migrate-incoming" command.
>
> Please review.  Thanks.
>
> [1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
>
> Peter Xu (15):
>   char-io: fix possible race on IOWatchPoll
>   qobject: allow NULL for qstring_get_str()
>   qobject: introduce qobject_to_str()
>   monitor: move skip_flush into monitor_data_init
>   qjson: add "opaque" field to JSONMessageParser
>   monitor: move the cur_mon hack deeper for QMP
>   monitor: unify global init
>   monitor: create IO thread
>   monitor: allow to use IO thread for parsing
>   monitor: introduce monitor_qmp_respond()
>   monitor: separate QMP parser and dispatcher

There should be a limit in the number of requests the thread can
queue. Before the patch, the limit was enforced by system socket
buffering I think. Now, should oob commands still be processed even if
the queue is full? If so, the thread can't be suspended.

>   monitor: enable IO thread for (qmp & !mux) typed
>   qapi: introduce new cmd option "allow-oob"
>   qmp: support out-of-band (oob) execution
>   qmp: let migrate-incoming allow out-of-band
>
>  chardev/char-io.c                |  15 ++-
>  docs/devel/qapi-code-gen.txt     |  51 ++++++-
>  include/monitor/monitor.h        |   2 +-
>  include/qapi/qmp/dispatch.h      |   2 +
>  include/qapi/qmp/json-streamer.h |   8 +-
>  include/qapi/qmp/qstring.h       |   1 +
>  monitor.c                        | 283 +++++++++++++++++++++++++++++++--------
>  qapi/introspect.json             |   6 +-
>  qapi/migration.json              |   3 +-
>  qapi/qmp-dispatch.c              |  34 +++++
>  qga/main.c                       |   5 +-
>  qobject/json-streamer.c          |   7 +-
>  qobject/qjson.c                  |   5 +-
>  qobject/qstring.c                |  13 +-
>  scripts/qapi-commands.py         |  19 ++-
>  scripts/qapi-introspect.py       |  10 +-
>  scripts/qapi.py                  |  15 ++-
>  scripts/qapi2texi.py             |   2 +-
>  tests/libqtest.c                 |   5 +-
>  tests/qapi-schema/test-qapi.py   |   2 +-
>  trace-events                     |   2 +
>  vl.c                             |   3 +-
>  22 files changed, 398 insertions(+), 95 deletions(-)
>
> --
> 2.7.4
>



-- 
Marc-André Lureau

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Stefan Hajnoczi]

On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> There should be a limit in the number of requests the thread can
> queue. Before the patch, the limit was enforced by system socket
> buffering I think. Now, should oob commands still be processed even if
> the queue is full? If so, the thread can't be suspended.

I agree.

Memory usage must be bounded.  The number of requests is less important
than the amount of memory consumed by them.

Existing QMP clients that send multiple QMP commands without waiting for
replies need to rethink their strategy because OOB commands cannot be
processed if queued non-OOB commands consume too much memory.

Stefan

Re: [Qemu-devel] [RFC 14/15] qmp: support out-of-band (oob) execution

[Stefan Hajnoczi]

On Thu, Sep 14, 2017 at 03:50:35PM +0800, Peter Xu wrote:
> diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
> index 61fa167..47d16bb 100644
> --- a/docs/devel/qapi-code-gen.txt
> +++ b/docs/devel/qapi-code-gen.txt
> @@ -665,6 +665,16 @@ allowed to run out-of-band can also be introspected using
>  query-qmp-schema command.  Please see the section "Client JSON
>  Protocol introspection" for more information.
>  
> +To execute a command in out-of-band way, we need to specify the
> +"control" field in the request, with "run-oob" set to true. Example:
> +
> + => { "execute": "command-support-oob",
> +      "arguments": { ... },
> +      "control": { "run-oob": true } }
> + <= { "return": { } }
> +
> +Without it, even the commands that supports out-of-band execution will
> +still be run in-band.

Is there a more relevant place to document QMP run-oob behavior than the
"How to use the QAPI code generator document"?

> @@ -3963,6 +3964,16 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
>      req_obj->id = id;
>      req_obj->req = req;
>  
> +    if (qmp_is_oob(req)) {
> +        /*
> +         * Trigger fast-path to handle the out-of-band request, by
> +         * executing the command directly in parser.
> +         */
> +        trace_monitor_qmp_cmd_out_of_band(qobject_get_str(req_obj->id));
> +        monitor_qmp_dispatch_one(req_obj);
> +        return;
> +    }

A "fast-path" is a performance optimization.  OOB is not a performance
optimization, it changes the semantics of command execution.  Please
mention the semantics of OOB command execution instead.

Stefa

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> Hi
> 
> On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
> > This series was born from this one:
> >
> >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> >
> > The design comes from Markus, and also the whole-bunch-of discussions
> > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
> > Stefan, etc. on discussing the topic (...again!), providing shiny
> > ideas and suggestions.  Finally we got such a solution that seems to
> > satisfy everyone.
> >
> > I re-started the versioning since this series is totally different
> > from previous one.  Now it's version 1.
> >
> > In case new reviewers come along the way without reading previous
> > discussions, I will try to do a summary on what this is all about.
> >
> > What is OOB execution?
> > ======================
> >
> > It's the shortcut of Out-Of-Band execution, its name is given by
> > Markus.  It's a way to quickly execute a QMP request.  Say, originally
> > QMP is going throw these steps:
> >
> >       JSON Parser --> QMP Dispatcher --> Respond
> >           /|\    (2)                (3)     |
> >        (1) |                               \|/ (4)
> >            +---------  main thread  --------+
> >
> > The requests are executed by the so-called QMP-dispatcher after the
> > JSON is parsed.  If OOB is on, we run the command directly in the
> > parser and quickly returns.
> 
> All commands should have the "id" field mandatory in this case, else
> the client will not distinguish the replies coming from the last/oob
> and the previous commands.
> 
> This should probably be enforced upfront by client capability checks,
> more below.
> 
> > Yeah I know in current code the parser calls dispatcher directly
> > (please see handle_qmp_command()).  However it's not true again after
> > this series (parser will has its own IO thread, and dispatcher will
> > still be run in main thread).  So this OOB does brings something
> > different.
> >
> > There are more details on why OOB and the difference/relationship
> > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
> > slightly out of topic (and believe me, it's not easy for me to
> > summarize that).  For more information, please refers to [1].
> >
> > Summary ends here.
> >
> > Some Implementation Details
> > ===========================
> >
> > Again, I mentioned that the old QMP workflow is this:
> >
> >       JSON Parser --> QMP Dispatcher --> Respond
> >           /|\    (2)                (3)     |
> >        (1) |                               \|/ (4)
> >            +---------  main thread  --------+
> >
> > What this series does is, firstly:
> >
> >       JSON Parser     QMP Dispatcher --> Respond
> >           /|\ |           /|\       (4)     |
> >            |  | (2)        | (3)            |  (5)
> >        (1) |  +----->      |               \|/
> >            +---------  main thread  <-------+
> >
> > And further:
> >
> >                queue/kick
> >      JSON Parser ======> QMP Dispatcher --> Respond
> >          /|\ |     (3)       /|\        (4)    |
> >       (1) |  | (2)            |                |  (5)
> >           | \|/               |               \|/
> >         IO thread         main thread  <-------+
> 
> Is the queue per monitor or per client? And is the dispatching going
> to be processed even if the client is disconnected, and are new
> clients going to receive the replies from previous clients commands? I
> believe there should be a per-client context, so there won't be "id"
> request conflicts.
> 
> >
> > Then it introduced the "allow-oob" parameter in QAPI schema to define
> > commands, and "run-oob" flag to let oob-allowed command to run in the
> > parser.
> 
> From a protocol point of view, I find that "run-oob" distinction per
> command a bit pointless. It helps with legacy client that wouldn't
> expect out-of-order replies if qemu were to run oob commands oob by
> default though. Clients shouldn't care about how/where a command is
> being queued or not. If they send a command, they want it processed as
> quickly as possible. However, it can be interesting to know if the
> implementation of the command will be able to deliver oob, so that
> data in the introspection could be useful.
> 
> I would rather propose a client/server capability in qmp_capabilities,
> call it "oob":
> 
> This capability indicates oob commands support.

The problem is indicating which commands support oob as opposed to
indicating whether oob is present at all.  Future versions will
probably make more commands oob-able and a client will want to know
whether it can rely on a particular command being non-blocking.

> An oob command is a regular client message request with the "id"
> member mandatory, but the reply may be delivered
> out of order by the server if the client supports
> it too.
> 
> If both the server and the client have the "oob" capability, the
> server can handle new client requests while previous requests are being
> processed.
> 
> If the client doesn't have the "oob" capability, it may still call
> an oob command, and make multiple outstanding calls. In this case,
> the commands are processed in order, so the replies will also be in
> order. The "id" member isn't mandatory in this case.
> 
> The client should match the replies with the "id" member associated
> with the requests.
> 
> When a client is disconnected, the pending commands are not
> necessarily cancelled. But the future clients will not get replies from
> commands they didn't make (they might, however, receive side-effects
> events).

What's the behaviour on the current monitor?


> Note that without "oob" support, a client may still receive
>  messages (or events) from the server between the time a
> request is handled by the server and the reply is received. It must
> thus be prepared to handle dispatching both events and reply after
> sending a request.
> 
> 
> (see also https://lists.gnu.org/archive/html/qemu-devel/2017-01/msg03641.html)
> 
> 
> > The last patch enables this for "migrate-incoming" command.
> >
> > Please review.  Thanks.
> >
> > [1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> >
> > Peter Xu (15):
> >   char-io: fix possible race on IOWatchPoll
> >   qobject: allow NULL for qstring_get_str()
> >   qobject: introduce qobject_to_str()
> >   monitor: move skip_flush into monitor_data_init
> >   qjson: add "opaque" field to JSONMessageParser
> >   monitor: move the cur_mon hack deeper for QMP
> >   monitor: unify global init
> >   monitor: create IO thread
> >   monitor: allow to use IO thread for parsing
> >   monitor: introduce monitor_qmp_respond()
> >   monitor: separate QMP parser and dispatcher
> 
> There should be a limit in the number of requests the thread can
> queue. Before the patch, the limit was enforced by system socket
> buffering I think. Now, should oob commands still be processed even if
> the queue is full? If so, the thread can't be suspended.

I think the previous discussion was expecting a pair of queues
per client and perhaps a pair of central queues; each pair being
for normal command and oob commands.
(I'm not expecting these queues to be deep; IMHO '1' is the
right size for this type of queue in both cases).

Dave

> >   monitor: enable IO thread for (qmp & !mux) typed
> >   qapi: introduce new cmd option "allow-oob"
> >   qmp: support out-of-band (oob) execution
> >   qmp: let migrate-incoming allow out-of-band
> >
> >  chardev/char-io.c                |  15 ++-
> >  docs/devel/qapi-code-gen.txt     |  51 ++++++-
> >  include/monitor/monitor.h        |   2 +-
> >  include/qapi/qmp/dispatch.h      |   2 +
> >  include/qapi/qmp/json-streamer.h |   8 +-
> >  include/qapi/qmp/qstring.h       |   1 +
> >  monitor.c                        | 283 +++++++++++++++++++++++++++++++--------
> >  qapi/introspect.json             |   6 +-
> >  qapi/migration.json              |   3 +-
> >  qapi/qmp-dispatch.c              |  34 +++++
> >  qga/main.c                       |   5 +-
> >  qobject/json-streamer.c          |   7 +-
> >  qobject/qjson.c                  |   5 +-
> >  qobject/qstring.c                |  13 +-
> >  scripts/qapi-commands.py         |  19 ++-
> >  scripts/qapi-introspect.py       |  10 +-
> >  scripts/qapi.py                  |  15 ++-
> >  scripts/qapi2texi.py             |   2 +-
> >  tests/libqtest.c                 |   5 +-
> >  tests/qapi-schema/test-qapi.py   |   2 +-
> >  trace-events                     |   2 +
> >  vl.c                             |   3 +-
> >  22 files changed, 398 insertions(+), 95 deletions(-)
> >
> > --
> > 2.7.4
> >
> 
> 
> 
> -- 
> Marc-André Lureau
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Peter Xu (peterx@redhat.com) wrote:
> This series was born from this one:
> 
>   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html

Are patches 1..6 separable and mergable without the rest ?

Dave

> The design comes from Markus, and also the whole-bunch-of discussions
> in previous thread.  My heartful thanks to Markus, Daniel, Dave,
> Stefan, etc. on discussing the topic (...again!), providing shiny
> ideas and suggestions.  Finally we got such a solution that seems to
> satisfy everyone.
> 
> I re-started the versioning since this series is totally different
> from previous one.  Now it's version 1.
> 
> In case new reviewers come along the way without reading previous
> discussions, I will try to do a summary on what this is all about.
> 
> What is OOB execution?
> ======================
> 
> It's the shortcut of Out-Of-Band execution, its name is given by
> Markus.  It's a way to quickly execute a QMP request.  Say, originally
> QMP is going throw these steps:
> 
>       JSON Parser --> QMP Dispatcher --> Respond
>           /|\    (2)                (3)     |
>        (1) |                               \|/ (4)
>            +---------  main thread  --------+
> 
> The requests are executed by the so-called QMP-dispatcher after the
> JSON is parsed.  If OOB is on, we run the command directly in the
> parser and quickly returns.
> 
> Yeah I know in current code the parser calls dispatcher directly
> (please see handle_qmp_command()).  However it's not true again after
> this series (parser will has its own IO thread, and dispatcher will
> still be run in main thread).  So this OOB does brings something
> different.
> 
> There are more details on why OOB and the difference/relationship
> between OOB, async QMP, block/general jobs, etc.. but IMHO that's
> slightly out of topic (and believe me, it's not easy for me to
> summarize that).  For more information, please refers to [1].
> 
> Summary ends here.
> 
> Some Implementation Details
> ===========================
> 
> Again, I mentioned that the old QMP workflow is this:
> 
>       JSON Parser --> QMP Dispatcher --> Respond
>           /|\    (2)                (3)     |
>        (1) |                               \|/ (4)
>            +---------  main thread  --------+
> 
> What this series does is, firstly:
> 
>       JSON Parser     QMP Dispatcher --> Respond
>           /|\ |           /|\       (4)     |
>            |  | (2)        | (3)            |  (5)
>        (1) |  +----->      |               \|/
>            +---------  main thread  <-------+
> 
> And further:
> 
>                queue/kick
>      JSON Parser ======> QMP Dispatcher --> Respond
>          /|\ |     (3)       /|\        (4)    |
>       (1) |  | (2)            |                |  (5)
>           | \|/               |               \|/
>         IO thread         main thread  <-------+
> 
> Then it introduced the "allow-oob" parameter in QAPI schema to define
> commands, and "run-oob" flag to let oob-allowed command to run in the
> parser.
> 
> The last patch enables this for "migrate-incoming" command.
> 
> Please review.  Thanks.
> 
> [1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> 
> Peter Xu (15):
>   char-io: fix possible race on IOWatchPoll
>   qobject: allow NULL for qstring_get_str()
>   qobject: introduce qobject_to_str()
>   monitor: move skip_flush into monitor_data_init
>   qjson: add "opaque" field to JSONMessageParser
>   monitor: move the cur_mon hack deeper for QMP
>   monitor: unify global init
>   monitor: create IO thread
>   monitor: allow to use IO thread for parsing
>   monitor: introduce monitor_qmp_respond()
>   monitor: separate QMP parser and dispatcher
>   monitor: enable IO thread for (qmp & !mux) typed
>   qapi: introduce new cmd option "allow-oob"
>   qmp: support out-of-band (oob) execution
>   qmp: let migrate-incoming allow out-of-band
> 
>  chardev/char-io.c                |  15 ++-
>  docs/devel/qapi-code-gen.txt     |  51 ++++++-
>  include/monitor/monitor.h        |   2 +-
>  include/qapi/qmp/dispatch.h      |   2 +
>  include/qapi/qmp/json-streamer.h |   8 +-
>  include/qapi/qmp/qstring.h       |   1 +
>  monitor.c                        | 283 +++++++++++++++++++++++++++++++--------
>  qapi/introspect.json             |   6 +-
>  qapi/migration.json              |   3 +-
>  qapi/qmp-dispatch.c              |  34 +++++
>  qga/main.c                       |   5 +-
>  qobject/json-streamer.c          |   7 +-
>  qobject/qjson.c                  |   5 +-
>  qobject/qstring.c                |  13 +-
>  scripts/qapi-commands.py         |  19 ++-
>  scripts/qapi-introspect.py       |  10 +-
>  scripts/qapi.py                  |  15 ++-
>  scripts/qapi2texi.py             |   2 +-
>  tests/libqtest.c                 |   5 +-
>  tests/qapi-schema/test-qapi.py   |   2 +-
>  trace-events                     |   2 +
>  vl.c                             |   3 +-
>  22 files changed, 398 insertions(+), 95 deletions(-)
> 
> -- 
> 2.7.4
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 14/15] qmp: support out-of-band (oob) execution

[Peter Xu]

On Thu, Sep 14, 2017 at 04:33:34PM +0100, Stefan Hajnoczi wrote:
> On Thu, Sep 14, 2017 at 03:50:35PM +0800, Peter Xu wrote:
> > diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
> > index 61fa167..47d16bb 100644
> > --- a/docs/devel/qapi-code-gen.txt
> > +++ b/docs/devel/qapi-code-gen.txt
> > @@ -665,6 +665,16 @@ allowed to run out-of-band can also be introspected using
> >  query-qmp-schema command.  Please see the section "Client JSON
> >  Protocol introspection" for more information.
> >  
> > +To execute a command in out-of-band way, we need to specify the
> > +"control" field in the request, with "run-oob" set to true. Example:
> > +
> > + => { "execute": "command-support-oob",
> > +      "arguments": { ... },
> > +      "control": { "run-oob": true } }
> > + <= { "return": { } }
> > +
> > +Without it, even the commands that supports out-of-band execution will
> > +still be run in-band.
> 
> Is there a more relevant place to document QMP run-oob behavior than the
> "How to use the QAPI code generator document"?

I agree, but I don't really know it. :(

Markus, could you provide a hint?

> 
> > @@ -3963,6 +3964,16 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> >      req_obj->id = id;
> >      req_obj->req = req;
> >  
> > +    if (qmp_is_oob(req)) {
> > +        /*
> > +         * Trigger fast-path to handle the out-of-band request, by
> > +         * executing the command directly in parser.
> > +         */
> > +        trace_monitor_qmp_cmd_out_of_band(qobject_get_str(req_obj->id));
> > +        monitor_qmp_dispatch_one(req_obj);
> > +        return;
> > +    }
> 
> A "fast-path" is a performance optimization.  OOB is not a performance
> optimization, it changes the semantics of command execution.  Please
> mention the semantics of OOB command execution instead.

I'll remove the "fast-path" wording and try to think out something
better than this comment.  After I know a good place to document, I
can put it there as well.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > There should be a limit in the number of requests the thread can
> > queue. Before the patch, the limit was enforced by system socket
> > buffering I think. Now, should oob commands still be processed even if
> > the queue is full? If so, the thread can't be suspended.
> 
> I agree.
> 
> Memory usage must be bounded.  The number of requests is less important
> than the amount of memory consumed by them.
> 
> Existing QMP clients that send multiple QMP commands without waiting for
> replies need to rethink their strategy because OOB commands cannot be
> processed if queued non-OOB commands consume too much memory.

Thanks for pointing out this.  Yes the memory usage problem is valid,
as Markus pointed out as well in previous discussions (in "Flow
Control" section of that long reply).  Hopefully this series basically
can work from design prospective, then I'll add this flow control in
next version.

Regarding to what we should do if the limit is reached: Markus
provided a few options, but the one I prefer most is that we don't
respond, but send an event showing that a command is dropped.
However, I would like it not queued, but a direct reply (after all,
it's an event, and we should not need to care much on ordering of it).
Then we can get rid of the babysitting of those "to be failed"
requests asap, meanwhile we don't lose anything IMHO.

I think I also missed at least a unit test for this new interface.
Again, I'll add it after the whole idea is proved solid.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Thu, Sep 14, 2017 at 07:56:04PM +0100, Dr. David Alan Gilbert wrote:
> * Peter Xu (peterx@redhat.com) wrote:
> > This series was born from this one:
> > 
> >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> 
> Are patches 1..6 separable and mergable without the rest ?

Yes I think so.

(I was always trying to put pre-requisite patches like these ones at
 the front of any of my series rather than separating them into more
 series, since I thought it is convenient for me to manage them (or
 add new ones when respin), and also easier for reviewers (so people
 don't need to try to find the dependencies).  And since I put them at
 the head, we can easily merge them without rebasing issue when they
 are good while the rest may still need further work.  Hopefully this
 is the right thing to do.)

-- 
Peter Xu

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> > Hi
> > 
> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
> > > This series was born from this one:
> > >
> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> > >
> > > The design comes from Markus, and also the whole-bunch-of discussions
> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
> > > Stefan, etc. on discussing the topic (...again!), providing shiny
> > > ideas and suggestions.  Finally we got such a solution that seems to
> > > satisfy everyone.
> > >
> > > I re-started the versioning since this series is totally different
> > > from previous one.  Now it's version 1.
> > >
> > > In case new reviewers come along the way without reading previous
> > > discussions, I will try to do a summary on what this is all about.
> > >
> > > What is OOB execution?
> > > ======================
> > >
> > > It's the shortcut of Out-Of-Band execution, its name is given by
> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
> > > QMP is going throw these steps:
> > >
> > >       JSON Parser --> QMP Dispatcher --> Respond
> > >           /|\    (2)                (3)     |
> > >        (1) |                               \|/ (4)
> > >            +---------  main thread  --------+
> > >
> > > The requests are executed by the so-called QMP-dispatcher after the
> > > JSON is parsed.  If OOB is on, we run the command directly in the
> > > parser and quickly returns.
> > 
> > All commands should have the "id" field mandatory in this case, else
> > the client will not distinguish the replies coming from the last/oob
> > and the previous commands.
> > 
> > This should probably be enforced upfront by client capability checks,
> > more below.

Hmm yes since the oob commands are actually running in async way,
request ID should be needed here.  However I'm not sure whether
enabling the whole "request ID" thing is too big for this "try to be
small" oob change... And IMHO it suites better to be part of the whole
async work (no matter which implementation we'll use).

How about this: we make "id" mandatory for "run-oob" requests only.
For oob commands, they will always have ID then no ordering issue, and
we can do it async; for the rest of non-oob commands, we still allow
them to go without ID, and since they are not oob, they'll always be
done in order as well.  Would this work?

> > 
> > > Yeah I know in current code the parser calls dispatcher directly
> > > (please see handle_qmp_command()).  However it's not true again after
> > > this series (parser will has its own IO thread, and dispatcher will
> > > still be run in main thread).  So this OOB does brings something
> > > different.
> > >
> > > There are more details on why OOB and the difference/relationship
> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
> > > slightly out of topic (and believe me, it's not easy for me to
> > > summarize that).  For more information, please refers to [1].
> > >
> > > Summary ends here.
> > >
> > > Some Implementation Details
> > > ===========================
> > >
> > > Again, I mentioned that the old QMP workflow is this:
> > >
> > >       JSON Parser --> QMP Dispatcher --> Respond
> > >           /|\    (2)                (3)     |
> > >        (1) |                               \|/ (4)
> > >            +---------  main thread  --------+
> > >
> > > What this series does is, firstly:
> > >
> > >       JSON Parser     QMP Dispatcher --> Respond
> > >           /|\ |           /|\       (4)     |
> > >            |  | (2)        | (3)            |  (5)
> > >        (1) |  +----->      |               \|/
> > >            +---------  main thread  <-------+
> > >
> > > And further:
> > >
> > >                queue/kick
> > >      JSON Parser ======> QMP Dispatcher --> Respond
> > >          /|\ |     (3)       /|\        (4)    |
> > >       (1) |  | (2)            |                |  (5)
> > >           | \|/               |               \|/
> > >         IO thread         main thread  <-------+
> > 
> > Is the queue per monitor or per client?

The queue is currently global. I think yes maybe at least we can do it
per monitor, but I am not sure whether that is urgent or can be
postponed.  After all now QMPRequest (please refer to patch 11) is
defined as (mon, id, req) tuple, so at least "id" namespace is
per-monitor.

> > And is the dispatching going
> > to be processed even if the client is disconnected, and are new
> > clients going to receive the replies from previous clients
> > commands?

[1]

(will discuss together below)

> > I
> > believe there should be a per-client context, so there won't be "id"
> > request conflicts.

I'd say I am not familiar with this "client" idea, since after all
IMHO one monitor is currently designed to mostly work with a single
client. Say, unix sockets, telnet, all these backends are only single
channeled, and one monitor instance can only work with one client at a
time.  Then do we really need to add this client layer upon it?  IMHO
the user can just provide more monitors if they wants more clients
(and at least these clients should know the existance of the others or
there might be problem, otherwise user2 will fail a migration, finally
noticed that user1 has already triggered one), and the user should
manage them well.

> > 
> > >
> > > Then it introduced the "allow-oob" parameter in QAPI schema to define
> > > commands, and "run-oob" flag to let oob-allowed command to run in the
> > > parser.
> > 
> > From a protocol point of view, I find that "run-oob" distinction per
> > command a bit pointless. It helps with legacy client that wouldn't
> > expect out-of-order replies if qemu were to run oob commands oob by
> > default though.

After all oob somehow breaks existing rules or sync execution.  I
thought the more important goal was at least to keep the legacy
behaviors when adding new things, no?

> > Clients shouldn't care about how/where a command is
> > being queued or not. If they send a command, they want it processed as
> > quickly as possible. However, it can be interesting to know if the
> > implementation of the command will be able to deliver oob, so that
> > data in the introspection could be useful.
> > 
> > I would rather propose a client/server capability in qmp_capabilities,
> > call it "oob":
> > 
> > This capability indicates oob commands support.
> 
> The problem is indicating which commands support oob as opposed to
> indicating whether oob is present at all.  Future versions will
> probably make more commands oob-able and a client will want to know
> whether it can rely on a particular command being non-blocking.

Yes.

And IMHO we don't urgently need that "whether the server globally
supports oob" thing.  Client can just know that from query-qmp-schema
already - there will always be the "allow-oob" new field for command
typed entries.  IMHO that's a solid hint.

But I don't object to return it as well in qmp_capabilities.

> 
> > An oob command is a regular client message request with the "id"
> > member mandatory, but the reply may be delivered
> > out of order by the server if the client supports
> > it too.
> > 
> > If both the server and the client have the "oob" capability, the
> > server can handle new client requests while previous requests are being
> > processed.
> > 
> > If the client doesn't have the "oob" capability, it may still call
> > an oob command, and make multiple outstanding calls. In this case,
> > the commands are processed in order, so the replies will also be in
> > order. The "id" member isn't mandatory in this case.
> > 
> > The client should match the replies with the "id" member associated
> > with the requests.
> > 
> > When a client is disconnected, the pending commands are not
> > necessarily cancelled. But the future clients will not get replies from
> > commands they didn't make (they might, however, receive side-effects
> > events).
> 
> What's the behaviour on the current monitor?

Yeah I want to ask the same question, along with questioning about
above [1].

IMHO this series will not change the behaviors of these, so IMHO the
behaviors will be the same before/after this series. E.g., when client
dropped right after the command is executed, I think we will still
execute the command, though we should encounter something odd in
monitor_json_emitter() somewhere when we want to respond.  And it will
happen the same after this series.

> 
> 
> > Note that without "oob" support, a client may still receive
> >  messages (or events) from the server between the time a
> > request is handled by the server and the reply is received. It must
> > thus be prepared to handle dispatching both events and reply after
> > sending a request.
> > 
> > 
> > (see also https://lists.gnu.org/archive/html/qemu-devel/2017-01/msg03641.html)
> > 
> > 
> > > The last patch enables this for "migrate-incoming" command.
> > >
> > > Please review.  Thanks.
> > >
> > > [1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> > >
> > > Peter Xu (15):
> > >   char-io: fix possible race on IOWatchPoll
> > >   qobject: allow NULL for qstring_get_str()
> > >   qobject: introduce qobject_to_str()
> > >   monitor: move skip_flush into monitor_data_init
> > >   qjson: add "opaque" field to JSONMessageParser
> > >   monitor: move the cur_mon hack deeper for QMP
> > >   monitor: unify global init
> > >   monitor: create IO thread
> > >   monitor: allow to use IO thread for parsing
> > >   monitor: introduce monitor_qmp_respond()
> > >   monitor: separate QMP parser and dispatcher
> > 
> > There should be a limit in the number of requests the thread can
> > queue. Before the patch, the limit was enforced by system socket
> > buffering I think. Now, should oob commands still be processed even if
> > the queue is full? If so, the thread can't be suspended.
> 
> I think the previous discussion was expecting a pair of queues
> per client and perhaps a pair of central queues; each pair being
> for normal command and oob commands.
> (I'm not expecting these queues to be deep; IMHO '1' is the
> right size for this type of queue in both cases).

Yes.  One thing to mention is that, if we see the graph above, I
didn't really introduce two queues (input/output), but only one input
queue.  The response is still handled in dispatcher for now, since I
think that's quite enough at least for OOB, and I didn't see much
benefit now to split that 2nd queue.

So I am thinking whether we can just quickly respond an event when the
queue is full, as I proposed in the other reply.

Regarding to queue size: I am afraid max_size=1 may not suffice?
Otherwise a simple batch of:

{"execute": "query-status"} {"execute": "query-status"}

Will trigger the failure.  But I definitely agree it should not be
something very large.  The total memory will be this:

  json limit * queue length limit * monitor count limit
      (X)            (Y)                    (Z)

Now we have (X) already (in form of a few tunables for JSON token
counts, etc.), we don't have (Z), and we definitely need (Y).

How about we add limits on Y=16 and Z=8?

We can do some math if we want some more exact number though.

> 
> Dave
> 
> > >   monitor: enable IO thread for (qmp & !mux) typed
> > >   qapi: introduce new cmd option "allow-oob"
> > >   qmp: support out-of-band (oob) execution
> > >   qmp: let migrate-incoming allow out-of-band
> > >
> > >  chardev/char-io.c                |  15 ++-
> > >  docs/devel/qapi-code-gen.txt     |  51 ++++++-
> > >  include/monitor/monitor.h        |   2 +-
> > >  include/qapi/qmp/dispatch.h      |   2 +
> > >  include/qapi/qmp/json-streamer.h |   8 +-
> > >  include/qapi/qmp/qstring.h       |   1 +
> > >  monitor.c                        | 283 +++++++++++++++++++++++++++++++--------
> > >  qapi/introspect.json             |   6 +-
> > >  qapi/migration.json              |   3 +-
> > >  qapi/qmp-dispatch.c              |  34 +++++
> > >  qga/main.c                       |   5 +-
> > >  qobject/json-streamer.c          |   7 +-
> > >  qobject/qjson.c                  |   5 +-
> > >  qobject/qstring.c                |  13 +-
> > >  scripts/qapi-commands.py         |  19 ++-
> > >  scripts/qapi-introspect.py       |  10 +-
> > >  scripts/qapi.py                  |  15 ++-
> > >  scripts/qapi2texi.py             |   2 +-
> > >  tests/libqtest.c                 |   5 +-
> > >  tests/qapi-schema/test-qapi.py   |   2 +-
> > >  trace-events                     |   2 +
> > >  vl.c                             |   3 +-
> > >  22 files changed, 398 insertions(+), 95 deletions(-)
> > >
> > > --
> > > 2.7.4
> > >
> > 
> > 
> > 
> > -- 
> > Marc-André Lureau
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

-- 
Peter Xu

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Stefan Hajnoczi]

On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > There should be a limit in the number of requests the thread can
> > > queue. Before the patch, the limit was enforced by system socket
> > > buffering I think. Now, should oob commands still be processed even if
> > > the queue is full? If so, the thread can't be suspended.
> > 
> > I agree.
> > 
> > Memory usage must be bounded.  The number of requests is less important
> > than the amount of memory consumed by them.
> > 
> > Existing QMP clients that send multiple QMP commands without waiting for
> > replies need to rethink their strategy because OOB commands cannot be
> > processed if queued non-OOB commands consume too much memory.
> 
> Thanks for pointing out this.  Yes the memory usage problem is valid,
> as Markus pointed out as well in previous discussions (in "Flow
> Control" section of that long reply).  Hopefully this series basically
> can work from design prospective, then I'll add this flow control in
> next version.
> 
> Regarding to what we should do if the limit is reached: Markus
> provided a few options, but the one I prefer most is that we don't
> respond, but send an event showing that a command is dropped.
> However, I would like it not queued, but a direct reply (after all,
> it's an event, and we should not need to care much on ordering of it).
> Then we can get rid of the babysitting of those "to be failed"
> requests asap, meanwhile we don't lose anything IMHO.
> 
> I think I also missed at least a unit test for this new interface.
> Again, I'll add it after the whole idea is proved solid.  Thanks,

Another solution: the server reports available receive buffer space to
the client.  The server only guarantees immediate OOB processing when
the client stays within the receive buffer size.

Clients wishing to take advantage of OOB must query the receive buffer
size and make sure to leave enough room.

The advantage of this approach is that the semantics are backwards
compatible (existing clients may continue to queue as many commands as
they wish) and it requires no new behavior in the client (no new QMP
event code path).

Stefan

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Marc-André Lureau]

Hi

On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu <peterx@redhat.com> wrote:
> On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
>> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
>> > Hi
>> >
>> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
>> > > This series was born from this one:
>> > >
>> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
>> > >
>> > > The design comes from Markus, and also the whole-bunch-of discussions
>> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
>> > > Stefan, etc. on discussing the topic (...again!), providing shiny
>> > > ideas and suggestions.  Finally we got such a solution that seems to
>> > > satisfy everyone.
>> > >
>> > > I re-started the versioning since this series is totally different
>> > > from previous one.  Now it's version 1.
>> > >
>> > > In case new reviewers come along the way without reading previous
>> > > discussions, I will try to do a summary on what this is all about.
>> > >
>> > > What is OOB execution?
>> > > ======================
>> > >
>> > > It's the shortcut of Out-Of-Band execution, its name is given by
>> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
>> > > QMP is going throw these steps:
>> > >
>> > >       JSON Parser --> QMP Dispatcher --> Respond
>> > >           /|\    (2)                (3)     |
>> > >        (1) |                               \|/ (4)
>> > >            +---------  main thread  --------+
>> > >
>> > > The requests are executed by the so-called QMP-dispatcher after the
>> > > JSON is parsed.  If OOB is on, we run the command directly in the
>> > > parser and quickly returns.
>> >
>> > All commands should have the "id" field mandatory in this case, else
>> > the client will not distinguish the replies coming from the last/oob
>> > and the previous commands.
>> >
>> > This should probably be enforced upfront by client capability checks,
>> > more below.
>
> Hmm yes since the oob commands are actually running in async way,
> request ID should be needed here.  However I'm not sure whether
> enabling the whole "request ID" thing is too big for this "try to be
> small" oob change... And IMHO it suites better to be part of the whole
> async work (no matter which implementation we'll use).
>
> How about this: we make "id" mandatory for "run-oob" requests only.
> For oob commands, they will always have ID then no ordering issue, and
> we can do it async; for the rest of non-oob commands, we still allow
> them to go without ID, and since they are not oob, they'll always be
> done in order as well.  Would this work?

This mixed-mode is imho more complicated to deal with than having the
protocol enforced one way or the other, but that should work.

>
>> >
>> > > Yeah I know in current code the parser calls dispatcher directly
>> > > (please see handle_qmp_command()).  However it's not true again after
>> > > this series (parser will has its own IO thread, and dispatcher will
>> > > still be run in main thread).  So this OOB does brings something
>> > > different.
>> > >
>> > > There are more details on why OOB and the difference/relationship
>> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
>> > > slightly out of topic (and believe me, it's not easy for me to
>> > > summarize that).  For more information, please refers to [1].
>> > >
>> > > Summary ends here.
>> > >
>> > > Some Implementation Details
>> > > ===========================
>> > >
>> > > Again, I mentioned that the old QMP workflow is this:
>> > >
>> > >       JSON Parser --> QMP Dispatcher --> Respond
>> > >           /|\    (2)                (3)     |
>> > >        (1) |                               \|/ (4)
>> > >            +---------  main thread  --------+
>> > >
>> > > What this series does is, firstly:
>> > >
>> > >       JSON Parser     QMP Dispatcher --> Respond
>> > >           /|\ |           /|\       (4)     |
>> > >            |  | (2)        | (3)            |  (5)
>> > >        (1) |  +----->      |               \|/
>> > >            +---------  main thread  <-------+
>> > >
>> > > And further:
>> > >
>> > >                queue/kick
>> > >      JSON Parser ======> QMP Dispatcher --> Respond
>> > >          /|\ |     (3)       /|\        (4)    |
>> > >       (1) |  | (2)            |                |  (5)
>> > >           | \|/               |               \|/
>> > >         IO thread         main thread  <-------+
>> >
>> > Is the queue per monitor or per client?
>
> The queue is currently global. I think yes maybe at least we can do it
> per monitor, but I am not sure whether that is urgent or can be
> postponed.  After all now QMPRequest (please refer to patch 11) is
> defined as (mon, id, req) tuple, so at least "id" namespace is
> per-monitor.
>
>> > And is the dispatching going
>> > to be processed even if the client is disconnected, and are new
>> > clients going to receive the replies from previous clients
>> > commands?
>
> [1]
>
> (will discuss together below)
>
>> > I
>> > believe there should be a per-client context, so there won't be "id"
>> > request conflicts.
>
> I'd say I am not familiar with this "client" idea, since after all
> IMHO one monitor is currently designed to mostly work with a single
> client. Say, unix sockets, telnet, all these backends are only single
> channeled, and one monitor instance can only work with one client at a
> time.  Then do we really need to add this client layer upon it?  IMHO
> the user can just provide more monitors if they wants more clients
> (and at least these clients should know the existance of the others or
> there might be problem, otherwise user2 will fail a migration, finally
> noticed that user1 has already triggered one), and the user should
> manage them well.

qemu should support a management layer / libvirt restart/reconnect.
Afaik, it mostly work today. There might be a cases where libvirt can
be confused if it receives a reply from a previous connection command,
but due to the sync processing of the chardev, I am not sure you can
get in this situation.  By adding "oob" commands and queuing, the
client will have to remember which was the last "id" used, or it will
create more conflict after a reconnect.

Imho we should introduce the client/connection concept to avoid this
confusion (unexpected reply & per client id space).

>
>> >
>> > >
>> > > Then it introduced the "allow-oob" parameter in QAPI schema to define
>> > > commands, and "run-oob" flag to let oob-allowed command to run in the
>> > > parser.
>> >
>> > From a protocol point of view, I find that "run-oob" distinction per
>> > command a bit pointless. It helps with legacy client that wouldn't
>> > expect out-of-order replies if qemu were to run oob commands oob by
>> > default though.
>
> After all oob somehow breaks existing rules or sync execution.  I
> thought the more important goal was at least to keep the legacy
> behaviors when adding new things, no?

Of course we have to keep compatibily. What do you mean by "oob
somehow breaks existing rules or sync execution"? oob means queuing
and unordered reply support, so clearly this is breaking the current
"mostly ordered" behaviour (mostly because events may still come any
time..., and the reconnect issue discussed above).

>> > Clients shouldn't care about how/where a command is
>> > being queued or not. If they send a command, they want it processed as
>> > quickly as possible. However, it can be interesting to know if the
>> > implementation of the command will be able to deliver oob, so that
>> > data in the introspection could be useful.
>> >
>> > I would rather propose a client/server capability in qmp_capabilities,
>> > call it "oob":
>> >
>> > This capability indicates oob commands support.
>>
>> The problem is indicating which commands support oob as opposed to
>> indicating whether oob is present at all.  Future versions will
>> probably make more commands oob-able and a client will want to know
>> whether it can rely on a particular command being non-blocking.
>
> Yes.
>
> And IMHO we don't urgently need that "whether the server globally
> supports oob" thing.  Client can just know that from query-qmp-schema
> already - there will always be the "allow-oob" new field for command
> typed entries.  IMHO that's a solid hint.
>
> But I don't object to return it as well in qmp_capabilities.

Does it feel right that the client can specify how the command are
processed / queued ? Isn't it preferable to leave that to the server
to decide? Why would a client specify that? And should the server be
expected to behave differently? What the client needs to be able is to
match the unordered replies, and that can be stated during cap
negotiation / qmp_capabilties. The server is expected to do a best
effort to handle commands and their priorities. If the client needs
several command queue, it is simpler to open several connection rather
than trying to fit that weird priority logic in the protocol imho.

>
>>
>> > An oob command is a regular client message request with the "id"
>> > member mandatory, but the reply may be delivered
>> > out of order by the server if the client supports
>> > it too.
>> >
>> > If both the server and the client have the "oob" capability, the
>> > server can handle new client requests while previous requests are being
>> > processed.
>> >
>> > If the client doesn't have the "oob" capability, it may still call
>> > an oob command, and make multiple outstanding calls. In this case,
>> > the commands are processed in order, so the replies will also be in
>> > order. The "id" member isn't mandatory in this case.
>> >
>> > The client should match the replies with the "id" member associated
>> > with the requests.
>> >
>> > When a client is disconnected, the pending commands are not
>> > necessarily cancelled. But the future clients will not get replies from
>> > commands they didn't make (they might, however, receive side-effects
>> > events).
>>
>> What's the behaviour on the current monitor?
>
> Yeah I want to ask the same question, along with questioning about
> above [1].
>
> IMHO this series will not change the behaviors of these, so IMHO the
> behaviors will be the same before/after this series. E.g., when client
> dropped right after the command is executed, I think we will still
> execute the command, though we should encounter something odd in
> monitor_json_emitter() somewhere when we want to respond.  And it will
> happen the same after this series.

I think it can get worse after your series, because you queue the
commands, so clearly a new client can get replies from an old client
commands. As said above, I am not convinced you can get in that
situation with current code.

>
>>
>>
>> > Note that without "oob" support, a client may still receive
>> >  messages (or events) from the server between the time a
>> > request is handled by the server and the reply is received. It must
>> > thus be prepared to handle dispatching both events and reply after
>> > sending a request.
>> >
>> >
>> > (see also https://lists.gnu.org/archive/html/qemu-devel/2017-01/msg03641.html)
>> >
>> >
>> > > The last patch enables this for "migrate-incoming" command.
>> > >
>> > > Please review.  Thanks.
>> > >
>> > > [1] https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
>> > >
>> > > Peter Xu (15):
>> > >   char-io: fix possible race on IOWatchPoll
>> > >   qobject: allow NULL for qstring_get_str()
>> > >   qobject: introduce qobject_to_str()
>> > >   monitor: move skip_flush into monitor_data_init
>> > >   qjson: add "opaque" field to JSONMessageParser
>> > >   monitor: move the cur_mon hack deeper for QMP
>> > >   monitor: unify global init
>> > >   monitor: create IO thread
>> > >   monitor: allow to use IO thread for parsing
>> > >   monitor: introduce monitor_qmp_respond()
>> > >   monitor: separate QMP parser and dispatcher
>> >
>> > There should be a limit in the number of requests the thread can
>> > queue. Before the patch, the limit was enforced by system socket
>> > buffering I think. Now, should oob commands still be processed even if
>> > the queue is full? If so, the thread can't be suspended.
>>
>> I think the previous discussion was expecting a pair of queues
>> per client and perhaps a pair of central queues; each pair being
>> for normal command and oob commands.
>> (I'm not expecting these queues to be deep; IMHO '1' is the
>> right size for this type of queue in both cases).
>
> Yes.  One thing to mention is that, if we see the graph above, I
> didn't really introduce two queues (input/output), but only one input
> queue.  The response is still handled in dispatcher for now, since I
> think that's quite enough at least for OOB, and I didn't see much
> benefit now to split that 2nd queue.
>
> So I am thinking whether we can just quickly respond an event when the
> queue is full, as I proposed in the other reply.
>
> Regarding to queue size: I am afraid max_size=1 may not suffice?
> Otherwise a simple batch of:
>
> {"execute": "query-status"} {"execute": "query-status"}
>
> Will trigger the failure.  But I definitely agree it should not be
> something very large.  The total memory will be this:
>
>   json limit * queue length limit * monitor count limit
>       (X)            (Y)                    (Z)
>
> Now we have (X) already (in form of a few tunables for JSON token
> counts, etc.), we don't have (Z), and we definitely need (Y).
>
> How about we add limits on Y=16 and Z=8?
>
> We can do some math if we want some more exact number though.
>
>>
>> Dave
>>
>> > >   monitor: enable IO thread for (qmp & !mux) typed
>> > >   qapi: introduce new cmd option "allow-oob"
>> > >   qmp: support out-of-band (oob) execution
>> > >   qmp: let migrate-incoming allow out-of-band
>> > >
>> > >  chardev/char-io.c                |  15 ++-
>> > >  docs/devel/qapi-code-gen.txt     |  51 ++++++-
>> > >  include/monitor/monitor.h        |   2 +-
>> > >  include/qapi/qmp/dispatch.h      |   2 +
>> > >  include/qapi/qmp/json-streamer.h |   8 +-
>> > >  include/qapi/qmp/qstring.h       |   1 +
>> > >  monitor.c                        | 283 +++++++++++++++++++++++++++++++--------
>> > >  qapi/introspect.json             |   6 +-
>> > >  qapi/migration.json              |   3 +-
>> > >  qapi/qmp-dispatch.c              |  34 +++++
>> > >  qga/main.c                       |   5 +-
>> > >  qobject/json-streamer.c          |   7 +-
>> > >  qobject/qjson.c                  |   5 +-
>> > >  qobject/qstring.c                |  13 +-
>> > >  scripts/qapi-commands.py         |  19 ++-
>> > >  scripts/qapi-introspect.py       |  10 +-
>> > >  scripts/qapi.py                  |  15 ++-
>> > >  scripts/qapi2texi.py             |   2 +-
>> > >  tests/libqtest.c                 |   5 +-
>> > >  tests/qapi-schema/test-qapi.py   |   2 +-
>> > >  trace-events                     |   2 +
>> > >  vl.c                             |   3 +-
>> > >  22 files changed, 398 insertions(+), 95 deletions(-)
>> > >
>> > > --
>> > > 2.7.4
>> > >
>> >
>> >
>> >
>> > --
>> > Marc-André Lureau
>> --
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> --
> Peter Xu



-- 
Marc-André Lureau

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Daniel P. Berrange]

On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > There should be a limit in the number of requests the thread can
> > > > queue. Before the patch, the limit was enforced by system socket
> > > > buffering I think. Now, should oob commands still be processed even if
> > > > the queue is full? If so, the thread can't be suspended.
> > > 
> > > I agree.
> > > 
> > > Memory usage must be bounded.  The number of requests is less important
> > > than the amount of memory consumed by them.
> > > 
> > > Existing QMP clients that send multiple QMP commands without waiting for
> > > replies need to rethink their strategy because OOB commands cannot be
> > > processed if queued non-OOB commands consume too much memory.
> > 
> > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > as Markus pointed out as well in previous discussions (in "Flow
> > Control" section of that long reply).  Hopefully this series basically
> > can work from design prospective, then I'll add this flow control in
> > next version.
> > 
> > Regarding to what we should do if the limit is reached: Markus
> > provided a few options, but the one I prefer most is that we don't
> > respond, but send an event showing that a command is dropped.
> > However, I would like it not queued, but a direct reply (after all,
> > it's an event, and we should not need to care much on ordering of it).
> > Then we can get rid of the babysitting of those "to be failed"
> > requests asap, meanwhile we don't lose anything IMHO.
> > 
> > I think I also missed at least a unit test for this new interface.
> > Again, I'll add it after the whole idea is proved solid.  Thanks,
> 
> Another solution: the server reports available receive buffer space to
> the client.  The server only guarantees immediate OOB processing when
> the client stays within the receive buffer size.
> 
> Clients wishing to take advantage of OOB must query the receive buffer
> size and make sure to leave enough room.

I don't think having to query it ahead of time is particularly nice,
and of course it is inherantly racy.

I would just have QEMU emit an event when it pausing processing of the
incoming commands due to a full queue.  If the event includes the ID
of the last queued command, the client will know which (if any) of
its outstanding commands are delayed. Another even can be sent when
it restarts reading.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Daniel P. Berrange (berrange@redhat.com) wrote:
> On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > There should be a limit in the number of requests the thread can
> > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > the queue is full? If so, the thread can't be suspended.
> > > > 
> > > > I agree.
> > > > 
> > > > Memory usage must be bounded.  The number of requests is less important
> > > > than the amount of memory consumed by them.
> > > > 
> > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > replies need to rethink their strategy because OOB commands cannot be
> > > > processed if queued non-OOB commands consume too much memory.
> > > 
> > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > as Markus pointed out as well in previous discussions (in "Flow
> > > Control" section of that long reply).  Hopefully this series basically
> > > can work from design prospective, then I'll add this flow control in
> > > next version.
> > > 
> > > Regarding to what we should do if the limit is reached: Markus
> > > provided a few options, but the one I prefer most is that we don't
> > > respond, but send an event showing that a command is dropped.
> > > However, I would like it not queued, but a direct reply (after all,
> > > it's an event, and we should not need to care much on ordering of it).
> > > Then we can get rid of the babysitting of those "to be failed"
> > > requests asap, meanwhile we don't lose anything IMHO.
> > > 
> > > I think I also missed at least a unit test for this new interface.
> > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > 
> > Another solution: the server reports available receive buffer space to
> > the client.  The server only guarantees immediate OOB processing when
> > the client stays within the receive buffer size.
> > 
> > Clients wishing to take advantage of OOB must query the receive buffer
> > size and make sure to leave enough room.
> 
> I don't think having to query it ahead of time is particularly nice,
> and of course it is inherantly racy.
> 
> I would just have QEMU emit an event when it pausing processing of the
> incoming commands due to a full queue.  If the event includes the ID
> of the last queued command, the client will know which (if any) of
> its outstanding commands are delayed. Another even can be sent when
> it restarts reading.

Hmm and now we're implementing flow control!

a) What exactly is the current semantics/buffer sizes?
b) When do clients send multiple QMP commands on one channel without
waiting for the response to the previous command?
c) Would one queue entry for each class of commands/channel work
  (Where a class of commands is currently 'normal' and 'oob')

Dave

> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Daniel P. Berrange]

On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> * Daniel P. Berrange (berrange@redhat.com) wrote:
> > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > There should be a limit in the number of requests the thread can
> > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > 
> > > > > I agree.
> > > > > 
> > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > than the amount of memory consumed by them.
> > > > > 
> > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > processed if queued non-OOB commands consume too much memory.
> > > > 
> > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > Control" section of that long reply).  Hopefully this series basically
> > > > can work from design prospective, then I'll add this flow control in
> > > > next version.
> > > > 
> > > > Regarding to what we should do if the limit is reached: Markus
> > > > provided a few options, but the one I prefer most is that we don't
> > > > respond, but send an event showing that a command is dropped.
> > > > However, I would like it not queued, but a direct reply (after all,
> > > > it's an event, and we should not need to care much on ordering of it).
> > > > Then we can get rid of the babysitting of those "to be failed"
> > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > 
> > > > I think I also missed at least a unit test for this new interface.
> > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > 
> > > Another solution: the server reports available receive buffer space to
> > > the client.  The server only guarantees immediate OOB processing when
> > > the client stays within the receive buffer size.
> > > 
> > > Clients wishing to take advantage of OOB must query the receive buffer
> > > size and make sure to leave enough room.
> > 
> > I don't think having to query it ahead of time is particularly nice,
> > and of course it is inherantly racy.
> > 
> > I would just have QEMU emit an event when it pausing processing of the
> > incoming commands due to a full queue.  If the event includes the ID
> > of the last queued command, the client will know which (if any) of
> > its outstanding commands are delayed. Another even can be sent when
> > it restarts reading.
> 
> Hmm and now we're implementing flow control!
> 
> a) What exactly is the current semantics/buffer sizes?
> b) When do clients send multiple QMP commands on one channel without
> waiting for the response to the previous command?
> c) Would one queue entry for each class of commands/channel work
>   (Where a class of commands is currently 'normal' and 'oob')

I do wonder if we need to worry about request limiting at all from the
client side.  For non-OOB commands clients will wait for a reply before
sending a 2nd non-OOB command, so you'll never get a deep queue for.

OOB commands are supposed to be things which can be handled quickly
without blocking, so even if a client sent several commands at once
without waiting for replies, they're going to be processed quickly,
so whether we temporarily block reading off the wire is a minor
detail.

IOW, I think we could just have a fixed 10 command queue and apps just
pretend that there's an infinite queue and nothing bad would happen from
the app's POV.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Daniel P. Berrange (berrange@redhat.com) wrote:
> On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > > There should be a limit in the number of requests the thread can
> > > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > > 
> > > > > > I agree.
> > > > > > 
> > > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > > than the amount of memory consumed by them.
> > > > > > 
> > > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > > processed if queued non-OOB commands consume too much memory.
> > > > > 
> > > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > > Control" section of that long reply).  Hopefully this series basically
> > > > > can work from design prospective, then I'll add this flow control in
> > > > > next version.
> > > > > 
> > > > > Regarding to what we should do if the limit is reached: Markus
> > > > > provided a few options, but the one I prefer most is that we don't
> > > > > respond, but send an event showing that a command is dropped.
> > > > > However, I would like it not queued, but a direct reply (after all,
> > > > > it's an event, and we should not need to care much on ordering of it).
> > > > > Then we can get rid of the babysitting of those "to be failed"
> > > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > > 
> > > > > I think I also missed at least a unit test for this new interface.
> > > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > > 
> > > > Another solution: the server reports available receive buffer space to
> > > > the client.  The server only guarantees immediate OOB processing when
> > > > the client stays within the receive buffer size.
> > > > 
> > > > Clients wishing to take advantage of OOB must query the receive buffer
> > > > size and make sure to leave enough room.
> > > 
> > > I don't think having to query it ahead of time is particularly nice,
> > > and of course it is inherantly racy.
> > > 
> > > I would just have QEMU emit an event when it pausing processing of the
> > > incoming commands due to a full queue.  If the event includes the ID
> > > of the last queued command, the client will know which (if any) of
> > > its outstanding commands are delayed. Another even can be sent when
> > > it restarts reading.
> > 
> > Hmm and now we're implementing flow control!
> > 
> > a) What exactly is the current semantics/buffer sizes?
> > b) When do clients send multiple QMP commands on one channel without
> > waiting for the response to the previous command?
> > c) Would one queue entry for each class of commands/channel work
> >   (Where a class of commands is currently 'normal' and 'oob')
> 
> I do wonder if we need to worry about request limiting at all from the
> client side.  For non-OOB commands clients will wait for a reply before
> sending a 2nd non-OOB command, so you'll never get a deep queue for.
> 
> OOB commands are supposed to be things which can be handled quickly
> without blocking, so even if a client sent several commands at once
> without waiting for replies, they're going to be processed quickly,
> so whether we temporarily block reading off the wire is a minor
> detail.

Lets just define it so that it can't - you send an OOB command and wait
for it's response before sending another on that channel.

> IOW, I think we could just have a fixed 10 command queue and apps just
> pretend that there's an infinite queue and nothing bad would happen from
> the app's POV.

Can you justify 10 as opposed to 1?

Dave

> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Daniel P. Berrange]

On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:
> * Daniel P. Berrange (berrange@redhat.com) wrote:
> > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > > > There should be a limit in the number of requests the thread can
> > > > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > > > 
> > > > > > > I agree.
> > > > > > > 
> > > > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > > > than the amount of memory consumed by them.
> > > > > > > 
> > > > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > > > processed if queued non-OOB commands consume too much memory.
> > > > > > 
> > > > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > > > Control" section of that long reply).  Hopefully this series basically
> > > > > > can work from design prospective, then I'll add this flow control in
> > > > > > next version.
> > > > > > 
> > > > > > Regarding to what we should do if the limit is reached: Markus
> > > > > > provided a few options, but the one I prefer most is that we don't
> > > > > > respond, but send an event showing that a command is dropped.
> > > > > > However, I would like it not queued, but a direct reply (after all,
> > > > > > it's an event, and we should not need to care much on ordering of it).
> > > > > > Then we can get rid of the babysitting of those "to be failed"
> > > > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > > > 
> > > > > > I think I also missed at least a unit test for this new interface.
> > > > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > > > 
> > > > > Another solution: the server reports available receive buffer space to
> > > > > the client.  The server only guarantees immediate OOB processing when
> > > > > the client stays within the receive buffer size.
> > > > > 
> > > > > Clients wishing to take advantage of OOB must query the receive buffer
> > > > > size and make sure to leave enough room.
> > > > 
> > > > I don't think having to query it ahead of time is particularly nice,
> > > > and of course it is inherantly racy.
> > > > 
> > > > I would just have QEMU emit an event when it pausing processing of the
> > > > incoming commands due to a full queue.  If the event includes the ID
> > > > of the last queued command, the client will know which (if any) of
> > > > its outstanding commands are delayed. Another even can be sent when
> > > > it restarts reading.
> > > 
> > > Hmm and now we're implementing flow control!
> > > 
> > > a) What exactly is the current semantics/buffer sizes?
> > > b) When do clients send multiple QMP commands on one channel without
> > > waiting for the response to the previous command?
> > > c) Would one queue entry for each class of commands/channel work
> > >   (Where a class of commands is currently 'normal' and 'oob')
> > 
> > I do wonder if we need to worry about request limiting at all from the
> > client side.  For non-OOB commands clients will wait for a reply before
> > sending a 2nd non-OOB command, so you'll never get a deep queue for.
> > 
> > OOB commands are supposed to be things which can be handled quickly
> > without blocking, so even if a client sent several commands at once
> > without waiting for replies, they're going to be processed quickly,
> > so whether we temporarily block reading off the wire is a minor
> > detail.
> 
> Lets just define it so that it can't - you send an OOB command and wait
> for it's response before sending another on that channel.
> 
> > IOW, I think we could just have a fixed 10 command queue and apps just
> > pretend that there's an infinite queue and nothing bad would happen from
> > the app's POV.
> 
> Can you justify 10 as opposed to 1?

Semantically I don't think it makes a difference if the OOB commands are
being processed sequentially by their thread. A >1 length queue would only
matter for non-OOB commands if an app was filling the pipeline with non-OOB
requests, as then that could block reading of OOB commands. 


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Daniel P. Berrange (berrange@redhat.com) wrote:
> On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:
> > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > > > > There should be a limit in the number of requests the thread can
> > > > > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > > > > 
> > > > > > > > I agree.
> > > > > > > > 
> > > > > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > > > > than the amount of memory consumed by them.
> > > > > > > > 
> > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > > > > processed if queued non-OOB commands consume too much memory.
> > > > > > > 
> > > > > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > > > > Control" section of that long reply).  Hopefully this series basically
> > > > > > > can work from design prospective, then I'll add this flow control in
> > > > > > > next version.
> > > > > > > 
> > > > > > > Regarding to what we should do if the limit is reached: Markus
> > > > > > > provided a few options, but the one I prefer most is that we don't
> > > > > > > respond, but send an event showing that a command is dropped.
> > > > > > > However, I would like it not queued, but a direct reply (after all,
> > > > > > > it's an event, and we should not need to care much on ordering of it).
> > > > > > > Then we can get rid of the babysitting of those "to be failed"
> > > > > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > > > > 
> > > > > > > I think I also missed at least a unit test for this new interface.
> > > > > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > > > > 
> > > > > > Another solution: the server reports available receive buffer space to
> > > > > > the client.  The server only guarantees immediate OOB processing when
> > > > > > the client stays within the receive buffer size.
> > > > > > 
> > > > > > Clients wishing to take advantage of OOB must query the receive buffer
> > > > > > size and make sure to leave enough room.
> > > > > 
> > > > > I don't think having to query it ahead of time is particularly nice,
> > > > > and of course it is inherantly racy.
> > > > > 
> > > > > I would just have QEMU emit an event when it pausing processing of the
> > > > > incoming commands due to a full queue.  If the event includes the ID
> > > > > of the last queued command, the client will know which (if any) of
> > > > > its outstanding commands are delayed. Another even can be sent when
> > > > > it restarts reading.
> > > > 
> > > > Hmm and now we're implementing flow control!
> > > > 
> > > > a) What exactly is the current semantics/buffer sizes?
> > > > b) When do clients send multiple QMP commands on one channel without
> > > > waiting for the response to the previous command?
> > > > c) Would one queue entry for each class of commands/channel work
> > > >   (Where a class of commands is currently 'normal' and 'oob')
> > > 
> > > I do wonder if we need to worry about request limiting at all from the
> > > client side.  For non-OOB commands clients will wait for a reply before
> > > sending a 2nd non-OOB command, so you'll never get a deep queue for.
> > > 
> > > OOB commands are supposed to be things which can be handled quickly
> > > without blocking, so even if a client sent several commands at once
> > > without waiting for replies, they're going to be processed quickly,
> > > so whether we temporarily block reading off the wire is a minor
> > > detail.
> > 
> > Lets just define it so that it can't - you send an OOB command and wait
> > for it's response before sending another on that channel.
> > 
> > > IOW, I think we could just have a fixed 10 command queue and apps just
> > > pretend that there's an infinite queue and nothing bad would happen from
> > > the app's POV.
> > 
> > Can you justify 10 as opposed to 1?
> 
> Semantically I don't think it makes a difference if the OOB commands are
> being processed sequentially by their thread. A >1 length queue would only
> matter for non-OOB commands if an app was filling the pipeline with non-OOB
> requests, as then that could block reading of OOB commands. 

But can't we just tell the app not to?

Dave

> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Daniel P. Berrange]

On Fri, Sep 15, 2017 at 03:29:46PM +0100, Dr. David Alan Gilbert wrote:
> * Daniel P. Berrange (berrange@redhat.com) wrote:
> > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:
> > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > > > > > There should be a limit in the number of requests the thread can
> > > > > > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > > > > > 
> > > > > > > > > I agree.
> > > > > > > > > 
> > > > > > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > > > > > than the amount of memory consumed by them.
> > > > > > > > > 
> > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > > > > > processed if queued non-OOB commands consume too much memory.
> > > > > > > > 
> > > > > > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > > > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > > > > > Control" section of that long reply).  Hopefully this series basically
> > > > > > > > can work from design prospective, then I'll add this flow control in
> > > > > > > > next version.
> > > > > > > > 
> > > > > > > > Regarding to what we should do if the limit is reached: Markus
> > > > > > > > provided a few options, but the one I prefer most is that we don't
> > > > > > > > respond, but send an event showing that a command is dropped.
> > > > > > > > However, I would like it not queued, but a direct reply (after all,
> > > > > > > > it's an event, and we should not need to care much on ordering of it).
> > > > > > > > Then we can get rid of the babysitting of those "to be failed"
> > > > > > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > > > > > 
> > > > > > > > I think I also missed at least a unit test for this new interface.
> > > > > > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > > > > > 
> > > > > > > Another solution: the server reports available receive buffer space to
> > > > > > > the client.  The server only guarantees immediate OOB processing when
> > > > > > > the client stays within the receive buffer size.
> > > > > > > 
> > > > > > > Clients wishing to take advantage of OOB must query the receive buffer
> > > > > > > size and make sure to leave enough room.
> > > > > > 
> > > > > > I don't think having to query it ahead of time is particularly nice,
> > > > > > and of course it is inherantly racy.
> > > > > > 
> > > > > > I would just have QEMU emit an event when it pausing processing of the
> > > > > > incoming commands due to a full queue.  If the event includes the ID
> > > > > > of the last queued command, the client will know which (if any) of
> > > > > > its outstanding commands are delayed. Another even can be sent when
> > > > > > it restarts reading.
> > > > > 
> > > > > Hmm and now we're implementing flow control!
> > > > > 
> > > > > a) What exactly is the current semantics/buffer sizes?
> > > > > b) When do clients send multiple QMP commands on one channel without
> > > > > waiting for the response to the previous command?
> > > > > c) Would one queue entry for each class of commands/channel work
> > > > >   (Where a class of commands is currently 'normal' and 'oob')
> > > > 
> > > > I do wonder if we need to worry about request limiting at all from the
> > > > client side.  For non-OOB commands clients will wait for a reply before
> > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.
> > > > 
> > > > OOB commands are supposed to be things which can be handled quickly
> > > > without blocking, so even if a client sent several commands at once
> > > > without waiting for replies, they're going to be processed quickly,
> > > > so whether we temporarily block reading off the wire is a minor
> > > > detail.
> > > 
> > > Lets just define it so that it can't - you send an OOB command and wait
> > > for it's response before sending another on that channel.
> > > 
> > > > IOW, I think we could just have a fixed 10 command queue and apps just
> > > > pretend that there's an infinite queue and nothing bad would happen from
> > > > the app's POV.
> > > 
> > > Can you justify 10 as opposed to 1?
> > 
> > Semantically I don't think it makes a difference if the OOB commands are
> > being processed sequentially by their thread. A >1 length queue would only
> > matter for non-OOB commands if an app was filling the pipeline with non-OOB
> > requests, as then that could block reading of OOB commands. 
> 
> But can't we just tell the app not to?

Yes, a sensible app would not do that. So this feels like mostly a documentation
problem.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Stefan Hajnoczi]

On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:
> On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:
> > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > > > > There should be a limit in the number of requests the thread can
> > > > > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > > > > 
> > > > > > > > I agree.
> > > > > > > > 
> > > > > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > > > > than the amount of memory consumed by them.
> > > > > > > > 
> > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > > > > processed if queued non-OOB commands consume too much memory.
> > > > > > > 
> > > > > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > > > > Control" section of that long reply).  Hopefully this series basically
> > > > > > > can work from design prospective, then I'll add this flow control in
> > > > > > > next version.
> > > > > > > 
> > > > > > > Regarding to what we should do if the limit is reached: Markus
> > > > > > > provided a few options, but the one I prefer most is that we don't
> > > > > > > respond, but send an event showing that a command is dropped.
> > > > > > > However, I would like it not queued, but a direct reply (after all,
> > > > > > > it's an event, and we should not need to care much on ordering of it).
> > > > > > > Then we can get rid of the babysitting of those "to be failed"
> > > > > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > > > > 
> > > > > > > I think I also missed at least a unit test for this new interface.
> > > > > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > > > > 
> > > > > > Another solution: the server reports available receive buffer space to
> > > > > > the client.  The server only guarantees immediate OOB processing when
> > > > > > the client stays within the receive buffer size.
> > > > > > 
> > > > > > Clients wishing to take advantage of OOB must query the receive buffer
> > > > > > size and make sure to leave enough room.
> > > > > 
> > > > > I don't think having to query it ahead of time is particularly nice,
> > > > > and of course it is inherantly racy.
> > > > > 
> > > > > I would just have QEMU emit an event when it pausing processing of the
> > > > > incoming commands due to a full queue.  If the event includes the ID
> > > > > of the last queued command, the client will know which (if any) of
> > > > > its outstanding commands are delayed. Another even can be sent when
> > > > > it restarts reading.
> > > > 
> > > > Hmm and now we're implementing flow control!
> > > > 
> > > > a) What exactly is the current semantics/buffer sizes?
> > > > b) When do clients send multiple QMP commands on one channel without
> > > > waiting for the response to the previous command?
> > > > c) Would one queue entry for each class of commands/channel work
> > > >   (Where a class of commands is currently 'normal' and 'oob')
> > > 
> > > I do wonder if we need to worry about request limiting at all from the
> > > client side.  For non-OOB commands clients will wait for a reply before
> > > sending a 2nd non-OOB command, so you'll never get a deep queue for.
> > > 
> > > OOB commands are supposed to be things which can be handled quickly
> > > without blocking, so even if a client sent several commands at once
> > > without waiting for replies, they're going to be processed quickly,
> > > so whether we temporarily block reading off the wire is a minor
> > > detail.
> > 
> > Lets just define it so that it can't - you send an OOB command and wait
> > for it's response before sending another on that channel.
> > 
> > > IOW, I think we could just have a fixed 10 command queue and apps just
> > > pretend that there's an infinite queue and nothing bad would happen from
> > > the app's POV.
> > 
> > Can you justify 10 as opposed to 1?
> 
> Semantically I don't think it makes a difference if the OOB commands are
> being processed sequentially by their thread. A >1 length queue would only
> matter for non-OOB commands if an app was filling the pipeline with non-OOB
> requests, as then that could block reading of OOB commands. 

To summarize:

The QMP server has a lookahead of 1 command so it can dispatch
out-of-band commands.  If 2 or more non-OOB commands are queued at the
same time then OOB processing will not occur.

Is that right?

Stefan

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Stefan Hajnoczi (stefanha@redhat.com) wrote:
> On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:
> > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:
> > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > > > > > There should be a limit in the number of requests the thread can
> > > > > > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > > > > > 
> > > > > > > > > I agree.
> > > > > > > > > 
> > > > > > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > > > > > than the amount of memory consumed by them.
> > > > > > > > > 
> > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > > > > > processed if queued non-OOB commands consume too much memory.
> > > > > > > > 
> > > > > > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > > > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > > > > > Control" section of that long reply).  Hopefully this series basically
> > > > > > > > can work from design prospective, then I'll add this flow control in
> > > > > > > > next version.
> > > > > > > > 
> > > > > > > > Regarding to what we should do if the limit is reached: Markus
> > > > > > > > provided a few options, but the one I prefer most is that we don't
> > > > > > > > respond, but send an event showing that a command is dropped.
> > > > > > > > However, I would like it not queued, but a direct reply (after all,
> > > > > > > > it's an event, and we should not need to care much on ordering of it).
> > > > > > > > Then we can get rid of the babysitting of those "to be failed"
> > > > > > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > > > > > 
> > > > > > > > I think I also missed at least a unit test for this new interface.
> > > > > > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > > > > > 
> > > > > > > Another solution: the server reports available receive buffer space to
> > > > > > > the client.  The server only guarantees immediate OOB processing when
> > > > > > > the client stays within the receive buffer size.
> > > > > > > 
> > > > > > > Clients wishing to take advantage of OOB must query the receive buffer
> > > > > > > size and make sure to leave enough room.
> > > > > > 
> > > > > > I don't think having to query it ahead of time is particularly nice,
> > > > > > and of course it is inherantly racy.
> > > > > > 
> > > > > > I would just have QEMU emit an event when it pausing processing of the
> > > > > > incoming commands due to a full queue.  If the event includes the ID
> > > > > > of the last queued command, the client will know which (if any) of
> > > > > > its outstanding commands are delayed. Another even can be sent when
> > > > > > it restarts reading.
> > > > > 
> > > > > Hmm and now we're implementing flow control!
> > > > > 
> > > > > a) What exactly is the current semantics/buffer sizes?
> > > > > b) When do clients send multiple QMP commands on one channel without
> > > > > waiting for the response to the previous command?
> > > > > c) Would one queue entry for each class of commands/channel work
> > > > >   (Where a class of commands is currently 'normal' and 'oob')
> > > > 
> > > > I do wonder if we need to worry about request limiting at all from the
> > > > client side.  For non-OOB commands clients will wait for a reply before
> > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.
> > > > 
> > > > OOB commands are supposed to be things which can be handled quickly
> > > > without blocking, so even if a client sent several commands at once
> > > > without waiting for replies, they're going to be processed quickly,
> > > > so whether we temporarily block reading off the wire is a minor
> > > > detail.
> > > 
> > > Lets just define it so that it can't - you send an OOB command and wait
> > > for it's response before sending another on that channel.
> > > 
> > > > IOW, I think we could just have a fixed 10 command queue and apps just
> > > > pretend that there's an infinite queue and nothing bad would happen from
> > > > the app's POV.
> > > 
> > > Can you justify 10 as opposed to 1?
> > 
> > Semantically I don't think it makes a difference if the OOB commands are
> > being processed sequentially by their thread. A >1 length queue would only
> > matter for non-OOB commands if an app was filling the pipeline with non-OOB
> > requests, as then that could block reading of OOB commands. 
> 
> To summarize:
> 
> The QMP server has a lookahead of 1 command so it can dispatch
> out-of-band commands.  If 2 or more non-OOB commands are queued at the
> same time then OOB processing will not occur.
> 
> Is that right?

I think my view is slightly more complex;
  a) There's a pair of queues for each channel
  b) There's a central pair of queues on the QMP server
    one for OOB commands and one for normal commands.
  c) Each queue is only really guaranteed to be one deep.

  That means that each one of the channels can send a non-OOB
command without getting in the way of a channel that wants
to send one.

Dave

> Stefan
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 14/15] qmp: support out-of-band (oob) execution

[Dr. David Alan Gilbert]

* Peter Xu (peterx@redhat.com) wrote:
> Having "allow-oob" to true for a command does not mean that this command
> will always be run in out-of-band mode.  The out-of-band quick path will
> only be executed if we specify the extra "run-oob" flag when sending the
> QMP request:
> 
>   { "execute": "command-that-allows-oob",
>     "arguments": { ... },
>     "control": { "run-oob": true } }
> 
> The "control" key is introduced to store this extra flag.  "control"
> field is used to store arguments that are shared by all the commands,
> rather than command specific arguments.  Let "run-oob" be the first.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>

I don't understand how this enforces the allowoob, that is it stops
other commands being called with run-oob=true

> ---
>  docs/devel/qapi-code-gen.txt | 10 ++++++++++
>  include/qapi/qmp/dispatch.h  |  1 +
>  monitor.c                    | 11 +++++++++++
>  qapi/qmp-dispatch.c          | 34 ++++++++++++++++++++++++++++++++++
>  trace-events                 |  2 ++
>  5 files changed, 58 insertions(+)
> 
> diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
> index 61fa167..47d16bb 100644
> --- a/docs/devel/qapi-code-gen.txt
> +++ b/docs/devel/qapi-code-gen.txt
> @@ -665,6 +665,16 @@ allowed to run out-of-band can also be introspected using
>  query-qmp-schema command.  Please see the section "Client JSON
>  Protocol introspection" for more information.
>  
> +To execute a command in out-of-band way, we need to specify the
> +"control" field in the request, with "run-oob" set to true. Example:
> +
> + => { "execute": "command-support-oob",
> +      "arguments": { ... },
> +      "control": { "run-oob": true } }
> + <= { "return": { } }
> +
> +Without it, even the commands that supports out-of-band execution will
> +still be run in-band.
>  
>  === Events ===
>  
> diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
> index b767988..ee2b8ce 100644
> --- a/include/qapi/qmp/dispatch.h
> +++ b/include/qapi/qmp/dispatch.h
> @@ -49,6 +49,7 @@ bool qmp_command_is_enabled(const QmpCommand *cmd);
>  const char *qmp_command_name(const QmpCommand *cmd);
>  bool qmp_has_success_response(const QmpCommand *cmd);
>  QObject *qmp_build_error_object(Error *err);
> +bool qmp_is_oob(const QObject *request);
>  
>  typedef void (*qmp_cmd_callback_fn)(QmpCommand *cmd, void *opaque);
>  
> diff --git a/monitor.c b/monitor.c
> index 599ea36..cb96204 100644
> --- a/monitor.c
> +++ b/monitor.c
> @@ -3928,6 +3928,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
>          if (!req_obj) {
>              break;
>          }
> +        trace_monitor_qmp_cmd_in_band(qobject_get_str(req_obj->id));
>          monitor_qmp_dispatch_one(req_obj);
>      }
>  }
> @@ -3963,6 +3964,16 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
>      req_obj->id = id;
>      req_obj->req = req;
>  
> +    if (qmp_is_oob(req)) {
> +        /*
> +         * Trigger fast-path to handle the out-of-band request, by
> +         * executing the command directly in parser.
> +         */
> +        trace_monitor_qmp_cmd_out_of_band(qobject_get_str(req_obj->id));
> +        monitor_qmp_dispatch_one(req_obj);
> +        return;
> +    }

I wonder if there is a way to run all allowoob commands in this way;
the only difference being if they're not started with run-oob
you wiat for completion of !oob commands.
That way the allowoob commands are always run from the same
thread/context which feels like it should simplify them.

Dave

>      /*
>       * Put the request to the end of queue so that requests will be
>       * handled in time order.  Ownership for req_obj, req, id,
> diff --git a/qapi/qmp-dispatch.c b/qapi/qmp-dispatch.c
> index b41fa17..9a05dfa 100644
> --- a/qapi/qmp-dispatch.c
> +++ b/qapi/qmp-dispatch.c
> @@ -52,6 +52,12 @@ static QDict *qmp_dispatch_check_obj(const QObject *request, Error **errp)
>                             "QMP input member 'arguments' must be an object");
>                  return NULL;
>              }
> +        } else if (!strcmp(arg_name, "control")) {
> +            if (qobject_type(arg_obj) != QTYPE_QDICT) {
> +                error_setg(errp,
> +                           "QMP input member 'control' must be an object");
> +                return NULL;
> +            }
>          } else {
>              error_setg(errp, "QMP input member '%s' is unexpected",
>                         arg_name);
> @@ -122,6 +128,34 @@ QObject *qmp_build_error_object(Error *err)
>                                error_get_pretty(err));
>  }
>  
> +/*
> + * Detect whether a request should be run out-of-band, by quickly
> + * peeking at whether we have: { "control": { "run-oob": True } }. By
> + * default commands are run in-band.
> + */
> +bool qmp_is_oob(const QObject *request)
> +{
> +    QDict *dict;
> +    QBool *bool_obj;
> +
> +    dict = qobject_to_qdict(request);
> +    if (!dict) {
> +        return false;
> +    }
> +
> +    dict = qdict_get_qdict(dict, "control");
> +    if (!dict) {
> +        return false;
> +    }
> +
> +    bool_obj = qobject_to_qbool(qdict_get(dict, "run-oob"));
> +    if (!bool_obj) {
> +        return false;
> +    }
> +
> +    return qbool_get_bool(bool_obj);
> +}
> +
>  QObject *qmp_dispatch(QmpCommandList *cmds, QObject *request)
>  {
>      Error *err = NULL;
> diff --git a/trace-events b/trace-events
> index 1f50f56..f7900a6 100644
> --- a/trace-events
> +++ b/trace-events
> @@ -47,6 +47,8 @@ monitor_protocol_event_emit(uint32_t event, void *data) "event=%d data=%p"
>  monitor_protocol_event_queue(uint32_t event, void *qdict, uint64_t rate) "event=%d data=%p rate=%" PRId64
>  handle_hmp_command(void *mon, const char *cmdline) "mon %p cmdline: %s"
>  handle_qmp_command(void *mon, const char *req) "mon %p req: %s"
> +monitor_qmp_cmd_in_band(const char *id) "%s"
> +monitor_qmp_cmd_out_of_band(const char *id) "%s"
>  
>  # dma-helpers.c
>  dma_blk_io(void *dbs, void *bs, int64_t offset, bool to_dev) "dbs=%p bs=%p offset=%" PRId64 " to_dev=%d"
> -- 
> 2.7.4
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 15/15] qmp: let migrate-incoming allow out-of-band

[Dr. David Alan Gilbert]

* Peter Xu (peterx@redhat.com) wrote:
> So it can get rid of being run on main thread.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  qapi/migration.json | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/qapi/migration.json b/qapi/migration.json
> index ee2b3b8..dedc4f8 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -986,7 +986,8 @@
>  # <- { "return": {} }
>  #
>  ##
> -{ 'command': 'migrate-incoming', 'data': {'uri': 'str' } }
> +{ 'command': 'migrate-incoming', 'data': {'uri': 'str' },
> +  'allow-oob': true }

Are you completely convinced migrate-incoming is non-blocking?
What about during the tls handshake if tls is enabled?
(We only set non-blocking on the fd later)

I don't understand in this structure on which thread
process_incoming_migration_co gets run.

Dave

>  ##
>  # @xen-save-devices-state:
> -- 
> 2.7.4
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 14/15] qmp: support out-of-band (oob) execution

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 777 bytes --]

On 09/14/2017 09:59 PM, Peter Xu wrote:
> On Thu, Sep 14, 2017 at 04:33:34PM +0100, Stefan Hajnoczi wrote:
>> On Thu, Sep 14, 2017 at 03:50:35PM +0800, Peter Xu wrote:
>>> diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
>>> index 61fa167..47d16bb 100644
>>> --- a/docs/devel/qapi-code-gen.txt
>>> +++ b/docs/devel/qapi-code-gen.txt

>>
>> Is there a more relevant place to document QMP run-oob behavior than the
>> "How to use the QAPI code generator document"?
> 
> I agree, but I don't really know it. :(
> 
> Markus, could you provide a hint?

I'm not Markus, but I suggest docs/interop/qmp-spec.txt

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC 14/15] qmp: support out-of-band (oob) execution

[Peter Xu]

On Fri, Sep 15, 2017 at 01:34:24PM -0500, Eric Blake wrote:
> On 09/14/2017 09:59 PM, Peter Xu wrote:
> > On Thu, Sep 14, 2017 at 04:33:34PM +0100, Stefan Hajnoczi wrote:
> >> On Thu, Sep 14, 2017 at 03:50:35PM +0800, Peter Xu wrote:
> >>> diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
> >>> index 61fa167..47d16bb 100644
> >>> --- a/docs/devel/qapi-code-gen.txt
> >>> +++ b/docs/devel/qapi-code-gen.txt
> 
> >>
> >> Is there a more relevant place to document QMP run-oob behavior than the
> >> "How to use the QAPI code generator document"?
> > 
> > I agree, but I don't really know it. :(
> > 
> > Markus, could you provide a hint?
> 
> I'm not Markus, but I suggest docs/interop/qmp-spec.txt

Sorry I should have noticed it before (I am always not sensitive on
the word "interop").  Thanks Eric!

-- 
Peter Xu

Re: [Qemu-devel] [RFC 14/15] qmp: support out-of-band (oob) execution

[Peter Xu]

On Fri, Sep 15, 2017 at 04:55:51PM +0100, Dr. David Alan Gilbert wrote:
> * Peter Xu (peterx@redhat.com) wrote:
> > Having "allow-oob" to true for a command does not mean that this command
> > will always be run in out-of-band mode.  The out-of-band quick path will
> > only be executed if we specify the extra "run-oob" flag when sending the
> > QMP request:
> > 
> >   { "execute": "command-that-allows-oob",
> >     "arguments": { ... },
> >     "control": { "run-oob": true } }
> > 
> > The "control" key is introduced to store this extra flag.  "control"
> > field is used to store arguments that are shared by all the commands,
> > rather than command specific arguments.  Let "run-oob" be the first.
> > 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> 
> I don't understand how this enforces the allowoob, that is it stops
> other commands being called with run-oob=true

Here's what I thought:

OOB commands are executed directly in the parser, and currently we
only have one single parser/IO thread, then we can't have two oob
commands in parallel, can we? Say, if we got one OOB command, we won't
handle anything else (no matter OOB or non-OOB) before we finished
processing that OOB command.

> 
> > ---
> >  docs/devel/qapi-code-gen.txt | 10 ++++++++++
> >  include/qapi/qmp/dispatch.h  |  1 +
> >  monitor.c                    | 11 +++++++++++
> >  qapi/qmp-dispatch.c          | 34 ++++++++++++++++++++++++++++++++++
> >  trace-events                 |  2 ++
> >  5 files changed, 58 insertions(+)
> > 
> > diff --git a/docs/devel/qapi-code-gen.txt b/docs/devel/qapi-code-gen.txt
> > index 61fa167..47d16bb 100644
> > --- a/docs/devel/qapi-code-gen.txt
> > +++ b/docs/devel/qapi-code-gen.txt
> > @@ -665,6 +665,16 @@ allowed to run out-of-band can also be introspected using
> >  query-qmp-schema command.  Please see the section "Client JSON
> >  Protocol introspection" for more information.
> >  
> > +To execute a command in out-of-band way, we need to specify the
> > +"control" field in the request, with "run-oob" set to true. Example:
> > +
> > + => { "execute": "command-support-oob",
> > +      "arguments": { ... },
> > +      "control": { "run-oob": true } }
> > + <= { "return": { } }
> > +
> > +Without it, even the commands that supports out-of-band execution will
> > +still be run in-band.
> >  
> >  === Events ===
> >  
> > diff --git a/include/qapi/qmp/dispatch.h b/include/qapi/qmp/dispatch.h
> > index b767988..ee2b8ce 100644
> > --- a/include/qapi/qmp/dispatch.h
> > +++ b/include/qapi/qmp/dispatch.h
> > @@ -49,6 +49,7 @@ bool qmp_command_is_enabled(const QmpCommand *cmd);
> >  const char *qmp_command_name(const QmpCommand *cmd);
> >  bool qmp_has_success_response(const QmpCommand *cmd);
> >  QObject *qmp_build_error_object(Error *err);
> > +bool qmp_is_oob(const QObject *request);
> >  
> >  typedef void (*qmp_cmd_callback_fn)(QmpCommand *cmd, void *opaque);
> >  
> > diff --git a/monitor.c b/monitor.c
> > index 599ea36..cb96204 100644
> > --- a/monitor.c
> > +++ b/monitor.c
> > @@ -3928,6 +3928,7 @@ static void monitor_qmp_bh_dispatcher(void *data)
> >          if (!req_obj) {
> >              break;
> >          }
> > +        trace_monitor_qmp_cmd_in_band(qobject_get_str(req_obj->id));
> >          monitor_qmp_dispatch_one(req_obj);
> >      }
> >  }
> > @@ -3963,6 +3964,16 @@ static void handle_qmp_command(JSONMessageParser *parser, GQueue *tokens,
> >      req_obj->id = id;
> >      req_obj->req = req;
> >  
> > +    if (qmp_is_oob(req)) {
> > +        /*
> > +         * Trigger fast-path to handle the out-of-band request, by
> > +         * executing the command directly in parser.
> > +         */
> > +        trace_monitor_qmp_cmd_out_of_band(qobject_get_str(req_obj->id));
> > +        monitor_qmp_dispatch_one(req_obj);
> > +        return;
> > +    }
> 
> I wonder if there is a way to run all allowoob commands in this way;
> the only difference being if they're not started with run-oob
> you wiat for completion of !oob commands.
> That way the allowoob commands are always run from the same
> thread/context which feels like it should simplify them.

Maybe I misread the comment... Even with current patch, all OOB
commands will be run from the same thread/context (which is the newly
created parser thread), right?  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC 15/15] qmp: let migrate-incoming allow out-of-band

[Peter Xu]

On Fri, Sep 15, 2017 at 05:09:28PM +0100, Dr. David Alan Gilbert wrote:
> * Peter Xu (peterx@redhat.com) wrote:
> > So it can get rid of being run on main thread.
> > 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  qapi/migration.json | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > diff --git a/qapi/migration.json b/qapi/migration.json
> > index ee2b3b8..dedc4f8 100644
> > --- a/qapi/migration.json
> > +++ b/qapi/migration.json
> > @@ -986,7 +986,8 @@
> >  # <- { "return": {} }
> >  #
> >  ##
> > -{ 'command': 'migrate-incoming', 'data': {'uri': 'str' } }
> > +{ 'command': 'migrate-incoming', 'data': {'uri': 'str' },
> > +  'allow-oob': true }
> 
> Are you completely convinced migrate-incoming is non-blocking?
> What about during the tls handshake if tls is enabled?
> (We only set non-blocking on the fd later)

IIUC after socket_start_incoming_migration() we are done in
qmp_migrate_incoming(), then all the rest of the socket work will all
be done in main thread.

Note: g_source_attach() along the way in the whole
qmp_migrate_incoming() should be using NULL as the second parameter,
which means they'll attach the objects all to the default gcontext,
which targets to the main thread.

> 
> I don't understand in this structure on which thread
> process_incoming_migration_co gets run.

IIUC it should be the main thread as well.

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:
> Hi
> 
> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu <peterx@redhat.com> wrote:
> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> >> > Hi
> >> >
> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
> >> > > This series was born from this one:
> >> > >
> >> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> >> > >
> >> > > The design comes from Markus, and also the whole-bunch-of discussions
> >> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny
> >> > > ideas and suggestions.  Finally we got such a solution that seems to
> >> > > satisfy everyone.
> >> > >
> >> > > I re-started the versioning since this series is totally different
> >> > > from previous one.  Now it's version 1.
> >> > >
> >> > > In case new reviewers come along the way without reading previous
> >> > > discussions, I will try to do a summary on what this is all about.
> >> > >
> >> > > What is OOB execution?
> >> > > ======================
> >> > >
> >> > > It's the shortcut of Out-Of-Band execution, its name is given by
> >> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
> >> > > QMP is going throw these steps:
> >> > >
> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> >> > >           /|\    (2)                (3)     |
> >> > >        (1) |                               \|/ (4)
> >> > >            +---------  main thread  --------+
> >> > >
> >> > > The requests are executed by the so-called QMP-dispatcher after the
> >> > > JSON is parsed.  If OOB is on, we run the command directly in the
> >> > > parser and quickly returns.
> >> >
> >> > All commands should have the "id" field mandatory in this case, else
> >> > the client will not distinguish the replies coming from the last/oob
> >> > and the previous commands.
> >> >
> >> > This should probably be enforced upfront by client capability checks,
> >> > more below.
> >
> > Hmm yes since the oob commands are actually running in async way,
> > request ID should be needed here.  However I'm not sure whether
> > enabling the whole "request ID" thing is too big for this "try to be
> > small" oob change... And IMHO it suites better to be part of the whole
> > async work (no matter which implementation we'll use).
> >
> > How about this: we make "id" mandatory for "run-oob" requests only.
> > For oob commands, they will always have ID then no ordering issue, and
> > we can do it async; for the rest of non-oob commands, we still allow
> > them to go without ID, and since they are not oob, they'll always be
> > done in order as well.  Would this work?
> 
> This mixed-mode is imho more complicated to deal with than having the
> protocol enforced one way or the other, but that should work.
> 
> >
> >> >
> >> > > Yeah I know in current code the parser calls dispatcher directly
> >> > > (please see handle_qmp_command()).  However it's not true again after
> >> > > this series (parser will has its own IO thread, and dispatcher will
> >> > > still be run in main thread).  So this OOB does brings something
> >> > > different.
> >> > >
> >> > > There are more details on why OOB and the difference/relationship
> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
> >> > > slightly out of topic (and believe me, it's not easy for me to
> >> > > summarize that).  For more information, please refers to [1].
> >> > >
> >> > > Summary ends here.
> >> > >
> >> > > Some Implementation Details
> >> > > ===========================
> >> > >
> >> > > Again, I mentioned that the old QMP workflow is this:
> >> > >
> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> >> > >           /|\    (2)                (3)     |
> >> > >        (1) |                               \|/ (4)
> >> > >            +---------  main thread  --------+
> >> > >
> >> > > What this series does is, firstly:
> >> > >
> >> > >       JSON Parser     QMP Dispatcher --> Respond
> >> > >           /|\ |           /|\       (4)     |
> >> > >            |  | (2)        | (3)            |  (5)
> >> > >        (1) |  +----->      |               \|/
> >> > >            +---------  main thread  <-------+
> >> > >
> >> > > And further:
> >> > >
> >> > >                queue/kick
> >> > >      JSON Parser ======> QMP Dispatcher --> Respond
> >> > >          /|\ |     (3)       /|\        (4)    |
> >> > >       (1) |  | (2)            |                |  (5)
> >> > >           | \|/               |               \|/
> >> > >         IO thread         main thread  <-------+
> >> >
> >> > Is the queue per monitor or per client?
> >
> > The queue is currently global. I think yes maybe at least we can do it
> > per monitor, but I am not sure whether that is urgent or can be
> > postponed.  After all now QMPRequest (please refer to patch 11) is
> > defined as (mon, id, req) tuple, so at least "id" namespace is
> > per-monitor.
> >
> >> > And is the dispatching going
> >> > to be processed even if the client is disconnected, and are new
> >> > clients going to receive the replies from previous clients
> >> > commands?
> >
> > [1]
> >
> > (will discuss together below)
> >
> >> > I
> >> > believe there should be a per-client context, so there won't be "id"
> >> > request conflicts.
> >
> > I'd say I am not familiar with this "client" idea, since after all
> > IMHO one monitor is currently designed to mostly work with a single
> > client. Say, unix sockets, telnet, all these backends are only single
> > channeled, and one monitor instance can only work with one client at a
> > time.  Then do we really need to add this client layer upon it?  IMHO
> > the user can just provide more monitors if they wants more clients
> > (and at least these clients should know the existance of the others or
> > there might be problem, otherwise user2 will fail a migration, finally
> > noticed that user1 has already triggered one), and the user should
> > manage them well.
> 
> qemu should support a management layer / libvirt restart/reconnect.
> Afaik, it mostly work today. There might be a cases where libvirt can
> be confused if it receives a reply from a previous connection command,
> but due to the sync processing of the chardev, I am not sure you can
> get in this situation.  By adding "oob" commands and queuing, the
> client will have to remember which was the last "id" used, or it will
> create more conflict after a reconnect.
> 
> Imho we should introduce the client/connection concept to avoid this
> confusion (unexpected reply & per client id space).

Hmm I agree that the reconnect feature would be nice, but if so IMHO
instead of throwing responses away when client disconnect, we should
really keep them, and when the client reconnects, we queue the
responses again.

I think we have other quite simple ways to solve the "unexpected
reply" and "per-client-id duplication" issues you have mentioned.

Firstly, when client gets unexpected replies ("id" field not in its
own request queue), the client should just ignore that reply, which
seems natural to me.

Then, if client disconnected and reconnected, it should not have the
problem to generate duplicated id for request, since it should know
what requests it has sent already.  A simplest case I can think of is,
the ID should contains the following tuple:

  (client name, client unique ID, request ID)

Here "client name" can be something like "libvirt", which is the name
of client application;

"client unique ID" can be anything generated when client starts, it
identifies a single client session, maybe a UUID.

"request ID" can be a unsigned integer starts from zero, and increases
each time the client sends one request.

I believe current libvirt is using "client name" + "request ID".  It's
something similar (after all I think we don't normally have >1 libvirt
to manage single QEMU, so I think it should be good enough).

Then even if client disconnect and reconnect, request ID won't lose,
and no duplication would happen IMHO.

> 
> >
> >> >
> >> > >
> >> > > Then it introduced the "allow-oob" parameter in QAPI schema to define
> >> > > commands, and "run-oob" flag to let oob-allowed command to run in the
> >> > > parser.
> >> >
> >> > From a protocol point of view, I find that "run-oob" distinction per
> >> > command a bit pointless. It helps with legacy client that wouldn't
> >> > expect out-of-order replies if qemu were to run oob commands oob by
> >> > default though.
> >
> > After all oob somehow breaks existing rules or sync execution.  I
> > thought the more important goal was at least to keep the legacy
> > behaviors when adding new things, no?
> 
> Of course we have to keep compatibily. What do you mean by "oob
> somehow breaks existing rules or sync execution"? oob means queuing
> and unordered reply support, so clearly this is breaking the current
> "mostly ordered" behaviour (mostly because events may still come any
> time..., and the reconnect issue discussed above).

Yes.  That's what I mean, it breaks the synchronous scemantic.  But
I should definitely not call it a "break" though since old clients
will work perfectly fine with it.  Sorry for the bad wording.

> 
> >> > Clients shouldn't care about how/where a command is
> >> > being queued or not. If they send a command, they want it processed as
> >> > quickly as possible. However, it can be interesting to know if the
> >> > implementation of the command will be able to deliver oob, so that
> >> > data in the introspection could be useful.
> >> >
> >> > I would rather propose a client/server capability in qmp_capabilities,
> >> > call it "oob":
> >> >
> >> > This capability indicates oob commands support.
> >>
> >> The problem is indicating which commands support oob as opposed to
> >> indicating whether oob is present at all.  Future versions will
> >> probably make more commands oob-able and a client will want to know
> >> whether it can rely on a particular command being non-blocking.
> >
> > Yes.
> >
> > And IMHO we don't urgently need that "whether the server globally
> > supports oob" thing.  Client can just know that from query-qmp-schema
> > already - there will always be the "allow-oob" new field for command
> > typed entries.  IMHO that's a solid hint.
> >
> > But I don't object to return it as well in qmp_capabilities.
> 
> Does it feel right that the client can specify how the command are
> processed / queued ? Isn't it preferable to leave that to the server
> to decide? Why would a client specify that? And should the server be
> expected to behave differently? What the client needs to be able is to
> match the unordered replies, and that can be stated during cap
> negotiation / qmp_capabilties. The server is expected to do a best
> effort to handle commands and their priorities. If the client needs
> several command queue, it is simpler to open several connection rather
> than trying to fit that weird priority logic in the protocol imho.

Sorry I may have missed the point here.  We were discussing about a
global hint for "oob" support, am I right?  Then, could I ask what's
the "weird priority logic" you mentioned?

> 
> >
> >>
> >> > An oob command is a regular client message request with the "id"
> >> > member mandatory, but the reply may be delivered
> >> > out of order by the server if the client supports
> >> > it too.
> >> >
> >> > If both the server and the client have the "oob" capability, the
> >> > server can handle new client requests while previous requests are being
> >> > processed.
> >> >
> >> > If the client doesn't have the "oob" capability, it may still call
> >> > an oob command, and make multiple outstanding calls. In this case,
> >> > the commands are processed in order, so the replies will also be in
> >> > order. The "id" member isn't mandatory in this case.
> >> >
> >> > The client should match the replies with the "id" member associated
> >> > with the requests.
> >> >
> >> > When a client is disconnected, the pending commands are not
> >> > necessarily cancelled. But the future clients will not get replies from
> >> > commands they didn't make (they might, however, receive side-effects
> >> > events).
> >>
> >> What's the behaviour on the current monitor?
> >
> > Yeah I want to ask the same question, along with questioning about
> > above [1].
> >
> > IMHO this series will not change the behaviors of these, so IMHO the
> > behaviors will be the same before/after this series. E.g., when client
> > dropped right after the command is executed, I think we will still
> > execute the command, though we should encounter something odd in
> > monitor_json_emitter() somewhere when we want to respond.  And it will
> > happen the same after this series.
> 
> I think it can get worse after your series, because you queue the
> commands, so clearly a new client can get replies from an old client
> commands. As said above, I am not convinced you can get in that
> situation with current code.

Hmm, seems so.  But would this a big problem?

I really think the new client should just throw that response away if
it does not really know that response (from peeking at "id" field),
just like my opinion above.

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Fri, Sep 15, 2017 at 04:17:07PM +0100, Dr. David Alan Gilbert wrote:
> * Stefan Hajnoczi (stefanha@redhat.com) wrote:
> > On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:
> > > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:
> > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > > > > > > There should be a limit in the number of requests the thread can
> > > > > > > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > > > > > > 
> > > > > > > > > > I agree.
> > > > > > > > > > 
> > > > > > > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > > > > > > than the amount of memory consumed by them.
> > > > > > > > > > 
> > > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > > > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > > > > > > processed if queued non-OOB commands consume too much memory.
> > > > > > > > > 
> > > > > > > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > > > > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > > > > > > Control" section of that long reply).  Hopefully this series basically
> > > > > > > > > can work from design prospective, then I'll add this flow control in
> > > > > > > > > next version.
> > > > > > > > > 
> > > > > > > > > Regarding to what we should do if the limit is reached: Markus
> > > > > > > > > provided a few options, but the one I prefer most is that we don't
> > > > > > > > > respond, but send an event showing that a command is dropped.
> > > > > > > > > However, I would like it not queued, but a direct reply (after all,
> > > > > > > > > it's an event, and we should not need to care much on ordering of it).
> > > > > > > > > Then we can get rid of the babysitting of those "to be failed"
> > > > > > > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > > > > > > 
> > > > > > > > > I think I also missed at least a unit test for this new interface.
> > > > > > > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > > > > > > 
> > > > > > > > Another solution: the server reports available receive buffer space to
> > > > > > > > the client.  The server only guarantees immediate OOB processing when
> > > > > > > > the client stays within the receive buffer size.
> > > > > > > > 
> > > > > > > > Clients wishing to take advantage of OOB must query the receive buffer
> > > > > > > > size and make sure to leave enough room.
> > > > > > > 
> > > > > > > I don't think having to query it ahead of time is particularly nice,
> > > > > > > and of course it is inherantly racy.
> > > > > > > 
> > > > > > > I would just have QEMU emit an event when it pausing processing of the
> > > > > > > incoming commands due to a full queue.  If the event includes the ID
> > > > > > > of the last queued command, the client will know which (if any) of
> > > > > > > its outstanding commands are delayed. Another even can be sent when
> > > > > > > it restarts reading.
> > > > > > 
> > > > > > Hmm and now we're implementing flow control!
> > > > > > 
> > > > > > a) What exactly is the current semantics/buffer sizes?
> > > > > > b) When do clients send multiple QMP commands on one channel without
> > > > > > waiting for the response to the previous command?
> > > > > > c) Would one queue entry for each class of commands/channel work
> > > > > >   (Where a class of commands is currently 'normal' and 'oob')
> > > > > 
> > > > > I do wonder if we need to worry about request limiting at all from the
> > > > > client side.  For non-OOB commands clients will wait for a reply before
> > > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.
> > > > > 
> > > > > OOB commands are supposed to be things which can be handled quickly
> > > > > without blocking, so even if a client sent several commands at once
> > > > > without waiting for replies, they're going to be processed quickly,
> > > > > so whether we temporarily block reading off the wire is a minor
> > > > > detail.
> > > > 
> > > > Lets just define it so that it can't - you send an OOB command and wait
> > > > for it's response before sending another on that channel.
> > > > 
> > > > > IOW, I think we could just have a fixed 10 command queue and apps just
> > > > > pretend that there's an infinite queue and nothing bad would happen from
> > > > > the app's POV.
> > > > 
> > > > Can you justify 10 as opposed to 1?
> > > 
> > > Semantically I don't think it makes a difference if the OOB commands are
> > > being processed sequentially by their thread. A >1 length queue would only
> > > matter for non-OOB commands if an app was filling the pipeline with non-OOB
> > > requests, as then that could block reading of OOB commands. 
> > 
> > To summarize:
> > 
> > The QMP server has a lookahead of 1 command so it can dispatch
> > out-of-band commands.  If 2 or more non-OOB commands are queued at the
> > same time then OOB processing will not occur.
> > 
> > Is that right?
> 
> I think my view is slightly more complex;
>   a) There's a pair of queues for each channel
>   b) There's a central pair of queues on the QMP server
>     one for OOB commands and one for normal commands.
>   c) Each queue is only really guaranteed to be one deep.
> 
>   That means that each one of the channels can send a non-OOB
> command without getting in the way of a channel that wants
> to send one.

But current version should not be that complex:

Firstly, parser thread will only be enabled for QMP+NO_MIXED monitors.

Then, we only have a single global queue for QMP non-oob commands, and
we don't have response queue yet.  We do respond just like before in a
synchronous way (I explained why - for OOB we don't need that
complexity IMHO).

When we parse commands, we execute it directly if OOB, otherwise we
put it onto request queue.  Request queue handling is done by a main
thread QEMUBH.  That's all.

Would this "simple version" suffice to implement this whole OOB idea?

(Again, I really don't think we need to specify queue length to 1,
 though we can make it small)

-- 
Peter Xu

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Marc-André Lureau]

Hi

On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu <peterx@redhat.com> wrote:
> On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:
>> Hi
>>
>> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu <peterx@redhat.com> wrote:
>> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
>> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
>> >> > Hi
>> >> >
>> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
>> >> > > This series was born from this one:
>> >> > >
>> >> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
>> >> > >
>> >> > > The design comes from Markus, and also the whole-bunch-of discussions
>> >> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
>> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny
>> >> > > ideas and suggestions.  Finally we got such a solution that seems to
>> >> > > satisfy everyone.
>> >> > >
>> >> > > I re-started the versioning since this series is totally different
>> >> > > from previous one.  Now it's version 1.
>> >> > >
>> >> > > In case new reviewers come along the way without reading previous
>> >> > > discussions, I will try to do a summary on what this is all about.
>> >> > >
>> >> > > What is OOB execution?
>> >> > > ======================
>> >> > >
>> >> > > It's the shortcut of Out-Of-Band execution, its name is given by
>> >> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
>> >> > > QMP is going throw these steps:
>> >> > >
>> >> > >       JSON Parser --> QMP Dispatcher --> Respond
>> >> > >           /|\    (2)                (3)     |
>> >> > >        (1) |                               \|/ (4)
>> >> > >            +---------  main thread  --------+
>> >> > >
>> >> > > The requests are executed by the so-called QMP-dispatcher after the
>> >> > > JSON is parsed.  If OOB is on, we run the command directly in the
>> >> > > parser and quickly returns.
>> >> >
>> >> > All commands should have the "id" field mandatory in this case, else
>> >> > the client will not distinguish the replies coming from the last/oob
>> >> > and the previous commands.
>> >> >
>> >> > This should probably be enforced upfront by client capability checks,
>> >> > more below.
>> >
>> > Hmm yes since the oob commands are actually running in async way,
>> > request ID should be needed here.  However I'm not sure whether
>> > enabling the whole "request ID" thing is too big for this "try to be
>> > small" oob change... And IMHO it suites better to be part of the whole
>> > async work (no matter which implementation we'll use).
>> >
>> > How about this: we make "id" mandatory for "run-oob" requests only.
>> > For oob commands, they will always have ID then no ordering issue, and
>> > we can do it async; for the rest of non-oob commands, we still allow
>> > them to go without ID, and since they are not oob, they'll always be
>> > done in order as well.  Would this work?
>>
>> This mixed-mode is imho more complicated to deal with than having the
>> protocol enforced one way or the other, but that should work.
>>
>> >
>> >> >
>> >> > > Yeah I know in current code the parser calls dispatcher directly
>> >> > > (please see handle_qmp_command()).  However it's not true again after
>> >> > > this series (parser will has its own IO thread, and dispatcher will
>> >> > > still be run in main thread).  So this OOB does brings something
>> >> > > different.
>> >> > >
>> >> > > There are more details on why OOB and the difference/relationship
>> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
>> >> > > slightly out of topic (and believe me, it's not easy for me to
>> >> > > summarize that).  For more information, please refers to [1].
>> >> > >
>> >> > > Summary ends here.
>> >> > >
>> >> > > Some Implementation Details
>> >> > > ===========================
>> >> > >
>> >> > > Again, I mentioned that the old QMP workflow is this:
>> >> > >
>> >> > >       JSON Parser --> QMP Dispatcher --> Respond
>> >> > >           /|\    (2)                (3)     |
>> >> > >        (1) |                               \|/ (4)
>> >> > >            +---------  main thread  --------+
>> >> > >
>> >> > > What this series does is, firstly:
>> >> > >
>> >> > >       JSON Parser     QMP Dispatcher --> Respond
>> >> > >           /|\ |           /|\       (4)     |
>> >> > >            |  | (2)        | (3)            |  (5)
>> >> > >        (1) |  +----->      |               \|/
>> >> > >            +---------  main thread  <-------+
>> >> > >
>> >> > > And further:
>> >> > >
>> >> > >                queue/kick
>> >> > >      JSON Parser ======> QMP Dispatcher --> Respond
>> >> > >          /|\ |     (3)       /|\        (4)    |
>> >> > >       (1) |  | (2)            |                |  (5)
>> >> > >           | \|/               |               \|/
>> >> > >         IO thread         main thread  <-------+
>> >> >
>> >> > Is the queue per monitor or per client?
>> >
>> > The queue is currently global. I think yes maybe at least we can do it
>> > per monitor, but I am not sure whether that is urgent or can be
>> > postponed.  After all now QMPRequest (please refer to patch 11) is
>> > defined as (mon, id, req) tuple, so at least "id" namespace is
>> > per-monitor.
>> >
>> >> > And is the dispatching going
>> >> > to be processed even if the client is disconnected, and are new
>> >> > clients going to receive the replies from previous clients
>> >> > commands?
>> >
>> > [1]
>> >
>> > (will discuss together below)
>> >
>> >> > I
>> >> > believe there should be a per-client context, so there won't be "id"
>> >> > request conflicts.
>> >
>> > I'd say I am not familiar with this "client" idea, since after all
>> > IMHO one monitor is currently designed to mostly work with a single
>> > client. Say, unix sockets, telnet, all these backends are only single
>> > channeled, and one monitor instance can only work with one client at a
>> > time.  Then do we really need to add this client layer upon it?  IMHO
>> > the user can just provide more monitors if they wants more clients
>> > (and at least these clients should know the existance of the others or
>> > there might be problem, otherwise user2 will fail a migration, finally
>> > noticed that user1 has already triggered one), and the user should
>> > manage them well.
>>
>> qemu should support a management layer / libvirt restart/reconnect.
>> Afaik, it mostly work today. There might be a cases where libvirt can
>> be confused if it receives a reply from a previous connection command,
>> but due to the sync processing of the chardev, I am not sure you can
>> get in this situation.  By adding "oob" commands and queuing, the
>> client will have to remember which was the last "id" used, or it will
>> create more conflict after a reconnect.
>>
>> Imho we should introduce the client/connection concept to avoid this
>> confusion (unexpected reply & per client id space).
>
> Hmm I agree that the reconnect feature would be nice, but if so IMHO
> instead of throwing responses away when client disconnect, we should
> really keep them, and when the client reconnects, we queue the
> responses again.
>
> I think we have other quite simple ways to solve the "unexpected
> reply" and "per-client-id duplication" issues you have mentioned.
>
> Firstly, when client gets unexpected replies ("id" field not in its
> own request queue), the client should just ignore that reply, which
> seems natural to me.

The trouble is that it may legitimately use the same "id" value for
new requests. And I don't see a simple way to handle that without
races.

>
> Then, if client disconnected and reconnected, it should not have the
> problem to generate duplicated id for request, since it should know
> what requests it has sent already.  A simplest case I can think of is,
> the ID should contains the following tuple:

If you assume the "same" client will recover its state, yes.

>
>   (client name, client unique ID, request ID)
>
> Here "client name" can be something like "libvirt", which is the name
> of client application;
>
> "client unique ID" can be anything generated when client starts, it
> identifies a single client session, maybe a UUID.
>
> "request ID" can be a unsigned integer starts from zero, and increases
> each time the client sends one request.

This is introducing  session handling, and can be done in server side
only without changes in the protocol I believe.

>
> I believe current libvirt is using "client name" + "request ID".  It's
> something similar (after all I think we don't normally have >1 libvirt
> to manage single QEMU, so I think it should be good enough).

I am not sure we should base our protocol usage assumptions based on
libvirt only, but rather on what is possible today (like queuing
requests in the socket etc..).

> Then even if client disconnect and reconnect, request ID won't lose,
> and no duplication would happen IMHO.
>
>>
>> >
>> >> >
>> >> > >
>> >> > > Then it introduced the "allow-oob" parameter in QAPI schema to define
>> >> > > commands, and "run-oob" flag to let oob-allowed command to run in the
>> >> > > parser.
>> >> >
>> >> > From a protocol point of view, I find that "run-oob" distinction per
>> >> > command a bit pointless. It helps with legacy client that wouldn't
>> >> > expect out-of-order replies if qemu were to run oob commands oob by
>> >> > default though.
>> >
>> > After all oob somehow breaks existing rules or sync execution.  I
>> > thought the more important goal was at least to keep the legacy
>> > behaviors when adding new things, no?
>>
>> Of course we have to keep compatibily. What do you mean by "oob
>> somehow breaks existing rules or sync execution"? oob means queuing
>> and unordered reply support, so clearly this is breaking the current
>> "mostly ordered" behaviour (mostly because events may still come any
>> time..., and the reconnect issue discussed above).
>
> Yes.  That's what I mean, it breaks the synchronous scemantic.  But
> I should definitely not call it a "break" though since old clients
> will work perfectly fine with it.  Sorry for the bad wording.
>
>>
>> >> > Clients shouldn't care about how/where a command is
>> >> > being queued or not. If they send a command, they want it processed as
>> >> > quickly as possible. However, it can be interesting to know if the
>> >> > implementation of the command will be able to deliver oob, so that
>> >> > data in the introspection could be useful.
>> >> >
>> >> > I would rather propose a client/server capability in qmp_capabilities,
>> >> > call it "oob":
>> >> >
>> >> > This capability indicates oob commands support.
>> >>
>> >> The problem is indicating which commands support oob as opposed to
>> >> indicating whether oob is present at all.  Future versions will
>> >> probably make more commands oob-able and a client will want to know
>> >> whether it can rely on a particular command being non-blocking.
>> >
>> > Yes.
>> >
>> > And IMHO we don't urgently need that "whether the server globally
>> > supports oob" thing.  Client can just know that from query-qmp-schema
>> > already - there will always be the "allow-oob" new field for command
>> > typed entries.  IMHO that's a solid hint.
>> >
>> > But I don't object to return it as well in qmp_capabilities.
>>
>> Does it feel right that the client can specify how the command are
>> processed / queued ? Isn't it preferable to leave that to the server
>> to decide? Why would a client specify that? And should the server be
>> expected to behave differently? What the client needs to be able is to
>> match the unordered replies, and that can be stated during cap
>> negotiation / qmp_capabilties. The server is expected to do a best
>> effort to handle commands and their priorities. If the client needs
>> several command queue, it is simpler to open several connection rather
>> than trying to fit that weird priority logic in the protocol imho.
>
> Sorry I may have missed the point here.  We were discussing about a
> global hint for "oob" support, am I right?  Then, could I ask what's
> the "weird priority logic" you mentioned?

I call per-message oob hint a kind of priority logic, since you can
make the same request without oob in the same session and in parallel.

>>
>> >
>> >>
>> >> > An oob command is a regular client message request with the "id"
>> >> > member mandatory, but the reply may be delivered
>> >> > out of order by the server if the client supports
>> >> > it too.
>> >> >
>> >> > If both the server and the client have the "oob" capability, the
>> >> > server can handle new client requests while previous requests are being
>> >> > processed.
>> >> >
>> >> > If the client doesn't have the "oob" capability, it may still call
>> >> > an oob command, and make multiple outstanding calls. In this case,
>> >> > the commands are processed in order, so the replies will also be in
>> >> > order. The "id" member isn't mandatory in this case.
>> >> >
>> >> > The client should match the replies with the "id" member associated
>> >> > with the requests.
>> >> >
>> >> > When a client is disconnected, the pending commands are not
>> >> > necessarily cancelled. But the future clients will not get replies from
>> >> > commands they didn't make (they might, however, receive side-effects
>> >> > events).
>> >>
>> >> What's the behaviour on the current monitor?
>> >
>> > Yeah I want to ask the same question, along with questioning about
>> > above [1].
>> >
>> > IMHO this series will not change the behaviors of these, so IMHO the
>> > behaviors will be the same before/after this series. E.g., when client
>> > dropped right after the command is executed, I think we will still
>> > execute the command, though we should encounter something odd in
>> > monitor_json_emitter() somewhere when we want to respond.  And it will
>> > happen the same after this series.
>>
>> I think it can get worse after your series, because you queue the
>> commands, so clearly a new client can get replies from an old client
>> commands. As said above, I am not convinced you can get in that
>> situation with current code.
>
> Hmm, seems so.  But would this a big problem?
>
> I really think the new client should just throw that response away if
> it does not really know that response (from peeking at "id" field),
> just like my opinion above.

This is a high expectation.


-- 
Marc-André Lureau

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Peter Xu (peterx@redhat.com) wrote:
> On Fri, Sep 15, 2017 at 04:17:07PM +0100, Dr. David Alan Gilbert wrote:
> > * Stefan Hajnoczi (stefanha@redhat.com) wrote:
> > > On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:
> > > > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:
> > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> > > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > > > > > > > There should be a limit in the number of requests the thread can
> > > > > > > > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > > > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > > > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > > > > > > > 
> > > > > > > > > > > I agree.
> > > > > > > > > > > 
> > > > > > > > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > > > > > > > than the amount of memory consumed by them.
> > > > > > > > > > > 
> > > > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > > > > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > > > > > > > processed if queued non-OOB commands consume too much memory.
> > > > > > > > > > 
> > > > > > > > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > > > > > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > > > > > > > Control" section of that long reply).  Hopefully this series basically
> > > > > > > > > > can work from design prospective, then I'll add this flow control in
> > > > > > > > > > next version.
> > > > > > > > > > 
> > > > > > > > > > Regarding to what we should do if the limit is reached: Markus
> > > > > > > > > > provided a few options, but the one I prefer most is that we don't
> > > > > > > > > > respond, but send an event showing that a command is dropped.
> > > > > > > > > > However, I would like it not queued, but a direct reply (after all,
> > > > > > > > > > it's an event, and we should not need to care much on ordering of it).
> > > > > > > > > > Then we can get rid of the babysitting of those "to be failed"
> > > > > > > > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > > > > > > > 
> > > > > > > > > > I think I also missed at least a unit test for this new interface.
> > > > > > > > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > > > > > > > 
> > > > > > > > > Another solution: the server reports available receive buffer space to
> > > > > > > > > the client.  The server only guarantees immediate OOB processing when
> > > > > > > > > the client stays within the receive buffer size.
> > > > > > > > > 
> > > > > > > > > Clients wishing to take advantage of OOB must query the receive buffer
> > > > > > > > > size and make sure to leave enough room.
> > > > > > > > 
> > > > > > > > I don't think having to query it ahead of time is particularly nice,
> > > > > > > > and of course it is inherantly racy.
> > > > > > > > 
> > > > > > > > I would just have QEMU emit an event when it pausing processing of the
> > > > > > > > incoming commands due to a full queue.  If the event includes the ID
> > > > > > > > of the last queued command, the client will know which (if any) of
> > > > > > > > its outstanding commands are delayed. Another even can be sent when
> > > > > > > > it restarts reading.
> > > > > > > 
> > > > > > > Hmm and now we're implementing flow control!
> > > > > > > 
> > > > > > > a) What exactly is the current semantics/buffer sizes?
> > > > > > > b) When do clients send multiple QMP commands on one channel without
> > > > > > > waiting for the response to the previous command?
> > > > > > > c) Would one queue entry for each class of commands/channel work
> > > > > > >   (Where a class of commands is currently 'normal' and 'oob')
> > > > > > 
> > > > > > I do wonder if we need to worry about request limiting at all from the
> > > > > > client side.  For non-OOB commands clients will wait for a reply before
> > > > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.
> > > > > > 
> > > > > > OOB commands are supposed to be things which can be handled quickly
> > > > > > without blocking, so even if a client sent several commands at once
> > > > > > without waiting for replies, they're going to be processed quickly,
> > > > > > so whether we temporarily block reading off the wire is a minor
> > > > > > detail.
> > > > > 
> > > > > Lets just define it so that it can't - you send an OOB command and wait
> > > > > for it's response before sending another on that channel.
> > > > > 
> > > > > > IOW, I think we could just have a fixed 10 command queue and apps just
> > > > > > pretend that there's an infinite queue and nothing bad would happen from
> > > > > > the app's POV.
> > > > > 
> > > > > Can you justify 10 as opposed to 1?
> > > > 
> > > > Semantically I don't think it makes a difference if the OOB commands are
> > > > being processed sequentially by their thread. A >1 length queue would only
> > > > matter for non-OOB commands if an app was filling the pipeline with non-OOB
> > > > requests, as then that could block reading of OOB commands. 
> > > 
> > > To summarize:
> > > 
> > > The QMP server has a lookahead of 1 command so it can dispatch
> > > out-of-band commands.  If 2 or more non-OOB commands are queued at the
> > > same time then OOB processing will not occur.
> > > 
> > > Is that right?
> > 
> > I think my view is slightly more complex;
> >   a) There's a pair of queues for each channel
> >   b) There's a central pair of queues on the QMP server
> >     one for OOB commands and one for normal commands.
> >   c) Each queue is only really guaranteed to be one deep.
> > 
> >   That means that each one of the channels can send a non-OOB
> > command without getting in the way of a channel that wants
> > to send one.
> 
> But current version should not be that complex:
> 
> Firstly, parser thread will only be enabled for QMP+NO_MIXED monitors.
> 
> Then, we only have a single global queue for QMP non-oob commands, and
> we don't have response queue yet.  We do respond just like before in a
> synchronous way (I explained why - for OOB we don't need that
> complexity IMHO).

I think  the discussion started because of two related comments:
  Marc-André said :
     'There should be a limit in the number of requests the thread can
queue'
and Stefan said :
     'Memory usage must be bounded.'

actually neither of those cases really worried me (because they only
happen if the client keeps pumping commands, and that seems it's fault).

However, once you start adding a limit, you've got to be careful - if
you just added a limit to the central queue, then what happens if that
queue is filled by non-OOB commands?

Dave

> When we parse commands, we execute it directly if OOB, otherwise we
> put it onto request queue.  Request queue handling is done by a main
> thread QEMUBH.  That's all.
> 
> Would this "simple version" suffice to implement this whole OOB idea?
> 
> (Again, I really don't think we need to specify queue length to 1,
>  though we can make it small)
> 
> -- 
> Peter Xu
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> Hi
> 
> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu <peterx@redhat.com> wrote:
> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:
> >> Hi
> >>
> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu <peterx@redhat.com> wrote:
> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> >> >> > Hi
> >> >> >
> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
> >> >> > > This series was born from this one:
> >> >> > >
> >> >> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> >> >> > >
> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions
> >> >> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny
> >> >> > > ideas and suggestions.  Finally we got such a solution that seems to
> >> >> > > satisfy everyone.
> >> >> > >
> >> >> > > I re-started the versioning since this series is totally different
> >> >> > > from previous one.  Now it's version 1.
> >> >> > >
> >> >> > > In case new reviewers come along the way without reading previous
> >> >> > > discussions, I will try to do a summary on what this is all about.
> >> >> > >
> >> >> > > What is OOB execution?
> >> >> > > ======================
> >> >> > >
> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by
> >> >> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
> >> >> > > QMP is going throw these steps:
> >> >> > >
> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> >> >> > >           /|\    (2)                (3)     |
> >> >> > >        (1) |                               \|/ (4)
> >> >> > >            +---------  main thread  --------+
> >> >> > >
> >> >> > > The requests are executed by the so-called QMP-dispatcher after the
> >> >> > > JSON is parsed.  If OOB is on, we run the command directly in the
> >> >> > > parser and quickly returns.
> >> >> >
> >> >> > All commands should have the "id" field mandatory in this case, else
> >> >> > the client will not distinguish the replies coming from the last/oob
> >> >> > and the previous commands.
> >> >> >
> >> >> > This should probably be enforced upfront by client capability checks,
> >> >> > more below.
> >> >
> >> > Hmm yes since the oob commands are actually running in async way,
> >> > request ID should be needed here.  However I'm not sure whether
> >> > enabling the whole "request ID" thing is too big for this "try to be
> >> > small" oob change... And IMHO it suites better to be part of the whole
> >> > async work (no matter which implementation we'll use).
> >> >
> >> > How about this: we make "id" mandatory for "run-oob" requests only.
> >> > For oob commands, they will always have ID then no ordering issue, and
> >> > we can do it async; for the rest of non-oob commands, we still allow
> >> > them to go without ID, and since they are not oob, they'll always be
> >> > done in order as well.  Would this work?
> >>
> >> This mixed-mode is imho more complicated to deal with than having the
> >> protocol enforced one way or the other, but that should work.
> >>
> >> >
> >> >> >
> >> >> > > Yeah I know in current code the parser calls dispatcher directly
> >> >> > > (please see handle_qmp_command()).  However it's not true again after
> >> >> > > this series (parser will has its own IO thread, and dispatcher will
> >> >> > > still be run in main thread).  So this OOB does brings something
> >> >> > > different.
> >> >> > >
> >> >> > > There are more details on why OOB and the difference/relationship
> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
> >> >> > > slightly out of topic (and believe me, it's not easy for me to
> >> >> > > summarize that).  For more information, please refers to [1].
> >> >> > >
> >> >> > > Summary ends here.
> >> >> > >
> >> >> > > Some Implementation Details
> >> >> > > ===========================
> >> >> > >
> >> >> > > Again, I mentioned that the old QMP workflow is this:
> >> >> > >
> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> >> >> > >           /|\    (2)                (3)     |
> >> >> > >        (1) |                               \|/ (4)
> >> >> > >            +---------  main thread  --------+
> >> >> > >
> >> >> > > What this series does is, firstly:
> >> >> > >
> >> >> > >       JSON Parser     QMP Dispatcher --> Respond
> >> >> > >           /|\ |           /|\       (4)     |
> >> >> > >            |  | (2)        | (3)            |  (5)
> >> >> > >        (1) |  +----->      |               \|/
> >> >> > >            +---------  main thread  <-------+
> >> >> > >
> >> >> > > And further:
> >> >> > >
> >> >> > >                queue/kick
> >> >> > >      JSON Parser ======> QMP Dispatcher --> Respond
> >> >> > >          /|\ |     (3)       /|\        (4)    |
> >> >> > >       (1) |  | (2)            |                |  (5)
> >> >> > >           | \|/               |               \|/
> >> >> > >         IO thread         main thread  <-------+
> >> >> >
> >> >> > Is the queue per monitor or per client?
> >> >
> >> > The queue is currently global. I think yes maybe at least we can do it
> >> > per monitor, but I am not sure whether that is urgent or can be
> >> > postponed.  After all now QMPRequest (please refer to patch 11) is
> >> > defined as (mon, id, req) tuple, so at least "id" namespace is
> >> > per-monitor.
> >> >
> >> >> > And is the dispatching going
> >> >> > to be processed even if the client is disconnected, and are new
> >> >> > clients going to receive the replies from previous clients
> >> >> > commands?
> >> >
> >> > [1]
> >> >
> >> > (will discuss together below)
> >> >
> >> >> > I
> >> >> > believe there should be a per-client context, so there won't be "id"
> >> >> > request conflicts.
> >> >
> >> > I'd say I am not familiar with this "client" idea, since after all
> >> > IMHO one monitor is currently designed to mostly work with a single
> >> > client. Say, unix sockets, telnet, all these backends are only single
> >> > channeled, and one monitor instance can only work with one client at a
> >> > time.  Then do we really need to add this client layer upon it?  IMHO
> >> > the user can just provide more monitors if they wants more clients
> >> > (and at least these clients should know the existance of the others or
> >> > there might be problem, otherwise user2 will fail a migration, finally
> >> > noticed that user1 has already triggered one), and the user should
> >> > manage them well.
> >>
> >> qemu should support a management layer / libvirt restart/reconnect.
> >> Afaik, it mostly work today. There might be a cases where libvirt can
> >> be confused if it receives a reply from a previous connection command,
> >> but due to the sync processing of the chardev, I am not sure you can
> >> get in this situation.  By adding "oob" commands and queuing, the
> >> client will have to remember which was the last "id" used, or it will
> >> create more conflict after a reconnect.
> >>
> >> Imho we should introduce the client/connection concept to avoid this
> >> confusion (unexpected reply & per client id space).
> >
> > Hmm I agree that the reconnect feature would be nice, but if so IMHO
> > instead of throwing responses away when client disconnect, we should
> > really keep them, and when the client reconnects, we queue the
> > responses again.
> >
> > I think we have other quite simple ways to solve the "unexpected
> > reply" and "per-client-id duplication" issues you have mentioned.
> >
> > Firstly, when client gets unexpected replies ("id" field not in its
> > own request queue), the client should just ignore that reply, which
> > seems natural to me.
> 
> The trouble is that it may legitimately use the same "id" value for
> new requests. And I don't see a simple way to handle that without
> races.

Under what circumstances can it reuse the same ID for new requests?
Can't we simply tell it not to?

Dave

> >
> > Then, if client disconnected and reconnected, it should not have the
> > problem to generate duplicated id for request, since it should know
> > what requests it has sent already.  A simplest case I can think of is,
> > the ID should contains the following tuple:
> 
> If you assume the "same" client will recover its state, yes.
> 
> >
> >   (client name, client unique ID, request ID)
> >
> > Here "client name" can be something like "libvirt", which is the name
> > of client application;
> >
> > "client unique ID" can be anything generated when client starts, it
> > identifies a single client session, maybe a UUID.
> >
> > "request ID" can be a unsigned integer starts from zero, and increases
> > each time the client sends one request.
> 
> This is introducing  session handling, and can be done in server side
> only without changes in the protocol I believe.
> 
> >
> > I believe current libvirt is using "client name" + "request ID".  It's
> > something similar (after all I think we don't normally have >1 libvirt
> > to manage single QEMU, so I think it should be good enough).
> 
> I am not sure we should base our protocol usage assumptions based on
> libvirt only, but rather on what is possible today (like queuing
> requests in the socket etc..).
> 
> > Then even if client disconnect and reconnect, request ID won't lose,
> > and no duplication would happen IMHO.
> >
> >>
> >> >
> >> >> >
> >> >> > >
> >> >> > > Then it introduced the "allow-oob" parameter in QAPI schema to define
> >> >> > > commands, and "run-oob" flag to let oob-allowed command to run in the
> >> >> > > parser.
> >> >> >
> >> >> > From a protocol point of view, I find that "run-oob" distinction per
> >> >> > command a bit pointless. It helps with legacy client that wouldn't
> >> >> > expect out-of-order replies if qemu were to run oob commands oob by
> >> >> > default though.
> >> >
> >> > After all oob somehow breaks existing rules or sync execution.  I
> >> > thought the more important goal was at least to keep the legacy
> >> > behaviors when adding new things, no?
> >>
> >> Of course we have to keep compatibily. What do you mean by "oob
> >> somehow breaks existing rules or sync execution"? oob means queuing
> >> and unordered reply support, so clearly this is breaking the current
> >> "mostly ordered" behaviour (mostly because events may still come any
> >> time..., and the reconnect issue discussed above).
> >
> > Yes.  That's what I mean, it breaks the synchronous scemantic.  But
> > I should definitely not call it a "break" though since old clients
> > will work perfectly fine with it.  Sorry for the bad wording.
> >
> >>
> >> >> > Clients shouldn't care about how/where a command is
> >> >> > being queued or not. If they send a command, they want it processed as
> >> >> > quickly as possible. However, it can be interesting to know if the
> >> >> > implementation of the command will be able to deliver oob, so that
> >> >> > data in the introspection could be useful.
> >> >> >
> >> >> > I would rather propose a client/server capability in qmp_capabilities,
> >> >> > call it "oob":
> >> >> >
> >> >> > This capability indicates oob commands support.
> >> >>
> >> >> The problem is indicating which commands support oob as opposed to
> >> >> indicating whether oob is present at all.  Future versions will
> >> >> probably make more commands oob-able and a client will want to know
> >> >> whether it can rely on a particular command being non-blocking.
> >> >
> >> > Yes.
> >> >
> >> > And IMHO we don't urgently need that "whether the server globally
> >> > supports oob" thing.  Client can just know that from query-qmp-schema
> >> > already - there will always be the "allow-oob" new field for command
> >> > typed entries.  IMHO that's a solid hint.
> >> >
> >> > But I don't object to return it as well in qmp_capabilities.
> >>
> >> Does it feel right that the client can specify how the command are
> >> processed / queued ? Isn't it preferable to leave that to the server
> >> to decide? Why would a client specify that? And should the server be
> >> expected to behave differently? What the client needs to be able is to
> >> match the unordered replies, and that can be stated during cap
> >> negotiation / qmp_capabilties. The server is expected to do a best
> >> effort to handle commands and their priorities. If the client needs
> >> several command queue, it is simpler to open several connection rather
> >> than trying to fit that weird priority logic in the protocol imho.
> >
> > Sorry I may have missed the point here.  We were discussing about a
> > global hint for "oob" support, am I right?  Then, could I ask what's
> > the "weird priority logic" you mentioned?
> 
> I call per-message oob hint a kind of priority logic, since you can
> make the same request without oob in the same session and in parallel.
> 
> >>
> >> >
> >> >>
> >> >> > An oob command is a regular client message request with the "id"
> >> >> > member mandatory, but the reply may be delivered
> >> >> > out of order by the server if the client supports
> >> >> > it too.
> >> >> >
> >> >> > If both the server and the client have the "oob" capability, the
> >> >> > server can handle new client requests while previous requests are being
> >> >> > processed.
> >> >> >
> >> >> > If the client doesn't have the "oob" capability, it may still call
> >> >> > an oob command, and make multiple outstanding calls. In this case,
> >> >> > the commands are processed in order, so the replies will also be in
> >> >> > order. The "id" member isn't mandatory in this case.
> >> >> >
> >> >> > The client should match the replies with the "id" member associated
> >> >> > with the requests.
> >> >> >
> >> >> > When a client is disconnected, the pending commands are not
> >> >> > necessarily cancelled. But the future clients will not get replies from
> >> >> > commands they didn't make (they might, however, receive side-effects
> >> >> > events).
> >> >>
> >> >> What's the behaviour on the current monitor?
> >> >
> >> > Yeah I want to ask the same question, along with questioning about
> >> > above [1].
> >> >
> >> > IMHO this series will not change the behaviors of these, so IMHO the
> >> > behaviors will be the same before/after this series. E.g., when client
> >> > dropped right after the command is executed, I think we will still
> >> > execute the command, though we should encounter something odd in
> >> > monitor_json_emitter() somewhere when we want to respond.  And it will
> >> > happen the same after this series.
> >>
> >> I think it can get worse after your series, because you queue the
> >> commands, so clearly a new client can get replies from an old client
> >> commands. As said above, I am not convinced you can get in that
> >> situation with current code.
> >
> > Hmm, seems so.  But would this a big problem?
> >
> > I really think the new client should just throw that response away if
> > it does not really know that response (from peeking at "id" field),
> > just like my opinion above.
> 
> This is a high expectation.
> 
> 
> -- 
> Marc-André Lureau
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Marc-André Lureau]

Hi

On Mon, Sep 18, 2017 at 12:55 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
>> Hi
>>
>> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu <peterx@redhat.com> wrote:
>> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:
>> >> Hi
>> >>
>> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu <peterx@redhat.com> wrote:
>> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
>> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
>> >> >> > Hi
>> >> >> >
>> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
>> >> >> > > This series was born from this one:
>> >> >> > >
>> >> >> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
>> >> >> > >
>> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions
>> >> >> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
>> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny
>> >> >> > > ideas and suggestions.  Finally we got such a solution that seems to
>> >> >> > > satisfy everyone.
>> >> >> > >
>> >> >> > > I re-started the versioning since this series is totally different
>> >> >> > > from previous one.  Now it's version 1.
>> >> >> > >
>> >> >> > > In case new reviewers come along the way without reading previous
>> >> >> > > discussions, I will try to do a summary on what this is all about.
>> >> >> > >
>> >> >> > > What is OOB execution?
>> >> >> > > ======================
>> >> >> > >
>> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by
>> >> >> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
>> >> >> > > QMP is going throw these steps:
>> >> >> > >
>> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
>> >> >> > >           /|\    (2)                (3)     |
>> >> >> > >        (1) |                               \|/ (4)
>> >> >> > >            +---------  main thread  --------+
>> >> >> > >
>> >> >> > > The requests are executed by the so-called QMP-dispatcher after the
>> >> >> > > JSON is parsed.  If OOB is on, we run the command directly in the
>> >> >> > > parser and quickly returns.
>> >> >> >
>> >> >> > All commands should have the "id" field mandatory in this case, else
>> >> >> > the client will not distinguish the replies coming from the last/oob
>> >> >> > and the previous commands.
>> >> >> >
>> >> >> > This should probably be enforced upfront by client capability checks,
>> >> >> > more below.
>> >> >
>> >> > Hmm yes since the oob commands are actually running in async way,
>> >> > request ID should be needed here.  However I'm not sure whether
>> >> > enabling the whole "request ID" thing is too big for this "try to be
>> >> > small" oob change... And IMHO it suites better to be part of the whole
>> >> > async work (no matter which implementation we'll use).
>> >> >
>> >> > How about this: we make "id" mandatory for "run-oob" requests only.
>> >> > For oob commands, they will always have ID then no ordering issue, and
>> >> > we can do it async; for the rest of non-oob commands, we still allow
>> >> > them to go without ID, and since they are not oob, they'll always be
>> >> > done in order as well.  Would this work?
>> >>
>> >> This mixed-mode is imho more complicated to deal with than having the
>> >> protocol enforced one way or the other, but that should work.
>> >>
>> >> >
>> >> >> >
>> >> >> > > Yeah I know in current code the parser calls dispatcher directly
>> >> >> > > (please see handle_qmp_command()).  However it's not true again after
>> >> >> > > this series (parser will has its own IO thread, and dispatcher will
>> >> >> > > still be run in main thread).  So this OOB does brings something
>> >> >> > > different.
>> >> >> > >
>> >> >> > > There are more details on why OOB and the difference/relationship
>> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
>> >> >> > > slightly out of topic (and believe me, it's not easy for me to
>> >> >> > > summarize that).  For more information, please refers to [1].
>> >> >> > >
>> >> >> > > Summary ends here.
>> >> >> > >
>> >> >> > > Some Implementation Details
>> >> >> > > ===========================
>> >> >> > >
>> >> >> > > Again, I mentioned that the old QMP workflow is this:
>> >> >> > >
>> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
>> >> >> > >           /|\    (2)                (3)     |
>> >> >> > >        (1) |                               \|/ (4)
>> >> >> > >            +---------  main thread  --------+
>> >> >> > >
>> >> >> > > What this series does is, firstly:
>> >> >> > >
>> >> >> > >       JSON Parser     QMP Dispatcher --> Respond
>> >> >> > >           /|\ |           /|\       (4)     |
>> >> >> > >            |  | (2)        | (3)            |  (5)
>> >> >> > >        (1) |  +----->      |               \|/
>> >> >> > >            +---------  main thread  <-------+
>> >> >> > >
>> >> >> > > And further:
>> >> >> > >
>> >> >> > >                queue/kick
>> >> >> > >      JSON Parser ======> QMP Dispatcher --> Respond
>> >> >> > >          /|\ |     (3)       /|\        (4)    |
>> >> >> > >       (1) |  | (2)            |                |  (5)
>> >> >> > >           | \|/               |               \|/
>> >> >> > >         IO thread         main thread  <-------+
>> >> >> >
>> >> >> > Is the queue per monitor or per client?
>> >> >
>> >> > The queue is currently global. I think yes maybe at least we can do it
>> >> > per monitor, but I am not sure whether that is urgent or can be
>> >> > postponed.  After all now QMPRequest (please refer to patch 11) is
>> >> > defined as (mon, id, req) tuple, so at least "id" namespace is
>> >> > per-monitor.
>> >> >
>> >> >> > And is the dispatching going
>> >> >> > to be processed even if the client is disconnected, and are new
>> >> >> > clients going to receive the replies from previous clients
>> >> >> > commands?
>> >> >
>> >> > [1]
>> >> >
>> >> > (will discuss together below)
>> >> >
>> >> >> > I
>> >> >> > believe there should be a per-client context, so there won't be "id"
>> >> >> > request conflicts.
>> >> >
>> >> > I'd say I am not familiar with this "client" idea, since after all
>> >> > IMHO one monitor is currently designed to mostly work with a single
>> >> > client. Say, unix sockets, telnet, all these backends are only single
>> >> > channeled, and one monitor instance can only work with one client at a
>> >> > time.  Then do we really need to add this client layer upon it?  IMHO
>> >> > the user can just provide more monitors if they wants more clients
>> >> > (and at least these clients should know the existance of the others or
>> >> > there might be problem, otherwise user2 will fail a migration, finally
>> >> > noticed that user1 has already triggered one), and the user should
>> >> > manage them well.
>> >>
>> >> qemu should support a management layer / libvirt restart/reconnect.
>> >> Afaik, it mostly work today. There might be a cases where libvirt can
>> >> be confused if it receives a reply from a previous connection command,
>> >> but due to the sync processing of the chardev, I am not sure you can
>> >> get in this situation.  By adding "oob" commands and queuing, the
>> >> client will have to remember which was the last "id" used, or it will
>> >> create more conflict after a reconnect.
>> >>
>> >> Imho we should introduce the client/connection concept to avoid this
>> >> confusion (unexpected reply & per client id space).
>> >
>> > Hmm I agree that the reconnect feature would be nice, but if so IMHO
>> > instead of throwing responses away when client disconnect, we should
>> > really keep them, and when the client reconnects, we queue the
>> > responses again.
>> >
>> > I think we have other quite simple ways to solve the "unexpected
>> > reply" and "per-client-id duplication" issues you have mentioned.
>> >
>> > Firstly, when client gets unexpected replies ("id" field not in its
>> > own request queue), the client should just ignore that reply, which
>> > seems natural to me.
>>
>> The trouble is that it may legitimately use the same "id" value for
>> new requests. And I don't see a simple way to handle that without
>> races.
>
> Under what circumstances can it reuse the same ID for new requests?
> Can't we simply tell it not to?

I don't see any restriction today in the protocol in connecting with a
new client that may not know anything from a previous client.

How would you tell it not to use old IDs? Just by writing an unwritten
rule, because we don't want to fix the per connection client session
handling in qemu?

>
> Dave
>
>> >
>> > Then, if client disconnected and reconnected, it should not have the
>> > problem to generate duplicated id for request, since it should know
>> > what requests it has sent already.  A simplest case I can think of is,
>> > the ID should contains the following tuple:
>>
>> If you assume the "same" client will recover its state, yes.
>>
>> >
>> >   (client name, client unique ID, request ID)
>> >
>> > Here "client name" can be something like "libvirt", which is the name
>> > of client application;
>> >
>> > "client unique ID" can be anything generated when client starts, it
>> > identifies a single client session, maybe a UUID.
>> >
>> > "request ID" can be a unsigned integer starts from zero, and increases
>> > each time the client sends one request.
>>
>> This is introducing  session handling, and can be done in server side
>> only without changes in the protocol I believe.
>>
>> >
>> > I believe current libvirt is using "client name" + "request ID".  It's
>> > something similar (after all I think we don't normally have >1 libvirt
>> > to manage single QEMU, so I think it should be good enough).
>>
>> I am not sure we should base our protocol usage assumptions based on
>> libvirt only, but rather on what is possible today (like queuing
>> requests in the socket etc..).
>>
>> > Then even if client disconnect and reconnect, request ID won't lose,
>> > and no duplication would happen IMHO.
>> >
>> >>
>> >> >
>> >> >> >
>> >> >> > >
>> >> >> > > Then it introduced the "allow-oob" parameter in QAPI schema to define
>> >> >> > > commands, and "run-oob" flag to let oob-allowed command to run in the
>> >> >> > > parser.
>> >> >> >
>> >> >> > From a protocol point of view, I find that "run-oob" distinction per
>> >> >> > command a bit pointless. It helps with legacy client that wouldn't
>> >> >> > expect out-of-order replies if qemu were to run oob commands oob by
>> >> >> > default though.
>> >> >
>> >> > After all oob somehow breaks existing rules or sync execution.  I
>> >> > thought the more important goal was at least to keep the legacy
>> >> > behaviors when adding new things, no?
>> >>
>> >> Of course we have to keep compatibily. What do you mean by "oob
>> >> somehow breaks existing rules or sync execution"? oob means queuing
>> >> and unordered reply support, so clearly this is breaking the current
>> >> "mostly ordered" behaviour (mostly because events may still come any
>> >> time..., and the reconnect issue discussed above).
>> >
>> > Yes.  That's what I mean, it breaks the synchronous scemantic.  But
>> > I should definitely not call it a "break" though since old clients
>> > will work perfectly fine with it.  Sorry for the bad wording.
>> >
>> >>
>> >> >> > Clients shouldn't care about how/where a command is
>> >> >> > being queued or not. If they send a command, they want it processed as
>> >> >> > quickly as possible. However, it can be interesting to know if the
>> >> >> > implementation of the command will be able to deliver oob, so that
>> >> >> > data in the introspection could be useful.
>> >> >> >
>> >> >> > I would rather propose a client/server capability in qmp_capabilities,
>> >> >> > call it "oob":
>> >> >> >
>> >> >> > This capability indicates oob commands support.
>> >> >>
>> >> >> The problem is indicating which commands support oob as opposed to
>> >> >> indicating whether oob is present at all.  Future versions will
>> >> >> probably make more commands oob-able and a client will want to know
>> >> >> whether it can rely on a particular command being non-blocking.
>> >> >
>> >> > Yes.
>> >> >
>> >> > And IMHO we don't urgently need that "whether the server globally
>> >> > supports oob" thing.  Client can just know that from query-qmp-schema
>> >> > already - there will always be the "allow-oob" new field for command
>> >> > typed entries.  IMHO that's a solid hint.
>> >> >
>> >> > But I don't object to return it as well in qmp_capabilities.
>> >>
>> >> Does it feel right that the client can specify how the command are
>> >> processed / queued ? Isn't it preferable to leave that to the server
>> >> to decide? Why would a client specify that? And should the server be
>> >> expected to behave differently? What the client needs to be able is to
>> >> match the unordered replies, and that can be stated during cap
>> >> negotiation / qmp_capabilties. The server is expected to do a best
>> >> effort to handle commands and their priorities. If the client needs
>> >> several command queue, it is simpler to open several connection rather
>> >> than trying to fit that weird priority logic in the protocol imho.
>> >
>> > Sorry I may have missed the point here.  We were discussing about a
>> > global hint for "oob" support, am I right?  Then, could I ask what's
>> > the "weird priority logic" you mentioned?
>>
>> I call per-message oob hint a kind of priority logic, since you can
>> make the same request without oob in the same session and in parallel.
>>
>> >>
>> >> >
>> >> >>
>> >> >> > An oob command is a regular client message request with the "id"
>> >> >> > member mandatory, but the reply may be delivered
>> >> >> > out of order by the server if the client supports
>> >> >> > it too.
>> >> >> >
>> >> >> > If both the server and the client have the "oob" capability, the
>> >> >> > server can handle new client requests while previous requests are being
>> >> >> > processed.
>> >> >> >
>> >> >> > If the client doesn't have the "oob" capability, it may still call
>> >> >> > an oob command, and make multiple outstanding calls. In this case,
>> >> >> > the commands are processed in order, so the replies will also be in
>> >> >> > order. The "id" member isn't mandatory in this case.
>> >> >> >
>> >> >> > The client should match the replies with the "id" member associated
>> >> >> > with the requests.
>> >> >> >
>> >> >> > When a client is disconnected, the pending commands are not
>> >> >> > necessarily cancelled. But the future clients will not get replies from
>> >> >> > commands they didn't make (they might, however, receive side-effects
>> >> >> > events).
>> >> >>
>> >> >> What's the behaviour on the current monitor?
>> >> >
>> >> > Yeah I want to ask the same question, along with questioning about
>> >> > above [1].
>> >> >
>> >> > IMHO this series will not change the behaviors of these, so IMHO the
>> >> > behaviors will be the same before/after this series. E.g., when client
>> >> > dropped right after the command is executed, I think we will still
>> >> > execute the command, though we should encounter something odd in
>> >> > monitor_json_emitter() somewhere when we want to respond.  And it will
>> >> > happen the same after this series.
>> >>
>> >> I think it can get worse after your series, because you queue the
>> >> commands, so clearly a new client can get replies from an old client
>> >> commands. As said above, I am not convinced you can get in that
>> >> situation with current code.
>> >
>> > Hmm, seems so.  But would this a big problem?
>> >
>> > I really think the new client should just throw that response away if
>> > it does not really know that response (from peeking at "id" field),
>> > just like my opinion above.
>>
>> This is a high expectation.
>>
>>
>> --
>> Marc-André Lureau
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



-- 
Marc-André Lureau

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> Hi
> 
> On Mon, Sep 18, 2017 at 12:55 PM, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> >> Hi
> >>
> >> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu <peterx@redhat.com> wrote:
> >> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:
> >> >> Hi
> >> >>
> >> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu <peterx@redhat.com> wrote:
> >> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
> >> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> >> >> >> > Hi
> >> >> >> >
> >> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
> >> >> >> > > This series was born from this one:
> >> >> >> > >
> >> >> >> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> >> >> >> > >
> >> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions
> >> >> >> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
> >> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny
> >> >> >> > > ideas and suggestions.  Finally we got such a solution that seems to
> >> >> >> > > satisfy everyone.
> >> >> >> > >
> >> >> >> > > I re-started the versioning since this series is totally different
> >> >> >> > > from previous one.  Now it's version 1.
> >> >> >> > >
> >> >> >> > > In case new reviewers come along the way without reading previous
> >> >> >> > > discussions, I will try to do a summary on what this is all about.
> >> >> >> > >
> >> >> >> > > What is OOB execution?
> >> >> >> > > ======================
> >> >> >> > >
> >> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by
> >> >> >> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
> >> >> >> > > QMP is going throw these steps:
> >> >> >> > >
> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> >> >> >> > >           /|\    (2)                (3)     |
> >> >> >> > >        (1) |                               \|/ (4)
> >> >> >> > >            +---------  main thread  --------+
> >> >> >> > >
> >> >> >> > > The requests are executed by the so-called QMP-dispatcher after the
> >> >> >> > > JSON is parsed.  If OOB is on, we run the command directly in the
> >> >> >> > > parser and quickly returns.
> >> >> >> >
> >> >> >> > All commands should have the "id" field mandatory in this case, else
> >> >> >> > the client will not distinguish the replies coming from the last/oob
> >> >> >> > and the previous commands.
> >> >> >> >
> >> >> >> > This should probably be enforced upfront by client capability checks,
> >> >> >> > more below.
> >> >> >
> >> >> > Hmm yes since the oob commands are actually running in async way,
> >> >> > request ID should be needed here.  However I'm not sure whether
> >> >> > enabling the whole "request ID" thing is too big for this "try to be
> >> >> > small" oob change... And IMHO it suites better to be part of the whole
> >> >> > async work (no matter which implementation we'll use).
> >> >> >
> >> >> > How about this: we make "id" mandatory for "run-oob" requests only.
> >> >> > For oob commands, they will always have ID then no ordering issue, and
> >> >> > we can do it async; for the rest of non-oob commands, we still allow
> >> >> > them to go without ID, and since they are not oob, they'll always be
> >> >> > done in order as well.  Would this work?
> >> >>
> >> >> This mixed-mode is imho more complicated to deal with than having the
> >> >> protocol enforced one way or the other, but that should work.
> >> >>
> >> >> >
> >> >> >> >
> >> >> >> > > Yeah I know in current code the parser calls dispatcher directly
> >> >> >> > > (please see handle_qmp_command()).  However it's not true again after
> >> >> >> > > this series (parser will has its own IO thread, and dispatcher will
> >> >> >> > > still be run in main thread).  So this OOB does brings something
> >> >> >> > > different.
> >> >> >> > >
> >> >> >> > > There are more details on why OOB and the difference/relationship
> >> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
> >> >> >> > > slightly out of topic (and believe me, it's not easy for me to
> >> >> >> > > summarize that).  For more information, please refers to [1].
> >> >> >> > >
> >> >> >> > > Summary ends here.
> >> >> >> > >
> >> >> >> > > Some Implementation Details
> >> >> >> > > ===========================
> >> >> >> > >
> >> >> >> > > Again, I mentioned that the old QMP workflow is this:
> >> >> >> > >
> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> >> >> >> > >           /|\    (2)                (3)     |
> >> >> >> > >        (1) |                               \|/ (4)
> >> >> >> > >            +---------  main thread  --------+
> >> >> >> > >
> >> >> >> > > What this series does is, firstly:
> >> >> >> > >
> >> >> >> > >       JSON Parser     QMP Dispatcher --> Respond
> >> >> >> > >           /|\ |           /|\       (4)     |
> >> >> >> > >            |  | (2)        | (3)            |  (5)
> >> >> >> > >        (1) |  +----->      |               \|/
> >> >> >> > >            +---------  main thread  <-------+
> >> >> >> > >
> >> >> >> > > And further:
> >> >> >> > >
> >> >> >> > >                queue/kick
> >> >> >> > >      JSON Parser ======> QMP Dispatcher --> Respond
> >> >> >> > >          /|\ |     (3)       /|\        (4)    |
> >> >> >> > >       (1) |  | (2)            |                |  (5)
> >> >> >> > >           | \|/               |               \|/
> >> >> >> > >         IO thread         main thread  <-------+
> >> >> >> >
> >> >> >> > Is the queue per monitor or per client?
> >> >> >
> >> >> > The queue is currently global. I think yes maybe at least we can do it
> >> >> > per monitor, but I am not sure whether that is urgent or can be
> >> >> > postponed.  After all now QMPRequest (please refer to patch 11) is
> >> >> > defined as (mon, id, req) tuple, so at least "id" namespace is
> >> >> > per-monitor.
> >> >> >
> >> >> >> > And is the dispatching going
> >> >> >> > to be processed even if the client is disconnected, and are new
> >> >> >> > clients going to receive the replies from previous clients
> >> >> >> > commands?
> >> >> >
> >> >> > [1]
> >> >> >
> >> >> > (will discuss together below)
> >> >> >
> >> >> >> > I
> >> >> >> > believe there should be a per-client context, so there won't be "id"
> >> >> >> > request conflicts.
> >> >> >
> >> >> > I'd say I am not familiar with this "client" idea, since after all
> >> >> > IMHO one monitor is currently designed to mostly work with a single
> >> >> > client. Say, unix sockets, telnet, all these backends are only single
> >> >> > channeled, and one monitor instance can only work with one client at a
> >> >> > time.  Then do we really need to add this client layer upon it?  IMHO
> >> >> > the user can just provide more monitors if they wants more clients
> >> >> > (and at least these clients should know the existance of the others or
> >> >> > there might be problem, otherwise user2 will fail a migration, finally
> >> >> > noticed that user1 has already triggered one), and the user should
> >> >> > manage them well.
> >> >>
> >> >> qemu should support a management layer / libvirt restart/reconnect.
> >> >> Afaik, it mostly work today. There might be a cases where libvirt can
> >> >> be confused if it receives a reply from a previous connection command,
> >> >> but due to the sync processing of the chardev, I am not sure you can
> >> >> get in this situation.  By adding "oob" commands and queuing, the
> >> >> client will have to remember which was the last "id" used, or it will
> >> >> create more conflict after a reconnect.
> >> >>
> >> >> Imho we should introduce the client/connection concept to avoid this
> >> >> confusion (unexpected reply & per client id space).
> >> >
> >> > Hmm I agree that the reconnect feature would be nice, but if so IMHO
> >> > instead of throwing responses away when client disconnect, we should
> >> > really keep them, and when the client reconnects, we queue the
> >> > responses again.
> >> >
> >> > I think we have other quite simple ways to solve the "unexpected
> >> > reply" and "per-client-id duplication" issues you have mentioned.
> >> >
> >> > Firstly, when client gets unexpected replies ("id" field not in its
> >> > own request queue), the client should just ignore that reply, which
> >> > seems natural to me.
> >>
> >> The trouble is that it may legitimately use the same "id" value for
> >> new requests. And I don't see a simple way to handle that without
> >> races.
> >
> > Under what circumstances can it reuse the same ID for new requests?
> > Can't we simply tell it not to?
> 
> I don't see any restriction today in the protocol in connecting with a
> new client that may not know anything from a previous client.

Well, it knows it's doing a reconnection.

> How would you tell it not to use old IDs? Just by writing an unwritten
> rule, because we don't want to fix the per connection client session
> handling in qemu?

BY writing a written rule!  This out of order stuff we're adding here
is a change to the interface and we can define what we require of the
client.  As long as what we expect is reasonable then we might end
up with something that's simpler for both the client and qemu.
And I worry this series keeps getting more and more complex for weird
edge cases.

Dave

> >
> > Dave
> >
> >> >
> >> > Then, if client disconnected and reconnected, it should not have the
> >> > problem to generate duplicated id for request, since it should know
> >> > what requests it has sent already.  A simplest case I can think of is,
> >> > the ID should contains the following tuple:
> >>
> >> If you assume the "same" client will recover its state, yes.
> >>
> >> >
> >> >   (client name, client unique ID, request ID)
> >> >
> >> > Here "client name" can be something like "libvirt", which is the name
> >> > of client application;
> >> >
> >> > "client unique ID" can be anything generated when client starts, it
> >> > identifies a single client session, maybe a UUID.
> >> >
> >> > "request ID" can be a unsigned integer starts from zero, and increases
> >> > each time the client sends one request.
> >>
> >> This is introducing  session handling, and can be done in server side
> >> only without changes in the protocol I believe.
> >>
> >> >
> >> > I believe current libvirt is using "client name" + "request ID".  It's
> >> > something similar (after all I think we don't normally have >1 libvirt
> >> > to manage single QEMU, so I think it should be good enough).
> >>
> >> I am not sure we should base our protocol usage assumptions based on
> >> libvirt only, but rather on what is possible today (like queuing
> >> requests in the socket etc..).
> >>
> >> > Then even if client disconnect and reconnect, request ID won't lose,
> >> > and no duplication would happen IMHO.
> >> >
> >> >>
> >> >> >
> >> >> >> >
> >> >> >> > >
> >> >> >> > > Then it introduced the "allow-oob" parameter in QAPI schema to define
> >> >> >> > > commands, and "run-oob" flag to let oob-allowed command to run in the
> >> >> >> > > parser.
> >> >> >> >
> >> >> >> > From a protocol point of view, I find that "run-oob" distinction per
> >> >> >> > command a bit pointless. It helps with legacy client that wouldn't
> >> >> >> > expect out-of-order replies if qemu were to run oob commands oob by
> >> >> >> > default though.
> >> >> >
> >> >> > After all oob somehow breaks existing rules or sync execution.  I
> >> >> > thought the more important goal was at least to keep the legacy
> >> >> > behaviors when adding new things, no?
> >> >>
> >> >> Of course we have to keep compatibily. What do you mean by "oob
> >> >> somehow breaks existing rules or sync execution"? oob means queuing
> >> >> and unordered reply support, so clearly this is breaking the current
> >> >> "mostly ordered" behaviour (mostly because events may still come any
> >> >> time..., and the reconnect issue discussed above).
> >> >
> >> > Yes.  That's what I mean, it breaks the synchronous scemantic.  But
> >> > I should definitely not call it a "break" though since old clients
> >> > will work perfectly fine with it.  Sorry for the bad wording.
> >> >
> >> >>
> >> >> >> > Clients shouldn't care about how/where a command is
> >> >> >> > being queued or not. If they send a command, they want it processed as
> >> >> >> > quickly as possible. However, it can be interesting to know if the
> >> >> >> > implementation of the command will be able to deliver oob, so that
> >> >> >> > data in the introspection could be useful.
> >> >> >> >
> >> >> >> > I would rather propose a client/server capability in qmp_capabilities,
> >> >> >> > call it "oob":
> >> >> >> >
> >> >> >> > This capability indicates oob commands support.
> >> >> >>
> >> >> >> The problem is indicating which commands support oob as opposed to
> >> >> >> indicating whether oob is present at all.  Future versions will
> >> >> >> probably make more commands oob-able and a client will want to know
> >> >> >> whether it can rely on a particular command being non-blocking.
> >> >> >
> >> >> > Yes.
> >> >> >
> >> >> > And IMHO we don't urgently need that "whether the server globally
> >> >> > supports oob" thing.  Client can just know that from query-qmp-schema
> >> >> > already - there will always be the "allow-oob" new field for command
> >> >> > typed entries.  IMHO that's a solid hint.
> >> >> >
> >> >> > But I don't object to return it as well in qmp_capabilities.
> >> >>
> >> >> Does it feel right that the client can specify how the command are
> >> >> processed / queued ? Isn't it preferable to leave that to the server
> >> >> to decide? Why would a client specify that? And should the server be
> >> >> expected to behave differently? What the client needs to be able is to
> >> >> match the unordered replies, and that can be stated during cap
> >> >> negotiation / qmp_capabilties. The server is expected to do a best
> >> >> effort to handle commands and their priorities. If the client needs
> >> >> several command queue, it is simpler to open several connection rather
> >> >> than trying to fit that weird priority logic in the protocol imho.
> >> >
> >> > Sorry I may have missed the point here.  We were discussing about a
> >> > global hint for "oob" support, am I right?  Then, could I ask what's
> >> > the "weird priority logic" you mentioned?
> >>
> >> I call per-message oob hint a kind of priority logic, since you can
> >> make the same request without oob in the same session and in parallel.
> >>
> >> >>
> >> >> >
> >> >> >>
> >> >> >> > An oob command is a regular client message request with the "id"
> >> >> >> > member mandatory, but the reply may be delivered
> >> >> >> > out of order by the server if the client supports
> >> >> >> > it too.
> >> >> >> >
> >> >> >> > If both the server and the client have the "oob" capability, the
> >> >> >> > server can handle new client requests while previous requests are being
> >> >> >> > processed.
> >> >> >> >
> >> >> >> > If the client doesn't have the "oob" capability, it may still call
> >> >> >> > an oob command, and make multiple outstanding calls. In this case,
> >> >> >> > the commands are processed in order, so the replies will also be in
> >> >> >> > order. The "id" member isn't mandatory in this case.
> >> >> >> >
> >> >> >> > The client should match the replies with the "id" member associated
> >> >> >> > with the requests.
> >> >> >> >
> >> >> >> > When a client is disconnected, the pending commands are not
> >> >> >> > necessarily cancelled. But the future clients will not get replies from
> >> >> >> > commands they didn't make (they might, however, receive side-effects
> >> >> >> > events).
> >> >> >>
> >> >> >> What's the behaviour on the current monitor?
> >> >> >
> >> >> > Yeah I want to ask the same question, along with questioning about
> >> >> > above [1].
> >> >> >
> >> >> > IMHO this series will not change the behaviors of these, so IMHO the
> >> >> > behaviors will be the same before/after this series. E.g., when client
> >> >> > dropped right after the command is executed, I think we will still
> >> >> > execute the command, though we should encounter something odd in
> >> >> > monitor_json_emitter() somewhere when we want to respond.  And it will
> >> >> > happen the same after this series.
> >> >>
> >> >> I think it can get worse after your series, because you queue the
> >> >> commands, so clearly a new client can get replies from an old client
> >> >> commands. As said above, I am not convinced you can get in that
> >> >> situation with current code.
> >> >
> >> > Hmm, seems so.  But would this a big problem?
> >> >
> >> > I really think the new client should just throw that response away if
> >> > it does not really know that response (from peeking at "id" field),
> >> > just like my opinion above.
> >>
> >> This is a high expectation.
> >>
> >>
> >> --
> >> Marc-André Lureau
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
> 
> 
> -- 
> Marc-André Lureau
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3274 bytes --]

On 09/18/2017 05:55 AM, Dr. David Alan Gilbert wrote:

>>> I think we have other quite simple ways to solve the "unexpected
>>> reply" and "per-client-id duplication" issues you have mentioned.
>>>
>>> Firstly, when client gets unexpected replies ("id" field not in its
>>> own request queue), the client should just ignore that reply, which
>>> seems natural to me.

That's probably reasonable, if we document it.

>>
>> The trouble is that it may legitimately use the same "id" value for
>> new requests. And I don't see a simple way to handle that without
>> races.
> 
> Under what circumstances can it reuse the same ID for new requests?

Libvirt uses distinct id's for every message on a single connection, but
there is always the possibility that it will use id 'libvirt-0' on one
connection, then restart libvirtd, then use id 'libvirt-0' on the second
connection (there's nothing that I saw in libvirt code that saves the
current 'mon->nextSerial' value in XML to survive libvirtd restarts).

> Can't we simply tell it not to?

Since use of OOB handling will require client opt-in, yes, we can make
part of the opt-in process be a contract that the client has to do a
better job of avoiding duplicate id's across reconnects, if we think
that is easier to maintain.

>>>
>>> Then, if client disconnected and reconnected, it should not have the
>>> problem to generate duplicated id for request, since it should know
>>> what requests it has sent already.  A simplest case I can think of is,
>>> the ID should contains the following tuple:
>>
>> If you assume the "same" client will recover its state, yes.
>>
>>>
>>>   (client name, client unique ID, request ID)
>>>
>>> Here "client name" can be something like "libvirt", which is the name
>>> of client application;
>>>
>>> "client unique ID" can be anything generated when client starts, it
>>> identifies a single client session, maybe a UUID.
>>>
>>> "request ID" can be a unsigned integer starts from zero, and increases
>>> each time the client sends one request.
>>
>> This is introducing  session handling, and can be done in server side
>> only without changes in the protocol I believe.

The 'id' field can be _any_ JSON object - libvirt currently sends a
string, but could just as easily send a dict, and then libvirt could
supply whatever it wanted in the dict, including uuids, to ensure that
future reconnects don't reuse the id of a previous connection.  But
right now, 'id' is opaque to qemu (and should stay that way) - if qemu
is going to do any work to ensure that it doesn't send a stale reply to
a new connection, then that has to be tracked externally from whatever
'id' is passed in; or we just document that clients wanting to use OOB
handling have to be careful of their choice of 'id' (and leave it up to
the client to avoid confusion, because qemu doesn't care about it).

I'm also fine with requiring that clients that opt-in to OOB handling be
documented as having to ignore unknown 'id' responses, since we already
document that clients must be able to ignore unknown 'event' messages.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Marc-André Lureau]

On Mon, Sep 18, 2017 at 1:26 PM, Dr. David Alan Gilbert
<dgilbert@redhat.com> wrote:
> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
>> Hi
>>
>> On Mon, Sep 18, 2017 at 12:55 PM, Dr. David Alan Gilbert
>> <dgilbert@redhat.com> wrote:
>> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
>> >> Hi
>> >>
>> >> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu <peterx@redhat.com> wrote:
>> >> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:
>> >> >> Hi
>> >> >>
>> >> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu <peterx@redhat.com> wrote:
>> >> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
>> >> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
>> >> >> >> > Hi
>> >> >> >> >
>> >> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
>> >> >> >> > > This series was born from this one:
>> >> >> >> > >
>> >> >> >> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
>> >> >> >> > >
>> >> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions
>> >> >> >> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
>> >> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny
>> >> >> >> > > ideas and suggestions.  Finally we got such a solution that seems to
>> >> >> >> > > satisfy everyone.
>> >> >> >> > >
>> >> >> >> > > I re-started the versioning since this series is totally different
>> >> >> >> > > from previous one.  Now it's version 1.
>> >> >> >> > >
>> >> >> >> > > In case new reviewers come along the way without reading previous
>> >> >> >> > > discussions, I will try to do a summary on what this is all about.
>> >> >> >> > >
>> >> >> >> > > What is OOB execution?
>> >> >> >> > > ======================
>> >> >> >> > >
>> >> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by
>> >> >> >> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
>> >> >> >> > > QMP is going throw these steps:
>> >> >> >> > >
>> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
>> >> >> >> > >           /|\    (2)                (3)     |
>> >> >> >> > >        (1) |                               \|/ (4)
>> >> >> >> > >            +---------  main thread  --------+
>> >> >> >> > >
>> >> >> >> > > The requests are executed by the so-called QMP-dispatcher after the
>> >> >> >> > > JSON is parsed.  If OOB is on, we run the command directly in the
>> >> >> >> > > parser and quickly returns.
>> >> >> >> >
>> >> >> >> > All commands should have the "id" field mandatory in this case, else
>> >> >> >> > the client will not distinguish the replies coming from the last/oob
>> >> >> >> > and the previous commands.
>> >> >> >> >
>> >> >> >> > This should probably be enforced upfront by client capability checks,
>> >> >> >> > more below.
>> >> >> >
>> >> >> > Hmm yes since the oob commands are actually running in async way,
>> >> >> > request ID should be needed here.  However I'm not sure whether
>> >> >> > enabling the whole "request ID" thing is too big for this "try to be
>> >> >> > small" oob change... And IMHO it suites better to be part of the whole
>> >> >> > async work (no matter which implementation we'll use).
>> >> >> >
>> >> >> > How about this: we make "id" mandatory for "run-oob" requests only.
>> >> >> > For oob commands, they will always have ID then no ordering issue, and
>> >> >> > we can do it async; for the rest of non-oob commands, we still allow
>> >> >> > them to go without ID, and since they are not oob, they'll always be
>> >> >> > done in order as well.  Would this work?
>> >> >>
>> >> >> This mixed-mode is imho more complicated to deal with than having the
>> >> >> protocol enforced one way or the other, but that should work.
>> >> >>
>> >> >> >
>> >> >> >> >
>> >> >> >> > > Yeah I know in current code the parser calls dispatcher directly
>> >> >> >> > > (please see handle_qmp_command()).  However it's not true again after
>> >> >> >> > > this series (parser will has its own IO thread, and dispatcher will
>> >> >> >> > > still be run in main thread).  So this OOB does brings something
>> >> >> >> > > different.
>> >> >> >> > >
>> >> >> >> > > There are more details on why OOB and the difference/relationship
>> >> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
>> >> >> >> > > slightly out of topic (and believe me, it's not easy for me to
>> >> >> >> > > summarize that).  For more information, please refers to [1].
>> >> >> >> > >
>> >> >> >> > > Summary ends here.
>> >> >> >> > >
>> >> >> >> > > Some Implementation Details
>> >> >> >> > > ===========================
>> >> >> >> > >
>> >> >> >> > > Again, I mentioned that the old QMP workflow is this:
>> >> >> >> > >
>> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
>> >> >> >> > >           /|\    (2)                (3)     |
>> >> >> >> > >        (1) |                               \|/ (4)
>> >> >> >> > >            +---------  main thread  --------+
>> >> >> >> > >
>> >> >> >> > > What this series does is, firstly:
>> >> >> >> > >
>> >> >> >> > >       JSON Parser     QMP Dispatcher --> Respond
>> >> >> >> > >           /|\ |           /|\       (4)     |
>> >> >> >> > >            |  | (2)        | (3)            |  (5)
>> >> >> >> > >        (1) |  +----->      |               \|/
>> >> >> >> > >            +---------  main thread  <-------+
>> >> >> >> > >
>> >> >> >> > > And further:
>> >> >> >> > >
>> >> >> >> > >                queue/kick
>> >> >> >> > >      JSON Parser ======> QMP Dispatcher --> Respond
>> >> >> >> > >          /|\ |     (3)       /|\        (4)    |
>> >> >> >> > >       (1) |  | (2)            |                |  (5)
>> >> >> >> > >           | \|/               |               \|/
>> >> >> >> > >         IO thread         main thread  <-------+
>> >> >> >> >
>> >> >> >> > Is the queue per monitor or per client?
>> >> >> >
>> >> >> > The queue is currently global. I think yes maybe at least we can do it
>> >> >> > per monitor, but I am not sure whether that is urgent or can be
>> >> >> > postponed.  After all now QMPRequest (please refer to patch 11) is
>> >> >> > defined as (mon, id, req) tuple, so at least "id" namespace is
>> >> >> > per-monitor.
>> >> >> >
>> >> >> >> > And is the dispatching going
>> >> >> >> > to be processed even if the client is disconnected, and are new
>> >> >> >> > clients going to receive the replies from previous clients
>> >> >> >> > commands?
>> >> >> >
>> >> >> > [1]
>> >> >> >
>> >> >> > (will discuss together below)
>> >> >> >
>> >> >> >> > I
>> >> >> >> > believe there should be a per-client context, so there won't be "id"
>> >> >> >> > request conflicts.
>> >> >> >
>> >> >> > I'd say I am not familiar with this "client" idea, since after all
>> >> >> > IMHO one monitor is currently designed to mostly work with a single
>> >> >> > client. Say, unix sockets, telnet, all these backends are only single
>> >> >> > channeled, and one monitor instance can only work with one client at a
>> >> >> > time.  Then do we really need to add this client layer upon it?  IMHO
>> >> >> > the user can just provide more monitors if they wants more clients
>> >> >> > (and at least these clients should know the existance of the others or
>> >> >> > there might be problem, otherwise user2 will fail a migration, finally
>> >> >> > noticed that user1 has already triggered one), and the user should
>> >> >> > manage them well.
>> >> >>
>> >> >> qemu should support a management layer / libvirt restart/reconnect.
>> >> >> Afaik, it mostly work today. There might be a cases where libvirt can
>> >> >> be confused if it receives a reply from a previous connection command,
>> >> >> but due to the sync processing of the chardev, I am not sure you can
>> >> >> get in this situation.  By adding "oob" commands and queuing, the
>> >> >> client will have to remember which was the last "id" used, or it will
>> >> >> create more conflict after a reconnect.
>> >> >>
>> >> >> Imho we should introduce the client/connection concept to avoid this
>> >> >> confusion (unexpected reply & per client id space).
>> >> >
>> >> > Hmm I agree that the reconnect feature would be nice, but if so IMHO
>> >> > instead of throwing responses away when client disconnect, we should
>> >> > really keep them, and when the client reconnects, we queue the
>> >> > responses again.
>> >> >
>> >> > I think we have other quite simple ways to solve the "unexpected
>> >> > reply" and "per-client-id duplication" issues you have mentioned.
>> >> >
>> >> > Firstly, when client gets unexpected replies ("id" field not in its
>> >> > own request queue), the client should just ignore that reply, which
>> >> > seems natural to me.
>> >>
>> >> The trouble is that it may legitimately use the same "id" value for
>> >> new requests. And I don't see a simple way to handle that without
>> >> races.
>> >
>> > Under what circumstances can it reuse the same ID for new requests?
>> > Can't we simply tell it not to?
>>
>> I don't see any restriction today in the protocol in connecting with a
>> new client that may not know anything from a previous client.
>
> Well, it knows it's doing a reconnection.

If you assume the "same client" reconnects to the monitor, I agree.
But this is a restriction of monitor usage.

>
>> How would you tell it not to use old IDs? Just by writing an unwritten
>> rule, because we don't want to fix the per connection client session
>> handling in qemu?
>
> BY writing a written rule!  This out of order stuff we're adding here
> is a change to the interface and we can define what we require of the
> client.  As long as what we expect is reasonable then we might end
> up with something that's simpler for both the client and qemu.

As long as we don't break existing qmp clients.

> And I worry this series keeps getting more and more complex for weird
> edge cases.

That's an interesting point-of-view. I see the point in fixing weird
edge cases in qemu RPC code. More than other code we develop with
weird edge cases in mind & tests, like the parsing/checking of the
json schema for ex, in a similar area with the same maintainer.

> Dave
>
>> >
>> > Dave
>> >
>> >> >
>> >> > Then, if client disconnected and reconnected, it should not have the
>> >> > problem to generate duplicated id for request, since it should know
>> >> > what requests it has sent already.  A simplest case I can think of is,
>> >> > the ID should contains the following tuple:
>> >>
>> >> If you assume the "same" client will recover its state, yes.
>> >>
>> >> >
>> >> >   (client name, client unique ID, request ID)
>> >> >
>> >> > Here "client name" can be something like "libvirt", which is the name
>> >> > of client application;
>> >> >
>> >> > "client unique ID" can be anything generated when client starts, it
>> >> > identifies a single client session, maybe a UUID.
>> >> >
>> >> > "request ID" can be a unsigned integer starts from zero, and increases
>> >> > each time the client sends one request.
>> >>
>> >> This is introducing  session handling, and can be done in server side
>> >> only without changes in the protocol I believe.
>> >>
>> >> >
>> >> > I believe current libvirt is using "client name" + "request ID".  It's
>> >> > something similar (after all I think we don't normally have >1 libvirt
>> >> > to manage single QEMU, so I think it should be good enough).
>> >>
>> >> I am not sure we should base our protocol usage assumptions based on
>> >> libvirt only, but rather on what is possible today (like queuing
>> >> requests in the socket etc..).
>> >>
>> >> > Then even if client disconnect and reconnect, request ID won't lose,
>> >> > and no duplication would happen IMHO.
>> >> >
>> >> >>
>> >> >> >
>> >> >> >> >
>> >> >> >> > >
>> >> >> >> > > Then it introduced the "allow-oob" parameter in QAPI schema to define
>> >> >> >> > > commands, and "run-oob" flag to let oob-allowed command to run in the
>> >> >> >> > > parser.
>> >> >> >> >
>> >> >> >> > From a protocol point of view, I find that "run-oob" distinction per
>> >> >> >> > command a bit pointless. It helps with legacy client that wouldn't
>> >> >> >> > expect out-of-order replies if qemu were to run oob commands oob by
>> >> >> >> > default though.
>> >> >> >
>> >> >> > After all oob somehow breaks existing rules or sync execution.  I
>> >> >> > thought the more important goal was at least to keep the legacy
>> >> >> > behaviors when adding new things, no?
>> >> >>
>> >> >> Of course we have to keep compatibily. What do you mean by "oob
>> >> >> somehow breaks existing rules or sync execution"? oob means queuing
>> >> >> and unordered reply support, so clearly this is breaking the current
>> >> >> "mostly ordered" behaviour (mostly because events may still come any
>> >> >> time..., and the reconnect issue discussed above).
>> >> >
>> >> > Yes.  That's what I mean, it breaks the synchronous scemantic.  But
>> >> > I should definitely not call it a "break" though since old clients
>> >> > will work perfectly fine with it.  Sorry for the bad wording.
>> >> >
>> >> >>
>> >> >> >> > Clients shouldn't care about how/where a command is
>> >> >> >> > being queued or not. If they send a command, they want it processed as
>> >> >> >> > quickly as possible. However, it can be interesting to know if the
>> >> >> >> > implementation of the command will be able to deliver oob, so that
>> >> >> >> > data in the introspection could be useful.
>> >> >> >> >
>> >> >> >> > I would rather propose a client/server capability in qmp_capabilities,
>> >> >> >> > call it "oob":
>> >> >> >> >
>> >> >> >> > This capability indicates oob commands support.
>> >> >> >>
>> >> >> >> The problem is indicating which commands support oob as opposed to
>> >> >> >> indicating whether oob is present at all.  Future versions will
>> >> >> >> probably make more commands oob-able and a client will want to know
>> >> >> >> whether it can rely on a particular command being non-blocking.
>> >> >> >
>> >> >> > Yes.
>> >> >> >
>> >> >> > And IMHO we don't urgently need that "whether the server globally
>> >> >> > supports oob" thing.  Client can just know that from query-qmp-schema
>> >> >> > already - there will always be the "allow-oob" new field for command
>> >> >> > typed entries.  IMHO that's a solid hint.
>> >> >> >
>> >> >> > But I don't object to return it as well in qmp_capabilities.
>> >> >>
>> >> >> Does it feel right that the client can specify how the command are
>> >> >> processed / queued ? Isn't it preferable to leave that to the server
>> >> >> to decide? Why would a client specify that? And should the server be
>> >> >> expected to behave differently? What the client needs to be able is to
>> >> >> match the unordered replies, and that can be stated during cap
>> >> >> negotiation / qmp_capabilties. The server is expected to do a best
>> >> >> effort to handle commands and their priorities. If the client needs
>> >> >> several command queue, it is simpler to open several connection rather
>> >> >> than trying to fit that weird priority logic in the protocol imho.
>> >> >
>> >> > Sorry I may have missed the point here.  We were discussing about a
>> >> > global hint for "oob" support, am I right?  Then, could I ask what's
>> >> > the "weird priority logic" you mentioned?
>> >>
>> >> I call per-message oob hint a kind of priority logic, since you can
>> >> make the same request without oob in the same session and in parallel.
>> >>
>> >> >>
>> >> >> >
>> >> >> >>
>> >> >> >> > An oob command is a regular client message request with the "id"
>> >> >> >> > member mandatory, but the reply may be delivered
>> >> >> >> > out of order by the server if the client supports
>> >> >> >> > it too.
>> >> >> >> >
>> >> >> >> > If both the server and the client have the "oob" capability, the
>> >> >> >> > server can handle new client requests while previous requests are being
>> >> >> >> > processed.
>> >> >> >> >
>> >> >> >> > If the client doesn't have the "oob" capability, it may still call
>> >> >> >> > an oob command, and make multiple outstanding calls. In this case,
>> >> >> >> > the commands are processed in order, so the replies will also be in
>> >> >> >> > order. The "id" member isn't mandatory in this case.
>> >> >> >> >
>> >> >> >> > The client should match the replies with the "id" member associated
>> >> >> >> > with the requests.
>> >> >> >> >
>> >> >> >> > When a client is disconnected, the pending commands are not
>> >> >> >> > necessarily cancelled. But the future clients will not get replies from
>> >> >> >> > commands they didn't make (they might, however, receive side-effects
>> >> >> >> > events).
>> >> >> >>
>> >> >> >> What's the behaviour on the current monitor?
>> >> >> >
>> >> >> > Yeah I want to ask the same question, along with questioning about
>> >> >> > above [1].
>> >> >> >
>> >> >> > IMHO this series will not change the behaviors of these, so IMHO the
>> >> >> > behaviors will be the same before/after this series. E.g., when client
>> >> >> > dropped right after the command is executed, I think we will still
>> >> >> > execute the command, though we should encounter something odd in
>> >> >> > monitor_json_emitter() somewhere when we want to respond.  And it will
>> >> >> > happen the same after this series.
>> >> >>
>> >> >> I think it can get worse after your series, because you queue the
>> >> >> commands, so clearly a new client can get replies from an old client
>> >> >> commands. As said above, I am not convinced you can get in that
>> >> >> situation with current code.
>> >> >
>> >> > Hmm, seems so.  But would this a big problem?
>> >> >
>> >> > I really think the new client should just throw that response away if
>> >> > it does not really know that response (from peeking at "id" field),
>> >> > just like my opinion above.
>> >>
>> >> This is a high expectation.
>> >>
>> >>
>> >> --
>> >> Marc-André Lureau
>> > --
>> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>
>>
>>
>> --
>> Marc-André Lureau
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK



-- 
Marc-André Lureau

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Mon, Sep 18, 2017 at 11:40:40AM +0100, Dr. David Alan Gilbert wrote:
> * Peter Xu (peterx@redhat.com) wrote:
> > On Fri, Sep 15, 2017 at 04:17:07PM +0100, Dr. David Alan Gilbert wrote:
> > > * Stefan Hajnoczi (stefanha@redhat.com) wrote:
> > > > On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:
> > > > > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:
> > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> > > > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > > > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > > > > > > > > There should be a limit in the number of requests the thread can
> > > > > > > > > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > > > > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > > > > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > > > > > > > > 
> > > > > > > > > > > > I agree.
> > > > > > > > > > > > 
> > > > > > > > > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > > > > > > > > than the amount of memory consumed by them.
> > > > > > > > > > > > 
> > > > > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > > > > > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > > > > > > > > processed if queued non-OOB commands consume too much memory.
> > > > > > > > > > > 
> > > > > > > > > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > > > > > > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > > > > > > > > Control" section of that long reply).  Hopefully this series basically
> > > > > > > > > > > can work from design prospective, then I'll add this flow control in
> > > > > > > > > > > next version.
> > > > > > > > > > > 
> > > > > > > > > > > Regarding to what we should do if the limit is reached: Markus
> > > > > > > > > > > provided a few options, but the one I prefer most is that we don't
> > > > > > > > > > > respond, but send an event showing that a command is dropped.
> > > > > > > > > > > However, I would like it not queued, but a direct reply (after all,
> > > > > > > > > > > it's an event, and we should not need to care much on ordering of it).
> > > > > > > > > > > Then we can get rid of the babysitting of those "to be failed"
> > > > > > > > > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > > > > > > > > 
> > > > > > > > > > > I think I also missed at least a unit test for this new interface.
> > > > > > > > > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > > > > > > > > 
> > > > > > > > > > Another solution: the server reports available receive buffer space to
> > > > > > > > > > the client.  The server only guarantees immediate OOB processing when
> > > > > > > > > > the client stays within the receive buffer size.
> > > > > > > > > > 
> > > > > > > > > > Clients wishing to take advantage of OOB must query the receive buffer
> > > > > > > > > > size and make sure to leave enough room.
> > > > > > > > > 
> > > > > > > > > I don't think having to query it ahead of time is particularly nice,
> > > > > > > > > and of course it is inherantly racy.
> > > > > > > > > 
> > > > > > > > > I would just have QEMU emit an event when it pausing processing of the
> > > > > > > > > incoming commands due to a full queue.  If the event includes the ID
> > > > > > > > > of the last queued command, the client will know which (if any) of
> > > > > > > > > its outstanding commands are delayed. Another even can be sent when
> > > > > > > > > it restarts reading.
> > > > > > > > 
> > > > > > > > Hmm and now we're implementing flow control!
> > > > > > > > 
> > > > > > > > a) What exactly is the current semantics/buffer sizes?
> > > > > > > > b) When do clients send multiple QMP commands on one channel without
> > > > > > > > waiting for the response to the previous command?
> > > > > > > > c) Would one queue entry for each class of commands/channel work
> > > > > > > >   (Where a class of commands is currently 'normal' and 'oob')
> > > > > > > 
> > > > > > > I do wonder if we need to worry about request limiting at all from the
> > > > > > > client side.  For non-OOB commands clients will wait for a reply before
> > > > > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.
> > > > > > > 
> > > > > > > OOB commands are supposed to be things which can be handled quickly
> > > > > > > without blocking, so even if a client sent several commands at once
> > > > > > > without waiting for replies, they're going to be processed quickly,
> > > > > > > so whether we temporarily block reading off the wire is a minor
> > > > > > > detail.
> > > > > > 
> > > > > > Lets just define it so that it can't - you send an OOB command and wait
> > > > > > for it's response before sending another on that channel.
> > > > > > 
> > > > > > > IOW, I think we could just have a fixed 10 command queue and apps just
> > > > > > > pretend that there's an infinite queue and nothing bad would happen from
> > > > > > > the app's POV.
> > > > > > 
> > > > > > Can you justify 10 as opposed to 1?
> > > > > 
> > > > > Semantically I don't think it makes a difference if the OOB commands are
> > > > > being processed sequentially by their thread. A >1 length queue would only
> > > > > matter for non-OOB commands if an app was filling the pipeline with non-OOB
> > > > > requests, as then that could block reading of OOB commands. 
> > > > 
> > > > To summarize:
> > > > 
> > > > The QMP server has a lookahead of 1 command so it can dispatch
> > > > out-of-band commands.  If 2 or more non-OOB commands are queued at the
> > > > same time then OOB processing will not occur.
> > > > 
> > > > Is that right?
> > > 
> > > I think my view is slightly more complex;
> > >   a) There's a pair of queues for each channel
> > >   b) There's a central pair of queues on the QMP server
> > >     one for OOB commands and one for normal commands.
> > >   c) Each queue is only really guaranteed to be one deep.
> > > 
> > >   That means that each one of the channels can send a non-OOB
> > > command without getting in the way of a channel that wants
> > > to send one.
> > 
> > But current version should not be that complex:
> > 
> > Firstly, parser thread will only be enabled for QMP+NO_MIXED monitors.
> > 
> > Then, we only have a single global queue for QMP non-oob commands, and
> > we don't have response queue yet.  We do respond just like before in a
> > synchronous way (I explained why - for OOB we don't need that
> > complexity IMHO).
> 
> I think  the discussion started because of two related comments:
>   Marc-André said :
>      'There should be a limit in the number of requests the thread can
> queue'
> and Stefan said :
>      'Memory usage must be bounded.'
> 
> actually neither of those cases really worried me (because they only
> happen if the client keeps pumping commands, and that seems it's fault).
> 
> However, once you start adding a limit, you've got to be careful - if
> you just added a limit to the central queue, then what happens if that
> queue is filled by non-OOB commands?

Ah!  So I misunderstood "a pair of queues for each channel".  I
thought it means the input and output of a single monitor, while I
think it actually means "OOB channel" and "non-OOB channel".

My plan (or say, this version) starts from only one global queue for
non-OOB commands.  There is no queue for OOB commands at all.  As
discussed below [1], if we receive one OOB command, we execute it
directly and reply to client.  And here the "request queue" will only
queue non-OOB commands.  Maybe the name "request queue" sounds
confusing here.

If so, we should not have above problem, right?  Since even if the
queue is full (of course there will only be non-OOB commands in the
queue), the parsing is still working, and we will still be able to
handle OOB ones:

  req = parse(stream);

  if (is_oob(req)) {
    execute(req);
    return;
  }

  if (queue_full(req_queue)) {
    emit_full_event(req);
    return;
  }

  enqueue(req_queue, req);

So again, this version is a simplified version from previous
discussion (no oob-queue but only non-oob-queue, no respond queue but
only request queue, etc...), but hope it can work.

Thanks,

> 
> Dave
> 
> > When we parse commands, we execute it directly if OOB, otherwise we
> > put it onto request queue.  Request queue handling is done by a main
> > thread QEMUBH.  That's all.

[1]

> > 
> > Would this "simple version" suffice to implement this whole OOB idea?
> > 
> > (Again, I really don't think we need to specify queue length to 1,
> >  though we can make it small)
> > 
> > -- 
> > Peter Xu
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

-- 
Peter Xu

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Mon, Sep 18, 2017 at 06:09:29PM +0200, Marc-André Lureau wrote:
> On Mon, Sep 18, 2017 at 1:26 PM, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> >> Hi
> >>
> >> On Mon, Sep 18, 2017 at 12:55 PM, Dr. David Alan Gilbert
> >> <dgilbert@redhat.com> wrote:
> >> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> >> >> Hi
> >> >>
> >> >> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu <peterx@redhat.com> wrote:
> >> >> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:
> >> >> >> Hi
> >> >> >>
> >> >> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu <peterx@redhat.com> wrote:
> >> >> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
> >> >> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> >> >> >> >> > Hi
> >> >> >> >> >
> >> >> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
> >> >> >> >> > > This series was born from this one:
> >> >> >> >> > >
> >> >> >> >> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> >> >> >> >> > >
> >> >> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions
> >> >> >> >> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
> >> >> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny
> >> >> >> >> > > ideas and suggestions.  Finally we got such a solution that seems to
> >> >> >> >> > > satisfy everyone.
> >> >> >> >> > >
> >> >> >> >> > > I re-started the versioning since this series is totally different
> >> >> >> >> > > from previous one.  Now it's version 1.
> >> >> >> >> > >
> >> >> >> >> > > In case new reviewers come along the way without reading previous
> >> >> >> >> > > discussions, I will try to do a summary on what this is all about.
> >> >> >> >> > >
> >> >> >> >> > > What is OOB execution?
> >> >> >> >> > > ======================
> >> >> >> >> > >
> >> >> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by
> >> >> >> >> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
> >> >> >> >> > > QMP is going throw these steps:
> >> >> >> >> > >
> >> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> >> >> >> >> > >           /|\    (2)                (3)     |
> >> >> >> >> > >        (1) |                               \|/ (4)
> >> >> >> >> > >            +---------  main thread  --------+
> >> >> >> >> > >
> >> >> >> >> > > The requests are executed by the so-called QMP-dispatcher after the
> >> >> >> >> > > JSON is parsed.  If OOB is on, we run the command directly in the
> >> >> >> >> > > parser and quickly returns.
> >> >> >> >> >
> >> >> >> >> > All commands should have the "id" field mandatory in this case, else
> >> >> >> >> > the client will not distinguish the replies coming from the last/oob
> >> >> >> >> > and the previous commands.
> >> >> >> >> >
> >> >> >> >> > This should probably be enforced upfront by client capability checks,
> >> >> >> >> > more below.
> >> >> >> >
> >> >> >> > Hmm yes since the oob commands are actually running in async way,
> >> >> >> > request ID should be needed here.  However I'm not sure whether
> >> >> >> > enabling the whole "request ID" thing is too big for this "try to be
> >> >> >> > small" oob change... And IMHO it suites better to be part of the whole
> >> >> >> > async work (no matter which implementation we'll use).
> >> >> >> >
> >> >> >> > How about this: we make "id" mandatory for "run-oob" requests only.
> >> >> >> > For oob commands, they will always have ID then no ordering issue, and
> >> >> >> > we can do it async; for the rest of non-oob commands, we still allow
> >> >> >> > them to go without ID, and since they are not oob, they'll always be
> >> >> >> > done in order as well.  Would this work?
> >> >> >>
> >> >> >> This mixed-mode is imho more complicated to deal with than having the
> >> >> >> protocol enforced one way or the other, but that should work.
> >> >> >>
> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > Yeah I know in current code the parser calls dispatcher directly
> >> >> >> >> > > (please see handle_qmp_command()).  However it's not true again after
> >> >> >> >> > > this series (parser will has its own IO thread, and dispatcher will
> >> >> >> >> > > still be run in main thread).  So this OOB does brings something
> >> >> >> >> > > different.
> >> >> >> >> > >
> >> >> >> >> > > There are more details on why OOB and the difference/relationship
> >> >> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
> >> >> >> >> > > slightly out of topic (and believe me, it's not easy for me to
> >> >> >> >> > > summarize that).  For more information, please refers to [1].
> >> >> >> >> > >
> >> >> >> >> > > Summary ends here.
> >> >> >> >> > >
> >> >> >> >> > > Some Implementation Details
> >> >> >> >> > > ===========================
> >> >> >> >> > >
> >> >> >> >> > > Again, I mentioned that the old QMP workflow is this:
> >> >> >> >> > >
> >> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> >> >> >> >> > >           /|\    (2)                (3)     |
> >> >> >> >> > >        (1) |                               \|/ (4)
> >> >> >> >> > >            +---------  main thread  --------+
> >> >> >> >> > >
> >> >> >> >> > > What this series does is, firstly:
> >> >> >> >> > >
> >> >> >> >> > >       JSON Parser     QMP Dispatcher --> Respond
> >> >> >> >> > >           /|\ |           /|\       (4)     |
> >> >> >> >> > >            |  | (2)        | (3)            |  (5)
> >> >> >> >> > >        (1) |  +----->      |               \|/
> >> >> >> >> > >            +---------  main thread  <-------+
> >> >> >> >> > >
> >> >> >> >> > > And further:
> >> >> >> >> > >
> >> >> >> >> > >                queue/kick
> >> >> >> >> > >      JSON Parser ======> QMP Dispatcher --> Respond
> >> >> >> >> > >          /|\ |     (3)       /|\        (4)    |
> >> >> >> >> > >       (1) |  | (2)            |                |  (5)
> >> >> >> >> > >           | \|/               |               \|/
> >> >> >> >> > >         IO thread         main thread  <-------+
> >> >> >> >> >
> >> >> >> >> > Is the queue per monitor or per client?
> >> >> >> >
> >> >> >> > The queue is currently global. I think yes maybe at least we can do it
> >> >> >> > per monitor, but I am not sure whether that is urgent or can be
> >> >> >> > postponed.  After all now QMPRequest (please refer to patch 11) is
> >> >> >> > defined as (mon, id, req) tuple, so at least "id" namespace is
> >> >> >> > per-monitor.
> >> >> >> >
> >> >> >> >> > And is the dispatching going
> >> >> >> >> > to be processed even if the client is disconnected, and are new
> >> >> >> >> > clients going to receive the replies from previous clients
> >> >> >> >> > commands?
> >> >> >> >
> >> >> >> > [1]
> >> >> >> >
> >> >> >> > (will discuss together below)
> >> >> >> >
> >> >> >> >> > I
> >> >> >> >> > believe there should be a per-client context, so there won't be "id"
> >> >> >> >> > request conflicts.
> >> >> >> >
> >> >> >> > I'd say I am not familiar with this "client" idea, since after all
> >> >> >> > IMHO one monitor is currently designed to mostly work with a single
> >> >> >> > client. Say, unix sockets, telnet, all these backends are only single
> >> >> >> > channeled, and one monitor instance can only work with one client at a
> >> >> >> > time.  Then do we really need to add this client layer upon it?  IMHO
> >> >> >> > the user can just provide more monitors if they wants more clients
> >> >> >> > (and at least these clients should know the existance of the others or
> >> >> >> > there might be problem, otherwise user2 will fail a migration, finally
> >> >> >> > noticed that user1 has already triggered one), and the user should
> >> >> >> > manage them well.
> >> >> >>
> >> >> >> qemu should support a management layer / libvirt restart/reconnect.
> >> >> >> Afaik, it mostly work today. There might be a cases where libvirt can
> >> >> >> be confused if it receives a reply from a previous connection command,
> >> >> >> but due to the sync processing of the chardev, I am not sure you can
> >> >> >> get in this situation.  By adding "oob" commands and queuing, the
> >> >> >> client will have to remember which was the last "id" used, or it will
> >> >> >> create more conflict after a reconnect.
> >> >> >>
> >> >> >> Imho we should introduce the client/connection concept to avoid this
> >> >> >> confusion (unexpected reply & per client id space).
> >> >> >
> >> >> > Hmm I agree that the reconnect feature would be nice, but if so IMHO
> >> >> > instead of throwing responses away when client disconnect, we should
> >> >> > really keep them, and when the client reconnects, we queue the
> >> >> > responses again.
> >> >> >
> >> >> > I think we have other quite simple ways to solve the "unexpected
> >> >> > reply" and "per-client-id duplication" issues you have mentioned.
> >> >> >
> >> >> > Firstly, when client gets unexpected replies ("id" field not in its
> >> >> > own request queue), the client should just ignore that reply, which
> >> >> > seems natural to me.
> >> >>
> >> >> The trouble is that it may legitimately use the same "id" value for
> >> >> new requests. And I don't see a simple way to handle that without
> >> >> races.
> >> >
> >> > Under what circumstances can it reuse the same ID for new requests?
> >> > Can't we simply tell it not to?
> >>
> >> I don't see any restriction today in the protocol in connecting with a
> >> new client that may not know anything from a previous client.
> >
> > Well, it knows it's doing a reconnection.
> 
> If you assume the "same client" reconnects to the monitor, I agree.
> But this is a restriction of monitor usage.

In monitor_qmp_event(), we can empty the request queue when got
CHR_EVENT_CLOSED.  Would that be a solution?

-- 
Peter Xu

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Peter Xu (peterx@redhat.com) wrote:
> On Mon, Sep 18, 2017 at 11:40:40AM +0100, Dr. David Alan Gilbert wrote:
> > * Peter Xu (peterx@redhat.com) wrote:
> > > On Fri, Sep 15, 2017 at 04:17:07PM +0100, Dr. David Alan Gilbert wrote:
> > > > * Stefan Hajnoczi (stefanha@redhat.com) wrote:
> > > > > On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:
> > > > > > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:
> > > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> > > > > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > > > > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > > > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > > > > > > > > > There should be a limit in the number of requests the thread can
> > > > > > > > > > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > > > > > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > > > > > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I agree.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > > > > > > > > > than the amount of memory consumed by them.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > > > > > > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > > > > > > > > > processed if queued non-OOB commands consume too much memory.
> > > > > > > > > > > > 
> > > > > > > > > > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > > > > > > > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > > > > > > > > > Control" section of that long reply).  Hopefully this series basically
> > > > > > > > > > > > can work from design prospective, then I'll add this flow control in
> > > > > > > > > > > > next version.
> > > > > > > > > > > > 
> > > > > > > > > > > > Regarding to what we should do if the limit is reached: Markus
> > > > > > > > > > > > provided a few options, but the one I prefer most is that we don't
> > > > > > > > > > > > respond, but send an event showing that a command is dropped.
> > > > > > > > > > > > However, I would like it not queued, but a direct reply (after all,
> > > > > > > > > > > > it's an event, and we should not need to care much on ordering of it).
> > > > > > > > > > > > Then we can get rid of the babysitting of those "to be failed"
> > > > > > > > > > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > > > > > > > > > 
> > > > > > > > > > > > I think I also missed at least a unit test for this new interface.
> > > > > > > > > > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > > > > > > > > > 
> > > > > > > > > > > Another solution: the server reports available receive buffer space to
> > > > > > > > > > > the client.  The server only guarantees immediate OOB processing when
> > > > > > > > > > > the client stays within the receive buffer size.
> > > > > > > > > > > 
> > > > > > > > > > > Clients wishing to take advantage of OOB must query the receive buffer
> > > > > > > > > > > size and make sure to leave enough room.
> > > > > > > > > > 
> > > > > > > > > > I don't think having to query it ahead of time is particularly nice,
> > > > > > > > > > and of course it is inherantly racy.
> > > > > > > > > > 
> > > > > > > > > > I would just have QEMU emit an event when it pausing processing of the
> > > > > > > > > > incoming commands due to a full queue.  If the event includes the ID
> > > > > > > > > > of the last queued command, the client will know which (if any) of
> > > > > > > > > > its outstanding commands are delayed. Another even can be sent when
> > > > > > > > > > it restarts reading.
> > > > > > > > > 
> > > > > > > > > Hmm and now we're implementing flow control!
> > > > > > > > > 
> > > > > > > > > a) What exactly is the current semantics/buffer sizes?
> > > > > > > > > b) When do clients send multiple QMP commands on one channel without
> > > > > > > > > waiting for the response to the previous command?
> > > > > > > > > c) Would one queue entry for each class of commands/channel work
> > > > > > > > >   (Where a class of commands is currently 'normal' and 'oob')
> > > > > > > > 
> > > > > > > > I do wonder if we need to worry about request limiting at all from the
> > > > > > > > client side.  For non-OOB commands clients will wait for a reply before
> > > > > > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.
> > > > > > > > 
> > > > > > > > OOB commands are supposed to be things which can be handled quickly
> > > > > > > > without blocking, so even if a client sent several commands at once
> > > > > > > > without waiting for replies, they're going to be processed quickly,
> > > > > > > > so whether we temporarily block reading off the wire is a minor
> > > > > > > > detail.
> > > > > > > 
> > > > > > > Lets just define it so that it can't - you send an OOB command and wait
> > > > > > > for it's response before sending another on that channel.
> > > > > > > 
> > > > > > > > IOW, I think we could just have a fixed 10 command queue and apps just
> > > > > > > > pretend that there's an infinite queue and nothing bad would happen from
> > > > > > > > the app's POV.
> > > > > > > 
> > > > > > > Can you justify 10 as opposed to 1?
> > > > > > 
> > > > > > Semantically I don't think it makes a difference if the OOB commands are
> > > > > > being processed sequentially by their thread. A >1 length queue would only
> > > > > > matter for non-OOB commands if an app was filling the pipeline with non-OOB
> > > > > > requests, as then that could block reading of OOB commands. 
> > > > > 
> > > > > To summarize:
> > > > > 
> > > > > The QMP server has a lookahead of 1 command so it can dispatch
> > > > > out-of-band commands.  If 2 or more non-OOB commands are queued at the
> > > > > same time then OOB processing will not occur.
> > > > > 
> > > > > Is that right?
> > > > 
> > > > I think my view is slightly more complex;
> > > >   a) There's a pair of queues for each channel
> > > >   b) There's a central pair of queues on the QMP server
> > > >     one for OOB commands and one for normal commands.
> > > >   c) Each queue is only really guaranteed to be one deep.
> > > > 
> > > >   That means that each one of the channels can send a non-OOB
> > > > command without getting in the way of a channel that wants
> > > > to send one.
> > > 
> > > But current version should not be that complex:
> > > 
> > > Firstly, parser thread will only be enabled for QMP+NO_MIXED monitors.
> > > 
> > > Then, we only have a single global queue for QMP non-oob commands, and
> > > we don't have response queue yet.  We do respond just like before in a
> > > synchronous way (I explained why - for OOB we don't need that
> > > complexity IMHO).
> > 
> > I think  the discussion started because of two related comments:
> >   Marc-André said :
> >      'There should be a limit in the number of requests the thread can
> > queue'
> > and Stefan said :
> >      'Memory usage must be bounded.'
> > 
> > actually neither of those cases really worried me (because they only
> > happen if the client keeps pumping commands, and that seems it's fault).
> > 
> > However, once you start adding a limit, you've got to be careful - if
> > you just added a limit to the central queue, then what happens if that
> > queue is filled by non-OOB commands?
> 
> Ah!  So I misunderstood "a pair of queues for each channel".  I
> thought it means the input and output of a single monitor, while I
> think it actually means "OOB channel" and "non-OOB channel".
> 
> My plan (or say, this version) starts from only one global queue for
> non-OOB commands.  There is no queue for OOB commands at all.  As
> discussed below [1], if we receive one OOB command, we execute it
> directly and reply to client.  And here the "request queue" will only
> queue non-OOB commands.  Maybe the name "request queue" sounds
> confusing here.
> 
> If so, we should not have above problem, right?  Since even if the
> queue is full (of course there will only be non-OOB commands in the
> queue), the parsing is still working, and we will still be able to
> handle OOB ones:
> 
>   req = parse(stream);
> 
>   if (is_oob(req)) {
>     execute(req);
>     return;
>   }
> 
>   if (queue_full(req_queue)) {
>     emit_full_event(req);
>     return;
>   }
> 
>   enqueue(req_queue, req);
> 
> So again, this version is a simplified version from previous
> discussion (no oob-queue but only non-oob-queue, no respond queue but
> only request queue, etc...), but hope it can work.

That might work.  You have to be really careful about allowing
OOB commands to be parsed, even if the non-OOB queue is full.

One problem is that one client could fill up that shared queue,
then another client would be surprised to find the queue is full
when it tries to send just one command - hence why I thought a separate
queue per client would solve that.

Dave

> Thanks,
> 
> > 
> > Dave
> > 
> > > When we parse commands, we execute it directly if OOB, otherwise we
> > > put it onto request queue.  Request queue handling is done by a main
> > > thread QEMUBH.  That's all.
> 
> [1]
> 
> > > 
> > > Would this "simple version" suffice to implement this whole OOB idea?
> > > 
> > > (Again, I really don't think we need to specify queue length to 1,
> > >  though we can make it small)
> > > 
> > > -- 
> > > Peter Xu
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
> -- 
> Peter Xu
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Peter Xu (peterx@redhat.com) wrote:
> On Mon, Sep 18, 2017 at 06:09:29PM +0200, Marc-André Lureau wrote:
> > On Mon, Sep 18, 2017 at 1:26 PM, Dr. David Alan Gilbert
> > <dgilbert@redhat.com> wrote:
> > > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> > >> Hi
> > >>
> > >> On Mon, Sep 18, 2017 at 12:55 PM, Dr. David Alan Gilbert
> > >> <dgilbert@redhat.com> wrote:
> > >> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> > >> >> Hi
> > >> >>
> > >> >> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu <peterx@redhat.com> wrote:
> > >> >> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:
> > >> >> >> Hi
> > >> >> >>
> > >> >> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu <peterx@redhat.com> wrote:
> > >> >> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
> > >> >> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> > >> >> >> >> > Hi
> > >> >> >> >> >
> > >> >> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
> > >> >> >> >> > > This series was born from this one:
> > >> >> >> >> > >
> > >> >> >> >> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> > >> >> >> >> > >
> > >> >> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions
> > >> >> >> >> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
> > >> >> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny
> > >> >> >> >> > > ideas and suggestions.  Finally we got such a solution that seems to
> > >> >> >> >> > > satisfy everyone.
> > >> >> >> >> > >
> > >> >> >> >> > > I re-started the versioning since this series is totally different
> > >> >> >> >> > > from previous one.  Now it's version 1.
> > >> >> >> >> > >
> > >> >> >> >> > > In case new reviewers come along the way without reading previous
> > >> >> >> >> > > discussions, I will try to do a summary on what this is all about.
> > >> >> >> >> > >
> > >> >> >> >> > > What is OOB execution?
> > >> >> >> >> > > ======================
> > >> >> >> >> > >
> > >> >> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by
> > >> >> >> >> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
> > >> >> >> >> > > QMP is going throw these steps:
> > >> >> >> >> > >
> > >> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> > >> >> >> >> > >           /|\    (2)                (3)     |
> > >> >> >> >> > >        (1) |                               \|/ (4)
> > >> >> >> >> > >            +---------  main thread  --------+
> > >> >> >> >> > >
> > >> >> >> >> > > The requests are executed by the so-called QMP-dispatcher after the
> > >> >> >> >> > > JSON is parsed.  If OOB is on, we run the command directly in the
> > >> >> >> >> > > parser and quickly returns.
> > >> >> >> >> >
> > >> >> >> >> > All commands should have the "id" field mandatory in this case, else
> > >> >> >> >> > the client will not distinguish the replies coming from the last/oob
> > >> >> >> >> > and the previous commands.
> > >> >> >> >> >
> > >> >> >> >> > This should probably be enforced upfront by client capability checks,
> > >> >> >> >> > more below.
> > >> >> >> >
> > >> >> >> > Hmm yes since the oob commands are actually running in async way,
> > >> >> >> > request ID should be needed here.  However I'm not sure whether
> > >> >> >> > enabling the whole "request ID" thing is too big for this "try to be
> > >> >> >> > small" oob change... And IMHO it suites better to be part of the whole
> > >> >> >> > async work (no matter which implementation we'll use).
> > >> >> >> >
> > >> >> >> > How about this: we make "id" mandatory for "run-oob" requests only.
> > >> >> >> > For oob commands, they will always have ID then no ordering issue, and
> > >> >> >> > we can do it async; for the rest of non-oob commands, we still allow
> > >> >> >> > them to go without ID, and since they are not oob, they'll always be
> > >> >> >> > done in order as well.  Would this work?
> > >> >> >>
> > >> >> >> This mixed-mode is imho more complicated to deal with than having the
> > >> >> >> protocol enforced one way or the other, but that should work.
> > >> >> >>
> > >> >> >> >
> > >> >> >> >> >
> > >> >> >> >> > > Yeah I know in current code the parser calls dispatcher directly
> > >> >> >> >> > > (please see handle_qmp_command()).  However it's not true again after
> > >> >> >> >> > > this series (parser will has its own IO thread, and dispatcher will
> > >> >> >> >> > > still be run in main thread).  So this OOB does brings something
> > >> >> >> >> > > different.
> > >> >> >> >> > >
> > >> >> >> >> > > There are more details on why OOB and the difference/relationship
> > >> >> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
> > >> >> >> >> > > slightly out of topic (and believe me, it's not easy for me to
> > >> >> >> >> > > summarize that).  For more information, please refers to [1].
> > >> >> >> >> > >
> > >> >> >> >> > > Summary ends here.
> > >> >> >> >> > >
> > >> >> >> >> > > Some Implementation Details
> > >> >> >> >> > > ===========================
> > >> >> >> >> > >
> > >> >> >> >> > > Again, I mentioned that the old QMP workflow is this:
> > >> >> >> >> > >
> > >> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> > >> >> >> >> > >           /|\    (2)                (3)     |
> > >> >> >> >> > >        (1) |                               \|/ (4)
> > >> >> >> >> > >            +---------  main thread  --------+
> > >> >> >> >> > >
> > >> >> >> >> > > What this series does is, firstly:
> > >> >> >> >> > >
> > >> >> >> >> > >       JSON Parser     QMP Dispatcher --> Respond
> > >> >> >> >> > >           /|\ |           /|\       (4)     |
> > >> >> >> >> > >            |  | (2)        | (3)            |  (5)
> > >> >> >> >> > >        (1) |  +----->      |               \|/
> > >> >> >> >> > >            +---------  main thread  <-------+
> > >> >> >> >> > >
> > >> >> >> >> > > And further:
> > >> >> >> >> > >
> > >> >> >> >> > >                queue/kick
> > >> >> >> >> > >      JSON Parser ======> QMP Dispatcher --> Respond
> > >> >> >> >> > >          /|\ |     (3)       /|\        (4)    |
> > >> >> >> >> > >       (1) |  | (2)            |                |  (5)
> > >> >> >> >> > >           | \|/               |               \|/
> > >> >> >> >> > >         IO thread         main thread  <-------+
> > >> >> >> >> >
> > >> >> >> >> > Is the queue per monitor or per client?
> > >> >> >> >
> > >> >> >> > The queue is currently global. I think yes maybe at least we can do it
> > >> >> >> > per monitor, but I am not sure whether that is urgent or can be
> > >> >> >> > postponed.  After all now QMPRequest (please refer to patch 11) is
> > >> >> >> > defined as (mon, id, req) tuple, so at least "id" namespace is
> > >> >> >> > per-monitor.
> > >> >> >> >
> > >> >> >> >> > And is the dispatching going
> > >> >> >> >> > to be processed even if the client is disconnected, and are new
> > >> >> >> >> > clients going to receive the replies from previous clients
> > >> >> >> >> > commands?
> > >> >> >> >
> > >> >> >> > [1]
> > >> >> >> >
> > >> >> >> > (will discuss together below)
> > >> >> >> >
> > >> >> >> >> > I
> > >> >> >> >> > believe there should be a per-client context, so there won't be "id"
> > >> >> >> >> > request conflicts.
> > >> >> >> >
> > >> >> >> > I'd say I am not familiar with this "client" idea, since after all
> > >> >> >> > IMHO one monitor is currently designed to mostly work with a single
> > >> >> >> > client. Say, unix sockets, telnet, all these backends are only single
> > >> >> >> > channeled, and one monitor instance can only work with one client at a
> > >> >> >> > time.  Then do we really need to add this client layer upon it?  IMHO
> > >> >> >> > the user can just provide more monitors if they wants more clients
> > >> >> >> > (and at least these clients should know the existance of the others or
> > >> >> >> > there might be problem, otherwise user2 will fail a migration, finally
> > >> >> >> > noticed that user1 has already triggered one), and the user should
> > >> >> >> > manage them well.
> > >> >> >>
> > >> >> >> qemu should support a management layer / libvirt restart/reconnect.
> > >> >> >> Afaik, it mostly work today. There might be a cases where libvirt can
> > >> >> >> be confused if it receives a reply from a previous connection command,
> > >> >> >> but due to the sync processing of the chardev, I am not sure you can
> > >> >> >> get in this situation.  By adding "oob" commands and queuing, the
> > >> >> >> client will have to remember which was the last "id" used, or it will
> > >> >> >> create more conflict after a reconnect.
> > >> >> >>
> > >> >> >> Imho we should introduce the client/connection concept to avoid this
> > >> >> >> confusion (unexpected reply & per client id space).
> > >> >> >
> > >> >> > Hmm I agree that the reconnect feature would be nice, but if so IMHO
> > >> >> > instead of throwing responses away when client disconnect, we should
> > >> >> > really keep them, and when the client reconnects, we queue the
> > >> >> > responses again.
> > >> >> >
> > >> >> > I think we have other quite simple ways to solve the "unexpected
> > >> >> > reply" and "per-client-id duplication" issues you have mentioned.
> > >> >> >
> > >> >> > Firstly, when client gets unexpected replies ("id" field not in its
> > >> >> > own request queue), the client should just ignore that reply, which
> > >> >> > seems natural to me.
> > >> >>
> > >> >> The trouble is that it may legitimately use the same "id" value for
> > >> >> new requests. And I don't see a simple way to handle that without
> > >> >> races.
> > >> >
> > >> > Under what circumstances can it reuse the same ID for new requests?
> > >> > Can't we simply tell it not to?
> > >>
> > >> I don't see any restriction today in the protocol in connecting with a
> > >> new client that may not know anything from a previous client.
> > >
> > > Well, it knows it's doing a reconnection.
> > 
> > If you assume the "same client" reconnects to the monitor, I agree.
> > But this is a restriction of monitor usage.
> 
> In monitor_qmp_event(), we can empty the request queue when got
> CHR_EVENT_CLOSED.  Would that be a solution?

What happens to commands that are in flight?

Dave

> -- 
> Peter Xu
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Tue, Sep 19, 2017 at 10:13:52AM +0100, Dr. David Alan Gilbert wrote:
> * Peter Xu (peterx@redhat.com) wrote:
> > On Mon, Sep 18, 2017 at 11:40:40AM +0100, Dr. David Alan Gilbert wrote:
> > > * Peter Xu (peterx@redhat.com) wrote:
> > > > On Fri, Sep 15, 2017 at 04:17:07PM +0100, Dr. David Alan Gilbert wrote:
> > > > > * Stefan Hajnoczi (stefanha@redhat.com) wrote:
> > > > > > On Fri, Sep 15, 2017 at 01:29:13PM +0100, Daniel P. Berrange wrote:
> > > > > > > On Fri, Sep 15, 2017 at 01:19:56PM +0100, Dr. David Alan Gilbert wrote:
> > > > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > > > > > On Fri, Sep 15, 2017 at 01:06:44PM +0100, Dr. David Alan Gilbert wrote:
> > > > > > > > > > * Daniel P. Berrange (berrange@redhat.com) wrote:
> > > > > > > > > > > On Fri, Sep 15, 2017 at 11:49:26AM +0100, Stefan Hajnoczi wrote:
> > > > > > > > > > > > On Fri, Sep 15, 2017 at 11:50:57AM +0800, Peter Xu wrote:
> > > > > > > > > > > > > On Thu, Sep 14, 2017 at 04:19:11PM +0100, Stefan Hajnoczi wrote:
> > > > > > > > > > > > > > On Thu, Sep 14, 2017 at 01:15:09PM +0200, Marc-André Lureau wrote:
> > > > > > > > > > > > > > > There should be a limit in the number of requests the thread can
> > > > > > > > > > > > > > > queue. Before the patch, the limit was enforced by system socket
> > > > > > > > > > > > > > > buffering I think. Now, should oob commands still be processed even if
> > > > > > > > > > > > > > > the queue is full? If so, the thread can't be suspended.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > I agree.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Memory usage must be bounded.  The number of requests is less important
> > > > > > > > > > > > > > than the amount of memory consumed by them.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Existing QMP clients that send multiple QMP commands without waiting for
> > > > > > > > > > > > > > replies need to rethink their strategy because OOB commands cannot be
> > > > > > > > > > > > > > processed if queued non-OOB commands consume too much memory.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Thanks for pointing out this.  Yes the memory usage problem is valid,
> > > > > > > > > > > > > as Markus pointed out as well in previous discussions (in "Flow
> > > > > > > > > > > > > Control" section of that long reply).  Hopefully this series basically
> > > > > > > > > > > > > can work from design prospective, then I'll add this flow control in
> > > > > > > > > > > > > next version.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Regarding to what we should do if the limit is reached: Markus
> > > > > > > > > > > > > provided a few options, but the one I prefer most is that we don't
> > > > > > > > > > > > > respond, but send an event showing that a command is dropped.
> > > > > > > > > > > > > However, I would like it not queued, but a direct reply (after all,
> > > > > > > > > > > > > it's an event, and we should not need to care much on ordering of it).
> > > > > > > > > > > > > Then we can get rid of the babysitting of those "to be failed"
> > > > > > > > > > > > > requests asap, meanwhile we don't lose anything IMHO.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > I think I also missed at least a unit test for this new interface.
> > > > > > > > > > > > > Again, I'll add it after the whole idea is proved solid.  Thanks,
> > > > > > > > > > > > 
> > > > > > > > > > > > Another solution: the server reports available receive buffer space to
> > > > > > > > > > > > the client.  The server only guarantees immediate OOB processing when
> > > > > > > > > > > > the client stays within the receive buffer size.
> > > > > > > > > > > > 
> > > > > > > > > > > > Clients wishing to take advantage of OOB must query the receive buffer
> > > > > > > > > > > > size and make sure to leave enough room.
> > > > > > > > > > > 
> > > > > > > > > > > I don't think having to query it ahead of time is particularly nice,
> > > > > > > > > > > and of course it is inherantly racy.
> > > > > > > > > > > 
> > > > > > > > > > > I would just have QEMU emit an event when it pausing processing of the
> > > > > > > > > > > incoming commands due to a full queue.  If the event includes the ID
> > > > > > > > > > > of the last queued command, the client will know which (if any) of
> > > > > > > > > > > its outstanding commands are delayed. Another even can be sent when
> > > > > > > > > > > it restarts reading.
> > > > > > > > > > 
> > > > > > > > > > Hmm and now we're implementing flow control!
> > > > > > > > > > 
> > > > > > > > > > a) What exactly is the current semantics/buffer sizes?
> > > > > > > > > > b) When do clients send multiple QMP commands on one channel without
> > > > > > > > > > waiting for the response to the previous command?
> > > > > > > > > > c) Would one queue entry for each class of commands/channel work
> > > > > > > > > >   (Where a class of commands is currently 'normal' and 'oob')
> > > > > > > > > 
> > > > > > > > > I do wonder if we need to worry about request limiting at all from the
> > > > > > > > > client side.  For non-OOB commands clients will wait for a reply before
> > > > > > > > > sending a 2nd non-OOB command, so you'll never get a deep queue for.
> > > > > > > > > 
> > > > > > > > > OOB commands are supposed to be things which can be handled quickly
> > > > > > > > > without blocking, so even if a client sent several commands at once
> > > > > > > > > without waiting for replies, they're going to be processed quickly,
> > > > > > > > > so whether we temporarily block reading off the wire is a minor
> > > > > > > > > detail.
> > > > > > > > 
> > > > > > > > Lets just define it so that it can't - you send an OOB command and wait
> > > > > > > > for it's response before sending another on that channel.
> > > > > > > > 
> > > > > > > > > IOW, I think we could just have a fixed 10 command queue and apps just
> > > > > > > > > pretend that there's an infinite queue and nothing bad would happen from
> > > > > > > > > the app's POV.
> > > > > > > > 
> > > > > > > > Can you justify 10 as opposed to 1?
> > > > > > > 
> > > > > > > Semantically I don't think it makes a difference if the OOB commands are
> > > > > > > being processed sequentially by their thread. A >1 length queue would only
> > > > > > > matter for non-OOB commands if an app was filling the pipeline with non-OOB
> > > > > > > requests, as then that could block reading of OOB commands. 
> > > > > > 
> > > > > > To summarize:
> > > > > > 
> > > > > > The QMP server has a lookahead of 1 command so it can dispatch
> > > > > > out-of-band commands.  If 2 or more non-OOB commands are queued at the
> > > > > > same time then OOB processing will not occur.
> > > > > > 
> > > > > > Is that right?
> > > > > 
> > > > > I think my view is slightly more complex;
> > > > >   a) There's a pair of queues for each channel
> > > > >   b) There's a central pair of queues on the QMP server
> > > > >     one for OOB commands and one for normal commands.
> > > > >   c) Each queue is only really guaranteed to be one deep.
> > > > > 
> > > > >   That means that each one of the channels can send a non-OOB
> > > > > command without getting in the way of a channel that wants
> > > > > to send one.
> > > > 
> > > > But current version should not be that complex:
> > > > 
> > > > Firstly, parser thread will only be enabled for QMP+NO_MIXED monitors.
> > > > 
> > > > Then, we only have a single global queue for QMP non-oob commands, and
> > > > we don't have response queue yet.  We do respond just like before in a
> > > > synchronous way (I explained why - for OOB we don't need that
> > > > complexity IMHO).
> > > 
> > > I think  the discussion started because of two related comments:
> > >   Marc-André said :
> > >      'There should be a limit in the number of requests the thread can
> > > queue'
> > > and Stefan said :
> > >      'Memory usage must be bounded.'
> > > 
> > > actually neither of those cases really worried me (because they only
> > > happen if the client keeps pumping commands, and that seems it's fault).
> > > 
> > > However, once you start adding a limit, you've got to be careful - if
> > > you just added a limit to the central queue, then what happens if that
> > > queue is filled by non-OOB commands?
> > 
> > Ah!  So I misunderstood "a pair of queues for each channel".  I
> > thought it means the input and output of a single monitor, while I
> > think it actually means "OOB channel" and "non-OOB channel".
> > 
> > My plan (or say, this version) starts from only one global queue for
> > non-OOB commands.  There is no queue for OOB commands at all.  As
> > discussed below [1], if we receive one OOB command, we execute it
> > directly and reply to client.  And here the "request queue" will only
> > queue non-OOB commands.  Maybe the name "request queue" sounds
> > confusing here.
> > 
> > If so, we should not have above problem, right?  Since even if the
> > queue is full (of course there will only be non-OOB commands in the
> > queue), the parsing is still working, and we will still be able to
> > handle OOB ones:
> > 
> >   req = parse(stream);
> > 
> >   if (is_oob(req)) {
> >     execute(req);
> >     return;
> >   }
> > 
> >   if (queue_full(req_queue)) {
> >     emit_full_event(req);
> >     return;
> >   }
> > 
> >   enqueue(req_queue, req);
> > 
> > So again, this version is a simplified version from previous
> > discussion (no oob-queue but only non-oob-queue, no respond queue but
> > only request queue, etc...), but hope it can work.
> 
> That might work.  You have to be really careful about allowing
> OOB commands to be parsed, even if the non-OOB queue is full.
> 
> One problem is that one client could fill up that shared queue,
> then another client would be surprised to find the queue is full
> when it tries to send just one command - hence why I thought a separate
> queue per client would solve that.

Ah yes.  Let me switch to one queue per monitor in my next post.  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Dr. David Alan Gilbert]

* Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> On Mon, Sep 18, 2017 at 1:26 PM, Dr. David Alan Gilbert
> <dgilbert@redhat.com> wrote:
> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> >> Hi
> >>
> >> On Mon, Sep 18, 2017 at 12:55 PM, Dr. David Alan Gilbert
> >> <dgilbert@redhat.com> wrote:
> >> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> >> >> Hi
> >> >>
> >> >> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu <peterx@redhat.com> wrote:
> >> >> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:
> >> >> >> Hi
> >> >> >>
> >> >> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu <peterx@redhat.com> wrote:
> >> >> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
> >> >> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> >> >> >> >> > Hi
> >> >> >> >> >
> >> >> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
> >> >> >> >> > > This series was born from this one:
> >> >> >> >> > >
> >> >> >> >> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> >> >> >> >> > >
> >> >> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions
> >> >> >> >> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
> >> >> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny
> >> >> >> >> > > ideas and suggestions.  Finally we got such a solution that seems to
> >> >> >> >> > > satisfy everyone.
> >> >> >> >> > >
> >> >> >> >> > > I re-started the versioning since this series is totally different
> >> >> >> >> > > from previous one.  Now it's version 1.
> >> >> >> >> > >
> >> >> >> >> > > In case new reviewers come along the way without reading previous
> >> >> >> >> > > discussions, I will try to do a summary on what this is all about.
> >> >> >> >> > >
> >> >> >> >> > > What is OOB execution?
> >> >> >> >> > > ======================
> >> >> >> >> > >
> >> >> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by
> >> >> >> >> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
> >> >> >> >> > > QMP is going throw these steps:
> >> >> >> >> > >
> >> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> >> >> >> >> > >           /|\    (2)                (3)     |
> >> >> >> >> > >        (1) |                               \|/ (4)
> >> >> >> >> > >            +---------  main thread  --------+
> >> >> >> >> > >
> >> >> >> >> > > The requests are executed by the so-called QMP-dispatcher after the
> >> >> >> >> > > JSON is parsed.  If OOB is on, we run the command directly in the
> >> >> >> >> > > parser and quickly returns.
> >> >> >> >> >
> >> >> >> >> > All commands should have the "id" field mandatory in this case, else
> >> >> >> >> > the client will not distinguish the replies coming from the last/oob
> >> >> >> >> > and the previous commands.
> >> >> >> >> >
> >> >> >> >> > This should probably be enforced upfront by client capability checks,
> >> >> >> >> > more below.
> >> >> >> >
> >> >> >> > Hmm yes since the oob commands are actually running in async way,
> >> >> >> > request ID should be needed here.  However I'm not sure whether
> >> >> >> > enabling the whole "request ID" thing is too big for this "try to be
> >> >> >> > small" oob change... And IMHO it suites better to be part of the whole
> >> >> >> > async work (no matter which implementation we'll use).
> >> >> >> >
> >> >> >> > How about this: we make "id" mandatory for "run-oob" requests only.
> >> >> >> > For oob commands, they will always have ID then no ordering issue, and
> >> >> >> > we can do it async; for the rest of non-oob commands, we still allow
> >> >> >> > them to go without ID, and since they are not oob, they'll always be
> >> >> >> > done in order as well.  Would this work?
> >> >> >>
> >> >> >> This mixed-mode is imho more complicated to deal with than having the
> >> >> >> protocol enforced one way or the other, but that should work.
> >> >> >>
> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > > Yeah I know in current code the parser calls dispatcher directly
> >> >> >> >> > > (please see handle_qmp_command()).  However it's not true again after
> >> >> >> >> > > this series (parser will has its own IO thread, and dispatcher will
> >> >> >> >> > > still be run in main thread).  So this OOB does brings something
> >> >> >> >> > > different.
> >> >> >> >> > >
> >> >> >> >> > > There are more details on why OOB and the difference/relationship
> >> >> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
> >> >> >> >> > > slightly out of topic (and believe me, it's not easy for me to
> >> >> >> >> > > summarize that).  For more information, please refers to [1].
> >> >> >> >> > >
> >> >> >> >> > > Summary ends here.
> >> >> >> >> > >
> >> >> >> >> > > Some Implementation Details
> >> >> >> >> > > ===========================
> >> >> >> >> > >
> >> >> >> >> > > Again, I mentioned that the old QMP workflow is this:
> >> >> >> >> > >
> >> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> >> >> >> >> > >           /|\    (2)                (3)     |
> >> >> >> >> > >        (1) |                               \|/ (4)
> >> >> >> >> > >            +---------  main thread  --------+
> >> >> >> >> > >
> >> >> >> >> > > What this series does is, firstly:
> >> >> >> >> > >
> >> >> >> >> > >       JSON Parser     QMP Dispatcher --> Respond
> >> >> >> >> > >           /|\ |           /|\       (4)     |
> >> >> >> >> > >            |  | (2)        | (3)            |  (5)
> >> >> >> >> > >        (1) |  +----->      |               \|/
> >> >> >> >> > >            +---------  main thread  <-------+
> >> >> >> >> > >
> >> >> >> >> > > And further:
> >> >> >> >> > >
> >> >> >> >> > >                queue/kick
> >> >> >> >> > >      JSON Parser ======> QMP Dispatcher --> Respond
> >> >> >> >> > >          /|\ |     (3)       /|\        (4)    |
> >> >> >> >> > >       (1) |  | (2)            |                |  (5)
> >> >> >> >> > >           | \|/               |               \|/
> >> >> >> >> > >         IO thread         main thread  <-------+
> >> >> >> >> >
> >> >> >> >> > Is the queue per monitor or per client?
> >> >> >> >
> >> >> >> > The queue is currently global. I think yes maybe at least we can do it
> >> >> >> > per monitor, but I am not sure whether that is urgent or can be
> >> >> >> > postponed.  After all now QMPRequest (please refer to patch 11) is
> >> >> >> > defined as (mon, id, req) tuple, so at least "id" namespace is
> >> >> >> > per-monitor.
> >> >> >> >
> >> >> >> >> > And is the dispatching going
> >> >> >> >> > to be processed even if the client is disconnected, and are new
> >> >> >> >> > clients going to receive the replies from previous clients
> >> >> >> >> > commands?
> >> >> >> >
> >> >> >> > [1]
> >> >> >> >
> >> >> >> > (will discuss together below)
> >> >> >> >
> >> >> >> >> > I
> >> >> >> >> > believe there should be a per-client context, so there won't be "id"
> >> >> >> >> > request conflicts.
> >> >> >> >
> >> >> >> > I'd say I am not familiar with this "client" idea, since after all
> >> >> >> > IMHO one monitor is currently designed to mostly work with a single
> >> >> >> > client. Say, unix sockets, telnet, all these backends are only single
> >> >> >> > channeled, and one monitor instance can only work with one client at a
> >> >> >> > time.  Then do we really need to add this client layer upon it?  IMHO
> >> >> >> > the user can just provide more monitors if they wants more clients
> >> >> >> > (and at least these clients should know the existance of the others or
> >> >> >> > there might be problem, otherwise user2 will fail a migration, finally
> >> >> >> > noticed that user1 has already triggered one), and the user should
> >> >> >> > manage them well.
> >> >> >>
> >> >> >> qemu should support a management layer / libvirt restart/reconnect.
> >> >> >> Afaik, it mostly work today. There might be a cases where libvirt can
> >> >> >> be confused if it receives a reply from a previous connection command,
> >> >> >> but due to the sync processing of the chardev, I am not sure you can
> >> >> >> get in this situation.  By adding "oob" commands and queuing, the
> >> >> >> client will have to remember which was the last "id" used, or it will
> >> >> >> create more conflict after a reconnect.
> >> >> >>
> >> >> >> Imho we should introduce the client/connection concept to avoid this
> >> >> >> confusion (unexpected reply & per client id space).
> >> >> >
> >> >> > Hmm I agree that the reconnect feature would be nice, but if so IMHO
> >> >> > instead of throwing responses away when client disconnect, we should
> >> >> > really keep them, and when the client reconnects, we queue the
> >> >> > responses again.
> >> >> >
> >> >> > I think we have other quite simple ways to solve the "unexpected
> >> >> > reply" and "per-client-id duplication" issues you have mentioned.
> >> >> >
> >> >> > Firstly, when client gets unexpected replies ("id" field not in its
> >> >> > own request queue), the client should just ignore that reply, which
> >> >> > seems natural to me.
> >> >>
> >> >> The trouble is that it may legitimately use the same "id" value for
> >> >> new requests. And I don't see a simple way to handle that without
> >> >> races.
> >> >
> >> > Under what circumstances can it reuse the same ID for new requests?
> >> > Can't we simply tell it not to?
> >>
> >> I don't see any restriction today in the protocol in connecting with a
> >> new client that may not know anything from a previous client.
> >
> > Well, it knows it's doing a reconnection.
> 
> If you assume the "same client" reconnects to the monitor, I agree.
> But this is a restriction of monitor usage.

I think I'm just requiring each monitor that connects to have a unique
set of IDs;  I don't really want the objects that Eric suggests; I'll
just take a string starting with a unique ID.

> >> How would you tell it not to use old IDs? Just by writing an unwritten
> >> rule, because we don't want to fix the per connection client session
> >> handling in qemu?
> >
> > BY writing a written rule!  This out of order stuff we're adding here
> > is a change to the interface and we can define what we require of the
> > client.  As long as what we expect is reasonable then we might end
> > up with something that's simpler for both the client and qemu.
> 
> As long as we don't break existing qmp clients.

Right.

> > And I worry this series keeps getting more and more complex for weird
> > edge cases.
> 
> That's an interesting point-of-view. I see the point in fixing weird
> edge cases in qemu RPC code. More than other code we develop with
> weird edge cases in mind & tests, like the parsing/checking of the
> json schema for ex, in a similar area with the same maintainer.

I'm more worried here about the ability to execute non-blocking
commands; and to be able to do it without rewriting the planet.
If we can avoid having edge-cases by just defining what's required
then I'm happy.

Dave

> > Dave
> >
> >> >
> >> > Dave
> >> >
> >> >> >
> >> >> > Then, if client disconnected and reconnected, it should not have the
> >> >> > problem to generate duplicated id for request, since it should know
> >> >> > what requests it has sent already.  A simplest case I can think of is,
> >> >> > the ID should contains the following tuple:
> >> >>
> >> >> If you assume the "same" client will recover its state, yes.
> >> >>
> >> >> >
> >> >> >   (client name, client unique ID, request ID)
> >> >> >
> >> >> > Here "client name" can be something like "libvirt", which is the name
> >> >> > of client application;
> >> >> >
> >> >> > "client unique ID" can be anything generated when client starts, it
> >> >> > identifies a single client session, maybe a UUID.
> >> >> >
> >> >> > "request ID" can be a unsigned integer starts from zero, and increases
> >> >> > each time the client sends one request.
> >> >>
> >> >> This is introducing  session handling, and can be done in server side
> >> >> only without changes in the protocol I believe.
> >> >>
> >> >> >
> >> >> > I believe current libvirt is using "client name" + "request ID".  It's
> >> >> > something similar (after all I think we don't normally have >1 libvirt
> >> >> > to manage single QEMU, so I think it should be good enough).
> >> >>
> >> >> I am not sure we should base our protocol usage assumptions based on
> >> >> libvirt only, but rather on what is possible today (like queuing
> >> >> requests in the socket etc..).
> >> >>
> >> >> > Then even if client disconnect and reconnect, request ID won't lose,
> >> >> > and no duplication would happen IMHO.
> >> >> >
> >> >> >>
> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > >
> >> >> >> >> > > Then it introduced the "allow-oob" parameter in QAPI schema to define
> >> >> >> >> > > commands, and "run-oob" flag to let oob-allowed command to run in the
> >> >> >> >> > > parser.
> >> >> >> >> >
> >> >> >> >> > From a protocol point of view, I find that "run-oob" distinction per
> >> >> >> >> > command a bit pointless. It helps with legacy client that wouldn't
> >> >> >> >> > expect out-of-order replies if qemu were to run oob commands oob by
> >> >> >> >> > default though.
> >> >> >> >
> >> >> >> > After all oob somehow breaks existing rules or sync execution.  I
> >> >> >> > thought the more important goal was at least to keep the legacy
> >> >> >> > behaviors when adding new things, no?
> >> >> >>
> >> >> >> Of course we have to keep compatibily. What do you mean by "oob
> >> >> >> somehow breaks existing rules or sync execution"? oob means queuing
> >> >> >> and unordered reply support, so clearly this is breaking the current
> >> >> >> "mostly ordered" behaviour (mostly because events may still come any
> >> >> >> time..., and the reconnect issue discussed above).
> >> >> >
> >> >> > Yes.  That's what I mean, it breaks the synchronous scemantic.  But
> >> >> > I should definitely not call it a "break" though since old clients
> >> >> > will work perfectly fine with it.  Sorry for the bad wording.
> >> >> >
> >> >> >>
> >> >> >> >> > Clients shouldn't care about how/where a command is
> >> >> >> >> > being queued or not. If they send a command, they want it processed as
> >> >> >> >> > quickly as possible. However, it can be interesting to know if the
> >> >> >> >> > implementation of the command will be able to deliver oob, so that
> >> >> >> >> > data in the introspection could be useful.
> >> >> >> >> >
> >> >> >> >> > I would rather propose a client/server capability in qmp_capabilities,
> >> >> >> >> > call it "oob":
> >> >> >> >> >
> >> >> >> >> > This capability indicates oob commands support.
> >> >> >> >>
> >> >> >> >> The problem is indicating which commands support oob as opposed to
> >> >> >> >> indicating whether oob is present at all.  Future versions will
> >> >> >> >> probably make more commands oob-able and a client will want to know
> >> >> >> >> whether it can rely on a particular command being non-blocking.
> >> >> >> >
> >> >> >> > Yes.
> >> >> >> >
> >> >> >> > And IMHO we don't urgently need that "whether the server globally
> >> >> >> > supports oob" thing.  Client can just know that from query-qmp-schema
> >> >> >> > already - there will always be the "allow-oob" new field for command
> >> >> >> > typed entries.  IMHO that's a solid hint.
> >> >> >> >
> >> >> >> > But I don't object to return it as well in qmp_capabilities.
> >> >> >>
> >> >> >> Does it feel right that the client can specify how the command are
> >> >> >> processed / queued ? Isn't it preferable to leave that to the server
> >> >> >> to decide? Why would a client specify that? And should the server be
> >> >> >> expected to behave differently? What the client needs to be able is to
> >> >> >> match the unordered replies, and that can be stated during cap
> >> >> >> negotiation / qmp_capabilties. The server is expected to do a best
> >> >> >> effort to handle commands and their priorities. If the client needs
> >> >> >> several command queue, it is simpler to open several connection rather
> >> >> >> than trying to fit that weird priority logic in the protocol imho.
> >> >> >
> >> >> > Sorry I may have missed the point here.  We were discussing about a
> >> >> > global hint for "oob" support, am I right?  Then, could I ask what's
> >> >> > the "weird priority logic" you mentioned?
> >> >>
> >> >> I call per-message oob hint a kind of priority logic, since you can
> >> >> make the same request without oob in the same session and in parallel.
> >> >>
> >> >> >>
> >> >> >> >
> >> >> >> >>
> >> >> >> >> > An oob command is a regular client message request with the "id"
> >> >> >> >> > member mandatory, but the reply may be delivered
> >> >> >> >> > out of order by the server if the client supports
> >> >> >> >> > it too.
> >> >> >> >> >
> >> >> >> >> > If both the server and the client have the "oob" capability, the
> >> >> >> >> > server can handle new client requests while previous requests are being
> >> >> >> >> > processed.
> >> >> >> >> >
> >> >> >> >> > If the client doesn't have the "oob" capability, it may still call
> >> >> >> >> > an oob command, and make multiple outstanding calls. In this case,
> >> >> >> >> > the commands are processed in order, so the replies will also be in
> >> >> >> >> > order. The "id" member isn't mandatory in this case.
> >> >> >> >> >
> >> >> >> >> > The client should match the replies with the "id" member associated
> >> >> >> >> > with the requests.
> >> >> >> >> >
> >> >> >> >> > When a client is disconnected, the pending commands are not
> >> >> >> >> > necessarily cancelled. But the future clients will not get replies from
> >> >> >> >> > commands they didn't make (they might, however, receive side-effects
> >> >> >> >> > events).
> >> >> >> >>
> >> >> >> >> What's the behaviour on the current monitor?
> >> >> >> >
> >> >> >> > Yeah I want to ask the same question, along with questioning about
> >> >> >> > above [1].
> >> >> >> >
> >> >> >> > IMHO this series will not change the behaviors of these, so IMHO the
> >> >> >> > behaviors will be the same before/after this series. E.g., when client
> >> >> >> > dropped right after the command is executed, I think we will still
> >> >> >> > execute the command, though we should encounter something odd in
> >> >> >> > monitor_json_emitter() somewhere when we want to respond.  And it will
> >> >> >> > happen the same after this series.
> >> >> >>
> >> >> >> I think it can get worse after your series, because you queue the
> >> >> >> commands, so clearly a new client can get replies from an old client
> >> >> >> commands. As said above, I am not convinced you can get in that
> >> >> >> situation with current code.
> >> >> >
> >> >> > Hmm, seems so.  But would this a big problem?
> >> >> >
> >> >> > I really think the new client should just throw that response away if
> >> >> > it does not really know that response (from peeking at "id" field),
> >> >> > just like my opinion above.
> >> >>
> >> >> This is a high expectation.
> >> >>
> >> >>
> >> >> --
> >> >> Marc-André Lureau
> >> > --
> >> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >>
> >>
> >>
> >> --
> >> Marc-André Lureau
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> 
> 
> 
> -- 
> Marc-André Lureau
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [RFC 01/15] char-io: fix possible race on IOWatchPoll

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1669 bytes --]

On 09/14/2017 02:50 AM, Peter Xu wrote:
> This is not a problem if we are only having one single loop thread like
> before.  However, after per-monitor thread is introduced, this is not
> true any more, and the race can happen.
> 
> The race can be triggered with "make check -j8" sometimes:
> 
>   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
>   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> 
> This patch keeps the reference for the watch object when creating in
> io_add_watch_poll(), so that the object will never be released in the
> context main loop, especially when the context loop is running in
> another standalone thread.  Meanwhile, when we want to remove the watch
> object, we always first detach the watch object from its owner context,
> then we continue with the cleanup.
> 
> Without this patch, calling io_remove_watch_poll() in main loop thread
> is not thread-safe, since the other per-monitor thread may be modifying
> the watch object at the same time.
> 
> Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---

> +     * Let's blame the glib bug mentioned in commit 2b3167 (again) for

That 6-char commit id may become ambiguous soon (it's still rare to see
ambiguity with an 8-char id, although I've seen it more in recent times
than in the past; and git itself has moved from a 7-char default
abbreviation length in the 2.0 days to what is now a 10-char default
abbreviation in 2.13.5).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC 02/15] qobject: allow NULL for qstring_get_str()

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1475 bytes --]

On 09/14/2017 02:50 AM, Peter Xu wrote:
> Then I can get NULL rather than crash when calling things like:
> 
>   qstring_get_str(qobject_to_qstring(object));
> 
> when key does not exist.

Right now, qdict_get_str() is documented as:

 * This function assumes that 'key' exists and it stores a
 * QString object.

Your code changes that, by making it now return NULL instead of crashing
on what used to be usage in violation of the contract; compared to
qdict_get_try_str() which gracefully returns NULL, but which could use
your new semantics for doing so in fewer lines of code.

I'm not necessarily opposed to the change, but I worry that it has
subtle ramifications that we haven't thought about, as well as
consistency with the rest of the QObject APIs.  It may be better to just
introduce qstring_get_try_str(), which gracefully handles NULL input,
and leave the existing function alone; and if you do introduce a new
helper, it may be worth converting existing clients (perhaps with help
from Coccinelle) to take advantage of the helper.

> +++ b/qobject/qstring.c
> @@ -125,7 +125,7 @@ QString *qobject_to_qstring(const QObject *obj)
>   */
>  const char *qstring_get_str(const QString *qstring)
>  {
> -    return qstring->string;
> +    return qstring ? qstring->string : NULL;
>  }
>  
>  /**
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC 05/15] qjson: add "opaque" field to JSONMessageParser

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2762 bytes --]

On 09/14/2017 02:50 AM, Peter Xu wrote:
> It'll be passed to emit() as well when it happens.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  include/qapi/qmp/json-streamer.h | 8 ++++++--
>  monitor.c                        | 7 ++++---
>  qga/main.c                       | 5 +++--
>  qobject/json-streamer.c          | 7 +++++--
>  qobject/qjson.c                  | 5 +++--
>  tests/libqtest.c                 | 5 +++--
>  6 files changed, 24 insertions(+), 13 deletions(-)
> 
> diff --git a/include/qapi/qmp/json-streamer.h b/include/qapi/qmp/json-streamer.h
> index 00d8a23..b83270c 100644
> --- a/include/qapi/qmp/json-streamer.h
> +++ b/include/qapi/qmp/json-streamer.h
> @@ -25,16 +25,20 @@ typedef struct JSONToken {
>  
>  typedef struct JSONMessageParser
>  {
> -    void (*emit)(struct JSONMessageParser *parser, GQueue *tokens);
> +    void (*emit)(struct JSONMessageParser *parser, GQueue *tokens, void *opaque);
>      JSONLexer lexer;
>      int brace_count;
>      int bracket_count;
>      GQueue *tokens;
>      uint64_t token_size;
> +    /* To be passed in when emit(). */

Reads awkwardly, better might be: /* Passed to emit() */

I might group void *opaque right next to emit, rather than separated by
the rest of the struct, to make it obvious they are related, at which
point the comment isn't necessary.

> +    void *opaque;
>  } JSONMessageParser;
>  
>  void json_message_parser_init(JSONMessageParser *parser,
> -                              void (*func)(JSONMessageParser *, GQueue *));
> +                              void (*func)(JSONMessageParser *, GQueue *,
> +                                           void *opaque),

Inconsistent to name only one of the three parameters in the inner
function pointer type for the outer parameter 'func'.  I wonder if a
typedef would make things more legible.

> +                              void *opaque);
>  

> +++ b/qobject/json-streamer.c
> @@ -102,18 +102,21 @@ out_emit:
>       */
>      tokens = parser->tokens;
>      parser->tokens = g_queue_new();
> -    parser->emit(parser, tokens);
> +    parser->emit(parser, tokens, parser->opaque);
>      parser->token_size = 0;
>  }
>  
>  void json_message_parser_init(JSONMessageParser *parser,
> -                              void (*func)(JSONMessageParser *, GQueue *))
> +                              void (*func)(JSONMessageParser *,
> +                                           GQueue *, void *opaque),

Again, inconsistent that you named only 1 of the three inner parameters.

Overall, the idea looks reasonable.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC 06/15] monitor: move the cur_mon hack deeper for QMP

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 935 bytes --]

On 09/14/2017 02:50 AM, Peter Xu wrote:
> In monitor_qmp_read(), we have the hack to temporarily replace the
> cur_mon pointer.  Now we move this hack deeper inside the QMP dispatcher
> routine since the Monitor pointer can be passed in to that using the new
> JSON Parser opaque field now.
> 
> This does not make much sense as a single patch.  However, this will be
> a big step for the next patch, when the QMP dispatcher routine will be
> splitted from the QMP parser.

English is weird: 'split' is the word for present, past, and past
participle tenses; there is no word 'splitted'.

> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  monitor.c | 19 ++++++++++---------
>  1 file changed, 10 insertions(+), 9 deletions(-)
> 

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC 07/15] monitor: unify global init

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2608 bytes --]

On 09/14/2017 02:50 AM, Peter Xu wrote:
> There are many places for monitor init its globals, at least:
> 
> - monitor_init_qmp_commands() at the very beginning
> - single function to init monitor_lock
> - in the first entry of monitor_init() using "is_first_init"
> 
> Unify them a bit.
> 
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  include/monitor/monitor.h |  2 +-
>  monitor.c                 | 25 ++++++++++---------------
>  vl.c                      |  3 ++-
>  3 files changed, 13 insertions(+), 17 deletions(-)
> 

>  
> +void monitor_init_globals(void)
> +{
> +    monitor_init_qmp_commands();
> +    monitor_qapi_event_init();
> +    sortcmdlist();
> +    qemu_mutex_init(&monitor_lock);
> +}

Are we sure that this new function is called sooner than any access to
monitor_lock,

> -static void __attribute__((constructor)) monitor_lock_init(void)
> -{
> -    qemu_mutex_init(&monitor_lock);
> -}

especially since the old code initialized the lock REALLY early?

> diff --git a/vl.c b/vl.c
> index fb1f05b..850cf55 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -3049,7 +3049,6 @@ int main(int argc, char **argv, char **envp)
>      qemu_init_exec_dir(argv[0]);
>  
>      module_call_init(MODULE_INIT_QOM);
> -    monitor_init_qmp_commands();
>  
>      qemu_add_opts(&qemu_drive_opts);
>      qemu_add_drive_opts(&qemu_legacy_drive_opts);
> @@ -4587,6 +4586,8 @@ int main(int argc, char **argv, char **envp)
>  
>      parse_numa_opts(current_machine);
>  
> +    monitor_init_globals();
> +

Pre-patch, a breakpoint on main() and on monitor_lock_init() fires on
monitor_lock_init() first, prior to main.

Breakpoint 2, monitor_lock_init () at /home/eblake/qemu/monitor.c:4089
4089	    qemu_mutex_init(&monitor_lock);
(gdb) c
Continuing.
[New Thread 0x7fffce225700 (LWP 26380)]

Thread 1 "qemu-system-x86" hit Breakpoint 1, main (argc=5,
    argv=0x7fffffffdc88, envp=0x7fffffffdcb8) at vl.c:3077
3077	{

Post-patch, the mutex is not initialized until well after main().  So
the real question is what (if anything) is using the lock in between
those two points?

Hmm - it may be that we needed it back before commit 05875687, when we
really did depend on MODULE_INIT_QAPI, but it is something we forgot to
cleanup in the meantime?

If nothing else, the commit message should call out that dropping
__attribute__((constructor)) nonsense is intentional (if it was indeed
nonsense).

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC 07/15] monitor: unify global init

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 3250 bytes --]

On 09/19/2017 04:35 PM, Eric Blake wrote:
> On 09/14/2017 02:50 AM, Peter Xu wrote:
>> There are many places for monitor init its globals, at least:
>>

> Are we sure that this new function is called sooner than any access to
> monitor_lock,
> 
>> -static void __attribute__((constructor)) monitor_lock_init(void)
>> -{
>> -    qemu_mutex_init(&monitor_lock);
>> -}
> 
> especially since the old code initialized the lock REALLY early?

(Partially) answering myself:

> 
> Pre-patch, a breakpoint on main() and on monitor_lock_init() fires on
> monitor_lock_init() first, prior to main.
> 
> Breakpoint 2, monitor_lock_init () at /home/eblake/qemu/monitor.c:4089
> 4089	    qemu_mutex_init(&monitor_lock);
> (gdb) c
> Continuing.
> [New Thread 0x7fffce225700 (LWP 26380)]
> 
> Thread 1 "qemu-system-x86" hit Breakpoint 1, main (argc=5,
>     argv=0x7fffffffdc88, envp=0x7fffffffdcb8) at vl.c:3077
> 3077	{

Also, pre-patch, 'watch monitor_lock.initialized' and 'watch
monitor_lock.lock.__data.__lock' show that the lock is first utilized at:

(gdb) bt
#0  0x00007fffdac59e12 in __GI___pthread_mutex_lock
(mutex=0x555556399340 <monitor_lock>) at ../nptl/pthread_mutex_lock.c:80
#1  0x0000555555ce01ed in qemu_mutex_lock (mutex=0x555556399340
<monitor_lock>)
    at util/qemu-thread-posix.c:65
#2  0x00005555557bc8b8 in monitor_init (chr=0x55555690bf70, flags=4)
    at /home/eblake/qemu/monitor.c:4126
#3  0x000055555591ae80 in mon_init_func (opaque=0x0,
opts=0x55555688e3d0, errp=0x0) at vl.c:2482
#4  0x0000555555cf3e63 in qemu_opts_foreach (list=0x555556225200
<qemu_mon_opts>, func=0x55555591ad33 <mon_init_func>, opaque=0x0, errp=0x0)
    at util/qemu-option.c:1104
#5  0x0000555555920128 in main (argc=5, argv=0x7fffffffdc88,
envp=0x7fffffffdcb8) at vl.c:4670

and double-checking qemu_mutex_lock, our .initialized member provides
NICE runtime checking that we don't use an uninitialized lock.  So the
fact that your patch doesn't assert means your later initialization is
still fine.

[TIL: the gdb 'watch' command is cool, but it's better if you watch only
4 or 8 bytes at a time; I first tried 'watch monitor_lock', but that's
hundreds of times slower as hardware can't watch that much data at once,
at which point gdb emulates it by single-stepping the entire program]

> 
> Post-patch, the mutex is not initialized until well after main().  So
> the real question is what (if anything) is using the lock in between
> those two points?

According to gdb watchpoints, no.

> 
> Hmm - it may be that we needed it back before commit 05875687, when we
> really did depend on MODULE_INIT_QAPI, but it is something we forgot to
> cleanup in the meantime?

So what I didn't debug was whether the constructor attribute was
mandatory in the past, and if so, which commit made it no longer
mandatory (my mention of commit 05875687 is only a guess).

> 
> If nothing else, the commit message should call out that dropping
> __attribute__((constructor)) nonsense is intentional (if it was indeed
> nonsense).
> 

This part is still true.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 619 bytes --]

Re: [Qemu-devel] [RFC 00/15] QMP: out-of-band (OOB) execution support

[Peter Xu]

On Tue, Sep 19, 2017 at 10:19:21AM +0100, Dr. David Alan Gilbert wrote:
> * Peter Xu (peterx@redhat.com) wrote:
> > On Mon, Sep 18, 2017 at 06:09:29PM +0200, Marc-André Lureau wrote:
> > > On Mon, Sep 18, 2017 at 1:26 PM, Dr. David Alan Gilbert
> > > <dgilbert@redhat.com> wrote:
> > > > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> > > >> Hi
> > > >>
> > > >> On Mon, Sep 18, 2017 at 12:55 PM, Dr. David Alan Gilbert
> > > >> <dgilbert@redhat.com> wrote:
> > > >> > * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> > > >> >> Hi
> > > >> >>
> > > >> >> On Mon, Sep 18, 2017 at 10:37 AM, Peter Xu <peterx@redhat.com> wrote:
> > > >> >> > On Fri, Sep 15, 2017 at 01:14:47PM +0200, Marc-André Lureau wrote:
> > > >> >> >> Hi
> > > >> >> >>
> > > >> >> >> On Thu, Sep 14, 2017 at 9:46 PM, Peter Xu <peterx@redhat.com> wrote:
> > > >> >> >> > On Thu, Sep 14, 2017 at 07:53:15PM +0100, Dr. David Alan Gilbert wrote:
> > > >> >> >> >> * Marc-André Lureau (marcandre.lureau@gmail.com) wrote:
> > > >> >> >> >> > Hi
> > > >> >> >> >> >
> > > >> >> >> >> > On Thu, Sep 14, 2017 at 9:50 AM, Peter Xu <peterx@redhat.com> wrote:
> > > >> >> >> >> > > This series was born from this one:
> > > >> >> >> >> > >
> > > >> >> >> >> > >   https://lists.gnu.org/archive/html/qemu-devel/2017-08/msg04310.html
> > > >> >> >> >> > >
> > > >> >> >> >> > > The design comes from Markus, and also the whole-bunch-of discussions
> > > >> >> >> >> > > in previous thread.  My heartful thanks to Markus, Daniel, Dave,
> > > >> >> >> >> > > Stefan, etc. on discussing the topic (...again!), providing shiny
> > > >> >> >> >> > > ideas and suggestions.  Finally we got such a solution that seems to
> > > >> >> >> >> > > satisfy everyone.
> > > >> >> >> >> > >
> > > >> >> >> >> > > I re-started the versioning since this series is totally different
> > > >> >> >> >> > > from previous one.  Now it's version 1.
> > > >> >> >> >> > >
> > > >> >> >> >> > > In case new reviewers come along the way without reading previous
> > > >> >> >> >> > > discussions, I will try to do a summary on what this is all about.
> > > >> >> >> >> > >
> > > >> >> >> >> > > What is OOB execution?
> > > >> >> >> >> > > ======================
> > > >> >> >> >> > >
> > > >> >> >> >> > > It's the shortcut of Out-Of-Band execution, its name is given by
> > > >> >> >> >> > > Markus.  It's a way to quickly execute a QMP request.  Say, originally
> > > >> >> >> >> > > QMP is going throw these steps:
> > > >> >> >> >> > >
> > > >> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> > > >> >> >> >> > >           /|\    (2)                (3)     |
> > > >> >> >> >> > >        (1) |                               \|/ (4)
> > > >> >> >> >> > >            +---------  main thread  --------+
> > > >> >> >> >> > >
> > > >> >> >> >> > > The requests are executed by the so-called QMP-dispatcher after the
> > > >> >> >> >> > > JSON is parsed.  If OOB is on, we run the command directly in the
> > > >> >> >> >> > > parser and quickly returns.
> > > >> >> >> >> >
> > > >> >> >> >> > All commands should have the "id" field mandatory in this case, else
> > > >> >> >> >> > the client will not distinguish the replies coming from the last/oob
> > > >> >> >> >> > and the previous commands.
> > > >> >> >> >> >
> > > >> >> >> >> > This should probably be enforced upfront by client capability checks,
> > > >> >> >> >> > more below.
> > > >> >> >> >
> > > >> >> >> > Hmm yes since the oob commands are actually running in async way,
> > > >> >> >> > request ID should be needed here.  However I'm not sure whether
> > > >> >> >> > enabling the whole "request ID" thing is too big for this "try to be
> > > >> >> >> > small" oob change... And IMHO it suites better to be part of the whole
> > > >> >> >> > async work (no matter which implementation we'll use).
> > > >> >> >> >
> > > >> >> >> > How about this: we make "id" mandatory for "run-oob" requests only.
> > > >> >> >> > For oob commands, they will always have ID then no ordering issue, and
> > > >> >> >> > we can do it async; for the rest of non-oob commands, we still allow
> > > >> >> >> > them to go without ID, and since they are not oob, they'll always be
> > > >> >> >> > done in order as well.  Would this work?
> > > >> >> >>
> > > >> >> >> This mixed-mode is imho more complicated to deal with than having the
> > > >> >> >> protocol enforced one way or the other, but that should work.
> > > >> >> >>
> > > >> >> >> >
> > > >> >> >> >> >
> > > >> >> >> >> > > Yeah I know in current code the parser calls dispatcher directly
> > > >> >> >> >> > > (please see handle_qmp_command()).  However it's not true again after
> > > >> >> >> >> > > this series (parser will has its own IO thread, and dispatcher will
> > > >> >> >> >> > > still be run in main thread).  So this OOB does brings something
> > > >> >> >> >> > > different.
> > > >> >> >> >> > >
> > > >> >> >> >> > > There are more details on why OOB and the difference/relationship
> > > >> >> >> >> > > between OOB, async QMP, block/general jobs, etc.. but IMHO that's
> > > >> >> >> >> > > slightly out of topic (and believe me, it's not easy for me to
> > > >> >> >> >> > > summarize that).  For more information, please refers to [1].
> > > >> >> >> >> > >
> > > >> >> >> >> > > Summary ends here.
> > > >> >> >> >> > >
> > > >> >> >> >> > > Some Implementation Details
> > > >> >> >> >> > > ===========================
> > > >> >> >> >> > >
> > > >> >> >> >> > > Again, I mentioned that the old QMP workflow is this:
> > > >> >> >> >> > >
> > > >> >> >> >> > >       JSON Parser --> QMP Dispatcher --> Respond
> > > >> >> >> >> > >           /|\    (2)                (3)     |
> > > >> >> >> >> > >        (1) |                               \|/ (4)
> > > >> >> >> >> > >            +---------  main thread  --------+
> > > >> >> >> >> > >
> > > >> >> >> >> > > What this series does is, firstly:
> > > >> >> >> >> > >
> > > >> >> >> >> > >       JSON Parser     QMP Dispatcher --> Respond
> > > >> >> >> >> > >           /|\ |           /|\       (4)     |
> > > >> >> >> >> > >            |  | (2)        | (3)            |  (5)
> > > >> >> >> >> > >        (1) |  +----->      |               \|/
> > > >> >> >> >> > >            +---------  main thread  <-------+
> > > >> >> >> >> > >
> > > >> >> >> >> > > And further:
> > > >> >> >> >> > >
> > > >> >> >> >> > >                queue/kick
> > > >> >> >> >> > >      JSON Parser ======> QMP Dispatcher --> Respond
> > > >> >> >> >> > >          /|\ |     (3)       /|\        (4)    |
> > > >> >> >> >> > >       (1) |  | (2)            |                |  (5)
> > > >> >> >> >> > >           | \|/               |               \|/
> > > >> >> >> >> > >         IO thread         main thread  <-------+
> > > >> >> >> >> >
> > > >> >> >> >> > Is the queue per monitor or per client?
> > > >> >> >> >
> > > >> >> >> > The queue is currently global. I think yes maybe at least we can do it
> > > >> >> >> > per monitor, but I am not sure whether that is urgent or can be
> > > >> >> >> > postponed.  After all now QMPRequest (please refer to patch 11) is
> > > >> >> >> > defined as (mon, id, req) tuple, so at least "id" namespace is
> > > >> >> >> > per-monitor.
> > > >> >> >> >
> > > >> >> >> >> > And is the dispatching going
> > > >> >> >> >> > to be processed even if the client is disconnected, and are new
> > > >> >> >> >> > clients going to receive the replies from previous clients
> > > >> >> >> >> > commands?
> > > >> >> >> >
> > > >> >> >> > [1]
> > > >> >> >> >
> > > >> >> >> > (will discuss together below)
> > > >> >> >> >
> > > >> >> >> >> > I
> > > >> >> >> >> > believe there should be a per-client context, so there won't be "id"
> > > >> >> >> >> > request conflicts.
> > > >> >> >> >
> > > >> >> >> > I'd say I am not familiar with this "client" idea, since after all
> > > >> >> >> > IMHO one monitor is currently designed to mostly work with a single
> > > >> >> >> > client. Say, unix sockets, telnet, all these backends are only single
> > > >> >> >> > channeled, and one monitor instance can only work with one client at a
> > > >> >> >> > time.  Then do we really need to add this client layer upon it?  IMHO
> > > >> >> >> > the user can just provide more monitors if they wants more clients
> > > >> >> >> > (and at least these clients should know the existance of the others or
> > > >> >> >> > there might be problem, otherwise user2 will fail a migration, finally
> > > >> >> >> > noticed that user1 has already triggered one), and the user should
> > > >> >> >> > manage them well.
> > > >> >> >>
> > > >> >> >> qemu should support a management layer / libvirt restart/reconnect.
> > > >> >> >> Afaik, it mostly work today. There might be a cases where libvirt can
> > > >> >> >> be confused if it receives a reply from a previous connection command,
> > > >> >> >> but due to the sync processing of the chardev, I am not sure you can
> > > >> >> >> get in this situation.  By adding "oob" commands and queuing, the
> > > >> >> >> client will have to remember which was the last "id" used, or it will
> > > >> >> >> create more conflict after a reconnect.
> > > >> >> >>
> > > >> >> >> Imho we should introduce the client/connection concept to avoid this
> > > >> >> >> confusion (unexpected reply & per client id space).
> > > >> >> >
> > > >> >> > Hmm I agree that the reconnect feature would be nice, but if so IMHO
> > > >> >> > instead of throwing responses away when client disconnect, we should
> > > >> >> > really keep them, and when the client reconnects, we queue the
> > > >> >> > responses again.
> > > >> >> >
> > > >> >> > I think we have other quite simple ways to solve the "unexpected
> > > >> >> > reply" and "per-client-id duplication" issues you have mentioned.
> > > >> >> >
> > > >> >> > Firstly, when client gets unexpected replies ("id" field not in its
> > > >> >> > own request queue), the client should just ignore that reply, which
> > > >> >> > seems natural to me.
> > > >> >>
> > > >> >> The trouble is that it may legitimately use the same "id" value for
> > > >> >> new requests. And I don't see a simple way to handle that without
> > > >> >> races.
> > > >> >
> > > >> > Under what circumstances can it reuse the same ID for new requests?
> > > >> > Can't we simply tell it not to?
> > > >>
> > > >> I don't see any restriction today in the protocol in connecting with a
> > > >> new client that may not know anything from a previous client.
> > > >
> > > > Well, it knows it's doing a reconnection.
> > > 
> > > If you assume the "same client" reconnects to the monitor, I agree.
> > > But this is a restriction of monitor usage.
> > 
> > In monitor_qmp_event(), we can empty the request queue when got
> > CHR_EVENT_CLOSED.  Would that be a solution?
> 
> What happens to commands that are in flight?

Good questioning...

I think we can track that one as well, say, provide a simple state
machine for Monitor (possibly with a lock) that can be either "idle",
"processing", "drop".

Then a normal routine to execution of command:

0. by default, monitor state "idle"
1. when dequeue the request, mark that monitor as "processing",
   execute the command
2. when reply: if still "processing", then do it; if "drop", then drop
   that reply for current command.  Here we'll reply.

Instead, if disconnect/reconnect happens:

0. by default, monitor state "idle"
1. when dequeue the request, mark that monitor as "processing",
   execute the command
2. port disconnected, in EVENT_CLOSED, we set state to "drop"
3. port reconnected, we do nothing (so the execution state keeps
   through reconnection)
4. when reply: if still "processing", then do it; if "drop", then drop
   that reply for current command.  Here we drop that reply.

But... IMHO this is too awkward only for this single "drop the last
command reply" purpose.  I would prefer to use documentation intead to
let client drop unknown responses directly if it's ok to everyone.

Thanks,

> 
> Dave
> 
> > -- 
> > Peter Xu
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

-- 
Peter Xu

Re: [Qemu-devel] [RFC 01/15] char-io: fix possible race on IOWatchPoll

[Peter Xu]

On Tue, Sep 19, 2017 at 02:59:37PM -0500, Eric Blake wrote:
> On 09/14/2017 02:50 AM, Peter Xu wrote:
> > This is not a problem if we are only having one single loop thread like
> > before.  However, after per-monitor thread is introduced, this is not
> > true any more, and the race can happen.
> > 
> > The race can be triggered with "make check -j8" sometimes:
> > 
> >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > 
> > This patch keeps the reference for the watch object when creating in
> > io_add_watch_poll(), so that the object will never be released in the
> > context main loop, especially when the context loop is running in
> > another standalone thread.  Meanwhile, when we want to remove the watch
> > object, we always first detach the watch object from its owner context,
> > then we continue with the cleanup.
> > 
> > Without this patch, calling io_remove_watch_poll() in main loop thread
> > is not thread-safe, since the other per-monitor thread may be modifying
> > the watch object at the same time.
> > 
> > Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> 
> > +     * Let's blame the glib bug mentioned in commit 2b3167 (again) for
> 
> That 6-char commit id may become ambiguous soon (it's still rare to see
> ambiguity with an 8-char id, although I've seen it more in recent times
> than in the past; and git itself has moved from a 7-char default
> abbreviation length in the 2.0 days to what is now a 10-char default
> abbreviation in 2.13.5).

Thanks for noticing this.  I'll use 10 chars in next post.

-- 
Peter Xu

Re: [Qemu-devel] [RFC 02/15] qobject: allow NULL for qstring_get_str()

[Peter Xu]

On Tue, Sep 19, 2017 at 03:48:35PM -0500, Eric Blake wrote:
> On 09/14/2017 02:50 AM, Peter Xu wrote:
> > Then I can get NULL rather than crash when calling things like:
> > 
> >   qstring_get_str(qobject_to_qstring(object));
> > 
> > when key does not exist.
> 
> Right now, qdict_get_str() is documented as:
> 
>  * This function assumes that 'key' exists and it stores a
>  * QString object.
> 
> Your code changes that, by making it now return NULL instead of crashing
> on what used to be usage in violation of the contract; compared to
> qdict_get_try_str() which gracefully returns NULL, but which could use
> your new semantics for doing so in fewer lines of code.
> 
> I'm not necessarily opposed to the change, but I worry that it has
> subtle ramifications that we haven't thought about, as well as
> consistency with the rest of the QObject APIs.  It may be better to just
> introduce qstring_get_try_str(), which gracefully handles NULL input,
> and leave the existing function alone; and if you do introduce a new
> helper, it may be worth converting existing clients (perhaps with help
> from Coccinelle) to take advantage of the helper.

I'll take your suggestion.

Though I didn't see much existing codes that can use the new
qstring_get_try_str().  One I found is object_property_get_str().
I can add one more patch to convert its usage.

Thanks,

> 
> > +++ b/qobject/qstring.c
> > @@ -125,7 +125,7 @@ QString *qobject_to_qstring(const QObject *obj)
> >   */
> >  const char *qstring_get_str(const QString *qstring)
> >  {
> > -    return qstring->string;
> > +    return qstring ? qstring->string : NULL;
> >  }
> >  
> >  /**
> > 
> 
> -- 
> Eric Blake, Principal Software Engineer
> Red Hat, Inc.           +1-919-301-3266
> Virtualization:  qemu.org | libvirt.org
> 

-- 
Peter Xu

Re: [Qemu-devel] [RFC 05/15] qjson: add "opaque" field to JSONMessageParser

[Peter Xu]

On Tue, Sep 19, 2017 at 03:55:41PM -0500, Eric Blake wrote:
> On 09/14/2017 02:50 AM, Peter Xu wrote:
> > It'll be passed to emit() as well when it happens.
> > 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  include/qapi/qmp/json-streamer.h | 8 ++++++--
> >  monitor.c                        | 7 ++++---
> >  qga/main.c                       | 5 +++--
> >  qobject/json-streamer.c          | 7 +++++--
> >  qobject/qjson.c                  | 5 +++--
> >  tests/libqtest.c                 | 5 +++--
> >  6 files changed, 24 insertions(+), 13 deletions(-)
> > 
> > diff --git a/include/qapi/qmp/json-streamer.h b/include/qapi/qmp/json-streamer.h
> > index 00d8a23..b83270c 100644
> > --- a/include/qapi/qmp/json-streamer.h
> > +++ b/include/qapi/qmp/json-streamer.h
> > @@ -25,16 +25,20 @@ typedef struct JSONToken {
> >  
> >  typedef struct JSONMessageParser
> >  {
> > -    void (*emit)(struct JSONMessageParser *parser, GQueue *tokens);
> > +    void (*emit)(struct JSONMessageParser *parser, GQueue *tokens, void *opaque);
> >      JSONLexer lexer;
> >      int brace_count;
> >      int bracket_count;
> >      GQueue *tokens;
> >      uint64_t token_size;
> > +    /* To be passed in when emit(). */
> 
> Reads awkwardly, better might be: /* Passed to emit() */
> 
> I might group void *opaque right next to emit, rather than separated by
> the rest of the struct, to make it obvious they are related, at which
> point the comment isn't necessary.

Will remove that comment and move it upwards to emit().

> 
> > +    void *opaque;
> >  } JSONMessageParser;
> >  
> >  void json_message_parser_init(JSONMessageParser *parser,
> > -                              void (*func)(JSONMessageParser *, GQueue *));
> > +                              void (*func)(JSONMessageParser *, GQueue *,
> > +                                           void *opaque),
> 
> Inconsistent to name only one of the three parameters in the inner
> function pointer type for the outer parameter 'func'.  I wonder if a
> typedef would make things more legible.

Yep, will do.

> 
> > +                              void *opaque);
> >  
> 
> > +++ b/qobject/json-streamer.c
> > @@ -102,18 +102,21 @@ out_emit:
> >       */
> >      tokens = parser->tokens;
> >      parser->tokens = g_queue_new();
> > -    parser->emit(parser, tokens);
> > +    parser->emit(parser, tokens, parser->opaque);
> >      parser->token_size = 0;
> >  }
> >  
> >  void json_message_parser_init(JSONMessageParser *parser,
> > -                              void (*func)(JSONMessageParser *, GQueue *))
> > +                              void (*func)(JSONMessageParser *,
> > +                                           GQueue *, void *opaque),
> 
> Again, inconsistent that you named only 1 of the three inner parameters.

Fixing up.

> 
> Overall, the idea looks reasonable.

Thanks!

-- 
Peter Xu

Re: [Qemu-devel] [RFC 06/15] monitor: move the cur_mon hack deeper for QMP

[Peter Xu]

On Tue, Sep 19, 2017 at 04:05:48PM -0500, Eric Blake wrote:
> On 09/14/2017 02:50 AM, Peter Xu wrote:
> > In monitor_qmp_read(), we have the hack to temporarily replace the
> > cur_mon pointer.  Now we move this hack deeper inside the QMP dispatcher
> > routine since the Monitor pointer can be passed in to that using the new
> > JSON Parser opaque field now.
> > 
> > This does not make much sense as a single patch.  However, this will be
> > a big step for the next patch, when the QMP dispatcher routine will be
> > splitted from the QMP parser.
> 
> English is weird: 'split' is the word for present, past, and past
> participle tenses; there is no word 'splitted'.

Fixed.

> 
> > 
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  monitor.c | 19 ++++++++++---------
> >  1 file changed, 10 insertions(+), 9 deletions(-)
> > 
> 
> Reviewed-by: Eric Blake <eblake@redhat.com>

Thanks!

-- 
Peter Xu

Re: [Qemu-devel] [RFC 07/15] monitor: unify global init

[Peter Xu]

On Tue, Sep 19, 2017 at 04:48:35PM -0500, Eric Blake wrote:
> On 09/19/2017 04:35 PM, Eric Blake wrote:
> > On 09/14/2017 02:50 AM, Peter Xu wrote:
> >> There are many places for monitor init its globals, at least:
> >>
> 
> > Are we sure that this new function is called sooner than any access to
> > monitor_lock,
> > 
> >> -static void __attribute__((constructor)) monitor_lock_init(void)
> >> -{
> >> -    qemu_mutex_init(&monitor_lock);
> >> -}
> > 
> > especially since the old code initialized the lock REALLY early?
> 
> (Partially) answering myself:
> 
> > 
> > Pre-patch, a breakpoint on main() and on monitor_lock_init() fires on
> > monitor_lock_init() first, prior to main.
> > 
> > Breakpoint 2, monitor_lock_init () at /home/eblake/qemu/monitor.c:4089
> > 4089	    qemu_mutex_init(&monitor_lock);
> > (gdb) c
> > Continuing.
> > [New Thread 0x7fffce225700 (LWP 26380)]
> > 
> > Thread 1 "qemu-system-x86" hit Breakpoint 1, main (argc=5,
> >     argv=0x7fffffffdc88, envp=0x7fffffffdcb8) at vl.c:3077
> > 3077	{
> 
> Also, pre-patch, 'watch monitor_lock.initialized' and 'watch
> monitor_lock.lock.__data.__lock' show that the lock is first utilized at:
> 
> (gdb) bt
> #0  0x00007fffdac59e12 in __GI___pthread_mutex_lock
> (mutex=0x555556399340 <monitor_lock>) at ../nptl/pthread_mutex_lock.c:80
> #1  0x0000555555ce01ed in qemu_mutex_lock (mutex=0x555556399340
> <monitor_lock>)
>     at util/qemu-thread-posix.c:65
> #2  0x00005555557bc8b8 in monitor_init (chr=0x55555690bf70, flags=4)
>     at /home/eblake/qemu/monitor.c:4126
> #3  0x000055555591ae80 in mon_init_func (opaque=0x0,
> opts=0x55555688e3d0, errp=0x0) at vl.c:2482
> #4  0x0000555555cf3e63 in qemu_opts_foreach (list=0x555556225200
> <qemu_mon_opts>, func=0x55555591ad33 <mon_init_func>, opaque=0x0, errp=0x0)
>     at util/qemu-option.c:1104
> #5  0x0000555555920128 in main (argc=5, argv=0x7fffffffdc88,
> envp=0x7fffffffdcb8) at vl.c:4670
> 
> and double-checking qemu_mutex_lock, our .initialized member provides
> NICE runtime checking that we don't use an uninitialized lock.  So the
> fact that your patch doesn't assert means your later initialization is
> still fine.

Yeah, that's something I liked as well.

> 
> [TIL: the gdb 'watch' command is cool, but it's better if you watch only
> 4 or 8 bytes at a time; I first tried 'watch monitor_lock', but that's
> hundreds of times slower as hardware can't watch that much data at once,
> at which point gdb emulates it by single-stepping the entire program]

Good to learn it!

Thanks for digging the whole thing up.

> 
> > 
> > Post-patch, the mutex is not initialized until well after main().  So
> > the real question is what (if anything) is using the lock in between
> > those two points?
> 
> According to gdb watchpoints, no.
> 
> > 
> > Hmm - it may be that we needed it back before commit 05875687, when we
> > really did depend on MODULE_INIT_QAPI, but it is something we forgot to
> > cleanup in the meantime?
> 
> So what I didn't debug was whether the constructor attribute was
> mandatory in the past, and if so, which commit made it no longer
> mandatory (my mention of commit 05875687 is only a guess).
> 
> > 
> > If nothing else, the commit message should call out that dropping
> > __attribute__((constructor)) nonsense is intentional (if it was indeed
> > nonsense).
> > 
> 
> This part is still true.

If this patch is doable, I'll add explicit reason to commit message.

Paolo/Markus, would any of you help confirm this change? (considering
Paolo introduced commit d622cb587)

One thing I slightly not sure of is that, some device realization has
this code path (take fsl_imx25_realize() as example):

  fsl_imx25_realize
    qemu_chr_new
      qemu_chr_new_noreplay
        char is_mux?
          monitor_init

(note: I never know why we create the monitor in chardev
 creation... would there be a better place?)

Especially considering some integrated devices can be created along
with machine init.

Anyway, this patch was trying to cleanup the things a bit, and also
more convenient for me to add new codes upon.  If any of us think it's
not safe enough, please say explicitly, and I can drop it and do the
rest in "the ugly way".

Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC 01/15] char-io: fix possible race on IOWatchPoll

[Daniel P. Berrange]

On Thu, Sep 14, 2017 at 03:50:22PM +0800, Peter Xu wrote:
> This is not a problem if we are only having one single loop thread like
> before.  However, after per-monitor thread is introduced, this is not
> true any more, and the race can happen.
> 
> The race can be triggered with "make check -j8" sometimes:
> 
>   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
>   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> 
> This patch keeps the reference for the watch object when creating in
> io_add_watch_poll(), so that the object will never be released in the
> context main loop, especially when the context loop is running in
> another standalone thread.  Meanwhile, when we want to remove the watch
> object, we always first detach the watch object from its owner context,
> then we continue with the cleanup.
> 
> Without this patch, calling io_remove_watch_poll() in main loop thread
> is not thread-safe, since the other per-monitor thread may be modifying
> the watch object at the same time.

This doesn't feel right to me. Why is the main loop thread doing anything
at all with the Chardev, if there is a per-monitor thread ? The Chardev
code isn't thread safe so it isn't safe to have two separate threads
accessing the same Chardev. IOW, if we want a per-monitor thread, then
we must make sure the main thread never touches that monitor's chardev
at all.  While your patch here might have avoided the assertion you
mention above, I fear this is just papering over a fundamental problem
that still exists, that can only be solved by not letting the mainloop
touch the chardev at all.

> 
> Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> Signed-off-by: Peter Xu <peterx@redhat.com>
> ---
>  chardev/char-io.c | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/chardev/char-io.c b/chardev/char-io.c
> index f810524..3828c20 100644
> --- a/chardev/char-io.c
> +++ b/chardev/char-io.c
> @@ -122,7 +122,6 @@ GSource *io_add_watch_poll(Chardev *chr,
>      g_free(name);
>  
>      g_source_attach(&iwp->parent, context);
> -    g_source_unref(&iwp->parent);
>      return (GSource *)iwp;
>  }
>  
> @@ -131,12 +130,24 @@ static void io_remove_watch_poll(GSource *source)
>      IOWatchPoll *iwp;
>  
>      iwp = io_watch_poll_from_source(source);
> +
> +    /*
> +     * Here the order of destruction really matters.  We need to first
> +     * detach the IOWatchPoll object from the context (which may still
> +     * be running in another loop thread), only after that could we
> +     * continue to operate on iwp->src, or there may be race condition
> +     * between current thread and the context loop thread.
> +     *
> +     * Let's blame the glib bug mentioned in commit 2b3167 (again) for
> +     * this extra complexity.
> +     */
> +    g_source_destroy(&iwp->parent);
>      if (iwp->src) {
>          g_source_destroy(iwp->src);
>          g_source_unref(iwp->src);
>          iwp->src = NULL;
>      }
> -    g_source_destroy(&iwp->parent);
> +    g_source_unref(&iwp->parent);
>  }
>  
>  void remove_fd_in_watch(Chardev *chr)
> -- 
> 2.7.4
> 

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC 01/15] char-io: fix possible race on IOWatchPoll

[Peter Xu]

On Wed, Sep 20, 2017 at 08:57:03AM +0100, Daniel P. Berrange wrote:
> On Thu, Sep 14, 2017 at 03:50:22PM +0800, Peter Xu wrote:
> > This is not a problem if we are only having one single loop thread like
> > before.  However, after per-monitor thread is introduced, this is not
> > true any more, and the race can happen.
> > 
> > The race can be triggered with "make check -j8" sometimes:
> > 
> >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > 
> > This patch keeps the reference for the watch object when creating in
> > io_add_watch_poll(), so that the object will never be released in the
> > context main loop, especially when the context loop is running in
> > another standalone thread.  Meanwhile, when we want to remove the watch
> > object, we always first detach the watch object from its owner context,
> > then we continue with the cleanup.
> > 
> > Without this patch, calling io_remove_watch_poll() in main loop thread
> > is not thread-safe, since the other per-monitor thread may be modifying
> > the watch object at the same time.
> 
> This doesn't feel right to me. Why is the main loop thread doing anything
> at all with the Chardev, if there is a per-monitor thread ? The Chardev
> code isn't thread safe so it isn't safe to have two separate threads
> accessing the same Chardev. IOW, if we want a per-monitor thread, then
> we must make sure the main thread never touches that monitor's chardev
> at all.  While your patch here might have avoided the assertion you
> mention above, I fear this is just papering over a fundamental problem
> that still exists, that can only be solved by not letting the mainloop
> touch the chardev at all.

The stack I encountered:

#0  0x00007f658234c765 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
#1  0x00007f658234e36a in __GI_abort () at abort.c:89
#2  0x00007f6582344f97 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x55c76345fce1 "iwp->src == NULL", file=file@entry=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=line@entry=91, function=function@entry=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:92
#3  0x00007f6582345042 in __GI___assert_fail (assertion=0x55c76345fce1 "iwp->src == NULL", file=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=91, function=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:101
#4  0x000055c7632c2be5 in io_watch_poll_finalize (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:91
#5  0x00007f65847bb859 in g_source_unref_internal () at /lib64/libglib-2.0.so.0
#6  0x00007f65847bca29 in g_source_destroy_internal () at /lib64/libglib-2.0.so.0
#7  0x000055c7632c2d30 in io_remove_watch_poll (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:139
#8  0x000055c7632c2d5c in remove_fd_in_watch (chr=0x55c7651ccdf0) at /root/git/qemu/chardev/char-io.c:145
#9  0x000055c7632c2368 in qemu_chr_fe_set_handlers (b=0x55c7651f6410, fd_can_read=0x0, fd_read=0x0, fd_event=0x0, be_change=0x0, opaque=0x0, context=0x0, set_open=true)
    at /root/git/qemu/chardev/char-fe.c:267
#10 0x000055c7632c2221 in qemu_chr_fe_deinit (b=0x55c7651f6410, del=false) at /root/git/qemu/chardev/char-fe.c:231
#11 0x000055c762e2b15c in monitor_data_destroy (mon=0x55c7651f6410) at /root/git/qemu/monitor.c:600
#12 0x000055c762e340ec in monitor_cleanup () at /root/git/qemu/monitor.c:4346
#13 0x000055c762f9445d in main (argc=19, argv=0x7ffc6846d0e8, envp=0x7ffc6846d188) at /root/git/qemu/vl.c:4889

So it's destroying the CharBackend, but it'll then call
qemu_chr_fe_set_handlers() which finally tries to remove the watch poll.

If without current patch, I can still encounter the same crash when
doing "make check -j8".

> 
> > 
> > Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
> > Signed-off-by: Peter Xu <peterx@redhat.com>
> > ---
> >  chardev/char-io.c | 15 +++++++++++++--
> >  1 file changed, 13 insertions(+), 2 deletions(-)
> > 
> > diff --git a/chardev/char-io.c b/chardev/char-io.c
> > index f810524..3828c20 100644
> > --- a/chardev/char-io.c
> > +++ b/chardev/char-io.c
> > @@ -122,7 +122,6 @@ GSource *io_add_watch_poll(Chardev *chr,
> >      g_free(name);
> >  
> >      g_source_attach(&iwp->parent, context);
> > -    g_source_unref(&iwp->parent);
> >      return (GSource *)iwp;
> >  }
> >  
> > @@ -131,12 +130,24 @@ static void io_remove_watch_poll(GSource *source)
> >      IOWatchPoll *iwp;
> >  
> >      iwp = io_watch_poll_from_source(source);
> > +
> > +    /*
> > +     * Here the order of destruction really matters.  We need to first
> > +     * detach the IOWatchPoll object from the context (which may still
> > +     * be running in another loop thread), only after that could we
> > +     * continue to operate on iwp->src, or there may be race condition
> > +     * between current thread and the context loop thread.
> > +     *
> > +     * Let's blame the glib bug mentioned in commit 2b3167 (again) for
> > +     * this extra complexity.
> > +     */
> > +    g_source_destroy(&iwp->parent);
> >      if (iwp->src) {
> >          g_source_destroy(iwp->src);
> >          g_source_unref(iwp->src);
> >          iwp->src = NULL;
> >      }
> > -    g_source_destroy(&iwp->parent);
> > +    g_source_unref(&iwp->parent);
> >  }
> >  
> >  void remove_fd_in_watch(Chardev *chr)
> > -- 
> > 2.7.4
> > 
> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

-- 
Peter Xu

Re: [Qemu-devel] [RFC 01/15] char-io: fix possible race on IOWatchPoll

[Daniel P. Berrange]

On Wed, Sep 20, 2017 at 05:09:26PM +0800, Peter Xu wrote:
> On Wed, Sep 20, 2017 at 08:57:03AM +0100, Daniel P. Berrange wrote:
> > On Thu, Sep 14, 2017 at 03:50:22PM +0800, Peter Xu wrote:
> > > This is not a problem if we are only having one single loop thread like
> > > before.  However, after per-monitor thread is introduced, this is not
> > > true any more, and the race can happen.
> > > 
> > > The race can be triggered with "make check -j8" sometimes:
> > > 
> > >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> > >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > > 
> > > This patch keeps the reference for the watch object when creating in
> > > io_add_watch_poll(), so that the object will never be released in the
> > > context main loop, especially when the context loop is running in
> > > another standalone thread.  Meanwhile, when we want to remove the watch
> > > object, we always first detach the watch object from its owner context,
> > > then we continue with the cleanup.
> > > 
> > > Without this patch, calling io_remove_watch_poll() in main loop thread
> > > is not thread-safe, since the other per-monitor thread may be modifying
> > > the watch object at the same time.
> > 
> > This doesn't feel right to me. Why is the main loop thread doing anything
> > at all with the Chardev, if there is a per-monitor thread ? The Chardev
> > code isn't thread safe so it isn't safe to have two separate threads
> > accessing the same Chardev. IOW, if we want a per-monitor thread, then
> > we must make sure the main thread never touches that monitor's chardev
> > at all.  While your patch here might have avoided the assertion you
> > mention above, I fear this is just papering over a fundamental problem
> > that still exists, that can only be solved by not letting the mainloop
> > touch the chardev at all.
> 
> The stack I encountered:
> 
> #0  0x00007f658234c765 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
> #1  0x00007f658234e36a in __GI_abort () at abort.c:89
> #2  0x00007f6582344f97 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x55c76345fce1 "iwp->src == NULL", file=file@entry=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=line@entry=91, function=function@entry=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:92
> #3  0x00007f6582345042 in __GI___assert_fail (assertion=0x55c76345fce1 "iwp->src == NULL", file=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=91, function=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:101
> #4  0x000055c7632c2be5 in io_watch_poll_finalize (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:91
> #5  0x00007f65847bb859 in g_source_unref_internal () at /lib64/libglib-2.0.so.0
> #6  0x00007f65847bca29 in g_source_destroy_internal () at /lib64/libglib-2.0.so.0
> #7  0x000055c7632c2d30 in io_remove_watch_poll (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:139
> #8  0x000055c7632c2d5c in remove_fd_in_watch (chr=0x55c7651ccdf0) at /root/git/qemu/chardev/char-io.c:145
> #9  0x000055c7632c2368 in qemu_chr_fe_set_handlers (b=0x55c7651f6410, fd_can_read=0x0, fd_read=0x0, fd_event=0x0, be_change=0x0, opaque=0x0, context=0x0, set_open=true)
>     at /root/git/qemu/chardev/char-fe.c:267
> #10 0x000055c7632c2221 in qemu_chr_fe_deinit (b=0x55c7651f6410, del=false) at /root/git/qemu/chardev/char-fe.c:231
> #11 0x000055c762e2b15c in monitor_data_destroy (mon=0x55c7651f6410) at /root/git/qemu/monitor.c:600
> #12 0x000055c762e340ec in monitor_cleanup () at /root/git/qemu/monitor.c:4346
> #13 0x000055c762f9445d in main (argc=19, argv=0x7ffc6846d0e8, envp=0x7ffc6846d188) at /root/git/qemu/vl.c:4889
> 
> So it's destroying the CharBackend, but it'll then call
> qemu_chr_fe_set_handlers() which finally tries to remove the watch poll.

Ok that code is broken - it must not call monitor_cleanup from the main
thread - it needs to be called from the monitor thread, unless it can
guarantee that the monitor thread has already exited, which seems unlikely

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC 01/15] char-io: fix possible race on IOWatchPoll

[Peter Xu]

On Wed, Sep 20, 2017 at 10:14:38AM +0100, Daniel P. Berrange wrote:
> On Wed, Sep 20, 2017 at 05:09:26PM +0800, Peter Xu wrote:
> > On Wed, Sep 20, 2017 at 08:57:03AM +0100, Daniel P. Berrange wrote:
> > > On Thu, Sep 14, 2017 at 03:50:22PM +0800, Peter Xu wrote:
> > > > This is not a problem if we are only having one single loop thread like
> > > > before.  However, after per-monitor thread is introduced, this is not
> > > > true any more, and the race can happen.
> > > > 
> > > > The race can be triggered with "make check -j8" sometimes:
> > > > 
> > > >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> > > >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > > > 
> > > > This patch keeps the reference for the watch object when creating in
> > > > io_add_watch_poll(), so that the object will never be released in the
> > > > context main loop, especially when the context loop is running in
> > > > another standalone thread.  Meanwhile, when we want to remove the watch
> > > > object, we always first detach the watch object from its owner context,
> > > > then we continue with the cleanup.
> > > > 
> > > > Without this patch, calling io_remove_watch_poll() in main loop thread
> > > > is not thread-safe, since the other per-monitor thread may be modifying
> > > > the watch object at the same time.
> > > 
> > > This doesn't feel right to me. Why is the main loop thread doing anything
> > > at all with the Chardev, if there is a per-monitor thread ? The Chardev
> > > code isn't thread safe so it isn't safe to have two separate threads
> > > accessing the same Chardev. IOW, if we want a per-monitor thread, then
> > > we must make sure the main thread never touches that monitor's chardev
> > > at all.  While your patch here might have avoided the assertion you
> > > mention above, I fear this is just papering over a fundamental problem
> > > that still exists, that can only be solved by not letting the mainloop
> > > touch the chardev at all.
> > 
> > The stack I encountered:
> > 
> > #0  0x00007f658234c765 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
> > #1  0x00007f658234e36a in __GI_abort () at abort.c:89
> > #2  0x00007f6582344f97 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x55c76345fce1 "iwp->src == NULL", file=file@entry=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=line@entry=91, function=function@entry=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:92
> > #3  0x00007f6582345042 in __GI___assert_fail (assertion=0x55c76345fce1 "iwp->src == NULL", file=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=91, function=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:101
> > #4  0x000055c7632c2be5 in io_watch_poll_finalize (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:91
> > #5  0x00007f65847bb859 in g_source_unref_internal () at /lib64/libglib-2.0.so.0
> > #6  0x00007f65847bca29 in g_source_destroy_internal () at /lib64/libglib-2.0.so.0
> > #7  0x000055c7632c2d30 in io_remove_watch_poll (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:139
> > #8  0x000055c7632c2d5c in remove_fd_in_watch (chr=0x55c7651ccdf0) at /root/git/qemu/chardev/char-io.c:145
> > #9  0x000055c7632c2368 in qemu_chr_fe_set_handlers (b=0x55c7651f6410, fd_can_read=0x0, fd_read=0x0, fd_event=0x0, be_change=0x0, opaque=0x0, context=0x0, set_open=true)
> >     at /root/git/qemu/chardev/char-fe.c:267
> > #10 0x000055c7632c2221 in qemu_chr_fe_deinit (b=0x55c7651f6410, del=false) at /root/git/qemu/chardev/char-fe.c:231
> > #11 0x000055c762e2b15c in monitor_data_destroy (mon=0x55c7651f6410) at /root/git/qemu/monitor.c:600
> > #12 0x000055c762e340ec in monitor_cleanup () at /root/git/qemu/monitor.c:4346
> > #13 0x000055c762f9445d in main (argc=19, argv=0x7ffc6846d0e8, envp=0x7ffc6846d188) at /root/git/qemu/vl.c:4889
> > 
> > So it's destroying the CharBackend, but it'll then call
> > qemu_chr_fe_set_handlers() which finally tries to remove the watch poll.
> 
> Ok that code is broken - it must not call monitor_cleanup from the main
> thread - it needs to be called from the monitor thread, unless it can
> guarantee that the monitor thread has already exited, which seems unlikely

The problem is that not all monitors are parsed in the IO thread, but
only those with use_io_thr=true set.

How about I move the calls of monitor_data_destroy() into that monitor
IO thread when use_io_thr=true?  And for the rest, I think they still
need to be destroyed in the main thread.

> 
> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

-- 
Peter Xu

Re: [Qemu-devel] [RFC 01/15] char-io: fix possible race on IOWatchPoll

[Daniel P. Berrange]

On Wed, Sep 20, 2017 at 06:49:58PM +0800, Peter Xu wrote:
> On Wed, Sep 20, 2017 at 10:14:38AM +0100, Daniel P. Berrange wrote:
> > On Wed, Sep 20, 2017 at 05:09:26PM +0800, Peter Xu wrote:
> > > On Wed, Sep 20, 2017 at 08:57:03AM +0100, Daniel P. Berrange wrote:
> > > > On Thu, Sep 14, 2017 at 03:50:22PM +0800, Peter Xu wrote:
> > > > > This is not a problem if we are only having one single loop thread like
> > > > > before.  However, after per-monitor thread is introduced, this is not
> > > > > true any more, and the race can happen.
> > > > > 
> > > > > The race can be triggered with "make check -j8" sometimes:
> > > > > 
> > > > >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> > > > >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > > > > 
> > > > > This patch keeps the reference for the watch object when creating in
> > > > > io_add_watch_poll(), so that the object will never be released in the
> > > > > context main loop, especially when the context loop is running in
> > > > > another standalone thread.  Meanwhile, when we want to remove the watch
> > > > > object, we always first detach the watch object from its owner context,
> > > > > then we continue with the cleanup.
> > > > > 
> > > > > Without this patch, calling io_remove_watch_poll() in main loop thread
> > > > > is not thread-safe, since the other per-monitor thread may be modifying
> > > > > the watch object at the same time.
> > > > 
> > > > This doesn't feel right to me. Why is the main loop thread doing anything
> > > > at all with the Chardev, if there is a per-monitor thread ? The Chardev
> > > > code isn't thread safe so it isn't safe to have two separate threads
> > > > accessing the same Chardev. IOW, if we want a per-monitor thread, then
> > > > we must make sure the main thread never touches that monitor's chardev
> > > > at all.  While your patch here might have avoided the assertion you
> > > > mention above, I fear this is just papering over a fundamental problem
> > > > that still exists, that can only be solved by not letting the mainloop
> > > > touch the chardev at all.
> > > 
> > > The stack I encountered:
> > > 
> > > #0  0x00007f658234c765 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
> > > #1  0x00007f658234e36a in __GI_abort () at abort.c:89
> > > #2  0x00007f6582344f97 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x55c76345fce1 "iwp->src == NULL", file=file@entry=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=line@entry=91, function=function@entry=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:92
> > > #3  0x00007f6582345042 in __GI___assert_fail (assertion=0x55c76345fce1 "iwp->src == NULL", file=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=91, function=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:101
> > > #4  0x000055c7632c2be5 in io_watch_poll_finalize (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:91
> > > #5  0x00007f65847bb859 in g_source_unref_internal () at /lib64/libglib-2.0.so.0
> > > #6  0x00007f65847bca29 in g_source_destroy_internal () at /lib64/libglib-2.0.so.0
> > > #7  0x000055c7632c2d30 in io_remove_watch_poll (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:139
> > > #8  0x000055c7632c2d5c in remove_fd_in_watch (chr=0x55c7651ccdf0) at /root/git/qemu/chardev/char-io.c:145
> > > #9  0x000055c7632c2368 in qemu_chr_fe_set_handlers (b=0x55c7651f6410, fd_can_read=0x0, fd_read=0x0, fd_event=0x0, be_change=0x0, opaque=0x0, context=0x0, set_open=true)
> > >     at /root/git/qemu/chardev/char-fe.c:267
> > > #10 0x000055c7632c2221 in qemu_chr_fe_deinit (b=0x55c7651f6410, del=false) at /root/git/qemu/chardev/char-fe.c:231
> > > #11 0x000055c762e2b15c in monitor_data_destroy (mon=0x55c7651f6410) at /root/git/qemu/monitor.c:600
> > > #12 0x000055c762e340ec in monitor_cleanup () at /root/git/qemu/monitor.c:4346
> > > #13 0x000055c762f9445d in main (argc=19, argv=0x7ffc6846d0e8, envp=0x7ffc6846d188) at /root/git/qemu/vl.c:4889
> > > 
> > > So it's destroying the CharBackend, but it'll then call
> > > qemu_chr_fe_set_handlers() which finally tries to remove the watch poll.
> > 
> > Ok that code is broken - it must not call monitor_cleanup from the main
> > thread - it needs to be called from the monitor thread, unless it can
> > guarantee that the monitor thread has already exited, which seems unlikely
> 
> The problem is that not all monitors are parsed in the IO thread, but
> only those with use_io_thr=true set.
> 
> How about I move the calls of monitor_data_destroy() into that monitor
> IO thread when use_io_thr=true?  And for the rest, I think they still
> need to be destroyed in the main thread.

I think having the monitor sometimes run in the main thread and sometimes
run in a background thread is a recipe for ongoing trouble, of which this
problem is just the first example that will hurt us. People will test
behaviour of a feature with one setup and then users will later run it in
a different setup and potentially experiance obscure bugs as a result.
IOW, use_io_thr flag should not exist, and every monitor should be run
unconditionally in the background thread from the point at which your
patch series merges.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC 01/15] char-io: fix possible race on IOWatchPoll

[Peter Xu]

On Wed, Sep 20, 2017 at 12:03:09PM +0100, Daniel P. Berrange wrote:
> On Wed, Sep 20, 2017 at 06:49:58PM +0800, Peter Xu wrote:
> > On Wed, Sep 20, 2017 at 10:14:38AM +0100, Daniel P. Berrange wrote:
> > > On Wed, Sep 20, 2017 at 05:09:26PM +0800, Peter Xu wrote:
> > > > On Wed, Sep 20, 2017 at 08:57:03AM +0100, Daniel P. Berrange wrote:
> > > > > On Thu, Sep 14, 2017 at 03:50:22PM +0800, Peter Xu wrote:
> > > > > > This is not a problem if we are only having one single loop thread like
> > > > > > before.  However, after per-monitor thread is introduced, this is not
> > > > > > true any more, and the race can happen.
> > > > > > 
> > > > > > The race can be triggered with "make check -j8" sometimes:
> > > > > > 
> > > > > >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> > > > > >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > > > > > 
> > > > > > This patch keeps the reference for the watch object when creating in
> > > > > > io_add_watch_poll(), so that the object will never be released in the
> > > > > > context main loop, especially when the context loop is running in
> > > > > > another standalone thread.  Meanwhile, when we want to remove the watch
> > > > > > object, we always first detach the watch object from its owner context,
> > > > > > then we continue with the cleanup.
> > > > > > 
> > > > > > Without this patch, calling io_remove_watch_poll() in main loop thread
> > > > > > is not thread-safe, since the other per-monitor thread may be modifying
> > > > > > the watch object at the same time.
> > > > > 
> > > > > This doesn't feel right to me. Why is the main loop thread doing anything
> > > > > at all with the Chardev, if there is a per-monitor thread ? The Chardev
> > > > > code isn't thread safe so it isn't safe to have two separate threads
> > > > > accessing the same Chardev. IOW, if we want a per-monitor thread, then
> > > > > we must make sure the main thread never touches that monitor's chardev
> > > > > at all.  While your patch here might have avoided the assertion you
> > > > > mention above, I fear this is just papering over a fundamental problem
> > > > > that still exists, that can only be solved by not letting the mainloop
> > > > > touch the chardev at all.
> > > > 
> > > > The stack I encountered:
> > > > 
> > > > #0  0x00007f658234c765 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
> > > > #1  0x00007f658234e36a in __GI_abort () at abort.c:89
> > > > #2  0x00007f6582344f97 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x55c76345fce1 "iwp->src == NULL", file=file@entry=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=line@entry=91, function=function@entry=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:92
> > > > #3  0x00007f6582345042 in __GI___assert_fail (assertion=0x55c76345fce1 "iwp->src == NULL", file=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=91, function=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:101
> > > > #4  0x000055c7632c2be5 in io_watch_poll_finalize (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:91
> > > > #5  0x00007f65847bb859 in g_source_unref_internal () at /lib64/libglib-2.0.so.0
> > > > #6  0x00007f65847bca29 in g_source_destroy_internal () at /lib64/libglib-2.0.so.0
> > > > #7  0x000055c7632c2d30 in io_remove_watch_poll (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:139
> > > > #8  0x000055c7632c2d5c in remove_fd_in_watch (chr=0x55c7651ccdf0) at /root/git/qemu/chardev/char-io.c:145
> > > > #9  0x000055c7632c2368 in qemu_chr_fe_set_handlers (b=0x55c7651f6410, fd_can_read=0x0, fd_read=0x0, fd_event=0x0, be_change=0x0, opaque=0x0, context=0x0, set_open=true)
> > > >     at /root/git/qemu/chardev/char-fe.c:267
> > > > #10 0x000055c7632c2221 in qemu_chr_fe_deinit (b=0x55c7651f6410, del=false) at /root/git/qemu/chardev/char-fe.c:231
> > > > #11 0x000055c762e2b15c in monitor_data_destroy (mon=0x55c7651f6410) at /root/git/qemu/monitor.c:600
> > > > #12 0x000055c762e340ec in monitor_cleanup () at /root/git/qemu/monitor.c:4346
> > > > #13 0x000055c762f9445d in main (argc=19, argv=0x7ffc6846d0e8, envp=0x7ffc6846d188) at /root/git/qemu/vl.c:4889
> > > > 
> > > > So it's destroying the CharBackend, but it'll then call
> > > > qemu_chr_fe_set_handlers() which finally tries to remove the watch poll.
> > > 
> > > Ok that code is broken - it must not call monitor_cleanup from the main
> > > thread - it needs to be called from the monitor thread, unless it can
> > > guarantee that the monitor thread has already exited, which seems unlikely
> > 
> > The problem is that not all monitors are parsed in the IO thread, but
> > only those with use_io_thr=true set.
> > 
> > How about I move the calls of monitor_data_destroy() into that monitor
> > IO thread when use_io_thr=true?  And for the rest, I think they still
> > need to be destroyed in the main thread.
> 
> I think having the monitor sometimes run in the main thread and sometimes
> run in a background thread is a recipe for ongoing trouble, of which this
> problem is just the first example that will hurt us. People will test
> behaviour of a feature with one setup and then users will later run it in
> a different setup and potentially experiance obscure bugs as a result.
> IOW, use_io_thr flag should not exist, and every monitor should be run
> unconditionally in the background thread from the point at which your
> patch series merges.

I agree with you that this may bring trouble in some aspect.  I just
don't know whether it'll bring more trouble if we move all the
monitor-related chardev IO into monitor thread.

The key is the muxed typed chardev.

If we don't have muxed typed chardev, I'll surely consider to use IO
thread for all the monitors.

However, the muxed chardevs can support e.g. one monitor plus a serial
port. Can we just run the IO stuff in monitor thread even part of its
frontend is a serial port?  And also I'm not sure what would happen if
it's a monitor plus something else I even don't aware of.

Any nicer thoughts?  Thanks,

-- 
Peter Xu

Re: [Qemu-devel] [RFC 01/15] char-io: fix possible race on IOWatchPoll

[Daniel P. Berrange]

On Wed, Sep 20, 2017 at 07:18:49PM +0800, Peter Xu wrote:
> On Wed, Sep 20, 2017 at 12:03:09PM +0100, Daniel P. Berrange wrote:
> > On Wed, Sep 20, 2017 at 06:49:58PM +0800, Peter Xu wrote:
> > > On Wed, Sep 20, 2017 at 10:14:38AM +0100, Daniel P. Berrange wrote:
> > > > On Wed, Sep 20, 2017 at 05:09:26PM +0800, Peter Xu wrote:
> > > > > On Wed, Sep 20, 2017 at 08:57:03AM +0100, Daniel P. Berrange wrote:
> > > > > > On Thu, Sep 14, 2017 at 03:50:22PM +0800, Peter Xu wrote:
> > > > > > > This is not a problem if we are only having one single loop thread like
> > > > > > > before.  However, after per-monitor thread is introduced, this is not
> > > > > > > true any more, and the race can happen.
> > > > > > > 
> > > > > > > The race can be triggered with "make check -j8" sometimes:
> > > > > > > 
> > > > > > >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> > > > > > >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > > > > > > 
> > > > > > > This patch keeps the reference for the watch object when creating in
> > > > > > > io_add_watch_poll(), so that the object will never be released in the
> > > > > > > context main loop, especially when the context loop is running in
> > > > > > > another standalone thread.  Meanwhile, when we want to remove the watch
> > > > > > > object, we always first detach the watch object from its owner context,
> > > > > > > then we continue with the cleanup.
> > > > > > > 
> > > > > > > Without this patch, calling io_remove_watch_poll() in main loop thread
> > > > > > > is not thread-safe, since the other per-monitor thread may be modifying
> > > > > > > the watch object at the same time.
> > > > > > 
> > > > > > This doesn't feel right to me. Why is the main loop thread doing anything
> > > > > > at all with the Chardev, if there is a per-monitor thread ? The Chardev
> > > > > > code isn't thread safe so it isn't safe to have two separate threads
> > > > > > accessing the same Chardev. IOW, if we want a per-monitor thread, then
> > > > > > we must make sure the main thread never touches that monitor's chardev
> > > > > > at all.  While your patch here might have avoided the assertion you
> > > > > > mention above, I fear this is just papering over a fundamental problem
> > > > > > that still exists, that can only be solved by not letting the mainloop
> > > > > > touch the chardev at all.
> > > > > 
> > > > > The stack I encountered:
> > > > > 
> > > > > #0  0x00007f658234c765 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
> > > > > #1  0x00007f658234e36a in __GI_abort () at abort.c:89
> > > > > #2  0x00007f6582344f97 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x55c76345fce1 "iwp->src == NULL", file=file@entry=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=line@entry=91, function=function@entry=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:92
> > > > > #3  0x00007f6582345042 in __GI___assert_fail (assertion=0x55c76345fce1 "iwp->src == NULL", file=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=91, function=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:101
> > > > > #4  0x000055c7632c2be5 in io_watch_poll_finalize (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:91
> > > > > #5  0x00007f65847bb859 in g_source_unref_internal () at /lib64/libglib-2.0.so.0
> > > > > #6  0x00007f65847bca29 in g_source_destroy_internal () at /lib64/libglib-2.0.so.0
> > > > > #7  0x000055c7632c2d30 in io_remove_watch_poll (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:139
> > > > > #8  0x000055c7632c2d5c in remove_fd_in_watch (chr=0x55c7651ccdf0) at /root/git/qemu/chardev/char-io.c:145
> > > > > #9  0x000055c7632c2368 in qemu_chr_fe_set_handlers (b=0x55c7651f6410, fd_can_read=0x0, fd_read=0x0, fd_event=0x0, be_change=0x0, opaque=0x0, context=0x0, set_open=true)
> > > > >     at /root/git/qemu/chardev/char-fe.c:267
> > > > > #10 0x000055c7632c2221 in qemu_chr_fe_deinit (b=0x55c7651f6410, del=false) at /root/git/qemu/chardev/char-fe.c:231
> > > > > #11 0x000055c762e2b15c in monitor_data_destroy (mon=0x55c7651f6410) at /root/git/qemu/monitor.c:600
> > > > > #12 0x000055c762e340ec in monitor_cleanup () at /root/git/qemu/monitor.c:4346
> > > > > #13 0x000055c762f9445d in main (argc=19, argv=0x7ffc6846d0e8, envp=0x7ffc6846d188) at /root/git/qemu/vl.c:4889
> > > > > 
> > > > > So it's destroying the CharBackend, but it'll then call
> > > > > qemu_chr_fe_set_handlers() which finally tries to remove the watch poll.
> > > > 
> > > > Ok that code is broken - it must not call monitor_cleanup from the main
> > > > thread - it needs to be called from the monitor thread, unless it can
> > > > guarantee that the monitor thread has already exited, which seems unlikely
> > > 
> > > The problem is that not all monitors are parsed in the IO thread, but
> > > only those with use_io_thr=true set.
> > > 
> > > How about I move the calls of monitor_data_destroy() into that monitor
> > > IO thread when use_io_thr=true?  And for the rest, I think they still
> > > need to be destroyed in the main thread.
> > 
> > I think having the monitor sometimes run in the main thread and sometimes
> > run in a background thread is a recipe for ongoing trouble, of which this
> > problem is just the first example that will hurt us. People will test
> > behaviour of a feature with one setup and then users will later run it in
> > a different setup and potentially experiance obscure bugs as a result.
> > IOW, use_io_thr flag should not exist, and every monitor should be run
> > unconditionally in the background thread from the point at which your
> > patch series merges.
> 
> I agree with you that this may bring trouble in some aspect.  I just
> don't know whether it'll bring more trouble if we move all the
> monitor-related chardev IO into monitor thread.
> 
> The key is the muxed typed chardev.
> 
> If we don't have muxed typed chardev, I'll surely consider to use IO
> thread for all the monitors.
> 
> However, the muxed chardevs can support e.g. one monitor plus a serial
> port. Can we just run the IO stuff in monitor thread even part of its
> frontend is a serial port?  And also I'm not sure what would happen if
> it's a monitor plus something else I even don't aware of.

Urgh, I forgot about the horrible mux chardev concept, that does rather
complicate life - moving the guest device interaction to the monitor
thread would be dubious.

So yeah, given that, it probably is simplest to change monitor_cleanup
to skip destroy of monitors which have a background thread.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [RFC 01/15] char-io: fix possible race on IOWatchPoll

[Peter Xu]

On Wed, Sep 20, 2017 at 12:29:21PM +0100, Daniel P. Berrange wrote:
> On Wed, Sep 20, 2017 at 07:18:49PM +0800, Peter Xu wrote:
> > On Wed, Sep 20, 2017 at 12:03:09PM +0100, Daniel P. Berrange wrote:
> > > On Wed, Sep 20, 2017 at 06:49:58PM +0800, Peter Xu wrote:
> > > > On Wed, Sep 20, 2017 at 10:14:38AM +0100, Daniel P. Berrange wrote:
> > > > > On Wed, Sep 20, 2017 at 05:09:26PM +0800, Peter Xu wrote:
> > > > > > On Wed, Sep 20, 2017 at 08:57:03AM +0100, Daniel P. Berrange wrote:
> > > > > > > On Thu, Sep 14, 2017 at 03:50:22PM +0800, Peter Xu wrote:
> > > > > > > > This is not a problem if we are only having one single loop thread like
> > > > > > > > before.  However, after per-monitor thread is introduced, this is not
> > > > > > > > true any more, and the race can happen.
> > > > > > > > 
> > > > > > > > The race can be triggered with "make check -j8" sometimes:
> > > > > > > > 
> > > > > > > >   qemu-system-x86_64: /root/git/qemu/chardev/char-io.c:91:
> > > > > > > >   io_watch_poll_finalize: Assertion `iwp->src == NULL' failed.
> > > > > > > > 
> > > > > > > > This patch keeps the reference for the watch object when creating in
> > > > > > > > io_add_watch_poll(), so that the object will never be released in the
> > > > > > > > context main loop, especially when the context loop is running in
> > > > > > > > another standalone thread.  Meanwhile, when we want to remove the watch
> > > > > > > > object, we always first detach the watch object from its owner context,
> > > > > > > > then we continue with the cleanup.
> > > > > > > > 
> > > > > > > > Without this patch, calling io_remove_watch_poll() in main loop thread
> > > > > > > > is not thread-safe, since the other per-monitor thread may be modifying
> > > > > > > > the watch object at the same time.
> > > > > > > 
> > > > > > > This doesn't feel right to me. Why is the main loop thread doing anything
> > > > > > > at all with the Chardev, if there is a per-monitor thread ? The Chardev
> > > > > > > code isn't thread safe so it isn't safe to have two separate threads
> > > > > > > accessing the same Chardev. IOW, if we want a per-monitor thread, then
> > > > > > > we must make sure the main thread never touches that monitor's chardev
> > > > > > > at all.  While your patch here might have avoided the assertion you
> > > > > > > mention above, I fear this is just papering over a fundamental problem
> > > > > > > that still exists, that can only be solved by not letting the mainloop
> > > > > > > touch the chardev at all.
> > > > > > 
> > > > > > The stack I encountered:
> > > > > > 
> > > > > > #0  0x00007f658234c765 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54
> > > > > > #1  0x00007f658234e36a in __GI_abort () at abort.c:89
> > > > > > #2  0x00007f6582344f97 in __assert_fail_base (fmt=<optimized out>, assertion=assertion@entry=0x55c76345fce1 "iwp->src == NULL", file=file@entry=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=line@entry=91, function=function@entry=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:92
> > > > > > #3  0x00007f6582345042 in __GI___assert_fail (assertion=0x55c76345fce1 "iwp->src == NULL", file=0x55c76345fcc0 "/root/git/qemu/chardev/char-io.c", line=91, function=0x55c76345fd10 <__PRETTY_FUNCTION__.21863> "io_watch_poll_finalize") at assert.c:101
> > > > > > #4  0x000055c7632c2be5 in io_watch_poll_finalize (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:91
> > > > > > #5  0x00007f65847bb859 in g_source_unref_internal () at /lib64/libglib-2.0.so.0
> > > > > > #6  0x00007f65847bca29 in g_source_destroy_internal () at /lib64/libglib-2.0.so.0
> > > > > > #7  0x000055c7632c2d30 in io_remove_watch_poll (source=0x55c7651cd450) at /root/git/qemu/chardev/char-io.c:139
> > > > > > #8  0x000055c7632c2d5c in remove_fd_in_watch (chr=0x55c7651ccdf0) at /root/git/qemu/chardev/char-io.c:145
> > > > > > #9  0x000055c7632c2368 in qemu_chr_fe_set_handlers (b=0x55c7651f6410, fd_can_read=0x0, fd_read=0x0, fd_event=0x0, be_change=0x0, opaque=0x0, context=0x0, set_open=true)
> > > > > >     at /root/git/qemu/chardev/char-fe.c:267
> > > > > > #10 0x000055c7632c2221 in qemu_chr_fe_deinit (b=0x55c7651f6410, del=false) at /root/git/qemu/chardev/char-fe.c:231
> > > > > > #11 0x000055c762e2b15c in monitor_data_destroy (mon=0x55c7651f6410) at /root/git/qemu/monitor.c:600
> > > > > > #12 0x000055c762e340ec in monitor_cleanup () at /root/git/qemu/monitor.c:4346
> > > > > > #13 0x000055c762f9445d in main (argc=19, argv=0x7ffc6846d0e8, envp=0x7ffc6846d188) at /root/git/qemu/vl.c:4889
> > > > > > 
> > > > > > So it's destroying the CharBackend, but it'll then call
> > > > > > qemu_chr_fe_set_handlers() which finally tries to remove the watch poll.
> > > > > 
> > > > > Ok that code is broken - it must not call monitor_cleanup from the main
> > > > > thread - it needs to be called from the monitor thread, unless it can
> > > > > guarantee that the monitor thread has already exited, which seems unlikely
> > > > 
> > > > The problem is that not all monitors are parsed in the IO thread, but
> > > > only those with use_io_thr=true set.
> > > > 
> > > > How about I move the calls of monitor_data_destroy() into that monitor
> > > > IO thread when use_io_thr=true?  And for the rest, I think they still
> > > > need to be destroyed in the main thread.
> > > 
> > > I think having the monitor sometimes run in the main thread and sometimes
> > > run in a background thread is a recipe for ongoing trouble, of which this
> > > problem is just the first example that will hurt us. People will test
> > > behaviour of a feature with one setup and then users will later run it in
> > > a different setup and potentially experiance obscure bugs as a result.
> > > IOW, use_io_thr flag should not exist, and every monitor should be run
> > > unconditionally in the background thread from the point at which your
> > > patch series merges.
> > 
> > I agree with you that this may bring trouble in some aspect.  I just
> > don't know whether it'll bring more trouble if we move all the
> > monitor-related chardev IO into monitor thread.
> > 
> > The key is the muxed typed chardev.
> > 
> > If we don't have muxed typed chardev, I'll surely consider to use IO
> > thread for all the monitors.
> > 
> > However, the muxed chardevs can support e.g. one monitor plus a serial
> > port. Can we just run the IO stuff in monitor thread even part of its
> > frontend is a serial port?  And also I'm not sure what would happen if
> > it's a monitor plus something else I even don't aware of.
> 
> Urgh, I forgot about the horrible mux chardev concept, that does rather
> complicate life - moving the guest device interaction to the monitor
> thread would be dubious.
> 
> So yeah, given that, it probably is simplest to change monitor_cleanup
> to skip destroy of monitors which have a background thread.

Thanks.  I'll add one more patch to split the cleanups to make sure
they are done in their own thread.

If you won't disagree, I would still prefer to keep this patch -
firstly, it never hurts; secondly, in case one day we would like to
make the whole chardev thread-safe, then we won't need to dig that
hole again.

-- 
Peter Xu