All of lore.kernel.org
 help / color / mirror / Atom feed
* [Qemu-devel] [PATCH v3 00/38] RFC: monitor: add asynchronous command type
@ 2018-03-26 15:08 Marc-André Lureau
  2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 01/38] HACK: add back OOB Marc-André Lureau
                   ` (43 more replies)
  0 siblings, 44 replies; 61+ messages in thread
From: Marc-André Lureau @ 2018-03-26 15:08 UTC (permalink / raw)
  To: qemu-devel
  Cc: Eduardo Habkost, Juan Quintela, Dr. David Alan Gilbert,
	Eric Blake, Cleber Rosa, Markus Armbruster, Gerd Hoffmann,
	Michael Roth, Marc-André Lureau

Hi,

One of initial design goals of QMP was to have "asynchronous command
completion" (http://wiki.qemu.org/Features/QAPI). Unfortunately, that
goal was not fully achieved, and some broken bits left were removed
progressively until commit 65207c59d that removed async command
support.

Yet there are clear benefits allowing the command handler to re-enter
the main loop if the command cannot be handled synchronously, or if it
is long-lasting. This is currently not possible and make some bugs
such as rhbz#1230527 difficult to solve.  Furthermore, many QMP
commands do IO and could be considered 'slow' and blocking the main
loop today. The unwritten solution so far is to use a pair of
command+event. But this approach has a number of issues, in particular
to fix existing commands, and inadequacy since the event is
broadcasted and may thus have command 'id' conflict, beside being
rather inefficient and incorrect.

The following series implements an async command solution instead. By
introducing a session context and a command return handler, it can:
- defer the return, allowing the mainloop to reenter
- return only to the caller (no broadcasted event)
- optionnally allow cancellation when the client is gone
- track on-going qapi command(s) per client/session

Existing qemu commands can be gradually replaced by async:true
variants when needed, and by carefully reviewing the concurrency
aspects. The async:true commands marshaller helpers are splitted in
half, the calling and return functions. The command is called with a
QmpReturn context, that can return immediately or later, using the
generated return helper, which allows for step-by-step conversion.

The screendump command is converted to an async:true version to solve
rhbz#1230527. The command shows basic cancellation (this could be
extended if needed). It could be further improved to do asynchronous
IO write as well.

v3:
- complete rework, dropping the asynchronous commands visibility from
  the protocol side entirely (until there is a real need for it)
- rebased, with a few preliminary cleanup patches
- teach asynchronous commands to HMP

v2:
- documentation fixes and improvements
- fix calling async commands sync without id
- fix bad hmp monitor assert
- add a few extra asserts
- add async with no-id failure and screendump test

Marc-André Lureau (38):
  HACK: add back OOB
  qmp-shell: learn to send commands with quoted arguments
  Revert "qmp: isolate responses into io thread"
  monitor: no need to save need_resume
  monitor: further simplify previous patch
  monitor: no need to remove desc before replacing it
  json-parser: always set an error if return NULL
  json-lexer: make it safe to call multiple times
  json: remove useless return value from lexer/parser
  tests: add a few qemu-qmp tests
  tests: change /0.15/* tests to /qmp/*
  tests: add a qmp success-response test
  qga: process_event() simplification
  monitor: simplify monitor_qmp_respond()
  qmp: pass and return a QDict to qmp_dispatch()
  qmp: move 'id' copy to qmp_dispatch()
  qmp: constify qmp_is_oob()
  qmp: add QmpSession
  QmpSession: add a return_cb
  QmpSession: add json parser and use it in qga
  QmpSession: add a dispatch callback
  monitor: use QmpSession parsing and common dispatch code
  QmpSession: introduce QmpReturn
  qmp: remove qmp_build_error_object()
  qmp: remove need for qobject_from_jsonf()
  qmp: fold do_qmp_dispatch() in qmp_dispatch()
  QmpSession: keep a queue of pending commands
  QmpSession: return orderly
  qmp: introduce asynchronous command type
  scripts: learn 'async' qapi commands
  qmp: add qmp_return_is_cancelled()
  monitor: add qmp_return_get_monitor()
  console: graphic_hw_update return true if async
  console: add graphic_hw_update_done()
  console: make screendump asynchronous
  monitor: start making qmp_human_monitor_command() asynchronous
  monitor: teach HMP about asynchronous commands
  hmp: call the asynchronous QMP screendump to fix outdated/glitches

 qapi/misc.json                          |   3 +-
 qapi/ui.json                            |   3 +-
 scripts/qapi/commands.py                | 149 +++++++--
 scripts/qapi/common.py                  |  12 +-
 scripts/qapi/doc.py                     |   2 +-
 scripts/qapi/introspect.py              |   2 +-
 hmp.h                                   |   3 +-
 hw/display/qxl.h                        |   2 +-
 include/monitor/monitor.h               |   3 +
 include/qapi/qmp/dispatch.h             |  85 ++++-
 include/qapi/qmp/json-lexer.h           |   4 +-
 include/qapi/qmp/json-streamer.h        |   4 +-
 include/ui/console.h                    |   7 +-
 hmp.c                                   |   6 +-
 hw/display/qxl-render.c                 |  14 +-
 hw/display/qxl.c                        |   8 +-
 migration/postcopy-ram.c                |   1 +
 monitor.c                               | 393 +++++++++---------------
 qapi/qmp-dispatch.c                     | 234 +++++++++++---
 qapi/qmp-registry.c                     |  27 +-
 qga/main.c                              |  73 +----
 qobject/json-lexer.c                    |  28 +-
 qobject/json-parser.c                   |   7 +-
 qobject/json-streamer.c                 |   8 +-
 tests/qmp-test.c                        | 146 ++++++++-
 tests/test-qmp-cmds.c                   | 206 +++++++++++--
 ui/console.c                            | 106 ++++++-
 hmp-commands.hx                         |   3 +-
 scripts/qmp/qmp-shell                   |   3 +-
 tests/Makefile.include                  |   6 +-
 tests/qapi-schema/qapi-schema-test.json |   7 +
 tests/qapi-schema/qapi-schema-test.out  |  10 +
 tests/qapi-schema/test-qapi.py          |   7 +-
 33 files changed, 1078 insertions(+), 494 deletions(-)
 mode change 100644 => 100755 scripts/qapi/doc.py

-- 
2.17.0.rc1.1.g4c4f2b46a3

^ permalink raw reply	[flat|nested] 61+ messages in thread

end of thread, other threads:[~2019-04-09 14:08 UTC | newest]

Thread overview: 61+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-03-26 15:08 [Qemu-devel] [PATCH v3 00/38] RFC: monitor: add asynchronous command type Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 01/38] HACK: add back OOB Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 02/38] qmp-shell: learn to send commands with quoted arguments Marc-André Lureau
2018-03-26 18:26   ` Eduardo Habkost
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 03/38] Revert "qmp: isolate responses into io thread" Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 04/38] monitor: no need to save need_resume Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 05/38] monitor: further simplify previous patch Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 06/38] monitor: no need to remove desc before replacing it Marc-André Lureau
2018-03-26 19:31   ` Eric Blake
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 07/38] json-parser: always set an error if return NULL Marc-André Lureau
2018-07-05 11:35   ` Markus Armbruster
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 08/38] json-lexer: make it safe to call multiple times Marc-André Lureau
2018-07-05 11:42   ` Markus Armbruster
2018-07-05 12:17     ` Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 09/38] json: remove useless return value from lexer/parser Marc-André Lureau
2018-07-05 11:49   ` Markus Armbruster
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 10/38] tests: add a few qemu-qmp tests Marc-André Lureau
2018-07-05 11:55   ` Markus Armbruster
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 11/38] tests: change /0.15/* tests to /qmp/* Marc-André Lureau
2018-07-05 11:56   ` Markus Armbruster
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 12/38] tests: add a qmp success-response test Marc-André Lureau
2018-07-05 11:59   ` Markus Armbruster
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 13/38] qga: process_event() simplification Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 14/38] monitor: simplify monitor_qmp_respond() Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 15/38] qmp: pass and return a QDict to qmp_dispatch() Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 16/38] qmp: move 'id' copy " Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 17/38] qmp: constify qmp_is_oob() Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 18/38] qmp: add QmpSession Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 19/38] QmpSession: add a return_cb Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 20/38] QmpSession: add json parser and use it in qga Marc-André Lureau
2018-03-26 15:08 ` [Qemu-devel] [PATCH v3 21/38] QmpSession: add a dispatch callback Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 22/38] monitor: use QmpSession parsing and common dispatch code Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 23/38] QmpSession: introduce QmpReturn Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 24/38] qmp: remove qmp_build_error_object() Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 25/38] qmp: remove need for qobject_from_jsonf() Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 26/38] qmp: fold do_qmp_dispatch() in qmp_dispatch() Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 27/38] QmpSession: keep a queue of pending commands Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 28/38] QmpSession: return orderly Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 29/38] qmp: introduce asynchronous command type Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 30/38] scripts: learn 'async' qapi commands Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 31/38] qmp: add qmp_return_is_cancelled() Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 32/38] monitor: add qmp_return_get_monitor() Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 33/38] console: graphic_hw_update return true if async Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 34/38] console: add graphic_hw_update_done() Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 35/38] console: make screendump asynchronous Marc-André Lureau
2018-04-12 14:48   ` Dr. David Alan Gilbert
2018-04-19 16:05     ` Marc-André Lureau
2019-04-09 14:06     ` Marc-André Lureau
2019-04-09 14:06       ` Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 36/38] monitor: start making qmp_human_monitor_command() asynchronous Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 37/38] monitor: teach HMP about asynchronous commands Marc-André Lureau
2018-03-26 15:09 ` [Qemu-devel] [PATCH v3 38/38] hmp: call the asynchronous QMP screendump to fix outdated/glitches Marc-André Lureau
2018-03-26 17:24 ` [Qemu-devel] [PATCH v3 00/38] RFC: monitor: add asynchronous command type Dr. David Alan Gilbert
2018-03-26 17:30   ` Marc-André Lureau
2018-03-26 17:43 ` no-reply
2018-03-26 17:55 ` no-reply
2018-04-04  9:34 ` Stefan Hajnoczi
2018-04-04 10:01   ` Marc-André Lureau
2018-04-04 13:45 ` Eric Blake
2018-04-04 13:57   ` Marc-André Lureau
2018-07-05 13:05 ` Markus Armbruster

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.