From: "Marc-André Lureau" <marcandre.lureau@redhat.com> To: qemu-devel@nongnu.org Cc: "Eric Blake" <eblake@redhat.com>, "Markus Armbruster" <armbru@redhat.com>, "Dr. David Alan Gilbert" <dgilbert@redhat.com>, "Gerd Hoffmann" <kraxel@redhat.com>, "Michael Roth" <mdroth@linux.vnet.ibm.com>, "Marc-André Lureau" <marcandre.lureau@redhat.com> Subject: [Qemu-devel] [PATCH v4 00/20] monitor: add asynchronous command type Date: Tue, 9 Apr 2019 18:09:49 +0200 [thread overview] Message-ID: <20190409161009.6322-1-marcandre.lureau@redhat.com> (raw) Hi, HMP and QMP commands are handled synchronously in qemu today. But there are benefits allowing the command handler to re-enter the main loop if the command cannot be handled synchronously, or if it is long-lasting. Some bugs such as rhbz#1230527 are difficult to solve without it. The common solution is to use a pair of command+event in this case. But this approach has a number of issues: - you can't "fix" an existing command: you need a new API, and ad-hoc documentation for that command+signal association, and old/broken command deprecation - since the reply event is broadcasted and 'id' is used for matching the request, it may conflict with other clients request 'id' space - it is arguably less efficient and elegant (weird API, useless return in most cases, broadcast reply, no cancelling on disconnect etc) The following series implements an async command solution instead. By introducing a session context and a command return handler, it can: - defer the return, allowing the mainloop to reenter - return only to the caller (instead of broadcast events for reply) - optionnally allow cancellation when the client is gone - track on-going qapi command(s) per client/session and without introduction of new QMP APIs or client visible change. Existing qemu commands can be gradually replaced by async:true variants when needed, while carefully reviewing the concurrency aspects. The async:true commands marshaller helpers are splitted in half, the calling and return functions. The command is called with a QmpReturn context, that can return immediately or later, using the generated return helper, which allows for a step-by-step conversion. The screendump command is converted to an async:true version to solve rhbz#1230527. The command shows basic cancellation (this could be extended if needed). It could be further improved to do asynchronous IO writes as well. v4: - rebased, mostly adapting to new OOB code (there was not much feedback in v3 for the async command part, but preliminary patches got merged!) - drop the RFC status v3: - complete rework, dropping the asynchronous commands visibility from the protocol side entirely (until there is a real need for it) - rebased, with a few preliminary cleanup patches - teach asynchronous commands to HMP v2: - documentation fixes and improvements - fix calling async commands sync without id - fix bad hmp monitor assert - add a few extra asserts - add async with no-id failure and screendump test Marc-André Lureau (20): qmp: constify QmpCommand and list json-lexer: make it safe to call destroy multiple times qmp: add QmpSession QmpSession: add a return callback QmpSession: add json parser and use it in qga monitor: use qmp session to parse json feed qga: simplify dispatch_return_cb QmpSession: introduce QmpReturn qmp: simplify qmp_return_error() QmpSession: keep a queue of pending commands QmpSession: return orderly qmp: introduce asynchronous command type scripts: learn 'async' qapi commands qmp: add qmp_return_is_cancelled() monitor: add qmp_return_get_monitor() console: add graphic_hw_update_done() console: make screendump asynchronous monitor: start making qmp_human_monitor_command() asynchronous monitor: teach HMP about asynchronous commands hmp: call the asynchronous QMP screendump to fix outdated/glitches qapi/misc.json | 3 +- qapi/ui.json | 3 +- scripts/qapi/commands.py | 151 ++++++++++++++--- scripts/qapi/common.py | 15 +- scripts/qapi/doc.py | 3 +- scripts/qapi/introspect.py | 3 +- hmp.h | 3 +- include/monitor/monitor.h | 3 + include/qapi/qmp/dispatch.h | 89 +++++++++- include/qapi/qmp/json-parser.h | 7 +- include/ui/console.h | 5 + hmp.c | 6 +- hw/display/qxl-render.c | 9 +- hw/display/qxl.c | 1 + monitor.c | 198 ++++++++++++++-------- qapi/qmp-dispatch.c | 214 +++++++++++++++++++----- qapi/qmp-registry.c | 33 +++- qga/commands.c | 2 +- qga/main.c | 51 ++---- qobject/json-lexer.c | 5 +- qobject/json-streamer.c | 3 +- tests/test-qmp-cmds.c | 206 +++++++++++++++++++---- ui/console.c | 100 +++++++++-- hmp-commands.hx | 3 +- tests/qapi-schema/qapi-schema-test.json | 5 + tests/qapi-schema/qapi-schema-test.out | 8 + tests/qapi-schema/test-qapi.py | 8 +- 27 files changed, 877 insertions(+), 260 deletions(-) -- 2.21.0.196.g041f5ea1cf
WARNING: multiple messages have this Message-ID (diff)
From: "Marc-André Lureau" <marcandre.lureau@redhat.com> To: qemu-devel@nongnu.org Cc: "Michael Roth" <mdroth@linux.vnet.ibm.com>, "Markus Armbruster" <armbru@redhat.com>, "Gerd Hoffmann" <kraxel@redhat.com>, "Marc-André Lureau" <marcandre.lureau@redhat.com>, "Dr. David Alan Gilbert" <dgilbert@redhat.com> Subject: [Qemu-devel] [PATCH v4 00/20] monitor: add asynchronous command type Date: Tue, 9 Apr 2019 18:09:49 +0200 [thread overview] Message-ID: <20190409161009.6322-1-marcandre.lureau@redhat.com> (raw) Message-ID: <20190409160949.YOAqAIjipA9aSS8kelj7995GeRUc1-7IQ9WCZRr2iyA@z> (raw) Hi, HMP and QMP commands are handled synchronously in qemu today. But there are benefits allowing the command handler to re-enter the main loop if the command cannot be handled synchronously, or if it is long-lasting. Some bugs such as rhbz#1230527 are difficult to solve without it. The common solution is to use a pair of command+event in this case. But this approach has a number of issues: - you can't "fix" an existing command: you need a new API, and ad-hoc documentation for that command+signal association, and old/broken command deprecation - since the reply event is broadcasted and 'id' is used for matching the request, it may conflict with other clients request 'id' space - it is arguably less efficient and elegant (weird API, useless return in most cases, broadcast reply, no cancelling on disconnect etc) The following series implements an async command solution instead. By introducing a session context and a command return handler, it can: - defer the return, allowing the mainloop to reenter - return only to the caller (instead of broadcast events for reply) - optionnally allow cancellation when the client is gone - track on-going qapi command(s) per client/session and without introduction of new QMP APIs or client visible change. Existing qemu commands can be gradually replaced by async:true variants when needed, while carefully reviewing the concurrency aspects. The async:true commands marshaller helpers are splitted in half, the calling and return functions. The command is called with a QmpReturn context, that can return immediately or later, using the generated return helper, which allows for a step-by-step conversion. The screendump command is converted to an async:true version to solve rhbz#1230527. The command shows basic cancellation (this could be extended if needed). It could be further improved to do asynchronous IO writes as well. v4: - rebased, mostly adapting to new OOB code (there was not much feedback in v3 for the async command part, but preliminary patches got merged!) - drop the RFC status v3: - complete rework, dropping the asynchronous commands visibility from the protocol side entirely (until there is a real need for it) - rebased, with a few preliminary cleanup patches - teach asynchronous commands to HMP v2: - documentation fixes and improvements - fix calling async commands sync without id - fix bad hmp monitor assert - add a few extra asserts - add async with no-id failure and screendump test Marc-André Lureau (20): qmp: constify QmpCommand and list json-lexer: make it safe to call destroy multiple times qmp: add QmpSession QmpSession: add a return callback QmpSession: add json parser and use it in qga monitor: use qmp session to parse json feed qga: simplify dispatch_return_cb QmpSession: introduce QmpReturn qmp: simplify qmp_return_error() QmpSession: keep a queue of pending commands QmpSession: return orderly qmp: introduce asynchronous command type scripts: learn 'async' qapi commands qmp: add qmp_return_is_cancelled() monitor: add qmp_return_get_monitor() console: add graphic_hw_update_done() console: make screendump asynchronous monitor: start making qmp_human_monitor_command() asynchronous monitor: teach HMP about asynchronous commands hmp: call the asynchronous QMP screendump to fix outdated/glitches qapi/misc.json | 3 +- qapi/ui.json | 3 +- scripts/qapi/commands.py | 151 ++++++++++++++--- scripts/qapi/common.py | 15 +- scripts/qapi/doc.py | 3 +- scripts/qapi/introspect.py | 3 +- hmp.h | 3 +- include/monitor/monitor.h | 3 + include/qapi/qmp/dispatch.h | 89 +++++++++- include/qapi/qmp/json-parser.h | 7 +- include/ui/console.h | 5 + hmp.c | 6 +- hw/display/qxl-render.c | 9 +- hw/display/qxl.c | 1 + monitor.c | 198 ++++++++++++++-------- qapi/qmp-dispatch.c | 214 +++++++++++++++++++----- qapi/qmp-registry.c | 33 +++- qga/commands.c | 2 +- qga/main.c | 51 ++---- qobject/json-lexer.c | 5 +- qobject/json-streamer.c | 3 +- tests/test-qmp-cmds.c | 206 +++++++++++++++++++---- ui/console.c | 100 +++++++++-- hmp-commands.hx | 3 +- tests/qapi-schema/qapi-schema-test.json | 5 + tests/qapi-schema/qapi-schema-test.out | 8 + tests/qapi-schema/test-qapi.py | 8 +- 27 files changed, 877 insertions(+), 260 deletions(-) -- 2.21.0.196.g041f5ea1cf
next reply other threads:[~2019-04-09 16:10 UTC|newest] Thread overview: 64+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-04-09 16:09 Marc-André Lureau [this message] 2019-04-09 16:09 ` [Qemu-devel] [PATCH v4 00/20] monitor: add asynchronous command type Marc-André Lureau 2019-04-09 16:09 ` [Qemu-devel] [PATCH v4 01/20] qmp: constify QmpCommand and list Marc-André Lureau 2019-04-09 16:09 ` Marc-André Lureau 2019-04-09 16:09 ` [Qemu-devel] [PATCH v4 02/20] json-lexer: make it safe to call destroy multiple times Marc-André Lureau 2019-04-09 16:09 ` Marc-André Lureau 2019-04-09 16:09 ` [Qemu-devel] [PATCH v4 03/20] qmp: add QmpSession Marc-André Lureau 2019-04-09 16:09 ` Marc-André Lureau 2019-04-09 16:09 ` [Qemu-devel] [PATCH v4 04/20] QmpSession: add a return callback Marc-André Lureau 2019-04-09 16:09 ` Marc-André Lureau 2019-04-09 16:09 ` [Qemu-devel] [PATCH v4 05/20] QmpSession: add json parser and use it in qga Marc-André Lureau 2019-04-09 16:09 ` Marc-André Lureau 2019-04-09 16:09 ` [Qemu-devel] [PATCH v4 06/20] monitor: use qmp session to parse json feed Marc-André Lureau 2019-04-09 16:09 ` Marc-André Lureau 2019-04-09 16:09 ` [Qemu-devel] [PATCH v4 07/20] qga: simplify dispatch_return_cb Marc-André Lureau 2019-04-09 16:09 ` Marc-André Lureau 2019-04-09 16:09 ` [Qemu-devel] [PATCH v4 08/20] QmpSession: introduce QmpReturn Marc-André Lureau 2019-04-09 16:09 ` Marc-André Lureau 2019-04-09 16:09 ` [Qemu-devel] [PATCH v4 09/20] qmp: simplify qmp_return_error() Marc-André Lureau 2019-04-09 16:09 ` Marc-André Lureau 2019-04-09 16:09 ` [Qemu-devel] [PATCH v4 10/20] QmpSession: keep a queue of pending commands Marc-André Lureau 2019-04-09 16:09 ` Marc-André Lureau 2019-04-09 16:10 ` [Qemu-devel] [PATCH v4 11/20] QmpSession: return orderly Marc-André Lureau 2019-04-09 16:10 ` Marc-André Lureau 2019-04-09 16:10 ` [Qemu-devel] [PATCH v4 12/20] qmp: introduce asynchronous command type Marc-André Lureau 2019-04-09 16:10 ` Marc-André Lureau 2019-04-09 16:10 ` [Qemu-devel] [PATCH v4 13/20] scripts: learn 'async' qapi commands Marc-André Lureau 2019-04-09 16:10 ` Marc-André Lureau 2019-04-09 16:10 ` [Qemu-devel] [PATCH v4 14/20] qmp: add qmp_return_is_cancelled() Marc-André Lureau 2019-04-09 16:10 ` Marc-André Lureau 2019-04-09 16:10 ` [Qemu-devel] [PATCH v4 15/20] monitor: add qmp_return_get_monitor() Marc-André Lureau 2019-04-09 16:10 ` Marc-André Lureau 2019-04-09 16:10 ` [Qemu-devel] [PATCH v4 16/20] console: add graphic_hw_update_done() Marc-André Lureau 2019-04-09 16:10 ` Marc-André Lureau 2019-04-10 8:43 ` Gerd Hoffmann 2019-04-10 8:43 ` Gerd Hoffmann 2019-04-09 16:10 ` [Qemu-devel] [PATCH v4 17/20] console: make screendump asynchronous Marc-André Lureau 2019-04-09 16:10 ` Marc-André Lureau 2019-04-10 8:48 ` Gerd Hoffmann 2019-04-10 8:48 ` Gerd Hoffmann 2019-07-15 19:09 ` Marc-André Lureau 2019-07-30 11:23 ` Gerd Hoffmann 2019-04-09 16:10 ` [Qemu-devel] [PATCH v4 18/20] monitor: start making qmp_human_monitor_command() asynchronous Marc-André Lureau 2019-04-09 16:10 ` Marc-André Lureau 2019-04-09 16:10 ` [Qemu-devel] [PATCH v4 19/20] monitor: teach HMP about asynchronous commands Marc-André Lureau 2019-04-09 16:10 ` Marc-André Lureau 2019-04-09 16:10 ` [Qemu-devel] [PATCH v4 20/20] hmp: call the asynchronous QMP screendump to fix outdated/glitches Marc-André Lureau 2019-04-09 16:10 ` Marc-André Lureau 2019-04-09 19:05 ` [Qemu-devel] [PATCH v4 00/20] monitor: add asynchronous command type no-reply 2019-04-09 19:05 ` no-reply 2019-04-09 19:07 ` no-reply 2019-04-09 19:07 ` no-reply 2019-05-17 22:21 ` Marc-André Lureau 2019-05-21 14:17 ` Markus Armbruster 2019-05-21 14:50 ` Marc-André Lureau 2019-05-23 7:52 ` Markus Armbruster 2019-05-25 12:12 ` Marc-André Lureau 2019-05-27 8:18 ` Markus Armbruster 2019-05-27 9:07 ` Gerd Hoffmann 2019-05-27 13:23 ` Markus Armbruster 2019-05-27 15:17 ` Marc-André Lureau 2019-05-28 6:09 ` Gerd Hoffmann 2019-05-31 9:15 ` Kevin Wolf 2019-05-21 20:50 ` John Snow
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190409161009.6322-1-marcandre.lureau@redhat.com \ --to=marcandre.lureau@redhat.com \ --cc=armbru@redhat.com \ --cc=dgilbert@redhat.com \ --cc=eblake@redhat.com \ --cc=kraxel@redhat.com \ --cc=mdroth@linux.vnet.ibm.com \ --cc=qemu-devel@nongnu.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.