From: John Snow <jsnow@redhat.com>
To: Markus Armbruster <armbru@redhat.com>
Cc: "Kevin Wolf" <kwolf@redhat.com>,
"Peter Maydell" <peter.maydell@linaro.org>,
"Daniel P. Berrangé" <berrange@redhat.com>,
"Denis V. Lunev" <den@virtuozzo.com>,
"Cleber Rosa" <cleber@redhat.com>,
"Stefan Hajnoczi" <stefanha@gmail.com>,
qemu-devel <qemu-devel@nongnu.org>,
"Eduardo Habkost" <ehabkost@redhat.com>,
"Paolo Bonzini" <pbonzini@redhat.com>,
"Marc-André Lureau" <marcandre.lureau@redhat.com>,
"Dominik Csapak" <d.csapak@proxmox.com>
Subject: Re: Making QEMU easier for management tools and applications
Date: Thu, 23 Jan 2020 12:58:45 -0500 [thread overview]
Message-ID: <a41ae09b-021f-2fda-0b03-7b37c5624ab3@redhat.com> (raw)
In-Reply-To: <878slyej29.fsf@dusky.pond.sub.org>
On 1/23/20 2:19 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
>
>> On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
>>>> * scripts/qmp/qmp-shell
>>>>
>>>> Half-hearted attempt at a human-friendly wrapper around the JSON
>>>> syntax. I have no use for this myself.
>>> I use this fairly often as its a useful debugging / experimentation
>>> / trouble shooting tool. There's similar ish functionality in
>>> virsh qemu-monitor-command. I think there's scope of a supported
>>> tool here that can talk to libvirt or a UNIX socket for doing
>>> QMP commands, with a friendlier syntax & pretty printing.
>>>
>>
>> qmp-shell is one of my go-to tools for working through bitmap workflows
>> where we don't have convenience commands yet, as some of the setups
>> required for fleecing et al involve quite a number of steps.
>>
>> I can copy-paste raw JSON into a socket, but personally I like seeing my
>> commands neatly organized in a format where I can visually reduce them
>> to their components at a glance.
>>
>> (What I mean is: It's hard to remember which QMP commands you've barfed
>> into a terminal because JSON is hard to read and looks very visually
>> repetitive.)
>>
>> I tried to rewrite qmp-shell late last year, actually. I wanted to write
>> a new REPL that was json-aware in some manner such that you could write
>> multi-line commands like this:
>>
>>> example-command arg={
>> "hello": "world"
>> }
>>
>> This requires, sadly, a streamable JSON parser. Most JSON parsers built
>> into Python as-is simply take a file pointer and consume the entirety of
>> the rest of the stream -- they don't play very nice with incomplete
>> input or input that may have trailing data, e.g.:
>>
>>> example-command arg={
>> "hello": "world"
>> } arg2={
>> "oops!": "more json!"
>> }
>
> QMP is in the same boat: it needs to process input that isn't
> necessarily full expressions (JSON-text in the RFC's grammar).
>
> Any conventional parser can be made streaming by turning it into a
> coroutine. This is probably the simplest solution for handwritten
> streaming LL parsers, because it permits recursive descent. In Python,
> I'd try a generator.
>
> Our actual solution for QMP predates coroutine support in QEMU, and is
> rather hamfisted:
>
> * Streaming lexer: it gets fed characters one at a time, and when its
> state machine says "token complete", it feeds the token to the
> "streamer".
>
> * "Streamer": gets fed tokens one at a time, buffers them up counting
> curly and square bracket nesting until the nesting is zero, then
> passes the buffered tokens to the parser.
>
> * Non-streaming parser: it gets fed a sequence of tokens that constitute
> a full expression.
>
> The best I can say about this is that it works. The streamer's token
> buffer eats a lot of memory compared to a real streaming parser, but in
> practice, it's a drop in the bucket.
>
I looked into this at one point. I forget why I didn't like it. I had
some notion that I should replace this one too, but forget exactly why.
Maybe it wasn't that bad, if I've forgotten.
>> Also, due to the nature of JSON as being a single discrete object and
>> never a stream of objects, no existing JSON parser really supports the
>> idea of ever seeing more than one object per buffer.
>
> That plainly sucks.
>
>> ...So I investigated writing a proper grammar for qmp-shell.
>
> Any parser must start with a proper grammar. If it doesn't, it's a toy,
> or a highway to madness.
>
>> Unfortunately, this basically means including the JSON grammar as a
>> subset of the shell grammar and writing your own parser for it entirely.
>
> Because qmp-shell is a half-hearted wrapper: we ran out of wrapping
> paper, so JSON sticks out left and right.
>
> Scrap and start over.
>
>> I looked into using Python's own lexer; but it's designed to lex
>> *python*, not *json*. I got a prototype lexer working for this purpose
>> under a grammar that I think reflects JSON, but I got that sinking
>> feeling that it was all more trouble than it was worth, and scrapped
>> working on it any further.
>
> Parsing JSON is pretty simple. Data point: QAPISchemaParser parses our
> weird derivative of JSON in 239 SLOC.
>
>> I did not find any other flex/yacc-like tools that seemed properly
>> idiomatic or otherwise heavily specialized. I gave up on the idea of
>> writing a new parser.
>
> While I recommend use of tools for parsing non-trivial grammars (you'll
> screw up, they won't), they're massive overkill for JSON.
>
>> I'd love to offer a nice robust QMP shell that is available for use by
>> end users, but the syntax of the shell will need some major considerations.
>
> Scrap and start over.
>
> [...]
>
Yes, I agree: Scrap and start over.
What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
is that it offers a convenient way to enter the top-level keys of the
arguments dict. This works absolutely fine right up until you need to
start providing nested definitions.
For the nesting, we say: "Go ahead and use JSON, but you have to take
all the spaces out."
This... works, charitably, but is hardly what I would call usable.
For the CLI, we offer a dot syntax notation that resembles nothing in
particular. It often seems the case that it isn't expressive enough to
map losslessly to JSON. I suspect it doesn't handle siblings very well.
A proper HMP-esque TUI would likely have need of coming up with its own
pet syntax for commands that avoid complicated nested JSON definitions,
but for effort:value ratio, having a QMP shorthand shell that works
arbitrarily with any command might be a better win.
Do we still have a general-case problem of how to represent QAPI
structures in plaintext? Will this need to be solved for the CLI, too?
--js
next prev parent reply other threads:[~2020-01-23 20:27 UTC|newest]
Thread overview: 183+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-12-20 16:13 Making QEMU easier for management tools and applications Stefan Hajnoczi
2019-12-20 21:07 ` Richard W.M. Jones
2020-01-02 11:26 ` Stefan Hajnoczi
2019-12-21 9:02 ` Markus Armbruster
2019-12-23 15:04 ` Michal Prívozník
2020-01-07 9:36 ` Kevin Wolf
2020-01-07 10:55 ` Michal Privoznik
2020-01-07 12:57 ` Kevin Wolf
2020-01-07 17:53 ` Christophe de Dinechin
2019-12-24 13:41 ` Daniel P. Berrangé
2020-01-22 22:28 ` John Snow
2020-01-23 7:19 ` Markus Armbruster
2020-01-23 17:58 ` John Snow [this message]
2020-01-23 19:01 ` Daniel P. Berrangé
2020-01-23 21:07 ` John Snow
2020-01-24 7:59 ` Markus Armbruster
2020-01-24 10:27 ` Daniel P. Berrangé
2020-01-24 14:38 ` Kevin Wolf
2020-01-24 18:23 ` John Snow
2020-01-24 18:30 ` Dr. David Alan Gilbert
2020-01-24 18:48 ` John Snow
2020-01-24 18:52 ` Dr. David Alan Gilbert
2020-01-24 18:58 ` John Snow
2020-01-25 10:18 ` Markus Armbruster
2020-01-27 10:18 ` Daniel P. Berrangé
2020-01-27 12:48 ` Markus Armbruster
2020-01-27 11:56 ` Kevin Wolf
2020-01-27 12:04 ` Peter Maydell
2020-01-27 20:11 ` John Snow
2020-01-27 22:38 ` Paolo Bonzini
2020-01-28 0:37 ` John Snow
2020-01-28 10:16 ` Daniel P. Berrangé
2020-01-28 10:39 ` Kevin Wolf
2020-01-28 15:36 ` Markus Armbruster
2020-01-31 12:25 ` Eric Blake
2020-01-28 10:28 ` Kevin Wolf
2020-01-28 12:36 ` Markus Armbruster
2020-01-28 12:54 ` Kevin Wolf
2020-01-28 13:45 ` Gerd Hoffmann
2020-01-31 6:50 ` Markus Armbruster
2020-01-31 7:48 ` Paolo Bonzini
2020-01-31 8:09 ` Markus Armbruster
2020-02-03 20:07 ` Andrea Bolognani
2020-02-04 9:58 ` Markus Armbruster
2020-01-31 12:27 ` Eric Blake
2020-02-02 9:21 ` Kevin Wolf
2020-02-02 10:44 ` Paolo Bonzini
2020-02-03 6:20 ` Markus Armbruster
2020-02-03 8:48 ` Markus Armbruster
2020-01-27 20:12 ` Dr. David Alan Gilbert
2020-01-24 20:34 ` John Snow
2020-01-27 8:35 ` Gerd Hoffmann
2020-01-27 12:13 ` Kevin Wolf
2020-01-27 16:18 ` Gerd Hoffmann
2020-01-24 9:50 ` Daniel P. Berrangé
2020-01-25 11:52 ` Paolo Bonzini
2020-01-27 10:05 ` Daniel P. Berrangé
2020-01-27 8:25 ` Tooling to help humans use JSON (was: Making QEMU easier for management tools and applications) Markus Armbruster
2020-01-27 9:06 ` Making QEMU easier for management tools and applications Markus Armbruster
2020-01-27 10:00 ` Daniel P. Berrangé
2020-01-27 14:35 ` Kevin Wolf
2020-01-27 20:29 ` Dr. David Alan Gilbert
2020-01-28 10:59 ` Kevin Wolf
2020-02-05 13:09 ` Kevin Wolf
2020-02-05 19:09 ` qmp-shell for GSoC/Outreachy? (Was: Re: Making QEMU easier for management tools and applications) John Snow
2020-02-05 19:49 ` Dr. David Alan Gilbert
2020-02-06 9:40 ` qmp-shell for GSoC/Outreachy? Markus Armbruster
2020-02-06 10:09 ` Daniel P. Berrangé
2020-02-06 12:11 ` Markus Armbruster
2020-02-06 12:15 ` Daniel P. Berrangé
2020-02-06 18:02 ` Dr. David Alan Gilbert
2020-02-07 21:03 ` John Snow
2020-02-08 7:17 ` Markus Armbruster
2020-02-06 14:21 ` Kevin Wolf
2020-02-06 18:26 ` Dr. David Alan Gilbert
2020-02-07 10:49 ` Kevin Wolf
2020-02-07 21:23 ` John Snow
2020-02-08 7:25 ` Markus Armbruster
2020-02-10 11:59 ` Kevin Wolf
2020-02-10 12:26 ` Kevin Wolf
2020-02-06 18:18 ` Dr. David Alan Gilbert
2020-02-07 7:47 ` Markus Armbruster
2020-02-07 21:31 ` Eric Blake
2020-02-08 7:34 ` Markus Armbruster
2020-02-07 21:56 ` John Snow
2020-02-07 20:56 ` John Snow
2020-01-27 20:59 ` Making QEMU easier for management tools and applications John Snow
2020-01-28 10:16 ` Markus Armbruster
2020-01-28 19:21 ` John Snow
2020-01-24 6:38 ` Markus Armbruster
2020-01-25 22:34 ` Christophe de Dinechin
2020-01-25 11:55 ` Paolo Bonzini
2020-01-02 14:47 ` Stefan Hajnoczi
2020-01-16 11:03 ` Kashyap Chamarthy
2020-01-20 9:55 ` Stefan Hajnoczi
2020-01-20 13:57 ` Kashyap Chamarthy
2020-01-25 11:41 ` Paolo Bonzini
2020-01-27 19:41 ` John Snow
2020-01-02 15:05 ` Dr. David Alan Gilbert
2020-01-13 13:44 ` Markus Armbruster
2019-12-24 13:00 ` Daniel P. Berrangé
2020-01-02 14:22 ` Stefan Hajnoczi
2020-01-22 22:42 ` John Snow
2020-01-23 7:21 ` Markus Armbruster
2020-01-23 10:27 ` Daniel P. Berrangé
2020-01-23 18:13 ` John Snow
2020-01-23 19:12 ` Daniel P. Berrangé
2020-01-02 15:10 ` Dr. David Alan Gilbert
2020-01-07 17:11 ` Christophe de Dinechin
2020-01-08 10:43 ` Kevin Wolf
2020-01-08 11:40 ` Christophe de Dinechin
2020-01-08 13:38 ` Kevin Wolf
2020-01-14 13:04 ` Markus Armbruster
2020-01-14 17:31 ` Christophe de Dinechin
2020-01-15 9:20 ` Markus Armbruster
2020-01-15 9:34 ` Christophe de Dinechin
2020-01-15 12:15 ` Markus Armbruster
2020-01-15 12:19 ` Daniel P. Berrangé
2020-01-15 14:02 ` Markus Armbruster
2020-01-30 21:09 ` Improving QOM documentation [Was: Re: Making QEMU easier for management tools and applications] Kashyap Chamarthy
2020-01-31 6:11 ` Markus Armbruster
2020-01-31 7:46 ` Paolo Bonzini
2020-01-31 15:37 ` Christophe de Dinechin
2020-01-31 16:28 ` Paolo Bonzini
2020-01-31 9:50 ` Kashyap Chamarthy
2020-01-31 10:35 ` Peter Maydell
2020-01-31 11:02 ` Paolo Bonzini
2020-01-31 15:22 ` Kashyap Chamarthy
2020-01-31 17:23 ` Markus Armbruster
2020-02-03 8:56 ` Paolo Bonzini
2020-02-03 9:54 ` Markus Armbruster
2020-02-03 15:21 ` Paolo Bonzini
2020-02-04 8:42 ` Markus Armbruster
2020-01-31 16:39 ` Markus Armbruster
2020-01-20 10:08 ` Making QEMU easier for management tools and applications Stefan Hajnoczi
2020-01-21 5:42 ` Markus Armbruster
2020-01-21 11:32 ` Stefan Hajnoczi
2020-01-21 12:03 ` Marc-André Lureau
2020-01-21 13:36 ` Integrating QOM into QAPI (was: Making QEMU easier for management tools and applications) Markus Armbruster
2020-01-21 14:36 ` Daniel P. Berrangé
2020-01-21 15:01 ` Integrating QOM into QAPI Markus Armbruster
2020-01-21 15:11 ` Marc-André Lureau
2020-01-21 16:21 ` Peter Maydell
2020-01-22 5:16 ` Getting whole-tree patches reviewed and merged (was: Integrating QOM into QAPI) Markus Armbruster
2020-02-07 21:53 ` Getting whole-tree patches reviewed and merged Eric Blake
2020-02-10 11:26 ` Paolo Bonzini
2020-02-10 16:04 ` Markus Armbruster
2020-02-10 16:12 ` Peter Maydell
2020-01-22 10:50 ` Integrating QOM into QAPI Alex Bennée
2020-01-22 12:24 ` Markus Armbruster
2020-01-22 12:42 ` Marc-André Lureau
2020-01-22 13:28 ` Peter Maydell
2020-01-22 13:32 ` Marc-André Lureau
2020-01-23 7:37 ` Markus Armbruster
2020-01-24 18:32 ` Paolo Bonzini
2020-01-25 4:44 ` Marc-André Lureau
2020-01-25 9:28 ` Paolo Bonzini
2020-01-25 21:25 ` Peter Maydell
2020-01-26 8:09 ` Christophe de Dinechin
2020-01-26 9:11 ` Marc-André Lureau
2020-01-26 16:47 ` Paolo Bonzini
2020-01-27 19:05 ` Christophe de Dinechin
2020-01-27 19:05 ` Christophe de Dinechin
2020-01-26 15:04 ` Peter Maydell
2020-01-27 19:05 ` Christophe de Dinechin
2020-01-28 8:00 ` Markus Armbruster
2020-01-28 10:03 ` Daniel P. Berrangé
2020-01-29 12:42 ` Christophe de Dinechin
2020-01-15 9:35 ` Making QEMU easier for management tools and applications Marc-André Lureau
2020-01-15 12:25 ` Markus Armbruster
2020-01-25 17:18 ` Paolo Bonzini
2020-01-27 9:30 ` Markus Armbruster
2020-01-13 16:30 ` Stefan Hajnoczi
2020-02-04 15:54 ` Summary of " Markus Armbruster
2020-02-05 6:38 ` Markus Armbruster
2020-02-10 10:56 ` Stefan Hajnoczi
2020-02-10 11:01 ` Peter Maydell
2020-02-10 11:08 ` Daniel P. Berrangé
2020-02-10 11:29 ` Peter Maydell
2020-02-10 11:04 ` Paolo Bonzini
2020-02-10 16:43 ` Markus Armbruster
2020-02-12 13:54 ` Stefan Hajnoczi
2020-02-12 14:03 ` Daniel P. Berrangé
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a41ae09b-021f-2fda-0b03-7b37c5624ab3@redhat.com \
--to=jsnow@redhat.com \
--cc=armbru@redhat.com \
--cc=berrange@redhat.com \
--cc=cleber@redhat.com \
--cc=d.csapak@proxmox.com \
--cc=den@virtuozzo.com \
--cc=ehabkost@redhat.com \
--cc=kwolf@redhat.com \
--cc=marcandre.lureau@redhat.com \
--cc=pbonzini@redhat.com \
--cc=peter.maydell@linaro.org \
--cc=qemu-devel@nongnu.org \
--cc=stefanha@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).