Re: Making QEMU easier for management tools and applications

[Christophe de Dinechin <dinechin@redhat.com>]

> On 23 Jan 2020, at 18:58, John Snow <jsnow@redhat.com> wrote:
> 
> 
> 
> On 1/23/20 2:19 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> On 12/24/19 8:41 AM, Daniel P. Berrangé wrote:
>>>>> * scripts/qmp/qmp-shell
>>>>> 
>>>>>  Half-hearted attempt at a human-friendly wrapper around the JSON
>>>>>  syntax.  I have no use for this myself.
>>>> I use this fairly often as its a useful debugging / experimentation
>>>> / trouble shooting tool. There's similar ish functionality in
>>>> virsh qemu-monitor-command. I think there's scope of a supported
>>>> tool here that can talk to libvirt or a UNIX socket for doing
>>>> QMP commands, with a friendlier syntax & pretty printing. 
>>>> 
>>> 
>>> qmp-shell is one of my go-to tools for working through bitmap workflows
>>> where we don't have convenience commands yet, as some of the setups
>>> required for fleecing et al involve quite a number of steps.
>>> 
>>> I can copy-paste raw JSON into a socket, but personally I like seeing my
>>> commands neatly organized in a format where I can visually reduce them
>>> to their components at a glance.
>>> 
>>> (What I mean is: It's hard to remember which QMP commands you've barfed
>>> into a terminal because JSON is hard to read and looks very visually
>>> repetitive.)
>>> 
>>> I tried to rewrite qmp-shell late last year, actually. I wanted to write
>>> a new REPL that was json-aware in some manner such that you could write
>>> multi-line commands like this:
>>> 
>>>> example-command arg={
>>>  "hello": "world"
>>> }
>>> 
>>> This requires, sadly, a streamable JSON parser. Most JSON parsers built
>>> into Python as-is simply take a file pointer and consume the entirety of
>>> the rest of the stream -- they don't play very nice with incomplete
>>> input or input that may have trailing data, e.g.:
>>> 
>>>> example-command arg={
>>>  "hello": "world"
>>> } arg2={
>>>  "oops!": "more json!"
>>> }
>> 
>> QMP is in the same boat: it needs to process input that isn't
>> necessarily full expressions (JSON-text in the RFC's grammar).
>> 
>> Any conventional parser can be made streaming by turning it into a
>> coroutine.  This is probably the simplest solution for handwritten
>> streaming LL parsers, because it permits recursive descent.  In Python,
>> I'd try a generator.
>> 
>> Our actual solution for QMP predates coroutine support in QEMU, and is
>> rather hamfisted:
>> 
>> * Streaming lexer: it gets fed characters one at a time, and when its
>>  state machine says "token complete", it feeds the token to the
>>  "streamer".
>> 
>> * "Streamer": gets fed tokens one at a time, buffers them up counting
>>  curly and square bracket nesting until the nesting is zero, then
>>  passes the buffered tokens to the parser.
>> 
>> * Non-streaming parser: it gets fed a sequence of tokens that constitute
>>  a full expression.
>> 
>> The best I can say about this is that it works.  The streamer's token
>> buffer eats a lot of memory compared to a real streaming parser, but in
>> practice, it's a drop in the bucket.
>> 
> 
> I looked into this at one point. I forget why I didn't like it. I had
> some notion that I should replace this one too, but forget exactly why.
> Maybe it wasn't that bad, if I've forgotten.
> 
>>> Also, due to the nature of JSON as being a single discrete object and
>>> never a stream of objects, no existing JSON parser really supports the
>>> idea of ever seeing more than one object per buffer.
>> 
>> That plainly sucks.
>> 
>>> ...So I investigated writing a proper grammar for qmp-shell.
>> 
>> Any parser must start with a proper grammar.  If it doesn't, it's a toy,
>> or a highway to madness.
>> 
>>> Unfortunately, this basically means including the JSON grammar as a
>>> subset of the shell grammar and writing your own parser for it entirely.
>> 
>> Because qmp-shell is a half-hearted wrapper: we ran out of wrapping
>> paper, so JSON sticks out left and right.
>> 
>> Scrap and start over.
>> 
>>> I looked into using Python's own lexer; but it's designed to lex
>>> *python*, not *json*. I got a prototype lexer working for this purpose
>>> under a grammar that I think reflects JSON, but I got that sinking
>>> feeling that it was all more trouble than it was worth, and scrapped
>>> working on it any further.
>> 
>> Parsing JSON is pretty simple.  Data point: QAPISchemaParser parses our
>> weird derivative of JSON in 239 SLOC.
>> 
>>> I did not find any other flex/yacc-like tools that seemed properly
>>> idiomatic or otherwise heavily specialized. I gave up on the idea of
>>> writing a new parser.
>> 
>> While I recommend use of tools for parsing non-trivial grammars (you'll
>> screw up, they won't), they're massive overkill for JSON.
>> 
>>> I'd love to offer a nice robust QMP shell that is available for use by
>>> end users, but the syntax of the shell will need some major considerations.
>> 
>> Scrap and start over.
>> 
>> [...]
>> 
> 
> Yes, I agree: Scrap and start over.
> 
> What SHOULD the syntax look like, though? Clearly the idea of qmp-shell
> is that it offers a convenient way to enter the top-level keys of the
> arguments dict. This works absolutely fine right up until you need to
> start providing nested definitions.

Well, if you are really ready to start from scratch, I might offer the XL syntax
as a starting point for a discussion of a user-visible syntax that is also
applicable for text-based or binary API exchanges.

I’m going to talk about it at FOSDEM in the “minimalist languages” design.
Those who are in Brussels might want to attend to get a better feel.
Source code is here: https://github.com/c3d/xl, but the only part you
care about for this discussion is src/{parser,scanner}.{c,h} and the
syntax configuration file src/xl.syntax. As well as renderer styles
src/xl.stylesheet, src/html.stylesheet, etc.

Key points for the use case considered:
- Tiny (~2000 lines of code for parser/scanner, a C and a C++ implementation)
- Fully introspectable, serializable in a cross-platform way, printable (with styles)
- Character-precise position tracking for error printing
- Parser preserves comments (for documentation generators)
- Small, if slow, interpreter in about 20K lines of code (~bash speed on some tests)
  meaning we would get a “qemu scripting language” with loops, tests, arithmetic, etc.

More detailed discussion at end of this mail if you think it warrants a second look.
In any case, if it helps, I’d be happy to help connecting it to qemu…

> 
> For the nesting, we say: "Go ahead and use JSON, but you have to take
> all the spaces out."

Here, that would be A.B.C, which parses as
(infix.
 (infix.
  A
  B)
 C
)

(result of `xl -nobuiltins -parse test.xl -style debug -show`)

Also, an example given earlier:

 { 'command': 'iothread-set-poll-params',
   'data': {
       'id': 'str',
	'*max-ns': 'uint64',
	'*grow': 'uint64',
	'*shrink': 'uint64'
   },
   'map-to-qom-set': 'IOThread'
 }

could be written as:

command iothread_set_poll_params
	data
		id : str
		*max_ns : uint64
		*grow : uint64
		*shrink : uint64
	map_to_qom_set IOThread

But if you want to keep the original syntax, it seems to parse and render practically OK:

% cat /tmp/a.xl 
{ 'command': 'iothread-set-poll-params',
   'data': {
       'id': 'str',
        '*max-ns': 'uint64',
        '*grow': 'uint64',
        '*shrink': 'uint64'
   },
   'map-to-qom-set': 'IOThread'
 }

%xl -nobuiltins -parse /tmp/a.xl  -show            
{ 'command':'iothread-set-poll-params', 'data': { 'id':'str', 
    '*max-ns':'uint64', '*grow':'uint64', '*shrink':'uint64'
}, 'map-to-qom-set':'IOThread' }

This is with no change to the XL parser / scanner code
whatsoever, not even to the syntax file. So that gives me hope
that we could have a “reasonably good” compatibility mode
that transforms the quasi-JSON format into the new form,
with a single parser accepting both.

> 
> This... works, charitably, but is hardly what I would call usable.
> 
> For the CLI, we offer a dot syntax notation that resembles nothing in
> particular. It often seems the case that it isn't expressive enough to
> map losslessly to JSON. I suspect it doesn't handle siblings very well.
> 
> A proper HMP-esque TUI would likely have need of coming up with its own
> pet syntax for commands that avoid complicated nested JSON definitions,
> but for effort:value ratio, having a QMP shorthand shell that works
> arbitrarily with any command might be a better win.

The XL proposal here would be to have a single format shared by
- The source definitions used to generate C code
- The monitor / internal shell syntax
- The command-line syntax
- The API data (possibly in serialized form for compactness)

> 
> Do we still have a general-case problem of how to represent QAPI
> structures in plaintext? Will this need to be solved for the CLI, too?
> 
> --js

More info below.

Here are some aspects that I think are interesting about it:

- Tiny (2000 lines of code for scanner and parser, ~20K for a full interpreter)
C:  wc parser.c parser.h scanner.c scanner.h 
     716    2183   26702 parser.c
     100     440    3372 parser.h
     926    2966   30537 scanner.c
     206     945    8249 scanner.h
    1948    6534   68860 total

C++:
     726    2372   26918 parser.cpp
     885    2480   26363 scanner.cpp
     248    1025    8867 ../include/scanner.h
     166     687    5958 ../include/parser.h
    2025    6564   68106 total

- Simple (parse tree with 8 node types, integer, real, name/symbol, text, infix, prefix, postfix and block)

	+ integer, e.g. 12, 1_000_000 or 16#33A or 2#10101
	+ real, e.g. 11.3, 16#1.FFF#e-3, 2#1.01
	+ text, e.g. “ABC”, ‘ABC’, <<Long text, multi-lines>> (configurable separators)
	+ name/symbols, a.g. Foo_Bar, +, <=, (precedence and spelling configurable)

	+ infix, e.g. A+B, A and B
	+ prefix, e.g. +3, sin X
	+ postfix, e.g. 3%, 3 km
	+ block, e.g. [A], (A), {A} and indentation blocks

- Fully introspectable (mostly because the parse tree is simple)
- Reversible, i.e. can be printed, including with formatting, e.g.:
	% xl -nobuiltins -parse demo/1-hello.xl -show
	tell "localhost”, 
	    print "Hello World”
	% xl -nobuiltins -parse demo/1-hello.xl -style debug -show   
	(prefix
	 tell
	 (infix,
	  “localhost"
	  (block indent
	   (prefix
	    print
	    "Hello World”
	   ))))
- Also has a binary serializer that produces a platform-independent format
- Has multiple implementations, notably C and C++ implementation (and even one in XL :-)
- Validated on thousands of lines of input, with various language styles (e.g. Ada-like or functional)
- Character-level position tracking for error messages in scripts / config files:
	/tmp/xl.xl:1007:8: Mismatched identation, expected “)"
	/tmp/xl.xl:2409:23: Mismatched identation, expected ""
- Designed to be easy to read and write
- Powerful enough to parse itself (https://github.com/c3d/xl/blob/master/xl2/native/xl.parser.xl)
- Dynamically configurable syntax (spelling and precedence of operators)
- Multi-line text with configurable separators, e.g. the following can be made a text constant by having XML and END_XML as text separators:
     XML
         <stuff>
            Insert your XML here
        </stuff> 
    END_XML
- Based-numbers in any base, e.g. 8#777, 16#FFFF_FFFF and 2#1.001 as valid numbers
- Has essentially a single contributor (me), so easy to relicense as needed
- There is an interpreter, e.g. potential evaluate expressions like 2+3*A
- Relatively fast (6.1s to parse 1M lines of code representing 40M of code, cpp ~1s)
% wc /tmp/tmp.xl
 1000000 3893922 41679700 /tmp/tmp.xl
% time xl -parse /tmp/tmp.xl
 6.10s user 0.21s system 99% cpu 6.346 total
- Support multiple styles, e.g. using { } for blocks or indentation, parentheses or not, etc.

Cons (but I’m not the better person to come up with cons on this pet project of mine ;-):
- Idiosyncratic
- Single contributor
- Not well maintained
- Definitely not production quality (even the makefiles are broken ;-)
- Has some CI testing, but it fails, and it’s totally insufficient
- Interpreter far from perfect
- Designed with another purpose in mind (a programming language)
- Syntax is not C-centric, e.g. 16#FFFF instead of 0xFFFF.
- Name syntax does not allow -, i.e. max-ns is “max minus ns”, max_ns OK.
- [insert probably about a thousand others here]

Precedences and other stuff can be configured dynamically, through a file
in the current implementations, eg. https://github.com/c3d/xl-c/blob/master/xl.syntax.
So that means we can have a “nice” syntax for the commands and objects,
and a format that can serve both as a config file format, as a command
language, and as a full shell-style language with if, loops, etc.

It also supports nested syntaxes, i.e. dynamic changes of precedence
between selected separators. Used to support simplified C syntax
with “extern int foo();”, where the “C” syntax is active between “extern”
and “;”. Could be useful for compatibility.

Parse tree is simple enough that it’s fully introspectable.

There is a (configurable) renderer, so you can generate source from the
internal data structure. The renderer can generate colorized source
code in HTML, so I guess we could generate C data structures relatively
easily.

I believe that it is relatively trivial to configure the parser syntax file 
to accept the QEMU quasi-JSON. (some code changes required to teach
it to totally ignore whitespace, toi avoid error messages).

More complete documentation about the language is here:
https://c3d.github.io/xl, but it’s quite light on implementation details.
So read only if you have a bit of time.