[Qemu-devel] Writing a C library to generate qemu command lines and configuration files

[Qemu-devel] Writing a C library to generate qemu command lines and configuration files

[Richard W.M. Jones]

Firstly, is there such a thing already?  (libvirt doesn't count since
it cannot generate -readconfig configuration files)

Well, I have written one.  It's in C and doesn't have any dependencies
beyond the standard C library:

  https://github.com/rwmjones/libguestfs/tree/max-disks/common/qemuopts

It models command line parameters as either flags (eg. ‘-nodefconfig’),
flags with a single argument (‘-name guest’), or flags taking a list
(eg. ‘-drive file=foo,id=bar’ modeled as a list of strings).

I'm immediately aware that there are shortcomings with this approach
(see below).  Is there a better model to use for qemu command line
options that also allows -readconfig config files to be generated?

Some of the more obvious problems:

* I don't know if single flags can be translated into config files.

* Config files treat the "id=.." parameter as special, eg:
  ‘-drive file=foo,id=bar’ is translated to:

  [drive "bar"]
    file = "foo"

* There's no clear mapping between some command line parameters and
  the config file, eg. ‘-m 2048 -kernel foo’ should be translated into:

  [memory]
    size = "2048"
  [machine]
    kernel = "foo"

  but my library couldn't do that translation without a look-up table
  that would duplicate the internals of qemu.

There are also unknown unknowns:

* Is comma-quoting (ie. doubling any commas) sufficient?  Or are there
  other forms of quoting?  A quick look at options parsing in qemu
  doesn't show any.

* From the point of view of a client generating command lines, is there
  any significance to dotted names (eg. ‘-drive file.driver=ssh,...’)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v

Re: [Qemu-devel] Writing a C library to generate qemu command lines and configuration files

[Markus Armbruster]

"Richard W.M. Jones" <rjones@redhat.com> writes:

> Firstly, is there such a thing already?  (libvirt doesn't count since
> it cannot generate -readconfig configuration files)

I'm not aware of no such thing.

> Well, I have written one.  It's in C and doesn't have any dependencies
> beyond the standard C library:
>
>   https://github.com/rwmjones/libguestfs/tree/max-disks/common/qemuopts
>
> It models command line parameters as either flags (eg. ‘-nodefconfig’),
> flags with a single argument (‘-name guest’), or flags taking a list
> (eg. ‘-drive file=foo,id=bar’ modeled as a list of strings).

See "option arguments values have become trees" below.

> I'm immediately aware that there are shortcomings with this approach
> (see below).  Is there a better model to use for qemu command line
> options that also allows -readconfig config files to be generated?
>
> Some of the more obvious problems:
>
> * I don't know if single flags can be translated into config files.

They can't.  -readconfig/-writeconfig cover only options implemented
with QemuOpts.

> * Config files treat the "id=.." parameter as special, eg:
>   ‘-drive file=foo,id=bar’ is translated to:
>
>   [drive "bar"]
>     file = "foo"

Yes.

> * There's no clear mapping between some command line parameters and
>   the config file, eg. ‘-m 2048 -kernel foo’ should be translated into:
>
>   [memory]
>     size = "2048"
>   [machine]
>     kernel = "foo"
>
>   but my library couldn't do that translation without a look-up table
>   that would duplicate the internals of qemu.

Yes.  Two problems:

* The configuration group name (the thing right after '[') by convention
  matches the command line option name.  Since the convention isn't
  enforced, screwups have crept in.  For instance, [boot-opts] matches
  -boot, and [memory] matches -m.  We suck.  Also plagues
  query-command-line-options.

* Even when it matches, it can only match the *unsugared* option name,
  not its various sugared forms.  For instance, -kernel ARG is sugar for
  -machine kernel=ARG.

> There are also unknown unknowns:
>
> * Is comma-quoting (ie. doubling any commas) sufficient?  Or are there
>   other forms of quoting?  A quick look at options parsing in qemu
>   doesn't show any.

A quick look at QemuOpts will at best confuse, and possibly deceive.

The beginnings of its replacement in util/keyval.c comes with a *gasp*
grammar.  It doesn't have all of QemuOpts bells & whistles, at least not
yet.  You might find it useful anyway.

To answer your question: you have to double comma after '=', or else it
terminates the value.  There is no other quoting.

> * From the point of view of a client generating command lines, is there
>   any significance to dotted names (eg. ‘-drive file.driver=ssh,...’)

Right now, you can still pretend dotted names are just names.

But option arguments have really become trees.  We've shoehorned them
into the existing command line syntax with dotted keys.  Design
discussion, if you're interested:

    Subject: Non-flat command line option argument syntax
    Message-ID: <87bmukmlau.fsf@dusky.pond.sub.org>
    https://lists.gnu.org/archive/html/qemu-devel/2017-02/msg00555.html

Implementation is the keyval.c mentioned above.

I'm not sure baking "option argument is a list of (key, string) pairs,
where both key and string are strings" into your library now is a good
idea.  Perhaps it would be better to embrace "option argument is a tree"
from the start.

In the longer run, I intend to make -readconfig (or its replacement)
accept JSON, because it supports trees directly, and is less badly
defined.

We might deprecate the Windows INI syntax then.

This might leave you with more questions.  Please don't hesitate to ask
me.

Re: [Qemu-devel] Writing a C library to generate qemu command lines and configuration files

[Richard W.M. Jones]

On Tue, May 02, 2017 at 04:28:52PM +0200, Markus Armbruster wrote:
> > * Is comma-quoting (ie. doubling any commas) sufficient?  Or are there
> >   other forms of quoting?  A quick look at options parsing in qemu
> >   doesn't show any.
> 
> A quick look at QemuOpts will at best confuse, and possibly deceive.
> 
> The beginnings of its replacement in util/keyval.c comes with a *gasp*
> grammar.  It doesn't have all of QemuOpts bells & whistles, at least not
> yet.  You might find it useful anyway.
> 
> To answer your question: you have to double comma after '=', or else it
> terminates the value.  There is no other quoting.

Hmm, is that really right?  It seems to me that any comma must be
doubled.  For example:

$ qemu-system-x86_64 -name foo,bar -writeconfig - -hda die
qemu-system-x86_64: -name foo,bar: Invalid parameter 'bar'

$ qemu-system-x86_64 -name foo,,bar -writeconfig - -hda die
# qemu config file

[name]
  guest = "foo,bar"

qemu-system-x86_64: -hda die: Could not open 'die': No such file or directory

--- Or:

$ qemu-system-x86_64 -drive file=foo,,bar -writeconfig - 
# qemu config file

[drive]
  file = "foo,bar"

qemu-system-x86_64: -drive file=foo,,bar: Could not open 'foo,bar': No such file or directory

> > * From the point of view of a client generating command lines, is there
> >   any significance to dotted names (eg. ‘-drive file.driver=ssh,...’)
> 
> Right now, you can still pretend dotted names are just names.
> 
> But option arguments have really become trees.  We've shoehorned them
> into the existing command line syntax with dotted keys.  Design
> discussion, if you're interested:
> 
>     Subject: Non-flat command line option argument syntax
>     Message-ID: <87bmukmlau.fsf@dusky.pond.sub.org>
>     https://lists.gnu.org/archive/html/qemu-devel/2017-02/msg00555.html
>
> Implementation is the keyval.c mentioned above.
> 
> I'm not sure baking "option argument is a list of (key, string) pairs,
> where both key and string are strings" into your library now is a good
> idea.  Perhaps it would be better to embrace "option argument is a tree"
> from the start.

OK, I'll think about that.

> In the longer run, I intend to make -readconfig (or its replacement)
> accept JSON, because it supports trees directly, and is less badly
> defined.

And JSON actually thinks about quoting too :-)

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org

Re: [Qemu-devel] Writing a C library to generate qemu command lines and configuration files

[Paolo Bonzini]

On 02/05/2017 16:59, Richard W.M. Jones wrote:
>> To answer your question: you have to double comma after '=', or else it
>> terminates the value.  There is no other quoting.
>
> Hmm, is that really right?  It seems to me that any comma must be
> doubled.  For example:
> 
> $ qemu-system-x86_64 -name foo,bar -writeconfig - -hda die
> qemu-system-x86_64: -name foo,bar: Invalid parameter 'bar'

That's because, as you found out, "-name" is not a simple option but a
compound one; "-name foo" is short for "-name guest=foo" (compare
"-device e1000").

The grammar mentioned in util/keyval.c says that you have to double
comma in a value, *and* comma is completely invalid in a key.

If you have '=', you have to double commas after it and cannot use it
before.

If you have no '=' at all, then you have to double comma too.

Paolo

> $ qemu-system-x86_64 -name foo,,bar -writeconfig - -hda die
> # qemu config file
> 
> [name]
>   guest = "foo,bar"

Re: [Qemu-devel] Writing a C library to generate qemu command lines and configuration files

[Markus Armbruster]

Paolo Bonzini <pbonzini@redhat.com> writes:

> On 02/05/2017 16:59, Richard W.M. Jones wrote:
>>> To answer your question: you have to double comma after '=', or else it
>>> terminates the value.  There is no other quoting.
>>
>> Hmm, is that really right?

I should avoid absolutes when it comes to QemuOpts...

>>                             It seems to me that any comma must be
>> doubled.  For example:
>> 
>> $ qemu-system-x86_64 -name foo,bar -writeconfig - -hda die
>> qemu-system-x86_64: -name foo,bar: Invalid parameter 'bar'
>
> That's because, as you found out, "-name" is not a simple option but a
> compound one; "-name foo" is short for "-name guest=foo" (compare
> "-device e1000").
>
> The grammar mentioned in util/keyval.c says that you have to double
> comma in a value, *and* comma is completely invalid in a key.
>
> If you have '=', you have to double commas after it and cannot use it
> before.
>
> If you have no '=' at all, then you have to double comma too.

Yes.  My description wasn't precise enough, sorry.

Please take note that the grammar in util/keyval.c does *not* produce
the exact QemuOpts language.  It's close, but it doesn't reproduce all
corner cases.  Anything it does produce should have the same meaning in
QemuOpts.

> Paolo
>
>> $ qemu-system-x86_64 -name foo,,bar -writeconfig - -hda die
>> # qemu config file
>> 
>> [name]
>>   guest = "foo,bar"