[Qemu-devel] Libvirt debug API

[Qemu-devel] Libvirt debug API

[Chris Lalancette]

Hello,
     In response to a lot of the talk of qemud lately on qemu-devel, the
libvirt community would like to put forward a proposal to help enable
debug/advanced options when using various hypervisors.  The goals of
this API are:

1)  To enable more rapid access to hypervisor features before proper
libvirt API's are designed around them.
2)  To facilitate debugging and access to advanced features that may
not fit into the normal libvirt world-view.

Caveats:
1)  Unlike other libvirt API's, this one will explicitly *not* be
guaranteed ABI/API compatible between libvirt updates.
2)  Again unlike other libvirt API's, access and configuration of
the debug section of a domain will be highly hypervisor dependent.
3)  Application developers will be strongly discouraged from using
this API because of the above 2 issues.  To help in this, the
API's will be in a separate library that developers will explicitly
have to link to, and it will have a different (but largely compatible)
wire protocol.
4)  We don't expect this API to solve all of the issues brought up
during the qemud discussion.  Our initial goal is just to give
ready access of the qemu command-line and monitor to developers.

With that being said, our initial proposal follows.  We expect this
to evolve over time as we get more feedback, but we think this
proposal addresses at least 2 of the major pain points qemu developers
have while trying to use libvirt.

The initial debug XML for qemu would be:

<domain type='kvm'>
  <name>myguest</name>
  ...
  <debug>
    <monitorpassthrough/>
    <commandline>
      <extra>qemu arguments</extra>
      <alter option="optname">
        <rename>newname</rename>
        <match>REGEXP</match>
        <modify>foo=on</modify>
        <extra>-bar</extra>
      </alter>
    </commandline>
  </debug>
</domain>

Raw access to the qemu monitor will be disabled by default; the
<monitorpassthrough/> tag enables the ability to send QMP (or
text, if you are using older qemu) messages straight through to the
monitor.  To do this there will be an additional API entry point
named virDomainDebugCommand() which takes an arbitrary string
and passes it to the monitor, and returns an arbitrary string as
a result.  Thus you could pass in either "info cpus" if using the
text monitor or '{ "execute": "query-cpus" }' if using QMP.

The <commandline><extra> tag does exactly what you might expect; appends
the exact string to the qemu command-line.

The <alter> tag gets more interesting.  The idea is that <alter> would
allow you to modify the libvirt-generated qemu command-line in arbitrary
ways.  How this would work is probably best explained with some examples:

<commandline>
  <alter option="-net">
    <rename>-netdev</rename>
  </alter>
</commandline>

In this example, all options named -net on the qemu command-line are
renamed to -netdev.

<commandline>
  <alter option="-net">
    <extra>-usbtablet</extra>
  </alter>
</commandline>

In this example, if (and only if) a -net option is seen, then -usbtablet is
appended to the qemu command-line.

<commandline>
  <alter option="-net">
    <match>\(.*name=hostnet0.*\)</match>
    <modify>\1,tap</modify>
  </alter>
</commandline>

This gets more complicated (but also more powerful).  In this case, any -net
option where the argument *also* matches the regex in <match> will be modified to append
the ",tap" string.  Think of it as a sed expression, s/match/modify/, against the
argument to the -net option, and it makes more sense.  

We are hoping to refine this proposal based on feedback, so comments and criticisms
are welcome!

-- 
Chris Lalancette

[Qemu-devel] Re: [libvirt] Libvirt debug API

[Daniel P. Berrange]

On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
> Hello,
>      In response to a lot of the talk of qemud lately on qemu-devel, the
> libvirt community would like to put forward a proposal to help enable
> debug/advanced options when using various hypervisors.  The goals of
> this API are:
> 
> 1)  To enable more rapid access to hypervisor features before proper
> libvirt API's are designed around them.
> 2)  To facilitate debugging and access to advanced features that may
> not fit into the normal libvirt world-view.
> 
> Caveats:
> 1)  Unlike other libvirt API's, this one will explicitly *not* be
> guaranteed ABI/API compatible between libvirt updates.

I think we'd still aim to keep the public API stable. The bit that we
can't guarentee is the interactions with QEMU & the libvirt driver. 
eg, if someone was using the API to send text monitor commands to
QEMU, and the next libvirt release switched to JSON mode, that use
would break. The API would still be valid, but the way they use it
might not be. Similarly if they add some custom extra command line
argument, this could potentially conflict with an extra command line
arg a subsquent libvirt release used. eg, they used -device to add
a PCI card with a specific PCI address. Then next libvirt release
specifies this same PCI address and then you get a clash. 

So in both cases the API and XML format can be reasonably guarenteed
between releases. What we can't guarentee is that usage  of these
features will be reliable across releases of libvirt.

> 2)  Again unlike other libvirt API's, access and configuration of
> the debug section of a domain will be highly hypervisor dependent.
> 3)  Application developers will be strongly discouraged from using
> this API because of the above 2 issues.  To help in this, the
> API's will be in a separate library that developers will explicitly
> have to link to, and it will have a different (but largely compatible)
> wire protocol.

In terms of wire protocol, this would still have to run over the same
existing data channel, because we don't want apps to have to open up
multiple connections. Fortunately our protocol is designed to allow
this kind of extension

  struct remote_message_header {
      unsigned prog;              /* REMOTE_PROGRAM */
      unsigned vers;              /* REMOTE_PROTOCOL_VERSION */
      remote_procedure proc;      /* REMOTE_PROC_x */
      remote_message_type type;
      unsigned serial;            /* Serial number of message. */
      remote_message_status status;
  };

We currently have a single program 'REMOTE_PROGRAM' and its associated
'vers' and 'proc' numbers. What we would be doing is to define a new 
program, say QEMU_PROGRAM. The 'remote_procedure proc' numbers can thus
start again from '1' without clashing with any of the existing APIs. 
All the RPC marshalling/demarshalling code should work more or less
unchanged. We just hook in a different function dispatch table in the
daemon for that program.

> 4)  We don't expect this API to solve all of the issues brought up
> during the qemud discussion.  Our initial goal is just to give
> ready access of the qemu command-line and monitor to developers.
> 
> With that being said, our initial proposal follows.  We expect this
> to evolve over time as we get more feedback, but we think this
> proposal addresses at least 2 of the major pain points qemu developers
> have while trying to use libvirt.
> 
> The initial debug XML for qemu would be:

I wouldn't call them 'debug' options, since that's just one of possible
use cases for this. What we're really doing there is exposing hypervisor
specific features, so we should just be upfront about that in our 
description / naming.

> 
> <domain type='kvm'>
>   <name>myguest</name>
>   ...
>   <debug>
>     <monitorpassthrough/>
>     <commandline>
>       <extra>qemu arguments</extra>
>       <alter option="optname">
>         <rename>newname</rename>
>         <match>REGEXP</match>
>         <modify>foo=on</modify>
>         <extra>-bar</extra>
>       </alter>
>     </commandline>
>   </debug>
> </domain>

The concept of command line & monitor is something that is QEMU specific
and thus is not suitable for the primary XML schema. IMHO, this needs to be
done as a separate schema, linked in via an XML namespace. For example

  <domain type='kvm' xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
    <name>myguest</name>
    ...
    <qemu:commandline>
      <qemu:arg>-device</arg>
      <qemu:arg>lsi</arg>
    </qemu:commandline>
  </domain>


> 
> Raw access to the qemu monitor will be disabled by default; the
> <monitorpassthrough/> tag enables the ability to send QMP (or
> text, if you are using older qemu) messages straight through to the
> monitor.  To do this there will be an additional API entry point
> named virDomainDebugCommand() which takes an arbitrary string
> and passes it to the monitor, and returns an arbitrary string as
> a result.  Thus you could pass in either "info cpus" if using the
> text monitor or '{ "execute": "query-cpus" }' if using QMP.

Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
other hypervisors have different approaches for low level extension/
debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
etc. So this should really live in a separate API namespace which is
specific to a hypervisor. For example, as a header file

  #include <libvirt/libvirt-qemu.h>

Containing APIs like

  int virDomainQEMUInvokeMonitor(virDomainPtr dom,
                                 const char *command,
                                 char **reply);

  typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
                                            virDomainPtr dom, 
                                            const char *eventname,
                                            const char *data,
                                            void *opaque)
  int virConnectQEMUDomainEventRegister(virConnectPtr conn,
                                        virDomainPtr dom,
                                        const char *eventname,
                                        virDomainQEMUMonitorCallback cb,
                                        void *opaque);


For an add-on library

  libvirt-qemu.so

I don't think there's much to be gained from having an XML element to
turn on/off use of these APIs. If an app doesn't want to use them, it
can simply not link to libvirt-qemu.so

> The <commandline><extra> tag does exactly what you might expect; appends
> the exact string to the qemu command-line.

Allowing many args at once in the <extra> blob means that libvirt will
need to parse & split this up into individual args which than then be
safely passed to 'exec'. It is better to specify one arg per element
to avoid this fragile parsing problem.

> The <alter> tag gets more interesting.  The idea is that <alter> would
> allow you to modify the libvirt-generated qemu command-line in arbitrary
> ways.  How this would work is probably best explained with some examples:
> 
> <commandline>
>   <alter option="-net">
>     <rename>-netdev</rename>
>   </alter>
> </commandline>
> 
> In this example, all options named -net on the qemu command-line are
> renamed to -netdev.
>
> In this example, if (and only if) a -net option is seen, then -usbtablet is
> appended to the qemu command-line.
> 
> <commandline>
>   <alter option="-net">
>     <match>\(.*name=hostnet0.*\)</match>
>     <modify>\1,tap</modify>
>   </alter>
> </commandline>
> 
> This gets more complicated (but also more powerful).  In this case, any -net

I think this alteration of existing args is faaaar too complex & fragile,
and way overkill. If the arg that libvirt generates isn't what someone 
needs, then remove the bit of the guest config responsible for that and
add a complete extra arg, rather than munging the existing one.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: [libvirt] Libvirt debug API

[Chris Lalancette]

On 04/09/2010 10:27 AM, Daniel P. Berrange wrote:
> The concept of command line & monitor is something that is QEMU specific
> and thus is not suitable for the primary XML schema. IMHO, this needs to be
> done as a separate schema, linked in via an XML namespace. For example
> 
>   <domain type='kvm' xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
>     <name>myguest</name>
>     ...
>     <qemu:commandline>
>       <qemu:arg>-device</arg>
>       <qemu:arg>lsi</arg>
>     </qemu:commandline>
>   </domain>

This is a very good idea.  I didn't really know about namespaces, but now
that you've pointed them out I think this is a great way to partition it up.
I'll start looking at this.

> 
> 
>>
>> Raw access to the qemu monitor will be disabled by default; the
>> <monitorpassthrough/> tag enables the ability to send QMP (or
>> text, if you are using older qemu) messages straight through to the
>> monitor.  To do this there will be an additional API entry point
>> named virDomainDebugCommand() which takes an arbitrary string
>> and passes it to the monitor, and returns an arbitrary string as
>> a result.  Thus you could pass in either "info cpus" if using the
>> text monitor or '{ "execute": "query-cpus" }' if using QMP.
> 
> Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
> other hypervisors have different approaches for low level extension/
> debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
> etc. So this should really live in a separate API namespace which is
> specific to a hypervisor. For example, as a header file
> 
>   #include <libvirt/libvirt-qemu.h>
> 
> Containing APIs like
> 
>   int virDomainQEMUInvokeMonitor(virDomainPtr dom,
>                                  const char *command,
>                                  char **reply);
> 
>   typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
>                                             virDomainPtr dom, 
>                                             const char *eventname,
>                                             const char *data,
>                                             void *opaque)
>   int virConnectQEMUDomainEventRegister(virConnectPtr conn,
>                                         virDomainPtr dom,
>                                         const char *eventname,
>                                         virDomainQEMUMonitorCallback cb,
>                                         void *opaque);
> 
> 
> For an add-on library
> 
>   libvirt-qemu.so
> 
> I don't think there's much to be gained from having an XML element to
> turn on/off use of these APIs. If an app doesn't want to use them, it
> can simply not link to libvirt-qemu.so

The reason I wanted to do this was mostly for debug/support reasons.
That is, with this element in place we can easily tell from the dumpxml
output whether a person was using the "unreliable" API's, and thus we can
tell them to try and reproduce without that in place.

> 
>> The <commandline><extra> tag does exactly what you might expect; appends
>> the exact string to the qemu command-line.
> 
> Allowing many args at once in the <extra> blob means that libvirt will
> need to parse & split this up into individual args which than then be
> safely passed to 'exec'. It is better to specify one arg per element
> to avoid this fragile parsing problem.

That is true.  However, it's nicer to the user (and more like the qemu
command-line) to specify the options as one big string.  In any case we
already have the code for parsing a qemu command-line (for the native-to-xml
API), so we can just adapt that.

> 
>> The <alter> tag gets more interesting.  The idea is that <alter> would
>> allow you to modify the libvirt-generated qemu command-line in arbitrary
>> ways.  How this would work is probably best explained with some examples:
>>
>> <commandline>
>>   <alter option="-net">
>>     <rename>-netdev</rename>
>>   </alter>
>> </commandline>
>>
>> In this example, all options named -net on the qemu command-line are
>> renamed to -netdev.
>>
>> In this example, if (and only if) a -net option is seen, then -usbtablet is
>> appended to the qemu command-line.
>>
>> <commandline>
>>   <alter option="-net">
>>     <match>\(.*name=hostnet0.*\)</match>
>>     <modify>\1,tap</modify>
>>   </alter>
>> </commandline>
>>
>> This gets more complicated (but also more powerful).  In this case, any -net
> 
> I think this alteration of existing args is faaaar too complex & fragile,
> and way overkill. If the arg that libvirt generates isn't what someone 
> needs, then remove the bit of the guest config responsible for that and
> add a complete extra arg, rather than munging the existing one.

It is complex, I agree.  I also think it is pretty powerful.  But if
the need arises, we can always add it later.  For now we'll go with just the
monitor passthrough and simple command-line addition, and we can evolve from
there.

-- 
Chris Lalancette

[Qemu-devel] Re: [libvirt] Libvirt debug API

[Daniel P. Berrange]

On Fri, Apr 09, 2010 at 02:16:06PM -0400, Chris Lalancette wrote:
> On 04/09/2010 10:27 AM, Daniel P. Berrange wrote:
> >> Raw access to the qemu monitor will be disabled by default; the
> >> <monitorpassthrough/> tag enables the ability to send QMP (or
> >> text, if you are using older qemu) messages straight through to the
> >> monitor.  To do this there will be an additional API entry point
> >> named virDomainDebugCommand() which takes an arbitrary string
> >> and passes it to the monitor, and returns an arbitrary string as
> >> a result.  Thus you could pass in either "info cpus" if using the
> >> text monitor or '{ "execute": "query-cpus" }' if using QMP.
> > 
> > Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
> > other hypervisors have different approaches for low level extension/
> > debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
> > etc. So this should really live in a separate API namespace which is
> > specific to a hypervisor. For example, as a header file
> > 
> >   #include <libvirt/libvirt-qemu.h>
> > 
> > Containing APIs like
> > 
> >   int virDomainQEMUInvokeMonitor(virDomainPtr dom,
> >                                  const char *command,
> >                                  char **reply);
> > 
> >   typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
> >                                             virDomainPtr dom, 
> >                                             const char *eventname,
> >                                             const char *data,
> >                                             void *opaque)
> >   int virConnectQEMUDomainEventRegister(virConnectPtr conn,
> >                                         virDomainPtr dom,
> >                                         const char *eventname,
> >                                         virDomainQEMUMonitorCallback cb,
> >                                         void *opaque);
> > 
> > 
> > For an add-on library
> > 
> >   libvirt-qemu.so
> > 
> > I don't think there's much to be gained from having an XML element to
> > turn on/off use of these APIs. If an app doesn't want to use them, it
> > can simply not link to libvirt-qemu.so
> 
> The reason I wanted to do this was mostly for debug/support reasons.
> That is, with this element in place we can easily tell from the dumpxml
> output whether a person was using the "unreliable" API's, and thus we can
> tell them to try and reproduce without that in place.

That doesn't tell you whether they have actually used any API or not.
It is also inconvenient if you start a guest without it, and only later
realize you want to use the extra APIs. If we want to track the actual
usage, then the first time a direct monitor command is issued, we should
simply log a warning message. 

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Chris Lalancette]

On 04/12/2010 08:41 AM, Daniel P. Berrange wrote:
>>> I don't think there's much to be gained from having an XML element to
>>> turn on/off use of these APIs. If an app doesn't want to use them, it
>>> can simply not link to libvirt-qemu.so
>>
>> The reason I wanted to do this was mostly for debug/support reasons.
>> That is, with this element in place we can easily tell from the dumpxml
>> output whether a person was using the "unreliable" API's, and thus we can
>> tell them to try and reproduce without that in place.
> 
> That doesn't tell you whether they have actually used any API or not.
> It is also inconvenient if you start a guest without it, and only later
> realize you want to use the extra APIs. If we want to track the actual
> usage, then the first time a direct monitor command is issued, we should
> simply log a warning message. 

The problem with logging a message is that it is easy to lose it.  What I'm
trying to avoid here is debugging somebody's setup for hours only to find
out that they did a "pci_del" behind libvirt's back.  Maybe we can just make the
<monitorpassthrough/> a read-only flag; it is ignored in the parsing, but
it is set by "GetXMLDesc" when it detects that the virDomainQemuInvokeMonitor
has been called.

-- 
Chris Lalancette

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Daniel P. Berrange]

On Mon, Apr 12, 2010 at 09:56:50AM -0400, Chris Lalancette wrote:
> On 04/12/2010 08:41 AM, Daniel P. Berrange wrote:
> >>> I don't think there's much to be gained from having an XML element to
> >>> turn on/off use of these APIs. If an app doesn't want to use them, it
> >>> can simply not link to libvirt-qemu.so
> >>
> >> The reason I wanted to do this was mostly for debug/support reasons.
> >> That is, with this element in place we can easily tell from the dumpxml
> >> output whether a person was using the "unreliable" API's, and thus we can
> >> tell them to try and reproduce without that in place.
> > 
> > That doesn't tell you whether they have actually used any API or not.
> > It is also inconvenient if you start a guest without it, and only later
> > realize you want to use the extra APIs. If we want to track the actual
> > usage, then the first time a direct monitor command is issued, we should
> > simply log a warning message. 
> 
> The problem with logging a message is that it is easy to lose it.  What I'm
> trying to avoid here is debugging somebody's setup for hours only to find
> out that they did a "pci_del" behind libvirt's back.  Maybe we can just make the
> <monitorpassthrough/> a read-only flag; it is ignored in the parsing, but
> it is set by "GetXMLDesc" when it detects that the virDomainQemuInvokeMonitor
> has been called.

If the logging message is at level 'WARN' it will be included in syslog
for libvirtd by default, which is basic support  data always collected.
The guest XML is a description of the guest configuration, which is not 
a place for ad-hoc flags related to API usage.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Jamie Lokier]

Daniel P. Berrange wrote:
> I think this alteration of existing args is faaaar too complex & fragile,
> and way overkill.

Would it not be simpler, for the target audience, for the config to
contain a one-line shell script to transform particular matched
arguments in any way that's wanted?

> If the arg that libvirt generates isn't what someone 
> needs, then remove the bit of the guest config responsible for that and
> add a complete extra arg, rather than munging the existing one.

Problem with that: You might want libvirt to think it has a device
that it's controlling, still show in the GUI, still tie up with the
rest of the stack, still do the management setup, but you just want to
change some qemu-specific parameters.  For example,
cache=somethingelse for block devices, iptables=tobedefined for net
devices, usbimplementation=hack for USB devices.

-- Jamie

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 862 bytes --]

On 04/09/2010 03:06 PM, Jamie Lokier wrote:
> Daniel P. Berrange wrote:
>> I think this alteration of existing args is faaaar too complex & fragile,
>> and way overkill.
> 
> Would it not be simpler, for the target audience, for the config to
> contain a one-line shell script to transform particular matched
> arguments in any way that's wanted?

Yeah, having the ability to specify an optional wrapper script, that
receives the name of the normal interpreter and all the arguments the
normal interpreter would have been given, sounds like the ultimate in
flexibility at a minimum of xml.  It comes at the expense of pushing the
burden of flexibility onto the shoulders of whoever writes the wrapper
script, but that seems okay to me.

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 323 bytes --]

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Paolo Bonzini]

On 04/09/2010 11:30 PM, Eric Blake wrote:
> Yeah, having the ability to specify an optional wrapper script, that
> receives the name of the normal interpreter and all the arguments the
> normal interpreter would have been given, sounds like the ultimate in
> flexibility at a minimum of xml.

You can also just have it piped into sed.

Paolo

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Richard W.M. Jones]

On Fri, Apr 09, 2010 at 10:06:51PM +0100, Jamie Lokier wrote:
> Daniel P. Berrange wrote:
> > I think this alteration of existing args is faaaar too complex & fragile,
> > and way overkill.
> 
> Would it not be simpler, for the target audience, for the config to
> contain a one-line shell script to transform particular matched
> arguments in any way that's wanted?

I might be missing something but I thought you could already do this
in current libvirt.  ie. Change the <emulator> element [1] to point to
your shell script:

  <emulator>/usr/local/bin/my-qemu-wrapper.sh</emulator>

The wrapper should get called with the command line arguments and you
can alter them however you like.

Isn't that right, Dan?

Rich.

[1] http://libvirt.org/formatdomain.html#elementsDevices

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming blog: http://rwmj.wordpress.com
Fedora now supports 80 OCaml packages (the OPEN alternative to F#)
http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Jamie Lokier]

Richard W.M. Jones wrote:
> On Fri, Apr 09, 2010 at 10:06:51PM +0100, Jamie Lokier wrote:
> > Daniel P. Berrange wrote:
> > > I think this alteration of existing args is faaaar too complex & fragile,
> > > and way overkill.
> > 
> > Would it not be simpler, for the target audience, for the config to
> > contain a one-line shell script to transform particular matched
> > arguments in any way that's wanted?
> 
> I might be missing something but I thought you could already do this
> in current libvirt.  ie. Change the <emulator> element [1] to point to
> your shell script:
> 
>   <emulator>/usr/local/bin/my-qemu-wrapper.sh</emulator>
> 
> The wrapper should get called with the command line arguments and you
> can alter them however you like.
> 
> Isn't that right, Dan?

That forces the shell script to be relatively complicated and
recognise all qemu arguments (including version specific ones) to skip
the ones with parameters - and to know where to look for the real
target executable is.

It's not a very friendly way to add or change parts of specific
subsystems, for example modifying a blockdev parameter, but it can be
done that way if necessary.

It's not that hard to write this for trivial extra options:

   <emulator>/bin/sh -c 'qemu "$0" "$@" -extra-flag'</emulator>

(if that works).

But for things like modifications to particular devices, wouldn't it
be better to associate "add cache=special option" to a section about
one of the blockdevs which libvirt knows about, rather than the awful
prospect of parsing libvirt's output to guess which of three -device
arguments corresponds with the second SCSI drive mentioned in the
libvirt config - which might not necessarily be the second on the
command line, even.

Parsing libvirt output and having to guess which option corresponds to
what from the libvirt config sounds very fragile and also a rather
large amount of effort for something which should be easy.

And then there's modifying how network tap devices are set up and
other networking, outside qemu, but requested by libvirt.  Doing that
in <emulator> sounds like a bad idea, but script hooks are the only
sane way to make networking fit every environment and requirement.
Ideally without forcing the hook writer to reimplement the normal
network setup from scratch just to tweak it a bit.  I apologise if
libvirt already provides such hooks - I haven't looked at that part of
it.

-- Jamie

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Jamie Lokier]

Daniel Veillard wrote:
> On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:
> > It's not that hard to write this for trivial extra options:
> > 
> >    <emulator>/bin/sh -c 'qemu "$0" "$@" -extra-flag'</emulator>
> > 
> > (if that works).
> 
>   That won't work because we expect the emulator to be a path to
> the emulator program, so yes that has to be done in the wrapper script.

Ok.  Being able to pass arguments would be about infinitely more
useful, so that you don't need a separate wrapper script for every
individual guest configuration - you can pass options to the single
wrapper for each variation.

Even <emulator>/path/to/wrapper qemu-0.11</emulator> is a big improvement
over <emulator>/path/to/wrapper-guest-config-6-with-qemu-0.11</emulator> ;-)

Some simple but versatile hook ideas:

   - <emulator-append-option>  (no space splitting, one option, appended)
   - <emulator-append-options> (space splitting multiple options)
   - <emulator-prepend-option>
   - <emulator-prepend-options>
   - <emulator-setenv name="NAME">VALUE</emulator-setenv>
   - <emulator-wrap>/path/to/script</emulator-wrap>
        This last one is to call the script, but pass all the options
        including the path to whichever version of qemu/kvm would get
        selected by libvirt.  (I presume it has some selection mechanism,
        given the variety of guests which break with each version of qemu
        and/or qemu-kvm and/or kvm?)

(It is most unfortunate that XML forces those long prefixes everywhere
due to lack of contextual naming, unlike less verbose config formats :-/)

> > Parsing libvirt output and having to guess which option corresponds to
> > what from the libvirt config sounds very fragile and also a rather
> > large amount of effort for something which should be easy.
> 
>   it's not that easy because we change the way we build the command line
> for qemu as qemu evolves. for example before and after the support for
> -device there have been a lot of changes.

I agree, which is why it's much less fragile if libvirt provides the
individual parts to hooks/wrappers, if that's asked, rather than
forcing the wrapper to parse libvirts output and guess what libvirt
does from version to version.

For sure, before and after -device, any script which is involved with
those options will probably have to change.  But that's quite unusual.
Most usually I'd expect things like the order of -device options might
change depending on how libvirt pulled together it's backing
resources, for example.

-- Jamie

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Daniel P. Berrange]

On Mon, Apr 12, 2010 at 01:23:08PM +0100, Jamie Lokier wrote:
> Daniel Veillard wrote:
> > On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:

> > > Parsing libvirt output and having to guess which option corresponds to
> > > what from the libvirt config sounds very fragile and also a rather
> > > large amount of effort for something which should be easy.
> > 
> >   it's not that easy because we change the way we build the command line
> > for qemu as qemu evolves. for example before and after the support for
> > -device there have been a lot of changes.
> 
> I agree, which is why it's much less fragile if libvirt provides the
> individual parts to hooks/wrappers, if that's asked, rather than
> forcing the wrapper to parse libvirts output and guess what libvirt
> does from version to version.
> 
> For sure, before and after -device, any script which is involved with
> those options will probably have to change.  But that's quite unusual.

I could easily see this changing again in the not too distant future.
eg, we might switch to writing out a temporary config file with all 
options and just passing qemu -readonly /var/lib/libvirt/qemu/guest.cfg
to avoid the QEMU arg parsing / escaping problems. Or we might stop
using command line at all for devices, and use monitor device_add to
setup all devices. Allowing ability to append extra args to the QEMU
command line is reasonably safe, while support for hooking into existing
command line args is not.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Anthony Liguori]

On 04/12/2010 07:23 AM, Jamie Lokier wrote:
> Daniel Veillard wrote:
>    
>> On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:
>>      
>>> It's not that hard to write this for trivial extra options:
>>>
>>>     <emulator>/bin/sh -c 'qemu "$0" "$@" -extra-flag'</emulator>
>>>
>>> (if that works).
>>>        
>>    That won't work because we expect the emulator to be a path to
>> the emulator program, so yes that has to be done in the wrapper script.
>>      
> Ok.  Being able to pass arguments would be about infinitely more
> useful, so that you don't need a separate wrapper script for every
> individual guest configuration - you can pass options to the single
> wrapper for each variation.
>
> Even<emulator>/path/to/wrapper qemu-0.11</emulator>  is a big improvement
> over<emulator>/path/to/wrapper-guest-config-6-with-qemu-0.11</emulator>  ;-)
>
> Some simple but versatile hook ideas:
>
>     -<emulator-append-option>   (no space splitting, one option, appended)
>     -<emulator-append-options>  (space splitting multiple options)
>     -<emulator-prepend-option>
>     -<emulator-prepend-options>
>     -<emulator-setenv name="NAME">VALUE</emulator-setenv>
>    

I'd strongly suggest not focusing on manipulating command line 
arguments.  I think in the not too distant future, few people will pass 
command line arguments as opposed to just using configuration files.

We are very close to being able to cover 90% of current users via the 
config file.

Regards,

Anthony Liguori

>     -<emulator-wrap>/path/to/script</emulator-wrap>
>          This last one is to call the script, but pass all the options
>          including the path to whichever version of qemu/kvm would get
>          selected by libvirt.  (I presume it has some selection mechanism,
>          given the variety of guests which break with each version of qemu
>          and/or qemu-kvm and/or kvm?)
>
> (It is most unfortunate that XML forces those long prefixes everywhere
> due to lack of contextual naming, unlike less verbose config formats :-/)
>
>    
>>> Parsing libvirt output and having to guess which option corresponds to
>>> what from the libvirt config sounds very fragile and also a rather
>>> large amount of effort for something which should be easy.
>>>        
>>    it's not that easy because we change the way we build the command line
>> for qemu as qemu evolves. for example before and after the support for
>> -device there have been a lot of changes.
>>      
> I agree, which is why it's much less fragile if libvirt provides the
> individual parts to hooks/wrappers, if that's asked, rather than
> forcing the wrapper to parse libvirts output and guess what libvirt
> does from version to version.
>
> For sure, before and after -device, any script which is involved with
> those options will probably have to change.  But that's quite unusual.
> Most usually I'd expect things like the order of -device options might
> change depending on how libvirt pulled together it's backing
> resources, for example.
>
> -- Jamie
>
>
>    

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Jes Sorensen]

On 04/22/10 20:47, Anthony Liguori wrote:
> On 04/12/2010 07:23 AM, Jamie Lokier wrote:
>> Some simple but versatile hook ideas:
>>
>>     -<emulator-append-option>   (no space splitting, one option,
>> appended)
>>     -<emulator-append-options>  (space splitting multiple options)
>>     -<emulator-prepend-option>
>>     -<emulator-prepend-options>
>>     -<emulator-setenv name="NAME">VALUE</emulator-setenv>
> 
> I'd strongly suggest not focusing on manipulating command line
> arguments.  I think in the not too distant future, few people will pass
> command line arguments as opposed to just using configuration files.
> 
> We are very close to being able to cover 90% of current users via the
> config file.

There will always be a place for adding command line parameters for
debugging purposes. When you try to debug a problem, the first port of
call is to look at the command line arguments libvirt uses to launch
qemu and replicate those with addons. It's a nightmare today and you end
up losing networking and other things. Having an optional debug options
thing that one can set quickly from virt-manager would make it so much
easier to retry different settings fast while replicating the setup that
is normally run.

Having to manually edit xml files for making changes is just not a
viable solution.

Speaking of which, one problem with the current XML format is that it
relies on hard-coding the path to QEMU. Instead if should rely on the
system default and provide an optional tag for debugging purposes. That
would make it easier to migrate specifications from one OS to another,
without manually having to edit the QEMU tag.

Cheers,
Jes

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Daniel P. Berrange]

On Thu, Apr 22, 2010 at 01:47:55PM -0500, Anthony Liguori wrote:
> On 04/12/2010 07:23 AM, Jamie Lokier wrote:
> >Daniel Veillard wrote:
> >   
> >>On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:
> >>     
> >>>It's not that hard to write this for trivial extra options:
> >>>
> >>>    <emulator>/bin/sh -c 'qemu "$0" "$@" -extra-flag'</emulator>
> >>>
> >>>(if that works).
> >>>       
> >>   That won't work because we expect the emulator to be a path to
> >>the emulator program, so yes that has to be done in the wrapper script.
> >>     
> >Ok.  Being able to pass arguments would be about infinitely more
> >useful, so that you don't need a separate wrapper script for every
> >individual guest configuration - you can pass options to the single
> >wrapper for each variation.
> >
> >Even<emulator>/path/to/wrapper qemu-0.11</emulator>  is a big improvement
> >over<emulator>/path/to/wrapper-guest-config-6-with-qemu-0.11</emulator>  
> >;-)
> >
> >Some simple but versatile hook ideas:
> >
> >    -<emulator-append-option>   (no space splitting, one option, appended)
> >    -<emulator-append-options>  (space splitting multiple options)
> >    -<emulator-prepend-option>
> >    -<emulator-prepend-options>
> >    -<emulator-setenv name="NAME">VALUE</emulator-setenv>
> >   
> 
> I'd strongly suggest not focusing on manipulating command line 
> arguments.  I think in the not too distant future, few people will pass 
> command line arguments as opposed to just using configuration files.

Agreed, we're not having anything in the XML for modifying command
line arguments explicitly. In tiny edge case where someone actually
needs that, QEMU already provides the '-set OPTION'  command line
arg that you can use to override any qdev property.


Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Daniel P. Berrange]

On Sun, Apr 11, 2010 at 11:17:38PM +0100, Jamie Lokier wrote:
> Richard W.M. Jones wrote:
> > On Fri, Apr 09, 2010 at 10:06:51PM +0100, Jamie Lokier wrote:
> > > Daniel P. Berrange wrote:
> > > > I think this alteration of existing args is faaaar too complex & fragile,
> > > > and way overkill.
> > > 
> > > Would it not be simpler, for the target audience, for the config to
> > > contain a one-line shell script to transform particular matched
> > > arguments in any way that's wanted?
> > 
> > I might be missing something but I thought you could already do this
> > in current libvirt.  ie. Change the <emulator> element [1] to point to
> > your shell script:
> > 
> >   <emulator>/usr/local/bin/my-qemu-wrapper.sh</emulator>
> > 
> > The wrapper should get called with the command line arguments and you
> > can alter them however you like.
> > 
> > Isn't that right, Dan?
> 
> That forces the shell script to be relatively complicated and
> recognise all qemu arguments (including version specific ones) to skip
> the ones with parameters - and to know where to look for the real
> target executable is.
> 
> It's not a very friendly way to add or change parts of specific
> subsystems, for example modifying a blockdev parameter, but it can be
> done that way if necessary.
> 
> It's not that hard to write this for trivial extra options:
> 
>    <emulator>/bin/sh -c 'qemu "$0" "$@" -extra-flag'</emulator>
> 
> (if that works).
> 
> But for things like modifications to particular devices, wouldn't it
> be better to associate "add cache=special option" to a section about
> one of the blockdevs which libvirt knows about, rather than the awful
> prospect of parsing libvirt's output to guess which of three -device
> arguments corresponds with the second SCSI drive mentioned in the
> libvirt config - which might not necessarily be the second on the
> command line, even.

Every device has a unique  alias in the XML, which maps onto the 'id='
parameter for QEMU so you can match up them. We need to consider the
tradeoff between what's likely to be neccessary vs cost of supporting
the extensions. Allowing addition of arbitrary extra command lines is
a trivial extension with low support cost & covers the most pressing
feature timelag problems. Munging existing dev options is a less common 
scenario with significantly higher support burden which I don't really
consider worth explicitly supporting beyond the existing ability to add
a wrapper script.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Luiz Capitulino]

On Fri, 9 Apr 2010 15:27:17 +0100
"Daniel P. Berrange" <berrange@redhat.com> wrote:

> On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
> > Hello,
> >      In response to a lot of the talk of qemud lately on qemu-devel, the
> > libvirt community would like to put forward a proposal to help enable
> > debug/advanced options when using various hypervisors.  The goals of
> > this API are:
> > 
> > 1)  To enable more rapid access to hypervisor features before proper
> > libvirt API's are designed around them.
> > 2)  To facilitate debugging and access to advanced features that may
> > not fit into the normal libvirt world-view.
> > 
> > Caveats:
> > 1)  Unlike other libvirt API's, this one will explicitly *not* be
> > guaranteed ABI/API compatible between libvirt updates.
> 
> I think we'd still aim to keep the public API stable. The bit that we
> can't guarentee is the interactions with QEMU & the libvirt driver. 
> eg, if someone was using the API to send text monitor commands to
> QEMU, and the next libvirt release switched to JSON mode, that use
> would break. The API would still be valid, but the way they use it
> might not be. Similarly if they add some custom extra command line
> argument, this could potentially conflict with an extra command line
> arg a subsquent libvirt release used. eg, they used -device to add
> a PCI card with a specific PCI address. Then next libvirt release
> specifies this same PCI address and then you get a clash.

 Another problem is issuing Monitor commands that could confuse libvirt's
real API. Say, adding a device libvirt doesn't know about or stopping the VM
while libvirt thinks it's still running or anything like that.

 When I suggested the monitor passthrough idea, I thought it could be a
workaround for the feature time lag problem, but as stated by Avi, I'm
afraid that this can potentially cause more problems than it solves.

 It's very desirable for debugging (item 2 above) and maybe to try new
things, but people couldn't use it for anything serious, which is the
major issue in this subject (item 1 above).

 Now, Anthony's idea of having libqemu has the same issues, which seems
to mean that the only way to really fix the time lag is to fix the
time lag :)

> > Raw access to the qemu monitor will be disabled by default; the
> > <monitorpassthrough/> tag enables the ability to send QMP (or
> > text, if you are using older qemu) messages straight through to the
> > monitor.  To do this there will be an additional API entry point
> > named virDomainDebugCommand() which takes an arbitrary string
> > and passes it to the monitor, and returns an arbitrary string as
> > a result.  Thus you could pass in either "info cpus" if using the
> > text monitor or '{ "execute": "query-cpus" }' if using QMP.
> 
> Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
> other hypervisors have different approaches for low level extension/
> debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
> etc. So this should really live in a separate API namespace which is
> specific to a hypervisor. For example, as a header file
> 
>   #include <libvirt/libvirt-qemu.h>
> 
> Containing APIs like
> 
>   int virDomainQEMUInvokeMonitor(virDomainPtr dom,
>                                  const char *command,
>                                  char **reply);
> 
>   typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
>                                             virDomainPtr dom, 
>                                             const char *eventname,
>                                             const char *data,
>                                             void *opaque)
>   int virConnectQEMUDomainEventRegister(virConnectPtr conn,
>                                         virDomainPtr dom,
>                                         const char *eventname,
>                                         virDomainQEMUMonitorCallback cb,
>                                         void *opaque);

 What about all the wonders of json? This will force clients to
have their own json <-> text converter, maybe we could provide two
APIs, just like qemu driver works internally?

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/12/2010 10:20 AM, Luiz Capitulino wrote:
> On Fri, 9 Apr 2010 15:27:17 +0100
> "Daniel P. Berrange"<berrange@redhat.com>  wrote:
>
>    
>> On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
>>      
>>> Hello,
>>>       In response to a lot of the talk of qemud lately on qemu-devel, the
>>> libvirt community would like to put forward a proposal to help enable
>>> debug/advanced options when using various hypervisors.  The goals of
>>> this API are:
>>>
>>> 1)  To enable more rapid access to hypervisor features before proper
>>> libvirt API's are designed around them.
>>> 2)  To facilitate debugging and access to advanced features that may
>>> not fit into the normal libvirt world-view.
>>>
>>> Caveats:
>>> 1)  Unlike other libvirt API's, this one will explicitly *not* be
>>> guaranteed ABI/API compatible between libvirt updates.
>>>        
>> I think we'd still aim to keep the public API stable. The bit that we
>> can't guarentee is the interactions with QEMU&  the libvirt driver.
>> eg, if someone was using the API to send text monitor commands to
>> QEMU, and the next libvirt release switched to JSON mode, that use
>> would break. The API would still be valid, but the way they use it
>> might not be. Similarly if they add some custom extra command line
>> argument, this could potentially conflict with an extra command line
>> arg a subsquent libvirt release used. eg, they used -device to add
>> a PCI card with a specific PCI address. Then next libvirt release
>> specifies this same PCI address and then you get a clash.
>>      
>   Another problem is issuing Monitor commands that could confuse libvirt's
> real API. Say, adding a device libvirt doesn't know about or stopping the VM
> while libvirt thinks it's still running or anything like that.
>    

We need to make libvirt and qemu smarter.

We already face this problem today with multiple libvirt users.  This is 
why sophisticated management mechanisms (like LDAP) have mechanisms to 
do transactions or at least a series of atomic operations.

Regards,

Anthony Liguori

>   When I suggested the monitor passthrough idea, I thought it could be a
> workaround for the feature time lag problem, but as stated by Avi, I'm
> afraid that this can potentially cause more problems than it solves.
>
>   It's very desirable for debugging (item 2 above) and maybe to try new
> things, but people couldn't use it for anything serious, which is the
> major issue in this subject (item 1 above).
>
>   Now, Anthony's idea of having libqemu has the same issues, which seems
> to mean that the only way to really fix the time lag is to fix the
> time lag :)
>
>    
>>> Raw access to the qemu monitor will be disabled by default; the
>>> <monitorpassthrough/>  tag enables the ability to send QMP (or
>>> text, if you are using older qemu) messages straight through to the
>>> monitor.  To do this there will be an additional API entry point
>>> named virDomainDebugCommand() which takes an arbitrary string
>>> and passes it to the monitor, and returns an arbitrary string as
>>> a result.  Thus you could pass in either "info cpus" if using the
>>> text monitor or '{ "execute": "query-cpus" }' if using QMP.
>>>        
>> Again the idea of a 'virDomainDebugCommand' API is QEMU specific, with
>> other hypervisors have different approaches for low level extension/
>> debug. For example, Xen would involve XenStore access, or XenD XMLRPC,
>> etc. So this should really live in a separate API namespace which is
>> specific to a hypervisor. For example, as a header file
>>
>>    #include<libvirt/libvirt-qemu.h>
>>
>> Containing APIs like
>>
>>    int virDomainQEMUInvokeMonitor(virDomainPtr dom,
>>                                   const char *command,
>>                                   char **reply);
>>
>>    typedef virConnectQEMUDomainEventCallback(virConnectPtr conn,
>>                                              virDomainPtr dom,
>>                                              const char *eventname,
>>                                              const char *data,
>>                                              void *opaque)
>>    int virConnectQEMUDomainEventRegister(virConnectPtr conn,
>>                                          virDomainPtr dom,
>>                                          const char *eventname,
>>                                          virDomainQEMUMonitorCallback cb,
>>                                          void *opaque);
>>      
>   What about all the wonders of json? This will force clients to
> have their own json<->  text converter, maybe we could provide two
> APIs, just like qemu driver works internally?
>
>
>    

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>> real API. Say, adding a device libvirt doesn't know about or stopping 
>> the VM
>> while libvirt thinks it's still running or anything like that.
>   Another problem is issuing Monitor commands that could confuse 
> libvirt's
>
> We need to make libvirt and qemu smarter.
>
> We already face this problem today with multiple libvirt users.  This 
> is why sophisticated management mechanisms (like LDAP) have mechanisms 
> to do transactions or at least a series of atomic operations.

And people said qmp/json was overengineered...

But seriously, transactions won't help anything.  qemu maintains state, 
and when you have two updaters touching a shared variable not excepting 
each other to, things break, no matter how much locking there is.  The 
only way that separate monitors could work is if they touch completely 
separate state, which is difficult to ensure if you upgrade your libvirt.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/23/2010 07:48 AM, Avi Kivity wrote:
> On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>>> real API. Say, adding a device libvirt doesn't know about or 
>>> stopping the VM
>>> while libvirt thinks it's still running or anything like that.
>>   Another problem is issuing Monitor commands that could confuse 
>> libvirt's
>>
>> We need to make libvirt and qemu smarter.
>>
>> We already face this problem today with multiple libvirt users.  This 
>> is why sophisticated management mechanisms (like LDAP) have 
>> mechanisms to do transactions or at least a series of atomic operations.
>
> And people said qmp/json was overengineered...
>
> But seriously, transactions won't help anything.  qemu maintains 
> state, and when you have two updaters touching a shared variable not 
> excepting each other to, things break, no matter how much locking 
> there is.

Let's consider some concrete examples.  I'm using libvirt and QMP and in 
QMP, I want to hot unplug a device.

Today, I do this by listing the pci devices, and issuing a pci_del that 
takes a PCI address.  This is intrinsically racy though because in the 
worst case scenario, in between when I enumerate pci devices and do the 
pci_del in QMP, in libvirt, I've done a pci_del and then a pci_add 
within libvirt of a completely different device.

There are a few ways to solve this, the simplest being that we give 
devices unique ids that are never reused and instead of pci_del taking a 
pci bus address, it takes a device id.  That would address this race.

You can get very far by just being clever about unique ids and 
notifications.  There are some cases where a true RMW may be required 
but I can't really think of one off hand.  The way LDAP addresses this 
is that it has a batched operation and a simple set of boolean 
comparison operations.  This lets you execute a batched operation that 
will do a RMW.

>   The only way that separate monitors could work is if they touch 
> completely separate state, which is difficult to ensure if you upgrade 
> your libvirt.
>

I don't think this is as difficult of a problem as you think it is.  If 
you look at Active Directory and the whole set of management tools based 
on it, they certainly allow concurrent management applications.  You can 
certainly get into trouble still but with just some careful 
considerations, you can make two management applications work together 
90% of the time without much fuss on the applications part.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/23/2010 04:48 PM, Anthony Liguori wrote:
> On 04/23/2010 07:48 AM, Avi Kivity wrote:
>> On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>>>> real API. Say, adding a device libvirt doesn't know about or 
>>>> stopping the VM
>>>> while libvirt thinks it's still running or anything like that.
>>>   Another problem is issuing Monitor commands that could confuse 
>>> libvirt's
>>>
>>> We need to make libvirt and qemu smarter.
>>>
>>> We already face this problem today with multiple libvirt users.  
>>> This is why sophisticated management mechanisms (like LDAP) have 
>>> mechanisms to do transactions or at least a series of atomic 
>>> operations.
>>
>> And people said qmp/json was overengineered...
>>
>> But seriously, transactions won't help anything.  qemu maintains 
>> state, and when you have two updaters touching a shared variable not 
>> excepting each other to, things break, no matter how much locking 
>> there is.
>
> Let's consider some concrete examples.  I'm using libvirt and QMP and 
> in QMP, I want to hot unplug a device.
>
> Today, I do this by listing the pci devices, and issuing a pci_del 
> that takes a PCI address.  This is intrinsically racy though because 
> in the worst case scenario, in between when I enumerate pci devices 
> and do the pci_del in QMP, in libvirt, I've done a pci_del and then a 
> pci_add within libvirt of a completely different device.

Obviously you should do the pci_del through libvirt.  Once libvirt 
supports an API, use it.

>
> There are a few ways to solve this, the simplest being that we give 
> devices unique ids that are never reused and instead of pci_del taking 
> a pci bus address, it takes a device id.  That would address this race.
>
> You can get very far by just being clever about unique ids and 
> notifications.  There are some cases where a true RMW may be required 
> but I can't really think of one off hand.  The way LDAP addresses this 
> is that it has a batched operation and a simple set of boolean 
> comparison operations.  This lets you execute a batched operation that 
> will do a RMW.

I'm sure we can be very clever, but I'd rather direct this cleverness to 
qemu core issues, not to the QMP (which in turn requires that users be 
clever to use it correctly).  QMP is a low bandwidth protocol, so races 
will never show up in testing.  We're laying mines here for users to 
step on that we will never encounter ourselves.

>
>>   The only way that separate monitors could work is if they touch 
>> completely separate state, which is difficult to ensure if you 
>> upgrade your libvirt.
>>
>
> I don't think this is as difficult of a problem as you think it is.  
> If you look at Active Directory and the whole set of management tools 
> based on it, they certainly allow concurrent management applications.  
> You can certainly get into trouble still but with just some careful 
> considerations, you can make two management applications work together 
> 90% of the time without much fuss on the applications part.

Maybe.  We'll still have issues.  For example, sVirt: if a QMP command 
names a labeled resource, the non-libvirt user will have no way of 
knowing how to label it.

Much better to exact a commitment from libvirt to track all QMP (and 
command line) capabilities.  Instead of adding cleverness to QMP, add 
APIs to libvirt.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Daniel P. Berrange]

On Fri, Apr 23, 2010 at 05:24:34PM +0300, Avi Kivity wrote:
> On 04/23/2010 04:48 PM, Anthony Liguori wrote:
> >On 04/23/2010 07:48 AM, Avi Kivity wrote:
> >>On 04/22/2010 09:49 PM, Anthony Liguori wrote:
> >>>>real API. Say, adding a device libvirt doesn't know about or 
> >>>>stopping the VM
> >>>>while libvirt thinks it's still running or anything like that.
> >>>  Another problem is issuing Monitor commands that could confuse 
> >>>libvirt's
> >>>
> >>>We need to make libvirt and qemu smarter.
> >>>
> >>>We already face this problem today with multiple libvirt users.  
> >>>This is why sophisticated management mechanisms (like LDAP) have 
> >>>mechanisms to do transactions or at least a series of atomic 
> >>>operations.
> >>
> >>And people said qmp/json was overengineered...
> >>
> >>But seriously, transactions won't help anything.  qemu maintains 
> >>state, and when you have two updaters touching a shared variable not 
> >>excepting each other to, things break, no matter how much locking 
> >>there is.
> >
> >Let's consider some concrete examples.  I'm using libvirt and QMP and 
> >in QMP, I want to hot unplug a device.
> >
> >Today, I do this by listing the pci devices, and issuing a pci_del 
> >that takes a PCI address.  This is intrinsically racy though because 
> >in the worst case scenario, in between when I enumerate pci devices 
> >and do the pci_del in QMP, in libvirt, I've done a pci_del and then a 
> >pci_add within libvirt of a completely different device.
> 
> Obviously you should do the pci_del through libvirt.  Once libvirt 
> supports an API, use it.
> 
> >
> >There are a few ways to solve this, the simplest being that we give 
> >devices unique ids that are never reused and instead of pci_del taking 
> >a pci bus address, it takes a device id.  That would address this race.
> >
> >You can get very far by just being clever about unique ids and 
> >notifications.  There are some cases where a true RMW may be required 
> >but I can't really think of one off hand.  The way LDAP addresses this 
> >is that it has a batched operation and a simple set of boolean 
> >comparison operations.  This lets you execute a batched operation that 
> >will do a RMW.
> 
> I'm sure we can be very clever, but I'd rather direct this cleverness to 
> qemu core issues, not to the QMP (which in turn requires that users be 
> clever to use it correctly).  QMP is a low bandwidth protocol, so races 
> will never show up in testing.  We're laying mines here for users to 
> step on that we will never encounter ourselves.
> 
> >
> >>  The only way that separate monitors could work is if they touch 
> >>completely separate state, which is difficult to ensure if you 
> >>upgrade your libvirt.
> >>
> >
> >I don't think this is as difficult of a problem as you think it is.  
> >If you look at Active Directory and the whole set of management tools 
> >based on it, they certainly allow concurrent management applications.  
> >You can certainly get into trouble still but with just some careful 
> >considerations, you can make two management applications work together 
> >90% of the time without much fuss on the applications part.
> 
> Maybe.  We'll still have issues.  For example, sVirt: if a QMP command 
> names a labeled resource, the non-libvirt user will have no way of 
> knowing how to label it.
> 
> Much better to exact a commitment from libvirt to track all QMP (and 
> command line) capabilities.  Instead of adding cleverness to QMP, add 
> APIs to libvirt.

Agreed. Despite adding this monitor / XML passthrough capability, we still
do not want apps to be using this at all. If there is some capability
missing that apps need then the default mode of operation is to add the
neccessary bits of libvirt. The monitor/XML pasthrough is just a short
term quick workaround until the official support is done. As such I do
not really think we need to put huge amounts of effort in the wierd 
complex racey edge cases. The effort is better spent on getting the 
features in libvirt.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Jamie Lokier]

Daniel P. Berrange wrote:
> > Much better to exact a commitment from libvirt to track all QMP (and 
> > command line) capabilities.  Instead of adding cleverness to QMP, add 
> > APIs to libvirt.
> 
> Agreed. Despite adding this monitor / XML passthrough capability, we still
> do not want apps to be using this at all. If there is some capability
> missing that apps need then the default mode of operation is to add the
> neccessary bits of libvirt. The monitor/XML pasthrough is just a short
> term quick workaround until the official support is done. As such I do
> not really think we need to put huge amounts of effort in the wierd 
> complex racey edge cases. The effort is better spent on getting the 
> features in libvirt.

All the features?  The qemu API is quite large already (look at all
the command line options and monitor commands).  I'll be very
surprised if libvirt provides all of it that obscure apps may use.

I'm thinking of features which are relatively obscure but nonetheless
useful to a small number of deployments.  Probably not enough to
justify the effort building data models, specifying the XML and remote
protocol and so on in libvirt.

(Unless that becomes so easily mapped to qemu's API that it's almost an
automatic thing... Which sounds like QMP, doesn't it?)

Is libvirt ever likely to go to the effort of providing all the
easily-usable API, or hooks, for:

    - sending keys to a guest, driven by a timed host script?

    - rebooting the guest while switching between USB touchpad and
      mouse devices, because one of them is needed during an OS
      install and the other is needed after?

    - changing the amount of RAM available to the guest at the next
      reboot, for OS install needing more memory than run time, in a
      scripted fashion when building new VMs from install disk images?

    - switching the guest between qemu mode and kvm mode on the next
      guest reset, because qemu is faster for some things (VGA
      updates) and kvm is faster for other things, so the best choice
      depends on which app you need to run on that guest

    - pausing a VM, making a copy, and resuming it, so as to fork it
      into two VMs (literally fork)?

    - setting up the host network container and NAT IP forwarding, on
      demand as guests are stopped and started, so that it works in
      the above scenario despite clashing IP addresses?

    - running a copy of the same guest, or perhaps an entire OS
      install process (scripted), many times for different qemu and
      qemu-kvm versions, different BIOSes, and different
      almost-equivalent hardware emulations (i.e. different NIC types,
      SMP count, CPU features, disk controller type, AIO/cache type) -
      for testing guests and apps on them - with some paralellism?

None of those, except perhaps the first, as what I think of as typical
virtualisation workloads, and they all seem obscure things probably
outside libvirt's remit.  Probably not many users either :-)

Yet you can do them all today with qemu and scripting the monitor, and
it's getting easier with QMP.

Which is fine, qemu works, but it would be great to be able to see
those guests and interact in the basic ways through the libvirt-based
GUIs?

QMP pass-through or QMP multiple monitors seems to provide most of
that, although I can see libvirt getting a bit confused about which
devices and how much RAM the guest has installed at different times.

The bit about forking guests, I'm not sure how complicated it is to
tie in to libvirt's notion of which disk images are being used, and
hooking into it's network configuration to handle the clashing
addresses.

If those things are considered to be entirely outside libvirt's remit,
that's fine with me.  Fair enough: I will continue to live with ssh
and vinagre.

I'm just raising my hand as a potential user who might like to monitor
a bunch of active and inactive guests, remotely, see how much memory
they report using, etc. launch VNC viewer from the GUI, even choose
the target host based on load and migrate on demand, while also
needing a fair bit of non-standardness and qemu-level scripting too.

Imho, that probably comes under the heading of apps using pass-through
or multiple QMP monitors, which use features that probably won't and
probably shouldn't ever be handled by libvirt itself.

-- Jamie

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Chris Lalancette]

On 04/26/2010 08:54 AM, Jamie Lokier wrote:
> All the features?  The qemu API is quite large already (look at all
> the command line options and monitor commands).  I'll be very
> surprised if libvirt provides all of it that obscure apps may use.
> 
> I'm thinking of features which are relatively obscure but nonetheless
> useful to a small number of deployments.  Probably not enough to
> justify the effort building data models, specifying the XML and remote
> protocol and so on in libvirt.
> 
> (Unless that becomes so easily mapped to qemu's API that it's almost an
> automatic thing... Which sounds like QMP, doesn't it?)
> 
> Is libvirt ever likely to go to the effort of providing all the
> easily-usable API, or hooks, for:
> 
>     - sending keys to a guest, driven by a timed host script?
> 
>     - rebooting the guest while switching between USB touchpad and
>       mouse devices, because one of them is needed during an OS
>       install and the other is needed after?
> 
>     - changing the amount of RAM available to the guest at the next
>       reboot, for OS install needing more memory than run time, in a
>       scripted fashion when building new VMs from install disk images?
> 
>     - switching the guest between qemu mode and kvm mode on the next
>       guest reset, because qemu is faster for some things (VGA
>       updates) and kvm is faster for other things, so the best choice
>       depends on which app you need to run on that guest
> 
>     - pausing a VM, making a copy, and resuming it, so as to fork it
>       into two VMs (literally fork)?
> 
>     - setting up the host network container and NAT IP forwarding, on
>       demand as guests are stopped and started, so that it works in
>       the above scenario despite clashing IP addresses?
> 
>     - running a copy of the same guest, or perhaps an entire OS
>       install process (scripted), many times for different qemu and
>       qemu-kvm versions, different BIOSes, and different
>       almost-equivalent hardware emulations (i.e. different NIC types,
>       SMP count, CPU features, disk controller type, AIO/cache type) -
>       for testing guests and apps on them - with some paralellism?
> 
> None of those, except perhaps the first, as what I think of as typical
> virtualisation workloads, and they all seem obscure things probably
> outside libvirt's remit.  Probably not many users either :-)
> 
> Yet you can do them all today with qemu and scripting the monitor, and
> it's getting easier with QMP.

In point of fact, you can also do some of them today more-or-less with libvirt
and some scripting.  However, you are right, there are probably obscure features
that libvirt will *never* be able to entirely implement, either because they
don't fit into libvirt's world-view or because they are too obscure or difficult
to do.

However, Dan's point stands; for most features that most users want to use,
it is far better to spend effort creating a cross-hypervisor, long-term API than
to spend too much effort on individual hypervisor hacks.

<snip>

> I'm just raising my hand as a potential user who might like to monitor
> a bunch of active and inactive guests, remotely, see how much memory
> they report using, etc. launch VNC viewer from the GUI, even choose
> the target host based on load and migrate on demand, while also
> needing a fair bit of non-standardness and qemu-level scripting too.
> 
> Imho, that probably comes under the heading of apps using pass-through
> or multiple QMP monitors, which use features that probably won't and
> probably shouldn't ever be handled by libvirt itself.

Right, and you are probably one of the users this work targets.  But in
general, for those not very familiar with virtualization/qemu, we want
to steer them far clear of this API.  That goes doubly true for application
developers; we want them to be able to use a stable, long-term API and
not have to worry about the nitty-gritty details of the monitor.  It's that
latter group that we want to make sure doesn't use this API.

-- 
Chris Lalancette

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Avi Kivity]

On 04/26/2010 05:25 PM, Chris Lalancette wrote:
> Right, and you are probably one of the users this work targets.  But in
> general, for those not very familiar with virtualization/qemu, we want
> to steer them far clear of this API.  That goes doubly true for application
> developers; we want them to be able to use a stable, long-term API and
> not have to worry about the nitty-gritty details of the monitor.  It's that
> latter group that we want to make sure doesn't use this API.
>    

With qmp, we have a stable long term API, and the nitty-gritty details 
are easily hidden behind a stock json parser (unfortunately some rpc 
details remain).  The command line is baroque, but the libvirt xml isn't 
so pretty either.

The problem is a user that starts with libvirt and outgrows its 
featureset.  Do we want them to fall back to qmp?

-- 
error compiling committee.c: too many arguments to function

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Daniel P. Berrange]

On Mon, Apr 26, 2010 at 05:34:22PM +0300, Avi Kivity wrote:
> On 04/26/2010 05:25 PM, Chris Lalancette wrote:
> >Right, and you are probably one of the users this work targets.  But in
> >general, for those not very familiar with virtualization/qemu, we want
> >to steer them far clear of this API.  That goes doubly true for application
> >developers; we want them to be able to use a stable, long-term API and
> >not have to worry about the nitty-gritty details of the monitor.  It's that
> >latter group that we want to make sure doesn't use this API.
> >   
> 
> With qmp, we have a stable long term API, and the nitty-gritty details 
> are easily hidden behind a stock json parser (unfortunately some rpc 
> details remain).  The command line is baroque, but the libvirt xml isn't 
> so pretty either.
> 
> The problem is a user that starts with libvirt and outgrows its 
> featureset.  Do we want them to fall back to qmp?

If were QMP were to be the standard mgmt API, then you are merely inverting
the problem. The problem is now a user that starts with QMP and outgrows
its featureset.

We have two layers in the stack here & neither provides a superset of the
other. As Anthony has mentioned, this has parallels with GTK (libvirt) vs
libX11 (libQMP). Programming against GTK does not give you all the features
you'd have if you programmed against X, but most people do not need them
and this is offset by the benefits GTK gives in portability across OS. GTK
does allow them to get direct access to X if really needed, but most apps
will avoid doing that if at all possible, and when it is found to be
neccessary new APIs are added to GTK so that direct X11 access remains a 
short-term hack.

We shouldn't have to frame this as a libvirt vs QMP problem. libvirt uses
QMP for its work, and apps use libvirt API for the vast majority of their
needs. If direct access to QMP is needed we can now provide that for apps
as a short term hack, until suitable APIs are added to libvirt. 

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Anthony Liguori]

On 04/26/2010 09:54 AM, Daniel P. Berrange wrote:
> On Mon, Apr 26, 2010 at 05:34:22PM +0300, Avi Kivity wrote:
>    
>> On 04/26/2010 05:25 PM, Chris Lalancette wrote:
>>      
>>> Right, and you are probably one of the users this work targets.  But in
>>> general, for those not very familiar with virtualization/qemu, we want
>>> to steer them far clear of this API.  That goes doubly true for application
>>> developers; we want them to be able to use a stable, long-term API and
>>> not have to worry about the nitty-gritty details of the monitor.  It's that
>>> latter group that we want to make sure doesn't use this API.
>>>
>>>        
>> With qmp, we have a stable long term API, and the nitty-gritty details
>> are easily hidden behind a stock json parser (unfortunately some rpc
>> details remain).  The command line is baroque, but the libvirt xml isn't
>> so pretty either.
>>
>> The problem is a user that starts with libvirt and outgrows its
>> featureset.  Do we want them to fall back to qmp?
>>      
> If were QMP were to be the standard mgmt API, then you are merely inverting
> the problem. The problem is now a user that starts with QMP and outgrows
> its featureset.
>
> We have two layers in the stack here&  neither provides a superset of the
> other. As Anthony has mentioned, this has parallels with GTK (libvirt) vs
> libX11 (libQMP). Programming against GTK does not give you all the features
> you'd have if you programmed against X, but most people do not need them
> and this is offset by the benefits GTK gives in portability across OS. GTK
> does allow them to get direct access to X if really needed, but most apps
> will avoid doing that if at all possible, and when it is found to be
> neccessary new APIs are added to GTK so that direct X11 access remains a
> short-term hack.
>
> We shouldn't have to frame this as a libvirt vs QMP problem. libvirt uses
> QMP for its work, and apps use libvirt API for the vast majority of their
> needs. If direct access to QMP is needed we can now provide that for apps
> as a short term hack, until suitable APIs are added to libvirt.
>    

I agree with you (for the most part).

The one difference here is that libgtk actually implements a full widget 
kit on top of libX11.  That's why people use gtk and not X11 directly.  
Portability was really an after thought with gtk.

If you look at libgtk and libcairo, instead of trying to replicate the 
features of cairo within gtk, gtk encourages people to use cairo 
directly for anything beyond the simplest 2d operations.

Regards,

Anthony Liguori

> Daniel
>    

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Daniel P. Berrange]

On Mon, Apr 26, 2010 at 10:08:33AM -0500, Anthony Liguori wrote:
> On 04/26/2010 09:54 AM, Daniel P. Berrange wrote:
> >On Mon, Apr 26, 2010 at 05:34:22PM +0300, Avi Kivity wrote:
> >   
> >>On 04/26/2010 05:25 PM, Chris Lalancette wrote:
> >>     
> >>>Right, and you are probably one of the users this work targets.  But in
> >>>general, for those not very familiar with virtualization/qemu, we want
> >>>to steer them far clear of this API.  That goes doubly true for 
> >>>application
> >>>developers; we want them to be able to use a stable, long-term API and
> >>>not have to worry about the nitty-gritty details of the monitor.  It's 
> >>>that
> >>>latter group that we want to make sure doesn't use this API.
> >>>
> >>>       
> >>With qmp, we have a stable long term API, and the nitty-gritty details
> >>are easily hidden behind a stock json parser (unfortunately some rpc
> >>details remain).  The command line is baroque, but the libvirt xml isn't
> >>so pretty either.
> >>
> >>The problem is a user that starts with libvirt and outgrows its
> >>featureset.  Do we want them to fall back to qmp?
> >>     
> >If were QMP were to be the standard mgmt API, then you are merely inverting
> >the problem. The problem is now a user that starts with QMP and outgrows
> >its featureset.
> >
> >We have two layers in the stack here&  neither provides a superset of the
> >other. As Anthony has mentioned, this has parallels with GTK (libvirt) vs
> >libX11 (libQMP). Programming against GTK does not give you all the features
> >you'd have if you programmed against X, but most people do not need them
> >and this is offset by the benefits GTK gives in portability across OS. GTK
> >does allow them to get direct access to X if really needed, but most apps
> >will avoid doing that if at all possible, and when it is found to be
> >neccessary new APIs are added to GTK so that direct X11 access remains a
> >short-term hack.
> >
> >We shouldn't have to frame this as a libvirt vs QMP problem. libvirt uses
> >QMP for its work, and apps use libvirt API for the vast majority of their
> >needs. If direct access to QMP is needed we can now provide that for apps
> >as a short term hack, until suitable APIs are added to libvirt.
> >   
> 
> I agree with you (for the most part).
> 
> The one difference here is that libgtk actually implements a full widget 
> kit on top of libX11.  That's why people use gtk and not X11 directly.  
> Portability was really an after thought with gtk.
>
> If you look at libgtk and libcairo, instead of trying to replicate the 
> features of cairo within gtk, gtk encourages people to use cairo 
> directly for anything beyond the simplest 2d operations.

For the sake of my analogy, I'd still group cairo in with gtk, not x11. It 
is really just a re-implmentation of the GDK drawing layer from GTK. It 
still provides a portable higher level abstraction over underlying graphics
& rendering systems it supports. eg, again an app might do the majority of
its drawing with Cairo, but occasionally need to jump outside to use XRender
or OpenGL directly for pieces that aren't supported currently in the cairo
API.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Anthony Liguori]

On 04/26/2010 10:20 AM, Daniel P. Berrange wrote:
>
> For the sake of my analogy, I'd still group cairo in with gtk, not x11. It
> is really just a re-implmentation of the GDK drawing layer from GTK. It
> still provides a portable higher level abstraction over underlying graphics
> &  rendering systems it supports. eg, again an app might do the majority of
> its drawing with Cairo, but occasionally need to jump outside to use XRender
> or OpenGL directly for pieces that aren't supported currently in the cairo
> API.
>    

Yeah, that's a fair point.

Regards,

Anthony Liguori

> Regards,
> Daniel
>    

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/23/2010 09:24 AM, Avi Kivity wrote:
> On 04/23/2010 04:48 PM, Anthony Liguori wrote:
>> On 04/23/2010 07:48 AM, Avi Kivity wrote:
>>> On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>>>>> real API. Say, adding a device libvirt doesn't know about or 
>>>>> stopping the VM
>>>>> while libvirt thinks it's still running or anything like that.
>>>>   Another problem is issuing Monitor commands that could confuse 
>>>> libvirt's
>>>>
>>>> We need to make libvirt and qemu smarter.
>>>>
>>>> We already face this problem today with multiple libvirt users.  
>>>> This is why sophisticated management mechanisms (like LDAP) have 
>>>> mechanisms to do transactions or at least a series of atomic 
>>>> operations.
>>>
>>> And people said qmp/json was overengineered...
>>>
>>> But seriously, transactions won't help anything.  qemu maintains 
>>> state, and when you have two updaters touching a shared variable not 
>>> excepting each other to, things break, no matter how much locking 
>>> there is.
>>
>> Let's consider some concrete examples.  I'm using libvirt and QMP and 
>> in QMP, I want to hot unplug a device.
>>
>> Today, I do this by listing the pci devices, and issuing a pci_del 
>> that takes a PCI address.  This is intrinsically racy though because 
>> in the worst case scenario, in between when I enumerate pci devices 
>> and do the pci_del in QMP, in libvirt, I've done a pci_del and then a 
>> pci_add within libvirt of a completely different device.
>
> Obviously you should do the pci_del through libvirt.  Once libvirt 
> supports an API, use it.

It was just an example...

>>
>> There are a few ways to solve this, the simplest being that we give 
>> devices unique ids that are never reused and instead of pci_del 
>> taking a pci bus address, it takes a device id.  That would address 
>> this race.
>>
>> You can get very far by just being clever about unique ids and 
>> notifications.  There are some cases where a true RMW may be required 
>> but I can't really think of one off hand.  The way LDAP addresses 
>> this is that it has a batched operation and a simple set of boolean 
>> comparison operations.  This lets you execute a batched operation 
>> that will do a RMW.
>
> I'm sure we can be very clever, but I'd rather direct this cleverness 
> to qemu core issues, not to the QMP (which in turn requires that users 
> be clever to use it correctly).  QMP is a low bandwidth protocol, so 
> races will never show up in testing.  We're laying mines here for 
> users to step on that we will never encounter ourselves.
>
>>
>>>   The only way that separate monitors could work is if they touch 
>>> completely separate state, which is difficult to ensure if you 
>>> upgrade your libvirt.
>>>
>>
>> I don't think this is as difficult of a problem as you think it is.  
>> If you look at Active Directory and the whole set of management tools 
>> based on it, they certainly allow concurrent management 
>> applications.  You can certainly get into trouble still but with just 
>> some careful considerations, you can make two management applications 
>> work together 90% of the time without much fuss on the applications 
>> part.
>
> Maybe.  We'll still have issues.  For example, sVirt: if a QMP command 
> names a labeled resource, the non-libvirt user will have no way of 
> knowing how to label it.

This is orthogonal to QMP and has to do strictly with how libvirt 
prepares a resource for qemu.

> Much better to exact a commitment from libvirt to track all QMP (and 
> command line) capabilities.  Instead of adding cleverness to QMP, add 
> APIs to libvirt.
>

Let's step back for a minute because I think we're missing the forest 
through the trees.

We're trying to address a few distinct problems:

1) Allow libvirt users to access features of qemu that are not exposed 
through libvirt

2) Provide a means for non-libvirt users to interact with qemu

3) Provide a unified and interoperable view of the world for non-libvirt 
and libvirt users

For (1), we all agree that the best case scenario would be for libvirt 
to support every qemu feature.  I think we can also all agree though 
that this is not really practical and certainly not practical for 
developers since there is a development cost associated with libvirt 
support (to model an API appropriately).

The new API proposed addresses (1) by allowing a user to drill down to 
the QMP context.  It's a good solution IMHO and I think we all agree 
that there's an inherent risk to this that users will have to evaluate 
on a case-by-case basis.  It's a good stop-gap though.

(2) is largely addressed by QMP and a config file.  I'd like to see a 
nice C library, but I think a lot of other folks are happy with JSON 
support in higher level languages.

(3) is the place where there are still potential challenges.  I think at 
the very least, our goal should be to enable conversion from (2) and (1) 
to be as easy as possible.  That's why I have proposed implementing a C 
library for the JSON transport because we could plumb that through the 
new libvirt API.  This would allow a user to very quickly port an 
application from QMP to libvirt.  In order to do this, we need the 
libvirt API to expose a dedicated monitor because we'll need to be able 
to manipulate events and negotiate features.

Beyond simple porting, there's a secondary question of having 
non-libvirt apps co-exist with libvirt apps.  I think it's a good long 
term goal, but I don't think we should worry too much about it now.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/23/2010 09:29 PM, Anthony Liguori wrote:
>> Maybe.  We'll still have issues.  For example, sVirt: if a QMP 
>> command names a labeled resource, the non-libvirt user will have no 
>> way of knowing how to label it.
>
>
> This is orthogonal to QMP and has to do strictly with how libvirt 
> prepares a resource for qemu.


It's not orthogonal.  If you allow qmp access behind libvirt's back, 
it's a problem that you will have.

>
>> Much better to exact a commitment from libvirt to track all QMP (and 
>> command line) capabilities.  Instead of adding cleverness to QMP, add 
>> APIs to libvirt.
>>
>
> Let's step back for a minute because I think we're missing the forest 
> through the trees.
>
> We're trying to address a few distinct problems:
>
> 1) Allow libvirt users to access features of qemu that are not exposed 
> through libvirt

That's an artificial problem.  If libvirt exposes all features, you 
don't need to solve it.

>
> 2) Provide a means for non-libvirt users to interact with qemu

We have qmp.  It doesn't do multiple guest management.  I think it's 
reasonable to have a qemud which does (and also does sVirt and the 
zillion other things libvirt does) provided we remove them from libvirt 
(long term).  The only problem is that it's a lot of effort.

>
> 3) Provide a unified and interoperable view of the world for 
> non-libvirt and libvirt users

This problem can be solved by the non-libvirt users adopting libvirt, or 
the libvirt users dropping libvirt.  I don't understand why we need to 
add interoperability between users who choose an interoperability 
library and users who don't choose an interoperability library.

>
> For (1), we all agree that the best case scenario would be for libvirt 
> to support every qemu feature.  I think we can also all agree though 
> that this is not really practical and certainly not practical for 
> developers since there is a development cost associated with libvirt 
> support (to model an API appropriately).

All except me, perhaps.

We already have two layers of feature modeling: first, we mostly emulate 
real life, not invent new features.  PCI hotplug existed long before 
qemu had support for it.  Second, we do give some thought into how we 
expose it through QMP.  libvirt doesn't have to invent it again, it only 
has to expose it through its lovely xml and C APIs.

>
> The new API proposed addresses (1) by allowing a user to drill down to 
> the QMP context.  It's a good solution IMHO and I think we all agree 
> that there's an inherent risk to this that users will have to evaluate 
> on a case-by-case basis.  It's a good stop-gap though.

Agree.

>
> (2) is largely addressed by QMP and a config file.  I'd like to see a 
> nice C library, but I think a lot of other folks are happy with JSON 
> support in higher level languages.

I agree with them.  C is a pretty bad choice for managing qemu (or even, 
C is a pretty bad choice).

>
> (3) is the place where there are still potential challenges.  I think 
> at the very least, our goal should be to enable conversion from (2) 
> and (1) to be as easy as possible.  That's why I have proposed 
> implementing a C library for the JSON transport because we could plumb 
> that through the new libvirt API.  This would allow a user to very 
> quickly port an application from QMP to libvirt.  In order to do this, 
> we need the libvirt API to expose a dedicated monitor because we'll 
> need to be able to manipulate events and negotiate features.

Most likely any application that talks QMP will hide the protocol behind 
a function call interface anyway.

> Beyond simple porting, there's a secondary question of having 
> non-libvirt apps co-exist with libvirt apps.  I think it's a good long 
> term goal, but I don't think we should worry too much about it now.

libvirt needs to either support all but the most esoteric use cases, or 
to get out of the way completely.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/24/2010 04:46 AM, Avi Kivity wrote:
> On 04/23/2010 09:29 PM, Anthony Liguori wrote:
>>> Maybe.  We'll still have issues.  For example, sVirt: if a QMP 
>>> command names a labeled resource, the non-libvirt user will have no 
>>> way of knowing how to label it.
>>
>>
>> This is orthogonal to QMP and has to do strictly with how libvirt 
>> prepares a resource for qemu.
>
>
> It's not orthogonal.  If you allow qmp access behind libvirt's back, 
> it's a problem that you will have.

My point was, if libvirt is just exposing raw qemu features, then it 
should be possible for qemu to arbitrate concurrent access.  If libvirt 
implements features on top of qemu, then no other third party will be 
able to co-exist with those features without interacting with qemu.  
It's an impossible problem for qemu to solve (arbitrating access to 
state stored in a third party management app).

>> 1) Allow libvirt users to access features of qemu that are not 
>> exposed through libvirt
>
> That's an artificial problem.  If libvirt exposes all features, you 
> don't need to solve it.

It won't.  Otherwise, we wouldn't be having this discussion.

>>
>> 2) Provide a means for non-libvirt users to interact with qemu
>
> We have qmp.  It doesn't do multiple guest management.  I think it's 
> reasonable to have a qemud which does (and also does sVirt and the 
> zillion other things libvirt does) provided we remove them from 
> libvirt (long term).  The only problem is that it's a lot of effort.

It depends on what things you think are important.  A lot of libvirt's 
complexity is based on the fact that it uses a daemon and needs to deal 
with the security implications of that.  You don't need explicit 
labelling if you don't use a daemon.  This is really the qemu model (as 
opposed to the xend model).  In theory, it does support this with the 
session urls but they are currently second-class citizens in libvirt.  
The remote dispatch also adds a fair bit of complexity and at least for 
the use-cases I'm interested in, it's not an important feature.

>>
>> 3) Provide a unified and interoperable view of the world for 
>> non-libvirt and libvirt users
>
> This problem can be solved by the non-libvirt users adopting libvirt, 
> or the libvirt users dropping libvirt.  I don't understand why we need 
> to add interoperability between users who choose an interoperability 
> library and users who don't choose an interoperability library.

What I'd like to avoid is user confusion.  Should a user use libvirt or 
libqemu?  If they make a decision to use libqemu and then down the road 
want to use libvirt, how hard is it to switch?  Fragmentation hurts the 
ecosystem and discourages good applications from existing.  I think it's 
our responsibility to ensure there's a good management API that exists 
for qemu that we can actively recommend to our users.  libvirt is very 
good at typical virtualization uses of qemu but qemu is much more than 
just that and has lots of advanced features.

>>
>> For (1), we all agree that the best case scenario would be for 
>> libvirt to support every qemu feature.  I think we can also all agree 
>> though that this is not really practical and certainly not practical 
>> for developers since there is a development cost associated with 
>> libvirt support (to model an API appropriately).
>
> All except me, perhaps.
>
> We already have two layers of feature modeling: first, we mostly 
> emulate real life, not invent new features.  PCI hotplug existed long 
> before qemu had support for it.  Second, we do give some thought into 
> how we expose it through QMP.  libvirt doesn't have to invent it 
> again, it only has to expose it through its lovely xml and C APIs.

That's not what the libvirt community wants to do.  We're very bias.  
We've made decisions about how features should be exposed and what 
features should be included.  We want all of those features exposed 
exactly how we've implemented them because we think it's the right way.

I'm not sure there's an obvious way forward unless we decide that there 
is going to be two ways to interact with qemu.  One way is through the 
libvirt world-view and the other is through a more qemu centric view.  
The problem then becomes allowing those two models to co-exist happily 
together.

The alternative is to get libvirt to just act as a thin layer to expose 
qemu features directly.  But honestly, what's the point of libvirt if 
they did that?  For the most part, the pool management is not 
virtualization specific and you could have libvirt provide that 
functionality without it knowing a thing about qemu.

>>
>> The new API proposed addresses (1) by allowing a user to drill down 
>> to the QMP context.  It's a good solution IMHO and I think we all 
>> agree that there's an inherent risk to this that users will have to 
>> evaluate on a case-by-case basis.  It's a good stop-gap though.
>
> Agree.
>
>>
>> (2) is largely addressed by QMP and a config file.  I'd like to see a 
>> nice C library, but I think a lot of other folks are happy with JSON 
>> support in higher level languages.
>
> I agree with them.  C is a pretty bad choice for managing qemu (or 
> even, C is a pretty bad choice).
>
>>
>> (3) is the place where there are still potential challenges.  I think 
>> at the very least, our goal should be to enable conversion from (2) 
>> and (1) to be as easy as possible.  That's why I have proposed 
>> implementing a C library for the JSON transport because we could 
>> plumb that through the new libvirt API.  This would allow a user to 
>> very quickly port an application from QMP to libvirt.  In order to do 
>> this, we need the libvirt API to expose a dedicated monitor because 
>> we'll need to be able to manipulate events and negotiate features.
>
> Most likely any application that talks QMP will hide the protocol 
> behind a function call interface anyway.
>
>> Beyond simple porting, there's a secondary question of having 
>> non-libvirt apps co-exist with libvirt apps.  I think it's a good 
>> long term goal, but I don't think we should worry too much about it now.
>
> libvirt needs to either support all but the most esoteric use cases, 
> or to get out of the way completely.

That's not where it is today or the path I think they're taking.  I'm 
also not the person you have to convince of this.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/25/2010 06:39 AM, Anthony Liguori wrote:
> On 04/24/2010 04:46 AM, Avi Kivity wrote:
>> On 04/23/2010 09:29 PM, Anthony Liguori wrote:
>>>> Maybe.  We'll still have issues.  For example, sVirt: if a QMP 
>>>> command names a labeled resource, the non-libvirt user will have no 
>>>> way of knowing how to label it.
>>>
>>>
>>> This is orthogonal to QMP and has to do strictly with how libvirt 
>>> prepares a resource for qemu.
>>
>>
>> It's not orthogonal.  If you allow qmp access behind libvirt's back, 
>> it's a problem that you will have.
>
> My point was, if libvirt is just exposing raw qemu features, then it 
> should be possible for qemu to arbitrate concurrent access.  If 
> libvirt implements features on top of qemu, then no other third party 
> will be able to co-exist with those features without interacting with 
> qemu.  It's an impossible problem for qemu to solve (arbitrating 
> access to state stored in a third party management app).

If libvirt implement features (like sVirt or network configuration) then 
it is indeed impossible for qemu to arbitrate.  If we take all those 
features into qemu[d], then it becomes possible to arbitrate so long as 
the libvirt and the other management app don't step on each others 
toes.  But that's impossible to guarantee if you upgrade your libvirt 
while keeping the other app unchanged.

>
>>> 1) Allow libvirt users to access features of qemu that are not 
>>> exposed through libvirt
>>
>> That's an artificial problem.  If libvirt exposes all features, you 
>> don't need to solve it.
>
> It won't.  Otherwise, we wouldn't be having this discussion.

Then libvirt will fade into uselessness.  A successful app using libvirt 
will grow, and will have new requirements.  As soon as libvirt doesn't 
meet those new requirements, the app will need to talk to qemu[d] 
directly.  Once it does that, it may as well use qemu[d] for everything; 
if you can talk QMP and generate qemu command lines, there's not much 
that libvirt buys you.  Even the cross-hypervisor support is not that 
hard to implement, especially if you only need to satisfy your own 
requirements.


>
>>>
>>> 2) Provide a means for non-libvirt users to interact with qemu
>>
>> We have qmp.  It doesn't do multiple guest management.  I think it's 
>> reasonable to have a qemud which does (and also does sVirt and the 
>> zillion other things libvirt does) provided we remove them from 
>> libvirt (long term).  The only problem is that it's a lot of effort.
>
> It depends on what things you think are important.  A lot of libvirt's 
> complexity is based on the fact that it uses a daemon and needs to 
> deal with the security implications of that.  You don't need explicit 
> labelling if you don't use a daemon. 

I don't follow.  If you have multiple guests that you want off each 
other's turf you have to label their resources, either statically or 
dynamically.  How is it related to a daemon being present?

> This is really the qemu model (as opposed to the xend model). 

(and the qemud model).

> In theory, it does support this with the session urls but they are 
> currently second-class citizens in libvirt.  The remote dispatch also 
> adds a fair bit of complexity and at least for the use-cases I'm 
> interested in, it's not an important feature.

If libvirt needs a local wrapper for interesting use cases, then it has 
failed.  You can't have a local wrapper with the esx driver, for example.

This is off-topic, but can you detail why you don't want remote dispatch 
(I assume we're talking about a multiple node deployment).

>
>>>
>>> 3) Provide a unified and interoperable view of the world for 
>>> non-libvirt and libvirt users
>>
>> This problem can be solved by the non-libvirt users adopting libvirt, 
>> or the libvirt users dropping libvirt.  I don't understand why we 
>> need to add interoperability between users who choose an 
>> interoperability library and users who don't choose an 
>> interoperability library.
>
> What I'd like to avoid is user confusion.  Should a user use libvirt 
> or libqemu?  If they make a decision to use libqemu and then down the 
> road want to use libvirt, how hard is it to switch?  Fragmentation 
> hurts the ecosystem and discourages good applications from existing.  
> I think it's our responsibility to ensure there's a good management 
> API that exists for qemu that we can actively recommend to our users.  
> libvirt is very good at typical virtualization uses of qemu but qemu 
> is much more than just that and has lots of advanced features.

Every typical virtualization use will eventually grow some non-typical 
requirements.  If libvirt explicitly refuses to support qemu features, I 
don't see how we can recommend it - even if it satisfies a user's 
requirements today, what about tomorrow? what about future qemu feature, 
will they be exposed or not?

If that is the case then we should develop qemud (which libvirt and 
other apps can use).

(even if it isn't the case I think qemud is a good idea)

>
>>>
>>> For (1), we all agree that the best case scenario would be for 
>>> libvirt to support every qemu feature.  I think we can also all 
>>> agree though that this is not really practical and certainly not 
>>> practical for developers since there is a development cost 
>>> associated with libvirt support (to model an API appropriately).
>>
>> All except me, perhaps.
>>
>> We already have two layers of feature modeling: first, we mostly 
>> emulate real life, not invent new features.  PCI hotplug existed long 
>> before qemu had support for it.  Second, we do give some thought into 
>> how we expose it through QMP.  libvirt doesn't have to invent it 
>> again, it only has to expose it through its lovely xml and C APIs.
>
> That's not what the libvirt community wants to do.  We're very bias.  
> We've made decisions about how features should be exposed and what 
> features should be included.  We want all of those features exposed 
> exactly how we've implemented them because we think it's the right way.
>
> I'm not sure there's an obvious way forward unless we decide that 
> there is going to be two ways to interact with qemu.  One way is 
> through the libvirt world-view and the other is through a more qemu 
> centric view.  The problem then becomes allowing those two models to 
> co-exist happily together.

I don't think there's a point in managing qemu through libvirt and 
directly in parallel.  It means a user has to learn both APIs, and for 
every operation they need to check both to see what's the best way of 
exploiting the feature.  There will invariably be some friction.

Layers need to stack on top of each other, not live side by side or 
bypass each other.

>
> The alternative is to get libvirt to just act as a thin layer to 
> expose qemu features directly.  But honestly, what's the point of 
> libvirt if they did that? 

For most hypervisors, that's exactly what libvirt does.  For Xen, it 
also bypasses Xend and the hypervisor's API, but it shouldn't really.  
Qemu is special due to the nonexistence of qemud.

Why is sVirt implemented in libvirt?  it's not the logical place for it; 
rather the logical place doesn't exist.

> For the most part, the pool management is not virtualization specific 
> and you could have libvirt provide that functionality without it 
> knowing a thing about qemu.

Yes.

>>
>>> Beyond simple porting, there's a secondary question of having 
>>> non-libvirt apps co-exist with libvirt apps.  I think it's a good 
>>> long term goal, but I don't think we should worry too much about it 
>>> now.
>>
>> libvirt needs to either support all but the most esoteric use cases, 
>> or to get out of the way completely.
>
> That's not where it is today or the path I think they're taking.  I'm 
> also not the person you have to convince of this.
>

It's fun trying all the same.  But you are the person to convince that 
side-by-side management is doomed to fail, yes?

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/25/2010 06:51 AM, Avi Kivity wrote:
>> It depends on what things you think are important.  A lot of 
>> libvirt's complexity is based on the fact that it uses a daemon and 
>> needs to deal with the security implications of that.  You don't need 
>> explicit labelling if you don't use a daemon. 
>
>
> I don't follow.  If you have multiple guests that you want off each 
> other's turf you have to label their resources, either statically or 
> dynamically.  How is it related to a daemon being present?

Because libvirt has to perform this labelling because it loses the 
original user's security context.

If you invoke qemu with the original user's credentials that launched 
the guest, then you don't need to do anything special with respect to 
security.

IOW, libvirt does not run guests as separate users which is why it needs 
to deal with security in the first place.

>> This is really the qemu model (as opposed to the xend model). 
>
> (and the qemud model).

And I've said in the past that I don't like the idea of a qemud :-)

>> In theory, it does support this with the session urls but they are 
>> currently second-class citizens in libvirt.  The remote dispatch also 
>> adds a fair bit of complexity and at least for the use-cases I'm 
>> interested in, it's not an important feature.
>
> If libvirt needs a local wrapper for interesting use cases, then it 
> has failed.  You can't have a local wrapper with the esx driver, for 
> example.
>
> This is off-topic, but can you detail why you don't want remote 
> dispatch (I assume we're talking about a multiple node deployment).

Because there are dozens of remote management APIs and then all have a 
concept of agents that run on the end nodes.  When fitting 
virtualization management into an existing management infrastructure, 
you are going to always use a local API.

If you use a remote management API, you then have to worry about 
security, authorization, and authentication.  Just supporting SASL 
really isn't enough because there are a lot fewer actual deployments of 
kerberos out there (most of them are actually Active Directory and SASL 
alone is not enough to interoperate with AD).

>>>>
>>>> 3) Provide a unified and interoperable view of the world for 
>>>> non-libvirt and libvirt users
>>>
>>> This problem can be solved by the non-libvirt users adopting 
>>> libvirt, or the libvirt users dropping libvirt.  I don't understand 
>>> why we need to add interoperability between users who choose an 
>>> interoperability library and users who don't choose an 
>>> interoperability library.
>>
>> What I'd like to avoid is user confusion.  Should a user use libvirt 
>> or libqemu?  If they make a decision to use libqemu and then down the 
>> road want to use libvirt, how hard is it to switch?  Fragmentation 
>> hurts the ecosystem and discourages good applications from existing.  
>> I think it's our responsibility to ensure there's a good management 
>> API that exists for qemu that we can actively recommend to our 
>> users.  libvirt is very good at typical virtualization uses of qemu 
>> but qemu is much more than just that and has lots of advanced features.
>
> Every typical virtualization use will eventually grow some non-typical 
> requirements.  If libvirt explicitly refuses to support qemu features, 
> I don't see how we can recommend it - even if it satisfies a user's 
> requirements today, what about tomorrow? what about future qemu 
> feature, will they be exposed or not?
>
> If that is the case then we should develop qemud (which libvirt and 
> other apps can use).
>
> (even if it isn't the case I think qemud is a good idea)

Yeah, that's where I'm at.  I'd eventually like libvirt to use our 
provided API and I can see where it would add value to the stack (by 
doing things like storage and network management).

>> That's not what the libvirt community wants to do.  We're very bias.  
>> We've made decisions about how features should be exposed and what 
>> features should be included.  We want all of those features exposed 
>> exactly how we've implemented them because we think it's the right way.
>>
>> I'm not sure there's an obvious way forward unless we decide that 
>> there is going to be two ways to interact with qemu.  One way is 
>> through the libvirt world-view and the other is through a more qemu 
>> centric view.  The problem then becomes allowing those two models to 
>> co-exist happily together.
>
> I don't think there's a point in managing qemu through libvirt and 
> directly in parallel.  It means a user has to learn both APIs, and for 
> every operation they need to check both to see what's the best way of 
> exploiting the feature.  There will invariably be some friction.
>
> Layers need to stack on top of each other, not live side by side or 
> bypass each other.

I agree with you theoretically but practically, I think it's immensely 
useful as a stop-gap.

>>
>> The alternative is to get libvirt to just act as a thin layer to 
>> expose qemu features directly.  But honestly, what's the point of 
>> libvirt if they did that? 
>
> For most hypervisors, that's exactly what libvirt does.  For Xen, it 
> also bypasses Xend and the hypervisor's API, but it shouldn't really.

Historically, xend was so incredibly slow (especially for frequent 
statistics collection) that it was a necessity.

>   Qemu is special due to the nonexistence of qemud.
>
> Why is sVirt implemented in libvirt?  it's not the logical place for 
> it; rather the logical place doesn't exist.

sVirt is not just implemented in libvirt.  libvirt implements a 
mechanism to set the context of a given domain and dynamically label 
it's resources to isolate it.

The reason it has to assign a context to a given domain is that all 
domains are launched from the same security context (the libvirtd 
context) as the original user's context (the consumer of the libvirt 
API) has been lost via the domain socket interface.

If you used the /session URL, then the domain would have the security 
context of whomever created the guest which means that dynamic labelling 
of the resources wouldn't be necessary (you would just do static labelling).

This is certainly a more secure model and it's a feature of qemu that I 
really wish didn't get lost in libvirt.  Again, /session can do this too 
but right now, /session really isn't usable in libvirt for qemu.

>
>>> libvirt needs to either support all but the most esoteric use cases, 
>>> or to get out of the way completely.
>>
>> That's not where it is today or the path I think they're taking.  I'm 
>> also not the person you have to convince of this.
>>
>
> It's fun trying all the same.  But you are the person to convince that 
> side-by-side management is doomed to fail, yes?

The world is full of large collections of code that are all doomed to 
fail :-)

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/26/2010 04:53 AM, Anthony Liguori wrote:
> On 04/25/2010 06:51 AM, Avi Kivity wrote:
>>> It depends on what things you think are important.  A lot of 
>>> libvirt's complexity is based on the fact that it uses a daemon and 
>>> needs to deal with the security implications of that.  You don't 
>>> need explicit labelling if you don't use a daemon. 
>>
>>
>> I don't follow.  If you have multiple guests that you want off each 
>> other's turf you have to label their resources, either statically or 
>> dynamically.  How is it related to a daemon being present?
>
> Because libvirt has to perform this labelling because it loses the 
> original user's security context.
>
> If you invoke qemu with the original user's credentials that launched 
> the guest, then you don't need to do anything special with respect to 
> security.
>
> IOW, libvirt does not run guests as separate users which is why it 
> needs to deal with security in the first place.

What if one user has multiple guests?  isolation is still needed.

One user per guest does not satisfy some security requirements.  The 'M' 
in selinux stands for mandatory, which means that the entities secured 
can't leak information even if they want to (scenario: G1 breaks into 
qemu, chmods files, G2 breaks into qemu, reads files).

>
>>> This is really the qemu model (as opposed to the xend model). 
>>
>> (and the qemud model).
>
> And I've said in the past that I don't like the idea of a qemud :-)

I must have missed it.  Why not?  Every other hypervisor has a central 
management entity.

>
>>> In theory, it does support this with the session urls but they are 
>>> currently second-class citizens in libvirt.  The remote dispatch 
>>> also adds a fair bit of complexity and at least for the use-cases 
>>> I'm interested in, it's not an important feature.
>>
>> If libvirt needs a local wrapper for interesting use cases, then it 
>> has failed.  You can't have a local wrapper with the esx driver, for 
>> example.
>>
>> This is off-topic, but can you detail why you don't want remote 
>> dispatch (I assume we're talking about a multiple node deployment).
>
> Because there are dozens of remote management APIs and then all have a 
> concept of agents that run on the end nodes.  When fitting 
> virtualization management into an existing management infrastructure, 
> you are going to always use a local API.

When you manage esx, do you deploy an agent?  I thought it was all done 
via their remote APIs.

>>
>> Every typical virtualization use will eventually grow some 
>> non-typical requirements.  If libvirt explicitly refuses to support 
>> qemu features, I don't see how we can recommend it - even if it 
>> satisfies a user's requirements today, what about tomorrow? what 
>> about future qemu feature, will they be exposed or not?
>>
>> If that is the case then we should develop qemud (which libvirt and 
>> other apps can use).
>>
>> (even if it isn't the case I think qemud is a good idea)
>
> Yeah, that's where I'm at.  I'd eventually like libvirt to use our 
> provided API and I can see where it would add value to the stack (by 
> doing things like storage and network management).

We do provide an API, qmp, and libvirt uses it?

>
>>> That's not what the libvirt community wants to do.  We're very 
>>> bias.  We've made decisions about how features should be exposed and 
>>> what features should be included.  We want all of those features 
>>> exposed exactly how we've implemented them because we think it's the 
>>> right way.
>>>
>>> I'm not sure there's an obvious way forward unless we decide that 
>>> there is going to be two ways to interact with qemu.  One way is 
>>> through the libvirt world-view and the other is through a more qemu 
>>> centric view.  The problem then becomes allowing those two models to 
>>> co-exist happily together.
>>
>> I don't think there's a point in managing qemu through libvirt and 
>> directly in parallel.  It means a user has to learn both APIs, and 
>> for every operation they need to check both to see what's the best 
>> way of exploiting the feature.  There will invariably be some friction.
>>
>> Layers need to stack on top of each other, not live side by side or 
>> bypass each other.
>
> I agree with you theoretically but practically, I think it's immensely 
> useful as a stop-gap.

Sure.  But please lets not start being clever with transactions and 
atomic operations and stuff, it has to come with a label that says, if 
you're using this, then something is wrong.


>>>
>>> The alternative is to get libvirt to just act as a thin layer to 
>>> expose qemu features directly.  But honestly, what's the point of 
>>> libvirt if they did that? 
>>
>> For most hypervisors, that's exactly what libvirt does.  For Xen, it 
>> also bypasses Xend and the hypervisor's API, but it shouldn't really.
>
> Historically, xend was so incredibly slow (especially for frequent 
> statistics collection) that it was a necessity.

Ah, reimplement rather than fix.

>>   Qemu is special due to the nonexistence of qemud.
>>
>> Why is sVirt implemented in libvirt?  it's not the logical place for 
>> it; rather the logical place doesn't exist.
>
> sVirt is not just implemented in libvirt.  libvirt implements a 
> mechanism to set the context of a given domain and dynamically label 
> it's resources to isolate it.
>
> The reason it has to assign a context to a given domain is that all 
> domains are launched from the same security context (the libvirtd 
> context) as the original user's context (the consumer of the libvirt 
> API) has been lost via the domain socket interface.
>
> If you used the /session URL, then the domain would have the security 
> context of whomever created the guest which means that dynamic 
> labelling of the resources wouldn't be necessary (you would just do 
> static labelling).
>
> This is certainly a more secure model and it's a feature of qemu that 
> I really wish didn't get lost in libvirt.  Again, /session can do this 
> too but right now, /session really isn't usable in libvirt for qemu.

That's wrong for three reasons.  First, selinux is not a uid replacement 
(if it was libvirt could just suid $random_user before launching qemu).  
Second, a single user's guests should be protected from each other.  
Third, in many deployments, the guest's owner isn't logged in to supply 
the credentials, it's system management that launches the guests.

There's also the case of resources that can't be permanently chowned or 
assigned a security label, like disk volumes or assignable devices.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [libvirt] [Qemu-devel] Re: Libvirt debug API

[Matthias Bolte]

2010/4/26 Avi Kivity <avi@redhat.com>:
>
[...]
>>
>>>> In theory, it does support this with the session urls but they are
>>>> currently second-class citizens in libvirt.  The remote dispatch also adds a
>>>> fair bit of complexity and at least for the use-cases I'm interested in,
>>>> it's not an important feature.
>>>
>>> If libvirt needs a local wrapper for interesting use cases, then it has
>>> failed.  You can't have a local wrapper with the esx driver, for example.
>>>
>>> This is off-topic, but can you detail why you don't want remote dispatch
>>> (I assume we're talking about a multiple node deployment).
>>
>> Because there are dozens of remote management APIs and then all have a
>> concept of agents that run on the end nodes.  When fitting virtualization
>> management into an existing management infrastructure, you are going to
>> always use a local API.
>
> When you manage esx, do you deploy an agent?  I thought it was all done via
> their remote APIs.
>

No, you don't deploy any agents.

If you manage an ESX server using VMware tools only, then you use
VMware management tools on the client PC. This tools directly use the
SOAP based remote API provided by the ESX server.

The ESX driver in libvirt does the same and uses this remote API as
the VMware management tools. No libvirt specific agents are installed
on the ESX server and no VMware specific agents (except libvirt
itself) are installed on the client PC.

Matthias

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/26/2010 12:56 AM, Avi Kivity wrote:
> On 04/26/2010 04:53 AM, Anthony Liguori wrote:
>> On 04/25/2010 06:51 AM, Avi Kivity wrote:
>>>> It depends on what things you think are important.  A lot of 
>>>> libvirt's complexity is based on the fact that it uses a daemon and 
>>>> needs to deal with the security implications of that.  You don't 
>>>> need explicit labelling if you don't use a daemon. 
>>>
>>>
>>> I don't follow.  If you have multiple guests that you want off each 
>>> other's turf you have to label their resources, either statically or 
>>> dynamically.  How is it related to a daemon being present?
>>
>> Because libvirt has to perform this labelling because it loses the 
>> original user's security context.
>>
>> If you invoke qemu with the original user's credentials that launched 
>> the guest, then you don't need to do anything special with respect to 
>> security.
>>
>> IOW, libvirt does not run guests as separate users which is why it 
>> needs to deal with security in the first place.
>
> What if one user has multiple guests?  isolation is still needed.

Don't confuse a management application's concept of users with using 
separate uid's to launch guests.

> One user per guest does not satisfy some security requirements.  The 
> 'M' in selinux stands for mandatory, which means that the entities 
> secured can't leak information even if they want to (scenario: G1 
> breaks into qemu, chmods files, G2 breaks into qemu, reads files).

If you're implementing a chinese firewall policy, then yes, you want to 
run each guest as a separate selinux context.  Starting as separate 
users and setting DAC privileges appropriately will achieve this.

But you're not always implementing that type of policy.  If the guest 
inherits the uid, selinux context, and namespaces of whatever launches 
the guest, then you have the most flexibility from a security perspective.

How do you launch a libvirt guest in a network namespace?  How do you 
put it in a chroot?  Today, you have to make changes to libvirt whereas 
in a direct launch model, you get all of the neat security features 
linux supports for free.

>> And I've said in the past that I don't like the idea of a qemud :-)
>
> I must have missed it.  Why not?  Every other hypervisor has a central 
> management entity.

Because you end up launching all guests from a single security context.

>>
>>>> In theory, it does support this with the session urls but they are 
>>>> currently second-class citizens in libvirt.  The remote dispatch 
>>>> also adds a fair bit of complexity and at least for the use-cases 
>>>> I'm interested in, it's not an important feature.
>>>
>>> If libvirt needs a local wrapper for interesting use cases, then it 
>>> has failed.  You can't have a local wrapper with the esx driver, for 
>>> example.
>>>
>>> This is off-topic, but can you detail why you don't want remote 
>>> dispatch (I assume we're talking about a multiple node deployment).
>>
>> Because there are dozens of remote management APIs and then all have 
>> a concept of agents that run on the end nodes.  When fitting 
>> virtualization management into an existing management infrastructure, 
>> you are going to always use a local API.
>
> When you manage esx, do you deploy an agent?  I thought it was all 
> done via their remote APIs.

Historically, people have deployed agents into the console OS.  In 
recent versions, ESX actually includes CIM agents by default.

>>>
>>> Every typical virtualization use will eventually grow some 
>>> non-typical requirements.  If libvirt explicitly refuses to support 
>>> qemu features, I don't see how we can recommend it - even if it 
>>> satisfies a user's requirements today, what about tomorrow? what 
>>> about future qemu feature, will they be exposed or not?
>>>
>>> If that is the case then we should develop qemud (which libvirt and 
>>> other apps can use).
>>>
>>> (even if it isn't the case I think qemud is a good idea)
>>
>> Yeah, that's where I'm at.  I'd eventually like libvirt to use our 
>> provided API and I can see where it would add value to the stack (by 
>> doing things like storage and network management).
>
> We do provide an API, qmp, and libvirt uses it?

Yeah, but we need to support more features (like guest enumeration).

>>>>
>>>> The alternative is to get libvirt to just act as a thin layer to 
>>>> expose qemu features directly.  But honestly, what's the point of 
>>>> libvirt if they did that? 
>>>
>>> For most hypervisors, that's exactly what libvirt does.  For Xen, it 
>>> also bypasses Xend and the hypervisor's API, but it shouldn't really.
>>
>> Historically, xend was so incredibly slow (especially for frequent 
>> statistics collection) that it was a necessity.
>
> Ah, reimplement rather than fix.

There's a complicated history there.

>>>   Qemu is special due to the nonexistence of qemud.
>>>
>>> Why is sVirt implemented in libvirt?  it's not the logical place for 
>>> it; rather the logical place doesn't exist.
>>
>> sVirt is not just implemented in libvirt.  libvirt implements a 
>> mechanism to set the context of a given domain and dynamically label 
>> it's resources to isolate it.
>>
>> The reason it has to assign a context to a given domain is that all 
>> domains are launched from the same security context (the libvirtd 
>> context) as the original user's context (the consumer of the libvirt 
>> API) has been lost via the domain socket interface.
>>
>> If you used the /session URL, then the domain would have the security 
>> context of whomever created the guest which means that dynamic 
>> labelling of the resources wouldn't be necessary (you would just do 
>> static labelling).
>>
>> This is certainly a more secure model and it's a feature of qemu that 
>> I really wish didn't get lost in libvirt.  Again, /session can do 
>> this too but right now, /session really isn't usable in libvirt for 
>> qemu.
>
> That's wrong for three reasons.  First, selinux is not a uid 
> replacement (if it was libvirt could just suid $random_user before 
> launching qemu).  Second, a single user's guests should be protected 
> from each other.  Third, in many deployments, the guest's owner isn't 
> logged in to supply the credentials, it's system management that 
> launches the guests.

(1) uid's are just one part of an applications security context.  
There's an selinux context, all of the various namespaces, capabilities, 
etc.  If you use a daemon to launch a guest, you lose all of that unless 
you have a very sophisticated api.

(2) If you want to implement a policy that only a single guest can 
access a single image, you can create an SELinux policy and use static 
labelling to achieve that.  That's just one type of policy though.

(3) The system management application can certainly create whatever 
context it wants to launch a vm from.  It's comes down to who's 
responsible for creating the context the guest runs under.  I think 
doing that at the libvirt level takes away a ton of flexibility from the 
management application.

Regards,

Anthony Liguori

> There's also the case of resources that can't be permanently chowned 
> or assigned a security label, like disk volumes or assignable devices.
>

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/26/2010 04:14 PM, Anthony Liguori wrote:
>>>
>>> IOW, libvirt does not run guests as separate users which is why it 
>>> needs to deal with security in the first place.
>>
>> What if one user has multiple guests?  isolation is still needed.
>
>
> Don't confuse a management application's concept of users with using 
> separate uid's to launch guests.

Then someone needs to manage those users.  A user can't suid to any 
random user.  You need someone privileged to allocate the new uid and su 
into it.

>
>> One user per guest does not satisfy some security requirements.  The 
>> 'M' in selinux stands for mandatory, which means that the entities 
>> secured can't leak information even if they want to (scenario: G1 
>> breaks into qemu, chmods files, G2 breaks into qemu, reads files).
>
> If you're implementing a chinese firewall policy, then yes, you want 
> to run each guest as a separate selinux context.  Starting as separate 
> users and setting DAC privileges appropriately will achieve this.
>
> But you're not always implementing that type of policy.  If the guest 
> inherits the uid, selinux context, and namespaces of whatever launches 
> the guest, then you have the most flexibility from a security 
> perspective.
>
> How do you launch a libvirt guest in a network namespace?  How do you 
> put it in a chroot? 

You pass the namespace fd and chroot fd using SCM_RIGHTS (except you 
probably can't do that).

> Today, you have to make changes to libvirt whereas in a direct launch 
> model, you get all of the neat security features linux supports for free.

But you lose tap networking, unless you have a privileged helper.  And 
how is the privileged helper to authenticate the qemu calling it?

>>> And I've said in the past that I don't like the idea of a qemud :-)
>>
>> I must have missed it.  Why not?  Every other hypervisor has a 
>> central management entity.
>
> Because you end up launching all guests from a single security context.

Run multiple qemuds?

But what you say makes sense.  It's similar to the fork()  /* do 
interesting stuff */ exec() model, compared to the spawn(..., hardcoded 
list of interesting stuff).

>>> Yeah, that's where I'm at.  I'd eventually like libvirt to use our 
>>> provided API and I can see where it would add value to the stack (by 
>>> doing things like storage and network management).
>>
>> We do provide an API, qmp, and libvirt uses it?
>
> Yeah, but we need to support more features (like guest enumeration).


What are our options?

1) qemud launches, enumerates
2) user launches, qemu registers in qemud
3) user launches, qemu registers in filesystem
4) you launched it, you enumerate it

>> That's wrong for three reasons.  First, selinux is not a uid 
>> replacement (if it was libvirt could just suid $random_user before 
>> launching qemu).  Second, a single user's guests should be protected 
>> from each other.  Third, in many deployments, the guest's owner isn't 
>> logged in to supply the credentials, it's system management that 
>> launches the guests.
>
> (1) uid's are just one part of an applications security context.  
> There's an selinux context, all of the various namespaces, 
> capabilities, etc.  If you use a daemon to launch a guest, you lose 
> all of that unless you have a very sophisticated api.

True.  In a perfect world, we'd use SCM_RIGHTS to channel all of these 
to libvirt or qemud.

On the other hand, users don't want to do all these things by hand.  
They want management to do things for them.  Self launch is very 
flexible, but it's not an API, and cannot be used remotely.

We could use qemud plugins to allow the user to customize the launch 
process.

>
> (2) If you want to implement a policy that only a single guest can 
> access a single image, you can create an SELinux policy and use static 
> labelling to achieve that.  That's just one type of policy though.

It's also not going to work in an environment that doesn't preserve all 
security labels (like direct access to volumes; /dev is on tmpfs these 
days).

> (3) The system management application can certainly create whatever 
> context it wants to launch a vm from.  It's comes down to who's 
> responsible for creating the context the guest runs under.  I think 
> doing that at the libvirt level takes away a ton of flexibility from 
> the management application.

If you want to push the flexibility slider all the way to the right you 
get bare qemu.  It exposes 100% of qemu capabilities.  And it's not so 
bad these days.  But it's not something that can be remoted.


-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/26/2010 08:41 AM, Avi Kivity wrote:
>> Today, you have to make changes to libvirt whereas in a direct launch 
>> model, you get all of the neat security features linux supports for 
>> free.
>
> But you lose tap networking, unless you have a privileged helper.  And 
> how is the privileged helper to authenticate the qemu calling it?

There are a variety of ways.  My original proposal used a policy file.

>>>> And I've said in the past that I don't like the idea of a qemud :-)
>>>
>>> I must have missed it.  Why not?  Every other hypervisor has a 
>>> central management entity.
>>
>> Because you end up launching all guests from a single security context.
>
> Run multiple qemuds?
>
> But what you say makes sense.  It's similar to the fork()  /* do 
> interesting stuff */ exec() model, compared to the spawn(..., 
> hardcoded list of interesting stuff).
>
>>>> Yeah, that's where I'm at.  I'd eventually like libvirt to use our 
>>>> provided API and I can see where it would add value to the stack 
>>>> (by doing things like storage and network management).
>>>
>>> We do provide an API, qmp, and libvirt uses it?
>>
>> Yeah, but we need to support more features (like guest enumeration).
>
>
> What are our options?
>
> 1) qemud launches, enumerates
> 2) user launches, qemu registers in qemud
> 3) user launches, qemu registers in filesystem
> 4) you launched it, you enumerate it

Both 2 and 3 are appealing to me.

>> (3) The system management application can certainly create whatever 
>> context it wants to launch a vm from.  It's comes down to who's 
>> responsible for creating the context the guest runs under.  I think 
>> doing that at the libvirt level takes away a ton of flexibility from 
>> the management application.
>
> If you want to push the flexibility slider all the way to the right 
> you get bare qemu.  It exposes 100% of qemu capabilities.  And it's 
> not so bad these days.  But it's not something that can be remoted.

As I mentioned earlier, remoting is not a very important use-case to me.

Does RHEV-M actually use the remote libvirt interface?  I assume it'll 
talk to vdsm via some protocol and vdsm will use the local libvirt API.  
I suspect most uses of libvirt are actually local uses.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/26/2010 04:46 PM, Anthony Liguori wrote:
>>> (3) The system management application can certainly create whatever 
>>> context it wants to launch a vm from.  It's comes down to who's 
>>> responsible for creating the context the guest runs under.  I think 
>>> doing that at the libvirt level takes away a ton of flexibility from 
>>> the management application.
>>
>> If you want to push the flexibility slider all the way to the right 
>> you get bare qemu.  It exposes 100% of qemu capabilities.  And it's 
>> not so bad these days.  But it's not something that can be remoted.
>
> As I mentioned earlier, remoting is not a very important use-case to me.
>
> Does RHEV-M actually use the remote libvirt interface?  I assume it'll 
> talk to vdsm via some protocol and vdsm will use the local libvirt API. 

Yes.

> I suspect most uses of libvirt are actually local uses.

I expect the same, though I'm sure a design goal was to make use of 
libvirt be reasonable through the remote API.  If we aren't able to 
fulfil it, much of the value of libvirt goes away.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Daniel P. Berrange]

On Mon, Apr 26, 2010 at 08:46:46AM -0500, Anthony Liguori wrote:
> On 04/26/2010 08:41 AM, Avi Kivity wrote:
> 
> >>(3) The system management application can certainly create whatever 
> >>context it wants to launch a vm from.  It's comes down to who's 
> >>responsible for creating the context the guest runs under.  I think 
> >>doing that at the libvirt level takes away a ton of flexibility from 
> >>the management application.
> >
> >If you want to push the flexibility slider all the way to the right 
> >you get bare qemu.  It exposes 100% of qemu capabilities.  And it's 
> >not so bad these days.  But it's not something that can be remoted.
> 
> As I mentioned earlier, remoting is not a very important use-case to me.

NB, the "remote" protocol is no different from the "local" protocol
the unprivileged app uses to talk to the privileged daemon. The only
difference is unix sockets vs tcp sockets. We essentially get the
remote access capability for free as part of the protocol used between
the apps & daemon. 

> Does RHEV-M actually use the remote libvirt interface?  I assume it'll 
> talk to vdsm via some protocol and vdsm will use the local libvirt API.  

That is correct. That doesn't mean the remote interface isn't useful.

> I suspect most uses of libvirt are actually local uses.

A key part of libvirt is interoperability between applications. So an app
like vsdm can be using libvirt locally, while at the same time virt-top
can be monitoring guests on a set of machines remotely, or virt-viewer can
be run from your desktop talking to libvirt remotely to discover the VNC 
console details. For flexibility there are several options for remotely
managing VMs. The native remote RPC system. The libvirt QPid/QMF model.
The CIM binding. Application specific remote protocols. Since they all
eventually talk to the same libvirtd daemon, their use is not mutually
exclusive which is a good thing.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/26/2010 08:58 AM, Daniel P. Berrange wrote:
> On Mon, Apr 26, 2010 at 08:46:46AM -0500, Anthony Liguori wrote:
>    
>> On 04/26/2010 08:41 AM, Avi Kivity wrote:
>>
>>      
>>>> (3) The system management application can certainly create whatever
>>>> context it wants to launch a vm from.  It's comes down to who's
>>>> responsible for creating the context the guest runs under.  I think
>>>> doing that at the libvirt level takes away a ton of flexibility from
>>>> the management application.
>>>>          
>>> If you want to push the flexibility slider all the way to the right
>>> you get bare qemu.  It exposes 100% of qemu capabilities.  And it's
>>> not so bad these days.  But it's not something that can be remoted.
>>>        
>> As I mentioned earlier, remoting is not a very important use-case to me.
>>      
> NB, the "remote" protocol is no different from the "local" protocol
> the unprivileged app uses to talk to the privileged daemon. The only
> difference is unix sockets vs tcp sockets. We essentially get the
> remote access capability for free as part of the protocol used between
> the apps&  daemon.
>    

I think Avi's concern is that if you expose direct launch, then that's 
not really something you can usefully exploit in a remote protocol.

For instance, if you're local, you can call setuid() before invoking the 
launch API whereas if you're remote, you will always launch guests as 
the context of the remoting daemon.

This makes the remote API second-class in terms of functionality.

Regards,

Anthony Liguori

>    
>> Does RHEV-M actually use the remote libvirt interface?  I assume it'll
>> talk to vdsm via some protocol and vdsm will use the local libvirt API.
>>      
> That is correct. That doesn't mean the remote interface isn't useful.
>
>    
>> I suspect most uses of libvirt are actually local uses.
>>      
> A key part of libvirt is interoperability between applications. So an app
> like vsdm can be using libvirt locally, while at the same time virt-top
> can be monitoring guests on a set of machines remotely, or virt-viewer can
> be run from your desktop talking to libvirt remotely to discover the VNC
> console details. For flexibility there are several options for remotely
> managing VMs. The native remote RPC system. The libvirt QPid/QMF model.
> The CIM binding. Application specific remote protocols. Since they all
> eventually talk to the same libvirtd daemon, their use is not mutually
> exclusive which is a good thing.
>
> Daniel
>    

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Daniel P. Berrange]

On Mon, Apr 26, 2010 at 09:26:55AM -0500, Anthony Liguori wrote:
> On 04/26/2010 08:58 AM, Daniel P. Berrange wrote:
> >On Mon, Apr 26, 2010 at 08:46:46AM -0500, Anthony Liguori wrote:
> >   
> >>On 04/26/2010 08:41 AM, Avi Kivity wrote:
> >>
> >>     
> >>>>(3) The system management application can certainly create whatever
> >>>>context it wants to launch a vm from.  It's comes down to who's
> >>>>responsible for creating the context the guest runs under.  I think
> >>>>doing that at the libvirt level takes away a ton of flexibility from
> >>>>the management application.
> >>>>         
> >>>If you want to push the flexibility slider all the way to the right
> >>>you get bare qemu.  It exposes 100% of qemu capabilities.  And it's
> >>>not so bad these days.  But it's not something that can be remoted.
> >>>       
> >>As I mentioned earlier, remoting is not a very important use-case to me.
> >>     
> >NB, the "remote" protocol is no different from the "local" protocol
> >the unprivileged app uses to talk to the privileged daemon. The only
> >difference is unix sockets vs tcp sockets. We essentially get the
> >remote access capability for free as part of the protocol used between
> >the apps&  daemon.
> >   
> 
> I think Avi's concern is that if you expose direct launch, then that's 
> not really something you can usefully exploit in a remote protocol.
> 
> For instance, if you're local, you can call setuid() before invoking the 
> launch API whereas if you're remote, you will always launch guests as 
> the context of the remoting daemon.
> 
> This makes the remote API second-class in terms of functionality.

It is not entirely outside the realms of possibility to consider
making a way for libvirt to attach to a QEMU spawned by a 3rd
party. eg, 

    virsh qemu-attach /path/to/monitor PID

Converting the QEMU config back into Libvirt XML is not trivial,
but neither is it impossible. Particularly now we have the custom
<qemu:arg> syntax for things libvirt can't represent, we can 
simply use that for bits we can't reverse map to XML. There's also
the UUID/name uniqueness, but we could fail the attach operation 
if that wwas violated. This would let you use most of the other 
libvirt APIs remotely from any app, even if the VM was spawned by
another one.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Daniel P. Berrange]

On Sun, Apr 25, 2010 at 08:53:17PM -0500, Anthony Liguori wrote:
> On 04/25/2010 06:51 AM, Avi Kivity wrote:
> >  Qemu is special due to the nonexistence of qemud.
> >
> >Why is sVirt implemented in libvirt?  it's not the logical place for 
> >it; rather the logical place doesn't exist.
> 
> sVirt is not just implemented in libvirt.  libvirt implements a 
> mechanism to set the context of a given domain and dynamically label 
> it's resources to isolate it.
> 
> The reason it has to assign a context to a given domain is that all 
> domains are launched from the same security context (the libvirtd 
> context) as the original user's context (the consumer of the libvirt 
> API) has been lost via the domain socket interface.
> 
> If you used the /session URL, then the domain would have the security 
> context of whomever created the guest which means that dynamic labelling 
> of the resources wouldn't be necessary (you would just do static labelling).

That is not correct. You do *not* ever want the guests to have the same 
security context as the thing that created them, because that would allow
the guest to access & compromise resources belonging to the management app.
Every guest must always have a unique context. The libvirt+sVirt dynamic 
labelling with unique contexts is applied even for the /session mode.
The thing you save with the /session mode, is not having todo the user/group
ownership management; sVirt labelling is always required.

> This is certainly a more secure model and it's a feature of qemu that I 
> really wish didn't get lost in libvirt.  Again, /session can do this too 
> but right now, /session really isn't usable in libvirt for qemu.

If you really want the qemu instance to inherit the context of the mgmt
app, then you can just declare in the guest XML that it should use a
static label, and pass in the apps' own label. This is *not* a more secure
model though.


Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/26/2010 04:59 AM, Daniel P. Berrange wrote:
> On Sun, Apr 25, 2010 at 08:53:17PM -0500, Anthony Liguori wrote:
>    
>> On 04/25/2010 06:51 AM, Avi Kivity wrote:
>>      
>>>   Qemu is special due to the nonexistence of qemud.
>>>
>>> Why is sVirt implemented in libvirt?  it's not the logical place for
>>> it; rather the logical place doesn't exist.
>>>        
>> sVirt is not just implemented in libvirt.  libvirt implements a
>> mechanism to set the context of a given domain and dynamically label
>> it's resources to isolate it.
>>
>> The reason it has to assign a context to a given domain is that all
>> domains are launched from the same security context (the libvirtd
>> context) as the original user's context (the consumer of the libvirt
>> API) has been lost via the domain socket interface.
>>
>> If you used the /session URL, then the domain would have the security
>> context of whomever created the guest which means that dynamic labelling
>> of the resources wouldn't be necessary (you would just do static labelling).
>>      
> That is not correct. You do *not* ever want the guests to have the same
> security context as the thing that created them, because that would allow
> the guest to access&  compromise resources belonging to the management app.
>    

You assume that the management app is not smart enough to create a new 
context for the guest to run in.

>> This is certainly a more secure model and it's a feature of qemu that I
>> really wish didn't get lost in libvirt.  Again, /session can do this too
>> but right now, /session really isn't usable in libvirt for qemu.
>>      
> If you really want the qemu instance to inherit the context of the mgmt
> app, then you can just declare in the guest XML that it should use a
> static label, and pass in the apps' own label. This is *not* a more secure
> model though.
>    

There is more context than just selinux labelling.  The problem with the 
daemon model is that to create a guest, you start with a lower set of 
privileges, escalate your privileges (by talking to libvirtd), then 
lower privileges to launch a guest.  Running a guest is essentially 
running arbitrary code (since you can set the emulator path) so now 
you've provided an environment where a user can launch arbitrary code as 
a different user in a different security context.

There is a new attack surface here.  I think it's undeniable that there 
is certainly the possibility that something goes wrong and a user will 
find a way to escalate it's privileges.

Compare that to a direct launch model.  There is not new attack 
surface.  The user's privileges never increase.  In fact, what's most 
likely to happen is that a caller will drop some of it's privileges 
before launching a guest.

Regards,

Anthony Liguori

> Daniel
>    

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Daniel P. Berrange]

On Mon, Apr 26, 2010 at 08:13:03AM -0500, Anthony Liguori wrote:
> On 04/26/2010 04:59 AM, Daniel P. Berrange wrote:
> >On Sun, Apr 25, 2010 at 08:53:17PM -0500, Anthony Liguori wrote:
> >   
> >>On 04/25/2010 06:51 AM, Avi Kivity wrote:
> >>     
> >>>  Qemu is special due to the nonexistence of qemud.
> >>>
> >>>Why is sVirt implemented in libvirt?  it's not the logical place for
> >>>it; rather the logical place doesn't exist.
> >>>       
> >>sVirt is not just implemented in libvirt.  libvirt implements a
> >>mechanism to set the context of a given domain and dynamically label
> >>it's resources to isolate it.
> >>
> >>The reason it has to assign a context to a given domain is that all
> >>domains are launched from the same security context (the libvirtd
> >>context) as the original user's context (the consumer of the libvirt
> >>API) has been lost via the domain socket interface.
> >>
> >>If you used the /session URL, then the domain would have the security
> >>context of whomever created the guest which means that dynamic labelling
> >>of the resources wouldn't be necessary (you would just do static 
> >>labelling).
> >>     
> >That is not correct. You do *not* ever want the guests to have the same
> >security context as the thing that created them, because that would allow
> >the guest to access&  compromise resources belonging to the management app.
> >   
> 
> You assume that the management app is not smart enough to create a new 
> context for the guest to run in.
> 
> >>This is certainly a more secure model and it's a feature of qemu that I
> >>really wish didn't get lost in libvirt.  Again, /session can do this too
> >>but right now, /session really isn't usable in libvirt for qemu.
> >>     
> >If you really want the qemu instance to inherit the context of the mgmt
> >app, then you can just declare in the guest XML that it should use a
> >static label, and pass in the apps' own label. This is *not* a more secure
> >model though.
> >   
> 
> There is more context than just selinux labelling.  The problem with the 
> daemon model is that to create a guest, you start with a lower set of 
> privileges, escalate your privileges (by talking to libvirtd), then 
> lower privileges to launch a guest.  Running a guest is essentially 
> running arbitrary code (since you can set the emulator path) so now 
> you've provided an environment where a user can launch arbitrary code as 
> a different user in a different security context.

No matter what emulator path is provided, that will never be run with
elevated privileges unless the host sysadmin has explicitly configured
that capability. The QEMU binaries always run in a dedicated account
which is completely unprivileged. This setuid is done *before* QEMU 
is even exec()'d. This is one of the reasons why we don't use the
QEMU -runas argument, which would allow custom emulator binaries to
potentially gain evelated privileges.

> There is a new attack surface here.  I think it's undeniable that there 
> is certainly the possibility that something goes wrong and a user will 
> find a way to escalate it's privileges.

Of course all software has bugs. 

> Compare that to a direct launch model.  There is not new attack 
> surface.  The user's privileges never increase.  In fact, what's most 
> likely to happen is that a caller will drop some of it's privileges 
> before launching a guest.

What you describe is not inherant to the daemon model. This is why we have
two separate models in libvirt. The system instance is pre-spawned with
high privileges, to allow use of hosts resources which require high 
privileges to access. The session instance is auto-spawned when the app
connects to libvirt, thus it inherits the privileges of the app that is
using it.

I don't deny that the system instance has a new attack surface, because it
is running privileged. If the app needs to connect VMs to privileged 
resources, then the architecture has to have some privileged component
to give access to those resoruces. You don't want the VM to be privileged,
nor the whole management app to be privileged. The system daemon is thus the
arbitrator for this privileged access. If you don't need todo anything that
requires higher privileges though, just use the session instance which 
always matches the apps privileges. 

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/26/2010 08:31 AM, Daniel P. Berrange wrote:
> What you describe is not inherant to the daemon model. This is why we have
> two separate models in libvirt. The system instance is pre-spawned with
> high privileges, to allow use of hosts resources which require high
> privileges to access. The session instance is auto-spawned when the app
> connects to libvirt, thus it inherits the privileges of the app that is
> using it.
>
> I don't deny that the system instance has a new attack surface, because it
> is running privileged. If the app needs to connect VMs to privileged
> resources, then the architecture has to have some privileged component
> to give access to those resoruces. You don't want the VM to be privileged,
> nor the whole management app to be privileged. The system daemon is thus the
> arbitrator for this privileged access. If you don't need todo anything that
> requires higher privileges though, just use the session instance which
> always matches the apps privileges.
>    

I regret saying "more secure" because I think it's a difficult concept 
to really quantify and that makes it hard to meaningfully discuss.

The reason I lean toward the direct launch model is that it gives the 
user a lot of flexibility in terms of using things like namespaces, DAC, 
cgroups, capabilities, etc.  A lot of potential features are lost when 
you do indirect launch because you have to teach the daemon how to 
support each of these features.

Regards,

Anthony Liguori

> Daniel
>    

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/26/2010 04:43 PM, Anthony Liguori wrote:
> The reason I lean toward the direct launch model is that it gives the 
> user a lot of flexibility in terms of using things like namespaces, 
> DAC, cgroups, capabilities, etc.  A lot of potential features are lost 
> when you do indirect launch because you have to teach the daemon how 
> to support each of these features.

But what's the alternative?  Teach the user how to do all these things?

It's infinitely flexible, but it's not an API you can give to a 
management tool developer.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/26/2010 09:01 AM, Avi Kivity wrote:
> On 04/26/2010 04:43 PM, Anthony Liguori wrote:
>> The reason I lean toward the direct launch model is that it gives the 
>> user a lot of flexibility in terms of using things like namespaces, 
>> DAC, cgroups, capabilities, etc.  A lot of potential features are 
>> lost when you do indirect launch because you have to teach the daemon 
>> how to support each of these features.
>
> But what's the alternative?  Teach the user how to do all these things?

You can expose layers of API.  The lowest layer makes no changes to the 
security context.  A higher (optional) layer could do dynamic labelling.

> It's infinitely flexible, but it's not an API you can give to a 
> management tool developer.

I think the goal of a management API should be to make common things 
very simple to do but not preclude doing even the most advanced things.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/26/2010 05:19 PM, Anthony Liguori wrote:
> On 04/26/2010 09:01 AM, Avi Kivity wrote:
>> On 04/26/2010 04:43 PM, Anthony Liguori wrote:
>>> The reason I lean toward the direct launch model is that it gives 
>>> the user a lot of flexibility in terms of using things like 
>>> namespaces, DAC, cgroups, capabilities, etc.  A lot of potential 
>>> features are lost when you do indirect launch because you have to 
>>> teach the daemon how to support each of these features.
>>
>> But what's the alternative?  Teach the user how to do all these things?
>
> You can expose layers of API.  The lowest layer makes no changes to 
> the security context.  A higher (optional) layer could do dynamic 
> labelling.

Or a library that the user-written launcher calls.  Or a plugin that 
qemud calls.

>> It's infinitely flexible, but it's not an API you can give to a 
>> management tool developer.
>
> I think the goal of a management API should be to make common things 
> very simple to do but not preclude doing even the most advanced things.

Agreed.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/26/2010 09:25 AM, Avi Kivity wrote:
> On 04/26/2010 05:19 PM, Anthony Liguori wrote:
>> On 04/26/2010 09:01 AM, Avi Kivity wrote:
>>> On 04/26/2010 04:43 PM, Anthony Liguori wrote:
>>>> The reason I lean toward the direct launch model is that it gives 
>>>> the user a lot of flexibility in terms of using things like 
>>>> namespaces, DAC, cgroups, capabilities, etc.  A lot of potential 
>>>> features are lost when you do indirect launch because you have to 
>>>> teach the daemon how to support each of these features.
>>>
>>> But what's the alternative?  Teach the user how to do all these things?
>>
>> You can expose layers of API.  The lowest layer makes no changes to 
>> the security context.  A higher (optional) layer could do dynamic 
>> labelling.
>
> Or a library that the user-written launcher calls.  Or a plugin that 
> qemud calls.

A plugin would lose the security context.  It could attempt to recreate 
it that seems like a lot of unnecessary complexity.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/26/2010 05:28 PM, Anthony Liguori wrote:
>> Or a library that the user-written launcher calls.  Or a plugin that 
>> qemud calls.
>
>
> A plugin would lose the security context.  It could attempt to 
> recreate it that seems like a lot of unnecessary complexity.
>

A plugin would create the security context instead of the launcher.

Currently security contexts are created by the login process.  We could 
easily reuse that.  Any other security context code would be custom 
written; so it can be written as a qemud plugin instead of a bit of code 
that goes before a qemu launch.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/26/2010 09:38 AM, Avi Kivity wrote:
> On 04/26/2010 05:28 PM, Anthony Liguori wrote:
>>> Or a library that the user-written launcher calls.  Or a plugin that 
>>> qemud calls.
>>
>>
>> A plugin would lose the security context.  It could attempt to 
>> recreate it that seems like a lot of unnecessary complexity.
>>
>
> A plugin would create the security context instead of the launcher.
>
> Currently security contexts are created by the login process.

It's not always that centralized.  An initial context is created by the 
login process, but then later something may come along and create a 
network namespace as part of containerization.

>   We could easily reuse that.  Any other security context code would 
> be custom written; so it can be written as a qemud plugin instead of a 
> bit of code that goes before a qemu launch.

I think we're mostly in agreement with respect to the need to have more 
control over the security context the qemu runs in.  Whether it's 
launched via a daemon or directly I think is an implementation detail 
that we can debate when we get closer to an actual implementation.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/26/2010 05:48 PM, Anthony Liguori wrote:
>
>>   We could easily reuse that.  Any other security context code would 
>> be custom written; so it can be written as a qemud plugin instead of 
>> a bit of code that goes before a qemu launch.
>
> I think we're mostly in agreement with respect to the need to have 
> more control over the security context the qemu runs in.  Whether it's 
> launched via a daemon or directly I think is an implementation detail 
> that we can debate when we get closer to an actual implementation.
>

Good, as I haven't decided yet which side I'm on.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Daniel P. Berrange]

On Fri, Apr 23, 2010 at 08:48:51AM -0500, Anthony Liguori wrote:
> On 04/23/2010 07:48 AM, Avi Kivity wrote:
> >On 04/22/2010 09:49 PM, Anthony Liguori wrote:
> >>>real API. Say, adding a device libvirt doesn't know about or 
> >>>stopping the VM
> >>>while libvirt thinks it's still running or anything like that.
> >>  Another problem is issuing Monitor commands that could confuse 
> >>libvirt's
> >>
> >>We need to make libvirt and qemu smarter.
> >>
> >>We already face this problem today with multiple libvirt users.  This 
> >>is why sophisticated management mechanisms (like LDAP) have 
> >>mechanisms to do transactions or at least a series of atomic operations.
> >
> >And people said qmp/json was overengineered...
> >
> >But seriously, transactions won't help anything.  qemu maintains 
> >state, and when you have two updaters touching a shared variable not 
> >excepting each other to, things break, no matter how much locking 
> >there is.
> 
> Let's consider some concrete examples.  I'm using libvirt and QMP and in 
> QMP, I want to hot unplug a device.
> 
> Today, I do this by listing the pci devices, and issuing a pci_del that 
> takes a PCI address.  This is intrinsically racy though because in the 
> worst case scenario, in between when I enumerate pci devices and do the 
> pci_del in QMP, in libvirt, I've done a pci_del and then a pci_add 
> within libvirt of a completely different device.

This is what already happens with any QEMU >= 0.12, where libvirt uses
the new -device syntax  with its 'id' parameter for all devices, and 
then uses 'device_id $ID' for unplug. The app still has to be careful
it doesn't try to add a device using the same naming scheme as a
device libvirt uses. This is easy enough though if they prepend some
random string to all device IDs thy use, that won't clash with libvirt
device ID names.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Markus Armbruster]

Anthony Liguori <anthony@codemonkey.ws> writes:

> On 04/23/2010 07:48 AM, Avi Kivity wrote:
>> On 04/22/2010 09:49 PM, Anthony Liguori wrote:
>>>> real API. Say, adding a device libvirt doesn't know about or
>>>> stopping the VM
>>>> while libvirt thinks it's still running or anything like that.
>>>   Another problem is issuing Monitor commands that could confuse
>>> libvirt's
>>>
>>> We need to make libvirt and qemu smarter.
>>>
>>> We already face this problem today with multiple libvirt users.
>>> This is why sophisticated management mechanisms (like LDAP) have
>>> mechanisms to do transactions or at least a series of atomic
>>> operations.
>>
>> And people said qmp/json was overengineered...
>>
>> But seriously, transactions won't help anything.  qemu maintains
>> state, and when you have two updaters touching a shared variable not
>> excepting each other to, things break, no matter how much locking
>> there is.
>
> Let's consider some concrete examples.  I'm using libvirt and QMP and
> in QMP, I want to hot unplug a device.
>
> Today, I do this by listing the pci devices, and issuing a pci_del
> that takes a PCI address.  This is intrinsically racy though because
> in the worst case scenario, in between when I enumerate pci devices
> and do the pci_del in QMP, in libvirt, I've done a pci_del and then a
> pci_add within libvirt of a completely different device.
>
> There are a few ways to solve this, the simplest being that we give
> devices unique ids that are never reused and instead of pci_del taking
> a pci bus address, it takes a device id.  That would address this
> race.

For what it's worth, that's how device_del works, except the ID is
chosen by the user, who is free to reuse them.

Really tangential to this thread, but it's worth repeating anyway: use
device_add & friends, not pci_add & friends.

[...]

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:
> On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
>    
>> Hello,
>>       In response to a lot of the talk of qemud lately on qemu-devel, the
>> libvirt community would like to put forward a proposal to help enable
>> debug/advanced options when using various hypervisors.  The goals of
>> this API are:
>>
>> 1)  To enable more rapid access to hypervisor features before proper
>> libvirt API's are designed around them.
>> 2)  To facilitate debugging and access to advanced features that may
>> not fit into the normal libvirt world-view.
>>
>> Caveats:
>> 1)  Unlike other libvirt API's, this one will explicitly *not* be
>> guaranteed ABI/API compatible between libvirt updates.
>>      
> I think we'd still aim to keep the public API stable. The bit that we
> can't guarentee is the interactions with QEMU&  the libvirt driver.
> eg, if someone was using the API to send text monitor commands to
> QEMU, and the next libvirt release switched to JSON mode, that use
> would break. The API would still be valid, but the way they use it
> might not be. Similarly if they add some custom extra command line
> argument, this could potentially conflict with an extra command line
> arg a subsquent libvirt release used. eg, they used -device to add
> a PCI card with a specific PCI address. Then next libvirt release
> specifies this same PCI address and then you get a clash.
>
> So in both cases the API and XML format can be reasonably guarenteed
> between releases. What we can't guarentee is that usage  of these
> features will be reliable across releases of libvirt.
>
>    
>> 2)  Again unlike other libvirt API's, access and configuration of
>> the debug section of a domain will be highly hypervisor dependent.
>> 3)  Application developers will be strongly discouraged from using
>> this API because of the above 2 issues.  To help in this, the
>> API's will be in a separate library that developers will explicitly
>> have to link to, and it will have a different (but largely compatible)
>> wire protocol.
>>      
> In terms of wire protocol, this would still have to run over the same
> existing data channel, because we don't want apps to have to open up
> multiple connections. Fortunately our protocol is designed to allow
> this kind of extension
>
>    struct remote_message_header {
>        unsigned prog;              /* REMOTE_PROGRAM */
>        unsigned vers;              /* REMOTE_PROTOCOL_VERSION */
>        remote_procedure proc;      /* REMOTE_PROC_x */
>        remote_message_type type;
>        unsigned serial;            /* Serial number of message. */
>        remote_message_status status;
>    };
>
> We currently have a single program 'REMOTE_PROGRAM' and its associated
> 'vers' and 'proc' numbers. What we would be doing is to define a new
> program, say QEMU_PROGRAM. The 'remote_procedure proc' numbers can thus
> start again from '1' without clashing with any of the existing APIs.
> All the RPC marshalling/demarshalling code should work more or less
> unchanged. We just hook in a different function dispatch table in the
> daemon for that program.
>
>    
>> 4)  We don't expect this API to solve all of the issues brought up
>> during the qemud discussion.  Our initial goal is just to give
>> ready access of the qemu command-line and monitor to developers.
>>
>> With that being said, our initial proposal follows.  We expect this
>> to evolve over time as we get more feedback, but we think this
>> proposal addresses at least 2 of the major pain points qemu developers
>> have while trying to use libvirt.
>>
>> The initial debug XML for qemu would be:
>>      
> I wouldn't call them 'debug' options, since that's just one of possible
> use cases for this. What we're really doing there is exposing hypervisor
> specific features, so we should just be upfront about that in our
> description / naming.
>
>    
>> <domain type='kvm'>
>>    <name>myguest</name>
>>    ...
>>    <debug>
>>      <monitorpassthrough/>
>>      <commandline>
>>        <extra>qemu arguments</extra>
>>        <alter option="optname">
>>          <rename>newname</rename>
>>          <match>REGEXP</match>
>>          <modify>foo=on</modify>
>>          <extra>-bar</extra>
>>        </alter>
>>      </commandline>
>>    </debug>
>> </domain>
>>      
> The concept of command line&  monitor is something that is QEMU specific
> and thus is not suitable for the primary XML schema. IMHO, this needs to be
> done as a separate schema, linked in via an XML namespace. For example
>
>    <domain type='kvm' xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
>      <name>myguest</name>
>      ...
>      <qemu:commandline>
>        <qemu:arg>-device</arg>
>        <qemu:arg>lsi</arg>
>      </qemu:commandline>
>    </domain>
>    

I think it's problematic to focus too much on command line arguments.  
We are not introducing new command line arguments to qemu for the most 
part that aren't usable in the config file.

What I really think we would like, is a way to express the qemu 
configuration file in terms of XML and then to use snippets of that 
within an XML namespace.  For instance:

[cpudef]
  name = "Opteron_G3"
  level = "5"
  vendor = "AuthenticAMD"
  ...

Could convert to:

<cpudef>
<name>Operon_G3</name>
<level>5</level>
<vendor>AuthenticAMD</vendor>
   ...
</cpudef>

Which in turn, could be embedded as:

<domain type='kvm' xmlns:qemu='http://qemu.org/schemas/qemu/1.0'>
<name>myguest</name>
    ...
<qemu:cpudef>
<name>Operon_G3</name>
<level>5</level>
<vendor>AuthenticAMD</vendor>
</qemu:cpudef>
</domain>

With respect to injecting QMP commands directly, I think the proposed 
debug API is probably reasonable.  We could build a libqemu that used 
that API as a transport which means that one could use libqemu and 
libvirt simultaneously which is certainly a key requirement of mine.

I think it's important that it's a dedicated monitor session though.  It 
shouldn't just be injecting commands within an existing QMP session IMHO.

Regards,

Anthony Liguori

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/22/2010 01:45 PM, Anthony Liguori wrote:
> Which in turn, could be embedded as:
>
> <domain type='kvm' xmlns:qemu='http://qemu.org/schemas/qemu/1.0'>
> <name>myguest</name>
>    ...
> <qemu:cpudef>
> <name>Operon_G3</name>
> <level>5</level>
> <vendor>AuthenticAMD</vendor>
> </qemu:cpudef>
> </domain>
>
> With respect to injecting QMP commands directly, I think the proposed 
> debug API is probably reasonable.  We could build a libqemu that used 
> that API as a transport which means that one could use libqemu and 
> libvirt simultaneously which is certainly a key requirement of mine.
>
> I think it's important that it's a dedicated monitor session though.  
> It shouldn't just be injecting commands within an existing QMP session 
> IMHO.
>

Another important thing to mention is that this type of conversion has 
two purposes.  A user can inject qemu config snippets into the libvirt 
description, but libvirt can also inject qemu config snippets that it 
cannot represent via it's own XML for the user.

For instance, let's say that a user adds a disk via QMP with an option 
that libvirt doesn't recognize.  When a virsh dumpxml is executed, 
libvirt could display the XML using the <qemu:drive> tag instead of the 
<disk> tag that it normally would use.

Regards,

Anthony Liguori

> Regards,
>
> Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Daniel P. Berrange]

On Thu, Apr 22, 2010 at 01:45:27PM -0500, Anthony Liguori wrote:
> On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:
> >On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
> >
> >   
> >><domain type='kvm'>
> >>   <name>myguest</name>
> >>   ...
> >>   <debug>
> >>     <monitorpassthrough/>
> >>     <commandline>
> >>       <extra>qemu arguments</extra>
> >>       <alter option="optname">
> >>         <rename>newname</rename>
> >>         <match>REGEXP</match>
> >>         <modify>foo=on</modify>
> >>         <extra>-bar</extra>
> >>       </alter>
> >>     </commandline>
> >>   </debug>
> >></domain>
> >>     
> >The concept of command line&  monitor is something that is QEMU specific
> >and thus is not suitable for the primary XML schema. IMHO, this needs to be
> >done as a separate schema, linked in via an XML namespace. For example
> >
> >   <domain type='kvm' 
> >   xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
> >     <name>myguest</name>
> >     ...
> >     <qemu:commandline>
> >       <qemu:arg>-device</arg>
> >       <qemu:arg>lsi</arg>
> >     </qemu:commandline>
> >   </domain>
> >   
> 
> I think it's problematic to focus too much on command line arguments.  
> We are not introducing new command line arguments to qemu for the most 
> part that aren't usable in the config file.

Currently  libvirt doesn't use any config file at all, so command line
is the only possible option. We're going to evaluate switching to the
config file at some point & so can easily add further XML options to
allow direct setting of config file entries at that point. There's no
problem supporting both command line & config file syntax in the XML.
The command line does have the immediate advantage that 99% of all
documentation about QEMU on the web today refers to command line args
so that's what people are most familiar with.


> With respect to injecting QMP commands directly, I think the proposed 
> debug API is probably reasonable.  We could build a libqemu that used 
> that API as a transport which means that one could use libqemu and 
> libvirt simultaneously which is certainly a key requirement of mine.
> 
> I think it's important that it's a dedicated monitor session though.  It 
> shouldn't just be injecting commands within an existing QMP session IMHO.

I think the opposite actually. If libvirt had two open monitor connections,
one for normal use & one for injection, then its open to racey usage where
2 monitor commands may be issued concurrently & it is tricky to determine
just which will be processed first, with all the scope for unexpected 
behaviour this entails. With a single monitor connection, libvirt's current
locking model for the monitor ensures that QMP monitor commands are reliably 
serialized onto the wire, giving unambiguous behaviour. 

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/23/2010 05:28 AM, Daniel P. Berrange wrote:
> On Thu, Apr 22, 2010 at 01:45:27PM -0500, Anthony Liguori wrote:
>    
>> On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:
>>      
>>> On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
>>>
>>>
>>>        
>>>> <domain type='kvm'>
>>>>    <name>myguest</name>
>>>>    ...
>>>>    <debug>
>>>>      <monitorpassthrough/>
>>>>      <commandline>
>>>>        <extra>qemu arguments</extra>
>>>>        <alter option="optname">
>>>>          <rename>newname</rename>
>>>>          <match>REGEXP</match>
>>>>          <modify>foo=on</modify>
>>>>          <extra>-bar</extra>
>>>>        </alter>
>>>>      </commandline>
>>>>    </debug>
>>>> </domain>
>>>>
>>>>          
>>> The concept of command line&   monitor is something that is QEMU specific
>>> and thus is not suitable for the primary XML schema. IMHO, this needs to be
>>> done as a separate schema, linked in via an XML namespace. For example
>>>
>>>    <domain type='kvm'
>>>    xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
>>>      <name>myguest</name>
>>>      ...
>>>      <qemu:commandline>
>>>        <qemu:arg>-device</arg>
>>>        <qemu:arg>lsi</arg>
>>>      </qemu:commandline>
>>>    </domain>
>>>
>>>        
>> I think it's problematic to focus too much on command line arguments.
>> We are not introducing new command line arguments to qemu for the most
>> part that aren't usable in the config file.
>>      
> Currently  libvirt doesn't use any config file at all, so command line
> is the only possible option.

The config file is really just a structured command line argument so the 
two are really equivalent.

 From an XML perspective, the advantage of using the config file format 
is that it's very well structured whereas the command line arguments 
aren't.  It's the difference between:

<qemu:commandline>
<arg>-drive</arg>
<arg>file=foo.img,if=virtio,cache=off</arg>
</qemu:commandline>

And:

<qemu:drive>
<file>foo.img</file>
<if>virtio</if>
<cache>off</cache>
</qemu:drive>

The later being much more friendly for things like XPath and XQuery.

>   We're going to evaluate switching to the
> config file at some point&  so can easily add further XML options to
> allow direct setting of config file entries at that point.

I don't necessary think there's a super compelling reason for you to 
switch to pure config.  Some things you'll have to do via -device and 
-set but it's still reasonable to keep using the command line.  A key 
requirement of the config file effort was that everything that can be 
done via config file be doable via command line.

>> With respect to injecting QMP commands directly, I think the proposed
>> debug API is probably reasonable.  We could build a libqemu that used
>> that API as a transport which means that one could use libqemu and
>> libvirt simultaneously which is certainly a key requirement of mine.
>>
>> I think it's important that it's a dedicated monitor session though.  It
>> shouldn't just be injecting commands within an existing QMP session IMHO.
>>      
> I think the opposite actually. If libvirt had two open monitor connections,
> one for normal use&  one for injection, then its open to racey usage where
> 2 monitor commands may be issued concurrently&  it is tricky to determine
> just which will be processed first, with all the scope for unexpected
> behaviour this entails.

Can you give an explicit example?  The nature of this debug extension is 
such that I don't think 1 monitor really helps this problem...

>   With a single monitor connection, libvirt's current
> locking model for the monitor ensures that QMP monitor commands are reliably
> serialized onto the wire, giving unambiguous behaviour.
>    

Except that libvirt doesn't know what the side effects of the debug 
commands are so it's intrinsically ambiguous :-)

Regards,

Anthony Liguori

> Regards,
> Daniel
>    

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Daniel P. Berrange]

On Fri, Apr 23, 2010 at 08:40:49AM -0500, Anthony Liguori wrote:
> On 04/23/2010 05:28 AM, Daniel P. Berrange wrote:
> >On Thu, Apr 22, 2010 at 01:45:27PM -0500, Anthony Liguori wrote:
> >   
> >>On 04/09/2010 09:27 AM, Daniel P. Berrange wrote:
> >>     
> >>>On Fri, Apr 09, 2010 at 09:41:39AM -0400, Chris Lalancette wrote:
> >>>
> >>>
> >>>       
> >>>><domain type='kvm'>
> >>>>   <name>myguest</name>
> >>>>   ...
> >>>>   <debug>
> >>>>     <monitorpassthrough/>
> >>>>     <commandline>
> >>>>       <extra>qemu arguments</extra>
> >>>>       <alter option="optname">
> >>>>         <rename>newname</rename>
> >>>>         <match>REGEXP</match>
> >>>>         <modify>foo=on</modify>
> >>>>         <extra>-bar</extra>
> >>>>       </alter>
> >>>>     </commandline>
> >>>>   </debug>
> >>>></domain>
> >>>>
> >>>>         
> >>>The concept of command line&   monitor is something that is QEMU specific
> >>>and thus is not suitable for the primary XML schema. IMHO, this needs to 
> >>>be
> >>>done as a separate schema, linked in via an XML namespace. For example
> >>>
> >>>   <domain type='kvm'
> >>>   xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0">
> >>>     <name>myguest</name>
> >>>     ...
> >>>     <qemu:commandline>
> >>>       <qemu:arg>-device</arg>
> >>>       <qemu:arg>lsi</arg>
> >>>     </qemu:commandline>
> >>>   </domain>
> >>>
> >>>       
> >>I think it's problematic to focus too much on command line arguments.
> >>We are not introducing new command line arguments to qemu for the most
> >>part that aren't usable in the config file.
> >>     
> >Currently  libvirt doesn't use any config file at all, so command line
> >is the only possible option.
> 
> The config file is really just a structured command line argument so the 
> two are really equivalent.
> 
> From an XML perspective, the advantage of using the config file format 
> is that it's very well structured whereas the command line arguments 
> aren't.  It's the difference between:
> 
> <qemu:commandline>
> <arg>-drive</arg>
> <arg>file=foo.img,if=virtio,cache=off</arg>
> </qemu:commandline>
> 
> And:
> 
> <qemu:drive>
> <file>foo.img</file>
> <if>virtio</if>
> <cache>off</cache>
> </qemu:drive>
> 
> The later being much more friendly for things like XPath and XQuery.
> 
> >  We're going to evaluate switching to the
> >config file at some point&  so can easily add further XML options to
> >allow direct setting of config file entries at that point.
> 
> I don't necessary think there's a super compelling reason for you to 
> switch to pure config.  Some things you'll have to do via -device and 
> -set but it's still reasonable to keep using the command line.  A key 
> requirement of the config file effort was that everything that can be 
> done via config file be doable via command line.

The main reason  we're interested in the config is that it uses a better
structured format, so we don't have such hairy escaping problems to deal
with. Also, the command line is getting outrageously long these days
with -device syntax.

> >>With respect to injecting QMP commands directly, I think the proposed
> >>debug API is probably reasonable.  We could build a libqemu that used
> >>that API as a transport which means that one could use libqemu and
> >>libvirt simultaneously which is certainly a key requirement of mine.
> >>
> >>I think it's important that it's a dedicated monitor session though.  It
> >>shouldn't just be injecting commands within an existing QMP session IMHO.
> >>     
> >I think the opposite actually. If libvirt had two open monitor connections,
> >one for normal use&  one for injection, then its open to racey usage where
> >2 monitor commands may be issued concurrently&  it is tricky to determine
> >just which will be processed first, with all the scope for unexpected
> >behaviour this entails.
> 
> Can you give an explicit example?  The nature of this debug extension is 
> such that I don't think 1 monitor really helps this problem...

Say libvirt is running a 'offline core dump' operation. This consists of
us invoking

   stop
   migrate exec:cat > foo.dump
   cont

I don't want other debug commands accidentally being issued in between
these steps. These 3 commands are in essence considered transactional
block. Internally our locking model ensures that no other APIs can be
invoked in this monitor connection during this sequence. Thus if someone
uses the libvirt debug API to issue a monitor command, thus command will
be blocked until the last 'cont' command is issued here. This provides us
a known command ordering in the logs. 

> >  With a single monitor connection, libvirt's current
> >locking model for the monitor ensures that QMP monitor commands are 
> >reliably
> >serialized onto the wire, giving unambiguous behaviour.
> 
> Except that libvirt doesn't know what the side effects of the debug 
> commands are so it's intrinsically ambiguous :-)

This is a different ambiguity, about the semantic results of the commands,
where as I'm refering to the execution order. If I look at a libvirt log
file and see a set of JSON commands logged, I want to know that this ordering
from the logs, was indeed the same as order in which qemu processed them. If
you have two separate monitor connection you can't be sure of the order of 
execution. It is key for our bug troubleshooting that given a libvirt log 
file, we can replay the JSON commands again and get the same results. Two
monitor connections is just increasing complexity of code without any 
tangible benefit.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/23/2010 09:21 AM, Daniel P. Berrange wrote:
>
> Say libvirt is running a 'offline core dump' operation. This consists of
> us invoking
>
>     stop
>     migrate exec:cat>  foo.dump
>     cont
>
> I don't want other debug commands accidentally being issued in between
> these steps. These 3 commands are in essence considered transactional
> block. Internally our locking model ensures that no other APIs can be
> invoked in this monitor connection during this sequence. Thus if someone
> uses the libvirt debug API to issue a monitor command, thus command will
> be blocked until the last 'cont' command is issued here. This provides us
> a known command ordering in the logs.
>    

Ah, so what'd I'd suggest is that you still use two monitors but that 
you use the same locking for both.  The reason a second monitor is 
useful is that you can allow different event masks to be setup on the 
second monitor which I think a QMP client will expect.

I don't see any trouble with respect to command ordering here.  libvirt 
is ultimately the one sending commands to monitor.  You know which one 
was sent and which one completed in what order.

>    
>>>   With a single monitor connection, libvirt's current
>>> locking model for the monitor ensures that QMP monitor commands are
>>> reliably
>>> serialized onto the wire, giving unambiguous behaviour.
>>>        
>> Except that libvirt doesn't know what the side effects of the debug
>> commands are so it's intrinsically ambiguous :-)
>>      
> This is a different ambiguity, about the semantic results of the commands,
> where as I'm refering to the execution order. If I look at a libvirt log
> file and see a set of JSON commands logged, I want to know that this ordering
> from the logs, was indeed the same as order in which qemu processed them. If
> you have two separate monitor connection you can't be sure of the order of
> execution. It is key for our bug troubleshooting that given a libvirt log
> file, we can replay the JSON commands again and get the same results. Two
> monitor connections is just increasing complexity of code without any
> tangible benefit.
>    

I think you're assuming direct access to the second monitor?  I'm not 
suggesting that.  I'm suggesting that libvirt is still the one 
submitting commands to the second monitor and that it submits those 
commands in lock step.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Avi Kivity]

On 04/23/2010 09:33 PM, Anthony Liguori wrote:
>> This is a different ambiguity, about the semantic results of the 
>> commands,
>> where as I'm refering to the execution order. If I look at a libvirt log
>> file and see a set of JSON commands logged, I want to know that this 
>> ordering
>> from the logs, was indeed the same as order in which qemu processed 
>> them. If
>> you have two separate monitor connection you can't be sure of the 
>> order of
>> execution. It is key for our bug troubleshooting that given a libvirt 
>> log
>> file, we can replay the JSON commands again and get the same results. 
>> Two
>> monitor connections is just increasing complexity of code without any
>> tangible benefit.
>
> I think you're assuming direct access to the second monitor?  I'm not 
> suggesting that.  I'm suggesting that libvirt is still the one 
> submitting commands to the second monitor and that it submits those 
> commands in lock step.
>

What about protocol extensions?  For instance, pretend libvirt doesn't 
support async messages, what would it do when it receives one from the 
user's monitor?

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Libvirt debug API

[Anthony Liguori]

On 04/25/2010 09:50 AM, Avi Kivity wrote:
> On 04/23/2010 09:33 PM, Anthony Liguori wrote:
>>> This is a different ambiguity, about the semantic results of the 
>>> commands,
>>> where as I'm refering to the execution order. If I look at a libvirt 
>>> log
>>> file and see a set of JSON commands logged, I want to know that this 
>>> ordering
>>> from the logs, was indeed the same as order in which qemu processed 
>>> them. If
>>> you have two separate monitor connection you can't be sure of the 
>>> order of
>>> execution. It is key for our bug troubleshooting that given a 
>>> libvirt log
>>> file, we can replay the JSON commands again and get the same 
>>> results. Two
>>> monitor connections is just increasing complexity of code without any
>>> tangible benefit.
>>
>> I think you're assuming direct access to the second monitor?  I'm not 
>> suggesting that.  I'm suggesting that libvirt is still the one 
>> submitting commands to the second monitor and that it submits those 
>> commands in lock step.
>>
>
> What about protocol extensions?  For instance, pretend libvirt doesn't 
> support async messages, what would it do when it receives one from the 
> user's monitor?

Protocol extensions could not be supported in this model.  I agree, 
that's unfortunate.

Regards,

Anthony Liguori

[Qemu-devel] Re: [libvirt] Libvirt debug API

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 1346 bytes --]

On 04/09/2010 07:41 AM, Chris Lalancette wrote:
> Caveats:
> 3)  Application developers will be strongly discouraged from using
> this API because of the above 2 issues.  To help in this, the
> API's will be in a separate library that developers will explicitly
> have to link to, and it will have a different (but largely compatible)
> wire protocol.

Providing two side-by-side libraries under the libvirt package umbrella
makes sense to me, but it means we need to start thinking more seriously
about library version numbers.  See
https://www.redhat.com/archives/libvir-list/2010-April/msg00226.html for
some food for thought; in particular, the idea that since we create our
library (or libraries, after this proposal is enacted) via libtool, we
should be using the libtool versioning scheme for the .so files, which
would be independent from the libvirt package number.  It is entirely
feasible to have a release that increments the version number of one but
not both .so, or which increments them in different manners (libvirt
increasing current and age, because it added new APIs but is still
backwards compatible, while the helper library resets age to 0 because
it added a backwards incompatible change).

-- 
Eric Blake   eblake@redhat.com    +1-801-349-2682
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 323 bytes --]