[Qemu-devel] Supporting hypervisor specific APIs in libvirt

[Qemu-devel] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

Hi,

I've mentioned this to a few folks already but I wanted to start a 
proper thread.

We're struggling in qemu with usability and one area that concerns me is 
the disparity in features that are supported by qemu vs what's 
implemented in libvirt.

This isn't necessarily libvirt's problem if it's mission is to provide a 
common hypervisor API that covers the most commonly used features.

However, for qemu, we need an API that covers all of our features that 
people can develop against.  The ultimate question we need to figure out 
is, should we encourage our users to always use libvirt or should we 
build our own API for people (and libvirt) to consume.

I don't think it's necessarily a big technical challenge for libvirt to 
support qemu more completely.  I think it amounts to introducing a 
series of virQemuXXXX APIs that implement qemu specific functions.  Over 
time, qemu specific APIs can be deprecated in favour of more generic 
virDomain APIs.

What's the feeling about this from the libvirt side of things?  Is there 
interest in support hypervisor specific interfaces should we be looking 
to provide our own management interface for libvirt to consume?

Regards,

Anthony Liguori

[Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Mon, Mar 22, 2010 at 02:25:00PM -0500, Anthony Liguori wrote:
> Hi,
> 
> I've mentioned this to a few folks already but I wanted to start a 
> proper thread.
> 
> We're struggling in qemu with usability and one area that concerns me is 
> the disparity in features that are supported by qemu vs what's 
> implemented in libvirt.
> 
> This isn't necessarily libvirt's problem if it's mission is to provide a 
> common hypervisor API that covers the most commonly used features.

That is more or less our current mission. If this mission leads to QEMU 
creating a non-libvirt based API & telling people to use that instead, 
then I'd say libvirt's mission needs to change to avoid that scenario !
I strongly believe that libvirt's strategy is good for application 
developers over the medium to long term. We need to figure out how to
get rid of the short term pain from the feature timelag, rather than
inventing a new library API for them to use.
 
> However, for qemu, we need an API that covers all of our features that 
> people can develop against.  The ultimate question we need to figure out 
> is, should we encourage our users to always use libvirt or should we 
> build our own API for people (and libvirt) to consume.
> 
> I don't think it's necessarily a big technical challenge for libvirt to 
> support qemu more completely.  I think it amounts to introducing a 
> series of virQemuXXXX APIs that implement qemu specific functions.  Over 
> time, qemu specific APIs can be deprecated in favour of more generic 
> virDomain APIs.

Stepping back a bit first, there are the two core areas in which people can
be limited by libvirt currently.

 1. Monitor commands
 2. Command line flags

Ultimately, IIUC, you are suggesting we need to allow arbitrary passthrough
for both of these in libvirt.

At the libvirt level, we have 3 core requirements

 1. The XML format is extend only (new elements allowed, or add attributes
    or children to existing elements)
 2. The C library API is append only (new symbols only)
 3. The RPC wire protocol is append only (maps 1-1 to the C API generally)

The core question for us as libvirt developers is how we could support
QEMU specific features that may change arbitrarily, without it impacting
on our ability to maintain these 3 requirements for the non-hypervisor
specific APIs.

We don't ever want to be in a situation where a QEMU specific API will 
require us to change the soname of the main libvirt library, or introduce
incompatible wire protocol changes. If we were to introduce QEMU specific
APIs, we also need a way to easily remove those over time, as & when we 
have them available as generic APIs.

At the C API level, this to me suggests that we'd want to introduce a
separate libvirt-qemu.so  library for the QEMU specific APIs. This 
library would not have the same requirements of fixed long term ABI
that the main libvirt.so did. We'd add QEMU APIs to libvirt-qemu.so 
any time needed, but remove them when the equivalent functionality
were in libvirt.so, and increment the soname of libvirt-qemu.so at
that point.

At the wire protocol level, the protocol allows us to support multiple
versioned protocols in parallel over the same data stream. So again
there, we could define a sub-protocol for QEMU specific features for
which we don't provide the indefinite ABI compatability.

Finally the XML format is "easy" - just have a versioned XML namespace
for extra pieces, that's distinct from the default namespace, again
without the permanent long term compatability guarentees.

There are, however, some bits that are unlikely to work when QEMU
is under libvirt. Specifically any of the device backends that use
stdio (eg, -serial stdio, or the ncurses graphics), simply because
all libvirt spawned VMs are fully daemonized & so stdio is /dev/null

Other items are hard, but not entirely impossible to solve. eg, any 
use of the 'script=' arg for -net devices doesn't work, because libvirt
clears all capabilities from the QEMU process so it'll be lacking
CAP_NET_ADMIN which most TAP device setup scripts in fact need.

Some parts of the C library/wire protocol here are related to another 
feature I'd like to introduce for libvirt, namely a administrative 
library. eg a API to configure and manage the libvirtd daemon itself 
on the fly. This could easily hook into the wire protocol, but live 
as a separate libvirt-daemon.so library API in similar way to what I 
suggest for QEMU specific API

> What's the feeling about this from the libvirt side of things?  Is there 
> interest in support hypervisor specific interfaces should we be looking 
> to provide our own management interface for libvirt to consume?

Adding yet another library in the stack isn't really going to solve the 
problem from the POV of libvirt users, but rather fork the community
effort which I imagine we'd all rather avoid. Having to tell people to
switch to a different library API to get access to a specific feature 
is a short term win, but with a significant long term cost/burden. This 
means we really do need to figure out how to better/fully support QEMU's 
features in libvirt, removing the feature timelag pain.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Gerd Hoffmann]

   Hi,

> Stepping back a bit first, there are the two core areas in which people can
> be limited by libvirt currently.

>   2. Command line flags

For me:  This one, and monitor access.

libvirt is very unfriendly to qemu hackers.  There is no easy way to add 
command line switches.  There is no easy way to get access to the 
monitor.  I can get it done by pointing <emulator> to a wrapper script 
and mangle the qemu command line there.  But this sucks big time.  And 
it doesn't integrate with libvirt at all.

When hacking qemu, especially when adding new command line options or 
monitor commands, I want to have a easy way to test this stuff.  Or I 
just wanna able to type some 'info $foo' commands for debugging and 
trouble shooting purposes.  libvirt makes it harder not easier to get 
the job done.

Image you could ask libvirt to create an additional monitor and expose 
it like a serial console.  virt-manager lists it as text console.  Two 
mouse clicks open a new window (or tab) with a terminal emulator linked 
to the monitor.  Wouldn't that be cool?

Other issues I've trap into:

-boot
   libvirt (or virt-manager?) supports only the very old single letter
   style.  You can't specify '-boot order=cd,menu=on'.

-enable-nested
   not available.

serial console doesn't work for remote connections.

cheers,
   Gerd

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/22/2010 04:33 PM, Gerd Hoffmann wrote:
>   Hi,
>
>> Stepping back a bit first, there are the two core areas in which 
>> people can
>> be limited by libvirt currently.
>
>>   2. Command line flags
>
> For me:  This one, and monitor access.
>
> libvirt is very unfriendly to qemu hackers.  There is no easy way to 
> add command line switches.  There is no easy way to get access to the 
> monitor.  I can get it done by pointing <emulator> to a wrapper script 
> and mangle the qemu command line there.  But this sucks big time.  And 
> it doesn't integrate with libvirt at all.

It's not just developers.  As we're doing deployments of qemu/kvm, we 
keep running into the same problem.  We realize that we need to use a 
feature of qemu/kvm that isn't modelled by libvirt today.  I've gone as 
far as to temporarily pausing libvirtd, finding the pty fd from 
/proc/<pid>, and hijacking the monitor session temporarily.

The problem is, it's not always easy to know what the most important 
features are.

>
> When hacking qemu, especially when adding new command line options or 
> monitor commands, I want to have a easy way to test this stuff.  Or I 
> just wanna able to type some 'info $foo' commands for debugging and 
> trouble shooting purposes.  libvirt makes it harder not easier to get 
> the job done.
>
> Image you could ask libvirt to create an additional monitor and expose 
> it like a serial console.  virt-manager lists it as text console.  Two 
> mouse clicks open a new window (or tab) with a terminal emulator 
> linked to the monitor.  Wouldn't that be cool?
>
> Other issues I've trap into:
>
> -boot
>   libvirt (or virt-manager?) supports only the very old single letter
>   style.  You can't specify '-boot order=cd,menu=on'.

You can, you specify multiple <boot> options (but you can't touch things 
like menu=on).

Regards,

Anthony Liguori

> -enable-nested
>   not available.
>
> serial console doesn't work for remote connections.
>
> cheers,
>   Gerd

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Jes Sorensen]

On 03/22/10 22:53, Anthony Liguori wrote:
> On 03/22/2010 04:33 PM, Gerd Hoffmann wrote:
>> libvirt is very unfriendly to qemu hackers. There is no easy way to
>> add command line switches. There is no easy way to get access to the
>> monitor. I can get it done by pointing <emulator> to a wrapper script
>> and mangle the qemu command line there. But this sucks big time. And
>> it doesn't integrate with libvirt at all.
>
> It's not just developers. As we're doing deployments of qemu/kvm, we
> keep running into the same problem. We realize that we need to use a
> feature of qemu/kvm that isn't modelled by libvirt today. I've gone as
> far as to temporarily pausing libvirtd, finding the pty fd from
> /proc/<pid>, and hijacking the monitor session temporarily.

One problem I have found, and I am not sure how to fix this in this
context. Sometimes when hacking on qemu, I want to try out a new
qemu binary on an existing image, without replacing the system wide
one and may want to pass new command line flags for testing those, plus
have access to the monitor.

What I do now is to look at the command line arguments of a guest using
ps and try and mimic it, but due to the random magic ptys and other
stuff, it's practically impossible to replicate a libvirt spawned qemu
on the command line. I end up having a somewhat similar command line
with everything removed that I cannot replicate.

I find it a real problem that libvirt tries to wrap things to the point
that an ordinary human cannot read, modify it's configuration or do a
simple command line spawn to replicate it, but as I said, I am not sure
how to solve the problem.

Regards,
Jes

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Gerd Hoffmann]

On 03/23/10 09:54, Jes Sorensen wrote:
> On 03/22/10 22:53, Anthony Liguori wrote:
>> On 03/22/2010 04:33 PM, Gerd Hoffmann wrote:
>>> libvirt is very unfriendly to qemu hackers. There is no easy way to
>>> add command line switches. There is no easy way to get access to the
>>> monitor. I can get it done by pointing <emulator> to a wrapper script
>>> and mangle the qemu command line there. But this sucks big time. And
>>> it doesn't integrate with libvirt at all.
>>
>> It's not just developers. As we're doing deployments of qemu/kvm, we
>> keep running into the same problem. We realize that we need to use a
>> feature of qemu/kvm that isn't modelled by libvirt today. I've gone as
>> far as to temporarily pausing libvirtd, finding the pty fd from
>> /proc/<pid>, and hijacking the monitor session temporarily.
>
> One problem I have found, and I am not sure how to fix this in this
> context. Sometimes when hacking on qemu, I want to try out a new
> qemu binary on an existing image, without replacing the system wide
> one and may want to pass new command line flags for testing those, plus
> have access to the monitor.

Works with the wrapper script trick mentioned above.

virsh edit $domain
grep for <emulator>
make it point to a wrapper script.

My setup:

[root@xenb ~]# virsh dumpxml fedora | grep emulator
     <emulator>/root/bin/qemu-wrapper</emulator>
[root@xenb ~]# cat /root/bin/qemu-wrapper
#!/bin/sh

# distro qemu-kvm
REAL_QEMU="/usr/bin/qemu-kvm"
MORE_ARGS="-boot menu=on -cpu host -enable-nesting"

# fresh build
#REAL_QEMU="/home/kraxel/git/kvm/x86_64-softmmu/qemu-system-x86_64"
#MORE_ARGS="-L /home/kraxel/git/kvm/pc-bios -boot menu=on"

# go!
case "$1" in
         "" | -h | -help | --help)
                 # libvirt capability check
                 exec $REAL_QEMU $1
                 ;;
         *)
                 # run qemu with additional args
                 exec $REAL_QEMU "$@" $MORE_ARGS
                 ;;
esac

# should never ever arrive here
echo "$0: exec $REAL_QEMU failed" >&2
exit 1

HTH,
   Gerd

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Jes Sorensen]

On 03/23/10 11:25, Gerd Hoffmann wrote:
> On 03/23/10 09:54, Jes Sorensen wrote:
>> One problem I have found, and I am not sure how to fix this in this
>> context. Sometimes when hacking on qemu, I want to try out a new
>> qemu binary on an existing image, without replacing the system wide
>> one and may want to pass new command line flags for testing those, plus
>> have access to the monitor.
>
> Works with the wrapper script trick mentioned above.
>
> virsh edit $domain
> grep for <emulator>
> make it point to a wrapper script.
>
> My setup:
>
> [root@xenb ~]# virsh dumpxml fedora | grep emulator
> <emulator>/root/bin/qemu-wrapper</emulator>
> [root@xenb ~]# cat /root/bin/qemu-wrapper

Ah right thanks! However, it's a hack to get around the real problem
with libvirt. Not to mention that the output from virsh dumpxml is
where you have to cover your eyes and try not getting sick while
editing :(

Having a normal config file in readable format where you could add
regular command line options manually would make life so much easier.

Cheers,
Jes

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Gerd Hoffmann]

On 03/23/10 11:31, Jes Sorensen wrote:
> On 03/23/10 11:25, Gerd Hoffmann wrote:
>> [root@xenb ~]# virsh dumpxml fedora | grep emulator
>> <emulator>/root/bin/qemu-wrapper</emulator>
>> [root@xenb ~]# cat /root/bin/qemu-wrapper
>
> Ah right thanks! However, it's a hack to get around the real problem
> with libvirt.

Sure.  It is a hack, not a real solution.  And it easily gets much more 
messy, for example in case you wanna have an additional monitor as 
unix:/var/tmp/mon/$vmname.

cheers,
   Gerd

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Cole Robinson]

On 03/22/2010 05:33 PM, Gerd Hoffmann wrote:
>   Hi,
> 
>> Stepping back a bit first, there are the two core areas in which
>> people can
>> be limited by libvirt currently.
> 
>>   2. Command line flags
> 
> For me:  This one, and monitor access.
> 
> libvirt is very unfriendly to qemu hackers.  There is no easy way to add
> command line switches.  There is no easy way to get access to the
> monitor.  I can get it done by pointing <emulator> to a wrapper script
> and mangle the qemu command line there.  But this sucks big time.  And
> it doesn't integrate with libvirt at all.
> 
> When hacking qemu, especially when adding new command line options or
> monitor commands, I want to have a easy way to test this stuff.  Or I
> just wanna able to type some 'info $foo' commands for debugging and
> trouble shooting purposes.  libvirt makes it harder not easier to get
> the job done.
> 
> Image you could ask libvirt to create an additional monitor and expose
> it like a serial console.  virt-manager lists it as text console.  Two
> mouse clicks open a new window (or tab) with a terminal emulator linked
> to the monitor.  Wouldn't that be cool?
> 
> Other issues I've trap into:
> 
> -boot
>   libvirt (or virt-manager?) supports only the very old single letter
>   style.  You can't specify '-boot order=cd,menu=on'.
> 

Libvirt has supported multiple boot options for a while, it just wasn't in
virt-manager. It's been upstream for a few weeks now though, and a new release
is coming in a matter of days.

I have a half implemented libvirt patch to allow setting boot menu, I guess
it's time to dust it off :)

> -enable-nested
>   not available.
> 
> serial console doesn't work for remote connections.
> 

Both of these have been requested a few times, so you aren't alone.

- Cole

[Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/22/2010 03:10 PM, Daniel P. Berrange wrote:
>> This isn't necessarily libvirt's problem if it's mission is to provide a
>> common hypervisor API that covers the most commonly used features.
>>      
> That is more or less our current mission. If this mission leads to QEMU
> creating a non-libvirt based API&  telling people to use that instead,
> then I'd say libvirt's mission needs to change to avoid that scenario !
> I strongly believe that libvirt's strategy is good for application
> developers over the medium to long term. We need to figure out how to
> get rid of the short term pain from the feature timelag, rather than
> inventing a new library API for them to use.
>    

Well that's certainly a good thing :-)

>> However, for qemu, we need an API that covers all of our features that
>> people can develop against.  The ultimate question we need to figure out
>> is, should we encourage our users to always use libvirt or should we
>> build our own API for people (and libvirt) to consume.
>>
>> I don't think it's necessarily a big technical challenge for libvirt to
>> support qemu more completely.  I think it amounts to introducing a
>> series of virQemuXXXX APIs that implement qemu specific functions.  Over
>> time, qemu specific APIs can be deprecated in favour of more generic
>> virDomain APIs.
>>      
> Stepping back a bit first, there are the two core areas in which people can
> be limited by libvirt currently.
>
>   1. Monitor commands
>   2. Command line flags
>
> Ultimately, IIUC, you are suggesting we need to allow arbitrary passthrough
> for both of these in libvirt.
>
> At the libvirt level, we have 3 core requirements
>
>   1. The XML format is extend only (new elements allowed, or add attributes
>      or children to existing elements)
>   2. The C library API is append only (new symbols only)
>   3. The RPC wire protocol is append only (maps 1-1 to the C API generally)
>    

We have a slightly different mentality within QEMU I think.  Here's 
roughly how I'd characterize our guarantees.

1. For any two versions of QEMU, we try to guarantee that the same VM, 
as far as the guest sees it, can be created.
2. We tend to avoid changing command line syntax unless the syntax was 
previously undefined.
3. QMP supports enumeration and feature negotiation.  This enables a 
client to discover which functions are supported.
4. We try to maintain monitor interfaces but provide no guarantees of 
compatibility.

> The core question for us as libvirt developers is how we could support
> QEMU specific features that may change arbitrarily, without it impacting
> on our ability to maintain these 3 requirements for the non-hypervisor
> specific APIs.
>
> We don't ever want to be in a situation where a QEMU specific API will
> require us to change the soname of the main libvirt library, or introduce
> incompatible wire protocol changes. If we were to introduce QEMU specific
> APIs, we also need a way to easily remove those over time, as&  when we
> have them available as generic APIs.
>
> At the C API level, this to me suggests that we'd want to introduce a
> separate libvirt-qemu.so  library for the QEMU specific APIs. This
> library would not have the same requirements of fixed long term ABI
> that the main libvirt.so did. We'd add QEMU APIs to libvirt-qemu.so
> any time needed, but remove them when the equivalent functionality
> were in libvirt.so, and increment the soname of libvirt-qemu.so at
> that point.
>    

How different is having a libvirt-qemu.so from having a libqemu.so that 
libvirt.so uses?

Practically speaking, if libvirt-qemu.so uses a separate XML namespace, 
does the fact that we use a different config format matter since you can 
transform our config format to XML and vice versa?

I think the problem is, if libvirt.so introduces a common API, removing 
it from libvirt-qemu.so is burdensome to an end-user.  For someone 
designing a QEMU specific management application, why should they have 
to update their implementation to a common API that they'll never use?

> At the wire protocol level, the protocol allows us to support multiple
> versioned protocols in parallel over the same data stream. So again
> there, we could define a sub-protocol for QEMU specific features for
> which we don't provide the indefinite ABI compatability.
>
> Finally the XML format is "easy" - just have a versioned XML namespace
> for extra pieces, that's distinct from the default namespace, again
> without the permanent long term compatability guarentees.
>
> There are, however, some bits that are unlikely to work when QEMU
> is under libvirt. Specifically any of the device backends that use
> stdio (eg, -serial stdio, or the ncurses graphics), simply because
> all libvirt spawned VMs are fully daemonized&  so stdio is /dev/null
>    

This seems fixable with //session.

> Other items are hard, but not entirely impossible to solve. eg, any
> use of the 'script=' arg for -net devices doesn't work, because libvirt
> clears all capabilities from the QEMU process so it'll be lacking
> CAP_NET_ADMIN which most TAP device setup scripts in fact need.
>
> Some parts of the C library/wire protocol here are related to another
> feature I'd like to introduce for libvirt, namely a administrative
> library. eg a API to configure and manage the libvirtd daemon itself
> on the fly. This could easily hook into the wire protocol, but live
> as a separate libvirt-daemon.so library API in similar way to what I
> suggest for QEMU specific API
>
>    
>> What's the feeling about this from the libvirt side of things?  Is there
>> interest in support hypervisor specific interfaces should we be looking
>> to provide our own management interface for libvirt to consume?
>>      
> Adding yet another library in the stack isn't really going to solve the
> problem from the POV of libvirt users, but rather fork the community
> effort which I imagine we'd all rather avoid. Having to tell people to
> switch to a different library API to get access to a specific feature
> is a short term win, but with a significant long term cost/burden. This
> means we really do need to figure out how to better/fully support QEMU's
> features in libvirt, removing the feature timelag pain.
>    

I think what we need to do is find a way to more tightly integrate the 
QEMU and libvirt communities in such a way that when a patch was 
submitted against QEMU adding a new feature, we could ask that that 
feature was implemented in libvirt.  I see two ways to do this.

One would be for libvirt to have a libvirt.so and libvirt-qemu.so.  The 
QEMU community would have to be much more heavily involved in 
libvirt-qemu.so and it probably suggests that libvirt-qemu.so should 
follow our release cycle.  libvirt would have to support using either 
libvirt.so or libvirt-qemu.so for it's users.

The alternative would be for the QEMU community to produce a libqemu.so 
and for libvirt.so to consume libqemu.so.  The libvirt community ought 
to be heavily engaged in the development of libqemu.so and certainly, 
shared maintainership would be appropriate.  A user using libvirt.so 
should see guests created with either libqemu.so or libvirt.so although 
libqemu.so would provide weaker long term compatibility guarantees (but 
more features).

I think both approaches are technically workable.  The question is which 
is the most appealing to both communities and which requires the least 
development effort.

At a high level, I think libqemu.so would require a heavy refactoring of 
the qemu libvirt driver.  Guest enumeration would have to be handled by 
qemu and libvirt would need to support translating qemu's config format 
to an XML format.  It would also need to translate the qemu XML format 
into it's own XML format relying on an xmlns to support the features 
that are not currently supported by the common hypervisor API.

The advantage though would be that any qemu instance launched would be 
visible to libvirt and libvirt tooling.

libvirt-qemu.so would require less change to the qemu driver in libvirt 
but would require some awkwardness in terms of translating a guest 
config to an XML config.  Some of the concepts that qemu supports (like 
running a guest as non-privileged user using stdio) would be difficult 
to support.

Regards,

Anthony Liguori

Regards,

Anthony Liguori

> Regards,
> Daniel
>    

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Alexander Graf]

On 22.03.2010, at 22:49, Anthony Liguori wrote:

> On 03/22/2010 03:10 PM, Daniel P. Berrange wrote:
>> 
>>   
>>> What's the feeling about this from the libvirt side of things?  Is there
>>> interest in support hypervisor specific interfaces should we be looking
>>> to provide our own management interface for libvirt to consume?
>>>     
>> Adding yet another library in the stack isn't really going to solve the
>> problem from the POV of libvirt users, but rather fork the community
>> effort which I imagine we'd all rather avoid. Having to tell people to
>> switch to a different library API to get access to a specific feature
>> is a short term win, but with a significant long term cost/burden. This
>> means we really do need to figure out how to better/fully support QEMU's
>> features in libvirt, removing the feature timelag pain.
>>   
> 
> I think what we need to do is find a way to more tightly integrate the QEMU and libvirt communities in such a way that when a patch was submitted against QEMU adding a new feature, we could ask that that feature was implemented in libvirt.  I see two ways to do this.
> 
> One would be for libvirt to have a libvirt.so and libvirt-qemu.so.  The QEMU community would have to be much more heavily involved in libvirt-qemu.so and it probably suggests that libvirt-qemu.so should follow our release cycle.  libvirt would have to support using either libvirt.so or libvirt-qemu.so for it's users.
> 
> The alternative would be for the QEMU community to produce a libqemu.so and for libvirt.so to consume libqemu.so.  The libvirt community ought to be heavily engaged in the development of libqemu.so and certainly, shared maintainership would be appropriate.  A user using libvirt.so should see guests created with either libqemu.so or libvirt.so although libqemu.so would provide weaker long term compatibility guarantees (but more features).

I don't see why we shouldn't be able to automatically generate libqemu.so. We have the *.hx files that describe the syntax of parameters plus list all available options / commands. I'm not sure how exactly QMP works, but having a generic QMP command to list all available options would be handy too.

By then we could automate most of the library, making the glueing really easy. If libvirt doesn't properly link against libqemu anymore we also know why: The ABI changed.

All that's needed then is a common Qemu object that libvirt users can get access to as well. That object is the magic key to all libqemu functions. If users need functionality not exposed in libvirt, they can then use libqemu calls directly. If they don't care about all the awesomeness of libvirt and don't want to be hypervisor agnostic, they can stick to libqemu completely.


Alex

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Jamie Lokier]

Alexander Graf wrote:
> I don't see why we shouldn't be able to automatically generate
> libqemu.so. We have the *.hx files that describe the syntax of
> parameters plus list all available options / commands. I'm not sure
> how exactly QMP works, but having a generic QMP command to list all
> available options would be handy too.
> 
> By then we could automate most of the library, making the glueing
> really easy. If libvirt doesn't properly link against libqemu anymore
> we also know why: The ABI changed.

I'm thinking most potential uses of the binary API, other than C
programmers, would be happier with a D-Bus version generated from
those same *.hx files.  Because then it's easy to call the API from
Python, Perl, even shell, etc.  Whereas libqemu.so would be relatively
difficult to use from those languages.

(Aside: I don't particularly like D-Bus.  But it does seem to work
for this sort of thing.)

-- Jamie

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/23/2010 06:25 PM, Jamie Lokier wrote:
> Alexander Graf wrote:
>    
>> I don't see why we shouldn't be able to automatically generate
>> libqemu.so. We have the *.hx files that describe the syntax of
>> parameters plus list all available options / commands. I'm not sure
>> how exactly QMP works, but having a generic QMP command to list all
>> available options would be handy too.
>>
>> By then we could automate most of the library, making the glueing
>> really easy. If libvirt doesn't properly link against libqemu anymore
>> we also know why: The ABI changed.
>>      
> I'm thinking most potential uses of the binary API, other than C
> programmers, would be happier with a D-Bus version generated from
> those same *.hx files.  Because then it's easy to call the API from
> Python, Perl, even shell, etc.  Whereas libqemu.so would be relatively
> difficult to use from those languages.
>    

My thinking with respect to libqemu.so is that it should be mostly 
autogenerated.

QMP supports introspection, we should be able to generate an idl 
description of QMP via introspection and then build all of the function 
stubs from that idl.  Then there is no opportunity for libqemu to be out 
of date.

All we really need to write for libqemu is some core bits to deal with 
transport specific issues.

> (Aside: I don't particularly like D-Bus.  But it does seem to work
> for this sort of thing.)
>    

I don't think d-bus is a good fit as a core qemu service.  It's not 
commonly used on other platform and it introduces quite a bit of 
overhead for non-Unix platforms.

But that certainly doesn't mean that a d-bus service implemented on top 
of libqemu (even autogenerated from our IDL) would be a bad project.

Regards,

Anthony Liguori

> -- Jamie
>    

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Markus Armbruster]

Anthony Liguori <anthony@codemonkey.ws> writes:

> On 03/23/2010 06:25 PM, Jamie Lokier wrote:
>> Alexander Graf wrote:
>>    
>>> I don't see why we shouldn't be able to automatically generate
>>> libqemu.so. We have the *.hx files that describe the syntax of
>>> parameters plus list all available options / commands. I'm not sure
>>> how exactly QMP works, but having a generic QMP command to list all
>>> available options would be handy too.

Yes, the plan is to have QMP describe itself.  Needs work.

>>> By then we could automate most of the library, making the glueing
>>> really easy. If libvirt doesn't properly link against libqemu anymore
>>> we also know why: The ABI changed.

I can't quite see what such a libqemu would buy us compared to straight
QMP.

Talking QMP should be easy, provided you got a suitable JSON library.

Generating a libqemu.so for (a particular version of) QMP may make
talking (that version of) QMP slightly easier in C, by turning a simple
text network protocol into a C API.  But it's not without drawbacks.

The text protocol is designed to be evolvable in backward-compatible
ways.  For instance, we can new add commands, new optional arguments,
and so forth.  But you can't add new optional arguments to C functions
without changing the API.  You can change the function and bump the
soname, or you can deprecate the function and add a new one, or you can
bypass static typing, e.g. by passing arguments in a dictionary.  In the
latter case, why not put the command in the dictionary as well, and cut
the number of functions from N to 1.

Ensured consistency of libqmp and QEMU sounds nice.  But it's consistent
with a *local* version of QEMU.  QMP is a *network* protocol.  If your
app talks QMP straight, it can handle any remote version it knows all by
itself.  If you interpose libqmp, you add a dependency: you need a
sufficiently current *local* QEMU/libqmp.

>> I'm thinking most potential uses of the binary API, other than C
>> programmers, would be happier with a D-Bus version generated from
>> those same *.hx files.  Because then it's easy to call the API from
>> Python, Perl, even shell, etc.  Whereas libqemu.so would be relatively
>> difficult to use from those languages.

I suspect importing a foreign libqmp.so C API into a non-C language is
no easier than using the language's JSON facilities and talk QMP
straight, and less flexible.

> My thinking with respect to libqemu.so is that it should be mostly
> autogenerated.
>
> QMP supports introspection, we should be able to generate an idl
> description of QMP via introspection and then build all of the
> function stubs from that idl.  Then there is no opportunity for
> libqemu to be out of date.
>
> All we really need to write for libqemu is some core bits to deal with
> transport specific issues.

I can't quite see the utility of that.

[D-Bus snipped...]

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Paul Brook]

> I can't quite see what such a libqemu would buy us compared to straight
> QMP.
> 
> Talking QMP should be easy, provided you got a suitable JSON library.

I agree. My undesranding is this was one of the large motivations behind using 
JSON: It's a common protocol that already has convenient bindings in most 
languages.  If it's hard[1] for third parties to bind QMP to their favourite 
language/framework then IMHO we've done it wrong.

Paul

[1] Hard compared to any other sane RPC mechanism. Some languages make 
everything hard :-)

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/24/2010 07:25 AM, Paul Brook wrote:
>> I can't quite see what such a libqemu would buy us compared to straight
>> QMP.
>>
>> Talking QMP should be easy, provided you got a suitable JSON library.
>>      
> I agree. My undesranding is this was one of the large motivations behind using
> JSON: It's a common protocol that already has convenient bindings in most
> languages.  If it's hard[1] for third parties to bind QMP to their favourite
> language/framework then IMHO we've done it wrong.
>    

You can't have convenient bindings to an RPC in C because it doesn't 
support dynamic dispatch.  With most types of RPC, you have an IDL 
description and a code generator.

But regardless of that, there are advantages to us providing a libqemu.  
The biggest one is that we can standardize transport implementations 
that include discovery mechanisms.

If the core of libqemu provided an extensible transport interface, and a 
generic QMP request/completion mechanism, in a Python binding, you would 
never use the IDL generated wrappers but instead use dynamic dispatch to 
invoke arbitrary QMP requests.

But the advantage is that if libvirt provided an API for a QMP transport 
encapsulated in their secure protocol, then provided the plumbed that 
API through their Python interface, you could use it for free in Python 
without having to reinvent the wheel.

Regards,

Anthony Liguori

> Paul
>
> [1] Hard compared to any other sane RPC mechanism. Some languages make
> everything hard :-)
>    

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Jamie Lokier]

Anthony Liguori wrote:
> But the advantage is that if libvirt provided an API for a QMP transport 
> encapsulated in their secure protocol, then provided the plumbed that 
> API through their Python interface, you could use it for free in Python 
> without having to reinvent the wheel.

It's not free if the only "free" way to access all qemu's capabilities
from Python requires you to switch all your config files to libvirt's
format and libvirt's way of doing things. There's quite a big jump
from the qemu/kvm way of doing things and the libvirt way, and the
latter isn't well matched to all uses of qemu.

But if libvirt exposes the same QMP as direct to qemu, or something
very similar (it could wrap it, and add it's own libvirt events,
commands and properties), that would be great for scripts that could
then work with either with minimal change.

-- Jamie

[Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Mon, Mar 22, 2010 at 04:49:21PM -0500, Anthony Liguori wrote:
> On 03/22/2010 03:10 PM, Daniel P. Berrange wrote:
> >>This isn't necessarily libvirt's problem if it's mission is to provide a
> >>common hypervisor API that covers the most commonly used features.
> >>     
> >That is more or less our current mission. If this mission leads to QEMU
> >creating a non-libvirt based API&  telling people to use that instead,
> >then I'd say libvirt's mission needs to change to avoid that scenario !
> >I strongly believe that libvirt's strategy is good for application
> >developers over the medium to long term. We need to figure out how to
> >get rid of the short term pain from the feature timelag, rather than
> >inventing a new library API for them to use.
> >   
> 
> Well that's certainly a good thing :-)
> 
> >>However, for qemu, we need an API that covers all of our features that
> >>people can develop against.  The ultimate question we need to figure out
> >>is, should we encourage our users to always use libvirt or should we
> >>build our own API for people (and libvirt) to consume.
> >>
> >>I don't think it's necessarily a big technical challenge for libvirt to
> >>support qemu more completely.  I think it amounts to introducing a
> >>series of virQemuXXXX APIs that implement qemu specific functions.  Over
> >>time, qemu specific APIs can be deprecated in favour of more generic
> >>virDomain APIs.
> >>     
> >Stepping back a bit first, there are the two core areas in which people can
> >be limited by libvirt currently.
> >
> >  1. Monitor commands
> >  2. Command line flags
> >
> >Ultimately, IIUC, you are suggesting we need to allow arbitrary passthrough
> >for both of these in libvirt.
> >
> >At the libvirt level, we have 3 core requirements
> >
> >  1. The XML format is extend only (new elements allowed, or add attributes
> >     or children to existing elements)
> >  2. The C library API is append only (new symbols only)
> >  3. The RPC wire protocol is append only (maps 1-1 to the C API generally)
> >   
> 
> We have a slightly different mentality within QEMU I think.  Here's 
> roughly how I'd characterize our guarantees.
> 
> 1. For any two versions of QEMU, we try to guarantee that the same VM, 
> as far as the guest sees it, can be created.
> 2. We tend to avoid changing command line syntax unless the syntax was 
> previously undefined.
> 3. QMP supports enumeration and feature negotiation.  This enables a 
> client to discover which functions are supported.
> 4. We try to maintain monitor interfaces but provide no guarantees of 
> compatibility.
> 
> >The core question for us as libvirt developers is how we could support
> >QEMU specific features that may change arbitrarily, without it impacting
> >on our ability to maintain these 3 requirements for the non-hypervisor
> >specific APIs.
> >
> >We don't ever want to be in a situation where a QEMU specific API will
> >require us to change the soname of the main libvirt library, or introduce
> >incompatible wire protocol changes. If we were to introduce QEMU specific
> >APIs, we also need a way to easily remove those over time, as&  when we
> >have them available as generic APIs.
> >
> >At the C API level, this to me suggests that we'd want to introduce a
> >separate libvirt-qemu.so  library for the QEMU specific APIs. This
> >library would not have the same requirements of fixed long term ABI
> >that the main libvirt.so did. We'd add QEMU APIs to libvirt-qemu.so
> >any time needed, but remove them when the equivalent functionality
> >were in libvirt.so, and increment the soname of libvirt-qemu.so at
> >that point.
> >   
> 
> How different is having a libvirt-qemu.so from having a libqemu.so that 
> libvirt.so uses?
> 
> Practically speaking, if libvirt-qemu.so uses a separate XML namespace, 
> does the fact that we use a different config format matter since you can 
> transform our config format to XML and vice versa?

If an application used libqemu.so directly, they would be excluding themselves
from much of the libvirt ecosystem which would otherwise be beneficial to their
needs. For example, with libvirt-qemu.so, you could still use libvirt's secure
remote API access, all the authentication & authorization capabilities on the
API. It could also be intergrated into the other libvirt language bindings, and
mapping layers such as libvirt-CIM, libvirt-qpid, etc. Applications may still
also want the benefit of the libvirt hypervisor-agnostic APIs, for example, 
virt-manager wants to use libvirt.so primary so it can support QEMU, Xen, VMWare
etc, but it might also want to access qemu specific features via libvirt-qemu.so
It could not use libqemu.so because it would not be accessible via the libvirt
remote RPC layer.

> I think the problem is, if libvirt.so introduces a common API, removing 
> it from libvirt-qemu.so is burdensome to an end-user.  For someone 
> designing a QEMU specific management application, why should they have 
> to update their implementation to a common API that they'll never use?

I think we'd have to have a formal deprecation cycle, to allow overlap
between adding a feature to libvirt.so & removing it from libvirt-qemu.so
Even if they are building a QEMU specific mgmt application, there is still
value in switching to a libvirt.so based API, because with that we are in
a stronger position to guarentee their app will continue working with future
QEMU that may change its command line or monitor syntax. 

For example, consider if a libvirt already had support for passing qemu
specific args through to QEMU, and that app were using '-net' for some
host style networking option libvirt.so didn't initially support. Now 
QEMU introduced '-netdev' as an improved option for networking. If the
QEMU specific mgmt app had switched to the generic libvirt API once it
was available, they would have got the benefit of the new improved -netdev
option automatically. This is even more compelling in cases where QEMU has
actually removed the existing argument.

> >>What's the feeling about this from the libvirt side of things?  Is there
> >>interest in support hypervisor specific interfaces should we be looking
> >>to provide our own management interface for libvirt to consume?
> >>     
> >Adding yet another library in the stack isn't really going to solve the
> >problem from the POV of libvirt users, but rather fork the community
> >effort which I imagine we'd all rather avoid. Having to tell people to
> >switch to a different library API to get access to a specific feature
> >is a short term win, but with a significant long term cost/burden. This
> >means we really do need to figure out how to better/fully support QEMU's
> >features in libvirt, removing the feature timelag pain.
> >   
> 
> I think what we need to do is find a way to more tightly integrate the 
> QEMU and libvirt communities in such a way that when a patch was 
> submitted against QEMU adding a new feature, we could ask that that 
> feature was implemented in libvirt.  I see two ways to do this.
> 
> One would be for libvirt to have a libvirt.so and libvirt-qemu.so.  The 
> QEMU community would have to be much more heavily involved in 
> libvirt-qemu.so and it probably suggests that libvirt-qemu.so should 
> follow our release cycle.  libvirt would have to support using either 
> libvirt.so or libvirt-qemu.so for it's users.

I'm not anticipating an either/or scenario for this. Apps would primarily
still use libvirt.so for everything. They would only use libvirt-qemu.so
to access indiviaul QEMU specific APIs they might need. In other words, it
augments the functionality of the base libvirt library.

> The alternative would be for the QEMU community to produce a libqemu.so 
> and for libvirt.so to consume libqemu.so.  The libvirt community ought 
> to be heavily engaged in the development of libqemu.so and certainly, 
> shared maintainership would be appropriate.  A user using libvirt.so 
> should see guests created with either libqemu.so or libvirt.so although 
> libqemu.so would provide weaker long term compatibility guarantees (but 
> more features).
> 
> I think both approaches are technically workable.  The question is which 
> is the most appealing to both communities and which requires the least 
> development effort.

I don't neccessarily think that the approaches are mutually exclusive
either, because in some ways they are addressing different problem 
spaces. Even if there was a libqemu.so that libvirt used, there would
still be demand for a libvirt-qemu.so, because apps using libvirt for
mgmt do not want to discard use of the entire libvirt API just to access
1 extra QEMU specific feature. A libqemu.so doesn't solve that problem,
its just an invisible internal impl detail for libvirt as far as end
users are concerned.

> At a high level, I think libqemu.so would require a heavy refactoring of 
> the qemu libvirt driver.  Guest enumeration would have to be handled by 
> qemu and libvirt would need to support translating qemu's config format 
> to an XML format.  It would also need to translate the qemu XML format 
> into it's own XML format relying on an xmlns to support the features 
> that are not currently supported by the common hypervisor API.

Both of those are minor implementation details ;-)

> The advantage though would be that any qemu instance launched would be 
> visible to libvirt and libvirt tooling.
> 
> libvirt-qemu.so would require less change to the qemu driver in libvirt 
> but would require some awkwardness in terms of translating a guest 
> config to an XML config.  Some of the concepts that qemu supports (like 
> running a guest as non-privileged user using stdio) would be difficult 
> to support.

Translating from libvirt XML to a QEMU specific config format is really
not a burden. This is a core task for any libvirt hypervisor driver. It
just happens that the current "config format" is a list of ARGV. Within
reason, any other config format is just as managable. It only becomes
hard when a hypervisor's config format arbitrarily changes semantics with
each new release :-(

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Mon, Mar 22, 2010 at 04:49:21PM -0500, Anthony Liguori wrote:
> On 03/22/2010 03:10 PM, Daniel P. Berrange wrote:
> >>This isn't necessarily libvirt's problem if it's mission is to provide a
> >>common hypervisor API that covers the most commonly used features.
> >>     
> >That is more or less our current mission. If this mission leads to QEMU
> >creating a non-libvirt based API&  telling people to use that instead,
> >then I'd say libvirt's mission needs to change to avoid that scenario !
> >I strongly believe that libvirt's strategy is good for application
> >developers over the medium to long term. We need to figure out how to
> >get rid of the short term pain from the feature timelag, rather than
> >inventing a new library API for them to use.
> >   
> 
> Well that's certainly a good thing :-)
> 
> >>However, for qemu, we need an API that covers all of our features that
> >>people can develop against.  The ultimate question we need to figure out
> >>is, should we encourage our users to always use libvirt or should we
> >>build our own API for people (and libvirt) to consume.
> >>
> >>I don't think it's necessarily a big technical challenge for libvirt to
> >>support qemu more completely.  I think it amounts to introducing a
> >>series of virQemuXXXX APIs that implement qemu specific functions.  Over
> >>time, qemu specific APIs can be deprecated in favour of more generic
> >>virDomain APIs.
> >>     
> >Stepping back a bit first, there are the two core areas in which people can
> >be limited by libvirt currently.
> >
> >  1. Monitor commands
> >  2. Command line flags
> >
> >Ultimately, IIUC, you are suggesting we need to allow arbitrary passthrough
> >for both of these in libvirt.
> >
> >At the libvirt level, we have 3 core requirements
> >
> >  1. The XML format is extend only (new elements allowed, or add attributes
> >     or children to existing elements)
> >  2. The C library API is append only (new symbols only)
> >  3. The RPC wire protocol is append only (maps 1-1 to the C API generally)
> >   
> 
> We have a slightly different mentality within QEMU I think.  Here's 
> roughly how I'd characterize our guarantees.
> 
> 1. For any two versions of QEMU, we try to guarantee that the same VM, 
> as far as the guest sees it, can be created.
> 2. We tend to avoid changing command line syntax unless the syntax was 
> previously undefined.
> 3. QMP supports enumeration and feature negotiation.  This enables a 
> client to discover which functions are supported.
> 4. We try to maintain monitor interfaces but provide no guarantees of 
> compatibility.

Points 2 & 4 make it very hard for libvirt to use any library API
that QEMU might expose. We need to support multiple concurrently
running versions of QEMU on a host, to cope with the package upgrade
scenario & adhoc testing of new versions. If a libqemu.so for talking 
to QEMU changed a monitor interface & didn't have backwards compatability
for older QEMU version, then it is not something we could use, because 
any particular libvirt build would be tied to only being able to talk 
to the specific QEMU version. Currently we internally deal with changes 
in syntax detecting which format/protocol we need to use at runtime and
need to maintain that ability.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Mon, Mar 22, 2010 at 02:25:00PM -0500, Anthony Liguori wrote:
> Hi,
> 
> I've mentioned this to a few folks already but I wanted to start a 
> proper thread.
> 
> We're struggling in qemu with usability and one area that concerns me is 
> the disparity in features that are supported by qemu vs what's 
> implemented in libvirt.
> 
> This isn't necessarily libvirt's problem if it's mission is to provide a 
> common hypervisor API that covers the most commonly used features.
> 
> However, for qemu, we need an API that covers all of our features that 
> people can develop against.  The ultimate question we need to figure out 
> is, should we encourage our users to always use libvirt or should we 
> build our own API for people (and libvirt) to consume.
> 
> I don't think it's necessarily a big technical challenge for libvirt to 
> support qemu more completely.  I think it amounts to introducing a 
> series of virQemuXXXX APIs that implement qemu specific functions.  Over 
> time, qemu specific APIs can be deprecated in favour of more generic 
> virDomain APIs.

The second thing I forgot to mention in my previous reply, is that we also
need to get a much better idea of just which QEMU features missing in libvirt.
Even if we provide a generic mechansim for setting QEMU specific features,
we still need visibility into what needs to be added to the main generic
libvirt XML / API format.  I think the qdev & QMP work will be very
useful in this respect, since both are pretty much self-describing. All that
is missing from QEMU is a way to query/extra the qdev/QMP metadata from the
QEMU binary in an automated fashion. eg, '-device qdev' can list all devices,
but following on from that we need to query the properties known against each
device type. I think it'd also be desirable to have a machine readable '-help'
output format, so we can more reliably detect what args are available, since
even with qdev + -device, we're still adding more command line args to QEMU
over time like -netdev, -blockdev, etc. 

If we can easily get these details of qdev properties, QMP commands/args and
ARGV from QEMU, then we can reliably identify just what is missing from
libvirt & use that to guide development of generic APIs for the missing
features.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Juan Quintela]

Anthony Liguori <anthony@codemonkey.ws> wrote:
> Hi,
>
> I've mentioned this to a few folks already but I wanted to start a
> proper thread.
>
> We're struggling in qemu with usability and one area that concerns me
> is the disparity in features that are supported by qemu vs what's
> implemented in libvirt.
>
> This isn't necessarily libvirt's problem if it's mission is to provide
> a common hypervisor API that covers the most commonly used features.
>
> However, for qemu, we need an API that covers all of our features that
> people can develop against.  The ultimate question we need to figure
> out is, should we encourage our users to always use libvirt or should
> we build our own API for people (and libvirt) to consume.
>
> I don't think it's necessarily a big technical challenge for libvirt
> to support qemu more completely.  I think it amounts to introducing a
> series of virQemuXXXX APIs that implement qemu specific functions.
> Over time, qemu specific APIs can be deprecated in favour of more
> generic virDomain APIs.
>
> What's the feeling about this from the libvirt side of things?  Is
> there interest in support hypervisor specific interfaces should we be
> looking to provide our own management interface for libvirt to
> consume?

My problems with libvirt are:
- I can't "hack" the qemu command line.  It would be great to be able to
  edit the command line that libvirt is about to launch (yes, this is
  useful for developers, but it is an easy hack).

- monitor: I need a way to get to the monitor when going through
  libvirt, in the past you couldn't allow this, but now it looks
  possible.

- changing the emulator: It is normal for me to be using several qemu
  binaries for testing, changing it with libvirt is just a mess.

Now, why I want to use libvirt in the 1st place?
- libvirt is good at remembering details like mac addresses, disks
  positions, etc.  Compare

   virsh start foo
  versus a 300+ character long qemu command line.
- virt-viewer: I want to be able to switch consoles, period.

- networking: man, setting networking is a mess, libvirt just does it
  for you.

Now, to do suggestions. What would make it for me:
- being able to use the qemu command line -> maximum flexibility
- if that is too complicated for any reason, being able to change the
  <emulator>...</emulator> would be good enough the mayority of the
  time.  What I normally need is: use a different emulator or add some
  additional flag to qemu.

Just my 2 cents

Juan.

Re: [Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Gerd Hoffmann]

> - networking: man, setting networking is a mess, libvirt just does it
>    for you.

+1

Even when not using libvirt for a reason or another I usually hook my 
virtual machines into virbr0 (libvirt default network).

cheers,
   Gerd

[Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Juan Quintela]

Gerd Hoffmann <kraxel@redhat.com> wrote:
>> - networking: man, setting networking is a mess, libvirt just does it
>>    for you.
>
> +1
>
> Even when not using libvirt for a reason or another I usually hook my
> virtual machines into virbr0 (libvirt default network).

This is a war for another day :-)

I have that very same setup on my laptop.  But I already use dnsmasq for
other reasons -> no way to share dnsmasq with libvirt, libvirt want to
use a different one -> have to configure two things, especially if I
want specific names (in my dnsmasq setup already) for some of my guests.

Later, Juan.

Re: [Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Tue, Mar 23, 2010 at 11:50:57AM +0100, Juan Quintela wrote:
> Gerd Hoffmann <kraxel@redhat.com> wrote:
> >> - networking: man, setting networking is a mess, libvirt just does it
> >>    for you.
> >
> > +1
> >
> > Even when not using libvirt for a reason or another I usually hook my
> > virtual machines into virbr0 (libvirt default network).
> 
> This is a war for another day :-)
> 
> I have that very same setup on my laptop.  But I already use dnsmasq for
> other reasons -> no way to share dnsmasq with libvirt, libvirt want to
> use a different one -> have to configure two things, especially if I
> want specific names (in my dnsmasq setup already) for some of my guests.

FYI, the dnsmasq maintainer has actually volunteered todo some enhancements
to dnsmasq to allow a prexisting instance of dnsmasq to play nicely with
libvirt's needs. This should pretty much solve that problem

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

[Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Juan Quintela]

"Daniel P. Berrange" <berrange@redhat.com> wrote:
> On Tue, Mar 23, 2010 at 11:50:57AM +0100, Juan Quintela wrote:
>> Gerd Hoffmann <kraxel@redhat.com> wrote:
>> >> - networking: man, setting networking is a mess, libvirt just does it
>> >>    for you.
>> >
>> > +1
>> >
>> > Even when not using libvirt for a reason or another I usually hook my
>> > virtual machines into virbr0 (libvirt default network).
>> 
>> This is a war for another day :-)
>> 
>> I have that very same setup on my laptop.  But I already use dnsmasq for
>> other reasons -> no way to share dnsmasq with libvirt, libvirt want to
>> use a different one -> have to configure two things, especially if I
>> want specific names (in my dnsmasq setup already) for some of my guests.
>
> FYI, the dnsmasq maintainer has actually volunteered todo some enhancements
> to dnsmasq to allow a prexisting instance of dnsmasq to play nicely with
> libvirt's needs. This should pretty much solve that problem

Great!!!!!
/me reserves beer for him.

Once there, making it to play well with selinux + dbus will make it just
perfect :-) (I haven't checked if this worked in the last year, so ...)

Later, Juan.

Re: [Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Jamie Lokier]

Gerd Hoffmann wrote:
> >- networking: man, setting networking is a mess, libvirt just does it
> >   for you.
> 
> +1
> 
> Even when not using libvirt for a reason or another I usually hook my 
> virtual machines into virbr0 (libvirt default network).

I had the opposite problem.  Needed to use multiple bridges and have
some VMs behind NAT without a bridge (private IPs), and some using
separately firewalled bridges (needed to behave like real attached
hardware with their original MACs, but be firewalled).  I couldn't see
how to do it easily with libvirt, so used qemu directly.

-- Jamie

Re: [Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Gerd Hoffmann]

On 03/24/10 00:13, Jamie Lokier wrote:
> Gerd Hoffmann wrote:
>>> - networking: man, setting networking is a mess, libvirt just does it
>>>    for you.
>>
>> +1
>>
>> Even when not using libvirt for a reason or another I usually hook my
>> virtual machines into virbr0 (libvirt default network).
>
> I had the opposite problem.  Needed to use multiple bridges and have
> some VMs behind NAT without a bridge (private IPs), and some using
> separately firewalled bridges (needed to behave like real attached
> hardware with their original MACs, but be firewalled).

No problem in theory.  libvirt should detect existing bridges and allow 
you to attach virtual machines to them.  So you can setup bridges and 
firewalling for them using usual distro tools and use them for virtual 
machines.

In practice I've seen this not working correctly in the past, i.e. my 
br0 didn't pop up in the virt-manager nic setup page.

cheers,
   Gerd

Re: [Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Cole Robinson]

On 03/24/2010 03:59 AM, Gerd Hoffmann wrote:
> On 03/24/10 00:13, Jamie Lokier wrote:
>> Gerd Hoffmann wrote:
>>>> - networking: man, setting networking is a mess, libvirt just does it
>>>>    for you.
>>>
>>> +1
>>>
>>> Even when not using libvirt for a reason or another I usually hook my
>>> virtual machines into virbr0 (libvirt default network).
>>
>> I had the opposite problem.  Needed to use multiple bridges and have
>> some VMs behind NAT without a bridge (private IPs), and some using
>> separately firewalled bridges (needed to behave like real attached
>> hardware with their original MACs, but be firewalled).
> 
> No problem in theory.  libvirt should detect existing bridges and allow 
> you to attach virtual machines to them.  So you can setup bridges and 
> firewalling for them using usual distro tools and use them for virtual 
> machines.
> 
> In practice I've seen this not working correctly in the past, i.e. my 
> br0 didn't pop up in the virt-manager nic setup page.
> 

Please file a bug: virt-manager has had bridge detection for years, so
something must be going wrong. In f13 we will use netcf for this, so
even bridge enumeration on remote hosts should work.

Additionally I recently made a change upstream to allow users to
manually enter a bridge name, since netcf isn't supported for all
distros yet.

- Cole

Re: [Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Gerd Hoffmann]

>> In practice I've seen this not working correctly in the past, i.e. my
                                                     ^^^^^^^^^^^
>> br0 didn't pop up in the virt-manager nic setup page.
>>
>
> Please file a bug: virt-manager has had bridge detection for years, so
> something must be going wrong.

Works for me now ;)

> In f13 we will use netcf for this, so
> even bridge enumeration on remote hosts should work.

Yes, remote connections.  F12 + virt-preview works.

cheers,
   Gerd

Re: [Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Jamie Lokier]

Juan Quintela wrote:
> - monitor: I need a way to get to the monitor when going through
>   libvirt, in the past you couldn't allow this, but now it looks
>   possible.

Now you can just start another monitor connection to qemu :-)

Previously I've used a multiplexing script which accepts multiple
monitor connections, and passes the commands to the real connection.
It's not ideal because a slow command blocks any others, but it
basically works.

> - changing the emulator: It is normal for me to be using several qemu
>   binaries for testing, changing it with libvirt is just a mess.

Same here, except not just for testing: I need to use different qemu
binaries for production use, because newer ones don't work with some
VM images that work on older ones.

> - virt-viewer: I want to be able to switch consoles, period.

A decent VNC client gets close to this.  If the qemus advertised
themselves, some VNC clients would show a list of them automatically.
As it is, I use Gnome's VNC client with a bookmark for each VM, which
works quite well.  It even has tabs :-)

> - networking: man, setting networking is a mess, libvirt just does it
>   for you.

networking is often a mess, and what libvirt does isn't always what
you want, even if it often is.  I need the option to set up networking
separately and have libvirt use what it's given, otherwise I cannot use it.

-- Jamie

[Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Andi Kleen]

Juan Quintela <quintela@redhat.com> writes:
>
> - networking: man, setting networking is a mess, libvirt just does it
>   for you.

Agreed it's messy, but isn't this something that the standard qemu
command line tool could potentially do better by itself? I don't see why you 
need a wrapper for that.

-Andi
-- 
ak@linux.intel.com -- Speaking for myself only.

[Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Juan Quintela]

Andi Kleen <andi@firstfloor.org> wrote:
> Juan Quintela <quintela@redhat.com> writes:
>>
>> - networking: man, setting networking is a mess, libvirt just does it
>>   for you.
>
> Agreed it's messy, but isn't this something that the standard qemu
> command line tool could potentially do better by itself? I don't see why you 
> need a wrapper for that.

In my case, basically it is MAC addresses.  I have dhcp setup, and it
always give the same IP to the same MAC.  But you have to remember to
type the MAC addresses.

This is the typical command line that virsh start launch for me:

/usr/libexec/qemu-kvm -S -M pc-0.12 -enable-kvm -m 1024 -smp
2,sockets=2,cores=1,threads=1 -name f12X-64 -uuid
1fbe73a6-f519-e848-03bd-6636f765d143 -nodefaults -chardev
socket,id=monitor,path=/var/lib/libvirt/qemu/f12X-64.monitor,server,nowait
-mon chardev=monitor,mode=readline -rtc base=utc -boot c -drive
file=/mnt/kvm/images/f12X-64.img,if=none,id=drive-virtio-disk0,boot=on,cache=none
-device
virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0
-device
virtio-net-pci,vlan=0,id=net0,mac=54:52:00:44:72:e6,bus=pci.0,addr=0x5
-net tap,fd=18,vlan=0,name=hostnet0 -chardev pty,id=serial0 -device
isa-serial,chardev=serial0 -usb -device usb-tablet,id=input0 -vnc
127.0.0.1:0 -k es -vga cirrus -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3

>From parts:

/usr/libexec/qemu-kvm
 -S

I don't want that
 -M pc-0.12

I don't care.

-enable-kvm

I _want_ :)
-m 1024

Also god idea

-smp 2,sockets=2,cores=1,threads=1

by hand it is always "-smp 2"
-name f12X-64

-uuid 1fbe73a6-f519-e848-03bd-6636f765d143

don't care

 -nodefaults

-chardev
socket,id=monitor,path=/var/lib/libvirt/qemu/f12X-64.monitor,server,nowait
-mon chardev=monitor,mode=readline

this is simplified as:
  -monitor stdio
when I launch it by hand.

 -rtc base=utc -boot c

don't care

-drive file=/mnt/kvm/images/f12X-64.img,if=none,id=drive-virtio-disk0,boot=on,cache=none
-device virtio-blk-pci,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0

this is _wow_, I only want to put the disk image path and convince it to
use virtio driver

-device virtio-net-pci,vlan=0,id=net0,mac=54:52:00:44:72:e6,bus=pci.0,addr=0x5
-net tap,fd=18,vlan=0,name=hostnet0

this always have to be changed. s/fd=18/script=/etc/kvm-ifup/
and then I normally found that I want "downscript=" to avoid the warning
at exit time.  If I don't put a mac address, qemu command line works
well, but as I normally also use vnc I have to:
- launch qemu
- kill it, relaunch with -vnc :0 instead of -vnc 127.0.0.1:0
- re-launch qemu
- connect to vnc
- check what address the dhcp server was giving to it this time
- I can ssh to the client now
with libvirt handling the command line, I just ssh to the same dhcp
address that was given the previous time/day/...

-chardev pty,id=serial0 -device isa-serial,chardev=serial0

I only use serial from time to time, and using -serial
tcp:0,server,nowait (or whatever is the sintax is easier by hand)

-usb -device usb-tablet,id=input0

usb tablet is mandatory, just in case the guest is able to _not_ grab
the mouse.

-vnc 127.0.0.1:0

Allways wrong in my case, because I want to run the vnc client in a
different machine.  a way to convince virt-viewer to connect to a qemu
launched by hand, or a way to convince libvirt to let me edit the
command line will be great.

 -k es -vga cirrus

this get right by default.

-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3

I normally don't use balloon.

Notice for the "normally I don't care bits", that at the end I always
care.  Why?  because then somebody arrives and told me that sound don't
work, and I have to edit the config file, and add sound option.  add a
sound option to the command line of qemu is not too complicate.

The other big problem for me are snapshots,  I have to remember
_exactly_ what was the qemu command line with which I saved the
snapshot.  Guess what, I normally don't remember and end:

- launching old qemu
- save a new snapshot
- test with the new qemu and new snapshot (because now I have the
  command line that I launched 5 mins before).

Just in case it helps.

Later, Juan.

[Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/23/2010 09:51 AM, Daniel Veillard wrote:
> On Mon, Mar 22, 2010 at 02:25:00PM -0500, Anthony Liguori wrote:
>    
>> Hi,
>>      
>    Hi Anthony,
>
>    
>> I've mentioned this to a few folks already but I wanted to start a
>> proper thread.
>>
>> We're struggling in qemu with usability and one area that concerns
>> me is the disparity in features that are supported by qemu vs what's
>> implemented in libvirt.
>>      
>    If you could come up with a list, then I would have an easier job
> answering, honnestly I have the feeling we spent the last 6 months
> filling that gap in a really fast way.
>    

qemu-doc.texi is a list of most of the command line features we 
support.  The help output of the monitor shows what we support in that 
interface.  It doesn't take a lot to read through it and see the things 
not supported by libvirt.  libvirt supports a relatively small amount of 
our overall features (although a good chunk of the most common set).

>> However, for qemu, we need an API that covers all of our features
>> that people can develop against.  The ultimate question we need to
>> figure out is, should we encourage our users to always use libvirt
>> or should we build our own API for people (and libvirt) to consume.
>>
>> I don't think it's necessarily a big technical challenge for libvirt
>> to support qemu more completely.  I think it amounts to introducing
>> a series of virQemuXXXX APIs that implement qemu specific functions.
>> Over time, qemu specific APIs can be deprecated in favour of more
>> generic virDomain APIs.
>>      
> But one point of libvirt is that once an API is there we don't break it.
>
> I think there is a serious divergence of approach there, instanciating
> API stating 'we are gonna deprecate them sooner or later' tell the
> application developper 'my time is more important than yours' and not
> really something I like to carry to the API users.
> The main goal of libvirt remains to provide APIs needed to unify the
> development of the virtualization layers. Having APIs which makes
> sense only for one or 2 virtualization engines is not a problem in
> itself, it just raises questions about the actual semantic of that API.
> If that semantic is sound, then I see no reason to not add it, really
> and we actually often do.
>    

Yeah, but the problem we're facing is, I want there to be an API added 
to the management layer as part of the feature commit in qemu.  If there 
has to be a discussion and decisions about how to model the API, it's 
not going to be successful.

Supporting legacy APIs forever is not a viable option for a project like 
qemu.  Things evolve quickly and we need a mechanism to deprecate APIs 
over time.

>> What's the feeling about this from the libvirt side of things?  Is
>> there interest in support hypervisor specific interfaces should we
>> be looking to provide our own management interface for libvirt to
>> consume?
>>      
>    The real question is what do you actually want to build.
>    

Any management application really.  Even with something like 
virt-manager, there's a ton of useful features that qemu supports (like 
migration status reporting) that libvirt doesn't support.

> Most of the feedback I have seen in this thread so far are mostly
> request to be able to hack on a qemu instance launched via libvirt.
>    

It's not about the "hacker" use-case.  It's about making sure that we've 
got 100% feature coverage in our management API.  All of the management 
tools that focus on KVM have had this problem that I am aware of.

We need to come up with a way that we can very easily plumb new qemu 
functions through the management interface.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Paul Brook]

> > I think there is a serious divergence of approach there, instanciating
> > API stating 'we are gonna deprecate them sooner or later' tell the
> > application developper 'my time is more important than yours' and not
> > really something I like to carry to the API users.
> > The main goal of libvirt remains to provide APIs needed to unify the
> > development of the virtualization layers. Having APIs which makes
> > sense only for one or 2 virtualization engines is not a problem in
> > itself, it just raises questions about the actual semantic of that API.
> > If that semantic is sound, then I see no reason to not add it, really
> > and we actually often do.
> 
> Yeah, but the problem we're facing is, I want there to be an API added
> to the management layer as part of the feature commit in qemu.  If there
> has to be a discussion and decisions about how to model the API, it's
> not going to be successful.

I thought the monitor protocol *was* our API. If not, why not?

Paul

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/23/2010 10:57 AM, Paul Brook wrote:
>>> I think there is a serious divergence of approach there, instanciating
>>> API stating 'we are gonna deprecate them sooner or later' tell the
>>> application developper 'my time is more important than yours' and not
>>> really something I like to carry to the API users.
>>> The main goal of libvirt remains to provide APIs needed to unify the
>>> development of the virtualization layers. Having APIs which makes
>>> sense only for one or 2 virtualization engines is not a problem in
>>> itself, it just raises questions about the actual semantic of that API.
>>> If that semantic is sound, then I see no reason to not add it, really
>>> and we actually often do.
>>>        
>> Yeah, but the problem we're facing is, I want there to be an API added
>> to the management layer as part of the feature commit in qemu.  If there
>> has to be a discussion and decisions about how to model the API, it's
>> not going to be successful.
>>      
> I thought the monitor protocol *was* our API. If not, why not?
>    

It is.  But our API is missing key components like guest enumeration.  
So the fundamental topic here is, do we introduce these missing 
components to allow people to build directly to our interface or do we 
make use of the functionality that libvirt already provides if they can 
plumb our API directly to users.

Regards,

Anthony Liguori

> Paul
>    

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/23/2010 06:06 PM, Anthony Liguori wrote:
>> I thought the monitor protocol *was* our API. If not, why not?
>
> It is.  But our API is missing key components like guest enumeration.  
> So the fundamental topic here is, do we introduce these missing 
> components to allow people to build directly to our interface or do we 
> make use of the functionality that libvirt already provides if they 
> can plumb our API directly to users.
>

Guest enumeration is another API.

Over the kvm call I suggested a qemu concentrator that would keep track 
of all running qemus, and would hand out monitor connections to users.  
It can do the enumeration (likely using qmp).  Libvirt could talk to 
that, like it does with other hypervisors.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [libvirt] [Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Tue, Mar 23, 2010 at 08:00:21PM +0200, Avi Kivity wrote:
> On 03/23/2010 06:06 PM, Anthony Liguori wrote:
> >>I thought the monitor protocol *was* our API. If not, why not?
> >
> >It is.  But our API is missing key components like guest enumeration.  
> >So the fundamental topic here is, do we introduce these missing 
> >components to allow people to build directly to our interface or do we 
> >make use of the functionality that libvirt already provides if they 
> >can plumb our API directly to users.
> >
> 
> Guest enumeration is another API.
> 
> Over the kvm call I suggested a qemu concentrator that would keep track 
> of all running qemus, and would hand out monitor connections to users.  
> It can do the enumeration (likely using qmp).  Libvirt could talk to 
> that, like it does with other hypervisors.

The libvirt QEMU driver started out as a fairly simple "concentrator" not
doing much beyond spawning QEMU with argv & issuing monitor commands. The
host concentrator inevitably needs to be involved in the OS level integration
with features such as cgroups, selinux/apparmounr, host NIC management, 
storage, iptables, etc. If you look at the daemons for Xen, VirtualBox, 
VMWare, that other libvirt drivers talk to, they all do faaaaar more than
just enumeration of VMs. A QEMU concentrator may start out simple, but it will 
end up growing over time to re-implememt much, if not all, the stuff that
libvirt already provides for QEMU in terms of host level APIs. If the core 
problem here is to provide app developers access to the full range of QEMU 
functionality, the re-implementing the entire of the libvirt QEMU driver is 
rather over the top way to achieve that.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [libvirt] [Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/23/2010 01:23 PM, Daniel P. Berrange wrote:
> On Tue, Mar 23, 2010 at 08:00:21PM +0200, Avi Kivity wrote:
>    
>> On 03/23/2010 06:06 PM, Anthony Liguori wrote:
>>      
>>>> I thought the monitor protocol *was* our API. If not, why not?
>>>>          
>>> It is.  But our API is missing key components like guest enumeration.
>>> So the fundamental topic here is, do we introduce these missing
>>> components to allow people to build directly to our interface or do we
>>> make use of the functionality that libvirt already provides if they
>>> can plumb our API directly to users.
>>>
>>>        
>> Guest enumeration is another API.
>>
>> Over the kvm call I suggested a qemu concentrator that would keep track
>> of all running qemus, and would hand out monitor connections to users.
>> It can do the enumeration (likely using qmp).  Libvirt could talk to
>> that, like it does with other hypervisors.
>>      
> The libvirt QEMU driver started out as a fairly simple "concentrator" not
> doing much beyond spawning QEMU with argv&  issuing monitor commands. The
> host concentrator inevitably needs to be involved in the OS level integration
> with features such as cgroups, selinux/apparmounr, host NIC management,
> storage, iptables, etc. If you look at the daemons for Xen, VirtualBox,
> VMWare, that other libvirt drivers talk to, they all do faaaaar more than
> just enumeration of VMs.

But with Xen, VirtualBox, and VMware, if you create a VM through the 
native interfaces, you'll see that virtual machine with libvirt.  
virt-top will show meaningful results for those VMs and almost all of 
libvirt's functionality will work as expected.

Likewise, if you create a virtual machine with libvirt, when you do an 
xm list, run the VirtualBox GUI, or use VirtualCenter, you see that 
guest and you can interact with it through those interfaces.

What that means is that if you write a libvirt application and need to 
use a feature that libvirt doesn't support for xen, you can also fall 
back to the Xend API.

That's currently missing with qemu.

>   A QEMU concentrator may start out simple, but it will
> end up growing over time to re-implememt much, if not all, the stuff that
> libvirt already provides for QEMU in terms of host level APIs. If the core
> problem here is to provide app developers access to the full range of QEMU
> functionality, the re-implementing the entire of the libvirt QEMU driver is
> rather over the top way to achieve that.
>    

The goal is not to replicate libvirt but to let a more complete, qemu 
specific API co-exist with libvirt just as is implemented for almost all 
of the other hypervisors libvirt supports.

Regards,

Anthony Liguori

> Regards,
> Daniel
>    

Re: [libvirt] [Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/23/2010 08:23 PM, Daniel P. Berrange wrote:
> On Tue, Mar 23, 2010 at 08:00:21PM +0200, Avi Kivity wrote:
>    
>> On 03/23/2010 06:06 PM, Anthony Liguori wrote:
>>      
>>>> I thought the monitor protocol *was* our API. If not, why not?
>>>>          
>>> It is.  But our API is missing key components like guest enumeration.
>>> So the fundamental topic here is, do we introduce these missing
>>> components to allow people to build directly to our interface or do we
>>> make use of the functionality that libvirt already provides if they
>>> can plumb our API directly to users.
>>>
>>>        
>> Guest enumeration is another API.
>>
>> Over the kvm call I suggested a qemu concentrator that would keep track
>> of all running qemus, and would hand out monitor connections to users.
>> It can do the enumeration (likely using qmp).  Libvirt could talk to
>> that, like it does with other hypervisors.
>>      
> The libvirt QEMU driver started out as a fairly simple "concentrator" not
> doing much beyond spawning QEMU with argv&  issuing monitor commands. The
> host concentrator inevitably needs to be involved in the OS level integration
> with features such as cgroups, selinux/apparmounr, host NIC management,
> storage, iptables, etc. If you look at the daemons for Xen, VirtualBox,
> VMWare, that other libvirt drivers talk to, they all do faaaaar more than
> just enumeration of VMs. A QEMU concentrator may start out simple, but it will
> end up growing over time to re-implememt much, if not all, the stuff that
> libvirt already provides for QEMU in terms of host level APIs.

The idea is not to replace libvirt, but provide something that sits 
underneath.  It wouldn't do any non-qemu host-level APIs.

>   If the core
> problem here is to provide app developers access to the full range of QEMU
> functionality, the re-implementing the entire of the libvirt QEMU driver is
> rather over the top way to achieve that.
>    

It's trivial to expose all qemu functionality by exposing a qmp connection.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/23/2010 01:00 PM, Avi Kivity wrote:
> On 03/23/2010 06:06 PM, Anthony Liguori wrote:
>>> I thought the monitor protocol *was* our API. If not, why not?
>>
>> It is.  But our API is missing key components like guest 
>> enumeration.  So the fundamental topic here is, do we introduce these 
>> missing components to allow people to build directly to our interface 
>> or do we make use of the functionality that libvirt already provides 
>> if they can plumb our API directly to users.
>>
>
> Guest enumeration is another API.
>
> Over the kvm call I suggested a qemu concentrator that would keep 
> track of all running qemus, and would hand out monitor connections to 
> users.  It can do the enumeration (likely using qmp).  Libvirt could 
> talk to that, like it does with other hypervisors.

If you think about network management, it's the difference between 
having a central management server that you add physical machines to, 
verses having physical machines use an advertisement mechanism (like 
mDNS or SLP).  The later mechanism scales better and tends to be more 
robust.

For instance, it's very common for VNC servers to advertise themselves 
via mDNS and it's also common for VNC clients to support this.  It 
requires no central server to keep track of VNC instances and generally 
provides much better usability.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Jamie Lokier]

Anthony Liguori wrote:
> (like mDNS or SLP).  The later mechanism scales better and tends to
> be more robust.

(Aside: mDNS is blocked on some larger networks because it creates too
much load.  On those networks, an aggregator is essential - or a
protocol which scales better (less broadcasting)).

Doesn't libvirt use mDNS already to discover multiple hosts on a
network, for remote access?

If so, why can't exactly the same protocol be used to enumerate
multiple VMs on each host?

If not, why not?

-- Jamie

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/23/2010 08:00 PM, Avi Kivity wrote:
> On 03/23/2010 06:06 PM, Anthony Liguori wrote:
>>> I thought the monitor protocol *was* our API. If not, why not?
>>
>> It is.  But our API is missing key components like guest 
>> enumeration.  So the fundamental topic here is, do we introduce these 
>> missing components to allow people to build directly to our interface 
>> or do we make use of the functionality that libvirt already provides 
>> if they can plumb our API directly to users.
>>
>
> Guest enumeration is another API.
>
> Over the kvm call I suggested a qemu concentrator that would keep 
> track of all running qemus, and would hand out monitor connections to 
> users.  It can do the enumeration (likely using qmp).  Libvirt could 
> talk to that, like it does with other hypervisors.
>

To elaborate

qemud
   - daemonaizes itself
   - listens on /var/lib/qemud/guests for incoming guest connections
   - listens on /var/lib/qemud/clients for incoming client connections
   - filters access according to uid (SCM_CREDENTIALS)
   - can pass a new monitor to client (SCM_RIGHTS)
   - supports 'list' command to query running guests
   - async messages on guest startup/exit

qemu
   - with -qemud option, connects to qemud (or maybe automatically?)

qemudc
   - command-line client, can access qemu human monitor

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Wed, Mar 24, 2010 at 07:17:26AM +0200, Avi Kivity wrote:
> On 03/23/2010 08:00 PM, Avi Kivity wrote:
> >On 03/23/2010 06:06 PM, Anthony Liguori wrote:
> >>>I thought the monitor protocol *was* our API. If not, why not?
> >>
> >>It is.  But our API is missing key components like guest 
> >>enumeration.  So the fundamental topic here is, do we introduce these 
> >>missing components to allow people to build directly to our interface 
> >>or do we make use of the functionality that libvirt already provides 
> >>if they can plumb our API directly to users.
> >>
> >
> >Guest enumeration is another API.
> >
> >Over the kvm call I suggested a qemu concentrator that would keep 
> >track of all running qemus, and would hand out monitor connections to 
> >users.  It can do the enumeration (likely using qmp).  Libvirt could 
> >talk to that, like it does with other hypervisors.
> >
> 
> To elaborate
> 
> qemud
>   - daemonaizes itself
>   - listens on /var/lib/qemud/guests for incoming guest connections
>   - listens on /var/lib/qemud/clients for incoming client connections
>   - filters access according to uid (SCM_CREDENTIALS)
>   - can pass a new monitor to client (SCM_RIGHTS)
>   - supports 'list' command to query running guests
>   - async messages on guest startup/exit

My concern is that once you provide this, then next someone wants it to
list inactive guests too. Once you list inactive guests, then you'll
want this to start a guest. Once you start guests then you want cgroups
integration, selinux labelling & so on, until it ends up replicating all
of libvirt's QEMU functionality.

To be able to use the list functionality from libvirt, we need this daemon
to also guarentee id, name & uuid uniqueness for all VMs, both running and
inactive, with separate namespaces for the system vs per-user lists. Or
we have to ignore any instances listed by qemud that were not started  by
libvirt, which rather defeats the purpose.

The filtering access part of this daemon is also not mapping well onto
libvirt's access model, because we don't soley filter based on UID in
libvirtd. We have it configurable based on UID, policykit, SASL, TLS/x509
already, and intend adding role based access control to further filter
things, integrating with the existing apparmour/selinux security models.
A qemud that filters based on UID only, gives users a side-channel to get
around libvirt's access control.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/24/2010 12:36 PM, Daniel P. Berrange wrote:
> On Wed, Mar 24, 2010 at 07:17:26AM +0200, Avi Kivity wrote:
>    
>> On 03/23/2010 08:00 PM, Avi Kivity wrote:
>>      
>>> On 03/23/2010 06:06 PM, Anthony Liguori wrote:
>>>        
>>>>> I thought the monitor protocol *was* our API. If not, why not?
>>>>>            
>>>> It is.  But our API is missing key components like guest
>>>> enumeration.  So the fundamental topic here is, do we introduce these
>>>> missing components to allow people to build directly to our interface
>>>> or do we make use of the functionality that libvirt already provides
>>>> if they can plumb our API directly to users.
>>>>
>>>>          
>>> Guest enumeration is another API.
>>>
>>> Over the kvm call I suggested a qemu concentrator that would keep
>>> track of all running qemus, and would hand out monitor connections to
>>> users.  It can do the enumeration (likely using qmp).  Libvirt could
>>> talk to that, like it does with other hypervisors.
>>>
>>>        
>> To elaborate
>>
>> qemud
>>    - daemonaizes itself
>>    - listens on /var/lib/qemud/guests for incoming guest connections
>>    - listens on /var/lib/qemud/clients for incoming client connections
>>    - filters access according to uid (SCM_CREDENTIALS)
>>    - can pass a new monitor to client (SCM_RIGHTS)
>>    - supports 'list' command to query running guests
>>    - async messages on guest startup/exit
>>      
> My concern is that once you provide this, then next someone wants it to
> list inactive guests too.

That's impossible, since qemud doesn't manage config files or disk 
images.  It can't even launch guests!

> Once you list inactive guests, then you'll
> want this to start a guest. Once you start guests then you want cgroups
> integration, selinux labelling&  so on, until it ends up replicating all
> of libvirt's QEMU functionality.
>
> To be able to use the list functionality from libvirt, we need this daemon
> to also guarentee id, name&  uuid uniqueness for all VMs, both running and
> inactive, with separate namespaces for the system vs per-user lists. Or
> we have to ignore any instances listed by qemud that were not started  by
> libvirt, which rather defeats the purpose.
>    

qemud won't guarantee name uniqueness or provide uuids.

> The filtering access part of this daemon is also not mapping well onto
> libvirt's access model, because we don't soley filter based on UID in
> libvirtd. We have it configurable based on UID, policykit, SASL, TLS/x509
> already, and intend adding role based access control to further filter
> things, integrating with the existing apparmour/selinux security models.
> A qemud that filters based on UID only, gives users a side-channel to get
> around libvirt's access control.
>    

That's true.  Any time you write a multiplexer these issues crop up.  
Much better to stay in single process land where everything is already 
taken care of.

So, at best qemud is a toy for people who are annoyed by libvirt.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/24/2010 05:42 AM, Avi Kivity wrote:
>
>> The filtering access part of this daemon is also not mapping well onto
>> libvirt's access model, because we don't soley filter based on UID in
>> libvirtd. We have it configurable based on UID, policykit, SASL, 
>> TLS/x509
>> already, and intend adding role based access control to further filter
>> things, integrating with the existing apparmour/selinux security models.
>> A qemud that filters based on UID only, gives users a side-channel to 
>> get
>> around libvirt's access control.
>
> That's true.  Any time you write a multiplexer these issues crop up.  
> Much better to stay in single process land where everything is already 
> taken care of.

What does a multiplexer give you that making individual qemu instances 
discoverable doesn't give you?  The later doesn't suffer from these 
problems.

Regards,

Anthony Liguori

> So, at best qemud is a toy for people who are annoyed by libvirt.
>

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/24/2010 02:23 PM, Anthony Liguori wrote:
> On 03/24/2010 05:42 AM, Avi Kivity wrote:
>>
>>> The filtering access part of this daemon is also not mapping well onto
>>> libvirt's access model, because we don't soley filter based on UID in
>>> libvirtd. We have it configurable based on UID, policykit, SASL, 
>>> TLS/x509
>>> already, and intend adding role based access control to further filter
>>> things, integrating with the existing apparmour/selinux security 
>>> models.
>>> A qemud that filters based on UID only, gives users a side-channel 
>>> to get
>>> around libvirt's access control.
>>
>> That's true.  Any time you write a multiplexer these issues crop up.  
>> Much better to stay in single process land where everything is 
>> already taken care of.
>
> What does a multiplexer give you that making individual qemu instances 
> discoverable doesn't give you?  The later doesn't suffer from these 
> problems.
>

You don't get a directory filled with a zillion socket files pointing at 
dead guests.  Agree that's a poor return on investment.

Maybe we want a O_UNLINK_ON_CLOSE for unix domain sockets - but no, 
that's not implementable.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/24/2010 07:29 AM, Avi Kivity wrote:
> On 03/24/2010 02:23 PM, Anthony Liguori wrote:
>> On 03/24/2010 05:42 AM, Avi Kivity wrote:
>>>
>>>> The filtering access part of this daemon is also not mapping well onto
>>>> libvirt's access model, because we don't soley filter based on UID in
>>>> libvirtd. We have it configurable based on UID, policykit, SASL, 
>>>> TLS/x509
>>>> already, and intend adding role based access control to further filter
>>>> things, integrating with the existing apparmour/selinux security 
>>>> models.
>>>> A qemud that filters based on UID only, gives users a side-channel 
>>>> to get
>>>> around libvirt's access control.
>>>
>>> That's true.  Any time you write a multiplexer these issues crop 
>>> up.  Much better to stay in single process land where everything is 
>>> already taken care of.
>>
>> What does a multiplexer give you that making individual qemu 
>> instances discoverable doesn't give you?  The later doesn't suffer 
>> from these problems.
>>
>
> You don't get a directory filled with a zillion socket files pointing 
> at dead guests.  Agree that's a poor return on investment.

Deleting it on atexit combined with flushing the whole directory at 
startup is a pretty reasonable solution to this (which is ultimately how 
the entirety of /var/run behaves).

If you're really paranoid, you can fork() a helper with a shared pipe to 
implement unlink on close.

Regards,

Anthony Liguori

> Maybe we want a O_UNLINK_ON_CLOSE for unix domain sockets - but no, 
> that's not implementable.
>

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/24/2010 02:32 PM, Anthony Liguori wrote:
>> You don't get a directory filled with a zillion socket files pointing 
>> at dead guests.  Agree that's a poor return on investment.
>
>
> Deleting it on atexit combined with flushing the whole directory at 
> startup is a pretty reasonable solution to this (which is ultimately 
> how the entirety of /var/run behaves).
>
> If you're really paranoid, you can fork() a helper with a shared pipe 
> to implement unlink on close.

My paranoia comes nowhere near to my dislike of forked helpers.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Jamie Lokier]

Avi Kivity wrote:
> On 03/24/2010 02:32 PM, Anthony Liguori wrote:
> >>You don't get a directory filled with a zillion socket files pointing 
> >>at dead guests.  Agree that's a poor return on investment.
> >
> >
> >Deleting it on atexit combined with flushing the whole directory at 
> >startup is a pretty reasonable solution to this (which is ultimately 
> >how the entirety of /var/run behaves).
> >
> >If you're really paranoid, you can fork() a helper with a shared pipe 
> >to implement unlink on close.
> 
> My paranoia comes nowhere near to my dislike of forked helpers.

Use clone() then, it's cheaper ;-)

Anyway, Linux at least *does* have unlink-on-exit unix sockets: use
the abstract unix namespace.

Enumeration is a different problem from being able to connect to an
instance, and there are several approaches to enumerating multiple
running instances.

One of the most well known at the moment is mDNS service discovery,
and each instance registering with freedesktop's Avahi enumeration
service.

-- Jamie

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Luiz Capitulino]

On Wed, 24 Mar 2010 12:42:16 +0200
Avi Kivity <avi@redhat.com> wrote:

> So, at best qemud is a toy for people who are annoyed by libvirt.

 Is the reason for doing this in qemu because libvirt is annoying? I don't see
how adding yet another layer/daemon is going to improve ours and user's life
(the same applies for libqemu).

 If I got it right, there were two complaints from the kvm-devel flamewar:

1. Qemu has usability problems
2. There's no way an external tool can get /proc/kallsyms info from Qemu

 I don't see how libqemu can help with 1) and having qemud doesn't seem
the best solution for 2) either.

 Still talking about 2), what's wrong in getting the PID or having a QMP
connection in a well known location as suggested by Anthony?

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/24/2010 06:42 PM, Luiz Capitulino wrote:
> On Wed, 24 Mar 2010 12:42:16 +0200
> Avi Kivity<avi@redhat.com>  wrote:
>
>    
>> So, at best qemud is a toy for people who are annoyed by libvirt.
>>      
>   Is the reason for doing this in qemu because libvirt is annoying?

Mostly.

> I don't see
> how adding yet another layer/daemon is going to improve ours and user's life
> (the same applies for libqemu).
>    

libvirt becomes optional.

>   If I got it right, there were two complaints from the kvm-devel flamewar:
>
> 1. Qemu has usability problems
> 2. There's no way an external tool can get /proc/kallsyms info from Qemu
>
>   I don't see how libqemu can help with 1) and having qemud doesn't seem
> the best solution for 2) either.
>
>   Still talking about 2), what's wrong in getting the PID or having a QMP
> connection in a well known location as suggested by Anthony?
>    

I now believe that's the best option.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Luiz Capitulino]

On Wed, 24 Mar 2010 21:49:45 +0200
Avi Kivity <avi@redhat.com> wrote:

> On 03/24/2010 06:42 PM, Luiz Capitulino wrote:
> > On Wed, 24 Mar 2010 12:42:16 +0200
> > Avi Kivity<avi@redhat.com>  wrote:
> >
> >    
> >> So, at best qemud is a toy for people who are annoyed by libvirt.
> >>      
> >   Is the reason for doing this in qemu because libvirt is annoying?
> 
> Mostly.
> 
> > I don't see
> > how adding yet another layer/daemon is going to improve ours and user's life
> > (the same applies for libqemu).
> >    
> 
> libvirt becomes optional.

 I think it should only be optional if all you want is to run a single VM
in this case what seems to be missing on our side is a _real_ GUI, bundled
with QEMU potentially written in a high-level language.

 Then we make virt-manager optional and this is good because we can sync
features way faster and we don't have to care about _managing_ several
VMs, our world in terms of usability and maintainability is about one VM.

 IMVHO, everything else should be done by third-party tools like libvirt,
we just provide the means for it.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/24/2010 03:12 PM, Luiz Capitulino wrote:
> On Wed, 24 Mar 2010 21:49:45 +0200
> Avi Kivity<avi@redhat.com>  wrote:
>
>    
>> On 03/24/2010 06:42 PM, Luiz Capitulino wrote:
>>      
>>> On Wed, 24 Mar 2010 12:42:16 +0200
>>> Avi Kivity<avi@redhat.com>   wrote:
>>>
>>>
>>>        
>>>> So, at best qemud is a toy for people who are annoyed by libvirt.
>>>>
>>>>          
>>>    Is the reason for doing this in qemu because libvirt is annoying?
>>>        
>> Mostly.
>>
>>      
>>> I don't see
>>> how adding yet another layer/daemon is going to improve ours and user's life
>>> (the same applies for libqemu).
>>>
>>>        
>> libvirt becomes optional.
>>      
>   I think it should only be optional if all you want is to run a single VM
> in this case what seems to be missing on our side is a _real_ GUI, bundled
> with QEMU potentially written in a high-level language.
>    

That's a separate problem.

>   Then we make virt-manager optional and this is good because we can sync
> features way faster and we don't have to care about _managing_ several
> VMs, our world in terms of usability and maintainability is about one VM.
>
>   IMVHO, everything else should be done by third-party tools like libvirt,
> we just provide the means for it.
>    

We need to have a common management interface for third party tools.  
libvirt cannot be that today because of the fact that it doesn't support 
all of our features.  What we need to figure out is how we can work with 
the libvirt team to fix this.

So far, a libqemu.so with a flexible transport that could be used 
directly by a libvirt user (ala cairo/gdk type interactions) seems like 
the best solution to me.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Alexander Graf]

On 24.03.2010, at 21:32, Anthony Liguori wrote:

> On 03/24/2010 03:12 PM, Luiz Capitulino wrote:
>> On Wed, 24 Mar 2010 21:49:45 +0200
>> Avi Kivity<avi@redhat.com>  wrote:
>> 
>>   
>>> On 03/24/2010 06:42 PM, Luiz Capitulino wrote:
>>>     
>>>> On Wed, 24 Mar 2010 12:42:16 +0200
>>>> Avi Kivity<avi@redhat.com>   wrote:
>>>> 
>>>> 
>>>>       
>>>>> So, at best qemud is a toy for people who are annoyed by libvirt.
>>>>> 
>>>>>         
>>>>   Is the reason for doing this in qemu because libvirt is annoying?
>>>>       
>>> Mostly.
>>> 
>>>     
>>>> I don't see
>>>> how adding yet another layer/daemon is going to improve ours and user's life
>>>> (the same applies for libqemu).
>>>> 
>>>>       
>>> libvirt becomes optional.
>>>     
>>  I think it should only be optional if all you want is to run a single VM
>> in this case what seems to be missing on our side is a _real_ GUI, bundled
>> with QEMU potentially written in a high-level language.
>>   
> 
> That's a separate problem.
> 
>>  Then we make virt-manager optional and this is good because we can sync
>> features way faster and we don't have to care about _managing_ several
>> VMs, our world in terms of usability and maintainability is about one VM.
>> 
>>  IMVHO, everything else should be done by third-party tools like libvirt,
>> we just provide the means for it.
>>   
> 
> We need to have a common management interface for third party tools.  libvirt cannot be that today because of the fact that it doesn't support all of our features.  What we need to figure out is how we can work with the libvirt team to fix this.

The feature problem isn't the only one. It's also about ease of use. I personally find the qemu command line easier to use than anything libvirt-derived.

> So far, a libqemu.so with a flexible transport that could be used directly by a libvirt user (ala cairo/gdk type interactions) seems like the best solution to me.

ACK.

One thing I was thinking is that it might be a good idea to split the qemu command line version off as well. The qemu backend would then only speak QMP with an empty device case and the actual "qemu" command would run that, connect to it via QMP and instanciate everything.

That way we'd get a consistent interface for management apps while keeping an easy to use CLI.


Alex

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Luiz Capitulino]

On Wed, 24 Mar 2010 21:54:09 +0100
Alexander Graf <agraf@suse.de> wrote:

> 
> On 24.03.2010, at 21:32, Anthony Liguori wrote:
> 
> > On 03/24/2010 03:12 PM, Luiz Capitulino wrote:
> >> On Wed, 24 Mar 2010 21:49:45 +0200
> >> Avi Kivity<avi@redhat.com>  wrote:
> >> 
> >>   
> >>> On 03/24/2010 06:42 PM, Luiz Capitulino wrote:
> >>>     
> >>>> On Wed, 24 Mar 2010 12:42:16 +0200
> >>>> Avi Kivity<avi@redhat.com>   wrote:
> >>>> 
> >>>> 
> >>>>       
> >>>>> So, at best qemud is a toy for people who are annoyed by libvirt.
> >>>>> 
> >>>>>         
> >>>>   Is the reason for doing this in qemu because libvirt is annoying?
> >>>>       
> >>> Mostly.
> >>> 
> >>>     
> >>>> I don't see
> >>>> how adding yet another layer/daemon is going to improve ours and user's life
> >>>> (the same applies for libqemu).
> >>>> 
> >>>>       
> >>> libvirt becomes optional.
> >>>     
> >>  I think it should only be optional if all you want is to run a single VM
> >> in this case what seems to be missing on our side is a _real_ GUI, bundled
> >> with QEMU potentially written in a high-level language.
> >>   
> > 
> > That's a separate problem.
> > 
> >>  Then we make virt-manager optional and this is good because we can sync
> >> features way faster and we don't have to care about _managing_ several
> >> VMs, our world in terms of usability and maintainability is about one VM.
> >> 
> >>  IMVHO, everything else should be done by third-party tools like libvirt,
> >> we just provide the means for it.
> >>   
> > 
> > We need to have a common management interface for third party tools.  libvirt cannot be that today because of the fact that it doesn't support all of our features.  What we need to figure out is how we can work with the libvirt team to fix this.
> 
> The feature problem isn't the only one. It's also about ease of use. I personally find the qemu command line easier to use than anything libvirt-derived.

 Because your a developer and it does make sense to have a good CLI,
on the other hand we also have use cases for a GUI bundled in QEMU
and libvirt-derived things, which know how to deal with several
VMs and integrates well with lots of other things.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Alexander Graf]

On 24.03.2010, at 22:33, Luiz Capitulino wrote:

> On Wed, 24 Mar 2010 21:54:09 +0100
> Alexander Graf <agraf@suse.de> wrote:
> 
>> 
>> On 24.03.2010, at 21:32, Anthony Liguori wrote:
>> 
>>> On 03/24/2010 03:12 PM, Luiz Capitulino wrote:
>>>> On Wed, 24 Mar 2010 21:49:45 +0200
>>>> Avi Kivity<avi@redhat.com>  wrote:
>>>> 
>>>> 
>>>>> On 03/24/2010 06:42 PM, Luiz Capitulino wrote:
>>>>> 
>>>>>> On Wed, 24 Mar 2010 12:42:16 +0200
>>>>>> Avi Kivity<avi@redhat.com>   wrote:
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> So, at best qemud is a toy for people who are annoyed by libvirt.
>>>>>>> 
>>>>>>> 
>>>>>>  Is the reason for doing this in qemu because libvirt is annoying?
>>>>>> 
>>>>> Mostly.
>>>>> 
>>>>> 
>>>>>> I don't see
>>>>>> how adding yet another layer/daemon is going to improve ours and user's life
>>>>>> (the same applies for libqemu).
>>>>>> 
>>>>>> 
>>>>> libvirt becomes optional.
>>>>> 
>>>> I think it should only be optional if all you want is to run a single VM
>>>> in this case what seems to be missing on our side is a _real_ GUI, bundled
>>>> with QEMU potentially written in a high-level language.
>>>> 
>>> 
>>> That's a separate problem.
>>> 
>>>> Then we make virt-manager optional and this is good because we can sync
>>>> features way faster and we don't have to care about _managing_ several
>>>> VMs, our world in terms of usability and maintainability is about one VM.
>>>> 
>>>> IMVHO, everything else should be done by third-party tools like libvirt,
>>>> we just provide the means for it.
>>>> 
>>> 
>>> We need to have a common management interface for third party tools.  libvirt cannot be that today because of the fact that it doesn't support all of our features.  What we need to figure out is how we can work with the libvirt team to fix this.
>> 
>> The feature problem isn't the only one. It's also about ease of use. I personally find the qemu command line easier to use than anything libvirt-derived.
> 
> Because your a developer and it does make sense to have a good CLI,
> on the other hand we also have use cases for a GUI bundled in QEMU
> and libvirt-derived things, which know how to deal with several
> VMs and integrates well with lots of other things.

Oh I certainly would use a GUI if it was easy to use for me.

Imagine we had a full machine description configuration, similar to a .vmx file. A GUI would simply use that. Whenever I add a feature or want to test something out, I'd use that .vmx file with the GUI and everything would be great.

In fact, I'd prefer that over remembering weird command line options for qemu. And you can always just expose something like the monitor interface using the GUI.


Alex

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Luiz Capitulino]

On Wed, 24 Mar 2010 15:32:42 -0500
Anthony Liguori <anthony@codemonkey.ws> wrote:

> On 03/24/2010 03:12 PM, Luiz Capitulino wrote:
> > On Wed, 24 Mar 2010 21:49:45 +0200
> > Avi Kivity<avi@redhat.com>  wrote:
> >
> >    
> >> On 03/24/2010 06:42 PM, Luiz Capitulino wrote:
> >>      
> >>> On Wed, 24 Mar 2010 12:42:16 +0200
> >>> Avi Kivity<avi@redhat.com>   wrote:
> >>>
> >>>
> >>>        
> >>>> So, at best qemud is a toy for people who are annoyed by libvirt.
> >>>>
> >>>>          
> >>>    Is the reason for doing this in qemu because libvirt is annoying?
> >>>        
> >> Mostly.
> >>
> >>      
> >>> I don't see
> >>> how adding yet another layer/daemon is going to improve ours and user's life
> >>> (the same applies for libqemu).
> >>>
> >>>        
> >> libvirt becomes optional.
> >>      
> >   I think it should only be optional if all you want is to run a single VM
> > in this case what seems to be missing on our side is a _real_ GUI, bundled
> > with QEMU potentially written in a high-level language.
> >    
> 
> That's a separate problem.

 I see it as a related problem, because what seems to be under discussion
is the quality of our interfaces with humans and tools.

 Also, when we were discussing the usuability problems I remember that
you

*WARNING: I might be wrong here, please correct me if so*

 you said that you don't push users to libvirt because it's out of sync with
our features. The point is that, even if this true and even if we solve that,
I don't think it will solve the problem of a good experience for a
'single VM user', because libvirt is more than that and people will likely
be annoyed as much as they are today.

 I believe this problem is up to us to solve.

> >   Then we make virt-manager optional and this is good because we can sync
> > features way faster and we don't have to care about _managing_ several
> > VMs, our world in terms of usability and maintainability is about one VM.
> >
> >   IMVHO, everything else should be done by third-party tools like libvirt,
> > we just provide the means for it.
> >    
> 
> We need to have a common management interface for third party tools.

 QMP? :-)
  
> libvirt cannot be that today because of the fact that it doesn't support 
> all of our features.  What we need to figure out is how we can work with 
> the libvirt team to fix this.

 Agreed.

> So far, a libqemu.so with a flexible transport that could be used 
> directly by a libvirt user (ala cairo/gdk type interactions) seems like 
> the best solution to me.

 I tend to disagree.

 First, I think we should invest our time and effort on the text protocol
business, which is QMP. Having yet another public interface will likely split
efforts a bit and will make clients' life harder (which one should I choose?
What if they get out of sync?). Not to mention that I think Paul has a point,
if QMP is not useful here, why do we have it in the first place (vs. a C library
from the beginning)?

 You mentioned dynamic dispatch, but this is useful only for C clients right?
If so, what C clients you expected beyond libvirt? Note that libvirt has added
a new events API recently.

 The second most important point for me is: why do you believe that
libqemu.so is going to improve things? Do you expect that libvirt will
sync faster? If this is the case, I think it will be as slower as it's
currently, as the problem is not the availability of interfaces, but
most likely community integration.

 I like the idea of having a transient qemu-specific API in libvirt,
as suggested by someone in this thread.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/24/2010 04:25 PM, Luiz Capitulino wrote:
>
>   I see it as a related problem, because what seems to be under discussion
> is the quality of our interfaces with humans and tools.
>
>   Also, when we were discussing the usuability problems I remember that
> you
>
> *WARNING: I might be wrong here, please correct me if so*
>
>   you said that you don't push users to libvirt because it's out of sync with
> our features.

Yes.

>   The point is that, even if this true and even if we solve that,
> I don't think it will solve the problem of a good experience for a
> 'single VM user', because libvirt is more than that and people will likely
> be annoyed as much as they are today.
>
>   I believe this problem is up to us to solve.
>    

With my qemu hat on, I'm happy to ignore libvirt and say we need to own 
our interfaces and to compete with libvirt for users.

But with my Linux virtualization hat on, I want to see a single 
management interface that users can use without having to make a choice 
between libvirt features or libqemu features.

>>>    Then we make virt-manager optional and this is good because we can sync
>>> features way faster and we don't have to care about _managing_ several
>>> VMs, our world in terms of usability and maintainability is about one VM.
>>>
>>>    IMVHO, everything else should be done by third-party tools like libvirt,
>>> we just provide the means for it.
>>>
>>>        
>> We need to have a common management interface for third party tools.
>>      
>   QMP? :-)

Only if QMP is compatible with libvirt.  I don't want a user to have to 
choose between QMP and libvirt.
>> So far, a libqemu.so with a flexible transport that could be used
>> directly by a libvirt user (ala cairo/gdk type interactions) seems like
>> the best solution to me.
>>      
>   I tend to disagree.
>
>   First, I think we should invest our time and effort on the text protocol
> business, which is QMP. Having yet another public interface will likely split
> efforts a bit and will make clients' life harder (which one should I choose?
> What if they get out of sync?). Not to mention that I think Paul has a point,
> if QMP is not useful here, why do we have it in the first place (vs. a C library
> from the beginning)?
>
>   You mentioned dynamic dispatch, but this is useful only for C clients right?
> If so, what C clients you expected beyond libvirt?

Users want a C API.  I don't agree that libvirt is the only C interface 
consumer out there.

>   Note that libvirt has added
> a new events API recently.
>
>   The second most important point for me is: why do you believe that
> libqemu.so is going to improve things? Do you expect that libvirt will
> sync faster?

With GDK and Cairo, when Cairo adds a new feature, GDK doesn't have to 
do anything to support it.  Users just get a cairo context from GDK and 
use the cairo API directly.

GDK provides a higher level interface for 2d operations that is more 
platform agnostic, and users can choice to use that or write directly to 
the cairo API.

>   If this is the case, I think it will be as slower as it's
> currently, as the problem is not the availability of interfaces, but
> most likely community integration.
>
>   I like the idea of having a transient qemu-specific API in libvirt,
> as suggested by someone in this thread.
>    

I really think what we want is for a libvirt user to be able to call 
libqemu functions directly.  There shouldn't have to be libvirt specific 
functions for every operation we expose.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Vincent Hanquez]

On 24/03/10 21:40, Anthony Liguori wrote:
>> If so, what C clients you expected beyond libvirt?
>
> Users want a C API.  I don't agree that libvirt is the only C 
> interface consumer out there.

(I've seen this written too many times ...)
How do you know that ? did you do a poll or something where *actual* 
users vote/tell ?

 From my point of view, i wouldn't want to write a high level management 
toolstack in C, specially
since the API is well defined JSON which is easily available in all high 
level language out there.

-- 
Vincent

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/25/2010 10:26 AM, Vincent Hanquez wrote:
> On 24/03/10 21:40, Anthony Liguori wrote:
>>> If so, what C clients you expected beyond libvirt?
>>
>> Users want a C API.  I don't agree that libvirt is the only C 
>> interface consumer out there.
>
> (I've seen this written too many times ...)
> How do you know that ? did you do a poll or something where *actual* 
> users vote/tell ?
>
> From my point of view, i wouldn't want to write a high level 
> management toolstack in C, specially
> since the API is well defined JSON which is easily available in all 
> high level language out there.
>

Strongly agreed.  Even the managementy bits of qemu (anything around 
QObject) are suffering from the lowleveledness of C.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/25/2010 03:26 AM, Vincent Hanquez wrote:
> On 24/03/10 21:40, Anthony Liguori wrote:
>>> If so, what C clients you expected beyond libvirt?
>>
>> Users want a C API.  I don't agree that libvirt is the only C 
>> interface consumer out there.
>
> (I've seen this written too many times ...)
> How do you know that ? did you do a poll or something where *actual* 
> users vote/tell ?
>
> From my point of view, i wouldn't want to write a high level 
> management toolstack in C, specially
> since the API is well defined JSON which is easily available in all 
> high level language out there.

There's a whole world of C based management toolstacks (CIM).

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/25/2010 02:33 PM, Anthony Liguori wrote:
>> From my point of view, i wouldn't want to write a high level 
>> management toolstack in C, specially
>> since the API is well defined JSON which is easily available in all 
>> high level language out there.
>
>
> There's a whole world of C based management toolstacks (CIM).
>

Gratefully I know very little about CIM, but isn't it language independent?

The prominent open source implementation, pegasus, is written in C++.

Or are you referring to specific management apps written in C?  If they 
go through CIM, how can they talk qmp?

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/25/2010 07:37 AM, Avi Kivity wrote:
> On 03/25/2010 02:33 PM, Anthony Liguori wrote:
>>> From my point of view, i wouldn't want to write a high level 
>>> management toolstack in C, specially
>>> since the API is well defined JSON which is easily available in all 
>>> high level language out there.
>>
>>
>> There's a whole world of C based management toolstacks (CIM).
>>
>
> Gratefully I know very little about CIM, but isn't it language 
> independent?
>
> The prominent open source implementation, pegasus, is written in C++.

There is also SFCB which is written in C.

But an awful lot of the providers for pegasus are written in C.

The point is, C is a lowest common denominator and it's important to 
support in a proper way.

Regards,

Anthony Liguori

> Or are you referring to specific management apps written in C?  If 
> they go through CIM, how can they talk qmp?
>

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/25/2010 03:44 PM, Anthony Liguori wrote:
> On 03/25/2010 07:37 AM, Avi Kivity wrote:
>> On 03/25/2010 02:33 PM, Anthony Liguori wrote:
>>>> From my point of view, i wouldn't want to write a high level 
>>>> management toolstack in C, specially
>>>> since the API is well defined JSON which is easily available in all 
>>>> high level language out there.
>>>
>>>
>>> There's a whole world of C based management toolstacks (CIM).
>>>
>>
>> Gratefully I know very little about CIM, but isn't it language 
>> independent?
>>
>> The prominent open source implementation, pegasus, is written in C++.
>
> There is also SFCB which is written in C.

Ok.

>
> But an awful lot of the providers for pegasus are written in C.

But we're concerned with only one, the virt provider.  None of the 
others will use libqemu?

> The point is, C is a lowest common denominator and it's important to 
> support in a proper way.

Problem is, it means horrible support for everyone else.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/25/2010 08:48 AM, Avi Kivity wrote:
>>
>> But an awful lot of the providers for pegasus are written in C.
>
> But we're concerned with only one, the virt provider.  None of the 
> others will use libqemu?
>
>> The point is, C is a lowest common denominator and it's important to 
>> support in a proper way.
>
> Problem is, it means horrible support for everyone else.

Why?

We can provide a generic QMP dispatch interface that high level 
languages can use.  Then they can do fancy dispatch, treat QErrors as 
exceptions, etc.

We just ought to also provide some simple C wrappers for all of the 
functions.  Yes, the C interface is inferior to the generic interface 
but that's fine.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Luiz Capitulino]

On Thu, 25 Mar 2010 08:57:36 -0500
Anthony Liguori <anthony@codemonkey.ws> wrote:

> On 03/25/2010 08:48 AM, Avi Kivity wrote:
> >>
> >> But an awful lot of the providers for pegasus are written in C.
> >
> > But we're concerned with only one, the virt provider.  None of the 
> > others will use libqemu?
> >
> >> The point is, C is a lowest common denominator and it's important to 
> >> support in a proper way.
> >
> > Problem is, it means horrible support for everyone else.
> 
> Why?

 Because it's useless for non C clients. QMP is language independent,
libqemu is a full machinery for C clients only.

> We can provide a generic QMP dispatch interface that high level 
> languages can use.  Then they can do fancy dispatch, treat QErrors as 
> exceptions, etc.

 They can do that by accessing QMP directly. Why would a Python developer
get in the mess of writing a Python binding for libqemu if they call do
the exactly same thing by using its native json module?

 Man, opening a QMP connection from Python and sending commands can be
done with a few lines.

> We just ought to also provide some simple C wrappers for all of the 
> functions.  Yes, the C interface is inferior to the generic interface 
> but that's fine.

 Why don't we start with my simple lib suggestion and wait one or two
releases to see what happens?

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/25/2010 09:09 AM, Luiz Capitulino wrote:
>
>> We can provide a generic QMP dispatch interface that high level
>> languages can use.  Then they can do fancy dispatch, treat QErrors as
>> exceptions, etc.
>>      
>   They can do that by accessing QMP directly. Why would a Python developer
> get in the mess of writing a Python binding for libqemu if they call do
> the exactly same thing by using its native json module?
>
>   Man, opening a QMP connection from Python and sending commands can be
> done with a few lines.
>    

Problem is, without a libqemu, libvirt cannot return a QMPContext that 
can be used by python bindings.   This is the problem that all high 
level languages have with respect to RPC transports.

You need libqemu to deal with establishing the transport.  That code 
needs to be common and shared across languages.

Dispatch can be handled by the high level language.

>> We just ought to also provide some simple C wrappers for all of the
>> functions.  Yes, the C interface is inferior to the generic interface
>> but that's fine.
>>      
>   Why don't we start with my simple lib suggestion and wait one or two
> releases to see what happens?
>    

I have no problem starting without the generated C wrappers.  But we 
need the core library to have the right abstractions with respect to 
transports.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Jamie Lokier]

Anthony Liguori wrote:
> On 03/25/2010 09:09 AM, Luiz Capitulino wrote:
> >
> >>We can provide a generic QMP dispatch interface that high level
> >>languages can use.  Then they can do fancy dispatch, treat QErrors as
> >>exceptions, etc.
> >>     
> >  They can do that by accessing QMP directly. Why would a Python developer
> >get in the mess of writing a Python binding for libqemu if they call do
> >the exactly same thing by using its native json module?
> >
> >  Man, opening a QMP connection from Python and sending commands can be
> >done with a few lines.
> >   
> 
> Problem is, without a libqemu, libvirt cannot return a QMPContext that 
> can be used by python bindings.   This is the problem that all high 
> level languages have with respect to RPC transports.
> 
> You need libqemu to deal with establishing the transport.  That code 
> needs to be common and shared across languages.

We can't libvirt talk QMG with Python over a pipe or local socket?

So that the Python can talk to native qemu and via libvirt with the
same code.

That would be much easier from Python person than writing a wrapper
around the C library.  Multiplied by each high-level language...

  -- Jamie

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/25/2010 03:57 PM, Anthony Liguori wrote:
> On 03/25/2010 08:48 AM, Avi Kivity wrote:
>>>
>>> But an awful lot of the providers for pegasus are written in C.
>>
>> But we're concerned with only one, the virt provider.  None of the 
>> others will use libqemu?
>>
>>> The point is, C is a lowest common denominator and it's important to 
>>> support in a proper way.
>>
>> Problem is, it means horrible support for everyone else.
>
> Why?
>
> We can provide a generic QMP dispatch interface that high level 
> languages can use.  Then they can do fancy dispatch, treat QErrors as 
> exceptions, etc.
>

Sure, with high level wrappers everything's fine.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Vincent Hanquez]

On Thu, Mar 25, 2010 at 08:57:36AM -0500, Anthony Liguori wrote:
> Why?
>
> We can provide a generic QMP dispatch interface that high level  
> languages can use.  Then they can do fancy dispatch, treat QErrors as  
> exceptions, etc.

Because more than likely it will be more efforts than doing the same work in
the native language, forcing certains designs [1] up to high-level-language
developers throats, and possibly less stability (segfault, memory corruption,
memory leak, ..) specially in development phase.

[1] lack of separation between IO and pure functions, file descriptor versus
stream, C memory functions instead of GC based, and probably lots of other
things easily accessible from high level language.

-- 
Vincent

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Markus Armbruster]

Anthony Liguori <anthony@codemonkey.ws> writes:

> On 03/25/2010 07:37 AM, Avi Kivity wrote:
>> On 03/25/2010 02:33 PM, Anthony Liguori wrote:
>>>> From my point of view, i wouldn't want to write a high level
>>>> management toolstack in C, specially
>>>> since the API is well defined JSON which is easily available in
>>>> all high level language out there.
>>>
>>>
>>> There's a whole world of C based management toolstacks (CIM).
>>>
>>
>> Gratefully I know very little about CIM, but isn't it language
>> independent?
>>
>> The prominent open source implementation, pegasus, is written in C++.
>
> There is also SFCB which is written in C.
>
> But an awful lot of the providers for pegasus are written in C.
>
> The point is, C is a lowest common denominator and it's important to
> support in a proper way.

No.  The lowest truly common denominator is plain text.  And we got that
covered already.

A developer encountered the problem of talking a simple text protocol.
He thought "I know, I'll create a 1:1 C API for that".  Now he got two
problems.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/25/2010 11:50 AM, Markus Armbruster wrote:
>
>> The point is, C is a lowest common denominator and it's important to
>> support in a proper way.
>>      
> No.  The lowest truly common denominator is plain text.  And we got that
> covered already.
>
> A developer encountered the problem of talking a simple text protocol.
> He thought "I know, I'll create a 1:1 C API for that".  Now he got two
> problems.
>    

I've done a poor job communicating in this thread.

The C API's primary purpose is *not* to providing 1:1 wrapping functions 
for QMP functions.  That's a minor, add on feature, that I really would 
like, but it not at all useful for high level languages.

The importances of libqemu is:

1) Providing a common QMP transport implementation that is extensible by 
third parties
2) Providing a set of common transports that support automatic discovery 
of command line launched guests
3) Providing a generic QMP dispatch function

Yes, this means you can't just create a JSON-RPC object in Python and 
talk QMP that way, but that's less desirable than you think it is.

You could if you really wanted to, but you wouldn't get the benefits of 
the common transports.

IOW, imagine qemu-cmd.  You want it to support:

# qmp_new_by_name("Fedora")
qemu-cmd Fedora set_link on

# libqemu-ssh.so - ssh_qmp_new()
qemu-cmd ssh://anthony@lab1.ibm/Fedora set_link on

# qmp_new_by_fd()
qemu-cmd -c /path/to/domain/socket set_link on

# libvirt-qemu.so - virDomainGetQMP()
qemu-cmd -b qemu+ssh://lab1.ibm/system Fedora set_link on

This requires a high level transport.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Markus Armbruster]

Anthony Liguori <anthony@codemonkey.ws> writes:

> On 03/25/2010 11:50 AM, Markus Armbruster wrote:
>>
>>> The point is, C is a lowest common denominator and it's important to
>>> support in a proper way.
>>>      
>> No.  The lowest truly common denominator is plain text.  And we got that
>> covered already.
>>
>> A developer encountered the problem of talking a simple text protocol.
>> He thought "I know, I'll create a 1:1 C API for that".  Now he got two
>> problems.
>>    
>
> I've done a poor job communicating in this thread.
>
> The C API's primary purpose is *not* to providing 1:1 wrapping
> functions for QMP functions.  That's a minor, add on feature, that I
> really would like, but it not at all useful for high level languages.
>
> The importances of libqemu is:
>
> 1) Providing a common QMP transport implementation that is extensible
> by third parties
> 2) Providing a set of common transports that support automatic
> discovery of command line launched guests
> 3) Providing a generic QMP dispatch function

Adding to this C wrappers for QMP commands threatens to make QMP command
arguments part of the library ABI.  Compatible QMP evolution (like
adding an optional argument) turns into a libqmp soname bump.
Counter-productive.  How do you plan to avoid that?

> Yes, this means you can't just create a JSON-RPC object in Python and
> talk QMP that way, but that's less desirable than you think it is.
>
> You could if you really wanted to, but you wouldn't get the benefits
> of the common transports.
>
> IOW, imagine qemu-cmd.  You want it to support:
>
> # qmp_new_by_name("Fedora")
> qemu-cmd Fedora set_link on
>
> # libqemu-ssh.so - ssh_qmp_new()
> qemu-cmd ssh://anthony@lab1.ibm/Fedora set_link on
>
> # qmp_new_by_fd()
> qemu-cmd -c /path/to/domain/socket set_link on
>
> # libvirt-qemu.so - virDomainGetQMP()
> qemu-cmd -b qemu+ssh://lab1.ibm/system Fedora set_link on
>
> This requires a high level transport.

All I'd want from such a transport is a file descriptor.  No need to
drag in yet another JSON library via libqmp.

Re: [libvirt] [Qemu-devel] Re: Supporting hypervisor specific APIs in libvirt

[Paolo Bonzini]

> Adding to this C wrappers for QMP commands threatens to make QMP command
> arguments part of the library ABI.  Compatible QMP evolution (like
> adding an optional argument) turns into a libqmp soname bump.
> Counter-productive.  How do you plan to avoid that?

.so versioning.  Ugly as hell to do manually, but if the library was 
automatically generated it would not be difficult.

Paolo

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/26/2010 10:37 AM, Markus Armbruster wrote:
>
>> The importances of libqemu is:
>>
>> 1) Providing a common QMP transport implementation that is extensible
>> by third parties
>> 2) Providing a set of common transports that support automatic
>> discovery of command line launched guests
>> 3) Providing a generic QMP dispatch function
>>      
> Adding to this C wrappers for QMP commands threatens to make QMP command
> arguments part of the library ABI.  Compatible QMP evolution (like
> adding an optional argument) turns into a libqmp soname bump.
> Counter-productive.  How do you plan to avoid that?
>    

You could make the API use QObjects; then you're completely isolated 
from high level protocol changes.  Of course, this is less useful than 
the full API.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/26/2010 04:51 AM, Avi Kivity wrote:
> On 03/26/2010 10:37 AM, Markus Armbruster wrote:
>>
>>> The importances of libqemu is:
>>>
>>> 1) Providing a common QMP transport implementation that is extensible
>>> by third parties
>>> 2) Providing a set of common transports that support automatic
>>> discovery of command line launched guests
>>> 3) Providing a generic QMP dispatch function
>> Adding to this C wrappers for QMP commands threatens to make QMP command
>> arguments part of the library ABI.  Compatible QMP evolution (like
>> adding an optional argument) turns into a libqmp soname bump.
>> Counter-productive.  How do you plan to avoid that?
>
> You could make the API use QObjects; then you're completely isolated 
> from high level protocol changes.  Of course, this is less useful than 
> the full API.

You want both.  You want a very high level QObject based API and then 
you want automatically generated wrappers with C friendly data types.  
The later API loses a little bit but that's okay.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/26/2010 02:37 AM, Markus Armbruster wrote:
> Adding to this C wrappers for QMP commands threatens to make QMP command
> arguments part of the library ABI.  Compatible QMP evolution (like
> adding an optional argument) turns into a libqmp soname bump.
> Counter-productive.  How do you plan to avoid that?
>    

I had thought about this.  I think there's a couple ways to handle it.  
You could ignore it and just not change existing symbols.  You could 
also introduce new functions any time an optional argument is added.  
Another option is to add a struct as a final argument whenever such a 
change happens that's padded.  Then new optional arguments can be added 
to the struct.



>> Yes, this means you can't just create a JSON-RPC object in Python and
>> talk QMP that way, but that's less desirable than you think it is.
>>
>> You could if you really wanted to, but you wouldn't get the benefits
>> of the common transports.
>>
>> IOW, imagine qemu-cmd.  You want it to support:
>>
>> # qmp_new_by_name("Fedora")
>> qemu-cmd Fedora set_link on
>>
>> # libqemu-ssh.so - ssh_qmp_new()
>> qemu-cmd ssh://anthony@lab1.ibm/Fedora set_link on
>>
>> # qmp_new_by_fd()
>> qemu-cmd -c /path/to/domain/socket set_link on
>>
>> # libvirt-qemu.so - virDomainGetQMP()
>> qemu-cmd -b qemu+ssh://lab1.ibm/system Fedora set_link on
>>
>> This requires a high level transport.
>>      
> All I'd want from such a transport is a file descriptor.  No need to
> drag in yet another JSON library via libqmp.
>    

Then you don't standardize creation which is probably where most of the 
complexity occurs.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Gildas Le Nadan]

Anthony Liguori wrote:
> On 03/25/2010 03:26 AM, Vincent Hanquez wrote:
>> On 24/03/10 21:40, Anthony Liguori wrote:
>>>> If so, what C clients you expected beyond libvirt?
>>>
>>> Users want a C API.  I don't agree that libvirt is the only C 
>>> interface consumer out there.
>>
>> (I've seen this written too many times ...)
>> How do you know that ? did you do a poll or something where *actual* 
>> users vote/tell ?
>>
>> From my point of view, i wouldn't want to write a high level 
>> management toolstack in C, specially
>> since the API is well defined JSON which is easily available in all 
>> high level language out there.
> 
> There's a whole world of C based management toolstacks (CIM).
> 
> Regards,
> 
> Anthony Liguori

That huge companies have developped over-complicated c-based management 
toolstacks doesn't necessarily mean that this is the best way to do it. 
It just mean that they have enough qualified ressources to do it.

A simple, language-neutral API is always preferable in my opinion, since 
it lowers the prerequisites/entry price, and will allow more people to 
use it, including system engineers.

Ensuring that the new API will be easy to use by new comers will also 
ensure that it will be easy to use by existing stacks including libvirt.

Also I second Avi's opinion in another mail that "all command line 
options [should] have qmp equivalents": it is vital for 
flexibility/manageability to be able to programatically change setups 
after a VM was started.

To quote the automation part of the "James White Manifesto"[1], a 
document that is gaining a lot of traction in the sysadmin/devops community:
"The provided API must have all functionality that the application provides.
The provided API must be tailored to more than one language and platform."

Regards,
Gildas

-- 
[1] You can find a copy of it here: 
http://www.kartar.net/2010/03/james-whites-rules-for-infrastructure/

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Thu, Mar 25, 2010 at 08:26:09AM +0000, Vincent Hanquez wrote:
> On 24/03/10 21:40, Anthony Liguori wrote:
> >>If so, what C clients you expected beyond libvirt?
> >
> >Users want a C API.  I don't agree that libvirt is the only C 
> >interface consumer out there.
> 
> (I've seen this written too many times ...)
> How do you know that ? did you do a poll or something where *actual* 
> users vote/tell ?
> 
> From my point of view, i wouldn't want to write a high level management 
> toolstack in C, specially
> since the API is well defined JSON which is easily available in all high 
> level language out there.

It was pretty straightforward for libvirt to talk to the JSON protocol
from C using the YAJL  library, so I don't think it is all that much of 
a barrier for low level languages like C either. If we want to make life
easy for app/library developers working against QEMU, then the far more
important aspect is to guarentee stability of all the QEMU interfaces 
since that is where all the serious pain occurs over time.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Vincent Hanquez]

On Thu, Mar 25, 2010 at 01:59:22PM +0000, Daniel P. Berrange wrote:
> > From my point of view, i wouldn't want to write a high level management 
> > toolstack in C, specially
> > since the API is well defined JSON which is easily available in all high 
> > level language out there.
> 
> It was pretty straightforward for libvirt to talk to the JSON protocol
> from C using the YAJL  library, so I don't think it is all that much of 
> a barrier for low level languages like C either.

note, that it's not the talking JSON part that's difficult to do in C (it's
just midly annoying compare to a highlevel language), but all the other part of
a toolstack. Since there's no performance requirements, writing in C is just a
bit of a waste ot time, but that's up to the developpers to choose the tools he
wants, even if it's not the most appropriate one ;)

> If we want to make life easy for app/library developers working against QEMU,
> then the far more important aspect is to guarentee stability of all the QEMU
> interfaces since that is where all the serious pain occurs over time.

if you're talking about the QMP interface then I agree with you. This need to
be back/forward compatible as much as possible and stable.

the other interface (i.e. the user monitor) has no business beeing
backward-compatible though, since it should never be used to talk a RPC.

-- 
Vincent

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Thu, Mar 25, 2010 at 02:56:52PM +0000, Vincent Hanquez wrote:
> On Thu, Mar 25, 2010 at 01:59:22PM +0000, Daniel P. Berrange wrote:
> > > From my point of view, i wouldn't want to write a high level management 
> > > toolstack in C, specially
> > > since the API is well defined JSON which is easily available in all high 
> > > level language out there.
> > 
> > It was pretty straightforward for libvirt to talk to the JSON protocol
> > from C using the YAJL  library, so I don't think it is all that much of 
> > a barrier for low level languages like C either.
> 
> note, that it's not the talking JSON part that's difficult to do in C (it's
> just midly annoying compare to a highlevel language), but all the other part of
> a toolstack. Since there's no performance requirements, writing in C is just a
> bit of a waste ot time, but that's up to the developpers to choose the tools he
> wants, even if it's not the most appropriate one ;)
> 
> > If we want to make life easy for app/library developers working against QEMU,
> > then the far more important aspect is to guarentee stability of all the QEMU
> > interfaces since that is where all the serious pain occurs over time.
> 
> if you're talking about the QMP interface then I agree with you. This need to
> be back/forward compatible as much as possible and stable.
> 
> the other interface (i.e. the user monitor) has no business beeing
> backward-compatible though, since it should never be used to talk a RPC.

I agree apps shouldn't use it for RPC, but admins using the interactive user 
monitor are just as deserving of stable commands & args. 

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Vincent Hanquez]

On Thu, Mar 25, 2010 at 03:07:20PM +0000, Daniel P. Berrange wrote:
> I agree apps shouldn't use it for RPC, but admins using the interactive user 
> monitor are just as deserving of stable commands & args. 

I think, once QMP is completely there, admins would be better using a qemu-cmd
that's just serialise it's command line arguments into a JSON command.
(something like dbus-send for dbus).

-- 
Vincent

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Thu, Mar 25, 2010 at 03:14:24PM +0000, Vincent Hanquez wrote:
> On Thu, Mar 25, 2010 at 03:07:20PM +0000, Daniel P. Berrange wrote:
> > I agree apps shouldn't use it for RPC, but admins using the interactive user 
> > monitor are just as deserving of stable commands & args. 
> 
> I think, once QMP is completely there, admins would be better using a qemu-cmd
> that's just serialise it's command line arguments into a JSON command.

Then, after a qemu-cmd is introduced, we should mark the user monitor deprecated
along with a specific date/release its future for removal.


Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/25/2010 10:16 AM, Daniel P. Berrange wrote:
> On Thu, Mar 25, 2010 at 03:14:24PM +0000, Vincent Hanquez wrote:
>    
>> On Thu, Mar 25, 2010 at 03:07:20PM +0000, Daniel P. Berrange wrote:
>>      
>>> I agree apps shouldn't use it for RPC, but admins using the interactive user
>>> monitor are just as deserving of stable commands&  args.
>>>        
>> I think, once QMP is completely there, admins would be better using a qemu-cmd
>> that's just serialise it's command line arguments into a JSON command.
>>      
> Then, after a qemu-cmd is introduced, we should mark the user monitor deprecated
> along with a specific date/release its future for removal.
>    

I'm not sure.  The human monitor has some features that are not 
appropriate for QMP.  For instance, the ability to deal with formula 
input and some commands meant to add debugging.

I guess you could do that in qemu-cmd but I don't see a compelling 
reason to.

Regards,

Anthony Liguori

> Daniel
>    

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Alexander Graf]

Anthony Liguori wrote:
> On 03/25/2010 10:16 AM, Daniel P. Berrange wrote:
>> On Thu, Mar 25, 2010 at 03:14:24PM +0000, Vincent Hanquez wrote:
>>   
>>> On Thu, Mar 25, 2010 at 03:07:20PM +0000, Daniel P. Berrange wrote:
>>>     
>>>> I agree apps shouldn't use it for RPC, but admins using the
>>>> interactive user
>>>> monitor are just as deserving of stable commands&  args.
>>>>        
>>> I think, once QMP is completely there, admins would be better using
>>> a qemu-cmd
>>> that's just serialise it's command line arguments into a JSON command.
>>>      
>> Then, after a qemu-cmd is introduced, we should mark the user monitor
>> deprecated
>> along with a specific date/release its future for removal.
>>    
>
> I'm not sure.  The human monitor has some features that are not
> appropriate for QMP.  For instance, the ability to deal with formula
> input and some commands meant to add debugging.
>
> I guess you could do that in qemu-cmd but I don't see a compelling
> reason to.

As I mentioned before, I'd love to see the qemu binary (incl. monitor
interface) being implemented as a pure QMP user. Then libvirt and
friends can be 100% that they can achieve everything using QMP because
we don't live in the same address space anymore and can't pull tricks.


Alex

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Jamie Lokier]

Anthony Liguori wrote:
> I'm not sure.  The human monitor has some features that are not 
> appropriate for QMP.  For instance, the ability to deal with formula 
> input and some commands meant to add debugging.
> 
> I guess you could do that in qemu-cmd but I don't see a compelling 
> reason to.

Would it be all that much work to just rip out the human monitor
altogether, moving it's functionality to qemu-cmd?  That'd be a great
way to confirm that QMP has all the functionality.

It doesn't have to be written in C, by the way ;-)

-- Jamie

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Luiz Capitulino]

On Wed, 24 Mar 2010 16:40:18 -0500
Anthony Liguori <anthony@codemonkey.ws> wrote:

> >> We need to have a common management interface for third party tools.
> >>      
> >   QMP? :-)
> 
> Only if QMP is compatible with libvirt.  I don't want a user to have to 
> choose between QMP and libvirt.

 Why not? If all they want is a simple qemu session, they can use
QMP directly, on the other hand if what they want is more complex,
what's the problem of using a management API like libivrt?

 Again, one important point of the this discussion is: what interfaces
should qemu provide and how/when should qemu rely on libvirt?

 IMO, we should rely on libvirt for a management interface for all the
reasons that have already been exposed: it has a established community,
it provides a well tested and stable API, it does VM management, it
integrates with several other services like cgroups, SELinux, etc.

 Now, regarding qemu:

> >   You mentioned dynamic dispatch, but this is useful only for C clients right?
> > If so, what C clients you expected beyond libvirt?
> 
> Users want a C API.  I don't agree that libvirt is the only C interface 
> consumer out there.

 Actually, I do agree. Maybe, we don't have other C consumers because they
weren't crazy enough to parse the crap of the user Monitor (or they do,
but for simple things).

 One possible future client is perf, for example.

 Here is my solution (actually it's not mine, you have suggested
it some time ago): let's provide a convenient way for C clients to
use QMP. That is, let's have an overly simple library which takes
QDitcs, sends them to qemu through QMP and returns others QDicts.

 Something like the _sketch_ below:

// Open a connection
int qmp_open(..., QDict **greeting);

// Register a callback for async messages, BUT note that the async message
// object is passed verbatim
void qmp_async_mes_handler(..., void (*async_mes_handler)(QDict *mes));

// Send a QMP command
int qmp_send(..., const char *command, QDict *params, QDict **res);

 Obviously that we'll need a QMPContext and maybe additional functions,
but the two main ideas are:

1. We don't do management

2. QMP is our standard interface

> I really think what we want is for a libvirt user to be able to call 
> libqemu functions directly.  There shouldn't have to be libvirt specific 
> functions for every operation we expose.

 Not sure if this is too crazy but, considering this user wants to
use qemu features not implemented by libvirt yet, what about using both
libqmp (above) and libvirt at the same time?

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/25/2010 08:23 AM, Luiz Capitulino wrote:
> On Wed, 24 Mar 2010 16:40:18 -0500
> Anthony Liguori<anthony@codemonkey.ws>  wrote:
>
>    
>>>> We need to have a common management interface for third party tools.
>>>>
>>>>          
>>>    QMP? :-)
>>>        
>> Only if QMP is compatible with libvirt.  I don't want a user to have to
>> choose between QMP and libvirt.
>>      
>   Why not? If all they want is a simple qemu session, they can use
> QMP directly, on the other hand if what they want is more complex,
> what's the problem of using a management API like libivrt?
>    

My point is that libvirt should not be a separate management API but 
effectively an add-on API that provides higher level features, better 
integration with Linux host services, etc.
You mentioned dynamic dispatch, but this is useful only for C clients right?

>>> If so, what C clients you expected beyond libvirt?
>>>        
>> Users want a C API.  I don't agree that libvirt is the only C interface
>> consumer out there.
>>      
>   Actually, I do agree. Maybe, we don't have other C consumers because they
> weren't crazy enough to parse the crap of the user Monitor (or they do,
> but for simple things).
>
>   One possible future client is perf, for example.
>
>   Here is my solution (actually it's not mine, you have suggested
> it some time ago): let's provide a convenient way for C clients to
> use QMP. That is, let's have an overly simple library which takes
> QDitcs, sends them to qemu through QMP and returns others QDicts.
>
>   Something like the _sketch_ below:
>
> // Open a connection
> int qmp_open(..., QDict **greeting);
>
> // Register a callback for async messages, BUT note that the async message
> // object is passed verbatim
> void qmp_async_mes_handler(..., void (*async_mes_handler)(QDict *mes));
>
> // Send a QMP command
> int qmp_send(..., const char *command, QDict *params, QDict **res);
>    

Yes, this is the core API.  It's missing a mechanism to create a 
QMPContext.  I'll also argue that we want a set of auto generated 
wrappers like:

QError *qmp_last_error(QMPContext *context);
int qmp_set_link(QMPContext *context, const char *name, bool up);

That's a thin wrapper around qmp_send().  You can autogenerate this 
fairly easy with an IDL like:

None:qmp_set_link:String:name,Boolean:up

Which is fairly easy to output using introspection on a QMP session.

>   Obviously that we'll need a QMPContext and maybe additional functions,
>    

And for a QMPContext, we need functions like:

/* use qemu's default advertisement mechanism to find a guest */
QMPContext *qmp_context_new_by_uuid(uuid_t uuid);
QMPContext *qmp_context_new_by_name(const char *name);

/* connect to an already established QMP session */
QMPContext *qmp_context_new_by_fd(int fd);
QMPContext *qmp_context_new_by_iops(QMPIOOps *ops);

libvirt could use qmp_context_new_by_iops() to implement a 
virQemuGetQMP().  It can support this either by having a second QMP 
connection or even by parsing the QMP traffic and relaying commands over 
it QMP session (which gives it a chance to snoop on anything it's 
interested in).  I think the former approach is less difficult technically.

We can certainly make it easier to create dynamic QMP sessions.

> but the two main ideas are:
>
> 1. We don't do management
>    

I really believe we need to stop thinking this way.  I'm not saying that 
qemu-devel is the place where we design virt-manager, but we ought to 
consider the whole stack as part of "we".

>> I really think what we want is for a libvirt user to be able to call
>> libqemu functions directly.  There shouldn't have to be libvirt specific
>> functions for every operation we expose.
>>      
>   Not sure if this is too crazy but, considering this user wants to
> use qemu features not implemented by libvirt yet, what about using both
> libqmp (above) and libvirt at the same time?
>    

Yes, that's *exactly* what I want.  Except I want to call it libqemu 
because qmp is an implementation detail.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Luiz Capitulino]

On Thu, 25 Mar 2010 08:55:42 -0500
Anthony Liguori <anthony@codemonkey.ws> wrote:

> On 03/25/2010 08:23 AM, Luiz Capitulino wrote:
> > On Wed, 24 Mar 2010 16:40:18 -0500
> > Anthony Liguori<anthony@codemonkey.ws>  wrote:
> >
> >    
> >>>> We need to have a common management interface for third party tools.
> >>>>
> >>>>          
> >>>    QMP? :-)
> >>>        
> >> Only if QMP is compatible with libvirt.  I don't want a user to have to
> >> choose between QMP and libvirt.
> >>      
> >   Why not? If all they want is a simple qemu session, they can use
> > QMP directly, on the other hand if what they want is more complex,
> > what's the problem of using a management API like libivrt?
> >    
> 
> My point is that libvirt should not be a separate management API but 
> effectively an add-on API that provides higher level features, better 
> integration with Linux host services, etc.

 Okay, I fully agree here.

> >>> If so, what C clients you expected beyond libvirt?
> >>>        
> >> Users want a C API.  I don't agree that libvirt is the only C interface
> >> consumer out there.
> >>      
> >   Actually, I do agree. Maybe, we don't have other C consumers because they
> > weren't crazy enough to parse the crap of the user Monitor (or they do,
> > but for simple things).
> >
> >   One possible future client is perf, for example.
> >
> >   Here is my solution (actually it's not mine, you have suggested
> > it some time ago): let's provide a convenient way for C clients to
> > use QMP. That is, let's have an overly simple library which takes
> > QDitcs, sends them to qemu through QMP and returns others QDicts.
> >
> >   Something like the _sketch_ below:
> >
> > // Open a connection
> > int qmp_open(..., QDict **greeting);
> >
> > // Register a callback for async messages, BUT note that the async message
> > // object is passed verbatim
> > void qmp_async_mes_handler(..., void (*async_mes_handler)(QDict *mes));
> >
> > // Send a QMP command
> > int qmp_send(..., const char *command, QDict *params, QDict **res);
> >    
> 
> Yes, this is the core API.  It's missing a mechanism to create a 
> QMPContext.  I'll also argue that we want a set of auto generated 
> wrappers like:

 Having the wrappers is one of the points we disagree, but as we have agreed
on starting with the core only, I don't see why keep arguing here.

 If, in the near feature, the need of having wrappers become evident
I'll be all for it (this statement is a bit dangerous though, as
this need can be subjective).

[...]

> > but the two main ideas are:
> >
> > 1. We don't do management
> >    
> 
> I really believe we need to stop thinking this way.  I'm not saying that 
> qemu-devel is the place where we design virt-manager, but we ought to 
> consider the whole stack as part of "we".

 Depends, if you mean that we should be involved with libvirt development,
than I completely agree.

 On the other hand, if you mean than qemu should provide its own management
API, than I tend to disagree. And I think this is a very important point of
the whole discussion, if you think this way I guess we should start a new
thread to collect feedback, listing pros and cons.

> >> I really think what we want is for a libvirt user to be able to call
> >> libqemu functions directly.  There shouldn't have to be libvirt specific
> >> functions for every operation we expose.
> >>      
> >   Not sure if this is too crazy but, considering this user wants to
> > use qemu features not implemented by libvirt yet, what about using both
> > libqmp (above) and libvirt at the same time?
> >    
> 
> Yes, that's *exactly* what I want.  Except I want to call it libqemu 
> because qmp is an implementation detail.

 libqemu is fine.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/24/2010 10:32 PM, Anthony Liguori wrote:
>
> So far, a libqemu.so with a flexible transport that could be used 
> directly by a libvirt user (ala cairo/gdk type interactions) seems 
> like the best solution to me.


libqemu.so would be a C API.  C is not the first choice for writing GUIs 
or management applications.  So it would need to be further wrapped.

We also need to allow qemu to control the display directly, without 
going through vnc.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Alexander Graf]

On 25.03.2010, at 07:37, Avi Kivity wrote:

> On 03/24/2010 10:32 PM, Anthony Liguori wrote:
>> 
>> So far, a libqemu.so with a flexible transport that could be used directly by a libvirt user (ala cairo/gdk type interactions) seems like the best solution to me.
> 
> 
> libqemu.so would be a C API.  C is not the first choice for writing GUIs or management applications.  So it would need to be further wrapped.
> 
> We also need to allow qemu to control the display directly, without going through vnc.

For the current functionality I tend to disagree. All that we need is an shm vnc extension that allows the GUI and qemu to not send image data over the wire, but only the dirtyness information.

As soon as we get to 3D things might start to look different.


Alex

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/25/2010 10:18 AM, Alexander Graf wrote:
>
>> libqemu.so would be a C API.  C is not the first choice for writing GUIs or management applications.  So it would need to be further wrapped.
>>
>> We also need to allow qemu to control the display directly, without going through vnc.
>>      
> For the current functionality I tend to disagree. All that we need is an shm vnc extension that allows the GUI and qemu to not send image data over the wire, but only the dirtyness information.
>    

It still means an extra copy.  I don't think we want to share the guest 
framebuffer (it includes offscreen bitmaps), so we'll need to copy it 
somewhere else.  It's even worse with qxl/spice where there is no 
framebuffer.

> As soon as we get to 3D things might start to look different.
>    

Very different.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/24/2010 12:17 AM, Avi Kivity wrote:
> On 03/23/2010 08:00 PM, Avi Kivity wrote:
>> On 03/23/2010 06:06 PM, Anthony Liguori wrote:
>>>> I thought the monitor protocol *was* our API. If not, why not?
>>>
>>> It is.  But our API is missing key components like guest 
>>> enumeration.  So the fundamental topic here is, do we introduce 
>>> these missing components to allow people to build directly to our 
>>> interface or do we make use of the functionality that libvirt 
>>> already provides if they can plumb our API directly to users.
>>>
>>
>> Guest enumeration is another API.
>>
>> Over the kvm call I suggested a qemu concentrator that would keep 
>> track of all running qemus, and would hand out monitor connections to 
>> users.  It can do the enumeration (likely using qmp).  Libvirt could 
>> talk to that, like it does with other hypervisors.
>>
>
> To elaborate
>
> qemud
>   - daemonaizes itself
>   - listens on /var/lib/qemud/guests for incoming guest connections
>   - listens on /var/lib/qemud/clients for incoming client connections
>   - filters access according to uid (SCM_CREDENTIALS)
>   - can pass a new monitor to client (SCM_RIGHTS)
>   - supports 'list' command to query running guests
>   - async messages on guest startup/exit

Then guests run with the wrong security context.

Regards,

Anthony Liguori

> qemu
>   - with -qemud option, connects to qemud (or maybe automatically?)
>
> qemudc
>   - command-line client, can access qemu human monitor
>

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/24/2010 02:19 PM, Anthony Liguori wrote:
>> qemud
>>   - daemonaizes itself
>>   - listens on /var/lib/qemud/guests for incoming guest connections
>>   - listens on /var/lib/qemud/clients for incoming client connections
>>   - filters access according to uid (SCM_CREDENTIALS)
>>   - can pass a new monitor to client (SCM_RIGHTS)
>>   - supports 'list' command to query running guests
>>   - async messages on guest startup/exit
>
>
> Then guests run with the wrong security context.

Why?  They run with the security context of whoever launched them (could 
be libvirtd).

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/24/2010 07:27 AM, Avi Kivity wrote:
> On 03/24/2010 02:19 PM, Anthony Liguori wrote:
>>> qemud
>>>   - daemonaizes itself
>>>   - listens on /var/lib/qemud/guests for incoming guest connections
>>>   - listens on /var/lib/qemud/clients for incoming client connections
>>>   - filters access according to uid (SCM_CREDENTIALS)
>>>   - can pass a new monitor to client (SCM_RIGHTS)
>>>   - supports 'list' command to query running guests
>>>   - async messages on guest startup/exit
>>
>>
>> Then guests run with the wrong security context.
>
> Why?  They run with the security context of whoever launched them 
> (could be libvirtd).

Because it doesn't have the same security context as qemud and since 
clients have to connect to qemud, qemud has to implement access control.

It's far better to have the qemu instance advertise itself such that and 
client connects directly to it.  Then all of the various authorization 
models will be applied correctly to it.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/24/2010 02:30 PM, Anthony Liguori wrote:
> On 03/24/2010 07:27 AM, Avi Kivity wrote:
>> On 03/24/2010 02:19 PM, Anthony Liguori wrote:
>>>> qemud
>>>>   - daemonaizes itself
>>>>   - listens on /var/lib/qemud/guests for incoming guest connections
>>>>   - listens on /var/lib/qemud/clients for incoming client connections
>>>>   - filters access according to uid (SCM_CREDENTIALS)
>>>>   - can pass a new monitor to client (SCM_RIGHTS)
>>>>   - supports 'list' command to query running guests
>>>>   - async messages on guest startup/exit
>>>
>>>
>>> Then guests run with the wrong security context.
>>
>> Why?  They run with the security context of whoever launched them 
>> (could be libvirtd).
>
> Because it doesn't have the same security context as qemud and since 
> clients have to connect to qemud, qemud has to implement access control.

Yeah.

> It's far better to have the qemu instance advertise itself such that 
> and client connects directly to it.  Then all of the various 
> authorization models will be applied correctly to it.

Agreed.  qemud->exit().

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Daniel P. Berrange]

On Tue, Mar 23, 2010 at 11:06:20AM -0500, Anthony Liguori wrote:
> On 03/23/2010 10:57 AM, Paul Brook wrote:
> >>>I think there is a serious divergence of approach there, instanciating
> >>>API stating 'we are gonna deprecate them sooner or later' tell the
> >>>application developper 'my time is more important than yours' and not
> >>>really something I like to carry to the API users.
> >>>The main goal of libvirt remains to provide APIs needed to unify the
> >>>development of the virtualization layers. Having APIs which makes
> >>>sense only for one or 2 virtualization engines is not a problem in
> >>>itself, it just raises questions about the actual semantic of that API.
> >>>If that semantic is sound, then I see no reason to not add it, really
> >>>and we actually often do.
> >>>       
> >>Yeah, but the problem we're facing is, I want there to be an API added
> >>to the management layer as part of the feature commit in qemu.  If there
> >>has to be a discussion and decisions about how to model the API, it's
> >>not going to be successful.
> >>     
> >I thought the monitor protocol *was* our API. If not, why not?
> 
> It is.  But our API is missing key components like guest enumeration.  
> So the fundamental topic here is, do we introduce these missing 
> components to allow people to build directly to our interface or do we 
> make use of the functionality that libvirt already provides if they can 
> plumb our API directly to users.

There's two levels of API here

 - VM level API - essentially APIs for the QMP protocol & qdev ARGV format

 - Host level API - guest enumeration, integration with other OS services
                    like cgroups, selinux, etc

QEMU has historically only cared about the per-VM level, but has not
actually provided any formal library APIs even for the monitor protocol
or command line syntax.

libvirt has obviously focused on the host level APIs, and directly figured
out the implicit VM level "API" that was exposed from QEMU. I think this is
a good split to maintain, because when you get to the host level API you 
start interacting / integrating beyond just QEMU with OS services like
cgroups, selinux, iptables, host networking, etc. QEMU might start with
a simple daemon for enumerating VMs....but that's how libvirt's QEMU driver
started off. Over time that "simple" demon would grow to end up doing all 
the things that libvirt currently does. This duplication of functionality
doesn't seem like a good use of development resources to me. 

Now libvirt does not currently directly expose the two VM level APIs that
QEMU has (qdev ARGV, and QMP protocol), which is where our feature timelag
comes from. If we can figure out a way to expose those two, then there 
shouldn't be a need for QEMU to get into duplicate host-level APIs like
enumeration.

There could still be useful APIs that QEMU can expose those. For example,
consider if QEMU provided 

 - libqmp.so - API(s) for each monitor command that serialized to/from JSON
               format string
 - libqdev.so - API(s) for constructing qdev strings, that can then be used
                as ARGV values, or QMP parameter values.

Next consider if libvirt provided a way to pass extra ARGV down to QEMU, 
and also provided a way to send/recv JSON commands/events.

The libqmp.so & libqdev.so could then be used both directly against a single
QEMU process spawned manually, but also indirectly via libvirt. eg, to use 
snapshots with libvirt, an app would use libqmp.so to generate a QMP command
for snapshotting, send it to the VM via the libvirt API for monitor injection
and get the response. The nice aspect of this is that libvirt is actually
adding value to libqmp.so, because users would now have secure remote
access to the QEMU monitor, tunnelled via libvirtd. It also avoids adding an
arbitrary number of extra APIs to libvirt - we just need the API to send
and recv JSON in libvirt, and libqmp.so can then be used ontop of that

In ascii art you'd end up with two models

Interacting with a single VM directly:

  Application   -> libqmp.so/libqdev.so -> QEMU

Interacting with many VMs via libvirt

  Application -------------> libvirt API ----> libvirtd ----> QEMU
      |                       ^
      |                       |
      +--------> libqmp.so ---+
      +--------> libqdev.so --+

So primarily an app would still use libvirt as the host level management
API, but libqmp.so proxied via libvirt would also allow access to arbitrary
extra features. This avoids the big overlap in functionality between libvirt
& QEMU apis, which would occurr if QEMU started doing multiple VM mgmt too.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/23/2010 01:07 PM, Daniel P. Berrange wrote:
> On Tue, Mar 23, 2010 at 11:06:20AM -0500, Anthony Liguori wrote:
>    
>> On 03/23/2010 10:57 AM, Paul Brook wrote:
>>      
>>>>> I think there is a serious divergence of approach there, instanciating
>>>>> API stating 'we are gonna deprecate them sooner or later' tell the
>>>>> application developper 'my time is more important than yours' and not
>>>>> really something I like to carry to the API users.
>>>>> The main goal of libvirt remains to provide APIs needed to unify the
>>>>> development of the virtualization layers. Having APIs which makes
>>>>> sense only for one or 2 virtualization engines is not a problem in
>>>>> itself, it just raises questions about the actual semantic of that API.
>>>>> If that semantic is sound, then I see no reason to not add it, really
>>>>> and we actually often do.
>>>>>
>>>>>            
>>>> Yeah, but the problem we're facing is, I want there to be an API added
>>>> to the management layer as part of the feature commit in qemu.  If there
>>>> has to be a discussion and decisions about how to model the API, it's
>>>> not going to be successful.
>>>>
>>>>          
>>> I thought the monitor protocol *was* our API. If not, why not?
>>>        
>> It is.  But our API is missing key components like guest enumeration.
>> So the fundamental topic here is, do we introduce these missing
>> components to allow people to build directly to our interface or do we
>> make use of the functionality that libvirt already provides if they can
>> plumb our API directly to users.
>>      
> There's two levels of API here
>
>   - VM level API - essentially APIs for the QMP protocol&  qdev ARGV format
>
>   - Host level API - guest enumeration, integration with other OS services
>                      like cgroups, selinux, etc
>
> QEMU has historically only cared about the per-VM level, but has not
> actually provided any formal library APIs even for the monitor protocol
> or command line syntax.
>    

We also provide an API for guest creation (the qemu command line).  When 
we create a guest, we don't integrate with things like cgroups and 
selinux and we probably never will.  This is a place where libvirt adds 
value.

The fundamental problem we have is that once you create a qemu instance, 
you cannot find it from a third party tool.  That's a problem we ought 
to solve and I'd like to see that be common across qemu and libvirt.  I 
don't see that as us growing our scope into libvirt's space.

I think libvirt does two things.  It provides a generic interface to 
hypervisors and if people write to this interface, they get better 
portability and the ability to management many platforms.  It also 
provides a certain amount of host services management that can include 
things not directly related to qemu (like network management) and 
services that further connect qemu to host services (like selinux 
labelling).

What I would like to see is that a user can write to the libvirt API and 
then call out to qemu specific functions when necessary.  I'd also like 
a user be able to interact directly with qemu without using the libvirt 
generic API.  The user should be able to still see the VMs and 
ultimately interact with them through libvirt.  The user should be able 
to use libvirt to deal with host services too (like storage and network 
pools).

The key is not to have two mutual exclusive management mechanisms but a 
set of complementary APIs.  The biggest obstacle I see is libvirt's 
remote management interface.  I think it's addressable though.  For 
instance, if libqemu.so provided a QMP IO interface, libvirt-qemu could 
basically provide an interface to create that context and otherwise have 
users use the libqemu.so interfaces directly.

IOW, libqemu.so would provide interfaces that looked like:

QMPContext *qemu_connect_by_name(const char *name);
int qemu_pci_add(QMPContext *ctxt, ...);

And libvirt would provide interfaces that looked like:

virQemuPtr *virDomainGetQemuPtr(virDomainPtr *ptr);
QMPContext *virQemuCreateQMPContext(virQemuPtr *ptr);

With respect to keep tracking of which operations are done through qemu, 
we should discuss the technical challenges of this and figure out how we 
can solve them.

> The libqmp.so&  libqdev.so could then be used both directly against a single
> QEMU process spawned manually, but also indirectly via libvirt.

That's not quite what I'm looking for because then it's really two 
separate interfaces.  I'd rather see complementary interfaces much like 
how Cairo integrates with GTK/GDK or even how GDK integrates with X11.

Regards,

Anthony Liguori

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/23/2010 09:24 PM, Anthony Liguori wrote:
>
> We also provide an API for guest creation (the qemu command line).


As an aside, I'd like to see all command line options have qmp 
equivalents (most of them can be implemented with a 'set' command that 
writes qdev values).  This allows a uniform way to control a guest, 
whether at startup or runtime.  You start with a case, cold-plug a 
motherboard, cpus, memory, disk controllers, and power it on.

I would also like a way to read the entire qdev tree from qmp.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Paul Brook]

> On 03/23/2010 09:24 PM, Anthony Liguori wrote:
> > We also provide an API for guest creation (the qemu command line).
> 
> As an aside, I'd like to see all command line options have qmp
> equivalents (most of them can be implemented with a 'set' command that
> writes qdev values).  This allows a uniform way to control a guest,
> whether at startup or runtime.  You start with a case, cold-plug a
> motherboard, cpus, memory, disk controllers, and power it on.

The main blocker to this is converting all the devices to qdev. "partial" 
conversions are not sufficient. It's approximately the same problem as a 
machine config file. If you have one then the other should be fairly trivial.
IMO the no_user flag is a bug, and should not exist.

Paul

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/24/2010 02:30 PM, Paul Brook wrote:
>> On 03/23/2010 09:24 PM, Anthony Liguori wrote:
>>      
>>> We also provide an API for guest creation (the qemu command line).
>>>        
>> As an aside, I'd like to see all command line options have qmp
>> equivalents (most of them can be implemented with a 'set' command that
>> writes qdev values).  This allows a uniform way to control a guest,
>> whether at startup or runtime.  You start with a case, cold-plug a
>> motherboard, cpus, memory, disk controllers, and power it on.
>>      
> The main blocker to this is converting all the devices to qdev. "partial"
> conversions are not sufficient. It's approximately the same problem as a
> machine config file. If you have one then the other should be fairly trivial.
>    

Agreed.

> IMO the no_user flag is a bug, and should not exist.
>    

Sorry, what's that?

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Paul Brook]

> > IMO the no_user flag is a bug, and should not exist.
> 
> Sorry, what's that?

Usually an indication that a device has been incorrectly or inproperly 
converted to the qdev interface.

Paul

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Markus Armbruster]

Paul Brook <paul@codesourcery.com> writes:

>> > IMO the no_user flag is a bug, and should not exist.
>> 
>> Sorry, what's that?
>
> Usually an indication that a device has been incorrectly or inproperly 
> converted to the qdev interface.

Can also be an indication that the device can't support multiple
instances.  For instance:

    commit 39a51dfda835a75c0ebbfd92705b96e4de77f795
    Author: Markus Armbruster <armbru@redhat.com>
    Date:   Tue Oct 27 13:52:13 2009 +0100

        qdev: Tag isa-fdc, PIIX3 IDE and PIIX4 IDE as no-user

        These devices are created automatically, and attempting to create
        another one with -device fails with "qemu: hardware error:
        register_ioport_write: invalid opaque".

        Signed-off-by: Markus Armbruster <armbru@redhat.com>
        Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>

With no-user, we at least fail with a decent error message.

I don't think it's fair to demand that a qdev conversion must relax
restrictions that haven't otherwise bothered us to be correct and
proper.  We'll relax them if and when they bother us enough to make
somebody send a decent patch.

And yes, there are better ways to disallow multiple instances of a
device than declaring it no-user.  Patches welcome.

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Paul Brook]

> Paul Brook <paul@codesourcery.com> writes:
> >> > IMO the no_user flag is a bug, and should not exist.
> >>
> >> Sorry, what's that?
> >
> > Usually an indication that a device has been incorrectly or inproperly
> > converted to the qdev interface.
> 
> Can also be an indication that the device can't support multiple
> instances.  For instance:
>        qdev: Tag isa-fdc, PIIX3 IDE and PIIX4 IDE as no-user

I still claim this is a bug, and see no good reason why we shouldn't support 
multiple floppy controllers, ISA busses, etc.

Paul

Re: [Qemu-devel] Re: [libvirt] Supporting hypervisor specific APIs in libvirt

[Jamie Lokier]

Anthony Liguori wrote:
> On 03/23/2010 10:57 AM, Paul Brook wrote:
> >I thought the monitor protocol *was* our API. If not, why not?
> 
> It is.  But our API is missing key components like guest enumeration.

Is that simply enumerating running qemu instances, and asking each one
about things like it's name, VNC port, etc.?

Having each qemu publish itself through D-Bus or Avahi (to find the
list of running instances), and every info query go through the
monitor, would seem a clean solution to that.

Are there any other missing key components?

-- Jamie

Re: [Qemu-devel] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/22/2010 09:25 PM, Anthony Liguori wrote:
> Hi,
>
> I've mentioned this to a few folks already but I wanted to start a 
> proper thread.
>
> We're struggling in qemu with usability and one area that concerns me 
> is the disparity in features that are supported by qemu vs what's 
> implemented in libvirt.
>
> This isn't necessarily libvirt's problem if it's mission is to provide 
> a common hypervisor API that covers the most commonly used features.
>
> However, for qemu, we need an API that covers all of our features that 
> people can develop against.  The ultimate question we need to figure 
> out is, should we encourage our users to always use libvirt or should 
> we build our own API for people (and libvirt) to consume.
>
> I don't think it's necessarily a big technical challenge for libvirt 
> to support qemu more completely.  I think it amounts to introducing a 
> series of virQemuXXXX APIs that implement qemu specific functions.  
> Over time, qemu specific APIs can be deprecated in favour of more 
> generic virDomain APIs.
>
> What's the feeling about this from the libvirt side of things?  Is 
> there interest in support hypervisor specific interfaces should we be 
> looking to provide our own management interface for libvirt to consume?
>

One option is to expose a qmp connection to the client.  Of course that 
introduces a consistency problem (libvirt plugs in a card, user plugs it 
own, libvirt is confused).  If the user promises to behave, it can work 
for stuff that's 100% orthogonal to libvirt.

One problem is that this is libvirt version specific.  For example, 
libvirt x doesn't support spice so we control that thorough qmp.  But 
libvirt x+1 does support spice and now it gets confused about all the 
spice messages.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Supporting hypervisor specific APIs in libvirt

[Anthony Liguori]

On 03/23/2010 12:57 PM, Avi Kivity wrote:
> On 03/22/2010 09:25 PM, Anthony Liguori wrote:
>> Hi,
>>
>> I've mentioned this to a few folks already but I wanted to start a 
>> proper thread.
>>
>> We're struggling in qemu with usability and one area that concerns me 
>> is the disparity in features that are supported by qemu vs what's 
>> implemented in libvirt.
>>
>> This isn't necessarily libvirt's problem if it's mission is to 
>> provide a common hypervisor API that covers the most commonly used 
>> features.
>>
>> However, for qemu, we need an API that covers all of our features 
>> that people can develop against.  The ultimate question we need to 
>> figure out is, should we encourage our users to always use libvirt or 
>> should we build our own API for people (and libvirt) to consume.
>>
>> I don't think it's necessarily a big technical challenge for libvirt 
>> to support qemu more completely.  I think it amounts to introducing a 
>> series of virQemuXXXX APIs that implement qemu specific functions.  
>> Over time, qemu specific APIs can be deprecated in favour of more 
>> generic virDomain APIs.
>>
>> What's the feeling about this from the libvirt side of things?  Is 
>> there interest in support hypervisor specific interfaces should we be 
>> looking to provide our own management interface for libvirt to consume?
>>
>
> One option is to expose a qmp connection to the client.  Of course 
> that introduces a consistency problem (libvirt plugs in a card, user 
> plugs it own, libvirt is confused).  If the user promises to behave, 
> it can work for stuff that's 100% orthogonal to libvirt.

With GTK/GDK/Cairo/etc, the expectation is that users behave.  For 
things like GDK, most people never break out to X11 unless they absolute 
have to.  The Cairo API is nice enough though that GDK doesn't even 
attempt to introduce wrappers.  If you muck with X11, accessing the same 
info with GDK might prove harmful.

>
> One problem is that this is libvirt version specific.  For example, 
> libvirt x doesn't support spice so we control that thorough qmp.  But 
> libvirt x+1 does support spice and now it gets confused about all the 
> spice messages.

That's only a problem if we only support a single QMP session.  This is 
exactly why we need to support multiple QMP sessions (and do).

Regards,

Anthony Liguori

Re: [Qemu-devel] Supporting hypervisor specific APIs in libvirt

[Avi Kivity]

On 03/23/2010 09:31 PM, Anthony Liguori wrote:
>
>>
>> One problem is that this is libvirt version specific.  For example, 
>> libvirt x doesn't support spice so we control that thorough qmp.  But 
>> libvirt x+1 does support spice and now it gets confused about all the 
>> spice messages.
>
> That's only a problem if we only support a single QMP session.  This 
> is exactly why we need to support multiple QMP sessions (and do).

It's unrelated to the number of sessions.  libvirt expects state that it 
manages in qemu not to change randomly.  Users know that, so they will 
only manage non-libvirt state in their private session.  But a new 
version of libvirt may expand its scope and start managing this area, 
leading to conflicts.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] Supporting hypervisor specific APIs in libvirt

[Jamie Lokier]

Avi Kivity wrote:
> On 03/23/2010 09:31 PM, Anthony Liguori wrote:
> >
> >>
> >>One problem is that this is libvirt version specific.  For example, 
> >>libvirt x doesn't support spice so we control that thorough qmp.  But 
> >>libvirt x+1 does support spice and now it gets confused about all the 
> >>spice messages.
> >
> >That's only a problem if we only support a single QMP session.  This 
> >is exactly why we need to support multiple QMP sessions (and do).
> 
> It's unrelated to the number of sessions.  libvirt expects state that it 
> manages in qemu not to change randomly.  Users know that, so they will 
> only manage non-libvirt state in their private session.  But a new 
> version of libvirt may expand its scope and start managing this area, 
> leading to conflicts.

We have events when state changes, for libvirt to be informed.

But that's prone to race conditions.

A fairly universal solution to that is "pre"-events, where something
that's going to change state emits a pre-event, waits for an ack from
all listeners, then proceeds or not according to the acks, then
finally emits the post-event to say the state is changed.  Between pre
and post events, as far as the listener is concerned, the state is
uncertain and it must query or wait when it needs the value.(*)

In other words, libvirt would register to listen to all the pre-events
for state that it cares about being synchronised with.  When libvirt
expands its scope, it would simply listen to some more.

-- Jamie

(*) Yes, this is like 2-phase transactions, and it's also a bit like
MESI caching.  But it's easier to understand than both :-)