Re: Enhance perf to support KVM

Re: Enhance perf to support KVM

[Zhang, Yanmin]

On Thu, 2010-02-25 at 10:20 +0100, Peter Zijlstra wrote:
> On Thu, 2010-02-25 at 11:27 +0800, Zhang, Yanmin wrote:
> > Ingo,
> > 
> > I did some testing with KVM virtualization. perf shows vmx_vcpu_run
> > consumes more than 50% cpu time. Actually, the info is incorrect because
> > when perf counter overflows and NMI is triggered, vm exit to function
> > vmx_vcpu_run, then vmx_vcpu_run triggers a software NMI so perf event is
> > notified. perf just checks regs which just saves the address of vmx_vcpu_run.
> > 
> > I want to enhance perf to collect real guest os address.
> > 
> > Below is the design.
> > KVM uses multi-thread model. Every guest os is a process of multi-thread.
> > 
> > 1) Kernel:
> > 	Add a per_cpu var and some functions, so KVM records interrupted
> > guest os address before triggering the software NMI. perf event would check
> > the per_cpu var to use it if it's not zero, or just goes though the old path.
> > 
> > 2) User space: Add a new parameter to perf-top and perf-report, such like
> > -g pid:guest_os_vmlinux_path. Command perf parses the guest os kernel image
> > to collect symbols. Change perf to summarize results based on pid.
> > Another direction is to use the new parameter -g only when old parameter
> > -p is defined. Perf just needs separate native kernel and guest os kernel.
> 
I really appreciate your kind comments, and will contact you again in
the future for help.

> -g is already taken :-)
We could use other flag or just -G.

> 
> One thing I worry about is making sense of the guest data, it might be
> possible to sorta make sense of the main kernel image, but after that
> its going to be 'interesting' in deed.
> 
> You're going to have to extend PERF_RECORD_MISC_* though, perhaps you
> can reuse CPUMODE_UNKNOWN for GUEST.
> 
> The callchain stuff already has GUEST context identifiers, however
> determining KERNEL/USER context might be hard and interpreting it is
> going to be harder still since we don't have map information for the
> guest.
Right. As for side #1 pointed in Ingo' email, we assume guest os is
linux. We couldn't support all capabilities of perf on KVM from host side.
1) We couldn't get module and process mapping info in guest os in an easy way,
so we can't support to collect guest kernel module and user space hot functions.
A work around is user could get guest os /proc/kallsym and pass it to tool perf
at host side so we could analyze module host functions.
2) We couldn't get guest os kernel/user stack data in an easy way, so we might not
support callchain feature of tool perf. A work around is KVM copies kernel stack
data out, so we could at least support guest os kernel callchain.

So the host side perf support on guest os:
  perf kvm list
  perf kvm record            # records the first running guest
  perf kvm stat              # stats the first running KVM guest
  perf kvm top               # shows the profile of the first running
guest
  perf kvm trace             # active the KVM specific tracepoints

As for record, doesn't support to record guest os user space stack
callchain and guest os user space hot functions.

Yanmin

Re: Enhance perf to support KVM

[Ingo Molnar]

* Zhang, Yanmin <yanmin_zhang@linux.intel.com> wrote:

> 2) We couldn't get guest os kernel/user stack data in an easy way, so we 
> might not support callchain feature of tool perf. A work around is KVM 
> copies kernel stack data out, so we could at least support guest os kernel 
> callchain.

If the guest is Linux, KVM can get all the info we need.

While the PMU event itself might trigger in an NMI (where we cannot access 
most of KVM's data structures safely), for this specific case of KVM 
instrumentation we can delay the processing to a more appropriate time - in 
fact we can do it in the KVM thread itself.

We can do that because we just triggered a VM exit, so the VM state is for all 
purposes frozen (as far as this virtual CPU goes).

Which egives us plenty of time and opportunity to piggy back to the KVM 
thread, look up the guest stack, process/fill the MMU cache as we walk the 
guest page tables, etc. etc.

It would need some minimal callback facility towards KVM, triggered by a perf 
event PMI.

One additional step needed is to get symbol information from the guest, and to 
integrate it into the symbol cache on the host side in ~/.debug. We already 
support cross-arch symbols and 'perf archive', so the basic facilities are 
there for that. So you can profile on 32-bit PA-RISC and type 'perf report' on 
64-bit x86 and get all the right info.

For this to work across a guest, a gateway is needed towards the guest. 
There's several ways to achieve this. The most practical would be two steps:

 - a user-space facility to access guest images/libraries. (say via ssh, or 
   just a plain TCP port) This would be useful for general 'remote profiling' 
   sessions as well, so it's not KVM specific - it would be useful for remote 
   debugging.

 - The guest /proc/kallsyms (and vmlinux) could be accessed via that channel 
   as well.

(Note that this is purely for guest symbol space access - all the profiling 
data itself comes via the host kernel.)

In theory we could build some sort of 'symbol server' facility into the 
kernel, which could be enabled in guest kernels too - but i suspect existing, 
user-space transports go most of the way already. (the only disadvantage of 
existing transports is that they all have to be configured, enabled and made 
user-accessible, which is one of the few weak points of KVM in general.)

Thanks,

	Ingo

Re: Enhance perf to support KVM

[Avi Kivity]

On 02/26/2010 11:01 AM, Ingo Molnar wrote:
> * Zhang, Yanmin<yanmin_zhang@linux.intel.com>  wrote:
>
>    
>> 2) We couldn't get guest os kernel/user stack data in an easy way, so we
>> might not support callchain feature of tool perf. A work around is KVM
>> copies kernel stack data out, so we could at least support guest os kernel
>> callchain.
>>      
> If the guest is Linux, KVM can get all the info we need.
>
> While the PMU event itself might trigger in an NMI (where we cannot access
> most of KVM's data structures safely), for this specific case of KVM
> instrumentation we can delay the processing to a more appropriate time - in
> fact we can do it in the KVM thread itself.
>    

The nmi will be a synchronous event: it happens in guest context, and we 
program the hardware to intercept nmis, so we just get an exit telling 
us that an nmi has happened.

(would also be interesting to allow the guest to process the nmi 
directly in some scenarios, though that would require that there be no 
nmi sources on the host).

> We can do that because we just triggered a VM exit, so the VM state is for all
> purposes frozen (as far as this virtual CPU goes).
>    

Yes.

> Which egives us plenty of time and opportunity to piggy back to the KVM
> thread, look up the guest stack, process/fill the MMU cache as we walk the
> guest page tables, etc. etc.
>
> It would need some minimal callback facility towards KVM, triggered by a perf
> event PMI.
>    

Since the event is synchronous and kvm is aware of it we don't need a 
callback; kvm can call directly into perf with all the information.

> One additional step needed is to get symbol information from the guest, and to
> integrate it into the symbol cache on the host side in ~/.debug. We already
> support cross-arch symbols and 'perf archive', so the basic facilities are
> there for that. So you can profile on 32-bit PA-RISC and type 'perf report' on
> 64-bit x86 and get all the right info.
>
> For this to work across a guest, a gateway is needed towards the guest.
> There's several ways to achieve this. The most practical would be two steps:
>
>   - a user-space facility to access guest images/libraries. (say via ssh, or
>     just a plain TCP port) This would be useful for general 'remote profiling'
>     sessions as well, so it's not KVM specific - it would be useful for remote
>     debugging.
>
>   - The guest /proc/kallsyms (and vmlinux) could be accessed via that channel
>     as well.
>
> (Note that this is purely for guest symbol space access - all the profiling
> data itself comes via the host kernel.)
>
> In theory we could build some sort of 'symbol server' facility into the
> kernel, which could be enabled in guest kernels too - but i suspect existing,
> user-space transports go most of the way already.

There is also vmchannel aka virtio-serial, a guest-to-host communication 
channel.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Enhance perf to support KVM

[Ingo Molnar]

* Avi Kivity <avi@redhat.com> wrote:

> On 02/26/2010 11:01 AM, Ingo Molnar wrote:
> >* Zhang, Yanmin<yanmin_zhang@linux.intel.com>  wrote:
> >
> >>2) We couldn't get guest os kernel/user stack data in an easy way, so we
> >>might not support callchain feature of tool perf. A work around is KVM
> >>copies kernel stack data out, so we could at least support guest os kernel
> >>callchain.
> >If the guest is Linux, KVM can get all the info we need.
> >
> >While the PMU event itself might trigger in an NMI (where we cannot access
> >most of KVM's data structures safely), for this specific case of KVM
> >instrumentation we can delay the processing to a more appropriate time - in
> >fact we can do it in the KVM thread itself.
> 
> The nmi will be a synchronous event: it happens in guest context,
> and we program the hardware to intercept nmis, so we just get an
> exit telling us that an nmi has happened.
> 
> (would also be interesting to allow the guest to process the nmi
> directly in some scenarios, though that would require that there be
> no nmi sources on the host).
> 
> >We can do that because we just triggered a VM exit, so the VM state is for all
> >purposes frozen (as far as this virtual CPU goes).
> 
> Yes.
> 
> >Which egives us plenty of time and opportunity to piggy back to the KVM
> >thread, look up the guest stack, process/fill the MMU cache as we walk the
> >guest page tables, etc. etc.
> >
> >It would need some minimal callback facility towards KVM, triggered by a perf
> >event PMI.
> 
> Since the event is synchronous and kvm is aware of it we don't need
> a callback; kvm can call directly into perf with all the
> information.

Yes - it's still a "callback" in the abstract sense. Much of it already all 
existing.

> >One additional step needed is to get symbol information from the guest, and to
> >integrate it into the symbol cache on the host side in ~/.debug. We already
> >support cross-arch symbols and 'perf archive', so the basic facilities are
> >there for that. So you can profile on 32-bit PA-RISC and type 'perf report' on
> >64-bit x86 and get all the right info.
> >
> >For this to work across a guest, a gateway is needed towards the guest.
> >There's several ways to achieve this. The most practical would be two steps:
> >
> >  - a user-space facility to access guest images/libraries. (say via ssh, or
> >    just a plain TCP port) This would be useful for general 'remote profiling'
> >    sessions as well, so it's not KVM specific - it would be useful for remote
> >    debugging.
> >
> >  - The guest /proc/kallsyms (and vmlinux) could be accessed via that channel
> >    as well.
> >
> >(Note that this is purely for guest symbol space access - all the profiling
> >data itself comes via the host kernel.)
> >
> >In theory we could build some sort of 'symbol server' facility into the
> >kernel, which could be enabled in guest kernels too - but i suspect existing,
> >user-space transports go most of the way already.
> 
> There is also vmchannel aka virtio-serial, a guest-to-host communication 
> channel.

Basically what is needed is plain filesystem access - properly privileged. So 
doing this via a vmchannel would be nice, but for the symbol extraction it 
would be a glorified NFS server in essence.

Do you have (or plan) any turn-key 'access to all files of the guest' kind of 
guest-transparent facility that could be used for such purposes? That would 
have various advantages over a traditional explicit file server approach:

 - it would not contaminate the guest port space

 - no guest side configuration needed (the various oprofile remote daemons 
   always sucked as they needed extra setup)

 - it might even be used with a guest that does no networking

 - if done fully in the kernel it could be done with a fully 'unaware' guest, etc.

Thanks,

	Ingo

Re: Enhance perf to support KVM

[Avi Kivity]

On 02/26/2010 12:35 PM, Ingo Molnar wrote:
>
>>> One additional step needed is to get symbol information from the guest, and to
>>> integrate it into the symbol cache on the host side in ~/.debug. We already
>>> support cross-arch symbols and 'perf archive', so the basic facilities are
>>> there for that. So you can profile on 32-bit PA-RISC and type 'perf report' on
>>> 64-bit x86 and get all the right info.
>>>
>>> For this to work across a guest, a gateway is needed towards the guest.
>>> There's several ways to achieve this. The most practical would be two steps:
>>>
>>>   - a user-space facility to access guest images/libraries. (say via ssh, or
>>>     just a plain TCP port) This would be useful for general 'remote profiling'
>>>     sessions as well, so it's not KVM specific - it would be useful for remote
>>>     debugging.
>>>
>>>   - The guest /proc/kallsyms (and vmlinux) could be accessed via that channel
>>>     as well.
>>>
>>> (Note that this is purely for guest symbol space access - all the profiling
>>> data itself comes via the host kernel.)
>>>
>>> In theory we could build some sort of 'symbol server' facility into the
>>> kernel, which could be enabled in guest kernels too - but i suspect existing,
>>> user-space transports go most of the way already.
>>>        
>> There is also vmchannel aka virtio-serial, a guest-to-host communication
>> channel.
>>      
> Basically what is needed is plain filesystem access - properly privileged. So
> doing this via a vmchannel would be nice, but for the symbol extraction it
> would be a glorified NFS server in essence.
>    

Well, we could run an nfs server over vmchannel, or over a private 
network interface.

> Do you have (or plan) any turn-key 'access to all files of the guest' kind of
> guest-transparent facility that could be used for such purposes?

Not really.  The guest and host admins are usually different people, who 
may, being admins, even actively hate each other.  The guest admin would 
probably regard it as a security hole.  It's probably useful for the 
single-host scenario, and of course for developers.

I guess sshfs can fill this role, with one command it gives you secure 
access to all guest files, provided you have the proper credentials.

> That would
> have various advantages over a traditional explicit file server approach:
>
>   - it would not contaminate the guest port space
>
>   - no guest side configuration needed (the various oprofile remote daemons
>     always sucked as they needed extra setup)
>
>   - it might even be used with a guest that does no networking
>
>   - if done fully in the kernel it could be done with a fully 'unaware' guest, etc.
>    

Seems sshfs fulfils the first two.  For the latter, we could do a 
vmchannelfs, but it seems quite a bit of work, and would require fairly 
new guest kernels, whereas sshfs would work out of the box on 10 year 
old guests and can be easily made to work on Windows.

Somewhat related, see libguestfs/guestfish, though that provides offline 
access only.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Enhance perf to support KVM

[Ingo Molnar]

* Avi Kivity <avi@redhat.com> wrote:

> > Do you have (or plan) any turn-key 'access to all files of the guest' kind 
> > of guest-transparent facility that could be used for such purposes?
> 
> Not really.  The guest and host admins are usually different people, who 
> may, being admins, even actively hate each other.  The guest admin would 
> probably regard it as a security hole.  It's probably useful for the 
> single-host scenario, and of course for developers.

Sounds like an exceedingly silly argument to me - the host admin is the king 
in any case.

Your argument boils down to: 'dont offer transparent, turn-key solutions 
because some might object to the functionality they offer for all the wrong 
reasons'. Which does not withstand elementary scrutiny.

This is a basic usability issue, and affects many parts of the KVM universe.

Really, it's by far the most fubar-ed notion of KVM. You are pushing _way_ too 
much to user-space into different modules and maintenance domains, and 
user-space forks those bits, fragments, diverts, delays and messes up basic 
features in the usual fashion.

The result is a basic out-of-box virtualization experience that sucks even 
these days.

Nobody is really 'in charge' of how KVM gets delivered to the user. You 
isolated the fun kernel part for you and pushed out the boring bits to 
user-space. So if mundane things like mouse integration sucks 'hey that's a 
user-space tooling problem', if file integration sucks then 'hey, that's an 
admin problem', if it cannot be used over the network 'hey, that's an Xorg 
problem', etc. etc.

You basically have given up control over the quality of KVM by pushing so many 
aspects of it to user-space and letting it rot there.

Sure the design looks somewhat cleaner on paper, but if the end result is not 
helped by it then over-modularization sure can hurt ...

( Note that i dont mind user-space tooling per se, as long as it sits together 
  with the kernel bits and gets developed, packaged and given to the user in 
  the same domain. )

And that's a key conceptual area were tools/perf/ differs: it's an integrated, 
turn-key solution that you can really rely on. We take responsibility for the 
full thing, no ifs and when. And if you cannot rely on your instrumentation 
tooling as a single unit you cannot use it, simple as that. (that is a key 
mistake Oprofile made a decade ago too btw.)

So i can see some upcoming culture friction with standing KVM principles there 
;-)

	Ingo

Re: Enhance perf to support KVM

[Avi Kivity]

On 02/26/2010 01:17 PM, Ingo Molnar wrote:
> * Avi Kivity<avi@redhat.com>  wrote:
>
>    
>>> Do you have (or plan) any turn-key 'access to all files of the guest' kind
>>> of guest-transparent facility that could be used for such purposes?
>>>        
>> Not really.  The guest and host admins are usually different people, who
>> may, being admins, even actively hate each other.  The guest admin would
>> probably regard it as a security hole.  It's probably useful for the
>> single-host scenario, and of course for developers.
>>      
> Sounds like an exceedingly silly argument to me - the host admin is the king
> in any case.
>
> Your argument boils down to: 'dont offer transparent, turn-key solutions
> because some might object to the functionality they offer for all the wrong
> reasons'. Which does not withstand elementary scrutiny.
>    

Again, the host admin and the guest admin are different people.  What 
would the host admin do with guest files?  Why would the guest admin 
want to run any code that exposes their files?


> This is a basic usability issue, and affects many parts of the KVM universe.
>
> Really, it's by far the most fubar-ed notion of KVM. You are pushing _way_ too
> much to user-space into different modules and maintenance domains, and
> user-space forks those bits, fragments, diverts, delays and messes up basic
> features in the usual fashion.
>
> The result is a basic out-of-box virtualization experience that sucks even
> these days.
>    


> Nobody is really 'in charge' of how KVM gets delivered to the user. You
> isolated the fun kernel part for you and pushed out the boring bits to
> user-space. So if mundane things like mouse integration sucks 'hey that's a
> user-space tooling problem', if file integration sucks then 'hey, that's an
> admin problem', if it cannot be used over the network 'hey, that's an Xorg
> problem', etc. etc.
>    

What would you have me do?  Push 200K lines of device emulation code 
into the kernel?  Write an X client, toolkit, and display in the kernel 
so that mouse integration works out of the box when you install Linux 
2.6.653?

As to "nobody is in charge", that's really insulting to the people who 
are in charge of the userspace components.  Perhaps the problems that we 
see are not the same problems that you see.  It might be that direct 
access to guest files from the host is only a pressing problem for you, 
but nobody else.  If there are features that you miss, post patches, if 
you will deign to code for lowly user space.

> You basically have given up control over the quality of KVM by pushing so many
> aspects of it to user-space and letting it rot there.
>    

That's wrong on so many levels.  First, nothing is rotting in userspace, 
qemu is evolving faster than kvm is.  If I pushed it into the kernel 
then development pace would be much slower (since kernel development is 
harder), quality would be lower (less infrastructure, any bug is a host 
crash or security issue), and I personally would be totally swamped.

> Sure the design looks somewhat cleaner on paper, but if the end result is not
> helped by it then over-modularization sure can hurt ...
>    

Run 'rpm -qa' one of these days.  Modern software is modular, that's the 
only way to manage it.

> ( Note that i dont mind user-space tooling per se, as long as it sits together
>    with the kernel bits and gets developed, packaged and given to the user in
>    the same domain. )
>    

Call me when glibc, the X servers and clients, and everything else qemu 
now uses is developed, packaged, and given to the user in the same domain.

> And that's a key conceptual area were tools/perf/ differs: it's an integrated,
> turn-key solution that you can really rely on. We take responsibility for the
> full thing, no ifs and when. And if you cannot rely on your instrumentation
> tooling as a single unit you cannot use it, simple as that. (that is a key
> mistake Oprofile made a decade ago too btw.)
>    

perf is a tool written by developers for developers.  kvm is written for 
users (most of them hidden behind management interfaces).  There's no 
point at all in shipping it as part of the kernel, users don't install 
and use kernels, they install and use distributions.

> So i can see some upcoming culture friction with standing KVM principles there
> ;-)
>    

No friction at all - I don't think any kvm developer agrees with you 
(but if anyone does please speak up).

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Enhance perf to support KVM

[Peter Zijlstra]

On Fri, 2010-02-26 at 12:47 +0200, Avi Kivity wrote:
> Not really.  The guest and host admins are usually different people, who 
> may, being admins, even actively hate each other.  The guest admin would 
> probably regard it as a security hole.  It's probably useful for the 
> single-host scenario, and of course for developers. 

LOL, let me be the malicious host admin, then you can be the guest,
there is no way you can protect yourself. If you don't trust the host,
don't use it.

All your IO flows through the host, all your sekrit keys are in memory,
there is no security.

Re: Enhance perf to support KVM

[Avi Kivity]

On 02/26/2010 01:48 PM, Peter Zijlstra wrote:
> On Fri, 2010-02-26 at 12:47 +0200, Avi Kivity wrote:
>    
>> Not really.  The guest and host admins are usually different people, who
>> may, being admins, even actively hate each other.  The guest admin would
>> probably regard it as a security hole.  It's probably useful for the
>> single-host scenario, and of course for developers.
>>      
> LOL, let me be the malicious host admin, then you can be the guest,
> there is no way you can protect yourself. If you don't trust the host,
> don't use it.
>
> All your IO flows through the host, all your sekrit keys are in memory,
> there is no security.
>    

That's true.  But guest admins are going to be unhappy about a file 
server serving their data to the host all the same.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Enhance perf to support KVM

[Ingo Molnar]

* Avi Kivity <avi@redhat.com> wrote:

> > You basically have given up control over the quality of KVM by pushing so 
> > many aspects of it to user-space and letting it rot there.
> 
> That's wrong on so many levels.  First, nothing is rotting in userspace, 
> qemu is evolving faster than kvm is.  If I pushed it into the kernel then 
> development pace would be much slower (since kernel development is harder), 
> quality would be lower (less infrastructure, any bug is a host crash or 
> security issue), and I personally would be totally swamped.

That was not what i suggested tho. tools/kvm/ would work plenty fine.

As i said:

> > [...] You are pushing _way_ too much to user-space into different modules 
> > and maintenance domains, [...]
> >
> > ( Note that i dont mind user-space tooling per se, as long as it sits together
> >   with the kernel bits and gets developed, packaged and given to the user 
> >   in the same domain. ) [...]


> > Sure the design looks somewhat cleaner on paper, but if the end result is 
> > not helped by it then over-modularization sure can hurt ...
> 
> Run 'rpm -qa' one of these days.  Modern software is modular, that's the 
> only way to manage it.

Of course rpm -qa shows cases where modularization works. But my point was 
over-modularization, which due to the KVM/qemu split we all suffer from.

Modularizing along the wrong interface is worse than not modularizing 
something that could be. So when designing software you generally want to err 
on the side of _under_-modularizing. It's always very easy to split stuff up, 
when there's a really strong technical argument for it. It's very hard to pull 
the broken pieces back together though once they are in difference domains of 
maintanence - as then it's usually social integration that has to happen, 
which is always harder than a technical split-up.

	Ingo

Re: Enhance perf to support KVM

[Avi Kivity]

On 02/26/2010 02:46 PM, Ingo Molnar wrote:
> * Avi Kivity<avi@redhat.com>  wrote:
>
>    
>>> You basically have given up control over the quality of KVM by pushing so
>>> many aspects of it to user-space and letting it rot there.
>>>        
>> That's wrong on so many levels.  First, nothing is rotting in userspace,
>> qemu is evolving faster than kvm is.  If I pushed it into the kernel then
>> development pace would be much slower (since kernel development is harder),
>> quality would be lower (less infrastructure, any bug is a host crash or
>> security issue), and I personally would be totally swamped.
>>      
> That was not what i suggested tho. tools/kvm/ would work plenty fine.
>    

I'll wait until we have tools/libc and tools/X.  After all, they affect 
a lot more people and are concerned with a lot more kernel/user 
interfaces than kvm.

> As i said:
>
>    
>>> [...] You are pushing _way_ too much to user-space into different modules
>>> and maintenance domains, [...]
>>>
>>> ( Note that i dont mind user-space tooling per se, as long as it sits together
>>>    with the kernel bits and gets developed, packaged and given to the user
>>>    in the same domain. ) [...]
>>>        
>
>    
>>> Sure the design looks somewhat cleaner on paper, but if the end result is
>>> not helped by it then over-modularization sure can hurt ...
>>>        
>> Run 'rpm -qa' one of these days.  Modern software is modular, that's the
>> only way to manage it.
>>      
> Of course rpm -qa shows cases where modularization works. But my point was
> over-modularization, which due to the KVM/qemu split we all suffer from.
>    

You're the only one who suffers from it.  Everyone else is happy with 
adding features in the modules that implements them, be it kvm, qemu, 
libvirt, or virt-manager (to name one tool stack out of several).

> Modularizing along the wrong interface is worse than not modularizing
> something that could be. So when designing software you generally want to err
> on the side of _under_-modularizing. It's always very easy to split stuff up,
> when there's a really strong technical argument for it. It's very hard to pull
> the broken pieces back together though once they are in difference domains of
> maintanence - as then it's usually social integration that has to happen,
> which is always harder than a technical split-up.
>    

As it happens, the kvm and qemu development community has a large 
overlap.  Many developers read both lists, contribute to both projects, 
and participate on the same weekly call.  While we had difficulties 
pushing patches to qemu in the past, that's behind us, and qemu is now 
accepting patches at a much higher rate than kvm.

Technically, it is obvious that the userspace and kernel components are 
separate projects.  All that remains is the social divide.  Since 
everyone (except you) is mostly happy, I see no reason to change.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Enhance perf to support KVM

[Ingo Molnar]

* Avi Kivity <avi@redhat.com> wrote:

> > That was not what i suggested tho. tools/kvm/ would work plenty fine.
> 
> I'll wait until we have tools/libc and tools/X.  After all, they affect a 
> lot more people and are concerned with a lot more kernel/user interfaces 
> than kvm.

So your answer can be summed up as: 'we wont do what makes sense technically 
because others suck even more' ?

And it's not just the kernel<->user interface (which btw., for the case of X 
is far narrower than what KVM currently has to Qemu).

The issue is a basic question of software design: does kvm-qemu really make as 
much sense without the kernel component as with it? The answer is: it will 
borderline-work with CPU emulation (and i'm sure there are people making use 
of it that way), but 90%+ of the userbase uses it with KVM and vice versa. It 
is really a single logical component as far as maintenance goes, and 
tools/kvm/ would make quite a bit of sense.

But i digress.

	Ingo

Re: Enhance perf to support KVM

[Jes Sorensen]

On 02/26/10 14:16, Ingo Molnar wrote:
>
> * Avi Kivity<avi@redhat.com>  wrote:
>
>>> That was not what i suggested tho. tools/kvm/ would work plenty fine.
>>
>> I'll wait until we have tools/libc and tools/X.  After all, they affect a
>> lot more people and are concerned with a lot more kernel/user interfaces
>> than kvm.
>
> So your answer can be summed up as: 'we wont do what makes sense technically
> because others suck even more' ?

Well in this discussion what makes sense technically differs depending
on who you ask.

I will argue that emulating the MSR access doesn't make sense
technically because there is no fixed specification we can rely on, 
since the spec seems to change randomly with every cpu family release
from Inte. In addition the overhead is making the resulting numbers
less if at all interesting.

Jes

Re: Enhance perf to support KVM

[Avi Kivity]

On 02/26/2010 03:16 PM, Ingo Molnar wrote:
> * Avi Kivity<avi@redhat.com>  wrote:
>
>    
>>> That was not what i suggested tho. tools/kvm/ would work plenty fine.
>>>        
>> I'll wait until we have tools/libc and tools/X.  After all, they affect a
>> lot more people and are concerned with a lot more kernel/user interfaces
>> than kvm.
>>      
> So your answer can be summed up as: 'we wont do what makes sense technically
> because others suck even more' ?
>    

I can sum up your this remark as 'whenever you disagree with me, I will 
rephrase your words to make you look like an idiot'.

If you believe I'm an idiot, there's no need to have this (or any) 
conversation.  If not, please refrain from this type of verbal gymnastics.

> And it's not just the kernel<->user interface (which btw., for the case of X
> is far narrower than what KVM currently has to Qemu).
>
> The issue is a basic question of software design: does kvm-qemu really make as
> much sense without the kernel component as with it? The answer is: it will
> borderline-work with CPU emulation (and i'm sure there are people making use
> of it that way), but 90%+ of the userbase uses it with KVM and vice versa. It
> is really a single logical component as far as maintenance goes, and
> tools/kvm/ would make quite a bit of sense.
>    

There are two separate questions.  Is there room for a kvm-only 
userspace component?  I believe so, but throwing away the momentum 
behind qemu would be foolish.

Does it make sense for such a component to live in linux.git?  IMO, no, 
and certainly a lot less than libc and X.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Enhance perf to support KVM

[Ingo Molnar]

* Avi Kivity <avi@redhat.com> wrote:

> On 02/26/2010 03:16 PM, Ingo Molnar wrote:
> >* Avi Kivity<avi@redhat.com>  wrote:
> >
> >>> That was not what i suggested tho. tools/kvm/ would work plenty fine.
> >>>
> >> I'll wait until we have tools/libc and tools/X.  After all, they affect a 
> >> lot more people and are concerned with a lot more kernel/user interfaces 
> >> than kvm.
> >
> > So your answer can be summed up as: 'we wont do what makes sense 
> > technically because others suck even more' ?
> 
> I can sum up your this remark as 'whenever you disagree with me, I will 
> rephrase your words to make you look like an idiot'.

Two points:

1)

You can try to ridicule me if you want, but do you actually claim that my 
summary is inaccurate?

I do claim it's a substantially accurate summary: you said you will (quote:) 
"wait with tools/kvm/ until we have tools/libc and tools/X".

I do think tools/X and tools/libc would make quite a bit of sense - this is 
one of the better design aspects of FreeBSD et al. It's a mistake that it's 
not being done.

2)

I used a question mark (the sentence was not a statement of fact), and you 
have no obligation to agree with the summary i provided.

	Ingo

Re: Enhance perf to support KVM

[Avi Kivity]

On 02/26/2010 04:23 PM, Ingo Molnar wrote:
> * Avi Kivity<avi@redhat.com>  wrote:
>
>    
>> On 02/26/2010 03:16 PM, Ingo Molnar wrote:
>>      
>>> * Avi Kivity<avi@redhat.com>   wrote:
>>>
>>>        
>>>>> That was not what i suggested tho. tools/kvm/ would work plenty fine.
>>>>>
>>>>>            
>>>> I'll wait until we have tools/libc and tools/X.  After all, they affect a
>>>> lot more people and are concerned with a lot more kernel/user interfaces
>>>> than kvm.
>>>>          
>>> So your answer can be summed up as: 'we wont do what makes sense
>>> technically because others suck even more' ?
>>>        
>> I can sum up your this remark as 'whenever you disagree with me, I will
>> rephrase your words to make you look like an idiot'.
>>      
> Two points:
>
> 1)
>
> You can try to ridicule me if you want,

I'd much prefer it if if no ridiculing was employed on either side.

>   but do you actually claim that my
> summary is inaccurate?
>
> I do claim it's a substantially accurate summary: you said you will (quote:)
> "wait with tools/kvm/ until we have tools/libc and tools/X".
>
> I do think tools/X and tools/libc would make quite a bit of sense - this is
> one of the better design aspects of FreeBSD et al. It's a mistake that it's
> not being done.
>    

There are arguments for libc to be developed in linux-2.6.git, and 
arguments against.  The fact is that they are not, so presumably the 
arguments against plus inertia outweigh the arguments for.

The same logic holds for kvm, except that there are less arguments for 
development in linux-2.6.git.  Only a small part of qemu is actually 
concerned with kvm; most of it is mucking around with X, emulating old 
devices, emulating instruction sets (irrelevant for tools/kvm) and doing 
boring managementy stuff.

Do we really want to add several hundered thousand lines to Linux, only 
a few thousand or of which talk to the kernel?

> 2)
>
> I used a question mark (the sentence was not a statement of fact), and you
> have no obligation to agree with the summary i provided.
>
>    

Thanks.  I hope you don't agree with it either.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Enhance perf to support KVM

[Avi Kivity]

On 02/26/2010 01:17 PM, Ingo Molnar wrote:
> Nobody is really 'in charge' of how KVM gets delivered to the user. You
> isolated the fun kernel part for you and pushed out the boring bits to
> user-space. So if mundane things like mouse integration sucks 'hey that's a
> user-space tooling problem', if file integration sucks then 'hey, that's an
> admin problem', if it cannot be used over the network 'hey, that's an Xorg
> problem', etc. etc.
>    

btw, mouse integration works with -usbdevice tablet and recent Fedoras, 
'it was an X.org driver problem'.

Really, I don't understand your problems.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: Enhance perf to support KVM

[Anthony Liguori]

On 02/26/2010 04:35 AM, Ingo Molnar wrote:
> Basically what is needed is plain filesystem access - properly privileged. So
> doing this via a vmchannel would be nice, but for the symbol extraction it
> would be a glorified NFS server in essence.
>
> Do you have (or plan) any turn-key 'access to all files of the guest' kind of
> guest-transparent facility that could be used for such purposes? That would
> have various advantages over a traditional explicit file server approach:
>
>   - it would not contaminate the guest port space
>
>   - no guest side configuration needed (the various oprofile remote daemons
>     always sucked as they needed extra setup)
>
>   - it might even be used with a guest that does no networking
>    

If you look at VMware, they provide a guest daemon that runs in 
userspace and communicates with the host to allow the host to do 
operations like execute a binary or read a file.

We've been trying to build this for years and virtio-console is what 
we've finally settled on.  The userspace bits are super easy to build.  
The problem is that we need guest kernel support.  We've not had a lot 
of success there as we've had a number of things Nacked.  The approach 
we originally started with was to introduce a new socket family for 
virtualization.  This is what s390 does and this is what VMware does.

But Dave Miller nacked that.  VMware happily carries their own module so 
they don't have to deal with problems like this.  I guess s390 was able 
to get away with it because noone pays attention to what they do.

Rusty had a proposal very similar to vhost-net that he posted two years 
ago but Andrew Morton nacked that because it wasn't generic enough 
(serves us right for trying to do anything generic in the first place).

Honestly, what's hardest for us to build a turn key solution is that 
unlike our competition, we've got to deal with getting people who don't 
necessarily care about virtualization to agree with what we're doing as 
opposed to just maintaining our own set of kernel modules.

Things limited to userspace have not slowed us down compared to getting 
things into the kernel.  Look at how difficult it was to get MMU 
notifiers merged as another example.

If you're volunteering to commit the things that we need to build a turn 
key solution, then that would certainly be a big help to us.

Regards,

Anthony Liguori

KVM usability

[Ingo Molnar]

* Avi Kivity <avi@redhat.com> wrote:

> On 02/26/2010 01:17 PM, Ingo Molnar wrote:
> >Nobody is really 'in charge' of how KVM gets delivered to the user. You
> >isolated the fun kernel part for you and pushed out the boring bits to
> >user-space. So if mundane things like mouse integration sucks 'hey that's a
> >user-space tooling problem', if file integration sucks then 'hey, that's an
> >admin problem', if it cannot be used over the network 'hey, that's an Xorg
> >problem', etc. etc.
> 
> btw, mouse integration works with -usbdevice tablet and recent
> Fedoras, 'it was an X.org driver problem'.
> 
> Really, I don't understand your problems.

I run bleeding edge rawhide on my main desktop so i just tried latest & 
greatest KVM and Qemu bits and started up kvm-qemu with some Fedora and XP 
images i had around:

  2.6.33-0.44.rc8.git0.fc13.x86_64
  qemu-system-x86-0.12.2-6.fc13.x86_64

Here's my experience with it:

 - qemu-kvm starts up with a miniature resolution by default. 640x480 - on my
   1680x1050 laptop screen. It's so small that initially i even overlooked 
   that i started it. It should multiplex pixels up to a reasonable screen 
   size by default.

 - The mouse is trapped straight away by default if you click into it. That's 
   very annoying if you actually try to integrate a guest OS into your desktop 
   - it's not just 'another, slightly weird app' but a sticky, opinionated GUI 
   component that you have to fight with all the time.

 - Once trapped it's not obvious how to untrap the mouse. The qemu window 
   title says:

          QEMU: Press Ctrl-ALT to exit grab

   Of course once you _know_ what a 'grab' is, you'll know where to look.
   At minimum it should say:

          QEMU: Press Ctrl-ALT to exit mouse grab

   But to first-time users it's an annoying trap of the mouse and with no
   obvious place to look for help. [besides, it doesnt tell which Ctrl and
   which ALT to use - it's the left side. The right side Ctrl does not work.]

 - Graphics performance is awful even with the 640x480 miniature version.
   During bootup I can see it drawing single characters. This is a Core2 
   2.8GHz.

 - Sound does not work by default. I have to go dig into command-line options
   to see that i need to add: "-soundhw all". Why isnt sound enabled by 
   default?

 - Qemu images are not integrated into the rest of the desktop. If i click on 
   a Qemu image it says:

     Could not display "/home/mingo/qemu/hda.img".

     The file is of an unknown type.

   10 years of Qemu and its base image format is still 'unknown' ?

 - Random bugs. I tried to boot some old Fedora image i had around, it says:

     spirit:~/qemu> qemu-kvm ./hda-fedora.img 
     kvm: unhandled exit 80000021
     kvm_run returned -22

   Bugs happen, but more important is what a user can do with it. To a user, 
   what does this tell? Is it actionable? Does it give any URL to check? Any bug
   submit mechanism to follow? Does it even tell what the code itself thinks
   that happened? Nope - it just prints that error message (on the console, so 
   to anyone starting it via a clicky interface wouldnt notice that there's 
   something wrong) - and the guest session hanging indefinitely.

 - When it boots up, the Qemu window flips around its size crazily, as the BIOS,
   the bootloader and the OS sets different screen resolutions. To the user that
   technical detail is immaterial, what matters is an amateurish-looking app 
   that flips its window size as if it was an adware popup window trying to 
   avoid being caught.

 - There's no obvious way to activate paravirt drivers on the Windows side.
   There's no friendly "install guest drivers" button to click on with Qemu.

   _Of course_ you will end up emulating hardware in KVM (and passing it through
   to the guest once it's clear that emulation performance sucks) and sooner 
   or later you will end up requesting unreasonable things of the host kernel 
   to achieve that ...

 - Another small detail: there is zero on-screen help (beyond the Ctrl-ALT 
   line) for a newbie to quickly find his way around it. No wiki address, no 
   help, no nothing. There's not even any hint about what this window does. 
   Which guest is it? In what state is that guest?

 - But i'm a more advanced user so i dont need help screens, i knew that the 
   "go full screen" hotkey is:

           LeftCtrl-LeftALT-F

   ... except that it is a one-way road: pressing it for a second time does 
   not restore the window, trapping me in the guest altogether! Ctrl-ALT does 
   not exit the trap either. I had to shut down the guest to get back my X 
   desktop.

etc., etc.

( I could go on and on about finer details of good integration, like the 
  difficulty of integrating host/guest files, networking, no way to quickly 
  attach ISOs via that window, no way to activate networking, sound and no way 
  to snapshot, no way to increase memory size except a command line option. )

etc - but that's not the point really: i only spent 10 minutes on this and i 
didnt try hard at all - _11_ bugs/annoyances from all across the functionality 
spectrum.

And the thing is _me_ reporting bugs does not matter at all in this picture, 
so please dont come with "why didnt you report this?".

_Anyone_ with half a brain who takes a critical look at this virtualization 
solution would notice the same. Still, it's essentially unchanged from 5 years 
ago.

Why is that so? I have outlined my opinion that this is due to the artificial 
package separation / over-modularization and no-one really being in charge of 
"KVM quality as a whole" - and i'm wondering what your theory is how such a 
state of affairs became possible.

I'm not trolling you at all: is it _really_ not obvious to you that the 
KVM/qemu usability status quo honestly sucks, to an unbiased observer?

And AFAICS KVM developers keep concentrating on all the wrong things due to 
that bad split/packaging: writing newer and newer low level kernel patches and 
optimizations which are nice but in large part irrelevant because the 
_fundamental basics_ of usability suck so much ... But to you it's probably 
just another external package so not really something you can do much about, 
right?

Really, the KVM design is so nice in many regards and Qemu has come forward 
leaps and bounds in the past few years as well, how can you miss such basic 
areas of weakness? 'First impression' is the thing that gives you new 
developers - it's any OSS project's bread and butter.

Thanks,

	Ingo

Re: KVM usability

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 8577 bytes --]

[ adding qemu-devel to CC as most issues concern that project as well ]

Ingo Molnar wrote:
> * Avi Kivity <avi@redhat.com> wrote:
> 
>> On 02/26/2010 01:17 PM, Ingo Molnar wrote:
>>> Nobody is really 'in charge' of how KVM gets delivered to the user. You
>>> isolated the fun kernel part for you and pushed out the boring bits to
>>> user-space. So if mundane things like mouse integration sucks 'hey that's a
>>> user-space tooling problem', if file integration sucks then 'hey, that's an
>>> admin problem', if it cannot be used over the network 'hey, that's an Xorg
>>> problem', etc. etc.
>> btw, mouse integration works with -usbdevice tablet and recent
>> Fedoras, 'it was an X.org driver problem'.
>>
>> Really, I don't understand your problems.
> 
> I run bleeding edge rawhide on my main desktop so i just tried latest & 
> greatest KVM and Qemu bits and started up kvm-qemu with some Fedora and XP 
> images i had around:
> 
>   2.6.33-0.44.rc8.git0.fc13.x86_64
>   qemu-system-x86-0.12.2-6.fc13.x86_64
> 
> Here's my experience with it:
> 
>  - qemu-kvm starts up with a miniature resolution by default. 640x480 - on my
>    1680x1050 laptop screen. It's so small that initially i even overlooked 
>    that i started it. It should multiplex pixels up to a reasonable screen 
>    size by default.
> 
>  - The mouse is trapped straight away by default if you click into it. That's 
>    very annoying if you actually try to integrate a guest OS into your desktop 
>    - it's not just 'another, slightly weird app' but a sticky, opinionated GUI 
>    component that you have to fight with all the time.
> 
>  - Once trapped it's not obvious how to untrap the mouse. The qemu window 
>    title says:
> 
>           QEMU: Press Ctrl-ALT to exit grab
> 
>    Of course once you _know_ what a 'grab' is, you'll know where to look.
>    At minimum it should say:
> 
>           QEMU: Press Ctrl-ALT to exit mouse grab
> 
>    But to first-time users it's an annoying trap of the mouse and with no
>    obvious place to look for help. [besides, it doesnt tell which Ctrl and
>    which ALT to use - it's the left side. The right side Ctrl does not work.]
> 
>  - Graphics performance is awful even with the 640x480 miniature version.
>    During bootup I can see it drawing single characters. This is a Core2 
>    2.8GHz.
> 
>  - Sound does not work by default. I have to go dig into command-line options
>    to see that i need to add: "-soundhw all". Why isnt sound enabled by 
>    default?
> 
>  - Qemu images are not integrated into the rest of the desktop. If i click on 
>    a Qemu image it says:
> 
>      Could not display "/home/mingo/qemu/hda.img".
> 
>      The file is of an unknown type.
> 
>    10 years of Qemu and its base image format is still 'unknown' ?
> 
>  - Random bugs. I tried to boot some old Fedora image i had around, it says:
> 
>      spirit:~/qemu> qemu-kvm ./hda-fedora.img 
>      kvm: unhandled exit 80000021
>      kvm_run returned -22
> 
>    Bugs happen, but more important is what a user can do with it. To a user, 
>    what does this tell? Is it actionable? Does it give any URL to check? Any bug
>    submit mechanism to follow? Does it even tell what the code itself thinks
>    that happened? Nope - it just prints that error message (on the console, so 
>    to anyone starting it via a clicky interface wouldnt notice that there's 
>    something wrong) - and the guest session hanging indefinitely.
> 
>  - When it boots up, the Qemu window flips around its size crazily, as the BIOS,
>    the bootloader and the OS sets different screen resolutions. To the user that
>    technical detail is immaterial, what matters is an amateurish-looking app 
>    that flips its window size as if it was an adware popup window trying to 
>    avoid being caught.
> 
>  - There's no obvious way to activate paravirt drivers on the Windows side.
>    There's no friendly "install guest drivers" button to click on with Qemu.
> 
>    _Of course_ you will end up emulating hardware in KVM (and passing it through
>    to the guest once it's clear that emulation performance sucks) and sooner 
>    or later you will end up requesting unreasonable things of the host kernel 
>    to achieve that ...
> 
>  - Another small detail: there is zero on-screen help (beyond the Ctrl-ALT 
>    line) for a newbie to quickly find his way around it. No wiki address, no 
>    help, no nothing. There's not even any hint about what this window does. 
>    Which guest is it? In what state is that guest?
> 
>  - But i'm a more advanced user so i dont need help screens, i knew that the 
>    "go full screen" hotkey is:
> 
>            LeftCtrl-LeftALT-F
> 
>    ... except that it is a one-way road: pressing it for a second time does 
>    not restore the window, trapping me in the guest altogether! Ctrl-ALT does 
>    not exit the trap either. I had to shut down the guest to get back my X 
>    desktop.
> 
> etc., etc.
> 
> ( I could go on and on about finer details of good integration, like the 
>   difficulty of integrating host/guest files, networking, no way to quickly 
>   attach ISOs via that window, no way to activate networking, sound and no way 
>   to snapshot, no way to increase memory size except a command line option. )
> 
> etc - but that's not the point really: i only spent 10 minutes on this and i 
> didnt try hard at all - _11_ bugs/annoyances from all across the functionality 
> spectrum.
> 
> And the thing is _me_ reporting bugs does not matter at all in this picture, 
> so please dont come with "why didnt you report this?".
> 
> _Anyone_ with half a brain who takes a critical look at this virtualization 
> solution would notice the same. Still, it's essentially unchanged from 5 years 
> ago.
> 
> Why is that so? I have outlined my opinion that this is due to the artificial 
> package separation / over-modularization and no-one really being in charge of 
> "KVM quality as a whole" - and i'm wondering what your theory is how such a 
> state of affairs became possible.
> 
> I'm not trolling you at all: is it _really_ not obvious to you that the 
> KVM/qemu usability status quo honestly sucks, to an unbiased observer?
> 
> And AFAICS KVM developers keep concentrating on all the wrong things due to 
> that bad split/packaging: writing newer and newer low level kernel patches and 
> optimizations which are nice but in large part irrelevant because the 
> _fundamental basics_ of usability suck so much ... But to you it's probably 
> just another external package so not really something you can do much about, 
> right?

Yes, there are quite a few frontends around, also for desktop
virtualization. AQEMU [1], e.g., should address at least some of your
valuable usability criticisms. If GUIs are not yet optimally integrated
with QEMU and if there is something that can be done inside QEMU or even
KVM, let's discuss that together with the related communities. However,
that is definitely not a valid reason for KVM developers to stop
improving the virtualization core for demanding use cases and start
writing GUIs.

Likely it is unfortunate that there is not The One And Only graphical
frontend for the desktop scenario we can focus on (But is it unfortunate
for desktop Linux that there is still Gnome vs. KDE? I bet there are
zillion opinions on that.). But maybe this discussion here will have the
positive effect that the QEMU/KVM community starts giving more feedback
to some of those projects, and vice versa.

> 
> Really, the KVM design is so nice in many regards and Qemu has come forward 
> leaps and bounds in the past few years as well, how can you miss such basic 
> areas of weakness? 'First impression' is the thing that gives you new 
> developers - it's any OSS project's bread and butter.

'First impression' mostly gives us more users - a worthwhile goal as well!

But developers are rather attracted by rich features, a flexible,
extensible design, a working community and - of course - a proper
licensing model. Look at Virtualbox, they probably meet your usability
requirements out of the box. Do they have a developer community? Not at
all because they miss some of that key requirements. Also for those
reasons, Vbox dropped from the list of candidates for many
virtualization projects @work, and we decided to invest in the true
Linux stack.

Jan

[1] http://sourceforge.net/projects/aqemu/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

[Qemu-devel] Re: KVM usability

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 8577 bytes --]

[ adding qemu-devel to CC as most issues concern that project as well ]

Ingo Molnar wrote:
> * Avi Kivity <avi@redhat.com> wrote:
> 
>> On 02/26/2010 01:17 PM, Ingo Molnar wrote:
>>> Nobody is really 'in charge' of how KVM gets delivered to the user. You
>>> isolated the fun kernel part for you and pushed out the boring bits to
>>> user-space. So if mundane things like mouse integration sucks 'hey that's a
>>> user-space tooling problem', if file integration sucks then 'hey, that's an
>>> admin problem', if it cannot be used over the network 'hey, that's an Xorg
>>> problem', etc. etc.
>> btw, mouse integration works with -usbdevice tablet and recent
>> Fedoras, 'it was an X.org driver problem'.
>>
>> Really, I don't understand your problems.
> 
> I run bleeding edge rawhide on my main desktop so i just tried latest & 
> greatest KVM and Qemu bits and started up kvm-qemu with some Fedora and XP 
> images i had around:
> 
>   2.6.33-0.44.rc8.git0.fc13.x86_64
>   qemu-system-x86-0.12.2-6.fc13.x86_64
> 
> Here's my experience with it:
> 
>  - qemu-kvm starts up with a miniature resolution by default. 640x480 - on my
>    1680x1050 laptop screen. It's so small that initially i even overlooked 
>    that i started it. It should multiplex pixels up to a reasonable screen 
>    size by default.
> 
>  - The mouse is trapped straight away by default if you click into it. That's 
>    very annoying if you actually try to integrate a guest OS into your desktop 
>    - it's not just 'another, slightly weird app' but a sticky, opinionated GUI 
>    component that you have to fight with all the time.
> 
>  - Once trapped it's not obvious how to untrap the mouse. The qemu window 
>    title says:
> 
>           QEMU: Press Ctrl-ALT to exit grab
> 
>    Of course once you _know_ what a 'grab' is, you'll know where to look.
>    At minimum it should say:
> 
>           QEMU: Press Ctrl-ALT to exit mouse grab
> 
>    But to first-time users it's an annoying trap of the mouse and with no
>    obvious place to look for help. [besides, it doesnt tell which Ctrl and
>    which ALT to use - it's the left side. The right side Ctrl does not work.]
> 
>  - Graphics performance is awful even with the 640x480 miniature version.
>    During bootup I can see it drawing single characters. This is a Core2 
>    2.8GHz.
> 
>  - Sound does not work by default. I have to go dig into command-line options
>    to see that i need to add: "-soundhw all". Why isnt sound enabled by 
>    default?
> 
>  - Qemu images are not integrated into the rest of the desktop. If i click on 
>    a Qemu image it says:
> 
>      Could not display "/home/mingo/qemu/hda.img".
> 
>      The file is of an unknown type.
> 
>    10 years of Qemu and its base image format is still 'unknown' ?
> 
>  - Random bugs. I tried to boot some old Fedora image i had around, it says:
> 
>      spirit:~/qemu> qemu-kvm ./hda-fedora.img 
>      kvm: unhandled exit 80000021
>      kvm_run returned -22
> 
>    Bugs happen, but more important is what a user can do with it. To a user, 
>    what does this tell? Is it actionable? Does it give any URL to check? Any bug
>    submit mechanism to follow? Does it even tell what the code itself thinks
>    that happened? Nope - it just prints that error message (on the console, so 
>    to anyone starting it via a clicky interface wouldnt notice that there's 
>    something wrong) - and the guest session hanging indefinitely.
> 
>  - When it boots up, the Qemu window flips around its size crazily, as the BIOS,
>    the bootloader and the OS sets different screen resolutions. To the user that
>    technical detail is immaterial, what matters is an amateurish-looking app 
>    that flips its window size as if it was an adware popup window trying to 
>    avoid being caught.
> 
>  - There's no obvious way to activate paravirt drivers on the Windows side.
>    There's no friendly "install guest drivers" button to click on with Qemu.
> 
>    _Of course_ you will end up emulating hardware in KVM (and passing it through
>    to the guest once it's clear that emulation performance sucks) and sooner 
>    or later you will end up requesting unreasonable things of the host kernel 
>    to achieve that ...
> 
>  - Another small detail: there is zero on-screen help (beyond the Ctrl-ALT 
>    line) for a newbie to quickly find his way around it. No wiki address, no 
>    help, no nothing. There's not even any hint about what this window does. 
>    Which guest is it? In what state is that guest?
> 
>  - But i'm a more advanced user so i dont need help screens, i knew that the 
>    "go full screen" hotkey is:
> 
>            LeftCtrl-LeftALT-F
> 
>    ... except that it is a one-way road: pressing it for a second time does 
>    not restore the window, trapping me in the guest altogether! Ctrl-ALT does 
>    not exit the trap either. I had to shut down the guest to get back my X 
>    desktop.
> 
> etc., etc.
> 
> ( I could go on and on about finer details of good integration, like the 
>   difficulty of integrating host/guest files, networking, no way to quickly 
>   attach ISOs via that window, no way to activate networking, sound and no way 
>   to snapshot, no way to increase memory size except a command line option. )
> 
> etc - but that's not the point really: i only spent 10 minutes on this and i 
> didnt try hard at all - _11_ bugs/annoyances from all across the functionality 
> spectrum.
> 
> And the thing is _me_ reporting bugs does not matter at all in this picture, 
> so please dont come with "why didnt you report this?".
> 
> _Anyone_ with half a brain who takes a critical look at this virtualization 
> solution would notice the same. Still, it's essentially unchanged from 5 years 
> ago.
> 
> Why is that so? I have outlined my opinion that this is due to the artificial 
> package separation / over-modularization and no-one really being in charge of 
> "KVM quality as a whole" - and i'm wondering what your theory is how such a 
> state of affairs became possible.
> 
> I'm not trolling you at all: is it _really_ not obvious to you that the 
> KVM/qemu usability status quo honestly sucks, to an unbiased observer?
> 
> And AFAICS KVM developers keep concentrating on all the wrong things due to 
> that bad split/packaging: writing newer and newer low level kernel patches and 
> optimizations which are nice but in large part irrelevant because the 
> _fundamental basics_ of usability suck so much ... But to you it's probably 
> just another external package so not really something you can do much about, 
> right?

Yes, there are quite a few frontends around, also for desktop
virtualization. AQEMU [1], e.g., should address at least some of your
valuable usability criticisms. If GUIs are not yet optimally integrated
with QEMU and if there is something that can be done inside QEMU or even
KVM, let's discuss that together with the related communities. However,
that is definitely not a valid reason for KVM developers to stop
improving the virtualization core for demanding use cases and start
writing GUIs.

Likely it is unfortunate that there is not The One And Only graphical
frontend for the desktop scenario we can focus on (But is it unfortunate
for desktop Linux that there is still Gnome vs. KDE? I bet there are
zillion opinions on that.). But maybe this discussion here will have the
positive effect that the QEMU/KVM community starts giving more feedback
to some of those projects, and vice versa.

> 
> Really, the KVM design is so nice in many regards and Qemu has come forward 
> leaps and bounds in the past few years as well, how can you miss such basic 
> areas of weakness? 'First impression' is the thing that gives you new 
> developers - it's any OSS project's bread and butter.

'First impression' mostly gives us more users - a worthwhile goal as well!

But developers are rather attracted by rich features, a flexible,
extensible design, a working community and - of course - a proper
licensing model. Look at Virtualbox, they probably meet your usability
requirements out of the box. Do they have a developer community? Not at
all because they miss some of that key requirements. Also for those
reasons, Vbox dropped from the list of candidates for many
virtualization projects @work, and we decided to invest in the true
Linux stack.

Jan

[1] http://sourceforge.net/projects/aqemu/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: KVM usability

[Ian Kirk]

Ingo Molnar wrote:

> I'm not trolling you at all: is it _really_ not obvious to you that the
> KVM/qemu usability status quo honestly sucks, to an unbiased observer?

I agree that for desktop-style (e.g. vmware workstation) style
virtualisation, the plain QEMU+KVM package is terrible.

However, in the case of Enterprise server virtualisation, a lot your
comments relate to things that do not matter - sound, graphics, GUI, X
integration.

In the former use-case, it definitely needs significant work on the
usability front. In the later use-case, libvirt, RHEV etc all provide a
(some argue) decent interface, which removes the remaining niggles around
obscure command line options.

Which one is QEMU+KVM trying to be?

IMO it is succeeding far, far better at server workloads than desktop
usage.

Ian

Re: KVM usability

[Zachary Amsden]

On 02/27/2010 12:56 AM, Ingo Molnar wrote:
> * Avi Kivity<avi@redhat.com>  wrote:
>
>    
>> On 02/26/2010 01:17 PM, Ingo Molnar wrote:
>>      
>>> Nobody is really 'in charge' of how KVM gets delivered to the user. You
>>> isolated the fun kernel part for you and pushed out the boring bits to
>>> user-space. So if mundane things like mouse integration sucks 'hey that's a
>>> user-space tooling problem', if file integration sucks then 'hey, that's an
>>> admin problem', if it cannot be used over the network 'hey, that's an Xorg
>>> problem', etc. etc.
>>>        
>> btw, mouse integration works with -usbdevice tablet and recent
>> Fedoras, 'it was an X.org driver problem'.
>>
>> Really, I don't understand your problems.
>>      
> I run bleeding edge rawhide on my main desktop so i just tried latest&
> greatest KVM and Qemu bits and started up kvm-qemu with some Fedora and XP
> images i had around:
>
>    2.6.33-0.44.rc8.git0.fc13.x86_64
>    qemu-system-x86-0.12.2-6.fc13.x86_64
>
> Here's my experience with it:
>
>   - qemu-kvm starts up with a miniature resolution by default. 640x480 - on my
>     1680x1050 laptop screen. It's so small that initially i even overlooked
>     that i started it. It should multiplex pixels up to a reasonable screen
>     size by default.
>    

No virtualization emulator today does this, it is not a reasonable thing 
to do.  Unless you are running compiz and use hardware scaling.  We 
should look into it.

>   - The mouse is trapped straight away by default if you click into it. That's
>     very annoying if you actually try to integrate a guest OS into your desktop
>     - it's not just 'another, slightly weird app' but a sticky, opinionated GUI
>     component that you have to fight with all the time.
>
>   - Once trapped it's not obvious how to untrap the mouse. The qemu window
>     title says:
>
>            QEMU: Press Ctrl-ALT to exit grab
>
>     Of course once you _know_ what a 'grab' is, you'll know where to look.
>     At minimum it should say:
>
>            QEMU: Press Ctrl-ALT to exit mouse grab
>
>     But to first-time users it's an annoying trap of the mouse and with no
>     obvious place to look for help. [besides, it doesnt tell which Ctrl and
>     which ALT to use - it's the left side. The right side Ctrl does not work.]
>    

As a long time user of other brand name virtualization software, this 
was immediately obvious to me and followed the paradigm I was used to - 
a two or three combination meta key press allows escape, while allowing 
you to send one or two combo meta keypresses to the VM.

>   - Graphics performance is awful even with the 640x480 miniature version.
>     During bootup I can see it drawing single characters. This is a Core2
>     2.8GHz.
>    

Yes it is.

>   - Sound does not work by default. I have to go dig into command-line options
>     to see that i need to add: "-soundhw all". Why isnt sound enabled by
>     default?
>    

Probably because qemu as a platform supports 800 different kinds of 
hosts, all with their own sound solutions, sometimes integrated "for 
ease" as part of the display library, sometimes split into a myriad of 
confusing and incompatible options, like on Linux.

>   - Qemu images are not integrated into the rest of the desktop. If i click on
>     a Qemu image it says:
>
>       Could not display "/home/mingo/qemu/hda.img".
>
>       The file is of an unknown type.
>
>     10 years of Qemu and its base image format is still 'unknown' ?
>    

This is a desktop distro issue and easily fixed.  Please file a bug.

>   - Random bugs. I tried to boot some old Fedora image i had around, it says:
>
>       spirit:~/qemu>  qemu-kvm ./hda-fedora.img
>       kvm: unhandled exit 80000021
>       kvm_run returned -22
>
>     Bugs happen, but more important is what a user can do with it. To a user,
>     what does this tell? Is it actionable? Does it give any URL to check? Any bug
>     submit mechanism to follow? Does it even tell what the code itself thinks
>     that happened? Nope - it just prints that error message (on the console, so
>     to anyone starting it via a clicky interface wouldnt notice that there's
>     something wrong) - and the guest session hanging indefinitely.
>    

Qemu code is very bad in general at error handling.  It will take years 
to fix this.

>   - When it boots up, the Qemu window flips around its size crazily, as the BIOS,
>     the bootloader and the OS sets different screen resolutions. To the user that
>     technical detail is immaterial, what matters is an amateurish-looking app
>     that flips its window size as if it was an adware popup window trying to
>     avoid being caught.
>    

This is true again of other major brand name virtualization solutions.  
The window resizes as the guest resizes the display.  There really isn't 
a better solution.

>   - There's no obvious way to activate paravirt drivers on the Windows side.
>     There's no friendly "install guest drivers" button to click on with Qemu.
>
>     _Of course_ you will end up emulating hardware in KVM (and passing it through
>     to the guest once it's clear that emulation performance sucks) and sooner
>     or later you will end up requesting unreasonable things of the host kernel
>     to achieve that ...
>
>   - Another small detail: there is zero on-screen help (beyond the Ctrl-ALT
>     line) for a newbie to quickly find his way around it. No wiki address, no
>     help, no nothing. There's not even any hint about what this window does.
>     Which guest is it? In what state is that guest?
>    

These are productized refinements we should look at implementing.

>   - But i'm a more advanced user so i dont need help screens, i knew that the
>     "go full screen" hotkey is:
>
>             LeftCtrl-LeftALT-F
>
>     ... except that it is a one-way road: pressing it for a second time does
>     not restore the window, trapping me in the guest altogether! Ctrl-ALT does
>     not exit the trap either. I had to shut down the guest to get back my X
>     desktop.
>
> etc., etc.
>
> ( I could go on and on about finer details of good integration, like the
>    difficulty of integrating host/guest files, networking, no way to quickly
>    attach ISOs via that window, no way to activate networking, sound and no way
>    to snapshot, no way to increase memory size except a command line option. )
>
> etc - but that's not the point really: i only spent 10 minutes on this and i
> didnt try hard at all - _11_ bugs/annoyances from all across the functionality
> spectrum.
>
> And the thing is _me_ reporting bugs does not matter at all in this picture,
> so please dont come with "why didnt you report this?".
>
> _Anyone_ with half a brain who takes a critical look at this virtualization
> solution would notice the same. Still, it's essentially unchanged from 5 years
> ago.
>
> Why is that so? I have outlined my opinion that this is due to the artificial
> package separation / over-modularization and no-one really being in charge of
> "KVM quality as a whole" - and i'm wondering what your theory is how such a
> state of affairs became possible.
>    

This is really packed with a bunch of loaded statements.

First of all, the package separation is not artificial, it is a natural 
consequence of the architecture.  There is a CPU, it manipulates and is 
tightly integrated with memory, and there are devices which are part of 
the chipset, and so on.

Second, it's not over-modularized.  The modules are the individual 
components of the architecture.  How would you propose to put it 
differently.  They really can't naturally combine.  And with the code 
quality of qemu in general being problematic by Linux kernel standards, 
it's not natural to move the device emulation directly into the kernel 
module.  So this is why we are where we are today.

Third, it's the maintainers upstream who are in charge of KVM quality as 
a whole - when you are talking about upstream code quality, and the 
package maintainers who are in charge of KVM quality as a whole - when 
you are talking about a distro.  This is nothing new - it's just a 
statement of decentralization.

Zach

Re: KVM usability

[Ingo Molnar]

* Zachary Amsden <zamsden@redhat.com> wrote:

[...]
> 
> Second, it's not over-modularized.  The modules are the individual
> components of the architecture.  How would you propose to put it
> differently.  They really can't naturally combine.  And with the
> code quality of qemu in general being problematic by Linux kernel
> standards, it's not natural to move the device emulation directly
> into the kernel module.  So this is why we are where we are today.

I'm not talking about moving it into a kernel _module_ - albeit that
alone is a worthwile thing to do for any performance sensitive hw
component.

I was talking about the option of a clean, stripped down Qemu base
hosted in the kernel proper, in linux/tools/kvm/ or so. If i were
running a virtualization effort it would be the first place i'd
consider to put my tooling into.

It would be a no-brainer: most of the devs come from the KVM side, and
KVM itself makes little sense without Qemu, and Qemu makes little sense
without KVM these days. (and i know about the non-KVM and non-x86
roots of Qemu - still, it's not a significant piece of usage today) 

> Third, it's the maintainers upstream who are in charge of KVM
> quality as a whole - when you are talking about upstream code
> quality, and the package maintainers who are in charge of KVM
> quality as a whole - when you are talking about a distro.  This is
> nothing new - it's just a statement of decentralization.

It's certainly nothing new. Nor was the suckage of CVS newup until
Git came along and changed the game on a fundamental basis.

Suckage is there to be fought all the time.

	Ingo

Re: KVM usability

[Ingo Molnar]

* Zachary Amsden <zamsden@redhat.com> wrote:

> > Here's my experience with it:
> >
> >  - qemu-kvm starts up with a miniature resolution by default. 640x480 - on my
> >    1680x1050 laptop screen. It's so small that initially i even overlooked
> >    that i started it. It should multiplex pixels up to a reasonable screen
> >    size by default.
> 
> No virtualization emulator today does this, it is not a reasonable thing to 
> do.  Unless you are running compiz and use hardware scaling.  We should look 
> into it.

I tried VirtualBox and it does something sane here: while it does not give a 
larger guest screen area, it at least creates a large enough X window, with 
any border area whited out.

That looks far more pleasing aesthetically than the Qemu method of resizing 
the full window wildly. The guest still resizes, but it stays within the same 
X window and the border is white.

Also, with Windows guests, the Windows side resolution resizes dynamically if 
the window is grown/shrunk by the user. (I suspect that is done via a paravirt 
driver on the guest side, through 'Guest Additions'.)

	Ingo

Re: KVM usability

[Anthony Liguori]

On 02/27/2010 04:56 AM, Ingo Molnar wrote:
> * Avi Kivity<avi@redhat.com>  wrote:
>
>    
>> On 02/26/2010 01:17 PM, Ingo Molnar wrote:
>>      
>>> Nobody is really 'in charge' of how KVM gets delivered to the user. You
>>> isolated the fun kernel part for you and pushed out the boring bits to
>>> user-space. So if mundane things like mouse integration sucks 'hey that's a
>>> user-space tooling problem', if file integration sucks then 'hey, that's an
>>> admin problem', if it cannot be used over the network 'hey, that's an Xorg
>>> problem', etc. etc.
>>>        
>> btw, mouse integration works with -usbdevice tablet and recent
>> Fedoras, 'it was an X.org driver problem'.
>>
>> Really, I don't understand your problems.
>>      
> I run bleeding edge rawhide on my main desktop so i just tried latest&
> greatest KVM and Qemu bits and started up kvm-qemu with some Fedora and XP
> images i had around:
>
>    2.6.33-0.44.rc8.git0.fc13.x86_64
>    qemu-system-x86-0.12.2-6.fc13.x86_64
>
> Here's my experience with it:
>
>   - qemu-kvm starts up with a miniature resolution by default. 640x480 - on my
>    

Probably 720x480 if you're referring to CGA mode.

>     1680x1050 laptop screen. It's so small that initially i even overlooked
>     that i started it. It should multiplex pixels up to a reasonable screen
>     size by default.
>    

Resize the window to whatever size you want it to be.  We'll 
automatically scale the screen.  This works both with SDL and with VNC 
(provided you use a gtk-vnc based client).

If you're suggesting we should scale by default, I disagree.  I have the 
same size screen and the QEMU window is a little bit larger than a 
standard gnome-terminal.

>   - The mouse is trapped straight away by default if you click into it. That's
>     very annoying if you actually try to integrate a guest OS into your desktop
>     - it's not just 'another, slightly weird app' but a sticky, opinionated GUI
>     component that you have to fight with all the time.
>    

All virtualization software behaves this way.

>   - Once trapped it's not obvious how to untrap the mouse. The qemu window
>     title says:
>
>            QEMU: Press Ctrl-ALT to exit grab
>
>     Of course once you _know_ what a 'grab' is, you'll know where to look.
>     At minimum it should say:
>
>            QEMU: Press Ctrl-ALT to exit mouse grab
>    

Reasonable suggestion.  I've changed it in my local branch.

>     But to first-time users it's an annoying trap of the mouse and with no
>     obvious place to look for help. [besides, it doesnt tell which Ctrl and
>     which ALT to use - it's the left side. The right side Ctrl does not work.]
>
>   - Graphics performance is awful even with the 640x480 miniature version.
>     During bootup I can see it drawing single characters. This is a Core2
>     2.8GHz.
>    

That's a bug.  Please report it.  Graphics performance isn't great, but 
it should not be that bad.

>   - Sound does not work by default. I have to go dig into command-line options
>     to see that i need to add: "-soundhw all". Why isnt sound enabled by
>     default?
>    

Because it's intrusive.  If you're running 20 guests, you don't want 
them all trying to play the Windows start up noise.  Even if you're 
running a single VM, you don't necessarily want your music interrupted 
to play the Ubuntu drums.  It's particularly annoying if you're 
repeatedly starting up guests.

>   - Qemu images are not integrated into the rest of the desktop. If i click on
>     a Qemu image it says:
>
>       Could not display "/home/mingo/qemu/hda.img".
>
>       The file is of an unknown type.
>
>     10 years of Qemu and its base image format is still 'unknown' ?
>    

It's not been 10 years, but I presume you're using a raw image or your 
distro is fubar because:

anthony@squirrel:~/images$ file linux.img
linux.img: Qemu Image, Format: Qcow , Version: 2

We intentionally don't want images to be directly executable.  An image 
doesn't include the necessary configuration information to launch a 
guest.  Adding that info to an image is not an obvious thing to do 
either because a virtual machine may consist of multiple images.

>   - Random bugs. I tried to boot some old Fedora image i had around, it says:
>
>       spirit:~/qemu>  qemu-kvm ./hda-fedora.img
>       kvm: unhandled exit 80000021
>       kvm_run returned -22
>    

This error message is horrible and it warrants a bug report.   See 
https://bugs.launchpad.net/qemu/+bug/530077

>   - When it boots up, the Qemu window flips around its size crazily, as the BIOS,
>     the bootloader and the OS sets different screen resolutions. To the user that
>     technical detail is immaterial, what matters is an amateurish-looking app
>     that flips its window size as if it was an adware popup window trying to
>     avoid being caught.
>    

All virtualization software does this and I think it's completely 
reasonable.  Again, you can resize the window manually and it'll keep 
that resolution if you are so inclined.  As KMS is better supported in 
Linux guests, we'll eventually get to the point where less screen 
flipping is done.  This is as much a usability issue on bare metal as it 
is in a VM.

>   - There's no obvious way to activate paravirt drivers on the Windows side.
>     There's no friendly "install guest drivers" button to click on with Qemu.
>
>     _Of course_ you will end up emulating hardware in KVM (and passing it through
>     to the guest once it's clear that emulation performance sucks) and sooner
>     or later you will end up requesting unreasonable things of the host kernel
>     to achieve that ...
>    

This is a licensing issue.  You cannot just build Windows drivers 
anymore.  They need to be signed by Microsoft in order to be loadable in 
the most recent versions of Windows.  If you look at the recent Fedora 
Advisory Board minutes, you'll see that this issue has been discussed 
specifically around how a distribution like Fedora can provide Windows 
drivers to end users in a friendly way.

I believe the current proposal is to host a signed binary on a site like 
Fedora Hosted, and then have virt-manager automatically download the ISO 
on demand.  Shipping the binary ISO as part of a distribution package 
though is not possible since it cannot be rebuilt like any other package.

>   - Another small detail: there is zero on-screen help (beyond the Ctrl-ALT
>     line) for a newbie to quickly find his way around it. No wiki address, no
>     help, no nothing. There's not even any hint about what this window does.
>     Which guest is it? In what state is that guest?
>    

This is a valid criticism but is tied to the fact that we use SDL.

>   - But i'm a more advanced user so i dont need help screens, i knew that the
>     "go full screen" hotkey is:
>
>             LeftCtrl-LeftALT-F
>
>     ... except that it is a one-way road: pressing it for a second time does
>     not restore the window, trapping me in the guest altogether! Ctrl-ALT does
>     not exit the trap either. I had to shut down the guest to get back my X
>     desktop.
>    

That's a bug.

> etc., etc.
>
> ( I could go on and on about finer details of good integration, like the
>    difficulty of integrating host/guest files, networking, no way to quickly
>    attach ISOs via that window, no way to activate networking, sound and no way
>    to snapshot, no way to increase memory size except a command line option. )
>
> etc - but that's not the point really: i only spent 10 minutes on this and i
> didnt try hard at all - _11_ bugs/annoyances from all across the functionality
> spectrum.
>
> And the thing is _me_ reporting bugs does not matter at all in this picture,
> so please dont come with "why didnt you report this?".
>
> _Anyone_ with half a brain who takes a critical look at this virtualization
> solution would notice the same. Still, it's essentially unchanged from 5 years
> ago.
>
> Why is that so? I have outlined my opinion that this is due to the artificial
> package separation / over-modularization and no-one really being in charge of
> "KVM quality as a whole" - and i'm wondering what your theory is how such a
> state of affairs became possible.
>
> I'm not trolling you at all: is it _really_ not obvious to you that the
> KVM/qemu usability status quo honestly sucks, to an unbiased observer?
>
> And AFAICS KVM developers keep concentrating on all the wrong things due to
> that bad split/packaging: writing newer and newer low level kernel patches and
> optimizations which are nice but in large part irrelevant because the
> _fundamental basics_ of usability suck so much ... But to you it's probably
> just another external package so not really something you can do much about,
> right?
>
> Really, the KVM design is so nice in many regards and Qemu has come forward
> leaps and bounds in the past few years as well, how can you miss such basic
> areas of weakness? 'First impression' is the thing that gives you new
> developers - it's any OSS project's bread and butter.
>    

Ultimately, the problem you're facing is choice.  It's the same problem 
that faces a lot of other open source projects.

Your expectation of QEMU/KVM is that the graphics experience is 
optimized for a casual desktop virtualization user.  That is not the 
goal of QEMU/KVM though.  That is the goal of virt-manager.  The real 
question to ask is, why are you using qemu directly instead of using 
virt-manager?

Qemu targets two types of users.  The first is third party management 
interfaces.  Qemu can be used for a lot of purposes beyond desktop 
virtualization.  Most cell phone SDKs use qemu arm emulation.  The 
interface for a cell phone SDK is obviously very different from the 
interface for desktop virtualization.  Taking the gui out of qemu lets 
us support both.

It makes very little sense to integrate a powerful gui directly into 
qemu because in many circumstances, the qemu instance that you are 
interacting with is on a completely different box.  This is where 
something like virt-manager comes into play because you can just as 
easily manage a remote instance of qemu as a local one.

The other type of user we target is power virtualization/emulation 
users.  We certainly could do better for this type of user but it's 
never going to fit what your expectation of desktop virtualization is.  
Qemu is never going to be like running VMware Workstation or VirtualBox.

There is very little split between qemu and kvm from a project 
perspective.  If anything, our lack of focus on desktop virtualization 
comes from the split between qemu/kvm and libvirt.  All of the end-user 
usability is done in the context of libvirt and virt-manager whereas the 
qemu/kvm developers tend to focus on performance, features, and robustness.

I think we don't spend enough time in qemu/kvm thinking about how we 
interact with libvirt with respect to end-to-end usability.  It's 
something I do believe we need to address and it's an area I have been 
focusing on recently.

Regards,

Anthony Liguori

> Thanks,
>
> 	Ingo
>    

Re: KVM usability

[Anthony Liguori]

On 02/27/2010 11:25 AM, Ingo Molnar wrote:
> * Zachary Amsden<zamsden@redhat.com>  wrote:
>
> [...]
>    
>> Second, it's not over-modularized.  The modules are the individual
>> components of the architecture.  How would you propose to put it
>> differently.  They really can't naturally combine.  And with the
>> code quality of qemu in general being problematic by Linux kernel
>> standards, it's not natural to move the device emulation directly
>> into the kernel module.  So this is why we are where we are today.
>>      
> I'm not talking about moving it into a kernel _module_ - albeit that
> alone is a worthwile thing to do for any performance sensitive hw
> component.
>
> I was talking about the option of a clean, stripped down Qemu base
> hosted in the kernel proper, in linux/tools/kvm/ or so. If i were
> running a virtualization effort it would be the first place i'd
> consider to put my tooling into.
>    

Let's ignore the suggestion of hosting it in the kernel.  There's no 
reason it couldn't be as successful hosted as a separate project.

Let's consider what you would strip of out qemu.  You would obviously 
pull out TCG and the device emulation that isn't useful for KVM.  You 
can't compile out TCG today but you actually can compile out most device 
emulation so this doesn't actually buy you much.  It certainly doesn't 
fix any of the problems you outlined.

The GUI wouldn't change at all.  You still have the same fundamental 
problem that whatever this userspace executable is, is not the place 
where you need to implement a user friendly GUI.  That has to be a 
separate process.  Maybe you could integrate that separate process into 
the same repository as the core process but we can still do this with qemu.

> It would be a no-brainer: most of the devs come from the KVM side, and
> KVM itself makes little sense without Qemu, and Qemu makes little sense
> without KVM these days. (and i know about the non-KVM and non-x86
> roots of Qemu - still, it's not a significant piece of usage today)
>    

Do you have statistics to back this up?  You would probably be surprised 
at how many people use TCG.

To be honest, every KVM developer including myself has considered and 
even prototyped exactly what you described.  We've all independently 
come to the same conclusion: it's easier to incrementally improve qemu 
than it is to split the code base and try to maintain the fork.

And a lot of the other vendors who have decided to fork qemu in the past 
have learned the hard way that it's more difficult to maintain a fork 
and are now merging back to upstream qemu.

We could certainly make the same argument about forking the kernel to 
make it optimized for virtualization.  If we took Linux and added it to 
the qemu git tree, we would instantly have transparent large page 
support for users instead of having to wait years to get it properly 
integrated.  We could also add gang scheduling and hard scheduler limits 
to the kernel.  But we know better and even though the process is more 
painful and drawn out, we end up with a much better solution in the long 
run by including the input and feedback from people like you.

Xen clearly made a different decision and is still suffering the 
consequences.  They've done the same thing with qemu as you describe and 
have now realized it was a mistake and are working to merge their 
changes into upstream qemu.

There are *plenty* of usability issues (like transparent large pages) 
that need to be addressed at the KVM/kernel level.  Today, a user has to 
choose between a ~30% decrease in performance on Java workloads or the 
ability to overcommit memory.  It's a pretty significant problem and 
there's been a lot of resistance within the kernel community to fix it.

Likewise, I'm seeing a good number of people hit problems with lock 
holder pre-emption in the field.  It's absolutely a usability problem 
when a user sees catastrophically bad performance running an 8-VCPU 
virtual machine on a 2 socket host.  Whether it's gang scheduling or 
directed yields + pause loop detection, we definitely need some 
scheduler changes to fix this problem.

Not having an option enabled by default is an annoyance that a user 
eventually overcomes with the help of documentation.  Performance 
problems are deal breakers that lead users to switch to another 
virtualization technology.

Just stripping down qemu and putting the result in the kernel source 
tree doesn't fix anything.  We have plenty of hard problems to solve 
already.

Regards,

Anthony Liguori

Re: KVM usability

[Anthony Liguori]

On 03/01/2010 03:25 AM, Ingo Molnar wrote:
> * Zachary Amsden<zamsden@redhat.com>  wrote:
>
>    
>>> Here's my experience with it:
>>>
>>>   - qemu-kvm starts up with a miniature resolution by default. 640x480 - on my
>>>     1680x1050 laptop screen. It's so small that initially i even overlooked
>>>     that i started it. It should multiplex pixels up to a reasonable screen
>>>     size by default.
>>>        
>> No virtualization emulator today does this, it is not a reasonable thing to
>> do.  Unless you are running compiz and use hardware scaling.  We should look
>> into it.
>>      
> I tried VirtualBox and it does something sane here: while it does not give a
> larger guest screen area, it at least creates a large enough X window, with
> any border area whited out.
>
> That looks far more pleasing aesthetically than the Qemu method of resizing
> the full window wildly. The guest still resizes, but it stays within the same
> X window and the border is white.
>
> Also, with Windows guests, the Windows side resolution resizes dynamically if
> the window is grown/shrunk by the user. (I suspect that is done via a paravirt
> driver on the guest side, through 'Guest Additions'.)
>    

We have to core pieces to do this in qemu.  The vmware vga driver 
supports this and it just takes a small userspace program to resize the 
VGA on demand.

The problem is, you need a way to communicate from the host to the 
guest's userspace.  We did not have a mechanism until very recently 
(virtio-serial) because we've encountered a ton of resistance within the 
kernel community for the things we've proposed.

VirtualBox gets around this by just shipping their own Linux drivers.

Regards,

Anthony Liguori

> 	Ingo
>    

Re: KVM usability

[Daniel P. Berrange]

On Mon, Mar 01, 2010 at 09:14:02AM -0600, Anthony Liguori wrote:
> On 02/27/2010 04:56 AM, Ingo Molnar wrote:
> 
> The other type of user we target is power virtualization/emulation 
> users.  We certainly could do better for this type of user but it's 
> never going to fit what your expectation of desktop virtualization is.  
> Qemu is never going to be like running VMware Workstation or VirtualBox.
> 
> There is very little split between qemu and kvm from a project 
> perspective.  If anything, our lack of focus on desktop virtualization 
> comes from the split between qemu/kvm and libvirt.  All of the end-user 
> usability is done in the context of libvirt and virt-manager whereas the 
> qemu/kvm developers tend to focus on performance, features, and robustness.
> 
> I think we don't spend enough time in qemu/kvm thinking about how we 
> interact with libvirt with respect to end-to-end usability.  It's 
> something I do believe we need to address and it's an area I have been 
> focusing on recently.

On the other side of things, we (in libvirt community) have tended to be
more focused on server virtualization use cases, than desktop ones. There
is certainly alot more we could be doing for end user desktop usability
in both libvirt & QEMU/KVM parts. As & when we move forward with spice
integration, we'll get some significant improvements in usability through
the improved graphics performance, flexible guest display resolution, and
audio tunnelling to the clients.

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: KVM usability

[Zachary Amsden]

On 02/27/2010 07:25 AM, Ingo Molnar wrote:
> * Zachary Amsden<zamsden@redhat.com>  wrote:
>
> [...]
>    
>> Second, it's not over-modularized.  The modules are the individual
>> components of the architecture.  How would you propose to put it
>> differently.  They really can't naturally combine.  And with the
>> code quality of qemu in general being problematic by Linux kernel
>> standards, it's not natural to move the device emulation directly
>> into the kernel module.  So this is why we are where we are today.
>>      
> I'm not talking about moving it into a kernel _module_ - albeit that
> alone is a worthwile thing to do for any performance sensitive hw
> component.
>
> I was talking about the option of a clean, stripped down Qemu base
> hosted in the kernel proper, in linux/tools/kvm/ or so. If i were
> running a virtualization effort it would be the first place i'd
> consider to put my tooling into.
>    

The problem is there is no way to clean and strip down the Qemu code.  
It's got nicely abstracted bus and device interfaces, for example, but 
then these go poking under the covers at things which require 
interacting with the display rendering library or remoting interface, 
which is not something to reasonably do in the kernel.

So ripping out a clean part interface like PCI bus infrastructure and 
using it in the kernel, for example, does nothing except put that 
infrastructure in two different places, because everything the kernel 
does, userspace will have to do again anyway.  So now you have twice as 
much code involving the same idea and you have to keep the pieces in 
sync and from trampling each other.

The only parts that warrant such complexity and high risk for bugs are 
performance critical things like the PIT and APIC.

Zach

Re: KVM usability

[Arnaldo Carvalho de Melo]

Em Mon, Mar 01, 2010 at 06:48:07AM -1000, Zachary Amsden escreveu:
> On 02/27/2010 07:25 AM, Ingo Molnar wrote:
>> I'm not talking about moving it into a kernel _module_ - albeit that
>> alone is a worthwile thing to do for any performance sensitive hw
>> component.
>>
>> I was talking about the option of a clean, stripped down Qemu base
>> hosted in the kernel proper, in linux/tools/kvm/ or so. If i were
>> running a virtualization effort it would be the first place i'd
>> consider to put my tooling into.

> So ripping out a clean part interface like PCI bus infrastructure and  
> using it in the kernel, for example, does nothing except put that  
> infrastructure in two different places, because everything the kernel  
> does, userspace will have to do again anyway.  So now you have twice as  
> much code involving the same idea and you have to keep the pieces in  
> sync and from trampling each other.
>
> The only parts that warrant such complexity and high risk for bugs are  
> performance critical things like the PIT and APIC.

I guess there is some misunderstanding here, the tools/ directory that
lives in the kernel _sources_, has no kernel source, its all userspace
stuff.

- Arnaldo

Re: KVM usability

[Zachary Amsden]

On 03/01/2010 07:41 AM, Arnaldo Carvalho de Melo wrote:
> Em Mon, Mar 01, 2010 at 06:48:07AM -1000, Zachary Amsden escreveu:
>    
>> On 02/27/2010 07:25 AM, Ingo Molnar wrote:
>>      
>>> I'm not talking about moving it into a kernel _module_ - albeit that
>>> alone is a worthwile thing to do for any performance sensitive hw
>>> component.
>>>
>>> I was talking about the option of a clean, stripped down Qemu base
>>> hosted in the kernel proper, in linux/tools/kvm/ or so. If i were
>>> running a virtualization effort it would be the first place i'd
>>> consider to put my tooling into.
>>>        
>    
>> So ripping out a clean part interface like PCI bus infrastructure and
>> using it in the kernel, for example, does nothing except put that
>> infrastructure in two different places, because everything the kernel
>> does, userspace will have to do again anyway.  So now you have twice as
>> much code involving the same idea and you have to keep the pieces in
>> sync and from trampling each other.
>>
>> The only parts that warrant such complexity and high risk for bugs are
>> performance critical things like the PIT and APIC.
>>      
> I guess there is some misunderstanding here, the tools/ directory that
> lives in the kernel _sources_, has no kernel source, its all userspace
> stuff.

Yeah, no morning coffee :=)

So instead we are advocating forking qemu into the kernel.  That makes 
even less sense.  It's not sustainable or eco-friendly to either community.

Zach

Re: KVM usability

[Ingo Molnar]

* Zachary Amsden <zamsden@redhat.com> wrote:

> > I guess there is some misunderstanding here, the tools/ directory that 
> > lives in the kernel _sources_, has no kernel source, its all userspace 
> > stuff.
> 
> Yeah, no morning coffee :=)
> 
> So instead we are advocating forking qemu into the kernel.  That makes even 
> less sense.  It's not sustainable or eco-friendly to either community.

Here's our experience with tools/perf/. Hosting the project in the kernel 
proper helped its quality immensely:

 - It's much easier to synchronize new features on the kernel side and on the
   user-space side. The two go hand in hand - they are often implemented in
   the same patch.

 - It's released together with the kernel, which gives a periodic 3 months
   release frequency. Not too slow, not too fast.

 - Lots of eyeballs and interest. In its mere 8 months of existence
   tools/perf/ has attracted more than 60 contributors already, and 35 KLOC of
   new code has been written.

 - Code quality requirements are that of the kernel's. No muck allowed and 
   it's not hard to explain what kind of code is preferred.

 - Tool breakage bisection is a breeze: there's never any mismatch between
   tools/perf and the kernel counterpart. With a separate package we'd
   have more complex test and bisection scenarios.

 - Code distribution is easy: it comes with the kernel. This spreads the code
   far and wide. It's easy for kernel developers to jump in and help out, the
   latest devel code is always there, a single 'cd tools/perf/; make -j install'
   command away.

 - Code reuse: we started sharing/librarizing code with the kernel: bitmap.h,
   hash.h, list.h, rbtree.h, bitops.h, prefetch.h.

etc.

In the KVM context this was obviously only a suggestion though. If i were 
hacking on kvm-qemu i wouldnt hesitate for a moment to do it: the project has 
very close ties to kernel-KVM and repo level unification would create various 
synergies - but you are hacking on it, not me ;-)

If i were doing it i'd probably start with a cleaned up and stripped down 
version of Qemu, to make it eligible for mainline kernel inclusion.

	Ingo

Re: KVM usability

[Dustin Kirkland]

On Sat, Feb 27, 2010 at 4:56 AM, Ingo Molnar <mingo@elte.hu> wrote:
> Here's my experience with it:
>
>  - qemu-kvm starts up with a miniature resolution by default. 640x480 - on my
>   1680x1050 laptop screen. It's so small that initially i even overlooked
>   that i started it. It should multiplex pixels up to a reasonable screen
>   size by default.
>
>  - The mouse is trapped straight away by default if you click into it. That's
>   very annoying if you actually try to integrate a guest OS into your desktop
>   - it's not just 'another, slightly weird app' but a sticky, opinionated GUI
>   component that you have to fight with all the time.
>
>  - Once trapped it's not obvious how to untrap the mouse. The qemu window
>   title says:
>
>          QEMU: Press Ctrl-ALT to exit grab
>
>   Of course once you _know_ what a 'grab' is, you'll know where to look.
>   At minimum it should say:
>
>          QEMU: Press Ctrl-ALT to exit mouse grab
>
>   But to first-time users it's an annoying trap of the mouse and with no
>   obvious place to look for help. [besides, it doesnt tell which Ctrl and
>   which ALT to use - it's the left side. The right side Ctrl does not work.]
>
>  - Graphics performance is awful even with the 640x480 miniature version.
>   During bootup I can see it drawing single characters. This is a Core2
>   2.8GHz.
>
>  - Sound does not work by default. I have to go dig into command-line options
>   to see that i need to add: "-soundhw all". Why isnt sound enabled by
>   default?

..snip..

> ( I could go on and on about finer details of good integration, like the
>  difficulty of integrating host/guest files, networking, no way to quickly
>  attach ISOs via that window, no way to activate networking, sound and no way
>  to snapshot, no way to increase memory size except a command line option. )

...snip...

> I'm not trolling you at all: is it _really_ not obvious to you that the
> KVM/qemu usability status quo honestly sucks, to an unbiased observer?

Hi there-

To a few of your comments shown above...I've noticed these two, and
they've bothered me a bit.

As Jan (and others) have mentioned, these are often "addressed" by
frontends running on top of kvm (and libvirt).  I look at kvm as a
swissarmy knife, with lots of buttons and knobs and whistles, but you
have to give it all of the "right" options to get it to do what you
want.

The graphical frontends make this easier.  But running KVM from the
command line, with the bare minimal options, is often a functional,
but non-satisfactory impression of KVM.

Maintaining the qemu-kvm package in Ubuntu, I've seen this over and
over and over again, with Ubuntu users expecting a much smoother
graphical experience.  "Why can't I get my mouse back?"  "Why doesn't
sound work?"  "How do I make networking work?"  "Why is the resolution
so low?"  Etc.

For these reasons (among others), we've created a little package and a
python script in Ubuntu called Testdrive [1], which basically
downloads an ISO (using rsync/zsync against cached images, if
possible), and launches a VM
with a "healthy" set of options.  The package puts a pointy/clicky
option in the Applications menu, and makes for a *very* easy way for
Ubuntu users to launch a VM and try out the daily ISO builds (as well
as install to the backing disk image).

Once it's running, the kvm process looks something like this:

kvm -m 512 -cdrom
/home/kirkland/.cache/testdrive/iso/lucid-desktop-amd64.iso -drive
file=/home/kirkland/.cache/testdrive/img/testdrive-disk-0086OD.img,if=virtio,index=0,boot=on
-usb -usbdevice tablet -net nic,model=virtio -net user -soundhw es1370

Among these:
 * 512MB is a nice step up from the 128MB by default (this one scales
based on your hosts memory)
 * virtio is used for disk and network for nice performance gains
 * usb device tablet greatly improves the mouse experience
 * sound card added
 * qcow2 sparse disk image
 * desktop is 1024 x 720

For what it's worth, you can just as easily run:
  testdrive -u http://download.fedoraproject.org/pub/fedora/linux/releases/12/Live/i686/Fedora-12-i686-Live.iso
or any other http://, ftp://, rsync://, or file:/// accessible ISO.

By no means is Testdrive supposed to be a production quality frontend
for KVM (or replace any of the libvirt-based utilities like
virt-manager).  But I think it shows that KVM's default set of options
is a little shy of what people expect for Desktop virtualization at
least.  For servers, lightweight defaults are perhaps more acceptable?
 But KVM has a number of new users by way of Ubuntu users just
pointing and clicking "Testdrive" and launching a VM.

[1] http://launchpad.net/testdrive

Cheers,
:-Dustin

Re: KVM usability

[Anthony Liguori]

On 03/01/2010 02:56 PM, Ingo Molnar wrote:
> Here's our experience with tools/perf/. Hosting the project in the kernel
> proper helped its quality immensely:
>
>   - It's much easier to synchronize new features on the kernel side and on the
>     user-space side. The two go hand in hand - they are often implemented in
>     the same patch.
>    

Kernel features and qemu features usually don't have a great amount of 
intersect.  All of the problems you've described are strictly in the 
qemu space.

>   - It's released together with the kernel, which gives a periodic 3 months
>     release frequency. Not too slow, not too fast.
>    

qemu release range in length from 3-6 months depending on distribution 
schedules.  They are very regular.

>   - Lots of eyeballs and interest. In its mere 8 months of existence
>     tools/perf/ has attracted more than 60 contributors already, and 35 KLOC of
>     new code has been written.
>    

In our last release, we had around 100 contributors and about 100 KLOC 
of code written.  We've got a lot of eyeballs and a lot of interest.

>   - Code quality requirements are that of the kernel's. No muck allowed and
>     it's not hard to explain what kind of code is preferred.
>    

Code quality is subjective.  We have a different coding style.

>   - Tool breakage bisection is a breeze: there's never any mismatch between
>     tools/perf and the kernel counterpart. With a separate package we'd
>     have more complex test and bisection scenarios.
>    

KVM has a backwards compatible ABI so there's no such thing as mismatch 
between user and kernel space.

>   - Code distribution is easy: it comes with the kernel. This spreads the code
>     far and wide. It's easy for kernel developers to jump in and help out, the
>     latest devel code is always there, a single 'cd tools/perf/; make -j install'
>     command away.
>   - Code reuse: we started sharing/librarizing code with the kernel: bitmap.h,
>     hash.h, list.h, rbtree.h, bitops.h, prefetch.h.
>    

You could argue that any project should be in the kernel for these 
reasons.  I see no reason why something as like KVM should be part of 
the kernel and udev shouldn't be.

> etc.
>
> In the KVM context this was obviously only a suggestion though. If i were
> hacking on kvm-qemu i wouldnt hesitate for a moment to do it: the project has
> very close ties to kernel-KVM and repo level unification would create various
> synergies - but you are hacking on it, not me ;-)
>
> If i were doing it i'd probably start with a cleaned up and stripped down
> version of Qemu, to make it eligible for mainline kernel inclusion.
>    

You should try it.  I think you'll find that it's not as obvious thing 
to do as you think it is.

Regards,

Anthony Liguori

> 	Ingo
>    

Re: KVM usability

[Anthony Liguori]

On 03/01/2010 03:12 PM, Dustin Kirkland wrote:
> kvm -m 512 -cdrom
> /home/kirkland/.cache/testdrive/iso/lucid-desktop-amd64.iso -drive
> file=/home/kirkland/.cache/testdrive/img/testdrive-disk-0086OD.img,if=virtio,index=0,boot=on
> -usb -usbdevice tablet -net nic,model=virtio -net user -soundhw es1370
>
> Among these:
>   * 512MB is a nice step up from the 128MB by default (this one scales
> based on your hosts memory)
>    

I'm not opposed to bumping from 128M to something else.  I'd like to see 
some indication though other than "seems reasonable" as to what the 
value should be.  For instance, if a Fedora 12 install doesn't work 
without 256M, that's a solid argument to bump the allocation.  Short 
term, I'd like to see this globally configurable.

>   * virtio is used for disk and network for nice performance gains
>    

The problem with virtio is that very few OSes have the drivers right 
now.  For networking, we can do better, and I've just written a spec for 
a new feature to attempt to address this[1].  For disk, I think we're 
stuck with ide as the default for a long time.

>   * usb device tablet greatly improves the mouse experience
>    

Defaulting usb to on and defaulting to a usb tablet is a reasonable 
thing to do IMHO.

>   * sound card added
>    

This personally would annoy me but if there was wide consensus that this 
was right, I'd be okay with it.

>   * qcow2 sparse disk image
>    

This is independent of qemu.

>   * desktop is 1024 x 720
>    

1024x768 and this is what the default is today anyway.

[1] http://wiki.qemu.org/Features/NetDeviceFailover

Regards,

Anthony Liguori

Re: KVM usability

[Zachary Amsden]

On 03/01/2010 11:45 AM, Anthony Liguori wrote:
> On 03/01/2010 02:56 PM, Ingo Molnar wrote:
>
>>   - Code distribution is easy: it comes with the kernel. This spreads 
>> the code
>>     far and wide. It's easy for kernel developers to jump in and help 
>> out, the
>>     latest devel code is always there, a single 'cd tools/perf/; make 
>> -j install'
>>     command away.
>>   - Code reuse: we started sharing/librarizing code with the kernel: 
>> bitmap.h,
>>     hash.h, list.h, rbtree.h, bitops.h, prefetch.h.
>
> You could argue that any project should be in the kernel for these 
> reasons.  I see no reason why something as like KVM should be part of 
> the kernel and udev shouldn't be.

gcc and the kernel are quite closely coupled, btw.

>
>> etc.
>>
>> In the KVM context this was obviously only a suggestion though. If i 
>> were
>> hacking on kvm-qemu i wouldnt hesitate for a moment to do it: the 
>> project has
>> very close ties to kernel-KVM and repo level unification would create 
>> various
>> synergies - but you are hacking on it, not me ;-)
>>
>> If i were doing it i'd probably start with a cleaned up and stripped 
>> down
>> version of Qemu, to make it eligible for mainline kernel inclusion.
>
> You should try it.  I think you'll find that it's not as obvious thing 
> to do as you think it is.

Yes.  Yes.

We've all looked at the hulking hairy behemoth of qemu and said, I think 
I can clean this beast up and make him look like a proper gentlemen.  
Then we tried to shave off the excess hair, only to find not only did it 
grow back as fast as we trimmed it, but his complexion underneath was 
mottled and inconsistent, so much so that the hair was actually helping.

In the end, we just settled for dressing him up in a suit and tie.

Zach

Re: KVM usability

[Ingo Molnar]

* Anthony Liguori <anthony@codemonkey.ws> wrote:

> On 03/01/2010 02:56 PM, Ingo Molnar wrote:
> >Here's our experience with tools/perf/. Hosting the project in the kernel
> >proper helped its quality immensely:
> >
> >  - It's much easier to synchronize new features on the kernel side and on the
> >    user-space side. The two go hand in hand - they are often implemented in
> >    the same patch.
> 
> Kernel features and qemu features usually don't have a great amount of 
> intersect.  All of the problems you've described are strictly in the qemu 
> space.

IMO that's a bug, not a feature. There should be a lot more interaction 
between kvm-qemu and KVM: for example Qemu should have a feature to install 
paravirt drivers in the guest, this would be helped by living in the kernel 
repo.

> 
> >  - It's released together with the kernel, which gives a periodic 3 months
> >    release frequency. Not too slow, not too fast.
> 
> qemu release range in length from 3-6 months depending on
> distribution schedules.  They are very regular.

The Linux kernel is released every 3 months, +- one week. Our experience is 
that even 6 months would be (way) too painful for distros.

> >  - Lots of eyeballs and interest. In its mere 8 months of existence
> >    tools/perf/ has attracted more than 60 contributors already, and 35 KLOC of
> >    new code has been written.
> 
> In our last release, we had around 100 contributors and about 100 KLOC of 
> code written.  We've got a lot of eyeballs and a lot of interest.

Congrats!

> >  - Code quality requirements are that of the kernel's. No muck allowed and
> >    it's not hard to explain what kind of code is preferred.
> 
> Code quality is subjective.  We have a different coding style.

That's somewhat of a problem when for example a KVM kernel-space developer 
crosses into Qemu code and back. Two separate styles, etc. I certainly 
remember a 'culture clash' when going from the kernel into Qemu and back. 
Different principles, different culture. It's better to standardize.

I.e. i think it would be useful to make it more of 'one' project, instead of 
this separation.

> >  - Tool breakage bisection is a breeze: there's never any mismatch between
> >    tools/perf and the kernel counterpart. With a separate package we'd
> >    have more complex test and bisection scenarios.
> 
> KVM has a backwards compatible ABI so there's no such thing as mismatch 
> between user and kernel space.

perf too is ABI compatible (between releases) - still bisection is a lot 
easier because the evolution of a particular feature can be bisected back to.

Btw., KVM certainly ha ABI breakages around 2.6.16(?) when it was added, even 
of released versions. Also, within a development version you sure sometimes 
iterate a new ABI component, right? With a time-coherent repository both 
intentional and unintentional breakages and variations can be bisected back to 
as they happened.

This is an unconditional advantage and i made use of it numerous times.

> >  - Code distribution is easy: it comes with the kernel. This spreads the code
> >    far and wide. It's easy for kernel developers to jump in and help out, the
> >    latest devel code is always there, a single 'cd tools/perf/; make -j install'
> >    command away.
> >  - Code reuse: we started sharing/librarizing code with the kernel: bitmap.h,
> >    hash.h, list.h, rbtree.h, bitops.h, prefetch.h.
> 
> You could argue that any project should be in the kernel for these
> reasons.  I see no reason why something as like KVM should be part
> of the kernel and udev shouldn't be.

Yes, you are quite correct: udev has been argued to be a prime candidate for 
tools/. (and some other kernel utilities as well)

>From a design POV all 'system/kernel utilities', which make little sense 
without the kernel and are license compatible can (and arguably should) move 
there.

Obviously there's no pressure to do so - it's only an opportunity.

> > etc.
> >
> > In the KVM context this was obviously only a suggestion though. If i were 
> > hacking on kvm-qemu i wouldnt hesitate for a moment to do it: the project 
> > has very close ties to kernel-KVM and repo level unification would create 
> > various synergies - but you are hacking on it, not me ;-)
> >
> > If i were doing it i'd probably start with a cleaned up and stripped down 
> > version of Qemu, to make it eligible for mainline kernel inclusion.
> 
> You should try it.  I think you'll find that it's not as obvious thing to do 
> as you think it is.

A few years ago I looked into cleaning up Qemu, when i hacked KVM and Qemu. I 
also wanted to have a 'qemu light', which is both smaller and cleaner, and 
still fits to KVM. It didnt look particularly hard back then - but it's 
certainly not zero amount of work.

Cleanups pay - they make a piece of code both more hackable, more debuggable 
and more appealing to new developers. (i suspect you have no argument with 
that notion) Also note that it wasnt me who suggested that Qemu wouldnt fit 
the kernel standards as-is - it was raised by others in this discussion.

	Ingo

Re: KVM usability

[Ingo Molnar]

* Zachary Amsden <zamsden@redhat.com> wrote:

> On 03/01/2010 11:45 AM, Anthony Liguori wrote:
> >On 03/01/2010 02:56 PM, Ingo Molnar wrote:
> >
> >>  - Code distribution is easy: it comes with the kernel. This
> >>spreads the code
> >>    far and wide. It's easy for kernel developers to jump in and
> >>help out, the
> >>    latest devel code is always there, a single 'cd tools/perf/;
> >>make -j install'
> >>    command away.
> >>  - Code reuse: we started sharing/librarizing code with the
> >>kernel: bitmap.h,
> >>    hash.h, list.h, rbtree.h, bitops.h, prefetch.h.
> >
> >You could argue that any project should be in the kernel for these
> >reasons.  I see no reason why something as like KVM should be part
> >of the kernel and udev shouldn't be.
> 
> gcc and the kernel are quite closely coupled, btw.

Note that GCC isnt very much coupled to the kernel. The _kernel_ is coupled to 
GCC pretty much. (although it's known to build with certain versions of LVVM 
and also perhaps ICC)

The most obvious utilities to move into tools/ are the ones that are 
bidirectionally coupled: udev, perhaps util-linux, etc.

	Ingo

Re: KVM usability

[Dustin Kirkland]

On Mon, Mar 1, 2010 at 9:14 AM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> On 02/27/2010 04:56 AM, Ingo Molnar wrote:
>>  - But i'm a more advanced user so i dont need help screens, i knew that
>> the
>>    "go full screen" hotkey is:
>>
>>            LeftCtrl-LeftALT-F
>>
>>    ... except that it is a one-way road: pressing it for a second time
>> does
>>    not restore the window, trapping me in the guest altogether! Ctrl-ALT
>> does
>>    not exit the trap either. I had to shut down the guest to get back my X
>>    desktop.
>>
>
> That's a bug.

The upstream QEMU bug for this is:
 * https://bugs.launchpad.net/qemu/+bug/483251

However, I just tried it on qemu-kvm_0.12.3 and I'm able to toggle in
and out of fullscreen mode with ctrl-alt-F ad nauseum.

:-Dustin

Re: KVM usability

[Anthony Liguori]

On 03/01/2010 06:30 PM, Ingo Molnar wrote:
> IMO that's a bug, not a feature. There should be a lot more interaction
> between kvm-qemu and KVM: for example Qemu should have a feature to install
> paravirt drivers in the guest, this would be helped by living in the kernel
> repo.
>    

Not in the slightest bit.

To support automatically installing paravirt drivers in a guest, we need 
to distribute an ISO containing *binary* versions of drivers.  For 
Windows, there's a licensing issue that I described earlier with respect 
to signing.  Figuring out distribution is non-trivial and is being 
worked on.  So far, Red Hat are the only ones actually capable of 
producing signed binaries (no mere mortal can do it).  For Linux 
drivers, we need to be able to ship different versions of the kernel 
drivers for different distribution kernels if we don't want to rely on 
what they ship.

The way we've tackled this in the past is by having an awk script that 
automatically converts the virtio drivers into something buildable 
across kernel versions.  It's incredibly difficult to maintain and we 
stopped maintaining it about a year ago when virtio drivers became 
common in all distro kernels.  See 
http://git.kernel.org/?p=virt/kvm/kvm-guest-drivers-linux.git if you're 
interested.

What would make this much easier for us is if we could add all of the 
#ifdef's for various kernel versions in the mainline source tree.  I'm 
not holding my breath for that though :-)

But once we had an ISO with binary drivers (and such a thing is 
available for Windows today), it's just a matter of adding an option to 
change the CDROM to the shipped ISO.  This is purely within qemu and 
doesn't touch kvm.ko at all.

Once the winpv driver's binary hosting is sorted out, virt-manager will 
have this feature.  There are zero changes required to the kvm kernel 
code to support this.

>>      
>>>   - It's released together with the kernel, which gives a periodic 3 months
>>>     release frequency. Not too slow, not too fast.
>>>        
>> qemu release range in length from 3-6 months depending on
>> distribution schedules.  They are very regular.
>>      
> The Linux kernel is released every 3 months, +- one week. Our experience is
> that even 6 months would be (way) too painful for distros.
>    

I expect that we'll eventually even out to a consistent release 
schedule. For now, we're still trying to see what fits us best.  The 
last 3 month release was very compressed so we're trying something a 
little longer this time.

>>>   - Code quality requirements are that of the kernel's. No muck allowed and
>>>     it's not hard to explain what kind of code is preferred.
>>>        
>> Code quality is subjective.  We have a different coding style.
>>      
> That's somewhat of a problem when for example a KVM kernel-space developer
> crosses into Qemu code and back. Two separate styles, etc. I certainly
> remember a 'culture clash' when going from the kernel into Qemu and back.
> Different principles, different culture. It's better to standardize.
>    

Some would argue that having diversity of culture is a good thing that 
breeds creative thinking :-)

It's annoying to switch coding styles but I don't think it's a major 
problem for anyone.

>>>   - Tool breakage bisection is a breeze: there's never any mismatch between
>>>     tools/perf and the kernel counterpart. With a separate package we'd
>>>     have more complex test and bisection scenarios.
>>>        
>> KVM has a backwards compatible ABI so there's no such thing as mismatch
>> between user and kernel space.
>>      
> perf too is ABI compatible (between releases) - still bisection is a lot
> easier because the evolution of a particular feature can be bisected back to.
>
> Btw., KVM certainly ha ABI breakages around 2.6.16(?) when it was added, even
> of released versions.

That was a one-time thing in the very early days of KVM.

>   Also, within a development version you sure sometimes
> iterate a new ABI component, right?

It's not really happened.  We introduce new ABIs very rarely.  KVM has a 
very defined purpose; it provides CPU virtualization.  We only extend 
the ABI to support new CPU features that we didn't previously support 
and since these things are defined by the Intel architecture, it's 
fairly easy to define the ABI properly up front.

>   With a time-coherent repository both
> intentional and unintentional breakages and variations can be bisected back to
> as they happened.
>
> This is an unconditional advantage and i made use of it numerous times.
>    

We used to keep the kernel code in the same repository as the userspace 
code.  We stopped doing that about a year ago and it's rare that we have 
a circumstance where joint bisecting is required.

>> You should try it.  I think you'll find that it's not as obvious thing to do
>> as you think it is.
>>      
> A few years ago I looked into cleaning up Qemu, when i hacked KVM and Qemu. I
> also wanted to have a 'qemu light', which is both smaller and cleaner, and
> still fits to KVM. It didnt look particularly hard back then - but it's
> certainly not zero amount of work.
>    

First impressions are deceptive.  My long term goal for qemu is to get 
to a point where the device models live independently of the rest of 
qemu.  I think it's reasonable to split these devices into a modular 
library that can then be used by other applications.

That would make it possible to create a kvm-specific virtualization tool 
that only supported tap and linux-aio and the bare minimum numbers of 
devices.  It would be easy to look at and for kernel hackers to play with.

But to be honest, it would never replace qemu.  Once you add a VNC/Spice 
server (you need remote connectivity), support for sparse file formats 
(because we can't wait forever for btrfs to solve all of our problems), 
live migration and snapshotting (required ticky marks for 
virtualization), a management layer, and all of the other bells and 
whistles, you'll find that you did an awful lot of work to recreate what 
qemu does.

Most people that have gone down this road believe that it's more 
efficient to just improve qemu's quality than it is to try and replicate 
it.  So far, we've been pretty successful IMHO.

Regards,

Anthony Liguori

> Cleanups pay - they make a piece of code both more hackable, more debuggable
> and more appealing to new developers. (i suspect you have no argument with
> that notion) Also note that it wasnt me who suggested that Qemu wouldnt fit
> the kernel standards as-is - it was raised by others in this discussion.
>
> 	Ingo
>    

Re: KVM usability

[Alexander Graf]

On 01.03.2010, at 22:59, Anthony Liguori wrote:

> On 03/01/2010 03:12 PM, Dustin Kirkland wrote:
>> kvm -m 512 -cdrom
>> /home/kirkland/.cache/testdrive/iso/lucid-desktop-amd64.iso -drive
>> file=/home/kirkland/.cache/testdrive/img/testdrive-disk-0086OD.img,if=virtio,index=0,boot=on
>> -usb -usbdevice tablet -net nic,model=virtio -net user -soundhw es1370
>> 
>> Among these:
>>  * 512MB is a nice step up from the 128MB by default (this one scales
>> based on your hosts memory)
>>   
> 
> I'm not opposed to bumping from 128M to something else.  I'd like to see some indication though other than "seems reasonable" as to what the value should be.  For instance, if a Fedora 12 install doesn't work without 256M, that's a solid argument to bump the allocation.  Short term, I'd like to see this globally configurable.

I keep a patch in the SUSE version for quite some time now that bumps the default to 384 for qemu-kvm. That was the first "round" number where an openSUSE installation worked.

Alex

Re: KVM usability

[Anthony Liguori]

On 03/01/2010 08:34 PM, Alexander Graf wrote:
> On 01.03.2010, at 22:59, Anthony Liguori wrote:
>
>    
>> On 03/01/2010 03:12 PM, Dustin Kirkland wrote:
>>      
>>> kvm -m 512 -cdrom
>>> /home/kirkland/.cache/testdrive/iso/lucid-desktop-amd64.iso -drive
>>> file=/home/kirkland/.cache/testdrive/img/testdrive-disk-0086OD.img,if=virtio,index=0,boot=on
>>> -usb -usbdevice tablet -net nic,model=virtio -net user -soundhw es1370
>>>
>>> Among these:
>>>   * 512MB is a nice step up from the 128MB by default (this one scales
>>> based on your hosts memory)
>>>
>>>        
>> I'm not opposed to bumping from 128M to something else.  I'd like to see some indication though other than "seems reasonable" as to what the value should be.  For instance, if a Fedora 12 install doesn't work without 256M, that's a solid argument to bump the allocation.  Short term, I'd like to see this globally configurable.
>>      
> I keep a patch in the SUSE version for quite some time now that bumps the default to 384 for qemu-kvm. That was the first "round" number where an openSUSE installation worked.
>    

If someone works up a patch and tests at least a couple types of guests 
to confirm that they all install with that number, I'd be happy to apply 
it (although we need some trickery to support older pc versions).

Regards,

Anthony Liguori

> Alex

Re: KVM usability

[Dustin Kirkland]

[-- Attachment #1: Type: text/plain, Size: 3055 bytes --]

On Mon, 2010-03-01 at 15:59 -0600, Anthony Liguori wrote:
> On 03/01/2010 03:12 PM, Dustin Kirkland wrote:
> > kvm -m 512 -cdrom
> > /home/kirkland/.cache/testdrive/iso/lucid-desktop-amd64.iso -drive
> > file=/home/kirkland/.cache/testdrive/img/testdrive-disk-0086OD.img,if=virtio,index=0,boot=on
> > -usb -usbdevice tablet -net nic,model=virtio -net user -soundhw es1370
> >
> > Among these:
> >   * 512MB is a nice step up from the 128MB by default (this one scales
> > based on your hosts memory)
> >    
> 
> I'm not opposed to bumping from 128M to something else.  I'd like to see 
> some indication though other than "seems reasonable" as to what the 
> value should be.  For instance, if a Fedora 12 install doesn't work 
> without 256M, that's a solid argument to bump the allocation.  Short 
> term, I'd like to see this globally configurable.

That would be very nice!  I think 256MB (or even 384MB) would be a
reasonable default.  Modern Ubuntu LiveCD's will work well with 256MB,
but 128MB is essentially unusable.  We should just make sure that the
default is intelligently chosen based on the host's available memory
resources.

The crude logic in testdrive looks like this:

if len(MEM) == 0:
        total = commands.getoutput("grep ^MemTotal /proc/meminfo | awk '{print $2}'")
        if total > 1000000:
                MEM = 512
        elif total > 750000:
                MEM = 384
        else:
                MEM = 256

> >   * virtio is used for disk and network for nice performance gains
> >    
> 
> The problem with virtio is that very few OSes have the drivers right 
> now.  For networking, we can do better, and I've just written a spec for 
> a new feature to attempt to address this[1].  For disk, I think we're 
> stuck with ide as the default for a long time.

That's safe, reasonable position for upstream to make.  And a sane one
for distro's to tweak depending on their intended target guest OS's.

> >   * usb device tablet greatly improves the mouse experience
> >    
> 
> Defaulting usb to on and defaulting to a usb tablet is a reasonable 
> thing to do IMHO.

\o/  Definitely a better user experience.

> >   * sound card added
> >    
> 
> This personally would annoy me but if there was wide consensus that this 
> was right, I'd be okay with it.

I agree that sound-on by default might be annoying to some.  Speaking
for myself, if I don't want to hear guest noises, I usually don't want
to hear host noises either.  In which case I mute the sound on my host
and solve both problems.  But I don't feel strongly about this one.

> >   * qcow2 sparse disk image
> >    
> 
> This is independent of qemu.

Understood.  This merely seemed like the best-practice,
most-recommended, best-performing backing disk format for unwitting kvm
users launching VMs when they don't actually care what the backing
format is.

> >   * desktop is 1024 x 720
> >    
> 
> 1024x768 and this is what the default is today anyway.

Ack.

:-Dustin

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: KVM usability

[Chris Webb]

Dustin Kirkland <kirkland@canonical.com> writes:

> On Mon, 2010-03-01 at 15:59 -0600, Anthony Liguori wrote:
> > 
> > Defaulting usb to on and defaulting to a usb tablet is a reasonable 
> > thing to do IMHO.
> 
> \o/  Definitely a better user experience.

I remember about a year ago, someone asserting on the list that -usbdevice
tablet was very CPU intensive even when not in use, and should be avoided if
mouse support wasn't needed, e.g. on non-graphical VMs. Was that actually a
significant hit, and is it still true today?

Cheers,

Chris.

Re: KVM usability

[Chris Webb]

Ingo Molnar <mingo@elte.hu> writes:

> Yes, you are quite correct: udev has been argued to be a prime candidate for 
> tools/. (and some other kernel utilities as well)

A small, static set of userspace like klibc (only 5M unpacked!) with enough
tools for rolling up in a standard initramfs would be especially nice, and
vastly less difficult to import than qemu.

It's a pain in the neck to have to build two versions of lots of bits of
userspace: one stripped down and statically linked for initramfs and one
full-featured for the main system. However, trying to avoid initramfs
altogether is an increasingly losing battle these days, and for quite
understandable reasons.  klibc + md* + mini lvm2 (enough to activate
volumes) perhaps?

Cheers,

Chris.

Re: KVM usability

[Peter Zijlstra]

On Mon, 2010-03-01 at 09:14 -0600, Anthony Liguori wrote:
> The real 
> question to ask is, why are you using qemu directly instead of using 
> virt-manager? 

Because I suspect Ingo, like me, is a command line user, launching a gui
to start kvm when there is a kvm command around just sounds daft.

Also, I just installed and tried it, virt-manager is a total piece of
shit, I wouldn't even know how to begin telling it how to start my
freshly baked kernel with serial console on stdio and some block image I
just created from the gentoo stage3 tarball.

That is, after 5 minutes clicking I have no idea how to even launch an
ISO with the thing, I prefer reading the kvm manpage over using some
mouse only gui crap like that.

Re: KVM usability

[Ingo Molnar]

* Ingo Molnar <mingo@elte.hu> wrote:

> Here's our experience with tools/perf/. Hosting the project in the kernel 
> proper helped its quality immensely:
> 
>  - It's much easier to synchronize new features on the kernel side and on the
>    user-space side. The two go hand in hand - they are often implemented in
>    the same patch.

Just look at an example from today, a perf+KVM feature patch posted by Yanmin 
Zhang:

  http://www.mail-archive.com/kvm@vger.kernel.org/msg29770.html

That single patch implements the following "perf kvm" commands:

  perf kvm top
  perf kvm record
  perf kvm report
  perf kvm diff

Both the kernel-space and the user-space changes are in that single patch.

Anyone who'd like to try it out can apply it and get an updated kernel plus 
updated tooling and can start profiling KVM guests straight away. You just 
check out the kernel, apply the patch and that's it - you can go. It doesnt 
get any more convenient than that to do development.

Such kind of a unified repository is a powerful concept, and we make use of 
those aspects of tools/perf/ every day. You could only pry it out of our cold, 
dead fingers ;-)

Btw., this is one of the things that FreeBSD does right - and i believe it is 
one of the technical concepts behind Apple's success as well. Apple, with a 
tenth's of Linux's effective R&D budget can consistently out-develop Linux. I 
think that's in part due to there not being a strict chinese wall between the 
Apple kernel, libraries and applications - it's one coherent project where 
everyone is well-connected to each piece, with no artificial project-cultural 
boundaries and barriers. People can and do move between those areas of the 
larger "Apple" project to achieve their goals - regardless of how many 
components need touching for a given area of interest.

IMHO we should learn from that - while we are good in many areas there's 
always aspects of Linux that can be improved. But i digress.

Thanks,

	Ingo

Re: KVM usability

[Nikolai K. Bochev]

I don't see where this argument is leading to. So far there are arguments that qemu/kvm sucks as a desktop virtualization, now suddenly the gui tools are shitty and everything should be done cli , because there's no man pages for virt-manager. Explain.

----- Original Message -----
From: Peter Zijlstra <peterz@infradead.org>
To: Anthony Liguori <anthony@codemonkey.ws>
Cc: Ingo Molnar <mingo@elte.hu>, Avi Kivity <avi@redhat.com>, Yanmin Zhang <yanmin_zhang@linux.intel.com>, ming m lin <ming.m.lin@intel.com>, sheng yang <sheng.yang@intel.com>, Jes Sorensen <Jes.Sorensen@redhat.com>, KVM General <kvm@vger.kernel.org>, Zachary Amsden <zamsden@redhat.com>, Gleb Natapov <gleb@redhat.com>, Arnaldo Carvalho de Melo <acme@redhat.com>, Fr??d??ric Weisbecker <fweisbec@gmail.com>, Thomas Gleixner <tglx@linutronix.de>, H. Peter Anvin <hpa@zytor.com>, Arjan van de Ven <arjan@infradead.org>, Cole Robinson <crobinso@redhat.com>
Sent: Tue, 02 Mar 2010 12:11:06 +0200 (EET)
Subject: Re: KVM usability
On Mon, 2010-03-01 at 09:14 -0600, Anthony Liguori wrote:
> The real 
> question to ask is, why are you using qemu directly instead of using 
> virt-manager? 
Because I suspect Ingo, like me, is a command line user, launching a gui
to start kvm when there is a kvm command around just sounds daft.
Also, I just installed and tried it, virt-manager is a total piece of
shit, I wouldn't even know how to begin telling it how to start my
freshly baked kernel with serial console on stdio and some block image I
just created from the gentoo stage3 tarball.
That is, after 5 minutes clicking I have no idea how to even launch an
ISO with the thing, I prefer reading the kvm manpage over using some
mouse only gui crap like that.
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html

Re: KVM usability

[Gerd Hoffmann]

On 03/02/10 14:37, Nikolai K. Bochev wrote:
> I don't see where this argument is leading to. So far there are
> arguments that qemu/kvm sucks as a desktop virtualization, now
> suddenly the gui tools are shitty and everything should be done cli ,
> because there's no man pages for virt-manager. Explain.

Lets face it:  virt-manager has a big bunch of problems.  For starters 
there is no (gui) way to create a virtual machine other than installing 
one from a iso image or via pxe.  So if you downloaded a disk image and 
want to boot it -> no cookie for you.  Likewise if you have a bunch of 
already installed guests and want to migrate from $othertool to 
virt-manager you can't do that easily.

virt-manager builds on top of libvirt, so you always have the option to 
use other tools (virsh command line shell for example) to get something 
done which virt-manager doesn't provide a gui for.  libvirt can handle 
everything Peter asked for without problems.  It still sucks though. 
The whole point of a GUI is that you do *not* have to go to libvirt.org 
to figure how to tweak the virtual machine config xml using "virsh edit 
$vmname".

cheers,
   Gerd

Re: KVM usability

[Ingo Molnar]

* Gerd Hoffmann <kraxel@redhat.com> wrote:

> On 03/02/10 14:37, Nikolai K. Bochev wrote:
> >I don't see where this argument is leading to. So far there are
> >arguments that qemu/kvm sucks as a desktop virtualization, now
> >suddenly the gui tools are shitty and everything should be done cli ,
> >because there's no man pages for virt-manager. Explain.
> 
> Lets face it:  virt-manager has a big bunch of problems.  For
> starters there is no (gui) way to create a virtual machine other
> than installing one from a iso image or via pxe.  So if you
> downloaded a disk image and want to boot it -> no cookie for you.
> Likewise if you have a bunch of already installed guests and want to
> migrate from $othertool to virt-manager you can't do that easily.
> 
> virt-manager builds on top of libvirt, so you always have the option
> to use other tools (virsh command line shell for example) to get
> something done which virt-manager doesn't provide a gui for.
> libvirt can handle everything Peter asked for without problems.  It
> still sucks though. The whole point of a GUI is that you do *not*
> have to go to libvirt.org to figure how to tweak the virtual machine
> config xml using "virsh edit $vmname".

>From the short exposure i had to the VirtualBox GUI, that app certainly looks 
usable. I was able to create & manage a virtual machine without any hassle, 
and everything Just Worked (tm). The various GUI controls for 
suspending/snapshotting/halting the virtual machines were intuitive as well, 
and mouse integration and various graphics details were what i'd expect from a 
modern solution.

So if Qemu/virt-manager/libvirt reached parity with the VirtualBox GUI, that 
would go a long, long way in satisfying non-expert virtualization users.

I guess the first step would be to move away from the 'lets support lots of 
crappy virtualization solutions at once, poorly' model, and pick one good 
combo (i'd go for Qemu+KVM) and turn it into a heck of an all-around solution. 
Then all the other combos will catch up as well. (or will wither away)

( Sidenote: i also looked at the VirtualBox kernel driver. Oh my ... i really 
  shouldnt have! They should migrate to the KVM kernel-side code ASAP ... )

Thanks,

	Ingo

Re: KVM usability

[Daniel P. Berrange]

On Tue, Mar 02, 2010 at 03:22:07PM +0100, Gerd Hoffmann wrote:
> On 03/02/10 14:37, Nikolai K. Bochev wrote:
> >I don't see where this argument is leading to. So far there are
> >arguments that qemu/kvm sucks as a desktop virtualization, now
> >suddenly the gui tools are shitty and everything should be done cli ,
> >because there's no man pages for virt-manager. Explain.
> 
> Lets face it:  virt-manager has a big bunch of problems.  For starters 
> there is no (gui) way to create a virtual machine other than installing 
> one from a iso image or via pxe.  So if you downloaded a disk image and 
> want to boot it -> no cookie for you.  Likewise if you have a bunch of 
> already installed guests and want to migrate from $othertool to 
> virt-manager you can't do that easily.

The ability to import existing KVM disk images is available in the 
latest release of virt-manager, targetted for Fedora 13. Migrating
from another tool like VMWare is a much harder than just importing
the disk image, since you have to update the drivers inside the guest,
likely reconfigure several config files, etc, etc. There is work on
a full v2v tool to automate this task, for both Linux & Windows guests.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: KVM usability

[Gerd Hoffmann]

On 03/02/10 15:37, Daniel P. Berrange wrote:
> On Tue, Mar 02, 2010 at 03:22:07PM +0100, Gerd Hoffmann wrote:
>> On 03/02/10 14:37, Nikolai K. Bochev wrote:
>>> I don't see where this argument is leading to. So far there are
>>> arguments that qemu/kvm sucks as a desktop virtualization, now
>>> suddenly the gui tools are shitty and everything should be done cli ,
>>> because there's no man pages for virt-manager. Explain.
>>
>> Lets face it:  virt-manager has a big bunch of problems.  For starters
>> there is no (gui) way to create a virtual machine other than installing
>> one from a iso image or via pxe.  So if you downloaded a disk image and
>> want to boot it ->  no cookie for you.  Likewise if you have a bunch of
>> already installed guests and want to migrate from $othertool to
>> virt-manager you can't do that easily.
>
> The ability to import existing KVM disk images is available in the
> latest release of virt-manager, targetted for Fedora 13.

Glad to hear that.  Are these bits in F12 virt-preview already?

> Migrating
> from another tool like VMWare  is a much harder than just importing
> the disk image, since you have to update the drivers inside the guest,
> likely reconfigure several config files, etc, etc. There is work on
> a full v2v tool to automate this task, for both Linux&  Windows guests.

I meant migrating from another qemu management tool, be it some other 
gui tool or self-baked shell scripts or something else.

Migrating from other virtualization products is harder of course.

cheers,
   Gerd

Re: KVM usability

[Dustin Kirkland]

On Tue, Mar 2, 2010 at 2:21 AM, Chris Webb <chris@arachsys.com> wrote:
> I remember about a year ago, someone asserting on the list that -usbdevice
> tablet was very CPU intensive even when not in use, and should be avoided if
> mouse support wasn't needed, e.g. on non-graphical VMs. Was that actually a
> significant hit, and is it still true today?

I don't notice a difference from a functional standpoint.

:-Dustin

Re: KVM usability

[Daniel P. Berrange]

On Tue, Mar 02, 2010 at 03:52:52PM +0100, Gerd Hoffmann wrote:
> On 03/02/10 15:37, Daniel P. Berrange wrote:
> >On Tue, Mar 02, 2010 at 03:22:07PM +0100, Gerd Hoffmann wrote:
> >>On 03/02/10 14:37, Nikolai K. Bochev wrote:
> >>>I don't see where this argument is leading to. So far there are
> >>>arguments that qemu/kvm sucks as a desktop virtualization, now
> >>>suddenly the gui tools are shitty and everything should be done cli ,
> >>>because there's no man pages for virt-manager. Explain.
> >>
> >>Lets face it:  virt-manager has a big bunch of problems.  For starters
> >>there is no (gui) way to create a virtual machine other than installing
> >>one from a iso image or via pxe.  So if you downloaded a disk image and
> >>want to boot it ->  no cookie for you.  Likewise if you have a bunch of
> >>already installed guests and want to migrate from $othertool to
> >>virt-manager you can't do that easily.
> >
> >The ability to import existing KVM disk images is available in the
> >latest release of virt-manager, targetted for Fedora 13.
> 
> Glad to hear that.  Are these bits in F12 virt-preview already?

I believe so.

> >Migrating
> >from another tool like VMWare  is a much harder than just importing
> >the disk image, since you have to update the drivers inside the guest,
> >likely reconfigure several config files, etc, etc. There is work on
> >a full v2v tool to automate this task, for both Linux&  Windows guests.
> 
> I meant migrating from another qemu management tool, be it some other 
> gui tool or self-baked shell scripts or something else.

That's pretty much what the 'disk import' could deal with, unless we wanted
to provide a way to read the config files from those other  qemu mgmt tools
directly on a case-by-case basis

Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: KVM usability

[Gerd Hoffmann]

On 03/02/10 15:56, Daniel P. Berrange wrote:
>> Glad to hear that.  Are these bits in F12 virt-preview already?
>
> I believe so.

/me goes fetch updates to check it out.

>>> Migrating
>> >from another tool like VMWare  is a much harder than just importing
>>> the disk image, since you have to update the drivers inside the guest,
>>> likely reconfigure several config files, etc, etc. There is work on
>>> a full v2v tool to automate this task, for both Linux&   Windows guests.
>>
>> I meant migrating from another qemu management tool, be it some other
>> gui tool or self-baked shell scripts or something else.
>
> That's pretty much what the 'disk import' could deal with,  unless we wanted
> to provide a way to read the config files from those other  qemu mgmt tools
> directly on a case-by-case basis

What I'll expect:

At minimum a way to create a virtual machine without the auto-started 
installation, with no or minimal hardware plugged in, so I can configure 
it the way I want via "add hardware", then boot it.

Better would be some dialog to shortcut the "add hardware" clicking for 
the most common stuff, i.e. a single dialog box asking for disk image, 
nic type, mac address, sound card, serial console.  I guess this is 
roughly what the "disk import" feature will do for me.

Analyzing the guest image, trying to figure what nic, disk and mac 
address it expects and pre-filling the dialog box with the results would 
be really great.  I suspect that is alot of work for little gains, but 
hey, maybe the v2v tool has most of the bits needed for that anyway, so 
we can get that nevertheless.

cheers,
   Gerd

Re: Enhance perf to support KVM

[Paolo Bonzini]

On 02/26/2010 03:23 PM, Ingo Molnar wrote:
> I do think tools/X and tools/libc would make quite a bit of sense - this is
> one of the better design aspects of FreeBSD et al. It's a mistake that it's
> not being done.

I don't see what it would buy to have tools/libc.  You cannot force 
users to update kernel-space and user-space in lockstep (Apple forced 
you to do that sometimes when I used Macs at work, and it was very very 
inconvenient), so it's not like libc would be able to always assume the 
latest system calls.  There is (relatively) a lot of 
backwards-compatibility code in libc; it's ugly code, but you have to 
live with it.

Any case in which you need lockstep upgrades of kernel and libc for 
bisectability is a bug in libc, and I haven't seen it happen enough to 
be a problem.

Paolo

ps: does it make sense to keep such a long list of recipients???

Re: Enhance perf to support KVM

[Arnaldo Carvalho de Melo]

Em Tue, Mar 02, 2010 at 05:46:03PM +0100, Paolo Bonzini escreveu:
> On 02/26/2010 03:23 PM, Ingo Molnar wrote:
>> I do think tools/X and tools/libc would make quite a bit of sense - this is
>> one of the better design aspects of FreeBSD et al. It's a mistake that it's
>> not being done.
>
> I don't see what it would buy to have tools/libc.  You cannot force  
> users to update kernel-space and user-space in lockstep (Apple forced  
> you to do that sometimes when I used Macs at work, and it was very very  
> inconvenient), so it's not like libc would be able to always assume the  
> latest system calls.  There is (relatively) a lot of  
> backwards-compatibility code in libc; it's ugly code, but you have to  
> live with it.
>
> Any case in which you need lockstep upgrades of kernel and libc for  
> bisectability is a bug in libc, and I haven't seen it happen enough to  
> be a problem.

You imply lockstep updates because both are on the same source tree? I
don't see why that would be required, there is an ABI contract to be
respected no matter where the sources for the signatories live.

perf lives in the kernel sources and while developing it I use all sorts
of combinations of userland and kernel perf bits so as not to be
rebooting the many machines of different arches where I test the
userspace bits all the time with a new kernel, even if it indeed has new
in-kernel perf bits.

As an instance where I could have worked on tools/libc recently was the
recvmmsg syscall I introduced.

Till it got into glibc (IIRC it was, but I wasn't involved in it, google
alert told me) interested parties had to use 'man syscall' and pick the
syscall number for their specific architecture, etc.

I could've done both the kernel and userspace bits in the same patch.

- Arnaldo

Re: Enhance perf to support KVM

[Ingo Molnar]

* Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 02/26/2010 03:23 PM, Ingo Molnar wrote:
> >I do think tools/X and tools/libc would make quite a bit of sense - this is
> >one of the better design aspects of FreeBSD et al. It's a mistake that it's
> >not being done.
> 
> I don't see what it would buy to have tools/libc.  You cannot force users to 
> update kernel-space and user-space in lockstep (Apple forced you to do that 
> sometimes when I used Macs at work, and it was very very inconvenient), so 
> it's not like libc would be able to always assume the latest system calls.  
> There is (relatively) a lot of backwards-compatibility code in libc; it's 
> ugly code, but you have to live with it.

If glibc was part of the kernel repo with a klibc alike approach we wouldnt 
have such problems: the kernel would provide the system library and that's it. 
You'd not want to upgrade them separately, just like you generally wouldnt 
want to upgrade your memory management code separately from the scheduler 
either.

The same argument could be made in a reverse fashion with just about any part 
of the kernel: 'it would be better to upgrade the ext3 driver separately' - 
etc. It's similar to all the classic microkernel versus monolithic kernel 
arguments.

There are costs with increasing size and increasing integration - but fact is 
that we can manage a 13+ MLOC kernel just fine and the benefits of an 
integrated approach far outweigh the costs.

In my experience it's far better to have one project for 'infrastructure' 
bits: developed, tested and offered as one coherent entity in essence. A bit 
like how Xorg does it.

These many splintered kernel facilities are historical legacies in essence 
(from the times when there was no single usable free OS available), and 
over-modularization has many costs and few advantages.

Thanks,

	Ingo

Re: Enhance perf to support KVM

[Paolo Bonzini]

On 03/02/2010 06:12 PM, Arnaldo Carvalho de Melo wrote:
> You imply lockstep updates because both are on the same source tree? I
> don't see why that would be required, there is an ABI contract to be
> respected no matter where the sources for the signatories live.

It's not about ABIs, it's about not being able to rely exclusively on 
new syscalls or flags (one for all: SOCK_NONBLOCK) because people can 
run new glibc on old kernels.

If libc had this luxury, it would possibly make a lot more sense to have 
it in the kernel tree.  But it doesn't.

Also, setting up a testing environment for glibc is not trivial, and the 
intersection of people that care about developing the kernel _and_ glibc 
is probably too small and opinionated to make this easier.

>I could've done both the kernel and userspace bits in the same patch.

You could add a glibc submodule to your git tree, and ask on the git 
list for help adding some kind of multi-destination support to 
git-{format-patch,send-email}.

Paolo

Re: Enhance perf to support KVM

[Ingo Molnar]

* Paolo Bonzini <pbonzini@redhat.com> wrote:

> On 03/02/2010 06:12 PM, Arnaldo Carvalho de Melo wrote:
> >You imply lockstep updates because both are on the same source tree? I
> >don't see why that would be required, there is an ABI contract to be
> >respected no matter where the sources for the signatories live.
> 
> It's not about ABIs, it's about not being able to rely exclusively on new 
> syscalls or flags (one for all: SOCK_NONBLOCK) because people can run new 
> glibc on old kernels.

That is not what i was suggesting though.

I am suggesting a klibc-alike approach, where glibc is not on the filesystem 
but mapped by the kernel, basically as an extension of the standard vDSO.

We always have most of glibc paged in all the time, so there's no real RAM 
usage difference.  (and ultra-embedded would sure make use of this to trim RAM 
usage in fact.)

That way the C library comes with the kernel. New kernel, new/updated system 
library.

	Ingo

Re: KVM usability

[Ross Boylan]

On Mon, 2010-03-01 at 15:59 -0600, Anthony Liguori wrote:
> 
> >   * desktop is 1024 x 720
> >    
> 
> 1024x768 and this is what the default is today anyway.
That was not my experience, as reported in my post a few days ago
("800x600 max resolution"), nor is it the experience reported in the
message that kicked off this thread.

I have been able to get a higher resolution, but it was far from
automatic.

I believe the root cause is the failure of the virtual monitor to
respond to or provide EDID (?) to tell the kernel available screen
resolutions, sync values, and similar information.  I recall seeing an
open bug on this, not sure if it was kvm or qemu.  I also recall it had
been open for a fairly long time.

Ross

Re: KVM usability

[Daniel P. Berrange]

On Tue, Mar 02, 2010 at 06:57:54PM -0800, Ross Boylan wrote:
> On Mon, 2010-03-01 at 15:59 -0600, Anthony Liguori wrote:
> > 
> > >   * desktop is 1024 x 720
> > >    
> > 
> > 1024x768 and this is what the default is today anyway.
> That was not my experience, as reported in my post a few days ago
> ("800x600 max resolution"), nor is it the experience reported in the
> message that kicked off this thread.
> 
> I have been able to get a higher resolution, but it was far from
> automatic.

It depends on the guest OS version. QEMU exposes a cirrus logic card by
defualt, and given the lack of vsync/hsync info, the Xorg driver will
pick 800x600 as the default resolution in absence of any Xorg.conf About
6 months or so back, we got Xorg guys to add a code to the Xorg cirrus
driver that looked for the QEMU PCI subsystem ID and if found, defaults
to 1024x768 instead. Of course this is itself still far from optimal
as a user experiance. We really want it to be fully configured to any
resolution as easily as the user would do with a real graphics card &
monitor.

Regards,
Daniel
-- 
|: Red Hat, Engineering, London    -o-   http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://deltacloud.org :|
|: http://autobuild.org        -o-         http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505  -o-   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

Re: KVM usability

[Ross Boylan]

On Wed, 2010-03-03 at 08:55 +0000, Daniel P. Berrange wrote:
> On Tue, Mar 02, 2010 at 06:57:54PM -0800, Ross Boylan wrote:
> > On Mon, 2010-03-01 at 15:59 -0600, Anthony Liguori wrote:
> > > 
> > > >   * desktop is 1024 x 720
> > > >    
> > > 
> > > 1024x768 and this is what the default is today anyway.
> > That was not my experience, as reported in my post a few days ago
> > ("800x600 max resolution"), nor is it the experience reported in the
> > message that kicked off this thread.
> > 
> > I have been able to get a higher resolution, but it was far from
> > automatic.
> 
> It depends on the guest OS version. QEMU exposes a cirrus logic card by
> defualt, 
QEMU docs recommend -std vga for higher resolutions; I used that.
> and given the lack of vsync/hsync info, the Xorg driver will
> pick 800x600 as the default resolution in absence of any Xorg.conf About
> 6 months or so back, we got Xorg guys to add a code to the Xorg cirrus
> driver that looked for the QEMU PCI subsystem ID and if found, defaults
> to 1024x768 instead. 
So presumably that logic wouldn't have kicked in.  I had xorg 7.5 on
Debian squeeze as the guest.
> Of course this is itself still far from optimal
> as a user experiance. We really want it to be fully configured to any
> resolution as easily as the user would do with a real graphics card &
> monitor.
Is there some obstacle to getting the virtual monitor to provide
configuration info when it's queried?  That seems like the most direct
solution.

Ross

Re: KVM usability

[Lucas Meneghel Rodrigues]

On Tue, 2010-03-02 at 11:11 +0100, Peter Zijlstra wrote:
> On Mon, 2010-03-01 at 09:14 -0600, Anthony Liguori wrote:
> > The real 
> > question to ask is, why are you using qemu directly instead of using 
> > virt-manager? 
> 
> Because I suspect Ingo, like me, is a command line user, launching a gui
> to start kvm when there is a kvm command around just sounds daft.
> 
> Also, I just installed and tried it, virt-manager is a total piece of
> shit, 

That statement is far from being fair. I use virt-manager quite a lot,
since I want to keep track of what's going on on KVM virtualization for
end users in Fedora. What's shipped with Fedora 12 is pretty decent in
many regards, but as in any other software there's plenty of room for
improvements.

> I wouldn't even know how to begin telling it how to start my
> freshly baked kernel with serial console on stdio and some block image I
> just created from the gentoo stage3 tarball.

Fair enough, it is convoluted to do what you want using virt-manager
(although possible), but mainly because this wasn't a use case for it.
You can't expect the application designers to support every single type
of work flow under the sun.

> That is, after 5 minutes clicking I have no idea how to even launch an
> ISO with the thing, I prefer reading the kvm manpage over using some
> mouse only gui crap like that.

For the 1st thing you wanted to do, I agree that it was cumbersome. But
to create a VM and make it boot an ISO available on your hard drive it
*is* trivial. There's a wizard to do it, because it's the main use case
of the thing.

If you want to point out problems on virt-manager that is fine, and the
developers will do what is possible to address problems, insults are not
necessary.

Re: KVM usability

[Zachary Amsden]

On 03/04/2010 10:00 AM, Lucas Meneghel Rodrigues wrote:
> On Tue, 2010-03-02 at 11:11 +0100, Peter Zijlstra wrote:
>    
>> On Mon, 2010-03-01 at 09:14 -0600, Anthony Liguori wrote:
>>      
>>> The real
>>> question to ask is, why are you using qemu directly instead of using
>>> virt-manager?
>>>        
>> Because I suspect Ingo, like me, is a command line user, launching a gui
>> to start kvm when there is a kvm command around just sounds daft.
>>
>> Also, I just installed and tried it, virt-manager is a total piece of
>> shit,
>>      
> That statement is far from being fair. I use virt-manager quite a lot,
> since I want to keep track of what's going on on KVM virtualization for
> end users in Fedora. What's shipped with Fedora 12 is pretty decent in
> many regards, but as in any other software there's plenty of room for
> improvements.
>    

The biggest problem with virt-manager isn't virt-manager, it's that it 
is trying to do a nearly intractable task.  Because a qemu virtual 
machine is not a machine at all, just a disk image without the proper 
metadata to track the important properties of the machine, like what 
revision of PCI chipset, how many disk controllers the thing is using, 
what kind of graphics card, etc.

These are all basic things that are left completely undefined by qemu's 
lack of a top-level configuration file, and it's an inexcusable disgrace.

So virt-manager or any other management tool has the burden of creating 
and maintaining a bunch of metadata around this workhorse tool called 
qemu and invoking libvirt to figure out which set of 100,000 blasted 
command line options to pass on.

That's why it falls short of expectations at times, not because 
virt-manager is crap, but because there is no well defined, well 
designed infrastructure for it to manage and the ad-hoc solution here is 
total crap.

Zach

Re: KVM usability

[Anthony Liguori]

On 03/04/2010 02:13 PM, Zachary Amsden wrote:
>
> The biggest problem with virt-manager isn't virt-manager, it's that it 
> is trying to do a nearly intractable task.  Because a qemu virtual 
> machine is not a machine at all, just a disk image without the proper 
> metadata to track the important properties of the machine, like what 
> revision of PCI chipset, how many disk controllers the thing is using, 
> what kind of graphics card, etc.
>
> These are all basic things that are left completely undefined by 
> qemu's lack of a top-level configuration file, and it's an inexcusable 
> disgrace.
>
> So virt-manager or any other management tool has the burden of 
> creating and maintaining a bunch of metadata around this workhorse 
> tool called qemu and invoking libvirt to figure out which set of 
> 100,000 blasted command line options to pass on.
>
> That's why it falls short of expectations at times, not because 
> virt-manager is crap, but because there is no well defined, well 
> designed infrastructure for it to manage and the ad-hoc solution here 
> is total crap.

And this is why we're doing QMP and qdev.  It's long overdue 
infrastructure.  It's not just the problem that you describe though.  
virt-manager is limited by what libvirt provides and today libvirt does 
not expose nearly enough qemu features for virt-manager to even attempt 
to solve the problem on it's own.

Regards,

Anthony Liguori

> Zach

Re: KVM usability

[H. Peter Anvin]

On 03/04/2010 12:13 PM, Zachary Amsden wrote:
> 
> These are all basic things that are left completely undefined by qemu's 
> lack of a top-level configuration file, and it's an inexcusable disgrace.
> 

There is a top-level configuration file for Qemu, at least in the
development tree.  It's optional, still, but it's there now.

	-hpa

Re: KVM usability

[Markus Armbruster]

"H. Peter Anvin" <hpa@zytor.com> writes:

> On 03/04/2010 12:13 PM, Zachary Amsden wrote:
>> 
>> These are all basic things that are left completely undefined by qemu's 
>> lack of a top-level configuration file, and it's an inexcusable disgrace.
>> 
>
> There is a top-level configuration file for Qemu, at least in the
> development tree.  It's optional, still, but it's there now.

It covers much but not all of the command line.

Re: KVM usability

[Avi Kivity]

On 03/02/2010 04:29 PM, Ingo Molnar wrote:
>
> I guess the first step would be to move away from the 'lets support lots of
> crappy virtualization solutions at once, poorly' model, and pick one good
> combo (i'd go for Qemu+KVM) and turn it into a heck of an all-around solution.
> Then all the other combos will catch up as well. (or will wither away)
>    

Should desktop integration be GNOME or KDE based?

IMO this should be addressed via a plugin system so that we can have a 
well integrated single process VM, without choosing one or the other.  
Also, the qemu community doesn't really have serious graphics design 
expertise; best to leave that to people who won't make it a total disaster.

> ( Sidenote: i also looked at the VirtualBox kernel driver. Oh my ... i really
>    shouldnt have! They should migrate to the KVM kernel-side code ASAP ... )
>    

They can't, if they want to continue to support hardware without 
virtualization extensions.

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Avi Kivity]

On 03/02/2010 12:30 PM, Ingo Molnar wrote:
> * Ingo Molnar<mingo@elte.hu>  wrote:
>
>    
>> Here's our experience with tools/perf/. Hosting the project in the kernel
>> proper helped its quality immensely:
>>
>>   - It's much easier to synchronize new features on the kernel side and on the
>>     user-space side. The two go hand in hand - they are often implemented in
>>     the same patch.
>>      
> Just look at an example from today, a perf+KVM feature patch posted by Yanmin
> Zhang:
>
>    http://www.mail-archive.com/kvm@vger.kernel.org/msg29770.html
>
> That single patch implements the following "perf kvm" commands:
>
>    perf kvm top
>    perf kvm record
>    perf kvm report
>    perf kvm diff
>
> Both the kernel-space and the user-space changes are in that single patch.
>
> Anyone who'd like to try it out can apply it and get an updated kernel plus
> updated tooling and can start profiling KVM guests straight away. You just
> check out the kernel, apply the patch and that's it - you can go. It doesnt
> get any more convenient than that to do development.
>
> Such kind of a unified repository is a powerful concept, and we make use of
> those aspects of tools/perf/ every day. You could only pry it out of our cold,
> dead fingers ;-)
>    

perf really is wonderful, but to be really competitive, and usable to 
more developers, it needs to be in a graphical environment.  I want 
'perf report' output to start out collapsed and drill down by clicking 
on a tree widget.  Clicking on a function name opens its definition.  
'perf annotate' should display annotations on my editor window, not in a 
pager.  I should be able to check events on a list, not using 'perf list'.

Is something like that suitable for tools/perf/?  I think you'll find 
the intersection of kernel developers and GUI developers to be fairly small.

> Btw., this is one of the things that FreeBSD does right - and i believe it is
> one of the technical concepts behind Apple's success as well. Apple, with a
> tenth's of Linux's effective R&D budget can consistently out-develop Linux. I
> think that's in part due to there not being a strict chinese wall between the
> Apple kernel, libraries and applications - it's one coherent project where
> everyone is well-connected to each piece, with no artificial project-cultural
> boundaries and barriers. People can and do move between those areas of the
> larger "Apple" project to achieve their goals - regardless of how many
> components need touching for a given area of interest.
>
> IMHO we should learn from that - while we are good in many areas there's
> always aspects of Linux that can be improved. But i digress.
>    

Folding everything into the kernel tree is one way to approach it; IMO 
it is completely unreasonable.  The kernel is a very small part of a 
complete system.

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Pekka Enberg]

Hi Avi,

(slightly off-topic)

On Sun, Mar 7, 2010 at 11:35 AM, Avi Kivity <avi@redhat.com> wrote:
> perf really is wonderful, but to be really competitive, and usable to more
> developers, it needs to be in a graphical environment.  I want 'perf report'
> output to start out collapsed and drill down by clicking on a tree widget.
>  Clicking on a function name opens its definition.  'perf annotate' should
> display annotations on my editor window, not in a pager.  I should be able
> to check events on a list, not using 'perf list'.

People keep bringing this up but I don't quite agree. Mac OS X has
"shark" which is pretty much what you describe above. However, having
used both, I prefer perf's simple UI for two reasons: it's much easier
to automate perf commands and text-based reports are superior for
sharing results (and keeping track of results when doing performance
optimizations).

That said, AFAICT, it should be pretty simple to implement a
shark-like UI with GTK as current perf code is pretty good fit for
that. I've pondered about doing that myself but quite frankly, I don't
see any big gains in that.

                        Pekka

Re: KVM usability

[Avi Kivity]

On 03/07/2010 11:56 AM, Pekka Enberg wrote:
> Hi Avi,
>
> (slightly off-topic)
>
> On Sun, Mar 7, 2010 at 11:35 AM, Avi Kivity<avi@redhat.com>  wrote:
>    
>> perf really is wonderful, but to be really competitive, and usable to more
>> developers, it needs to be in a graphical environment.  I want 'perf report'
>> output to start out collapsed and drill down by clicking on a tree widget.
>>   Clicking on a function name opens its definition.  'perf annotate' should
>> display annotations on my editor window, not in a pager.  I should be able
>> to check events on a list, not using 'perf list'.
>>      
> People keep bringing this up but I don't quite agree. Mac OS X has
> "shark" which is pretty much what you describe above. However, having
> used both, I prefer perf's simple UI for two reasons: it's much easier
> to automate perf commands and text-based reports are superior for
> sharing results (and keeping track of results when doing performance
> optimizations).
>    

Yes, you (and me as well).  But most people are quite unlike new and 
me.  There's a reason GUIs dominate today, and there are even a few 
kernel developers that don't use mutt for reading email.

Even for command-line-happy people, GUIs still have an advantage in that 
it is much easier to discover features by exploring the UI vs. reading 
manual pages.

> That said, AFAICT, it should be pretty simple to implement a
> shark-like UI with GTK as current perf code is pretty good fit for
> that. I've pondered about doing that myself but quite frankly, I don't
> see any big gains in that.
>    

Because you are only interested in your own itches (which is perfectly 
legitimate, but will keep perf's userbase down).

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Avi Kivity]

On 03/01/2010 05:14 PM, Anthony Liguori wrote:
>>   - Graphics performance is awful even with the 640x480 miniature 
>> version.
>>     During bootup I can see it drawing single characters. This is a 
>> Core2
>>     2.8GHz.
>
>
> That's a bug.  Please report it.  Graphics performance isn't great, 
> but it should not be that bad.

(This is in grub, right?)

That's not fixable in kvm.  In 4-bit vga mode you cannot allow the guest 
to access emulated VRAM directly (since each access has side effects).  
A Core 2 will only do about 250K emulations per second, so screen 
redraws take about 3 seconds.

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Avi Kivity]

On 03/02/2010 11:34 AM, Jernej Simončič wrote:
> On Tuesday, March 2, 2010, 9:21:18, Chris Webb wrote:
>
>    
>> I remember about a year ago, someone asserting on the list that -usbdevice
>> tablet was very CPU intensive even when not in use, and should be avoided if
>> mouse support wasn't needed, e.g. on non-graphical VMs. Was that actually a
>> significant hit, and is it still true today?
>>      
> It would appear that this is still the case, at least on slower hosts
> - on Atom Z530 (1,6GHz), the XP VM uses ~30% CPU when idle with
> -usbdevice tablet, but only ~4% without it. However, on a faster host
> (Core2 Quad 2,66GHz), there's practically no difference (Vista x64 VM
> uses ~1% CPU when idle regardless of -usbdevice tablet).
>    

Looks like the tablet is set to 100 Hz polling rate.  We may be able to 
get away with 30 Hz or even less (ep_bInterval, in ms, in hw/usb-wacom.c).

-- 
error compiling committee.c: too many arguments to function

Re: Enhance perf to support KVM

[Avi Kivity]

On 03/02/2010 07:17 PM, Ingo Molnar wrote:
> * Paolo Bonzini<pbonzini@redhat.com>  wrote:
>
>    
>> On 02/26/2010 03:23 PM, Ingo Molnar wrote:
>>      
>>> I do think tools/X and tools/libc would make quite a bit of sense - this is
>>> one of the better design aspects of FreeBSD et al. It's a mistake that it's
>>> not being done.
>>>        
>> I don't see what it would buy to have tools/libc.  You cannot force users to
>> update kernel-space and user-space in lockstep (Apple forced you to do that
>> sometimes when I used Macs at work, and it was very very inconvenient), so
>> it's not like libc would be able to always assume the latest system calls.
>> There is (relatively) a lot of backwards-compatibility code in libc; it's
>> ugly code, but you have to live with it.
>>      
> If glibc was part of the kernel repo with a klibc alike approach we wouldnt
> have such problems: the kernel would provide the system library and that's it.
> You'd not want to upgrade them separately, just like you generally wouldnt
> want to upgrade your memory management code separately from the scheduler
> either.
>
> The same argument could be made in a reverse fashion with just about any part
> of the kernel: 'it would be better to upgrade the ext3 driver separately' -
> etc. It's similar to all the classic microkernel versus monolithic kernel
> arguments.
>    

No, that's Documentation/stable_api_nonsense.txt.  It's perfectly 
possible to have a filesystem driver that is decoupled from the kernel, 
yet both are in the same address space.

> There are costs with increasing size and increasing integration - but fact is
> that we can manage a 13+ MLOC kernel just fine and the benefits of an
> integrated approach far outweigh the costs.
>
> In my experience it's far better to have one project for 'infrastructure'
> bits: developed, tested and offered as one coherent entity in essence. A bit
> like how Xorg does it.
>
> These many splintered kernel facilities are historical legacies in essence
> (from the times when there was no single usable free OS available), and
> over-modularization has many costs and few advantages.
>    

You're probably right for core libraries, but I don't think a GUI for 
kvm qualifies.  A server oriented application, maybe.

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Avi Kivity]

On 03/03/2010 05:42 PM, Ross Boylan wrote:
>> Of course this is itself still far from optimal
>> as a user experiance. We really want it to be fully configured to any
>> resolution as easily as the user would do with a real graphics card&
>> monitor.
>>      
> Is there some obstacle to getting the virtual monitor to provide
> configuration info when it's queried?  That seems like the most direct
> solution.
>    

That was my conclusion as well.  All that's missing is something to code 
it up.

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Luca Barbieri]

> perf really is wonderful, but to be really competitive, and usable to more
> developers, it needs to be in a graphical environment.  I want 'perf report'
> output to start out collapsed and drill down by clicking on a tree widget.
>  Clicking on a function name opens its definition.  'perf annotate' should
> display annotations on my editor window, not in a pager.  I should be able
> to check events on a list, not using 'perf list'.
>
> Is something like that suitable for tools/perf/?  I think you'll find the
> intersection of kernel developers and GUI developers to be fairly small.

The latest versions of Gnome Sysprof use perf and provide a GTK+ tree
interface for the profiling output.

However, they are not configurable at all and don't support anything
but call graph profiling, unless they added more features very
recently.
It would be nice to extend sysprof into a more capable tool, and one
that can read perf output files and do so when launched from the
command line.

Re: KVM usability

[Arnaldo Carvalho de Melo]

Em Sun, Mar 07, 2010 at 11:35:31AM +0200, Avi Kivity escreveu:
> perf really is wonderful, but to be really competitive, and usable to  
> more developers, it needs to be in a graphical environment.  I want  
> 'perf report' output to start out collapsed and drill down by clicking  
> on a tree widget.  Clicking on a function name opens its definition.   
> 'perf annotate' should display annotations on my editor window, not in a  
> pager.  I should be able to check events on a list, not using 'perf 
> list'.

Do you really think that more kernel developers would use perf more
frequently if it had some GUI?

I plan to work on a ncurses tool combining aspects of the existing perf
tools, integrating them more, like you suggest above, but even having
worked on a pygtk tool that is close to the kernel [1], I'm unsure if
doing it using gtk or QT would be something that would entice more
developers to use it.

- Arnaldo

[1] http://www.osadl.org/Single-View.111+M52212cb1379.0.html

Re: KVM usability

[Avi Kivity]

On 03/07/2010 08:01 PM, Arnaldo Carvalho de Melo wrote:
> Em Sun, Mar 07, 2010 at 11:35:31AM +0200, Avi Kivity escreveu:
>    
>> perf really is wonderful, but to be really competitive, and usable to
>> more developers, it needs to be in a graphical environment.  I want
>> 'perf report' output to start out collapsed and drill down by clicking
>> on a tree widget.  Clicking on a function name opens its definition.
>> 'perf annotate' should display annotations on my editor window, not in a
>> pager.  I should be able to check events on a list, not using 'perf
>> list'.
>>      
> Do you really think that more kernel developers would use perf more
> frequently if it had some GUI?
>    

Not much.  Is perf's target kernel developers exclusively?  Who are we 
writing this kernel for?

No wonder everything is benchmarked using kbuild.

> I plan to work on a ncurses tool combining aspects of the existing perf
> tools, integrating them more, like you suggest above, but even having
> worked on a pygtk tool that is close to the kernel [1], I'm unsure if
> doing it using gtk or QT would be something that would entice more
> developers to use it.
>    

Even for kernel developers there are advantages in a GUI, namely that 
features are easily discovered, the amount of information is easily 
controlled, and in that you can interact (not redo everything from 
scratch every time you want to change something).  The difference 
between a curses based tool and a true GUI are minimal for this audience.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: KVM usability

[Avi Kivity]

On 03/07/2010 05:14 PM, Luca Barbieri wrote:
>> perf really is wonderful, but to be really competitive, and usable to more
>> developers, it needs to be in a graphical environment.  I want 'perf report'
>> output to start out collapsed and drill down by clicking on a tree widget.
>>   Clicking on a function name opens its definition.  'perf annotate' should
>> display annotations on my editor window, not in a pager.  I should be able
>> to check events on a list, not using 'perf list'.
>>
>> Is something like that suitable for tools/perf/?  I think you'll find the
>> intersection of kernel developers and GUI developers to be fairly small.
>>      
> The latest versions of Gnome Sysprof use perf and provide a GTK+ tree
> interface for the profiling output.
>
> However, they are not configurable at all and don't support anything
> but call graph profiling, unless they added more features very
> recently.
> It would be nice to extend sysprof into a more capable tool, and one
> that can read perf output files and do so when launched from the
> command line.
>    

Looks like a step in the right direction.  I don't think this belong in 
tools/, though.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: KVM usability

[Avi Kivity]

On 02/27/2010 07:25 PM, Ingo Molnar wrote:
> * Zachary Amsden<zamsden@redhat.com>  wrote:
>
> [...]
>    
>> Second, it's not over-modularized.  The modules are the individual
>> components of the architecture.  How would you propose to put it
>> differently.  They really can't naturally combine.  And with the
>> code quality of qemu in general being problematic by Linux kernel
>> standards, it's not natural to move the device emulation directly
>> into the kernel module.  So this is why we are where we are today.
>>      
> I'm not talking about moving it into a kernel _module_ - albeit that
> alone is a worthwile thing to do for any performance sensitive hw
> component.
>    

Moving qemu into the kernel is preposterous.  It's a huge code base 
(even a stripped down version) with many external dependencies.  You're 
converting a guest crash into a host crash, and a simple binary update 
into a reboot.

What makes sense is identifying the performance critical bits and 
creating generally useful kernel interfaces that can be used to speed 
them up, or kvm-specific kernel accelerations.  Example include 
preadv/pwritev (generic) and vhost-net (specific).

> I was talking about the option of a clean, stripped down Qemu base
> hosted in the kernel proper, in linux/tools/kvm/ or so. If i were
> running a virtualization effort it would be the first place i'd
> consider to put my tooling into.
>
> It would be a no-brainer: most of the devs come from the KVM side, and
> KVM itself makes little sense without Qemu, and Qemu makes little sense
> without KVM these days. (and i know about the non-KVM and non-x86
> roots of Qemu - still, it's not a significant piece of usage today)
>    

It's true that qemu development is dominated these days by kvm 
development.  If the qemu community was dysfunctional (as it once was) 
then it makes sense to fork it.  But it isn't, so it doesn't.

Note also that kvm is not x86 only.

>> Third, it's the maintainers upstream who are in charge of KVM
>> quality as a whole - when you are talking about upstream code
>> quality, and the package maintainers who are in charge of KVM
>> quality as a whole - when you are talking about a distro.  This is
>> nothing new - it's just a statement of decentralization.
>>      
> It's certainly nothing new. Nor was the suckage of CVS newup until
> Git came along and changed the game on a fundamental basis.
>    

Bitkeeper deserves some credit here.

> Suckage is there to be fought all the time.
>    

Send patches, not flames.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: KVM usability

[Avi Kivity]

On 03/02/2010 02:30 AM, Ingo Molnar wrote:
> * Anthony Liguori<anthony@codemonkey.ws>  wrote:
>
>    
>> On 03/01/2010 02:56 PM, Ingo Molnar wrote:
>>      
>>> Here's our experience with tools/perf/. Hosting the project in the kernel
>>> proper helped its quality immensely:
>>>
>>>   - It's much easier to synchronize new features on the kernel side and on the
>>>     user-space side. The two go hand in hand - they are often implemented in
>>>     the same patch.
>>>        
>> Kernel features and qemu features usually don't have a great amount of
>> intersect.  All of the problems you've described are strictly in the qemu
>> space.
>>      
> IMO that's a bug, not a feature. There should be a lot more interaction
> between kvm-qemu and KVM: for example Qemu should have a feature to install
> paravirt drivers in the guest, this would be helped by living in the kernel
> repo.
>    

The paravirt drivers are completely disassociated from kvm.  You can run 
a virtio driver with qemu but without kvm (or even with virtualbox, 
without either qemu or kvm).

For Linux, installing drivers automatically in older guests is 
impossible due to Documentation/stable_api_nonsense.txt and unnecessary 
in newer Linux (same reason).  For non-Linux, this feature certainly 
makes sense, but I don't see how putting qemu in tools/kvm helps it much.


>>>   - It's released together with the kernel, which gives a periodic 3 months
>>>     release frequency. Not too slow, not too fast.
>>>        
>> qemu release range in length from 3-6 months depending on
>> distribution schedules.  They are very regular.
>>      
> The Linux kernel is released every 3 months, +- one week. Our experience is
> that even 6 months would be (way) too painful for distros.
>    

It would also be horrible for internal synchronization.  That's not an 
issue with qemu, nor do I think that six months would hurt distros any.  
In any case, we respond to feedback (which we happen to generate in the 
first place).

>>>   - Code quality requirements are that of the kernel's. No muck allowed and
>>>     it's not hard to explain what kind of code is preferred.
>>>        
>> Code quality is subjective.  We have a different coding style.
>>      
> That's somewhat of a problem when for example a KVM kernel-space developer
> crosses into Qemu code and back. Two separate styles, etc. I certainly
> remember a 'culture clash' when going from the kernel into Qemu and back.
> Different principles, different culture. It's better to standardize.
>    

That sounds like a trivial thing.

>>>   - Tool breakage bisection is a breeze: there's never any mismatch between
>>>     tools/perf and the kernel counterpart. With a separate package we'd
>>>     have more complex test and bisection scenarios.
>>>        
>> KVM has a backwards compatible ABI so there's no such thing as mismatch
>> between user and kernel space.
>>      
> perf too is ABI compatible (between releases) - still bisection is a lot
> easier because the evolution of a particular feature can be bisected back to.
>
> Btw., KVM certainly ha ABI breakages around 2.6.16(?) when it was added, even
> of released versions. Also, within a development version you sure sometimes
> iterate a new ABI component, right? With a time-coherent repository both
> intentional and unintentional breakages and variations can be bisected back to
> as they happened.
>
> This is an unconditional advantage and i made use of it numerous times.
>    

Try old qemu on new kernel.  If it works, bisect qemu.  If it fails, 
bisect the kernel.

If you're lucky it is qemu that was broken, so no kernel rebuilds and 
reboots.  Since qemu is much larger than kvm, it is more likely to have 
introduced the problem, so the bisect goes faster.

>> You could argue that any project should be in the kernel for these
>> reasons.  I see no reason why something as like KVM should be part
>> of the kernel and udev shouldn't be.
>>      
> Yes, you are quite correct: udev has been argued to be a prime candidate for
> tools/. (and some other kernel utilities as well)
>
>  From a design POV all 'system/kernel utilities', which make little sense
> without the kernel and are license compatible can (and arguably should) move
> there.
>
> Obviously there's no pressure to do so - it's only an opportunity.
>    

Only a small part of qemu, especially the desktop oriented qemu that you 
seem to want, actually interfaces with kvm.  Mostly it involves 
emulating hardware, issuing I/O, talking to management layers, 
presenting a user interface, etc.  It's not a system/kernel utility.

>>> etc.
>>>
>>> In the KVM context this was obviously only a suggestion though. If i were
>>> hacking on kvm-qemu i wouldnt hesitate for a moment to do it: the project
>>> has very close ties to kernel-KVM and repo level unification would create
>>> various synergies - but you are hacking on it, not me ;-)
>>>
>>> If i were doing it i'd probably start with a cleaned up and stripped down
>>> version of Qemu, to make it eligible for mainline kernel inclusion.
>>>        
>> You should try it.  I think you'll find that it's not as obvious thing to do
>> as you think it is.
>>      
> A few years ago I looked into cleaning up Qemu, when i hacked KVM and Qemu. I
> also wanted to have a 'qemu light', which is both smaller and cleaner, and
> still fits to KVM. It didnt look particularly hard back then - but it's
> certainly not zero amount of work.
>
> Cleanups pay - they make a piece of code both more hackable, more debuggable
> and more appealing to new developers. (i suspect you have no argument with
> that notion) Also note that it wasnt me who suggested that Qemu wouldnt fit
> the kernel standards as-is - it was raised by others in this discussion.
>    

I'm sure patches to clean up qemu will be more than welcome on qemu-devel.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: KVM usability

[Ingo Molnar]

* Pekka Enberg <penberg@cs.helsinki.fi> wrote:

> That said, AFAICT, it should be pretty simple to implement a shark-like UI 
> with GTK as current perf code is pretty good fit for that. I've pondered 
> about doing that myself but quite frankly, I don't see any big gains in 
> that.

There's a perf events based GUI: sysprof.

It's not integrated into tools/perf/ (it's using the syscall) and we'd welcome 
it integrated as 'perf view' or 'perf gui' - just like we'd welcome a new 
from-scratch GUI frontend as well.

The latter would have the advantage of growing gradually and naturally - so if 
you feel like implementing it, please do not hesitate ;-) I had a good hard 
look at the perf GUI situation a few months ago and GTK looks like the best 
fit.

Even if it's simple initially it would clearly show us the exact demand of GUI 
versus TUI. I.e. if you are right and it doesnt matter, it will stay simple, 
if Avi is right and it matters then people will use it as a base and improve 
it. A win-win scenario.

Thanks,

	Ingo

Re: KVM usability

[Arnaldo Carvalho de Melo]

Em Sun, Mar 07, 2010 at 08:15:40PM +0200, Avi Kivity escreveu:
> On 03/07/2010 08:01 PM, Arnaldo Carvalho de Melo wrote:
> >Em Sun, Mar 07, 2010 at 11:35:31AM +0200, Avi Kivity escreveu:
> >>perf really is wonderful, but to be really competitive, and usable to
> >>more developers, it needs to be in a graphical environment.  I want
> >>'perf report' output to start out collapsed and drill down by clicking
> >>on a tree widget.  Clicking on a function name opens its definition.
> >>'perf annotate' should display annotations on my editor window, not in a
> >>pager.  I should be able to check events on a list, not using 'perf
> >>list'.
> >Do you really think that more kernel developers would use perf more
> >frequently if it had some GUI?
> 
> Not much.  Is perf's target kernel developers exclusively?  Who are
> we writing this kernel for?

No, we aren't writing this tool only for kernel developers exclusively,
but that wasn't my question, it was badly formulated, sorry, I shouldn't
have included "kernel" in it :-\
 
> No wonder everything is benchmarked using kbuild.
> 
> >I plan to work on a ncurses tool combining aspects of the existing perf
> >tools, integrating them more, like you suggest above, but even having
> >worked on a pygtk tool that is close to the kernel [1], I'm unsure if
> >doing it using gtk or QT would be something that would entice more
> >developers to use it.
> 
> Even for kernel developers there are advantages in a GUI, namely
> that features are easily discovered, the amount of information is
> easily controlled, and in that you can interact (not redo everything
> from scratch every time you want to change something).  The
> difference between a curses based tool and a true GUI are minimal
> for this audience.

Ok, I agree with you about easier discoverability of features, path
shortened from report to annotate to starting the editor right at the
line where some event of interest happened, will try to keep the
routines not much coupled with ncurses, but definetely ncurses will be
the first step.

- Arnaldo

Re: KVM usability

[Avi Kivity]

On 03/07/2010 08:53 PM, Arnaldo Carvalho de Melo wrote:
>>> Do you really think that more kernel developers would use perf more
>>> frequently if it had some GUI?
>>>        
>> Not much.  Is perf's target kernel developers exclusively?  Who are
>> we writing this kernel for?
>>      
> No, we aren't writing this tool only for kernel developers exclusively,
> but that wasn't my question, it was badly formulated, sorry, I shouldn't
> have included "kernel" in it :-\
>    

In this case, I will reformulate my answer.  Very much.

>> Even for kernel developers there are advantages in a GUI, namely
>> that features are easily discovered, the amount of information is
>> easily controlled, and in that you can interact (not redo everything
>> from scratch every time you want to change something).  The
>> difference between a curses based tool and a true GUI are minimal
>> for this audience.
>>      
> Ok, I agree with you about easier discoverability of features, path
> shortened from report to annotate to starting the editor right at the
> line where some event of interest happened,

Another path is browse some function, start profiling, see perf data 
fill up in the margin.  Or, jump to callers.  etc.  You need an 
integrated browser for that (or an emacs perf mode).

> will try to keep the
> routines not much coupled with ncurses, but definetely ncurses will be
> the first step.
>    

Great.  ncurses is certainly much easier to experiment with and will 
likely provide useful experience.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: KVM usability

[Avi Kivity]

On 03/02/2010 04:36 AM, Anthony Liguori wrote:
>> I keep a patch in the SUSE version for quite some time now that bumps 
>> the default to 384 for qemu-kvm. That was the first "round" number 
>> where an openSUSE installation worked.
>
> If someone works up a patch and tests at least a couple types of 
> guests to confirm that they all install with that number, I'd be happy 
> to apply it (although we need some trickery to support older pc 
> versions).

We should avoid changing defaults.  I don't think in this case it 
matters, since everyone specifies -m anyway, but as a general rule 
changing defaults = breakage for the unwary.  At least make the default 
part of the machine type to preserve compatibility.

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Dustin Kirkland]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On Tue, 2010-03-09 at 15:32 +0200, Avi Kivity wrote:
> On 03/02/2010 04:36 AM, Anthony Liguori wrote:
> >> I keep a patch in the SUSE version for quite some time now that bumps 
> >> the default to 384 for qemu-kvm. That was the first "round" number 
> >> where an openSUSE installation worked.
> >
> > If someone works up a patch and tests at least a couple types of 
> > guests to confirm that they all install with that number, I'd be happy 
> > to apply it (although we need some trickery to support older pc 
> > versions).
> 
> We should avoid changing defaults.  I don't think in this case it 
> matters, since everyone specifies -m anyway, but as a general rule 
> changing defaults = breakage for the unwary.  At least make the default 
> part of the machine type to preserve compatibility.

In that case, Alex, where can I find your +384M patch, because I'd like
to carry the same one in Ubuntu...

Dustin

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: KVM usability

[Alexander Graf]

On 09.03.2010, at 15:32, Dustin Kirkland wrote:

> On Tue, 2010-03-09 at 15:32 +0200, Avi Kivity wrote:
>> On 03/02/2010 04:36 AM, Anthony Liguori wrote:
>>>> I keep a patch in the SUSE version for quite some time now that bumps 
>>>> the default to 384 for qemu-kvm. That was the first "round" number 
>>>> where an openSUSE installation worked.
>>> 
>>> If someone works up a patch and tests at least a couple types of 
>>> guests to confirm that they all install with that number, I'd be happy 
>>> to apply it (although we need some trickery to support older pc 
>>> versions).
>> 
>> We should avoid changing defaults.  I don't think in this case it 
>> matters, since everyone specifies -m anyway, but as a general rule 
>> changing defaults = breakage for the unwary.  At least make the default 
>> part of the machine type to preserve compatibility.
> 
> In that case, Alex, where can I find your +384M patch, because I'd like
> to carry the same one in Ubuntu...

It's all in the openSUSE build service. The direct access URL (login required FWIW) is here:

https://build.opensuse.org/package/view_file?file=kvm-qemu-default-memsize.patch&package=kvm&project=Virtualization

It merely changes DEFAULT_RAM_SIZE in vl.c from 128 to 384 for the "kvm" package.


Alex

Re: KVM usability

[Anthony Liguori]

On 03/09/2010 07:32 AM, Avi Kivity wrote:
> On 03/02/2010 04:36 AM, Anthony Liguori wrote:
>>> I keep a patch in the SUSE version for quite some time now that 
>>> bumps the default to 384 for qemu-kvm. That was the first "round" 
>>> number where an openSUSE installation worked.
>>
>> If someone works up a patch and tests at least a couple types of 
>> guests to confirm that they all install with that number, I'd be 
>> happy to apply it (although we need some trickery to support older pc 
>> versions).
>
> We should avoid changing defaults.

I disagree.  IMHO, the defaults should represent our best suggestion for 
any given release.  The compatibility machines make it very easier for a 
user to fix on a particular version of a machine type.

Regards,

Anthony Liguori

Re: KVM usability

[Anthony Liguori]

On 03/09/2010 08:38 AM, Alexander Graf wrote:
> On 09.03.2010, at 15:32, Dustin Kirkland wrote:
>
>    
>> On Tue, 2010-03-09 at 15:32 +0200, Avi Kivity wrote:
>>      
>>> On 03/02/2010 04:36 AM, Anthony Liguori wrote:
>>>        
>>>>> I keep a patch in the SUSE version for quite some time now that bumps
>>>>> the default to 384 for qemu-kvm. That was the first "round" number
>>>>> where an openSUSE installation worked.
>>>>>            
>>>> If someone works up a patch and tests at least a couple types of
>>>> guests to confirm that they all install with that number, I'd be happy
>>>> to apply it (although we need some trickery to support older pc
>>>> versions).
>>>>          
>>> We should avoid changing defaults.  I don't think in this case it
>>> matters, since everyone specifies -m anyway, but as a general rule
>>> changing defaults = breakage for the unwary.  At least make the default
>>> part of the machine type to preserve compatibility.
>>>        
>> In that case, Alex, where can I find your +384M patch, because I'd like
>> to carry the same one in Ubuntu...
>>      
> It's all in the openSUSE build service. The direct access URL (login required FWIW) is here:
>
> https://build.opensuse.org/package/view_file?file=kvm-qemu-default-memsize.patch&package=kvm&project=Virtualization
>
> It merely changes DEFAULT_RAM_SIZE in vl.c from 128 to 384 for the "kvm" package.
>    

We should attempt to do three things with default ram size:

1) bump it up to a more reasonable number
2) make it specified in the global default config
3) make sure we can provide compatibility support for older machine types

> Alex
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>    

Re: KVM usability

[Avi Kivity]

On 03/09/2010 04:50 PM, Anthony Liguori wrote:
>> It's all in the openSUSE build service. The direct access URL (login 
>> required FWIW) is here:
>>
>> https://build.opensuse.org/package/view_file?file=kvm-qemu-default-memsize.patch&package=kvm&project=Virtualization 
>>
>>
>> It merely changes DEFAULT_RAM_SIZE in vl.c from 128 to 384 for the 
>> "kvm" package.
>
> We should attempt to do three things with default ram size:
>
> 1) bump it up to a more reasonable number
> 2) make it specified in the global default config
> 3) make sure we can provide compatibility support for older machine types

It's really sad, the amount of code needed to change a number.

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Avi Kivity]

On 03/09/2010 04:49 PM, Anthony Liguori wrote:
> On 03/09/2010 07:32 AM, Avi Kivity wrote:
>> On 03/02/2010 04:36 AM, Anthony Liguori wrote:
>>>> I keep a patch in the SUSE version for quite some time now that 
>>>> bumps the default to 384 for qemu-kvm. That was the first "round" 
>>>> number where an openSUSE installation worked.
>>>
>>> If someone works up a patch and tests at least a couple types of 
>>> guests to confirm that they all install with that number, I'd be 
>>> happy to apply it (although we need some trickery to support older 
>>> pc versions).
>>
>> We should avoid changing defaults.
>
> I disagree.  IMHO, the defaults should represent our best suggestion 
> for any given release.  The compatibility machines make it very easier 
> for a user to fix on a particular version of a machine type.
>

Agreed, should have said, avoid changing defaults without taking care of 
backwards compatibility.

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Anthony Liguori]

On 03/09/2010 08:52 AM, Avi Kivity wrote:
> On 03/09/2010 04:50 PM, Anthony Liguori wrote:
>>> It's all in the openSUSE build service. The direct access URL (login 
>>> required FWIW) is here:
>>>
>>> https://build.opensuse.org/package/view_file?file=kvm-qemu-default-memsize.patch&package=kvm&project=Virtualization 
>>>
>>>
>>> It merely changes DEFAULT_RAM_SIZE in vl.c from 128 to 384 for the 
>>> "kvm" package.
>>
>> We should attempt to do three things with default ram size:
>>
>> 1) bump it up to a more reasonable number
>> 2) make it specified in the global default config
>> 3) make sure we can provide compatibility support for older machine 
>> types
>
> It's really sad, the amount of code needed to change a number.

We don't do enough via a config.  If we did, we could just have a 0.12 
config version that got frozen over time.

So really, if we can make the mem readable by global config, and we can 
have machine specific configs, it would simplify the problem in the 
future so that we just had to bump a number.

Regards,

Anthony Liguori

Re: KVM usability

[Avi Kivity]

On 03/09/2010 04:57 PM, Anthony Liguori wrote:
> On 03/09/2010 08:52 AM, Avi Kivity wrote:
>> On 03/09/2010 04:50 PM, Anthony Liguori wrote:
>>>> It's all in the openSUSE build service. The direct access URL 
>>>> (login required FWIW) is here:
>>>>
>>>> https://build.opensuse.org/package/view_file?file=kvm-qemu-default-memsize.patch&package=kvm&project=Virtualization 
>>>>
>>>>
>>>> It merely changes DEFAULT_RAM_SIZE in vl.c from 128 to 384 for the 
>>>> "kvm" package.
>>>
>>> We should attempt to do three things with default ram size:
>>>
>>> 1) bump it up to a more reasonable number
>>> 2) make it specified in the global default config
>>> 3) make sure we can provide compatibility support for older machine 
>>> types
>>
>> It's really sad, the amount of code needed to change a number.
>
> We don't do enough via a config.  If we did, we could just have a 0.12 
> config version that got frozen over time.
>
> So really, if we can make the mem readable by global config, and we 
> can have machine specific configs, it would simplify the problem in 
> the future so that we just had to bump a number.

Perhaps a json representation of things.  We already have the parser.

-- 
error compiling committee.c: too many arguments to function

Re: KVM usability

[Anthony Liguori]

On 03/09/2010 11:11 AM, Avi Kivity wrote:
> On 03/09/2010 04:57 PM, Anthony Liguori wrote:
>> On 03/09/2010 08:52 AM, Avi Kivity wrote:
>>> On 03/09/2010 04:50 PM, Anthony Liguori wrote:
>>>>> It's all in the openSUSE build service. The direct access URL 
>>>>> (login required FWIW) is here:
>>>>>
>>>>> https://build.opensuse.org/package/view_file?file=kvm-qemu-default-memsize.patch&package=kvm&project=Virtualization 
>>>>>
>>>>>
>>>>> It merely changes DEFAULT_RAM_SIZE in vl.c from 128 to 384 for the 
>>>>> "kvm" package.
>>>>
>>>> We should attempt to do three things with default ram size:
>>>>
>>>> 1) bump it up to a more reasonable number
>>>> 2) make it specified in the global default config
>>>> 3) make sure we can provide compatibility support for older machine 
>>>> types
>>>
>>> It's really sad, the amount of code needed to change a number.
>>
>> We don't do enough via a config.  If we did, we could just have a 
>> 0.12 config version that got frozen over time.
>>
>> So really, if we can make the mem readable by global config, and we 
>> can have machine specific configs, it would simplify the problem in 
>> the future so that we just had to bump a number.
>
> Perhaps a json representation of things.  We already have the parser.

Please no :-)

We have a config format, QemuOpts ties nicely into it as does qdev.  We 
just need to represent machine information via QemuOpts and tie -m to 
manipulating the memory assigned to a machine.  IOW, instead of:

(machine_init)(ram_addr_t ram_size,
                         const char *boot_device,
                         const char *kernel_filename,
                         const char *kernel_cmdline,
                         const char *initrd_filename,
                         const char *cpu_model)

It should be:

(machine_init)(QemuOpts *opts);

Then we can have a [machine] section in the config where we describe all 
of these things.

Regards,

Anthony Liguori

Re: KVM usability

[Avi Kivity]

On 03/09/2010 07:27 PM, Anthony Liguori wrote:
>> Perhaps a json representation of things.  We already have the parser.
>
>
> Please no :-)
>
> We have a config format, QemuOpts ties nicely into it as does qdev.  
> We just need to represent machine information via QemuOpts and tie -m 
> to manipulating the memory assigned to a machine.  IOW, instead of:
>
> (machine_init)(ram_addr_t ram_size,
>                         const char *boot_device,
>                         const char *kernel_filename,
>                         const char *kernel_cmdline,
>                         const char *initrd_filename,
>                         const char *cpu_model)
>
> It should be:
>
> (machine_init)(QemuOpts *opts);
>
> Then we can have a [machine] section in the config where we describe 
> all of these things.

Looks good.

One day we'll read VHDL descriptions of the device model from the 
machine config file and tcg them to host native code, and qemu will be 
pure infrastructure with zero details.

-- 
error compiling committee.c: too many arguments to function

High CPU use of -usbdevice tablet (was Re: KVM usability)

[Chris Webb]

Avi Kivity <avi@redhat.com> writes:

> On 03/02/2010 11:34 AM, Jernej Simončič wrote:
> >On Tuesday, March 2, 2010, 9:21:18, Chris Webb wrote:
> >
> >>I remember about a year ago, someone asserting on the list that -usbdevice
> >>tablet was very CPU intensive even when not in use, and should be avoided if
> >>mouse support wasn't needed, e.g. on non-graphical VMs. Was that actually a
> >>significant hit, and is it still true today?
> >It would appear that this is still the case, at least on slower hosts
> >- on Atom Z530 (1,6GHz), the XP VM uses ~30% CPU when idle with
> >-usbdevice tablet, but only ~4% without it. However, on a faster host
> >(Core2 Quad 2,66GHz), there's practically no difference (Vista x64 VM
> >uses ~1% CPU when idle regardless of -usbdevice tablet).
> 
> Looks like the tablet is set to 100 Hz polling rate.  We may be able
> to get away with 30 Hz or even less (ep_bInterval, in ms, in
> hw/usb-wacom.c).

Hi Avi. Sorry for the very late follow-up, but I decided to experiment with
this. The cpu impact of the usb tablet device shows up fairly clearly on a
crude test on my (relatively low-spec) desktop. Running an idle Fedora 11
livecd on qemu-kvm 0.12.3, top shows around 0.1% of my cpu in use, but this
increases to roughly 5% when specifying -usbdevice tablet, and more detailed
examination with perf record/report suggests about a factor of thirty too.

It's actually a more general symptom with USB or at least HID devices by the
look of things: although -usb doesn't increase CPU use on its own, the same
increase in load can also be triggered by -usbdevice keyboard or mouse.
However, running with all three of -usbdevice mouse, keyboard and tablet
doesn't increase load any more than just one of these.

Changing the USB tablet polling interval from 10ms to 100ms in both
hw/usb-wacom.c and hw/usb-hid.c made no difference except the an increase in
bInterval shown in lsusb -v in the guest and the hint of jerky mouse
movement I expected from setting this value so high. A similar change to the
polling interval for the keyboard and mouse also made no difference to their
performance impact.

Taking the FRAME_TIMER_FREQ down to 100 in hw/usb-uhci.c does seem to reduce
the CPU load quite a bit, but at the expense of making the USB tablet (and
presumably all other USB devices) very laggy.

Could there be some bug here that causes the usb hid devices to wake qemu at
the maximum rate possible (FRAME_TIMER_FREQ?) rather than the configured
polling interval?

Best wishes,

Chris.

PS Vmmouse works fine as an absolute pointing device in the place of
-usbdevice tablet without the performance impact, but this isn't supported
out of the box with typical linux live CDs (e.g. Fedora 11 and 12 or
Knoppix) so unfortunately it's probably less suitable as a default
configuration to expose to end-users.

[Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Chris Webb]

Avi Kivity <avi@redhat.com> writes:

> On 03/02/2010 11:34 AM, Jernej Simončič wrote:
> >On Tuesday, March 2, 2010, 9:21:18, Chris Webb wrote:
> >
> >>I remember about a year ago, someone asserting on the list that -usbdevice
> >>tablet was very CPU intensive even when not in use, and should be avoided if
> >>mouse support wasn't needed, e.g. on non-graphical VMs. Was that actually a
> >>significant hit, and is it still true today?
> >It would appear that this is still the case, at least on slower hosts
> >- on Atom Z530 (1,6GHz), the XP VM uses ~30% CPU when idle with
> >-usbdevice tablet, but only ~4% without it. However, on a faster host
> >(Core2 Quad 2,66GHz), there's practically no difference (Vista x64 VM
> >uses ~1% CPU when idle regardless of -usbdevice tablet).
> 
> Looks like the tablet is set to 100 Hz polling rate.  We may be able
> to get away with 30 Hz or even less (ep_bInterval, in ms, in
> hw/usb-wacom.c).

Hi Avi. Sorry for the very late follow-up, but I decided to experiment with
this. The cpu impact of the usb tablet device shows up fairly clearly on a
crude test on my (relatively low-spec) desktop. Running an idle Fedora 11
livecd on qemu-kvm 0.12.3, top shows around 0.1% of my cpu in use, but this
increases to roughly 5% when specifying -usbdevice tablet, and more detailed
examination with perf record/report suggests about a factor of thirty too.

It's actually a more general symptom with USB or at least HID devices by the
look of things: although -usb doesn't increase CPU use on its own, the same
increase in load can also be triggered by -usbdevice keyboard or mouse.
However, running with all three of -usbdevice mouse, keyboard and tablet
doesn't increase load any more than just one of these.

Changing the USB tablet polling interval from 10ms to 100ms in both
hw/usb-wacom.c and hw/usb-hid.c made no difference except the an increase in
bInterval shown in lsusb -v in the guest and the hint of jerky mouse
movement I expected from setting this value so high. A similar change to the
polling interval for the keyboard and mouse also made no difference to their
performance impact.

Taking the FRAME_TIMER_FREQ down to 100 in hw/usb-uhci.c does seem to reduce
the CPU load quite a bit, but at the expense of making the USB tablet (and
presumably all other USB devices) very laggy.

Could there be some bug here that causes the usb hid devices to wake qemu at
the maximum rate possible (FRAME_TIMER_FREQ?) rather than the configured
polling interval?

Best wishes,

Chris.

PS Vmmouse works fine as an absolute pointing device in the place of
-usbdevice tablet without the performance impact, but this isn't supported
out of the box with typical linux live CDs (e.g. Fedora 11 and 12 or
Knoppix) so unfortunately it's probably less suitable as a default
configuration to expose to end-users.

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Paul Brook]

> > Looks like the tablet is set to 100 Hz polling rate.  We may be able
> > to get away with 30 Hz or even less (ep_bInterval, in ms, in
> > hw/usb-wacom.c).
>
> Changing the USB tablet polling interval from 10ms to 100ms in both
> hw/usb-wacom.c and hw/usb-hid.c made no difference except the an increase
>  in bInterval shown in lsusb -v in the guest and the hint of jerky mouse
>  movement I expected from setting this value so high. A similar change to
>  the polling interval for the keyboard and mouse also made no difference to
>  their performance impact.

The USB HID devices implement the SET_IDLE command, so the polling interval 
will have no real effect on performance.

My guess is that the overhead you're seeing is entirely from the USB host 
adapter having to wake up and check the transport descriptor lists. This will 
only result in the guest being woken if a device actually responds (as 
mentioned above it should not).

>Taking the FRAME_TIMER_FREQ down to 100 in hw/usb-uhci.c does seem to reduce
>the CPU load quite a bit, but at the expense of making the USB tablet (and
>presumably all other USB devices) very laggy.

The guest USB driver explicitly decides which devices to poll each frame. 
Slowing down the frame rate will effectively change the polling period by the 
same factor. e.g. the HID device requests a polling rate of 10ms, you slowed 
down frame rate by 10x, so you're efectively only polling every 100ms.

If you want a quick and nasty hack then you can probably make the device wake 
up less often, and process multiple frames every wakeup.  However this is 
probably going to do bad things (at best extremely poor performance) when 
using actual USB devices.

Fixing this properly is hard because the transport descriptor lists are stores 
in system RAM, and polled by the host adapter.  The first step is to read the 
whole table of descriptors, and calculate when the next event is due. However 
the guest will not explicitly notify the HBA when these tables are modified, 
so you also need some sort of MMU trap to trigger recalculation.

This only gets you down to the base polling interval requested by the device.  
Increasing this interval causes significant user visible latency, so 
increasing it is not an option. The guest is also likely to distribute polling 
events evenly, further reducing the effective sleep interval.  To fix this you 
need additional APIs so that a device can report when an endpoint will become 
unblocked, rather than just waiting to be polled and NAKing the request.

Paul

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Paul Brook]

> > Looks like the tablet is set to 100 Hz polling rate.  We may be able
> > to get away with 30 Hz or even less (ep_bInterval, in ms, in
> > hw/usb-wacom.c).
>
> Changing the USB tablet polling interval from 10ms to 100ms in both
> hw/usb-wacom.c and hw/usb-hid.c made no difference except the an increase
>  in bInterval shown in lsusb -v in the guest and the hint of jerky mouse
>  movement I expected from setting this value so high. A similar change to
>  the polling interval for the keyboard and mouse also made no difference to
>  their performance impact.

The USB HID devices implement the SET_IDLE command, so the polling interval 
will have no real effect on performance.

My guess is that the overhead you're seeing is entirely from the USB host 
adapter having to wake up and check the transport descriptor lists. This will 
only result in the guest being woken if a device actually responds (as 
mentioned above it should not).

>Taking the FRAME_TIMER_FREQ down to 100 in hw/usb-uhci.c does seem to reduce
>the CPU load quite a bit, but at the expense of making the USB tablet (and
>presumably all other USB devices) very laggy.

The guest USB driver explicitly decides which devices to poll each frame. 
Slowing down the frame rate will effectively change the polling period by the 
same factor. e.g. the HID device requests a polling rate of 10ms, you slowed 
down frame rate by 10x, so you're efectively only polling every 100ms.

If you want a quick and nasty hack then you can probably make the device wake 
up less often, and process multiple frames every wakeup.  However this is 
probably going to do bad things (at best extremely poor performance) when 
using actual USB devices.

Fixing this properly is hard because the transport descriptor lists are stores 
in system RAM, and polled by the host adapter.  The first step is to read the 
whole table of descriptors, and calculate when the next event is due. However 
the guest will not explicitly notify the HBA when these tables are modified, 
so you also need some sort of MMU trap to trigger recalculation.

This only gets you down to the base polling interval requested by the device.  
Increasing this interval causes significant user visible latency, so 
increasing it is not an option. The guest is also likely to distribute polling 
events evenly, further reducing the effective sleep interval.  To fix this you 
need additional APIs so that a device can report when an endpoint will become 
unblocked, rather than just waiting to be polled and NAKing the request.

Paul

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Avi Kivity]

On 04/04/2010 05:25 PM, Paul Brook wrote:
>>> Looks like the tablet is set to 100 Hz polling rate.  We may be able
>>> to get away with 30 Hz or even less (ep_bInterval, in ms, in
>>> hw/usb-wacom.c).
>>>        
>> Changing the USB tablet polling interval from 10ms to 100ms in both
>> hw/usb-wacom.c and hw/usb-hid.c made no difference except the an increase
>>   in bInterval shown in lsusb -v in the guest and the hint of jerky mouse
>>   movement I expected from setting this value so high. A similar change to
>>   the polling interval for the keyboard and mouse also made no difference to
>>   their performance impact.
>>      
> The USB HID devices implement the SET_IDLE command, so the polling interval
> will have no real effect on performance.
>    

On a Linux guest (F12), I see 125 USB interrupts per second with no 
mouse movement, so something is broken (on the guest or host).

> My guess is that the overhead you're seeing is entirely from the USB host
> adapter having to wake up and check the transport descriptor lists. This will
> only result in the guest being woken if a device actually responds (as
> mentioned above it should not).
>    

A quick profile on the host side doesn't show this.  Instead, I see a 
lot of select() overhead.  Surprising as there are ~10 descriptors being 
polled, so ~1200 polls per second.  Maybe epoll will help here.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Avi Kivity]

On 04/04/2010 05:25 PM, Paul Brook wrote:
>>> Looks like the tablet is set to 100 Hz polling rate.  We may be able
>>> to get away with 30 Hz or even less (ep_bInterval, in ms, in
>>> hw/usb-wacom.c).
>>>        
>> Changing the USB tablet polling interval from 10ms to 100ms in both
>> hw/usb-wacom.c and hw/usb-hid.c made no difference except the an increase
>>   in bInterval shown in lsusb -v in the guest and the hint of jerky mouse
>>   movement I expected from setting this value so high. A similar change to
>>   the polling interval for the keyboard and mouse also made no difference to
>>   their performance impact.
>>      
> The USB HID devices implement the SET_IDLE command, so the polling interval
> will have no real effect on performance.
>    

On a Linux guest (F12), I see 125 USB interrupts per second with no 
mouse movement, so something is broken (on the guest or host).

> My guess is that the overhead you're seeing is entirely from the USB host
> adapter having to wake up and check the transport descriptor lists. This will
> only result in the guest being woken if a device actually responds (as
> mentioned above it should not).
>    

A quick profile on the host side doesn't show this.  Instead, I see a 
lot of select() overhead.  Surprising as there are ~10 descriptors being 
polled, so ~1200 polls per second.  Maybe epoll will help here.


-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Paul Brook]

> > The USB HID devices implement the SET_IDLE command, so the polling
> > interval will have no real effect on performance.
> 
> On a Linux guest (F12), I see 125 USB interrupts per second with no
> mouse movement, so something is broken (on the guest or host).

Turns out to be a a bug in the UHCI emulation. It should only raise an 
interrupt if the transfer actually completes (i.e. the active bit is set to 
zero). Fixed by 5bd2c0d7.

I was testing with an OHCI controller, which does not have this bug.

Paul

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Paul Brook]

> > The USB HID devices implement the SET_IDLE command, so the polling
> > interval will have no real effect on performance.
> 
> On a Linux guest (F12), I see 125 USB interrupts per second with no
> mouse movement, so something is broken (on the guest or host).

Turns out to be a a bug in the UHCI emulation. It should only raise an 
interrupt if the transfer actually completes (i.e. the active bit is set to 
zero). Fixed by 5bd2c0d7.

I was testing with an OHCI controller, which does not have this bug.

Paul

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Paul Brook]

> > My guess is that the overhead you're seeing is entirely from the USB host
> > adapter having to wake up and check the transport descriptor lists. This
> > will only result in the guest being woken if a device actually responds
> > (as mentioned above it should not).
> 
> A quick profile on the host side doesn't show this.  Instead, I see a
> lot of select() overhead.

This actually confirms my hypothesis. After fixing the UHCI bug the guest is 
completely idle, but the host still needs to wake up at 1ms intervals to do 
UHCI emulation. I can believe that the most visible part of this is the 
select() syscall.

> Surprising as there are ~10 descriptors being
> polled, so ~1200 polls per second.  Maybe epoll will help here.

I'm not sure where you get 1200 from.  select will be called once per host 
wakeup. i.e. if the USB controller is enabled then 1k times per second due to 
the frame tick.

Are you sure there are actually 10 descriptors being polled? Remember that the 
nfds argument is the value of the largest fd in the set (+1), not the number 
of descriptors in the set.

Paul

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Paul Brook]

> > My guess is that the overhead you're seeing is entirely from the USB host
> > adapter having to wake up and check the transport descriptor lists. This
> > will only result in the guest being woken if a device actually responds
> > (as mentioned above it should not).
> 
> A quick profile on the host side doesn't show this.  Instead, I see a
> lot of select() overhead.

This actually confirms my hypothesis. After fixing the UHCI bug the guest is 
completely idle, but the host still needs to wake up at 1ms intervals to do 
UHCI emulation. I can believe that the most visible part of this is the 
select() syscall.

> Surprising as there are ~10 descriptors being
> polled, so ~1200 polls per second.  Maybe epoll will help here.

I'm not sure where you get 1200 from.  select will be called once per host 
wakeup. i.e. if the USB controller is enabled then 1k times per second due to 
the frame tick.

Are you sure there are actually 10 descriptors being polled? Remember that the 
nfds argument is the value of the largest fd in the set (+1), not the number 
of descriptors in the set.

Paul

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Avi Kivity]

On 04/05/2010 12:53 AM, Paul Brook wrote:
>
>> Surprising as there are ~10 descriptors being
>> polled, so ~1200 polls per second.  Maybe epoll will help here.
>>      
> I'm not sure where you get 1200 from.  select will be called once per host
> wakeup. i.e. if the USB controller is enabled then 1k times per second due to
> the frame tick.
>    

That was 125 interrupts/sec x 10 fds ('poll' here was the per-file 
callback in the kernel, not the select call).  With your numbers, it's 
more like 10k polls/sec which can be eliminated.

> Are you sure there are actually 10 descriptors being polled? Remember that the
> nfds argument is the value of the largest fd in the set (+1), not the number
> of descriptors in the set.
>    

I was estimating from the strace parsed output, the real number is 7.  
So the kernel calls 7k callbacks/sec only to return with a timeout.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.

Re: [Qemu-devel] High CPU use of -usbdevice tablet (was Re: KVM usability)

[Avi Kivity]

On 04/05/2010 12:53 AM, Paul Brook wrote:
>
>> Surprising as there are ~10 descriptors being
>> polled, so ~1200 polls per second.  Maybe epoll will help here.
>>      
> I'm not sure where you get 1200 from.  select will be called once per host
> wakeup. i.e. if the USB controller is enabled then 1k times per second due to
> the frame tick.
>    

That was 125 interrupts/sec x 10 fds ('poll' here was the per-file 
callback in the kernel, not the select call).  With your numbers, it's 
more like 10k polls/sec which can be eliminated.

> Are you sure there are actually 10 descriptors being polled? Remember that the
> nfds argument is the value of the largest fd in the set (+1), not the number
> of descriptors in the set.
>    

I was estimating from the strace parsed output, the real number is 7.  
So the kernel calls 7k callbacks/sec only to return with a timeout.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.