[ANNOUNCE] kvm-14 release

[ANNOUNCE] kvm-14 release

[Avi Kivity]

The big news here is the update to qemu 0.9.0.

Changes from kvm-13:
- qemu 0.9.0
   - too many goodies to list
   - kvm can no longer share qemu's bios on Intel hosts due to real mode 
trouble.  use the supplied bios.
- migration now based on Anthony Liguori's live migration patches (Uri 
Lublin)
   - currently, only non-live migration is supported under kvm
- handle smi on host on AMD hosts (Joerg Roedel)
- random fixes

Note that if you use the modules from Linux 2.6.20, you need to use 
kvm-12.  You can use kvm-14 with Linux 2.6.20, provided you use the 
external module included in kvm-14.

http://kvm.qumranet.com

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Daniel P. Berrange]

On Mon, Feb 19, 2007 at 01:01:36PM +0200, Avi Kivity wrote:
> The big news here is the update to qemu 0.9.0.
> 
> Changes from kvm-13:
> - qemu 0.9.0
>    - too many goodies to list
>    - kvm can no longer share qemu's bios on Intel hosts due to real mode 
> trouble.  use the supplied bios.
> - migration now based on Anthony Liguori's live migration patches (Uri 
> Lublin)
>    - currently, only non-live migration is supported under kvm
> - handle smi on host on AMD hosts (Joerg Roedel)
> - random fixes
> 
> Note that if you use the modules from Linux 2.6.20, you need to use 
> kvm-12.  You can use kvm-14 with Linux 2.6.20, provided you use the 
> external module included in kvm-14.

Is there back-compatability in the reverse direction ? ie, will the
new kmod from kvm-14 work with the older kvm-12 userspace ? 

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Avi Kivity]

Daniel P. Berrange wrote:
>> Note that if you use the modules from Linux 2.6.20, you need to use 
>> kvm-12.  You can use kvm-14 with Linux 2.6.20, provided you use the 
>> external module included in kvm-14.
>>     
>
> Is there back-compatability in the reverse direction ? ie, will the
> new kmod from kvm-14 work with the older kvm-12 userspace ? 
>   

No.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Andreas Hasenack]

On Monday 19 February 2007 09:01:36 Avi Kivity wrote:
> The big news here is the update to qemu 0.9.0.
>
> Changes from kvm-13:
> - qemu 0.9.0
>    - too many goodies to list
>    - kvm can no longer share qemu's bios on Intel hosts due to real mode
> trouble.  use the supplied bios.
> - migration now based on Anthony Liguori's live migration patches (Uri
> Lublin)
>    - currently, only non-live migration is supported under kvm
> - handle smi on host on AMD hosts (Joerg Roedel)
> - random fixes
>
> Note that if you use the modules from Linux 2.6.20, you need to use
> kvm-12.  You can use kvm-14 with Linux 2.6.20, provided you use the
> external module included in kvm-14.

How are you guys in general dealing with the gcc-3.x requirement? Most distros 
already use gcc-4.x.

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Avi Kivity]

Andreas Hasenack wrote:
> How are you guys in general dealing with the gcc-3.x requirement? Most distros 
> already use gcc-4.x.
>   

If you're not planning to use the -no-kvm option, you can use gcc 4.  
Otherwise, you can use the gcc 3 compat packages that most distros provide.

Note that only qemu needs gcc 3.  The kernel module and libkvm.a can and 
should use the system compiler.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Omar Khan]

Andreas Hasenack <ahasenack@...> writes:
> 
> How are you guys in general dealing with the gcc-3.x requirement? Most distros 
> already use gcc-4.x.
> 

gcc-3.x for qemu? Here is how you can install gcc-3.x without breaking gcc-4.x:

http://en.opensuse.org/Qemu_with_kqemu_kernel_module_support

you can then configure qemu with the appropriate options.


Omar Khan


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Avi Kivity]

Omar Khan wrote:
> Andreas Hasenack <ahasenack@...> writes:
>   
>> How are you guys in general dealing with the gcc-3.x requirement? Most distros 
>> already use gcc-4.x.
>>
>>     
>
> gcc-3.x for qemu? Here is how you can install gcc-3.x without breaking gcc-4.x:
>
> http://en.opensuse.org/Qemu_with_kqemu_kernel_module_support
>
> you can then configure qemu with the appropriate options.
>
>   

Some distros provide a gcc 3 package in addition to gcc 4.  For example, 
on Fedora Core 6, it is called compat-gcc-34.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[James Morris]

On Mon, 19 Feb 2007, Avi Kivity wrote:

> Daniel P. Berrange wrote:
> >> Note that if you use the modules from Linux 2.6.20, you need to use 
> >> kvm-12.  You can use kvm-14 with Linux 2.6.20, provided you use the 
> >> external module included in kvm-14.
> >>     
> >
> > Is there back-compatability in the reverse direction ? ie, will the
> > new kmod from kvm-14 work with the older kvm-12 userspace ? 
> >   
> 
> No.

It's traditional for the mainline kernel to maintain backward 
compatibility with existing userspace ABIs (e.g. syscalls).

The idea of a userspace application breaking because of a mainline kernel 
upgrade is not acceptable to users and also pretty much unsupportable by 
distros.


- James
-- 
James Morris
<jmorris-gx6/JNMH7DfYtjvyW6yDsg@public.gmane.org>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Avi Kivity]

James Morris wrote:
> It's traditional for the mainline kernel to maintain backward 
> compatibility with existing userspace ABIs (e.g. syscalls).
>
> The idea of a userspace application breaking because of a mainline kernel 
> upgrade is not acceptable to users and also pretty much unsupportable by 
> distros.
>   

That is of course very true, and it is a goal of kvm to have a stable 
userspace interface.  I plan to target 2.6.21 as the stable ABI release.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Aurelien Jarno]

Avi Kivity a écrit :
> The big news here is the update to qemu 0.9.0.
> 
> Changes from kvm-13:
> - qemu 0.9.0
>    - too many goodies to list
>    - kvm can no longer share qemu's bios on Intel hosts due to real mode 
> trouble.  use the supplied bios.
> - migration now based on Anthony Liguori's live migration patches (Uri 
> Lublin)
>    - currently, only non-live migration is supported under kvm
> - handle smi on host on AMD hosts (Joerg Roedel)
> - random fixes
> 

Thanks a lot for your work. I have just tried it.

The good news is that kvm is now able to boot a Hurd system, this what
no the case with kvm-12.

The bad news is that kvm-14 seems to be a lot slower than kvm-12 +
modules from kernel 2.6.20. This is the case with a GNU/kFreeBSD guest.
kvm-12 was about 1.5 time faster than qemu + kqemu. kvm-20 is slower
than qemu without kqemu...

Does anybody have an idea about this performance regression?


-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org         | aurelien-rXXEIb44qovR7s880joybQ@public.gmane.org
   `-    people.debian.org/~aurel32 | www.aurel32.net

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Rusty Russell]

On Mon, 2007-02-19 at 10:37 -0500, James Morris wrote:
> On Mon, 19 Feb 2007, Avi Kivity wrote:
> 
> > Daniel P. Berrange wrote:
> > >> Note that if you use the modules from Linux 2.6.20, you need to use 
> > >> kvm-12.  You can use kvm-14 with Linux 2.6.20, provided you use the 
> > >> external module included in kvm-14.
> > >>     
> > >
> > > Is there back-compatability in the reverse direction ? ie, will the
> > > new kmod from kvm-14 work with the older kvm-12 userspace ? 
> > >   
> > 
> > No.
> 
> It's traditional for the mainline kernel to maintain backward 
> compatibility with existing userspace ABIs (e.g. syscalls).

Stable ABIs are hard, and this is CONFIG_EXPERIMENTAL after all.  I
don't think KVM needs ABI stability until CONFIG_EXPERIMENTAL is removed
(of course, sooner is always nice if it doesn't create problems).

Rusty.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[James Morris]

On Tue, 20 Feb 2007, Rusty Russell wrote:

> Stable ABIs are hard, and this is CONFIG_EXPERIMENTAL after all.  I
> don't think KVM needs ABI stability until CONFIG_EXPERIMENTAL is removed
> (of course, sooner is always nice if it doesn't create problems).

FC7 will ship with kvm and 2.6.21, so changing the ABI after that will 
cause problems for real users and distro maintainers.  We already have 
enough of this to deal with due to Xen.



- James
-- 
James Morris
<jmorris-gx6/JNMH7DfYtjvyW6yDsg@public.gmane.org>

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Avi Kivity]

Aurelien Jarno wrote:
> The bad news is that kvm-14 seems to be a lot slower than kvm-12 +
> modules from kernel 2.6.20. This is the case with a GNU/kFreeBSD guest.
> kvm-12 was about 1.5 time faster than qemu + kqemu. kvm-20 is slower
> than qemu without kqemu...
>
> Does anybody have an idea about this performance regression?
>   

What is your workload?  How are you measuring performance?

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Aurelien Jarno]

Avi Kivity a écrit :
> Aurelien Jarno wrote:
>> The bad news is that kvm-14 seems to be a lot slower than kvm-12 +
>> modules from kernel 2.6.20. This is the case with a GNU/kFreeBSD guest.
>> kvm-12 was about 1.5 time faster than qemu + kqemu. kvm-20 is slower
>> than qemu without kqemu...
>>
>> Does anybody have an idea about this performance regression?
>>   
> 
> What is your workload?  How are you measuring performance?

Sorry to answer only now, it took me some time to do some more
measurements and have some numbers. I am simply building a Debian
package (simulpic) and measuring the build time. Basically the command is:

  apt-get source simulpic
  cd simulpic-2005-1-28
 time debuild -uc us

It surely not a performance index, but I guess it is ok to compare
performances between version. It is also quite representative of my use
of the machine.

The guest is Debian GNU/kFreeBSD amd64 (ie FreeBSD kernel + GNU libc).
It is accessed via ssh, and kvm is started with -nographic, so there is
no influence of xorg.

I am doing my tests on an Athlon X2 3800+ machine, running a 2.6.20
kernel. During all my tests, the machine is not loaded with other tasks
(except systems tasks), so qemu or kvm have a full core available. Top
shows that the core is used between 95 and 100% during the whole build
in all cases.

The tests I have made are presented below. In all cases I have verified
that the real time correspond to the time of my wall clock, it is
correct in all case given the resolution of my wall clock (1 s):

qemu
----
real    3m16.626s
user    2m22.654s
sys     0m41.738s

qemu + kqemu
------------
real    0m51.529s
user    0m11.775s
sys     0m36.215s

kvm 12 + modules from kernel 2.6.20
-----------------------------------
real    0m30.635s
user    0m16.357s
sys     0m8.511s

kvm 12
------
real    0m25.357s
user    0m16.259s
sys     0m6.496s

kvm 13
------
real    0m23.415s
user    0m15.177s
sys     0m5.811s

kvm 14
------
real    7m47.310s
user    5m17.359s
sys     2m3.184s


Using kvm 14, the system is clearly not responsive at all. You can see
that without running a benchmark.

Also running Hurd under kvm shows the same problem, though I can't make
such measurements, because it does not run under kvm < 14. Note however
that the ping time of the guest machine running Hurd during I/O
increases from a few milliseconds to a dozen of seconds.

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org         | aurelien-rXXEIb44qovR7s880joybQ@public.gmane.org
   `-    people.debian.org/~aurel32 | www.aurel32.net

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Avi Kivity]

Aurelien Jarno wrote:
> Avi Kivity a écrit :
>   
>> Aurelien Jarno wrote:
>>     
>>> The bad news is that kvm-14 seems to be a lot slower than kvm-12 +
>>> modules from kernel 2.6.20. This is the case with a GNU/kFreeBSD guest.
>>> kvm-12 was about 1.5 time faster than qemu + kqemu. kvm-20 is slower
>>> than qemu without kqemu...
>>>
>>> Does anybody have an idea about this performance regression?
>>>   
>>>       
>> What is your workload?  How are you measuring performance?
>>     
>
> Sorry to answer only now, it took me some time to do some more
> measurements and have some numbers. I am simply building a Debian
> package (simulpic) and measuring the build time. Basically the command is:
>
>   apt-get source simulpic
>   cd simulpic-2005-1-28
>  time debuild -uc us
>
> It surely not a performance index, but I guess it is ok to compare
> performances between version. It is also quite representative of my use
> of the machine.
>   

Real workloads (likr this) are more important than synthetic benchmarks.

> The guest is Debian GNU/kFreeBSD amd64 (ie FreeBSD kernel + GNU libc).
> It is accessed via ssh, and kvm is started with -nographic, so there is
> no influence of xorg.
>
> I am doing my tests on an Athlon X2 3800+ machine, running a 2.6.20
> kernel. During all my tests, the machine is not loaded with other tasks
> (except systems tasks), so qemu or kvm have a full core available. Top
> shows that the core is used between 95 and 100% during the whole build
> in all cases.
>
> The tests I have made are presented below. In all cases I have verified
> that the real time correspond to the time of my wall clock, it is
> correct in all case given the resolution of my wall clock (1 s):
>
> qemu
> ----
> real    3m16.626s
> user    2m22.654s
> sys     0m41.738s
>   

Is this qemu 0.8.2 or qemu 0.9.0?

> qemu + kqemu
> ------------
> real    0m51.529s
> user    0m11.775s
> sys     0m36.215s
>
> kvm 12 + modules from kernel 2.6.20
> -----------------------------------
> real    0m30.635s
> user    0m16.357s
> sys     0m8.511s
>
> kvm 12
> ------
> real    0m25.357s
> user    0m16.259s
> sys     0m6.496s
>
> kvm 13
> ------
> real    0m23.415s
> user    0m15.177s
> sys     0m5.811s
>   

So far so good.  Steady improvement.  The low system time indicates a 
lot of I/O and inefficiency in the qemu device emulation (guest time is 
charged as system time).

> kvm 14
> ------
> real    7m47.310s
> user    5m17.359s
> sys     2m3.184s
>
>
> Using kvm 14, the system is clearly not responsive at all. You can see
> that without running a benchmark.
>   

kvm-14 is mostly qemu 0.9.0.  Do you get the same results with kvm-14 
-no-kvm?

What is your disk image file format, or are you using a partition?

Do the results change (on kvm-14) if you pin the guest to a core with 
'taskset 1 qemu ...'



Thank you for taking the time to do real measurements and report the 
results clearly.  That makes it possible (I hope) to find the cause and 
fix it.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Aurelien Jarno]

Avi Kivity a écrit :
> Aurelien Jarno wrote:
>> Avi Kivity a écrit :
>>   
>>> Aurelien Jarno wrote:
>>>     
>>>> The bad news is that kvm-14 seems to be a lot slower than kvm-12 +
>>>> modules from kernel 2.6.20. This is the case with a GNU/kFreeBSD guest.
>>>> kvm-12 was about 1.5 time faster than qemu + kqemu. kvm-20 is slower
>>>> than qemu without kqemu...
>>>>
>>>> Does anybody have an idea about this performance regression?
>>>>   
>>>>       
>>> What is your workload?  How are you measuring performance?
>>>     
>> Sorry to answer only now, it took me some time to do some more
>> measurements and have some numbers. I am simply building a Debian
>> package (simulpic) and measuring the build time. Basically the command is:
>>
>>   apt-get source simulpic
>>   cd simulpic-2005-1-28
>>  time debuild -uc us
>>
>> It surely not a performance index, but I guess it is ok to compare
>> performances between version. It is also quite representative of my use
>> of the machine.
>>   
> 
> Real workloads (likr this) are more important than synthetic benchmarks.
> 
>> The guest is Debian GNU/kFreeBSD amd64 (ie FreeBSD kernel + GNU libc).
>> It is accessed via ssh, and kvm is started with -nographic, so there is
>> no influence of xorg.
>>
>> I am doing my tests on an Athlon X2 3800+ machine, running a 2.6.20
>> kernel. During all my tests, the machine is not loaded with other tasks
>> (except systems tasks), so qemu or kvm have a full core available. Top
>> shows that the core is used between 95 and 100% during the whole build
>> in all cases.
>>
>> The tests I have made are presented below. In all cases I have verified
>> that the real time correspond to the time of my wall clock, it is
>> correct in all case given the resolution of my wall clock (1 s):
>>
>> qemu
>> ----
>> real    3m16.626s
>> user    2m22.654s
>> sys     0m41.738s
>>   
> 
> Is this qemu 0.8.2 or qemu 0.9.0?

It's qemu 0.9.0 + kqemu 1.3.0pre11. kqemu is enabled in user mode only
(kernel mode does not work well on amd64)

>> qemu + kqemu
>> ------------
>> real    0m51.529s
>> user    0m11.775s
>> sys     0m36.215s
>>
>> kvm 12 + modules from kernel 2.6.20
>> -----------------------------------
>> real    0m30.635s
>> user    0m16.357s
>> sys     0m8.511s
>>
>> kvm 12
>> ------
>> real    0m25.357s
>> user    0m16.259s
>> sys     0m6.496s
>>
>> kvm 13
>> ------
>> real    0m23.415s
>> user    0m15.177s
>> sys     0m5.811s
>>   
> 
> So far so good.  Steady improvement.  The low system time indicates a 
> lot of I/O and inefficiency in the qemu device emulation (guest time is 
> charged as system time).
> 
>> kvm 14
>> ------
>> real    7m47.310s
>> user    5m17.359s
>> sys     2m3.184s
>>
>>
>> Using kvm 14, the system is clearly not responsive at all. You can see
>> that without running a benchmark.
>>   
> 
> kvm-14 is mostly qemu 0.9.0.  Do you get the same results with kvm-14 
> -no-kvm?

Here are the results:

real    3m45.459s
user    2m48.581s
sys     0m48.585s

It's a bit slower than qemu 0.9.0, but a lot faster than kvm 0.9.0 with
kvm enable.

> What is your disk image file format, or are you using a partition?

I am using a raw image file on an ext3 partition.

> Do the results change (on kvm-14) if you pin the guest to a core with 
> 'taskset 1 qemu ...'

Bingo. It now works even faster than kvm-13!

real    0m22.307s
user    0m13.935s
sys     0m4.720


> Thank you for taking the time to do real measurements and report the 
> results clearly.  That makes it possible (I hope) to find the cause and 
> fix it.

Thanks for your help! Do you think this problem is fixable? On my final
machine, I have a dozen of qemu/kvm running, and when I start them I
don't know how they will be used, and so how to pin them on the two cores.

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org         | aurelien-rXXEIb44qovR7s880joybQ@public.gmane.org
   `-    people.debian.org/~aurel32 | www.aurel32.net

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Avi Kivity]

Aurelien Jarno wrote:
>   
>> What is your disk image file format, or are you using a partition?
>>     
>
> I am using a raw image file on an ext3 partition.
>
>   
>> Do the results change (on kvm-14) if you pin the guest to a core with 
>> 'taskset 1 qemu ...'
>>     
>
> Bingo. It now works even faster than kvm-13!
>
> real    0m22.307s
> user    0m13.935s
> sys     0m4.720
>
>   

I'm guessing this is due to the glibc aio implementation, which uses 
threads instead of true aio.  The threads may cause the vcpu to migrate 
frequently from one code to another.

There are two possible solutions:

 - use native aio from http://www.bullopensource.org/posix/.  I think 
the aio signal patches are not yet in, so this may not work.
 - teach the scheduler about the cost of migrating vcpus

The first approach will solve itself eventually, though slowly if the 
current slow rate of aio merging continues.  We'll have to do the second.

>   
>> Thank you for taking the time to do real measurements and report the 
>> results clearly.  That makes it possible (I hope) to find the cause and 
>> fix it.
>>     
>
> Thanks for your help! Do you think this problem is fixable? On my final
> machine, I have a dozen of qemu/kvm running, and when I start them I
> don't know how they will be used, and so how to pin them on the two cores.
>   

For a temporary solution, I'd just pick a random core in a startup 
script and trust to probability.  Hopefully the scheduler will get fixed 
to handle vcpus soon.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Laurent Vivier]

[-- Attachment #1.1: Type: text/plain, Size: 1436 bytes --]

Avi Kivity wrote:
> Aurelien Jarno wrote:
>>   
>>> What is your disk image file format, or are you using a partition?
>>>     
>> I am using a raw image file on an ext3 partition.
>>
>>   
>>> Do the results change (on kvm-14) if you pin the guest to a core with 
>>> 'taskset 1 qemu ...'
>>>     
>> Bingo. It now works even faster than kvm-13!
>>
>> real    0m22.307s
>> user    0m13.935s
>> sys     0m4.720
>>
>>   
> 
> I'm guessing this is due to the glibc aio implementation, which uses 
> threads instead of true aio.  The threads may cause the vcpu to migrate 
> frequently from one code to another.
> 
> There are two possible solutions:
> 
>  - use native aio from http://www.bullopensource.org/posix/.  I think 
> the aio signal patches are not yet in, so this may not work.
>  - teach the scheduler about the cost of migrating vcpus
> The first approach will solve itself eventually, though slowly if the 
> current slow rate of aio merging continues.  We'll have to do the second.
> 

if you prefer the first one, Sébastien will release very soon aio patches for
2.6.20 with an up-to-date libposix-aio.

[advertising] Keep an eye on the bullopensource website. [/advertising]
:-P

Regards,
Laurent
-- 
------------- Laurent.Vivier-6ktuUTfB/bM@public.gmane.org  --------------
       "Any sufficiently advanced technology is
  indistinguishable from magic." - Arthur C. Clarke


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 345 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: [ANNOUNCE] kvm-14 release

[Anthony Liguori]

Laurent Vivier wrote:
> Avi Kivity wrote:
>   
>> Aurelien Jarno wrote:
>>     
>>>   
>>>       
>>>> What is your disk image file format, or are you using a partition?
>>>>     
>>>>         
>>> I am using a raw image file on an ext3 partition.
>>>
>>>   
>>>       
>>>> Do the results change (on kvm-14) if you pin the guest to a core with 
>>>> 'taskset 1 qemu ...'
>>>>     
>>>>         
>>> Bingo. It now works even faster than kvm-13!
>>>
>>> real    0m22.307s
>>> user    0m13.935s
>>> sys     0m4.720
>>>
>>>   
>>>       
>> I'm guessing this is due to the glibc aio implementation, which uses 
>> threads instead of true aio.  The threads may cause the vcpu to migrate 
>> frequently from one code to another.
>>
>> There are two possible solutions:
>>
>>  - use native aio from http://www.bullopensource.org/posix/.  I think 
>> the aio signal patches are not yet in, so this may not work.
>>  - teach the scheduler about the cost of migrating vcpus
>> The first approach will solve itself eventually, though slowly if the 
>> current slow rate of aio merging continues.  We'll have to do the second.
>>
>>     
>
> if you prefer the first one, Sébastien will release very soon aio patches for
> 2.6.20 with an up-to-date libposix-aio.
>   

Hi Laurent,

I gave that a shot a little bit ago.  Ran into two problems.

1) Couldn't avoid linking to -lrt as QEMU uses time functions from it.
2) While I could get things compiling (with patches), QEMU would SEGV 
almost immediately.

Could you guys maybe give compiling QEMU w/libposix-aio a shot?  I'm 
really interested to see if it makes a difference.

Regards,

Anthony Liguori

> [advertising] Keep an eye on the bullopensource website. [/advertising]
> :-P
>
> Regards,
> Laurent
>   
> ------------------------------------------------------------------------
>
> -------------------------------------------------------------------------
> Take Surveys. Earn Cash. Influence the Future of IT
> Join SourceForge.net's Techsay panel and you'll get the chance to share your
> opinions on IT & business topics through brief surveys-and earn cash
> http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
> ------------------------------------------------------------------------
>
> _______________________________________________
> kvm-devel mailing list
> kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>   


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Laurent Vivier]

[-- Attachment #1.1: Type: text/plain, Size: 2004 bytes --]

Anthony Liguori wrote:
> Laurent Vivier wrote:
>> Avi Kivity wrote:
>>   
>>> Aurelien Jarno wrote:
>>>     
>>>>   
>>>>       
>>>>> What is your disk image file format, or are you using a partition?
>>>>>     
>>>>>         
>>>> I am using a raw image file on an ext3 partition.
>>>>
>>>>   
>>>>       
>>>>> Do the results change (on kvm-14) if you pin the guest to a core with 
>>>>> 'taskset 1 qemu ...'
>>>>>     
>>>>>         
>>>> Bingo. It now works even faster than kvm-13!
>>>>
>>>> real    0m22.307s
>>>> user    0m13.935s
>>>> sys     0m4.720
>>>>
>>>>   
>>>>       
>>> I'm guessing this is due to the glibc aio implementation, which uses 
>>> threads instead of true aio.  The threads may cause the vcpu to migrate 
>>> frequently from one code to another.
>>>
>>> There are two possible solutions:
>>>
>>>  - use native aio from http://www.bullopensource.org/posix/.  I think 
>>> the aio signal patches are not yet in, so this may not work.
>>>  - teach the scheduler about the cost of migrating vcpus
>>> The first approach will solve itself eventually, though slowly if the 
>>> current slow rate of aio merging continues.  We'll have to do the second.
>>>
>>>     
>> if you prefer the first one, Sébastien will release very soon aio patches for
>> 2.6.20 with an up-to-date libposix-aio.
>>   
> 
> Hi Laurent,

Hi Anthony,

> I gave that a shot a little bit ago.  Ran into two problems.
> 
> 1) Couldn't avoid linking to -lrt as QEMU uses time functions from it.
> 2) While I could get things compiling (with patches), QEMU would SEGV 
> almost immediately.
> 
> Could you guys maybe give compiling QEMU w/libposix-aio a shot?  I'm 
> really interested to see if it makes a difference.

Ok, I take a look at this.

Regards,
Laurent
-- 
------------- Laurent.Vivier-6ktuUTfB/bM@public.gmane.org  --------------
       "Any sufficiently advanced technology is
  indistinguishable from magic." - Arthur C. Clarke


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 345 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: [ANNOUNCE] kvm-14 release

[Laurent Vivier]

[-- Attachment #1.1.1: Type: text/plain, Size: 3199 bytes --]

Anthony Liguori wrote:
> Laurent Vivier wrote:
>> Avi Kivity wrote:
>>   
>>> Aurelien Jarno wrote:
>>>     
>>>>   
>>>>       
>>>>> What is your disk image file format, or are you using a partition?
>>>>>     
>>>>>         
>>>> I am using a raw image file on an ext3 partition.
>>>>
>>>>   
>>>>       
>>>>> Do the results change (on kvm-14) if you pin the guest to a core with 
>>>>> 'taskset 1 qemu ...'
>>>>>     
>>>>>         
>>>> Bingo. It now works even faster than kvm-13!
>>>>
>>>> real    0m22.307s
>>>> user    0m13.935s
>>>> sys     0m4.720
>>>>
>>>>   
>>>>       
>>> I'm guessing this is due to the glibc aio implementation, which uses 
>>> threads instead of true aio.  The threads may cause the vcpu to migrate 
>>> frequently from one code to another.
>>>
>>> There are two possible solutions:
>>>
>>>  - use native aio from http://www.bullopensource.org/posix/.  I think 
>>> the aio signal patches are not yet in, so this may not work.
>>>  - teach the scheduler about the cost of migrating vcpus
>>> The first approach will solve itself eventually, though slowly if the 
>>> current slow rate of aio merging continues.  We'll have to do the second.
>>>
>>>     
>> if you prefer the first one, Sébastien will release very soon aio patches for
>> 2.6.20 with an up-to-date libposix-aio.
>>   
> 
> Hi Laurent,
> 
> I gave that a shot a little bit ago.  Ran into two problems.
> 
> 1) Couldn't avoid linking to -lrt as QEMU uses time functions from it.
> 2) While I could get things compiling (with patches), QEMU would SEGV 
> almost immediately.
> 
> Could you guys maybe give compiling QEMU w/libposix-aio a shot?  I'm 
> really interested to see if it makes a difference.
> 
> Regards,
> 
> Anthony Liguori
> 
>> [advertising] Keep an eye on the bullopensource website. [/advertising]
>> :-P
>>
>> Regards,
>> Laurent

OK, I didn' have time to test the performance of the result, but you can find
attached some patches to enable libposix-aio with kvm-14.

first take last patches for linux-2.6.20 and libposix-aio-.0.8.2 from website :

http://sourceforge.net/projects/paiol

if you are using AMD64, you must patch libposix-aio because there is a little
problem remaining to detect lio_submit syscall (first attachment)

then apply following patch to kvm-14 (second attachment).

It works fine on my system except when I use "-hda /dev/sdb" : qemu crashes just
after mounting filesystems when "init" tries to set kernel parameters with
"sysctl" (I use a debian 4.0). If I boot in emergency mode, mounting manually
filesystems and running manually sysctl, all works fine. It looks like a
synchronization problem.

There are remaining issues : libposix-aio uses kernel AIO, so files must be
opened using O_DIRECT and buffers must be aligned. libposix-aio is able to
manage other cases but this has a performance cost.

Aurélien, do you have any time to test this on your system ?

Regards,
Laurent
-- 
------------- Laurent.Vivier-6ktuUTfB/bM@public.gmane.org  --------------
       "Any sufficiently advanced technology is
  indistinguishable from magic." - Arthur C. Clarke

[-- Attachment #1.1.2: configure_ac-x86_64.patch --]
[-- Type: text/plain, Size: 1191 bytes --]

--- libposix-aio-0.8.2-org/configure.ac	2007-02-21 15:58:43.000000000 +0100
+++ libposix-aio-0.8.2/configure.ac	2007-02-22 09:43:39.000000000 +0100
@@ -217,6 +217,24 @@
 
 # test if kernel supports signal on LIO completion
 
+case $host_cpu in
+
+        x86_64*)
+                lio_submit_syscall=280
+        ;;
+  
+        i386*)
+                lio_submit_syscall=320
+        ;;
+	*)
+		AC_MSG_WARN(unknown syscall number for lio_submit)
+		lio_submit_syscall=
+        ;;
+esac
+
+if test "${lio_submit_syscall}" != ""
+then
+
 AC_ARG_ENABLE(
 	[lio-signal],
 	[AS_HELP_STRING(--enable-lio-signal,
@@ -236,7 +254,7 @@
 
 #define IO_CMD_PWRITE 1
 
-#define SYS_lio_submit 320
+#define SYS_lio_submit ${lio_submit_syscall}
 
 #define LIO_WAIT   0
 #define LIO_NOWAIT 1
@@ -397,7 +415,7 @@
 
 #define IO_CMD_PWRITE 1
 
-#define SYS_lio_submit 320
+#define SYS_lio_submit ${lio_submit_syscall}
 
 #define LIO_WAIT   0
 #define LIO_NOWAIT 1
@@ -514,6 +532,7 @@
 			Define to 1 if kernel supports LIO wait)],[ac_lio_wait=no
 	AC_MSG_RESULT(no)
 ],[]))
+fi # end of test lio_submit_syscall
 
 # check for support of fd for io_cancel
 

[-- Attachment #1.1.3: posixaio.diff --]
[-- Type: text/plain, Size: 3432 bytes --]

Index: kvm-14/configure
===================================================================
--- kvm-14.orig/configure	2007-02-22 11:46:09.000000000 +0100
+++ kvm-14/configure	2007-02-22 11:46:55.000000000 +0100
@@ -87,7 +87,7 @@
     --enable-kvm --kernel-path="$libkvm_kerneldir" \
     --enable-alsa \
     ${disable_gcc_check:+"--disable-gcc-check"} \
-    --prefix="$prefix"
+    --prefix="$prefix" --enable-libposix-aio
 )
 
 
Index: kvm-14/qemu/Makefile
===================================================================
--- kvm-14.orig/qemu/Makefile	2007-02-22 11:46:09.000000000 +0100
+++ kvm-14/qemu/Makefile	2007-02-22 11:46:12.000000000 +0100
@@ -27,10 +27,14 @@
 ifndef CONFIG_DARWIN
 ifndef CONFIG_WIN32
 ifndef CONFIG_SOLARIS
+ifdef USE_LIBPOSIX_AIO
+LIBS+=-lposix-aio
+else
 LIBS+=-lrt
 endif
 endif
 endif
+endif
 
 all: $(TOOLS) $(DOCS) recurse-all
 
Index: kvm-14/qemu/Makefile.target
===================================================================
--- kvm-14.orig/qemu/Makefile.target	2007-02-22 11:46:09.000000000 +0100
+++ kvm-14/qemu/Makefile.target	2007-02-22 11:46:12.000000000 +0100
@@ -452,7 +452,11 @@
 ifndef CONFIG_DARWIN
 ifndef CONFIG_WIN32
 ifndef CONFIG_SOLARIS
-VL_LIBS=-lutil -lrt -luuid
+VL_LIBS=-lutil
+ifdef USE_LIBPOSIX_AIO
+VL_LIBS+=-lposix-aio
+endif
+VL_LIBS+=-lrt -luuid
 endif
 endif
 endif
Index: kvm-14/qemu/block-raw.c
===================================================================
--- kvm-14.orig/qemu/block-raw.c	2007-02-22 11:46:09.000000000 +0100
+++ kvm-14/qemu/block-raw.c	2007-02-22 11:46:12.000000000 +0100
@@ -197,7 +197,7 @@
     act.sa_handler = aio_signal_handler;
     sigaction(aio_sig_num, &act, NULL);
 
-#if defined(__GLIBC__) && defined(__linux__)
+#if defined(__GLIBC__) && defined(__linux__) && !defined(USE_LIBPOSIX_AIO)
     {
         /* XXX: aio thread exit seems to hang on RedHat 9 and this init
            seems to fix the problem. */
Index: kvm-14/qemu/configure
===================================================================
--- kvm-14.orig/qemu/configure	2007-02-22 11:46:09.000000000 +0100
+++ kvm-14/qemu/configure	2007-02-22 11:46:12.000000000 +0100
@@ -99,6 +99,7 @@
 linux_user="no"
 darwin_user="no"
 build_docs="no"
+libposix_aio="no"
 uname_release=""
 
 # OS specific
@@ -262,6 +263,8 @@
   ;;
   --enable-uname-release=*) uname_release="$optarg"
   ;;
+  --enable-libposix-aio) libposix_aio="yes"
+  ;;
   esac
 done
 
@@ -312,6 +315,7 @@
 echo "  --fmod-lib               path to FMOD library"
 echo "  --fmod-inc               path to FMOD includes"
 echo "  --enable-uname-release=R Return R for uname -r in usermode emulation"
+echo "  --enable-libposix-aio    use libposix-aio instead of glibc aio"
 echo ""
 echo "NOTE: The object files are built at the place where configure is launched"
 exit 1
@@ -635,6 +639,7 @@
 echo "kqemu support     $kqemu"
 echo "kvm support       $kvm"
 echo "Documentation     $build_docs"
+echo "libposix-aio      $libposix_aio"
 [ ! -z "$uname_release" ] && \
 echo "uname -r          $uname_release"
 
@@ -795,6 +800,8 @@
   echo "#define MAP_ANONYMOUS MAP_ANON" >> $config_h
   echo "#define _BSD 1" >> $config_h
 fi
+echo "USE_LIBPOSIX_AIO=yes" >> $config_mak
+echo "#define USE_LIBPOSIX_AIO 1" >> $config_h
 
 echo "#define CONFIG_UNAME_RELEASE \"$uname_release\"" >> $config_h
 

[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 345 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: [ANNOUNCE] kvm-14 release

[Avi Kivity]

Laurent Vivier wrote:
> There are remaining issues : libposix-aio uses kernel AIO, so files must be
> opened using O_DIRECT and buffers must be aligned. libposix-aio is able to
> manage other cases but this has a performance cost.
>
>   

Because the guest is also doing dma, the buffers are expected to be 
aligned (it might be different if using pio, but unlikely).


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Anthony Liguori]

Avi Kivity wrote:
> Laurent Vivier wrote:
>> There are remaining issues : libposix-aio uses kernel AIO, so files 
>> must be
>> opened using O_DIRECT and buffers must be aligned. libposix-aio is 
>> able to
>> manage other cases but this has a performance cost.
>>
>>   
>
> Because the guest is also doing dma, the buffers are expected to be 
> aligned (it might be different if using pio, but unlikely).

The IDE emulation always uses a temporary buffer.  This is partially 
because you cannot always rely on being able to use an offset on 
phys_ram_base (when doing user only emulation).

We can pretty easily patch this code though to use the DMA address 
directly instead of the temporary buffer in the non PIO case.  We still 
have to define a raw_pread() that is capable of emulating sector 
alignment though since Linux will use PIO at first.

Regards,

Anthony Liguori


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Avi Kivity]

Anthony Liguori wrote:
> Avi Kivity wrote:
>> Laurent Vivier wrote:
>>> There are remaining issues : libposix-aio uses kernel AIO, so files 
>>> must be
>>> opened using O_DIRECT and buffers must be aligned. libposix-aio is 
>>> able to
>>> manage other cases but this has a performance cost.
>>>
>>>   
>>
>> Because the guest is also doing dma, the buffers are expected to be 
>> aligned (it might be different if using pio, but unlikely).
>
> The IDE emulation always uses a temporary buffer.  This is partially 
> because you cannot always rely on being able to use an offset on 
> phys_ram_base (when doing user only emulation).

Why is that?


>
> We can pretty easily patch this code though to use the DMA address 
> directly instead of the temporary buffer in the non PIO case.  We 
> still have to define a raw_pread() that is capable of emulating sector 
> alignment though since Linux will use PIO at first.

Sure.  It seems worthwhile for mainline qemu too.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Anthony Liguori]

Avi Kivity wrote:
> Anthony Liguori wrote:
>> Avi Kivity wrote:
>>> Laurent Vivier wrote:
>>>> There are remaining issues : libposix-aio uses kernel AIO, so files 
>>>> must be
>>>> opened using O_DIRECT and buffers must be aligned. libposix-aio is 
>>>> able to
>>>> manage other cases but this has a performance cost.
>>>>
>>>>   
>>>
>>> Because the guest is also doing dma, the buffers are expected to be 
>>> aligned (it might be different if using pio, but unlikely).
>>
>> The IDE emulation always uses a temporary buffer.  This is partially 
>> because you cannot always rely on being able to use an offset on 
>> phys_ram_base (when doing user only emulation).
>
> Why is that?

Well, user won't do IDE emulation, but cpu_physical_memory_rw has to 
jump through more hoops for IDE.

What's more relevant though, is it's possible for the guest to setup DMA 
to device memory, in which case, you need to route the IO to callbacks.  
That's pretty unlikely though so I think we can stream line it so that 
we maintain correctness while also getting less copies.

Regards,

Anthony Liguori

>
>>
>> We can pretty easily patch this code though to use the DMA address 
>> directly instead of the temporary buffer in the non PIO case.  We 
>> still have to define a raw_pread() that is capable of emulating 
>> sector alignment though since Linux will use PIO at first.
>
> Sure.  It seems worthwhile for mainline qemu too.
>
>


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Aurelien Jarno]

Laurent Vivier a écrit :
> Anthony Liguori wrote:
>> Laurent Vivier wrote:
>>> Avi Kivity wrote:
>>>   
>>>> Aurelien Jarno wrote:
>>>>     
>>>>>   
>>>>>       
>>>>>> What is your disk image file format, or are you using a partition?
>>>>>>     
>>>>>>         
>>>>> I am using a raw image file on an ext3 partition.
>>>>>
>>>>>   
>>>>>       
>>>>>> Do the results change (on kvm-14) if you pin the guest to a core with 
>>>>>> 'taskset 1 qemu ...'
>>>>>>     
>>>>>>         
>>>>> Bingo. It now works even faster than kvm-13!
>>>>>
>>>>> real    0m22.307s
>>>>> user    0m13.935s
>>>>> sys     0m4.720
>>>>>
>>>>>   
>>>>>       
>>>> I'm guessing this is due to the glibc aio implementation, which uses 
>>>> threads instead of true aio.  The threads may cause the vcpu to migrate 
>>>> frequently from one code to another.
>>>>
>>>> There are two possible solutions:
>>>>
>>>>  - use native aio from http://www.bullopensource.org/posix/.  I think 
>>>> the aio signal patches are not yet in, so this may not work.
>>>>  - teach the scheduler about the cost of migrating vcpus
>>>> The first approach will solve itself eventually, though slowly if the 
>>>> current slow rate of aio merging continues.  We'll have to do the second.
>>>>
>>>>     
>>> if you prefer the first one, Sébastien will release very soon aio patches for
>>> 2.6.20 with an up-to-date libposix-aio.
>>>   
>> Hi Laurent,
>>
>> I gave that a shot a little bit ago.  Ran into two problems.
>>
>> 1) Couldn't avoid linking to -lrt as QEMU uses time functions from it.
>> 2) While I could get things compiling (with patches), QEMU would SEGV 
>> almost immediately.
>>
>> Could you guys maybe give compiling QEMU w/libposix-aio a shot?  I'm 
>> really interested to see if it makes a difference.
>>
>> Regards,
>>
>> Anthony Liguori
>>
>>> [advertising] Keep an eye on the bullopensource website. [/advertising]
>>> :-P
>>>
>>> Regards,
>>> Laurent
> 
> OK, I didn' have time to test the performance of the result, but you can find
> attached some patches to enable libposix-aio with kvm-14.
> 
> first take last patches for linux-2.6.20 and libposix-aio-.0.8.2 from website :
> 
> http://sourceforge.net/projects/paiol
> 
> if you are using AMD64, you must patch libposix-aio because there is a little
> problem remaining to detect lio_submit syscall (first attachment)
> 
> then apply following patch to kvm-14 (second attachment).
> 
> It works fine on my system except when I use "-hda /dev/sdb" : qemu crashes just
> after mounting filesystems when "init" tries to set kernel parameters with
> "sysctl" (I use a debian 4.0). If I boot in emergency mode, mounting manually
> filesystems and running manually sysctl, all works fine. It looks like a
> synchronization problem.
> 
> There are remaining issues : libposix-aio uses kernel AIO, so files must be
> opened using O_DIRECT and buffers must be aligned. libposix-aio is able to
> manage other cases but this has a performance cost.
> 
> Aurélien, do you have any time to test this on your system ?

It took me some time to test it, I was not at home and I prefer to be
near the machine when I change the kernel!

I have just tried your patches, and I can say it works. I get the same
times (minor measurement issues) as when using kvm without your patches
and with taskset. Thanks a lot!

Would it be possible to merge the kvm part, and maybe to send it to the
qemu mailing list?

Regards,
Aurelien

-- 
  .''`.  Aurelien Jarno	            | GPG: 1024D/F1BCDB73
 : :' :  Debian developer           | Electrical Engineer
 `. `'   aurel32-8fiUuRrzOP0dnm+yROfE0A@public.gmane.org         | aurelien-rXXEIb44qovR7s880joybQ@public.gmane.org
   `-    people.debian.org/~aurel32 | www.aurel32.net

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Laurent Vivier]

[-- Attachment #1.1: Type: text/plain, Size: 4167 bytes --]

Aurelien Jarno wrote:
> Laurent Vivier a écrit :
>> Anthony Liguori wrote:
>>> Laurent Vivier wrote:
>>>> Avi Kivity wrote:
>>>>   
>>>>> Aurelien Jarno wrote:
>>>>>     
>>>>>>   
>>>>>>       
>>>>>>> What is your disk image file format, or are you using a partition?
>>>>>>>     
>>>>>>>         
>>>>>> I am using a raw image file on an ext3 partition.
>>>>>>
>>>>>>   
>>>>>>       
>>>>>>> Do the results change (on kvm-14) if you pin the guest to a core with 
>>>>>>> 'taskset 1 qemu ...'
>>>>>>>     
>>>>>>>         
>>>>>> Bingo. It now works even faster than kvm-13!
>>>>>>
>>>>>> real    0m22.307s
>>>>>> user    0m13.935s
>>>>>> sys     0m4.720
>>>>>>
>>>>>>   
>>>>>>       
>>>>> I'm guessing this is due to the glibc aio implementation, which uses 
>>>>> threads instead of true aio.  The threads may cause the vcpu to migrate 
>>>>> frequently from one code to another.
>>>>>
>>>>> There are two possible solutions:
>>>>>
>>>>>  - use native aio from http://www.bullopensource.org/posix/.  I think 
>>>>> the aio signal patches are not yet in, so this may not work.
>>>>>  - teach the scheduler about the cost of migrating vcpus
>>>>> The first approach will solve itself eventually, though slowly if the 
>>>>> current slow rate of aio merging continues.  We'll have to do the second.
>>>>>
>>>>>     
>>>> if you prefer the first one, Sébastien will release very soon aio patches for
>>>> 2.6.20 with an up-to-date libposix-aio.
>>>>   
>>> Hi Laurent,
>>>
>>> I gave that a shot a little bit ago.  Ran into two problems.
>>>
>>> 1) Couldn't avoid linking to -lrt as QEMU uses time functions from it.
>>> 2) While I could get things compiling (with patches), QEMU would SEGV 
>>> almost immediately.
>>>
>>> Could you guys maybe give compiling QEMU w/libposix-aio a shot?  I'm 
>>> really interested to see if it makes a difference.
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>>>
>>>> [advertising] Keep an eye on the bullopensource website. [/advertising]
>>>> :-P
>>>>
>>>> Regards,
>>>> Laurent
>> OK, I didn' have time to test the performance of the result, but you can find
>> attached some patches to enable libposix-aio with kvm-14.
>>
>> first take last patches for linux-2.6.20 and libposix-aio-.0.8.2 from website :
>>
>> http://sourceforge.net/projects/paiol
>>
>> if you are using AMD64, you must patch libposix-aio because there is a little
>> problem remaining to detect lio_submit syscall (first attachment)
>>
>> then apply following patch to kvm-14 (second attachment).
>>
>> It works fine on my system except when I use "-hda /dev/sdb" : qemu crashes just
>> after mounting filesystems when "init" tries to set kernel parameters with
>> "sysctl" (I use a debian 4.0). If I boot in emergency mode, mounting manually
>> filesystems and running manually sysctl, all works fine. It looks like a
>> synchronization problem.
>>
>> There are remaining issues : libposix-aio uses kernel AIO, so files must be
>> opened using O_DIRECT and buffers must be aligned. libposix-aio is able to
>> manage other cases but this has a performance cost.
>>
>> Aurélien, do you have any time to test this on your system ?
> 
> It took me some time to test it, I was not at home and I prefer to be
> near the machine when I change the kernel!

thank you very much to have tested it : I was not able to reproduce your problem
with debian/linux guest on my intel xeon MP.

> I have just tried your patches, and I can say it works. I get the same
> times (minor measurement issues) as when using kvm without your patches
> and with taskset. Thanks a lot!

Good news.

> Would it be possible to merge the kvm part, and maybe to send it to the
> qemu mailing list?

I can try to send it to qemu mailing list but I think it is not useful for pure
qemu. It seems to improve performance only with kvm.

Avi, your opinion ?

> Regards,
> Aurelien

Regards,
Laurent
-- 
------------- Laurent.Vivier-6ktuUTfB/bM@public.gmane.org  --------------
       "Any sufficiently advanced technology is
  indistinguishable from magic." - Arthur C. Clarke


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 345 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: [ANNOUNCE] kvm-14 release

[Avi Kivity]

Laurent Vivier wrote:
>   
>> Would it be possible to merge the kvm part, and maybe to send it to the
>> qemu mailing list?
>>     
>
> I can try to send it to qemu mailing list but I think it is not useful for pure
> qemu. It seems to improve performance only with kvm.
>
> Avi, your opinion ?

As it doesn't involve code changes, I see no reason why it can't be 
merged into qemu.  The case would probably be stronger if distributions 
carried libposix-aio -- do they?

What are the long term plans for libposix-aio?  Do you plan to 
contribute it into glibc?

Also, it probably relies on kernel patches to get signal notification to 
work.  When are these patches going to be merged?


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

Re: [ANNOUNCE] kvm-14 release

[Laurent Vivier]

[-- Attachment #1.1: Type: text/plain, Size: 1341 bytes --]

Avi Kivity wrote:
> Laurent Vivier wrote:
>>  
>>> Would it be possible to merge the kvm part, and maybe to send it to the
>>> qemu mailing list?
>>>     
>>
>> I can try to send it to qemu mailing list but I think it is not useful
>> for pure
>> qemu. It seems to improve performance only with kvm.
>>
>> Avi, your opinion ?
> 
> As it doesn't involve code changes, I see no reason why it can't be
> merged into qemu.  The case would probably be stronger if distributions
> carried libposix-aio -- do they?
> 
> What are the long term plans for libposix-aio?  Do you plan to
> contribute it into glibc?
> 
> Also, it probably relies on kernel patches to get signal notification to
> work.  When are these patches going to be merged?

AIO completion signal has been added to the -mm tree, but support in glibc is
not ready. Ulrich Drepper should work on this, but I think Sébastien already did
this work.

So, libposix-aio should be considered only as experimental until its integration
in glibc.

Sébastien Dugué is the maintainer and main contributor of libposix-aio, he could
correct me if I'm wrong.

Regards,
Laurent
-- 
------------- Laurent.Vivier-6ktuUTfB/bM@public.gmane.org  --------------
       "Any sufficiently advanced technology is
  indistinguishable from magic." - Arthur C. Clarke


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 345 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: [ANNOUNCE] kvm-14 release

[Laurent Vivier]

[-- Attachment #1.1: Type: text/plain, Size: 3921 bytes --]

Anthony Liguori wrote:
> Avi Kivity wrote:
>> Anthony Liguori wrote:
>>> Avi Kivity wrote:
>>>> Laurent Vivier wrote:
>>>>> There are remaining issues : libposix-aio uses kernel AIO, so files 
>>>>> must be
>>>>> opened using O_DIRECT and buffers must be aligned. libposix-aio is 
>>>>> able to
>>>>> manage other cases but this has a performance cost.
>>>>>
>>>>>   
>>>> Because the guest is also doing dma, the buffers are expected to be 
>>>> aligned (it might be different if using pio, but unlikely).
>>> The IDE emulation always uses a temporary buffer.  This is partially 
>>> because you cannot always rely on being able to use an offset on 
>>> phys_ram_base (when doing user only emulation).
>> Why is that?
> 
> Well, user won't do IDE emulation, but cpu_physical_memory_rw has to 
> jump through more hoops for IDE.
> 
> What's more relevant though, is it's possible for the guest to setup DMA 
> to device memory, in which case, you need to route the IO to callbacks.  
> That's pretty unlikely though so I think we can stream line it so that 
> we maintain correctness while also getting less copies.

I'm not sure I had understood correctly your explanation, but I tried to remove
buffer from IDE device. I made a (pseudo-)benchmark with "dd"

Applied patch is here:

http://www.bullopensource.org/posix/qemu/ide-directio.diff

(pseudo-)Benchmark is here:

kvm-14

dd if=/dev/hda of=/dev/null count=100000
51200000 bytes transferred in 4.735706 seconds (10811482 bytes/sec)
dd if=/dev/hda of=/dev/null count=1000000
512000000 bytes transferred in 25.218732 seconds (20302369 bytes/sec)

kvm-14 + libposix-aio

dd if=/dev/hda of=/dev/null count=100000
51200000 bytes transferred in 3.335737 seconds (15348932 bytes/sec)
dd if=/dev/hda of=/dev/null count=1000000
512000000 bytes transferred in 19.242217 seconds (26608161 bytes/sec)

kvm-14 + libposix-aio + ide-directio

dd if=/dev/hda of=/dev/null count=100000
51200000 bytes transferred in 2.695296 seconds (18996058 bytes/sec)
dd if=/dev/hda of=/dev/null count=1000000
512000000 bytes transferred in 14.992513 seconds (34150379 bytes/sec)

qemu-0.9.0

dd if=/dev/hda of=/dev/null count=100000
1200000 bytes transferred in 2.694546 seconds (19001345 bytes/sec)
dd if=/dev/hda of=/dev/null count=1000000
512000000 bytes transferred in 25.524641 seconds (20059048 bytes/sec)

qemu-0.9.0 + libposix-aio

dd if=/dev/hda of=/dev/null count=100000
51200000 bytes transferred in 2.859584 seconds (17904702 bytes/sec)
dd if=/dev/hda of=/dev/null count=1000000
512000000 bytes transferred in 25.045071 seconds (20443144 bytes/sec)

qemu-0.9.0 + libposix-aio + ide-directio

dd if=/dev/hda of=/dev/null count=100000
51200000 bytes transferred in 2.897503 seconds (17670387 bytes/sec)
dd if=/dev/hda of=/dev/null count=1000000
512000000 bytes transferred in 25.900654 seconds (19767841 bytes/sec)

qemu-0.9.0 + kqemu

dd if=/dev/hda of=/dev/null count=100000
51200000 bytes transferred in 6.204987 seconds (8251427 bytes/sec)
dd if=/dev/hda of=/dev/null count=1000000
512000000 bytes transferred in 65.149972 seconds (7858791 bytes/sec)

qemu-0.9.0 + kqemu + libposix-aio

dd if=/dev/hda of=/dev/null count=100000
51200000 bytes transferred in 6.075537 seconds (8427239 bytes/sec)
dd if=/dev/hda of=/dev/null count=1000000
512000000 bytes transferred in 57.506392 seconds (8903358 bytes/sec)

qemu-0.9.0 + kqemu + libposix-aio + ide-directio

dd if=/dev/hda of=/dev/null count=100000
51200000 bytes transferred in 6.083927 seconds (8415617 bytes/sec)
dd if=/dev/hda of=/dev/null count=1000000
512000000 bytes transferred in 57.602499 seconds (8888503 bytes/sec)

Regards,
Laurent
-- 
------------- Laurent.Vivier-6ktuUTfB/bM@public.gmane.org  --------------
       "Any sufficiently advanced technology is
  indistinguishable from magic." - Arthur C. Clarke


[-- Attachment #1.2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 345 bytes --]

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV

[-- Attachment #3: Type: text/plain, Size: 186 bytes --]

_______________________________________________
kvm-devel mailing list
kvm-devel-5NWGOfrQmneRv+LV9MX5uipxlwaOVQ5f@public.gmane.org
https://lists.sourceforge.net/lists/listinfo/kvm-devel

Re: [ANNOUNCE] kvm-14 release

[Avi Kivity]

Avi Kivity wrote:
> James Morris wrote:
>> It's traditional for the mainline kernel to maintain backward 
>> compatibility with existing userspace ABIs (e.g. syscalls).
>>
>> The idea of a userspace application breaking because of a mainline 
>> kernel upgrade is not acceptable to users and also pretty much 
>> unsupportable by distros.
>>   
>
> That is of course very true, and it is a goal of kvm to have a stable 
> userspace interface.  I plan to target 2.6.21 as the stable ABI release.
>

As the alert reader will have guessed, this is now postponed to 2.6.22.  
I will provide a nonintrusive patch backporting the stable ABI to 2.6.21 
if desired.

-- 
Do not meddle in the internals of kernels, for they are subtle and quick to panic.


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV