IO on guest is 20 times slower than host

IO on guest is 20 times slower than host

[Kurt Yoder]

Hello list,

I'm puzzled as to why my KVM guest has IO that is 20 times slower than  
the KVM host:

me@guest:~$ dd if=/dev/zero of=/tmp/bigfile count=100000
100000+0 records in
100000+0 records out
51200000 bytes (51 MB) copied, 8.36612 s, 6.1 MB/s
me@guest:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/vda1             9.2G  587M  8.2G   7% /
tmpfs                 123M     0  123M   0% /lib/init/rw
varrun                123M   28K  123M   1% /var/run
varlock               123M     0  123M   0% /var/lock
udev                  123M  2.6M  120M   3% /dev
tmpfs                 123M     0  123M   0% /dev/shm



me@host:~$ dd if=/dev/zero of=/tmp/bigfile count=100000
100000+0 records in
100000+0 records out
51200000 bytes (51 MB) copied, 0.47188 s, 109 MB/s



The VM is started via libvirt and is the only VM running on an  
otherwise-unused system:

root     19915     1  0 Mar24 ?        00:00:00   /usr/sbin/libvirtd -d
nobody   19937 19915  0 Mar24 ?        00:00:00     dnsmasq --keep-in- 
foreground --strict-order --bind-interfaces --pid-file  --conf-file  -- 
listen-address 192.168.122.1 --except-interface lo --dhcp-leasefile=/ 
var/lib/libvirt/dhcp-default.leases --dhcp-range  
192.168.122.2,192.168.122.254
root     24354 19915  2 09:52 ?        00:02:33     /usr/bin/kvm -S -M  
pc -m 256 -smp 1 -name guest -monitor pty -boot c -drive file=/dev/ 
HW_RAID/Guest,if=virtio,index=0,boot=on -net  
nic,macaddr=52:54:00:ba:e6:db,vlan=0,model=virtio -net  
tap,fd=11,script=,vlan=0,ifname=vnet1 -serial none -parallel none -usb  
-vnc 127.0.0.1:1



Can someone suggest where to start looking? Thanks,

-Kurt

Re: IO on guest is 20 times slower than host

[Avi Kivity]

Kurt Yoder wrote:
> Hello list,
>
> I'm puzzled as to why my KVM guest has IO that is 20 times slower than 
> the KVM host:
>
> me@guest:~$ dd if=/dev/zero of=/tmp/bigfile count=100000
> 100000+0 records in
> 100000+0 records out
> 51200000 bytes (51 MB) copied, 8.36612 s, 6.1 MB/s
> me@guest:~$ df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/vda1             9.2G  587M  8.2G   7% /
> tmpfs                 123M     0  123M   0% /lib/init/rw
> varrun                123M   28K  123M   1% /var/run
> varlock               123M     0  123M   0% /var/lock
> udev                  123M  2.6M  120M   3% /dev
> tmpfs                 123M     0  123M   0% /dev/shm
>
>
>
> me@host:~$ dd if=/dev/zero of=/tmp/bigfile count=100000
> 100000+0 records in
> 100000+0 records out
> 51200000 bytes (51 MB) copied, 0.47188 s, 109 MB/s
>
>
>
> The VM is started via libvirt and is the only VM running on an 
> otherwise-unused system:
>
> root     19915     1  0 Mar24 ?        00:00:00   /usr/sbin/libvirtd -d
> nobody   19937 19915  0 Mar24 ?        00:00:00     dnsmasq 
> --keep-in-foreground --strict-order --bind-interfaces --pid-file  
> --conf-file  --listen-address 192.168.122.1 --except-interface lo 
> --dhcp-leasefile=/var/lib/libvirt/dhcp-default.leases --dhcp-range 
> 192.168.122.2,192.168.122.254
> root     24354 19915  2 09:52 ?        00:02:33     /usr/bin/kvm -S -M 
> pc -m 256 -smp 1 -name guest -monitor pty -boot c -drive 
> file=/dev/HW_RAID/Guest,if=virtio,index=0,boot=on -net 
> nic,macaddr=52:54:00:ba:e6:db,vlan=0,model=virtio -net 
> tap,fd=11,script=,vlan=0,ifname=vnet1 -serial none -parallel none -usb 
> -vnc 127.0.0.1:1
>
>
>
> Can someone suggest where to start looking? Thanks,

I get 141 MB/s on the same test (it only copies to memory anyway).  Is 
your VM under memory pressure?

-- 
error compiling committee.c: too many arguments to function

Re: IO on guest is 20 times slower than host

[Kurt Yoder]

On Mar 25, 2009, at 12:13 PM, Avi Kivity wrote:

> Kurt Yoder wrote:
>> Hello list,
>>
>> I'm puzzled as to why my KVM guest has IO that is 20 times slower  
>> than the KVM host:
>>
>> me@guest:~$ dd if=/dev/zero of=/tmp/bigfile count=100000
>> 100000+0 records in
>> 100000+0 records out
>> 51200000 bytes (51 MB) copied, 8.36612 s, 6.1 MB/s
>> me@guest:~$ df -h
>> Filesystem            Size  Used Avail Use% Mounted on
>> /dev/vda1             9.2G  587M  8.2G   7% /
>> tmpfs                 123M     0  123M   0% /lib/init/rw
>> varrun                123M   28K  123M   1% /var/run
>> varlock               123M     0  123M   0% /var/lock
>> udev                  123M  2.6M  120M   3% /dev
>> tmpfs                 123M     0  123M   0% /dev/shm
>>
>>
>>
>> me@host:~$ dd if=/dev/zero of=/tmp/bigfile count=100000
>> 100000+0 records in
>> 100000+0 records out
>> 51200000 bytes (51 MB) copied, 0.47188 s, 109 MB/s
>>
>>
>>
>> The VM is started via libvirt and is the only VM running on an  
>> otherwise-unused system:
>>
>> root     19915     1  0 Mar24 ?        00:00:00   /usr/sbin/ 
>> libvirtd -d
>> nobody   19937 19915  0 Mar24 ?        00:00:00     dnsmasq --keep- 
>> in-foreground --strict-order --bind-interfaces --pid-file  --conf- 
>> file  --listen-address 192.168.122.1 --except-interface lo --dhcp- 
>> leasefile=/var/lib/libvirt/dhcp-default.leases --dhcp-range  
>> 192.168.122.2,192.168.122.254
>> root     24354 19915  2 09:52 ?        00:02:33     /usr/bin/kvm -S  
>> -M pc -m 256 -smp 1 -name guest -monitor pty -boot c -drive file=/ 
>> dev/HW_RAID/Guest,if=virtio,index=0,boot=on -net  
>> nic,macaddr=52:54:00:ba:e6:db,vlan=0,model=virtio -net  
>> tap,fd=11,script=,vlan=0,ifname=vnet1 -serial none -parallel none - 
>> usb -vnc 127.0.0.1:1
>>
>>
>>
>> Can someone suggest where to start looking? Thanks,
>
> I get 141 MB/s on the same test (it only copies to memory anyway).   
> Is your VM under memory pressure?

I don't think so. The host has 128 GB of memory, and even the pre- 
emptive kernel caching doesn't come close to filling it:

me@host:~$ free
              total       used       free     shared    buffers      
cached
Mem:     132043960    1742480  130301480          0      84628      
826040
-/+ buffers/cache:     831812  131212148
Swap:      1048568          0    1048568

My /tmp should be physical disk in both cases. To be thorough, I also  
ran the test while writing the output file to /, and got the same  
results (5.7 MB/s on guest, 144 MB/s on host).

-Kurt

Re: IO on guest is 20 times slower than host

[Avi Kivity]

Kurt Yoder wrote:
>>
>> I get 141 MB/s on the same test (it only copies to memory anyway).  
>> Is your VM under memory pressure?
>
> I don't think so. The host has 128 GB of memory, and even the 
> pre-emptive kernel caching doesn't come close to filling it:
>
> me@host:~$ free
>              total       used       free     shared    buffers     cached
> Mem:     132043960    1742480  130301480          0      84628     826040
> -/+ buffers/cache:     831812  131212148
> Swap:      1048568          0    1048568
>
> My /tmp should be physical disk in both cases. To be thorough, I also 
> ran the test while writing the output file to /, and got the same 
> results (5.7 MB/s on guest, 144 MB/s on host).

Your guest is only assigned 256MB though.

-- 
error compiling committee.c: too many arguments to function

Re: IO on guest is 20 times slower than host

[Kurt Yoder]

On Mar 25, 2009, at 12:35 PM, Avi Kivity wrote:

> Kurt Yoder wrote:
>>>
>>> I get 141 MB/s on the same test (it only copies to memory  
>>> anyway).  Is your VM under memory pressure?
>>
>> I don't think so. The host has 128 GB of memory, and even the pre- 
>> emptive kernel caching doesn't come close to filling it:
>>
>> me@host:~$ free
>>             total       used       free     shared    buffers      
>> cached
>> Mem:     132043960    1742480  130301480          0      84628      
>> 826040
>> -/+ buffers/cache:     831812  131212148
>> Swap:      1048568          0    1048568
>>
>> My /tmp should be physical disk in both cases. To be thorough, I  
>> also ran the test while writing the output file to /, and got the  
>> same results (5.7 MB/s on guest, 144 MB/s on host).
>
> Your guest is only assigned 256MB though.


I upped the memory on the VM to 1 GB and ran the same test. It  
actually got *slower*:

me@guest:~$ free
              total       used       free     shared    buffers      
cached
Mem:       1024224     131324     892900          0       1816       
18328
-/+ buffers/cache:     111180     913044
Swap:       488248          0     488248
me@guest:~$ sudo -s
me@guest:~# dd if=/dev/zero of=/bigfile count=100000
100000+0 records in
100000+0 records out
51200000 bytes (51 MB) copied, 22.2141 s, 2.3 MB/s

I ran this three times to make sure it wasn't a fluke, and got those  
same rates each time. The guest is now running thus:

root     19915     1  0 Mar24 ?        00:00:00   /usr/sbin/libvirtd -d
nobody   19937 19915  0 Mar24 ?        00:00:00     dnsmasq --keep-in- 
foreground --strict-order --bind-interfaces --pid-file  --conf-file  -- 
listen-address 192.168.122.1 --except-interface lo --dhcp-leasefile=/ 
var/lib/libvirt/dhcp-default.leases --dhcp-range  
192.168.122.2,192.168.122.254
root     25051 19915 54 12:40 ?        00:05:05     /usr/bin/kvm -S -M  
pc -m 1024 -smp 1 -name guest -monitor pty -boot c -drive file=/dev/ 
HW_RAID/Guest,if=virtio,index=0,boot=on -net  
nic,macaddr=52:54:00:ba:e6:db,vlan=0,model=virtio -net  
tap,fd=11,script=,vlan=0,ifname=vnet1 -serial none -parallel none -usb  
-vnc 127.0.0.1:1

Re: IO on guest is 20 times slower than host

[Avi Kivity]

Kurt Yoder wrote:
>
> I upped the memory on the VM to 1 GB and ran the same test. It 
> actually got *slower*:
>
> me@guest:~$ free
>              total       used       free     shared    buffers     cached
> Mem:       1024224     131324     892900          0       1816      18328
> -/+ buffers/cache:     111180     913044
> Swap:       488248          0     488248
> me@guest:~$ sudo -s
> me@guest:~# dd if=/dev/zero of=/bigfile count=100000
> 100000+0 records in
> 100000+0 records out
> 51200000 bytes (51 MB) copied, 22.2141 s, 2.3 MB/s
>


Wierd.  Can you change the disk interface to ide to rule out virtio issues?

(though dd should be running entirely in cache)

Is your disk a raw volume or qcow file?

-- 
error compiling committee.c: too many arguments to function

Re: IO on guest is 20 times slower than host

[Kurt Yoder]

On Mar 25, 2009, at 12:54 PM, Avi Kivity wrote:

> Kurt Yoder wrote:
>>
>> I upped the memory on the VM to 1 GB and ran the same test. It  
>> actually got *slower*:
>>
>> me@guest:~$ free
>>             total       used       free     shared    buffers      
>> cached
>> Mem:       1024224     131324     892900          0       1816       
>> 18328
>> -/+ buffers/cache:     111180     913044
>> Swap:       488248          0     488248
>> me@guest:~$ sudo -s
>> me@guest:~# dd if=/dev/zero of=/bigfile count=100000
>> 100000+0 records in
>> 100000+0 records out
>> 51200000 bytes (51 MB) copied, 22.2141 s, 2.3 MB/s
>>
>
>
> Wierd.  Can you change the disk interface to ide to rule out virtio  
> issues?
>
> (though dd should be running entirely in cache)
>
> Is your disk a raw volume or qcow file?



It was actually set as an ide disk before it was a virtio disk. I  
switched it to virtio to try to get rid of this problem, but no luck  
so far.

The disk lives on an LVM logical volume.

What do you mean about the cache? Is my test fundamentally flawed? I  
*thought* I was testing write speed on the disk...

-Kurt

Re: IO on guest is 20 times slower than host

[Avi Kivity]

Kurt Yoder wrote:
>
> What do you mean about the cache? Is my test fundamentally flawed? I 
> *thought* I was testing write speed on the disk...

'dd', without further arguments, will write to the page cache and let 
the kernel write the data back at a later time.  If you increase the 
block size (bs=1M count=1000) you should see much faster times on the 
host, I wouldn't be surprised to see 1GB/s.

If  you want to test disk speed, use of=/dev/blah oflag=direct.  Beware 
of destroying your data disk.

Something weird is happening with your system.  If you extend the test, 
what does 'top' show?  On both guest and host.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: IO on guest is 20 times slower than host

[Kurt Yoder]

On Mar 25, 2009, at 1:55 PM, Avi Kivity wrote:

> Kurt Yoder wrote:
>>
>> What do you mean about the cache? Is my test fundamentally flawed?  
>> I *thought* I was testing write speed on the disk...
>
> 'dd', without further arguments, will write to the page cache and  
> let the kernel write the data back at a later time.  If you increase  
> the block size (bs=1M count=1000) you should see much faster times  
> on the host, I wouldn't be surprised to see 1GB/s.
>
> If  you want to test disk speed, use of=/dev/blah oflag=direct.   
> Beware of destroying your data disk.
>

I see. I looked up another test: using hdparm -t. It doesn't show the  
situation as quite so bad, but the guest is still a little over half  
the speed of the host:

me@host:~$ sudo hdparm -t /dev/mapper/HW_RAID-ROOT

/dev/mapper/HW_RAID-ROOT:
  Timing buffered disk reads:  282 MB in  3.00 seconds =  93.92 MB/sec

me@guest:~# hdparm -t /dev/vda

/dev/vda:
  Timing buffered disk reads:  156 MB in  3.03 seconds =  51.56 MB/sec



> Something weird is happening with your system.  If you extend the  
> test, what does 'top' show?  On both guest and host.

If I extend the test thusly on the guest:

dd if=/dev/zero of=/bigfile count=10000000

I see 100% CPU utilization on the guest, and 100% CPU utilization on  
one of the host cores.

Re: IO on guest is 20 times slower than host

[Avi Kivity]

Kurt Yoder wrote:
>
> I see. I looked up another test: using hdparm -t. It doesn't show the 
> situation as quite so bad, but the guest is still a little over half 
> the speed of the host:
>
> me@host:~$ sudo hdparm -t /dev/mapper/HW_RAID-ROOT
>
> /dev/mapper/HW_RAID-ROOT:
>  Timing buffered disk reads:  282 MB in  3.00 seconds =  93.92 MB/sec
>
> me@guest:~# hdparm -t /dev/vda
>
> /dev/vda:
>  Timing buffered disk reads:  156 MB in  3.03 seconds =  51.56 MB/sec

This is reasonable.  IDE emulation is not expected to be as fast as the 
host.

>> Something weird is happening with your system.  If you extend the 
>> test, what does 'top' show?  On both guest and host.
>
> If I extend the test thusly on the guest:
>
> dd if=/dev/zero of=/bigfile count=10000000
>
> I see 100% CPU utilization on the guest, and 100% CPU utilization on 
> one of the host cores.

Can you post 'vmstat 1' snippets on both guest and host while this is 
running?

Oh, and try switching to the deadline I/O scheduler on the host instead 
of the default cfq.


-- 
error compiling committee.c: too many arguments to function

Re: IO on guest is 20 times slower than host

[Kurt Yoder]

> Kurt Yoder wrote:
>>
>> I see. I looked up another test: using hdparm -t. It doesn't show the
>> situation as quite so bad, but the guest is still a little over half
>> the speed of the host:
>>
>> me@host:~$ sudo hdparm -t /dev/mapper/HW_RAID-ROOT
>>
>> /dev/mapper/HW_RAID-ROOT:
>>  Timing buffered disk reads:  282 MB in  3.00 seconds =  93.92 MB/sec
>>
>> me@guest:~# hdparm -t /dev/vda
>>
>> /dev/vda:
>>  Timing buffered disk reads:  156 MB in  3.03 seconds =  51.56 MB/sec
>
> This is reasonable.  IDE emulation is not expected to be as fast as the
> host.

Yes, it's not as bad as I originally thought. Half as much though? I'm
also running KVM on a different machine, not even using virtio, and the
same tests show virtually no difference between host and guest IO. I hope
the vmstat pinpoints a fixable problem...






>
>>> Something weird is happening with your system.  If you extend the
>>> test, what does 'top' show?  On both guest and host.
>>
>> If I extend the test thusly on the guest:
>>
>> dd if=/dev/zero of=/bigfile count=10000000
>>
>> I see 100% CPU utilization on the guest, and 100% CPU utilization on
>> one of the host cores.
>
> Can you post 'vmstat 1' snippets on both guest and host while this is
> running?

me@host:~$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0 122708608 5179880 1290152    0    0     0     2    2    0  0 
0 100  0
 1  0      0 122708360 5179936 1290120    0    0     0     0 1151  988 10 
1 89  0
 1  0      0 122708352 5179936 1290176    0    0     0     0  714  746  5 
0 95  0
 1  0      0 122708352 5179936 1290176    0    0     0     0  716  746  5 
0 95  0
 1  0      0 122708608 5179936 1290176    0    0     0     0  717  731  9 
0 91  0
 2  0      0 122708608 5179936 1290176    0    0     0     0  714  735  8 
0 92  0
 2  0      0 122708608 5179936 1290176    0    0     0     0  894  815  8 
0 92  0
 1  0      0 122708640 5179992 1290176    0    0     0     0  963  850  5 
0 95  0
 1  0      0 122708640 5179992 1290176    0    0     0     0  721  726  5 
0 95  0
 1  0      0 122708640 5179992 1290176    0    0     0     0  717  739  8 
0 92  0
 1  0      0 122708640 5179992 1290176    0    0     0     0  717  749  5 
0 95  0
 1  0      0 122708640 5179992 1290176    0    0     0     0  714  746  6 
0 94  0
 1  0      0 122708640 5180048 1290176    0    0     0     0  723 1973  8 
0 92  0

me@guest:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0   2616 686488   7756 208004    0    0     2    34   10    3  0  1 99  0
 1  0   2616 680280   7764 213920    0    0     0     0  106    8  7 93  0  0
 1  0   2616 674076   7768 219768    0    0     0     0  104    5  8 92  0  0
 1  0   2616 669000   7780 224864    0    0     0 15124  188   14  1 99  0  0
 1  0   2616 663168   7788 230700    0    0     0     0  104    5  4 96  0  0
 1  0   2616 657836   7792 235472    0    0     0  5684  139   21  5 95  0  0
 1  0   2616 651648   7796 241288    0    0     0     0  104    5  7 93  0  0
 1  0   2616 645428   7804 247136    0    0     0     0  104    5  2 98  0  0
 1  0   2616 640956   7816 251908    0    0     0 22164  232   31  3 97  0  0
 1  0   2616 635016   7820 257732    0    0     0     0  104    5  2 98  0  0
 1  0   2616 629204   7828 263552    0    0     0     0  105    9  2 98  0  0
 1  0   2616 622732   7832 269392    0    0     0     0  104    5  5 95  0  0
 1  0   2616 616676   7840 275096    0    0     0     0  104    7  5 95  0  0
 1  0   2616 612200   7852 279540    0    0     0 27872  263   39  1 99  0  0
 1  0   2616 606384   7856 285204    0    0     0     0  105    7  3 97  0  0
 1  0   2616 600536   7860 290716    0    0     0     0  104   11  7 93  0  0
 1  0   2616 594352   7868 296408    0    0     0     0  104    7  9 91  0  0






> Oh, and try switching to the deadline I/O scheduler on the host instead
> of the default cfq.

It seems it was already using deadline?

me@host:~$ cat /sys/block/cciss\!c0d0/queue/scheduler
noop anticipatory [deadline] cfq

I also checked on the other KVM host that has faster IO on its guests. It
is using software raid, and each individual disk likewise uses deadline
scheduler.





Thanks for your help on this,

-Kurt

Re: IO on guest is 20 times slower than host

[Kurt Yoder]

>> Kurt Yoder wrote:

<snip>

> me@guest:~# vmstat 1
> procs -----------memory---------- ---swap-- -----io---- -system--
> ----cpu----
>  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id
> wa
>  1  0   2616 686488   7756 208004    0    0     2    34   10    3  0  1 99
>  0
>  1  0   2616 680280   7764 213920    0    0     0     0  106    8  7 93  0
>  0
>  1  0   2616 674076   7768 219768    0    0     0     0  104    5  8 92  0
>  0
>  1  0   2616 669000   7780 224864    0    0     0 15124  188   14  1 99  0
>  0
>  1  0   2616 663168   7788 230700    0    0     0     0  104    5  4 96  0
>  0
>  1  0   2616 657836   7792 235472    0    0     0  5684  139   21  5 95  0
>  0
>  1  0   2616 651648   7796 241288    0    0     0     0  104    5  7 93  0
>  0
>  1  0   2616 645428   7804 247136    0    0     0     0  104    5  2 98  0
>  0
>  1  0   2616 640956   7816 251908    0    0     0 22164  232   31  3 97  0
>  0
>  1  0   2616 635016   7820 257732    0    0     0     0  104    5  2 98  0
>  0
>  1  0   2616 629204   7828 263552    0    0     0     0  105    9  2 98  0
>  0
>  1  0   2616 622732   7832 269392    0    0     0     0  104    5  5 95  0
>  0
>  1  0   2616 616676   7840 275096    0    0     0     0  104    7  5 95  0
>  0
>  1  0   2616 612200   7852 279540    0    0     0 27872  263   39  1 99  0
>  0
>  1  0   2616 606384   7856 285204    0    0     0     0  105    7  3 97  0
>  0
>  1  0   2616 600536   7860 290716    0    0     0     0  104   11  7 93  0
>  0
>  1  0   2616 594352   7868 296408    0    0     0     0  104    7  9 91  0
>  0


BTW, for comparison I did the same test on the other KVM host/guest, and
here is the vmstat -1:

me@otherhost:/etc/apache2# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 3  0  55636   4520    956  37504    0    0    12    12   26   19  0  0 99  0
 3  0  55636   3332    620  38516    0    0     0 81880  537  355  4 92  0  4
 2  1  55636   3232    684  38508    0    0     8 56660 1708  550  2 97  0  1
 2  1  55636   3200    764  38440    0    0     0 87696  409  271  4 96  0  0
 2  1  55636   3096    852  38532    0    0     4 82824  394  254  4 95  0  1
 3  1  55636   3072    848  38776    0    0    12 83520  429  293  8 91  0  1
 3  1  55636   3204    536  38644    0    0    12 64856 1313  464  4 94  0  2
 3  1  55636   3088    620  38484  192    0   280 85632  467  314  5 94  0  1
 2  1  55636   3180    684  38428    0    0     0 67368 1011  443  3 97  0  0
 2  1  55636   3080    768  38456  148    0   220 81432  414  270  8 91  0  1
 3  0  55636   3124    784  38952    0    0    16 80416  554  284  5 94  0  1
 3  1  55636   3004    488  38968    0    0     4 57756 1195  439  4 96  0  0
 1  0  55636   4208    556  37644    0    0     4 75680  545  336  6 92  0  2
 1  1  55636   3296    644  38388    0    0     0 87024  299  339  3 97  0  0
 2  1  55636   2992    724  38648    0    0     4 78792  293  341  4 92  0  4
 3  1  55636   3164    804  38660    0    0     8 73648  462  369  4 83  0 13
 2  0  55636   2988    492  39044    0    0     8 73168  492  424  3 86  0 11


So clearly there is something odd going on with the first one. It looks
like it's CPU-bound, and this is throttling the IO?


-Kurt

Re: IO on guest is 20 times slower than host

[Kurt Yoder]

sorry to keep replying to myself:

>>> Kurt Yoder wrote:

<snip>

>
>
> So clearly there is something odd going on with the first one. It looks
> like it's CPU-bound, and this is throttling the IO?


slow guest cpu information:

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 2
model name      : QEMU Virtual CPU version 0.9.1
stepping        : 3
cpu MHz         : 2612.149
cache size      : 512 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm 3dnowext 3dnow up
nopl pni svm
bogomips        : 5224.29
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

slow host cpu information, core 1 of 16:

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 16
model           : 4
model name      : Quad-Core AMD Opteron(tm) Processor 8382
stepping        : 2
cpu MHz         : 2611.998
cache size      : 512 KB
physical id     : 0
siblings        : 4
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 5
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall mmxext fxsr_opt
pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl pni monitor
cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
misalignsse 3dnowprefetch osvw ibs skinit wdt
bogomips        : 5223.97
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm stc 100mhzsteps hwpstate



fast guest cpu information:

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 2
model name      : QEMU Virtual CPU version 0.9.1
stepping        : 3
cpu MHz         : 2792.466
cache size      : 512 KB
fpu             : yes
fpu_exception   : yes
cpuid level     : 2
wp              : yes
flags           : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx lm up nopl pni
bogomips        : 5584.93
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management:

fast host cpu information, core 1 of 16:

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 15
model           : 65
model name      : Dual-Core AMD Opteron(tm) Processor 8220
stepping        : 3
cpu MHz         : 2792.573
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 4
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt
rdtscp lm 3dnowext 3dnow rep_good nopl pni cx16 lahf_lm cmp_legacy svm
extapic cr8_legacy
bogomips        : 5585.14
TLB size        : 1024 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc



-Kurt

Re: IO on guest is 20 times slower than host

[Avi Kivity]

Kurt Yoder wrote:
> slow host cpu information, core 1 of 16:
>
> processor       : 0
> vendor_id       : AuthenticAMD
> cpu family      : 16
> model           : 4
> model name      : Quad-Core AMD Opteron(tm) Processor 8382
> stepping        : 2
> cpu MHz         : 2611.998
> cache size      : 512 KB
> physical id     : 0
> siblings        : 4
> core id         : 0
> cpu cores       : 4
> apicid          : 0
> initial apicid  : 0
> fpu             : yes
> fpu_exception   : yes
> cpuid level     : 5
> wp              : yes
> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca
> cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall mmxext fxsr_opt
> pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl pni monitor
> cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
> misalignsse 3dnowprefetch osvw ibs skinit wdt
> bogomips        : 5223.97
> TLB size        : 1024 4K pages
> clflush size    : 64
> cache_alignment : 64
> address sizes   : 48 bits physical, 48 bits virtual
> power management: ts ttp tm stc 100mhzsteps hwpstate
>
>
>   

Can you loading kvm_amd on this host with 'modprobe kvm-amd npt=0'?

-- 
error compiling committee.c: too many arguments to function

Re: IO on guest is 20 times slower than host

[Avi Kivity]

[-- Attachment #1: Type: text/plain, Size: 1409 bytes --]

Avi Kivity wrote:
> Kurt Yoder wrote:
>> slow host cpu information, core 1 of 16:
>>
>> processor       : 0
>> vendor_id       : AuthenticAMD
>> cpu family      : 16
>> model           : 4
>> model name      : Quad-Core AMD Opteron(tm) Processor 8382
>> stepping        : 2
>> cpu MHz         : 2611.998
>> cache size      : 512 KB
>> physical id     : 0
>> siblings        : 4
>> core id         : 0
>> cpu cores       : 4
>> apicid          : 0
>> initial apicid  : 0
>> fpu             : yes
>> fpu_exception   : yes
>> cpuid level     : 5
>> wp              : yes
>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr 
>> pge mca
>> cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall mmxext fxsr_opt
>> pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl pni monitor
>> cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
>> misalignsse 3dnowprefetch osvw ibs skinit wdt
>> bogomips        : 5223.97
>> TLB size        : 1024 4K pages
>> clflush size    : 64
>> cache_alignment : 64
>> address sizes   : 48 bits physical, 48 bits virtual
>> power management: ts ttp tm stc 100mhzsteps hwpstate
>>
>>
>>   
>
> Can you loading kvm_amd on this host with 'modprobe kvm-amd npt=0'?
>

If it helps, then the guest is messing up the cpu cache.  Try the 
attached patch.

-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.


[-- Attachment #2: svm-pat-writeback.patch --]
[-- Type: text/x-patch, Size: 507 bytes --]

diff --git a/kernel/x86/kvm/svm.c b/kernel/x86/kvm/svm.c
index 1fcbc17..d9774e9 100644
--- a/kernel/x86/kvm/svm.c
+++ b/kernel/x86/kvm/svm.c
@@ -575,7 +575,7 @@ static void init_vmcb(struct vcpu_svm *svm)
 						INTERCEPT_CR3_MASK);
 		control->intercept_cr_write &= ~(INTERCEPT_CR0_MASK|
 						 INTERCEPT_CR3_MASK);
-		save->g_pat = 0x0007040600070406ULL;
+		save->g_pat = 0x0606060606060606ULL;
 		/* enable caching because the QEMU Bios doesn't enable it */
 		save->cr0 = X86_CR0_ET;
 		save->cr3 = 0;

Re: IO on guest is 20 times slower than host

[Joerg Roedel]

On Sun, Mar 29, 2009 at 10:10:26PM +0300, Avi Kivity wrote:
> Avi Kivity wrote:
>> Kurt Yoder wrote:
>>> slow host cpu information, core 1 of 16:
>>>
>>> processor       : 0
>>> vendor_id       : AuthenticAMD
>>> cpu family      : 16
>>> model           : 4
>>> model name      : Quad-Core AMD Opteron(tm) Processor 8382
>>> stepping        : 2
>>> cpu MHz         : 2611.998
>>> cache size      : 512 KB
>>> physical id     : 0
>>> siblings        : 4
>>> core id         : 0
>>> cpu cores       : 4
>>> apicid          : 0
>>> initial apicid  : 0
>>> fpu             : yes
>>> fpu_exception   : yes
>>> cpuid level     : 5
>>> wp              : yes
>>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr  
>>> pge mca
>>> cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall mmxext fxsr_opt
>>> pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl pni monitor
>>> cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
>>> misalignsse 3dnowprefetch osvw ibs skinit wdt
>>> bogomips        : 5223.97
>>> TLB size        : 1024 4K pages
>>> clflush size    : 64
>>> cache_alignment : 64
>>> address sizes   : 48 bits physical, 48 bits virtual
>>> power management: ts ttp tm stc 100mhzsteps hwpstate
>>>
>>>
>>>   
>>
>> Can you loading kvm_amd on this host with 'modprobe kvm-amd npt=0'?
>>
>
> If it helps, then the guest is messing up the cpu cache.  Try the  
> attached patch.
>
> -- 
> I have a truly marvellous patch that fixes the bug which this
> signature is too narrow to contain.
>

> diff --git a/kernel/x86/kvm/svm.c b/kernel/x86/kvm/svm.c
> index 1fcbc17..d9774e9 100644
> --- a/kernel/x86/kvm/svm.c
> +++ b/kernel/x86/kvm/svm.c
> @@ -575,7 +575,7 @@ static void init_vmcb(struct vcpu_svm *svm)
>  						INTERCEPT_CR3_MASK);
>  		control->intercept_cr_write &= ~(INTERCEPT_CR0_MASK|
>  						 INTERCEPT_CR3_MASK);
> -		save->g_pat = 0x0007040600070406ULL;
> +		save->g_pat = 0x0606060606060606ULL;
>  		/* enable caching because the QEMU Bios doesn't enable it */
>  		save->cr0 = X86_CR0_ET;
>  		save->cr3 = 0;

Yeah, that patch makes sense. But I think we need some more work on this
because the guest may change the pat msr afterwards. Best would be a simple
shadow of the pat msr. Last question is how this will effect pci passthrough.

	Joerg


-- 
           | Advanced Micro Devices GmbH
 Operating | Karl-Hammerschmidt-Str. 34, 85609 Dornach bei München
 System    | 
 Research  | Geschäftsführer: Jochen Polster, Thomas M. McCoy, Giuliano Meroni
 Center    | Sitz: Dornach, Gemeinde Aschheim, Landkreis München
           | Registergericht München, HRB Nr. 43632

Re: IO on guest is 20 times slower than host

[Avi Kivity]

Joerg Roedel wrote:
>> --- a/kernel/x86/kvm/svm.c
>> +++ b/kernel/x86/kvm/svm.c
>> @@ -575,7 +575,7 @@ static void init_vmcb(struct vcpu_svm *svm)
>>  						INTERCEPT_CR3_MASK);
>>  		control->intercept_cr_write &= ~(INTERCEPT_CR0_MASK|
>>  						 INTERCEPT_CR3_MASK);
>> -		save->g_pat = 0x0007040600070406ULL;
>> +		save->g_pat = 0x0606060606060606ULL;
>>  		/* enable caching because the QEMU Bios doesn't enable it */
>>  		save->cr0 = X86_CR0_ET;
>>  		save->cr3 = 0;
>>     
>
> Yeah, that patch makes sense. But I think we need some more work on this
> because the guest may change the pat msr afterwards. Best would be a simple
> shadow of the pat msr. Last question is how this will effect pci passthrough.	

This is just a stopgap; we can later add proper pat shadowing.

-- 
error compiling committee.c: too many arguments to function

Re: IO on guest is 20 times slower than host

[Avi Kivity]

Joerg Roedel wrote:
>> index 1fcbc17..d9774e9 100644
>> --- a/kernel/x86/kvm/svm.c
>> +++ b/kernel/x86/kvm/svm.c
>> @@ -575,7 +575,7 @@ static void init_vmcb(struct vcpu_svm *svm)
>>  						INTERCEPT_CR3_MASK);
>>  		control->intercept_cr_write &= ~(INTERCEPT_CR0_MASK|
>>  						 INTERCEPT_CR3_MASK);
>> -		save->g_pat = 0x0007040600070406ULL;
>> +		save->g_pat = 0x0606060606060606ULL;
>>  		/* enable caching because the QEMU Bios doesn't enable it */
>>  		save->cr0 = X86_CR0_ET;
>>  		save->cr3 = 0;
>>     
>
> Yeah, that patch makes sense. But I think we need some more work on this
> because the guest may change the pat msr afterwards. Best would be a simple
> shadow of the pat msr. Last question is how this will effect pci passthrough.
>
>   

I've noticed that Windows (and likely Linux, didn't test) maps the 
cirrus framebuffer with PWT=1, which should slow down the emulated 
framebuffer.  So this patch should speed up things.

If a device is assigned, we must respect the guest PAT, so cirrus 
performance will be low.  On Intel there's an 'ignore PAT' bit which can 
be set on an ept pte for the framebuffer.  Any trick we can do on AMD to 
achieve a similar result?


-- 
I have a truly marvellous patch that fixes the bug which this
signature is too narrow to contain.

Re: IO on guest is 20 times slower than host

[Kurt Yoder]

On Mar 29, 2009, at 10:29 AM, Avi Kivity wrote:

> Kurt Yoder wrote:
>> slow host cpu information, core 1 of 16:
>>
>> processor       : 0
>> vendor_id       : AuthenticAMD
>> cpu family      : 16
>> model           : 4
>> model name      : Quad-Core AMD Opteron(tm) Processor 8382
>> stepping        : 2
>> cpu MHz         : 2611.998
>> cache size      : 512 KB
>> physical id     : 0
>> siblings        : 4
>> core id         : 0
>> cpu cores       : 4
>> apicid          : 0
>> initial apicid  : 0
>> fpu             : yes
>> fpu_exception   : yes
>> cpuid level     : 5
>> wp              : yes
>> flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr  
>> pge mca
>> cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall mmxext fxsr_opt
>> pdpe1gb rdtscp lm 3dnowext 3dnow constant_tsc rep_good nopl pni  
>> monitor
>> cx16 popcnt lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a
>> misalignsse 3dnowprefetch osvw ibs skinit wdt
>> bogomips        : 5223.97
>> TLB size        : 1024 4K pages
>> clflush size    : 64
>> cache_alignment : 64
>> address sizes   : 48 bits physical, 48 bits virtual
>> power management: ts ttp tm stc 100mhzsteps hwpstate
>>
>>
>>
>
> Can you loading kvm_amd on this host with 'modprobe kvm-amd npt=0'?

So that's most likely the problem for me:

me@host:/etc/nagios/nrpe_directives$ sudo modprobe kvm-amd npt=0
FATAL: Error inserting kvm_amd (/lib/modules/2.6.27-11-server/kernel/ 
arch/x86/kvm/kvm-amd.ko): Operation not supported
me@host:/etc/nagios/nrpe_directives$ uname -a
Linux boron 2.6.27-11-server #1 SMP Thu Jan 29 20:13:12 UTC 2009  
x86_64 GNU/Linux


It looks like I need to enable SVM in my BIOS. I'll do that and report  
back on the results.

-Kurt

Re: IO on guest is 20 times slower than host

[Kurt Yoder]

>
> On Mar 29, 2009, at 10:29 AM, Avi Kivity wrote:
>
>> Kurt Yoder wrote:

<snip>

>>
>> Can you loading kvm_amd on this host with 'modprobe kvm-amd npt=0'?
>
> So that's most likely the problem for me:
>
> me@host:/etc/nagios/nrpe_directives$ sudo modprobe kvm-amd npt=0
> FATAL: Error inserting kvm_amd (/lib/modules/2.6.27-11-server/kernel/
> arch/x86/kvm/kvm-amd.ko): Operation not supported
> me@host:/etc/nagios/nrpe_directives$ uname -a
> Linux boron 2.6.27-11-server #1 SMP Thu Jan 29 20:13:12 UTC 2009
> x86_64 GNU/Linux
>
>
> It looks like I need to enable SVM in my BIOS. I'll do that and report
> back on the results.




The AMD virtualization option was disabled in my BIOS. Once I enabled it,
all my problems disappeared:

me@guest:~$ sudo hdparm -t /dev/sda

/dev/sda:
 Timing buffered disk reads:  846 MB in  3.00 seconds = 281.73 MB/sec
me@guest:~$ sudo dd if=/dev/zero of=/tmp/bigfile count=1000000
1000000+0 records in
1000000+0 records out
512000000 bytes (512 MB) copied, 3.84358 s, 133 MB/s


I do recall seeing a warning at boot about an error loading kvm-amd, but
it was not onscreen for long. It was something like "error loading module
kvm-amd"; looking back now, I see in the syslog "kvm: disabled by bios".
Perhaps a warning about slow IO should be issued/logged on AMD hosts every
time KVM is brought up without the benefit of the kvm-amd kernel module?
Anyway, I'm happy now.



Thanks Avi for your help.