Re: IO on guest is 20 times slower than host

From: "Kurt Yoder" <ktykvm@yoderhome.com>
To: "Avi Kivity" <avi@redhat.com>
Cc: kvm@vger.kernel.org
Subject: Re: IO on guest is 20 times slower than host
Date: Fri, 27 Mar 2009 09:53:57 -0400 (EDT)	[thread overview]
Message-ID: <52808.140.147.222.3.1238162037.squirrel@dyn.yoderhome.com> (raw)
In-Reply-To: <49CB55F3.5030208@redhat.com>

> Kurt Yoder wrote:
>>
>> I see. I looked up another test: using hdparm -t. It doesn't show the
>> situation as quite so bad, but the guest is still a little over half
>> the speed of the host:
>>
>> me@host:~$ sudo hdparm -t /dev/mapper/HW_RAID-ROOT
>>
>> /dev/mapper/HW_RAID-ROOT:
>>  Timing buffered disk reads:  282 MB in  3.00 seconds =  93.92 MB/sec
>>
>> me@guest:~# hdparm -t /dev/vda
>>
>> /dev/vda:
>>  Timing buffered disk reads:  156 MB in  3.03 seconds =  51.56 MB/sec
>
> This is reasonable.  IDE emulation is not expected to be as fast as the
> host.

Yes, it's not as bad as I originally thought. Half as much though? I'm
also running KVM on a different machine, not even using virtio, and the
same tests show virtually no difference between host and guest IO. I hope
the vmstat pinpoints a fixable problem...

>
>>> Something weird is happening with your system.  If you extend the
>>> test, what does 'top' show?  On both guest and host.
>>
>> If I extend the test thusly on the guest:
>>
>> dd if=/dev/zero of=/bigfile count=10000000
>>
>> I see 100% CPU utilization on the guest, and 100% CPU utilization on
>> one of the host cores.
>
> Can you post 'vmstat 1' snippets on both guest and host while this is
> running?

me@host:~$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0      0 122708608 5179880 1290152    0    0     0     2    2    0  0 
0 100  0
 1  0      0 122708360 5179936 1290120    0    0     0     0 1151  988 10 
1 89  0
 1  0      0 122708352 5179936 1290176    0    0     0     0  714  746  5 
0 95  0
 1  0      0 122708352 5179936 1290176    0    0     0     0  716  746  5 
0 95  0
 1  0      0 122708608 5179936 1290176    0    0     0     0  717  731  9 
0 91  0
 2  0      0 122708608 5179936 1290176    0    0     0     0  714  735  8 
0 92  0
 2  0      0 122708608 5179936 1290176    0    0     0     0  894  815  8 
0 92  0
 1  0      0 122708640 5179992 1290176    0    0     0     0  963  850  5 
0 95  0
 1  0      0 122708640 5179992 1290176    0    0     0     0  721  726  5 
0 95  0
 1  0      0 122708640 5179992 1290176    0    0     0     0  717  739  8 
0 92  0
 1  0      0 122708640 5179992 1290176    0    0     0     0  717  749  5 
0 95  0
 1  0      0 122708640 5179992 1290176    0    0     0     0  714  746  6 
0 94  0
 1  0      0 122708640 5180048 1290176    0    0     0     0  723 1973  8 
0 92  0

me@guest:~# vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 1  0   2616 686488   7756 208004    0    0     2    34   10    3  0  1 99  0
 1  0   2616 680280   7764 213920    0    0     0     0  106    8  7 93  0  0
 1  0   2616 674076   7768 219768    0    0     0     0  104    5  8 92  0  0
 1  0   2616 669000   7780 224864    0    0     0 15124  188   14  1 99  0  0
 1  0   2616 663168   7788 230700    0    0     0     0  104    5  4 96  0  0
 1  0   2616 657836   7792 235472    0    0     0  5684  139   21  5 95  0  0
 1  0   2616 651648   7796 241288    0    0     0     0  104    5  7 93  0  0
 1  0   2616 645428   7804 247136    0    0     0     0  104    5  2 98  0  0
 1  0   2616 640956   7816 251908    0    0     0 22164  232   31  3 97  0  0
 1  0   2616 635016   7820 257732    0    0     0     0  104    5  2 98  0  0
 1  0   2616 629204   7828 263552    0    0     0     0  105    9  2 98  0  0
 1  0   2616 622732   7832 269392    0    0     0     0  104    5  5 95  0  0
 1  0   2616 616676   7840 275096    0    0     0     0  104    7  5 95  0  0
 1  0   2616 612200   7852 279540    0    0     0 27872  263   39  1 99  0  0
 1  0   2616 606384   7856 285204    0    0     0     0  105    7  3 97  0  0
 1  0   2616 600536   7860 290716    0    0     0     0  104   11  7 93  0  0
 1  0   2616 594352   7868 296408    0    0     0     0  104    7  9 91  0  0

> Oh, and try switching to the deadline I/O scheduler on the host instead
> of the default cfq.

It seems it was already using deadline?

me@host:~$ cat /sys/block/cciss\!c0d0/queue/scheduler
noop anticipatory [deadline] cfq

I also checked on the other KVM host that has faster IO on its guests. It
is using software raid, and each individual disk likewise uses deadline
scheduler.

Thanks for your help on this,

-Kurt