From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Kurt Yoder" Subject: Re: IO on guest is 20 times slower than host Date: Fri, 27 Mar 2009 09:53:57 -0400 (EDT) Message-ID: <52808.140.147.222.3.1238162037.squirrel@dyn.yoderhome.com> References: <49CA5825.7030201@redhat.com> <4AC00930-1EBB-4704-94CF-29478D07F03A@yoderhome.com> <49CA5D53.1080401@redhat.com> <49CA61E2.3050400@redhat.com> <49CA6FFC.2030807@redhat.com> <49CB55F3.5030208@redhat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7BIT Cc: kvm@vger.kernel.org To: "Avi Kivity" Return-path: Received: from vms173001pub.verizon.net ([206.46.173.1]:37234 "EHLO vms173001pub.verizon.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752157AbZC0Ny0 (ORCPT ); Fri, 27 Mar 2009 09:54:26 -0400 Received: from yoderhome.com ([173.66.247.170]) by vms173001.mailsrvcs.net (Sun Java(tm) System Messaging Server 6.3-7.04 (built Sep 26 2008; 32bit)) with ESMTPA id <0KH600H9O3XXF3RE@vms173001.mailsrvcs.net> for kvm@vger.kernel.org; Fri, 27 Mar 2009 08:54:02 -0500 (CDT) In-reply-to: <49CB55F3.5030208@redhat.com> Sender: kvm-owner@vger.kernel.org List-ID: > Kurt Yoder wrote: >> >> I see. I looked up another test: using hdparm -t. It doesn't show the >> situation as quite so bad, but the guest is still a little over half >> the speed of the host: >> >> me@host:~$ sudo hdparm -t /dev/mapper/HW_RAID-ROOT >> >> /dev/mapper/HW_RAID-ROOT: >> Timing buffered disk reads: 282 MB in 3.00 seconds = 93.92 MB/sec >> >> me@guest:~# hdparm -t /dev/vda >> >> /dev/vda: >> Timing buffered disk reads: 156 MB in 3.03 seconds = 51.56 MB/sec > > This is reasonable. IDE emulation is not expected to be as fast as the > host. Yes, it's not as bad as I originally thought. Half as much though? I'm also running KVM on a different machine, not even using virtio, and the same tests show virtually no difference between host and guest IO. I hope the vmstat pinpoints a fixable problem... > >>> Something weird is happening with your system. If you extend the >>> test, what does 'top' show? On both guest and host. >> >> If I extend the test thusly on the guest: >> >> dd if=/dev/zero of=/bigfile count=10000000 >> >> I see 100% CPU utilization on the guest, and 100% CPU utilization on >> one of the host cores. > > Can you post 'vmstat 1' snippets on both guest and host while this is > running? me@host:~$ vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 0 122708608 5179880 1290152 0 0 0 2 2 0 0 0 100 0 1 0 0 122708360 5179936 1290120 0 0 0 0 1151 988 10 1 89 0 1 0 0 122708352 5179936 1290176 0 0 0 0 714 746 5 0 95 0 1 0 0 122708352 5179936 1290176 0 0 0 0 716 746 5 0 95 0 1 0 0 122708608 5179936 1290176 0 0 0 0 717 731 9 0 91 0 2 0 0 122708608 5179936 1290176 0 0 0 0 714 735 8 0 92 0 2 0 0 122708608 5179936 1290176 0 0 0 0 894 815 8 0 92 0 1 0 0 122708640 5179992 1290176 0 0 0 0 963 850 5 0 95 0 1 0 0 122708640 5179992 1290176 0 0 0 0 721 726 5 0 95 0 1 0 0 122708640 5179992 1290176 0 0 0 0 717 739 8 0 92 0 1 0 0 122708640 5179992 1290176 0 0 0 0 717 749 5 0 95 0 1 0 0 122708640 5179992 1290176 0 0 0 0 714 746 6 0 94 0 1 0 0 122708640 5180048 1290176 0 0 0 0 723 1973 8 0 92 0 me@guest:~# vmstat 1 procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu---- r b swpd free buff cache si so bi bo in cs us sy id wa 1 0 2616 686488 7756 208004 0 0 2 34 10 3 0 1 99 0 1 0 2616 680280 7764 213920 0 0 0 0 106 8 7 93 0 0 1 0 2616 674076 7768 219768 0 0 0 0 104 5 8 92 0 0 1 0 2616 669000 7780 224864 0 0 0 15124 188 14 1 99 0 0 1 0 2616 663168 7788 230700 0 0 0 0 104 5 4 96 0 0 1 0 2616 657836 7792 235472 0 0 0 5684 139 21 5 95 0 0 1 0 2616 651648 7796 241288 0 0 0 0 104 5 7 93 0 0 1 0 2616 645428 7804 247136 0 0 0 0 104 5 2 98 0 0 1 0 2616 640956 7816 251908 0 0 0 22164 232 31 3 97 0 0 1 0 2616 635016 7820 257732 0 0 0 0 104 5 2 98 0 0 1 0 2616 629204 7828 263552 0 0 0 0 105 9 2 98 0 0 1 0 2616 622732 7832 269392 0 0 0 0 104 5 5 95 0 0 1 0 2616 616676 7840 275096 0 0 0 0 104 7 5 95 0 0 1 0 2616 612200 7852 279540 0 0 0 27872 263 39 1 99 0 0 1 0 2616 606384 7856 285204 0 0 0 0 105 7 3 97 0 0 1 0 2616 600536 7860 290716 0 0 0 0 104 11 7 93 0 0 1 0 2616 594352 7868 296408 0 0 0 0 104 7 9 91 0 0 > Oh, and try switching to the deadline I/O scheduler on the host instead > of the default cfq. It seems it was already using deadline? me@host:~$ cat /sys/block/cciss\!c0d0/queue/scheduler noop anticipatory [deadline] cfq I also checked on the other KVM host that has faster IO on its guests. It is using software raid, and each individual disk likewise uses deadline scheduler. Thanks for your help on this, -Kurt