From mboxrd@z Thu Jan 1 00:00:00 1970 From: Avi Kivity Subject: Re: slow guest performance with build load, looking for ideas Date: Thu, 02 Jul 2009 08:48:24 +0300 Message-ID: <4A4C4A28.1060105@redhat.com> References: <20090612210443.GA21840@sgi.com> <4A34C3D2.9020009@redhat.com> <20090618230744.GA19307@sgi.com> <4A477B8C.2010502@redhat.com> <20090628190500.GA7347@sgi.com> <4A47E06A.7040700@redhat.com> <20090701214128.GA1380@sgi.com> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: kvm@vger.kernel.org To: Erik Jacobson Return-path: Received: from mx2.redhat.com ([66.187.237.31]:55116 "EHLO mx2.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750713AbZGBFsL (ORCPT ); Thu, 2 Jul 2009 01:48:11 -0400 In-Reply-To: <20090701214128.GA1380@sgi.com> Sender: kvm-owner@vger.kernel.org List-ID: On 07/02/2009 12:41 AM, Erik Jacobson wrote: > I wanted to post in to the thread the lastest test run. > > Avi Kivity provided some ideas to try. I had mixed luck. I'd like to try > this again if we have any thoughts on the vpid/ept issue, or any other > ideas for drilling down on this. Avi Kivity mentioned LVM in the thread. > I continued to just export the whole /dev/sdb to the guest. I'm happy to > try LVM in some form if we think it would help? > Exporting an entire drive is even better than LVM (in terms of performance; flexibility obviously suffers). Just make sure to use cache=none (which I see in your command line below). > * I could NOT find vpid and ept parameters on the host. They weren't here: > /sys/module/kvm_intel/parameters > nor here > /sys/module/kvm/parameters > So the check for those parameters resulted in no information. > Didn't see them elsewhere either: > # pwd > /sys > # find . -name vpid -print > # find . -name ept -print > > Apparently the parameters were only exposed in 2.6.30. Previously they were only available during modprobe. Since you're using nehalem, let's assume they're set correctly (since that's the default). > > I had done some stuff to set up the test including a build I didn't count. > > GUEST time (make -j12&& make -j12 modules), work area disk no cache param > -------------------------------------------------------------------------- > kvm_stat output BEFORE running this test: > > kvm statistics > > efer_reload 13 0 > exits 27145076 1142 > fpu_reload 1298729 0 > halt_exits 2152011 189 > halt_wakeup 494689 123 > host_state_reload 4998646 837 > hypercalls 0 0 > insn_emulation 10165593 302 > insn_emulation_fail 0 0 > invlpg 0 0 > io_exits 2096834 643 > irq_exits 6469071 8 > irq_injections 4765189 190 > irq_window 279385 0 > largepages 0 0 > mmio_exits 0 0 > mmu_cache_miss 18670 0 > mmu_flooded 0 0 > mmu_pde_zapped 0 0 > mmu_pte_updated 0 0 > mmu_pte_write 10440 0 > mmu_recycled 0 0 > Nice and quiet. > qemu-kvm command: > /usr/bin/qemu-kvm -M pc -m 4096 -smp 8 -name f11-test -uuid b7b4b7e4-9c07-22aa-0c95-d5c8a24176c5 -monitor pty -pidfile /var/run/libvirt/qemu//f11-test.pid -drive file=/var/lib/libvirt/images/f11-test.img,if=virtio,index=0,boot=on -drive file=/dev/sdb,if=virtio,index=1 -net nic,macaddr=54:52:00:46:48:0e,model=virtio -net user -serial pty -parallel none -usb -usbdevice tablet -vnc cct201:1 -soundhw es1370 -redir tcp:5555::22 > -usbdevice tablet is known to cause large interrupt loads. I suggest dropping it. If it helps your vnc session, drop your vnc client and use vinagre instead. > test run timing: > real 12m36.165s > user 27m28.976s > sys 8m32.245s > 12 minutes real vs 35 cpu minutes -> scaling only 3:1 on smp 8. > > kvm_stat output after this test run > kvm statistics > > efer_reload 13 0 > exits 47097981 2003 > fpu_reload 2168308 0 > halt_exits 3378761 301 > halt_wakeup 707171 241 > host_state_reload 7545990 1538 > hypercalls 0 0 > insn_emulation 17809066 462 > insn_emulation_fail 0 0 > invlpg 0 0 > io_exits 2801221 1232 > irq_exits 11959063 7 > irq_injections 8395980 304 > irq_window 531641 3 > largepages 0 0 > mmio_exits 0 0 > mmu_cache_miss 28419 0 > mmu_flooded 0 0 > mmu_pde_zapped 0 0 > mmu_pte_updated 0 0 > mmu_pte_write 10440 0 > mmu_recycled 7193 0 > > Nice and quiet too, but what's needed is kvm_stat (or kvm_stat -1) during the run. Many of the 47M exists are unaccounted for, there's a lack in the stats gathering code. vmstat 1 on host and guest during the run would also help. > HOST time (make -j12&& make -j12 modules) with no guest running > ---------------------------------------------------------------- > real 6m50.936s > user 29m12.051s > sys 5m50.867s > > 35 minutes cpu run on 7 minutes real time, so scaling 1:7. User time almost the same, system time different but not enough to account for the large difference in run time. I'm due to get my own Nehalem next week, I'll try to reproduce your results here. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain.