Re: kvm scaling question

[Andre Przywara <andre.przywara@amd.com>]

Marcelo Tosatti wrote:
> On Fri, Sep 11, 2009 at 09:36:10AM -0600, Bruce Rogers wrote:
>> I am wondering if anyone has investigated how well kvm scales when supporting many guests, or many vcpus or both.
>>
>> I'll do some investigations into the per vm memory overhead and
>> play with bumping the max vcpu limit way beyond 16, but hopefully
>> someone can comment on issues such as locking problems that are known
>> to exist and needing to be addressed to increased parallellism,
>> general overhead percentages which can help provide consolidation
>> expectations, etc.
> 
> I suppose it depends on the guest and workload. With an EPT host and
> 16-way Linux guest doing kernel compilations, on recent kernel, i see:
 > ...
> 
>> Also, when I did a simple experiment with vcpu overcommitment, I was
>> surprised how quickly performance suffered (just bringing a Linux vm
>> up), since I would have assumed the additional vcpus would have been
>> halted the vast majority of the time. On a 2 proc box, overcommitment
>> to 8 vcpus in a guest (I know this isn't a good usage scenario, but
>> does provide some insights) caused the boot time to increase to almost
>> exponential levels. At 16 vcpus, it took hours to just reach the gui
>> login prompt.
> 
> One probable reason for that are vcpus which hold spinlocks in the guest
> are scheduled out in favour of vcpus which spin on that same lock.
We have encountered this issue some time ago in Xen. Ticket spinlocks 
make this even worse. More detailed info can be found here:
http://www.amd64.org/research/virtualization.html#Lock_holder_preemption

Have you tried using paravirtualized spinlock in the guest kernel?
http://lkml.indiana.edu/hypermail/linux/kernel/0807.0/2808.html

Regards,
Andre.

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 488-3567-12
----to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Andrew Bowd; Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632