kvm scaling question

kvm scaling question

[Bruce Rogers]

I am wondering if anyone has investigated how well kvm scales when supporting many guests, or many vcpus or both.

I'll do some investigations into the per vm memory overhead and play with bumping the max vcpu limit way beyond 16, but hopefully someone can comment on issues such as locking problems that are known to exist and needing to be addressed to increased parallellism, general overhead percentages which can help provide consolidation expectations, etc.

Also, when I did a simple experiment with vcpu overcommitment, I was surprised how quickly performance suffered (just bringing a Linux vm up), since I would have assumed the additional vcpus would have been halted the vast majority of the time.  On a 2 proc box, overcommitment to 8 vcpus in a guest (I know this isn't a good usage scenario, but does provide some insights) caused the boot time to increase to almost exponential levels. At 16 vcpus, it took hours to just reach the gui login prompt.

Any perspective you can offer would be appreciated.

Bruce

Re: kvm scaling question

[Javier Guerra]

On Fri, Sep 11, 2009 at 10:36 AM, Bruce Rogers <brogers@novell.com> wrote:
> Also, when I did a simple experiment with vcpu overcommitment, I was surprised how quickly performance suffered (just bringing a Linux vm up), since I would have assumed the additional vcpus would have been halted the vast majority of the time.  On a 2 proc box, overcommitment to 8 vcpus in a guest (I know this isn't a good usage scenario, but does provide some insights) caused the boot time to increase to almost exponential levels. At 16 vcpus, it took hours to just reach the gui login prompt.

I'd guess (and hope!) that having many 1- or 2-cpu guests won't kill
performance as sharply as having a single guest with more vcpus than
the physical cpus available.  have you tested that?

-- 
Javier

Re: kvm scaling question

[Marcelo Tosatti]

On Fri, Sep 11, 2009 at 09:36:10AM -0600, Bruce Rogers wrote:
> I am wondering if anyone has investigated how well kvm scales when supporting many guests, or many vcpus or both.
> 
> I'll do some investigations into the per vm memory overhead and
> play with bumping the max vcpu limit way beyond 16, but hopefully
> someone can comment on issues such as locking problems that are known
> to exist and needing to be addressed to increased parallellism,
> general overhead percentages which can help provide consolidation
> expectations, etc.

I suppose it depends on the guest and workload. With an EPT host and
16-way Linux guest doing kernel compilations, on recent kernel, i see:

# Samples: 98703304
#
# Overhead          Command                      Shared Object  Symbol
# ........  ...............  .................................  ......
#
    97.15%               sh  [kernel]                           [k] vmx_vcpu_run
     0.27%               sh  [kernel]                           [k] kvm_arch_vcpu_ioctl_
     0.12%               sh  [kernel]                           [k] default_send_IPI_mas
     0.09%               sh  [kernel]                           [k] _spin_lock_irq

Which is pretty good. Without EPT/NPT the mmu_lock seems to be the major
bottleneck to parallelism.

> Also, when I did a simple experiment with vcpu overcommitment, I was
> surprised how quickly performance suffered (just bringing a Linux vm
> up), since I would have assumed the additional vcpus would have been
> halted the vast majority of the time. On a 2 proc box, overcommitment
> to 8 vcpus in a guest (I know this isn't a good usage scenario, but
> does provide some insights) caused the boot time to increase to almost
> exponential levels. At 16 vcpus, it took hours to just reach the gui
> login prompt.

One probable reason for that are vcpus which hold spinlocks in the guest
are scheduled out in favour of vcpus which spin on that same lock.

> Any perspective you can offer would be appreciated.
> 
> Bruce

Re: kvm scaling question

[Andre Przywara]

Marcelo Tosatti wrote:
> On Fri, Sep 11, 2009 at 09:36:10AM -0600, Bruce Rogers wrote:
>> I am wondering if anyone has investigated how well kvm scales when supporting many guests, or many vcpus or both.
>>
>> I'll do some investigations into the per vm memory overhead and
>> play with bumping the max vcpu limit way beyond 16, but hopefully
>> someone can comment on issues such as locking problems that are known
>> to exist and needing to be addressed to increased parallellism,
>> general overhead percentages which can help provide consolidation
>> expectations, etc.
> 
> I suppose it depends on the guest and workload. With an EPT host and
> 16-way Linux guest doing kernel compilations, on recent kernel, i see:
 > ...
> 
>> Also, when I did a simple experiment with vcpu overcommitment, I was
>> surprised how quickly performance suffered (just bringing a Linux vm
>> up), since I would have assumed the additional vcpus would have been
>> halted the vast majority of the time. On a 2 proc box, overcommitment
>> to 8 vcpus in a guest (I know this isn't a good usage scenario, but
>> does provide some insights) caused the boot time to increase to almost
>> exponential levels. At 16 vcpus, it took hours to just reach the gui
>> login prompt.
> 
> One probable reason for that are vcpus which hold spinlocks in the guest
> are scheduled out in favour of vcpus which spin on that same lock.
We have encountered this issue some time ago in Xen. Ticket spinlocks 
make this even worse. More detailed info can be found here:
http://www.amd64.org/research/virtualization.html#Lock_holder_preemption

Have you tried using paravirtualized spinlock in the guest kernel?
http://lkml.indiana.edu/hypermail/linux/kernel/0807.0/2808.html

Regards,
Andre.

-- 
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 488-3567-12
----to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Andrew Bowd; Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

Proxmox VE 1.4beta1 released

[Martin Maurer]

Hi all,

We just released Proxmox VE 1.4beta1 - KVM live-migration is working great as its now possible to configure NFS, iSCSI, DRBD, LVM with just a view clicks (for DRBD, you still need to do the first config on CLI)

All info: http://www.proxmox.com/forum/showthread.php?t=2066

Best Regards,

Martin Maurer

martin@proxmox.com
http://www.proxmox.com

____________________________________________________________________
Proxmox Server Solutions GmbH
Kohlgasse 51/10, 1050 Vienna, Austria
Phone: +43 1 545 4497 11 Fax: +43 1 545 4497 22
Commercial register no.: FN 258879 f
Registration office: Handelsgericht Wien

Re: kvm scaling question

[Bruce Rogers]

 On 9/11/2009 at 9:46 AM, Javier Guerra <javier@guerrag.com> wrote:
> On Fri, Sep 11, 2009 at 10:36 AM, Bruce Rogers <brogers@novell.com> wrote:
>> Also, when I did a simple experiment with vcpu overcommitment, I was 
> surprised how quickly performance suffered (just bringing a Linux vm up), 
> since I would have assumed the additional vcpus would have been halted the 
> vast majority of the time.  On a 2 proc box, overcommitment to 8 vcpus in a 
> guest (I know this isn't a good usage scenario, but does provide some 
> insights) caused the boot time to increase to almost exponential levels. At 
> 16 vcpus, it took hours to just reach the gui login prompt.
> 
> I'd guess (and hope!) that having many 1- or 2-cpu guests won't kill
> performance as sharply as having a single guest with more vcpus than
> the physical cpus available.  have you tested that?
> 
> -- 
> Javier

Yes, but not empirically.  I'll certainly be doing that, but wanted to see what perspective there was on the results I was seeing.
And I've gotten the response that explains why overcommitment is performing so poorly in another email.

Bruce

Re: kvm scaling question

[Bruce Rogers]

 On 9/11/2009 at 3:53 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Fri, Sep 11, 2009 at 09:36:10AM -0600, Bruce Rogers wrote:
>> I am wondering if anyone has investigated how well kvm scales when 
> supporting many guests, or many vcpus or both.
>> 
>> I'll do some investigations into the per vm memory overhead and
>> play with bumping the max vcpu limit way beyond 16, but hopefully
>> someone can comment on issues such as locking problems that are known
>> to exist and needing to be addressed to increased parallellism,
>> general overhead percentages which can help provide consolidation
>> expectations, etc.
> 
> I suppose it depends on the guest and workload. With an EPT host and
> 16-way Linux guest doing kernel compilations, on recent kernel, i see:
> 
> # Samples: 98703304
> #
> # Overhead          Command                      Shared Object  Symbol
> # ........  ...............  .................................  ......
> #
>     97.15%               sh  [kernel]                           [k] 
> vmx_vcpu_run
>      0.27%               sh  [kernel]                           [k] 
> kvm_arch_vcpu_ioctl_
>      0.12%               sh  [kernel]                           [k] 
> default_send_IPI_mas
>      0.09%               sh  [kernel]                           [k] 
> _spin_lock_irq
> 
> Which is pretty good. Without EPT/NPT the mmu_lock seems to be the major
> bottleneck to parallelism.
> 
>> Also, when I did a simple experiment with vcpu overcommitment, I was
>> surprised how quickly performance suffered (just bringing a Linux vm
>> up), since I would have assumed the additional vcpus would have been
>> halted the vast majority of the time. On a 2 proc box, overcommitment
>> to 8 vcpus in a guest (I know this isn't a good usage scenario, but
>> does provide some insights) caused the boot time to increase to almost
>> exponential levels. At 16 vcpus, it took hours to just reach the gui
>> login prompt.
> 
> One probable reason for that are vcpus which hold spinlocks in the guest
> are scheduled out in favour of vcpus which spin on that same lock.

I suspected it might be a whole lot of spinning happening. That does seems most likely. I was just surprised how bad the behavior was.

Bruce

Re: kvm scaling question

[Bruce Rogers]

 On 9/11/2009 at 5:02 PM, Andre Przywara <andre.przywara@amd.com> wrote:
> Marcelo Tosatti wrote:
>> On Fri, Sep 11, 2009 at 09:36:10AM -0600, Bruce Rogers wrote:
>>> I am wondering if anyone has investigated how well kvm scales when 
> supporting many guests, or many vcpus or both.
>>>
>>> I'll do some investigations into the per vm memory overhead and
>>> play with bumping the max vcpu limit way beyond 16, but hopefully
>>> someone can comment on issues such as locking problems that are known
>>> to exist and needing to be addressed to increased parallellism,
>>> general overhead percentages which can help provide consolidation
>>> expectations, etc.
>> 
>> I suppose it depends on the guest and workload. With an EPT host and
>> 16-way Linux guest doing kernel compilations, on recent kernel, i see:
>  > ...
>> 
>>> Also, when I did a simple experiment with vcpu overcommitment, I was
>>> surprised how quickly performance suffered (just bringing a Linux vm
>>> up), since I would have assumed the additional vcpus would have been
>>> halted the vast majority of the time. On a 2 proc box, overcommitment
>>> to 8 vcpus in a guest (I know this isn't a good usage scenario, but
>>> does provide some insights) caused the boot time to increase to almost
>>> exponential levels. At 16 vcpus, it took hours to just reach the gui
>>> login prompt.
>> 
>> One probable reason for that are vcpus which hold spinlocks in the guest
>> are scheduled out in favour of vcpus which spin on that same lock.
> We have encountered this issue some time ago in Xen. Ticket spinlocks 
> make this even worse. More detailed info can be found here:
> http://www.amd64.org/research/virtualization.html#Lock_holder_preemption 
> 
> Have you tried using paravirtualized spinlock in the guest kernel?
> http://lkml.indiana.edu/hypermail/linux/kernel/0807.0/2808.html 


I'll try to give that a try.  Thanks for the tips.

Bruce

Re: kvm scaling question

[Andrew Theurer]

On Mon, 2009-09-14 at 17:19 -0600, Bruce Rogers wrote:
> On 9/11/2009 at 3:53 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> > On Fri, Sep 11, 2009 at 09:36:10AM -0600, Bruce Rogers wrote:
> >> I am wondering if anyone has investigated how well kvm scales when 
> > supporting many guests, or many vcpus or both.
> >> 
> >> I'll do some investigations into the per vm memory overhead and
> >> play with bumping the max vcpu limit way beyond 16, but hopefully
> >> someone can comment on issues such as locking problems that are known
> >> to exist and needing to be addressed to increased parallellism,
> >> general overhead percentages which can help provide consolidation
> >> expectations, etc.
> > 
> > I suppose it depends on the guest and workload. With an EPT host and
> > 16-way Linux guest doing kernel compilations, on recent kernel, i see:
> > 
> > # Samples: 98703304
> > #
> > # Overhead          Command                      Shared Object  Symbol
> > # ........  ...............  .................................  ......
> > #
> >     97.15%               sh  [kernel]                           [k] 
> > vmx_vcpu_run
> >      0.27%               sh  [kernel]                           [k] 
> > kvm_arch_vcpu_ioctl_
> >      0.12%               sh  [kernel]                           [k] 
> > default_send_IPI_mas
> >      0.09%               sh  [kernel]                           [k] 
> > _spin_lock_irq
> > 
> > Which is pretty good. Without EPT/NPT the mmu_lock seems to be the major
> > bottleneck to parallelism.
> > 
> >> Also, when I did a simple experiment with vcpu overcommitment, I was
> >> surprised how quickly performance suffered (just bringing a Linux vm
> >> up), since I would have assumed the additional vcpus would have been
> >> halted the vast majority of the time. On a 2 proc box, overcommitment
> >> to 8 vcpus in a guest (I know this isn't a good usage scenario, but
> >> does provide some insights) caused the boot time to increase to almost
> >> exponential levels. At 16 vcpus, it took hours to just reach the gui
> >> login prompt.
> > 
> > One probable reason for that are vcpus which hold spinlocks in the guest
> > are scheduled out in favour of vcpus which spin on that same lock.
> 
> I suspected it might be a whole lot of spinning happening. That does seems most likely. I was just surprised how bad the behavior was.

I have collected lock_stat info on a similar vcpu over-commit
configuration, but with EPT system, and saw a very significant amount of
spinning.  However, if you don't have EPT or NPT, I would bet that's the
first problem.  IMO, I am a little surprised simply booting is such a
problem.  I would be interesting to see what lock_stat shows on your
guest after booting with 16 vcpus.  

I have observed that shortening the time between vcpus being scheduled
can help mitigate the problem with lock holder preemption (presumably
because the spinning vcpu is de-scheduled earlier and the vcpu holding
the lock is scheduled sooner), but I imagine there are other unwanted
side-effects like lower cache hits.

-Andrew

> 
> Bruce
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html