* kvm scaling question
@ 2009-09-11 15:36 Bruce Rogers
2009-09-11 15:46 ` Javier Guerra
` (2 more replies)
0 siblings, 3 replies; 9+ messages in thread
From: Bruce Rogers @ 2009-09-11 15:36 UTC (permalink / raw)
To: kvm
I am wondering if anyone has investigated how well kvm scales when supporting many guests, or many vcpus or both.
I'll do some investigations into the per vm memory overhead and play with bumping the max vcpu limit way beyond 16, but hopefully someone can comment on issues such as locking problems that are known to exist and needing to be addressed to increased parallellism, general overhead percentages which can help provide consolidation expectations, etc.
Also, when I did a simple experiment with vcpu overcommitment, I was surprised how quickly performance suffered (just bringing a Linux vm up), since I would have assumed the additional vcpus would have been halted the vast majority of the time. On a 2 proc box, overcommitment to 8 vcpus in a guest (I know this isn't a good usage scenario, but does provide some insights) caused the boot time to increase to almost exponential levels. At 16 vcpus, it took hours to just reach the gui login prompt.
Any perspective you can offer would be appreciated.
Bruce
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: kvm scaling question 2009-09-11 15:36 kvm scaling question Bruce Rogers @ 2009-09-11 15:46 ` Javier Guerra 2009-09-14 23:12 ` Bruce Rogers 2009-09-11 21:53 ` Marcelo Tosatti 2009-09-12 9:52 ` Proxmox VE 1.4beta1 released Martin Maurer 2 siblings, 1 reply; 9+ messages in thread From: Javier Guerra @ 2009-09-11 15:46 UTC (permalink / raw) To: Bruce Rogers; +Cc: kvm On Fri, Sep 11, 2009 at 10:36 AM, Bruce Rogers <brogers@novell.com> wrote: > Also, when I did a simple experiment with vcpu overcommitment, I was surprised how quickly performance suffered (just bringing a Linux vm up), since I would have assumed the additional vcpus would have been halted the vast majority of the time. On a 2 proc box, overcommitment to 8 vcpus in a guest (I know this isn't a good usage scenario, but does provide some insights) caused the boot time to increase to almost exponential levels. At 16 vcpus, it took hours to just reach the gui login prompt. I'd guess (and hope!) that having many 1- or 2-cpu guests won't kill performance as sharply as having a single guest with more vcpus than the physical cpus available. have you tested that? -- Javier ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kvm scaling question 2009-09-11 15:46 ` Javier Guerra @ 2009-09-14 23:12 ` Bruce Rogers 0 siblings, 0 replies; 9+ messages in thread From: Bruce Rogers @ 2009-09-14 23:12 UTC (permalink / raw) To: Javier Guerra; +Cc: kvm On 9/11/2009 at 9:46 AM, Javier Guerra <javier@guerrag.com> wrote: > On Fri, Sep 11, 2009 at 10:36 AM, Bruce Rogers <brogers@novell.com> wrote: >> Also, when I did a simple experiment with vcpu overcommitment, I was > surprised how quickly performance suffered (just bringing a Linux vm up), > since I would have assumed the additional vcpus would have been halted the > vast majority of the time. On a 2 proc box, overcommitment to 8 vcpus in a > guest (I know this isn't a good usage scenario, but does provide some > insights) caused the boot time to increase to almost exponential levels. At > 16 vcpus, it took hours to just reach the gui login prompt. > > I'd guess (and hope!) that having many 1- or 2-cpu guests won't kill > performance as sharply as having a single guest with more vcpus than > the physical cpus available. have you tested that? > > -- > Javier Yes, but not empirically. I'll certainly be doing that, but wanted to see what perspective there was on the results I was seeing. And I've gotten the response that explains why overcommitment is performing so poorly in another email. Bruce ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kvm scaling question 2009-09-11 15:36 kvm scaling question Bruce Rogers 2009-09-11 15:46 ` Javier Guerra @ 2009-09-11 21:53 ` Marcelo Tosatti 2009-09-11 23:02 ` Andre Przywara 2009-09-14 23:19 ` Bruce Rogers 2009-09-12 9:52 ` Proxmox VE 1.4beta1 released Martin Maurer 2 siblings, 2 replies; 9+ messages in thread From: Marcelo Tosatti @ 2009-09-11 21:53 UTC (permalink / raw) To: Bruce Rogers; +Cc: kvm On Fri, Sep 11, 2009 at 09:36:10AM -0600, Bruce Rogers wrote: > I am wondering if anyone has investigated how well kvm scales when supporting many guests, or many vcpus or both. > > I'll do some investigations into the per vm memory overhead and > play with bumping the max vcpu limit way beyond 16, but hopefully > someone can comment on issues such as locking problems that are known > to exist and needing to be addressed to increased parallellism, > general overhead percentages which can help provide consolidation > expectations, etc. I suppose it depends on the guest and workload. With an EPT host and 16-way Linux guest doing kernel compilations, on recent kernel, i see: # Samples: 98703304 # # Overhead Command Shared Object Symbol # ........ ............... ................................. ...... # 97.15% sh [kernel] [k] vmx_vcpu_run 0.27% sh [kernel] [k] kvm_arch_vcpu_ioctl_ 0.12% sh [kernel] [k] default_send_IPI_mas 0.09% sh [kernel] [k] _spin_lock_irq Which is pretty good. Without EPT/NPT the mmu_lock seems to be the major bottleneck to parallelism. > Also, when I did a simple experiment with vcpu overcommitment, I was > surprised how quickly performance suffered (just bringing a Linux vm > up), since I would have assumed the additional vcpus would have been > halted the vast majority of the time. On a 2 proc box, overcommitment > to 8 vcpus in a guest (I know this isn't a good usage scenario, but > does provide some insights) caused the boot time to increase to almost > exponential levels. At 16 vcpus, it took hours to just reach the gui > login prompt. One probable reason for that are vcpus which hold spinlocks in the guest are scheduled out in favour of vcpus which spin on that same lock. > Any perspective you can offer would be appreciated. > > Bruce ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kvm scaling question 2009-09-11 21:53 ` Marcelo Tosatti @ 2009-09-11 23:02 ` Andre Przywara 2009-09-14 23:21 ` Bruce Rogers 2009-09-14 23:19 ` Bruce Rogers 1 sibling, 1 reply; 9+ messages in thread From: Andre Przywara @ 2009-09-11 23:02 UTC (permalink / raw) To: Marcelo Tosatti; +Cc: Bruce Rogers, kvm, Thomas Friebel Marcelo Tosatti wrote: > On Fri, Sep 11, 2009 at 09:36:10AM -0600, Bruce Rogers wrote: >> I am wondering if anyone has investigated how well kvm scales when supporting many guests, or many vcpus or both. >> >> I'll do some investigations into the per vm memory overhead and >> play with bumping the max vcpu limit way beyond 16, but hopefully >> someone can comment on issues such as locking problems that are known >> to exist and needing to be addressed to increased parallellism, >> general overhead percentages which can help provide consolidation >> expectations, etc. > > I suppose it depends on the guest and workload. With an EPT host and > 16-way Linux guest doing kernel compilations, on recent kernel, i see: > ... > >> Also, when I did a simple experiment with vcpu overcommitment, I was >> surprised how quickly performance suffered (just bringing a Linux vm >> up), since I would have assumed the additional vcpus would have been >> halted the vast majority of the time. On a 2 proc box, overcommitment >> to 8 vcpus in a guest (I know this isn't a good usage scenario, but >> does provide some insights) caused the boot time to increase to almost >> exponential levels. At 16 vcpus, it took hours to just reach the gui >> login prompt. > > One probable reason for that are vcpus which hold spinlocks in the guest > are scheduled out in favour of vcpus which spin on that same lock. We have encountered this issue some time ago in Xen. Ticket spinlocks make this even worse. More detailed info can be found here: http://www.amd64.org/research/virtualization.html#Lock_holder_preemption Have you tried using paravirtualized spinlock in the guest kernel? http://lkml.indiana.edu/hypermail/linux/kernel/0807.0/2808.html Regards, Andre. -- Andre Przywara AMD-Operating System Research Center (OSRC), Dresden, Germany Tel: +49 351 488-3567-12 ----to satisfy European Law for business letters: Advanced Micro Devices GmbH Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen Geschaeftsfuehrer: Andrew Bowd; Thomas M. McCoy; Giuliano Meroni Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen Registergericht Muenchen, HRB Nr. 43632 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kvm scaling question 2009-09-11 23:02 ` Andre Przywara @ 2009-09-14 23:21 ` Bruce Rogers 0 siblings, 0 replies; 9+ messages in thread From: Bruce Rogers @ 2009-09-14 23:21 UTC (permalink / raw) To: Andre Przywara, Marcelo Tosatti; +Cc: Thomas Friebel, kvm On 9/11/2009 at 5:02 PM, Andre Przywara <andre.przywara@amd.com> wrote: > Marcelo Tosatti wrote: >> On Fri, Sep 11, 2009 at 09:36:10AM -0600, Bruce Rogers wrote: >>> I am wondering if anyone has investigated how well kvm scales when > supporting many guests, or many vcpus or both. >>> >>> I'll do some investigations into the per vm memory overhead and >>> play with bumping the max vcpu limit way beyond 16, but hopefully >>> someone can comment on issues such as locking problems that are known >>> to exist and needing to be addressed to increased parallellism, >>> general overhead percentages which can help provide consolidation >>> expectations, etc. >> >> I suppose it depends on the guest and workload. With an EPT host and >> 16-way Linux guest doing kernel compilations, on recent kernel, i see: > > ... >> >>> Also, when I did a simple experiment with vcpu overcommitment, I was >>> surprised how quickly performance suffered (just bringing a Linux vm >>> up), since I would have assumed the additional vcpus would have been >>> halted the vast majority of the time. On a 2 proc box, overcommitment >>> to 8 vcpus in a guest (I know this isn't a good usage scenario, but >>> does provide some insights) caused the boot time to increase to almost >>> exponential levels. At 16 vcpus, it took hours to just reach the gui >>> login prompt. >> >> One probable reason for that are vcpus which hold spinlocks in the guest >> are scheduled out in favour of vcpus which spin on that same lock. > We have encountered this issue some time ago in Xen. Ticket spinlocks > make this even worse. More detailed info can be found here: > http://www.amd64.org/research/virtualization.html#Lock_holder_preemption > > Have you tried using paravirtualized spinlock in the guest kernel? > http://lkml.indiana.edu/hypermail/linux/kernel/0807.0/2808.html I'll try to give that a try. Thanks for the tips. Bruce ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kvm scaling question 2009-09-11 21:53 ` Marcelo Tosatti 2009-09-11 23:02 ` Andre Przywara @ 2009-09-14 23:19 ` Bruce Rogers 2009-09-15 14:10 ` Andrew Theurer 1 sibling, 1 reply; 9+ messages in thread From: Bruce Rogers @ 2009-09-14 23:19 UTC (permalink / raw) To: Marcelo Tosatti; +Cc: kvm On 9/11/2009 at 3:53 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote: > On Fri, Sep 11, 2009 at 09:36:10AM -0600, Bruce Rogers wrote: >> I am wondering if anyone has investigated how well kvm scales when > supporting many guests, or many vcpus or both. >> >> I'll do some investigations into the per vm memory overhead and >> play with bumping the max vcpu limit way beyond 16, but hopefully >> someone can comment on issues such as locking problems that are known >> to exist and needing to be addressed to increased parallellism, >> general overhead percentages which can help provide consolidation >> expectations, etc. > > I suppose it depends on the guest and workload. With an EPT host and > 16-way Linux guest doing kernel compilations, on recent kernel, i see: > > # Samples: 98703304 > # > # Overhead Command Shared Object Symbol > # ........ ............... ................................. ...... > # > 97.15% sh [kernel] [k] > vmx_vcpu_run > 0.27% sh [kernel] [k] > kvm_arch_vcpu_ioctl_ > 0.12% sh [kernel] [k] > default_send_IPI_mas > 0.09% sh [kernel] [k] > _spin_lock_irq > > Which is pretty good. Without EPT/NPT the mmu_lock seems to be the major > bottleneck to parallelism. > >> Also, when I did a simple experiment with vcpu overcommitment, I was >> surprised how quickly performance suffered (just bringing a Linux vm >> up), since I would have assumed the additional vcpus would have been >> halted the vast majority of the time. On a 2 proc box, overcommitment >> to 8 vcpus in a guest (I know this isn't a good usage scenario, but >> does provide some insights) caused the boot time to increase to almost >> exponential levels. At 16 vcpus, it took hours to just reach the gui >> login prompt. > > One probable reason for that are vcpus which hold spinlocks in the guest > are scheduled out in favour of vcpus which spin on that same lock. I suspected it might be a whole lot of spinning happening. That does seems most likely. I was just surprised how bad the behavior was. Bruce ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: kvm scaling question 2009-09-14 23:19 ` Bruce Rogers @ 2009-09-15 14:10 ` Andrew Theurer 0 siblings, 0 replies; 9+ messages in thread From: Andrew Theurer @ 2009-09-15 14:10 UTC (permalink / raw) To: Bruce Rogers; +Cc: Marcelo Tosatti, kvm On Mon, 2009-09-14 at 17:19 -0600, Bruce Rogers wrote: > On 9/11/2009 at 3:53 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote: > > On Fri, Sep 11, 2009 at 09:36:10AM -0600, Bruce Rogers wrote: > >> I am wondering if anyone has investigated how well kvm scales when > > supporting many guests, or many vcpus or both. > >> > >> I'll do some investigations into the per vm memory overhead and > >> play with bumping the max vcpu limit way beyond 16, but hopefully > >> someone can comment on issues such as locking problems that are known > >> to exist and needing to be addressed to increased parallellism, > >> general overhead percentages which can help provide consolidation > >> expectations, etc. > > > > I suppose it depends on the guest and workload. With an EPT host and > > 16-way Linux guest doing kernel compilations, on recent kernel, i see: > > > > # Samples: 98703304 > > # > > # Overhead Command Shared Object Symbol > > # ........ ............... ................................. ...... > > # > > 97.15% sh [kernel] [k] > > vmx_vcpu_run > > 0.27% sh [kernel] [k] > > kvm_arch_vcpu_ioctl_ > > 0.12% sh [kernel] [k] > > default_send_IPI_mas > > 0.09% sh [kernel] [k] > > _spin_lock_irq > > > > Which is pretty good. Without EPT/NPT the mmu_lock seems to be the major > > bottleneck to parallelism. > > > >> Also, when I did a simple experiment with vcpu overcommitment, I was > >> surprised how quickly performance suffered (just bringing a Linux vm > >> up), since I would have assumed the additional vcpus would have been > >> halted the vast majority of the time. On a 2 proc box, overcommitment > >> to 8 vcpus in a guest (I know this isn't a good usage scenario, but > >> does provide some insights) caused the boot time to increase to almost > >> exponential levels. At 16 vcpus, it took hours to just reach the gui > >> login prompt. > > > > One probable reason for that are vcpus which hold spinlocks in the guest > > are scheduled out in favour of vcpus which spin on that same lock. > > I suspected it might be a whole lot of spinning happening. That does seems most likely. I was just surprised how bad the behavior was. I have collected lock_stat info on a similar vcpu over-commit configuration, but with EPT system, and saw a very significant amount of spinning. However, if you don't have EPT or NPT, I would bet that's the first problem. IMO, I am a little surprised simply booting is such a problem. I would be interesting to see what lock_stat shows on your guest after booting with 16 vcpus. I have observed that shortening the time between vcpus being scheduled can help mitigate the problem with lock holder preemption (presumably because the spinning vcpu is de-scheduled earlier and the vcpu holding the lock is scheduled sooner), but I imagine there are other unwanted side-effects like lower cache hits. -Andrew > > Bruce > > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Proxmox VE 1.4beta1 released 2009-09-11 15:36 kvm scaling question Bruce Rogers 2009-09-11 15:46 ` Javier Guerra 2009-09-11 21:53 ` Marcelo Tosatti @ 2009-09-12 9:52 ` Martin Maurer 2 siblings, 0 replies; 9+ messages in thread From: Martin Maurer @ 2009-09-12 9:52 UTC (permalink / raw) To: kvm Hi all, We just released Proxmox VE 1.4beta1 - KVM live-migration is working great as its now possible to configure NFS, iSCSI, DRBD, LVM with just a view clicks (for DRBD, you still need to do the first config on CLI) All info: http://www.proxmox.com/forum/showthread.php?t=2066 Best Regards, Martin Maurer martin@proxmox.com http://www.proxmox.com ____________________________________________________________________ Proxmox Server Solutions GmbH Kohlgasse 51/10, 1050 Vienna, Austria Phone: +43 1 545 4497 11 Fax: +43 1 545 4497 22 Commercial register no.: FN 258879 f Registration office: Handelsgericht Wien ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2009-09-15 14:10 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2009-09-11 15:36 kvm scaling question Bruce Rogers 2009-09-11 15:46 ` Javier Guerra 2009-09-14 23:12 ` Bruce Rogers 2009-09-11 21:53 ` Marcelo Tosatti 2009-09-11 23:02 ` Andre Przywara 2009-09-14 23:21 ` Bruce Rogers 2009-09-14 23:19 ` Bruce Rogers 2009-09-15 14:10 ` Andrew Theurer 2009-09-12 9:52 ` Proxmox VE 1.4beta1 released Martin Maurer
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.