Xen CPU limit?

Xen CPU limit?

[Martin Lukasik]

Hello,

Can somebody explain why the number of CPUs Xen can handle is limited to 32?
I have a machine with 48 cores, and would love to virtualise it, but if 
I do it, I loose 16 CPUs, which isnt very good... :-(
Where is this limit coming from? Or maybe it is possible to change it 
somewhere in the source files?

Cheers,
Martin


This communication is from Cancer Research UK. Our website is at www.cancerresearchuk.org. We are a charity registered under number 1089464 and a company limited by guarantee registered in England & Wales under number 4325234. Our registered address is 61 Lincoln's Inn Fields, London WC2A 3PX. Our central telephone number is 020 7242 0200.
 
This communication and any attachments contain information which is confidential and may also be privileged.   It is for the exclusive use of the intended recipient(s).  If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful.  If you have received this communication in error, please notify the sender and delete the email and destroy any copies of it.
 
E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses.  We do not accept liability for any such matters or their consequences.  Anyone who communicates with us by e-mail is taken to accept the risks in doing so.

Re: Xen CPU limit?

[Pasi Kärkkäinen]

On Thu, Mar 11, 2010 at 04:08:38PM +0000, Martin Lukasik wrote:
> Hello,
>
> Can somebody explain why the number of CPUs Xen can handle is limited to 32?
> I have a machine with 48 cores, and would love to virtualise it, but if  
> I do it, I loose 16 CPUs, which isnt very good... :-(
> Where is this limit coming from? Or maybe it is possible to change it  
> somewhere in the source files?
>

I think the number of virtual cpus (vcpus) per guest is (was) 32,
and now in Xen 4.0.0 you can have up to 64 vcpus per guest.

The max number of physical cpus is 128 or so..

-- Pasi

> Cheers,
> Martin
>
>
> This communication is from Cancer Research UK. Our website is at www.cancerresearchuk.org. We are a charity registered under number 1089464 and a company limited by guarantee registered in England & Wales under number 4325234. Our registered address is 61 Lincoln's Inn Fields, London WC2A 3PX. Our central telephone number is 020 7242 0200.
>
> This communication and any attachments contain information which is confidential and may also be privileged.   It is for the exclusive use of the intended recipient(s).  If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful.  If you have received this communication in error, please notify the sender and delete the email and destroy any copies of it.
>
> E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses.  We do not accept liability for any such matters or their consequences.  Anyone who communicates with us by e-mail is taken to accept the risks in doing so.
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Xen CPU limit?

[Jan Beulich]

>>> Martin Lukasik <martin.lukasik@cancer.org.uk> 11.03.10 17:08 >>>
>Can somebody explain why the number of CPUs Xen can handle is limited to 32?
>I have a machine with 48 cores, and would love to virtualise it, but if 
>I do it, I loose 16 CPUs, which isnt very good... :-(
>Where is this limit coming from? Or maybe it is possible to change it 
>somewhere in the source files?

This is done on the make command line, by adding "max_phys_cpus=...".

Jan

Re: Xen CPU limit?

[Martin Lukasik]

Jan Beulich wrote:
>>>> Martin Lukasik <martin.lukasik@cancer.org.uk> 11.03.10 17:08 >>>
>>>>         
>> Can somebody explain why the number of CPUs Xen can handle is limited to 32?
>> I have a machine with 48 cores, and would love to virtualise it, but if 
>> I do it, I loose 16 CPUs, which isnt very good... :-(
>> Where is this limit coming from? Or maybe it is possible to change it 
>> somewhere in the source files?
>>     
>
> This is done on the make command line, by adding "max_phys_cpus=...".
>
> Jan
>   

Yeah... I thought that installing it from RPM wasn't a good idea... ;-)
Thanks for the info. Will give it a try some time this week.

BTW.: Wouldn't it make sense to include support for max number of CPUs 
when building RPMs?

Martin

-- 
-----------------------------------------------------------------------
Martin Lukasik                                    tel: +44 20 7269 3115
Bioinformatics Officer
Bioinformatics and Biostatistics
Cancer Research UK, London Research Institute
44 Lincoln's Inn Fields, London, WC2A 3PX, UK
-----------------------------------------------------------------------


This communication is from Cancer Research UK. Our website is at www.cancerresearchuk.org. We are a charity registered under number 1089464 and a company limited by guarantee registered in England & Wales under number 4325234. Our registered address is 61 Lincoln's Inn Fields, London WC2A 3PX. Our central telephone number is 020 7242 0200.
 
This communication and any attachments contain information which is confidential and may also be privileged.   It is for the exclusive use of the intended recipient(s).  If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful.  If you have received this communication in error, please notify the sender and delete the email and destroy any copies of it.
 
E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses.  We do not accept liability for any such matters or their consequences.  Anyone who communicates with us by e-mail is taken to accept the risks in doing so.

Re: Xen CPU limit?

[Jan Beulich]

>>> Martin Lukasik <martin.lukasik@cancer.org.uk> 29.03.10 18:03 >>>
>BTW.: Wouldn't it make sense to include support for max number of CPUs 
>when building RPMs?

Sure - anyone can easily do so in their .spec file.

Jan

Re: Xen CPU limit?

[Keir Fraser]

Latest Xen supports up to 128 CPUs by default. Earlier versions you can try
rebuilding (for example): "make xen max_phys_cpus=64".

 -- Keir

On 11/03/2010 17:08, "Martin Lukasik" <martin.lukasik@cancer.org.uk> wrote:

> Hello,
> 
> Can somebody explain why the number of CPUs Xen can handle is limited to 32?
> I have a machine with 48 cores, and would love to virtualise it, but if
> I do it, I loose 16 CPUs, which isnt very good... :-(
> Where is this limit coming from? Or maybe it is possible to change it
> somewhere in the source files?
> 
> Cheers,
> Martin
> 
> 
> This communication is from Cancer Research UK. Our website is at
> www.cancerresearchuk.org. We are a charity registered under number 1089464 and
> a company limited by guarantee registered in England & Wales under number
> 4325234. Our registered address is 61 Lincoln's Inn Fields, London WC2A 3PX.
> Our central telephone number is 020 7242 0200.
>  
> This communication and any attachments contain information which is
> confidential and may also be privileged.   It is for the exclusive use of the
> intended recipient(s).  If you are not the intended recipient(s) please note
> that any form of disclosure, distribution, copying or use of this
> communication or the information in it or in any attachments is strictly
> prohibited and may be unlawful.  If you have received this communication in
> error, please notify the sender and delete the email and destroy any copies of
> it.
>  
> E-mail communications cannot be guaranteed to be secure or error free, as
> information could be intercepted, corrupted, amended, lost, destroyed, arrive
> late or incomplete, or contain viruses.  We do not accept liability for any
> such matters or their consequences.  Anyone who communicates with us by e-mail
> is taken to accept the risks in doing so.
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Xen CPU limit?

[Jan Beulich]

>>> Pasi Kärkkäinen<pasik@iki.fi> 29.03.10 17:49 >>>
>I think the number of virtual cpus (vcpus) per guest is (was) 32,
>and now in Xen 4.0.0 you can have up to 64 vcpus per guest.

I don't think the tools are up to anything beyond 32 yet. The (64-bit)
hypervisor allows up to 8192 vCPU-s iirc (but that's a truly
theoretical limit, as Dom0 or a guest likely won't be able to bring up
that many due to there only being 4096 event channels; current
Linux requires 5-6 of them per vCPU for IPIs and timer vIRQ).

Jan

Re: Xen CPU limit?

[Pasi Kärkkäinen]

On Tue, Mar 30, 2010 at 08:16:26AM +0100, Jan Beulich wrote:
> >>> Pasi Kärkkäinen<pasik@iki.fi> 29.03.10 17:49 >>>
> >I think the number of virtual cpus (vcpus) per guest is (was) 32,
> >and now in Xen 4.0.0 you can have up to 64 vcpus per guest.
> 
> I don't think the tools are up to anything beyond 32 yet. 

Hmm.. what's actually missing from the tools to handle >32 vcpus?

> The (64-bit) hypervisor allows up to 8192 vCPU-s iirc (but that's a truly
> theoretical limit, as Dom0 or a guest likely won't be able to bring up
> that many due to there only being 4096 event channels; current
> Linux requires 5-6 of them per vCPU for IPIs and timer vIRQ).
>

Wow.. I wasn't aware the (theoretical) limit is that big :)

-- Pasi
 

Re: Xen CPU limit?

[Jan Beulich]

>>> Pasi Kärkkäinen<pasik@iki.fi> 30.03.10 09:53 >>>
>On Tue, Mar 30, 2010 at 08:16:26AM +0100, Jan Beulich wrote:
>> >>> Pasi Kärkkäinen<pasik@iki.fi> 29.03.10 17:49 >>>
>> >I think the number of virtual cpus (vcpus) per guest is (was) 32,
>> >and now in Xen 4.0.0 you can have up to 64 vcpus per guest.
>> 
>> I don't think the tools are up to anything beyond 32 yet. 
>
>Hmm.. what's actually missing from the tools to handle >32 vcpus?

Any place referencing XEN_LEGACY_MAX_VCPUS needs revisiting
(seems like the one obvious place that needs fixing is in
xc_domain_restore()). Beyond that I really don't know - I just know
too little about the tools.

Jan

Re: Xen CPU limit?

[Martin Lukasik]

Pasi Kärkkäinen wrote:
> I think the number of virtual cpus (vcpus) per guest is (was) 32,
> and now in Xen 4.0.0 you can have up to 64 vcpus per guest.
>
> The max number of physical cpus is 128 or so..
>   

When I said "CPUs" I meant the number of physical CPUs (or strictly 
speaking: cores) visible by Dom0.
I thought I got my RPM from RedHat's repo, but there is a possibility 
that I'm wrong here... and I thought that Xen won't support more than 32 
cores.
I'm not really after setting up 48 cores for one virtual machine, just 
wanted to be able to have all 48 visible in Dom0 (and then have up to 16 
per VM).

Thanks for the answers; interesting read.

Kind regards,
Martin

-- 
-----------------------------------------------------------------------
Martin Lukasik                                    tel: +44 20 7269 3115
Bioinformatics Officer
Bioinformatics and Biostatistics
Cancer Research UK, London Research Institute
44 Lincoln's Inn Fields, London, WC2A 3PX, UK
-----------------------------------------------------------------------


This communication is from Cancer Research UK. Our website is at www.cancerresearchuk.org. We are a charity registered under number 1089464 and a company limited by guarantee registered in England & Wales under number 4325234. Our registered address is 61 Lincoln's Inn Fields, London WC2A 3PX. Our central telephone number is 020 7242 0200.
 
This communication and any attachments contain information which is confidential and may also be privileged.   It is for the exclusive use of the intended recipient(s).  If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful.  If you have received this communication in error, please notify the sender and delete the email and destroy any copies of it.
 
E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses.  We do not accept liability for any such matters or their consequences.  Anyone who communicates with us by e-mail is taken to accept the risks in doing so.

Re: Xen CPU limit?

[Keir Fraser]

On 30/03/2010 10:16, "Martin Lukasik" <martin.lukasik@cancer.org.uk> wrote:

> I'm not really after setting up 48 cores for one virtual machine, just
> wanted to be able to have all 48 visible in Dom0 (and then have up to 16
> per VM).

Dom0 is itself a VM. Though of course it doesn't need tools support to set
it up since it gets auto-constructed by the hypervisor itself. But it's
subject to hypervisor limits just like any other VM.

 -- Keir

Re: Xen CPU limit?

[Pasi Kärkkäinen]

On Tue, Mar 30, 2010 at 10:16:36AM +0100, Martin Lukasik wrote:
> Pasi Kärkkäinen wrote:
>> I think the number of virtual cpus (vcpus) per guest is (was) 32,
>> and now in Xen 4.0.0 you can have up to 64 vcpus per guest.
>>
>> The max number of physical cpus is 128 or so..
>>   
>
> When I said "CPUs" I meant the number of physical CPUs (or strictly  
> speaking: cores) visible by Dom0.
> I thought I got my RPM from RedHat's repo, but there is a possibility  
> that I'm wrong here... and I thought that Xen won't support more than 32  
> cores.
> I'm not really after setting up 48 cores for one virtual machine, just  
> wanted to be able to have all 48 visible in Dom0 (and then have up to 16  
> per VM).
>
> Thanks for the answers; interesting read.
>

Why do you need to have all the cores visible to dom0? 

It's usually enough to have a couple of cores for dom0, 
and the rest free/available in Xen hypervisor for other guests.

-- Pasi

Re: Xen CPU limit?

[Martin Lukasik]

Pasi Kärkkäinen wrote:
>
> Why do you need to have all the cores visible to dom0? 
>
> It's usually enough to have a couple of cores for dom0, 
> and the rest free/available in Xen hypervisor for other guests.
>
>   

I read three books about Xen, but surely they weren't a good read. Many 
things haven't been even mentioned (btw: can anyone recommend a *good* 
book on Xen?)
I thought that the way it works is: you set up Dom0 with max number of 
CPUs and then allocate them to VMs.
But since Dom0 is a VM as Keir said, then it all must be up to a hypervisor.

So I need a hypervisor to support 48 cores, then set up Dom0 with let's 
say 2 vCPUs and 1GB of RAM, and create other VMs according to my 
requirements.
Am I right?


Thank you,
Martin

-- 
-----------------------------------------------------------------------
Martin Lukasik                                    tel: +44 20 7269 3115
Bioinformatics Officer
Bioinformatics and Biostatistics
Cancer Research UK, London Research Institute
44 Lincoln's Inn Fields, London, WC2A 3PX, UK
-----------------------------------------------------------------------


This communication is from Cancer Research UK. Our website is at www.cancerresearchuk.org. We are a charity registered under number 1089464 and a company limited by guarantee registered in England & Wales under number 4325234. Our registered address is 61 Lincoln's Inn Fields, London WC2A 3PX. Our central telephone number is 020 7242 0200.
 
This communication and any attachments contain information which is confidential and may also be privileged.   It is for the exclusive use of the intended recipient(s).  If you are not the intended recipient(s) please note that any form of disclosure, distribution, copying or use of this communication or the information in it or in any attachments is strictly prohibited and may be unlawful.  If you have received this communication in error, please notify the sender and delete the email and destroy any copies of it.
 
E-mail communications cannot be guaranteed to be secure or error free, as information could be intercepted, corrupted, amended, lost, destroyed, arrive late or incomplete, or contain viruses.  We do not accept liability for any such matters or their consequences.  Anyone who communicates with us by e-mail is taken to accept the risks in doing so.

Re: Xen CPU limit?

[Pasi Kärkkäinen]

On Tue, Mar 30, 2010 at 10:32:29AM +0100, Martin Lukasik wrote:
> Pasi Kärkkäinen wrote:
>>
>> Why do you need to have all the cores visible to dom0? 
>>
>> It's usually enough to have a couple of cores for dom0, and the rest 
>> free/available in Xen hypervisor for other guests.
>>
>>   
>
> I read three books about Xen, but surely they weren't a good read. Many  
> things haven't been even mentioned (btw: can anyone recommend a *good*  
> book on Xen?)
> I thought that the way it works is: you set up Dom0 with max number of  
> CPUs and then allocate them to VMs.
> But since Dom0 is a VM as Keir said, then it all must be up to a hypervisor.
>
> So I need a hypervisor to support 48 cores, then set up Dom0 with let's  
> say 2 vCPUs and 1GB of RAM, and create other VMs according to my  
> requirements.
> Am I right?
>

Yes, that's correct. 

See here for some tips: http://wiki.xensource.com/xenwiki/XenBestPractices

-- Pasi

Re: Xen CPU limit?

[Keir Fraser]

On 30/03/2010 10:32, "Martin Lukasik" <martin.lukasik@cancer.org.uk> wrote:

> I read three books about Xen, but surely they weren't a good read. Many
> things haven't been even mentioned (btw: can anyone recommend a *good*
> book on Xen?)
> I thought that the way it works is: you set up Dom0 with max number of
> CPUs and then allocate them to VMs.
> But since Dom0 is a VM as Keir said, then it all must be up to a hypervisor.
>
> So I need a hypervisor to support 48 cores, then set up Dom0 with let's
> say 2 vCPUs and 1GB of RAM, and create other VMs according to my
> requirements.
> Am I right?

Yeah, that would be sensible. All dom0 is doing is VM management (i.e.,
running control tools like xend/xm) and also handling I/O, so two cores is
probably plenty.

 -- Keir

RE: Xen CPU limit?

[Dan Magenheimer]

> -----Original Message-----
> From: Pasi Kärkkäinen [mailto:pasik@iki.fi]
> Sent: Tuesday, March 30, 2010 1:53 AM
> To: Jan Beulich
> Cc: xen-devel@lists.xensource.com; Martin Lukasik
> Subject: Re: [Xen-devel] Xen CPU limit?
> 
> On Tue, Mar 30, 2010 at 08:16:26AM +0100, Jan Beulich wrote:
> > >>> Pasi Kärkkäinen<pasik@iki.fi> 29.03.10 17:49 >>>
> > >I think the number of virtual cpus (vcpus) per guest is (was) 32,
> > >and now in Xen 4.0.0 you can have up to 64 vcpus per guest.
> >
> > I don't think the tools are up to anything beyond 32 yet.
> 
> Hmm.. what's actually missing from the tools to handle >32 vcpus?
> 
> > The (64-bit) hypervisor allows up to 8192 vCPU-s iirc (but that's a
> truly
> > theoretical limit, as Dom0 or a guest likely won't be able to bring
> up
> > that many due to there only being 4096 event channels; current
> > Linux requires 5-6 of them per vCPU for IPIs and timer vIRQ).
> >
> 
> Wow.. I wasn't aware the (theoretical) limit is that big :)

But, if I am reading this correctly, 4096 event channels
is the next scalability barrier.  If, on a "big" machine, one
tries to run 64 (nearly always idle) guests each configured
with 16 vcpus (because they run a busy database load when
they are not idle), it won't work due to the event channel limit?

This scenario seems quite possible in a cloud/hosting environment.

RE: Xen CPU limit?

[Jan Beulich]

>>> Dan Magenheimer <dan.magenheimer@oracle.com> 30.03.10 18:14 >>>
>But, if I am reading this correctly, 4096 event channels
>is the next scalability barrier.  If, on a "big" machine, one
>tries to run 64 (nearly always idle) guests each configured
>with 16 vcpus (because they run a busy database load when
>they are not idle), it won't work due to the event channel limit?

No, the event channel limit is a per-domain one.

Jan