All of lore.kernel.org
 help / color / mirror / Atom feed
* Max. PV and HVM Guests
@ 2009-11-08 11:45 Mr. Teo En Ming (Zhang Enming)
  2009-11-08 11:46 ` Mr. Teo En Ming (Zhang Enming)
                   ` (2 more replies)
  0 siblings, 3 replies; 20+ messages in thread
From: Mr. Teo En Ming (Zhang Enming) @ 2009-11-08 11:45 UTC (permalink / raw)
  To: xen-devel, xen-users; +Cc: space.time.universe


[-- Attachment #1.1: Type: text/plain, Size: 1226 bytes --]

Hi,

I have observed that there is a maximum limit to the number of PV or HVM
virtual machines you can start before dom0 hangs or crashes.

For Fedora 11 Linux x86-64 PV guests, dom0 will crash when I start the 7th
instance. Max I can start is 6 without crashing.

For CentOS 5.2 Linux x86-64 HVM guests, dom0 will crash when I start the 4th
instance. Max I can start is 3 without crashing.

I have 6 GB of DDR2-800 with Intel Pentium Dual Core E6300 2.8 GHz on Intel
DQ45CB motherboard.

Are the above limits reasonable considering the hardware specifications of
my computer?

I am using Xen 3.5-unstable changeset 20143 with pv-ops dom0 kernels
2.6.30-rc3, 2.6.31-rc6, 2.6.31.1, 2.6.31.4, and 2.6.31.5. My host operating
system is Fedora 11 Linux x86-64.

-- 
Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics) BEng(Hons)(Mechanical
Engineering)
Alma Maters:
(1) Singapore Polytechnic
(2) National University of Singapore
My Primary Blog: http://teo-en-ming-aka-zhang-enming.blogspot.com
My Secondary Blog: http://enmingteo.wordpress.com
My Youtube videos: http://www.youtube.com/user/enmingteo
Email: space.time.universe@gmail.com
Mobile Phone (Starhub Prepaid): +65-8369-2618
Street: Bedok Reservoir Road
Country: Singapore

[-- Attachment #1.2: Type: text/html, Size: 1539 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: Re: [Xen-users] Max. PV and HVM Guests
@ 2009-11-09 15:06 Nick Couchman
  2009-11-09 15:18 ` [Xen-devel] " Keir Fraser
  0 siblings, 1 reply; 20+ messages in thread
From: Nick Couchman @ 2009-11-09 15:06 UTC (permalink / raw)
  To: Mr. Teo En Ming (Zhang Enming), Pasi Kärkkäinen
  Cc: xen-devel, xen-users, Robert Dunkley

[-- Attachment #1: Type: text/plain, Size: 6184 bytes --]



>>> On 2009/11/09 at 05:05, Pasi Kärkkäinen<pasik@iki.fi> wrote: 
> On Mon, Nov 09, 2009 at 08:01:00PM +0800, Mr. Teo En Ming (Zhang
Enming) 
> wrote:
>>    No, I didn't limit dom0 memory in grub.conf.
>> 
> 
> You should. 

Really?  I thought current conventional wisdom was to allow Xen to
self-manage memory in both dom0 and domUs, and not to manually adjust
this?  I run several Xen systems with anywhere from 8 to 24 GB of RAM
and 20 to 30 domUs on some of these systems and have *never* specified
the dom0 memory at boot time - the Xen ballooning has always functioned
perfectly fine, and never crashed my dom0.  Furthermore, while I'm not
Linux developer and so not familiar with how Linux calculates buffering
and caching, I do know that my Linux systems dynamically manage buffers
and caches, and when memory is reduced or some application requires a
larger amount of physical memory, Linux reduces the amount of data in
buffers and caches.

Of course, a lot of this depends on what you're doing in dom0 - on my
Xen servers, my dom0 is strictly for Xen management - I'm not running
anything else in dom0 that would require large amounts of memory, memory
buffers and caches, etc.

-Nick

> 
> If dom0 has all the memory at boot time, you need to balloon down
dom0 
> memory every time you create a new guest - this can (and will) cause

> problems with the dom0 linux kernel.
> 
> Linux calculates some internal parameters/buffers/values based on
the
> _boot time_ amount of memory. And when the amount of memory goes down
to
> only a small fraction of that while creating new guests bad things
can
> happen..
> 
> It still shouldn't crash though.. I bet your problem will get fixed
when
> you limit the dom0 memory to say dom0_mem=512M and reboot.
> 
> -- Pasi
> 
>>    Here's my xm info output after I have shutdown all the virtual
machines.
>> 
>>    [root@fedora11-x86-64-host ~]# xm list
>>    Name                                        ID   Mem VCPUs     
State
>>    Time(s)
>>    Domain-0                                     0  2812     2    
r-----
>>    3242.5
>>    [root@fedora11-x86-64-host ~]# xm info
>>    host                   : fedora11-x86-64-host
>>    release                : 2.6.30-rc3-enming.teo-tip
>>    version                : #1 SMP Wed Aug 19 23:14:15 SGT 2009
>>    machine                : x86_64
>>    nr_cpus                : 2
>>    nr_nodes               : 1
>>    cores_per_socket       : 2
>>    threads_per_core       : 1
>>    cpu_mhz                : 2800
>>    hw_caps                :
>>   
bfebfbff:20100800:00000000:00000140:0400e3bd:00000000:00000001:00000000
>>    virt_caps              : hvm hvm_directio
>>    total_memory           : 6039
>>    free_memory            : 3124
>>    node_to_cpu            : node0:0-1
>>    node_to_memory         : node0:3124
>>    xen_major              : 3
>>    xen_minor              : 5
>>    xen_extra              : -unstable
>>    xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
hvm-3.0-x86_32
>>    hvm-3.0-x86_32p hvm-3.0-x86_64
>>    xen_scheduler          : credit
>>    xen_pagesize           : 4096
>>    platform_params        : virt_start=0xffff800000000000
>>    xen_changeset          : Tue Sep 01 11:34:31 2009 +0100 
> 20143:a7de5bd776ca
>>    xen_commandline        : iommu=1
>>    cc_compiler            : gcc version 4.4.1 20090725 (Red Hat
4.4.1-2)
>>    (GCC)
>>    cc_compile_by          : root
>>    cc_compile_domain      : (none)
>>    cc_compile_date        : Thu Sep 10 07:01:13 SGT 2009
>>    xend_config_format     : 4
>> 
>>    --
>>    Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics)
BEng(Hons)(Mechanical
>>    Engineering)
>>    Alma Maters:
>>    (1) Singapore Polytechnic
>>    (2) National University of Singapore
>>    My Primary Blog:
[1]http://teo-en-ming-aka-zhang-enming.blogspot.com
>>    My Secondary Blog: [2]http://enmingteo.wordpress.com
>>    My Youtube videos: [3]http://www.youtube.com/user/enmingteo
>>    Email: [4]space.time.universe@gmail.com
>>    Mobile Phone (Starhub Prepaid): +65-8369-2618
>>    Street: Bedok Reservoir Road
>>    Country: Singapore
>> 
>>    On Mon, Nov 9, 2009 at 7:54 PM, Pasi Kärkkäinen <[5]pasik@iki.fi>
wrote:
>> 
>>      On Mon, Nov 09, 2009 at 06:52:37PM +0800, Mr. Teo En Ming
(Zhang 
> Enming)
>>      wrote:
>>      >    Hi,
>>      >
>>      >    Please watch this 4-minute video at
>>      >    [1][6]http://www.youtube.com/watch?v=LbLaPpwNAx4
>>      >
>>      >    I have only started 3 HVM Linux guests with 1 GB ram each.
I can't
>>      start
>>      >    the 4th HVM guest. If I attempt to start the 4th instance,
it will
>>      crash
>>      >    dom0.
>>      >
>>      >    Are there anything in the xm dmesg output that could
explain the
>>      low limit
>>      >    to the number of VMs that I could start before dom0
becomes
>>      unresponsive?
>>      >
>> 
>>      Have you limited dom0 memory (by specifying dom0_mem=XMB option
in
>>      grub.conf for xen.gz) ?
>> 
>>      What does "xm info" say about free memory before starting any
guests?
>>      -- Pasi
>> 
>> References
>> 
>>    Visible links
>>    1. http://teo-en-ming-aka-zhang-enming.blogspot.com/
>>    2. http://enmingteo.wordpress.com/
>>    3. http://www.youtube.com/user/enmingteo
>>    4. mailto:space.time.universe@gmail.com
>>    5. mailto:pasik@iki.fi
>>    6. http://www.youtube.com/watch?v=LbLaPpwNAx4



--------
This e-mail may contain confidential and privileged material for the sole use of the intended recipient.  If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information.  In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way.  If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox.  Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR.

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

^ permalink raw reply	[flat|nested] 20+ messages in thread
* Re: [Xen-devel] Re: Max. PV and HVM Guests
  2009-11-09 15:06 Re: [Xen-users] " Nick Couchman
@ 2009-11-09 15:17 Pasi Kärkkäinen
  2009-11-09 15:24 ` Keir Fraser
  2009-11-09 15:39 ` Dan Magenheimer
  0 siblings, 2 replies; 20+ messages in thread
From: Pasi Kärkkäinen @ 2009-11-09 15:17 UTC (permalink / raw)
  To: Nick Couchman
  Cc: xen-devel, Mr. Teo En Ming (Zhang Enming), xen-users, Robert Dunkley

On Mon, Nov 09, 2009 at 08:06:54AM -0700, Nick Couchman wrote:
> 
> 
> >>> On 2009/11/09 at 05:05, Pasi Kärkkäinen<pasik@iki.fi> wrote: 
> > On Mon, Nov 09, 2009 at 08:01:00PM +0800, Mr. Teo En Ming (Zhang
> Enming) 
> > wrote:
> >>    No, I didn't limit dom0 memory in grub.conf.
> >> 
> > 
> > You should. 
> 
> Really?  I thought current conventional wisdom was to allow Xen to
> self-manage memory in both dom0 and domUs, and not to manually adjust
> this?  I run several Xen systems with anywhere from 8 to 24 GB of RAM
> and 20 to 30 domUs on some of these systems and have *never* specified
> the dom0 memory at boot time - the Xen ballooning has always functioned
> perfectly fine, and never crashed my dom0.  
>

Yes, Xen is totally OK with this, but dom0 Linux has more problems..

> Furthermore, while I'm not
> Linux developer and so not familiar with how Linux calculates buffering
> and caching, I do know that my Linux systems dynamically manage buffers
> and caches, and when memory is reduced or some application requires a
> larger amount of physical memory, Linux reduces the amount of data in
> buffers and caches.
> 

Yeah, it has to do with sizing the network buffers, caches etc.. 

It shouldn't _crash_, so Teo is seeing some bug I believe. But it has
always been "best practice" to limit dom0 memory - and prevent weird
things happening later (like "memory squeeze in netback driver").

> Of course, a lot of this depends on what you're doing in dom0 - on my
> Xen servers, my dom0 is strictly for Xen management - I'm not running
> anything else in dom0 that would require large amounts of memory, memory
> buffers and caches, etc.
> 

Teo is running graphical stuff, X etc so it's a bit different..

-- Pasi

> 
> > 
> > If dom0 has all the memory at boot time, you need to balloon down
> dom0 
> > memory every time you create a new guest - this can (and will) cause
> 
> > problems with the dom0 linux kernel.
> > 
> > Linux calculates some internal parameters/buffers/values based on
> the
> > _boot time_ amount of memory. And when the amount of memory goes down
> to
> > only a small fraction of that while creating new guests bad things
> can
> > happen..
> > 
> > It still shouldn't crash though.. I bet your problem will get fixed
> when
> > you limit the dom0 memory to say dom0_mem=512M and reboot.
> > 
> > -- Pasi
> > 
> >>    Here's my xm info output after I have shutdown all the virtual
> machines.
> >> 
> >>    [root@fedora11-x86-64-host ~]# xm list
> >>    Name                                        ID   Mem VCPUs     
> State
> >>    Time(s)
> >>    Domain-0                                     0  2812     2    
> r-----
> >>    3242.5
> >>    [root@fedora11-x86-64-host ~]# xm info
> >>    host                   : fedora11-x86-64-host
> >>    release                : 2.6.30-rc3-enming.teo-tip
> >>    version                : #1 SMP Wed Aug 19 23:14:15 SGT 2009
> >>    machine                : x86_64
> >>    nr_cpus                : 2
> >>    nr_nodes               : 1
> >>    cores_per_socket       : 2
> >>    threads_per_core       : 1
> >>    cpu_mhz                : 2800
> >>    hw_caps                :
> >>   
> bfebfbff:20100800:00000000:00000140:0400e3bd:00000000:00000001:00000000
> >>    virt_caps              : hvm hvm_directio
> >>    total_memory           : 6039
> >>    free_memory            : 3124
> >>    node_to_cpu            : node0:0-1
> >>    node_to_memory         : node0:3124
> >>    xen_major              : 3
> >>    xen_minor              : 5
> >>    xen_extra              : -unstable
> >>    xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p
> hvm-3.0-x86_32
> >>    hvm-3.0-x86_32p hvm-3.0-x86_64
> >>    xen_scheduler          : credit
> >>    xen_pagesize           : 4096
> >>    platform_params        : virt_start=0xffff800000000000
> >>    xen_changeset          : Tue Sep 01 11:34:31 2009 +0100 
> > 20143:a7de5bd776ca
> >>    xen_commandline        : iommu=1
> >>    cc_compiler            : gcc version 4.4.1 20090725 (Red Hat
> 4.4.1-2)
> >>    (GCC)
> >>    cc_compile_by          : root
> >>    cc_compile_domain      : (none)
> >>    cc_compile_date        : Thu Sep 10 07:01:13 SGT 2009
> >>    xend_config_format     : 4
> >> 
> >>    --
> >>    Mr. Teo En Ming (Zhang Enming) Dip(Mechatronics)
> BEng(Hons)(Mechanical
> >>    Engineering)
> >>    Alma Maters:
> >>    (1) Singapore Polytechnic
> >>    (2) National University of Singapore
> >>    My Primary Blog:
> [1]http://teo-en-ming-aka-zhang-enming.blogspot.com
> >>    My Secondary Blog: [2]http://enmingteo.wordpress.com
> >>    My Youtube videos: [3]http://www.youtube.com/user/enmingteo
> >>    Email: [4]space.time.universe@gmail.com
> >>    Mobile Phone (Starhub Prepaid): +65-8369-2618
> >>    Street: Bedok Reservoir Road
> >>    Country: Singapore
> >> 
> >>    On Mon, Nov 9, 2009 at 7:54 PM, Pasi Kärkkäinen <[5]pasik@iki.fi>
> wrote:
> >> 
> >>      On Mon, Nov 09, 2009 at 06:52:37PM +0800, Mr. Teo En Ming
> (Zhang 
> > Enming)
> >>      wrote:
> >>      >    Hi,
> >>      >
> >>      >    Please watch this 4-minute video at
> >>      >    [1][6]http://www.youtube.com/watch?v=LbLaPpwNAx4
> >>      >
> >>      >    I have only started 3 HVM Linux guests with 1 GB ram each.
> I can't
> >>      start
> >>      >    the 4th HVM guest. If I attempt to start the 4th instance,
> it will
> >>      crash
> >>      >    dom0.
> >>      >
> >>      >    Are there anything in the xm dmesg output that could
> explain the
> >>      low limit
> >>      >    to the number of VMs that I could start before dom0
> becomes
> >>      unresponsive?
> >>      >
> >> 
> >>      Have you limited dom0 memory (by specifying dom0_mem=XMB option
> in
> >>      grub.conf for xen.gz) ?
> >> 
> >>      What does "xm info" say about free memory before starting any
> guests?
> >>      -- Pasi
> >> 
> >> References
> >> 
> >>    Visible links
> >>    1. http://teo-en-ming-aka-zhang-enming.blogspot.com/
> >>    2. http://enmingteo.wordpress.com/
> >>    3. http://www.youtube.com/user/enmingteo
> >>    4. mailto:space.time.universe@gmail.com
> >>    5. mailto:pasik@iki.fi
> >>    6. http://www.youtube.com/watch?v=LbLaPpwNAx4
> 
> 
> 
> --------
> This e-mail may contain confidential and privileged material for the sole use of the intended recipient.  If this email is not intended for you, or you are not responsible for the delivery of this message to the intended recipient, please note that this message may contain SEAKR Engineering (SEAKR) Privileged/Proprietary Information.  In such a case, you are strictly prohibited from downloading, photocopying, distributing or otherwise using this message, its contents or attachments in any way.  If you have received this message in error, please notify us immediately by replying to this e-mail and delete the message from your mailbox.  Information contained in this message that does not relate to the business of SEAKR is neither endorsed by nor attributable to SEAKR.

^ permalink raw reply	[flat|nested] 20+ messages in thread
end of thread, other threads:[~2009-11-09 15:39 UTC | newest]

Thread overview: 20+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2009-11-08 11:45 Max. PV and HVM Guests Mr. Teo En Ming (Zhang Enming)
2009-11-08 11:46 ` Mr. Teo En Ming (Zhang Enming)
2009-11-08 13:01 ` Moi meme
2009-11-09  8:17   ` [Xen-users] " Robert Dunkley
2009-11-09 10:52     ` Mr. Teo En Ming (Zhang Enming)
2009-11-09 11:53       ` Mr. Teo En Ming (Zhang Enming)
2009-11-09 11:54       ` [Xen-devel] " Pasi Kärkkäinen
2009-11-09 12:01         ` Re: [Xen-users] " Mr. Teo En Ming (Zhang Enming)
2009-11-09 12:05           ` Pasi Kärkkäinen
2009-11-09 12:14             ` Mr. Teo En Ming (Zhang Enming)
2009-11-09 12:18               ` Pasi Kärkkäinen
2009-11-09 13:10                 ` Mr. Teo En Ming (Zhang Enming)
2009-11-09 14:37                   ` [Xen-devel] " Mr. Teo En Ming (Zhang Enming)
2009-11-08 13:50 ` Pasi Kärkkäinen
2009-11-08 16:30   ` [Xen-devel] " Grant McWilliams
2009-11-09  3:35   ` Mr. Teo En Ming (Zhang Enming)
2009-11-09 15:06 Re: [Xen-users] " Nick Couchman
2009-11-09 15:18 ` [Xen-devel] " Keir Fraser
2009-11-09 15:29   ` Re: [Xen-users] " Nick Couchman
2009-11-09 15:17 [Xen-devel] " Pasi Kärkkäinen
2009-11-09 15:24 ` Keir Fraser
2009-11-09 15:27   ` Re: [Xen-users] " Pasi Kärkkäinen
2009-11-09 15:39 ` Dan Magenheimer

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.