failed to start centos 5 domU with maxmem=30000

failed to start centos 5 domU with maxmem=30000

[Vasiliy G Tolstov]

Hello. I'm try to start centos domU pvm under sles 11 sp1 xen kernel
with memory=512 and maxmem=30000. Centos doing boot and freeze.
If i'm try to use maxmem=15000 all works.

Can You tell me what's wrong?

dom0:
xen24:~ # xm info
host                   : xen24
release                : 2.6.32.12-0.7-xen
version                : #1 SMP 2010-05-20 11:14:20 +0200
machine                : x86_64
nr_cpus                : 16
nr_nodes               : 2
cores_per_socket       : 4
threads_per_core       : 2
cpu_mhz                : 2266
hw_caps                :
bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000
virt_caps              : hvm hvm_directio
total_memory           : 65455
free_memory            : 62133
free_cpus              : 0
max_free_memory        : 62133
max_para_memory        : 62129
max_hvm_memory         : 61975
node_to_cpu            : node0:0-3,8-11
                         node1:4-7,12-15
node_to_memory         : node0:30485
                         node1:31647
node_to_dma32_mem      : node0:2995
                         node1:0
max_node_id            : 1
xen_major              : 4
xen_minor              : 0
xen_extra              : .0_21091_04-0.2
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : 21091
xen_commandline        : loglvl=all noreboot dom0_mem=2048M
dom0_max_vcpus=2 dom0_vcpus_pin console=vga vga=current
cc_compiler            : gcc version 4.3.4 [gcc-4_3-branch revision
152973] (SUSE Linux)
cc_compile_by          : abuild
cc_compile_domain      : 
cc_compile_date        : Thu May 20 11:44:41 UTC 2010
xend_config_format     : 4




-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

Re: failed to start centos 5 domU with maxmem=30000

[Pasi Kärkkäinen]

On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote:
> Hello. I'm try to start centos domU pvm under sles 11 sp1 xen kernel
> with memory=512 and maxmem=30000. Centos doing boot and freeze.
> If i'm try to use maxmem=15000 all works.
> 
> Can You tell me what's wrong?
> 

Is the centos PV domU 32bit or 64bit ?

-- Pasi


> dom0:
> xen24:~ # xm info
> host                   : xen24
> release                : 2.6.32.12-0.7-xen
> version                : #1 SMP 2010-05-20 11:14:20 +0200
> machine                : x86_64
> nr_cpus                : 16
> nr_nodes               : 2
> cores_per_socket       : 4
> threads_per_core       : 2
> cpu_mhz                : 2266
> hw_caps                :
> bfebfbff:28100800:00000000:00001b40:009ce3bd:00000000:00000001:00000000
> virt_caps              : hvm hvm_directio
> total_memory           : 65455
> free_memory            : 62133
> free_cpus              : 0
> max_free_memory        : 62133
> max_para_memory        : 62129
> max_hvm_memory         : 61975
> node_to_cpu            : node0:0-3,8-11
>                          node1:4-7,12-15
> node_to_memory         : node0:30485
>                          node1:31647
> node_to_dma32_mem      : node0:2995
>                          node1:0
> max_node_id            : 1
> xen_major              : 4
> xen_minor              : 0
> xen_extra              : .0_21091_04-0.2
> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
> hvm-3.0-x86_32p hvm-3.0-x86_64 
> xen_scheduler          : credit
> xen_pagesize           : 4096
> platform_params        : virt_start=0xffff800000000000
> xen_changeset          : 21091
> xen_commandline        : loglvl=all noreboot dom0_mem=2048M
> dom0_max_vcpus=2 dom0_vcpus_pin console=vga vga=current
> cc_compiler            : gcc version 4.3.4 [gcc-4_3-branch revision
> 152973] (SUSE Linux)
> cc_compile_by          : abuild
> cc_compile_domain      : 
> cc_compile_date        : Thu May 20 11:44:41 UTC 2010
> xend_config_format     : 4
> 
> 
> 
> 
> -- 
> Vasiliy G Tolstov <v.tolstov@selfip.ru>
> Selfip.Ru
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: failed to start centos 5 domU with maxmem=30000

[Vasiliy G Tolstov]

В Чтв, 12/08/2010 в 14:37 +0300, Pasi Kärkkäinen пишет:
> On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote:
> > Hello. I'm try to start centos domU pvm under sles 11 sp1 xen kernel
> > with memory=512 and maxmem=30000. Centos doing boot and freeze.
> > If i'm try to use maxmem=15000 all works.
> > 
> > Can You tell me what's wrong?
> > 
> 
> Is the centos PV domU 32bit or 64bit ?
> 
> -- Pasi

64 bit kernel-xen.

I found this in xen sources:

/* Simple continuous piecewiese linear function:
         *  max MiB -> min MiB  gradient
         *       0         0
         *      16        16
         *      32        24
         *     128        72    (1/2)
         *     512       168    (1/4)
         *    2048       360    (1/8)
         *    8192       552    (1/32)
         *   32768      1320
         *  131072      4392
         */

Why this is not provided in documentation or on web site?


-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

RE: failed to start centos 5 domU with maxmem=30000

[Dan Magenheimer]

> From: Vasiliy G Tolstov [mailto:v.tolstov@selfip.ru]
> Sent: Thursday, August 12, 2010 5:38 AM
> To: Pasi Kärkkäinen
> Cc: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] failed to start centos 5 domU with
> maxmem=30000
> 
> В Чтв, 12/08/2010 в 14:37 +0300, Pasi Kärkkäinen пишет:
> > On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote:
> > > Hello. I'm try to start centos domU pvm under sles 11 sp1 xen
> kernel
> > > with memory=512 and maxmem=30000. Centos doing boot and freeze.
> > > If i'm try to use maxmem=15000 all works.
> > >
> > > Can You tell me what's wrong?
> > >
> >
> > Is the centos PV domU 32bit or 64bit ?
> 
> 64 bit kernel-xen.
> 
> I found this in xen sources:
> 
> /* Simple continuous piecewiese linear function:
>          *  max MiB -> min MiB  gradient
>          *       0         0
>          *      16        16
>          *      32        24
>          *     128        72    (1/2)
>          *     512       168    (1/4)
>          *    2048       360    (1/8)
>          *    8192       552    (1/32)
>          *   32768      1320
>          *  131072      4392
>          */
> 
> Why this is not provided in documentation or on web site?

Hi Vasily --

This function limits how far memory can be reduced when
ballooning a guest (including dom0).  It is only a heuristic
but is intended to take into account the various overheads
a guest Linux kernel requires to manage memory to avoid
out-of-memory conditions.

But I think you are correct... the same (or similar)
function should be published as it also serves as a
guideline for the ratio between memory= and maxmem=
parameters when creating a guest:  If the ratio
of maxmem divided by memory is too high, the guest
will not even boot.

I am curious as to why you would specify memory= so
much smaller than maxmem=.  Are you trying to overcommit
memory for guests that are often idle but sometimes use
a very large amount of memory?

Thanks,
Dan

RE: failed to start centos 5 domU with maxmem=30000

[Vasiliy G Tolstov]

В Чтв, 12/08/2010 в 05:17 -0700, Dan Magenheimer пишет:
> > From: Vasiliy G Tolstov [mailto:v.tolstov@selfip.ru]
> > Sent: Thursday, August 12, 2010 5:38 AM
> > To: Pasi Kärkkäinen
> > Cc: xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] failed to start centos 5 domU with
> > maxmem=30000
> > 
> > В Чтв, 12/08/2010 в 14:37 +0300, Pasi Kärkkäinen пишет:
> > > On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote:
> > > > Hello. I'm try to start centos domU pvm under sles 11 sp1 xen
> > kernel
> > > > with memory=512 and maxmem=30000. Centos doing boot and freeze.
> > > > If i'm try to use maxmem=15000 all works.
> > > >
> > > > Can You tell me what's wrong?
> > > >
> > >
> > > Is the centos PV domU 32bit or 64bit ?
> > 
> > 64 bit kernel-xen.
> > 
> > I found this in xen sources:
> > 
> > /* Simple continuous piecewiese linear function:
> >          *  max MiB -> min MiB  gradient
> >          *       0         0
> >          *      16        16
> >          *      32        24
> >          *     128        72    (1/2)
> >          *     512       168    (1/4)
> >          *    2048       360    (1/8)
> >          *    8192       552    (1/32)
> >          *   32768      1320
> >          *  131072      4392
> >          */
> > 
> > Why this is not provided in documentation or on web site?
> 
> Hi Vasily --
> 
> This function limits how far memory can be reduced when
> ballooning a guest (including dom0).  It is only a heuristic
> but is intended to take into account the various overheads
> a guest Linux kernel requires to manage memory to avoid
> out-of-memory conditions.
> 
> But I think you are correct... the same (or similar)
> function should be published as it also serves as a
> guideline for the ratio between memory= and maxmem=
> parameters when creating a guest:  If the ratio
> of maxmem divided by memory is too high, the guest
> will not even boot.
> 

Is that possible to use memory=32 and maxmem=60000 ?

> I am curious as to why you would specify memory= so
> much smaller than maxmem=.  Are you trying to overcommit
> memory for guests that are often idle but sometimes use
> a very large amount of memory?

We want to provide ability to use small as possible memory if the guest
is idle. And much as possible when the guest under heavy load.

Thank You.
-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

RE: failed to start centos 5 domU with maxmem=30000

[Dan Magenheimer]

> > > Why this is not provided in documentation or on web site?
> >
> > Hi Vasily --
> >
> > This function limits how far memory can be reduced when
> > ballooning a guest (including dom0).  It is only a heuristic
> > but is intended to take into account the various overheads
> > a guest Linux kernel requires to manage memory to avoid
> > out-of-memory conditions.
> >
> > But I think you are correct... the same (or similar)
> > function should be published as it also serves as a
> > guideline for the ratio between memory= and maxmem=
> > parameters when creating a guest:  If the ratio
> > of maxmem divided by memory is too high, the guest
> > will not even boot.
> 
> Is that possible to use memory=32 and maxmem=60000 ?

I think the answer is no.  I believe there is a kernel
data structure for each 4K page in physical memory
I don't remember the size of this data structure, but
assume it is 4 bytes.  That means that for 1GB of
physical memory (as specified by maxmem) this data
structure requires 1MB of physical memory just to track
the 1GB.  So for your maxmem=60000, memory=60 would
only be enough for this one kernel data structure (and
the kernel requires many other data structures to
be functional).

Daniel Kiper is developing virtual hotplug memory for Xen
guests.  This may be a good use case for it.
 
> > I am curious as to why you would specify memory= so
> > much smaller than maxmem=.  Are you trying to overcommit
> > memory for guests that are often idle but sometimes use
> > a very large amount of memory?
> 
> We want to provide ability to use small as possible memory if the guest
> is idle. And much as possible when the guest under heavy load.

This is a much harder problem than it seems.  You may want
to look at some of the presentations on Transcendent Memory
that explain why it is hard.  (Most can be found at
http://oss.oracle.com/projects/tmem )

Dan

Re: failed to start centos 5 domU with maxmem=30000

[Jeremy Fitzhardinge]

  On 08/12/2010 05:24 AM, Vasiliy G Tolstov wrote:
> В Чтв, 12/08/2010 в 05:17 -0700, Dan Magenheimer пишет:
>>> From: Vasiliy G Tolstov [mailto:v.tolstov@selfip.ru]
>>> Sent: Thursday, August 12, 2010 5:38 AM
>>> To: Pasi Kärkkäinen
>>> Cc: xen-devel@lists.xensource.com
>>> Subject: Re: [Xen-devel] failed to start centos 5 domU with
>>> maxmem=30000
>>>
>>> В Чтв, 12/08/2010 в 14:37 +0300, Pasi Kärkkäinen пишет:
>>>> On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote:
>>>>> Hello. I'm try to start centos domU pvm under sles 11 sp1 xen
>>> kernel
>>>>> with memory=512 and maxmem=30000. Centos doing boot and freeze.
>>>>> If i'm try to use maxmem=15000 all works.
>>>>>
>>>>> Can You tell me what's wrong?
>>>>>
>>>> Is the centos PV domU 32bit or 64bit ?
>>> 64 bit kernel-xen.
>>>
>>> I found this in xen sources:
>>>
>>> /* Simple continuous piecewiese linear function:
>>>           *  max MiB ->  min MiB  gradient
>>>           *       0         0
>>>           *      16        16
>>>           *      32        24
>>>           *     128        72    (1/2)
>>>           *     512       168    (1/4)
>>>           *    2048       360    (1/8)
>>>           *    8192       552    (1/32)
>>>           *   32768      1320
>>>           *  131072      4392
>>>           */
>>>
>>> Why this is not provided in documentation or on web site?
>> Hi Vasily --
>>
>> This function limits how far memory can be reduced when
>> ballooning a guest (including dom0).  It is only a heuristic
>> but is intended to take into account the various overheads
>> a guest Linux kernel requires to manage memory to avoid
>> out-of-memory conditions.
>>
>> But I think you are correct... the same (or similar)
>> function should be published as it also serves as a
>> guideline for the ratio between memory= and maxmem=
>> parameters when creating a guest:  If the ratio
>> of maxmem divided by memory is too high, the guest
>> will not even boot.
>>
> Is that possible to use memory=32 and maxmem=60000 ?

A factor of ~2000 between smallest and largest size won't work.  The 
problem is that lots of kernel structures are scaled with the amount of 
boot-time memory, and they'll simply be overloaded when you add more 
memory.  For example, the page structure array for 60GB needs ~15.3 
million entries, which at 64 bytes per entry will take ~980MB...

>> I am curious as to why you would specify memory= so
>> much smaller than maxmem=.  Are you trying to overcommit
>> memory for guests that are often idle but sometimes use
>> a very large amount of memory?
> We want to provide ability to use small as possible memory if the guest
> is idle. And much as possible when the guest under heavy load.

This is exactly the use-case for the hotplug memory stuff Daniel is 
working on - which I guess is why you've been testing it.

     J

Re: failed to start centos 5 domU with maxmem=30000

[Vasiliy G Tolstov]

В Чтв, 12/08/2010 в 11:34 -0700, Jeremy Fitzhardinge пишет:
> On 08/12/2010 05:24 AM, Vasiliy G Tolstov wrote:
> > В Чтв, 12/08/2010 в 05:17 -0700, Dan Magenheimer пишет:
> >>> From: Vasiliy G Tolstov [mailto:v.tolstov@selfip.ru]
> >>> Sent: Thursday, August 12, 2010 5:38 AM
> >>> To: Pasi Kärkkäinen
> >>> Cc: xen-devel@lists.xensource.com
> >>> Subject: Re: [Xen-devel] failed to start centos 5 domU with
> >>> maxmem=30000
> >>>
> >>> В Чтв, 12/08/2010 в 14:37 +0300, Pasi Kärkkäinen пишет:
> >>>> On Thu, Aug 12, 2010 at 02:24:05PM +0400, Vasiliy G Tolstov wrote:
> >>>>> Hello. I'm try to start centos domU pvm under sles 11 sp1 xen
> >>> kernel
> >>>>> with memory=512 and maxmem=30000. Centos doing boot and freeze.
> >>>>> If i'm try to use maxmem=15000 all works.
> >>>>>
> >>>>> Can You tell me what's wrong?
> >>>>>
> >>>> Is the centos PV domU 32bit or 64bit ?
> >>> 64 bit kernel-xen.
> >>>
> >>> I found this in xen sources:
> >>>
> >>> /* Simple continuous piecewiese linear function:
> >>>           *  max MiB ->  min MiB  gradient
> >>>           *       0         0
> >>>           *      16        16
> >>>           *      32        24
> >>>           *     128        72    (1/2)
> >>>           *     512       168    (1/4)
> >>>           *    2048       360    (1/8)
> >>>           *    8192       552    (1/32)
> >>>           *   32768      1320
> >>>           *  131072      4392
> >>>           */
> >>>
> >>> Why this is not provided in documentation or on web site?
> >> Hi Vasily --
> >>
> >> This function limits how far memory can be reduced when
> >> ballooning a guest (including dom0).  It is only a heuristic
> >> but is intended to take into account the various overheads
> >> a guest Linux kernel requires to manage memory to avoid
> >> out-of-memory conditions.
> >>
> >> But I think you are correct... the same (or similar)
> >> function should be published as it also serves as a
> >> guideline for the ratio between memory= and maxmem=
> >> parameters when creating a guest:  If the ratio
> >> of maxmem divided by memory is too high, the guest
> >> will not even boot.
> >>
> > Is that possible to use memory=32 and maxmem=60000 ?
> 
> A factor of ~2000 between smallest and largest size won't work.  The 
> problem is that lots of kernel structures are scaled with the amount of 
> boot-time memory, and they'll simply be overloaded when you add more 
> memory.  For example, the page structure array for 60GB needs ~15.3 
> million entries, which at 64 bytes per entry will take ~980MB...
> 

Ok. Thank You. Bu can some one provide correct ratio for memory and
maxmemory that others not to make mistakes and not try to post bugs on
that? =)

> >> I am curious as to why you would specify memory= so
> >> much smaller than maxmem=.  Are you trying to overcommit
> >> memory for guests that are often idle but sometimes use
> >> a very large amount of memory?
> > We want to provide ability to use small as possible memory if the guest
> > is idle. And much as possible when the guest under heavy load.
> 
> This is exactly the use-case for the hotplug memory stuff Daniel is 
> working on - which I guess is why you've been testing it.

This is problem, because a'm using centos, debian, ubuntu, opensuse and
other systems in domU, but i read that tmem support needed in dom0 and
domU. Debian and Ubuntu i think, does not support this... But i'm try.


-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

RE: failed to start centos 5 domU with maxmem=30000

[Vasiliy G Tolstov]

В Чтв, 12/08/2010 в 11:00 -0700, Dan Magenheimer пишет:
> > > > Why this is not provided in documentation or on web site?
> > >
> > > Hi Vasily --
> > >
> > > This function limits how far memory can be reduced when
> > > ballooning a guest (including dom0).  It is only a heuristic
> > > but is intended to take into account the various overheads
> > > a guest Linux kernel requires to manage memory to avoid
> > > out-of-memory conditions.
> > >
> > > But I think you are correct... the same (or similar)
> > > function should be published as it also serves as a
> > > guideline for the ratio between memory= and maxmem=
> > > parameters when creating a guest:  If the ratio
> > > of maxmem divided by memory is too high, the guest
> > > will not even boot.
> > 
> > Is that possible to use memory=32 and maxmem=60000 ?
> 
> I think the answer is no.  I believe there is a kernel
> data structure for each 4K page in physical memory
> I don't remember the size of this data structure, but
> assume it is 4 bytes.  That means that for 1GB of
> physical memory (as specified by maxmem) this data
> structure requires 1MB of physical memory just to track
> the 1GB.  So for your maxmem=60000, memory=60 would
> only be enough for this one kernel data structure (and
> the kernel requires many other data structures to
> be functional).
> 

Ok. Thank's.

> Daniel Kiper is developing virtual hotplug memory for Xen
> guests.  This may be a good use case for it.
>  

I'm try for my systems. Thank's

> > > I am curious as to why you would specify memory= so
> > > much smaller than maxmem=.  Are you trying to overcommit
> > > memory for guests that are often idle but sometimes use
> > > a very large amount of memory?
> > 
> > We want to provide ability to use small as possible memory if the guest
> > is idle. And much as possible when the guest under heavy load.
> 
> This is a much harder problem than it seems.  You may want
> to look at some of the presentations on Transcendent Memory
> that explain why it is hard.  (Most can be found at
> http://oss.oracle.com/projects/tmem )


Need to support in domU... this is bad. But i'm try it. 


-- 
Vasiliy G Tolstov <v.tolstov@selfip.ru>
Selfip.Ru

RE: failed to start centos 5 domU with maxmem=30000

[Dan Magenheimer]

> This is problem, because a'm using centos, debian, ubuntu, opensuse and
> other systems in domU, but i read that tmem support needed in dom0 and
> domU. Debian and Ubuntu i think, does not support this... But i'm try.

I have kernel rpm's for some versions of Oracle Enterprise Linux
that might work on the same version of centos.  I'm told that an
earlier version of the tmem patch (without self-ballooning) was
added to opensuse.

Due to upstream Linux kernel change requests, the tmem Linux patches
continue to change... cosmetic changes, all core functionality
is the same.  I've been focused on that rather than on keeping
the patches for older versions up to date.  But if you are eager
to try tmem on a particular distro and you have experience in
patching kernels for that distro, I would be happy to help.

Also, tmem does not require dom0 kernel changes, but the toolset
that create domains must be aware of tmem because tmem introduces
the concept of "freeable" memory... since tmem eventually uses all
free memory in the system, creating a domain may require freeable
memory to be freed and the tools need to know how to do that.
Xend (xm) works fine as of 4.0.0.

Again, efficient utilization of memory is a very hard problem.
"Simple" solutions that have been tried to date (such as externally
ballooning all guests from the host, and page sharing with host
swapping) have unacceptable side-effects.  Tmem uses paravirtualization
to achieve a much more efficient solution to the problem, but it
requires guest changes.

Thanks,
Dan