Possible regression: XEN 4.0.0 total_memory decrease

Possible regression: XEN 4.0.0 total_memory decrease

[Alex Williams]

Hello,

I noticed a significant *decrease* (150MB) in "total_memory" on a host 
build using XEN 4.0.0 with kernel 2.6.18.

Here are the details:

Server = 4096MB RAM
XEN 3.4.2 + kernel 2.6.18: xm info | grep total_memory = 4084
XEN 4.0.0 + kernel 2.6.18: xm info | grep total_memory = 3946

The exact same kernel config file was used for both builds. I also made 
sure there is absolutely no difference on the hosts (while the builds 
are created on a different machine).

This seems like a regression, unless someone can provide suggestions on 
how to correct this?

Thanks.


-- 
Alexander Williams
IT Architecture Specialist
Spécialiste de l'Architecture TI
http://www.iWeb.com/

Re: Possible regression: XEN 4.0.0 total_memory decrease

[Keir Fraser]

Are the hypervisor and dom0 kernel 64-bit or 32-bit? Xen prints a message
"System RAM: xxx" early during boot. Do these differ significantly for you
between 3.4.2 and 4.0.0?

 K.

On 15/04/2010 16:54, "Alex Williams" <awilliams@iweb.com> wrote:

> Hello,
> 
> I noticed a significant *decrease* (150MB) in "total_memory" on a host
> build using XEN 4.0.0 with kernel 2.6.18.
> 
> Here are the details:
> 
> Server = 4096MB RAM
> XEN 3.4.2 + kernel 2.6.18: xm info | grep total_memory = 4084
> XEN 4.0.0 + kernel 2.6.18: xm info | grep total_memory = 3946
> 
> The exact same kernel config file was used for both builds. I also made
> sure there is absolutely no difference on the hosts (while the builds
> are created on a different machine).
> 
> This seems like a regression, unless someone can provide suggestions on
> how to correct this?
> 
> Thanks.
> 

Re: Possible regression: XEN 4.0.0 total_memory decrease

[Alex Williams]

Hello,

Both builds run a 64-bit Xen kernel and 64-bit Dom0 kernel.

(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 -> 
0xffffffff805d388c

As for the RAM:

Xen 3.4.2: (XEN) System RAM: 3946MB (4041012kB)
Xen 4.0.0: (XEN) System RAM: 4084MB (4182064kB)

I have also verified with only 2048MB installed in the system. Here are 
the results:

Xen 3.4.2: (XEN) System RAM: 2036MB (2084912kB)
Xen 4.0.0: (XEN) System RAM: 1898MB (1943860kB)

Still 150MB RAM missing... it appears this problem is static and 
independant of the amount of RAM installed in the system.

Regards,



Keir Fraser wrote:
> Are the hypervisor and dom0 kernel 64-bit or 32-bit? Xen prints a message
> "System RAM: xxx" early during boot. Do these differ significantly for you
> between 3.4.2 and 4.0.0?
> 
>  K.
> 
> On 15/04/2010 16:54, "Alex Williams" <awilliams@iweb.com> wrote:
> 
>> Hello,
>>
>> I noticed a significant *decrease* (150MB) in "total_memory" on a host
>> build using XEN 4.0.0 with kernel 2.6.18.
>>
>> Here are the details:
>>
>> Server = 4096MB RAM
>> XEN 3.4.2 + kernel 2.6.18: xm info | grep total_memory = 4084
>> XEN 4.0.0 + kernel 2.6.18: xm info | grep total_memory = 3946
>>
>> The exact same kernel config file was used for both builds. I also made
>> sure there is absolutely no difference on the hosts (while the builds
>> are created on a different machine).
>>
>> This seems like a regression, unless someone can provide suggestions on
>> how to correct this?
>>
>> Thanks.
>>
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

-- 
Alexander Williams
IT Architecture Specialist
Spécialiste de l'Architecture TI
http://www.iWeb.com/

Re: Possible regression: XEN 4.0.0 total_memory decrease

[Alex Williams]

Hi,

I also want to add, my build process is practically automated. The only 
difference I had to make for 4.0.0 was:

- Install Debian package dependencies: uuid-dev iasl gcc-multilib

Let me know if you need more information.

Thanks.


Alex Williams wrote:
> Hello,
> 
> Both builds run a 64-bit Xen kernel and 64-bit Dom0 kernel.
> 
> (XEN)  Xen  kernel: 64-bit, lsb, compat32
> (XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 -> 
> 0xffffffff805d388c
> 
> As for the RAM:
> 
> Xen 3.4.2: (XEN) System RAM: 3946MB (4041012kB)
> Xen 4.0.0: (XEN) System RAM: 4084MB (4182064kB)
> 
> I have also verified with only 2048MB installed in the system. Here are 
> the results:
> 
> Xen 3.4.2: (XEN) System RAM: 2036MB (2084912kB)
> Xen 4.0.0: (XEN) System RAM: 1898MB (1943860kB)
> 
> Still 150MB RAM missing... it appears this problem is static and 
> independant of the amount of RAM installed in the system.
> 
> Regards,
> 
> 
> 
> Keir Fraser wrote:
>> Are the hypervisor and dom0 kernel 64-bit or 32-bit? Xen prints a message
>> "System RAM: xxx" early during boot. Do these differ significantly for 
>> you
>> between 3.4.2 and 4.0.0?
>>
>>  K.
>>
>> On 15/04/2010 16:54, "Alex Williams" <awilliams@iweb.com> wrote:
>>
>>> Hello,
>>>
>>> I noticed a significant *decrease* (150MB) in "total_memory" on a host
>>> build using XEN 4.0.0 with kernel 2.6.18.
>>>
>>> Here are the details:
>>>
>>> Server = 4096MB RAM
>>> XEN 3.4.2 + kernel 2.6.18: xm info | grep total_memory = 4084
>>> XEN 4.0.0 + kernel 2.6.18: xm info | grep total_memory = 3946
>>>
>>> The exact same kernel config file was used for both builds. I also made
>>> sure there is absolutely no difference on the hosts (while the builds
>>> are created on a different machine).
>>>
>>> This seems like a regression, unless someone can provide suggestions on
>>> how to correct this?
>>>
>>> Thanks.
>>>
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>>
> 

-- 
Alexander Williams
IT Architecture Specialist
Spécialiste de l'Architecture TI
http://www.iWeb.com/

Re: Possible regression: XEN 4.0.0 total_memory decrease

[Keir Fraser]

Ah, hm, I think this is a bug in total memory reporting only. I think the
memory is still there in the free pools though. :-) How does xm info's
free_memory compare in the two cases? I bet it's a lot closer than 150MB
(assuming dom0 is allocated the same amount of memory in both cases, e.g.,
by explicitly specifying dom0_mem=xxx on Xen command line).

 -- Keir

On 15/04/2010 18:33, "Alex Williams" <awilliams@iweb.com> wrote:

> Hello,
> 
> Both builds run a 64-bit Xen kernel and 64-bit Dom0 kernel.
> 
> (XEN)  Xen  kernel: 64-bit, lsb, compat32
> (XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 ->
> 0xffffffff805d388c
> 
> As for the RAM:
> 
> Xen 3.4.2: (XEN) System RAM: 3946MB (4041012kB)
> Xen 4.0.0: (XEN) System RAM: 4084MB (4182064kB)
> 
> I have also verified with only 2048MB installed in the system. Here are
> the results:
> 
> Xen 3.4.2: (XEN) System RAM: 2036MB (2084912kB)
> Xen 4.0.0: (XEN) System RAM: 1898MB (1943860kB)
> 
> Still 150MB RAM missing... it appears this problem is static and
> independant of the amount of RAM installed in the system.
> 
> Regards,
> 
> 
> 
> Keir Fraser wrote:
>> Are the hypervisor and dom0 kernel 64-bit or 32-bit? Xen prints a message
>> "System RAM: xxx" early during boot. Do these differ significantly for you
>> between 3.4.2 and 4.0.0?
>> 
>>  K.
>> 
>> On 15/04/2010 16:54, "Alex Williams" <awilliams@iweb.com> wrote:
>> 
>>> Hello,
>>> 
>>> I noticed a significant *decrease* (150MB) in "total_memory" on a host
>>> build using XEN 4.0.0 with kernel 2.6.18.
>>> 
>>> Here are the details:
>>> 
>>> Server = 4096MB RAM
>>> XEN 3.4.2 + kernel 2.6.18: xm info | grep total_memory = 4084
>>> XEN 4.0.0 + kernel 2.6.18: xm info | grep total_memory = 3946
>>> 
>>> The exact same kernel config file was used for both builds. I also made
>>> sure there is absolutely no difference on the hosts (while the builds
>>> are created on a different machine).
>>> 
>>> This seems like a regression, unless someone can provide suggestions on
>>> how to correct this?
>>> 
>>> Thanks.
>>> 
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>> 

Re: Possible regression: XEN 4.0.0 total_memory decrease

[Keir Fraser]

I've fixed this regression as xen-unstable:21190 and xen-4.0-testing:21114.
The fix will appear in Xen 4.0.1.

 Thanks,
 Keir

On 15/04/2010 19:05, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

> Ah, hm, I think this is a bug in total memory reporting only. I think the
> memory is still there in the free pools though. :-) How does xm info's
> free_memory compare in the two cases? I bet it's a lot closer than 150MB
> (assuming dom0 is allocated the same amount of memory in both cases, e.g.,
> by explicitly specifying dom0_mem=xxx on Xen command line).
> 
>  -- Keir
> 
> On 15/04/2010 18:33, "Alex Williams" <awilliams@iweb.com> wrote:
> 
>> Hello,
>> 
>> Both builds run a 64-bit Xen kernel and 64-bit Dom0 kernel.
>> 
>> (XEN)  Xen  kernel: 64-bit, lsb, compat32
>> (XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 ->
>> 0xffffffff805d388c
>> 
>> As for the RAM:
>> 
>> Xen 3.4.2: (XEN) System RAM: 3946MB (4041012kB)
>> Xen 4.0.0: (XEN) System RAM: 4084MB (4182064kB)
>> 
>> I have also verified with only 2048MB installed in the system. Here are
>> the results:
>> 
>> Xen 3.4.2: (XEN) System RAM: 2036MB (2084912kB)
>> Xen 4.0.0: (XEN) System RAM: 1898MB (1943860kB)
>> 
>> Still 150MB RAM missing... it appears this problem is static and
>> independant of the amount of RAM installed in the system.
>> 
>> Regards,
>> 
>> 
>> 
>> Keir Fraser wrote:
>>> Are the hypervisor and dom0 kernel 64-bit or 32-bit? Xen prints a message
>>> "System RAM: xxx" early during boot. Do these differ significantly for you
>>> between 3.4.2 and 4.0.0?
>>> 
>>>  K.
>>> 
>>> On 15/04/2010 16:54, "Alex Williams" <awilliams@iweb.com> wrote:
>>> 
>>>> Hello,
>>>> 
>>>> I noticed a significant *decrease* (150MB) in "total_memory" on a host
>>>> build using XEN 4.0.0 with kernel 2.6.18.
>>>> 
>>>> Here are the details:
>>>> 
>>>> Server = 4096MB RAM
>>>> XEN 3.4.2 + kernel 2.6.18: xm info | grep total_memory = 4084
>>>> XEN 4.0.0 + kernel 2.6.18: xm info | grep total_memory = 3946
>>>> 
>>>> The exact same kernel config file was used for both builds. I also made
>>>> sure there is absolutely no difference on the hosts (while the builds
>>>> are created on a different machine).
>>>> 
>>>> This seems like a regression, unless someone can provide suggestions on
>>>> how to correct this?
>>>> 
>>>> Thanks.
>>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>> 
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Possible regression: XEN 4.0.0 total_memory decrease

[Alex Williams]

Yes you're correct! Both hosts have dom0_mem=512M set.

Xen 3.4.2: free_memory            : 3514
Xen 4.0.0: free_memory            : 3523

So now it appears to be a free memory INCREASE caused by this 'bug' ? ;)

Thank you!!!


Keir Fraser wrote:
> Ah, hm, I think this is a bug in total memory reporting only. I think the
> memory is still there in the free pools though. :-) How does xm info's
> free_memory compare in the two cases? I bet it's a lot closer than 150MB
> (assuming dom0 is allocated the same amount of memory in both cases, e.g.,
> by explicitly specifying dom0_mem=xxx on Xen command line).
> 
>  -- Keir
> 
> On 15/04/2010 18:33, "Alex Williams" <awilliams@iweb.com> wrote:
> 
>> Hello,
>>
>> Both builds run a 64-bit Xen kernel and 64-bit Dom0 kernel.
>>
>> (XEN)  Xen  kernel: 64-bit, lsb, compat32
>> (XEN)  Dom0 kernel: 64-bit, lsb, paddr 0xffffffff80200000 ->
>> 0xffffffff805d388c
>>
>> As for the RAM:
>>
>> Xen 3.4.2: (XEN) System RAM: 3946MB (4041012kB)
>> Xen 4.0.0: (XEN) System RAM: 4084MB (4182064kB)
>>
>> I have also verified with only 2048MB installed in the system. Here are
>> the results:
>>
>> Xen 3.4.2: (XEN) System RAM: 2036MB (2084912kB)
>> Xen 4.0.0: (XEN) System RAM: 1898MB (1943860kB)
>>
>> Still 150MB RAM missing... it appears this problem is static and
>> independant of the amount of RAM installed in the system.
>>
>> Regards,
>>
>>
>>
>> Keir Fraser wrote:
>>> Are the hypervisor and dom0 kernel 64-bit or 32-bit? Xen prints a message
>>> "System RAM: xxx" early during boot. Do these differ significantly for you
>>> between 3.4.2 and 4.0.0?
>>>
>>>  K.
>>>
>>> On 15/04/2010 16:54, "Alex Williams" <awilliams@iweb.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I noticed a significant *decrease* (150MB) in "total_memory" on a host
>>>> build using XEN 4.0.0 with kernel 2.6.18.
>>>>
>>>> Here are the details:
>>>>
>>>> Server = 4096MB RAM
>>>> XEN 3.4.2 + kernel 2.6.18: xm info | grep total_memory = 4084
>>>> XEN 4.0.0 + kernel 2.6.18: xm info | grep total_memory = 3946
>>>>
>>>> The exact same kernel config file was used for both builds. I also made
>>>> sure there is absolutely no difference on the hosts (while the builds
>>>> are created on a different machine).
>>>>
>>>> This seems like a regression, unless someone can provide suggestions on
>>>> how to correct this?
>>>>
>>>> Thanks.
>>>>

-- 
Alexander Williams
IT Architecture Specialist
Spécialiste de l'Architecture TI
http://www.iWeb.com/

Re: Possible regression: XEN 4.0.0 total_memory decrease

[Keir Fraser]

On 15/04/2010 19:26, "Alex Williams" <awilliams@iweb.com> wrote:

> Yes you're correct! Both hosts have dom0_mem=512M set.
> 
> Xen 3.4.2: free_memory            : 3514
> Xen 4.0.0: free_memory            : 3523
> 
> So now it appears to be a free memory INCREASE caused by this 'bug' ? ;)

The reason for the increase is that for various technical reasons we used to
carve off 16MB of memory into a separate memory allocator used for Xen's own
internal book-keeping structures. We no longer have that and so that 16MB is
now part of the ordinary allocation pool. So you see 9MB extra in that pool,
which is that 16MB minus the book-keeping allocations (about 7MB of them)
which now occur from the unified pool.

 -- Keir

> Thank you!!!

Re: Possible regression: XEN 4.0.0 total_memory decrease

[Jan Beulich]

>>> Keir Fraser <keir.fraser@eu.citrix.com> 15.04.10 20:23 >>>
>I've fixed this regression as xen-unstable:21190 and xen-4.0-testing:21114.
>The fix will appear in Xen 4.0.1.

That seems wrong to me - I specifically changed the accounting so
that pieces not used from the E820 map (which can no longer be cut
off in e820.c, as that code doesn't know *where* to cut off) won't
get reported as available memory. The real question is where (for
the non-cut-off case) the two calculations differ.

Jan

Re: Possible regression: XEN 4.0.0 total_memory decrease

[Keir Fraser]

On 16/04/2010 08:53, "Jan Beulich" <JBeulich@novell.com> wrote:

>>>> Keir Fraser <keir.fraser@eu.citrix.com> 15.04.10 20:23 >>>
>> I've fixed this regression as xen-unstable:21190 and xen-4.0-testing:21114.
>> The fix will appear in Xen 4.0.1.
> 
> That seems wrong to me - I specifically changed the accounting so
> that pieces not used from the E820 map (which can no longer be cut
> off in e820.c, as that code doesn't know *where* to cut off) won't
> get reported as available memory. The real question is where (for
> the non-cut-off case) the two calculations differ.

boot_e820 has chunks cut out of it for stashing kexec stuff, as well as all
the multiboot modules. The value thereby obtained is just confusing to users
who think we've binned possibly 100s of megabytes (if they run a big
initrd).

 -- Keir

Re: Possible regression: XEN 4.0.0 total_memory decrease

[Jan Beulich]

>>> Keir Fraser <keir.fraser@eu.citrix.com> 16.04.10 19:37 >>>
>On 16/04/2010 08:53, "Jan Beulich" <JBeulich@novell.com> wrote:
>
>>>>> Keir Fraser <keir.fraser@eu.citrix.com> 15.04.10 20:23 >>>
>>> I've fixed this regression as xen-unstable:21190 and xen-4.0-testing:21114.
>>> The fix will appear in Xen 4.0.1.
>> 
>> That seems wrong to me - I specifically changed the accounting so
>> that pieces not used from the E820 map (which can no longer be cut
>> off in e820.c, as that code doesn't know *where* to cut off) won't
>> get reported as available memory. The real question is where (for
>> the non-cut-off case) the two calculations differ.
>
>boot_e820 has chunks cut out of it for stashing kexec stuff, as well as all
>the multiboot modules. The value thereby obtained is just confusing to users
>who think we've binned possibly 100s of megabytes (if they run a big
>initrd).

The piece cut off for kexec imo shouldn't be counted as (usable)
system RAM; the piece for the multiboot modules certainly should,
but perhaps it would then be better to account for that explicitly
instead of reporting a possibly much higher value than is actually
available at runtime?

Jan

Re: Possible regression: XEN 4.0.0 total_memory decrease

[Keir Fraser]

On 19/04/2010 08:35, "Jan Beulich" <JBeulich@novell.com> wrote:

>> boot_e820 has chunks cut out of it for stashing kexec stuff, as well as all
>> the multiboot modules. The value thereby obtained is just confusing to users
>> who think we've binned possibly 100s of megabytes (if they run a big
>> initrd).
> 
> The piece cut off for kexec imo shouldn't be counted as (usable)
> system RAM; the piece for the multiboot modules certainly should,
> but perhaps it would then be better to account for that explicitly
> instead of reporting a possibly much higher value than is actually
> available at runtime?

I think it's quite open to debate what should be included in total_memory.
Certainly I think kexec can be, as why is it different from any other in-use
memory in the system in that regard?

 -- Keir