Xen runs out of memory!

Xen runs out of memory!

[Jayaraman, Bhaskar]

[-- Attachment #1.1: Type: text/plain, Size: 900 bytes --]

Hi all, I'm trying to increase max grant table frames from the current 32 pages to a user defined value. I'm basically trying to map an entire domain into another so I have say a Linux HVM with 3GB memory and I'm trying to map the whole domain into another domain with 4GB memory.

So I set the max grant frames to 1536 (needed to map 3GB of memory into any domain), but when I setup the grant table for that many frames Xen is only able to setup 1427 grant table frames at the most and Xen fails after this in alloc_heap_pages.

I'm also not able to bring up any other HVM after this, i.e. xen has no memory for any other domains, which makes me wonder if anyone has tested the maximum domUs that can be brought up on Xen (255 right?).

Is there anyway I can make this memory allocation succeed? i.e. by increasing Xen memory or some other means? (Xen in HVM uses 64MB)

Regards,
Bhaskar.

[-- Attachment #1.2: Type: text/html, Size: 3323 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Xen runs out of memory!

[Keir Fraser]

Are you running a 64-bit hypervisor? You want to do that, and with latest
xen-unstable, to lift Xen-heap size restrictions. Beyond that, make sure
that other domains (especially dom0) are not using all the memory. Restrict
dom0¹s memory usage by using the dom0_mem= option on Xen¹s command line, for
example.

 -- Keir

On 12/03/2009 10:20, "Jayaraman, Bhaskar" <Bhaskar.Jayaraman@lsi.com> wrote:

> Hi all, I¹m trying to increase max grant table frames from the current 32
> pages to a user defined value. I¹m basically trying to map an entire domain
> into another so I have say a Linux HVM with 3GB memory and I¹m trying to map
> the whole domain into another domain with 4GB memory.
>  
> So I set the max grant frames to 1536 (needed to map 3GB of memory into any
> domain), but when I setup the grant table for that many frames Xen is only
> able to setup 1427 grant table frames at the most and Xen fails after this in
> alloc_heap_pages.
>  
> I¹m also not able to bring up any other HVM after this, i.e. xen has no memory
> for any other domains, which makes me wonder if anyone has tested the maximum
> domUs that can be brought up on Xen (255 right?).
>  
> Is there anyway I can make this memory allocation succeed? i.e. by increasing
> Xen memory or some other means? (Xen in HVM uses 64MB)
>  

RE: Xen runs out of memory!

[Jayaraman, Bhaskar]

Keir, I'm using a 64 bit unstable version of Xen. 4 Core single socket CPU with 8GB memory and Dom0 is using around 2.5GB. What do I do to lift heap size restrictions? 

The only other domain that comes up is my HVM which squeezes Xen out of memory.

Do you think I can increase contiguous pages by increasing the value of MAX_ORDER, which is right now defined as 
#define MAX_ORDER 20 /* 2^20 contiguous pages */

Which is a hell of a lot of pages and all I'm asking for are 1536 contiguous ones.

However when I checked in xen/common/page_alloc.c I see the following: -
typedef struct list_head heap_by_zone_and_order_t[NR_ZONES][MAX_ORDER+1]

Should this have been: -
typedef struct list_head heap_by_zone_and_order_t[NR_ZONES][(int)pow(2,MAX_ORDER * +1)]

??


Bhaskar.

-----Original Message-----
From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
Sent: Thursday, March 12, 2009 4:03 PM
To: Jayaraman, Bhaskar; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Xen runs out of memory!

Are you running a 64-bit hypervisor? You want to do that, and with latest
xen-unstable, to lift Xen-heap size restrictions. Beyond that, make sure
that other domains (especially dom0) are not using all the memory. Restrict
dom0¹s memory usage by using the dom0_mem= option on Xen¹s command line, for
example.

 -- Keir

On 12/03/2009 10:20, "Jayaraman, Bhaskar" <Bhaskar.Jayaraman@lsi.com> wrote:

> Hi all, I¹m trying to increase max grant table frames from the current 32
> pages to a user defined value. I¹m basically trying to map an entire domain
> into another so I have say a Linux HVM with 3GB memory and I¹m trying to map
> the whole domain into another domain with 4GB memory.
>  
> So I set the max grant frames to 1536 (needed to map 3GB of memory into any
> domain), but when I setup the grant table for that many frames Xen is only
> able to setup 1427 grant table frames at the most and Xen fails after this in
> alloc_heap_pages.
>  
> I¹m also not able to bring up any other HVM after this, i.e. xen has no memory
> for any other domains, which makes me wonder if anyone has tested the maximum
> domUs that can be brought up on Xen (255 right?).
>  
> Is there anyway I can make this memory allocation succeed? i.e. by increasing
> Xen memory or some other means? (Xen in HVM uses 64MB)
>  

Re: Xen runs out of memory!

[Keir Fraser]

On 12/03/2009 11:01, "Jayaraman, Bhaskar" <Bhaskar.Jayaraman@lsi.com> wrote:

> Do you think I can increase contiguous pages by increasing the value of
> MAX_ORDER, which is right now defined as
> #define MAX_ORDER 20 /* 2^20 contiguous pages */
> 
> Which is a hell of a lot of pages and all I'm asking for are 1536 contiguous
> ones.

You're not needing contiguous memory. You're simply out of memory. Not much
Xen can do about that. Your domains are using it all.

 -- Keir

RE: Xen runs out of memory! - succeeds now

[Jayaraman, Bhaskar]

Thanks Keir, reducing Dom0 memory usage alone helps but I'm not able to figure how. This is why: -

Case 1: - 

I have 8GB of memory in the system and Dom0 was initially using 2.5 GB. I brought up a DomU that came up with 4GB memory which still leaves out 1.5GB free and still I wasn't able to setup the 1536 grant table pages which I wished to.

Case 2: -

Next I reduce Dom0 usage to 800MB and then bring up DomU with 3GB (~4.5GB free) of memory and I am able to setup 1536 pages ----> This is great

Next I bring up another DomU with 3GB memory (~1.5GB free now) and once again I'm able to setup 1536 grant table pages ----> Even better.

I'm don't know why case 1 with 1.5GB memory free on the system, 1536 grant table pages weren't setup with just one domU, while in the 2nd case, with again ~ 1.5GB free I'm able to setup the grant table on the second domU?


Regards,
Bhaskar.

-----Original Message-----
From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
Sent: Thursday, March 12, 2009 4:50 PM
To: Jayaraman, Bhaskar; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] Xen runs out of memory!

On 12/03/2009 11:01, "Jayaraman, Bhaskar" <Bhaskar.Jayaraman@lsi.com> wrote:

> Do you think I can increase contiguous pages by increasing the value of
> MAX_ORDER, which is right now defined as
> #define MAX_ORDER 20 /* 2^20 contiguous pages */
> 
> Which is a hell of a lot of pages and all I'm asking for are 1536 contiguous
> ones.

You're not needing contiguous memory. You're simply out of memory. Not much
Xen can do about that. Your domains are using it all.

 -- Keir