Can't always start 32 bit domains after 64 bit domains

Can't always start 32 bit domains after 64 bit domains

[Sarah Newman]

Last night on a 288GiB server with less than 64GiB of 32 bit
domUs, we used the standard xendomains script which starts VMs
in alphabetical order.

Some 32 bit domUs at the very end were unable to start. The
error message we received is the following:

xc: error: panic: xc_dom_x86.c:944: arch_setup_meminit: failed to allocate 0x80000 pages: Internal error
xc: error: panic: xc_dom_boot.c:154: xc_dom_boot_mem_init: can't allocate low memory for domain: Out of memory
libxl: error: libxl_dom.c:719:libxl__build_pv: xc_dom_boot_mem_init failed: Device or resource busy
libxl: error: libxl_create.c:1144:domcreate_rebuild_done: cannot (re-)build domain: -3

After shutting down some 64 bit domUs, we could start the remainder
of the 32 bit domUs. Finally we started the rest of the 64 bit
domUs such that everything was booted.

My current understanding is that on a server with more than 168GiB
of memory, I should still be able to around 128GiB of 32-bit PV
domUs, regardless of what order the domUs are started in.

The version of xen we're using is Xen4CentOS 4.6.3-3.

The current workaround plan is to start all the 32-bit domUs
in smallest to largest order, and then all the 64 bit domUs in
smallest to largest order, but we are not 100% confident this will
work long term given it should have worked in the first place.

Is there any other information I can provide to help with duplicating the issue?

Please keep me CC'ed as I am not subscribed to xen-devel right now.

--Sarah

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Jan Beulich]

>>> On 19.11.16 at 22:22, <srn@prgmr.com> wrote:
> Last night on a 288GiB server with less than 64GiB of 32 bit
> domUs, we used the standard xendomains script which starts VMs
> in alphabetical order.
> 
> Some 32 bit domUs at the very end were unable to start. The
> error message we received is the following:
> 
> xc: error: panic: xc_dom_x86.c:944: arch_setup_meminit: failed to allocate 
> 0x80000 pages: Internal error
> xc: error: panic: xc_dom_boot.c:154: xc_dom_boot_mem_init: can't allocate 
> low memory for domain: Out of memory
> libxl: error: libxl_dom.c:719:libxl__build_pv: xc_dom_boot_mem_init failed: 
> Device or resource busy
> libxl: error: libxl_create.c:1144:domcreate_rebuild_done: cannot (re-)build 
> domain: -3
> 
> After shutting down some 64 bit domUs, we could start the remainder
> of the 32 bit domUs. Finally we started the rest of the 64 bit
> domUs such that everything was booted.
> 
> My current understanding is that on a server with more than 168GiB
> of memory, I should still be able to around 128GiB of 32-bit PV
> domUs, regardless of what order the domUs are started in.

You don't clarify what you base this understanding of yours on; I don't
think this is the case. What exact memory will be allocated for 64-bit
guests isn't very predictable. In particular, the preference of allocating
1Gb pages as long as available may result in allocations eating into the
low 128Gb of memory earlier than you expect. You may want to analyze
system state by looking at debug key output at relevant points in time.

Back in the xend days someone here had invented a (crude) mechanism
to set aside memory for 32-bit PV domains, but I don't think dealing with
this situation in xl has ever seen any interest.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Sarah Newman]

On 11/21/2016 12:20 AM, Jan Beulich wrote:
>>>> On 19.11.16 at 22:22, <srn@prgmr.com> wrote:
>> My current understanding is that on a server with more than 168GiB
>> of memory, I should still be able to around 128GiB of 32-bit PV
>> domUs, regardless of what order the domUs are started in.
> 
> You don't clarify what you base this understanding of yours on; I don't
> think this is the case. What exact memory will be allocated for 64-bit
> guests isn't very predictable. In particular, the preference of allocating
> 1Gb pages as long as available may result in allocations eating into the
> low 128Gb of memory earlier than you expect. You may want to analyze
> system state by looking at debug key output at relevant points in time.

Because whenever I read about the 168GiB limit and 32 bit domains there was no mention that the initial 128GiB might not all be available if there
were also 64 bit domains. Given how little visibility there typically is into where memory is physically allocated, there is no way to know this
doesn't work as a normal user until it doesn't. Are you saying all Xen users have to be developers too?

In my opinion this is a bug until/unless it's very clearly documented that it's not supported behavior to mix 32 and 64 bit PV domains when there is
more than 168GiB RAM, or maybe just period if that's too complicated to explain to people. I don't know where to find that documentation. It seems
like https://wiki.xenproject.org/wiki/Xen_Project_Release_Features might be the right place, I'm not sure. If you know of a better place I'd love to know.

> Back in the xend days someone here had invented a (crude) mechanism
> to set aside memory for 32-bit PV domains, but I don't think dealing with
> this situation in xl has ever seen any interest.

If I wanted to add that, where would it go?

Thanks, Sarah

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Jan Beulich]

>>> On 21.11.16 at 10:45, <srn@prgmr.com> wrote:
> On 11/21/2016 12:20 AM, Jan Beulich wrote:
>>>>> On 19.11.16 at 22:22, <srn@prgmr.com> wrote:
>>> My current understanding is that on a server with more than 168GiB
>>> of memory, I should still be able to around 128GiB of 32-bit PV
>>> domUs, regardless of what order the domUs are started in.
>> 
>> You don't clarify what you base this understanding of yours on; I don't
>> think this is the case. What exact memory will be allocated for 64-bit
>> guests isn't very predictable. In particular, the preference of allocating
>> 1Gb pages as long as available may result in allocations eating into the
>> low 128Gb of memory earlier than you expect. You may want to analyze
>> system state by looking at debug key output at relevant points in time.
> 
> Because whenever I read about the 168GiB limit and 32 bit domains there was 
> no mention that the initial 128GiB might not all be available if there
> were also 64 bit domains. Given how little visibility there typically is 
> into where memory is physically allocated, there is no way to know this
> doesn't work as a normal user until it doesn't. Are you saying all Xen users 
> have to be developers too?

No, I'm not saying this, but please note that you didn't post the
question on xen-users, but xen-devel.

> In my opinion this is a bug until/unless it's very clearly documented that 
> it's not supported behavior to mix 32 and 64 bit PV domains when there is
> more than 168GiB RAM, or maybe just period if that's too complicated to 
> explain to people. I don't know where to find that documentation. It seems
> like https://wiki.xenproject.org/wiki/Xen_Project_Release_Features might be 
> the right place, I'm not sure. If you know of a better place I'd love to 
> know.

I view this as a bug too, fwiw. Me previously bringing up the topic
didn't result in any activity though, presumably because 32-bit
guests are being considered rare enough these days.

>> Back in the xend days someone here had invented a (crude) mechanism
>> to set aside memory for 32-bit PV domains, but I don't think dealing with
>> this situation in xl has ever seen any interest.
> 
> If I wanted to add that, where would it go?

I don't know, that's a question for xl folks I guess.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Andrew Cooper]

On 21/11/16 10:05, Jan Beulich wrote:
>>>> On 21.11.16 at 10:45, <srn@prgmr.com> wrote:
>> On 11/21/2016 12:20 AM, Jan Beulich wrote:
>>>>>> On 19.11.16 at 22:22, <srn@prgmr.com> wrote:
>>>> My current understanding is that on a server with more than 168GiB
>>>> of memory, I should still be able to around 128GiB of 32-bit PV
>>>> domUs, regardless of what order the domUs are started in.
>>> You don't clarify what you base this understanding of yours on; I don't
>>> think this is the case. What exact memory will be allocated for 64-bit
>>> guests isn't very predictable. In particular, the preference of allocating
>>> 1Gb pages as long as available may result in allocations eating into the
>>> low 128Gb of memory earlier than you expect. You may want to analyze
>>> system state by looking at debug key output at relevant points in time.
>> Because whenever I read about the 168GiB limit and 32 bit domains there was 
>> no mention that the initial 128GiB might not all be available if there
>> were also 64 bit domains. Given how little visibility there typically is 
>> into where memory is physically allocated, there is no way to know this
>> doesn't work as a normal user until it doesn't. Are you saying all Xen users 
>> have to be developers too?
> No, I'm not saying this, but please note that you didn't post the
> question on xen-users, but xen-devel.
>
>> In my opinion this is a bug until/unless it's very clearly documented that 
>> it's not supported behavior to mix 32 and 64 bit PV domains when there is
>> more than 168GiB RAM, or maybe just period if that's too complicated to 
>> explain to people. I don't know where to find that documentation. It seems
>> like https://wiki.xenproject.org/wiki/Xen_Project_Release_Features might be 
>> the right place, I'm not sure. If you know of a better place I'd love to 
>> know.
> I view this as a bug too, fwiw. Me previously bringing up the topic
> didn't result in any activity though, presumably because 32-bit
> guests are being considered rare enough these days.
>
>>> Back in the xend days someone here had invented a (crude) mechanism
>>> to set aside memory for 32-bit PV domains, but I don't think dealing with
>>> this situation in xl has ever seen any interest.
>> If I wanted to add that, where would it go?
> I don't know, that's a question for xl folks I guess.

IIRC, given no other constraints, the Xen heap allocation allocates from
the top down to help this exact case.

I suspect that libxl's preference towards NUMA allocation of domains
interferes with this, by adding a NUMA constraints to memory allocations
for 64bit PV guests.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Sarah Newman]

On 11/21/2016 05:21 AM, Andrew Cooper wrote:
> On 21/11/16 10:05, Jan Beulich wrote:

>>>> Back in the xend days someone here had invented a (crude) mechanism
>>>> to set aside memory for 32-bit PV domains, but I don't think dealing with
>>>> this situation in xl has ever seen any interest.
>>> If I wanted to add that, where would it go?
>> I don't know, that's a question for xl folks I guess.
> 
> IIRC, given no other constraints, the Xen heap allocation allocates from
> the top down to help this exact case.

Yes, that's what I thought it did too, though I can't find my source information for that.

> 
> I suspect that libxl's preference towards NUMA allocation of domains
> interferes with this, by adding a NUMA constraints to memory allocations
> for 64bit PV guests.

I ran xl info -n (which I didn't know about before) and that shows the problem much more clearly.

If that's the reason not all the higher memory is being used first: is a potential workaround to pin 64 bit domains to the second physical core on
boot, and 32 bit domains to the first physical core on boot, and then change the allowed cores with 'xl vcpu-pin' after the domain is loaded?

Thanks, Sarah

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Sarah Newman]

On 11/21/2016 11:37 AM, Sarah Newman wrote:
> On 11/21/2016 05:21 AM, Andrew Cooper wrote:
>> On 21/11/16 10:05, Jan Beulich wrote:
> 
>>>>> Back in the xend days someone here had invented a (crude) mechanism
>>>>> to set aside memory for 32-bit PV domains, but I don't think dealing with
>>>>> this situation in xl has ever seen any interest.
>>>> If I wanted to add that, where would it go?
>>> I don't know, that's a question for xl folks I guess.
>>
>> IIRC, given no other constraints, the Xen heap allocation allocates from
>> the top down to help this exact case.
> 
> Yes, that's what I thought it did too, though I can't find my source information for that.
> 
>>
>> I suspect that libxl's preference towards NUMA allocation of domains
>> interferes with this, by adding a NUMA constraints to memory allocations
>> for 64bit PV guests.
> 
> I ran xl info -n (which I didn't know about before) and that shows the problem much more clearly.
> 
> If that's the reason not all the higher memory is being used first: is a potential workaround to pin 64 bit domains to the second physical core on
> boot, and 32 bit domains to the first physical core on boot, and then change the allowed cores with 'xl vcpu-pin' after the domain is loaded?

Free memory on a test server with no domains:
node:    memsize    memfree    distances
   0:    148480     142983      10,21
   1:    147456     144645      21,10

Free memory booting 116 256M 64-bit domains, limited to cpus='all,^0-1' on boot:
node:    memsize    memfree    distances
   0:    148480     128416      10,21
   1:    147456     129669      21,10

Free memory booting 116 256M 64-bit domains, limited to cpus='12-23' on boot:
node:    memsize    memfree    distances
   0:    148480     143397      10,21
   1:    147456     114693      21,10

This looks like a viable workaround. Where should I document it?

--Sarah

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2229 bytes --]

On Mon, 2016-11-21 at 13:06 -0800, Sarah Newman wrote:
> On 11/21/2016 11:37 AM, Sarah Newman wrote:
> > 
> > If that's the reason not all the higher memory is being used first:
> > is a potential workaround to pin 64 bit domains to the second
> > physical core on
> > boot, and 32 bit domains to the first physical core on boot, and
> > then change the allowed cores with 'xl vcpu-pin' after the domain
> > is loaded?
> 
> Free memory on a test server with no domains:
> node:    memsize    memfree    distances
>    0:    148480     142983      10,21
>    1:    147456     144645      21,10
> 
> Free memory booting 116 256M 64-bit domains, limited to cpus='all,^0-
> 1' on boot:
> node:    memsize    memfree    distances
>    0:    148480     128416      10,21
>    1:    147456     129669      21,10
> 
> Free memory booting 116 256M 64-bit domains, limited to cpus='12-23'
> on boot:
> node:    memsize    memfree    distances
>    0:    148480     143397      10,21
>    1:    147456     114693      21,10
> 
> This looks like a viable workaround. Where should I document it?
> 
It's documented in here (although, the text can be improved a bit, I
think)... look for 'cpus' and 'cpus_soft' within the page:
https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html

A more clear mention to using "all" can perhaps be added to the wiki
pages I listed in my other email.

However, what I think is totally missing, is any documentation about
the fact that, in use cases like yours, domain creation should be done
in a certain order, for what reasons, which order is that, and the fact
that NUMA placement may interfere.

I'm not sure where and how to properly document all this [adding Lars],
but I'd say it probably deserves a dedicated wiki page.

Thoughts?

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Sarah Newman]

On 11/22/2016 10:46 AM, Dario Faggioli wrote:
> On Mon, 2016-11-21 at 13:06 -0800, Sarah Newman wrote:
>> On 11/21/2016 11:37 AM, Sarah Newman wrote:
>>> 
>>> If that's the reason not all the higher memory is being used first: is a potential workaround to pin 64 bit domains to the second physical
>>> core on boot, and 32 bit domains to the first physical core on boot, and then change the allowed cores with 'xl vcpu-pin' after the domain is
>>> loaded?
>> 
>> Free memory on a test server with no domains: node:    memsize    memfree    distances 0:    148480     142983      10,21 1:    147456
>> 144645      21,10
>> 
>> Free memory booting 116 256M 64-bit domains, limited to cpus='all,^0- 1' on boot: node:    memsize    memfree    distances 0:    148480
>> 128416      10,21 1:    147456     129669      21,10
>> 
>> Free memory booting 116 256M 64-bit domains, limited to cpus='12-23' on boot: node:    memsize    memfree    distances 0:    148480     143397
>> 10,21 1:    147456     114693      21,10
>> 
>> This looks like a viable workaround. Where should I document it?
>> 
> It's documented in here (although, the text can be improved a bit, I think)... look for 'cpus' and 'cpus_soft' within the page: 
> https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html
> 
> A more clear mention to using "all" can perhaps be added to the wiki pages I listed in my other email.
> 

Testing shows that memory allocation is still alternated between the two physical nodes, even after setting cpus='all' or removing cpus=... from the
configuration file entirely.

If you're saying not specifying "cpus=..." will keep libxl from interfering with the Xens default allocation policy, then Xens default allocation
policy no longer starts from the top of memory for 64 bit PV domains, at least for 4.6.3. Maybe it's starting from the top of memory per node.

> However, what I think is totally missing, is any documentation about the fact that, in use cases like yours, domain creation should be done in a
> certain order, for what reasons, which order is that, and the fact that NUMA placement may interfere.
> 
> I'm not sure where and how to properly document all this [adding Lars], but I'd say it probably deserves a dedicated wiki page.
> 
> Thoughts?

A dedicated wiki page would be fine if it is sufficiently linked to. I would link to it from
https://wiki.xenproject.org/wiki/Xen_Project_Release_Features, maybe adding a footnote for "Host Limits" and/or various parameters listed therein.

--Sarah

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 2662 bytes --]

On Tue, 2016-11-22 at 11:37 -0800, Sarah Newman wrote:
> On 11/22/2016 10:46 AM, Dario Faggioli wrote:
> > It's documented in here (although, the text can be improved a bit,
> > I think)... look for 'cpus' and 'cpus_soft' within the page: 
> > https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html
> > 
> > A more clear mention to using "all" can perhaps be added to the
> > wiki pages I listed in my other email.
> > 
> 
> Testing shows that memory allocation is still alternated between the
> two physical nodes, even after setting cpus='all' or removing
> cpus=... from the
> configuration file entirely.
> 
Not sure I'm getting. So, libxl default is to try to place a new domain
on a NUMA node (or on the smallest possible set of NUMA nodes). If you
don't specify neither cpus= nor cpus_soft= such default applies, and
you should _not_ get memory spread on more than one node (unless
placement fails for some reason, or the domain does not fit there).

If you specify cpus="node:0" or cpus_soft="node:0", libxl automatic
NUMA placement won't be execute, and you'll get memory only from NUMA
node 0.

If you specify cpus="all", libxl automatic NUMA placement won't be
executed, and you get memory from all the NUMA nodes, which AFAICT, is
what Xen does, if not instructed otherwise by the toolstack.

Are you seeing something different from this?

> If you're saying not specifying "cpus=..." will keep libxl from
> interfering with the Xens default allocation policy, then Xens
> default allocation
> policy no longer starts from the top of memory for 64 bit PV domains,
> at least for 4.6.3. Maybe it's starting from the top of memory per
> node.
> 
No, I'm saying that _not_ specifying cpus= will trigger libxl
interference. _Do_ specifying it, will either limit or disable it,
depending on how you specify it.

That's all I'm saying.

As far as I can remember, Xen's always been striping the memory among
all the available nodes, if not told otherwise. This has at least been
the case for all my test and experiments, on any NUMA box I've touched.

I have to admit I've not played much with 32 bit guests, though.

And let me add that, if we figure out what and where the issue is, and
come up with a sensible plan, I'm happy to think and help improving xl
and libxl NUMA placement to take these constraints into account.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1925 bytes --]

On Tue, 2016-11-22 at 22:40 +0100, Dario Faggioli wrote:
> On Tue, 2016-11-22 at 11:37 -0800, Sarah Newman wrote:
> > If you're saying not specifying "cpus=..." will keep libxl from
> > interfering with the Xens default allocation policy, then Xens
> > default allocation
> > policy no longer starts from the top of memory for 64 bit PV
> > domains,
> > at least for 4.6.3. Maybe it's starting from the top of memory per
> > node.
> > 
> No, I'm saying that _not_ specifying cpus= will trigger libxl
> interference. _Do_ specifying it, will either limit or disable it,
> depending on how you specify it.
> 
> That's all I'm saying.
> 
And let me just add that, the automatic NUMA placement algorithm is
very flexible and extendible.

As I said, I didn't pay much attention at this 32 bit guests memory
issue when doing it (neither did anyone reviewing the code), and I'm
sorry if this is causing troubles.

But, really, if we want to feed a constraint, or a preference about
this inside the algorithm, we can do that without much hassle, and I'm
happy to help with that.

For instance (and I'm really just thinking out loud here), let's assume
that we know memory on NUMA node 0 is what 32 bit guests need: we can
instruct the placement algorithm to either disfavour or ignore node 1
when placing a 64 bit guest.

Or something like that...

If you think this could be interesting, let's talk about it. As soon as
I'll fully understand the constraint, and we'll have agreed on an
interface (e.g., a xl.conf flag?) and on the best course of action
(e.g., disfavour vs. forbid), I can write the code myself. :-)

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Sarah Newman]

On 11/26/2016 05:14 PM, Dario Faggioli wrote:
> On Tue, 2016-11-22 at 22:40 +0100, Dario Faggioli wrote:
>> On Tue, 2016-11-22 at 11:37 -0800, Sarah Newman wrote:
>>> If you're saying not specifying "cpus=..." will keep libxl from
>>> interfering with the Xens default allocation policy, then Xens
>>> default allocation
>>> policy no longer starts from the top of memory for 64 bit PV
>>> domains,
>>> at least for 4.6.3. Maybe it's starting from the top of memory per
>>> node.
>>>
>> No, I'm saying that _not_ specifying cpus= will trigger libxl
>> interference. _Do_ specifying it, will either limit or disable it,
>> depending on how you specify it.
>>
>> That's all I'm saying.
>>
> And let me just add that, the automatic NUMA placement algorithm is
> very flexible and extendible.
>
> As I said, I didn't pay much attention at this 32 bit guests memory
> issue when doing it (neither did anyone reviewing the code), and I'm
> sorry if this is causing troubles.
>
> But, really, if we want to feed a constraint, or a preference about
> this inside the algorithm, we can do that without much hassle, and I'm
> happy to help with that.
>
> For instance (and I'm really just thinking out loud here), let's assume
> that we know memory on NUMA node 0 is what 32 bit guests need: we can
> instruct the placement algorithm to either disfavour or ignore node 1
> when placing a 64 bit guest.
>
> Or something like that...
>
> If you think this could be interesting, let's talk about it. As soon as
> I'll fully understand the constraint, and we'll have agreed on an
> interface (e.g., a xl.conf flag?) and on the best course of action
> (e.g., disfavour vs. forbid), I can write the code myself. :-)

I don't think unconditionally allocating from the top of RAM would be kind
to most other users. Presumably most people are no longer using 32 bit
guests or this would have been found already.

It seems to me like an xl config option for setting aside memory for 32 bit
guests, similar to the one for xm, would be the best option. If  that entire
amount could be considered allocated when considering where to put 64 bit
domains, and the constraint was to balance the amount of RAM in use by each
node, I think it would do the right thing.

Or if that's too much effort, having an option to allocate down from the
top of memory for 64 bit guests (regardless of NUMA) would also work.

Thanks, Sarah



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

Re: Can't always start 32 bit domains after 64 bit domains

[Dario Faggioli]

[-- Attachment #1.1: Type: text/plain, Size: 1973 bytes --]

On Mon, 2016-11-21 at 11:37 -0800, Sarah Newman wrote:
> On 11/21/2016 05:21 AM, Andrew Cooper wrote:
> > I suspect that libxl's preference towards NUMA allocation of
> > domains
> > interferes with this, by adding a NUMA constraints to memory
> > allocations
> > for 64bit PV guests.
> 
> I ran xl info -n (which I didn't know about before) and that shows
> the problem much more clearly.
> 
> If that's the reason not all the higher memory is being used first:
> is a potential workaround to pin 64 bit domains to the second
> physical core on
> boot, and 32 bit domains to the first physical core on boot, and then
> change the allowed cores with 'xl vcpu-pin' after the domain is
> loaded?
> 
If you're looking for a way to disable libxl's NUMA-aware domain
placement --which does indeed interfere whit what memory (as in, the
memory of what NUMA node) is going to be used for the domains-- you can
"just" specify this, in all the domains' config files:

cpus="all"

This will leave all the vcpus free to run everywhere, and stop libxl to
pass down to Xen any hint on memory allocation.

Using

cpus="foo"

and/or

cpus_soft="bar"

would allow to tweak things more.

And the same is true for creating cpupools, with specific cpus from
specific NUMA nodes in them, and creating the domain direcly inside
those various pools.

That being said, what values are the best for your use case, I'm not
really sure... But maybe have a look at this.

Some more info:
https://wiki.xen.org/wiki/Xen_on_NUMA_Machines
https://wiki.xen.org/wiki/Xen_4.3_NUMA_Aware_Scheduling
https://wiki.xen.org/wiki/Tuning_Xen_for_Performance#vCPU_Soft_Affinity_for_guests

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel