Re: Can't always start 32 bit domains after 64 bit domains

[Sarah Newman <srn@prgmr.com>]

On 11/22/2016 10:46 AM, Dario Faggioli wrote:
> On Mon, 2016-11-21 at 13:06 -0800, Sarah Newman wrote:
>> On 11/21/2016 11:37 AM, Sarah Newman wrote:
>>> 
>>> If that's the reason not all the higher memory is being used first: is a potential workaround to pin 64 bit domains to the second physical
>>> core on boot, and 32 bit domains to the first physical core on boot, and then change the allowed cores with 'xl vcpu-pin' after the domain is
>>> loaded?
>> 
>> Free memory on a test server with no domains: node:    memsize    memfree    distances 0:    148480     142983      10,21 1:    147456
>> 144645      21,10
>> 
>> Free memory booting 116 256M 64-bit domains, limited to cpus='all,^0- 1' on boot: node:    memsize    memfree    distances 0:    148480
>> 128416      10,21 1:    147456     129669      21,10
>> 
>> Free memory booting 116 256M 64-bit domains, limited to cpus='12-23' on boot: node:    memsize    memfree    distances 0:    148480     143397
>> 10,21 1:    147456     114693      21,10
>> 
>> This looks like a viable workaround. Where should I document it?
>> 
> It's documented in here (although, the text can be improved a bit, I think)... look for 'cpus' and 'cpus_soft' within the page: 
> https://xenbits.xen.org/docs/unstable/man/xl.cfg.5.html
> 
> A more clear mention to using "all" can perhaps be added to the wiki pages I listed in my other email.
> 

Testing shows that memory allocation is still alternated between the two physical nodes, even after setting cpus='all' or removing cpus=... from the
configuration file entirely.

If you're saying not specifying "cpus=..." will keep libxl from interfering with the Xens default allocation policy, then Xens default allocation
policy no longer starts from the top of memory for 64 bit PV domains, at least for 4.6.3. Maybe it's starting from the top of memory per node.

> However, what I think is totally missing, is any documentation about the fact that, in use cases like yours, domain creation should be done in a
> certain order, for what reasons, which order is that, and the fact that NUMA placement may interfere.
> 
> I'm not sure where and how to properly document all this [adding Lars], but I'd say it probably deserves a dedicated wiki page.
> 
> Thoughts?

A dedicated wiki page would be fine if it is sufficiently linked to. I would link to it from
https://wiki.xenproject.org/wiki/Xen_Project_Release_Features, maybe adding a footnote for "Host Limits" and/or various parameters listed therein.

--Sarah

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel