* RFC: automatic NUMA placement
@ 2010-09-16 13:38 Juergen Gross
2010-09-16 15:52 ` Dan Magenheimer
2010-09-27 21:46 ` Andre Przywara
0 siblings, 2 replies; 5+ messages in thread
From: Juergen Gross @ 2010-09-16 13:38 UTC (permalink / raw)
To: xen-devel
Hi,
I just stumbled upon the automatic pinning of vcpus on domain creation in
case of NUMA.
This behaviour is questionable IMO, as it breaks correct handling of
scheduling weights on NUMA machines.
I would suggest to switch this feature off per default and make it a
configuration option of xend. It would make sense, however, to change cpu pool
processor allocation to be NUMA-aware.
Switching NUMA off via boot option would remove NUMA-optimized memory
allocation, which would be sub-optimal :-)
What do you think?
Juergen
--
Juergen Gross Principal Developer Operating Systems
TSP ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions e-mail: juergen.gross@ts.fujitsu.com
Domagkstr. 28 Internet: ts.fujitsu.com
D-80807 Muenchen Company details: ts.fujitsu.com/imprint.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* RE: RFC: automatic NUMA placement
2010-09-16 13:38 RFC: automatic NUMA placement Juergen Gross
@ 2010-09-16 15:52 ` Dan Magenheimer
2010-09-17 6:26 ` Juergen Gross
2010-09-27 21:46 ` Andre Przywara
1 sibling, 1 reply; 5+ messages in thread
From: Dan Magenheimer @ 2010-09-16 15:52 UTC (permalink / raw)
To: Juergen Gross, xen-devel
I am in favor of this being optional rather than default.
It is another case of the tradeoff between performance
and flexibility that I have ranted about several times
on xen-devel, and Xen's policy seems to be fairly random
(sometimes Xen chooses performance over flexibility and
sometimes the opposite).
I went looking through xen-devel archives for a previous
discussion on this (and to find when the code got added)
but couldn't find it... if you have links to the patch
and discussion, please post.
> -----Original Message-----
> From: Juergen Gross [mailto:juergen.gross@ts.fujitsu.com]
> Sent: Thursday, September 16, 2010 7:39 AM
> To: xen-devel@lists.xensource.com
> Subject: [Xen-devel] RFC: automatic NUMA placement
>
> Hi,
>
> I just stumbled upon the automatic pinning of vcpus on domain creation
> in
> case of NUMA.
> This behaviour is questionable IMO, as it breaks correct handling of
> scheduling weights on NUMA machines.
> I would suggest to switch this feature off per default and make it a
> configuration option of xend. It would make sense, however, to change
> cpu pool
> processor allocation to be NUMA-aware.
> Switching NUMA off via boot option would remove NUMA-optimized memory
> allocation, which would be sub-optimal :-)
>
> What do you think?
>
>
> Juergen
>
> --
> Juergen Gross Principal Developer Operating Systems
> TSP ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967
> Fujitsu Technology Solutions e-mail:
> juergen.gross@ts.fujitsu.com
> Domagkstr. 28 Internet: ts.fujitsu.com
> D-80807 Muenchen Company details:
> ts.fujitsu.com/imprint.html
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: RFC: automatic NUMA placement
2010-09-16 15:52 ` Dan Magenheimer
@ 2010-09-17 6:26 ` Juergen Gross
0 siblings, 0 replies; 5+ messages in thread
From: Juergen Gross @ 2010-09-17 6:26 UTC (permalink / raw)
To: Dan Magenheimer; +Cc: xen-devel
On 09/16/10 17:52, Dan Magenheimer wrote:
> I am in favor of this being optional rather than default.
> It is another case of the tradeoff between performance
> and flexibility that I have ranted about several times
> on xen-devel, and Xen's policy seems to be fairly random
> (sometimes Xen chooses performance over flexibility and
> sometimes the opposite).
>
> I went looking through xen-devel archives for a previous
> discussion on this (and to find when the code got added)
> but couldn't find it... if you have links to the patch
> and discussion, please post.
Took a while to find it, but hg log helped :-)
Patch:
http://xenbits.xensource.com/staging/xen-unstable.hg?diff/b58180cf8ab8/tools/python/xen/xend/XendDomainInfo.py
Discussion:
http://lists.xensource.com/archives/html/xen-devel/2008-02/msg00883.html
>
>> -----Original Message-----
>> From: Juergen Gross [mailto:juergen.gross@ts.fujitsu.com]
>> Sent: Thursday, September 16, 2010 7:39 AM
>> To: xen-devel@lists.xensource.com
>> Subject: [Xen-devel] RFC: automatic NUMA placement
>>
>> Hi,
>>
>> I just stumbled upon the automatic pinning of vcpus on domain creation
>> in
>> case of NUMA.
>> This behaviour is questionable IMO, as it breaks correct handling of
>> scheduling weights on NUMA machines.
>> I would suggest to switch this feature off per default and make it a
>> configuration option of xend. It would make sense, however, to change
>> cpu pool
>> processor allocation to be NUMA-aware.
>> Switching NUMA off via boot option would remove NUMA-optimized memory
>> allocation, which would be sub-optimal :-)
>>
>> What do you think?
Juergen
--
Juergen Gross Principal Developer Operating Systems
TSP ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions e-mail: juergen.gross@ts.fujitsu.com
Domagkstr. 28 Internet: ts.fujitsu.com
D-80807 Muenchen Company details: ts.fujitsu.com/imprint.html
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: RFC: automatic NUMA placement
2010-09-16 13:38 RFC: automatic NUMA placement Juergen Gross
2010-09-16 15:52 ` Dan Magenheimer
@ 2010-09-27 21:46 ` Andre Przywara
2010-09-28 4:48 ` Juergen Gross
1 sibling, 1 reply; 5+ messages in thread
From: Andre Przywara @ 2010-09-27 21:46 UTC (permalink / raw)
To: Juergen Gross; +Cc: xen-devel
Juergen Gross wrote:
> Hi,
>
> I just stumbled upon the automatic pinning of vcpus on domain creation in
> case of NUMA.
> This behaviour is questionable IMO, as it breaks correct handling of
> scheduling weights on NUMA machines.
> I would suggest to switch this feature off per default and make it a
> configuration option of xend. It would make sense, however, to change cpu pool
> processor allocation to be NUMA-aware.
> Switching NUMA off via boot option would remove NUMA-optimized memory
> allocation, which would be sub-optimal :-)
Hi Jürgen,
stumbled over your mail just now, so sorry for the delay.
First: Don't turn off automatic NUMA placement ;-)
In my tests it helped a lot to preserve performance on NUMA machines.
I was just browsing through the ML archive to find your original CPU
pools description from April, and it seems to fit the requirements in
NUMA machines quite well.
I haven't done any experiments with Cpupools nor haven't looked at the
code yet, but just a quick idea:
What about if we marry static NUMA placement and Cpupools?
I'd suggest to introduce static NUMA pools, one for each node. The CPUs
assigned to each pool are fixed and cannot be removed nor added (because
the NUMA topology is fixed).
Is that possible? Can we assign one physical CPUs to multiple pools (to
Pool-0 and to NUMA-0?) Or are they exclusive or hierarchical like the
Linux' cpusets?
We could introduce magic names for each NUMA pool, so that people just
say cpupool="NUMA-2" and get their domain pinned to that pool. Without
any explicit assignment the system would pick a NUMA node (like it does
today) and would just use the respective Cpupool. I think that is very
similar to what it does today, only that the pinning nature is more
evident to the user (as it uses the Cpupool name space).
Also it would allow for users to override the pinning by specifying a
different Cpupool explicitly (like Pool-0).
Just tell me what you think about this and whether I am wrong with my
thinking ;-)
Regards,
Andre.
--
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 448-3567-12
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: RFC: automatic NUMA placement
2010-09-27 21:46 ` Andre Przywara
@ 2010-09-28 4:48 ` Juergen Gross
0 siblings, 0 replies; 5+ messages in thread
From: Juergen Gross @ 2010-09-28 4:48 UTC (permalink / raw)
To: Andre Przywara; +Cc: xen-devel
Hi Andre,
thanks for your thoughts.
On 09/27/10 23:46, Andre Przywara wrote:
> Juergen Gross wrote:
>> Hi,
>>
>> I just stumbled upon the automatic pinning of vcpus on domain creation in
>> case of NUMA.
>> This behaviour is questionable IMO, as it breaks correct handling of
>> scheduling weights on NUMA machines.
>> I would suggest to switch this feature off per default and make it a
>> configuration option of xend. It would make sense, however, to change
>> cpu pool
>> processor allocation to be NUMA-aware.
>> Switching NUMA off via boot option would remove NUMA-optimized memory
>> allocation, which would be sub-optimal :-)
> Hi Jürgen,
>
> stumbled over your mail just now, so sorry for the delay.
> First: Don't turn off automatic NUMA placement ;-)
> In my tests it helped a lot to preserve performance on NUMA machines.
>
> I was just browsing through the ML archive to find your original CPU
> pools description from April, and it seems to fit the requirements in
> NUMA machines quite well.
> I haven't done any experiments with Cpupools nor haven't looked at the
> code yet, but just a quick idea:
> What about if we marry static NUMA placement and Cpupools?
>
> I'd suggest to introduce static NUMA pools, one for each node. The CPUs
> assigned to each pool are fixed and cannot be removed nor added (because
> the NUMA topology is fixed).
> Is that possible? Can we assign one physical CPUs to multiple pools (to
> Pool-0 and to NUMA-0?) Or are they exclusive or hierarchical like the
> Linux' cpusets?
A cpu is always member of only one pool.
> We could introduce magic names for each NUMA pool, so that people just
> say cpupool="NUMA-2" and get their domain pinned to that pool. Without
> any explicit assignment the system would pick a NUMA node (like it does
> today) and would just use the respective Cpupool. I think that is very
> similar to what it does today, only that the pinning nature is more
> evident to the user (as it uses the Cpupool name space).
> Also it would allow for users to override the pinning by specifying a
> different Cpupool explicitly (like Pool-0).
>
> Just tell me what you think about this and whether I am wrong with my
> thinking ;-)
With your proposal it isn't possible to start a domU with more vcpus than
cpus in a node without changing cpu pools.
I would suggest to do it the following way:
- use automatic NUMA placement only in Pool-0 (this won't change anything
for users not using cpu pools), perhaps with an option to switch it off
- change the cpu allocation for pools to be NUMA aware
- optionally add a xl and/or xm command to create one cpu pool per NUMA node
Juergen
--
Juergen Gross Principal Developer Operating Systems
TSP ES&S SWE OS6 Telephone: +49 (0) 89 3222 2967
Fujitsu Technology Solutions e-mail: juergen.gross@ts.fujitsu.com
Domagkstr. 28 Internet: ts.fujitsu.com
D-80807 Muenchen Company details: ts.fujitsu.com/imprint.html
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2010-09-28 4:48 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2010-09-16 13:38 RFC: automatic NUMA placement Juergen Gross
2010-09-16 15:52 ` Dan Magenheimer
2010-09-17 6:26 ` Juergen Gross
2010-09-27 21:46 ` Andre Przywara
2010-09-28 4:48 ` Juergen Gross
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.