* numactl using it's own affinity to determine CPU set
@ 2012-10-22 19:35 Jon Stanley
2012-10-22 19:41 ` Andi Kleen
0 siblings, 1 reply; 6+ messages in thread
From: Jon Stanley @ 2012-10-22 19:35 UTC (permalink / raw)
To: linux-numa, pholasek
If you set the affinity of your shell to be a subset of the available
CPUs, and then attempt to use numactl to bind to something that is not
in that affinity mask, the attempt will fail. I thought that this was
fixed in the current numactl 2.0.8, but it doesn't seem to be.
A simple reproducer on a 2-node 8-core system is as such:
$ numactl --physcpubind=5 ls
<see directory goodness>
$ taskset -p f $$
$ numactl --physcpubind=5 ls
libnuma: Warning: cpu argument 5 is out of range
<snip>
Let me know if more info is required.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: numactl using it's own affinity to determine CPU set
2012-10-22 19:35 numactl using it's own affinity to determine CPU set Jon Stanley
@ 2012-10-22 19:41 ` Andi Kleen
2012-10-22 20:12 ` Jon Stanley
0 siblings, 1 reply; 6+ messages in thread
From: Andi Kleen @ 2012-10-22 19:41 UTC (permalink / raw)
To: Jon Stanley; +Cc: linux-numa, pholasek
On Mon, Oct 22, 2012 at 03:35:42PM -0400, Jon Stanley wrote:
> If you set the affinity of your shell to be a subset of the available
> CPUs, and then attempt to use numactl to bind to something that is not
> in that affinity mask, the attempt will fail. I thought that this was
> fixed in the current numactl 2.0.8, but it doesn't seem to be.
>
> A simple reproducer on a 2-node 8-core system is as such:
>
> $ numactl --physcpubind=5 ls
> <see directory goodness>
> $ taskset -p f $$
> $ numactl --physcpubind=5 ls
> libnuma: Warning: cpu argument 5 is out of range
> <snip>
That was done intentional at some point to handle cpusets.
numactl 1.0 or so didn't have that problem.
However it's unclear if it's really bug.
Presumably could have an option to override.
-Andi
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: numactl using it's own affinity to determine CPU set
2012-10-22 19:41 ` Andi Kleen
@ 2012-10-22 20:12 ` Jon Stanley
2012-10-23 12:21 ` Petr Holasek
0 siblings, 1 reply; 6+ messages in thread
From: Jon Stanley @ 2012-10-22 20:12 UTC (permalink / raw)
To: Andi Kleen; +Cc: linux-numa, pholasek
On Mon, Oct 22, 2012 at 3:41 PM, Andi Kleen <andi@firstfloor.org> wrote:
> That was done intentional at some point to handle cpusets.
> numactl 1.0 or so didn't have that problem.
I'll happily admit to not being an expert here :)
However, I thought that the concept of cpusets was that if you
attempted to set affinity outside of your cpuset, that would silently
fail. I assume that the reason that libnuma didn't want to set outside
of it's own affinity mask is to avoid such a silent failure?
Also, I question the Real World(TM) prevalence of cpusets.
> However it's unclear if it's really bug.
Not sure if I'd call it a bug since it was intentional, but sure would
call it an unexpected change in behavior :)
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: numactl using it's own affinity to determine CPU set
2012-10-22 20:12 ` Jon Stanley
@ 2012-10-23 12:21 ` Petr Holasek
2012-10-23 14:19 ` Cliff Wickman
0 siblings, 1 reply; 6+ messages in thread
From: Petr Holasek @ 2012-10-23 12:21 UTC (permalink / raw)
To: Jon Stanley; +Cc: Andi Kleen, linux-numa, Cliff Wickman
Hi everyone,
On Mon, 22 Oct 2012, Jon Stanley wrote:
> On Mon, Oct 22, 2012 at 3:41 PM, Andi Kleen <andi@firstfloor.org> wrote:
> > That was done intentional at some point to handle cpusets.
> > numactl 1.0 or so didn't have that problem.
>
> I'll happily admit to not being an expert here :)
I've added new interface for parsing cpusets/nodesets with _all() suffix
in 2.0.8-rc5 libnuma, but numactl utility still uses old ones with cpuset
awareness. I didn't want to dig into some libnuma internals and change
behaviour of dependent utils in such way.
>
> However, I thought that the concept of cpusets was that if you
> attempted to set affinity outside of your cpuset, that would silently
> fail. I assume that the reason that libnuma didn't want to set outside
> of it's own affinity mask is to avoid such a silent failure?
>
> Also, I question the Real World(TM) prevalence of cpusets.
>
> > However it's unclear if it's really bug.
>
> Not sure if I'd call it a bug since it was intentional, but sure would
> call it an unexpected change in behavior :)
I like the idea of some overriding option proposed by Andi.
btw, Cliff, can you remember what was the reason for this change in some
version after 1.0?
Petr
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: numactl using it's own affinity to determine CPU set
2012-10-23 12:21 ` Petr Holasek
@ 2012-10-23 14:19 ` Cliff Wickman
2012-10-23 21:17 ` Petr Holasek
0 siblings, 1 reply; 6+ messages in thread
From: Cliff Wickman @ 2012-10-23 14:19 UTC (permalink / raw)
To: Petr Holasek; +Cc: Jon Stanley, Andi Kleen, linux-numa
Hi Petr,
On Tue, Oct 23, 2012 at 02:21:40PM +0200, Petr Holasek wrote:
> Hi everyone,
>
> On Mon, 22 Oct 2012, Jon Stanley wrote:
> > On Mon, Oct 22, 2012 at 3:41 PM, Andi Kleen <andi@firstfloor.org> wrote:
> > > That was done intentional at some point to handle cpusets.
> > > numactl 1.0 or so didn't have that problem.
> >
> > I'll happily admit to not being an expert here :)
>
> I've added new interface for parsing cpusets/nodesets with _all() suffix
> in 2.0.8-rc5 libnuma, but numactl utility still uses old ones with cpuset
> awareness. I didn't want to dig into some libnuma internals and change
> behaviour of dependent utils in such way.
>
> >
> > However, I thought that the concept of cpusets was that if you
> > attempted to set affinity outside of your cpuset, that would silently
> > fail. I assume that the reason that libnuma didn't want to set outside
> > of it's own affinity mask is to avoid such a silent failure?
> >
> > Also, I question the Real World(TM) prevalence of cpusets.
> >
> > > However it's unclear if it's really bug.
> >
> > Not sure if I'd call it a bug since it was intentional, but sure would
> > call it an unexpected change in behavior :)
>
> I like the idea of some overriding option proposed by Andi.
>
> btw, Cliff, can you remember what was the reason for this change in some
> version after 1.0?
>
> Petr
Memory nodes allowed was made cpuset-aware by the below patch.
I'm not quite sure if this is the change you are asking about.
patch 0804_ls_patch2
From: Lee Schermerhorn <lee.schermerhorn@hp.com>
Date: Wed, 02 Apr 2008 14:34:37 -0400
Depends-on: MPOL_F_MEMS_ALLOWED kernel patch
Provide libnuma API for MPOL_F_MEMS_ALLOWED flag.
Return nodes allowed by the application's current cpuset context
via new API numa_get_mems_allowed().
+.BR numa_get_mems_allowed()
+returns the mask of nodes from which the process is allowed to allocate
+memory in it's current cpuset context.
-Cliff
--
Cliff Wickman
SGI
cpw@sgi.com
(651) 683-3824
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: numactl using it's own affinity to determine CPU set
2012-10-23 14:19 ` Cliff Wickman
@ 2012-10-23 21:17 ` Petr Holasek
0 siblings, 0 replies; 6+ messages in thread
From: Petr Holasek @ 2012-10-23 21:17 UTC (permalink / raw)
To: Cliff Wickman; +Cc: Jon Stanley, Andi Kleen, linux-numa
On Tue, 23 Oct 2012, Cliff Wickman wrote:
> Hi Petr,
>
> On Tue, Oct 23, 2012 at 02:21:40PM +0200, Petr Holasek wrote:
> > btw, Cliff, can you remember what was the reason for this change in some
> > version after 1.0?
> >
> > Petr
>
> Memory nodes allowed was made cpuset-aware by the below patch.
> I'm not quite sure if this is the change you are asking about.
>
> patch 0804_ls_patch2
> From: Lee Schermerhorn <lee.schermerhorn@hp.com>
> Date: Wed, 02 Apr 2008 14:34:37 -0400
>
> Depends-on: MPOL_F_MEMS_ALLOWED kernel patch
> Provide libnuma API for MPOL_F_MEMS_ALLOWED flag.
> Return nodes allowed by the application's current cpuset context
> via new API numa_get_mems_allowed().
>
> +.BR numa_get_mems_allowed()
> +returns the mask of nodes from which the process is allowed to allocate
> +memory in it's current cpuset context.
Thank you!
I'll check it.
Petr
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-10-23 21:17 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-10-22 19:35 numactl using it's own affinity to determine CPU set Jon Stanley
2012-10-22 19:41 ` Andi Kleen
2012-10-22 20:12 ` Jon Stanley
2012-10-23 12:21 ` Petr Holasek
2012-10-23 14:19 ` Cliff Wickman
2012-10-23 21:17 ` Petr Holasek
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).