* Re: Processor sets (pset) for linux kernel 2.5/2.6?
2003-04-12 11:20 Processor sets (pset) for linux kernel 2.5/2.6? Shaheed R. Haque
@ 2003-04-12 12:11 ` Dave Jones
2003-04-12 12:24 ` Antonio Vargas
` (3 subsequent siblings)
4 siblings, 0 replies; 31+ messages in thread
From: Dave Jones @ 2003-04-12 12:11 UTC (permalink / raw)
To: Shaheed R. Haque; +Cc: linux-kernel, thockin
On Sat, Apr 12, 2003 at 12:20:35PM +0100, Shaheed R. Haque wrote:
> announcing an updated patch in April 1999. Are there any plans out there to
> include this, or similar functionality in 2.5/2.6? I'm particularly interested
> in getting exclusive access to a CPU (plus or minus HT support, for now anyway).
sched_getaffinity() and sched_setaffinity()
You'll also need schedutils from http://tech9.net/rml/schedutils
all this (and more) documented at http://www.codemonkey.org.uk/post-halloween-2.5.txt
Dave
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Processor sets (pset) for linux kernel 2.5/2.6?
2003-04-12 11:20 Processor sets (pset) for linux kernel 2.5/2.6? Shaheed R. Haque
2003-04-12 12:11 ` Dave Jones
@ 2003-04-12 12:24 ` Antonio Vargas
2003-04-12 14:54 ` Martin J. Bligh
2003-04-12 18:23 ` Robert Love
2003-04-12 19:56 ` Shaheed R. Haque
` (2 subsequent siblings)
4 siblings, 2 replies; 31+ messages in thread
From: Antonio Vargas @ 2003-04-12 12:24 UTC (permalink / raw)
To: Shaheed R. Haque; +Cc: linux-kernel, thockin
On Sat, Apr 12, 2003 at 12:20:35PM +0100, Shaheed R. Haque wrote:
>
> Hi,
>
> Looking through the archive etc., I cannot see any references to pset
> functionality more recently than
>
> http://www.ussg.iu.edu/hypermail/linux/kernel/9904.1/0521.html
>
> announcing an updated patch in April 1999. Are there any plans out there to
> include this, or similar functionality in 2.5/2.6? I'm particularly interested
> in getting exclusive access to a CPU (plus or minus HT support, for now anyway).
Shaheed, I think that task->cpu_allowed covers your need, since it
describes a mask of cpu�'s allowed to run the task. CPU migration, for
example, is implemented in current scheduler by saving the
current cpu_allowed, forcing it to contain only the destination CPU and
then restoring it.
I think it's carried over by fork(), but have been unable to find
it on the sources.
HT is accounted as a NUMA SMP system with strong memory affinity
for his 2 cores, so that when running 2 HT processors (2+2 cores),
the tasks are kept preferably on the same HT processor and just bounce
cores, since they share the same cache (don't know exactly L1, L2 or
both, tough).
Greets, Antonio.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Processor sets (pset) for linux kernel 2.5/2.6?
2003-04-12 12:24 ` Antonio Vargas
@ 2003-04-12 14:54 ` Martin J. Bligh
2003-04-12 18:23 ` Robert Love
1 sibling, 0 replies; 31+ messages in thread
From: Martin J. Bligh @ 2003-04-12 14:54 UTC (permalink / raw)
To: Antonio Vargas, Shaheed R. Haque; +Cc: linux-kernel, thockin
> HT is accounted as a NUMA SMP system with strong memory affinity
> for his 2 cores, so that when running 2 HT processors (2+2 cores),
> the tasks are kept preferably on the same HT processor and just bounce
> cores, since they share the same cache (don't know exactly L1, L2 or
> both, tough).
We don't actually do that right now (though it was discussed) - we treat it
as flat SMP. Ingo had some patches to share 1 runqueue between each HT pair,
which I think is a better plan for this, but we're not seeing much
performance improvment from it.
M.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Processor sets (pset) for linux kernel 2.5/2.6?
2003-04-12 12:24 ` Antonio Vargas
2003-04-12 14:54 ` Martin J. Bligh
@ 2003-04-12 18:23 ` Robert Love
1 sibling, 0 replies; 31+ messages in thread
From: Robert Love @ 2003-04-12 18:23 UTC (permalink / raw)
To: Antonio Vargas; +Cc: Shaheed R. Haque, linux-kernel, thockin
On Sat, 2003-04-12 at 08:24, Antonio Vargas wrote:
> I think it's carried over by fork(), but have been unable to find
> it on the sources.
Yep, its inherited over fork().
Robert Love
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Re: Processor sets (pset) for linux kernel 2.5/2.6?
2003-04-12 11:20 Processor sets (pset) for linux kernel 2.5/2.6? Shaheed R. Haque
2003-04-12 12:11 ` Dave Jones
2003-04-12 12:24 ` Antonio Vargas
@ 2003-04-12 19:56 ` Shaheed R. Haque
2003-04-12 20:02 ` Robert Love
` (2 more replies)
2003-04-14 22:40 ` [RFC] patch to allow CPUs to be reserved to callers of sys_setaffinity [was Processor sets (pset) for linux kernel 2.5/2.6?] shaheed
2003-05-01 20:19 ` Working .config for a Dell 2650 for 2.5.6x? (was Re: Processor sets (pset) for linux kernel 2.5/2.6?) shaheed
4 siblings, 3 replies; 31+ messages in thread
From: Shaheed R. Haque @ 2003-04-12 19:56 UTC (permalink / raw)
To: linux-kernel; +Cc: thockin
Hmmm, AFAICS, sched_getaffinity() and sched_setaffinity()
allow the calling process to be bound to the nominated CPU(s), but that is not
the same as giving them exclusive access, is it? In other words, other
processes which have no particualr affinity needs can presumably still be
scheduled to run on the same processor.
I am looking for something more akin to the patch I referred to...or did I miss
something in the effect of set_cpus_allowed()?
Thanks for the replies, Shaheed
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Re: Processor sets (pset) for linux kernel 2.5/2.6?
2003-04-12 19:56 ` Shaheed R. Haque
@ 2003-04-12 20:02 ` Robert Love
2003-04-13 8:30 ` Shaheed R. Haque
2003-04-13 3:52 ` Re: Processor sets (pset) for linux kernel 2.5/2.6? Martin J. Bligh
2003-07-15 11:39 ` [HOWTO] Emulate processor sets (pset) for linux kernel 2.5/2.6 \"shaheed r. haque\"
2 siblings, 1 reply; 31+ messages in thread
From: Robert Love @ 2003-04-12 20:02 UTC (permalink / raw)
To: Shaheed R. Haque; +Cc: linux-kernel, thockin
On Sat, 2003-04-12 at 15:56, Shaheed R. Haque wrote:
> Hmmm, AFAICS, sched_getaffinity() and sched_setaffinity()
> allow the calling process to be bound to the nominated CPU(s), but that is not
> the same as giving them exclusive access, is it? In other words, other
> processes which have no particualr affinity needs can presumably still be
> scheduled to run on the same processor.
>
> I am looking for something more akin to the patch I referred to...or did I miss
> something in the effect of set_cpus_allowed()?
We strive for simple interfaces here in Linux :)
If you want to give them exclusive access, you need to bind all the
other processes to the other processors. One easy way to do this is to
have init bind itself elsewhere on boot.
Robert Love
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Re: Processor sets (pset) for linux kernel 2.5/2.6?
2003-04-12 20:02 ` Robert Love
@ 2003-04-13 8:30 ` Shaheed R. Haque
2003-04-13 14:28 ` Robert Love
0 siblings, 1 reply; 31+ messages in thread
From: Shaheed R. Haque @ 2003-04-13 8:30 UTC (permalink / raw)
To: Robert Love; +Cc: linux-kernel
Quoting Robert Love <rml@tech9.net>:
> We strive for simple interfaces here in Linux :)
>
> If you want to give them exclusive access, you need to bind all the
> other processes to the other processors. One easy way to do this is to
> have init bind itself elsewhere on boot.
Interesting idea. AFAICS, this involves either changing the code of /sbin/init
to set its affinity to a default cpu mask (provided by a kernel boot flag, I
presume), or using taskset-like functionality and then hoping that all current
shells etc. die before the programs I care about start.
Would it not be better to simply have the kernel use the boot flag directly as
the default CPU mask setting?
[ I can see the argument that "if you need such control, you should set up your
distro to deal with it", but that is a level of tweaking I would like to avoid
for my customers ]
Thanks, Shaheed
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Re: Processor sets (pset) for linux kernel 2.5/2.6?
2003-04-13 8:30 ` Shaheed R. Haque
@ 2003-04-13 14:28 ` Robert Love
2003-05-13 11:49 ` 2.6 must-fix list, v2 Shaheed R. Haque
0 siblings, 1 reply; 31+ messages in thread
From: Robert Love @ 2003-04-13 14:28 UTC (permalink / raw)
To: Shaheed R. Haque; +Cc: linux-kernel
On Sun, 2003-04-13 at 04:30, Shaheed R. Haque wrote:
> Interesting idea. AFAICS, this involves either changing the code of /sbin/init
> to set its affinity to a default cpu mask (provided by a kernel boot flag, I
> presume), or using taskset-like functionality and then hoping that all current
> shells etc. die before the programs I care about start.
Well, you can get away with it easier by having a program (like taskset
in schedutils) run in rc.sysinit. Have it bind init to the appropriate
processor.
> Would it not be better to simply have the kernel use the boot flag directly as
> the default CPU mask setting?
This is not a bad idea either, as it involves minimal code.
Robert Love
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-04-13 14:28 ` Robert Love
@ 2003-05-13 11:49 ` Shaheed R. Haque
2003-05-13 20:02 ` Andrew Morton
0 siblings, 1 reply; 31+ messages in thread
From: Shaheed R. Haque @ 2003-05-13 11:49 UTC (permalink / raw)
To: akpm; +Cc: linux-kernel
Can I propose an addition?
Not-ready features and speedups
===============================
kernel/fork.c
-------------
- Add ability to restrict the the default CPU affinity mask so that
sys_setaffinity() can be used to implement exclusive access to a CPU. Patch
exists on LKML.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-13 11:49 ` 2.6 must-fix list, v2 Shaheed R. Haque
@ 2003-05-13 20:02 ` Andrew Morton
2003-05-13 22:46 ` Shaheed R. Haque
2003-05-13 22:49 ` Shaheed R. Haque
0 siblings, 2 replies; 31+ messages in thread
From: Andrew Morton @ 2003-05-13 20:02 UTC (permalink / raw)
To: Shaheed R. Haque; +Cc: linux-kernel
"Shaheed R. Haque" <srhaque@iee.org> wrote:
>
> - Add ability to restrict the the default CPU affinity mask so that
> sys_setaffinity() can be used to implement exclusive access to a CPU.
Why is this useful?
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-13 20:02 ` Andrew Morton
@ 2003-05-13 22:46 ` Shaheed R. Haque
2003-05-14 2:42 ` Steven Cole
2003-05-13 22:49 ` Shaheed R. Haque
1 sibling, 1 reply; 31+ messages in thread
From: Shaheed R. Haque @ 2003-05-13 22:46 UTC (permalink / raw)
To: Andrew Morton; +Cc: linux-kernel
Quoting Andrew Morton <akpm@digeo.com>:
> "Shaheed R. Haque" <srhaque@iee.org> wrote:
> >
> > - Add ability to restrict the the default CPU affinity mask so that
> > sys_setaffinity() can be used to implement exclusive access to a CPU.
>
> Why is this useful?
Because it allows one to dedicate a CPU to a process. For example, lets say you
have a quad processor,and want to run joe-random stuff on CPU 0, but a
specialised program on CPUs 1, 2, 3 that does not want to compete with
joe-random stuff.
With sys_setaffinity(), one can set the affinity of the special program to
0xe...but the default affinity for all the joe-random stuff is still 0xf (from
cpu_online_map)! Since its impractical to to modify every single joe-random
executable to set its affinity to 0x1, a way is needed to set the default. The
logical place is in init(), a.k.a. kernel/fork.c.
I hope that make sense.
Thanks, Shaheed
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-13 22:46 ` Shaheed R. Haque
@ 2003-05-14 2:42 ` Steven Cole
2003-05-14 11:49 ` Shaheed R. Haque
0 siblings, 1 reply; 31+ messages in thread
From: Steven Cole @ 2003-05-14 2:42 UTC (permalink / raw)
To: Shaheed R. Haque; +Cc: Andrew Morton, linux-kernel
On Tue, 2003-05-13 at 16:46, Shaheed R. Haque wrote:
> Quoting Andrew Morton <akpm@digeo.com>:
>
> > "Shaheed R. Haque" <srhaque@iee.org> wrote:
> > >
> > > - Add ability to restrict the the default CPU affinity mask so that
> > > sys_setaffinity() can be used to implement exclusive access to a CPU.
> >
> > Why is this useful?
>
> Because it allows one to dedicate a CPU to a process. For example, lets say you
> have a quad processor,and want to run joe-random stuff on CPU 0, but a
> specialised program on CPUs 1, 2, 3 that does not want to compete with
> joe-random stuff.
>
> With sys_setaffinity(), one can set the affinity of the special program to
> 0xe...but the default affinity for all the joe-random stuff is still 0xf (from
> cpu_online_map)! Since its impractical to to modify every single joe-random
> executable to set its affinity to 0x1, a way is needed to set the default. The
> logical place is in init(), a.k.a. kernel/fork.c.
>
> I hope that make sense.
>
> Thanks, Shaheed
>
Is this related or not to processor shielding used by RedHawk Linux?
Here is a link to their page:
http://www.ccur.com/realtime/sys_rdhwklnx.html
I saw a presentation by these guys over a year ago. I'm not sure what
they're up to now.
Steven
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-14 2:42 ` Steven Cole
@ 2003-05-14 11:49 ` Shaheed R. Haque
2003-05-14 13:08 ` Steven Cole
0 siblings, 1 reply; 31+ messages in thread
From: Shaheed R. Haque @ 2003-05-14 11:49 UTC (permalink / raw)
To: Steven Cole; +Cc: Andrew Morton, linux-kernel
Quoting Steven Cole <elenstev@mesatop.com>:
> Is this related or not to processor shielding used by RedHawk Linux?
> Here is a link to their page:
>
> http://www.ccur.com/realtime/sys_rdhwklnx.html
>
> I saw a presentation by these guys over a year ago. I'm not sure what
> they're up to now.
Yes, if I correctly read the description of this feature, it seems to be the
same thing.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-14 11:49 ` Shaheed R. Haque
@ 2003-05-14 13:08 ` Steven Cole
0 siblings, 0 replies; 31+ messages in thread
From: Steven Cole @ 2003-05-14 13:08 UTC (permalink / raw)
To: Shaheed R. Haque; +Cc: Andrew Morton, linux-kernel
On Wed, 2003-05-14 at 05:49, Shaheed R. Haque wrote:
> Quoting Steven Cole <elenstev@mesatop.com>:
>
> > Is this related or not to processor shielding used by RedHawk Linux?
> > Here is a link to their page:
> >
> > http://www.ccur.com/realtime/sys_rdhwklnx.html
> >
> > I saw a presentation by these guys over a year ago. I'm not sure what
> > they're up to now.
>
> Yes, if I correctly read the description of this feature, it seems to be the
> same thing.
>
Thanks, that is what I suspected.
There seemed to be quite a bit of interest in this from the other
customers, although our facility doesn't presently need this
functionality. In the spirit of the "squeaky wheel", I'll squeak softly
for them.
>From the above web page, thus quoth the RedHawk:
"In tightly-coupled symmetric multiprocessing systems such as
Concurrent’s iHawk real-time systems, RedHawk Linux allows individual
CPUs to be shielded from interrupt processing, daemons, bottom halves,
and other Linux tasks. Processor shielding provides a highly
deterministic execution environment where interrupt response is
guaranteed. RedHawk implements shielding via the industry-accepted
shield(1) command."
Steven
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-13 20:02 ` Andrew Morton
2003-05-13 22:46 ` Shaheed R. Haque
@ 2003-05-13 22:49 ` Shaheed R. Haque
2003-05-14 11:02 ` Felipe Alfaro Solana
1 sibling, 1 reply; 31+ messages in thread
From: Shaheed R. Haque @ 2003-05-13 22:49 UTC (permalink / raw)
To: Andrew Morton; +Cc: Shaheed R. Haque, linux-kernel
Quoting Andrew Morton <akpm@digeo.com>:
> > - Add ability to restrict the the default CPU affinity mask so that
> > sys_setaffinity() can be used to implement exclusive access to a CPU.
>
> Why is this useful?
I forgot to add that the result is the rough equivalent of Digital UNIX's psets
and Irix's sysmp for my prurposes at least.
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-13 22:49 ` Shaheed R. Haque
@ 2003-05-14 11:02 ` Felipe Alfaro Solana
2003-05-14 15:59 ` Robert Love
0 siblings, 1 reply; 31+ messages in thread
From: Felipe Alfaro Solana @ 2003-05-14 11:02 UTC (permalink / raw)
To: Shaheed R. Haque; +Cc: Andrew Morton, LKML
On Wed, 2003-05-14 at 00:49, Shaheed R. Haque wrote:
> Quoting Andrew Morton <akpm@digeo.com>:
>
> > > - Add ability to restrict the the default CPU affinity mask so that
> > > sys_setaffinity() can be used to implement exclusive access to a CPU.
> >
> > Why is this useful?
>
> I forgot to add that the result is the rough equivalent of Digital UNIX's psets
> and Irix's sysmp for my prurposes at least.
And psets and fencing in Solaris too...
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-14 11:02 ` Felipe Alfaro Solana
@ 2003-05-14 15:59 ` Robert Love
2003-05-14 16:04 ` Robert Love
2003-05-14 21:01 ` shaheed
0 siblings, 2 replies; 31+ messages in thread
From: Robert Love @ 2003-05-14 15:59 UTC (permalink / raw)
To: Felipe Alfaro Solana; +Cc: Shaheed R. Haque, Andrew Morton, LKML
On Wed, 2003-05-14 at 07:02, Felipe Alfaro Solana wrote:
> > > > - Add ability to restrict the the default CPU affinity mask so that
> > > > sys_setaffinity() can be used to implement exclusive access to a CPU.
> > >
> > > Why is this useful?
> >
> > I forgot to add that the result is the rough equivalent of Digital UNIX's psets
> > and Irix's sysmp for my prurposes at least.
>
> And psets and fencing in Solaris too...
You can get exclusive access with mangling the system call, simply by
having init bind itself to the non-exclusive processors on boot.
Try it. Every task will then end up on only the non-exclusive
processors. Seems a very simple change to me, and one that can be done
in user-space.
You do not even have to modify init, if you do not want. Grab
http://tech9.net/rml/schedutils and put a taskset call in your rc.d
Robert Love
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-14 15:59 ` Robert Love
@ 2003-05-14 16:04 ` Robert Love
2003-05-14 21:01 ` shaheed
1 sibling, 0 replies; 31+ messages in thread
From: Robert Love @ 2003-05-14 16:04 UTC (permalink / raw)
To: Felipe Alfaro Solana; +Cc: Shaheed R. Haque, Andrew Morton, LKML
On Wed, 2003-05-14 at 11:59, Robert Love wrote:
> You can get exclusive access with mangling the system call, simply by
> having init bind itself to the non-exclusive processors on boot.
Whoops. s/with/without/
Sorry,
Robert Love
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-14 15:59 ` Robert Love
2003-05-14 16:04 ` Robert Love
@ 2003-05-14 21:01 ` shaheed
2003-05-14 21:15 ` Robert Love
1 sibling, 1 reply; 31+ messages in thread
From: shaheed @ 2003-05-14 21:01 UTC (permalink / raw)
To: Robert Love, Felipe Alfaro Solana; +Cc: Andrew Morton, LKML
On Wednesday 14 May 2003 4:59 pm, Robert Love wrote:
> You can get exclusive access with mangling the system call, simply by
> having init bind itself to the non-exclusive processors on boot.
>
> Try it. Every task will then end up on only the non-exclusive
> processors. Seems a very simple change to me, and one that can be done
> in user-space.
>
> You do not even have to modify init, if you do not want. Grab
> http://tech9.net/rml/schedutils and put a taskset call in your rc.d
Ah. I think I misread your previous note to me on this...that's why my patch
modifies init itself (it does not muck with the syscall in any way). I'll try
this as soon as I have my 2.5 multiprocessor back. BTW: what are the plans
for getting schedutils (and specifically taskset) into a normal 2.6-based
distribution? Can I be reasonably sure that this will happen?
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-14 21:01 ` shaheed
@ 2003-05-14 21:15 ` Robert Love
2003-05-15 9:19 ` Shaheed R. Haque
0 siblings, 1 reply; 31+ messages in thread
From: Robert Love @ 2003-05-14 21:15 UTC (permalink / raw)
To: shaheed; +Cc: Felipe Alfaro Solana, Andrew Morton, LKML
On Wed, 2003-05-14 at 17:01, shaheed wrote:
> Ah. I think I misread your previous note to me on this...that's why my patch
> modifies init itself (it does not muck with the syscall in any way). I'll try
> this as soon as I have my 2.5 multiprocessor back. BTW: what are the plans
> for getting schedutils (and specifically taskset) into a normal 2.6-based
> distribution? Can I be reasonably sure that this will happen?
I think it is in Debian (unstable at least).
More important to me is getting it into Red Hat and SuSE. I have heard
encouring words from Matt Wilson at Red Hat about schedutils possibly
going into Rawhide soon. It would not hurt to let Red Hat/SuSE/whoever
know that schedutils is something their customers want.
Both Red Hat and SuSE's kernels have the CPU affinity system calls
merged, so you do not need to wait until 2.6 is out to use them.
Robert Love
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-14 21:15 ` Robert Love
@ 2003-05-15 9:19 ` Shaheed R. Haque
2003-05-15 15:32 ` Robert Love
0 siblings, 1 reply; 31+ messages in thread
From: Shaheed R. Haque @ 2003-05-15 9:19 UTC (permalink / raw)
To: Robert Love; +Cc: shaheed, Felipe Alfaro Solana, Andrew Morton, LKML
Quoting Robert Love <rml@tech9.net>:
> More important to me is getting it into Red Hat and SuSE. I have heard
> encouring words from Matt Wilson at Red Hat about schedutils possibly
> going into Rawhide soon. It would not hurt to let Red Hat/SuSE/whoever
> know that schedutils is something their customers want.
>
> Both Red Hat and SuSE's kernels have the CPU affinity system calls
> merged, so you do not need to wait until 2.6 is out to use them.
These are the distros I am interested in too. I knew it was in RH AS/ES, but
are you saying it is in RH9.0? That would be good news.
On the technical point, I tried out taskset in rc.sysinit, and as you said, it
works just fine. On reflection, I feel that editing rc.sysinit is not the right
answer given the confidence/competence level of our customers' typical
sysadmins: but I can see that a carefully crafted rc5.d/S00aaaaa script could
set the affinity of the executing shell, and its parent(s) upto init to fix all
subsequent rcN.d children in the desired manner.
I do suspect that other commercial users will also baulk at editing rc.sysint,
and so have to brew the same rcN.d solution. Now, the rcN.d script hackery
would be greatly simplified if taskset had a mode of "set the affinity of the
identified process, and all its parent processes upto init". Would you accept a
patch to taskset along those lines?
I think that would be a very acceptable, easy to deploy, solution.
Thoughts? Shaheed
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-15 9:19 ` Shaheed R. Haque
@ 2003-05-15 15:32 ` Robert Love
2003-05-15 20:07 ` shaheed
0 siblings, 1 reply; 31+ messages in thread
From: Robert Love @ 2003-05-15 15:32 UTC (permalink / raw)
To: Shaheed R. Haque; +Cc: Felipe Alfaro Solana, Andrew Morton, LKML
On Thu, 2003-05-15 at 05:19, Shaheed R. Haque wrote:
> These are the distros I am interested in too. I knew it was in RH AS/ES, but
> are you saying it is in RH9.0? That would be good news.
No, I am saying with luck it will be in the next RH release.
> On the technical point, I tried out taskset in rc.sysinit, and as you said, it
> works just fine.
Good :)
> On reflection, I feel that editing rc.sysinit is not the right
> answer given the confidence/competence level of our customers' typical
> sysadmins: but I can see that a carefully crafted rc5.d/S00aaaaa script could
> set the affinity of the executing shell, and its parent(s) upto init to fix all
> subsequent rcN.d children in the desired manner.
>
> I do suspect that other commercial users will also baulk at editing rc.sysint,
> and so have to brew the same rcN.d solution. Now, the rcN.d script hackery
> would be greatly simplified if taskset had a mode of "set the affinity of the
> identified process, and all its parent processes upto init". Would you accept a
> patch to taskset along those lines?
It is racey to do this, so its something that should remain a hack and
not part of taskset, I think.
If you do it in rc.d, you don't need to set all the parents. rc.d is the
first thing run, so if you do it at the top of the script, nothing else
is running. Just put:
taskset <mask> 1
taskset <mask> $$
at the top of rc.d.
Another consideration is modifying init (and hopefully having said
changes merged back). Init could call sched_setaffinity() when it is
first created, based on a setting in /etc/inittab or a command line
parameter passed during boot.
My reservation is against doing it in the kernel. I do not particularly
care _how_ its done in user-space.
Robert Love
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-15 15:32 ` Robert Love
@ 2003-05-15 20:07 ` shaheed
2003-05-15 20:20 ` Robert Love
2003-05-15 20:24 ` Robert Love
0 siblings, 2 replies; 31+ messages in thread
From: shaheed @ 2003-05-15 20:07 UTC (permalink / raw)
To: Robert Love; +Cc: Felipe Alfaro Solana, Andrew Morton, LKML
On Thursday 15 May 2003 4:32 pm, Robert Love wrote:
> It is racey to do this, so its something that should remain a hack and
> not part of taskset, I think.
Hmm. I guess you are thinking of daemons?
> If you do it in rc.d, you don't need to set all the parents. rc.d is the
> first thing run, so if you do it at the top of the script, nothing else
> is running. Just put:
>
> taskset <mask> 1
> taskset <mask> $$
>
> at the top of rc.d.
Perhaps we are talking at cross purposes. As I understand it the calling chain
is:
1. kernel bootstrap
2. /sbin/init
3. bash to run /etc/rc.sysinit
4. bash to run individual /etc/rcN.d/whatever
I feel wary of doing it in 3 as you seem to suggest because I am pretty sure
this will intimidate my customers. I am happy to do it in 4 - I can avoid the
races by only doing it for the distros I care about.
That leaves options 1 and 2 for a community-wide solution. I guess I haven't
quite understood the reluctance to do it in 1 given that:
- we know who owns 1 (!!)
- AFAICS, it isn't conceptual bloat because the utility of the implementation
of sys_setaffinity() in 1 is greatly limited by not including this feature.
- its hardly physical bloat because the number of bytes required to implement
this is absolutely in the noise, and virtually all __init()ed away.
> Another consideration is modifying init (and hopefully having said
> changes merged back). Init could call sched_setaffinity() when it is
> first created, based on a setting in /etc/inittab or a command line
> parameter passed during boot.
I have no idea with whom to persue this path, and as I say, I feel that
solving this once for each distro is crazy IMHO.
> My reservation is against doing it in the kernel. I do not particularly
> care _how_ its done in user-space.
I'm sorry to appear foolish, but as explained above, I genuinely don't
understand why this does not belong in the kernel. I would be grateful for
elaboration. If I really am being thick, then just ignore me and I'll just
solve this for myself using route 4.
In any case, thanks for all the patience and kind suggestions so far - it is
appreciated.
Regards, Shaheed
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-15 20:07 ` shaheed
@ 2003-05-15 20:20 ` Robert Love
2003-05-15 20:24 ` Robert Love
1 sibling, 0 replies; 31+ messages in thread
From: Robert Love @ 2003-05-15 20:20 UTC (permalink / raw)
To: shaheed; +Cc: Felipe Alfaro Solana, Andrew Morton, LKML
On Thu, 2003-05-15 at 16:07, shaheed wrote:
> I have no idea with whom to persue this path, and as I say, I feel that
> solving this once for each distro is crazy IMHO.
It does not have to be done for each distribution. Modify the SysVinit
package directly to support this feature. All init needs to do is bind
itself to the allowed processors prior to its first fork(). This can be
done as part of the core init package, and thus all distributions
automatically reap the benefits.
If init is not modified, then it can be done in rc.d or wherever by
hand.
> I'm sorry to appear foolish, but as explained above, I genuinely don't
> understand why this does not belong in the kernel. I would be grateful for
> elaboration. If I really am being thick, then just ignore me and I'll just
> solve this for myself using route 4.
Things which can be done in user-space should be done in user-space.
There is absolutely zero reason to do this in the kernel. init can do
it.
Submit a patch to the init maintainer to have it bind itself on boot to
a given command line value. Maybe I will do this if I find the time...
Robert Love
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-15 20:07 ` shaheed
2003-05-15 20:20 ` Robert Love
@ 2003-05-15 20:24 ` Robert Love
2003-05-15 21:30 ` shaheed
1 sibling, 1 reply; 31+ messages in thread
From: Robert Love @ 2003-05-15 20:24 UTC (permalink / raw)
To: shaheed; +Cc: Felipe Alfaro Solana, Andrew Morton, LKML
On Thu, 2003-05-15 at 16:07, shaheed wrote:
> I'm sorry to appear foolish, but as explained above, I genuinely don't
> understand why this does not belong in the kernel. I would be grateful for
> elaboration. If I really am being thick, then just ignore me and I'll just
> solve this for myself using route 4.
Oh, one other problem with doing it in the kernel via INIT_TASK:
You end up affining any kernel threads, which you absolutely do not want
to do _implicitly_. Maybe explicitly, but certainly not implicitly as a
blind consequence.
Doing it via init is really the way to go.
Regards,
Robert Love
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: 2.6 must-fix list, v2
2003-05-15 20:24 ` Robert Love
@ 2003-05-15 21:30 ` shaheed
0 siblings, 0 replies; 31+ messages in thread
From: shaheed @ 2003-05-15 21:30 UTC (permalink / raw)
To: Robert Love; +Cc: Felipe Alfaro Solana, Andrew Morton, LKML
On Thursday 15 May 2003 9:24 pm, Robert Love wrote:
> Oh, one other problem with doing it in the kernel via INIT_TASK:
>
> You end up affining any kernel threads, which you absolutely do not want
> to do _implicitly_. Maybe explicitly, but certainly not implicitly as a
> blind consequence.
>
> Doing it via init is really the way to go.
OK, that does make sense.
>Submit a patch to the init maintainer to have it bind itself on boot to
>a given command line value. Maybe I will do this if I find the time...
I will have a go myself too...
[srhaque@chiswick srhaque]$ rpm -q --whatprovides /sbin/init
SysVinit-2.84-2mdk
and rpmfind.net points to ftp://ftp.cistron.nl/pub/people/miquels/software.
I'll drop you a line if I make progress.
Thanks, Shaheed
^ permalink raw reply [flat|nested] 31+ messages in thread
* Re: Re: Processor sets (pset) for linux kernel 2.5/2.6?
2003-04-12 19:56 ` Shaheed R. Haque
2003-04-12 20:02 ` Robert Love
@ 2003-04-13 3:52 ` Martin J. Bligh
2003-07-15 11:39 ` [HOWTO] Emulate processor sets (pset) for linux kernel 2.5/2.6 \"shaheed r. haque\"
2 siblings, 0 replies; 31+ messages in thread
From: Martin J. Bligh @ 2003-04-13 3:52 UTC (permalink / raw)
To: Shaheed R. Haque, linux-kernel; +Cc: thockin
> Hmmm, AFAICS, sched_getaffinity() and sched_setaffinity()
> allow the calling process to be bound to the nominated CPU(s), but that is not
> the same as giving them exclusive access, is it? In other words, other
> processes which have no particualr affinity needs can presumably still be
> scheduled to run on the same processor.
>
> I am looking for something more akin to the patch I referred to...or did I miss
> something in the effect of set_cpus_allowed()?
The NUMA scheduler work can trivially be converted into arbitrary
scheduler pools - they are not designed for dynamic modification
at the moment, but could be without too much effort I think.
Around 2.5.59 or so I posted a patch to rename them to pools, though
we had a few other things to sort out at the time. I might revive it
at some point - it's pretty simple.
M.
^ permalink raw reply [flat|nested] 31+ messages in thread
* [HOWTO] Emulate processor sets (pset) for linux kernel 2.5/2.6
2003-04-12 19:56 ` Shaheed R. Haque
2003-04-12 20:02 ` Robert Love
2003-04-13 3:52 ` Re: Processor sets (pset) for linux kernel 2.5/2.6? Martin J. Bligh
@ 2003-07-15 11:39 ` \"shaheed r. haque\"
2 siblings, 0 replies; 31+ messages in thread
From: \"shaheed r. haque\" @ 2003-07-15 11:39 UTC (permalink / raw)
To: linux-kernel; +Cc: Robert Love, Miquel van Smoorenburg
A while ago, I was banging on about kernel patches and whatnot to implement the
equivalent of "pset" functionality. After some shoves/pointers from Robert and
Miquel, I found that init(8) has the concept of an initscript(5), so that the
trivial solution to my problem is to say:
#
# initscript Executed by init(8) for every program it
# wants to spawn like this:
#
# /bin/sh /etc/initscript <id> <level> <action> <process>
#
taskset -p 1 $$
eval exec "$4"
and then modify the relevant programs to set their affinity to the other
processors as required. Bingo!
Hope this helps, Shaheed
^ permalink raw reply [flat|nested] 31+ messages in thread
* [RFC] patch to allow CPUs to be reserved to callers of sys_setaffinity [was Processor sets (pset) for linux kernel 2.5/2.6?]
2003-04-12 11:20 Processor sets (pset) for linux kernel 2.5/2.6? Shaheed R. Haque
` (2 preceding siblings ...)
2003-04-12 19:56 ` Shaheed R. Haque
@ 2003-04-14 22:40 ` shaheed
2003-05-01 20:19 ` Working .config for a Dell 2650 for 2.5.6x? (was Re: Processor sets (pset) for linux kernel 2.5/2.6?) shaheed
4 siblings, 0 replies; 31+ messages in thread
From: shaheed @ 2003-04-14 22:40 UTC (permalink / raw)
To: mingo; +Cc: linux-kernel, rml
Hi,
Following up from http://www.ussg.iu.edu/hypermail/linux/kernel/0304.1/1113.html, here is a prototype patch against 2.5.67 to allow CPUs to be reserved to callers of sys_setaffinity(). It seems to work safely on a uniprocessor, and I'll test it on a multi-processor in a few days time. I'm not requesting inclusion of this patch until then.
Any feedback on form as well as content appreciated...I'm particularly wondering if it is worth stealing sys_setaffinity()/sys_getaffinity() for PID 0 to set/return this value since that would allow user-mode code to dynamically choose a non-restricted processor if required.
Thanks, Shaheed
1. The patch is against a clean 2.5.67
2. There is a new boot flag "restricted_cpus". This takes a bit mask of CPUs.
3. The initial task has it's cpus_allowed mask set to ~restricted_cpus in the fork initialisation code (it cannot be done in TASK_INIT since the value is not static). This should mean that any task that does not override cpus_allowed with sys_setaffinity does not get to run on the restricted processors.
4. The parsing of the boot flag does a sanity check to prevent all available CPUs being restricted.
5. Patch follows...
diff -urN -X dontdiff linux-2.5.67/kernel/fork.c newlinux-2.5.67/kernel/fork.c
--- linux-2.5.67/kernel/fork.c 2003-04-07 18:30:42.000000000 +0100
+++ newlinux-2.5.67/kernel/fork.c 2003-04-14 21:36:59.000000000 +0100
@@ -206,6 +206,7 @@
init_task.rlim[RLIMIT_NPROC].rlim_cur = max_threads/2;
init_task.rlim[RLIMIT_NPROC].rlim_max = max_threads/2;
+ init_task.cpus_allowed = ~restricted_cpus;
}
static struct task_struct *dup_task_struct(struct task_struct *orig)
diff -urN -X dontdiff linux-2.5.67/kernel/sched.c newlinux-2.5.67/kernel/sched.c
--- linux-2.5.67/kernel/sched.c 2003-04-07 18:32:23.000000000 +0100
+++ newlinux-2.5.67/kernel/sched.c 2003-04-14 22:28:31.000000000 +0100
@@ -40,6 +40,13 @@
#endif
/*
+ * The boot flag "restricted_cpus" is negated to form the default per-task
+ * cpus_allowed mask. This has the effect of only allowing processes which
+ * call sys_setaffinity() to run on restricted_cpus.
+ */
+unsigned long restricted_cpus = 0;
+
+/*
* Convert user-nice values [ -20 ... 0 ... 19 ]
* to static priority [ MAX_RT_PRIO..MAX_PRIO-1 ],
* and back.
@@ -2338,6 +2345,34 @@
}
/*
+ * set_restricted_cpus - process boot flag to set restricted_cpus.
+ *
+ * NOTE: This function is a __setup and __init function.
+ */
+int __init set_restricted_cpus(char *options)
+{
+ unsigned long temp;
+ char *value;
+
+ if (!options || !*options)
+ return 0;
+ temp = simple_strtoul(value, &value, 0);
+
+ /*
+ * Make sure that the user has not restricted all available CPUs,
+ * otherwise a functioning system won't result!
+ */
+ if (~temp & cpu_online_map) {
+ restricted_cpus = temp;
+ } else {
+ printk(KERN_ERR "cannot restrict all 0x%lx online CPUs!\n", cpu_online_map);
+ }
+ return 0;
+}
+
+__setup("restricted_cpus=", set_restricted_cpus);
+
+/*
* migration_thread - this is a highprio system thread that performs
* thread migration by 'pulling' threads into the target runqueue.
*/
diff -urN -X dontdiff linux-2.5.67/Documentation/kernel-parameters.txt newlinux-2.5.67/Documentation/kernel-parameters.txt
--- linux-2.5.67/Documentation/kernel-parameters.txt 2003-04-07 18:33:03.000000000 +0100
+++ newlinux-2.5.67/Documentation/kernel-parameters.txt 2003-04-14 22:45:25.000000000 +0100
@@ -809,6 +809,10 @@
reserve= [KNL,BUGS] Force the kernel to ignore some iomem area
+ restricted_cpus=
+ [KNL] Don't use these CPUs for processes by default.
+ Format: <cpu_mask>
+
resume= [SWSUSP] Specify the partition device for software suspension
riscom8= [HW,SERIAL]
diff -urN -X dontdiff linux-2.5.67/include/linux/sched.h newlinux-2.5.67/include/linux/sched.h
--- linux-2.5.67/include/linux/sched.h 2003-04-07 18:30:42.000000000 +0100
+++ newlinux-2.5.67/include/linux/sched.h 2003-04-14 21:37:42.000000000 +0100
@@ -88,6 +88,7 @@
extern int nr_threads;
extern int last_pid;
+extern unsigned long restricted_cpus;
DECLARE_PER_CPU(unsigned long, process_counts);
extern int nr_processes(void);
extern unsigned long nr_running(void);
^ permalink raw reply [flat|nested] 31+ messages in thread
* Working .config for a Dell 2650 for 2.5.6x? (was Re: Processor sets (pset) for linux kernel 2.5/2.6?)
2003-04-12 11:20 Processor sets (pset) for linux kernel 2.5/2.6? Shaheed R. Haque
` (3 preceding siblings ...)
2003-04-14 22:40 ` [RFC] patch to allow CPUs to be reserved to callers of sys_setaffinity [was Processor sets (pset) for linux kernel 2.5/2.6?] shaheed
@ 2003-05-01 20:19 ` shaheed
4 siblings, 0 replies; 31+ messages in thread
From: shaheed @ 2003-05-01 20:19 UTC (permalink / raw)
To: linux-kernel
Hi,
Can any kind soul with a working .config (no modules, boot from AIC7xxx JBOD
disk is fine) for a Dell 2650 for 2.5.6anything kindly send me a copy?
Off-list is probably best - I'm trying to get this patch going, but am
having an unbeliveable amount of difficulty getting this system booted.
Thanks, Shaheed
^ permalink raw reply [flat|nested] 31+ messages in thread