Kernel thread's CPU affinity with isolcpus kernel boot argument

Kernel thread's CPU affinity with isolcpus kernel boot argument

[Mathias Koehrer]

Hi all,

we use an Intel Core i7 machine (64bit) to use the cores > 0 for real 
time tasks.

For this the kernel boot parameter “isolcpus=1-31” is provided.
That works fine for all user space tasks.

However, I noticed that most kernel threads use the affinity mask 0xff.
I can change this by using “taskset”. However, I am wondering if there 
is mechanism that forces the kernel to consider the value of “isolcpus” 
also for kernel threads.

Kernel version in use is: 3.18.20-rt18.

Thanks for any feedback on this.

Best regards

Mathias


--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel thread's CPU affinity with isolcpus kernel boot argument

[Sebastian Andrzej Siewior]

* Mathias Koehrer | 2016-01-15 12:15:28 [+0100]:

>However, I noticed that most kernel threads use the affinity mask 0xff.
>I can change this by using “taskset”. However, I am wondering if
>there is mechanism that forces the kernel to consider the value of
>“isolcpus” also for kernel threads.

not that I am aware of. There are a few per-CPU threads which have to
stay the way they are. I would have expected that others like kworker/u*
respect the isol CPUs and stay away.

>Best regards
>
>Mathias

Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel thread's CPU affinity with isolcpus kernel boot argument

[Josh Cartwright]

[-- Attachment #1: Type: text/plain, Size: 772 bytes --]

On Thu, Feb 25, 2016 at 03:18:06PM +0100, Sebastian Andrzej Siewior wrote:
> * Mathias Koehrer | 2016-01-15 12:15:28 [+0100]:
> 
> >However, I noticed that most kernel threads use the affinity mask 0xff.
> >I can change this by using ???taskset???. However, I am wondering if
> >there is mechanism that forces the kernel to consider the value of
> >???isolcpus??? also for kernel threads.
> 
> not that I am aware of. There are a few per-CPU threads which have to
> stay the way they are. I would have expected that others like kworker/u*
> respect the isol CPUs and stay away.

I suppose the higher level question to Mathias is: are you seeing these
threads perturb your application?  Or, are you just observing they are
affinitized to isolcpus?

  Josh

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: Kernel thread's CPU affinity with isolcpus kernel boot argument

[Daniel Bristot de Oliveira]

On 02/25/2016 11:18 AM, Sebastian Andrzej Siewior wrote:
> not that I am aware of. There are a few per-CPU threads which have to
> stay the way they are. I would have expected that others like kworker/u*
> respect the isol CPUs and stay away.

Isolcpus is applied only for user-space threads. For instance, you
still need to move rcu offload callbacks away from isolated CPUs, and
all other kernel threads as well.

In the special case of kworker/u* threads, you need to use a special
interface to do it. The interface is the following file:

/sys/devices/virtual/workqueue/cpumask

It was implemented on the following commit:

042f7df workqueue: Allow modifying low level unbound workqueue cpumask

But the default CPU mask is the "cpu_possible_mask".

Sometime ago I sent a patch to modify the default mask of the kworker/u*
threads, to avoid them on isolated CPUs by default, but it was not
acked. It was not explicitly nacked, though.

http://www.gossamer-threads.com/lists/linux/kernel/2218495

Should I try it again?

-- Daniel

Re: Kernel thread's CPU affinity with isolcpus kernel boot argument

[Sebastian Andrzej Siewior]

On 02/25/2016 04:13 PM, Daniel Bristot de Oliveira wrote:
> On 02/25/2016 11:18 AM, Sebastian Andrzej Siewior wrote:
>> not that I am aware of. There are a few per-CPU threads which have to
>> stay the way they are. I would have expected that others like kworker/u*
>> respect the isol CPUs and stay away.
> 
> Isolcpus is applied only for user-space threads. For instance, you
> still need to move rcu offload callbacks away from isolated CPUs, and
> all other kernel threads as well.

RCU is special since this is a per-CPU thread. However Paul did make an
option where you can move those to another CPU.

> In the special case of kworker/u* threads, you need to use a special
> interface to do it. The interface is the following file:
> 
> /sys/devices/virtual/workqueue/cpumask

I see it documented in "Documentation/ABI". Oh wait…

> It was implemented on the following commit:
> 
> 042f7df workqueue: Allow modifying low level unbound workqueue cpumask
> 
> But the default CPU mask is the "cpu_possible_mask".
> 
> Sometime ago I sent a patch to modify the default mask of the kworker/u*
> threads, to avoid them on isolated CPUs by default, but it was not
> acked. It was not explicitly nacked, though.
> 
> http://www.gossamer-threads.com/lists/linux/kernel/2218495
> 
> Should I try it again?

Atleast they do have a default cpumask which you can modify. I don't
see why the kworker should be special here and avoid the isolcpu mask.
I *do* support the default mask being !isolcpus and not cpu_possible.
If you have the time to try again, please do so.

> 
> -- Daniel
> 
Sebastian
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Kernel thread's CPU affinity with isolcpus kernel boot argument

[Mathias Koehrer]

Am 25.02.2016 um 15:43 schrieb Josh Cartwright:
> On Thu, Feb 25, 2016 at 03:18:06PM +0100, Sebastian Andrzej Siewior wrote:
>> * Mathias Koehrer | 2016-01-15 12:15:28 [+0100]:
>>
>>> However, I noticed that most kernel threads use the affinity mask 0xff.
>>> I can change this by using ???taskset???. However, I am wondering if
>>> there is mechanism that forces the kernel to consider the value of
>>> ???isolcpus??? also for kernel threads.
>>
>> not that I am aware of. There are a few per-CPU threads which have to
>> stay the way they are. I would have expected that others like kworker/u*
>> respect the isol CPUs and stay away.
>
> I suppose the higher level question to Mathias is: are you seeing these
> threads perturb your application?  Or, are you just observing they are
> affinitized to isolcpus?
We use isolcpus=1-31 to reserve all cores but core 0 for real time stuff.
I just noticed, that some kernel threads have an affinity to cores > 0.
So far I did not observe any disturbance from these threads.

Thanks

Mathias