question about rcuc/X tasks

question about rcuc/X tasks

[Chris Friesen]

Hi,

I'm testing the CentOS 7.3 RT kernel, 3.10.0-514.rt56.420

(I realize this is a heavily-patched kernel, but I'm hoping general information 
will still apply.)

My kernel is booted with "isolcpus=1-15 rcu_nocbs=1-15" as kernel parameters, 
and /dev/cpu_dma_latency is set to 3.  The relevent config options are set as 
follows:

CONFIG_PREEMPT=y
CONFIG_PREEMPT_RT_BASE=y
CONFIG_HAVE_PREEMPT_LAZY=y
CONFIG_PREEMPT_LAZY=y
CONFIG_PREEMPT_RT_FULL=y
CONFIG_RCU_BOOST=y
CONFIG_RCU_KTHREAD_PRIO=2
CONFIG_RCU_BOOST_DELAY=500
CONFIG_RCU_NOCB_CPU=y
CONFIG_RCU_NOCB_CPU_NONE=y

The issue I'm seeing is that when I run cyclictest, ftrace shows rcuc/<cpu> 
running for 5-6 usec quite a few times per second.

Based on the fact that Documentation/kernel-per-CPU-kthreads.txt describes 
CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the rcuc/%u kthreads from 
having any work to do, I had expected that the "rcu_nocbs=1-15" kernel parameter 
would have a similar effect.

Does anyone have any idea what's going on here, and how I can prevent the 
rcuc/<cpu> tasks from waking up?

Thanks,
Chris

Re: question about rcuc/X tasks

[Daniel Bristot de Oliveira]

Hi Chris,

On 12/12/2016 11:42 PM, Chris Friesen wrote:
> Based on the fact that Documentation/kernel-per-CPU-kthreads.txt
> describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the
> rcuc/%u kthreads from having any work to do, I had expected that the
> "rcu_nocbs=1-15" kernel parameter would have a similar effect.

As far as I recall, using CONFIG_RCU_NOCB_CPU_ALL=y will prevent the RCU
callbacks from running on rcuc/X threads (which are pinned to the CPU
X), but the rcuc/X thread is still in charge of awaking the ruco*
threads of the CPU X.

> Does anyone have any idea what's going on here, and how I can prevent
> the rcuc/<cpu> tasks from waking up?

You can try to use rcu_nocb_poll kernel parameter to unload the rcuc/X
thread of the task of awaking its respective ruco*s threads.

>From [1]:

rcu_nocb_poll	[KNL]
			Rather than requiring that offloaded CPUs
			(specified by rcu_nocbs= above) explicitly
			awaken the corresponding "rcuoN" kthreads,
			make these kthreads poll for callbacks.
			This improves the real-time response for the
			offloaded CPUs by relieving them of the need to
			wake up the corresponding kthread, but degrades
			energy efficiency by requiring that the kthreads
			periodically wake up to do the polling.

[1] https://www.kernel.org/doc/Documentation/kernel-parameters.txt

-- Daniel

Re: question about rcuc/X tasks

[Steven Rostedt]

On Thu, 15 Dec 2016 14:47:37 +0100
Daniel Bristot de Oliveira <daniel@bristot.me> wrote:

> Hi Chris,
> 
> On 12/12/2016 11:42 PM, Chris Friesen wrote:
> > Based on the fact that Documentation/kernel-per-CPU-kthreads.txt
> > describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the
> > rcuc/%u kthreads from having any work to do, I had expected that the
> > "rcu_nocbs=1-15" kernel parameter would have a similar effect.  

Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming
this is on a 16 CPUs box, in which case I don't see much of a difference
for not just using ALL as it is almost there anyway ;-)

-- Steve

> 
> As far as I recall, using CONFIG_RCU_NOCB_CPU_ALL=y will prevent the RCU
> callbacks from running on rcuc/X threads (which are pinned to the CPU
> X), but the rcuc/X thread is still in charge of awaking the ruco*
> threads of the CPU X.
> 
> > Does anyone have any idea what's going on here, and how I can prevent
> > the rcuc/<cpu> tasks from waking up?  
> 
> You can try to use rcu_nocb_poll kernel parameter to unload the rcuc/X
> thread of the task of awaking its respective ruco*s threads.
> 
> >From [1]:  
> 
> rcu_nocb_poll	[KNL]
> 			Rather than requiring that offloaded CPUs
> 			(specified by rcu_nocbs= above) explicitly
> 			awaken the corresponding "rcuoN" kthreads,
> 			make these kthreads poll for callbacks.
> 			This improves the real-time response for the
> 			offloaded CPUs by relieving them of the need to
> 			wake up the corresponding kthread, but degrades
> 			energy efficiency by requiring that the kthreads
> 			periodically wake up to do the polling.
> 
> [1] https://www.kernel.org/doc/Documentation/kernel-parameters.txt
> 
> -- Daniel
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: question about rcuc/X tasks

[Chris Friesen]

On 12/15/2016 08:07 AM, Steven Rostedt wrote:
> On Thu, 15 Dec 2016 14:47:37 +0100
> Daniel Bristot de Oliveira <daniel@bristot.me> wrote:
>
>> Hi Chris,
>>
>> On 12/12/2016 11:42 PM, Chris Friesen wrote:
>>> Based on the fact that Documentation/kernel-per-CPU-kthreads.txt
>>> describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the
>>> rcuc/%u kthreads from having any work to do, I had expected that the
>>> "rcu_nocbs=1-15" kernel parameter would have a similar effect.
>
> Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming
> this is on a 16 CPUs box, in which case I don't see much of a difference
> for not just using ALL as it is almost there anyway ;-)
>
> -- Steve

Yes, this was a 16 CPU box.

The blocker for CONFIG_RCU_NOCB_CPU_ALL is that the set of 
management/housekeeping CPUs is configurable by the end-user, so while it 
defaults to only CPU0 as management it's not guaranteed that it will always be 
that way.

On a related note, I found an old email from Paul suggesting that the various 
rcuc/X threads could be affined to the management CPUs to free up the "realtime" 
cores, but when I try that it doesn't let me change affinity.  Was that 
disallowed for technical reasons?  (It's also possible it's something local, in 
which case I need to go digging.)

Thanks,
Chris

Re: question about rcuc/X tasks

[Paul E. McKenney]

On Thu, Dec 15, 2016 at 09:07:14AM -0500, Steven Rostedt wrote:
> On Thu, 15 Dec 2016 14:47:37 +0100
> Daniel Bristot de Oliveira <daniel@bristot.me> wrote:
> 
> > Hi Chris,
> > 
> > On 12/12/2016 11:42 PM, Chris Friesen wrote:
> > > Based on the fact that Documentation/kernel-per-CPU-kthreads.txt
> > > describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the
> > > rcuc/%u kthreads from having any work to do, I had expected that the
> > > "rcu_nocbs=1-15" kernel parameter would have a similar effect.  
> 
> Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming
> this is on a 16 CPUs box, in which case I don't see much of a difference
> for not just using ALL as it is almost there anyway ;-)

Using ALL is equivalent to rcu_nocbs=0-15 (as opposed to rcu_nocbs=1-15) on
a 16-CPU box, assuming that the CPUs are numbered consecutively.  The only
case where I could see a difference is if you were running something like
Rik's nasty Java workload on CPU 0, in which case rcu_nocbs=1-15 would be
the better choice.

							Thanx, Paul

> -- Steve
> 
> > 
> > As far as I recall, using CONFIG_RCU_NOCB_CPU_ALL=y will prevent the RCU
> > callbacks from running on rcuc/X threads (which are pinned to the CPU
> > X), but the rcuc/X thread is still in charge of awaking the ruco*
> > threads of the CPU X.
> > 
> > > Does anyone have any idea what's going on here, and how I can prevent
> > > the rcuc/<cpu> tasks from waking up?  
> > 
> > You can try to use rcu_nocb_poll kernel parameter to unload the rcuc/X
> > thread of the task of awaking its respective ruco*s threads.
> > 
> > >From [1]:  
> > 
> > rcu_nocb_poll	[KNL]
> > 			Rather than requiring that offloaded CPUs
> > 			(specified by rcu_nocbs= above) explicitly
> > 			awaken the corresponding "rcuoN" kthreads,
> > 			make these kthreads poll for callbacks.
> > 			This improves the real-time response for the
> > 			offloaded CPUs by relieving them of the need to
> > 			wake up the corresponding kthread, but degrades
> > 			energy efficiency by requiring that the kthreads
> > 			periodically wake up to do the polling.
> > 
> > [1] https://www.kernel.org/doc/Documentation/kernel-parameters.txt
> > 
> > -- Daniel
> > --
> > To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: question about rcuc/X tasks

[Paul E. McKenney]

On Thu, Dec 15, 2016 at 09:20:24AM -0600, Chris Friesen wrote:
> On 12/15/2016 08:07 AM, Steven Rostedt wrote:
> >On Thu, 15 Dec 2016 14:47:37 +0100
> >Daniel Bristot de Oliveira <daniel@bristot.me> wrote:
> >
> >>Hi Chris,
> >>
> >>On 12/12/2016 11:42 PM, Chris Friesen wrote:
> >>>Based on the fact that Documentation/kernel-per-CPU-kthreads.txt
> >>>describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the
> >>>rcuc/%u kthreads from having any work to do, I had expected that the
> >>>"rcu_nocbs=1-15" kernel parameter would have a similar effect.
> >
> >Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming
> >this is on a 16 CPUs box, in which case I don't see much of a difference
> >for not just using ALL as it is almost there anyway ;-)
> >
> >-- Steve
> 
> Yes, this was a 16 CPU box.
> 
> The blocker for CONFIG_RCU_NOCB_CPU_ALL is that the set of
> management/housekeeping CPUs is configurable by the end-user, so
> while it defaults to only CPU0 as management it's not guaranteed
> that it will always be that way.
> 
> On a related note, I found an old email from Paul suggesting that
> the various rcuc/X threads could be affined to the management CPUs
> to free up the "realtime" cores, but when I try that it doesn't let
> me change affinity.  Was that disallowed for technical reasons?
> (It's also possible it's something local, in which case I need to go
> digging.)

The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on
the corresponding CPU for correctness reasons -- they communicate with
RCU core using protocols that are only single-CPU-safe.  But if you are
running NO_HZ_FULL, these kthreads should never run unless your user
threads are doing syscalls.

So, are they actually running in your setup?

							Thanx, Paul

Re: question about rcuc/X tasks

[Chris Friesen]

On 12/15/2016 01:04 PM, Paul E. McKenney wrote:
> On Thu, Dec 15, 2016 at 09:20:24AM -0600, Chris Friesen wrote:

>> On a related note, I found an old email from Paul suggesting that
>> the various rcuc/X threads could be affined to the management CPUs
>> to free up the "realtime" cores, but when I try that it doesn't let
>> me change affinity.  Was that disallowed for technical reasons?
>> (It's also possible it's something local, in which case I need to go
>> digging.)
>
> The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on
> the corresponding CPU for correctness reasons -- they communicate with
> RCU core using protocols that are only single-CPU-safe.  But if you are
> running NO_HZ_FULL, these kthreads should never run unless your user
> threads are doing syscalls.
>
> So, are they actually running in your setup?

Yes, but I wasn't setting nohz_full.  With "rcu_nocb_poll isolcpus=1-15 
rcu_nocbs=1-15 nohz_full=1-15" I'm not seeing the rcuc/X kthreads running.

So in the non-nohz_full case, what are they waking up to do?  Something 
timer-related?

Thanks,
Chris

Re: question about rcuc/X tasks

[Paul E. McKenney]

On Thu, Dec 15, 2016 at 04:23:27PM -0600, Chris Friesen wrote:
> On 12/15/2016 01:04 PM, Paul E. McKenney wrote:
> >On Thu, Dec 15, 2016 at 09:20:24AM -0600, Chris Friesen wrote:
> 
> >>On a related note, I found an old email from Paul suggesting that
> >>the various rcuc/X threads could be affined to the management CPUs
> >>to free up the "realtime" cores, but when I try that it doesn't let
> >>me change affinity.  Was that disallowed for technical reasons?
> >>(It's also possible it's something local, in which case I need to go
> >>digging.)
> >
> >The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on
> >the corresponding CPU for correctness reasons -- they communicate with
> >RCU core using protocols that are only single-CPU-safe.  But if you are
> >running NO_HZ_FULL, these kthreads should never run unless your user
> >threads are doing syscalls.
> >
> >So, are they actually running in your setup?
> 
> Yes, but I wasn't setting nohz_full.  With "rcu_nocb_poll
> isolcpus=1-15 rcu_nocbs=1-15 nohz_full=1-15" I'm not seeing the
> rcuc/X kthreads running.
> 
> So in the non-nohz_full case, what are they waking up to do?
> Something timer-related?

Interesting.  I need to look into this a bit.  I would not expect
that rcuc/X kthreads corresponding to NOCB CPUs to ever wake up.
(They are created by a per-CPU facility that creates a kthread per
CPU no matter what.)

							Thanx, Paul

Re: question about rcuc/X tasks

[Chris Friesen]

On 12/15/2016 05:34 PM, Paul E. McKenney wrote:
> On Thu, Dec 15, 2016 at 04:23:27PM -0600, Chris Friesen wrote:
>> On 12/15/2016 01:04 PM, Paul E. McKenney wrote:
>>> On Thu, Dec 15, 2016 at 09:20:24AM -0600, Chris Friesen wrote:
>>
>>>> On a related note, I found an old email from Paul suggesting that
>>>> the various rcuc/X threads could be affined to the management CPUs
>>>> to free up the "realtime" cores, but when I try that it doesn't let
>>>> me change affinity.  Was that disallowed for technical reasons?
>>>> (It's also possible it's something local, in which case I need to go
>>>> digging.)
>>>
>>> The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on
>>> the corresponding CPU for correctness reasons -- they communicate with
>>> RCU core using protocols that are only single-CPU-safe.  But if you are
>>> running NO_HZ_FULL, these kthreads should never run unless your user
>>> threads are doing syscalls.
>>>
>>> So, are they actually running in your setup?
>>
>> Yes, but I wasn't setting nohz_full.  With "rcu_nocb_poll
>> isolcpus=1-15 rcu_nocbs=1-15 nohz_full=1-15" I'm not seeing the
>> rcuc/X kthreads running.
>>
>> So in the non-nohz_full case, what are they waking up to do?
>> Something timer-related?
>
> Interesting.  I need to look into this a bit.  I would not expect
> that rcuc/X kthreads corresponding to NOCB CPUs to ever wake up.
> (They are created by a per-CPU facility that creates a kthread per
> CPU no matter what.)


Just be aware that this is Centos 7.3, so who knows what mishmash they've got 
going on. :)

This is a typical function trace of rcuc/9, the only thing running on CPU 9 is a 
qemu thread corresponding to a virtual CPU that is pinned to CPU 9.

<idle>-0     [009] dN..2..  3335.422089: pick_next_task_dl <-__schedule
<idle>-0     [009] dN..2..  3335.422089: pick_next_task_rt <-__schedule
rcuc/9-97    [009] d...2..  3335.422089: __switch_to_xtra <-__switch_to
rcuc/9-97    [009] d...2..  3335.422089: finish_task_switch <-__schedule
rcuc/9-97    [009] d...2..  3335.422089: _raw_spin_unlock_irq <-finish_task_switch
rcuc/9-97    [009] ....1..  3335.422090: kthread_should_stop <-smpboot_thread_fn
rcuc/9-97    [009] ....1..  3335.422090: kthread_should_park <-smpboot_thread_fn
rcuc/9-97    [009] ....1..  3335.422090: rcu_cpu_kthread_should_run 
<-smpboot_thread_fn
rcuc/9-97    [009] .......  3335.422090: rcu_cpu_kthread <-smpboot_thread_fn
rcuc/9-97    [009] .......  3335.422090: local_bh_disable <-rcu_cpu_kthread
rcuc/9-97    [009] .......  3335.422090: migrate_disable <-local_bh_disable
rcuc/9-97    [009] ....11.  3335.422090: pin_current_cpu <-migrate_disable
rcuc/9-97    [009] .....11  3335.422090: rcu_process_gp_end <-rcu_cpu_kthread
rcuc/9-97    [009] .....11  3335.422090: check_for_new_grace_period.isra.26 
<-rcu_cpu_kthread
rcuc/9-97    [009] .....11  3335.422090: _raw_spin_lock_irqsave <-rcu_cpu_kthread
rcuc/9-97    [009] d...111  3335.422091: rcu_accelerate_cbs <-rcu_cpu_kthread
rcuc/9-97    [009] d...111  3335.422091: rcu_report_qs_rnp <-rcu_cpu_kthread
rcuc/9-97    [009] d...111  3335.422091: _raw_spin_unlock_irqrestore 
<-rcu_report_qs_rnp
rcuc/9-97    [009] d....11  3335.422091: cpu_needs_another_gp <-rcu_cpu_kthread
rcuc/9-97    [009] .....11  3335.422091: rcu_process_gp_end <-rcu_cpu_kthread
rcuc/9-97    [009] .....11  3335.422091: check_for_new_grace_period.isra.26 
<-rcu_cpu_kthread
rcuc/9-97    [009] d....11  3335.422091: cpu_needs_another_gp <-rcu_cpu_kthread
rcuc/9-97    [009] .....11  3335.422091: rcu_process_gp_end <-rcu_cpu_kthread
rcuc/9-97    [009] .....11  3335.422091: check_for_new_grace_period.isra.26 
<-rcu_cpu_kthread
rcuc/9-97    [009] d....11  3335.422091: cpu_needs_another_gp <-rcu_cpu_kthread
rcuc/9-97    [009] .....11  3335.422091: local_bh_enable <-rcu_cpu_kthread
rcuc/9-97    [009] .....11  3335.422092: migrate_enable <-local_bh_enable
rcuc/9-97    [009] ....11.  3335.422092: unpin_current_cpu <-migrate_enable
rcuc/9-97    [009] .......  3335.422092: _raw_spin_lock_irq <-rcu_cpu_kthread
rcuc/9-97    [009] d...1..  3335.422092: rt_mutex_getprio <-rcu_cpu_kthread
rcuc/9-97    [009] d...1..  3335.422092: _raw_spin_unlock_irq <-rcu_cpu_kthread
rcuc/9-97    [009] ....1..  3335.422092: kthread_should_stop <-smpboot_thread_fn
rcuc/9-97    [009] ....1..  3335.422092: kthread_should_park <-smpboot_thread_fn
rcuc/9-97    [009] ....1..  3335.422092: rcu_cpu_kthread_should_run 
<-smpboot_thread_fn
rcuc/9-97    [009] .......  3335.422092: schedule <-smpboot_thread_fn


Does this give any useful clues as to why it's waking up?

Looking at the code, rcu_cpu_kthread() is calling rcu_process_callbacks(), which 
will loop calling __rcu_process_callbacks() for each rcu flavor.

The fact that rcu_accelerate_cbs() and rcu_report_qs_rnp() are called within the 
spinlock for the first rcu flavor processed indicates that (rnp->qsmask & 
rdp->grpmask) is nonzero in rcu_report_qs_rdp().  I'm not sure what that 
actually means real-world though.

Then we loop through the other two rcu flavors and it doesn't look like we 
really do anything for them.

Then we return from rcu_process_callbacks() and *workp is 0 so we set the 
priority and return to the caller.

Chris