* question about rcuc/X tasks @ 2016-12-12 22:42 Chris Friesen 2016-12-15 13:47 ` Daniel Bristot de Oliveira 0 siblings, 1 reply; 9+ messages in thread From: Chris Friesen @ 2016-12-12 22:42 UTC (permalink / raw) To: linux-rt-users Hi, I'm testing the CentOS 7.3 RT kernel, 3.10.0-514.rt56.420 (I realize this is a heavily-patched kernel, but I'm hoping general information will still apply.) My kernel is booted with "isolcpus=1-15 rcu_nocbs=1-15" as kernel parameters, and /dev/cpu_dma_latency is set to 3. The relevent config options are set as follows: CONFIG_PREEMPT=y CONFIG_PREEMPT_RT_BASE=y CONFIG_HAVE_PREEMPT_LAZY=y CONFIG_PREEMPT_LAZY=y CONFIG_PREEMPT_RT_FULL=y CONFIG_RCU_BOOST=y CONFIG_RCU_KTHREAD_PRIO=2 CONFIG_RCU_BOOST_DELAY=500 CONFIG_RCU_NOCB_CPU=y CONFIG_RCU_NOCB_CPU_NONE=y The issue I'm seeing is that when I run cyclictest, ftrace shows rcuc/<cpu> running for 5-6 usec quite a few times per second. Based on the fact that Documentation/kernel-per-CPU-kthreads.txt describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the rcuc/%u kthreads from having any work to do, I had expected that the "rcu_nocbs=1-15" kernel parameter would have a similar effect. Does anyone have any idea what's going on here, and how I can prevent the rcuc/<cpu> tasks from waking up? Thanks, Chris ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: question about rcuc/X tasks 2016-12-12 22:42 question about rcuc/X tasks Chris Friesen @ 2016-12-15 13:47 ` Daniel Bristot de Oliveira 2016-12-15 14:07 ` Steven Rostedt 0 siblings, 1 reply; 9+ messages in thread From: Daniel Bristot de Oliveira @ 2016-12-15 13:47 UTC (permalink / raw) To: Chris Friesen, linux-rt-users Hi Chris, On 12/12/2016 11:42 PM, Chris Friesen wrote: > Based on the fact that Documentation/kernel-per-CPU-kthreads.txt > describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the > rcuc/%u kthreads from having any work to do, I had expected that the > "rcu_nocbs=1-15" kernel parameter would have a similar effect. As far as I recall, using CONFIG_RCU_NOCB_CPU_ALL=y will prevent the RCU callbacks from running on rcuc/X threads (which are pinned to the CPU X), but the rcuc/X thread is still in charge of awaking the ruco* threads of the CPU X. > Does anyone have any idea what's going on here, and how I can prevent > the rcuc/<cpu> tasks from waking up? You can try to use rcu_nocb_poll kernel parameter to unload the rcuc/X thread of the task of awaking its respective ruco*s threads. >From [1]: rcu_nocb_poll [KNL] Rather than requiring that offloaded CPUs (specified by rcu_nocbs= above) explicitly awaken the corresponding "rcuoN" kthreads, make these kthreads poll for callbacks. This improves the real-time response for the offloaded CPUs by relieving them of the need to wake up the corresponding kthread, but degrades energy efficiency by requiring that the kthreads periodically wake up to do the polling. [1] https://www.kernel.org/doc/Documentation/kernel-parameters.txt -- Daniel ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: question about rcuc/X tasks 2016-12-15 13:47 ` Daniel Bristot de Oliveira @ 2016-12-15 14:07 ` Steven Rostedt 2016-12-15 15:20 ` Chris Friesen 2016-12-15 19:01 ` Paul E. McKenney 0 siblings, 2 replies; 9+ messages in thread From: Steven Rostedt @ 2016-12-15 14:07 UTC (permalink / raw) To: Paul E. McKenney Cc: Daniel Bristot de Oliveira, Chris Friesen, linux-rt-users On Thu, 15 Dec 2016 14:47:37 +0100 Daniel Bristot de Oliveira <daniel@bristot.me> wrote: > Hi Chris, > > On 12/12/2016 11:42 PM, Chris Friesen wrote: > > Based on the fact that Documentation/kernel-per-CPU-kthreads.txt > > describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the > > rcuc/%u kthreads from having any work to do, I had expected that the > > "rcu_nocbs=1-15" kernel parameter would have a similar effect. Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming this is on a 16 CPUs box, in which case I don't see much of a difference for not just using ALL as it is almost there anyway ;-) -- Steve > > As far as I recall, using CONFIG_RCU_NOCB_CPU_ALL=y will prevent the RCU > callbacks from running on rcuc/X threads (which are pinned to the CPU > X), but the rcuc/X thread is still in charge of awaking the ruco* > threads of the CPU X. > > > Does anyone have any idea what's going on here, and how I can prevent > > the rcuc/<cpu> tasks from waking up? > > You can try to use rcu_nocb_poll kernel parameter to unload the rcuc/X > thread of the task of awaking its respective ruco*s threads. > > >From [1]: > > rcu_nocb_poll [KNL] > Rather than requiring that offloaded CPUs > (specified by rcu_nocbs= above) explicitly > awaken the corresponding "rcuoN" kthreads, > make these kthreads poll for callbacks. > This improves the real-time response for the > offloaded CPUs by relieving them of the need to > wake up the corresponding kthread, but degrades > energy efficiency by requiring that the kthreads > periodically wake up to do the polling. > > [1] https://www.kernel.org/doc/Documentation/kernel-parameters.txt > > -- Daniel > -- > To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: question about rcuc/X tasks 2016-12-15 14:07 ` Steven Rostedt @ 2016-12-15 15:20 ` Chris Friesen 2016-12-15 19:04 ` Paul E. McKenney 2016-12-15 19:01 ` Paul E. McKenney 1 sibling, 1 reply; 9+ messages in thread From: Chris Friesen @ 2016-12-15 15:20 UTC (permalink / raw) To: Steven Rostedt, Paul E. McKenney Cc: Daniel Bristot de Oliveira, linux-rt-users On 12/15/2016 08:07 AM, Steven Rostedt wrote: > On Thu, 15 Dec 2016 14:47:37 +0100 > Daniel Bristot de Oliveira <daniel@bristot.me> wrote: > >> Hi Chris, >> >> On 12/12/2016 11:42 PM, Chris Friesen wrote: >>> Based on the fact that Documentation/kernel-per-CPU-kthreads.txt >>> describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the >>> rcuc/%u kthreads from having any work to do, I had expected that the >>> "rcu_nocbs=1-15" kernel parameter would have a similar effect. > > Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming > this is on a 16 CPUs box, in which case I don't see much of a difference > for not just using ALL as it is almost there anyway ;-) > > -- Steve Yes, this was a 16 CPU box. The blocker for CONFIG_RCU_NOCB_CPU_ALL is that the set of management/housekeeping CPUs is configurable by the end-user, so while it defaults to only CPU0 as management it's not guaranteed that it will always be that way. On a related note, I found an old email from Paul suggesting that the various rcuc/X threads could be affined to the management CPUs to free up the "realtime" cores, but when I try that it doesn't let me change affinity. Was that disallowed for technical reasons? (It's also possible it's something local, in which case I need to go digging.) Thanks, Chris ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: question about rcuc/X tasks 2016-12-15 15:20 ` Chris Friesen @ 2016-12-15 19:04 ` Paul E. McKenney 2016-12-15 22:23 ` Chris Friesen 0 siblings, 1 reply; 9+ messages in thread From: Paul E. McKenney @ 2016-12-15 19:04 UTC (permalink / raw) To: Chris Friesen; +Cc: Steven Rostedt, Daniel Bristot de Oliveira, linux-rt-users On Thu, Dec 15, 2016 at 09:20:24AM -0600, Chris Friesen wrote: > On 12/15/2016 08:07 AM, Steven Rostedt wrote: > >On Thu, 15 Dec 2016 14:47:37 +0100 > >Daniel Bristot de Oliveira <daniel@bristot.me> wrote: > > > >>Hi Chris, > >> > >>On 12/12/2016 11:42 PM, Chris Friesen wrote: > >>>Based on the fact that Documentation/kernel-per-CPU-kthreads.txt > >>>describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the > >>>rcuc/%u kthreads from having any work to do, I had expected that the > >>>"rcu_nocbs=1-15" kernel parameter would have a similar effect. > > > >Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming > >this is on a 16 CPUs box, in which case I don't see much of a difference > >for not just using ALL as it is almost there anyway ;-) > > > >-- Steve > > Yes, this was a 16 CPU box. > > The blocker for CONFIG_RCU_NOCB_CPU_ALL is that the set of > management/housekeeping CPUs is configurable by the end-user, so > while it defaults to only CPU0 as management it's not guaranteed > that it will always be that way. > > On a related note, I found an old email from Paul suggesting that > the various rcuc/X threads could be affined to the management CPUs > to free up the "realtime" cores, but when I try that it doesn't let > me change affinity. Was that disallowed for technical reasons? > (It's also possible it's something local, in which case I need to go > digging.) The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on the corresponding CPU for correctness reasons -- they communicate with RCU core using protocols that are only single-CPU-safe. But if you are running NO_HZ_FULL, these kthreads should never run unless your user threads are doing syscalls. So, are they actually running in your setup? Thanx, Paul ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: question about rcuc/X tasks 2016-12-15 19:04 ` Paul E. McKenney @ 2016-12-15 22:23 ` Chris Friesen 2016-12-15 23:34 ` Paul E. McKenney 0 siblings, 1 reply; 9+ messages in thread From: Chris Friesen @ 2016-12-15 22:23 UTC (permalink / raw) To: paulmck; +Cc: Steven Rostedt, Daniel Bristot de Oliveira, linux-rt-users On 12/15/2016 01:04 PM, Paul E. McKenney wrote: > On Thu, Dec 15, 2016 at 09:20:24AM -0600, Chris Friesen wrote: >> On a related note, I found an old email from Paul suggesting that >> the various rcuc/X threads could be affined to the management CPUs >> to free up the "realtime" cores, but when I try that it doesn't let >> me change affinity. Was that disallowed for technical reasons? >> (It's also possible it's something local, in which case I need to go >> digging.) > > The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on > the corresponding CPU for correctness reasons -- they communicate with > RCU core using protocols that are only single-CPU-safe. But if you are > running NO_HZ_FULL, these kthreads should never run unless your user > threads are doing syscalls. > > So, are they actually running in your setup? Yes, but I wasn't setting nohz_full. With "rcu_nocb_poll isolcpus=1-15 rcu_nocbs=1-15 nohz_full=1-15" I'm not seeing the rcuc/X kthreads running. So in the non-nohz_full case, what are they waking up to do? Something timer-related? Thanks, Chris ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: question about rcuc/X tasks 2016-12-15 22:23 ` Chris Friesen @ 2016-12-15 23:34 ` Paul E. McKenney 2016-12-16 2:43 ` Chris Friesen 0 siblings, 1 reply; 9+ messages in thread From: Paul E. McKenney @ 2016-12-15 23:34 UTC (permalink / raw) To: Chris Friesen; +Cc: Steven Rostedt, Daniel Bristot de Oliveira, linux-rt-users On Thu, Dec 15, 2016 at 04:23:27PM -0600, Chris Friesen wrote: > On 12/15/2016 01:04 PM, Paul E. McKenney wrote: > >On Thu, Dec 15, 2016 at 09:20:24AM -0600, Chris Friesen wrote: > > >>On a related note, I found an old email from Paul suggesting that > >>the various rcuc/X threads could be affined to the management CPUs > >>to free up the "realtime" cores, but when I try that it doesn't let > >>me change affinity. Was that disallowed for technical reasons? > >>(It's also possible it's something local, in which case I need to go > >>digging.) > > > >The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on > >the corresponding CPU for correctness reasons -- they communicate with > >RCU core using protocols that are only single-CPU-safe. But if you are > >running NO_HZ_FULL, these kthreads should never run unless your user > >threads are doing syscalls. > > > >So, are they actually running in your setup? > > Yes, but I wasn't setting nohz_full. With "rcu_nocb_poll > isolcpus=1-15 rcu_nocbs=1-15 nohz_full=1-15" I'm not seeing the > rcuc/X kthreads running. > > So in the non-nohz_full case, what are they waking up to do? > Something timer-related? Interesting. I need to look into this a bit. I would not expect that rcuc/X kthreads corresponding to NOCB CPUs to ever wake up. (They are created by a per-CPU facility that creates a kthread per CPU no matter what.) Thanx, Paul ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: question about rcuc/X tasks 2016-12-15 23:34 ` Paul E. McKenney @ 2016-12-16 2:43 ` Chris Friesen 0 siblings, 0 replies; 9+ messages in thread From: Chris Friesen @ 2016-12-16 2:43 UTC (permalink / raw) To: paulmck; +Cc: Steven Rostedt, Daniel Bristot de Oliveira, linux-rt-users On 12/15/2016 05:34 PM, Paul E. McKenney wrote: > On Thu, Dec 15, 2016 at 04:23:27PM -0600, Chris Friesen wrote: >> On 12/15/2016 01:04 PM, Paul E. McKenney wrote: >>> On Thu, Dec 15, 2016 at 09:20:24AM -0600, Chris Friesen wrote: >> >>>> On a related note, I found an old email from Paul suggesting that >>>> the various rcuc/X threads could be affined to the management CPUs >>>> to free up the "realtime" cores, but when I try that it doesn't let >>>> me change affinity. Was that disallowed for technical reasons? >>>> (It's also possible it's something local, in which case I need to go >>>> digging.) >>> >>> The rcuo/X kthreads can be affined, but the rcuc/X kthreads must run on >>> the corresponding CPU for correctness reasons -- they communicate with >>> RCU core using protocols that are only single-CPU-safe. But if you are >>> running NO_HZ_FULL, these kthreads should never run unless your user >>> threads are doing syscalls. >>> >>> So, are they actually running in your setup? >> >> Yes, but I wasn't setting nohz_full. With "rcu_nocb_poll >> isolcpus=1-15 rcu_nocbs=1-15 nohz_full=1-15" I'm not seeing the >> rcuc/X kthreads running. >> >> So in the non-nohz_full case, what are they waking up to do? >> Something timer-related? > > Interesting. I need to look into this a bit. I would not expect > that rcuc/X kthreads corresponding to NOCB CPUs to ever wake up. > (They are created by a per-CPU facility that creates a kthread per > CPU no matter what.) Just be aware that this is Centos 7.3, so who knows what mishmash they've got going on. :) This is a typical function trace of rcuc/9, the only thing running on CPU 9 is a qemu thread corresponding to a virtual CPU that is pinned to CPU 9. <idle>-0 [009] dN..2.. 3335.422089: pick_next_task_dl <-__schedule <idle>-0 [009] dN..2.. 3335.422089: pick_next_task_rt <-__schedule rcuc/9-97 [009] d...2.. 3335.422089: __switch_to_xtra <-__switch_to rcuc/9-97 [009] d...2.. 3335.422089: finish_task_switch <-__schedule rcuc/9-97 [009] d...2.. 3335.422089: _raw_spin_unlock_irq <-finish_task_switch rcuc/9-97 [009] ....1.. 3335.422090: kthread_should_stop <-smpboot_thread_fn rcuc/9-97 [009] ....1.. 3335.422090: kthread_should_park <-smpboot_thread_fn rcuc/9-97 [009] ....1.. 3335.422090: rcu_cpu_kthread_should_run <-smpboot_thread_fn rcuc/9-97 [009] ....... 3335.422090: rcu_cpu_kthread <-smpboot_thread_fn rcuc/9-97 [009] ....... 3335.422090: local_bh_disable <-rcu_cpu_kthread rcuc/9-97 [009] ....... 3335.422090: migrate_disable <-local_bh_disable rcuc/9-97 [009] ....11. 3335.422090: pin_current_cpu <-migrate_disable rcuc/9-97 [009] .....11 3335.422090: rcu_process_gp_end <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422090: check_for_new_grace_period.isra.26 <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422090: _raw_spin_lock_irqsave <-rcu_cpu_kthread rcuc/9-97 [009] d...111 3335.422091: rcu_accelerate_cbs <-rcu_cpu_kthread rcuc/9-97 [009] d...111 3335.422091: rcu_report_qs_rnp <-rcu_cpu_kthread rcuc/9-97 [009] d...111 3335.422091: _raw_spin_unlock_irqrestore <-rcu_report_qs_rnp rcuc/9-97 [009] d....11 3335.422091: cpu_needs_another_gp <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422091: rcu_process_gp_end <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422091: check_for_new_grace_period.isra.26 <-rcu_cpu_kthread rcuc/9-97 [009] d....11 3335.422091: cpu_needs_another_gp <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422091: rcu_process_gp_end <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422091: check_for_new_grace_period.isra.26 <-rcu_cpu_kthread rcuc/9-97 [009] d....11 3335.422091: cpu_needs_another_gp <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422091: local_bh_enable <-rcu_cpu_kthread rcuc/9-97 [009] .....11 3335.422092: migrate_enable <-local_bh_enable rcuc/9-97 [009] ....11. 3335.422092: unpin_current_cpu <-migrate_enable rcuc/9-97 [009] ....... 3335.422092: _raw_spin_lock_irq <-rcu_cpu_kthread rcuc/9-97 [009] d...1.. 3335.422092: rt_mutex_getprio <-rcu_cpu_kthread rcuc/9-97 [009] d...1.. 3335.422092: _raw_spin_unlock_irq <-rcu_cpu_kthread rcuc/9-97 [009] ....1.. 3335.422092: kthread_should_stop <-smpboot_thread_fn rcuc/9-97 [009] ....1.. 3335.422092: kthread_should_park <-smpboot_thread_fn rcuc/9-97 [009] ....1.. 3335.422092: rcu_cpu_kthread_should_run <-smpboot_thread_fn rcuc/9-97 [009] ....... 3335.422092: schedule <-smpboot_thread_fn Does this give any useful clues as to why it's waking up? Looking at the code, rcu_cpu_kthread() is calling rcu_process_callbacks(), which will loop calling __rcu_process_callbacks() for each rcu flavor. The fact that rcu_accelerate_cbs() and rcu_report_qs_rnp() are called within the spinlock for the first rcu flavor processed indicates that (rnp->qsmask & rdp->grpmask) is nonzero in rcu_report_qs_rdp(). I'm not sure what that actually means real-world though. Then we loop through the other two rcu flavors and it doesn't look like we really do anything for them. Then we return from rcu_process_callbacks() and *workp is 0 so we set the priority and return to the caller. Chris ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: question about rcuc/X tasks 2016-12-15 14:07 ` Steven Rostedt 2016-12-15 15:20 ` Chris Friesen @ 2016-12-15 19:01 ` Paul E. McKenney 1 sibling, 0 replies; 9+ messages in thread From: Paul E. McKenney @ 2016-12-15 19:01 UTC (permalink / raw) To: Steven Rostedt; +Cc: Daniel Bristot de Oliveira, Chris Friesen, linux-rt-users On Thu, Dec 15, 2016 at 09:07:14AM -0500, Steven Rostedt wrote: > On Thu, 15 Dec 2016 14:47:37 +0100 > Daniel Bristot de Oliveira <daniel@bristot.me> wrote: > > > Hi Chris, > > > > On 12/12/2016 11:42 PM, Chris Friesen wrote: > > > Based on the fact that Documentation/kernel-per-CPU-kthreads.txt > > > describes CONFIG_RCU_NOCB_CPU_ALL=y as a solution by preventing the > > > rcuc/%u kthreads from having any work to do, I had expected that the > > > "rcu_nocbs=1-15" kernel parameter would have a similar effect. > > Paul, would rcu_nocbs=1-15 work? Or should ALL be used ? I'm assuming > this is on a 16 CPUs box, in which case I don't see much of a difference > for not just using ALL as it is almost there anyway ;-) Using ALL is equivalent to rcu_nocbs=0-15 (as opposed to rcu_nocbs=1-15) on a 16-CPU box, assuming that the CPUs are numbered consecutively. The only case where I could see a difference is if you were running something like Rik's nasty Java workload on CPU 0, in which case rcu_nocbs=1-15 would be the better choice. Thanx, Paul > -- Steve > > > > > As far as I recall, using CONFIG_RCU_NOCB_CPU_ALL=y will prevent the RCU > > callbacks from running on rcuc/X threads (which are pinned to the CPU > > X), but the rcuc/X thread is still in charge of awaking the ruco* > > threads of the CPU X. > > > > > Does anyone have any idea what's going on here, and how I can prevent > > > the rcuc/<cpu> tasks from waking up? > > > > You can try to use rcu_nocb_poll kernel parameter to unload the rcuc/X > > thread of the task of awaking its respective ruco*s threads. > > > > >From [1]: > > > > rcu_nocb_poll [KNL] > > Rather than requiring that offloaded CPUs > > (specified by rcu_nocbs= above) explicitly > > awaken the corresponding "rcuoN" kthreads, > > make these kthreads poll for callbacks. > > This improves the real-time response for the > > offloaded CPUs by relieving them of the need to > > wake up the corresponding kthread, but degrades > > energy efficiency by requiring that the kthreads > > periodically wake up to do the polling. > > > > [1] https://www.kernel.org/doc/Documentation/kernel-parameters.txt > > > > -- Daniel > > -- > > To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2016-12-16 4:07 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-12-12 22:42 question about rcuc/X tasks Chris Friesen 2016-12-15 13:47 ` Daniel Bristot de Oliveira 2016-12-15 14:07 ` Steven Rostedt 2016-12-15 15:20 ` Chris Friesen 2016-12-15 19:04 ` Paul E. McKenney 2016-12-15 22:23 ` Chris Friesen 2016-12-15 23:34 ` Paul E. McKenney 2016-12-16 2:43 ` Chris Friesen 2016-12-15 19:01 ` Paul E. McKenney
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.