PATCH documentation 0/2] OS-jitter documentation

PATCH documentation 0/2] OS-jitter documentation

[Paul E. McKenney]

Hello!

This is v3 of the OS-jitter-reduction documentation.

Changes from v2 (https://lkml.org/lkml/2013/4/11/337 and
https://lkml.org/lkml/2013/4/11/338):

o	Updated both patches based on review feedback.

Changes from v1 (https://lkml.org/lkml/2013/3/18/462):

o	Updated the nohz1 patch based on feedback from Frederic Weisbecker,
	Steven Rostedt, Borislav Petkov, Arjan van de Ven, Kevin Hilman,
	and Christoph Lameter.

o	Added a second file describing how to reduce OS jitter from
	per-CPU kthreads.  This is quite rough, but is hopefully a
	good starting point.

							Thanx, Paul

------------------------------------------------------------------------

 b/Documentation/kernel-per-CPU-kthreads.txt |  186 ++++++++++++++++++++
 b/Documentation/timers/NO_HZ.txt            |  255 ++++++++++++++++++++++++++++
 2 files changed, 441 insertions(+)

[PATCH documentation 1/2] nohz_full: Add documentation.

[Paul E. McKenney]

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Kevin Hilman <khilman@linaro.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Olivier Baetz <olivier.baetz@novasparks.com>
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
Reviewed-by: Borislav Petkov <bp@suse.de>
---
 Documentation/timers/NO_HZ.txt | 255 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 255 insertions(+)
 create mode 100644 Documentation/timers/NO_HZ.txt

diff --git a/Documentation/timers/NO_HZ.txt b/Documentation/timers/NO_HZ.txt
new file mode 100644
index 0000000..e3dd9af
--- /dev/null
+++ b/Documentation/timers/NO_HZ.txt
@@ -0,0 +1,255 @@
+		NO_HZ: Reducing Scheduling-Clock Ticks
+
+
+This document describes Kconfig options and boot parameters that can
+reduce the number of scheduling-clock interrupts, thereby improving energy
+efficiency and reducing OS jitter.  Reducing OS jitter is important for
+some types of computationally intensive high-performance computing (HPC)
+applications and for real-time applications.
+
+There are two main contexts in which the number of scheduling-clock
+interrupts can be reduced compared to the old-school approach of sending
+a scheduling-clock interrupt to all CPUs every jiffy whether they need
+it or not (CONFIG_HZ_PERIODIC=y or CONFIG_NO_HZ=n for older kernels):
+
+1.	Idle CPUs (CONFIG_NO_HZ_IDLE=y or CONFIG_NO_HZ=y for older kernels).
+
+2.	CPUs having only one runnable task (CONFIG_NO_HZ_FULL=y).
+
+These two cases are described in the following two sections, followed
+by a third section on RCU-specific considerations and a fourth and final
+section listing known issues.
+
+
+IDLE CPUs
+
+If a CPU is idle, there is little point in sending it a scheduling-clock
+interrupt.  After all, the primary purpose of a scheduling-clock interrupt
+is to force a busy CPU to shift its attention among multiple duties,
+and an idle CPU has no duties to shift its attention among.
+
+The CONFIG_NO_HZ_IDLE=y Kconfig option causes the kernel to avoid sending
+scheduling-clock interrupts to idle CPUs, which is critically important
+both to battery-powered devices and to highly virtualized mainframes.
+A battery-powered device running a CONFIG_HZ_PERIODIC=y kernel would
+drain its battery very quickly, easily 2-3 times as fast as would the
+same device running a CONFIG_NO_HZ_IDLE=y kernel.  A mainframe running
+1,500 OS instances might find that half of its CPU time was consumed by
+unnecessary scheduling-clock interrupts.  In these situations, there
+is strong motivation to avoid sending scheduling-clock interrupts to
+idle CPUs.  That said, dyntick-idle mode is not free:
+
+1.	It increases the number of instructions executed on the path
+	to and from the idle loop.
+
+2.	On many architectures, dyntick-idle mode also increases the
+	number of expensive clock-reprogramming operations.
+
+Therefore, systems with aggressive real-time response constraints often
+run CONFIG_HZ_PERIODIC=y kernels (or CONFIG_NO_HZ=n for older kernels)
+in order to avoid degrading from-idle transition latencies.
+
+An idle CPU that is not receiving scheduling-clock interrupts is said to
+be "dyntick-idle", "in dyntick-idle mode", "in nohz mode", or "running
+tickless".  The remainder of this document will use "dyntick-idle mode".
+
+There is also a boot parameter "nohz=" that can be used to disable
+dyntick-idle mode in CONFIG_NO_HZ_IDLE=y kernels by specifying "nohz=off".
+By default, CONFIG_NO_HZ_IDLE=y kernels boot with "nohz=on", enabling
+dyntick-idle mode.
+
+
+CPUs WITH ONLY ONE RUNNABLE TASK
+
+If a CPU has only one runnable task, there is little point in sending it
+a scheduling-clock interrupt because there is no other task to switch to.
+
+The CONFIG_NO_HZ_FULL=y Kconfig option causes the kernel to avoid
+sending scheduling-clock interrupts to CPUs with a single runnable task,
+and such CPUs are said to be "adaptive-ticks CPUs".  This is important
+for applications with aggressive real-time response constraints because
+it allows them to improve their worst-case response times by the maximum
+duration of a scheduling-clock interrupt.  It is also important for
+computationally intensive short-iteration workloads:  If any CPU is
+delayed during a given iteration, all the other CPUs will be forced to
+wait idle while the delayed CPU finishes.  Thus, the delay is multiplied
+by one less than the number of CPUs.  In these situations, there is
+again strong motivation to avoid sending scheduling-clock interrupts.
+
+By default, no CPU will be an adaptive-ticks CPU.  The "nohz_full="
+boot parameter specifies the adaptive-ticks CPUs.  For example,
+"nohz_full=1,6-8" says that CPUs 1, 6, 7, and 8 are to be adaptive-ticks
+CPUs.  Note that you are prohibited from marking all of the CPUs as
+adaptive-tick CPUs:  At least one non-adaptive-tick CPU must remain
+online to handle timekeeping tasks in order to ensure that system calls
+like gettimeofday() returns accurate values on adaptive-tick CPUs.
+(This is not an issue for CONFIG_NO_HZ_IDLE=y because there are no
+running user processes to observe slight drifts in clock rate.)
+
+Normally, a CPU remains in adaptive-ticks mode as long as possible.
+In particular, transitioning to kernel mode does not automatically change
+the mode.  Instead, the CPU will exit adaptive-ticks mode only if needed,
+for example, if that CPU enqueues an RCU callback.
+
+Just as with dyntick-idle mode, the benefits of adaptive-tick mode do
+not come for free:
+
+1.	CONFIG_NO_HZ_FULL selects CONFIG_NO_HZ_COMMON, so you cannot run
+	adaptive ticks without also running dyntick idle.  This dependency
+	extends down into the implementation, so that all of the costs
+	of CONFIG_NO_HZ_IDLE are also incurred by CONFIG_NO_HZ_FULL.
+
+2.	The user/kernel transitions are slightly more expensive due
+	to the need to inform kernel subsystems (such as RCU) about
+	the change in mode.
+
+3.	POSIX CPU timers on adaptive-tick CPUs may miss their deadlines
+	(perhaps indefinitely) because they currently rely on
+	scheduling-tick interrupts.  This will likely be fixed in
+	one of two ways: (1) Prevent CPUs with POSIX CPU timers from
+	entering adaptive-tick mode, or (2) Use hrtimers or other
+	adaptive-ticks-immune mechanism to cause the POSIX CPU timer to
+	fire properly.
+
+4.	If there are more perf events pending than the hardware can
+	accommodate, they are normally round-robined so as to collect
+	all of them over time.  Adaptive-tick mode may prevent this
+	round-robining from happening.  This will likely be fixed by
+	preventing CPUs with large numbers of perf events pending from
+	entering adaptive-tick mode.
+
+5.	Scheduler statistics for adaptive-tick CPUs may be computed
+	slightly differently than those for non-adaptive-tick CPUs.
+	This might in turn perturb load-balancing of real-time tasks.
+
+6.	The LB_BIAS scheduler feature is disabled by adaptive ticks.
+
+Although improvements are expected over time, adaptive ticks is quite
+useful for many types of real-time and compute-intensive applications.
+However, the drawbacks listed above mean that adaptive ticks should not
+(yet) be enabled by default.
+
+
+RCU IMPLICATIONS
+
+There are situations in which idle CPUs cannot be permitted to
+enter either dyntick-idle mode or adaptive-tick mode, the most
+common being when that CPU has RCU callbacks pending.
+
+The CONFIG_RCU_FAST_NO_HZ=y Kconfig option may be used to cause such CPUs
+to enter dyntick-idle mode or adaptive-tick mode anyway.  In this case,
+a timer will awaken these CPUs every four jiffies in order to ensure
+that the RCU callbacks are processed in a timely fashion.
+
+Another approach is to offload RCU callback processing to "rcuo" kthreads
+using the CONFIG_RCU_NOCB_CPU=y Kconfig option.  The specific CPUs to
+offload may be selected via several methods:
+
+1.	One of three mutually exclusive Kconfig options specify a
+	build-time default for the CPUs to offload:
+
+	a.	The CONFIG_RCU_NOCB_CPU_NONE=y Kconfig option results in
+		no CPUs being offloaded.
+
+	b.	The CONFIG_RCU_NOCB_CPU_ZERO=y Kconfig option causes
+		CPU 0 to be offloaded.
+
+	c.	The CONFIG_RCU_NOCB_CPU_ALL=y Kconfig option causes all
+		CPUs to be offloaded.  Note that the callbacks will be
+		offloaded to "rcuo" kthreads, and that those kthreads
+		will in fact run on some CPU.  However, this approach
+		gives fine-grained control on exactly which CPUs the
+		callbacks run on, along with their scheduling priority
+		(including the default of SCHED_OTHER), and it further
+		allows this control to be varied dynamically at runtime.
+
+2.	The "rcu_nocbs=" kernel boot parameter, which takes a comma-separated
+	list of CPUs and CPU ranges, for example, "1,3-5" selects CPUs 1,
+	3, 4, and 5.  The specified CPUs will be offloaded in addition to
+	any CPUs specified as offloaded by CONFIG_RCU_NOCB_CPU_ZERO=y or
+	CONFIG_RCU_NOCB_CPU_ALL=y.  This means that the "rcu_nocbs=" boot
+	parameter has no effect for kernels built with RCU_NOCB_CPU_ALL=y.
+
+The offloaded CPUs will never queue RCU callbacks, and therefore RCU
+never prevents offloaded CPUs from entering either dyntick-idle mode
+or adaptive-tick mode.  That said, note that it is up to userspace to
+pin the "rcuo" kthreads to specific CPUs if desired.  Otherwise, the
+scheduler will decide where to run them, which might or might not be
+where you want them to run.
+
+
+KNOWN ISSUES
+
+o	Dyntick-idle slows transitions to and from idle slightly.
+	In practice, this has not been a problem except for the most
+	aggressive real-time workloads, which have the option of disabling
+	dyntick-idle mode, an option that most of them take.  However,
+	some workloads will no doubt want to use adaptive ticks to
+	eliminate scheduling-clock interrupt latencies.  Here are some
+	options for these workloads:
+
+	a.	Use PMQOS from userspace to inform the kernel of your
+		latency requirements (preferred).
+
+	b.	On x86 systems, use the "idle=mwait" boot parameter.
+
+	c.	On x86 systems, use the "intel_idle.max_cstate=" to limit
+	`	the maximum C-state depth.
+
+	d.	On x86 systems, use the "idle=poll" boot parameter.
+		However, please note that use of this parameter can cause
+		your CPU to overheat, which may cause thermal throttling
+		to degrade your latencies -- and that this degradation can
+		be even worse than that of dyntick-idle.  Furthermore,
+		this parameter effectively disables Turbo Mode on Intel
+		CPUs, which can significantly reduce maximum performance.
+
+o	Adaptive-ticks slows user/kernel transitions slightly.
+	This is not expected to be a problem for computationally intensive
+	workloads, which have few such transitions.  Careful benchmarking
+	will be required to determine whether or not other workloads
+	are significantly affected by this effect.
+
+o	Adaptive-ticks does not do anything unless there is only one
+	runnable task for a given CPU, even though there are a number
+	of other situations where the scheduling-clock tick is not
+	needed.  To give but one example, consider a CPU that has one
+	runnable high-priority SCHED_FIFO task and an arbitrary number
+	of low-priority SCHED_OTHER tasks.  In this case, the CPU is
+	required to run the SCHED_FIFO task until it either blocks or
+	some other higher-priority task awakens on (or is assigned to)
+	this CPU, so there is no point in sending a scheduling-clock
+	interrupt to this CPU.	However, the current implementation
+	nevertheless sends scheduling-clock interrupts to CPUs having a
+	single runnable SCHED_FIFO task and multiple runnable SCHED_OTHER
+	tasks, even though these interrupts are unnecessary.
+
+	Better handling of these sorts of situations is future work.
+
+o	A reboot is required to reconfigure both adaptive idle and RCU
+	callback offloading.  Runtime reconfiguration could be provided
+	if needed, however, due to the complexity of reconfiguring RCU at
+	runtime, there would need to be an earthshakingly good reason.
+	Especially given that you have the straightforward option of
+	simply offloading RCU callbacks from all CPUs and pinning them
+	where you want them whenever you want them pinned.
+
+o	Additional configuration is required to deal with other sources
+	of OS jitter, including interrupts and system-utility tasks
+	and processes.  This configuration normally involves binding
+	interrupts and tasks to particular CPUs.
+
+o	Some sources of OS jitter can currently be eliminated only by
+	constraining the workload.  For example, the only way to eliminate
+	OS jitter due to global TLB shootdowns is to avoid the unmapping
+	operations (such as kernel module unload operations) that
+	result in these shootdowns.  For another example, page faults
+	and TLB misses can be reduced (and in some cases eliminated) by
+	using huge pages and by constraining the amount of memory used
+	by the application.  Pre-faulting the working set can also be
+	helpful, especially when combined with the mlock() and mlockall()
+	system calls.
+
+o	Unless all CPUs are idle, at least one CPU must keep the
+	scheduling-clock interrupt going in order to support accurate
+	timekeeping.
-- 
1.8.1.5

[PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Paul E. McKenney]

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

The Linux kernel uses a number of per-CPU kthreads, any of which might
contribute to OS jitter at any time.  The usual approach to normal
kthreads, namely to bind them to a "housekeeping" CPU, does not work
with these kthreads because they cannot operate correctly if moved to
some other CPU.  This commit therefore lists ways of controlling OS
jitter from the Linux kernel's per-CPU kthreads.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Kevin Hilman <khilman@linaro.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Olivier Baetz <olivier.baetz@novasparks.com>
Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
---
 Documentation/kernel-per-CPU-kthreads.txt | 186 ++++++++++++++++++++++++++++++
 1 file changed, 186 insertions(+)
 create mode 100644 Documentation/kernel-per-CPU-kthreads.txt

diff --git a/Documentation/kernel-per-CPU-kthreads.txt b/Documentation/kernel-per-CPU-kthreads.txt
new file mode 100644
index 0000000..bfecc1c
--- /dev/null
+++ b/Documentation/kernel-per-CPU-kthreads.txt
@@ -0,0 +1,186 @@
+REDUCING OS JITTER DUE TO PER-CPU KTHREADS
+
+This document lists per-CPU kthreads in the Linux kernel and presents
+options to control OS jitter due to these kthreads.  Note that kthreads
+that are not per-CPU are not listed here -- to reduce OS jitter from
+non-per-CPU kthreads, bind them to a "housekeeping" CPU that is dedicated
+to such work.
+
+
+REFERENCES
+
+o	Documentation/IRQ-affinity.txt:  Binding interrupts to sets of CPUs.
+
+o	Documentation/cgroups:  Using cgroups to bind tasks to sets of CPUs.
+
+o	man taskset:  Using the taskset command to bind tasks to sets
+	of CPUs.
+
+o	man sched_setaffinity:  Using the sched_setaffinity() system
+	call to bind tasks to sets of CPUs.
+
+
+KTHREADS
+
+Name: ehca_comp/%u
+Purpose: Periodically process Infiniband-related work.
+To reduce corresponding OS jitter, do any of the following:
+1.	Don't use EHCA Infiniband hardware.  This will prevent these
+	kthreads from being created in the first place.  (This will
+	work for most people, as this hardware, though important,
+	is relatively old and is produced in relatively low unit
+	volumes.)
+2.	Do all EHCA-Infiniband-related work on other CPUs, including
+	interrupts.
+
+
+Name: irq/%d-%s
+Purpose: Handle threaded interrupts.
+To reduce corresponding OS jitter, do the following:
+1.	Use irq affinity to force the irq threads to execute on
+	some other CPU.
+
+Name: kcmtpd_ctr_%d
+Purpose: Handle Bluetooth work.
+To reduce corresponding OS jitter, do one of the following:
+1.	Don't use Bluetooth, in which case these kthreads won't be
+	created in the first place.
+2.	Use irq affinity to force Bluetooth-related interrupts to
+	occur on some other CPU and furthermore initiate all
+	Bluetooth activity on some other CPU.
+
+Name: ksoftirqd/%u
+Purpose: Execute softirq handlers when threaded or when under heavy load.
+To reduce corresponding OS jitter, each softirq vector must be handled
+separately as follows:
+TIMER_SOFTIRQ:  Do all of the following:
+1.	To the extent possible, keep the CPU out of the kernel when it
+	is non-idle, for example, by avoiding system calls and by forcing
+	both kernel threads and interrupts to execute elsewhere.
+2.	Build with CONFIG_HOTPLUG_CPU=y.  After boot completes, force
+	the CPU offline, then bring it back online.  This forces
+	recurring timers to migrate elsewhere.	If you are concerned
+	with multiple CPUs, force them all offline before bringing the
+	first one back online.
+NET_TX_SOFTIRQ and NET_RX_SOFTIRQ:  Do all of the following:
+1.	Force networking interrupts onto other CPUs.
+2.	Initiate any network I/O on other CPUs.
+3.	Once your application has started, prevent CPU-hotplug operations
+	from being initiated from tasks that might run on the CPU to
+	be de-jittered.  (It is OK to force this CPU offline and then
+	bring it back online before you start your application.)
+BLOCK_SOFTIRQ:  Do all of the following:
+1.	Force block-device interrupts onto some other CPU.
+2.	Initiate any block I/O on other CPUs.
+3.	Once your application has started, prevent CPU-hotplug operations
+	from being initiated from tasks that might run on the CPU to
+	be de-jittered.  (It is OK to force this CPU offline and then
+	bring it back online before you start your application.)
+BLOCK_IOPOLL_SOFTIRQ:  Do all of the following:
+1.	Force block-device interrupts onto some other CPU.
+2.	Initiate any block I/O and block-I/O polling on other CPUs.
+3.	Once your application has started, prevent CPU-hotplug operations
+	from being initiated from tasks that might run on the CPU to
+	be de-jittered.  (It is OK to force this CPU offline and then
+	bring it back online before you start your application.)
+TASKLET_SOFTIRQ: Do one or more of the following:
+1.	Avoid use of drivers that use tasklets.
+2.	Convert all drivers that you must use from tasklets to workqueues.
+3.	Force interrupts for drivers using tasklets onto other CPUs,
+	and also do I/O involving these drivers on other CPUs.
+SCHED_SOFTIRQ: Do all of the following:
+1.	Avoid sending scheduler IPIs to the CPU to be de-jittered,
+	for example, ensure that at most one runnable kthread is
+	present on that CPU.  If a thread awakens that expects
+	to run on the de-jittered CPU, the scheduler will send
+	an IPI that can result in a subsequent SCHED_SOFTIRQ.
+2.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
+	CONFIG_NO_HZ_FULL=y, and in addition ensure that the CPU
+	to be de-jittered is marked as an adaptive-ticks CPU using the
+	"nohz_full=" boot parameter.  This reduces the number of
+	scheduler-clock interrupts that the de-jittered CPU receives,
+	minimizing its chances of being selected to do load balancing,
+	which happens in SCHED_SOFTIRQ context.
+3.	To the extent possible, keep the CPU out of the kernel when it
+	is non-idle, for example, by avoiding system calls and by
+	forcing both kernel threads and interrupts to execute elsewhere.
+	This further reduces the number of scheduler-clock interrupts
+	that the de-jittered CPU receives.
+HRTIMER_SOFTIRQ:  Do all of the following:
+1.	To the extent possible, keep the CPU out of the kernel when it
+	is non-idle, for example, by avoiding system calls and by forcing
+	both kernel threads and interrupts to execute elsewhere.
+2.	Build with CONFIG_HOTPLUG_CPU=y.  Once boot completes, force the
+	CPU offline, then bring it back online.  This forces recurring
+	timers to migrate elsewhere.  If you are concerned with multiple
+	CPUs, force them all offline before bringing the first one
+	back online.
+RCU_SOFTIRQ:  Do at least one of the following:
+1.	Offload callbacks and keep the CPU in either dyntick-idle or
+	adaptive-ticks state by doing all of the following:
+	a.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
+		CONFIG_NO_HZ_FULL=y, and in addition ensure that the CPU
+		to be de-jittered is marked as an adaptive-ticks CPU
+		using the "nohz_full=" boot parameter.	Bind the rcuo
+		kthreads to housekeeping CPUs that can tolerate OS jitter.
+	b.	To the extent possible, keep the CPU out of the kernel
+		when it is non-idle, for example, by avoiding system
+		calls and by forcing both kernel threads and interrupts
+		to execute elsewhere.
+2.	Enable RCU to do its processing remotely via dyntick-idle by
+	doing all of the following:
+	a.	Build with CONFIG_NO_HZ=y and CONFIG_RCU_FAST_NO_HZ=y.
+	b.	Ensure that the CPU goes idle frequently, allowing other
+		CPUs to detect that it has passed through an RCU quiescent
+		state.	If the kernel is built with CONFIG_NO_HZ_FULL=y,
+		userspace execution also allows other CPUs to detect that
+		the CPU in question has passed through a quiescent state.
+	c.	To the extent possible, keep the CPU out of the kernel
+		when it is non-idle, for example, by avoiding system
+		calls and by forcing both kernel threads and interrupts
+		to execute elsewhere.
+
+Name: rcuc/%u
+Purpose: Execute RCU callbacks in CONFIG_RCU_BOOST=y kernels.
+To reduce corresponding OS jitter, do at least one of the following:
+1.	Build the kernel with CONFIG_PREEMPT=n.  This prevents these
+	kthreads from being created in the first place, and also prevents
+	RCU priority boosting from ever being required.  This approach
+	is feasible for workloads that do not require high degrees of
+	responsiveness.
+2.	Build the kernel with CONFIG_RCU_BOOST=n.  This prevents these
+	kthreads from being created in the first place.  This approach
+	is feasible only if your workload never requires RCU priority
+	boosting, for example, if you ensure frequent idle time on all
+	CPUs that might execute within the kernel.
+3.	Build with CONFIG_RCU_NOCB_CPU=y and CONFIG_RCU_NOCB_CPU_ALL=y,
+	which offloads all RCU callbacks to kthreads that can be moved
+	off of CPUs susceptible to OS jitter.  This approach prevents the
+	rcuc/%u kthreads from having any work to do, so that they are
+	never awakened.
+4.	Ensure that the CPU never enters the kernel and in particular
+	avoid initiating any CPU hotplug operations on this CPU.  This is
+	another way of preventing any callbacks from being queued on the
+	CPU, again preventing the rcuc/%u kthreads from having any work
+	to do.
+
+Name: rcuob/%d, rcuop/%d, and rcuos/%d
+Purpose: Offload RCU callbacks from the corresponding CPU.
+To reduce corresponding OS jitter, do at least one of the following:
+1.	Use affinity, cgroups, or other mechanism to force these kthreads
+	to execute on some other CPU.
+2.	Build with CONFIG_RCU_NOCB_CPUS=n, which will prevent these
+	kthreads from being created in the first place.  However,
+	please note that this will not eliminate the corresponding
+	OS jitter, but will instead shift it to RCU_SOFTIRQ.
+
+Name: watchdog/%u
+Purpose: Detect software lockups on each CPU.
+To reduce corresponding OS jitter, do at least one of the following:
+1.	Build with CONFIG_LOCKUP_DETECTOR=n, which will prevent these
+	kthreads from being created in the first place.
+2.	Echo a zero to /proc/sys/kernel/watchdog to disable the
+	watchdog timer.
+3.	Echo a large number of /proc/sys/kernel/watchdog_thresh in
+	order to reduce the frequency of OS jitter due to the watchdog
+	timer down to a level that is acceptable for your workload.
-- 
1.8.1.5

Re: [PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Borislav Petkov]

On Tue, Apr 16, 2013 at 09:41:30AM -0700, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> 
> The Linux kernel uses a number of per-CPU kthreads, any of which might
> contribute to OS jitter at any time.  The usual approach to normal
> kthreads, namely to bind them to a "housekeeping" CPU, does not work
> with these kthreads because they cannot operate correctly if moved to
> some other CPU.  This commit therefore lists ways of controlling OS
> jitter from the Linux kernel's per-CPU kthreads.
> 
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Borislav Petkov <bp@alien8.de>
> Cc: Arjan van de Ven <arjan@linux.intel.com>
> Cc: Kevin Hilman <khilman@linaro.org>
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Olivier Baetz <olivier.baetz@novasparks.com>
> Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
> ---
>  Documentation/kernel-per-CPU-kthreads.txt | 186 ++++++++++++++++++++++++++++++
>  1 file changed, 186 insertions(+)
>  create mode 100644 Documentation/kernel-per-CPU-kthreads.txt
> 
> diff --git a/Documentation/kernel-per-CPU-kthreads.txt b/Documentation/kernel-per-CPU-kthreads.txt
> new file mode 100644
> index 0000000..bfecc1c
> --- /dev/null
> +++ b/Documentation/kernel-per-CPU-kthreads.txt
> @@ -0,0 +1,186 @@
> +REDUCING OS JITTER DUE TO PER-CPU KTHREADS
> +
> +This document lists per-CPU kthreads in the Linux kernel and presents
> +options to control OS jitter due to these kthreads.  Note that kthreads

s/due to/which can be caused by/

> +that are not per-CPU are not listed here -- to reduce OS jitter from

one too many "that"s:

s/that/which/

> +non-per-CPU kthreads, bind them to a "housekeeping" CPU that is dedicated

s/that/which/

> +to such work.
> +
> +
> +REFERENCES
> +
> +o	Documentation/IRQ-affinity.txt:  Binding interrupts to sets of CPUs.
> +
> +o	Documentation/cgroups:  Using cgroups to bind tasks to sets of CPUs.
> +
> +o	man taskset:  Using the taskset command to bind tasks to sets
> +	of CPUs.
> +
> +o	man sched_setaffinity:  Using the sched_setaffinity() system
> +	call to bind tasks to sets of CPUs.
> +
> +
> +KTHREADS
> +
> +Name: ehca_comp/%u
> +Purpose: Periodically process Infiniband-related work.
> +To reduce corresponding OS jitter, do any of the following:
> +1.	Don't use EHCA Infiniband hardware.  This will prevent these

Sounds like this particular hardware is slow and its IRQ handler/softirq
needs a lot of time. Yes, no?

Can we have a reason why people shouldn't use that hw.

> +	kthreads from being created in the first place.  (This will
> +	work for most people, as this hardware, though important,
> +	is relatively old and is produced in relatively low unit
> +	volumes.)
> +2.	Do all EHCA-Infiniband-related work on other CPUs, including
> +	interrupts.
> +
> +
> +Name: irq/%d-%s
> +Purpose: Handle threaded interrupts.
> +To reduce corresponding OS jitter, do the following:

This sentence keeps repeating; maybe explain the purpose of this doc in
the beginning once and drop this sentence in the later sections.

> +1.	Use irq affinity to force the irq threads to execute on
> +	some other CPU.
> +
> +Name: kcmtpd_ctr_%d
> +Purpose: Handle Bluetooth work.
> +To reduce corresponding OS jitter, do one of the following:
> +1.	Don't use Bluetooth, in which case these kthreads won't be
> +	created in the first place.
> +2.	Use irq affinity to force Bluetooth-related interrupts to
> +	occur on some other CPU and furthermore initiate all
> +	Bluetooth activity on some other CPU.
> +
> +Name: ksoftirqd/%u
> +Purpose: Execute softirq handlers when threaded or when under heavy load.
> +To reduce corresponding OS jitter, each softirq vector must be handled
> +separately as follows:
> +TIMER_SOFTIRQ:  Do all of the following:
> +1.	To the extent possible, keep the CPU out of the kernel when it
> +	is non-idle, for example, by avoiding system calls and by forcing
> +	both kernel threads and interrupts to execute elsewhere.
> +2.	Build with CONFIG_HOTPLUG_CPU=y.  After boot completes, force
> +	the CPU offline, then bring it back online.  This forces
> +	recurring timers to migrate elsewhere.	If you are concerned

We don't migrate them back to that CPU when we online it again, do we?

> +	with multiple CPUs, force them all offline before bringing the
> +	first one back online.
> +NET_TX_SOFTIRQ and NET_RX_SOFTIRQ:  Do all of the following:
> +1.	Force networking interrupts onto other CPUs.
> +2.	Initiate any network I/O on other CPUs.
> +3.	Once your application has started, prevent CPU-hotplug operations
> +	from being initiated from tasks that might run on the CPU to
> +	be de-jittered.  (It is OK to force this CPU offline and then
> +	bring it back online before you start your application.)
> +BLOCK_SOFTIRQ:  Do all of the following:
> +1.	Force block-device interrupts onto some other CPU.
> +2.	Initiate any block I/O on other CPUs.
> +3.	Once your application has started, prevent CPU-hotplug operations
> +	from being initiated from tasks that might run on the CPU to
> +	be de-jittered.  (It is OK to force this CPU offline and then
> +	bring it back online before you start your application.)
> +BLOCK_IOPOLL_SOFTIRQ:  Do all of the following:
> +1.	Force block-device interrupts onto some other CPU.
> +2.	Initiate any block I/O and block-I/O polling on other CPUs.
> +3.	Once your application has started, prevent CPU-hotplug operations
> +	from being initiated from tasks that might run on the CPU to
> +	be de-jittered.  (It is OK to force this CPU offline and then
> +	bring it back online before you start your application.)

more repeated text in brackets, maybe a footnote somewhere instead...

> +TASKLET_SOFTIRQ: Do one or more of the following:
> +1.	Avoid use of drivers that use tasklets.
> +2.	Convert all drivers that you must use from tasklets to workqueues.
> +3.	Force interrupts for drivers using tasklets onto other CPUs,
> +	and also do I/O involving these drivers on other CPUs.

How do I check which drivers use tasklets?

> +SCHED_SOFTIRQ: Do all of the following:
> +1.	Avoid sending scheduler IPIs to the CPU to be de-jittered,
> +	for example, ensure that at most one runnable kthread is

To which sentence does "for example" belong to? Depending on the answer,
you can split that sentence.

> +	present on that CPU.  If a thread awakens that expects
> +	to run on the de-jittered CPU, the scheduler will send

"If a thread expecting to run ont the de-jittered CPU awakens, the
scheduler..."

> +	an IPI that can result in a subsequent SCHED_SOFTIRQ.
> +2.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
> +	CONFIG_NO_HZ_FULL=y, and in addition ensure that the CPU

commas:

			  , and, in addition, ensure...


> +	to be de-jittered is marked as an adaptive-ticks CPU using the
> +	"nohz_full=" boot parameter.  This reduces the number of
> +	scheduler-clock interrupts that the de-jittered CPU receives,
> +	minimizing its chances of being selected to do load balancing,

I don't think there's a "," if the "which... " part refers to the
previous "load balancing" and not to the whole sentence.

> +	which happens in SCHED_SOFTIRQ context.
> +3.	To the extent possible, keep the CPU out of the kernel when it
> +	is non-idle, for example, by avoiding system calls and by
> +	forcing both kernel threads and interrupts to execute elsewhere.

This time "for example" reads ok.

> +	This further reduces the number of scheduler-clock interrupts
> +	that the de-jittered CPU receives.

s/that/which/ would suit better here IMHO.

> +HRTIMER_SOFTIRQ:  Do all of the following:
> +1.	To the extent possible, keep the CPU out of the kernel when it
> +	is non-idle, for example, by avoiding system calls and by forcing
> +	both kernel threads and interrupts to execute elsewhere.

Ok, I think I get your "for example" usage pattern.

"blabablabla. For example, do blabalbal."

I think that would be a bit more readable.

> +2.	Build with CONFIG_HOTPLUG_CPU=y.  Once boot completes, force the
> +	CPU offline, then bring it back online.  This forces recurring
> +	timers to migrate elsewhere.  If you are concerned with multiple
> +	CPUs, force them all offline before bringing the first one
> +	back online.

Same question: do the timers get migrated back when the CPU reappears
online?

> +RCU_SOFTIRQ:  Do at least one of the following:
> +1.	Offload callbacks and keep the CPU in either dyntick-idle or
> +	adaptive-ticks state by doing all of the following:
> +	a.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
> +		CONFIG_NO_HZ_FULL=y, and in addition ensure that the CPU

				   , and, in addition, 

> +		to be de-jittered is marked as an adaptive-ticks CPU
> +		using the "nohz_full=" boot parameter.	Bind the rcuo
> +		kthreads to housekeeping CPUs that can tolerate OS jitter.

					      which

> +	b.	To the extent possible, keep the CPU out of the kernel
> +		when it is non-idle, for example, by avoiding system
> +		calls and by forcing both kernel threads and interrupts
> +		to execute elsewhere.
> +2.	Enable RCU to do its processing remotely via dyntick-idle by
> +	doing all of the following:
> +	a.	Build with CONFIG_NO_HZ=y and CONFIG_RCU_FAST_NO_HZ=y.
> +	b.	Ensure that the CPU goes idle frequently, allowing other

I'm ensuring that by selecting the proper workload which has idle
breathers?

> +		CPUs to detect that it has passed through an RCU quiescent
> +		state.	If the kernel is built with CONFIG_NO_HZ_FULL=y,
> +		userspace execution also allows other CPUs to detect that
> +		the CPU in question has passed through a quiescent state.
> +	c.	To the extent possible, keep the CPU out of the kernel
> +		when it is non-idle, for example, by avoiding system
> +		calls and by forcing both kernel threads and interrupts
> +		to execute elsewhere.
> +
> +Name: rcuc/%u
> +Purpose: Execute RCU callbacks in CONFIG_RCU_BOOST=y kernels.
> +To reduce corresponding OS jitter, do at least one of the following:
> +1.	Build the kernel with CONFIG_PREEMPT=n.  This prevents these
> +	kthreads from being created in the first place, and also prevents
> +	RCU priority boosting from ever being required.  This approach

"... this obviates the need for RCU priority boosting."

> +	is feasible for workloads that do not require high degrees of
> +	responsiveness.
> +2.	Build the kernel with CONFIG_RCU_BOOST=n.  This prevents these
> +	kthreads from being created in the first place.  This approach
> +	is feasible only if your workload never requires RCU priority
> +	boosting, for example, if you ensure frequent idle time on all
> +	CPUs that might execute within the kernel.
> +3.	Build with CONFIG_RCU_NOCB_CPU=y and CONFIG_RCU_NOCB_CPU_ALL=y,
> +	which offloads all RCU callbacks to kthreads that can be moved
> +	off of CPUs susceptible to OS jitter.  This approach prevents the
> +	rcuc/%u kthreads from having any work to do, so that they are
> +	never awakened.
> +4.	Ensure that the CPU never enters the kernel and in particular

						   , and, in particular, 

> +	avoid initiating any CPU hotplug operations on this CPU.  This is
> +	another way of preventing any callbacks from being queued on the
> +	CPU, again preventing the rcuc/%u kthreads from having any work
> +	to do.
> +
> +Name: rcuob/%d, rcuop/%d, and rcuos/%d
> +Purpose: Offload RCU callbacks from the corresponding CPU.
> +To reduce corresponding OS jitter, do at least one of the following:
> +1.	Use affinity, cgroups, or other mechanism to force these kthreads
> +	to execute on some other CPU.
> +2.	Build with CONFIG_RCU_NOCB_CPUS=n, which will prevent these
> +	kthreads from being created in the first place.  However,
> +	please note that this will not eliminate the corresponding

can we drop "corresponding" here?

> +	OS jitter, but will instead shift it to RCU_SOFTIRQ.
> +
> +Name: watchdog/%u
> +Purpose: Detect software lockups on each CPU.
> +To reduce corresponding OS jitter, do at least one of the following:

ditto.

> +1.	Build with CONFIG_LOCKUP_DETECTOR=n, which will prevent these
> +	kthreads from being created in the first place.
> +2.	Echo a zero to /proc/sys/kernel/watchdog to disable the
> +	watchdog timer.
> +3.	Echo a large number of /proc/sys/kernel/watchdog_thresh in
> +	order to reduce the frequency of OS jitter due to the watchdog
> +	timer down to a level that is acceptable for your workload.


-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Paul E. McKenney]

On Sun, Apr 21, 2013 at 09:37:05PM +0200, Borislav Petkov wrote:
> On Tue, Apr 16, 2013 at 09:41:30AM -0700, Paul E. McKenney wrote:
> > From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> > 
> > The Linux kernel uses a number of per-CPU kthreads, any of which might
> > contribute to OS jitter at any time.  The usual approach to normal
> > kthreads, namely to bind them to a "housekeeping" CPU, does not work
> > with these kthreads because they cannot operate correctly if moved to
> > some other CPU.  This commit therefore lists ways of controlling OS
> > jitter from the Linux kernel's per-CPU kthreads.
> > 
> > Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Cc: Frederic Weisbecker <fweisbec@gmail.com>
> > Cc: Steven Rostedt <rostedt@goodmis.org>
> > Cc: Borislav Petkov <bp@alien8.de>
> > Cc: Arjan van de Ven <arjan@linux.intel.com>
> > Cc: Kevin Hilman <khilman@linaro.org>
> > Cc: Christoph Lameter <cl@linux.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Olivier Baetz <olivier.baetz@novasparks.com>
> > Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
> > ---
> >  Documentation/kernel-per-CPU-kthreads.txt | 186 ++++++++++++++++++++++++++++++
> >  1 file changed, 186 insertions(+)
> >  create mode 100644 Documentation/kernel-per-CPU-kthreads.txt
> > 
> > diff --git a/Documentation/kernel-per-CPU-kthreads.txt b/Documentation/kernel-per-CPU-kthreads.txt
> > new file mode 100644
> > index 0000000..bfecc1c
> > --- /dev/null
> > +++ b/Documentation/kernel-per-CPU-kthreads.txt
> > @@ -0,0 +1,186 @@
> > +REDUCING OS JITTER DUE TO PER-CPU KTHREADS
> > +
> > +This document lists per-CPU kthreads in the Linux kernel and presents
> > +options to control OS jitter due to these kthreads.  Note that kthreads
> 
> s/due to/which can be caused by/

Same meaning, but "due to" is probably a bit more arcane.  But how
about "and presents options to control these kthreads' OS jitter"?

> > +that are not per-CPU are not listed here -- to reduce OS jitter from
> 
> one too many "that"s:
> 
> s/that/which/

Fair point, but I can shorten it as follows:

	Note that non-per-CPU kthreads CPU are not listed here --
	to reduce OS jitter from non-per-CPU kthreads, bind them to a
	"housekeeping" CPU that is dedicated to such work.

> > +non-per-CPU kthreads, bind them to a "housekeeping" CPU that is dedicated
> 
> s/that/which/

Good catch -- I chose s/that is//.

> > +to such work.
> > +
> > +
> > +REFERENCES
> > +
> > +o	Documentation/IRQ-affinity.txt:  Binding interrupts to sets of CPUs.
> > +
> > +o	Documentation/cgroups:  Using cgroups to bind tasks to sets of CPUs.
> > +
> > +o	man taskset:  Using the taskset command to bind tasks to sets
> > +	of CPUs.
> > +
> > +o	man sched_setaffinity:  Using the sched_setaffinity() system
> > +	call to bind tasks to sets of CPUs.
> > +
> > +
> > +KTHREADS
> > +
> > +Name: ehca_comp/%u
> > +Purpose: Periodically process Infiniband-related work.
> > +To reduce corresponding OS jitter, do any of the following:
> > +1.	Don't use EHCA Infiniband hardware.  This will prevent these
> 
> Sounds like this particular hardware is slow and its IRQ handler/softirq
> needs a lot of time. Yes, no?
> 
> Can we have a reason why people shouldn't use that hw.

Because it has per-CPU kthreads that can cause OS jitter.  ;-)

> > +	kthreads from being created in the first place.  (This will
> > +	work for most people, as this hardware, though important,
> > +	is relatively old and is produced in relatively low unit
> > +	volumes.)
> > +2.	Do all EHCA-Infiniband-related work on other CPUs, including
> > +	interrupts.
> > +
> > +
> > +Name: irq/%d-%s
> > +Purpose: Handle threaded interrupts.
> > +To reduce corresponding OS jitter, do the following:
> 
> This sentence keeps repeating; maybe explain the purpose of this doc in
> the beginning once and drop this sentence in the later sections.

There are "any of" and "all of" qualifiers.  Also, I cannot count on
someone reading the document beginning to end.  I would instead expect
many of them to search for the name of the kthread that is bothering
them and read only that part.

> > +1.	Use irq affinity to force the irq threads to execute on
> > +	some other CPU.
> > +
> > +Name: kcmtpd_ctr_%d
> > +Purpose: Handle Bluetooth work.
> > +To reduce corresponding OS jitter, do one of the following:
> > +1.	Don't use Bluetooth, in which case these kthreads won't be
> > +	created in the first place.
> > +2.	Use irq affinity to force Bluetooth-related interrupts to
> > +	occur on some other CPU and furthermore initiate all
> > +	Bluetooth activity on some other CPU.
> > +
> > +Name: ksoftirqd/%u
> > +Purpose: Execute softirq handlers when threaded or when under heavy load.
> > +To reduce corresponding OS jitter, each softirq vector must be handled
> > +separately as follows:
> > +TIMER_SOFTIRQ:  Do all of the following:
> > +1.	To the extent possible, keep the CPU out of the kernel when it
> > +	is non-idle, for example, by avoiding system calls and by forcing
> > +	both kernel threads and interrupts to execute elsewhere.
> > +2.	Build with CONFIG_HOTPLUG_CPU=y.  After boot completes, force
> > +	the CPU offline, then bring it back online.  This forces
> > +	recurring timers to migrate elsewhere.	If you are concerned
> 
> We don't migrate them back to that CPU when we online it again, do we?

Not unless the CPU it migrated to later is taken offline.  Good point,
added words to that effect.

> > +	with multiple CPUs, force them all offline before bringing the
> > +	first one back online.
> > +NET_TX_SOFTIRQ and NET_RX_SOFTIRQ:  Do all of the following:
> > +1.	Force networking interrupts onto other CPUs.
> > +2.	Initiate any network I/O on other CPUs.
> > +3.	Once your application has started, prevent CPU-hotplug operations
> > +	from being initiated from tasks that might run on the CPU to
> > +	be de-jittered.  (It is OK to force this CPU offline and then
> > +	bring it back online before you start your application.)
> > +BLOCK_SOFTIRQ:  Do all of the following:
> > +1.	Force block-device interrupts onto some other CPU.
> > +2.	Initiate any block I/O on other CPUs.
> > +3.	Once your application has started, prevent CPU-hotplug operations
> > +	from being initiated from tasks that might run on the CPU to
> > +	be de-jittered.  (It is OK to force this CPU offline and then
> > +	bring it back online before you start your application.)
> > +BLOCK_IOPOLL_SOFTIRQ:  Do all of the following:
> > +1.	Force block-device interrupts onto some other CPU.
> > +2.	Initiate any block I/O and block-I/O polling on other CPUs.
> > +3.	Once your application has started, prevent CPU-hotplug operations
> > +	from being initiated from tasks that might run on the CPU to
> > +	be de-jittered.  (It is OK to force this CPU offline and then
> > +	bring it back online before you start your application.)
> 
> more repeated text in brackets, maybe a footnote somewhere instead...

Indeed, it is a bit repetitive, but I expect that people will tend
to look just at the part that seems relevant rather than reading the
whole thing.

> > +TASKLET_SOFTIRQ: Do one or more of the following:
> > +1.	Avoid use of drivers that use tasklets.
> > +2.	Convert all drivers that you must use from tasklets to workqueues.
> > +3.	Force interrupts for drivers using tasklets onto other CPUs,
> > +	and also do I/O involving these drivers on other CPUs.
> 
> How do I check which drivers use tasklets?

Good point -- I added "(Such drivers will contain calls to things like
tasklet_schedule().)"

> > +SCHED_SOFTIRQ: Do all of the following:
> > +1.	Avoid sending scheduler IPIs to the CPU to be de-jittered,
> > +	for example, ensure that at most one runnable kthread is
> 
> To which sentence does "for example" belong to? Depending on the answer,
> you can split that sentence.

It belongs with the first sentence.

> > +	present on that CPU.  If a thread awakens that expects
> > +	to run on the de-jittered CPU, the scheduler will send
> 
> "If a thread expecting to run ont the de-jittered CPU awakens, the
> scheduler..."

Sold!

> > +	an IPI that can result in a subsequent SCHED_SOFTIRQ.
> > +2.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
> > +	CONFIG_NO_HZ_FULL=y, and in addition ensure that the CPU
> 
> commas:
> 
> 			  , and, in addition, ensure...

Good catch, fixed.

> > +	to be de-jittered is marked as an adaptive-ticks CPU using the
> > +	"nohz_full=" boot parameter.  This reduces the number of
> > +	scheduler-clock interrupts that the de-jittered CPU receives,
> > +	minimizing its chances of being selected to do load balancing,
> 
> I don't think there's a "," if the "which... " part refers to the
> previous "load balancing" and not to the whole sentence.

Good point -- I can reword to:

	This reduces the number of scheduler-clock interrupts that the
	de-jittered CPU receives, minimizing its chances of being selected
	to do the load balancing work that runs in SCHED_SOFTIRQ context.

> > +	which happens in SCHED_SOFTIRQ context.
> > +3.	To the extent possible, keep the CPU out of the kernel when it
> > +	is non-idle, for example, by avoiding system calls and by
> > +	forcing both kernel threads and interrupts to execute elsewhere.
> 
> This time "for example" reads ok.

Glad you like it.  ;-)

> > +	This further reduces the number of scheduler-clock interrupts
> > +	that the de-jittered CPU receives.
> 
> s/that/which/ would suit better here IMHO.

Fair point, but how about this?

	This further reduces the number of scheduler-clock interrupts
	received by the de-jittered CPU.

> > +HRTIMER_SOFTIRQ:  Do all of the following:
> > +1.	To the extent possible, keep the CPU out of the kernel when it
> > +	is non-idle, for example, by avoiding system calls and by forcing
> > +	both kernel threads and interrupts to execute elsewhere.
> 
> Ok, I think I get your "for example" usage pattern.
> 
> "blabablabla. For example, do blabalbal."
> 
> I think that would be a bit more readable.

In this case, agreed:

	To the extent possible, keep the CPU out of the kernel when it
	is non-idle.  For example, avoid system calls and force both
	kernel threads and interrupts to execute elsewhere.

> > +2.	Build with CONFIG_HOTPLUG_CPU=y.  Once boot completes, force the
> > +	CPU offline, then bring it back online.  This forces recurring
> > +	timers to migrate elsewhere.  If you are concerned with multiple
> > +	CPUs, force them all offline before bringing the first one
> > +	back online.
> 
> Same question: do the timers get migrated back when the CPU reappears
> online?

Good point, applied the same change here.

> > +RCU_SOFTIRQ:  Do at least one of the following:
> > +1.	Offload callbacks and keep the CPU in either dyntick-idle or
> > +	adaptive-ticks state by doing all of the following:
> > +	a.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
> > +		CONFIG_NO_HZ_FULL=y, and in addition ensure that the CPU
> 
> 				   , and, in addition, 
> 
> > +		to be de-jittered is marked as an adaptive-ticks CPU
> > +		using the "nohz_full=" boot parameter.	Bind the rcuo
> > +		kthreads to housekeeping CPUs that can tolerate OS jitter.
> 
> 					      which

Good point, took both.

> > +	b.	To the extent possible, keep the CPU out of the kernel
> > +		when it is non-idle, for example, by avoiding system
> > +		calls and by forcing both kernel threads and interrupts
> > +		to execute elsewhere.
> > +2.	Enable RCU to do its processing remotely via dyntick-idle by
> > +	doing all of the following:
> > +	a.	Build with CONFIG_NO_HZ=y and CONFIG_RCU_FAST_NO_HZ=y.
> > +	b.	Ensure that the CPU goes idle frequently, allowing other
> 
> I'm ensuring that by selecting the proper workload which has idle
> breathers?

Yep!  Or, equivalently, by adding enough CPUs so that the workload
has idle breathers.

> > +		CPUs to detect that it has passed through an RCU quiescent
> > +		state.	If the kernel is built with CONFIG_NO_HZ_FULL=y,
> > +		userspace execution also allows other CPUs to detect that
> > +		the CPU in question has passed through a quiescent state.
> > +	c.	To the extent possible, keep the CPU out of the kernel
> > +		when it is non-idle, for example, by avoiding system
> > +		calls and by forcing both kernel threads and interrupts
> > +		to execute elsewhere.
> > +
> > +Name: rcuc/%u
> > +Purpose: Execute RCU callbacks in CONFIG_RCU_BOOST=y kernels.
> > +To reduce corresponding OS jitter, do at least one of the following:
> > +1.	Build the kernel with CONFIG_PREEMPT=n.  This prevents these
> > +	kthreads from being created in the first place, and also prevents
> > +	RCU priority boosting from ever being required.  This approach
> 
> "... this obviates the need for RCU priority boosting."

Sold!

> > +	is feasible for workloads that do not require high degrees of
> > +	responsiveness.
> > +2.	Build the kernel with CONFIG_RCU_BOOST=n.  This prevents these
> > +	kthreads from being created in the first place.  This approach
> > +	is feasible only if your workload never requires RCU priority
> > +	boosting, for example, if you ensure frequent idle time on all
> > +	CPUs that might execute within the kernel.
> > +3.	Build with CONFIG_RCU_NOCB_CPU=y and CONFIG_RCU_NOCB_CPU_ALL=y,
> > +	which offloads all RCU callbacks to kthreads that can be moved
> > +	off of CPUs susceptible to OS jitter.  This approach prevents the
> > +	rcuc/%u kthreads from having any work to do, so that they are
> > +	never awakened.
> > +4.	Ensure that the CPU never enters the kernel and in particular
> 
> 						   , and, in particular, 

Good, fixed.

> > +	avoid initiating any CPU hotplug operations on this CPU.  This is
> > +	another way of preventing any callbacks from being queued on the
> > +	CPU, again preventing the rcuc/%u kthreads from having any work
> > +	to do.
> > +
> > +Name: rcuob/%d, rcuop/%d, and rcuos/%d
> > +Purpose: Offload RCU callbacks from the corresponding CPU.
> > +To reduce corresponding OS jitter, do at least one of the following:
> > +1.	Use affinity, cgroups, or other mechanism to force these kthreads
> > +	to execute on some other CPU.
> > +2.	Build with CONFIG_RCU_NOCB_CPUS=n, which will prevent these
> > +	kthreads from being created in the first place.  However,
> > +	please note that this will not eliminate the corresponding
> 
> can we drop "corresponding" here?

Yep!  Dropped the preceding "the" as well, just to be on the safe side.

> > +	OS jitter, but will instead shift it to RCU_SOFTIRQ.
> > +
> > +Name: watchdog/%u
> > +Purpose: Detect software lockups on each CPU.
> > +To reduce corresponding OS jitter, do at least one of the following:
> 
> ditto.

I changed "corresponding" to "its" globally for this lead-in sentence.

> > +1.	Build with CONFIG_LOCKUP_DETECTOR=n, which will prevent these
> > +	kthreads from being created in the first place.
> > +2.	Echo a zero to /proc/sys/kernel/watchdog to disable the
> > +	watchdog timer.
> > +3.	Echo a large number of /proc/sys/kernel/watchdog_thresh in
> > +	order to reduce the frequency of OS jitter due to the watchdog
> > +	timer down to a level that is acceptable for your workload.

Thank you for the thorough review and comments!  Please see below for
an update.

							Thanx, Paul

------------------------------------------------------------------------

REDUCING OS JITTER DUE TO PER-CPU KTHREADS

This document lists per-CPU kthreads in the Linux kernel and presents
options to control these kthreads' OS jitter.  Note that non-per-CPU
kthreads CPU are not listed here.  To reduce OS jitter from non-per-CPU
kthreads, bind them to a "housekeeping" CPU dedicated to such work.


REFERENCES

o	Documentation/IRQ-affinity.txt:  Binding interrupts to sets of CPUs.

o	Documentation/cgroups:  Using cgroups to bind tasks to sets of CPUs.

o	man taskset:  Using the taskset command to bind tasks to sets
	of CPUs.

o	man sched_setaffinity:  Using the sched_setaffinity() system
	call to bind tasks to sets of CPUs.


KTHREADS

Name: ehca_comp/%u
Purpose: Periodically process Infiniband-related work.
To reduce its OS jitter, do any of the following:
1.	Don't use eHCA Infiniband hardware.  This will prevent these
	kthreads from being created in the first place.  (This will
	work for most people, as this hardware, though important,
	is relatively old and is produced in relatively low unit
	volumes.)
2.	Do all eHCA-Infiniband-related work on other CPUs, including
	interrupts.


Name: irq/%d-%s
Purpose: Handle threaded interrupts.
To reduce its OS jitter, do the following:
1.	Use irq affinity to force the irq threads to execute on
	some other CPU.

Name: kcmtpd_ctr_%d
Purpose: Handle Bluetooth work.
To reduce its OS jitter, do one of the following:
1.	Don't use Bluetooth, in which case these kthreads won't be
	created in the first place.
2.	Use irq affinity to force Bluetooth-related interrupts to
	occur on some other CPU and furthermore initiate all
	Bluetooth activity on some other CPU.

Name: ksoftirqd/%u
Purpose: Execute softirq handlers when threaded or when under heavy load.
To reduce its OS jitter, each softirq vector must be handled
separately as follows:
TIMER_SOFTIRQ:  Do all of the following:
1.	To the extent possible, keep the CPU out of the kernel when it
	is non-idle, for example, by avoiding system calls and by forcing
	both kernel threads and interrupts to execute elsewhere.
2.	Build with CONFIG_HOTPLUG_CPU=y.  After boot completes, force
	the CPU offline, then bring it back online.  This forces
	recurring timers to migrate elsewhere.	If you are concerned
	with multiple CPUs, force them all offline before bringing the
	first one back online.  Once you have onlined the CPUs in question,
	do not offline any other CPUs, because doing so could force the
	timer back onto one of the CPUs in question.
NET_TX_SOFTIRQ and NET_RX_SOFTIRQ:  Do all of the following:
1.	Force networking interrupts onto other CPUs.
2.	Initiate any network I/O on other CPUs.
3.	Once your application has started, prevent CPU-hotplug operations
	from being initiated from tasks that might run on the CPU to
	be de-jittered.  (It is OK to force this CPU offline and then
	bring it back online before you start your application.)
BLOCK_SOFTIRQ:  Do all of the following:
1.	Force block-device interrupts onto some other CPU.
2.	Initiate any block I/O on other CPUs.
3.	Once your application has started, prevent CPU-hotplug operations
	from being initiated from tasks that might run on the CPU to
	be de-jittered.  (It is OK to force this CPU offline and then
	bring it back online before you start your application.)
BLOCK_IOPOLL_SOFTIRQ:  Do all of the following:
1.	Force block-device interrupts onto some other CPU.
2.	Initiate any block I/O and block-I/O polling on other CPUs.
3.	Once your application has started, prevent CPU-hotplug operations
	from being initiated from tasks that might run on the CPU to
	be de-jittered.  (It is OK to force this CPU offline and then
	bring it back online before you start your application.)
TASKLET_SOFTIRQ: Do one or more of the following:
1.	Avoid use of drivers that use tasklets.  (Such drivers will contain
	calls to things like tasklet_schedule().)
2.	Convert all drivers that you must use from tasklets to workqueues.
3.	Force interrupts for drivers using tasklets onto other CPUs,
	and also do I/O involving these drivers on other CPUs.
SCHED_SOFTIRQ: Do all of the following:
1.	Avoid sending scheduler IPIs to the CPU to be de-jittered,
	for example, ensure that at most one runnable kthread is present
	on that CPU.  If a thread that expects to run on the de-jittered
	CPU awakens, the scheduler will send an IPI that can result in
	a subsequent SCHED_SOFTIRQ.
2.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
	CONFIG_NO_HZ_FULL=y, and, in addition, ensure that the CPU
	to be de-jittered is marked as an adaptive-ticks CPU using the
	"nohz_full=" boot parameter.  This reduces the number of
	scheduler-clock interrupts that the de-jittered CPU receives,
	minimizing its chances of being selected to do the load balancing
	work that runs in SCHED_SOFTIRQ context.
3.	To the extent possible, keep the CPU out of the kernel when it
	is non-idle, for example, by avoiding system calls and by
	forcing both kernel threads and interrupts to execute elsewhere.
	This further reduces the number of scheduler-clock interrupts
	received by the de-jittered CPU.
HRTIMER_SOFTIRQ:  Do all of the following:
1.	To the extent possible, keep the CPU out of the kernel when it
	is non-idle.  For example, avoid system calls and force both
	kernel threads and interrupts to execute elsewhere.
2.	Build with CONFIG_HOTPLUG_CPU=y.  Once boot completes, force the
	CPU offline, then bring it back online.  This forces recurring
	timers to migrate elsewhere.  If you are concerned with multiple
	CPUs, force them all offline before bringing the first one
	back online.  Once you have onlined the CPUs in question, do not
	offline any other CPUs, because doing so could force the timer
	back onto one of the CPUs in question.
RCU_SOFTIRQ:  Do at least one of the following:
1.	Offload callbacks and keep the CPU in either dyntick-idle or
	adaptive-ticks state by doing all of the following:
	a.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
		CONFIG_NO_HZ_FULL=y, and, in addition ensure that the CPU
		to be de-jittered is marked as an adaptive-ticks CPU using
		the "nohz_full=" boot parameter.  Bind the rcuo kthreads
		to housekeeping CPUs, which can tolerate OS jitter.
	b.	To the extent possible, keep the CPU out of the kernel
		when it is non-idle, for example, by avoiding system
		calls and by forcing both kernel threads and interrupts
		to execute elsewhere.
2.	Enable RCU to do its processing remotely via dyntick-idle by
	doing all of the following:
	a.	Build with CONFIG_NO_HZ=y and CONFIG_RCU_FAST_NO_HZ=y.
	b.	Ensure that the CPU goes idle frequently, allowing other
		CPUs to detect that it has passed through an RCU quiescent
		state.	If the kernel is built with CONFIG_NO_HZ_FULL=y,
		userspace execution also allows other CPUs to detect that
		the CPU in question has passed through a quiescent state.
	c.	To the extent possible, keep the CPU out of the kernel
		when it is non-idle, for example, by avoiding system
		calls and by forcing both kernel threads and interrupts
		to execute elsewhere.

Name: rcuc/%u
Purpose: Execute RCU callbacks in CONFIG_RCU_BOOST=y kernels.
To reduce its OS jitter, do at least one of the following:
1.	Build the kernel with CONFIG_PREEMPT=n.  This prevents these
	kthreads from being created in the first place, and also obviates
	the need for RCU priority boosting.  This approach is feasible
	for workloads that do not require high degrees of responsiveness.
2.	Build the kernel with CONFIG_RCU_BOOST=n.  This prevents these
	kthreads from being created in the first place.  This approach
	is feasible only if your workload never requires RCU priority
	boosting, for example, if you ensure frequent idle time on all
	CPUs that might execute within the kernel.
3.	Build with CONFIG_RCU_NOCB_CPU=y and CONFIG_RCU_NOCB_CPU_ALL=y,
	which offloads all RCU callbacks to kthreads that can be moved
	off of CPUs susceptible to OS jitter.  This approach prevents the
	rcuc/%u kthreads from having any work to do, so that they are
	never awakened.
4.	Ensure that the CPU never enters the kernel, and, in particular,
	avoid initiating any CPU hotplug operations on this CPU.  This is
	another way of preventing any callbacks from being queued on the
	CPU, again preventing the rcuc/%u kthreads from having any work
	to do.

Name: rcuob/%d, rcuop/%d, and rcuos/%d
Purpose: Offload RCU callbacks from the corresponding CPU.
To reduce its OS jitter, do at least one of the following:
1.	Use affinity, cgroups, or other mechanism to force these kthreads
	to execute on some other CPU.
2.	Build with CONFIG_RCU_NOCB_CPUS=n, which will prevent these
	kthreads from being created in the first place.  However, please
	note that this will not eliminate OS jitter, but will instead
	shift it to RCU_SOFTIRQ.

Name: watchdog/%u
Purpose: Detect software lockups on each CPU.
To reduce its OS jitter, do at least one of the following:
1.	Build with CONFIG_LOCKUP_DETECTOR=n, which will prevent these
	kthreads from being created in the first place.
2.	Echo a zero to /proc/sys/kernel/watchdog to disable the
	watchdog timer.
3.	Echo a large number of /proc/sys/kernel/watchdog_thresh in
	order to reduce the frequency of OS jitter due to the watchdog
	timer down to a level that is acceptable for your workload.

Re: [PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Borislav Petkov]

On Mon, Apr 22, 2013 at 09:03:29PM -0700, Paul E. McKenney wrote:
> > > +This document lists per-CPU kthreads in the Linux kernel and presents
> > > +options to control OS jitter due to these kthreads.  Note that kthreads
> > 
> > s/due to/which can be caused by/
> 
> Same meaning, but "due to" is probably a bit more arcane.

Yeah, "due to" kinda didn't read right in the context, arcane could be
one way to put it.

> But how about "and presents options to control these kthreads' OS
> jitter"?

Yep.

> > > +that are not per-CPU are not listed here -- to reduce OS jitter from
> > 
> > one too many "that"s:
> > 
> > s/that/which/
> 
> Fair point, but I can shorten it as follows:
> 
> 	Note that non-per-CPU kthreads CPU are not listed here --

that second "CPU" is kinda superfluous...?

> 	to reduce OS jitter from non-per-CPU kthreads, bind them to a
> 	"housekeeping" CPU that is dedicated to such work.

Yep, reads ok, except "that is" but you've removed it in the final
version below.

> > > +non-per-CPU kthreads, bind them to a "housekeeping" CPU that is dedicated
> > 
> > s/that/which/
> 
> Good catch -- I chose s/that is//.

Yep.

> > > +Name: ehca_comp/%u
> > > +Purpose: Periodically process Infiniband-related work.
> > > +To reduce corresponding OS jitter, do any of the following:
> > > +1.	Don't use EHCA Infiniband hardware.  This will prevent these
> > 
> > Sounds like this particular hardware is slow and its IRQ handler/softirq
> > needs a lot of time. Yes, no?
> > 
> > Can we have a reason why people shouldn't use that hw.
> 
> Because it has per-CPU kthreads that can cause OS jitter.  ;-)

Yeah, I stumbled over this specific brand of Infiniband hw. It looks
like this particular Infiniband driver uses per-CPU kthreads and the
others in drivers/infiniband/hw/ don't?

I hope this explains my head-scratching moment here...

> > This sentence keeps repeating; maybe explain the purpose of this doc in
> > the beginning once and drop this sentence in the later sections.
> 
> There are "any of" and "all of" qualifiers.  Also, I cannot count on
> someone reading the document beginning to end.  I would instead expect
> many of them to search for the name of the kthread that is bothering
> them and read only that part.

Ha! Very good point. :-)

> > > +2.	Build with CONFIG_HOTPLUG_CPU=y.  After boot completes, force
> > > +	the CPU offline, then bring it back online.  This forces
> > > +	recurring timers to migrate elsewhere.	If you are concerned
> > 
> > We don't migrate them back to that CPU when we online it again, do we?
> 
> Not unless the CPU it migrated to later is taken offline.  Good point,
> added words to that effect.

Yep, good.

> > > +	to be de-jittered is marked as an adaptive-ticks CPU using the
> > > +	"nohz_full=" boot parameter.  This reduces the number of
> > > +	scheduler-clock interrupts that the de-jittered CPU receives,
> > > +	minimizing its chances of being selected to do load balancing,
> > 
> > I don't think there's a "," if the "which... " part refers to the
> > previous "load balancing" and not to the whole sentence.
> 
> Good point -- I can reword to:
> 
> 	This reduces the number of scheduler-clock interrupts that the
> 	de-jittered CPU receives, minimizing its chances of being selected
> 	to do the load balancing work that runs in SCHED_SOFTIRQ context.

Yep.

> > > +	This further reduces the number of scheduler-clock interrupts
> > > +	that the de-jittered CPU receives.
> > 
> > s/that/which/ would suit better here IMHO.
> 
> Fair point, but how about this?
> 
> 	This further reduces the number of scheduler-clock interrupts
> 	received by the de-jittered CPU.

Even better.

> > > +	b.	To the extent possible, keep the CPU out of the kernel
> > > +		when it is non-idle, for example, by avoiding system
> > > +		calls and by forcing both kernel threads and interrupts
> > > +		to execute elsewhere.
> > > +2.	Enable RCU to do its processing remotely via dyntick-idle by
> > > +	doing all of the following:
> > > +	a.	Build with CONFIG_NO_HZ=y and CONFIG_RCU_FAST_NO_HZ=y.
> > > +	b.	Ensure that the CPU goes idle frequently, allowing other
> > 
> > I'm ensuring that by selecting the proper workload which has idle
> > breathers?
> 
> Yep!  Or, equivalently, by adding enough CPUs so that the workload
> has idle breathers.

Yeah, this sentence could be in the text, since we're explaining
everything! :-)

> Thank you for the thorough review and comments!  Please see below for
> an update.

Sure, thank you for writing this up for others to read.

Reviewed-by: Borislav Petkov <bp@suse.de>

> ------------------------------------------------------------------------
> 
> REDUCING OS JITTER DUE TO PER-CPU KTHREADS
> 
> This document lists per-CPU kthreads in the Linux kernel and presents
> options to control these kthreads' OS jitter.  Note that non-per-CPU

s /these kthreads'/their/

Sorry, I can't help it :) I promise I won't read too much in the rest so
as not to beat it to death again :-)

> kthreads CPU are not listed here.  To reduce OS jitter from non-per-CPU

s/CPU //

see above.

> kthreads, bind them to a "housekeeping" CPU dedicated to such work.

[ … ]

Ok, it looks good, ship it.

:-)

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Paul E. McKenney]

On Thu, Apr 25, 2013 at 12:23:12PM +0200, Borislav Petkov wrote:
> On Mon, Apr 22, 2013 at 09:03:29PM -0700, Paul E. McKenney wrote:
> > > > +This document lists per-CPU kthreads in the Linux kernel and presents
> > > > +options to control OS jitter due to these kthreads.  Note that kthreads
> > > 
> > > s/due to/which can be caused by/
> > 
> > Same meaning, but "due to" is probably a bit more arcane.
> 
> Yeah, "due to" kinda didn't read right in the context, arcane could be
> one way to put it.
> 
> > But how about "and presents options to control these kthreads' OS
> > jitter"?
> 
> Yep.
> 
> > > > +that are not per-CPU are not listed here -- to reduce OS jitter from
> > > 
> > > one too many "that"s:
> > > 
> > > s/that/which/
> > 
> > Fair point, but I can shorten it as follows:
> > 
> > 	Note that non-per-CPU kthreads CPU are not listed here --
> 
> that second "CPU" is kinda superfluous...?
> 
> > 	to reduce OS jitter from non-per-CPU kthreads, bind them to a
> > 	"housekeeping" CPU that is dedicated to such work.
> 
> Yep, reads ok, except "that is" but you've removed it in the final
> version below.
> 
> > > > +non-per-CPU kthreads, bind them to a "housekeeping" CPU that is dedicated
> > > 
> > > s/that/which/
> > 
> > Good catch -- I chose s/that is//.
> 
> Yep.
> 
> > > > +Name: ehca_comp/%u
> > > > +Purpose: Periodically process Infiniband-related work.
> > > > +To reduce corresponding OS jitter, do any of the following:
> > > > +1.	Don't use EHCA Infiniband hardware.  This will prevent these
> > > 
> > > Sounds like this particular hardware is slow and its IRQ handler/softirq
> > > needs a lot of time. Yes, no?
> > > 
> > > Can we have a reason why people shouldn't use that hw.
> > 
> > Because it has per-CPU kthreads that can cause OS jitter.  ;-)
> 
> Yeah, I stumbled over this specific brand of Infiniband hw. It looks
> like this particular Infiniband driver uses per-CPU kthreads and the
> others in drivers/infiniband/hw/ don't?
> 
> I hope this explains my head-scratching moment here...

Ah!  I rewrote the first sentence to read:

	Don't use eHCA Infiniband hardware, instead choosing hardware
	that does not require per-CPU kthreads.

> > > This sentence keeps repeating; maybe explain the purpose of this doc in
> > > the beginning once and drop this sentence in the later sections.
> > 
> > There are "any of" and "all of" qualifiers.  Also, I cannot count on
> > someone reading the document beginning to end.  I would instead expect
> > many of them to search for the name of the kthread that is bothering
> > them and read only that part.
> 
> Ha! Very good point. :-)
> 
> > > > +2.	Build with CONFIG_HOTPLUG_CPU=y.  After boot completes, force
> > > > +	the CPU offline, then bring it back online.  This forces
> > > > +	recurring timers to migrate elsewhere.	If you are concerned
> > > 
> > > We don't migrate them back to that CPU when we online it again, do we?
> > 
> > Not unless the CPU it migrated to later is taken offline.  Good point,
> > added words to that effect.
> 
> Yep, good.
> 
> > > > +	to be de-jittered is marked as an adaptive-ticks CPU using the
> > > > +	"nohz_full=" boot parameter.  This reduces the number of
> > > > +	scheduler-clock interrupts that the de-jittered CPU receives,
> > > > +	minimizing its chances of being selected to do load balancing,
> > > 
> > > I don't think there's a "," if the "which... " part refers to the
> > > previous "load balancing" and not to the whole sentence.
> > 
> > Good point -- I can reword to:
> > 
> > 	This reduces the number of scheduler-clock interrupts that the
> > 	de-jittered CPU receives, minimizing its chances of being selected
> > 	to do the load balancing work that runs in SCHED_SOFTIRQ context.
> 
> Yep.
> 
> > > > +	This further reduces the number of scheduler-clock interrupts
> > > > +	that the de-jittered CPU receives.
> > > 
> > > s/that/which/ would suit better here IMHO.
> > 
> > Fair point, but how about this?
> > 
> > 	This further reduces the number of scheduler-clock interrupts
> > 	received by the de-jittered CPU.
> 
> Even better.
> 
> > > > +	b.	To the extent possible, keep the CPU out of the kernel
> > > > +		when it is non-idle, for example, by avoiding system
> > > > +		calls and by forcing both kernel threads and interrupts
> > > > +		to execute elsewhere.
> > > > +2.	Enable RCU to do its processing remotely via dyntick-idle by
> > > > +	doing all of the following:
> > > > +	a.	Build with CONFIG_NO_HZ=y and CONFIG_RCU_FAST_NO_HZ=y.
> > > > +	b.	Ensure that the CPU goes idle frequently, allowing other
> > > 
> > > I'm ensuring that by selecting the proper workload which has idle
> > > breathers?
> > 
> > Yep!  Or, equivalently, by adding enough CPUs so that the workload
> > has idle breathers.
> 
> Yeah, this sentence could be in the text, since we're explaining
> everything! :-)
> 
> > Thank you for the thorough review and comments!  Please see below for
> > an update.
> 
> Sure, thank you for writing this up for others to read.
> 
> Reviewed-by: Borislav Petkov <bp@suse.de>

Thank you, added!

> > ------------------------------------------------------------------------
> > 
> > REDUCING OS JITTER DUE TO PER-CPU KTHREADS
> > 
> > This document lists per-CPU kthreads in the Linux kernel and presents
> > options to control these kthreads' OS jitter.  Note that non-per-CPU
> 
> s /these kthreads'/their/
> 
> Sorry, I can't help it :) I promise I won't read too much in the rest so
> as not to beat it to death again :-)

Good change, though, applied.

> > kthreads CPU are not listed here.  To reduce OS jitter from non-per-CPU
> 
> s/CPU //
> 
> see above.

Good point, fixed

> > kthreads, bind them to a "housekeeping" CPU dedicated to such work.
> 
> [ … ]
> 
> Ok, it looks good, ship it.
> 
> :-)

Will do!  ;-)

							Thanx, Paul

Re: [PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Thomas Gleixner]

On Thu, 25 Apr 2013, Paul E. McKenney wrote:
> On Thu, Apr 25, 2013 at 12:23:12PM +0200, Borislav Petkov wrote:
> > On Mon, Apr 22, 2013 at 09:03:29PM -0700, Paul E. McKenney wrote:
> > > > > +Name: ehca_comp/%u
> > > > > +Purpose: Periodically process Infiniband-related work.
> > > > > +To reduce corresponding OS jitter, do any of the following:
> > > > > +1.	Don't use EHCA Infiniband hardware.  This will prevent these
> > > > 
> > > > Sounds like this particular hardware is slow and its IRQ handler/softirq
> > > > needs a lot of time. Yes, no?
> > > > 
> > > > Can we have a reason why people shouldn't use that hw.
> > > 
> > > Because it has per-CPU kthreads that can cause OS jitter.  ;-)
> > 
> > Yeah, I stumbled over this specific brand of Infiniband hw. It looks
> > like this particular Infiniband driver uses per-CPU kthreads and the
> > others in drivers/infiniband/hw/ don't?
> > 
> > I hope this explains my head-scratching moment here...
> 
> Ah!  I rewrote the first sentence to read:
> 
> 	Don't use eHCA Infiniband hardware, instead choosing hardware
> 	that does not require per-CPU kthreads.

Another option would be to teach that eHCA driver to be configurable
on which cpus kthreads are desired and on which not. I can't see a
reason (aside of throughput) why that hardware can't cope with a
single thread.

Thanks,

	tglx

Re: [PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Paul E. McKenney]

On Thu, Apr 25, 2013 at 10:59:05PM +0200, Thomas Gleixner wrote:
> On Thu, 25 Apr 2013, Paul E. McKenney wrote:
> > On Thu, Apr 25, 2013 at 12:23:12PM +0200, Borislav Petkov wrote:
> > > On Mon, Apr 22, 2013 at 09:03:29PM -0700, Paul E. McKenney wrote:
> > > > > > +Name: ehca_comp/%u
> > > > > > +Purpose: Periodically process Infiniband-related work.
> > > > > > +To reduce corresponding OS jitter, do any of the following:
> > > > > > +1.	Don't use EHCA Infiniband hardware.  This will prevent these
> > > > > 
> > > > > Sounds like this particular hardware is slow and its IRQ handler/softirq
> > > > > needs a lot of time. Yes, no?
> > > > > 
> > > > > Can we have a reason why people shouldn't use that hw.
> > > > 
> > > > Because it has per-CPU kthreads that can cause OS jitter.  ;-)
> > > 
> > > Yeah, I stumbled over this specific brand of Infiniband hw. It looks
> > > like this particular Infiniband driver uses per-CPU kthreads and the
> > > others in drivers/infiniband/hw/ don't?
> > > 
> > > I hope this explains my head-scratching moment here...
> > 
> > Ah!  I rewrote the first sentence to read:
> > 
> > 	Don't use eHCA Infiniband hardware, instead choosing hardware
> > 	that does not require per-CPU kthreads.
> 
> Another option would be to teach that eHCA driver to be configurable
> on which cpus kthreads are desired and on which not. I can't see a
> reason (aside of throughput) why that hardware can't cope with a
> single thread.

Good point!  I have added a third item to the eHCA list:

	Rework the eHCA driver so that its per-CPU kthreads are
	provisioned only on selected CPUs.

							Thanx, Paul

Re: [PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Paul E. McKenney]

On Thu, Apr 11, 2013 at 01:09:28PM -0700, Randy Dunlap wrote:
> On 04/11/13 11:40, Paul E. McKenney wrote:
> > On Thu, Apr 11, 2013 at 10:18:26AM -0700, Randy Dunlap wrote:
> >> On 04/11/2013 09:05 AM, Paul E. McKenney wrote:
> >>> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> >>>
> >>> The Linux kernel uses a number of per-CPU kthreads, any of which might
> >>> contribute to OS jitter at any time.  The usual approach to normal
> >>> kthreads, namely to affinity them to a "housekeeping" CPU, does not
> >>
> >> ugh.               to affine them
> > 
> > How about s/affinity/bind/ instead?
> 
> Yes, that's good.
> 
> >>> work with these kthreads because they cannot operate correctly if moved
> >>> to some other CPU.  This commit therefore lists ways of controlling OS
> >>> jitter from the Linux kernel's per-CPU kthreads.
> >>>
> >>> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> >>> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> >>> Cc: Steven Rostedt <rostedt@goodmis.org>
> >>> Cc: Borislav Petkov <bp@alien8.de>
> >>> Cc: Arjan van de Ven <arjan@linux.intel.com>
> >>> Cc: Kevin Hilman <khilman@linaro.org>
> >>> Cc: Christoph Lameter <cl@linux.com>
> >>> ---
> >>>  Documentation/kernel-per-CPU-kthreads.txt | 159 ++++++++++++++++++++++++++++++
> >>>  1 file changed, 159 insertions(+)
> >>>  create mode 100644 Documentation/kernel-per-CPU-kthreads.txt
> >>>
> >>> diff --git a/Documentation/kernel-per-CPU-kthreads.txt b/Documentation/kernel-per-CPU-kthreads.txt
> >>> new file mode 100644
> >>> index 0000000..495dacf
> >>> --- /dev/null
> >>> +++ b/Documentation/kernel-per-CPU-kthreads.txt
> >>> @@ -0,0 +1,159 @@
> >>> +REDUCING OS JITTER DUE TO PER-CPU KTHREADS
> >>> +
> >>> +
> >>> +Name: irq/%d-%s
> >>> +Purpose: Handle threaded interrupts.
> >>> +To reduce corresponding OS jitter, do the following:
> >>> +1.	Use irq affinity to force the irq threads to execute on
> >>> +	some other CPU.
> >>
> >> It would be very nice to explain here how that is done.
> > 
> > Documentation/IRQ-affinity.txt
> > 
> > I added a pointer to this near the beginning.
> > 
> 
> Good.
> 
> > Thank you for your review and comments!  Given my rationale above,
> > are you still comfortable with my applying your Reviewed-by?
> 
> Sure.  Thanks.
> 
> >> Reviewed-by: Randy Dunlap <rdunlap@infradead.org>

I have added your Reviewed-by, thank you again!

							Thanx, Paul

Re: [PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Randy Dunlap]

On 04/11/13 11:40, Paul E. McKenney wrote:
> On Thu, Apr 11, 2013 at 10:18:26AM -0700, Randy Dunlap wrote:
>> On 04/11/2013 09:05 AM, Paul E. McKenney wrote:
>>> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
>>>
>>> The Linux kernel uses a number of per-CPU kthreads, any of which might
>>> contribute to OS jitter at any time.  The usual approach to normal
>>> kthreads, namely to affinity them to a "housekeeping" CPU, does not
>>
>> ugh.               to affine them
> 
> How about s/affinity/bind/ instead?

Yes, that's good.

>>> work with these kthreads because they cannot operate correctly if moved
>>> to some other CPU.  This commit therefore lists ways of controlling OS
>>> jitter from the Linux kernel's per-CPU kthreads.
>>>
>>> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
>>> Cc: Frederic Weisbecker <fweisbec@gmail.com>
>>> Cc: Steven Rostedt <rostedt@goodmis.org>
>>> Cc: Borislav Petkov <bp@alien8.de>
>>> Cc: Arjan van de Ven <arjan@linux.intel.com>
>>> Cc: Kevin Hilman <khilman@linaro.org>
>>> Cc: Christoph Lameter <cl@linux.com>
>>> ---
>>>  Documentation/kernel-per-CPU-kthreads.txt | 159 ++++++++++++++++++++++++++++++
>>>  1 file changed, 159 insertions(+)
>>>  create mode 100644 Documentation/kernel-per-CPU-kthreads.txt
>>>
>>> diff --git a/Documentation/kernel-per-CPU-kthreads.txt b/Documentation/kernel-per-CPU-kthreads.txt
>>> new file mode 100644
>>> index 0000000..495dacf
>>> --- /dev/null
>>> +++ b/Documentation/kernel-per-CPU-kthreads.txt
>>> @@ -0,0 +1,159 @@
>>> +REDUCING OS JITTER DUE TO PER-CPU KTHREADS
>>> +
>>> +
>>> +Name: irq/%d-%s
>>> +Purpose: Handle threaded interrupts.
>>> +To reduce corresponding OS jitter, do the following:
>>> +1.	Use irq affinity to force the irq threads to execute on
>>> +	some other CPU.
>>
>> It would be very nice to explain here how that is done.
> 
> Documentation/IRQ-affinity.txt
> 
> I added a pointer to this near the beginning.
> 

Good.

> Thank you for your review and comments!  Given my rationale above,
> are you still comfortable with my applying your Reviewed-by?

Sure.  Thanks.

>> Reviewed-by: Randy Dunlap <rdunlap@infradead.org>


-- 
~Randy

Re: [PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Paul E. McKenney]

On Thu, Apr 11, 2013 at 10:18:26AM -0700, Randy Dunlap wrote:
> On 04/11/2013 09:05 AM, Paul E. McKenney wrote:
> >From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
> >
> >The Linux kernel uses a number of per-CPU kthreads, any of which might
> >contribute to OS jitter at any time.  The usual approach to normal
> >kthreads, namely to affinity them to a "housekeeping" CPU, does not
> 
> ugh.               to affine them

How about s/affinity/bind/ instead?

> >work with these kthreads because they cannot operate correctly if moved
> >to some other CPU.  This commit therefore lists ways of controlling OS
> >jitter from the Linux kernel's per-CPU kthreads.
> >
> >Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> >Cc: Frederic Weisbecker <fweisbec@gmail.com>
> >Cc: Steven Rostedt <rostedt@goodmis.org>
> >Cc: Borislav Petkov <bp@alien8.de>
> >Cc: Arjan van de Ven <arjan@linux.intel.com>
> >Cc: Kevin Hilman <khilman@linaro.org>
> >Cc: Christoph Lameter <cl@linux.com>
> >---
> >  Documentation/kernel-per-CPU-kthreads.txt | 159 ++++++++++++++++++++++++++++++
> >  1 file changed, 159 insertions(+)
> >  create mode 100644 Documentation/kernel-per-CPU-kthreads.txt
> >
> >diff --git a/Documentation/kernel-per-CPU-kthreads.txt b/Documentation/kernel-per-CPU-kthreads.txt
> >new file mode 100644
> >index 0000000..495dacf
> >--- /dev/null
> >+++ b/Documentation/kernel-per-CPU-kthreads.txt
> >@@ -0,0 +1,159 @@
> >+REDUCING OS JITTER DUE TO PER-CPU KTHREADS
> >+
> >+This document lists per-CPU kthreads in the Linux kernel and presents
> >+options to control OS jitter due to these kthreads.  Note that kthreads
> >+that are not per-CPU are not listed here -- to reduce OS jitter from
> >+non-per-CPU kthreads, bind them to a "housekeeping" CPU that is dedicated
> >+to such work.
> >+
> >+
> >+Name: ehca_comp/%u
> >+Purpose: Periodically process Infiniband-related work.
> >+To reduce corresponding OS jitter, do any of the following:
> >+1.	Don't use EHCA Infiniband hardware.  This will prevent these
> >+	kthreads from being created in the first place.  (This will
> >+	work for most people, as this hardware, though important,
> >+	is relatively old as is produced in relatively low unit
> >+	volumes.)
> >+2.	Do all EHCA-Infiniband-related work on other CPUs, including
> >+	interrupts.
> >+
> >+
> >+Name: irq/%d-%s
> >+Purpose: Handle threaded interrupts.
> >+To reduce corresponding OS jitter, do the following:
> >+1.	Use irq affinity to force the irq threads to execute on
> >+	some other CPU.
> 
> It would be very nice to explain here how that is done.

Documentation/IRQ-affinity.txt

I added a pointer to this near the beginning.

> >+
> >+Name: kcmtpd_ctr_%d
> >+Purpose: Handle Bluetooth work.
> >+To reduce corresponding OS jitter, do one of the following:
> >+1.	Don't use Bluetooth, in cwhich case these kthreads won't be
> 
> 	                        which

Good catch, fixed.

> >+	created in the first place.
> >+2.	Use irq affinity to force Bluetooth-related interrupts to
> >+	occur on some other CPU and furthermore initiate all
> >+	Bluetooth activity from some other CPU.
> >+
> >+Name: ksoftirqd/%u
> >+Purpose: Execute softirq handlers when threaded or when under heavy load.
> >+To reduce corresponding OS jitter, each softirq vector must be handled
> >+separately as follows:
> >+TIMER_SOFTIRQ:
> >+1.	Build with CONFIG_HOTPLUG_CPU=y.
> >+2.	To the extent possible, keep the CPU out of the kernel when it
> 
> I guess I have a different viewpoint.  I would say:  keep the kernel
> off of that CPU ....

The rationale for the viewpoint that I chose is that many workloads that
care about OS jitter run CPU-bound userspace threads.  The more that
these threads avoid system calls, the less opportunity for OS jitter to
slip in.  So in this case, the application writer really is keeping the
CPU out of the kernel.

> >+	is non-idle, for example, by forcing user and kernel threads as
> >+	well as interrupts to execute elsewhere.
> >+3.	Force the CPU offline, then bring it back online.  This forces
> >+	recurring timers to migrate elsewhere.  If you are concerned
> >+	with multiple CPUs, force them all offline before bringing the
> >+	first one back online.
> >+NET_TX_SOFTIRQ and NET_RX_SOFTIRQ:  Do all of the following:
> >+1.	Force networking interrupts onto other CPUs.
> >+2.	Initiate any network I/O on other CPUs.
> >+3.	Prevent CPU-hotplug operations from being initiated from tasks
> >+	that might run on the CPU to be de-jittered.
> >+BLOCK_SOFTIRQ:  Do all of the following:
> >+1.	Force block-device interrupts onto some other CPU.
> >+2.	Initiate any block I/O on other CPUs.
> >+3.	Prevent CPU-hotplug operations from being initiated from tasks
> >+	that might run on the CPU to be de-jittered.
> >+BLOCK_IOPOLL_SOFTIRQ:  Do all of the following:
> >+1.	Force block-device interrupts onto some other CPU.
> >+2.	Initiate any block I/O and block-I/O polling on other CPUs.
> >+3.	Prevent CPU-hotplug operations from being initiated from tasks
> >+	that might run on the CPU to be de-jittered.
> >+TASKLET_SOFTIRQ: Do one or more of the following:
> >+1.	Avoid use of drivers that use tasklets.
> >+2.	Convert all drivers that you must use from tasklets to workqueues.
> >+3.	Force interrupts for drivers using tasklets onto other CPUs,
> >+	and also do I/O involving these drivers on other CPUs.
> >+SCHED_SOFTIRQ: Do all of the following:
> >+1.	Avoid sending scheduler IPIs to the CPU to be de-jittered,
> >+	for example, ensure that at most one runnable kthread is
> >+	present on that CPU.  If a thread awakens that expects
> >+	to run on the de-jittered CPU, the scheduler will send
> >+	an IPI that can result in a subsequent SCHED_SOFTIRQ.
> >+2.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
> >+	CONFIG_NO_HZ_EXTENDED=y, and in addition ensure that the CPU
> >+	to be de-jittered is marked as an adaptive-ticks CPU using the
> >+	"nohz_extended=" boot parameter.  This reduces the number of
> >+	scheduler-clock interrupts that the de-jittered CPU receives,
> >+	minimizing its chances of being selected to do load balancing,
> >+	which happens in SCHED_SOFTIRQ context.
> >+3.	To the extent possible, keep the CPU out of the kernel when it
> 
> same viewpoint point.

Same rationale.  ;-)

> >+	is non-idle, for example, by forcing user and kernel threads as
> >+	well as interrupts to execute elsewhere.  This further reduces
> >+	the number of scheduler-clock interrupts that the de-jittered
> >+	CPU receives.
> >+HRTIMER_SOFTIRQ:  Do all of the following:
> >+1.	Build with CONFIG_HOTPLUG_CPU=y.
> >+2.	To the extent possible, keep the CPU out of the kernel when it
> >+	is non-idle, for example, by forcing user and kernel threads as
> >+	well as interrupts to execute elsewhere.
> >+3.	Force the CPU offline, then bring it back online.  This forces
> >+	recurring timers to migrate elsewhere.  If you are concerned
> >+	with multiple CPUs, force them all offline before bringing the
> >+	first one back online.
> >+RCU_SOFTIRQ:  Do at least one of the following:
> >+1.	Offload callbacks and keep the CPU in either dyntick-idle or
> >+	adaptive-ticks state by doing all of the following:
> >+	a.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
> >+		CONFIG_NO_HZ_EXTENDED=y, and in addition ensure that
> >+		the CPU to be de-jittered is marked as an adaptive-ticks CPU
> >+		using the "nohz_extended=" boot parameter.
> >+	b.	To the extent possible, keep the CPU out of the kernel
> 
> viewpoint?

Ditto.

> >+		when it is non-idle, for example, by forcing user and
> >+		kernel threads as well as interrupts to execute elsewhere.
> >+2.	Enable RCU to do its processing remotely via dyntick-idle by
> >+	doing all of the following:
> >+	a.	Build with CONFIG_NO_HZ=y and CONFIG_RCU_FAST_NO_HZ=y.
> >+	b.	To the extent possible, keep the CPU out of the kernel
> 
> viewpoint?

Ditto.

> >+		when it is non-idle, for example, by forcing user and
> >+		kernel threads as well as interrupts to execute elsewhere.
> >+	c.	Ensure that the CPU goes idle frequently, allowing other
> >+		CPUs to detect that it has passed through an RCU
> >+		quiescent state.
> >+
> >+Name: rcuc/%u
> >+Purpose: Execute RCU callbacks in CONFIG_RCU_BOOST=y kernels.
> >+To reduce corresponding OS jitter, do at least one of the following:
> >+1.	Build the kernel with CONFIG_PREEMPT=n.  This prevents these
> >+	kthreads from being created in the first place, and also prevents
> >+	RCU priority boosting from ever being required.  This approach
> >+	is feasible for workloads that do not require high degrees of
> >+	responsiveness.
> >+2.	Build the kernel with CONFIG_RCU_BOOST=n.  This prevents these
> >+	kthreads from being created in the first place.  This approach
> >+	is feasible only if your workload never requires RCU priority
> >+	boosting, for example, if you ensure ample idle time on all CPUs
> >+	that might execute within the kernel.
> >+3.	Build with CONFIG_RCU_NOCB_CPU=y and CONFIG_RCU_NOCB_CPU_ALL=y,
> >+	which offloads all RCU callbacks to kthreads that can be moved
> >+	off of CPUs susceptible to OS jitter.  This approach prevents the
> >+	rcuc/%u kthreads from having any work to do, and are therefore
> >+	never awakened.
> >+4.	Ensure that then CPU never enters the kernel and avoid any
> 
> 	            the

Good catch, fixed.

> viewpoint?

Rationale.

> >+	CPU hotplug operations.  This is another way of preventing any
> >+	callbacks from being queued on the CPU, again preventing the
> >+	rcuc/%u kthreads from having any work to do.
> >+
> >+Name: rcuob/%d, rcuop/%d, and rcuos/%d
> >+Purpose: Offload RCU callbacks from the corresponding CPU.
> >+To reduce corresponding OS jitter, do at least one of the following:
> >+1.	Use affinity, cgroups, or other mechanism to force these kthreads
> >+	to execute on some other CPU.
> >+2.	Build with CONFIG_RCU_NOCB_CPUS=n, which will prevent these
> >+	kthreads from being created in the first place.  However,
> >+	please note that this will not eliminate the corresponding
> >+	OS jitter, but will instead merely shift it to softirq.
> >+
> >+Name: watchdog/%u
> >+Purpose: Detect software lockups on each CPU.
> >+To reduce corresponding OS jitter, do at least one of the following:
> >+1.	Build with CONFIG_LOCKUP_DETECTOR=n, which will prevent these
> >+	kthreads from being created in the first place.
> >+2.	Echo a zero to /proc/sys/kernel/watchdog to disable the
> >+	watchdog timer.
> >+3.	Echo a large number of /proc/sys/kernel/watchdog_thresh in
> >+	order to reduce the frequency of OS jitter due to the watchdog
> >+	timer down to a level that is acceptable for your workload.

Thank you for your review and comments!  Given my rationale above,
are you still comfortable with my applying your Reviewed-by?

							Thanx, Paul

> Reviewed-by: Randy Dunlap <rdunlap@infradead.org>
> 
> 
> -- 
> ~Randy
> 

Re: [PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Randy Dunlap]

On 04/11/2013 09:05 AM, Paul E. McKenney wrote:
> From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
>
> The Linux kernel uses a number of per-CPU kthreads, any of which might
> contribute to OS jitter at any time.  The usual approach to normal
> kthreads, namely to affinity them to a "housekeeping" CPU, does not

ugh.               to affine them

> work with these kthreads because they cannot operate correctly if moved
> to some other CPU.  This commit therefore lists ways of controlling OS
> jitter from the Linux kernel's per-CPU kthreads.
>
> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> Cc: Frederic Weisbecker <fweisbec@gmail.com>
> Cc: Steven Rostedt <rostedt@goodmis.org>
> Cc: Borislav Petkov <bp@alien8.de>
> Cc: Arjan van de Ven <arjan@linux.intel.com>
> Cc: Kevin Hilman <khilman@linaro.org>
> Cc: Christoph Lameter <cl@linux.com>
> ---
>   Documentation/kernel-per-CPU-kthreads.txt | 159 ++++++++++++++++++++++++++++++
>   1 file changed, 159 insertions(+)
>   create mode 100644 Documentation/kernel-per-CPU-kthreads.txt
>
> diff --git a/Documentation/kernel-per-CPU-kthreads.txt b/Documentation/kernel-per-CPU-kthreads.txt
> new file mode 100644
> index 0000000..495dacf
> --- /dev/null
> +++ b/Documentation/kernel-per-CPU-kthreads.txt
> @@ -0,0 +1,159 @@
> +REDUCING OS JITTER DUE TO PER-CPU KTHREADS
> +
> +This document lists per-CPU kthreads in the Linux kernel and presents
> +options to control OS jitter due to these kthreads.  Note that kthreads
> +that are not per-CPU are not listed here -- to reduce OS jitter from
> +non-per-CPU kthreads, bind them to a "housekeeping" CPU that is dedicated
> +to such work.
> +
> +
> +Name: ehca_comp/%u
> +Purpose: Periodically process Infiniband-related work.
> +To reduce corresponding OS jitter, do any of the following:
> +1.	Don't use EHCA Infiniband hardware.  This will prevent these
> +	kthreads from being created in the first place.  (This will
> +	work for most people, as this hardware, though important,
> +	is relatively old as is produced in relatively low unit
> +	volumes.)
> +2.	Do all EHCA-Infiniband-related work on other CPUs, including
> +	interrupts.
> +
> +
> +Name: irq/%d-%s
> +Purpose: Handle threaded interrupts.
> +To reduce corresponding OS jitter, do the following:
> +1.	Use irq affinity to force the irq threads to execute on
> +	some other CPU.

It would be very nice to explain here how that is done.

> +
> +Name: kcmtpd_ctr_%d
> +Purpose: Handle Bluetooth work.
> +To reduce corresponding OS jitter, do one of the following:
> +1.	Don't use Bluetooth, in cwhich case these kthreads won't be

	                        which

> +	created in the first place.
> +2.	Use irq affinity to force Bluetooth-related interrupts to
> +	occur on some other CPU and furthermore initiate all
> +	Bluetooth activity from some other CPU.
> +
> +Name: ksoftirqd/%u
> +Purpose: Execute softirq handlers when threaded or when under heavy load.
> +To reduce corresponding OS jitter, each softirq vector must be handled
> +separately as follows:
> +TIMER_SOFTIRQ:
> +1.	Build with CONFIG_HOTPLUG_CPU=y.
> +2.	To the extent possible, keep the CPU out of the kernel when it

I guess I have a different viewpoint.  I would say:  keep the kernel
off of that CPU ....

> +	is non-idle, for example, by forcing user and kernel threads as
> +	well as interrupts to execute elsewhere.
> +3.	Force the CPU offline, then bring it back online.  This forces
> +	recurring timers to migrate elsewhere.  If you are concerned
> +	with multiple CPUs, force them all offline before bringing the
> +	first one back online.
> +NET_TX_SOFTIRQ and NET_RX_SOFTIRQ:  Do all of the following:
> +1.	Force networking interrupts onto other CPUs.
> +2.	Initiate any network I/O on other CPUs.
> +3.	Prevent CPU-hotplug operations from being initiated from tasks
> +	that might run on the CPU to be de-jittered.
> +BLOCK_SOFTIRQ:  Do all of the following:
> +1.	Force block-device interrupts onto some other CPU.
> +2.	Initiate any block I/O on other CPUs.
> +3.	Prevent CPU-hotplug operations from being initiated from tasks
> +	that might run on the CPU to be de-jittered.
> +BLOCK_IOPOLL_SOFTIRQ:  Do all of the following:
> +1.	Force block-device interrupts onto some other CPU.
> +2.	Initiate any block I/O and block-I/O polling on other CPUs.
> +3.	Prevent CPU-hotplug operations from being initiated from tasks
> +	that might run on the CPU to be de-jittered.
> +TASKLET_SOFTIRQ: Do one or more of the following:
> +1.	Avoid use of drivers that use tasklets.
> +2.	Convert all drivers that you must use from tasklets to workqueues.
> +3.	Force interrupts for drivers using tasklets onto other CPUs,
> +	and also do I/O involving these drivers on other CPUs.
> +SCHED_SOFTIRQ: Do all of the following:
> +1.	Avoid sending scheduler IPIs to the CPU to be de-jittered,
> +	for example, ensure that at most one runnable kthread is
> +	present on that CPU.  If a thread awakens that expects
> +	to run on the de-jittered CPU, the scheduler will send
> +	an IPI that can result in a subsequent SCHED_SOFTIRQ.
> +2.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
> +	CONFIG_NO_HZ_EXTENDED=y, and in addition ensure that the CPU
> +	to be de-jittered is marked as an adaptive-ticks CPU using the
> +	"nohz_extended=" boot parameter.  This reduces the number of
> +	scheduler-clock interrupts that the de-jittered CPU receives,
> +	minimizing its chances of being selected to do load balancing,
> +	which happens in SCHED_SOFTIRQ context.
> +3.	To the extent possible, keep the CPU out of the kernel when it

same viewpoint point.

> +	is non-idle, for example, by forcing user and kernel threads as
> +	well as interrupts to execute elsewhere.  This further reduces
> +	the number of scheduler-clock interrupts that the de-jittered
> +	CPU receives.
> +HRTIMER_SOFTIRQ:  Do all of the following:
> +1.	Build with CONFIG_HOTPLUG_CPU=y.
> +2.	To the extent possible, keep the CPU out of the kernel when it
> +	is non-idle, for example, by forcing user and kernel threads as
> +	well as interrupts to execute elsewhere.
> +3.	Force the CPU offline, then bring it back online.  This forces
> +	recurring timers to migrate elsewhere.  If you are concerned
> +	with multiple CPUs, force them all offline before bringing the
> +	first one back online.
> +RCU_SOFTIRQ:  Do at least one of the following:
> +1.	Offload callbacks and keep the CPU in either dyntick-idle or
> +	adaptive-ticks state by doing all of the following:
> +	a.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
> +		CONFIG_NO_HZ_EXTENDED=y, and in addition ensure that
> +		the CPU to be de-jittered is marked as an adaptive-ticks CPU
> +		using the "nohz_extended=" boot parameter.
> +	b.	To the extent possible, keep the CPU out of the kernel

viewpoint?

> +		when it is non-idle, for example, by forcing user and
> +		kernel threads as well as interrupts to execute elsewhere.
> +2.	Enable RCU to do its processing remotely via dyntick-idle by
> +	doing all of the following:
> +	a.	Build with CONFIG_NO_HZ=y and CONFIG_RCU_FAST_NO_HZ=y.
> +	b.	To the extent possible, keep the CPU out of the kernel

viewpoint?

> +		when it is non-idle, for example, by forcing user and
> +		kernel threads as well as interrupts to execute elsewhere.
> +	c.	Ensure that the CPU goes idle frequently, allowing other
> +		CPUs to detect that it has passed through an RCU
> +		quiescent state.
> +
> +Name: rcuc/%u
> +Purpose: Execute RCU callbacks in CONFIG_RCU_BOOST=y kernels.
> +To reduce corresponding OS jitter, do at least one of the following:
> +1.	Build the kernel with CONFIG_PREEMPT=n.  This prevents these
> +	kthreads from being created in the first place, and also prevents
> +	RCU priority boosting from ever being required.  This approach
> +	is feasible for workloads that do not require high degrees of
> +	responsiveness.
> +2.	Build the kernel with CONFIG_RCU_BOOST=n.  This prevents these
> +	kthreads from being created in the first place.  This approach
> +	is feasible only if your workload never requires RCU priority
> +	boosting, for example, if you ensure ample idle time on all CPUs
> +	that might execute within the kernel.
> +3.	Build with CONFIG_RCU_NOCB_CPU=y and CONFIG_RCU_NOCB_CPU_ALL=y,
> +	which offloads all RCU callbacks to kthreads that can be moved
> +	off of CPUs susceptible to OS jitter.  This approach prevents the
> +	rcuc/%u kthreads from having any work to do, and are therefore
> +	never awakened.
> +4.	Ensure that then CPU never enters the kernel and avoid any

	            the
viewpoint?

> +	CPU hotplug operations.  This is another way of preventing any
> +	callbacks from being queued on the CPU, again preventing the
> +	rcuc/%u kthreads from having any work to do.
> +
> +Name: rcuob/%d, rcuop/%d, and rcuos/%d
> +Purpose: Offload RCU callbacks from the corresponding CPU.
> +To reduce corresponding OS jitter, do at least one of the following:
> +1.	Use affinity, cgroups, or other mechanism to force these kthreads
> +	to execute on some other CPU.
> +2.	Build with CONFIG_RCU_NOCB_CPUS=n, which will prevent these
> +	kthreads from being created in the first place.  However,
> +	please note that this will not eliminate the corresponding
> +	OS jitter, but will instead merely shift it to softirq.
> +
> +Name: watchdog/%u
> +Purpose: Detect software lockups on each CPU.
> +To reduce corresponding OS jitter, do at least one of the following:
> +1.	Build with CONFIG_LOCKUP_DETECTOR=n, which will prevent these
> +	kthreads from being created in the first place.
> +2.	Echo a zero to /proc/sys/kernel/watchdog to disable the
> +	watchdog timer.
> +3.	Echo a large number of /proc/sys/kernel/watchdog_thresh in
> +	order to reduce the frequency of OS jitter due to the watchdog
> +	timer down to a level that is acceptable for your workload.
>


Reviewed-by: Randy Dunlap <rdunlap@infradead.org>


-- 
~Randy

[PATCH documentation 2/2] kthread: Document ways of reducing OS jitter due to per-CPU kthreads

[Paul E. McKenney]

From: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>

The Linux kernel uses a number of per-CPU kthreads, any of which might
contribute to OS jitter at any time.  The usual approach to normal
kthreads, namely to affinity them to a "housekeeping" CPU, does not
work with these kthreads because they cannot operate correctly if moved
to some other CPU.  This commit therefore lists ways of controlling OS
jitter from the Linux kernel's per-CPU kthreads.

Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Borislav Petkov <bp@alien8.de>
Cc: Arjan van de Ven <arjan@linux.intel.com>
Cc: Kevin Hilman <khilman@linaro.org>
Cc: Christoph Lameter <cl@linux.com>
---
 Documentation/kernel-per-CPU-kthreads.txt | 159 ++++++++++++++++++++++++++++++
 1 file changed, 159 insertions(+)
 create mode 100644 Documentation/kernel-per-CPU-kthreads.txt

diff --git a/Documentation/kernel-per-CPU-kthreads.txt b/Documentation/kernel-per-CPU-kthreads.txt
new file mode 100644
index 0000000..495dacf
--- /dev/null
+++ b/Documentation/kernel-per-CPU-kthreads.txt
@@ -0,0 +1,159 @@
+REDUCING OS JITTER DUE TO PER-CPU KTHREADS
+
+This document lists per-CPU kthreads in the Linux kernel and presents
+options to control OS jitter due to these kthreads.  Note that kthreads
+that are not per-CPU are not listed here -- to reduce OS jitter from
+non-per-CPU kthreads, bind them to a "housekeeping" CPU that is dedicated
+to such work.
+
+
+Name: ehca_comp/%u
+Purpose: Periodically process Infiniband-related work.
+To reduce corresponding OS jitter, do any of the following:
+1.	Don't use EHCA Infiniband hardware.  This will prevent these
+	kthreads from being created in the first place.  (This will
+	work for most people, as this hardware, though important,
+	is relatively old as is produced in relatively low unit
+	volumes.)
+2.	Do all EHCA-Infiniband-related work on other CPUs, including
+	interrupts.
+
+
+Name: irq/%d-%s
+Purpose: Handle threaded interrupts.
+To reduce corresponding OS jitter, do the following:
+1.	Use irq affinity to force the irq threads to execute on
+	some other CPU.
+
+Name: kcmtpd_ctr_%d
+Purpose: Handle Bluetooth work.
+To reduce corresponding OS jitter, do one of the following:
+1.	Don't use Bluetooth, in cwhich case these kthreads won't be
+	created in the first place.
+2.	Use irq affinity to force Bluetooth-related interrupts to
+	occur on some other CPU and furthermore initiate all
+	Bluetooth activity from some other CPU.
+
+Name: ksoftirqd/%u
+Purpose: Execute softirq handlers when threaded or when under heavy load.
+To reduce corresponding OS jitter, each softirq vector must be handled
+separately as follows:
+TIMER_SOFTIRQ:
+1.	Build with CONFIG_HOTPLUG_CPU=y.
+2.	To the extent possible, keep the CPU out of the kernel when it
+	is non-idle, for example, by forcing user and kernel threads as
+	well as interrupts to execute elsewhere.
+3.	Force the CPU offline, then bring it back online.  This forces
+	recurring timers to migrate elsewhere.  If you are concerned
+	with multiple CPUs, force them all offline before bringing the
+	first one back online.
+NET_TX_SOFTIRQ and NET_RX_SOFTIRQ:  Do all of the following:
+1.	Force networking interrupts onto other CPUs.
+2.	Initiate any network I/O on other CPUs.
+3.	Prevent CPU-hotplug operations from being initiated from tasks
+	that might run on the CPU to be de-jittered.
+BLOCK_SOFTIRQ:  Do all of the following:
+1.	Force block-device interrupts onto some other CPU.
+2.	Initiate any block I/O on other CPUs.
+3.	Prevent CPU-hotplug operations from being initiated from tasks
+	that might run on the CPU to be de-jittered.
+BLOCK_IOPOLL_SOFTIRQ:  Do all of the following:
+1.	Force block-device interrupts onto some other CPU.
+2.	Initiate any block I/O and block-I/O polling on other CPUs.
+3.	Prevent CPU-hotplug operations from being initiated from tasks
+	that might run on the CPU to be de-jittered.
+TASKLET_SOFTIRQ: Do one or more of the following:
+1.	Avoid use of drivers that use tasklets.
+2.	Convert all drivers that you must use from tasklets to workqueues.
+3.	Force interrupts for drivers using tasklets onto other CPUs,
+	and also do I/O involving these drivers on other CPUs.
+SCHED_SOFTIRQ: Do all of the following:
+1.	Avoid sending scheduler IPIs to the CPU to be de-jittered,
+	for example, ensure that at most one runnable kthread is
+	present on that CPU.  If a thread awakens that expects
+	to run on the de-jittered CPU, the scheduler will send
+	an IPI that can result in a subsequent SCHED_SOFTIRQ.
+2.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
+	CONFIG_NO_HZ_EXTENDED=y, and in addition ensure that the CPU
+	to be de-jittered is marked as an adaptive-ticks CPU using the
+	"nohz_extended=" boot parameter.  This reduces the number of
+	scheduler-clock interrupts that the de-jittered CPU receives,
+	minimizing its chances of being selected to do load balancing,
+	which happens in SCHED_SOFTIRQ context.
+3.	To the extent possible, keep the CPU out of the kernel when it
+	is non-idle, for example, by forcing user and kernel threads as
+	well as interrupts to execute elsewhere.  This further reduces
+	the number of scheduler-clock interrupts that the de-jittered
+	CPU receives.
+HRTIMER_SOFTIRQ:  Do all of the following:
+1.	Build with CONFIG_HOTPLUG_CPU=y.
+2.	To the extent possible, keep the CPU out of the kernel when it
+	is non-idle, for example, by forcing user and kernel threads as
+	well as interrupts to execute elsewhere.
+3.	Force the CPU offline, then bring it back online.  This forces
+	recurring timers to migrate elsewhere.  If you are concerned
+	with multiple CPUs, force them all offline before bringing the
+	first one back online.
+RCU_SOFTIRQ:  Do at least one of the following:
+1.	Offload callbacks and keep the CPU in either dyntick-idle or
+	adaptive-ticks state by doing all of the following:
+	a.	Build with CONFIG_RCU_NOCB_CPU=y, CONFIG_RCU_NOCB_CPU_ALL=y,
+		CONFIG_NO_HZ_EXTENDED=y, and in addition ensure that
+		the CPU to be de-jittered is marked as an adaptive-ticks CPU
+		using the "nohz_extended=" boot parameter.
+	b.	To the extent possible, keep the CPU out of the kernel
+		when it is non-idle, for example, by forcing user and
+		kernel threads as well as interrupts to execute elsewhere.
+2.	Enable RCU to do its processing remotely via dyntick-idle by
+	doing all of the following:
+	a.	Build with CONFIG_NO_HZ=y and CONFIG_RCU_FAST_NO_HZ=y.
+	b.	To the extent possible, keep the CPU out of the kernel
+		when it is non-idle, for example, by forcing user and
+		kernel threads as well as interrupts to execute elsewhere.
+	c.	Ensure that the CPU goes idle frequently, allowing other
+		CPUs to detect that it has passed through an RCU
+		quiescent state.
+
+Name: rcuc/%u
+Purpose: Execute RCU callbacks in CONFIG_RCU_BOOST=y kernels.
+To reduce corresponding OS jitter, do at least one of the following:
+1.	Build the kernel with CONFIG_PREEMPT=n.  This prevents these
+	kthreads from being created in the first place, and also prevents
+	RCU priority boosting from ever being required.  This approach
+	is feasible for workloads that do not require high degrees of
+	responsiveness.
+2.	Build the kernel with CONFIG_RCU_BOOST=n.  This prevents these
+	kthreads from being created in the first place.  This approach
+	is feasible only if your workload never requires RCU priority
+	boosting, for example, if you ensure ample idle time on all CPUs
+	that might execute within the kernel.
+3.	Build with CONFIG_RCU_NOCB_CPU=y and CONFIG_RCU_NOCB_CPU_ALL=y,
+	which offloads all RCU callbacks to kthreads that can be moved
+	off of CPUs susceptible to OS jitter.  This approach prevents the
+	rcuc/%u kthreads from having any work to do, and are therefore
+	never awakened.
+4.	Ensure that then CPU never enters the kernel and avoid any
+	CPU hotplug operations.  This is another way of preventing any
+	callbacks from being queued on the CPU, again preventing the
+	rcuc/%u kthreads from having any work to do.
+
+Name: rcuob/%d, rcuop/%d, and rcuos/%d
+Purpose: Offload RCU callbacks from the corresponding CPU.
+To reduce corresponding OS jitter, do at least one of the following:
+1.	Use affinity, cgroups, or other mechanism to force these kthreads
+	to execute on some other CPU.
+2.	Build with CONFIG_RCU_NOCB_CPUS=n, which will prevent these
+	kthreads from being created in the first place.  However,
+	please note that this will not eliminate the corresponding
+	OS jitter, but will instead merely shift it to softirq.
+
+Name: watchdog/%u
+Purpose: Detect software lockups on each CPU.
+To reduce corresponding OS jitter, do at least one of the following:
+1.	Build with CONFIG_LOCKUP_DETECTOR=n, which will prevent these
+	kthreads from being created in the first place.
+2.	Echo a zero to /proc/sys/kernel/watchdog to disable the
+	watchdog timer.
+3.	Echo a large number of /proc/sys/kernel/watchdog_thresh in
+	order to reduce the frequency of OS jitter due to the watchdog
+	timer down to a level that is acceptable for your workload.
-- 
1.8.1.5