Re: [PATCH 6/6] sched/isolation: Update nohz documentation to explain tick offload

From: Frederic Weisbecker <frederic@kernel.org>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Chris Metcalf <cmetcalf@mellanox.com>,
	Luiz Capitulino <lcapitulino@redhat.com>,
	Christoph Lameter <cl@linux.com>,
	"Paul E . McKenney" <paulmck@linux.vnet.ibm.com>,
	Wanpeng Li <kernellwp@gmail.com>, Mike Galbraith <efault@gmx.de>,
	Rik van Riel <riel@redhat.com>
Subject: Re: [PATCH 6/6] sched/isolation: Update nohz documentation to explain tick offload
Date: Tue, 20 Feb 2018 17:40:22 +0100	[thread overview]
Message-ID: <20180220164021.GB13879@lerouge> (raw)
In-Reply-To: <alpine.DEB.2.21.1802171152060.2087@nanos.tec.linutronix.de>

On Sat, Feb 17, 2018 at 11:53:22AM +0100, Thomas Gleixner wrote:
> On Thu, 15 Feb 2018, Frederic Weisbecker wrote:
> 
> > Update the documentation to reflect the 1Hz tick offload changes.
> > 
> > Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
> > Cc: Chris Metcalf <cmetcalf@mellanox.com>
> > Cc: Christoph Lameter <cl@linux.com>
> > Cc: Luiz Capitulino <lcapitulino@redhat.com>
> > Cc: Mike Galbraith <efault@gmx.de>
> > Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
> > Cc: Peter Zijlstra <peterz@infradead.org>
> > Cc: Rik van Riel <riel@redhat.com>
> > Cc: Thomas Gleixner <tglx@linutronix.de>
> > Cc: Wanpeng Li <kernellwp@gmail.com>
> > Cc: Ingo Molnar <mingo@kernel.org>
> > ---
> >  Documentation/admin-guide/kernel-parameters.txt | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> > 
> > diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt
> > index 1d1d53f..50b9837 100644
> > --- a/Documentation/admin-guide/kernel-parameters.txt
> > +++ b/Documentation/admin-guide/kernel-parameters.txt
> > @@ -1766,6 +1766,17 @@
> >  
> >  			nohz
> >  			  Disable the tick when a single task runs.
> > +
> > +			  A residual 1Hz tick is offloaded to workqueues, which you
> > +			  need to affine to housekeeping through the global
> > +			  workqueue's affinity configured via the
> > +			  /sys/devices/virtual/workqueue/cpumask sysfs file, or
> > +			  by using the 'domain' flag described below.
> > +
> > +			  NOTE: by default the global workqueue runs on all CPUs,
> > +			  so to protect individual CPUs the 'cpumask' file has to
> > +			  be configured manually after bootup.
> 
> Hmm. Why don't you set that to the housekeeping cpus by default?

"nohz_full=" does affine to housekeeping by default but "isolcpus=nohz" doesn't
as it is designed for more finegrained settings.

In fact nohz_full= is what people want to use right now as isolcpus= is still
under construction. I'm waiting to be able to make its effects mutable through
cpusets before extending it further. But in the end I would like to have things
such as:

     isolcpus=nohz,async

This would affine the tick and everything that is async (workqueues, unbound timers,
unbound kthreads, etc...) to housekeeping.

And an even simpler toggle to fully isolate a CPU:

     isolcpus=all

This is to avoid that nohz alone implies side effects. For example if we affine
workqueues right now and later the 1hz offloaded tick isn't necessary anymore,
we'll stick with affined workqueues as a side effect without reason, and reverting
that would end up into regression.

Thanks.