Re: [PATCH 4/4] nohz: Set isolcpus when nohz_full is set

From: Afzal Mohammed <afzal.mohd.ma@gmail.com>
To: Frederic Weisbecker <fweisbec@gmail.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
	Mike Galbraith <umgwanakikbuti@gmail.com>,
	Sasha Levin <levinsasha928@gmail.com>,
	Ingo Molnar <mingo@kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Chris Metcalf <cmetcalf@ezchip.com>,
	"Rafael J . Wysocki" <rafael.j.wysocki@intel.com>,
	Peter Zijlstra <peterz@infradead.org>,
	Dave Jones <davej@redhat.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	Oleg Nesterov <oleg@redhat.com>, Ingo Molnar <mingo@redhat.com>,
	Rik van Riel <riel@redhat.com>,
	Martin Schwidefsky <schwidefsky@de.ibm.com>
Subject: Re: [PATCH 4/4] nohz: Set isolcpus when nohz_full is set
Date: Thu, 21 May 2015 18:59:05 +0530	[thread overview]
Message-ID: <20150521132905.GA8706@afzalpc> (raw)
In-Reply-To: <20150521130621.GA14324@lerouge>

Hi,

On Thu, May 21, 2015 at 03:06:23PM +0200, Frederic Weisbecker wrote:
> On Thu, May 21, 2015 at 05:57:59AM -0700, Paul E. McKenney wrote:

> > Indeed, NO_HZ_FULL is special purpose.  You normally would select
> > NO_HZ_FULL_ALL only on a system intended for heavy compute without
> > normal-workload distractions or for some real-time systems.  For mixed
> > workloads, you would build with NO_HZ_FULL (but not NO_HZ_FULL_ALL) and
> > use the boot parameters to select which CPUs are to be running the
> > specialized portion of the workload.
> > 
> > And you would of course need to lead enough CPUs running normally to
> > handle the non-specialized portion of the workload.
> > 
> > This sort of thing has traditionally required specialized kernels,
> > so the cool thing here is that we can make Linux do it.  Though, as
> > you noticed, careful configuration is still required.
> > 
> > Seem reasonable?

Yes, thanks, some dots got connected :)

> That said if he saw a big performance regression after applying these patches,
> then there is likely a problem in the patchset. Well it could be due to that mode
> which loops on full dynticks before resuming to userspace. Indeed when that is
> enabled, I expect real throughput issues on workloads doing lots of kernel <->
> userspace roundtrips. We just need to make sure this thing only works when requested.

With this change (& having NO_HZ_FULL_ALL), hackbench was being served
only by the boot cpu, while w/o this change, all 8 (this is a quad
core HT processor) was being used - observation based on 'top'.

Regards
Afzal