Re: [PATCH v7 05/11] task_isolation: add debug boot flag

From: Luiz Capitulino <lcapitulino@redhat.com>
To: Chris Metcalf <cmetcalf@ezchip.com>
Cc: Gilad Ben Yossef <giladb@ezchip.com>,
	Steven Rostedt <rostedt@goodmis.org>,
	Ingo Molnar <mingo@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	Rik van Riel <riel@redhat.com>, Tejun Heo <tj@kernel.org>,
	Frederic Weisbecker <fweisbec@gmail.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	"Paul E. McKenney" <paulmck@linux.vnet.ibm.com>,
	Christoph Lameter <cl@linux.com>,
	Viresh Kumar <viresh.kumar@linaro.org>,
	Catalin Marinas <catalin.marinas@arm.com>,
	Will Deacon <will.deacon@arm.com>,
	Andy Lutomirski <luto@amacapital.net>,
	<linux-doc@vger.kernel.org>, <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH v7 05/11] task_isolation: add debug boot flag
Date: Thu, 8 Oct 2015 16:28:05 -0400	[thread overview]
Message-ID: <20151008162805.7e4a4f25@redhat.com> (raw)
In-Reply-To: <5615B9F4.5010902@ezchip.com>

On Wed, 7 Oct 2015 20:33:56 -0400
Chris Metcalf <cmetcalf@ezchip.com> wrote:

> On 10/5/2015 1:07 PM, Luiz Capitulino wrote:
> > On Mon, 28 Sep 2015 11:17:20 -0400
> > Chris Metcalf <cmetcalf@ezchip.com> wrote:
> >
> >> The new "task_isolation_debug" flag simplifies debugging
> >> of TASK_ISOLATION kernels when processes are running in
> >> PR_TASK_ISOLATION_ENABLE mode.  Such processes should get no
> >> interrupts from the kernel, and if they do, when this boot flag is
> >> specified a kernel stack dump on the console is generated.
> >>
> >> It's possible to use ftrace to simply detect whether a task_isolation
> >> core has unexpectedly entered the kernel.  But what this boot flag
> >> does is allow the kernel to provide better diagnostics, e.g. by
> >> reporting in the IPI-generating code what remote core and context
> >> is preparing to deliver an interrupt to a task_isolation core.
> >>
> >> It may be worth considering other ways to generate useful debugging
> >> output rather than console spew, but for now that is simple and direct.
> > Honest question: does any of the task_isolation_debug() calls added
> > by this patch take care of the case where vmstat_shepherd() may
> > schedule vmstat_update() to run because a TASK_ISOLATION process is
> > changing memory stats?
> 
> The task_isolation_debug() calls don't "take care of" any cases - they are
> really just there to generate console dumps when the kernel unexpectedly
> interrupts a task_isolated task.
> 
> The idea with vmstat is that before a task_isolated task returns to
> userspace, it quiesces the vmstat thread (does a final sweep to collect
> the stats and turns off the scheduled work item).  As a result, the vmstat
> shepherd won't run while the task is in userspace.  When and if it returns
> to the kernel, it will again sweep up the stats before returning to userspace.
> 
> The usual shepherd mechanism on a housekeeping core might notice
> that the task had entered the kernel and started changing stats, and
> might then asynchronously restart the scheduled work, but it should be
> quiesced again regardless on the way back out to userspace.

OK, I've missed the (obvious) fact that the process has to enter the
kernel to change stats. Thanks a lot for your explanation.

> > If that's not taken care of yet, should we? I just don't know if we
> > should call task_isolation_exception() or task_isolation_debug().
> 
> task_isolation_exception() is called when an exception (page fault or
> similar) is generated synchronously by the running task and we want
> to make sure to notify the task with a signal if it has set up STRICT mode
> to indicate that it is not planning to enter the kernel.
> 
> > In the case of the latter, wouldn't it be interesting to add it to
> > __queue_work() then?
> 
> Well, queuing remote work involves sending an IPI, and we already tag
> both the SMP send side AND the client side IRQ side with a task_isolation_debug(),
> so I expect in practice it would be detected.
> 