Re: [PATCH 3/6] dataplane nohz: run softirqs synchronously on user entry

[Chris Metcalf <cmetcalf@ezchip.com>]

On 05/09/2015 03:04 AM, Mike Galbraith wrote:
> On Fri, 2015-05-08 at 13:58 -0400, Chris Metcalf wrote:
>> For tasks which have elected dataplane functionality, we run
>> any pending softirqs for the core before returning to userspace,
>> rather than ever scheduling ksoftirqd to run.  The problem we
>> fix is that by allowing another task to run on the core, we
>> guarantee more interrupts in the future to the dataplane task,
>> which is exactly what dataplane mode is required to prevent.
> If ksoftirqd were rt class

I realize I actually don't know if this is true or not.  Is
ksoftirqd rt class?  If not, it does seem pretty plausible that
it should be...

> softirqs would be gone when the soloist gets
> the CPU back and heads to userspace.  Being a soloist, it has no use for
> a priority, so why can't it just let ksoftirqd run if it raises the
> occasional softirq?  Meeting a contended lock while processing it will
> wreck the soloist regardless of who does that processing.

The thing you want to avoid is having two processes both
runnable at once, since then the "quiesce" mode can't make
forward progress and basically spins in cpu_idle() until ksoftirqd
can come in.  Alas, my recollection of the precise failure mode
is somewhat dimmed; my commit notes from a year ago (for
a variant of the patch I'm upstreaming now):

         - Trying to return to userspace with pending softirqs is not
           currently allowed.  Prior to this patch, when this happened
           we would just wait in cpu_idle.  Instead, what we now do is
           directly run any pending softirqs, then go back and retry the
           path where we return to userspace.
         
         - Raising softirqs (in this case for hrtimer support) could
           cause the ksoftirqd daemon to be woken on a core.  This is
           bad because on a dataplane core, a QUIESCE process will
           then block until the ksoftirqd runs, and the system sometimes
           seems to flag that soft irqs are available but not schedule
           the timer to arrange for a context switch to ksoftirqd.
           To handle this, we avoid bailing out in __do_softirq() when
           we've been working for a while, if we're on a dataplane core,
           and just keep working until done.  Similarly, on a dataplane
           core running a userspace task, we don't wake ksoftirqd when
           we are raising a softirq, even if we're not in an interrupt
           context where it will run promptly, since a non-interrupt
           context will also run promptly.

I'm happy to drop this patch entirely from the series for now, and
if ksoftirqd shows up as a problem going forward, we can address it
as necessary at that time.   What do you think?

-- 
Chris Metcalf, EZChip Semiconductor
http://www.ezchip.com