Re: [PATCH 4/5] irq_work: Handle some irq_work in SOFTIRQ on PREEMPT_RT

[Peter Zijlstra <peterz@infradead.org>]

On Mon, Sep 27, 2021 at 11:19:18PM +0200, Sebastian Andrzej Siewior wrote:
> The irq_work callback is invoked in hard IRQ context. By default all
> callbacks are scheduled for invocation right away (given supported by
> the architecture) except for the ones marked IRQ_WORK_LAZY which are
> delayed until the next timer-tick.
> 
> While looking over the callbacks, some of them may acquire locks
> (spinlock_t, rwlock_t) which are transformed into sleeping locks on
> PREEMPT_RT and must not be acquired in hard IRQ context.
> Changing the locks into locks which could be acquired in this context
> will lead to other problems such as increased latencies if everything
> in the chain has IRQ-off locks. This will not solve all the issues as
> one callback has been noticed which invoked kref_put() and its callback
> invokes kfree() and this can not be invoked in hardirq context.
> 
> Some callbacks are required to be invoked in hardirq context even on
> PREEMPT_RT to work properly. This includes for instance the NO_HZ
> callback which needs to be able to observe the idle context.
> 
> The callbacks which require to be run in hardirq have already been
> marked. Use this information to split the callbacks onto the two lists
> on PREEMPT_RT:
> - lazy_list
>   Work items which are not marked with IRQ_WORK_HARD_IRQ will be added
>   to this list. Callbacks on this list will be invoked from timer
>   softirq handler. The handler here may acquire sleeping locks such as
>   spinlock_t and invoke kfree().
> 
> - raised_list
>   Work items which are marked with IRQ_WORK_HARD_IRQ will be added to
>   this list. They will be invoked in hardirq context and must not
>   acquire any sleeping locks.
> 
> [bigeasy: melt tglx's irq_work_tick_soft() which splits irq_work_tick() into a
>           hard and soft variant. Collected fixes over time from Steven
> 	  Rostedt and Mike Galbraith. ]
> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de>

IIRC we have existing problems in -RT due to this irq_work softirq muck.

I think the problem was something Jolsa found a while ago, where perf
defers to an irq_work (from NMI context) and that irq_work wants to
deliver signals, which it can't on -RT, so the whole thing gets punted
to softirq. With the end-result that if you self-profile RT tasks,
things come apart or something.

There might have been others as well, I don't know. But generally I
think we want *less* softirq, not more.