Re: [PATCH 4/5] irq_work: Handle some irq_work in SOFTIRQ on PREEMPT_RT

[Peter Zijlstra <peterz@infradead.org>]

On Thu, Sep 30, 2021 at 06:38:58PM +0200, Sebastian Andrzej Siewior wrote:
> On 2021-09-30 16:39:51 [+0200], Peter Zijlstra wrote:

> > Runing them all at the same prio still sucks (much like the single
> > net-RX thing), but at least a kthread is somewhat controllable.
> 
> I could replace the softirq processing with a per-CPU thread. This
> should work. But I would have to (still) delay the wake-up of the thread
> to the timer tick - or - we try the wake from the irqwork-self-IPI.

That, just wake the thread from the hardirq.

> I
> just don't know how many will arrive back-to-back. The RCU callback
> (rcu_preempt_deferred_qs_handler()) pops up a lot. By my naive guesswork
> I would say that the irqwork is not needed since preempt-enable
> somewhere should do needed scheduling. But then commit
>   0864f057b050b ("rcu: Use irq_work to get scheduler's attention in clean context")
> 
> claims it is not enough.

Oh gawd, that was something really nasty. I'm not sure that Changelog
captures all (at least I'm not sure I fully understand the problem again
reading it).

But basically that thing wants to reschedule, but suffers the same
problem as:

	preempt_disable();

	<TIF_NEED_RESCHED gets set>

	local_irq_disable();
	preempt_enable();
	  // cannea schedule because IRQs are disabled
	local_irq_enable();
	// lost a reschedule


Yes, that will _eventually_ reschedule, but violates the PREEMPT rules
because there is an unspecified amount of time until it does actually do
reschedule.

So what RCU does there is basically trigger a self-IPI, which guarantees
that we reschedule after IRQs are finally enabled, which then triggers a
resched.

I see no problem marking that particular irq_work as HARD tho, it really
doesn't do anything (other than tell RCU the GP is no longer blocked)
and triggering the return-from-interrupt path.

There's also a fun comment in perf_lock_task_context() that possibly
predates the above RCU fix.