Re: [PATCH 4/5] irq_work: Handle some irq_work in SOFTIRQ on PREEMPT_RT

[Sebastian Andrzej Siewior <bigeasy@linutronix.de>]

On 2021-10-01 12:32:38 [+0200], Peter Zijlstra wrote:
> On Thu, Sep 30, 2021 at 06:38:58PM +0200, Sebastian Andrzej Siewior wrote:
> > On 2021-09-30 16:39:51 [+0200], Peter Zijlstra wrote:
> 
> > > Runing them all at the same prio still sucks (much like the single
> > > net-RX thing), but at least a kthread is somewhat controllable.
> > 
> > I could replace the softirq processing with a per-CPU thread. This
> > should work. But I would have to (still) delay the wake-up of the thread
> > to the timer tick - or - we try the wake from the irqwork-self-IPI.
> 
> That, just wake the thread from the hardirq.

"just". Let me do that and see how bad it gets ;)

> > I
> > just don't know how many will arrive back-to-back. The RCU callback
> > (rcu_preempt_deferred_qs_handler()) pops up a lot. By my naive guesswork
> > I would say that the irqwork is not needed since preempt-enable
> > somewhere should do needed scheduling. But then commit
> >   0864f057b050b ("rcu: Use irq_work to get scheduler's attention in clean context")
> > 
> > claims it is not enough.
> 
> Oh gawd, that was something really nasty. I'm not sure that Changelog
> captures all (at least I'm not sure I fully understand the problem again
> reading it).
> 
> But basically that thing wants to reschedule, but suffers the same
> problem as:
> 
> 	preempt_disable();
> 
> 	<TIF_NEED_RESCHED gets set>
> 
> 	local_irq_disable();
> 	preempt_enable();
> 	  // cannea schedule because IRQs are disabled
> 	local_irq_enable();
> 	// lost a reschedule
> 
> 
> Yes, that will _eventually_ reschedule, but violates the PREEMPT rules
> because there is an unspecified amount of time until it does actually do
> reschedule.

Yeah but buh. We could let local_irq_enable/restore() check that
need-resched bit if the above is considered pretty and supported _or_
start to yell if it is not. A middle way would be to trigger that
self-IPI in such a case. I mean everyone suffers from that lost
reschedule and, if I'm not mistaken, you don't receive a remote wakeup
because the remote CPU notices need-resched bit and assumes that it is
about to be handled. So RCU isn't special here.

> So what RCU does there is basically trigger a self-IPI, which guarantees
> that we reschedule after IRQs are finally enabled, which then triggers a
> resched.
> 
> I see no problem marking that particular irq_work as HARD tho, it really
> doesn't do anything (other than tell RCU the GP is no longer blocked)
> and triggering the return-from-interrupt path.

Hmm. Your Highness. I'm going back to my peasant village to build the
thread you asked for. I will look into this. I see two of those irq-work
things that is the scheduler thingy and this.

Thanks.

> There's also a fun comment in perf_lock_task_context() that possibly
> predates the above RCU fix.

Sebastian