Re: [PATCH net-next 1/2] net: napi: wake up ksoftirqd if needed after scheduling NAPI

[Eric Dumazet <edumazet@google.com>]

On Thu, Feb 3, 2022 at 3:40 PM Yannick Vignon
<yannick.vignon@oss.nxp.com> wrote:
>
> On 2/3/2022 8:08 PM, Eric Dumazet wrote:
> > On Thu, Feb 3, 2022 at 11:06 AM Yannick Vignon
> > <yannick.vignon@oss.nxp.com> wrote:
> >>
> >> From: Yannick Vignon <yannick.vignon@nxp.com>
> >>
> >> If NAPI was not scheduled from interrupt or softirq,
> >> __raise_softirq_irqoff would mark the softirq pending, but not
> >> wake up ksoftirqd. With force threaded IRQs, this is
> >> compensated by the fact that the interrupt handlers are
> >> protected inside a local_bh_disable()/local_bh_enable()
> >> section, and bh_enable will call do_softirq if needed. With
> >> normal threaded IRQs however, this is no longer the case
> >> (unless the interrupt handler itself calls local_bh_enable()),
> >> whic results in a pending softirq not being handled, and the
> >> following message being printed out from tick-sched.c:
> >> "NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #%02x!!!\n"
> >>
> >> Call raise_softirq_irqoff instead to make sure ksoftirqd is
> >> woken up in such a case, ensuring __napi_schedule, etc behave
> >> normally in more situations than just from an interrupt,
> >> softirq or from within a bh_disable/bh_enable section.
> >>
> >
> > This is buggy. NAPI is called from the right context.
> >
> > Can you provide a stack trace or something, so that the buggy driver
> > can be fixed ?
> >
>
> Maybe some background on how I came to this would be helpful. I have
> been chasing down sources of latencies in processing rx packets on a
> PREEMPT_RT kernel and the stmmac driver. I observed that the main ones
> were bh_dis/en sections, preventing even my high-prio, (force-)threaded
> rx irq from being handled in a timely manner. Given that explicitly
> threaded irq handlers were not enclosed in a bh_dis/en section, and that
> from what I saw the stmmac interrupt handler didn't need such a
> protection anyway, I modified the stmmac driver to request threaded
> interrupts. This worked, safe for that "NOHZ" message: because
> __napi_schedule was now called from a kernel thread context, the softirq
> was no longer triggered.
> (note that the problem solves itself when enabling threaded NAPI)
>
> Is there a rule saying we shouldn't call __napi_schedule from a regular
> kernel thread, and in particular a threaded interrupt handler?

The rule is that you need to be in a safe context before calling
__napi_schedule()

This has been the case for more than a decade.

We are not going to slow down the stack just in case a process context
does not care about BH.

Please check:

commit ec13ee80145ccb95b00e6e610044bbd94a170051
Author: Michael S. Tsirkin <mst@redhat.com>
Date:   Wed May 16 10:57:12 2012 +0300

    virtio_net: invoke softirqs after __napi_schedule


>
> >> Signed-off-by: Yannick Vignon <yannick.vignon@nxp.com>
> >> ---
> >>   net/core/dev.c | 2 +-
> >>   1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/net/core/dev.c b/net/core/dev.c
> >> index 1baab07820f6..f93b3173454c 100644
> >> --- a/net/core/dev.c
> >> +++ b/net/core/dev.c
> >> @@ -4239,7 +4239,7 @@ static inline void ____napi_schedule(struct softnet_data *sd,
> >>          }
> >>
> >>          list_add_tail(&napi->poll_list, &sd->poll_list);
> >> -       __raise_softirq_irqoff(NET_RX_SOFTIRQ);
> >> +       raise_softirq_irqoff(NET_RX_SOFTIRQ);
> >>   }
> >>
> >>   #ifdef CONFIG_RPS
> >> --
> >> 2.25.1
> >>
>