PREEMPT_RT causes scheduling errors with f_rndis USB gadget

PREEMPT_RT causes scheduling errors with f_rndis USB gadget

[Yunhao Tian]

Hi everyone,

I'm using Linux 5.15.24-rt31 kernel with PREEMPT_RT enabled, on my
RK3308 board. I set up f_rndis gadget with the following script, and
plugged my board to a x86 Linux computer running 5.15.25 kernel:

#!/bin/sh
cd /sys/kernel/config/usb_gadget
mkdir g_smartcross; cd g_smartcross
echo 0xF055 > idVendor
echo 0xCAFE > idProduct
mkdir strings/0x409
echo "SmartCross" > strings/0x409/manufacturer
echo "SC-1 USB Device" > strings/0x409/product
mkdir configs/c.1
mkdir functions/rndis.usb0
echo EF > functions/rndis.usb0/class
echo 4 > functions/rndis.usb0/subclass
echo 1 > functions/rndis.usb0/protocol
ln -s functions/rndis.usb0 configs/c.1
echo ff400000.usb > UDC

 I started a HTTP server using python3 -m http.server on the x86
computer, and executed curl on my board to download a 100MB file from
the computer. I got the following kernel logs from serial console of
the board:

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
7 100M 7 7500k 0 0 7217k 0 0:00:14 0:00:01 0:00:13 7218k[ 23.002846]
sched: RT throttling activated
23 100M 23 23.4M 0 0 5948k 0 0:00:17 0:00:04 0:00:13 5948k[ 25.992869]
NOHZ tick-stop error: Non-RCU local softi!
[ 25.993834] NOHZ tick-stop error: Non-RCU local softirq work is
pending, handler #80!!!
[ 25.994833] NOHZ tick-stop error: Non-RCU local softirq work is
pending, handler #80!!!
[ 25.995832] NOHZ tick-stop error: Non-RCU local softirq work is
pending, handler #80!!!
[ 25.995885] NOHZ tick-stop error: Non-RCU local softirq work is
pending, handler #180!!!
[ 25.996831] NOHZ tick-stop error: Non-RCU local softirq work is
pending, handler #180!!!
[ 25.997830] NOHZ tick-stop error: Non-RCU local softirq work is
pending, handler #180!!!

If I turn off PREEMPT_RT, there won't be any errors while downloading.

I believe this problem is not tied to any specific board, and anyone
can reproduce this problem using a Raspberry Pi (Although I didn't try
because I don't have one). I would like to ask for assistance
regarding this problem.

Re: PREEMPT_RT causes scheduling errors with f_rndis USB gadget

[Greg KH]

On Sun, Mar 06, 2022 at 06:17:54PM +0800, Yunhao Tian wrote:
> Hi everyone,
> 
> I'm using Linux 5.15.24-rt31 kernel with PREEMPT_RT enabled, on my
> RK3308 board. I set up f_rndis gadget with the following script, and
> plugged my board to a x86 Linux computer running 5.15.25 kernel:

Does the same thing happen with the non-rt kernel?

thanks,

greg k-h

Re: PREEMPT_RT causes scheduling errors with f_rndis USB gadget

[Yunhao Tian]

Greg KH <gregkh@linuxfoundation.org> 于2022年3月6日周日 18:36写道：
>
> On Sun, Mar 06, 2022 at 06:17:54PM +0800, Yunhao Tian wrote:
> > Hi everyone,
> >
> > I'm using Linux 5.15.24-rt31 kernel with PREEMPT_RT enabled, on my
> > RK3308 board. I set up f_rndis gadget with the following script, and
> > plugged my board to a x86 Linux computer running 5.15.25 kernel:
>
> Does the same thing happen with the non-rt kernel?
>
> thanks,
>
> greg k-h

Hi Greg,

Thanks for your reply!

This doesn't happen to non-rt kernel. It will only happen when
PREEMPT_RT is turned on, with rt kernel.

Regards,
Yunhao

Re: PREEMPT_RT causes scheduling errors with f_rndis USB gadget

[Greg KH]

On Sun, Mar 06, 2022 at 06:57:08PM +0800, Yunhao Tian wrote:
> Greg KH <gregkh@linuxfoundation.org> 于2022年3月6日周日 18:36写道：
> >
> > On Sun, Mar 06, 2022 at 06:17:54PM +0800, Yunhao Tian wrote:
> > > Hi everyone,
> > >
> > > I'm using Linux 5.15.24-rt31 kernel with PREEMPT_RT enabled, on my
> > > RK3308 board. I set up f_rndis gadget with the following script, and
> > > plugged my board to a x86 Linux computer running 5.15.25 kernel:
> >
> > Does the same thing happen with the non-rt kernel?
> >
> > thanks,
> >
> > greg k-h
> 
> Hi Greg,
> 
> Thanks for your reply!
> 
> This doesn't happen to non-rt kernel. It will only happen when
> PREEMPT_RT is turned on, with rt kernel.

Great, thanks for letting us know.  I'll let you work this out with the
-rt developers :)

Re: PREEMPT_RT causes scheduling errors with f_rndis USB gadget

[Sebastian Andrzej Siewior]

On 2022-03-06 18:17:54 [+0800], Yunhao Tian wrote:
> Hi everyone,
Hi,

> I'm using Linux 5.15.24-rt31 kernel with PREEMPT_RT enabled, on my
> RK3308 board. I set up f_rndis gadget with the following script, and
> plugged my board to a x86 Linux computer running 5.15.25 kernel:

Is this 5.15.24-rt31 as-is or are there other patches involved?
What UDC is used here?

…
> [ 25.993834] NOHZ tick-stop error: Non-RCU local softirq work is
> pending, handler #80!!!
that is sched

…
> [ 25.995885] NOHZ tick-stop error: Non-RCU local softirq work is
> pending, handler #180!!!

that is additionally hrtimer.
I did touch hrtimer but I did not yet touch sched so I account myself as
50% innocent.
You don't have by chance change the thread affinity of tasks/ interrupt
threads? The SoC has 4 cores if I see this right.

> If I turn off PREEMPT_RT, there won't be any errors while downloading.
> 
> I believe this problem is not tied to any specific board, and anyone
> can reproduce this problem using a Raspberry Pi (Although I didn't try
> because I don't have one). I would like to ask for assistance
> regarding this problem.

Let me try dummy_udc…

Sebastian