Weird NET_RX softirq behavior

Weird NET_RX softirq behavior

[Jisheng Zhang]

Hi list,

I observed one weird NET_RX softirq behavior:


1. CONFIG_RPS=y in kernel config file.

2. only one netdev in the system: eth0. eth0 only has one interrupt which is
all handled by CPU0. eth0 doesn't support RFS_ACCEL

"cat /proc/interrupts" shows 

~ # cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       
...
 49:         52          0          0          0       GIC  mmc1
 52:         53          0          0          0       GIC  mmc2
 56:     430365          0          0          0       GIC  eth0
 60:         61          0          0          0       GIC  mmc0
...

3. RPS and RFS remains disabled as default.

~ # ls /sys/class/net/eth0/queues/
rx-0  rx-1  rx-2  rx-3  tx-0
~ # cat /sys/class/net/eth0/queues/rx-*/rps_cpus
0
0
0
0
~ # cat /proc/sys/net/core/rps_sock_flow_entries
0
~ # cat /sys/class/net/eth0/queues/rx-*/rps_flow_cnt
0
0
0
0

4. But NET_RX seems abnormal
~ # cat /proc/softirqs 
                    CPU0       CPU1       CPU2       CPU3       
          HI:          0          0          0          0
       TIMER:      92386        767        367        263
      NET_TX:       6338          0          0          1
      NET_RX:     445587        322        983          0
       BLOCK:          0          0          0          0
BLOCK_IOPOLL:          0          0          0          0
     TASKLET:        150          0          0          0
       SCHED:      31414        490        275        179
     HRTIMER:          1          0          0          0
         RCU:      49029        570        285        182

I'm expecting NET_RX under CPU1, 2, 3 should be zero. Any suggestions
about this abnormal behavior?

Thanks in advance,
Jisheng

Re: Weird NET_RX softirq behavior

[Dmitry Popov]

On Thu, 7 Aug 2014 17:10:50 +0800
Jisheng Zhang <jszhang@marvell.com> wrote:

> 2. only one netdev in the system: eth0.

There should also be lo (loopback) at least.

> 4. But NET_RX seems abnormal
> ~ # cat /proc/softirqs 
>                     CPU0       CPU1       CPU2       CPU3       
>       NET_RX:     445587        322        983          0
> 
> I'm expecting NET_RX under CPU1, 2, 3 should be zero. Any suggestions
> about this abnormal behavior?

Do you have any loopback traffic? It could be handled by CPU1/2 explaining
non-zero NET_RX counters.

Re: Weird NET_RX softirq behavior

[Jisheng Zhang]

Hi Dmitry,

On Thu, 7 Aug 2014 07:18:13 -0700
Dmitry Popov <ixaphire@qrator.net> wrote:

> On Thu, 7 Aug 2014 17:10:50 +0800
> Jisheng Zhang <jszhang@marvell.com> wrote:
> 
> > 2. only one netdev in the system: eth0.
> 
> There should also be lo (loopback) at least.

Yep, I forget that ;)

> 
> > 4. But NET_RX seems abnormal
> > ~ # cat /proc/softirqs 
> >                     CPU0       CPU1       CPU2       CPU3       
> >       NET_RX:     445587        322        983          0
> > 
> > I'm expecting NET_RX under CPU1, 2, 3 should be zero. Any suggestions
> > about this abnormal behavior?
> 
> Do you have any loopback traffic? It could be handled by CPU1/2 explaining
> non-zero NET_RX counters.

Yes. lo only send and recive 7 packets so far, about 1400bytes. Seems small
comparing with the CPU1 and CPU2 NET_RX softirq numbers, right?

Any other possible case?

Thanks for your reply,
Jisheng

Re: Weird NET_RX softirq behavior

[Eric Dumazet]

On Fri, 2014-08-08 at 10:37 +0800, Jisheng Zhang wrote:
> nd and recive 7 packets so far, about 1400bytes. Seems small
> comparing with the CPU1 and CPU2 NET_RX softirq numbers, right?
> 
> Any other possible case?

Multicast loop. Check dev_loopback_xmit() and its callers.

Re: Weird NET_RX softirq behavior

[Jisheng Zhang]

On Thu, 7 Aug 2014 22:47:58 -0700
Eric Dumazet <eric.dumazet@gmail.com> wrote:

> On Fri, 2014-08-08 at 10:37 +0800, Jisheng Zhang wrote:
> > nd and recive 7 packets so far, about 1400bytes. Seems small
> > comparing with the CPU1 and CPU2 NET_RX softirq numbers, right?
> > 
> > Any other possible case?
> 
> Multicast loop. Check dev_loopback_xmit() and its callers.
> 
> 

That's it! This make sense

I didn't see any NET_RX on CPU1,2,3 if I disabled mulitcast

Thanks very much.