FEC on i.MX 7 transmit queue timeout

FEC on i.MX 7 transmit queue timeout

[Stefan Agner]

Hi,

I noticed last week on upstream (v4.11-rc6) on a Colibri iMX7 board that
after a while (~10 minutes) the detdev wachdog prints a stacktrace and
the driver then continuously dumps the TX ring. I then did a quick test
with 4.10, and realized it actually suffers the same issue, so it seems
not to be a regression. I use a rootfs mounted over NFS...

------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
dev_watchdog+0x240/0x244
NETDEV WATCHDOG: eth0 (fec): transmit queue 2 timed out
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted
4.11.0-rc7-00030-g2c4e6bd0c4f0-dirty #330
Hardware name: Freescale i.MX7 Dual (Device Tree)
[<c02293f0>] (unwind_backtrace) from [<c0225820>] (show_stack+0x10/0x14)
[<c0225820>] (show_stack) from [<c050db6c>] (dump_stack+0x90/0xa0)
[<c050db6c>] (dump_stack) from [<c023ae68>] (__warn+0xac/0x11c)
[<c023ae68>] (__warn) from [<c023af10>] (warn_slowpath_fmt+0x38/0x48)
[<c023af10>] (warn_slowpath_fmt) from [<c088bb8c>]
(dev_watchdog+0x240/0x244)
[<c088bb8c>] (dev_watchdog) from [<c0294798>]
(run_timer_softirq+0x24c/0x708)
[<c0294798>] (run_timer_softirq) from [<c023f584>]
(__do_softirq+0x12c/0x2a8)
[<c023f584>] (__do_softirq) from [<c023f8c4>] (irq_exit+0xdc/0x13c)
[<c023f8c4>] (irq_exit) from [<c02818ac>]
(__handle_domain_irq+0xa4/0xf8)
[<c02818ac>] (__handle_domain_irq) from [<c0201624>]
(gic_handle_irq+0x34/0xa4)
[<c0201624>] (gic_handle_irq) from [<c0226338>] (__irq_svc+0x58/0x8c)
Exception stack(0xc1201f30 to 0xc1201f78)
1f20:                                     c0233320 00000000 00000000
01400000
1f40: c1203d80 ffffe000 00000000 00000000 c107bf10 c0e055b5 c1203d34
00000001
1f60: c07d2324 c1201f80 c0222ac8 c0222acc 60000013 ffffffff
[<c0226338>] (__irq_svc) from [<c0222acc>] (arch_cpu_idle+0x38/0x3c)
[<c0222acc>] (arch_cpu_idle) from [<c0275f24>] (do_idle+0xa8/0x250)
[<c0275f24>] (do_idle) from [<c02760e4>] (cpu_startup_entry+0x18/0x1c)
[<c02760e4>] (cpu_startup_entry) from [<c1000aa0>]
(start_kernel+0x3fc/0x45c)
---[ end trace 5b0c6dc3466a7918 ]---
fec 30be0000.ethernet eth0: TX ring dump
Nr     SC     addr       len  SKB
  0    0x1c00 0x00000000  590   (null)
  1    0x1c00 0x00000000  590   (null)
  2    0x1c00 0x00000000   42   (null)
  3  H 0x1c00 0x00000000   42   (null)
  4 S  0x0000 0x00000000    0   (null)
  5    0x0000 0x00000000    0   (null)
  6    0x0000 0x00000000    0   (null)
  7    0x0000 0x00000000    0   (null)
  8    0x0000 0x00000000    0   (null)
  9    0x0000 0x00000000    0   (null)
 10    0x0000 0x00000000    0   (null)
 11    0x0000 0x00000000    0   (null)
 12    0x0000 0x00000000    0   (null)
 13    0x0000 0x00000000    0   (null)
 14    0x0000 0x00000000    0   (null)
 15    0x0000 0x00000000    0   (null)
 16    0x0000 0x00000000    0   (null)
 17    0x0000 0x00000000    0   (null)
 18    0x0000 0x00000000    0   (null)
...


A second TX ring dump from 4.10:
fec 30be0000.ethernet eth0: TX ring dump
Nr     SC     addr       len  SKB
  0    0x1c00 0x00000000   42   (null)
  1    0x1c00 0x00000000   42   (null)
  2    0x1c00 0x00000000   90   (null)
  3    0x1c00 0x00000000   90   (null)
  4    0x1c00 0x00000000   90   (null)
  5    0x1c00 0x00000000  218   (null)
  6    0x1c00 0x00000000  218   (null)
  7    0x1c00 0x00000000  218   (null)
  8    0x1c00 0x00000000   90   (null)
  9    0x1c00 0x00000000  206   (null)
 10    0x1c00 0x00000000  216   (null)
 11    0x1c00 0x00000000  216   (null)
 12    0x1c00 0x00000000  216   (null)
 13    0x1c00 0x00000000  311   (null)
 14    0x1c00 0x00000000  178   (null)
 15    0x1c00 0x00000000  311   (null)
 16    0x1c00 0x00000000  206   (null)
 17  H 0x1c00 0x00000000  311   (null)
 18 S  0x0000 0x00000000    0   (null)
 19    0x0000 0x00000000    0   (null)

The ring dump prints continously, but I can access console every now and
then. I noticed that the second interrupt seems static (66441, TX
interrupt?):
 58:         18     GIC-0 150 Level     30be0000.ethernet
 59:      66441     GIC-0 151 Level     30be0000.ethernet
 60:      70477     GIC-0 152 Level     30be0000.ethernet

Anybody else seen this? Any idea?

In 4.10 as well as 4.11-rc6 the interrupt counts were just over 65536...
pure chance?

Re: FEC on i.MX 7 transmit queue timeout

[Andy Duan]

On 2017年04月19日 03:46, Stefan Agner wrote:
> Hi,
>
> I noticed last week on upstream (v4.11-rc6) on a Colibri iMX7 board that
> after a while (~10 minutes) the detdev wachdog prints a stacktrace and
> the driver then continuously dumps the TX ring. I then did a quick test
> with 4.10, and realized it actually suffers the same issue, so it seems
> not to be a regression. I use a rootfs mounted over NFS...
>
> ------------[ cut here ]------------
> WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
> dev_watchdog+0x240/0x244
> NETDEV WATCHDOG: eth0 (fec): transmit queue 2 timed out
> Modules linked in:
> CPU: 0 PID: 0 Comm: swapper/0 Not tainted
> 4.11.0-rc7-00030-g2c4e6bd0c4f0-dirty #330
> Hardware name: Freescale i.MX7 Dual (Device Tree)
> [<c02293f0>] (unwind_backtrace) from [<c0225820>] (show_stack+0x10/0x14)
> [<c0225820>] (show_stack) from [<c050db6c>] (dump_stack+0x90/0xa0)
> [<c050db6c>] (dump_stack) from [<c023ae68>] (__warn+0xac/0x11c)
> [<c023ae68>] (__warn) from [<c023af10>] (warn_slowpath_fmt+0x38/0x48)
> [<c023af10>] (warn_slowpath_fmt) from [<c088bb8c>]
> (dev_watchdog+0x240/0x244)
> [<c088bb8c>] (dev_watchdog) from [<c0294798>]
> (run_timer_softirq+0x24c/0x708)
> [<c0294798>] (run_timer_softirq) from [<c023f584>]
> (__do_softirq+0x12c/0x2a8)
> [<c023f584>] (__do_softirq) from [<c023f8c4>] (irq_exit+0xdc/0x13c)
> [<c023f8c4>] (irq_exit) from [<c02818ac>]
> (__handle_domain_irq+0xa4/0xf8)
> [<c02818ac>] (__handle_domain_irq) from [<c0201624>]
> (gic_handle_irq+0x34/0xa4)
> [<c0201624>] (gic_handle_irq) from [<c0226338>] (__irq_svc+0x58/0x8c)
> Exception stack(0xc1201f30 to 0xc1201f78)
> 1f20:                                     c0233320 00000000 00000000
> 01400000
> 1f40: c1203d80 ffffe000 00000000 00000000 c107bf10 c0e055b5 c1203d34
> 00000001
> 1f60: c07d2324 c1201f80 c0222ac8 c0222acc 60000013 ffffffff
> [<c0226338>] (__irq_svc) from [<c0222acc>] (arch_cpu_idle+0x38/0x3c)
> [<c0222acc>] (arch_cpu_idle) from [<c0275f24>] (do_idle+0xa8/0x250)
> [<c0275f24>] (do_idle) from [<c02760e4>] (cpu_startup_entry+0x18/0x1c)
> [<c02760e4>] (cpu_startup_entry) from [<c1000aa0>]
> (start_kernel+0x3fc/0x45c)
> ---[ end trace 5b0c6dc3466a7918 ]---
> fec 30be0000.ethernet eth0: TX ring dump
> Nr     SC     addr       len  SKB
>    0    0x1c00 0x00000000  590   (null)
>    1    0x1c00 0x00000000  590   (null)
>    2    0x1c00 0x00000000   42   (null)
>    3  H 0x1c00 0x00000000   42   (null)
>    4 S  0x0000 0x00000000    0   (null)
>    5    0x0000 0x00000000    0   (null)
>    6    0x0000 0x00000000    0   (null)
>    7    0x0000 0x00000000    0   (null)
>    8    0x0000 0x00000000    0   (null)
>    9    0x0000 0x00000000    0   (null)
>   10    0x0000 0x00000000    0   (null)
>   11    0x0000 0x00000000    0   (null)
>   12    0x0000 0x00000000    0   (null)
>   13    0x0000 0x00000000    0   (null)
>   14    0x0000 0x00000000    0   (null)
>   15    0x0000 0x00000000    0   (null)
>   16    0x0000 0x00000000    0   (null)
>   17    0x0000 0x00000000    0   (null)
>   18    0x0000 0x00000000    0   (null)
> ...
>
>
> A second TX ring dump from 4.10:
> fec 30be0000.ethernet eth0: TX ring dump
> Nr     SC     addr       len  SKB
>    0    0x1c00 0x00000000   42   (null)
>    1    0x1c00 0x00000000   42   (null)
>    2    0x1c00 0x00000000   90   (null)
>    3    0x1c00 0x00000000   90   (null)
>    4    0x1c00 0x00000000   90   (null)
>    5    0x1c00 0x00000000  218   (null)
>    6    0x1c00 0x00000000  218   (null)
>    7    0x1c00 0x00000000  218   (null)
>    8    0x1c00 0x00000000   90   (null)
>    9    0x1c00 0x00000000  206   (null)
>   10    0x1c00 0x00000000  216   (null)
>   11    0x1c00 0x00000000  216   (null)
>   12    0x1c00 0x00000000  216   (null)
>   13    0x1c00 0x00000000  311   (null)
>   14    0x1c00 0x00000000  178   (null)
>   15    0x1c00 0x00000000  311   (null)
>   16    0x1c00 0x00000000  206   (null)
>   17  H 0x1c00 0x00000000  311   (null)
>   18 S  0x0000 0x00000000    0   (null)
>   19    0x0000 0x00000000    0   (null)
The dump show tx ring is fine.

>
> The ring dump prints continously, but I can access console every now and
> then. I noticed that the second interrupt seems static (66441, TX
> interrupt?):
>   58:         18     GIC-0 150 Level     30be0000.ethernet
>   59:      66441     GIC-0 151 Level     30be0000.ethernet
>   60:      70477     GIC-0 152 Level     30be0000.ethernet
150 irq number is for tx/rx queue 1 receive/transmit buffer/frame done.
151 irq number is for tx/rx queue 2 receive/transmit buffer/frame done.
152 irq number is for tx/rx queue 0 receive/transmit buffer/frame done, 
mii interrupt and others.

i.MX7D enet has three queues for tx and rx.
It seems netdev pick tx queue 1 rate is very rare by __netdev_pick_tx().
> Anybody else seen this? Any idea?
>
> In 4.10 as well as 4.11-rc6 the interrupt counts were just over 65536...
> pure chance?
>
>
you can use ethtool to set the irq coalesce like:
ethtool -c eth0 rx-frames 80
ethtool -c eth0 rx-usecs 600
ethtool -c eth0 tx-frames 64
ethtool -c eth0 tx-usenc 700


You don't run any test case, just nfs mount rootfs ?
I will setup one imx7d sdb board to run it.

Andy

Re: FEC on i.MX 7 transmit queue timeout

[Stefan Agner]

Hi Andy,

On 2017-04-18 19:24, Andy Duan wrote:
> On 2017年04月19日 03:46, Stefan Agner wrote:
>> Hi,
>>
>> I noticed last week on upstream (v4.11-rc6) on a Colibri iMX7 board that
>> after a while (~10 minutes) the detdev wachdog prints a stacktrace and
>> the driver then continuously dumps the TX ring. I then did a quick test
>> with 4.10, and realized it actually suffers the same issue, so it seems
>> not to be a regression. I use a rootfs mounted over NFS...
>>
>> ------------[ cut here ]------------
>> WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
>> dev_watchdog+0x240/0x244
>> NETDEV WATCHDOG: eth0 (fec): transmit queue 2 timed out
>> Modules linked in:
>> CPU: 0 PID: 0 Comm: swapper/0 Not tainted
>> 4.11.0-rc7-00030-g2c4e6bd0c4f0-dirty #330
>> Hardware name: Freescale i.MX7 Dual (Device Tree)
>> [<c02293f0>] (unwind_backtrace) from [<c0225820>] (show_stack+0x10/0x14)
>> [<c0225820>] (show_stack) from [<c050db6c>] (dump_stack+0x90/0xa0)
>> [<c050db6c>] (dump_stack) from [<c023ae68>] (__warn+0xac/0x11c)
>> [<c023ae68>] (__warn) from [<c023af10>] (warn_slowpath_fmt+0x38/0x48)
>> [<c023af10>] (warn_slowpath_fmt) from [<c088bb8c>]
>> (dev_watchdog+0x240/0x244)
>> [<c088bb8c>] (dev_watchdog) from [<c0294798>]
>> (run_timer_softirq+0x24c/0x708)
>> [<c0294798>] (run_timer_softirq) from [<c023f584>]
>> (__do_softirq+0x12c/0x2a8)
>> [<c023f584>] (__do_softirq) from [<c023f8c4>] (irq_exit+0xdc/0x13c)
>> [<c023f8c4>] (irq_exit) from [<c02818ac>]
>> (__handle_domain_irq+0xa4/0xf8)
>> [<c02818ac>] (__handle_domain_irq) from [<c0201624>]
>> (gic_handle_irq+0x34/0xa4)
>> [<c0201624>] (gic_handle_irq) from [<c0226338>] (__irq_svc+0x58/0x8c)
>> Exception stack(0xc1201f30 to 0xc1201f78)
>> 1f20:                                     c0233320 00000000 00000000
>> 01400000
>> 1f40: c1203d80 ffffe000 00000000 00000000 c107bf10 c0e055b5 c1203d34
>> 00000001
>> 1f60: c07d2324 c1201f80 c0222ac8 c0222acc 60000013 ffffffff
>> [<c0226338>] (__irq_svc) from [<c0222acc>] (arch_cpu_idle+0x38/0x3c)
>> [<c0222acc>] (arch_cpu_idle) from [<c0275f24>] (do_idle+0xa8/0x250)
>> [<c0275f24>] (do_idle) from [<c02760e4>] (cpu_startup_entry+0x18/0x1c)
>> [<c02760e4>] (cpu_startup_entry) from [<c1000aa0>]
>> (start_kernel+0x3fc/0x45c)
>> ---[ end trace 5b0c6dc3466a7918 ]---
>> fec 30be0000.ethernet eth0: TX ring dump
>> Nr     SC     addr       len  SKB
>>    0    0x1c00 0x00000000  590   (null)
>>    1    0x1c00 0x00000000  590   (null)
>>    2    0x1c00 0x00000000   42   (null)
>>    3  H 0x1c00 0x00000000   42   (null)
>>    4 S  0x0000 0x00000000    0   (null)
>>    5    0x0000 0x00000000    0   (null)
>>    6    0x0000 0x00000000    0   (null)
>>    7    0x0000 0x00000000    0   (null)
>>    8    0x0000 0x00000000    0   (null)
>>    9    0x0000 0x00000000    0   (null)
>>   10    0x0000 0x00000000    0   (null)
>>   11    0x0000 0x00000000    0   (null)
>>   12    0x0000 0x00000000    0   (null)
>>   13    0x0000 0x00000000    0   (null)
>>   14    0x0000 0x00000000    0   (null)
>>   15    0x0000 0x00000000    0   (null)
>>   16    0x0000 0x00000000    0   (null)
>>   17    0x0000 0x00000000    0   (null)
>>   18    0x0000 0x00000000    0   (null)
>> ...
>>
>>
>> A second TX ring dump from 4.10:
>> fec 30be0000.ethernet eth0: TX ring dump
>> Nr     SC     addr       len  SKB
>>    0    0x1c00 0x00000000   42   (null)
>>    1    0x1c00 0x00000000   42   (null)
>>    2    0x1c00 0x00000000   90   (null)
>>    3    0x1c00 0x00000000   90   (null)
>>    4    0x1c00 0x00000000   90   (null)
>>    5    0x1c00 0x00000000  218   (null)
>>    6    0x1c00 0x00000000  218   (null)
>>    7    0x1c00 0x00000000  218   (null)
>>    8    0x1c00 0x00000000   90   (null)
>>    9    0x1c00 0x00000000  206   (null)
>>   10    0x1c00 0x00000000  216   (null)
>>   11    0x1c00 0x00000000  216   (null)
>>   12    0x1c00 0x00000000  216   (null)
>>   13    0x1c00 0x00000000  311   (null)
>>   14    0x1c00 0x00000000  178   (null)
>>   15    0x1c00 0x00000000  311   (null)
>>   16    0x1c00 0x00000000  206   (null)
>>   17  H 0x1c00 0x00000000  311   (null)
>>   18 S  0x0000 0x00000000    0   (null)
>>   19    0x0000 0x00000000    0   (null)
> The dump show tx ring is fine.
> 
>>
>> The ring dump prints continously, but I can access console every now and
>> then. I noticed that the second interrupt seems static (66441, TX
>> interrupt?):
>>   58:         18     GIC-0 150 Level     30be0000.ethernet
>>   59:      66441     GIC-0 151 Level     30be0000.ethernet
>>   60:      70477     GIC-0 152 Level     30be0000.ethernet
> 150 irq number is for tx/rx queue 1 receive/transmit buffer/frame done.
> 151 irq number is for tx/rx queue 2 receive/transmit buffer/frame done.
> 152 irq number is for tx/rx queue 0 receive/transmit buffer/frame done, 
> mii interrupt and others.
> 
> i.MX7D enet has three queues for tx and rx.
> It seems netdev pick tx queue 1 rate is very rare by __netdev_pick_tx().

Oh ok I see, and it seems to choose queue 2 fairly often...

>> Anybody else seen this? Any idea?
>>
>> In 4.10 as well as 4.11-rc6 the interrupt counts were just over 65536...
>> pure chance?
>>
>>
> you can use ethtool to set the irq coalesce like:
> ethtool -c eth0 rx-frames 80
> ethtool -c eth0 rx-usecs 600
> ethtool -c eth0 tx-frames 64
> ethtool -c eth0 tx-usenc 700
> 
> 
> You don't run any test case, just nfs mount rootfs ?
> I will setup one imx7d sdb board to run it.

I noticed it without doing anything, just boot via NFS. There was always
a little bit of activity, at least according to the link (blinks every
~5s).

It seemd that it happened a bit earlier when using iperf to exacerbate
the problem...

I noticed that errata 7885 is not mentioned in the i.MX 7 errata, so I
created a new devtype:

        }, {
                .name = "imx7d-fec",
                .driver_data = FEC_QUIRK_ENET_MAC | FEC_QUIRK_HAS_GBIT |
                                FEC_QUIRK_HAS_BUFDESC_EX |
FEC_QUIRK_HAS_CSUM |
                                FEC_QUIRK_HAS_VLAN |
FEC_QUIRK_BUG_CAPTURE |
                                FEC_QUIRK_HAS_RACC |
FEC_QUIRK_HAS_COALESCE,
        }, {

I had that running for about 6h with iperf, it did not seem to happen
despite lots of traffic and interrupts:
 58:   12782877     GIC-0 150 Level     30be0000.ethernet
 59:   14607039     GIC-0 151 Level     30be0000.ethernet
 60:   32356307     GIC-0 152 Level     30be0000.ethernet

But just when I restarted the same stack trace appeared again....

RE: FEC on i.MX 7 transmit queue timeout

[Andy Duan]

From: Stefan Agner <stefan@agner.ch> Sent: Wednesday, April 19, 2017 1:02 PM
>To: Andy Duan <fugang.duan@nxp.com>
>Cc: fugang.duan@freescale.com; festevam@gmail.com;
>netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>Subject: Re: FEC on i.MX 7 transmit queue timeout
>
>Hi Andy,
>
>On 2017-04-18 19:24, Andy Duan wrote:
>> On 2017年04月19日 03:46, Stefan Agner wrote:
>>> Hi,
>>>
>>> I noticed last week on upstream (v4.11-rc6) on a Colibri iMX7 board
>>> that after a while (~10 minutes) the detdev wachdog prints a
>>> stacktrace and the driver then continuously dumps the TX ring. I then
>>> did a quick test with 4.10, and realized it actually suffers the same
>>> issue, so it seems not to be a regression. I use a rootfs mounted over NFS...
>>>
>>> ------------[ cut here ]------------
>>> WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
>>> dev_watchdog+0x240/0x244
>>> NETDEV WATCHDOG: eth0 (fec): transmit queue 2 timed out Modules
>>> linked in:
>>> CPU: 0 PID: 0 Comm: swapper/0 Not tainted
>>> 4.11.0-rc7-00030-g2c4e6bd0c4f0-dirty #330 Hardware name: Freescale
>>> i.MX7 Dual (Device Tree) [<c02293f0>] (unwind_backtrace) from
>>> [<c0225820>] (show_stack+0x10/0x14) [<c0225820>] (show_stack) from
>>> [<c050db6c>] (dump_stack+0x90/0xa0) [<c050db6c>] (dump_stack) from
>>> [<c023ae68>] (__warn+0xac/0x11c) [<c023ae68>] (__warn) from
>>> [<c023af10>] (warn_slowpath_fmt+0x38/0x48) [<c023af10>]
>>> (warn_slowpath_fmt) from [<c088bb8c>]
>>> (dev_watchdog+0x240/0x244)
>>> [<c088bb8c>] (dev_watchdog) from [<c0294798>]
>>> (run_timer_softirq+0x24c/0x708)
>>> [<c0294798>] (run_timer_softirq) from [<c023f584>]
>>> (__do_softirq+0x12c/0x2a8)
>>> [<c023f584>] (__do_softirq) from [<c023f8c4>] (irq_exit+0xdc/0x13c)
>>> [<c023f8c4>] (irq_exit) from [<c02818ac>]
>>> (__handle_domain_irq+0xa4/0xf8)
>>> [<c02818ac>] (__handle_domain_irq) from [<c0201624>]
>>> (gic_handle_irq+0x34/0xa4)
>>> [<c0201624>] (gic_handle_irq) from [<c0226338>] (__irq_svc+0x58/0x8c)
>>> Exception stack(0xc1201f30 to 0xc1201f78)
>>> 1f20:                                     c0233320 00000000 00000000
>>> 01400000
>>> 1f40: c1203d80 ffffe000 00000000 00000000 c107bf10 c0e055b5 c1203d34
>>> 00000001
>>> 1f60: c07d2324 c1201f80 c0222ac8 c0222acc 60000013 ffffffff
>>> [<c0226338>] (__irq_svc) from [<c0222acc>] (arch_cpu_idle+0x38/0x3c)
>>> [<c0222acc>] (arch_cpu_idle) from [<c0275f24>] (do_idle+0xa8/0x250)
>>> [<c0275f24>] (do_idle) from [<c02760e4>]
>>> (cpu_startup_entry+0x18/0x1c) [<c02760e4>] (cpu_startup_entry) from
>>> [<c1000aa0>]
>>> (start_kernel+0x3fc/0x45c)
>>> ---[ end trace 5b0c6dc3466a7918 ]---
>>> fec 30be0000.ethernet eth0: TX ring dump
>>> Nr     SC     addr       len  SKB
>>>    0    0x1c00 0x00000000  590   (null)
>>>    1    0x1c00 0x00000000  590   (null)
>>>    2    0x1c00 0x00000000   42   (null)
>>>    3  H 0x1c00 0x00000000   42   (null)
>>>    4 S  0x0000 0x00000000    0   (null)
>>>    5    0x0000 0x00000000    0   (null)
>>>    6    0x0000 0x00000000    0   (null)
>>>    7    0x0000 0x00000000    0   (null)
>>>    8    0x0000 0x00000000    0   (null)
>>>    9    0x0000 0x00000000    0   (null)
>>>   10    0x0000 0x00000000    0   (null)
>>>   11    0x0000 0x00000000    0   (null)
>>>   12    0x0000 0x00000000    0   (null)
>>>   13    0x0000 0x00000000    0   (null)
>>>   14    0x0000 0x00000000    0   (null)
>>>   15    0x0000 0x00000000    0   (null)
>>>   16    0x0000 0x00000000    0   (null)
>>>   17    0x0000 0x00000000    0   (null)
>>>   18    0x0000 0x00000000    0   (null)
>>> ...
>>>
>>>
>>> A second TX ring dump from 4.10:
>>> fec 30be0000.ethernet eth0: TX ring dump
>>> Nr     SC     addr       len  SKB
>>>    0    0x1c00 0x00000000   42   (null)
>>>    1    0x1c00 0x00000000   42   (null)
>>>    2    0x1c00 0x00000000   90   (null)
>>>    3    0x1c00 0x00000000   90   (null)
>>>    4    0x1c00 0x00000000   90   (null)
>>>    5    0x1c00 0x00000000  218   (null)
>>>    6    0x1c00 0x00000000  218   (null)
>>>    7    0x1c00 0x00000000  218   (null)
>>>    8    0x1c00 0x00000000   90   (null)
>>>    9    0x1c00 0x00000000  206   (null)
>>>   10    0x1c00 0x00000000  216   (null)
>>>   11    0x1c00 0x00000000  216   (null)
>>>   12    0x1c00 0x00000000  216   (null)
>>>   13    0x1c00 0x00000000  311   (null)
>>>   14    0x1c00 0x00000000  178   (null)
>>>   15    0x1c00 0x00000000  311   (null)
>>>   16    0x1c00 0x00000000  206   (null)
>>>   17  H 0x1c00 0x00000000  311   (null)
>>>   18 S  0x0000 0x00000000    0   (null)
>>>   19    0x0000 0x00000000    0   (null)
>> The dump show tx ring is fine.
>>
>>>
>>> The ring dump prints continously, but I can access console every now
>>> and then. I noticed that the second interrupt seems static (66441, TX
>>> interrupt?):
>>>   58:         18     GIC-0 150 Level     30be0000.ethernet
>>>   59:      66441     GIC-0 151 Level     30be0000.ethernet
>>>   60:      70477     GIC-0 152 Level     30be0000.ethernet
>> 150 irq number is for tx/rx queue 1 receive/transmit buffer/frame done.
>> 151 irq number is for tx/rx queue 2 receive/transmit buffer/frame done.
>> 152 irq number is for tx/rx queue 0 receive/transmit buffer/frame
>> done, mii interrupt and others.
>>
>> i.MX7D enet has three queues for tx and rx.
>> It seems netdev pick tx queue 1 rate is very rare by __netdev_pick_tx().
>
>Oh ok I see, and it seems to choose queue 2 fairly often...
>
>>> Anybody else seen this? Any idea?
>>>
>>> In 4.10 as well as 4.11-rc6 the interrupt counts were just over 65536...
>>> pure chance?
>>>
>>>
>> you can use ethtool to set the irq coalesce like:
>> ethtool -c eth0 rx-frames 80
>> ethtool -c eth0 rx-usecs 600
>> ethtool -c eth0 tx-frames 64
>> ethtool -c eth0 tx-usenc 700
>>
>>
>> You don't run any test case, just nfs mount rootfs ?
>> I will setup one imx7d sdb board to run it.
>
>I noticed it without doing anything, just boot via NFS. There was always a little
>bit of activity, at least according to the link (blinks every ~5s).
>
>It seemd that it happened a bit earlier when using iperf to exacerbate the
>problem...
>
>I noticed that errata 7885 is not mentioned in the i.MX 7 errata, so I created a
>new devtype:
>
>        }, {
>                .name = "imx7d-fec",
>                .driver_data = FEC_QUIRK_ENET_MAC | FEC_QUIRK_HAS_GBIT |
>                                FEC_QUIRK_HAS_BUFDESC_EX | FEC_QUIRK_HAS_CSUM |
>                                FEC_QUIRK_HAS_VLAN | FEC_QUIRK_BUG_CAPTURE |
>                                FEC_QUIRK_HAS_RACC | FEC_QUIRK_HAS_COALESCE,
>        }, {
>

Upstreaming driver doesn't have the platform_device_id for "imx7d-fec", imx7d enet still use imx6sx-fec device id driver.
It lost FEC_QUIRK_ERR007885 and FEC_QUIRK_HAS_AVB quirk flags.

You can add these.
I validate imx7d sdb board with 4.11.0-rc6, no such problem after nfs mount more than 3.5 hours.

>I had that running for about 6h with iperf, it did not seem to happen despite
>lots of traffic and interrupts:
> 58:   12782877     GIC-0 150 Level     30be0000.ethernet
> 59:   14607039     GIC-0 151 Level     30be0000.ethernet
> 60:   32356307     GIC-0 152 Level     30be0000.ethernet
>
>But just when I restarted the same stack trace appeared again....
>
>--
>Stefan

RE: FEC on i.MX 7 transmit queue timeout

[Stefan Agner]

On 2017-04-18 22:28, Andy Duan wrote:
> From: Stefan Agner <stefan@agner.ch> Sent: Wednesday, April 19, 2017 1:02 PM
>>To: Andy Duan <fugang.duan@nxp.com>
>>Cc: fugang.duan@freescale.com; festevam@gmail.com;
>>netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>>Subject: Re: FEC on i.MX 7 transmit queue timeout
>>
>>Hi Andy,
>>
>>On 2017-04-18 19:24, Andy Duan wrote:
>>> On 2017年04月19日 03:46, Stefan Agner wrote:
>>>> Hi,
>>>>
>>>> I noticed last week on upstream (v4.11-rc6) on a Colibri iMX7 board
>>>> that after a while (~10 minutes) the detdev wachdog prints a
>>>> stacktrace and the driver then continuously dumps the TX ring. I then
>>>> did a quick test with 4.10, and realized it actually suffers the same
>>>> issue, so it seems not to be a regression. I use a rootfs mounted over NFS...
>>>>
>>>> ------------[ cut here ]------------
>>>> WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
>>>> dev_watchdog+0x240/0x244
>>>> NETDEV WATCHDOG: eth0 (fec): transmit queue 2 timed out Modules
>>>> linked in:
>>>> CPU: 0 PID: 0 Comm: swapper/0 Not tainted
>>>> 4.11.0-rc7-00030-g2c4e6bd0c4f0-dirty #330 Hardware name: Freescale
>>>> i.MX7 Dual (Device Tree) [<c02293f0>] (unwind_backtrace) from
>>>> [<c0225820>] (show_stack+0x10/0x14) [<c0225820>] (show_stack) from
>>>> [<c050db6c>] (dump_stack+0x90/0xa0) [<c050db6c>] (dump_stack) from
>>>> [<c023ae68>] (__warn+0xac/0x11c) [<c023ae68>] (__warn) from
>>>> [<c023af10>] (warn_slowpath_fmt+0x38/0x48) [<c023af10>]
>>>> (warn_slowpath_fmt) from [<c088bb8c>]
>>>> (dev_watchdog+0x240/0x244)
>>>> [<c088bb8c>] (dev_watchdog) from [<c0294798>]
>>>> (run_timer_softirq+0x24c/0x708)
>>>> [<c0294798>] (run_timer_softirq) from [<c023f584>]
>>>> (__do_softirq+0x12c/0x2a8)
>>>> [<c023f584>] (__do_softirq) from [<c023f8c4>] (irq_exit+0xdc/0x13c)
>>>> [<c023f8c4>] (irq_exit) from [<c02818ac>]
>>>> (__handle_domain_irq+0xa4/0xf8)
>>>> [<c02818ac>] (__handle_domain_irq) from [<c0201624>]
>>>> (gic_handle_irq+0x34/0xa4)
>>>> [<c0201624>] (gic_handle_irq) from [<c0226338>] (__irq_svc+0x58/0x8c)
>>>> Exception stack(0xc1201f30 to 0xc1201f78)
>>>> 1f20:                                     c0233320 00000000 00000000
>>>> 01400000
>>>> 1f40: c1203d80 ffffe000 00000000 00000000 c107bf10 c0e055b5 c1203d34
>>>> 00000001
>>>> 1f60: c07d2324 c1201f80 c0222ac8 c0222acc 60000013 ffffffff
>>>> [<c0226338>] (__irq_svc) from [<c0222acc>] (arch_cpu_idle+0x38/0x3c)
>>>> [<c0222acc>] (arch_cpu_idle) from [<c0275f24>] (do_idle+0xa8/0x250)
>>>> [<c0275f24>] (do_idle) from [<c02760e4>]
>>>> (cpu_startup_entry+0x18/0x1c) [<c02760e4>] (cpu_startup_entry) from
>>>> [<c1000aa0>]
>>>> (start_kernel+0x3fc/0x45c)
>>>> ---[ end trace 5b0c6dc3466a7918 ]---
>>>> fec 30be0000.ethernet eth0: TX ring dump
>>>> Nr     SC     addr       len  SKB
>>>>    0    0x1c00 0x00000000  590   (null)
>>>>    1    0x1c00 0x00000000  590   (null)
>>>>    2    0x1c00 0x00000000   42   (null)
>>>>    3  H 0x1c00 0x00000000   42   (null)
>>>>    4 S  0x0000 0x00000000    0   (null)
>>>>    5    0x0000 0x00000000    0   (null)
>>>>    6    0x0000 0x00000000    0   (null)
>>>>    7    0x0000 0x00000000    0   (null)
>>>>    8    0x0000 0x00000000    0   (null)
>>>>    9    0x0000 0x00000000    0   (null)
>>>>   10    0x0000 0x00000000    0   (null)
>>>>   11    0x0000 0x00000000    0   (null)
>>>>   12    0x0000 0x00000000    0   (null)
>>>>   13    0x0000 0x00000000    0   (null)
>>>>   14    0x0000 0x00000000    0   (null)
>>>>   15    0x0000 0x00000000    0   (null)
>>>>   16    0x0000 0x00000000    0   (null)
>>>>   17    0x0000 0x00000000    0   (null)
>>>>   18    0x0000 0x00000000    0   (null)
>>>> ...
>>>>
>>>>
>>>> A second TX ring dump from 4.10:
>>>> fec 30be0000.ethernet eth0: TX ring dump
>>>> Nr     SC     addr       len  SKB
>>>>    0    0x1c00 0x00000000   42   (null)
>>>>    1    0x1c00 0x00000000   42   (null)
>>>>    2    0x1c00 0x00000000   90   (null)
>>>>    3    0x1c00 0x00000000   90   (null)
>>>>    4    0x1c00 0x00000000   90   (null)
>>>>    5    0x1c00 0x00000000  218   (null)
>>>>    6    0x1c00 0x00000000  218   (null)
>>>>    7    0x1c00 0x00000000  218   (null)
>>>>    8    0x1c00 0x00000000   90   (null)
>>>>    9    0x1c00 0x00000000  206   (null)
>>>>   10    0x1c00 0x00000000  216   (null)
>>>>   11    0x1c00 0x00000000  216   (null)
>>>>   12    0x1c00 0x00000000  216   (null)
>>>>   13    0x1c00 0x00000000  311   (null)
>>>>   14    0x1c00 0x00000000  178   (null)
>>>>   15    0x1c00 0x00000000  311   (null)
>>>>   16    0x1c00 0x00000000  206   (null)
>>>>   17  H 0x1c00 0x00000000  311   (null)
>>>>   18 S  0x0000 0x00000000    0   (null)
>>>>   19    0x0000 0x00000000    0   (null)
>>> The dump show tx ring is fine.
>>>
>>>>
>>>> The ring dump prints continously, but I can access console every now
>>>> and then. I noticed that the second interrupt seems static (66441, TX
>>>> interrupt?):
>>>>   58:         18     GIC-0 150 Level     30be0000.ethernet
>>>>   59:      66441     GIC-0 151 Level     30be0000.ethernet
>>>>   60:      70477     GIC-0 152 Level     30be0000.ethernet
>>> 150 irq number is for tx/rx queue 1 receive/transmit buffer/frame done.
>>> 151 irq number is for tx/rx queue 2 receive/transmit buffer/frame done.
>>> 152 irq number is for tx/rx queue 0 receive/transmit buffer/frame
>>> done, mii interrupt and others.
>>>
>>> i.MX7D enet has three queues for tx and rx.
>>> It seems netdev pick tx queue 1 rate is very rare by __netdev_pick_tx().
>>
>>Oh ok I see, and it seems to choose queue 2 fairly often...
>>
>>>> Anybody else seen this? Any idea?
>>>>
>>>> In 4.10 as well as 4.11-rc6 the interrupt counts were just over 65536...
>>>> pure chance?
>>>>
>>>>
>>> you can use ethtool to set the irq coalesce like:
>>> ethtool -c eth0 rx-frames 80
>>> ethtool -c eth0 rx-usecs 600
>>> ethtool -c eth0 tx-frames 64
>>> ethtool -c eth0 tx-usenc 700
>>>
>>>
>>> You don't run any test case, just nfs mount rootfs ?
>>> I will setup one imx7d sdb board to run it.
>>
>>I noticed it without doing anything, just boot via NFS. There was always a little
>>bit of activity, at least according to the link (blinks every ~5s).
>>
>>It seemd that it happened a bit earlier when using iperf to exacerbate the
>>problem...
>>
>>I noticed that errata 7885 is not mentioned in the i.MX 7 errata, so I created a
>>new devtype:
>>
>>        }, {
>>                .name = "imx7d-fec",
>>                .driver_data = FEC_QUIRK_ENET_MAC | FEC_QUIRK_HAS_GBIT |
>>                                FEC_QUIRK_HAS_BUFDESC_EX | FEC_QUIRK_HAS_CSUM |
>>                                FEC_QUIRK_HAS_VLAN | FEC_QUIRK_BUG_CAPTURE |
>>                                FEC_QUIRK_HAS_RACC | FEC_QUIRK_HAS_COALESCE,
>>        }, {
>>
> 
> Upstreaming driver doesn't have the platform_device_id for
> "imx7d-fec", imx7d enet still use imx6sx-fec device id driver.
> It lost FEC_QUIRK_ERR007885 and FEC_QUIRK_HAS_AVB quirk flags.

Also downstream uses imx6sx-fec, at least 4.1.15 GA 2.0.0 release:
http://git.freescale.com/git/cgit.cgi/imx/linux-imx.git/tree/arch/arm/boot/dts/imx7d.dtsi?h=imx_4.1.15_2.0.0_ga#n1380

However, with downstream Linux 4.1 the kernel seems to only use queue 0:
292:          0     GPCV2 118 Edge      30be0000.ethernet
293:          0     GPCV2 119 Edge      30be0000.ethernet
294:     204929     GPCV2 120 Edge      30be0000.ethernet


> 
> You can add these.

I guess if i.MX 7 does not suffer ERR007885 it would be good to add a
new devtype, correct? This also needs a device tree change, since
imx6sx-fec is still in the compatible list... I saw that you sent a
patch to add ERR007885 for imx6ul as well ("net: fec: add ERR007885 for
i.MX6ul enet IP").

My earlier run which showed the stack trace again actually still had
imx6sx-fec in the device tree compatible string, and hence used
ERR007885! So I need to test again...


> I validate imx7d sdb board with 4.11.0-rc6, no such problem after nfs
> mount more than 3.5 hours.
> 

Hm, the Colibri iMX7 uses a different PHY and only supports fast
ethernet. Also, I do tests on a i.MX 7Solo actually, but I can do test
on a i.MX 7Dual tomorrow. But again, with downstream which only uses
queue 0 the issue did never appear.

Re: FEC on i.MX 7 transmit queue timeout

[Andy Duan]

On 2017年04月19日 13:56, Stefan Agner wrote:
> On 2017-04-18 22:28, Andy Duan wrote:
>> From: Stefan Agner <stefan@agner.ch> Sent: Wednesday, April 19, 2017 1:02 PM
>>> To: Andy Duan <fugang.duan@nxp.com>
>>> Cc: fugang.duan@freescale.com; festevam@gmail.com;
>>> netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>>> Subject: Re: FEC on i.MX 7 transmit queue timeout
>>>
>>> Hi Andy,
>>>
>>> On 2017-04-18 19:24, Andy Duan wrote:
>>>> On 2017年04月19日 03:46, Stefan Agner wrote:
>>>>> Hi,
>>>>>
>>>>> I noticed last week on upstream (v4.11-rc6) on a Colibri iMX7 board
>>>>> that after a while (~10 minutes) the detdev wachdog prints a
>>>>> stacktrace and the driver then continuously dumps the TX ring. I then
>>>>> did a quick test with 4.10, and realized it actually suffers the same
>>>>> issue, so it seems not to be a regression. I use a rootfs mounted over NFS...
>>>>>
>>>>> ------------[ cut here ]------------
>>>>> WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
>>>>> dev_watchdog+0x240/0x244
>>>>> NETDEV WATCHDOG: eth0 (fec): transmit queue 2 timed out Modules
>>>>> linked in:
>>>>> CPU: 0 PID: 0 Comm: swapper/0 Not tainted
>>>>> 4.11.0-rc7-00030-g2c4e6bd0c4f0-dirty #330 Hardware name: Freescale
>>>>> i.MX7 Dual (Device Tree) [<c02293f0>] (unwind_backtrace) from
>>>>> [<c0225820>] (show_stack+0x10/0x14) [<c0225820>] (show_stack) from
>>>>> [<c050db6c>] (dump_stack+0x90/0xa0) [<c050db6c>] (dump_stack) from
>>>>> [<c023ae68>] (__warn+0xac/0x11c) [<c023ae68>] (__warn) from
>>>>> [<c023af10>] (warn_slowpath_fmt+0x38/0x48) [<c023af10>]
>>>>> (warn_slowpath_fmt) from [<c088bb8c>]
>>>>> (dev_watchdog+0x240/0x244)
>>>>> [<c088bb8c>] (dev_watchdog) from [<c0294798>]
>>>>> (run_timer_softirq+0x24c/0x708)
>>>>> [<c0294798>] (run_timer_softirq) from [<c023f584>]
>>>>> (__do_softirq+0x12c/0x2a8)
>>>>> [<c023f584>] (__do_softirq) from [<c023f8c4>] (irq_exit+0xdc/0x13c)
>>>>> [<c023f8c4>] (irq_exit) from [<c02818ac>]
>>>>> (__handle_domain_irq+0xa4/0xf8)
>>>>> [<c02818ac>] (__handle_domain_irq) from [<c0201624>]
>>>>> (gic_handle_irq+0x34/0xa4)
>>>>> [<c0201624>] (gic_handle_irq) from [<c0226338>] (__irq_svc+0x58/0x8c)
>>>>> Exception stack(0xc1201f30 to 0xc1201f78)
>>>>> 1f20:                                     c0233320 00000000 00000000
>>>>> 01400000
>>>>> 1f40: c1203d80 ffffe000 00000000 00000000 c107bf10 c0e055b5 c1203d34
>>>>> 00000001
>>>>> 1f60: c07d2324 c1201f80 c0222ac8 c0222acc 60000013 ffffffff
>>>>> [<c0226338>] (__irq_svc) from [<c0222acc>] (arch_cpu_idle+0x38/0x3c)
>>>>> [<c0222acc>] (arch_cpu_idle) from [<c0275f24>] (do_idle+0xa8/0x250)
>>>>> [<c0275f24>] (do_idle) from [<c02760e4>]
>>>>> (cpu_startup_entry+0x18/0x1c) [<c02760e4>] (cpu_startup_entry) from
>>>>> [<c1000aa0>]
>>>>> (start_kernel+0x3fc/0x45c)
>>>>> ---[ end trace 5b0c6dc3466a7918 ]---
>>>>> fec 30be0000.ethernet eth0: TX ring dump
>>>>> Nr     SC     addr       len  SKB
>>>>>     0    0x1c00 0x00000000  590   (null)
>>>>>     1    0x1c00 0x00000000  590   (null)
>>>>>     2    0x1c00 0x00000000   42   (null)
>>>>>     3  H 0x1c00 0x00000000   42   (null)
>>>>>     4 S  0x0000 0x00000000    0   (null)
>>>>>     5    0x0000 0x00000000    0   (null)
>>>>>     6    0x0000 0x00000000    0   (null)
>>>>>     7    0x0000 0x00000000    0   (null)
>>>>>     8    0x0000 0x00000000    0   (null)
>>>>>     9    0x0000 0x00000000    0   (null)
>>>>>    10    0x0000 0x00000000    0   (null)
>>>>>    11    0x0000 0x00000000    0   (null)
>>>>>    12    0x0000 0x00000000    0   (null)
>>>>>    13    0x0000 0x00000000    0   (null)
>>>>>    14    0x0000 0x00000000    0   (null)
>>>>>    15    0x0000 0x00000000    0   (null)
>>>>>    16    0x0000 0x00000000    0   (null)
>>>>>    17    0x0000 0x00000000    0   (null)
>>>>>    18    0x0000 0x00000000    0   (null)
>>>>> ...
>>>>>
>>>>>
>>>>> A second TX ring dump from 4.10:
>>>>> fec 30be0000.ethernet eth0: TX ring dump
>>>>> Nr     SC     addr       len  SKB
>>>>>     0    0x1c00 0x00000000   42   (null)
>>>>>     1    0x1c00 0x00000000   42   (null)
>>>>>     2    0x1c00 0x00000000   90   (null)
>>>>>     3    0x1c00 0x00000000   90   (null)
>>>>>     4    0x1c00 0x00000000   90   (null)
>>>>>     5    0x1c00 0x00000000  218   (null)
>>>>>     6    0x1c00 0x00000000  218   (null)
>>>>>     7    0x1c00 0x00000000  218   (null)
>>>>>     8    0x1c00 0x00000000   90   (null)
>>>>>     9    0x1c00 0x00000000  206   (null)
>>>>>    10    0x1c00 0x00000000  216   (null)
>>>>>    11    0x1c00 0x00000000  216   (null)
>>>>>    12    0x1c00 0x00000000  216   (null)
>>>>>    13    0x1c00 0x00000000  311   (null)
>>>>>    14    0x1c00 0x00000000  178   (null)
>>>>>    15    0x1c00 0x00000000  311   (null)
>>>>>    16    0x1c00 0x00000000  206   (null)
>>>>>    17  H 0x1c00 0x00000000  311   (null)
>>>>>    18 S  0x0000 0x00000000    0   (null)
>>>>>    19    0x0000 0x00000000    0   (null)
>>>> The dump show tx ring is fine.
>>>>
>>>>> The ring dump prints continously, but I can access console every now
>>>>> and then. I noticed that the second interrupt seems static (66441, TX
>>>>> interrupt?):
>>>>>    58:         18     GIC-0 150 Level     30be0000.ethernet
>>>>>    59:      66441     GIC-0 151 Level     30be0000.ethernet
>>>>>    60:      70477     GIC-0 152 Level     30be0000.ethernet
>>>> 150 irq number is for tx/rx queue 1 receive/transmit buffer/frame done.
>>>> 151 irq number is for tx/rx queue 2 receive/transmit buffer/frame done.
>>>> 152 irq number is for tx/rx queue 0 receive/transmit buffer/frame
>>>> done, mii interrupt and others.
>>>>
>>>> i.MX7D enet has three queues for tx and rx.
>>>> It seems netdev pick tx queue 1 rate is very rare by __netdev_pick_tx().
>>> Oh ok I see, and it seems to choose queue 2 fairly often...
>>>
>>>>> Anybody else seen this? Any idea?
>>>>>
>>>>> In 4.10 as well as 4.11-rc6 the interrupt counts were just over 65536...
>>>>> pure chance?
>>>>>
>>>>>
>>>> you can use ethtool to set the irq coalesce like:
>>>> ethtool -c eth0 rx-frames 80
>>>> ethtool -c eth0 rx-usecs 600
>>>> ethtool -c eth0 tx-frames 64
>>>> ethtool -c eth0 tx-usenc 700
>>>>
>>>>
>>>> You don't run any test case, just nfs mount rootfs ?
>>>> I will setup one imx7d sdb board to run it.
>>> I noticed it without doing anything, just boot via NFS. There was always a little
>>> bit of activity, at least according to the link (blinks every ~5s).
>>>
>>> It seemd that it happened a bit earlier when using iperf to exacerbate the
>>> problem...
>>>
>>> I noticed that errata 7885 is not mentioned in the i.MX 7 errata, so I created a
>>> new devtype:
>>>
>>>         }, {
>>>                 .name = "imx7d-fec",
This is added by you, we never added the platform_device_id.

>>>                 .driver_data = FEC_QUIRK_ENET_MAC | FEC_QUIRK_HAS_GBIT |
>>>                                 FEC_QUIRK_HAS_BUFDESC_EX | FEC_QUIRK_HAS_CSUM |
>>>                                 FEC_QUIRK_HAS_VLAN | FEC_QUIRK_BUG_CAPTURE |
>>>                                 FEC_QUIRK_HAS_RACC | FEC_QUIRK_HAS_COALESCE,
>>>         }, {
>>>
>> Upstreaming driver doesn't have the platform_device_id for
>> "imx7d-fec", imx7d enet still use imx6sx-fec device id driver.
>> It lost FEC_QUIRK_ERR007885 and FEC_QUIRK_HAS_AVB quirk flags.
> Also downstream uses imx6sx-fec, at least 4.1.15 GA 2.0.0 release:
> http://git.freescale.com/git/cgit.cgi/imx/linux-imx.git/tree/arch/arm/boot/dts/imx7d.dtsi?h=imx_4.1.15_2.0.0_ga#n1380
>
> However, with downstream Linux 4.1 the kernel seems to only use queue 0:
> 292:          0     GPCV2 118 Edge      30be0000.ethernet
> 293:          0     GPCV2 119 Edge      30be0000.ethernet
> 294:     204929     GPCV2 120 Edge      30be0000.ethernet
>
yes, queue 0 is for best effort, queue 1 and 2 are for audio/video.

>> You can add these.
> I guess if i.MX 7 does not suffer ERR007885 it would be good to add a
> new devtype, correct? This also needs a device tree change, since
> imx6sx-fec is still in the compatible list... I saw that you sent a
> patch to add ERR007885 for imx6ul as well ("net: fec: add ERR007885 for
> i.MX6ul enet IP").
ERR007885 just to add some cycles before set TDAR that don't take side 
effort.
I will confirm the hw issue is fixed or not.

> My earlier run which showed the stack trace again actually still had
> imx6sx-fec in the device tree compatible string, and hence used
> ERR007885! So I need to test again...
>
pls use compatible string "imx6sx-fec" and test again.

>> I validate imx7d sdb board with 4.11.0-rc6, no such problem after nfs
>> mount more than 3.5 hours
> Hm, the Colibri iMX7 uses a different PHY and only supports fast
> ethernet. Also, I do tests on a i.MX 7Solo actually, but I can do test
> on a i.MX 7Dual tomorrow. But again, with downstream which only uses
> queue 0 the issue did never appear.
>
> --
no, my imx7d sdb board running upstreaming kernel 4.11.0-rc6 with three 
queues.
So far so good (about 6.5 hours).

Re: FEC on i.MX 7 transmit queue timeout

[Stefan Agner]

On 2017-04-19 01:45, Andy Duan wrote:
> On 2017年04月19日 13:56, Stefan Agner wrote:
>> On 2017-04-18 22:28, Andy Duan wrote:
>>> From: Stefan Agner <stefan@agner.ch> Sent: Wednesday, April 19, 2017 1:02 PM
>>>> To: Andy Duan <fugang.duan@nxp.com>
>>>> Cc: fugang.duan@freescale.com; festevam@gmail.com;
>>>> netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>>>> Subject: Re: FEC on i.MX 7 transmit queue timeout
>>>>
>>>> Hi Andy,
>>>>
>>>> On 2017-04-18 19:24, Andy Duan wrote:
>>>>> On 2017年04月19日 03:46, Stefan Agner wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I noticed last week on upstream (v4.11-rc6) on a Colibri iMX7 board
>>>>>> that after a while (~10 minutes) the detdev wachdog prints a
>>>>>> stacktrace and the driver then continuously dumps the TX ring. I then
>>>>>> did a quick test with 4.10, and realized it actually suffers the same
>>>>>> issue, so it seems not to be a regression. I use a rootfs mounted over NFS...
>>>>>>
>>>>>> ------------[ cut here ]------------
>>>>>> WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
>>>>>> dev_watchdog+0x240/0x244
>>>>>> NETDEV WATCHDOG: eth0 (fec): transmit queue 2 timed out Modules
>>>>>> linked in:
>>>>>> CPU: 0 PID: 0 Comm: swapper/0 Not tainted
>>>>>> 4.11.0-rc7-00030-g2c4e6bd0c4f0-dirty #330 Hardware name: Freescale
>>>>>> i.MX7 Dual (Device Tree) [<c02293f0>] (unwind_backtrace) from
>>>>>> [<c0225820>] (show_stack+0x10/0x14) [<c0225820>] (show_stack) from
>>>>>> [<c050db6c>] (dump_stack+0x90/0xa0) [<c050db6c>] (dump_stack) from
>>>>>> [<c023ae68>] (__warn+0xac/0x11c) [<c023ae68>] (__warn) from
>>>>>> [<c023af10>] (warn_slowpath_fmt+0x38/0x48) [<c023af10>]
>>>>>> (warn_slowpath_fmt) from [<c088bb8c>]
>>>>>> (dev_watchdog+0x240/0x244)
>>>>>> [<c088bb8c>] (dev_watchdog) from [<c0294798>]
>>>>>> (run_timer_softirq+0x24c/0x708)
>>>>>> [<c0294798>] (run_timer_softirq) from [<c023f584>]
>>>>>> (__do_softirq+0x12c/0x2a8)
>>>>>> [<c023f584>] (__do_softirq) from [<c023f8c4>] (irq_exit+0xdc/0x13c)
>>>>>> [<c023f8c4>] (irq_exit) from [<c02818ac>]
>>>>>> (__handle_domain_irq+0xa4/0xf8)
>>>>>> [<c02818ac>] (__handle_domain_irq) from [<c0201624>]
>>>>>> (gic_handle_irq+0x34/0xa4)
>>>>>> [<c0201624>] (gic_handle_irq) from [<c0226338>] (__irq_svc+0x58/0x8c)
>>>>>> Exception stack(0xc1201f30 to 0xc1201f78)
>>>>>> 1f20:                                     c0233320 00000000 00000000
>>>>>> 01400000
>>>>>> 1f40: c1203d80 ffffe000 00000000 00000000 c107bf10 c0e055b5 c1203d34
>>>>>> 00000001
>>>>>> 1f60: c07d2324 c1201f80 c0222ac8 c0222acc 60000013 ffffffff
>>>>>> [<c0226338>] (__irq_svc) from [<c0222acc>] (arch_cpu_idle+0x38/0x3c)
>>>>>> [<c0222acc>] (arch_cpu_idle) from [<c0275f24>] (do_idle+0xa8/0x250)
>>>>>> [<c0275f24>] (do_idle) from [<c02760e4>]
>>>>>> (cpu_startup_entry+0x18/0x1c) [<c02760e4>] (cpu_startup_entry) from
>>>>>> [<c1000aa0>]
>>>>>> (start_kernel+0x3fc/0x45c)
>>>>>> ---[ end trace 5b0c6dc3466a7918 ]---
>>>>>> fec 30be0000.ethernet eth0: TX ring dump
>>>>>> Nr     SC     addr       len  SKB
>>>>>>     0    0x1c00 0x00000000  590   (null)
>>>>>>     1    0x1c00 0x00000000  590   (null)
>>>>>>     2    0x1c00 0x00000000   42   (null)
>>>>>>     3  H 0x1c00 0x00000000   42   (null)
>>>>>>     4 S  0x0000 0x00000000    0   (null)
>>>>>>     5    0x0000 0x00000000    0   (null)
>>>>>>     6    0x0000 0x00000000    0   (null)
>>>>>>     7    0x0000 0x00000000    0   (null)
>>>>>>     8    0x0000 0x00000000    0   (null)
>>>>>>     9    0x0000 0x00000000    0   (null)
>>>>>>    10    0x0000 0x00000000    0   (null)
>>>>>>    11    0x0000 0x00000000    0   (null)
>>>>>>    12    0x0000 0x00000000    0   (null)
>>>>>>    13    0x0000 0x00000000    0   (null)
>>>>>>    14    0x0000 0x00000000    0   (null)
>>>>>>    15    0x0000 0x00000000    0   (null)
>>>>>>    16    0x0000 0x00000000    0   (null)
>>>>>>    17    0x0000 0x00000000    0   (null)
>>>>>>    18    0x0000 0x00000000    0   (null)
>>>>>> ...
>>>>>>
>>>>>>
>>>>>> A second TX ring dump from 4.10:
>>>>>> fec 30be0000.ethernet eth0: TX ring dump
>>>>>> Nr     SC     addr       len  SKB
>>>>>>     0    0x1c00 0x00000000   42   (null)
>>>>>>     1    0x1c00 0x00000000   42   (null)
>>>>>>     2    0x1c00 0x00000000   90   (null)
>>>>>>     3    0x1c00 0x00000000   90   (null)
>>>>>>     4    0x1c00 0x00000000   90   (null)
>>>>>>     5    0x1c00 0x00000000  218   (null)
>>>>>>     6    0x1c00 0x00000000  218   (null)
>>>>>>     7    0x1c00 0x00000000  218   (null)
>>>>>>     8    0x1c00 0x00000000   90   (null)
>>>>>>     9    0x1c00 0x00000000  206   (null)
>>>>>>    10    0x1c00 0x00000000  216   (null)
>>>>>>    11    0x1c00 0x00000000  216   (null)
>>>>>>    12    0x1c00 0x00000000  216   (null)
>>>>>>    13    0x1c00 0x00000000  311   (null)
>>>>>>    14    0x1c00 0x00000000  178   (null)
>>>>>>    15    0x1c00 0x00000000  311   (null)
>>>>>>    16    0x1c00 0x00000000  206   (null)
>>>>>>    17  H 0x1c00 0x00000000  311   (null)
>>>>>>    18 S  0x0000 0x00000000    0   (null)
>>>>>>    19    0x0000 0x00000000    0   (null)
>>>>> The dump show tx ring is fine.
>>>>>
>>>>>> The ring dump prints continously, but I can access console every now
>>>>>> and then. I noticed that the second interrupt seems static (66441, TX
>>>>>> interrupt?):
>>>>>>    58:         18     GIC-0 150 Level     30be0000.ethernet
>>>>>>    59:      66441     GIC-0 151 Level     30be0000.ethernet
>>>>>>    60:      70477     GIC-0 152 Level     30be0000.ethernet
>>>>> 150 irq number is for tx/rx queue 1 receive/transmit buffer/frame done.
>>>>> 151 irq number is for tx/rx queue 2 receive/transmit buffer/frame done.
>>>>> 152 irq number is for tx/rx queue 0 receive/transmit buffer/frame
>>>>> done, mii interrupt and others.
>>>>>
>>>>> i.MX7D enet has three queues for tx and rx.
>>>>> It seems netdev pick tx queue 1 rate is very rare by __netdev_pick_tx().
>>>> Oh ok I see, and it seems to choose queue 2 fairly often...
>>>>
>>>>>> Anybody else seen this? Any idea?
>>>>>>
>>>>>> In 4.10 as well as 4.11-rc6 the interrupt counts were just over 65536...
>>>>>> pure chance?
>>>>>>
>>>>>>
>>>>> you can use ethtool to set the irq coalesce like:
>>>>> ethtool -c eth0 rx-frames 80
>>>>> ethtool -c eth0 rx-usecs 600
>>>>> ethtool -c eth0 tx-frames 64
>>>>> ethtool -c eth0 tx-usenc 700
>>>>>
>>>>>
>>>>> You don't run any test case, just nfs mount rootfs ?
>>>>> I will setup one imx7d sdb board to run it.
>>>> I noticed it without doing anything, just boot via NFS. There was always a little
>>>> bit of activity, at least according to the link (blinks every ~5s).
>>>>
>>>> It seemd that it happened a bit earlier when using iperf to exacerbate the
>>>> problem...
>>>>
>>>> I noticed that errata 7885 is not mentioned in the i.MX 7 errata, so I created a
>>>> new devtype:
>>>>
>>>>         }, {
>>>>                 .name = "imx7d-fec",
> This is added by you, we never added the platform_device_id.
> 
>>>>                 .driver_data = FEC_QUIRK_ENET_MAC | FEC_QUIRK_HAS_GBIT |
>>>>                                 FEC_QUIRK_HAS_BUFDESC_EX | FEC_QUIRK_HAS_CSUM |
>>>>                                 FEC_QUIRK_HAS_VLAN | FEC_QUIRK_BUG_CAPTURE |
>>>>                                 FEC_QUIRK_HAS_RACC | FEC_QUIRK_HAS_COALESCE,
>>>>         }, {
>>>>
>>> Upstreaming driver doesn't have the platform_device_id for
>>> "imx7d-fec", imx7d enet still use imx6sx-fec device id driver.
>>> It lost FEC_QUIRK_ERR007885 and FEC_QUIRK_HAS_AVB quirk flags.
>> Also downstream uses imx6sx-fec, at least 4.1.15 GA 2.0.0 release:
>> http://git.freescale.com/git/cgit.cgi/imx/linux-imx.git/tree/arch/arm/boot/dts/imx7d.dtsi?h=imx_4.1.15_2.0.0_ga#n1380
>>
>> However, with downstream Linux 4.1 the kernel seems to only use queue 0:
>> 292:          0     GPCV2 118 Edge      30be0000.ethernet
>> 293:          0     GPCV2 119 Edge      30be0000.ethernet
>> 294:     204929     GPCV2 120 Edge      30be0000.ethernet
>>
> yes, queue 0 is for best effort, queue 1 and 2 are for audio/video.
> 
>>> You can add these.
>> I guess if i.MX 7 does not suffer ERR007885 it would be good to add a
>> new devtype, correct? This also needs a device tree change, since
>> imx6sx-fec is still in the compatible list... I saw that you sent a
>> patch to add ERR007885 for imx6ul as well ("net: fec: add ERR007885 for
>> i.MX6ul enet IP").
> ERR007885 just to add some cycles before set TDAR that don't take side 
> effort.
> I will confirm the hw issue is fixed or not.
> 
>> My earlier run which showed the stack trace again actually still had
>> imx6sx-fec in the device tree compatible string, and hence used
>> ERR007885! So I need to test again...
>>
> pls use compatible string "imx6sx-fec" and test again.
> 

I tested again with imx6sx-fec compatible string. I could reproduce it
on a Colibri with i.MX 7Dual. But not always: It really depends whether
queue 2 is counting up or not. Just after boot, I check /proc/interrupts
twice, if queue 2 is counting it will happen!

But if only queue 0 is mostly in use, then it seems to work just fine.

I also tried i.MX 7Dual SabreSD here, and the same thing. I had to
reboot 3 times, then queue 2 was counting:
 57:          8     GIC-0 150 Level     30be0000.ethernet
 58:      20137     GIC-0 151 Level     30be0000.ethernet
 59:       9269     GIC-0 152 Level     30be0000.ethernet

It took me about 40 minutes on Sabre until it happened, and I had to
force it using iperf, but then I got the ring dumps:

(I do not have the first dump unfortunately)
fec 30be0000.ethernet eth0: TX ring dump
Nr     SC     addr       len  SKB
  0  H 0x1c00 0x00000000   78   (null)
  1 S  0x0000 0x00000000   66   (null)
  2    0x0000 0x00000000   78   (null)
  3    0x0000 0x00000000   78   (null)
  4    0x0000 0x00000000   78   (null)
...



>>> I validate imx7d sdb board with 4.11.0-rc6, no such problem after nfs
>>> mount more than 3.5 hours
>> Hm, the Colibri iMX7 uses a different PHY and only supports fast
>> ethernet. Also, I do tests on a i.MX 7Solo actually, but I can do test
>> on a i.MX 7Dual tomorrow. But again, with downstream which only uses
>> queue 0 the issue did never appear.
>>
>> --
> no, my imx7d sdb board running upstreaming kernel 4.11.0-rc6 with three 
> queues.
> So far so good (about 6.5 hours).

Re: FEC on i.MX 7 transmit queue timeout

[Andy Duan]

On 2017年04月20日 07:15, Stefan Agner wrote:
> I tested again with imx6sx-fec compatible string. I could reproduce it
> on a Colibri with i.MX 7Dual. But not always: It really depends whether
> queue 2 is counting up or not. Just after boot, I check /proc/interrupts
> twice, if queue 2 is counting it will happen!
>
> But if only queue 0 is mostly in use, then it seems to work just fine.
If your case is only running best effort like tcp/udp, you can re-set 
the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts file.
Other two queues are for AVB audio/video queues, they have high priority 
than queue 0. If running iperf tcp test on the three queues, then
the tcp segment may be out-of-order that cause net watchdog timeout.
>
> I also tried i.MX 7Dual SabreSD here, and the same thing. I had to
> reboot 3 times, then queue 2 was counting:
>   57:          8     GIC-0 150 Level     30be0000.ethernet
>   58:      20137     GIC-0 151 Level     30be0000.ethernet
>   59:       9269     GIC-0 152 Level     30be0000.ethernet
>
> It took me about 40 minutes on Sabre until it happened, and I had to
> force it using iperf, but then I got the ring dumps:
My board had ran more than 47 hours with nfs rootfs in 4.11.0-rc6, but 
not running iperf.
I am testing with iperf.

Andy

Re: FEC on i.MX 7 transmit queue timeout

[Stefan Agner]

Hi Andy,

On 2017-04-20 19:48, Andy Duan wrote:
> On 2017年04月20日 07:15, Stefan Agner wrote:
>> I tested again with imx6sx-fec compatible string. I could reproduce it
>> on a Colibri with i.MX 7Dual. But not always: It really depends whether
>> queue 2 is counting up or not. Just after boot, I check /proc/interrupts
>> twice, if queue 2 is counting it will happen!
>>
>> But if only queue 0 is mostly in use, then it seems to work just fine.
> If your case is only running best effort like tcp/udp, you can re-set 
> the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts file.
> Other two queues are for AVB audio/video queues, they have high priority 
> than queue 0. If running iperf tcp test on the three queues, then
> the tcp segment may be out-of-order that cause net watchdog timeout.
>>
>> I also tried i.MX 7Dual SabreSD here, and the same thing. I had to
>> reboot 3 times, then queue 2 was counting:
>>   57:          8     GIC-0 150 Level     30be0000.ethernet
>>   58:      20137     GIC-0 151 Level     30be0000.ethernet
>>   59:       9269     GIC-0 152 Level     30be0000.ethernet
>>
>> It took me about 40 minutes on Sabre until it happened, and I had to
>> force it using iperf, but then I got the ring dumps:
> My board had ran more than 47 hours with nfs rootfs in 4.11.0-rc6, but 
> not running iperf.
> I am testing with iperf.

Any update on this issue?

When using iperf (server) on the board with Linux 4.11 the issue appears
within a few iperf iterations on a Sabre (TO 1.2, Board Rev C, if that
matters)...

root@colibri-imx7:~# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.10.70 port 5001 connected with 192.168.10.1 port
60524
random: crng init done
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.06 GBytes   909 Mbits/sec
[  5] local 192.168.10.70 port 5001 connected with 192.168.10.1 port
60528
[  5]  0.0-10.0 sec  1.07 GBytes   919 Mbits/sec
[  4] local 192.168.10.70 port 5001 connected with 192.168.10.1 port
60562
------------[ cut here ]------------
WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:316
dev_watchdog+0x248/0x24c
NETDEV WATCHDOG: eth0 (fec): transmit queue 1 timed out
Modules linked in:
CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.11.0 #360
Hardware name: Freescale i.MX7 Dual (Device Tree)
[<c0226930>] (unwind_backtrace) from [<c0222ffc>] (show_stack+0x10/0x14)
[<c0222ffc>] (show_stack) from [<c04d4f78>] (dump_stack+0x78/0x8c)
[<c04d4f78>] (dump_stack) from [<c0236b38>] (__warn+0xe8/0x100)
[<c0236b38>] (__warn) from [<c0236b88>] (warn_slowpath_fmt+0x38/0x48)
[<c0236b88>] (warn_slowpath_fmt) from [<c0806154>]
(dev_watchdog+0x248/0x24c)
[<c0806154>] (dev_watchdog) from [<c028a9a0>] (call_timer_fn+0x28/0x98)
[<c028a9a0>] (call_timer_fn) from [<c028aab0>] (expire_timers+0xa0/0xac)
[<c028aab0>] (expire_timers) from [<c028ab58>]
(run_timer_softirq+0x9c/0x194)
[<c028ab58>] (run_timer_softirq) from [<c023b110>]
(__do_softirq+0x114/0x234)
[<c023b110>] (__do_softirq) from [<c023b4fc>] (irq_exit+0xcc/0x108)
[<c023b4fc>] (irq_exit) from [<c027a1a0>]
(__handle_domain_irq+0x80/0xec)
[<c027a1a0>] (__handle_domain_irq) from [<c0201544>]
(gic_handle_irq+0x48/0x8c)
[<c0201544>] (gic_handle_irq) from [<c0223ab8>] (__irq_svc+0x58/0x8c)
Exception stack(0xc1001f28 to 0xc1001f70)
1f20:                   00000001 00000000 00000000 c0230060 c1000000
c1003d80
1f40: c1003d34 c0e72f50 c0bd9a04 c1001f80 00000000 00000000 0000320a
c1001f78
1f60: c022070c c0220710 600e0013 ffffffff
[<c0223ab8>] (__irq_svc) from [<c0220710>] (arch_cpu_idle+0x38/0x3c)
[<c0220710>] (arch_cpu_idle) from [<c026f4e0>] (do_idle+0x170/0x204)
[<c026f4e0>] (do_idle) from [<c026f82c>] (cpu_startup_entry+0x18/0x1c)
[<c026f82c>] (cpu_startup_entry) from [<c0e00c88>]
(start_kernel+0x394/0x3a0)
---[ end trace 86a38600d1b9e2a5 ]---
fec 30be0000.ethernet eth0: TX ring dump
Nr     SC     addr       len  SKB
  0    0x1c00 0x00000000   42   (null)
  1  H 0x1c00 0x00000000   86   (null)

RE: FEC on i.MX 7 transmit queue timeout

[Andy Duan]

From: Stefan Agner <stefan@agner.ch> Sent: Thursday, May 04, 2017 9:22 AM
>To: Andy Duan <fugang.duan@nxp.com>
>Cc: fugang.duan@freescale.com; festevam@gmail.com;
>netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>Subject: Re: FEC on i.MX 7 transmit queue timeout
>
>Hi Andy,
>
>On 2017-04-20 19:48, Andy Duan wrote:
>> On 2017年04月20日 07:15, Stefan Agner wrote:
>>> I tested again with imx6sx-fec compatible string. I could reproduce
>>> it on a Colibri with i.MX 7Dual. But not always: It really depends
>>> whether queue 2 is counting up or not. Just after boot, I check
>>> /proc/interrupts twice, if queue 2 is counting it will happen!
>>>
>>> But if only queue 0 is mostly in use, then it seems to work just fine.
>> If your case is only running best effort like tcp/udp, you can re-set
>> the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts file.
>> Other two queues are for AVB audio/video queues, they have high
>> priority than queue 0. If running iperf tcp test on the three queues,
>> then the tcp segment may be out-of-order that cause net watchdog
>timeout.
>>>
>>> I also tried i.MX 7Dual SabreSD here, and the same thing. I had to
>>> reboot 3 times, then queue 2 was counting:
>>>   57:          8     GIC-0 150 Level     30be0000.ethernet
>>>   58:      20137     GIC-0 151 Level     30be0000.ethernet
>>>   59:       9269     GIC-0 152 Level     30be0000.ethernet
>>>
>>> It took me about 40 minutes on Sabre until it happened, and I had to
>>> force it using iperf, but then I got the ring dumps:
>> My board had ran more than 47 hours with nfs rootfs in 4.11.0-rc6, but
>> not running iperf.
>> I am testing with iperf.
>
>Any update on this issue?
>
>When using iperf (server) on the board with Linux 4.11 the issue appears
>within a few iperf iterations on a Sabre (TO 1.2, Board Rev C, if that matters)...
>
I don’t know whether you received my last mail. (maybe failed due to I received some rejection mails)

If your case is only running best effort like tcp/udp, you can re-set the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts file.
Other two queues are for AVB audio/video queues, they have high priority than queue 0. If running iperf tcp test on the three queues, then the tcp segment may be out-of-order that cause net watchdog timeout.
In fsl kernel tree, there have one patch that only select the queue0 for best effort like tcp/udp. Pls test again in your board, if no problem I will upstream the patch.

Re: FEC on i.MX 7 transmit queue timeout

[Stefan Agner]

On 2017-05-03 20:08, Andy Duan wrote:
> From: Stefan Agner <stefan@agner.ch> Sent: Thursday, May 04, 2017 9:22 AM
>>To: Andy Duan <fugang.duan@nxp.com>
>>Cc: fugang.duan@freescale.com; festevam@gmail.com;
>>netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>>Subject: Re: FEC on i.MX 7 transmit queue timeout
>>
>>Hi Andy,
>>
>>On 2017-04-20 19:48, Andy Duan wrote:
>>> On 2017年04月20日 07:15, Stefan Agner wrote:
>>>> I tested again with imx6sx-fec compatible string. I could reproduce
>>>> it on a Colibri with i.MX 7Dual. But not always: It really depends
>>>> whether queue 2 is counting up or not. Just after boot, I check
>>>> /proc/interrupts twice, if queue 2 is counting it will happen!
>>>>
>>>> But if only queue 0 is mostly in use, then it seems to work just fine.
>>> If your case is only running best effort like tcp/udp, you can re-set
>>> the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts file.
>>> Other two queues are for AVB audio/video queues, they have high
>>> priority than queue 0. If running iperf tcp test on the three queues,
>>> then the tcp segment may be out-of-order that cause net watchdog
>>timeout.
>>>>
>>>> I also tried i.MX 7Dual SabreSD here, and the same thing. I had to
>>>> reboot 3 times, then queue 2 was counting:
>>>>   57:          8     GIC-0 150 Level     30be0000.ethernet
>>>>   58:      20137     GIC-0 151 Level     30be0000.ethernet
>>>>   59:       9269     GIC-0 152 Level     30be0000.ethernet
>>>>
>>>> It took me about 40 minutes on Sabre until it happened, and I had to
>>>> force it using iperf, but then I got the ring dumps:
>>> My board had ran more than 47 hours with nfs rootfs in 4.11.0-rc6, but
>>> not running iperf.
>>> I am testing with iperf.
>>
>>Any update on this issue?
>>
>>When using iperf (server) on the board with Linux 4.11 the issue appears
>>within a few iperf iterations on a Sabre (TO 1.2, Board Rev C, if that matters)...
>>
> I don’t know whether you received my last mail. (maybe failed due to I
> received some rejection mails)

I think I did not... The last email I received was Fri, 21 Apr 2017
02:48:23 UTC.

 
> If your case is only running best effort like tcp/udp, you can re-set
> the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts
> file.

I did test that, and it seems to work fine with those properties set to
1.

> Other two queues are for AVB audio/video queues, they have high
> priority than queue 0. If running iperf tcp test on the three queues,
> then the tcp segment may be out-of-order that cause net watchdog
> timeout.

Okay. A single event would be understandable, but it seems to enter some
kind of loop after that (continuously printing "fec 30be0000.ethernet
eth0: TX ring dump ...").

In a quick test I commented out the fec_dump call, with that it seems to
print only once and continues working afterwards (although, speed starts
to decrease, so something is not good at that point).

> In fsl kernel tree, there have one patch that only select the queue0
> for best effort like tcp/udp. Pls test again in your board, if no
> problem I will upstream the patch.

That sounds like a reasonable fix.

IP, no matter whether TCP/UDP, is the most common use case, so IMHO this
should "just work" by default.

Re: FEC on i.MX 7 transmit queue timeout

[Andy Duan]

On 2017年05月05日 05:36, Stefan Agner wrote:
> On 2017-05-03 20:08, Andy Duan wrote:
>> From: Stefan Agner <stefan@agner.ch> Sent: Thursday, May 04, 2017 9:22 AM
>>> To: Andy Duan <fugang.duan@nxp.com>
>>> Cc: fugang.duan@freescale.com; festevam@gmail.com;
>>> netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>>> Subject: Re: FEC on i.MX 7 transmit queue timeout
>>>
>>> Hi Andy,
>>>
>>> On 2017-04-20 19:48, Andy Duan wrote:
>>>> On 2017年04月20日 07:15, Stefan Agner wrote:
>>>>> I tested again with imx6sx-fec compatible string. I could reproduce
>>>>> it on a Colibri with i.MX 7Dual. But not always: It really depends
>>>>> whether queue 2 is counting up or not. Just after boot, I check
>>>>> /proc/interrupts twice, if queue 2 is counting it will happen!
>>>>>
>>>>> But if only queue 0 is mostly in use, then it seems to work just fine.
>>>> If your case is only running best effort like tcp/udp, you can re-set
>>>> the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts file.
>>>> Other two queues are for AVB audio/video queues, they have high
>>>> priority than queue 0. If running iperf tcp test on the three queues,
>>>> then the tcp segment may be out-of-order that cause net watchdog
>>> timeout.
>>>>> I also tried i.MX 7Dual SabreSD here, and the same thing. I had to
>>>>> reboot 3 times, then queue 2 was counting:
>>>>>    57:          8     GIC-0 150 Level     30be0000.ethernet
>>>>>    58:      20137     GIC-0 151 Level     30be0000.ethernet
>>>>>    59:       9269     GIC-0 152 Level     30be0000.ethernet
>>>>>
>>>>> It took me about 40 minutes on Sabre until it happened, and I had to
>>>>> force it using iperf, but then I got the ring dumps:
>>>> My board had ran more than 47 hours with nfs rootfs in 4.11.0-rc6, but
>>>> not running iperf.
>>>> I am testing with iperf.
>>> Any update on this issue?
>>>
>>> When using iperf (server) on the board with Linux 4.11 the issue appears
>>> within a few iperf iterations on a Sabre (TO 1.2, Board Rev C, if that matters)...
>>>
>> I don’t know whether you received my last mail. (maybe failed due to I
>> received some rejection mails)
> I think I did not... The last email I received was Fri, 21 Apr 2017
> 02:48:23 UTC.
>
>   
>> If your case is only running best effort like tcp/udp, you can re-set
>> the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts
>> file.
> I did test that, and it seems to work fine with those properties set to
> 1.
So it can fix your problem after long time test?
>> Other two queues are for AVB audio/video queues, they have high
>> priority than queue 0. If running iperf tcp test on the three queues,
>> then the tcp segment may be out-of-order that cause net watchdog
>> timeout.
> Okay. A single event would be understandable, but it seems to enter some
> kind of loop after that (continuously printing "fec 30be0000.ethernet
> eth0: TX ring dump ...").
>
> In a quick test I commented out the fec_dump call, with that it seems to
> print only once and continues working afterwards (although, speed starts
> to decrease, so something is not good at that point).
The test base on above change ? One queue still bring watchdog timeout ?
>> In fsl kernel tree, there have one patch that only select the queue0
>> for best effort like tcp/udp. Pls test again in your board, if no
>> problem I will upstream the patch.
> That sounds like a reasonable fix.
>
> IP, no matter whether TCP/UDP, is the most common use case, so IMHO this
> should "just work" by default.
>
> --
> Stefan

Re: FEC on i.MX 7 transmit queue timeout

[Stefan Agner]

On 2017-05-04 19:03, Andy Duan wrote:
> On 2017年05月05日 05:36, Stefan Agner wrote:
>> On 2017-05-03 20:08, Andy Duan wrote:
>>> From: Stefan Agner <stefan@agner.ch> Sent: Thursday, May 04, 2017 9:22 AM
>>>> To: Andy Duan <fugang.duan@nxp.com>
>>>> Cc: fugang.duan@freescale.com; festevam@gmail.com;
>>>> netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>>>> Subject: Re: FEC on i.MX 7 transmit queue timeout
>>>>
>>>> Hi Andy,
>>>>
>>>> On 2017-04-20 19:48, Andy Duan wrote:
>>>>> On 2017年04月20日 07:15, Stefan Agner wrote:
>>>>>> I tested again with imx6sx-fec compatible string. I could reproduce
>>>>>> it on a Colibri with i.MX 7Dual. But not always: It really depends
>>>>>> whether queue 2 is counting up or not. Just after boot, I check
>>>>>> /proc/interrupts twice, if queue 2 is counting it will happen!
>>>>>>
>>>>>> But if only queue 0 is mostly in use, then it seems to work just fine.
>>>>> If your case is only running best effort like tcp/udp, you can re-set
>>>>> the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts file.
>>>>> Other two queues are for AVB audio/video queues, they have high
>>>>> priority than queue 0. If running iperf tcp test on the three queues,
>>>>> then the tcp segment may be out-of-order that cause net watchdog
>>>> timeout.
>>>>>> I also tried i.MX 7Dual SabreSD here, and the same thing. I had to
>>>>>> reboot 3 times, then queue 2 was counting:
>>>>>>    57:          8     GIC-0 150 Level     30be0000.ethernet
>>>>>>    58:      20137     GIC-0 151 Level     30be0000.ethernet
>>>>>>    59:       9269     GIC-0 152 Level     30be0000.ethernet
>>>>>>
>>>>>> It took me about 40 minutes on Sabre until it happened, and I had to
>>>>>> force it using iperf, but then I got the ring dumps:
>>>>> My board had ran more than 47 hours with nfs rootfs in 4.11.0-rc6, but
>>>>> not running iperf.
>>>>> I am testing with iperf.
>>>> Any update on this issue?
>>>>
>>>> When using iperf (server) on the board with Linux 4.11 the issue appears
>>>> within a few iperf iterations on a Sabre (TO 1.2, Board Rev C, if that matters)...
>>>>
>>> I don’t know whether you received my last mail. (maybe failed due to I
>>> received some rejection mails)
>> I think I did not... The last email I received was Fri, 21 Apr 2017
>> 02:48:23 UTC.
>>
>>
>>> If your case is only running best effort like tcp/udp, you can re-set
>>> the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts
>>> file.
>> I did test that, and it seems to work fine with those properties set to
>> 1.
> So it can fix your problem after long time test?

Yes, seems to work fine after more than 2 hours.

>>> Other two queues are for AVB audio/video queues, they have high
>>> priority than queue 0. If running iperf tcp test on the three queues,
>>> then the tcp segment may be out-of-order that cause net watchdog
>>> timeout.
>> Okay. A single event would be understandable, but it seems to enter some
>> kind of loop after that (continuously printing "fec 30be0000.ethernet
>> eth0: TX ring dump ...").
>>
>> In a quick test I commented out the fec_dump call, with that it seems to
>> print only once and continues working afterwards (although, speed starts
>> to decrease, so something is not good at that point).
> The test base on above change ? One queue still bring watchdog timeout ?

No, sorry for the confusion: This was without the fix above. So use
multiple queues, and disable fec_dump... I was just wondering, because
disabling the multiple queues seems to me somewhat a workaround for
now... :-)

Re: FEC on i.MX 7 transmit queue timeout

[Andy Duan]

On 2017年05月05日 10:09, Stefan Agner wrote:
> On 2017-05-04 19:03, Andy Duan wrote:
>> On 2017年05月05日 05:36, Stefan Agner wrote:
>>> On 2017-05-03 20:08, Andy Duan wrote:
>>>> From: Stefan Agner <stefan@agner.ch> Sent: Thursday, May 04, 2017 9:22 AM
>>>>> To: Andy Duan <fugang.duan@nxp.com>
>>>>> Cc: fugang.duan@freescale.com; festevam@gmail.com;
>>>>> netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>>>>> Subject: Re: FEC on i.MX 7 transmit queue timeout
>>>>>
>>>>> Hi Andy,
>>>>>
>>>>> On 2017-04-20 19:48, Andy Duan wrote:
>>>>>> On 2017年04月20日 07:15, Stefan Agner wrote:
>>>>>>> I tested again with imx6sx-fec compatible string. I could reproduce
>>>>>>> it on a Colibri with i.MX 7Dual. But not always: It really depends
>>>>>>> whether queue 2 is counting up or not. Just after boot, I check
>>>>>>> /proc/interrupts twice, if queue 2 is counting it will happen!
>>>>>>>
>>>>>>> But if only queue 0 is mostly in use, then it seems to work just fine.
>>>>>> If your case is only running best effort like tcp/udp, you can re-set
>>>>>> the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts file.
>>>>>> Other two queues are for AVB audio/video queues, they have high
>>>>>> priority than queue 0. If running iperf tcp test on the three queues,
>>>>>> then the tcp segment may be out-of-order that cause net watchdog
>>>>> timeout.
>>>>>>> I also tried i.MX 7Dual SabreSD here, and the same thing. I had to
>>>>>>> reboot 3 times, then queue 2 was counting:
>>>>>>>     57:          8     GIC-0 150 Level     30be0000.ethernet
>>>>>>>     58:      20137     GIC-0 151 Level     30be0000.ethernet
>>>>>>>     59:       9269     GIC-0 152 Level     30be0000.ethernet
>>>>>>>
>>>>>>> It took me about 40 minutes on Sabre until it happened, and I had to
>>>>>>> force it using iperf, but then I got the ring dumps:
>>>>>> My board had ran more than 47 hours with nfs rootfs in 4.11.0-rc6, but
>>>>>> not running iperf.
>>>>>> I am testing with iperf.
>>>>> Any update on this issue?
>>>>>
>>>>> When using iperf (server) on the board with Linux 4.11 the issue appears
>>>>> within a few iperf iterations on a Sabre (TO 1.2, Board Rev C, if that matters)...
>>>>>
>>>> I don’t know whether you received my last mail. (maybe failed due to I
>>>> received some rejection mails)
>>> I think I did not... The last email I received was Fri, 21 Apr 2017
>>> 02:48:23 UTC.
>>>
>>>
>>>> If your case is only running best effort like tcp/udp, you can re-set
>>>> the "fsl,num-tx-queues" and "fsl,num-rx-queues" to 1 in board dts
>>>> file.
>>> I did test that, and it seems to work fine with those properties set to
>>> 1.
>> So it can fix your problem after long time test?
> Yes, seems to work fine after more than 2 hours.
>
>>>> Other two queues are for AVB audio/video queues, they have high
>>>> priority than queue 0. If running iperf tcp test on the three queues,
>>>> then the tcp segment may be out-of-order that cause net watchdog
>>>> timeout.
>>> Okay. A single event would be understandable, but it seems to enter some
>>> kind of loop after that (continuously printing "fec 30be0000.ethernet
>>> eth0: TX ring dump ...").
>>>
>>> In a quick test I commented out the fec_dump call, with that it seems to
>>> print only once and continues working afterwards (although, speed starts
>>> to decrease, so something is not good at that point).
>> The test base on above change ? One queue still bring watchdog timeout ?
> No, sorry for the confusion: This was without the fix above. So use
> multiple queues, and disable fec_dump... I was just wondering, because
> disabling the multiple queues seems to me somewhat a workaround for
> now... :-)
>
No, it is not workaround. As i said, quque1 and queue2 are for AVB paths 
have higher priority in transmition.
It bring the trouble for your case. I will submit one patch to fix it 
that best effort go queue0, AVB streaming go
quque1 and queue2.

>
>>>> In fsl kernel tree, there have one patch that only select the queue0
>>>> for best effort like tcp/udp. Pls test again in your board, if no
>>>> problem I will upstream the patch.
>>> That sounds like a reasonable fix.
>>>
>>> IP, no matter whether TCP/UDP, is the most common use case, so IMHO this
>>> should "just work" by default.
>>>
>>> --
>>> Stefan

Re: FEC on i.MX 7 transmit queue timeout

[Andrew Lunn]

> No, it is not workaround. As i said, quque1 and queue2 are for AVB paths 
> have higher priority in transmition.

Does this higher priority result in the low priority queue being
starved? Is that why the timer goes off? What happens when somebody
does use AVB. Are we back to the same problem? This is what seems to
make is sounds like a work around, not a fix.

     Andrew

RE: FEC on i.MX 7 transmit queue timeout

[Andy Duan]

From: Andrew Lunn <andrew@lunn.ch> Sent: Friday, May 05, 2017 8:24 PM
>To: Andy Duan <fugang.duan@nxp.com>
>Cc: Stefan Agner <stefan@agner.ch>; festevam@gmail.com;
>netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>Subject: Re: FEC on i.MX 7 transmit queue timeout
>
>> No, it is not workaround. As i said, quque1 and queue2 are for AVB
>> paths have higher priority in transmition.
>
>Does this higher priority result in the low priority queue being starved? Is that
>why the timer goes off? What happens when somebody does use AVB. Are
>we back to the same problem? This is what seems to make is sounds like a
>work around, not a fix.
>
>     Andrew
Yes, queue0 may be blocked by queue1 and queue2, then the queue0 watchdog time maybe triggered.
If somebody use AVB quque1 and queue2, the remaining bandwidth is for queue0, for example, in 100Mbps system, quque1 cost 50Mbps bandwidth and queue2 cost 50Mbps bandwidth for audio and video streaming, then queue0 (best effort) has 0 bandwidth that limit user case cannot have  asynchronous frames (IP(tcp/udp)) on networking. Of course these is extreme case. 
In essentially,  asynchronous frames (IP) go queue0 for the original design. To do these just implement .ndo_select_queue() callback in driver like fsl tree.

Regards,
Andy

Re: FEC on i.MX 7 transmit queue timeout

[Stefan Agner]

On 2017-05-07 19:13, Andy Duan wrote:
> From: Andrew Lunn <andrew@lunn.ch> Sent: Friday, May 05, 2017 8:24 PM
>>To: Andy Duan <fugang.duan@nxp.com>
>>Cc: Stefan Agner <stefan@agner.ch>; festevam@gmail.com;
>>netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>>Subject: Re: FEC on i.MX 7 transmit queue timeout
>>
>>> No, it is not workaround. As i said, quque1 and queue2 are for AVB
>>> paths have higher priority in transmition.
>>
>>Does this higher priority result in the low priority queue being starved? Is that
>>why the timer goes off? What happens when somebody does use AVB. Are
>>we back to the same problem? This is what seems to make is sounds like a
>>work around, not a fix.
>>
>>     Andrew
> Yes, queue0 may be blocked by queue1 and queue2, then the queue0
> watchdog time maybe triggered.
> If somebody use AVB quque1 and queue2, the remaining bandwidth is for
> queue0, for example, in 100Mbps system, quque1 cost 50Mbps bandwidth
> and queue2 cost 50Mbps bandwidth for audio and video streaming, then
> queue0 (best effort) has 0 bandwidth that limit user case cannot have 
> asynchronous frames (IP(tcp/udp)) on networking. Of course these is
> extreme case.
> In essentially,  asynchronous frames (IP) go queue0 for the original
> design. To do these just implement .ndo_select_queue() callback in
> driver like fsl tree.

I guess you refer to this commit?

http://git.freescale.com/git/cgit.cgi/imx/linux-imx.git/commit/?h=imx_4.1.15_2.0.0_ga&id=b0d8fa989651baf847287f6888f4d7b723e2a207

It seems that by default there is a Credit-based scheme enabled
(ENETx_QOS[TX_SCHEME] = 000). The driver enables the queue1/2 and
assigns it each 50% of the bandwidth (IDLE_SLOPE_1/2 is set to 0x200,
which according to the register description of IDLE_SLOPE means BW
fraction of 0.5!). This actually violates even the note in register
ENETx_DMAnCFG:

"NOTE: For AVB applications, the BW fraction of class 1 and class 2
combined must not exceed .75."

Instead of using the credit based scheme we could switch to round robin,
but not sure if that is what we want.

What is the default criteria to select queues when .ndo_select_queue is
not provided? I guess it tries to balance individual streams/processes
for better SMP performance?

RE: FEC on i.MX 7 transmit queue timeout

[Andy Duan]

From: Stefan Agner <stefan@agner.ch> Sent: Tuesday, May 09, 2017 2:23 AM
>To: Andy Duan <fugang.duan@nxp.com>; Andrew Lunn <andrew@lunn.ch>
>Cc: festevam@gmail.com; netdev@vger.kernel.org; netdev-
>owner@vger.kernel.org
>Subject: Re: FEC on i.MX 7 transmit queue timeout
>
>On 2017-05-07 19:13, Andy Duan wrote:
>> From: Andrew Lunn <andrew@lunn.ch> Sent: Friday, May 05, 2017 8:24 PM
>>>To: Andy Duan <fugang.duan@nxp.com>
>>>Cc: Stefan Agner <stefan@agner.ch>; festevam@gmail.com;
>>>netdev@vger.kernel.org; netdev-owner@vger.kernel.org
>>>Subject: Re: FEC on i.MX 7 transmit queue timeout
>>>
>>>> No, it is not workaround. As i said, quque1 and queue2 are for AVB
>>>> paths have higher priority in transmition.
>>>
>>>Does this higher priority result in the low priority queue being
>>>starved? Is that why the timer goes off? What happens when somebody
>>>does use AVB. Are we back to the same problem? This is what seems to
>>>make is sounds like a work around, not a fix.
>>>
>>>     Andrew
>> Yes, queue0 may be blocked by queue1 and queue2, then the queue0
>> watchdog time maybe triggered.
>> If somebody use AVB quque1 and queue2, the remaining bandwidth is for
>> queue0, for example, in 100Mbps system, quque1 cost 50Mbps bandwidth
>> and queue2 cost 50Mbps bandwidth for audio and video streaming, then
>> queue0 (best effort) has 0 bandwidth that limit user case cannot have
>> asynchronous frames (IP(tcp/udp)) on networking. Of course these is
>> extreme case.
>> In essentially,  asynchronous frames (IP) go queue0 for the original
>> design. To do these just implement .ndo_select_queue() callback in
>> driver like fsl tree.
>
>I guess you refer to this commit?
>
>http://git.freescale.com/git/cgit.cgi/imx/linux-
>imx.git/commit/?h=imx_4.1.15_2.0.0_ga&id=b0d8fa989651baf847287f6888f4d
>7b723e2a207
>
>It seems that by default there is a Credit-based scheme enabled
>(ENETx_QOS[TX_SCHEME] = 000). The driver enables the queue1/2 and
>assigns it each 50% of the bandwidth (IDLE_SLOPE_1/2 is set to 0x200, which
>according to the register description of IDLE_SLOPE means BW fraction of 0.5!).
>This actually violates even the note in register
>ENETx_DMAnCFG:
>
>"NOTE: For AVB applications, the BW fraction of class 1 and class 2 combined
>must not exceed .75."
>
Yes, the suggested BW fraction for class1 and class2 must not exceed .75, otherwise best effort
Queue will be blocked then watchdog timeout on queue0.

How to set the BW fraction that is up to user case.

>Instead of using the credit based scheme we could switch to round robin, but
>not sure if that is what we want.
> 
Using round robin has no meaning for AVB feature.

>What is the default criteria to select queues when .ndo_select_queue is not
>provided? I guess it tries to balance individual streams/processes for better
>SMP performance?
>
Since AVB audio/video streaming is based on IEEE1722 format,  .ndo_select_queue just parse
the avb streaming and insect the skb into class1 and class2 queues, which are individual streaming and processes.