Regarding eth0 (ftgmac100): transmit queue 0 timed out

Regarding eth0 (ftgmac100): transmit queue 0 timed out

[Mohammed.Habeeb ISV]

[-- Attachment #1.1: Type: text/plain, Size: 980 bytes --]

Hi All,

We see below backtrace on our runBmc board. And networking does not work on eth0.
No DHCP and static IP address can work. Pings and ARP fails, no packets received seen running tcpdump.
Using 2.7 version of openBmc. Please let me know if there is a known fix or patch for this.

transformers:~# [  158.082251] ------------[ cut here ]------------
[  158.087442] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x2d4/0x2f8
[  158.096681] NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0 timed out
[  158.104279] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.4.8-671f622-dirty-f5845fd #1

Few issues were reported but no resolution.
https://github.com/openbmc/openbmc/issues/3192
https://github.com/openbmc/openbmc/issues/2228

Tried below patch , seems does not work.
http://lkml.iu.edu/hypermail/linux/kernel/2010.1/05957.html

FYI ..We tried openBmc 2.9 version. It can work fine.

Attached is the complete backtrace.

Thanks


[-- Attachment #1.2: Type: text/html, Size: 3621 bytes --]

[-- Attachment #2: stacktrace.txt --]
[-- Type: text/plain, Size: 3347 bytes --]

transformers:~# [  158.082251] ------------[ cut here ]------------
[  158.087442] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x2d4/0x2f8
[  158.096681] NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0 timed out
[  158.104279] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.4.8-671f622-dirty-f5845fd #1
[  158.112921] Hardware name: Generic DT based system
[  158.118264] Backtrace:
[  158.121002] [<8010ea54>] (dump_backtrace) from [<8010ec88>] (show_stack+0x20/0x24)
[  158.129449]  r7:000001bf r6:600e0113 r5:00000000 r4:80c55eb8
[  158.135767] [<8010ec68>] (show_stack) from [<80834110>] (dump_stack+0xc8/0xe4)
[  158.143830] [<80834048>] (dump_stack) from [<80123468>] (__warn+0xcc/0x104)
[  158.151597]  r7:000001bf r6:8072f80c r5:00000009 r4:80a85c10
[  158.157913] [<8012339c>] (__warn) from [<801238b0>] (warn_slowpath_fmt+0x9c/0xc4)
[  158.166262]  r7:8072f80c r6:000001bf r5:80a85c10 r4:80a85bd4
[  158.172581] [<80123818>] (warn_slowpath_fmt) from [<8072f80c>] (dev_watchdog+0x2d4/0x2f8)
[  158.181706]  r8:b4402800 r7:00000001 r6:80c03d00 r5:b4402a64 r4:00000000
[  158.189185] [<8072f538>] (dev_watchdog) from [<80197c08>] (call_timer_fn+0x40/0x19c)
[  158.197826]  r8:ffffc890 r7:00000100 r6:8072f538 r5:b4402a64 r4:b4402a64
[  158.205303] [<80197bc8>] (call_timer_fn) from [<80198394>] (run_timer_softirq+0x1fc/0x5d8)
[  158.214525]  r8:ffffc890 r7:ffffe000 r6:00000000 r5:b4402a64 r4:bf0fde34
[  158.222004] [<80198198>] (run_timer_softirq) from [<801024a0>] (__do_softirq+0xf0/0x374)
[  158.231033]  r10:00000082 r9:bf0fdf38 r8:00000100 r7:bf0fc000 r6:00000001 r5:00000002
[  158.239767]  r4:80c03084
[  158.242592] [<801023b0>] (__do_softirq) from [<8012a018>] (irq_exit+0xb0/0xe8)
[  158.250651]  r10:00000000 r9:bf0fdf38 r8:bf00c800 r7:00000001 r6:00000000 r5:80b7d1c4
[  158.259385]  r4:ffffe000
[  158.262202] [<80129f68>] (irq_exit) from [<8017865c>] (__handle_domain_irq+0x68/0xc4)
[  158.270936]  r5:80b7d1c4 r4:00000000
[  158.274924] [<801785f4>] (__handle_domain_irq) from [<801022f8>] (gic_handle_irq+0x4c/0x94)
[  158.284245]  r9:bf0fdf38 r8:00000000 r7:bf802000 r6:80c062e8 r5:000003ff r4:bf80200c
[  158.292886] [<801022ac>] (gic_handle_irq) from [<80101aac>] (__irq_svc+0x6c/0x90)
[  158.301233] Exception stack(0xbf0fdf38 to 0xbf0fdf80)
[  158.306866] df20:                                                       0001ba3c 00000000
[  158.315993] df40: bf59bdf4 8011d780 bf0fc000 00000001 80c05c68 80c05ca4 80c5a2d2 809ffff0
[  158.325119] df60: 00000000 bf0fdf94 bf0fdf98 bf0fdf88 8010a03c 8010a040 600e0013 ffffffff
[  158.334244]  r9:bf0fc000 r8:80c5a2d2 r7:bf0fdf6c r6:ffffffff r5:600e0013 r4:8010a040
[  158.342890] [<8010a000>] (arch_cpu_idle) from [<80853e28>] (default_idle_call+0x30/0x3c)
[  158.351923] [<80853df8>] (default_idle_call) from [<80156624>] (do_idle+0xd8/0x16c)
[  158.360467] [<8015654c>] (do_idle) from [<8015698c>] (cpu_startup_entry+0x28/0x2c)
[  158.368914]  r9:410fc075 r8:8000406a r7:80c76200 r6:10c0387d r5:00000001 r4:00000089
[  158.377557] [<80156964>] (cpu_startup_entry) from [<8010fb5c>] (secondary_start_kernel+0x148/0x18c)
[  158.387653] [<8010fa14>] (secondary_start_kernel) from [<801027cc>] (__enable_mmu+0x0/0x14)
[  158.396970]  r5:00000051 r4:bf0f006a
[  158.401016] ---[ end trace a4f73b4bff414075 ]---

Re: Regarding eth0 (ftgmac100): transmit queue 0 timed out

[Joel Stanley]

On Thu, 14 Jan 2021 at 23:48, Mohammed.Habeeb ISV
<mohammed.habeeb@inventec.com> wrote:
>
> Hi All,
>
>
>
> We see below backtrace on our runBmc board. And networking does not work on eth0.
>
> No DHCP and static IP address can work. Pings and ARP fails, no packets received seen running tcpdump.
>
> Using 2.7 version of openBmc. Please let me know if there is a known fix or patch for this.
>
>
>
> transformers:~# [  158.082251] ------------[ cut here ]------------
>
> [  158.087442] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:447 dev_watchdog+0x2d4/0x2f8
>
> [  158.096681] NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0 timed out
>
> [  158.104279] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.4.8-671f622-dirty-f5845fd #1

This is a symptom of networking not working; there's many possible root causes.

Is this an ast2500?

From your log, it appears you're using a 5.4 kernel:

[  158.104279] CPU: 1 PID: 0 Comm: swapper/1 Not tainted
5.4.8-671f622-dirty-f5845fd #1

This is not maintained by anyone, and is probably missing patches. It
would be best if you moved to the latest openbmc kernel tree, but if
you can't do that then look at the history of the ethernet driver in
dev-5.8 and see which patches you're missing:

$ git log --no-merges --oneline v5.4...dev-5.8
drivers/net/ethernet/faraday/ftgmac100.*
9f55d7825b96 net: ftgmac100: Ensure tx descriptor updates are visible
8de0141c2524 net: ftgmac100: Fix Aspeed ast2600 TX hang issue
e6b45ee790ec net/faraday: fix grammar in function
ftgmac100_setup_clk() in ftgmac100.c
86eeb97de710 ftgmac100: Remove redundant judgement
d560b733ed53 net/faraday: Delete driver version from the drivers
f1294617d2f3 net: convert suitable network drivers to use phy_do_ioctl
0290bd291cc0 netdev: pass the stuck queue to the timeout handler
0c65b2b90d13 net: of_get_phy_mode: Change API to solve int/unit warnings
9bce4b27f3ec net: ftgmac100: Ungate RCLK for RMII on ASPEED MACs
0ac624f47dd3 docs: fix some broken references

The top two patches "net: ftgmac100: Ensure tx descriptor updates are
visible" and "net: ftgmac100: Fix Aspeed ast2600 TX hang issue" are
ast2600 specific.

>
>
>
> Few issues were reported but no resolution.
>
> https://github.com/openbmc/openbmc/issues/3192
>
> https://github.com/openbmc/openbmc/issues/2228

These two issues were specific to the broadcom network device used on
that platform. I doubt they relate to your issue.

>
>
>
> Tried below patch , seems does not work.
>
> http://lkml.iu.edu/hypermail/linux/kernel/2010.1/05957.html
>
>
>
> FYI ..We tried openBmc 2.9 version. It can work fine.
>
>
>
> Attached is the complete backtrace.
>
>
>
> Thanks
>
>

RE: Regarding eth0 (ftgmac100): transmit queue 0 timed out

[Mohammed.Habeeb ISV]

-----Original Message-----
From: Joel Stanley <joel@jms.id.au> 
Sent: Thursday, January 14, 2021 4:00 PM
To: Mohammed.Habeeb ISV <mohammed.habeeb@inventec.com>
Cc: openbmc@lists.ozlabs.org
Subject: Re: Regarding eth0 (ftgmac100): transmit queue 0 timed out

On Thu, 14 Jan 2021 at 23:48, Mohammed.Habeeb ISV <mohammed.habeeb@inventec.com> wrote:
>
> Hi All,
>
>
>
> We see below backtrace on our runBmc board. And networking does not work on eth0.
>
> No DHCP and static IP address can work. Pings and ARP fails, no packets received seen running tcpdump.
>
> Using 2.7 version of openBmc. Please let me know if there is a known fix or patch for this.
>
>
>
> transformers:~# [  158.082251] ------------[ cut here ]------------
>
> [  158.087442] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:447 
> dev_watchdog+0x2d4/0x2f8
>
> [  158.096681] NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0 
> timed out
>
> [  158.104279] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 
> 5.4.8-671f622-dirty-f5845fd #1

This is a symptom of networking not working; there's many possible root causes.

Is this an ast2500?
-It is an ast2600.

From your log, it appears you're using a 5.4 kernel:

[  158.104279] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 5.4.8-671f622-dirty-f5845fd #1

This is not maintained by anyone, and is probably missing patches. It would be best if you moved to the latest openbmc kernel tree, but if you can't do that then look at the history of the ethernet driver in
dev-5.8 and see which patches you're missing:

$ git log --no-merges --oneline v5.4...dev-5.8
drivers/net/ethernet/faraday/ftgmac100.*
9f55d7825b96 net: ftgmac100: Ensure tx descriptor updates are visible
8de0141c2524 net: ftgmac100: Fix Aspeed ast2600 TX hang issue e6b45ee790ec net/faraday: fix grammar in function
ftgmac100_setup_clk() in ftgmac100.c
86eeb97de710 ftgmac100: Remove redundant judgement
d560b733ed53 net/faraday: Delete driver version from the drivers
f1294617d2f3 net: convert suitable network drivers to use phy_do_ioctl
0290bd291cc0 netdev: pass the stuck queue to the timeout handler
0c65b2b90d13 net: of_get_phy_mode: Change API to solve int/unit warnings 9bce4b27f3ec net: ftgmac100: Ungate RCLK for RMII on ASPEED MACs
0ac624f47dd3 docs: fix some broken references

The top two patches "net: ftgmac100: Ensure tx descriptor updates are visible" and "net: ftgmac100: Fix Aspeed ast2600 TX hang issue" are
ast2600 specific.

-Applied all these patches to sync drivers/net/ethernet/faraday/* with latest dev-5.8, still I see the backtrace. 
>
>
>
> Few issues were reported but no resolution.
>
> https://github.com/openbmc/openbmc/issues/3192
>
> https://github.com/openbmc/openbmc/issues/2228

These two issues were specific to the broadcom network device used on that platform. I doubt they relate to your issue.

>
>
>
> Tried below patch , seems does not work.
>
> http://lkml.iu.edu/hypermail/linux/kernel/2010.1/05957.html
>
>
>
> FYI ..We tried openBmc 2.9 version. It can work fine.
>
>
>
> Attached is the complete backtrace.
>
>
>
> Thanks
>
>

RE: Regarding eth0 (ftgmac100): transmit queue 0 timed out

[Dylan Hung]

Hi Mohammed,

Before this backtrace showed up, is there anything you've done? E.g. getting IP via DHCP, or ping some addresses?

--
Dylan

> -----Original Message-----
> From: openbmc
> [mailto:openbmc-bounces+dylan_hung=aspeedtech.com@lists.ozlabs.org] On
> Behalf Of Mohammed.Habeeb ISV
> Sent: 2021年1月19日 4:09 AM
> To: Joel Stanley <joel@jms.id.au>
> Cc: openbmc@lists.ozlabs.org
> Subject: RE: Regarding eth0 (ftgmac100): transmit queue 0 timed out
> 
> 
> 
> -----Original Message-----
> From: Joel Stanley <joel@jms.id.au>
> Sent: Thursday, January 14, 2021 4:00 PM
> To: Mohammed.Habeeb ISV <mohammed.habeeb@inventec.com>
> Cc: openbmc@lists.ozlabs.org
> Subject: Re: Regarding eth0 (ftgmac100): transmit queue 0 timed out
> 
> On Thu, 14 Jan 2021 at 23:48, Mohammed.Habeeb ISV
> <mohammed.habeeb@inventec.com> wrote:
> >
> > Hi All,
> >
> >
> >
> > We see below backtrace on our runBmc board. And networking does not
> work on eth0.
> >
> > No DHCP and static IP address can work. Pings and ARP fails, no packets
> received seen running tcpdump.
> >
> > Using 2.7 version of openBmc. Please let me know if there is a known fix or
> patch for this.
> >
> >
> >
> > transformers:~# [  158.082251] ------------[ cut here ]------------
> >
> > [  158.087442] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:447
> > dev_watchdog+0x2d4/0x2f8
> >
> > [  158.096681] NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0
> > timed out
> >
> > [  158.104279] CPU: 1 PID: 0 Comm: swapper/1 Not tainted
> > 5.4.8-671f622-dirty-f5845fd #1
> 
> This is a symptom of networking not working; there's many possible root
> causes.
> 
> Is this an ast2500?
> -It is an ast2600.
> 
> From your log, it appears you're using a 5.4 kernel:
> 
> [  158.104279] CPU: 1 PID: 0 Comm: swapper/1 Not tainted
> 5.4.8-671f622-dirty-f5845fd #1
> 
> This is not maintained by anyone, and is probably missing patches. It would be
> best if you moved to the latest openbmc kernel tree, but if you can't do that
> then look at the history of the ethernet driver in
> dev-5.8 and see which patches you're missing:
> 
> $ git log --no-merges --oneline v5.4...dev-5.8
> drivers/net/ethernet/faraday/ftgmac100.*
> 9f55d7825b96 net: ftgmac100: Ensure tx descriptor updates are visible
> 8de0141c2524 net: ftgmac100: Fix Aspeed ast2600 TX hang issue
> e6b45ee790ec net/faraday: fix grammar in function
> ftgmac100_setup_clk() in ftgmac100.c
> 86eeb97de710 ftgmac100: Remove redundant judgement
> d560b733ed53 net/faraday: Delete driver version from the drivers
> f1294617d2f3 net: convert suitable network drivers to use phy_do_ioctl
> 0290bd291cc0 netdev: pass the stuck queue to the timeout handler
> 0c65b2b90d13 net: of_get_phy_mode: Change API to solve int/unit warnings
> 9bce4b27f3ec net: ftgmac100: Ungate RCLK for RMII on ASPEED MACs
> 0ac624f47dd3 docs: fix some broken references
> 
> The top two patches "net: ftgmac100: Ensure tx descriptor updates are visible"
> and "net: ftgmac100: Fix Aspeed ast2600 TX hang issue" are
> ast2600 specific.
> 
> -Applied all these patches to sync drivers/net/ethernet/faraday/* with latest
> dev-5.8, still I see the backtrace.
> >
> >
> >
> > Few issues were reported but no resolution.
> >
> > https://github.com/openbmc/openbmc/issues/3192
> >
> > https://github.com/openbmc/openbmc/issues/2228
> 
> These two issues were specific to the broadcom network device used on that
> platform. I doubt they relate to your issue.
> 
> >
> >
> >
> > Tried below patch , seems does not work.
> >
> > http://lkml.iu.edu/hypermail/linux/kernel/2010.1/05957.html
> >
> >
> >
> > FYI ..We tried openBmc 2.9 version. It can work fine.
> >
> >
> >
> > Attached is the complete backtrace.
> >
> >
> >
> > Thanks
> >
> >

RE: Regarding eth0 (ftgmac100): transmit queue 0 timed out

[Mohammed.Habeeb ISV]

-----Original Message-----
From: Dylan Hung <dylan_hung@aspeedtech.com> 
Sent: Wednesday, January 20, 2021 2:19 AM
To: Mohammed.Habeeb ISV <mohammed.habeeb@inventec.com>; Joel Stanley <joel@jms.id.au>
Cc: openbmc@lists.ozlabs.org
Subject: RE: Regarding eth0 (ftgmac100): transmit queue 0 timed out

Hi Mohammed,

Before this backtrace showed up, is there anything you've done? E.g. getting IP via DHCP, or ping some addresses?

HB> No pings. DHCP was enabled by default . I tried disabling DHCP to check if it is the reason for backtrace. But same result in both cases.
--
Dylan

> -----Original Message-----
> From: openbmc
> [mailto:openbmc-bounces+dylan_hung=aspeedtech.com@lists.ozlabs.org] On 
> Behalf Of Mohammed.Habeeb ISV
> Sent: 2021年1月19日 4:09 AM
> To: Joel Stanley <joel@jms.id.au>
> Cc: openbmc@lists.ozlabs.org
> Subject: RE: Regarding eth0 (ftgmac100): transmit queue 0 timed out
> 
> 
> 
> -----Original Message-----
> From: Joel Stanley <joel@jms.id.au>
> Sent: Thursday, January 14, 2021 4:00 PM
> To: Mohammed.Habeeb ISV <mohammed.habeeb@inventec.com>
> Cc: openbmc@lists.ozlabs.org
> Subject: Re: Regarding eth0 (ftgmac100): transmit queue 0 timed out
> 
> On Thu, 14 Jan 2021 at 23:48, Mohammed.Habeeb ISV 
> <mohammed.habeeb@inventec.com> wrote:
> >
> > Hi All,
> >
> >
> >
> > We see below backtrace on our runBmc board. And networking does not
> work on eth0.
> >
> > No DHCP and static IP address can work. Pings and ARP fails, no 
> > packets
> received seen running tcpdump.
> >
> > Using 2.7 version of openBmc. Please let me know if there is a known 
> > fix or
> patch for this.
> >
> >
> >
> > transformers:~# [  158.082251] ------------[ cut here ]------------
> >
> > [  158.087442] WARNING: CPU: 1 PID: 0 at net/sched/sch_generic.c:447
> > dev_watchdog+0x2d4/0x2f8
> >
> > [  158.096681] NETDEV WATCHDOG: eth0 (ftgmac100): transmit queue 0 
> > timed out
> >
> > [  158.104279] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 
> > 5.4.8-671f622-dirty-f5845fd #1
> 
> This is a symptom of networking not working; there's many possible 
> root causes.
> 
> Is this an ast2500?
> -It is an ast2600.
> 
> From your log, it appears you're using a 5.4 kernel:
> 
> [  158.104279] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 
> 5.4.8-671f622-dirty-f5845fd #1
> 
> This is not maintained by anyone, and is probably missing patches. It 
> would be best if you moved to the latest openbmc kernel tree, but if 
> you can't do that then look at the history of the ethernet driver in
> dev-5.8 and see which patches you're missing:
> 
> $ git log --no-merges --oneline v5.4...dev-5.8
> drivers/net/ethernet/faraday/ftgmac100.*
> 9f55d7825b96 net: ftgmac100: Ensure tx descriptor updates are visible
> 8de0141c2524 net: ftgmac100: Fix Aspeed ast2600 TX hang issue 
> e6b45ee790ec net/faraday: fix grammar in function
> ftgmac100_setup_clk() in ftgmac100.c
> 86eeb97de710 ftgmac100: Remove redundant judgement
> d560b733ed53 net/faraday: Delete driver version from the drivers
> f1294617d2f3 net: convert suitable network drivers to use phy_do_ioctl
> 0290bd291cc0 netdev: pass the stuck queue to the timeout handler
> 0c65b2b90d13 net: of_get_phy_mode: Change API to solve int/unit 
> warnings 9bce4b27f3ec net: ftgmac100: Ungate RCLK for RMII on ASPEED 
> MACs
> 0ac624f47dd3 docs: fix some broken references
> 
> The top two patches "net: ftgmac100: Ensure tx descriptor updates are visible"
> and "net: ftgmac100: Fix Aspeed ast2600 TX hang issue" are
> ast2600 specific.
> 
> -Applied all these patches to sync drivers/net/ethernet/faraday/* with 
> latest dev-5.8, still I see the backtrace.
> >
> >
> >
> > Few issues were reported but no resolution.
> >
> > https://github.com/openbmc/openbmc/issues/3192
> >
> > https://github.com/openbmc/openbmc/issues/2228
> 
> These two issues were specific to the broadcom network device used on 
> that platform. I doubt they relate to your issue.
> 
> >
> >
> >
> > Tried below patch , seems does not work.
> >
> > http://lkml.iu.edu/hypermail/linux/kernel/2010.1/05957.html
> >
> >
> >
> > FYI ..We tried openBmc 2.9 version. It can work fine.
> >
> >
> >
> > Attached is the complete backtrace.
> >
> >
> >
> > Thanks
> >
> >