[PATCH net] net: stmmac: fix call trace when stmmac_xdp_xmit() is invoked

[PATCH net] net: stmmac: fix call trace when stmmac_xdp_xmit() is invoked

[wei.fang]

From: Wei Fang <wei.fang@nxp.com>

We encountered a kernel call trace issue which was related to
ndo_xdp_xmit callback on our i.MX8MP platform. The reproduce
steps show as follows.
1. The FEC port (eth0) connects to a PC port, and the PC uses
pktgen_sample03_burst_single_flow.sh to generate packets and
send these packets to the FEC port. Notice that the script must
be executed before step 2.
2. Run the "./xdp_redirect eth0 eth1" command on i.MX8MP, the
eth1 interface is the dwmac. Then there will be a call trace
issue soon. Please see the log for more details.
The root cause is that the NETDEV_XDP_ACT_NDO_XMIT feature is
enabled by default, so when the step 2 command is exexcuted
and packets have already been sent to eth0, the stmmac_xdp_xmit()
starts running before the stmmac_xdp_set_prog() finishes. To
resolve this issue, we disable the NETDEV_XDP_ACT_NDO_XMIT
feature by default and turn on/off this feature when the bpf
program is installed/uninstalled which just like the other
ethernet drivers.

Call Trace log:
[  306.311271] ------------[ cut here ]------------
[  306.315910] WARNING: CPU: 0 PID: 15 at lib/timerqueue.c:55 timerqueue_del+0x68/0x70
[  306.323590] Modules linked in:
[  306.326654] CPU: 0 PID: 15 Comm: ksoftirqd/0 Not tainted 6.4.0-rc1+ #37
[  306.333277] Hardware name: NXP i.MX8MPlus EVK board (DT)
[  306.338591] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  306.345561] pc : timerqueue_del+0x68/0x70
[  306.349577] lr : __remove_hrtimer+0x5c/0xa0
[  306.353777] sp : ffff80000b7c3920
[  306.357094] x29: ffff80000b7c3920 x28: 0000000000000000 x27: 0000000000000001
[  306.364244] x26: ffff80000a763a40 x25: ffff0000d0285a00 x24: 0000000000000001
[  306.371390] x23: 0000000000000001 x22: ffff000179389a40 x21: 0000000000000000
[  306.378537] x20: ffff000179389aa0 x19: ffff0000d2951308 x18: 0000000000001000
[  306.385686] x17: f1d3000000000000 x16: 00000000c39c1000 x15: 55e99bbe00001a00
[  306.392835] x14: 09000900120aa8c0 x13: e49af1d300000000 x12: 000000000000c39c
[  306.399987] x11: 100055e99bbe0000 x10: ffff8000090b1048 x9 : ffff8000081603fc
[  306.407133] x8 : 000000000000003c x7 : 000000000000003c x6 : 0000000000000001
[  306.414284] x5 : ffff0000d2950980 x4 : 0000000000000000 x3 : 0000000000000000
[  306.421432] x2 : 0000000000000001 x1 : ffff0000d2951308 x0 : ffff0000d2951308
[  306.428585] Call trace:
[  306.431035]  timerqueue_del+0x68/0x70
[  306.434706]  __remove_hrtimer+0x5c/0xa0
[  306.438549]  hrtimer_start_range_ns+0x2bc/0x370
[  306.443089]  stmmac_xdp_xmit+0x174/0x1b0
[  306.447021]  bq_xmit_all+0x194/0x4b0
[  306.450612]  __dev_flush+0x4c/0x98
[  306.454024]  xdp_do_flush+0x18/0x38
[  306.457522]  fec_enet_rx_napi+0x6c8/0xc68
[  306.461539]  __napi_poll+0x40/0x220
[  306.465038]  net_rx_action+0xf8/0x240
[  306.468707]  __do_softirq+0x128/0x3a8
[  306.472378]  run_ksoftirqd+0x40/0x58
[  306.475961]  smpboot_thread_fn+0x1c4/0x288
[  306.480068]  kthread+0x124/0x138
[  306.483305]  ret_from_fork+0x10/0x20
[  306.486889] ---[ end trace 0000000000000000 ]---

Fixes: 66c0e13ad236 ("drivers: net: turn on XDP features")
Signed-off-by: Wei Fang <wei.fang@nxp.com>
---
 drivers/net/ethernet/stmicro/stmmac/stmmac_main.c | 3 +--
 drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c  | 6 ++++++
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index 0fca81507a77..52cab9de05f2 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -7233,8 +7233,7 @@ int stmmac_dvr_probe(struct device *device,
 	ndev->hw_features = NETIF_F_SG | NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM |
 			    NETIF_F_RXCSUM;
 	ndev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT |
-			     NETDEV_XDP_ACT_XSK_ZEROCOPY |
-			     NETDEV_XDP_ACT_NDO_XMIT;
+			     NETDEV_XDP_ACT_XSK_ZEROCOPY;
 
 	ret = stmmac_tc_init(priv, priv);
 	if (!ret) {
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c
index 9d4d8c3dad0a..aa6f16d3df64 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_xdp.c
@@ -117,6 +117,9 @@ int stmmac_xdp_set_prog(struct stmmac_priv *priv, struct bpf_prog *prog,
 		return -EOPNOTSUPP;
 	}
 
+	if (!prog)
+		xdp_features_clear_redirect_target(dev);
+
 	need_update = !!priv->xdp_prog != !!prog;
 	if (if_running && need_update)
 		stmmac_xdp_release(dev);
@@ -131,5 +134,8 @@ int stmmac_xdp_set_prog(struct stmmac_priv *priv, struct bpf_prog *prog,
 	if (if_running && need_update)
 		stmmac_xdp_open(dev);
 
+	if (prog)
+		xdp_features_set_redirect_target(dev, false);
+
 	return 0;
 }
-- 
2.25.1

Re: [PATCH net] net: stmmac: fix call trace when stmmac_xdp_xmit() is invoked

[Simon Horman]

On Wed, May 24, 2023 at 08:57:14PM +0800, wei.fang@nxp.com wrote:
> From: Wei Fang <wei.fang@nxp.com>
> 
> We encountered a kernel call trace issue which was related to
> ndo_xdp_xmit callback on our i.MX8MP platform. The reproduce
> steps show as follows.
> 1. The FEC port (eth0) connects to a PC port, and the PC uses
> pktgen_sample03_burst_single_flow.sh to generate packets and
> send these packets to the FEC port. Notice that the script must
> be executed before step 2.
> 2. Run the "./xdp_redirect eth0 eth1" command on i.MX8MP, the
> eth1 interface is the dwmac. Then there will be a call trace
> issue soon. Please see the log for more details.
> The root cause is that the NETDEV_XDP_ACT_NDO_XMIT feature is
> enabled by default, so when the step 2 command is exexcuted
> and packets have already been sent to eth0, the stmmac_xdp_xmit()
> starts running before the stmmac_xdp_set_prog() finishes. To
> resolve this issue, we disable the NETDEV_XDP_ACT_NDO_XMIT
> feature by default and turn on/off this feature when the bpf
> program is installed/uninstalled which just like the other
> ethernet drivers.
> 
> Call Trace log:
> [  306.311271] ------------[ cut here ]------------
> [  306.315910] WARNING: CPU: 0 PID: 15 at lib/timerqueue.c:55 timerqueue_del+0x68/0x70
> [  306.323590] Modules linked in:
> [  306.326654] CPU: 0 PID: 15 Comm: ksoftirqd/0 Not tainted 6.4.0-rc1+ #37
> [  306.333277] Hardware name: NXP i.MX8MPlus EVK board (DT)
> [  306.338591] pstate: 600000c5 (nZCv daIF -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> [  306.345561] pc : timerqueue_del+0x68/0x70
> [  306.349577] lr : __remove_hrtimer+0x5c/0xa0
> [  306.353777] sp : ffff80000b7c3920
> [  306.357094] x29: ffff80000b7c3920 x28: 0000000000000000 x27: 0000000000000001
> [  306.364244] x26: ffff80000a763a40 x25: ffff0000d0285a00 x24: 0000000000000001
> [  306.371390] x23: 0000000000000001 x22: ffff000179389a40 x21: 0000000000000000
> [  306.378537] x20: ffff000179389aa0 x19: ffff0000d2951308 x18: 0000000000001000
> [  306.385686] x17: f1d3000000000000 x16: 00000000c39c1000 x15: 55e99bbe00001a00
> [  306.392835] x14: 09000900120aa8c0 x13: e49af1d300000000 x12: 000000000000c39c
> [  306.399987] x11: 100055e99bbe0000 x10: ffff8000090b1048 x9 : ffff8000081603fc
> [  306.407133] x8 : 000000000000003c x7 : 000000000000003c x6 : 0000000000000001
> [  306.414284] x5 : ffff0000d2950980 x4 : 0000000000000000 x3 : 0000000000000000
> [  306.421432] x2 : 0000000000000001 x1 : ffff0000d2951308 x0 : ffff0000d2951308
> [  306.428585] Call trace:
> [  306.431035]  timerqueue_del+0x68/0x70
> [  306.434706]  __remove_hrtimer+0x5c/0xa0
> [  306.438549]  hrtimer_start_range_ns+0x2bc/0x370
> [  306.443089]  stmmac_xdp_xmit+0x174/0x1b0
> [  306.447021]  bq_xmit_all+0x194/0x4b0
> [  306.450612]  __dev_flush+0x4c/0x98
> [  306.454024]  xdp_do_flush+0x18/0x38
> [  306.457522]  fec_enet_rx_napi+0x6c8/0xc68
> [  306.461539]  __napi_poll+0x40/0x220
> [  306.465038]  net_rx_action+0xf8/0x240
> [  306.468707]  __do_softirq+0x128/0x3a8
> [  306.472378]  run_ksoftirqd+0x40/0x58
> [  306.475961]  smpboot_thread_fn+0x1c4/0x288
> [  306.480068]  kthread+0x124/0x138
> [  306.483305]  ret_from_fork+0x10/0x20
> [  306.486889] ---[ end trace 0000000000000000 ]---
> 
> Fixes: 66c0e13ad236 ("drivers: net: turn on XDP features")
> Signed-off-by: Wei Fang <wei.fang@nxp.com>

This looks good to me.

Reviewed-by: Simon Horman <simon.horman@corigine.com>

Alexandre, is there any feedback from your side?

Re: [PATCH net] net: stmmac: fix call trace when stmmac_xdp_xmit() is invoked

[patchwork-bot+netdevbpf]

Hello:

This patch was applied to netdev/net.git (main)
by Jakub Kicinski <kuba@kernel.org>:

On Wed, 24 May 2023 20:57:14 +0800 you wrote:
> From: Wei Fang <wei.fang@nxp.com>
> 
> We encountered a kernel call trace issue which was related to
> ndo_xdp_xmit callback on our i.MX8MP platform. The reproduce
> steps show as follows.
> 1. The FEC port (eth0) connects to a PC port, and the PC uses
> pktgen_sample03_burst_single_flow.sh to generate packets and
> send these packets to the FEC port. Notice that the script must
> be executed before step 2.
> 2. Run the "./xdp_redirect eth0 eth1" command on i.MX8MP, the
> eth1 interface is the dwmac. Then there will be a call trace
> issue soon. Please see the log for more details.
> The root cause is that the NETDEV_XDP_ACT_NDO_XMIT feature is
> enabled by default, so when the step 2 command is exexcuted
> and packets have already been sent to eth0, the stmmac_xdp_xmit()
> starts running before the stmmac_xdp_set_prog() finishes. To
> resolve this issue, we disable the NETDEV_XDP_ACT_NDO_XMIT
> feature by default and turn on/off this feature when the bpf
> program is installed/uninstalled which just like the other
> ethernet drivers.
> 
> [...]

Here is the summary with links:
  - [net] net: stmmac: fix call trace when stmmac_xdp_xmit() is invoked
    https://git.kernel.org/netdev/net/c/ffb3322181d9

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html