* [0/1 bpf-next] fix panic bringing up veth with xdp progs
@ 2022-11-08 22:16 John Fastabend
2022-11-08 22:16 ` [1/1 bpf-next] bpf: veth driver panics when xdp prog attached before veth_open John Fastabend
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: John Fastabend @ 2022-11-08 22:16 UTC (permalink / raw)
To: daniel, kuba, davem, ast; +Cc: netdev, bpf, john.fastabend
Not sure if folks want to take this through BPF tree or networking tree.
I took a quick look and didn't see any pending fixes so seems no one
has noticed the panic yet. It reproducible and easy to repro.
I put bpf in the title thinking it woudl be great to run through the
BPF selftests given its XDP triggering the panic.
Sorry maintainers resent with CC'ing actual lists. Had a scripting
issue. Also dropped henqqi has they are bouncing.
Thanks!
John Fastabend (1):
bpf: veth driver panics when xdp prog attached before veth_open
drivers/net/veth.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
2.33.0
^ permalink raw reply [flat|nested] 4+ messages in thread* [1/1 bpf-next] bpf: veth driver panics when xdp prog attached before veth_open 2022-11-08 22:16 [0/1 bpf-next] fix panic bringing up veth with xdp progs John Fastabend @ 2022-11-08 22:16 ` John Fastabend 2022-11-10 2:25 ` [0/1 bpf-next] fix panic bringing up veth with xdp progs Jakub Kicinski 2022-11-10 5:10 ` patchwork-bot+netdevbpf 2 siblings, 0 replies; 4+ messages in thread From: John Fastabend @ 2022-11-08 22:16 UTC (permalink / raw) To: daniel, kuba, davem, ast; +Cc: netdev, bpf, john.fastabend The following panic is observed when bringing up (veth_open) a veth device that has an XDP program attached. [ 61.519185] kernel BUG at net/core/dev.c:6442! [ 61.519456] invalid opcode: 0000 [#1] PREEMPT SMP PTI [ 61.519752] CPU: 0 PID: 408 Comm: ip Tainted: G W 6.1.0-rc2-185930-gd9095f92950b-dirty #26 [ 61.520288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014 [ 61.520806] RIP: 0010:napi_enable+0x3d/0x40 [ 61.521077] Code: f6 f6 80 61 08 00 00 02 74 0d 48 83 bf 88 01 00 00 00 74 03 80 cd 01 48 89 d0 f0 48 0f b1 4f 10 48 39 c2 75 c8 c3 cc cc cc cc <0f> 0b 90 48 8b 87 b0 00 00 00 48 81 c7 b0 00 00 00 45 31 c0 48 39 [ 61.522226] RSP: 0018:ffffbc9800cc36f8 EFLAGS: 00010246 [ 61.522557] RAX: 0000000000000001 RBX: 0000000000000300 RCX: 0000000000000001 [ 61.523004] RDX: 0000000000000010 RSI: ffffffff8b0de852 RDI: ffff9f03848e5000 [ 61.523452] RBP: 0000000000000000 R08: 0000000000000800 R09: 0000000000000000 [ 61.523899] R10: ffff9f0384a96800 R11: ffffffffffa48061 R12: ffff9f03849c3000 [ 61.524345] R13: 0000000000000300 R14: ffff9f03848e5000 R15: 0000001000000100 [ 61.524792] FS: 00007f58cb64d2c0(0000) GS:ffff9f03bbc00000(0000) knlGS:0000000000000000 [ 61.525301] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 61.525673] CR2: 00007f6cc629b498 CR3: 000000010498c000 CR4: 00000000000006f0 [ 61.526121] Call Trace: [ 61.526284] <TASK> [ 61.526425] __veth_napi_enable_range+0xd6/0x230 [ 61.526723] veth_enable_xdp+0xd0/0x160 [ 61.526969] veth_open+0x2e/0xc0 [ 61.527180] __dev_open+0xe2/0x1b0 [ 61.527405] __dev_change_flags+0x1a1/0x210 [ 61.527673] dev_change_flags+0x1c/0x60 This happens because we are calling veth_napi_enable() on already enabled queues. The root cause is in commit 2e0de6366ac16 changed the control logic dropping this case, if (priv->_xdp_prog) { err = veth_enable_xdp(dev); if (err) return err; - } else if (veth_gro_requested(dev)) { + /* refer to the logic in veth_xdp_set() */ + if (!rtnl_dereference(peer_rq->napi)) { + err = veth_napi_enable(peer); + if (err) + return err; + } so that now veth_napi_enable is called if the peer has not yet initialiazed its peer_rq->napi. The issue is this will happen even if the NIC is not up. Then in veth_enable_xdp just above we have similar path, veth_enable_xdp napi_already_on = (dev->flags & IFF_UP) && rcu_access_pointer(rq->napi) err = veth_enable_xdp_range(dev, 0, dev->real_num_rx_queues, napi_already_on); The trouble is an xdp prog is assigned before bringing the device up each of the veth_open path will enable the peers xdp napi structs. But then when we bring the peer up it will similar try to enable again because from veth_open the IFF_UP flag is not set until after the op in __dev_open so we believe napi_alread_on = false. To fix this just drop the IFF_UP test and rely on checking if the napi struct is enabled. This also matches the peer check in veth_xdp for disabling. To reproduce run ./test_xdp_meta.sh I found adding Cilium/Tetragon tests for XDP. Fixes: 2e0de6366ac16 ("veth: Avoid drop packets when xdp_redirect performs") Signed-off-by: John Fastabend <john.fastabend@gmail.com> --- drivers/net/veth.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/net/veth.c b/drivers/net/veth.c index b1ed5a93b6c5..2a4592780141 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -1125,7 +1125,7 @@ static int veth_enable_xdp(struct net_device *dev) int err, i; rq = &priv->rq[0]; - napi_already_on = (dev->flags & IFF_UP) && rcu_access_pointer(rq->napi); + napi_already_on = rcu_access_pointer(rq->napi); if (!xdp_rxq_info_is_reg(&priv->rq[0].xdp_rxq)) { err = veth_enable_xdp_range(dev, 0, dev->real_num_rx_queues, napi_already_on); -- 2.33.0 ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [0/1 bpf-next] fix panic bringing up veth with xdp progs 2022-11-08 22:16 [0/1 bpf-next] fix panic bringing up veth with xdp progs John Fastabend 2022-11-08 22:16 ` [1/1 bpf-next] bpf: veth driver panics when xdp prog attached before veth_open John Fastabend @ 2022-11-10 2:25 ` Jakub Kicinski 2022-11-10 5:10 ` patchwork-bot+netdevbpf 2 siblings, 0 replies; 4+ messages in thread From: Jakub Kicinski @ 2022-11-10 2:25 UTC (permalink / raw) To: John Fastabend; +Cc: daniel, davem, ast, netdev, bpf, Paolo Abeni On Tue, 8 Nov 2022 14:16:49 -0800 John Fastabend wrote: > Not sure if folks want to take this through BPF tree or networking tree. Whatever's easiest :) FWIW Acked-by: Jakub Kicinski <kuba@kernel.org> > I took a quick look and didn't see any pending fixes so seems no one > has noticed the panic yet. It reproducible and easy to repro. > > I put bpf in the title thinking it woudl be great to run through the > BPF selftests given its XDP triggering the panic. > > Sorry maintainers resent with CC'ing actual lists. Had a scripting > issue. Also dropped henqqi has they are bouncing. Adding Paolo, who had comments on this patch in the past. The entire concept is worth mainintaining in your opinion, I take it? ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [0/1 bpf-next] fix panic bringing up veth with xdp progs 2022-11-08 22:16 [0/1 bpf-next] fix panic bringing up veth with xdp progs John Fastabend 2022-11-08 22:16 ` [1/1 bpf-next] bpf: veth driver panics when xdp prog attached before veth_open John Fastabend 2022-11-10 2:25 ` [0/1 bpf-next] fix panic bringing up veth with xdp progs Jakub Kicinski @ 2022-11-10 5:10 ` patchwork-bot+netdevbpf 2 siblings, 0 replies; 4+ messages in thread From: patchwork-bot+netdevbpf @ 2022-11-10 5:10 UTC (permalink / raw) To: John Fastabend; +Cc: daniel, kuba, davem, ast, netdev, bpf Hello: This patch was applied to bpf/bpf-next.git (master) by Martin KaFai Lau <martin.lau@kernel.org>: On Tue, 8 Nov 2022 14:16:49 -0800 you wrote: > Not sure if folks want to take this through BPF tree or networking tree. > I took a quick look and didn't see any pending fixes so seems no one > has noticed the panic yet. It reproducible and easy to repro. > > I put bpf in the title thinking it woudl be great to run through the > BPF selftests given its XDP triggering the panic. > > [...] Here is the summary with links: - [1/1,bpf-next] bpf: veth driver panics when xdp prog attached before veth_open https://git.kernel.org/bpf/bpf-next/c/5e5dc33d5dac You are awesome, thank you! -- Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-11-10 5:10 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2022-11-08 22:16 [0/1 bpf-next] fix panic bringing up veth with xdp progs John Fastabend 2022-11-08 22:16 ` [1/1 bpf-next] bpf: veth driver panics when xdp prog attached before veth_open John Fastabend 2022-11-10 2:25 ` [0/1 bpf-next] fix panic bringing up veth with xdp progs Jakub Kicinski 2022-11-10 5:10 ` patchwork-bot+netdevbpf
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.