* Re: [PATCH 5.13 06/35] bpf, sockmap: On cleanup we additionally need to remove cached skb
2021-08-07 17:45 ` [PATCH 5.13 06/35] bpf, sockmap: On cleanup we additionally need to remove cached skb Naresh Kamboju
@ 2021-08-09 17:33 ` John Fastabend
0 siblings, 0 replies; 2+ messages in thread
From: John Fastabend @ 2021-08-09 17:33 UTC (permalink / raw)
To: Naresh Kamboju, Greg Kroah-Hartman, John Fastabend
Cc: open list, linux-stable, Andrii Nakryiko, Jakub Sitnicki,
Martin KaFai Lau, Sasha Levin, lkft-triage, Netdev
Naresh Kamboju wrote:
> While running packetdrill test suite on qemu aarch64 the following warning
> noticed with stable-rc 5.13.9-rc1 kernel intermittently.
>
> On Fri, 6 Aug 2021 at 13:52, Greg Kroah-Hartman
> <gregkh@linuxfoundation.org> wrote:
> >
> > From: John Fastabend <john.fastabend@gmail.com>
> >
> > [ Upstream commit 476d98018f32e68e7c5d4e8456940cf2b6d66f10 ]
>
> <trim>
>
> > With this fix we no longer see this warning splat from tcp side on
> > socket close when we hit the above case with redirect to ingress self.
> >
> > [224913.935822] WARNING: CPU: 3 PID: 32100 at net/core/stream.c:208 sk_stream_kill_queues+0x212/0x220
> > [224913.935841] Modules linked in: fuse overlay bpf_preload x86_pkg_temp_thermal intel_uncore wmi_bmof squashfs sch_fq_codel efivarfs ip_tables x_tables uas xhci_pci ixgbe mdio xfrm_algo xhci_hcd wmi
> > [224913.935897] CPU: 3 PID: 32100 Comm: fgs-bench Tainted: G I 5.14.0-rc1alu+ #181
> > [224913.935908] Hardware name: Dell Inc. Precision 5820 Tower/002KVM, BIOS 1.9.2 01/24/2019
> > [224913.935914] RIP: 0010:sk_stream_kill_queues+0x212/0x220
>
>
> steps to reproduce:
> -------------------
> # boot qemu aarch64 with stable-rc 5.13.9-rc1 kernel
> # /usr/bin/qemu-system-aarch64 -cpu max -machine virt,accel=kvm
> -nographic -net nic,model=virtio,macaddr=BC:DD:AD:CC:09:01 -net tap -m
> 4096 -monitor none -kernel kernel/Image.gz --append "console=ttyAMA0
> root=/dev/vda rw" -hda
> rpb-console-image-lkft-juno-20210525221209.rootfs.ext4 -m 4096 -smp 4
> -nographic
>
> # Run test
> # cd ./automated/linux/packetdrill/
> # ./configure
> # make all
> # python3 ./packetdrill/run_all.py -v -l -L
>
> ## Build
> * kernel: 5.13.9-rc1
> * git: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> * git branch: linux-5.13.y
> * git commit: 1eb1590ab470d5f73dd2d20a7196bca35fa3d3e7
> * git describe: v5.13.8-36-g1eb1590ab470
> * test details:
> https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-5.13.y/build/v5.13.8-36-g1eb1590ab470
Hi Naresh,
The fix here should only be visible with sockmap BPF programs running. The
trace below doesn't seem to have any of the BPF calls. I tried to parse
the test details, but I didn't see how packetdrill and the BPF tests
are related. The test that would be relevant linked here seems to be
passing in your case.
https://qa-reports.linaro.org/lkft/linux-stable-rc-linux-5.13.y/build/v5.13.8-36-g1eb1590ab470/testrun/5382761/suite/kselftest-bpf/test/bpf.test_sockmap/details/
Am I correct in assuming that you bisected to this patch somehow, but
did not have any BPF programs running?
Thanks,
John
>
>
> Crash log:
> -----------
> INFO: Skip installing package dependency for packetdrill
> /opt/packetdrill /lava-3242370/0/tests/1_packetdrill/automated/linux/packetdrill
> [ 11.329564] tun: Universal TUN/TAP device driver, 1.6
> [ 14.801347] TCP: tun0: Driver has suspect GRO implementation, TCP
> performance may be compromised.
> [ 15.113626] ------------[ cut here ]------------
> [ 15.115380] WARNING: CPU: 3 PID: 671 at net/core/stream.c:207
> sk_stream_kill_queues+0x104/0x130
> [ 15.118527] Modules linked in: tun crct10dif_ce rfkill fuse
> [ 15.120361] CPU: 3 PID: 671 Comm: packetdrill Not tainted 5.13.9-rc1 #1
> [ 15.122587] Hardware name: linux,dummy-virt (DT)
> [ 15.124123] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--)
> [ 15.126117] pc : sk_stream_kill_queues+0x104/0x130
> [ 15.127764] lr : inet_csk_destroy_sock+0x68/0x130
> [ 15.129326] sp : ffff8000109f36d0
> [ 15.130484] x29: ffff8000109f36d0 x28: 0000000000000005 x27: fffffffffffffff2
> [ 15.132807] x26: 0000000000000001 x25: ffffa05141000900 x24: ffff6f1e0b51dc40
> [ 15.136643] x23: 0000000000000000 x22: 0000000000000000 x21: ffff6f1e11e7e054
> [ 15.139117] x20: ffff6f1e08a0dd30 x19: ffff6f1e08a0dc80 x18: 0000000000000000
> [ 15.141494] x17: 0000000000000000 x16: 0000000000000000 x15: 00000000238ecea0
> [ 15.143903] x14: 0000000000000000 x13: 000000000000dd86 x12: 000000007ffff000
> [ 15.146292] x11: 0000000000000004 x10: 0000000000000000 x9 : ffffa051410d5148
> [ 15.148660] x8 : 0000000000000000 x7 : ffffffffd3039400 x6 : 0000000000000202
> [ 15.151071] x5 : ffff6f1e08a0dd00 x4 : 0000000000000004 x3 : 0000000000000007
> [ 15.153435] x2 : ffff6f1e08a0e560 x1 : 0000000000000180 x0 : 00000000fffffe80
> [ 15.155835] Call trace:
> [ 15.156675] sk_stream_kill_queues+0x104/0x130
> [ 15.163667] inet_csk_destroy_sock+0x68/0x130
> [ 15.165139] tcp_done+0x120/0x1b0
> [ 15.166274] tcp_reset+0x74/0x130
> [ 15.167445] tcp_validate_incoming+0x394/0x510
> [ 15.168953] tcp_rcv_state_process+0x2d8/0x15c0
> [ 15.170512] tcp_v4_do_rcv+0x15c/0x2d4
> [ 15.171798] tcp_v4_rcv+0x9c0/0xaa4
> [ 15.173009] ip_protocol_deliver_rcu+0x4c/0x184
> [ 15.174597] ip_local_deliver_finish+0x74/0x90
> [ 15.176103] ip_local_deliver+0x88/0x130
> [ 15.177429] ip_rcv+0x7c/0x130
> [ 15.178512] __netif_receive_skb_one_core+0x60/0x8c
> [ 15.180143] __netif_receive_skb+0x20/0x70
> [ 15.181527] netif_receive_skb+0x48/0x1e0
> [ 15.182930] tun_get_user+0xbe4/0xd70 [tun]
> [ 15.184368] tun_chr_write_iter+0x68/0xf0 [tun]
> [ 15.185936] do_iter_readv_writev+0x100/0x1a4
> [ 15.187448] do_iter_write+0x98/0x1fc
> [ 15.188735] vfs_writev+0xb4/0x170
> [ 15.189956] do_writev+0x7c/0x140
> [ 15.191115] __arm64_sys_writev+0x2c/0x40
> [ 15.192481] invoke_syscall+0x50/0x120
> [ 15.193754] el0_svc_common.constprop.0+0xf4/0x104
> [ 15.195391] do_el0_svc+0x34/0x9c
> [ 15.196558] el0_svc+0x2c/0x54
> [ 15.197653] el0_sync_handler+0xa4/0x130
> [ 15.199054] el0_sync+0x198/0x1c0
> [ 15.200182] ---[ end trace c9faa1be6c93e4fb ]---
> [ 15.201864] ------------[ cut here ]------------
> [ 15.203404] WARNING: CPU: 3 PID: 671 at net/core/stream.c:208
> sk_stream_kill_queues+0x110/0x130
> [ 15.206141] Modules linked in: tun crct10dif_ce rfkill fuse
> [ 15.207976] CPU: 3 PID: 671 Comm: packetdrill Tainted: G W
> 5.13.9-rc1 #1
> [ 15.210542] Hardware name: linux,dummy-virt (DT)
> [ 15.212029] pstate: 80000005 (Nzcv daif -PAN -UAO -TCO BTYPE=--)
> [ 15.213947] pc : sk_stream_kill_queues+0x110/0x130
> [ 15.215546] lr : inet_csk_destroy_sock+0x68/0x130
> [ 15.217097] sp : ffff8000109f36d0
> [ 15.218204] x29: ffff8000109f36d0 x28: 0000000000000005 x27: fffffffffffffff2
> [ 15.220548] x26: 0000000000000001 x25: ffffa05141000900 x24: ffff6f1e0b51dc40
> [ 15.223012] x23: 0000000000000000 x22: 0000000000000000 x21: ffff6f1e11e7e054
> [ 15.225350] x20: ffff6f1e08a0dd30 x19: ffff6f1e08a0dc80 x18: 0000000000000000
> [ 15.227705] x17: 0000000000000000 x16: 0000000000000000 x15: 00000000238ecea0
> [ 15.230042] x14: 0000000000000000 x13: 000000000000dd86 x12: 000000007ffff000
> [ 15.232454] x11: 0000000000000004 x10: 0000000000000000 x9 : ffffa051410d5148
> [ 15.234896] x8 : 0000000000000000 x7 : ffffffffd3039400 x6 : 0000000000000202
> [ 15.237544] x5 : ffff6f1e08a0dd00 x4 : 0000000000000004 x3 : 0000000000000007
> [ 15.239933] x2 : ffff6f1e08a0e560 x1 : 0000000000000180 x0 : 0000000000000180
> [ 15.242315] Call trace:
> [ 15.243203] sk_stream_kill_queues+0x110/0x130
> [ 15.244721] inet_csk_destroy_sock+0x68/0x130
> [ 15.246191] tcp_done+0x120/0x1b0
> [ 15.247388] tcp_reset+0x74/0x130
> [ 15.248540] tcp_validate_incoming+0x394/0x510
> [ 15.250057] tcp_rcv_state_process+0x2d8/0x15c0
> [ 15.251636] tcp_v4_do_rcv+0x15c/0x2d4
> [ 15.252908] tcp_v4_rcv+0x9c0/0xaa4
> [ 15.254095] ip_protocol_deliver_rcu+0x4c/0x184
> [ 15.255643] ip_local_deliver_finish+0x74/0x90
> [ 15.257138] ip_local_deliver+0x88/0x130
> [ 15.258516] ip_rcv+0x7c/0x130
> [ 15.259573] __netif_receive_skb_one_core+0x60/0x8c
> [ 15.261561] __netif_receive_skb+0x20/0x70
> [ 15.263355] netif_receive_skb+0x48/0x1e0
> [ 15.264721] tun_get_user+0xbe4/0xd70 [tun]
> [ 15.266110] tun_chr_write_iter+0x68/0xf0 [tun]
> [ 15.267704] do_iter_readv_writev+0x100/0x1a4
> [ 15.269162] do_iter_write+0x98/0x1fc
> [ 15.270721] vfs_writev+0xb4/0x170
> [ 15.271948] do_writev+0x7c/0x140
> [ 15.273120] __arm64_sys_writev+0x2c/0x40
> [ 15.274527] invoke_syscall+0x50/0x120
> [ 15.275805] el0_svc_common.constprop.0+0xf4/0x104
> [ 15.277278] do_el0_svc+0x34/0x9c
> [ 15.278832] el0_svc+0x2c/0x54
> [ 15.280081] el0_sync_handler+0xa4/0x130
> [ 15.281614] el0_sync+0x198/0x1c0
> [ 15.282899] ---[ end trace c9faa1be6c93e4fc ]---
> [ 15.284650] ------------[ cut here ]------------
> [ 15.286116] WARNING: CPU: 3 PID: 671 at net/ipv4/af_inet.c:156
> inet_sock_destruct+0x190/0x1b0
> [ 15.288999] Modules linked in: tun crct10dif_ce rfkill fuse
> [ 15.290884] CPU: 3 PID: 671 Comm: packetdrill Tainted: G W
> 5.13.9-rc1 #1
> [ 15.294511] Hardware name: linux,dummy-virt (DT)
> [ 15.296041] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
> [ 15.298073] pc : inet_sock_destruct+0x190/0x1b0
> [ 15.299743] lr : __sk_destruct+0x38/0x23c
> [ 15.301154] sp : ffff8000109f37c0
> [ 15.302291] x29: ffff8000109f37c0 x28: 0000000000000005 x27: fffffffffffffff2
> [ 15.304703] x26: 0000000000000001 x25: ffffa05141000900 x24: ffffa05142a8be80
> [ 15.307100] x23: 0000000000000000 x22: ffff6f1e08a0dd08 x21: ffff6f1e08a0dc80
> [ 15.309419] x20: ffff6f1e08a0dd30 x19: ffff6f1e08a0dc80 x18: 0000000000000000
> [ 15.311764] x17: 0000000000000000 x16: 0000000000000000 x15: 00000000238ecea0
> [ 15.314056] x14: 0000000000000000 x13: 000000000000dd86 x12: 000000007ffff000
> [ 15.316392] x11: 0000000000000004 x10: 0000000000000000 x9 : ffffa05141003acc
> [ 15.318680] x8 : 0000000000000000 x7 : ffffffffd3039400 x6 : ffff6f1e08a0ddbc
> [ 15.320920] x5 : 0000000000000001 x4 : 0000000000000000 x3 : ffffceccfd9db000
> [ 15.323197] x2 : ffff6f1e02c00810 x1 : 0000000000000180 x0 : 00000000fffffe80
> [ 15.325465] Call trace:
> [ 15.326270] inet_sock_destruct+0x190/0x1b0
> [ 15.327643] __sk_destruct+0x38/0x23c
> [ 15.328842] __sk_free+0x80/0x120
> [ 15.329923] sk_free+0x68/0x90
> [ 15.330941] sock_put+0x5c/0x80
> [ 15.331963] tcp_v4_rcv+0xa40/0xaa4
> [ 15.333091] ip_protocol_deliver_rcu+0x4c/0x184
> [ 15.334585] ip_local_deliver_finish+0x74/0x90
> [ 15.336000] ip_local_deliver+0x88/0x130
> [ 15.337273] ip_rcv+0x7c/0x130
> [ 15.338277] __netif_receive_skb_one_core+0x60/0x8c
> [ 15.339869] __netif_receive_skb+0x20/0x70
> [ 15.341221] netif_receive_skb+0x48/0x1e0
> [ 15.342558] tun_get_user+0xbe4/0xd70 [tun]
> [ 15.343929] tun_chr_write_iter+0x68/0xf0 [tun]
> [ 15.345395] do_iter_readv_writev+0x100/0x1a4
> [ 15.346831] do_iter_write+0x98/0x1fc
> [ 15.348025] vfs_writev+0xb4/0x170
> [ 15.349136] do_writev+0x7c/0x140
> [ 15.350224] __arm64_sys_writev+0x2c/0x40
> [ 15.351556] invoke_syscall+0x50/0x120
> [ 15.352789] el0_svc_common.constprop.0+0xf4/0x104
> [ 15.354335] do_el0_svc+0x34/0x9c
> [ 15.355451] el0_svc+0x2c/0x54
> [ 15.356460] el0_sync_handler+0xa4/0x130
> [ 15.357740] el0_sync+0x198/0x1c0
> [ 15.358861] ---[ end trace c9faa1be6c93e4fd ]---
> [ 15.360555] ------------[ cut here ]------------
> [ 15.361959] WARNING: CPU: 3 PID: 671 at net/ipv4/af_inet.c:157
> inet_sock_destruct+0x16c/0x1b0
> [ 15.364603] Modules linked in: tun crct10dif_ce rfkill fuse
> [ 15.366276] CPU: 3 PID: 671 Comm: packetdrill Tainted: G W
> 5.13.9-rc1 #1
> [ 15.373031] Hardware name: linux,dummy-virt (DT)
> [ 15.377359] pstate: 60000005 (nZCv daif -PAN -UAO -TCO BTYPE=--)
> [ 15.381502] pc : inet_sock_destruct+0x16c/0x1b0
> [ 15.384348] lr : __sk_destruct+0x38/0x23c
> [ 15.387724] sp : ffff8000109f37c0
> [ 15.388842] x29: ffff8000109f37c0 x28: 0000000000000005 x27: fffffffffffffff2
> [ 15.391840] x26: 0000000000000001 x25: ffffa05141000900 x24: ffffa05142a8be80
> [ 15.394837] x23: 0000000000000000 x22: ffff6f1e08a0dd08 x21: ffff6f1e08a0dc80
> [ 15.398292] x20: ffff6f1e08a0dd30 x19: ffff6f1e08a0dc80 x18: 0000000000000000
> [ 15.401315] x17: 0000000000000000 x16: 0000000000000000 x15: 00000000238ecea0
> [ 15.403655] x14: 0000000000000000 x13: 000000000000dd86 x12: 000000007ffff000
> [ 15.405967] x11: 0000000000000004 x10: 0000000000000000 x9 : ffffa05141003acc
> [ 15.408287] x8 : 0000000000000000 x7 : ffffffffd3039400 x6 : ffff6f1e08a0ddbc
> [ 15.410616] x5 : 0000000000000001 x4 : 0000000000000000 x3 : ffffceccfd9db000
> [ 15.412969] x2 : ffff6f1e02c00810 x1 : 0000000000000180 x0 : 0000000000000180
> [ 15.415332] Call trace:
> [ 15.416187] inet_sock_destruct+0x16c/0x1b0
> [ 15.417551] __sk_destruct+0x38/0x23c
> [ 15.418793] __sk_free+0x80/0x120
> [ 15.419886] sk_free+0x68/0x90
> [ 15.420916] sock_put+0x5c/0x80
> [ 15.421974] tcp_v4_rcv+0xa40/0xaa4
> [ 15.423182] ip_protocol_deliver_rcu+0x4c/0x184
> [ 15.424649] ip_local_deliver_finish+0x74/0x90
> [ 15.426214] ip_local_deliver+0x88/0x130
> [ 15.427869] ip_rcv+0x7c/0x130
> [ 15.428884] __netif_receive_skb_one_core+0x60/0x8c
> [ 15.430545] __netif_receive_skb+0x20/0x70
> [ 15.432679] netif_receive_skb+0x48/0x1e0
> [ 15.435651] tun_get_user+0xbe4/0xd70 [tun]
> [ 15.437662] tun_chr_write_iter+0x68/0xf0 [tun]
> [ 15.439835] do_iter_readv_writev+0x100/0x1a4
> [ 15.441836] do_iter_write+0x98/0x1fc
> [ 15.443588] vfs_writev+0xb4/0x170
> [ 15.445150] do_writev+0x7c/0x140
> [ 15.446713] __arm64_sys_writev+0x2c/0x40
> [ 15.448639] invoke_syscall+0x50/0x120
> [ 15.450373] el0_svc_common.constprop.0+0xf4/0x104
> [ 15.452642] do_el0_svc+0x34/0x9c
> [ 15.454230] el0_svc+0x2c/0x54
> [ 15.455743] el0_sync_handler+0xa4/0x130
> [ 15.457486] el0_sync+0x198/0x1c0
> [ 15.458659] ---[ end trace c9faa1be6c93e4fe ]---
>
> Reported-by: Linux Kernel Functional Testing <lkft@linaro.org>
>
> --
> Linaro LKFT
> https://lkft.linaro.org
^ permalink raw reply [flat|nested] 2+ messages in thread