* BUG: selftest/net/tun: Hang in unregister_netdevice
@ 2023-03-14 11:45 Mirsad Todorovac
2023-03-14 13:52 ` Mirsad Todorovac
0 siblings, 1 reply; 8+ messages in thread
From: Mirsad Todorovac @ 2023-03-14 11:45 UTC (permalink / raw)
To: netdev, linux-kselftest
Cc: David S. Miller, Jakub Kicinski, Paolo Abeni, Shuah Khan,
Eric Dumazet, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 2792 bytes --]
Hi, all!
After running tools/testing/selftests/net/tun, there seems to be some kind of hang
in test "FAIL tun.reattach_delete_close" or "FAIL tun.reattach_close_delete".
Two tests exit by timeout, but the processes left are unkillable, even with kill -9 PID:
[root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned
root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun
root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun
root 4056 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun
[root@pc-mtodorov linux_torvalds]# kill -9 3930 3952
[root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned
root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun
root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun
root 4060 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun
[root@pc-mtodorov linux_torvalds]#
The kernel seems to be stuck in some loop, and filling the log with the
following messages until reboot, where it is also waiting very long on the
situation to timeout, which apparently never happens.
Mar 14 11:54:09 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 11:54:19 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 11:54:29 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 11:54:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 11:54:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
The platform is kernel 6.3.0-rc2 on AlmaLinux 8.7 and a LENOVO_MT_10TX_BU_Lenovo_FM_V530S-07ICB
(lshw output attached).
The .config is here:
https://domac.alu.hr/~mtodorov/linux/selftests/net-tun/config-6.3.0-rc2-mg-andy-devres-00006-gfc89d7fb499b
Basically, it is a vanilla Torvalds tree kernel with MGLRU, KMEMLEAK, and CONFIG_DEBUG_KOBJECT enabled.
And devres patch.
Please find the strace of the net/tun run attached.
I am available for additional diagnostics.
Hope this helps.
Best regards,
Mirsad
--
Mirsad Goran Todorovac
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb, Republic of Croatia
[-- Attachment #2: tun-log.txt --]
[-- Type: text/plain, Size: 7052 bytes --]
execve("tools/testing/selftests/net/tun", ["tools/testing/selftests/net/tun"], 0x7fff8e683510 /* 46 vars */) = 0
brk(NULL) = 0x11d7000
arch_prctl(0x3001 /* ARCH_??? */, 0x7ffef3ef5740) = -1 EINVAL (Invalid argument)
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=62335, ...}) = 0
mmap(NULL, 62335, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fb12d140000
close(3) = 0
openat(AT_FDCWD, "/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\256\3\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2093744, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fb12d13e000
lseek(3, 808, SEEK_SET) = 808
read(3, "\4\0\0\0\20\0\0\0\5\0\0\0GNU\0\2\0\0\300\4\0\0\0\3\0\0\0\0\0\0\0", 32) = 32
mmap(NULL, 3954880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb12ca00000
mprotect(0x7fb12cbbc000, 2097152, PROT_NONE) = 0
mmap(0x7fb12cdbc000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1bc000) = 0x7fb12cdbc000
mmap(0x7fb12cdc2000, 14528, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fb12cdc2000
close(3) = 0
arch_prctl(ARCH_SET_FS, 0x7fb12d13f500) = 0
mprotect(0x7fb12cdbc000, 16384, PROT_READ) = 0
mprotect(0x604000, 4096, PROT_READ) = 0
mprotect(0x7fb12d02d000, 4096, PROT_READ) = 0
munmap(0x7fb12d140000, 62335) = 0
mmap(NULL, 1024, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_ANONYMOUS, -1, 0) = 0x7fb12d14f000
fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
brk(NULL) = 0x11d7000
brk(0x11f8000) = 0x11f8000
brk(NULL) = 0x11f8000
write(1, "TAP version 13\n1..5\n# Starting 5"..., 103TAP version 13
1..5
# Starting 5 tests from 1 test cases.
# RUN tun.delete_detach_close ...
) = 103
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb12d13f7d0) = 3871
rt_sigaction(SIGALRM, {sa_handler=0x401bf0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fb12ca4eb80}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
alarm(30) = 0
wait4(3871, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 3871
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3871, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
alarm(0) = 26
rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fb12ca4eb80}, NULL, 8) = 0
write(1, "# OK tun.delete_deta"..., 115# OK tun.delete_detach_close
ok 1 tun.delete_detach_close
# RUN tun.detach_delete_close ...
) = 115
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb12d13f7d0) = 3890
rt_sigaction(SIGALRM, {sa_handler=0x401bf0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fb12ca4eb80}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fb12ca4eb80}, 8) = 0
alarm(30) = 0
wait4(3890, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 3890
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3890, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
alarm(0) = 26
rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fb12ca4eb80}, NULL, 8) = 0
write(1, "# OK tun.detach_dele"..., 115# OK tun.detach_delete_close
ok 2 tun.detach_delete_close
# RUN tun.detach_close_delete ...
) = 115
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb12d13f7d0) = 3909
rt_sigaction(SIGALRM, {sa_handler=0x401bf0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fb12ca4eb80}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fb12ca4eb80}, 8) = 0
alarm(30) = 0
wait4(3909, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 3909
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3909, si_uid=0, si_status=0, si_utime=0, si_stime=0} ---
alarm(0) = 26
rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fb12ca4eb80}, NULL, 8) = 0
write(1, "# OK tun.detach_clos"..., 117# OK tun.detach_close_delete
ok 3 tun.detach_close_delete
# RUN tun.reattach_delete_close ...
) = 117
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb12d13f7d0) = 3930
rt_sigaction(SIGALRM, {sa_handler=0x401bf0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fb12ca4eb80}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fb12ca4eb80}, 8) = 0
alarm(30) = 0
wait4(3930, 0x7ffef3ef55cc, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
kill(-3930, SIGKILL) = 0
rt_sigreturn({mask=[]}) = -1 EINTR (Interrupted system call)
alarm(0) = 0
rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fb12ca4eb80}, NULL, 8) = 0
write(2, "# reattach_delete_close: Test te"..., 52# reattach_delete_close: Test terminated by timeout
) = 52
write(1, "# FAIL tun.reattach_de"..., 125# FAIL tun.reattach_delete_close
not ok 4 tun.reattach_delete_close
# RUN tun.reattach_close_delete ...
) = 125
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fb12d13f7d0) = 3952
rt_sigaction(SIGALRM, {sa_handler=0x401bf0, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7fb12ca4eb80}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fb12ca4eb80}, 8) = 0
alarm(30) = 0
wait4(3952, 0x7ffef3ef55cc, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
kill(-3952, SIGKILL) = 0
rt_sigreturn({mask=[]}) = -1 EINTR (Interrupted system call)
alarm(0) = 0
rt_sigaction(SIGALRM, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7fb12ca4eb80}, NULL, 8) = 0
write(2, "# reattach_close_delete: Test te"..., 52# reattach_close_delete: Test terminated by timeout
) = 52
munmap(0x7fb12d14f000, 1024) = 0
write(1, "# FAIL tun.reattach_cl"..., 163# FAIL tun.reattach_close_delete
not ok 5 tun.reattach_close_delete
# FAILED: 3 / 5 tests passed.
# Totals: pass:3 fail:2 xfail:0 xpass:0 skip:0 error:0
) = 163
exit_group(1) = ?
+++ exited with 1 +++
[-- Attachment #3: lshw-lenovo-desktop.txt --]
[-- Type: text/plain, Size: 23889 bytes --]
pc-mtodorov.slava.alu.hr
description: Desktop Computer
product: 10TX000VCR (LENOVO_MT_10TX_BU_Lenovo_FM_V530S-07ICB)
vendor: LENOVO
version: V530S-07ICB
serial: YL004CDC
width: 64 bits
capabilities: smbios-3.2.1 dmi-3.2.1 smp vsyscall32
configuration: administrator_password=disabled boot=normal chassis=desktop family=V530S-07ICB keyboard_password=enabled power-on_password=disabled sku=LENOVO_MT_10TX_BU_Lenovo_FM_V530S-07ICB uuid=4fb93a00-bb35-11e8-bb0a-59599e0a2e00
*-core
description: Motherboard
product: 3140
vendor: LENOVO
physical id: 0
version: SDK0J40697 WIN 3305148204708
slot: Default string
*-firmware
description: BIOS
vendor: LENOVO
physical id: 0
version: M22KT49A
date: 11/10/2022
size: 64KiB
capacity: 16MiB
capabilities: pci upgrade shadowing cdboot bootselect socketedrom edd int13floppy1200 int13floppy720 int13floppy2880 int5printscreen int9keyboard int14serial int17printer acpi usb biosbootspecification uefi
*-memory
description: System Memory
physical id: 3b
slot: System board or motherboard
size: 16GiB
*-bank:0
description: DIMM DDR4 Synchronous 2666 MHz (0.4 ns)
product: RMUA5110ME78HAF-2666
vendor: Fujitsu
physical id: 0
serial: 14903C7C
slot: ChannelA-DIMM1
size: 8GiB
width: 64 bits
clock: 2666MHz (0.4ns)
*-bank:1
description: DIMM DDR4 Synchronous 2666 MHz (0.4 ns)
product: JM2666HLB-8G
vendor: Transcend
physical id: 1
serial: 00002814
slot: ChannelB-DIMM2
size: 8GiB
width: 64 bits
clock: 2666MHz (0.4ns)
*-cache:0
description: L1 cache
physical id: 46
slot: L1 Cache
size: 384KiB
capacity: 384KiB
capabilities: synchronous internal write-back unified
configuration: level=1
*-cache:1
description: L2 cache
physical id: 47
slot: L2 Cache
size: 1536KiB
capacity: 1536KiB
capabilities: synchronous internal write-back unified
configuration: level=2
*-cache:2
description: L3 cache
physical id: 48
slot: L3 Cache
size: 9MiB
capacity: 9MiB
capabilities: synchronous internal write-back unified
configuration: level=3
*-cpu
description: CPU
product: Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz
vendor: Intel Corp.
physical id: 49
bus info: cpu@0
version: 6.158.10
serial: To Be Filled By O.E.M.
slot: U3E1
size: 3856MHz
capacity: 4005MHz
width: 64 bits
clock: 100MHz
capabilities: lm fpu fpu_exception wp vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp x86-64 constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d arch_capabilities cpufreq
configuration: cores=6 enabledcores=6 microcode=240 threads=6
*-pci
description: Host bridge
product: 8th Gen Core Processor Host Bridge/DRAM Registers
vendor: Intel Corporation
physical id: 100
bus info: pci@0000:00:00.0
version: 07
width: 32 bits
clock: 33MHz
configuration: driver=skl_uncore
resources: irq:0
*-display
description: VGA compatible controller
product: CoffeeLake-S GT2 [UHD Graphics 630]
vendor: Intel Corporation
physical id: 2
bus info: pci@0000:00:02.0
logical name: /dev/fb0
version: 00
width: 64 bits
clock: 33MHz
capabilities: pciexpress msi pm vga_controller bus_master cap_list rom fb
configuration: depth=32 driver=i915 latency=0 resolution=1920,1080
resources: irq:124 memory:a0000000-a0ffffff memory:90000000-9fffffff ioport:4000(size=64) memory:c0000-dffff
*-generic:0 UNCLAIMED
description: System peripheral
product: Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
vendor: Intel Corporation
physical id: 8
bus info: pci@0000:00:08.0
version: 00
width: 64 bits
clock: 33MHz
capabilities: msi pm cap_list
configuration: latency=0
resources: memory:a1221000-a1221fff
*-generic:1
description: Signal processing controller
product: Cannon Lake PCH Thermal Controller
vendor: Intel Corporation
physical id: 12
bus info: pci@0000:00:12.0
version: 10
width: 64 bits
clock: 33MHz
capabilities: pm msi cap_list
configuration: driver=intel_pch_thermal latency=0
resources: irq:16 memory:a1220000-a1220fff
*-usb
description: USB controller
product: Cannon Lake PCH USB 3.1 xHCI Host Controller
vendor: Intel Corporation
physical id: 14
bus info: pci@0000:00:14.0
version: 10
width: 64 bits
clock: 33MHz
capabilities: pm msi xhci bus_master cap_list
configuration: driver=xhci_hcd latency=0
resources: irq:121 memory:a1200000-a120ffff
*-usbhost:0
product: xHCI Host Controller
vendor: Linux 6.3.0-rc2-mg-andy-devres-00006-gfc89d7fb499b xhci-hcd
physical id: 0
bus info: usb@1
logical name: usb1
version: 6.03
capabilities: usb-2.00
configuration: driver=hub slots=16 speed=480Mbit/s
*-usb:0
description: Keyboard
product: Lenovo Lenovo Calliope USB Keyboard Consumer Control
vendor: Lenovo
physical id: 1
bus info: usb@1:1
logical name: input3
logical name: /dev/input/event3
logical name: input3::capslock
logical name: input3::numlock
logical name: input3::scrolllock
logical name: input5
logical name: /dev/input/event5
logical name: input6
logical name: /dev/input/event6
logical name: /dev/input/js0
version: 2.18
capabilities: usb-2.00 usb
configuration: driver=usbhid maxpower=100mA speed=2Mbit/s
*-usb:1
description: Mouse
product: PixArt lenovo USB Optical Mouse
vendor: PixArt
physical id: 2
bus info: usb@1:2
logical name: input4
logical name: /dev/input/event4
logical name: /dev/input/mouse0
version: 1.00
capabilities: usb-2.00 usb
configuration: driver=usbhid maxpower=100mA speed=2Mbit/s
*-usb:2
description: MMC Host
product: USB2.0-CRW
vendor: Generic
physical id: 6
bus info: usb@1:6
logical name: mmc0
version: 39.60
serial: 20100201396000000
capabilities: usb-2.00
configuration: driver=rtsx_usb maxpower=500mA speed=480Mbit/s
*-usbhost:1
product: xHCI Host Controller
vendor: Linux 6.3.0-rc2-mg-andy-devres-00006-gfc89d7fb499b xhci-hcd
physical id: 1
bus info: usb@2
logical name: usb2
version: 6.03
capabilities: usb-3.10
configuration: driver=hub slots=6 speed=10000Mbit/s
*-memory UNCLAIMED
description: RAM memory
product: Cannon Lake PCH Shared SRAM
vendor: Intel Corporation
physical id: 14.2
bus info: pci@0000:00:14.2
version: 10
width: 64 bits
clock: 33MHz (30.3ns)
capabilities: pm cap_list
configuration: latency=0
resources: memory:a1216000-a1217fff memory:a121f000-a121ffff
*-serial:0
description: Serial bus controller
product: Cannon Lake PCH Serial IO I2C Controller #0
vendor: Intel Corporation
physical id: 15
bus info: pci@0000:00:15.0
version: 10
width: 64 bits
clock: 33MHz
capabilities: pm bus_master cap_list
configuration: driver=intel-lpss latency=0
resources: irq:16 memory:a1219000-a1219fff
*-serial:1
description: Serial bus controller
product: Cannon Lake PCH Serial IO I2C Controller #1
vendor: Intel Corporation
physical id: 15.1
bus info: pci@0000:00:15.1
version: 10
width: 64 bits
clock: 33MHz
capabilities: pm bus_master cap_list
configuration: driver=intel-lpss latency=0
resources: irq:17 memory:a121d000-a121dfff
*-communication:0
description: Communication controller
product: Cannon Lake PCH HECI Controller
vendor: Intel Corporation
physical id: 16
bus info: pci@0000:00:16.0
version: 10
width: 64 bits
clock: 33MHz
capabilities: pm msi bus_master cap_list
configuration: driver=mei_me latency=0
resources: irq:125 memory:a121c000-a121cfff
*-sata
description: SATA controller
product: Cannon Lake PCH SATA AHCI Controller
vendor: Intel Corporation
physical id: 17
bus info: pci@0000:00:17.0
logical name: scsi0
logical name: scsi1
version: 10
width: 32 bits
clock: 66MHz
capabilities: sata msi pm ahci_1.0 bus_master cap_list emulated
configuration: driver=ahci latency=0
resources: irq:122 memory:a1214000-a1215fff memory:a121b000-a121b0ff ioport:4090(size=8) ioport:4080(size=4) ioport:4060(size=32) memory:a121a000-a121a7ff
*-disk
description: ATA Disk
product: WDC WDS500G2B0A
vendor: Western Digital
physical id: 0
bus info: scsi@0:0.0.0
logical name: /dev/sda
version: 20WD
serial: 21251V800740
size: 465GiB (500GB)
capabilities: gpt-1.00 partitioned partitioned:gpt
configuration: ansiversion=5 guid=337420f3-fbd8-437d-ab9c-1a5f5f14c1d1 logicalsectorsize=512 sectorsize=512
*-volume:0
description: reserved partition
vendor: Windows
physical id: 1
bus info: scsi@0:0.0.0,1
logical name: /dev/sda1
serial: 9e53775b-0aa1-4b97-93da-0e4e1531d755
capacity: 15MiB
capabilities: nofs
configuration: name=Microsoft reserved partition
*-volume:1
description: Windows NTFS volume
vendor: Windows
physical id: 2
bus info: scsi@0:0.0.0,2
logical name: /dev/sda2
logical name: /mnt/win
version: 3.1
serial: e40c9218-ef8c-4e46-947e-2f1c19b65ae3
size: 292GiB
capacity: 292GiB
capabilities: ntfs initialized
configuration: clustersize=4096 created=2021-09-30 16:24:41 filesystem=ntfs label=New Volume mount.fstype=fuseblk mount.options=rw,relatime,user_id=0,group_id=0,allow_other,blksize=4096 name=Basic data partition state=mounted
*-volume:2
description: Windows NTFS volume
vendor: Windows
physical id: 3
bus info: scsi@0:0.0.0,3
logical name: /dev/sda3
version: 3.1
serial: 28ff-3849
size: 488MiB
capacity: 507MiB
capabilities: boot precious ntfs initialized
configuration: clustersize=4096 created=2021-10-20 10:53:48 filesystem=ntfs state=clean
*-volume:3
description: Windows FAT volume
vendor: MSDOS5.0
physical id: 4
bus info: scsi@0:0.0.0,4
logical name: /dev/sda4
logical name: /boot/efi
version: FAT32
serial: 161a-4187
size: 80MiB
capacity: 99MiB
capabilities: boot fat initialized
configuration: FATs=2 filesystem=fat mount.fstype=vfat mount.options=rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=iso8859-1,shortname=winnt,errors=remount-ro name=EFI System Partition state=mounted
*-volume:4
description: EFI partition
physical id: 5
bus info: scsi@0:0.0.0,5
logical name: /dev/sda5
logical name: /boot
serial: 215919e7-bba2-422f-b239-ce4814c65c12
capacity: 1023MiB
configuration: mount.fstype=xfs mount.options=rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota state=mounted
*-volume:5
description: LVM Physical Volume
vendor: Linux
physical id: 6
bus info: scsi@0:0.0.0,6
logical name: /dev/sda6
serial: 5mgsOO-lX9e-wV9r-GPcp-1VBx-UJpx-3uWLde
size: 170GiB
capabilities: multi lvm2
*-cdrom
description: DVD-RAM writer
product: DVDRAM GUE0N
vendor: HL-DT-ST
physical id: 1
bus info: scsi@1:0.0.0
logical name: /dev/cdrom
logical name: /dev/sr0
version: T.02
capabilities: removable audio cd-r cd-rw dvd dvd-r dvd-ram
configuration: ansiversion=5 status=nodisc
*-pci
description: PCI bridge
product: Cannon Lake PCH PCI Express Root Port #7
vendor: Intel Corporation
physical id: 1c
bus info: pci@0000:00:1c.0
version: f0
width: 32 bits
clock: 33MHz
capabilities: pci pciexpress msi pm normal_decode bus_master cap_list
configuration: driver=pcieport
resources: irq:120 ioport:3000(size=4096) memory:a1100000-a11fffff
*-network
description: Ethernet interface
product: RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
vendor: Realtek Semiconductor Co., Ltd.
physical id: 0
bus info: pci@0000:01:00.0
logical name: enp1s0
version: 15
serial: f4:93:9f:f0:a5:f5
size: 1Gbit/s
capacity: 1Gbit/s
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress msix vpd bus_master cap_list ethernet physical tp mii 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
configuration: autonegotiation=on broadcast=yes driver=r8169 driverversion=6.3.0-rc2-mg-andy-devres-00006- duplex=full firmware=rtl8168h-2_0.0.2 02/26/15 ip=193.198.186.200 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
resources: irq:18 ioport:3000(size=256) memory:a1104000-a1104fff memory:a1100000-a1103fff
*-communication:1
description: Communication controller
product: Cannon Lake PCH Serial IO UART Host Controller
vendor: Intel Corporation
physical id: 1e
bus info: pci@0000:00:1e.0
version: 10
width: 64 bits
clock: 33MHz
capabilities: pm bus_master cap_list
configuration: driver=intel-lpss latency=0
resources: irq:20 memory:a121e000-a121efff
*-isa
description: ISA bridge
product: Intel Corporation
vendor: Intel Corporation
physical id: 1f
bus info: pci@0000:00:1f.0
version: 10
width: 32 bits
clock: 33MHz
capabilities: isa bus_master
configuration: latency=0
*-pnp00:00
product: PnP device PNP0c02
physical id: 0
capabilities: pnp
configuration: driver=system
*-pnp00:01
product: PnP device PNP0c02
physical id: 1
capabilities: pnp
configuration: driver=system
*-pnp00:02
product: PnP device PNP0501
physical id: 2
capabilities: pnp
configuration: driver=serial
*-pnp00:03
product: PnP device PNP0c02
physical id: 3
capabilities: pnp
configuration: driver=system
*-pnp00:04
product: PnP device INT3f0d
vendor: Interphase Corporation
physical id: 4
capabilities: pnp
configuration: driver=system
*-pnp00:05
product: PnP device PNP0c02
physical id: 5
capabilities: pnp
configuration: driver=system
*-pnp00:06
product: PnP device PNP0c02
physical id: 6
capabilities: pnp
configuration: driver=system
*-pnp00:07
product: PnP device PNP0c02
physical id: 7
capabilities: pnp
configuration: driver=system
*-pnp00:08
product: PnP device PNP0c02
physical id: 8
capabilities: pnp
configuration: driver=system
*-multimedia
description: Audio device
product: Cannon Lake PCH cAVS
vendor: Intel Corporation
physical id: 1f.3
bus info: pci@0000:00:1f.3
logical name: card0
logical name: /dev/snd/controlC0
logical name: /dev/snd/hwC0D0
logical name: /dev/snd/hwC0D2
logical name: /dev/snd/pcmC0D0c
logical name: /dev/snd/pcmC0D0p
logical name: /dev/snd/pcmC0D3p
logical name: /dev/snd/pcmC0D7p
logical name: /dev/snd/pcmC0D8p
version: 10
width: 64 bits
clock: 33MHz
capabilities: pm msi bus_master cap_list
configuration: driver=snd_hda_intel latency=32
resources: irq:126 memory:a1210000-a1213fff memory:a1000000-a10fffff
*-input:0
product: HDA Intel PCH Front Mic
physical id: 0
logical name: input10
logical name: /dev/input/event10
*-input:1
product: HDA Intel PCH Line Out
physical id: 1
logical name: input11
logical name: /dev/input/event11
*-input:2
product: HDA Intel PCH Front Headphone
physical id: 2
logical name: input12
logical name: /dev/input/event12
*-input:3
product: HDA Intel PCH HDMI/DP,pcm=3
physical id: 3
logical name: input13
logical name: /dev/input/event13
*-input:4
product: HDA Intel PCH HDMI/DP,pcm=7
physical id: 4
logical name: input14
logical name: /dev/input/event14
*-input:5
product: HDA Intel PCH HDMI/DP,pcm=8
physical id: 5
logical name: input15
logical name: /dev/input/event15
*-input:6
product: HDA Intel PCH Mic
physical id: 6
logical name: input9
logical name: /dev/input/event9
*-serial:2
description: SMBus
product: Cannon Lake PCH SMBus Controller
vendor: Intel Corporation
physical id: 1f.4
bus info: pci@0000:00:1f.4
version: 10
width: 64 bits
clock: 33MHz
configuration: driver=i801_smbus latency=0
resources: irq:16 memory:a1218000-a12180ff ioport:efa0(size=32)
*-serial:3 UNCLAIMED
description: Serial bus controller
product: Cannon Lake PCH SPI Controller
vendor: Intel Corporation
physical id: 1f.5
bus info: pci@0000:00:1f.5
version: 10
width: 32 bits
clock: 33MHz
configuration: latency=0
resources: memory:fe010000-fe010fff
*-power UNCLAIMED
description: To Be Filled By O.E.M.
product: To Be Filled By O.E.M.
vendor: To Be Filled By O.E.M.
physical id: 1
version: To Be Filled By O.E.M.
serial: To Be Filled By O.E.M.
capacity: 32768mWh
*-input:0
product: Sleep Button
physical id: 2
logical name: input0
logical name: /dev/input/event0
capabilities: platform
*-input:1
product: Power Button
physical id: 3
logical name: input1
logical name: /dev/input/event1
capabilities: platform
*-input:2
product: Power Button
physical id: 4
logical name: input2
logical name: /dev/input/event2
capabilities: platform
*-input:3
product: Video Bus
physical id: 5
logical name: input7
logical name: /dev/input/event7
capabilities: platform
*-input:4
product: PC Speaker
physical id: 6
logical name: input8
logical name: /dev/input/event8
capabilities: isa
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: BUG: selftest/net/tun: Hang in unregister_netdevice
2023-03-14 11:45 BUG: selftest/net/tun: Hang in unregister_netdevice Mirsad Todorovac
@ 2023-03-14 13:52 ` Mirsad Todorovac
2023-03-14 16:00 ` Mirsad Todorovac
0 siblings, 1 reply; 8+ messages in thread
From: Mirsad Todorovac @ 2023-03-14 13:52 UTC (permalink / raw)
To: netdev, linux-kselftest
Cc: David S. Miller, Jakub Kicinski, Paolo Abeni, Shuah Khan,
Eric Dumazet, linux-kernel
On 3/14/23 12:45, Mirsad Todorovac wrote:
> Hi, all!
>
> After running tools/testing/selftests/net/tun, there seems to be some kind of hang
> in test "FAIL tun.reattach_delete_close" or "FAIL tun.reattach_close_delete".
>
> Two tests exit by timeout, but the processes left are unkillable, even with kill -9 PID:
>
> [root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
> root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned
> root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
> root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun
> root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun
> root 4056 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun
> [root@pc-mtodorov linux_torvalds]# kill -9 3930 3952
> [root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
> root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned
> root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
> root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun
> root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun
> root 4060 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun
> [root@pc-mtodorov linux_torvalds]#
>
> The kernel seems to be stuck in some loop, and filling the log with the
> following messages until reboot, where it is also waiting very long on the
> situation to timeout, which apparently never happens.
>
> Mar 14 11:54:09 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 11:54:19 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 11:54:29 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 11:54:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 11:54:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>
> The platform is kernel 6.3.0-rc2 on AlmaLinux 8.7 and a LENOVO_MT_10TX_BU_Lenovo_FM_V530S-07ICB
> (lshw output attached).
>
> The .config is here:
>
> https://domac.alu.hr/~mtodorov/linux/selftests/net-tun/config-6.3.0-rc2-mg-andy-devres-00006-gfc89d7fb499b
>
> Basically, it is a vanilla Torvalds tree kernel with MGLRU, KMEMLEAK, and CONFIG_DEBUG_KOBJECT enabled.
> And devres patch.
>
> Please find the strace of the net/tun run attached.
>
> I am available for additional diagnostics.
Hi, again!
I've been busy while waiting for reply, so I wondered how would a vanilla kernel
go through the test, considering that I've been testing a number of patches
lately.
I did a fresh git clone from repo and woa.
Surprisingly, the test with CONFIG_DEBUG_KOBJECT turned off passes:
[root@pc-mtodorov linux_torvalds]# tools/testing/selftests/net/tun
TAP version 13
1..5
# Starting 5 tests from 1 test cases.
# RUN tun.delete_detach_close ...
# OK tun.delete_detach_close
ok 1 tun.delete_detach_close
# RUN tun.detach_delete_close ...
# OK tun.detach_delete_close
ok 2 tun.detach_delete_close
# RUN tun.detach_close_delete ...
# OK tun.detach_close_delete
ok 3 tun.detach_close_delete
# RUN tun.reattach_delete_close ...
# OK tun.reattach_delete_close
ok 4 tun.reattach_delete_close
# RUN tun.reattach_close_delete ...
# OK tun.reattach_close_delete
ok 5 tun.reattach_close_delete
# PASSED: 5 / 5 tests passed.
# Totals: pass:5 fail:0 xfail:0 xpass:0 skip:0 error:0
[root@pc-mtodorov linux_torvalds]#
So, no hanging processes that cannot be killed now.
If you think it is worthy to explore the lockup that occurs when turning
CONFIG_DEBUG_KOBJECT=y, I will rebuild once again with these turned on,
to clear any doubts.
Until later.
Best regards,
Mirsad
--
Mirsad Goran Todorovac
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb, Republic of Croatia
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: BUG: selftest/net/tun: Hang in unregister_netdevice
2023-03-14 13:52 ` Mirsad Todorovac
@ 2023-03-14 16:00 ` Mirsad Todorovac
2023-03-14 16:02 ` Eric Dumazet
0 siblings, 1 reply; 8+ messages in thread
From: Mirsad Todorovac @ 2023-03-14 16:00 UTC (permalink / raw)
To: netdev, linux-kselftest
Cc: David S. Miller, Jakub Kicinski, Paolo Abeni, Shuah Khan,
Eric Dumazet, linux-kernel
On 3/14/23 14:52, Mirsad Todorovac wrote:
> On 3/14/23 12:45, Mirsad Todorovac wrote:
>> Hi, all!
>>
>> After running tools/testing/selftests/net/tun, there seems to be some kind of hang
>> in test "FAIL tun.reattach_delete_close" or "FAIL tun.reattach_close_delete".
>>
>> Two tests exit by timeout, but the processes left are unkillable, even with kill -9 PID:
>>
>> [root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
>> root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned
>> root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
>> root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun
>> root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun
>> root 4056 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun
>> [root@pc-mtodorov linux_torvalds]# kill -9 3930 3952
>> [root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
>> root 1140 1 0 12:16 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned
>> root 1333 1 0 12:16 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
>> root 3930 2309 0 12:20 pts/1 00:00:00 tools/testing/selftests/net/tun
>> root 3952 2309 0 12:21 pts/1 00:00:00 tools/testing/selftests/net/tun
>> root 4060 3765 0 12:25 pts/1 00:00:00 grep --color=auto tun
>> [root@pc-mtodorov linux_torvalds]#
>>
>> The kernel seems to be stuck in some loop, and filling the log with the
>> following messages until reboot, where it is also waiting very long on the
>> situation to timeout, which apparently never happens.
>>
>> Mar 14 11:54:09 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>> Mar 14 11:54:19 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>> Mar 14 11:54:29 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>> Mar 14 11:54:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>> Mar 14 11:54:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>>
>> The platform is kernel 6.3.0-rc2 on AlmaLinux 8.7 and a LENOVO_MT_10TX_BU_Lenovo_FM_V530S-07ICB
>> (lshw output attached).
>>
>> The .config is here:
>>
>> https://domac.alu.hr/~mtodorov/linux/selftests/net-tun/config-6.3.0-rc2-mg-andy-devres-00006-gfc89d7fb499b
>>
>> Basically, it is a vanilla Torvalds tree kernel with MGLRU, KMEMLEAK, and CONFIG_DEBUG_KOBJECT enabled.
>> And devres patch.
>>
>> Please find the strace of the net/tun run attached.
>>
>> I am available for additional diagnostics.
>
> Hi, again!
>
> I've been busy while waiting for reply, so I wondered how would a vanilla kernel
> go through the test, considering that I've been testing a number of patches
> lately.
>
> I did a fresh git clone from repo and woa.
>
> Surprisingly, the test with CONFIG_DEBUG_KOBJECT turned off passes:
>
> [root@pc-mtodorov linux_torvalds]# tools/testing/selftests/net/tun
> TAP version 13
> 1..5
> # Starting 5 tests from 1 test cases.
> # RUN tun.delete_detach_close ...
> # OK tun.delete_detach_close
> ok 1 tun.delete_detach_close
> # RUN tun.detach_delete_close ...
> # OK tun.detach_delete_close
> ok 2 tun.detach_delete_close
> # RUN tun.detach_close_delete ...
> # OK tun.detach_close_delete
> ok 3 tun.detach_close_delete
> # RUN tun.reattach_delete_close ...
> # OK tun.reattach_delete_close
> ok 4 tun.reattach_delete_close
> # RUN tun.reattach_close_delete ...
> # OK tun.reattach_close_delete
> ok 5 tun.reattach_close_delete
> # PASSED: 5 / 5 tests passed.
> # Totals: pass:5 fail:0 xfail:0 xpass:0 skip:0 error:0
> [root@pc-mtodorov linux_torvalds]#
>
> So, no hanging processes that cannot be killed now.
>
> If you think it is worthy to explore the lockup that occurs when turning
> CONFIG_DEBUG_KOBJECT=y, I will rebuild once again with these turned on,
> to clear any doubts.
Confirmed.
With the sole difference of:
[marvin@pc-mtodorov linux_torvalds]$ grep KOBJECT /boot/config-6.3.0-rc2-vanilla-00006-gfc89d7fb499b
CONFIG_DEBUG_KOBJECT=y
CONFIG_DEBUG_KOBJECT_RELEASE=y
# CONFIG_SAMPLE_KOBJECT is not set
[marvin@pc-mtodorov linux_torvalds]$
we get again unkillable processes:
[root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
root 1157 1 0 16:44 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned
root 1331 1 0 16:44 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
root 3479 2315 0 16:45 pts/1 00:00:00 tools/testing/selftests/net/tun
root 3512 2315 0 16:45 pts/1 00:00:00 tools/testing/selftests/net/tun
root 4091 3364 0 16:49 pts/1 00:00:00 grep --color=auto tun
[root@pc-mtodorov linux_torvalds]# kill -9 3479 3512
[root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
root 1157 1 0 16:44 ? 00:00:00 /bin/bash /usr/sbin/ksmtuned
root 1331 1 0 16:44 ? 00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
root 3479 2315 0 16:45 pts/1 00:00:00 tools/testing/selftests/net/tun
root 3512 2315 0 16:45 pts/1 00:00:00 tools/testing/selftests/net/tun
root 4095 3364 0 16:50 pts/1 00:00:00 grep --color=auto tun
[root@pc-mtodorov linux_torvalds]#
Possibly the kernel /proc/cmdline is also important:
[root@pc-mtodorov linux_torvalds]# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt5)/vmlinuz-6.3.0-rc2-vanilla-00006-gfc89d7fb499b root=/dev/mapper/almalinux_desktop--mtodorov-root ro
crashkernel=auto resume=/dev/mapper/almalinux_desktop--mtodorov-swap rd.lvm.lv=almalinux_desktop-mtodorov/root
rd.lvm.lv=almalinux_desktop-mtodorov/swap loglevel=7 i915.alpha_support=1 debug devres.log=1
[root@pc-mtodorov linux_torvalds]#
After a while, kernel message start looping:
kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:15 ...
kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:24 ...
kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Message from syslogd@pc-mtodorov at Mar 14 16:57:26 ...
kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
This hangs processes until very late stage of shutdown.
I can confirm that CONFIG_DEBUG_{KOBJECT,KOBJECT_RELEASE}=y were the only changes
to .config in between builds.
Best regards,
Mirsad
--
Mirsad Goran Todorovac
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb, Republic of Croatia
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: BUG: selftest/net/tun: Hang in unregister_netdevice
2023-03-14 16:00 ` Mirsad Todorovac
@ 2023-03-14 16:02 ` Eric Dumazet
2023-03-14 20:10 ` Mirsad Goran Todorovac
0 siblings, 1 reply; 8+ messages in thread
From: Eric Dumazet @ 2023-03-14 16:02 UTC (permalink / raw)
To: Mirsad Todorovac
Cc: netdev, linux-kselftest, David S. Miller, Jakub Kicinski,
Paolo Abeni, Shuah Khan, linux-kernel
On Tue, Mar 14, 2023 at 9:01 AM Mirsad Todorovac
<mirsad.todorovac@alu.unizg.hr> wrote:
> After a while, kernel message start looping:
>
> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>
> Message from syslogd@pc-mtodorov at Mar 14 16:57:15 ...
> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>
> Message from syslogd@pc-mtodorov at Mar 14 16:57:24 ...
> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>
> Message from syslogd@pc-mtodorov at Mar 14 16:57:26 ...
> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>
> This hangs processes until very late stage of shutdown.
>
> I can confirm that CONFIG_DEBUG_{KOBJECT,KOBJECT_RELEASE}=y were the only changes
> to .config in between builds.
>
> Best regards,
> Mirsad
>
Try adding in your config
CONFIG_NET_DEV_REFCNT_TRACKER=y
CONFIG_NET_NS_REFCNT_TRACKER=y
Thanks.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: BUG: selftest/net/tun: Hang in unregister_netdevice
2023-03-14 16:02 ` Eric Dumazet
@ 2023-03-14 20:10 ` Mirsad Goran Todorovac
2023-03-15 20:56 ` Kuniyuki Iwashima
0 siblings, 1 reply; 8+ messages in thread
From: Mirsad Goran Todorovac @ 2023-03-14 20:10 UTC (permalink / raw)
To: Eric Dumazet
Cc: netdev, linux-kselftest, David S. Miller, Jakub Kicinski,
Paolo Abeni, Shuah Khan, linux-kernel
On 14. 03. 2023. 17:02, Eric Dumazet wrote:
> On Tue, Mar 14, 2023 at 9:01 AM Mirsad Todorovac
> <mirsad.todorovac@alu.unizg.hr> wrote:
>
>> After a while, kernel message start looping:
>>
>> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>>
>> Message from syslogd@pc-mtodorov at Mar 14 16:57:15 ...
>> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>>
>> Message from syslogd@pc-mtodorov at Mar 14 16:57:24 ...
>> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>>
>> Message from syslogd@pc-mtodorov at Mar 14 16:57:26 ...
>> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>>
>> This hangs processes until very late stage of shutdown.
>>
>> I can confirm that CONFIG_DEBUG_{KOBJECT,KOBJECT_RELEASE}=y were the only changes
>> to .config in between builds.
>>
>> Best regards,
>> Mirsad
>>
>
> Try adding in your config
>
> CONFIG_NET_DEV_REFCNT_TRACKER=y
> CONFIG_NET_NS_REFCNT_TRACKER=y
>
> Thanks.
Not at all.
According to the info here: https://cateee.net/lkddb/web-lkddb/NET_DEV_REFCNT_TRACKER.html
no kerenel param was needed.
I have got the same hang, and additional debug information appears to be this
(in /var/log/messages):
Mar 14 20:58:20 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:58:20 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:20 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:20 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:20 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:20 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:58:20 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:20 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:20 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:20 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:30 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:58:30 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:30 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:30 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:30 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:30 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:58:30 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:30 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:30 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:30 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:58:40 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:40 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:40 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:40 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:58:40 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:40 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:40 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:40 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:58:50 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:50 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:50 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:50 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:58:50 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:50 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:50 pc-mtodorov kernel: leaked reference.
Mar 14 20:58:50 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:58:57 pc-mtodorov kernel: kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
Mar 14 20:59:00 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:59:00 pc-mtodorov kernel: leaked reference.
Mar 14 20:59:00 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:59:00 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:59:00 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:59:00 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:59:00 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:59:00 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:59:00 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:59:00 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:59:00 pc-mtodorov kernel: leaked reference.
Mar 14 20:59:00 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:59:00 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:59:00 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:59:00 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:59:00 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:59:00 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:59:00 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:59:00 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:59:01 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:59:01 pc-mtodorov kernel: leaked reference.
Mar 14 20:59:01 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:59:01 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:59:01 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:59:01 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:59:01 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:59:01 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:59:01 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:59:01 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:59:01 pc-mtodorov kernel: leaked reference.
Mar 14 20:59:01 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:59:01 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:59:01 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:59:01 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:59:01 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:59:01 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:59:01 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:59:01 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:59:10 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:59:10 pc-mtodorov kernel: leaked reference.
Mar 14 20:59:10 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:59:10 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:59:10 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:59:10 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:59:10 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:59:10 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:59:10 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:59:10 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:59:10 pc-mtodorov kernel: leaked reference.
Mar 14 20:59:10 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:59:10 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:59:10 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:59:10 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:59:10 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:59:10 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:59:10 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:59:10 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:59:11 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
Mar 14 20:59:11 pc-mtodorov kernel: leaked reference.
Mar 14 20:59:11 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
Mar 14 20:59:11 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
Mar 14 20:59:11 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
Mar 14 20:59:11 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:59:11 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:59:11 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:59:11 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:59:11 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Mar 14 20:59:11 pc-mtodorov kernel: leaked reference.
Mar 14 20:59:11 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
Mar 14 20:59:11 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
Mar 14 20:59:11 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
Mar 14 20:59:11 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
Mar 14 20:59:11 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
Mar 14 20:59:11 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
Mar 14 20:59:11 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
Mar 14 20:59:11 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
[root@pc-mtodorov marvin]#
I see those "leaked reference" lines are being printed here:
https://elixir.bootlin.com/linux/v6.3-rc2/source/lib/ref_tracker.c#L55
However, it is beyond the scope of my knowledge to track the actual leak.
Hope this helps.
Best regards,
Mirsad
--
Mirsad Goran Todorovac
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb, Republic of Croatia
The European Union
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: BUG: selftest/net/tun: Hang in unregister_netdevice
2023-03-14 20:10 ` Mirsad Goran Todorovac
@ 2023-03-15 20:56 ` Kuniyuki Iwashima
2023-03-15 20:59 ` Eric Dumazet
0 siblings, 1 reply; 8+ messages in thread
From: Kuniyuki Iwashima @ 2023-03-15 20:56 UTC (permalink / raw)
To: mirsad.todorovac
Cc: davem, edumazet, kuba, linux-kernel, linux-kselftest, netdev,
pabeni, shuah, kuniyu
From: Mirsad Goran Todorovac <mirsad.todorovac@alu.unizg.hr>
Date: Tue, 14 Mar 2023 21:10:55 +0100
> On 14. 03. 2023. 17:02, Eric Dumazet wrote:
> > On Tue, Mar 14, 2023 at 9:01 AM Mirsad Todorovac
> > <mirsad.todorovac@alu.unizg.hr> wrote:
> >
> >> After a while, kernel message start looping:
> >>
> >> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> >>
> >> Message from syslogd@pc-mtodorov at Mar 14 16:57:15 ...
> >> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> >>
> >> Message from syslogd@pc-mtodorov at Mar 14 16:57:24 ...
> >> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> >>
> >> Message from syslogd@pc-mtodorov at Mar 14 16:57:26 ...
> >> kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> >>
> >> This hangs processes until very late stage of shutdown.
> >>
> >> I can confirm that CONFIG_DEBUG_{KOBJECT,KOBJECT_RELEASE}=y were the only changes
> >> to .config in between builds.
KOBJECT_RELEASE delays freeing kobject with delayed_work.
In the failing tests, e.g. reattach_delete_close, we create
a tun dev and do
1. detach (tun_detach())
2. re-attach (tun_attach())
3. close
. We release kobjects (tun->dev->(_tx|_rx).kobj) in 1. and
recycle the same memory region in 2.
But the kobjects are not actually released due to the delay,
thus netdev_queue_add_kobject() and rx_queue_add_kobject()
fail.
---8<---
# RUN tun.reattach_delete_close ...
[ 179.087589] kobject: 'tx-1' (00000000ee182e2f): kobject_release, parent 000000000643eb5f (delayed 3000)
[ 179.088105] kobject: 'rx-1' (000000001c44852d): kobject_release, parent 000000000643eb5f (delayed 1000)
[ 179.089097] kobject (00000000ee182e2f): tried to init an initialized object, something is seriously wrong.
---8<---
However, we don't assume the delay and also the failure in
tun_set_real_num_queues().
In this case, we have to re-initialise the queues without
touching kobjects.
Eric,
Are you working on this?
If not, let me try fixing this :)
Thanks,
Kuniyuki
> >>
> >> Best regards,
> >> Mirsad
> >>
> >
> > Try adding in your config
> >
> > CONFIG_NET_DEV_REFCNT_TRACKER=y
> > CONFIG_NET_NS_REFCNT_TRACKER=y
> >
> > Thanks.
>
> Not at all.
>
> According to the info here: https://cateee.net/lkddb/web-lkddb/NET_DEV_REFCNT_TRACKER.html
> no kerenel param was needed.
>
> I have got the same hang, and additional debug information appears to be this
> (in /var/log/messages):
>
> Mar 14 20:58:20 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:58:20 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:20 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:20 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:20 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:20 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:58:20 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:20 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:20 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:20 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:58:20 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:58:20 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:58:20 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:20 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:20 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:20 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:20 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:30 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:58:30 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:30 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:30 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:30 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:30 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:58:30 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:30 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:30 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:30 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:58:30 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:58:30 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:58:30 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:30 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:30 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:30 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:30 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:58:40 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:40 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:40 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:40 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:58:40 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:40 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:40 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:40 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:58:40 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:58:40 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:58:40 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:40 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:40 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:40 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:40 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:58:50 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:50 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:50 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:50 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:58:50 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:50 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:50 pc-mtodorov kernel: leaked reference.
> Mar 14 20:58:50 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:58:50 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:58:50 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:58:50 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:58:50 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:58:50 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:58:50 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:58:50 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:58:57 pc-mtodorov kernel: kmemleak: 1 new suspected memory leaks (see /sys/kernel/debug/kmemleak)
> Mar 14 20:59:00 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:59:00 pc-mtodorov kernel: leaked reference.
> Mar 14 20:59:00 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:59:00 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:59:00 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:59:00 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:59:00 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:59:00 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:59:00 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:59:00 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:59:00 pc-mtodorov kernel: leaked reference.
> Mar 14 20:59:00 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:59:00 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:59:00 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:59:00 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:59:00 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:59:00 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:59:00 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:59:00 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:59:01 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:59:01 pc-mtodorov kernel: leaked reference.
> Mar 14 20:59:01 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:59:01 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:59:01 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:59:01 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:59:01 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:59:01 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:59:01 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:59:01 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:59:01 pc-mtodorov kernel: leaked reference.
> Mar 14 20:59:01 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:59:01 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:59:01 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:59:01 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:59:01 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:59:01 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:59:01 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:59:01 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:59:10 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:59:10 pc-mtodorov kernel: leaked reference.
> Mar 14 20:59:10 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:59:10 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:59:10 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:59:10 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:59:10 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:59:10 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:59:10 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:59:10 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:59:10 pc-mtodorov kernel: leaked reference.
> Mar 14 20:59:10 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:59:10 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:59:10 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:59:10 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:59:10 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:59:10 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:59:10 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:59:10 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:59:11 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
> Mar 14 20:59:11 pc-mtodorov kernel: leaked reference.
> Mar 14 20:59:11 pc-mtodorov kernel: net_rx_queue_update_kobjects+0x75/0x1d0
> Mar 14 20:59:11 pc-mtodorov kernel: netif_set_real_num_rx_queues+0x5b/0xb0
> Mar 14 20:59:11 pc-mtodorov kernel: tun_attach+0x1ec/0x5a0
> Mar 14 20:59:11 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:59:11 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:59:11 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:59:11 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:59:11 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> Mar 14 20:59:11 pc-mtodorov kernel: leaked reference.
> Mar 14 20:59:11 pc-mtodorov kernel: netdev_queue_update_kobjects+0x86/0x190
> Mar 14 20:59:11 pc-mtodorov kernel: netif_set_real_num_tx_queues+0x86/0x250
> Mar 14 20:59:11 pc-mtodorov kernel: tun_attach+0x1d7/0x5a0
> Mar 14 20:59:11 pc-mtodorov kernel: __tun_chr_ioctl+0xa58/0x17d0
> Mar 14 20:59:11 pc-mtodorov kernel: tun_chr_ioctl+0x17/0x20
> Mar 14 20:59:11 pc-mtodorov kernel: __x64_sys_ioctl+0x97/0xd0
> Mar 14 20:59:11 pc-mtodorov kernel: do_syscall_64+0x5c/0x90
> Mar 14 20:59:11 pc-mtodorov kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
> [root@pc-mtodorov marvin]#
>
> I see those "leaked reference" lines are being printed here:
> https://elixir.bootlin.com/linux/v6.3-rc2/source/lib/ref_tracker.c#L55
>
> However, it is beyond the scope of my knowledge to track the actual leak.
>
> Hope this helps.
>
> Best regards,
> Mirsad
>
> --
> Mirsad Goran Todorovac
> Sistem inženjer
> Grafički fakultet | Akademija likovnih umjetnosti
> Sveučilište u Zagrebu
>
> System engineer
> Faculty of Graphic Arts | Academy of Fine Arts
> University of Zagreb, Republic of Croatia
> The European Union
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: BUG: selftest/net/tun: Hang in unregister_netdevice
2023-03-15 20:56 ` Kuniyuki Iwashima
@ 2023-03-15 20:59 ` Eric Dumazet
2023-03-16 20:28 ` Mirsad Goran Todorovac
0 siblings, 1 reply; 8+ messages in thread
From: Eric Dumazet @ 2023-03-15 20:59 UTC (permalink / raw)
To: Kuniyuki Iwashima
Cc: mirsad.todorovac, davem, kuba, linux-kernel, linux-kselftest,
netdev, pabeni, shuah
On Wed, Mar 15, 2023 at 1:57 PM Kuniyuki Iwashima <kuniyu@amazon.com> wrote:
>
> However, we don't assume the delay and also the failure in
> tun_set_real_num_queues().
>
> In this case, we have to re-initialise the queues without
> touching kobjects.
>
> Eric,
> Are you working on this?
> If not, let me try fixing this :)
I am not working on this, please go ahead, thanks !
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: BUG: selftest/net/tun: Hang in unregister_netdevice
2023-03-15 20:59 ` Eric Dumazet
@ 2023-03-16 20:28 ` Mirsad Goran Todorovac
0 siblings, 0 replies; 8+ messages in thread
From: Mirsad Goran Todorovac @ 2023-03-16 20:28 UTC (permalink / raw)
To: Eric Dumazet, Kuniyuki Iwashima
Cc: davem, kuba, linux-kernel, linux-kselftest, netdev, pabeni, shuah
On 15. 03. 2023. 21:59, Eric Dumazet wrote:
> On Wed, Mar 15, 2023 at 1:57 PM Kuniyuki Iwashima <kuniyu@amazon.com> wrote:
>>
>> However, we don't assume the delay and also the failure in
>> tun_set_real_num_queues().
>>
>> In this case, we have to re-initialise the queues without
>> touching kobjects.
>>
>> Eric,
>> Are you working on this?
>> If not, let me try fixing this :)
>
> I am not working on this, please go ahead, thanks !
Hi,
It's me again. I just have new findings.
[root@pc-mtodorov linux_torvalds]# grep -E '(KOBJECT|TRACKER)' /boot/config-6.3.0-rc2-00006-gfc89d7fb499b
CONFIG_REF_TRACKER=y
CONFIG_NET_DEV_REFCNT_TRACKER=y
CONFIG_NET_NS_REFCNT_TRACKER=y
CONFIG_DEBUG_KOBJECT=y
# CONFIG_DEBUG_KOBJECT_RELEASE is not set
# CONFIG_SAMPLE_KOBJECT is not set
# CONFIG_TEST_REF_TRACKER is not set
[root@pc-mtodorov linux_torvalds]# uname -rms
Linux 6.3.0-rc2-00006-gfc89d7fb499b x86_64
[root@pc-mtodorov linux_torvalds]# grep -E '(KOBJECT|TRACKER)' /boot/config-6.3.0-rc2-00006-gfc89d7fb499b
CONFIG_REF_TRACKER=y
CONFIG_NET_DEV_REFCNT_TRACKER=y
CONFIG_NET_NS_REFCNT_TRACKER=y
CONFIG_DEBUG_KOBJECT=y
# CONFIG_DEBUG_KOBJECT_RELEASE is not set
# CONFIG_SAMPLE_KOBJECT is not set
# CONFIG_TEST_REF_TRACKER is not set
[root@pc-mtodorov linux_torvalds]# tools/testing/selftests/net/tun
TAP version 13
1..5
# Starting 5 tests from 1 test cases.
# RUN tun.delete_detach_close ...
# OK tun.delete_detach_close
ok 1 tun.delete_detach_close
# RUN tun.detach_delete_close ...
# OK tun.detach_delete_close
ok 2 tun.detach_delete_close
# RUN tun.detach_close_delete ...
# OK tun.detach_close_delete
ok 3 tun.detach_close_delete
# RUN tun.reattach_delete_close ...
# OK tun.reattach_delete_close
ok 4 tun.reattach_delete_close
# RUN tun.reattach_close_delete ...
# OK tun.reattach_close_delete
ok 5 tun.reattach_close_delete
# PASSED: 5 / 5 tests passed.
# Totals: pass:5 fail:0 xfail:0 xpass:0 skip:0 error:0
[root@pc-mtodorov linux_torvalds]#
My interpretation if you allow it is that the bug search can be narrowed to the code
that depends on CONFIG_DEBUG_KOBJECT_RELEASE=y.
Best regards,
Mirsad
CONFIG_DEBUG_KOBJECT=y alone doesn't seem to be sufficient to trigger the reference leak.
Hope this helps narrow down the search.
--
Mirsad Goran Todorovac
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu
System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb, Republic of Croatia
The European Union
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2023-03-16 20:28 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-03-14 11:45 BUG: selftest/net/tun: Hang in unregister_netdevice Mirsad Todorovac
2023-03-14 13:52 ` Mirsad Todorovac
2023-03-14 16:00 ` Mirsad Todorovac
2023-03-14 16:02 ` Eric Dumazet
2023-03-14 20:10 ` Mirsad Goran Todorovac
2023-03-15 20:56 ` Kuniyuki Iwashima
2023-03-15 20:59 ` Eric Dumazet
2023-03-16 20:28 ` Mirsad Goran Todorovac
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).