Re: BUG: selftest/net/tun: Hang in unregister_netdevice

[Mirsad Todorovac <mirsad.todorovac@alu.unizg.hr>]

On 3/14/23 14:52, Mirsad Todorovac wrote:
> On 3/14/23 12:45, Mirsad Todorovac wrote:
>> Hi, all!
>>
>> After running tools/testing/selftests/net/tun, there seems to be some kind of hang
>> in test "FAIL  tun.reattach_delete_close" or "FAIL  tun.reattach_close_delete".
>>
>> Two tests exit by timeout, but the processes left are unkillable, even with kill -9 PID:
>>
>> [root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
>> root        1140       1  0 12:16 ?        00:00:00 /bin/bash /usr/sbin/ksmtuned
>> root        1333       1  0 12:16 ?        00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
>> root        3930    2309  0 12:20 pts/1    00:00:00 tools/testing/selftests/net/tun
>> root        3952    2309  0 12:21 pts/1    00:00:00 tools/testing/selftests/net/tun
>> root        4056    3765  0 12:25 pts/1    00:00:00 grep --color=auto tun
>> [root@pc-mtodorov linux_torvalds]# kill -9 3930 3952
>> [root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
>> root        1140       1  0 12:16 ?        00:00:00 /bin/bash /usr/sbin/ksmtuned
>> root        1333       1  0 12:16 ?        00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
>> root        3930    2309  0 12:20 pts/1    00:00:00 tools/testing/selftests/net/tun
>> root        3952    2309  0 12:21 pts/1    00:00:00 tools/testing/selftests/net/tun
>> root        4060    3765  0 12:25 pts/1    00:00:00 grep --color=auto tun
>> [root@pc-mtodorov linux_torvalds]#
>>
>> The kernel seems to be stuck in some loop, and filling the log with the
>> following messages until reboot, where it is also waiting very long on the
>> situation to timeout, which apparently never happens.
>>
>> Mar 14 11:54:09 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>> Mar 14 11:54:19 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>> Mar 14 11:54:29 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>> Mar 14 11:54:40 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>> Mar 14 11:54:50 pc-mtodorov kernel: unregister_netdevice: waiting for tap0 to become free. Usage count = 3
>>
>> The platform is kernel 6.3.0-rc2 on AlmaLinux 8.7 and a LENOVO_MT_10TX_BU_Lenovo_FM_V530S-07ICB
>> (lshw output attached).
>>
>> The .config is here:
>>
>> https://domac.alu.hr/~mtodorov/linux/selftests/net-tun/config-6.3.0-rc2-mg-andy-devres-00006-gfc89d7fb499b
>>
>> Basically, it is a vanilla Torvalds tree kernel with MGLRU, KMEMLEAK, and CONFIG_DEBUG_KOBJECT enabled.
>> And devres patch.
>>
>> Please find the strace of the net/tun run attached.
>>
>> I am available for additional diagnostics.
> 
> Hi, again!
> 
> I've been busy while waiting for reply, so I wondered how would a vanilla kernel
> go through the test, considering that I've been testing a number of patches
> lately.
> 
> I did a fresh git clone from repo and woa.
> 
> Surprisingly, the test with CONFIG_DEBUG_KOBJECT turned off passes:
> 
> [root@pc-mtodorov linux_torvalds]# tools/testing/selftests/net/tun
> TAP version 13
> 1..5
> # Starting 5 tests from 1 test cases.
> #  RUN           tun.delete_detach_close ...
> #            OK  tun.delete_detach_close
> ok 1 tun.delete_detach_close
> #  RUN           tun.detach_delete_close ...
> #            OK  tun.detach_delete_close
> ok 2 tun.detach_delete_close
> #  RUN           tun.detach_close_delete ...
> #            OK  tun.detach_close_delete
> ok 3 tun.detach_close_delete
> #  RUN           tun.reattach_delete_close ...
> #            OK  tun.reattach_delete_close
> ok 4 tun.reattach_delete_close
> #  RUN           tun.reattach_close_delete ...
> #            OK  tun.reattach_close_delete
> ok 5 tun.reattach_close_delete
> # PASSED: 5 / 5 tests passed.
> # Totals: pass:5 fail:0 xfail:0 xpass:0 skip:0 error:0
> [root@pc-mtodorov linux_torvalds]#
> 
> So, no hanging processes that cannot be killed now.
> 
> If you think it is worthy to explore the lockup that occurs when turning
> CONFIG_DEBUG_KOBJECT=y, I will rebuild once again with these turned on,
> to clear any doubts.

Confirmed.

With the sole difference of:

[marvin@pc-mtodorov linux_torvalds]$ grep KOBJECT /boot/config-6.3.0-rc2-vanilla-00006-gfc89d7fb499b
CONFIG_DEBUG_KOBJECT=y
CONFIG_DEBUG_KOBJECT_RELEASE=y
# CONFIG_SAMPLE_KOBJECT is not set
[marvin@pc-mtodorov linux_torvalds]$

we get again unkillable processes:

[root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
root        1157       1  0 16:44 ?        00:00:00 /bin/bash /usr/sbin/ksmtuned
root        1331       1  0 16:44 ?        00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
root        3479    2315  0 16:45 pts/1    00:00:00 tools/testing/selftests/net/tun
root        3512    2315  0 16:45 pts/1    00:00:00 tools/testing/selftests/net/tun
root        4091    3364  0 16:49 pts/1    00:00:00 grep --color=auto tun
[root@pc-mtodorov linux_torvalds]# kill -9 3479 3512
[root@pc-mtodorov linux_torvalds]# ps -ef | grep tun
root        1157       1  0 16:44 ?        00:00:00 /bin/bash /usr/sbin/ksmtuned
root        1331       1  0 16:44 ?        00:00:01 /usr/libexec/platform-python -Es /usr/sbin/tuned -l -P
root        3479    2315  0 16:45 pts/1    00:00:00 tools/testing/selftests/net/tun
root        3512    2315  0 16:45 pts/1    00:00:00 tools/testing/selftests/net/tun
root        4095    3364  0 16:50 pts/1    00:00:00 grep --color=auto tun
[root@pc-mtodorov linux_torvalds]#

Possibly the kernel /proc/cmdline is also important:

[root@pc-mtodorov linux_torvalds]# cat /proc/cmdline
BOOT_IMAGE=(hd0,gpt5)/vmlinuz-6.3.0-rc2-vanilla-00006-gfc89d7fb499b root=/dev/mapper/almalinux_desktop--mtodorov-root ro 
crashkernel=auto resume=/dev/mapper/almalinux_desktop--mtodorov-swap rd.lvm.lv=almalinux_desktop-mtodorov/root 
rd.lvm.lv=almalinux_desktop-mtodorov/swap loglevel=7 i915.alpha_support=1 debug devres.log=1
[root@pc-mtodorov linux_torvalds]#

After a while, kernel message start looping:

  kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3

Message from syslogd@pc-mtodorov at Mar 14 16:57:15 ...
  kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3

Message from syslogd@pc-mtodorov at Mar 14 16:57:24 ...
  kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3

Message from syslogd@pc-mtodorov at Mar 14 16:57:26 ...
  kernel:unregister_netdevice: waiting for tap0 to become free. Usage count = 3

This hangs processes until very late stage of shutdown.

I can confirm that CONFIG_DEBUG_{KOBJECT,KOBJECT_RELEASE}=y were the only changes
to .config in between builds.

Best regards,
Mirsad

-- 
Mirsad Goran Todorovac
Sistem inženjer
Grafički fakultet | Akademija likovnih umjetnosti
Sveučilište u Zagrebu

System engineer
Faculty of Graphic Arts | Academy of Fine Arts
University of Zagreb, Republic of Croatia