[sparc64] possible circular locking / deadlock

[Anatoly Pugachev <matorola@gmail.com>]

Hello!

Getting the following git kernel trace on boot with rc.local having :

ipset create sshguard4 hash:net
iptables -A INPUT -p tcp --dport 22 -m set --match-set sshguard4 src -j DROP

current git kernel:

$ uname -a
Linux ttip 5.2.0-rc5 #981 SMP Mon Jun 17 09:52:04 MSK 2019 sparc64 GNU/Linux
linux-2.6$ git desc
v5.2-rc5


$ dmesg
<cut>
[   10.356388] Adding 787176k swap on /dev/vdiska4.  Priority:-2
extents:1 across:787176k FS
[   10.471900] EXT4-fs (vdiska1): mounting ext3 file system using the
ext4 subsystem
[   10.487226] EXT4-fs (vdiska1): mounted filesystem with ordered data
mode. Opts: (null)
[   11.158102] random: crng init done
[   11.158155] random: 7 urandom warning(s) missed due to ratelimiting

[   11.697866] ======================================================
[   11.697875] WARNING: possible circular locking dependency detected
[   11.697886] 5.2.0-rc5 #981 Not tainted
[   11.697894] ------------------------------------------------------
[   11.697902] iptables/732 is trying to acquire lock:
[   11.697913] 000000004f61aa56 (&table[i].mutex){+.+.}, at:
nfnl_lock+0x24/0x40 [nfnetlink]
[   11.697937]
               but task is already holding lock:
[   11.697946] 000000000d652829 (&net->nft.commit_mutex){+.+.}, at:
nf_tables_valid_genid+0x18/0x60 [nf_tables]
[   11.697973]
               which lock already depends on the new lock.

[   11.697983]
               the existing dependency chain (in reverse order) is:
[   11.697992]
               -> #1 (&net->nft.commit_mutex){+.+.}:
[   11.698012]        __mutex_lock+0x48/0x920
[   11.698021]        mutex_lock_nested+0x1c/0x40
[   11.698033]        nf_tables_valid_genid+0x18/0x60 [nf_tables]
[   11.698043]        nfnetlink_rcv_batch+0x24c/0x620 [nfnetlink]
[   11.698053]        nfnetlink_rcv+0x110/0x140 [nfnetlink]
[   11.698067]        netlink_unicast+0x12c/0x1e0
[   11.698076]        netlink_sendmsg+0x324/0x360
[   11.698091]        sock_sendmsg+0x34/0x80
[   11.698099]        ___sys_sendmsg+0x228/0x240
[   11.698108]        __sys_sendmsg+0x4c/0x80
[   11.698116]        sys_sendmsg+0x18/0x40
[   11.698131]        linux_sparc_syscall+0x34/0x44
[   11.698138]
               -> #0 (&table[i].mutex){+.+.}:
[   11.698157]        lock_acquire+0x1a4/0x1c0
[   11.698165]        __mutex_lock+0x48/0x920
[   11.698173]        mutex_lock_nested+0x1c/0x40
[   11.698181]        nfnl_lock+0x24/0x40 [nfnetlink]
[   11.698196]        ip_set_nfnl_get_byindex+0x19c/0x280 [ip_set]
[   11.698207]        set_match_v1_checkentry+0x14/0xc0 [xt_set]
[   11.698222]        xt_check_match+0x238/0x260 [x_tables]
[   11.698234]        __nft_match_init+0x160/0x180 [nft_compat]
[   11.698244]        nft_match_init+0x18/0x40 [nft_compat]
[   11.698256]        nf_tables_newrule+0x57c/0x7a0 [nf_tables]
[   11.698266]        nfnetlink_rcv_batch+0x3f8/0x620 [nfnetlink]
[   11.698275]        nfnetlink_rcv+0x110/0x140 [nfnetlink]
[   11.698284]        netlink_unicast+0x12c/0x1e0
[   11.698292]        netlink_sendmsg+0x324/0x360
[   11.698300]        sock_sendmsg+0x34/0x80
[   11.698309]        ___sys_sendmsg+0x228/0x240
[   11.698317]        __sys_sendmsg+0x4c/0x80
[   11.698325]        sys_sendmsg+0x18/0x40
[   11.698334]        linux_sparc_syscall+0x34/0x44
[   11.698340]
               other info that might help us debug this:

[   11.698351]  Possible unsafe locking scenario:

[   11.698359]        CPU0                    CPU1
[   11.698366]        ----                    ----
[   11.698372]   lock(&net->nft.commit_mutex);
[   11.698381]                                lock(&table[i].mutex);
[   11.698390]                                lock(&net->nft.commit_mutex);
[   11.698400]   lock(&table[i].mutex);
[   11.698408]
                *** DEADLOCK ***

[   11.698418] 1 lock held by iptables/732:
[   11.698424]  #0: 000000000d652829 (&net->nft.commit_mutex){+.+.},
at: nf_tables_valid_genid+0x18/0x60 [nf_tables]
[   11.698444]
               stack backtrace:
[   11.698454] CPU: 6 PID: 732 Comm: iptables Not tainted 5.2.0-rc5 #981
[   11.698463] Call Trace:
[   11.698471]  [00000000004cfde0] print_circular_bug+0x2e0/0x320
[   11.698480]  [00000000004d4bd8] __lock_acquire+0x1d38/0x2900
[   11.698489]  [00000000004d6084] lock_acquire+0x1a4/0x1c0
[   11.698498]  [0000000000a06508] __mutex_lock+0x48/0x920
[   11.698506]  [0000000000a06dfc] mutex_lock_nested+0x1c/0x40
[   11.698516]  [000000001071c024] nfnl_lock+0x24/0x40 [nfnetlink]
[   11.698527]  [00000000107568dc] ip_set_nfnl_get_byindex+0x19c/0x280 [ip_set]
[   11.698537]  [000000001078e5d4] set_match_v1_checkentry+0x14/0xc0 [xt_set]
[   11.698549]  [0000000010310ed8] xt_check_match+0x238/0x260 [x_tables]
[   11.698559]  [000000001077cc00] __nft_match_init+0x160/0x180 [nft_compat]
[   11.698569]  [000000001077ccb8] nft_match_init+0x18/0x40 [nft_compat]
[   11.698582]  [0000000010731c3c] nf_tables_newrule+0x57c/0x7a0 [nf_tables]
[   11.698592]  [000000001071d238] nfnetlink_rcv_batch+0x3f8/0x620 [nfnetlink]
[   11.698602]  [000000001071d570] nfnetlink_rcv+0x110/0x140 [nfnetlink]
[   11.698611]  [000000000093e82c] netlink_unicast+0x12c/0x1e0
[   11.698620]  [000000000093f484] netlink_sendmsg+0x324/0x360



Full kernel configuration file as well full dmesg messages are
available at https://github.com/mator/sparc64-dmesg/

system info:

$ gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/sparc64-linux-gnu/8/lto-wrapper
Target: sparc64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Debian
8.3.0-7' --with-bugurl=file:///usr/share/doc/gcc-8/README.Bugs
--enable-languages=c,ada,c++,go,d,fortran,objc,obj-c++ --prefix=/usr
--with-gcc-major-version-only --program-suffix=-8
--program-prefix=sparc64-linux-gnu- --enable-shared
--enable-linker-build-id --libexecdir=/usr/lib
--without-included-gettext --enable-threads=posix --libdir=/usr/lib
--enable-nls --enable-clocale=gnu --enable-libstdcxx-debug
--enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new
--enable-gnu-unique-object --disable-libquadmath
--disable-libquadmath-support --enable-plugin --enable-default-pie
--with-system-zlib --disable-libphobos --enable-objc-gc=auto
--enable-multiarch --disable-werror --with-cpu-32=ultrasparc
--enable-targets=all --with-long-double-128 --enable-multilib
--enable-checking=release --build=sparc64-linux-gnu
--host=sparc64-linux-gnu --target=sparc64-linux-gnu
Thread model: posix
gcc version 8.3.0 (Debian 8.3.0-7)

# ldconfig -V
ldconfig (Debian GLIBC 2.28-10) 2.28

# ld -V
GNU ld (GNU Binutils for Debian) 2.31.1

PS: i wasn't able to trace which kernel version introduced this
possible deadlock... but tried (from top git tag v5.2-rc1 to bottom)
up to 4.13 kernel version...