* [syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2) @ 2021-05-19 17:35 syzbot 2021-05-19 19:48 ` Dmitry Vyukov 2022-11-08 13:40 ` syzbot 0 siblings, 2 replies; 8+ messages in thread From: syzbot @ 2021-05-19 17:35 UTC (permalink / raw) To: Jason, davem, kuba, linux-kernel, netdev, syzkaller-bugs, wireguard Hello, syzbot found the following issue on: HEAD commit: b81ac784 net: cdc_eem: fix URL to CDC EEM 1.0 spec git tree: net console output: https://syzkaller.appspot.com/x/log.txt?x=15a257c3d00000 kernel config: https://syzkaller.appspot.com/x/.config?x=5b86a12e0d1933b5 dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f Unfortunately, I don't have any reproducer for this issue yet. IMPORTANT: if you fix the issue, please add the following tag to the commit: Reported-by: syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com BUG: MAX_LOCKDEP_KEYS too low! turning off the locking correctness validator. CPU: 0 PID: 5917 Comm: syz-executor.4 Not tainted 5.12.0-syzkaller #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 Call Trace: __dump_stack lib/dump_stack.c:79 [inline] dump_stack+0x141/0x1d7 lib/dump_stack.c:120 register_lock_class.cold+0x14/0x19 kernel/locking/lockdep.c:1281 __lock_acquire+0x102/0x5230 kernel/locking/lockdep.c:4781 lock_acquire kernel/locking/lockdep.c:5512 [inline] lock_acquire+0x1ab/0x740 kernel/locking/lockdep.c:5477 flush_workqueue+0x110/0x13e0 kernel/workqueue.c:2786 drain_workqueue+0x1a5/0x3c0 kernel/workqueue.c:2951 destroy_workqueue+0x71/0x800 kernel/workqueue.c:4382 alloc_workqueue+0xc40/0xef0 kernel/workqueue.c:4343 wg_newlink+0x43d/0x9e0 drivers/net/wireguard/device.c:335 __rtnl_newlink+0x1062/0x1710 net/core/rtnetlink.c:3452 rtnl_newlink+0x64/0xa0 net/core/rtnetlink.c:3500 rtnetlink_rcv_msg+0x44e/0xad0 net/core/rtnetlink.c:5562 netlink_rcv_skb+0x153/0x420 net/netlink/af_netlink.c:2502 netlink_unicast_kernel net/netlink/af_netlink.c:1312 [inline] netlink_unicast+0x533/0x7d0 net/netlink/af_netlink.c:1338 netlink_sendmsg+0x856/0xd90 net/netlink/af_netlink.c:1927 sock_sendmsg_nosec net/socket.c:654 [inline] sock_sendmsg+0xcf/0x120 net/socket.c:674 ____sys_sendmsg+0x6e8/0x810 net/socket.c:2350 ___sys_sendmsg+0xf3/0x170 net/socket.c:2404 __sys_sendmsg+0xe5/0x1b0 net/socket.c:2433 do_syscall_64+0x3a/0xb0 arch/x86/entry/common.c:47 entry_SYSCALL_64_after_hwframe+0x44/0xae RIP: 0033:0x4665d9 Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007fb25febe188 EFLAGS: 00000246 ORIG_RAX: 000000000000002e RAX: ffffffffffffffda RBX: 000000000056c0b0 RCX: 00000000004665d9 RDX: 0000000000000000 RSI: 0000000020000080 RDI: 0000000000000005 RBP: 00000000004bfcb9 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056c0b0 R13: 00007fff30a5021f R14: 00007fb25febe300 R15: 0000000000022000 --- This report is generated by a bot. It may contain errors. See https://goo.gl/tpsmEJ for more information about syzbot. syzbot engineers can be reached at syzkaller@googlegroups.com. syzbot will keep track of this issue. See: https://goo.gl/tpsmEJ#status for how to communicate with syzbot. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2) 2021-05-19 17:35 [syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2) syzbot @ 2021-05-19 19:48 ` Dmitry Vyukov 2021-05-19 19:57 ` Randy Dunlap 2022-11-08 13:40 ` syzbot 1 sibling, 1 reply; 8+ messages in thread From: Dmitry Vyukov @ 2021-05-19 19:48 UTC (permalink / raw) To: syzbot, Peter Zijlstra, Tetsuo Handa Cc: Jason A. Donenfeld, David Miller, Jakub Kicinski, LKML, netdev, syzkaller-bugs, WireGuard mailing list On Wed, May 19, 2021 at 7:35 PM syzbot <syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com> wrote: > > Hello, > > syzbot found the following issue on: > > HEAD commit: b81ac784 net: cdc_eem: fix URL to CDC EEM 1.0 spec > git tree: net > console output: https://syzkaller.appspot.com/x/log.txt?x=15a257c3d00000 > kernel config: https://syzkaller.appspot.com/x/.config?x=5b86a12e0d1933b5 > dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f > > Unfortunately, I don't have any reproducer for this issue yet. > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > Reported-by: syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com > > BUG: MAX_LOCKDEP_KEYS too low! What config controls this? I don't see "MAX_LOCKDEP_KEYS too low" in any of the config descriptions... Here is what syzbot used: CONFIG_LOCKDEP=y CONFIG_LOCKDEP_BITS=16 CONFIG_LOCKDEP_CHAINS_BITS=17 CONFIG_LOCKDEP_STACK_TRACE_BITS=20 CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14 CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12 We already bumped most of these. The log contains dump of the lockdep debug files, is there any offender? Also looking at the log I noticed a memory safety bug in lockdep implementation: [ 2023.605505][ T6807] ================================================================== [ 2023.613589][ T6807] BUG: KASAN: global-out-of-bounds in print_name+0x1b0/0x1d0 [ 2023.624553][ T6807] Read of size 8 at addr ffffffff90225cb0 by task cat/6807 [ 2023.631765][ T6807] [ 2023.634096][ T6807] CPU: 1 PID: 6807 Comm: cat Not tainted 5.12.0-syzkaller #0 [ 2023.641488][ T6807] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 [ 2023.651745][ T6807] Call Trace: [ 2023.655031][ T6807] dump_stack+0x141/0x1d7 [ 2023.659375][ T6807] ? print_name+0x1b0/0x1d0 [ 2023.663890][ T6807] print_address_description.constprop.0.cold+0x5/0x2f8 [ 2023.670895][ T6807] ? print_name+0x1b0/0x1d0 [ 2023.675413][ T6807] ? print_name+0x1b0/0x1d0 [ 2023.679948][ T6807] kasan_report.cold+0x7c/0xd8 [ 2023.684725][ T6807] ? print_name+0x1b0/0x1d0 [ 2023.689248][ T6807] print_name+0x1b0/0x1d0 [ 2023.694196][ T6807] ? lockdep_stats_show+0xa20/0xa20 [ 2023.699940][ T6807] ? seq_file_path+0x30/0x30 [ 2023.704721][ T6807] ? mutex_lock_io_nested+0xf70/0xf70 [ 2023.710118][ T6807] ? lock_acquire+0x58a/0x740 [ 2023.715156][ T6807] ? kasan_unpoison+0x3c/0x60 [ 2023.719843][ T6807] lc_show+0x10a/0x210 [ 2023.723924][ T6807] seq_read_iter+0xb66/0x1220 [ 2023.728617][ T6807] proc_reg_read_iter+0x1fb/0x2d0 [ 2023.733651][ T6807] new_sync_read+0x41e/0x6e0 [ 2023.738272][ T6807] ? ksys_lseek+0x1b0/0x1b0 [ 2023.742784][ T6807] ? lock_acquire+0x58a/0x740 [ 2023.747563][ T6807] vfs_read+0x35c/0x570 [ 2023.751737][ T6807] ksys_read+0x12d/0x250 [ 2023.756003][ T6807] ? vfs_write+0xa30/0xa30 [ 2023.760429][ T6807] ? syscall_enter_from_user_mode+0x27/0x70 [ 2023.766335][ T6807] do_syscall_64+0x3a/0xb0 [ 2023.770764][ T6807] entry_SYSCALL_64_after_hwframe+0x44/0xae [ 2023.776685][ T6807] RIP: 0033:0x7f99856e2910 [ 2023.781104][ T6807] Code: b6 fe ff ff 48 8d 3d 0f be 08 00 48 83 ec 08 e8 06 db 01 00 66 0f 1f 44 00 00 83 3d f9 2d 2c 00 00 75 10 b8 00 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 31 c3 48 83 ec 08 e8 de 9b 01 00 48 89 04 24 [ 2023.800719][ T6807] RSP: 002b:00007ffee7328628 EFLAGS: 00000246 ORIG_RAX: 0000000000000000 [ 2023.809169][ T6807] RAX: ffffffffffffffda RBX: 0000000000020000 RCX: 00007f99856e2910 [ 2023.817150][ T6807] RDX: 0000000000020000 RSI: 0000564290b2a000 RDI: 0000000000000003 [ 2023.825123][ T6807] RBP: 0000564290b2a000 R08: 0000000000000003 R09: 0000000000021010 [ 2023.833107][ T6807] R10: 0000000000000002 R11: 0000000000000246 R12: 0000564290b2a000 [ 2023.841091][ T6807] R13: 0000000000000003 R14: 0000000000020000 R15: 0000000000001000 [ 2023.849074][ T6807] [ 2023.851408][ T6807] The buggy address belongs to the variable: [ 2023.857388][ T6807] lock_classes_in_use+0x410/0x420 [ 2023.862510][ T6807] [ 2023.864826][ T6807] Memory state around the buggy address: [ 2023.870450][ T6807] ffffffff90225b80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2023.878511][ T6807] ffffffff90225c00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2023.886572][ T6807] >ffffffff90225c80: 00 00 00 00 f9 f9 f9 f9 00 00 00 00 00 00 00 00 [ 2023.894628][ T6807] ^ [ 2023.900256][ T6807] ffffffff90225d00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2023.908317][ T6807] ffffffff90225d80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [ 2023.916377][ T6807] ================================================================== > turning off the locking correctness validator. > CPU: 0 PID: 5917 Comm: syz-executor.4 Not tainted 5.12.0-syzkaller #0 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011 > Call Trace: > __dump_stack lib/dump_stack.c:79 [inline] > dump_stack+0x141/0x1d7 lib/dump_stack.c:120 > register_lock_class.cold+0x14/0x19 kernel/locking/lockdep.c:1281 > __lock_acquire+0x102/0x5230 kernel/locking/lockdep.c:4781 > lock_acquire kernel/locking/lockdep.c:5512 [inline] > lock_acquire+0x1ab/0x740 kernel/locking/lockdep.c:5477 > flush_workqueue+0x110/0x13e0 kernel/workqueue.c:2786 > drain_workqueue+0x1a5/0x3c0 kernel/workqueue.c:2951 > destroy_workqueue+0x71/0x800 kernel/workqueue.c:4382 > alloc_workqueue+0xc40/0xef0 kernel/workqueue.c:4343 > wg_newlink+0x43d/0x9e0 drivers/net/wireguard/device.c:335 > __rtnl_newlink+0x1062/0x1710 net/core/rtnetlink.c:3452 > rtnl_newlink+0x64/0xa0 net/core/rtnetlink.c:3500 > rtnetlink_rcv_msg+0x44e/0xad0 net/core/rtnetlink.c:5562 > netlink_rcv_skb+0x153/0x420 net/netlink/af_netlink.c:2502 > netlink_unicast_kernel net/netlink/af_netlink.c:1312 [inline] > netlink_unicast+0x533/0x7d0 net/netlink/af_netlink.c:1338 > netlink_sendmsg+0x856/0xd90 net/netlink/af_netlink.c:1927 > sock_sendmsg_nosec net/socket.c:654 [inline] > sock_sendmsg+0xcf/0x120 net/socket.c:674 > ____sys_sendmsg+0x6e8/0x810 net/socket.c:2350 > ___sys_sendmsg+0xf3/0x170 net/socket.c:2404 > __sys_sendmsg+0xe5/0x1b0 net/socket.c:2433 > do_syscall_64+0x3a/0xb0 arch/x86/entry/common.c:47 > entry_SYSCALL_64_after_hwframe+0x44/0xae > RIP: 0033:0x4665d9 > Code: ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 bc ff ff ff f7 d8 64 89 01 48 > RSP: 002b:00007fb25febe188 EFLAGS: 00000246 ORIG_RAX: 000000000000002e > RAX: ffffffffffffffda RBX: 000000000056c0b0 RCX: 00000000004665d9 > RDX: 0000000000000000 RSI: 0000000020000080 RDI: 0000000000000005 > RBP: 00000000004bfcb9 R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000000000000 R11: 0000000000000246 R12: 000000000056c0b0 > R13: 00007fff30a5021f R14: 00007fb25febe300 R15: 0000000000022000 > > > --- > This report is generated by a bot. It may contain errors. > See https://goo.gl/tpsmEJ for more information about syzbot. > syzbot engineers can be reached at syzkaller@googlegroups.com. > > syzbot will keep track of this issue. See: > https://goo.gl/tpsmEJ#status for how to communicate with syzbot. > > -- > You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group. > To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/0000000000003687bd05c2b2401d%40google.com. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2) 2021-05-19 19:48 ` Dmitry Vyukov @ 2021-05-19 19:57 ` Randy Dunlap 2021-05-19 20:09 ` Dmitry Vyukov 0 siblings, 1 reply; 8+ messages in thread From: Randy Dunlap @ 2021-05-19 19:57 UTC (permalink / raw) To: Dmitry Vyukov, syzbot, Peter Zijlstra, Tetsuo Handa Cc: Jason A. Donenfeld, David Miller, Jakub Kicinski, LKML, netdev, syzkaller-bugs, WireGuard mailing list On 5/19/21 12:48 PM, Dmitry Vyukov wrote: > On Wed, May 19, 2021 at 7:35 PM syzbot > <syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com> wrote: >> >> Hello, >> >> syzbot found the following issue on: >> >> HEAD commit: b81ac784 net: cdc_eem: fix URL to CDC EEM 1.0 spec >> git tree: net >> console output: https://syzkaller.appspot.com/x/log.txt?x=15a257c3d00000 >> kernel config: https://syzkaller.appspot.com/x/.config?x=5b86a12e0d1933b5 >> dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f >> >> Unfortunately, I don't have any reproducer for this issue yet. >> >> IMPORTANT: if you fix the issue, please add the following tag to the commit: >> Reported-by: syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com >> >> BUG: MAX_LOCKDEP_KEYS too low! > include/linux/lockdep.h #define MAX_LOCKDEP_KEYS_BITS 13 #define MAX_LOCKDEP_KEYS (1UL << MAX_LOCKDEP_KEYS_BITS) Documentation/locking/lockdep-design.rst: Troubleshooting: ---------------- The validator tracks a maximum of MAX_LOCKDEP_KEYS number of lock classes. Exceeding this number will trigger the following lockdep warning:: (DEBUG_LOCKS_WARN_ON(id >= MAX_LOCKDEP_KEYS)) By default, MAX_LOCKDEP_KEYS is currently set to 8191, and typical desktop systems have less than 1,000 lock classes, so this warning normally results from lock-class leakage or failure to properly initialize locks. These two problems are illustrated below: > > What config controls this? I don't see "MAX_LOCKDEP_KEYS too low" in > any of the config descriptions... > Here is what syzbot used: > > CONFIG_LOCKDEP=y > CONFIG_LOCKDEP_BITS=16 > CONFIG_LOCKDEP_CHAINS_BITS=17 > CONFIG_LOCKDEP_STACK_TRACE_BITS=20 > CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14 > CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12 > > We already bumped most of these. > The log contains dump of the lockdep debug files, is there any offender? > > Also looking at the log I noticed a memory safety bug in lockdep implementation: ... -- ~Randy ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2) 2021-05-19 19:57 ` Randy Dunlap @ 2021-05-19 20:09 ` Dmitry Vyukov 2021-05-20 5:02 ` Tetsuo Handa 0 siblings, 1 reply; 8+ messages in thread From: Dmitry Vyukov @ 2021-05-19 20:09 UTC (permalink / raw) To: Randy Dunlap Cc: syzbot, Peter Zijlstra, Tetsuo Handa, Jason A. Donenfeld, David Miller, Jakub Kicinski, LKML, netdev, syzkaller-bugs, WireGuard mailing list On Wed, May 19, 2021 at 9:58 PM Randy Dunlap <rdunlap@infradead.org> wrote: > > On 5/19/21 12:48 PM, Dmitry Vyukov wrote: > > On Wed, May 19, 2021 at 7:35 PM syzbot > > <syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com> wrote: > >> > >> Hello, > >> > >> syzbot found the following issue on: > >> > >> HEAD commit: b81ac784 net: cdc_eem: fix URL to CDC EEM 1.0 spec > >> git tree: net > >> console output: https://syzkaller.appspot.com/x/log.txt?x=15a257c3d00000 > >> kernel config: https://syzkaller.appspot.com/x/.config?x=5b86a12e0d1933b5 > >> dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f > >> > >> Unfortunately, I don't have any reproducer for this issue yet. > >> > >> IMPORTANT: if you fix the issue, please add the following tag to the commit: > >> Reported-by: syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com > >> > >> BUG: MAX_LOCKDEP_KEYS too low! > > > > include/linux/lockdep.h > > #define MAX_LOCKDEP_KEYS_BITS 13 > #define MAX_LOCKDEP_KEYS (1UL << MAX_LOCKDEP_KEYS_BITS) Ouch, so it's not configurable yet :( Unless, of course, we identify the offender that produced thousands of lock classes in the log and fix it. > Documentation/locking/lockdep-design.rst: > > Troubleshooting: > ---------------- > > The validator tracks a maximum of MAX_LOCKDEP_KEYS number of lock classes. > Exceeding this number will trigger the following lockdep warning:: > > (DEBUG_LOCKS_WARN_ON(id >= MAX_LOCKDEP_KEYS)) > > By default, MAX_LOCKDEP_KEYS is currently set to 8191, and typical > desktop systems have less than 1,000 lock classes, so this warning > normally results from lock-class leakage or failure to properly > initialize locks. These two problems are illustrated below: > > > > > What config controls this? I don't see "MAX_LOCKDEP_KEYS too low" in > > any of the config descriptions... > > Here is what syzbot used: > > > > CONFIG_LOCKDEP=y > > CONFIG_LOCKDEP_BITS=16 > > CONFIG_LOCKDEP_CHAINS_BITS=17 > > CONFIG_LOCKDEP_STACK_TRACE_BITS=20 > > CONFIG_LOCKDEP_STACK_TRACE_HASH_BITS=14 > > CONFIG_LOCKDEP_CIRCULAR_QUEUE_BITS=12 > > > > We already bumped most of these. > > The log contains dump of the lockdep debug files, is there any offender? > > > > Also looking at the log I noticed a memory safety bug in lockdep implementation: > > ... > > -- > ~Randy > > -- > You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group. > To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com. > To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/c545268c-fe62-883c-4c46-974b3bb3cea1%40infradead.org. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2) 2021-05-19 20:09 ` Dmitry Vyukov @ 2021-05-20 5:02 ` Tetsuo Handa 2021-05-21 7:04 ` Dmitry Vyukov 0 siblings, 1 reply; 8+ messages in thread From: Tetsuo Handa @ 2021-05-20 5:02 UTC (permalink / raw) To: Dmitry Vyukov, Randy Dunlap, David Miller Cc: syzbot, Peter Zijlstra, Jason A. Donenfeld, Jakub Kicinski, LKML, netdev, syzkaller-bugs, WireGuard mailing list On 2021/05/20 5:09, Dmitry Vyukov wrote: > On Wed, May 19, 2021 at 9:58 PM Randy Dunlap <rdunlap@infradead.org> wrote: >> >> On 5/19/21 12:48 PM, Dmitry Vyukov wrote: >>> On Wed, May 19, 2021 at 7:35 PM syzbot >>> <syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com> wrote: >>>> >>>> Hello, >>>> >>>> syzbot found the following issue on: >>>> >>>> HEAD commit: b81ac784 net: cdc_eem: fix URL to CDC EEM 1.0 spec >>>> git tree: net >>>> console output: https://syzkaller.appspot.com/x/log.txt?x=15a257c3d00000 >>>> kernel config: https://syzkaller.appspot.com/x/.config?x=5b86a12e0d1933b5 >>>> dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f >>>> >>>> Unfortunately, I don't have any reproducer for this issue yet. >>>> >>>> IMPORTANT: if you fix the issue, please add the following tag to the commit: >>>> Reported-by: syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com >>>> >>>> BUG: MAX_LOCKDEP_KEYS too low! >>> >> >> include/linux/lockdep.h >> >> #define MAX_LOCKDEP_KEYS_BITS 13 >> #define MAX_LOCKDEP_KEYS (1UL << MAX_LOCKDEP_KEYS_BITS) > > Ouch, so it's not configurable yet :( I didn't try to make this value configurable, for > Unless, of course, we identify the offender that produced thousands of > lock classes in the log and fix it. number of currently active locks should decrease over time. If this message is printed, increasing this value unlikely helps. We have https://lkml.kernel.org/r/c099ad52-0c2c-b886-bae2-c64bd8626452@ozlabs.ru which seems to be unresolved. Regarding this report, cleanup of bonding device is too slow to catch up to creation of bonding device? We might need to throttle creation of BPF, bonding etc. which involve WQ operation for clean up ? ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2) 2021-05-20 5:02 ` Tetsuo Handa @ 2021-05-21 7:04 ` Dmitry Vyukov 0 siblings, 0 replies; 8+ messages in thread From: Dmitry Vyukov @ 2021-05-21 7:04 UTC (permalink / raw) To: Tetsuo Handa Cc: Randy Dunlap, David Miller, syzbot, Peter Zijlstra, Jason A. Donenfeld, Jakub Kicinski, LKML, netdev, syzkaller-bugs, WireGuard mailing list On Thu, May 20, 2021 at 7:02 AM Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> wrote: > > On 2021/05/20 5:09, Dmitry Vyukov wrote: > > On Wed, May 19, 2021 at 9:58 PM Randy Dunlap <rdunlap@infradead.org> wrote: > >> > >> On 5/19/21 12:48 PM, Dmitry Vyukov wrote: > >>> On Wed, May 19, 2021 at 7:35 PM syzbot > >>> <syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com> wrote: > >>>> > >>>> Hello, > >>>> > >>>> syzbot found the following issue on: > >>>> > >>>> HEAD commit: b81ac784 net: cdc_eem: fix URL to CDC EEM 1.0 spec > >>>> git tree: net > >>>> console output: https://syzkaller.appspot.com/x/log.txt?x=15a257c3d00000 > >>>> kernel config: https://syzkaller.appspot.com/x/.config?x=5b86a12e0d1933b5 > >>>> dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f > >>>> > >>>> Unfortunately, I don't have any reproducer for this issue yet. > >>>> > >>>> IMPORTANT: if you fix the issue, please add the following tag to the commit: > >>>> Reported-by: syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com > >>>> > >>>> BUG: MAX_LOCKDEP_KEYS too low! > >>> > >> > >> include/linux/lockdep.h > >> > >> #define MAX_LOCKDEP_KEYS_BITS 13 > >> #define MAX_LOCKDEP_KEYS (1UL << MAX_LOCKDEP_KEYS_BITS) > > > > Ouch, so it's not configurable yet :( > > I didn't try to make this value configurable, for > > > Unless, of course, we identify the offender that produced thousands of > > lock classes in the log and fix it. > > number of currently active locks should decrease over time. > If this message is printed, increasing this value unlikely helps. > > We have https://lkml.kernel.org/r/c099ad52-0c2c-b886-bae2-c64bd8626452@ozlabs.ru > which seems to be unresolved. > > Regarding this report, cleanup of bonding device is too slow to catch up to > creation of bonding device? > > We might need to throttle creation of BPF, bonding etc. which involve WQ operation for clean up? I see, thanks for digging into it. Unbounded asynchronous queueing is always a recipe for disaster... I assume such issues can affect production as well, if some program creates namespaces/devices in a loop. So I think ideally such things are throttled/restricted in the kernel, e.g. new namespaces/devices are not created if some threshold is reached. Potentially syzkaller could throttle creation of new namespaces/devices if we find a good and reliable way to monitor backlog. Something like the length of a particular workqueue. It may also help with OOMs. But so far I haven't found it. ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2) 2021-05-19 17:35 [syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2) syzbot 2021-05-19 19:48 ` Dmitry Vyukov @ 2022-11-08 13:40 ` syzbot 2022-11-08 18:36 ` Dmitry Vyukov 1 sibling, 1 reply; 8+ messages in thread From: syzbot @ 2022-11-08 13:40 UTC (permalink / raw) To: Jason, davem, dvyukov, edumazet, jiri, kuba, linux-kernel, netdev, pabeni, penguin-kernel, peterz, rdunlap, syzkaller-bugs, tonymarislogistics, wireguard syzbot has found a reproducer for the following issue on: HEAD commit: 3577a7611842 Merge branches 'for-next/acpi', 'for-next/kbu.. git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci console output: https://syzkaller.appspot.com/x/log.txt?x=15ea3e61880000 kernel config: https://syzkaller.appspot.com/x/.config?x=606e57fd25c5c6cc dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2 userspace arch: arm64 syz repro: https://syzkaller.appspot.com/x/repro.syz?x=168c4c99880000 C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145d6376880000 Downloadable assets: disk image: https://storage.googleapis.com/syzbot-assets/054b1f56af52/disk-3577a761.raw.xz vmlinux: https://storage.googleapis.com/syzbot-assets/c616835b2a22/vmlinux-3577a761.xz kernel image: https://storage.googleapis.com/syzbot-assets/9825c28b2090/Image-3577a761.gz.xz IMPORTANT: if you fix the issue, please add the following tag to the commit: Reported-by: syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com netlink: 4 bytes leftover after parsing attributes in process `syz-executor357'. device team6635 entered promiscuous mode 8021q: adding VLAN 0 to HW filter on device team6635 BUG: MAX_LOCKDEP_KEYS too low! turning off the locking correctness validator. CPU: 0 PID: 9692 Comm: syz-executor357 Not tainted 6.1.0-rc4-syzkaller-31844-g3577a7611842 #0 Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/30/2022 Call trace: dump_backtrace+0x1c4/0x1f0 arch/arm64/kernel/stacktrace.c:156 show_stack+0x2c/0x54 arch/arm64/kernel/stacktrace.c:163 __dump_stack lib/dump_stack.c:88 [inline] dump_stack_lvl+0x104/0x16c lib/dump_stack.c:106 dump_stack+0x1c/0x58 lib/dump_stack.c:113 register_lock_class+0x2e4/0x2f8 kernel/locking/lockdep.c:1326 __lock_acquire+0xa8/0x3084 kernel/locking/lockdep.c:4934 lock_acquire+0x100/0x1f8 kernel/locking/lockdep.c:5668 __mutex_lock_common+0xd4/0xca8 kernel/locking/mutex.c:603 __mutex_lock kernel/locking/mutex.c:747 [inline] mutex_lock_nested+0x38/0x44 kernel/locking/mutex.c:799 team_vlan_rx_add_vid+0x38/0xd8 drivers/net/team/team.c:1904 vlan_add_rx_filter_info net/8021q/vlan_core.c:211 [inline] __vlan_vid_add net/8021q/vlan_core.c:306 [inline] vlan_vid_add+0x328/0x38c net/8021q/vlan_core.c:336 vlan_device_event+0x200/0xc4c net/8021q/vlan.c:385 notifier_call_chain kernel/notifier.c:87 [inline] raw_notifier_call_chain+0x7c/0x108 kernel/notifier.c:455 __dev_notify_flags+0x170/0x2e8 rtnl_newlink_create+0x460/0x6bc net/core/rtnetlink.c:3372 __rtnl_newlink net/core/rtnetlink.c:3581 [inline] rtnl_newlink+0x728/0xa04 net/core/rtnetlink.c:3594 rtnetlink_rcv_msg+0x484/0x82c net/core/rtnetlink.c:6091 netlink_rcv_skb+0xe8/0x1d4 net/netlink/af_netlink.c:2540 rtnetlink_rcv+0x28/0x38 net/core/rtnetlink.c:6109 netlink_unicast_kernel+0xfc/0x1dc net/netlink/af_netlink.c:1319 netlink_unicast+0x164/0x248 net/netlink/af_netlink.c:1345 netlink_sendmsg+0x484/0x584 net/netlink/af_netlink.c:1921 sock_sendmsg_nosec net/socket.c:714 [inline] sock_sendmsg net/socket.c:734 [inline] ____sys_sendmsg+0x2f8/0x440 net/socket.c:2482 ___sys_sendmsg net/socket.c:2536 [inline] __sys_sendmsg+0x1ac/0x228 net/socket.c:2565 __do_sys_sendmsg net/socket.c:2574 [inline] __se_sys_sendmsg net/socket.c:2572 [inline] __arm64_sys_sendmsg+0x2c/0x3c net/socket.c:2572 __invoke_syscall arch/arm64/kernel/syscall.c:38 [inline] invoke_syscall arch/arm64/kernel/syscall.c:52 [inline] el0_svc_common+0x138/0x220 arch/arm64/kernel/syscall.c:142 do_el0_svc+0x48/0x164 arch/arm64/kernel/syscall.c:206 el0_svc+0x58/0x150 arch/arm64/kernel/entry-common.c:637 el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655 el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:581 ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2) 2022-11-08 13:40 ` syzbot @ 2022-11-08 18:36 ` Dmitry Vyukov 0 siblings, 0 replies; 8+ messages in thread From: Dmitry Vyukov @ 2022-11-08 18:36 UTC (permalink / raw) To: syzbot Cc: Jason, davem, edumazet, jiri, kuba, linux-kernel, netdev, pabeni, penguin-kernel, peterz, rdunlap, syzkaller-bugs, tonymarislogistics, wireguard On Tue, 8 Nov 2022 at 05:40, syzbot <syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com> wrote: > > syzbot has found a reproducer for the following issue on: > > HEAD commit: 3577a7611842 Merge branches 'for-next/acpi', 'for-next/kbu.. > git tree: git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git for-kernelci > console output: https://syzkaller.appspot.com/x/log.txt?x=15ea3e61880000 > kernel config: https://syzkaller.appspot.com/x/.config?x=606e57fd25c5c6cc > dashboard link: https://syzkaller.appspot.com/bug?extid=a70a6358abd2c3f9550f > compiler: Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2 > userspace arch: arm64 > syz repro: https://syzkaller.appspot.com/x/repro.syz?x=168c4c99880000 > C reproducer: https://syzkaller.appspot.com/x/repro.c?x=145d6376880000 > > Downloadable assets: > disk image: https://storage.googleapis.com/syzbot-assets/054b1f56af52/disk-3577a761.raw.xz > vmlinux: https://storage.googleapis.com/syzbot-assets/c616835b2a22/vmlinux-3577a761.xz > kernel image: https://storage.googleapis.com/syzbot-assets/9825c28b2090/Image-3577a761.gz.xz > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > Reported-by: syzbot+a70a6358abd2c3f9550f@syzkaller.appspotmail.com I see team device code uses lockdep_register_key()/lockdep_set_class() in some interesting ways: https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/tree/drivers/net/team/team.c?id=3577a76118426e4409ecf4f75520925eff67d42c#n1657 https://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux.git/tree/drivers/net/team/team.c?id=3577a76118426e4409ecf4f75520925eff67d42c#n2008 I guess that's what causes lockdep capacity exhaustion. Does team use it incorrectly? > netlink: 4 bytes leftover after parsing attributes in process `syz-executor357'. > device team6635 entered promiscuous mode > 8021q: adding VLAN 0 to HW filter on device team6635 > BUG: MAX_LOCKDEP_KEYS too low! > turning off the locking correctness validator. > CPU: 0 PID: 9692 Comm: syz-executor357 Not tainted 6.1.0-rc4-syzkaller-31844-g3577a7611842 #0 > Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 09/30/2022 > Call trace: > dump_backtrace+0x1c4/0x1f0 arch/arm64/kernel/stacktrace.c:156 > show_stack+0x2c/0x54 arch/arm64/kernel/stacktrace.c:163 > __dump_stack lib/dump_stack.c:88 [inline] > dump_stack_lvl+0x104/0x16c lib/dump_stack.c:106 > dump_stack+0x1c/0x58 lib/dump_stack.c:113 > register_lock_class+0x2e4/0x2f8 kernel/locking/lockdep.c:1326 > __lock_acquire+0xa8/0x3084 kernel/locking/lockdep.c:4934 > lock_acquire+0x100/0x1f8 kernel/locking/lockdep.c:5668 > __mutex_lock_common+0xd4/0xca8 kernel/locking/mutex.c:603 > __mutex_lock kernel/locking/mutex.c:747 [inline] > mutex_lock_nested+0x38/0x44 kernel/locking/mutex.c:799 > team_vlan_rx_add_vid+0x38/0xd8 drivers/net/team/team.c:1904 > vlan_add_rx_filter_info net/8021q/vlan_core.c:211 [inline] > __vlan_vid_add net/8021q/vlan_core.c:306 [inline] > vlan_vid_add+0x328/0x38c net/8021q/vlan_core.c:336 > vlan_device_event+0x200/0xc4c net/8021q/vlan.c:385 > notifier_call_chain kernel/notifier.c:87 [inline] > raw_notifier_call_chain+0x7c/0x108 kernel/notifier.c:455 > __dev_notify_flags+0x170/0x2e8 > rtnl_newlink_create+0x460/0x6bc net/core/rtnetlink.c:3372 > __rtnl_newlink net/core/rtnetlink.c:3581 [inline] > rtnl_newlink+0x728/0xa04 net/core/rtnetlink.c:3594 > rtnetlink_rcv_msg+0x484/0x82c net/core/rtnetlink.c:6091 > netlink_rcv_skb+0xe8/0x1d4 net/netlink/af_netlink.c:2540 > rtnetlink_rcv+0x28/0x38 net/core/rtnetlink.c:6109 > netlink_unicast_kernel+0xfc/0x1dc net/netlink/af_netlink.c:1319 > netlink_unicast+0x164/0x248 net/netlink/af_netlink.c:1345 > netlink_sendmsg+0x484/0x584 net/netlink/af_netlink.c:1921 > sock_sendmsg_nosec net/socket.c:714 [inline] > sock_sendmsg net/socket.c:734 [inline] > ____sys_sendmsg+0x2f8/0x440 net/socket.c:2482 > ___sys_sendmsg net/socket.c:2536 [inline] > __sys_sendmsg+0x1ac/0x228 net/socket.c:2565 > __do_sys_sendmsg net/socket.c:2574 [inline] > __se_sys_sendmsg net/socket.c:2572 [inline] > __arm64_sys_sendmsg+0x2c/0x3c net/socket.c:2572 > __invoke_syscall arch/arm64/kernel/syscall.c:38 [inline] > invoke_syscall arch/arm64/kernel/syscall.c:52 [inline] > el0_svc_common+0x138/0x220 arch/arm64/kernel/syscall.c:142 > do_el0_svc+0x48/0x164 arch/arm64/kernel/syscall.c:206 > el0_svc+0x58/0x150 arch/arm64/kernel/entry-common.c:637 > el0t_64_sync_handler+0x84/0xf0 arch/arm64/kernel/entry-common.c:655 > el0t_64_sync+0x18c/0x190 arch/arm64/kernel/entry.S:581 > ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2022-11-08 18:36 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-05-19 17:35 [syzbot] BUG: MAX_LOCKDEP_KEYS too low! (2) syzbot 2021-05-19 19:48 ` Dmitry Vyukov 2021-05-19 19:57 ` Randy Dunlap 2021-05-19 20:09 ` Dmitry Vyukov 2021-05-20 5:02 ` Tetsuo Handa 2021-05-21 7:04 ` Dmitry Vyukov 2022-11-08 13:40 ` syzbot 2022-11-08 18:36 ` Dmitry Vyukov
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).