BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[syzbot]

Hello,

syzbot found the following crash on:

HEAD commit:    cf08baa29613 Add linux-next specific files for 20190306
git tree:       linux-next
console output: https://syzkaller.appspot.com/x/log.txt?x=15bc76c7200000
kernel config:  https://syzkaller.appspot.com/x/.config?x=c8b6073d992e8217
dashboard link: https://syzkaller.appspot.com/bug?extid=91fd909b6e62ebe06131
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)

Unfortunately, I don't have any reproducer for this crash yet.

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+91fd909b6e62ebe06131@syzkaller.appspotmail.com

BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
turning off the locking correctness validator.
CPU: 0 PID: 11902 Comm: kworker/u5:27 Not tainted 5.0.0-next-20190306 #4
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Workqueue: hci94 hci_power_on
Call Trace:
  __dump_stack lib/dump_stack.c:77 [inline]
  dump_stack+0x172/0x1f0 lib/dump_stack.c:113
  add_chain_cache kernel/locking/lockdep.c:2582 [inline]
  lookup_chain_cache_add kernel/locking/lockdep.c:2656 [inline]
  validate_chain kernel/locking/lockdep.c:2676 [inline]
  __lock_acquire.cold+0x250/0x50d kernel/locking/lockdep.c:3692
  lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:4202
  __flush_work+0x677/0x8a0 kernel/workqueue.c:3034
  flush_work+0x18/0x20 kernel/workqueue.c:3060
  hci_dev_do_open+0xa92/0x1780 net/bluetooth/hci_core.c:1543
  hci_power_on+0x10d/0x580 net/bluetooth/hci_core.c:2173
  process_one_work+0x98e/0x1790 kernel/workqueue.c:2269
  worker_thread+0x98/0xe40 kernel/workqueue.c:2415
  kthread+0x357/0x430 kernel/kthread.c:253
  ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352


---
This bug is generated by a bot. It may contain errors.
See https://goo.gl/tpsmEJ for more information about syzbot.
syzbot engineers can be reached at syzkaller@googlegroups.com.

syzbot will keep track of this bug report. See:
https://goo.gl/tpsmEJ#bug-status-tracking for how to communicate with  
syzbot.

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[syzbot]

syzbot has found a reproducer for the following crash on:

HEAD commit:    771acc7e Bluetooth: btusb: request wake pin with NOAUTOEN
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=15f58add200000
kernel config:  https://syzkaller.appspot.com/x/.config?x=4fb64439e07a1ec0
dashboard link: https://syzkaller.appspot.com/bug?extid=91fd909b6e62ebe06131
compiler:       gcc (GCC) 9.0.0 20181231 (experimental)
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=11770a8f200000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=128c945b200000

IMPORTANT: if you fix the bug, please add the following tag to the commit:
Reported-by: syzbot+91fd909b6e62ebe06131@syzkaller.appspotmail.com

BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
turning off the locking correctness validator.
CPU: 0 PID: 1174 Comm: kworker/u5:0 Not tainted 5.1.0-rc4+ #63
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS  
Google 01/01/2011
Workqueue: hci1 hci_power_on
Call Trace:
  __dump_stack lib/dump_stack.c:77 [inline]
  dump_stack+0x172/0x1f0 lib/dump_stack.c:113
  add_chain_cache kernel/locking/lockdep.c:2591 [inline]
  lookup_chain_cache_add kernel/locking/lockdep.c:2665 [inline]
  validate_chain kernel/locking/lockdep.c:2685 [inline]
  __lock_acquire.cold+0x250/0x50d kernel/locking/lockdep.c:3701
  lock_acquire+0x16f/0x3f0 kernel/locking/lockdep.c:4211
  __mutex_lock_common kernel/locking/mutex.c:925 [inline]
  __mutex_lock+0xf7/0x1310 kernel/locking/mutex.c:1072
  mutex_lock_nested+0x16/0x20 kernel/locking/mutex.c:1087
  hci_dev_do_close+0x317/0xf20 net/bluetooth/hci_core.c:1696
  hci_power_on+0x1d2/0x580 net/bluetooth/hci_core.c:2191
  process_one_work+0x98e/0x1790 kernel/workqueue.c:2269
  worker_thread+0x98/0xe40 kernel/workqueue.c:2415
  kthread+0x357/0x430 kernel/kthread.c:253
  ret_from_fork+0x3a/0x50 arch/x86/entry/entry_64.S:352

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[syzbot]

syzbot has bisected this bug to:

commit 726e41097920a73e4c7c33385dcc0debb1281e18
Author: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date:   Tue Jul 10 00:29:10 2018 +0000

     drivers: core: Remove glue dirs from sysfs earlier

bisection log:  https://syzkaller.appspot.com/x/bisect.txt?x=15f69eaf200000
start commit:   771acc7e Bluetooth: btusb: request wake pin with NOAUTOEN
git tree:       upstream
final crash:    https://syzkaller.appspot.com/x/report.txt?x=17f69eaf200000
console output: https://syzkaller.appspot.com/x/log.txt?x=13f69eaf200000
kernel config:  https://syzkaller.appspot.com/x/.config?x=4fb64439e07a1ec0
dashboard link: https://syzkaller.appspot.com/bug?extid=91fd909b6e62ebe06131
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=11770a8f200000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=128c945b200000

Reported-by: syzbot+91fd909b6e62ebe06131@syzkaller.appspotmail.com
Fixes: 726e41097920 ("drivers: core: Remove glue dirs from sysfs earlier")

For information about bisection process see: https://goo.gl/tpsmEJ#bisection

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Benjamin Herrenschmidt]

On Thu, 2019-04-11 at 05:14 -0700, syzbot wrote:
> syzbot has bisected this bug to:
> 
> commit 726e41097920a73e4c7c33385dcc0debb1281e18
> Author: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> Date:   Tue Jul 10 00:29:10 2018 +0000
> 
>      drivers: core: Remove glue dirs from sysfs earlier

Greg, any idea what this is ? The log isn't terribly readable. The
above patch fixes a real bug that causes use after free and memory
corruption under some circumstances. I wonder if the BT stack is itself
manipulating stale objects ?

Ben.

> bisection log:  
> https://syzkaller.appspot.com/x/bisect.txt?x=15f69eaf200000
> start commit:   771acc7e Bluetooth: btusb: request wake pin with
> NOAUTOEN
> git tree:       upstream
> final crash:    
> https://syzkaller.appspot.com/x/report.txt?x=17f69eaf200000
> console output: 
> https://syzkaller.appspot.com/x/log.txt?x=13f69eaf200000
> kernel config:  
> https://syzkaller.appspot.com/x/.config?x=4fb64439e07a1ec0
> dashboard link: 
> https://syzkaller.appspot.com/bug?extid=91fd909b6e62ebe06131
> syz repro:      
> https://syzkaller.appspot.com/x/repro.syz?x=11770a8f200000
> C reproducer:   
> https://syzkaller.appspot.com/x/repro.c?x=128c945b200000
> 
> Reported-by: syzbot+91fd909b6e62ebe06131@syzkaller.appspotmail.com
> Fixes: 726e41097920 ("drivers: core: Remove glue dirs from sysfs
> earlier")
> 
> For information about bisection process see: 
> https://goo.gl/tpsmEJ#bisection

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Chris Murphy]

On Fri, Jan 27, 2023, at 9:26 AM, Waiman Long wrote:
> On 1/26/23 23:07, Boqun Feng wrote:
>> On Thu, Jan 26, 2023 at 10:37:56PM -0500, Chris Murphy wrote:
>>>
>>> On Thu, Jan 26, 2023, at 7:20 PM, Waiman Long wrote:
>>>> On 1/26/23 17:42, Mikhail Gavrilov wrote:
>>>>>> I'm not sure whether these options are better than just increasing the
>>>>>> number, maybe to unblock your ASAP, you can try make it 30 and make sure
>>>>>> you have large enough memory to test.
>>>>> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
>>>>> be done? In vanilla kernel on kernel.org? In a specific distribution?
>>>>> or the user must rebuild the kernel himself? Maybe increase
>>>>> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
>>>>> to distribute to end users because the meaning of using packaged
>>>>> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
>>>>> config and rebuild the kernel by yourself).
>>>> Note that lockdep is typically only enabled in a debug kernel shipped by
>>>> a distro because of the high performance overhead. The non-debug kernel
>>>> doesn't have lockdep enabled. When LOCKDEP_CHAINS_BITS isn't big enough
>>>> when testing on the debug kernel, you can file a ticket to the distro
>>>> asking for an increase in CONFIG_LOCKDEP_CHAIN_BITS. Or you can build
>>>> your own debug kernel with a bigger CONFIG_LOCKDEP_CHAIN_BITS.
>>> Fedora bumped CONFIG_LOCKDEP_CHAINS_BITS=17 to 18 just 6 months ago for debug kernels.
>>> https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921
>>>
>>> If 19 the recommended value I don't mind sending an MR for it. But if
>>> the idea is we're going to be back here talking about bumping it to 20
>>> in six months, I'd like to avoid that.
>>>
>> How about a boot parameter then?
>
> A boot parameter doesn't work for a statically allocated array which is 
> determined at compile time. Dynamic memory allocation isn't enabled yet 
> at early boot when lockdep will be used.

Also, at least in Fedora Rawhide where the mainline debug kernels appear, mostly get used non-interactively with automated tests. So if we're going to discover lockdep issues, we need the kernel logs to be reliable at the time those tests are run, and we don't have a practical way of adding another boot parameter just for these tests.

Anyway I went ahead and submitted an MR to bump this to 19.
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2271



-- 
Chris Murphy

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Waiman Long]

On 1/26/23 23:07, Boqun Feng wrote:
> On Thu, Jan 26, 2023 at 10:37:56PM -0500, Chris Murphy wrote:
>>
>> On Thu, Jan 26, 2023, at 7:20 PM, Waiman Long wrote:
>>> On 1/26/23 17:42, Mikhail Gavrilov wrote:
>>>>> I'm not sure whether these options are better than just increasing the
>>>>> number, maybe to unblock your ASAP, you can try make it 30 and make sure
>>>>> you have large enough memory to test.
>>>> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
>>>> be done? In vanilla kernel on kernel.org? In a specific distribution?
>>>> or the user must rebuild the kernel himself? Maybe increase
>>>> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
>>>> to distribute to end users because the meaning of using packaged
>>>> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
>>>> config and rebuild the kernel by yourself).
>>> Note that lockdep is typically only enabled in a debug kernel shipped by
>>> a distro because of the high performance overhead. The non-debug kernel
>>> doesn't have lockdep enabled. When LOCKDEP_CHAINS_BITS isn't big enough
>>> when testing on the debug kernel, you can file a ticket to the distro
>>> asking for an increase in CONFIG_LOCKDEP_CHAIN_BITS. Or you can build
>>> your own debug kernel with a bigger CONFIG_LOCKDEP_CHAIN_BITS.
>> Fedora bumped CONFIG_LOCKDEP_CHAINS_BITS=17 to 18 just 6 months ago for debug kernels.
>> https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921
>>
>> If 19 the recommended value I don't mind sending an MR for it. But if
>> the idea is we're going to be back here talking about bumping it to 20
>> in six months, I'd like to avoid that.
>>
> How about a boot parameter then?

A boot parameter doesn't work for a statically allocated array which is 
determined at compile time. Dynamic memory allocation isn't enabled yet 
at early boot when lockdep will be used.

Cheers,
Longman

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Mikhail Gavrilov]

On Fri, Jan 27, 2023 at 9:08 AM Boqun Feng <boqun.feng@gmail.com> wrote:
>
> On Thu, Jan 26, 2023 at 10:37:56PM -0500, Chris Murphy wrote:
> >
> >
> > On Thu, Jan 26, 2023, at 7:20 PM, Waiman Long wrote:
> > > On 1/26/23 17:42, Mikhail Gavrilov wrote:
> > >>> I'm not sure whether these options are better than just increasing the
> > >>> number, maybe to unblock your ASAP, you can try make it 30 and make sure
> > >>> you have large enough memory to test.
> > >> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
> > >> be done? In vanilla kernel on kernel.org? In a specific distribution?
> > >> or the user must rebuild the kernel himself? Maybe increase
> > >> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
> > >> to distribute to end users because the meaning of using packaged
> > >> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
> > >> config and rebuild the kernel by yourself).
> > >
> > > Note that lockdep is typically only enabled in a debug kernel shipped by
> > > a distro because of the high performance overhead. The non-debug kernel
> > > doesn't have lockdep enabled. When LOCKDEP_CHAINS_BITS isn't big enough
> > > when testing on the debug kernel, you can file a ticket to the distro
> > > asking for an increase in CONFIG_LOCKDEP_CHAIN_BITS. Or you can build
> > > your own debug kernel with a bigger CONFIG_LOCKDEP_CHAIN_BITS.
> >
> > Fedora bumped CONFIG_LOCKDEP_CHAINS_BITS=17 to 18 just 6 months ago for debug kernels.
> > https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921
> >
> > If 19 the recommended value I don't mind sending an MR for it. But if
> > the idea is we're going to be back here talking about bumping it to 20
> > in six months, I'd like to avoid that.
> >
>
> How about a boot parameter then?

I would like this option.
This is better than rebuilding the kernel yourself and asking the
distribution's maintainers to increase this value.

Thanks.

-- 
Best Regards,
Mike Gavrilov.

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Boqun Feng]

On Thu, Jan 26, 2023 at 10:37:56PM -0500, Chris Murphy wrote:
> 
> 
> On Thu, Jan 26, 2023, at 7:20 PM, Waiman Long wrote:
> > On 1/26/23 17:42, Mikhail Gavrilov wrote:
> >>> I'm not sure whether these options are better than just increasing the
> >>> number, maybe to unblock your ASAP, you can try make it 30 and make sure
> >>> you have large enough memory to test.
> >> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
> >> be done? In vanilla kernel on kernel.org? In a specific distribution?
> >> or the user must rebuild the kernel himself? Maybe increase
> >> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
> >> to distribute to end users because the meaning of using packaged
> >> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
> >> config and rebuild the kernel by yourself).
> >
> > Note that lockdep is typically only enabled in a debug kernel shipped by 
> > a distro because of the high performance overhead. The non-debug kernel 
> > doesn't have lockdep enabled. When LOCKDEP_CHAINS_BITS isn't big enough 
> > when testing on the debug kernel, you can file a ticket to the distro 
> > asking for an increase in CONFIG_LOCKDEP_CHAIN_BITS. Or you can build 
> > your own debug kernel with a bigger CONFIG_LOCKDEP_CHAIN_BITS.
> 
> Fedora bumped CONFIG_LOCKDEP_CHAINS_BITS=17 to 18 just 6 months ago for debug kernels.
> https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921
> 
> If 19 the recommended value I don't mind sending an MR for it. But if
> the idea is we're going to be back here talking about bumping it to 20
> in six months, I'd like to avoid that.
> 

How about a boot parameter then?

Regards,
Boqun

> 
> 
> -- 
> Chris Murphy

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Chris Murphy]

On Thu, Jan 26, 2023, at 7:20 PM, Waiman Long wrote:
> On 1/26/23 17:42, Mikhail Gavrilov wrote:
>>> I'm not sure whether these options are better than just increasing the
>>> number, maybe to unblock your ASAP, you can try make it 30 and make sure
>>> you have large enough memory to test.
>> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
>> be done? In vanilla kernel on kernel.org? In a specific distribution?
>> or the user must rebuild the kernel himself? Maybe increase
>> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
>> to distribute to end users because the meaning of using packaged
>> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
>> config and rebuild the kernel by yourself).
>
> Note that lockdep is typically only enabled in a debug kernel shipped by 
> a distro because of the high performance overhead. The non-debug kernel 
> doesn't have lockdep enabled. When LOCKDEP_CHAINS_BITS isn't big enough 
> when testing on the debug kernel, you can file a ticket to the distro 
> asking for an increase in CONFIG_LOCKDEP_CHAIN_BITS. Or you can build 
> your own debug kernel with a bigger CONFIG_LOCKDEP_CHAIN_BITS.

Fedora bumped CONFIG_LOCKDEP_CHAINS_BITS=17 to 18 just 6 months ago for debug kernels.
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921

If 19 the recommended value I don't mind sending an MR for it. But if the idea is we're going to be back here talking about bumping it to 20 in six months, I'd like to avoid that.



-- 
Chris Murphy

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Waiman Long]

On 1/26/23 17:42, Mikhail Gavrilov wrote:
>> I'm not sure whether these options are better than just increasing the
>> number, maybe to unblock your ASAP, you can try make it 30 and make sure
>> you have large enough memory to test.
> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
> be done? In vanilla kernel on kernel.org? In a specific distribution?
> or the user must rebuild the kernel himself? Maybe increase
> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
> to distribute to end users because the meaning of using packaged
> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
> config and rebuild the kernel by yourself).

Note that lockdep is typically only enabled in a debug kernel shipped by 
a distro because of the high performance overhead. The non-debug kernel 
doesn't have lockdep enabled. When LOCKDEP_CHAINS_BITS isn't big enough 
when testing on the debug kernel, you can file a ticket to the distro 
asking for an increase in CONFIG_LOCKDEP_CHAIN_BITS. Or you can build 
your own debug kernel with a bigger CONFIG_LOCKDEP_CHAIN_BITS.

Cheers,
Longman

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Boqun Feng]

On Fri, Jan 27, 2023 at 03:42:52AM +0500, Mikhail Gavrilov wrote:
> On Thu, Jan 26, 2023 at 10:39 PM Boqun Feng <boqun.feng@gmail.com> wrote:
> >
> > [Cc lock folks]
> >
> > On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
> > > On Wed, Jan 25, 2023 at 10:21 PM David Sterba <dsterba@suse.cz> wrote:
> > > >
> > > > On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > > > > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
> > > > > >
> > > > > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > > > > Hi guys.
> > > > > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > > > > >
> > > > > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > > > > tends to work.
> > > > >
> > > > > Hi,
> > > > > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > > > > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> > > > >
> > > > > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > > > > CONFIG_LOCKDEP_CHAINS_BITS=18
> > > > >
> > > > > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > > > > [88685.088124] turning off the locking correctness validator.
> > > > > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > > > > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G        W    L
> > > > >   -------  ---  6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > > > > [88685.088154] Hardware name: System manufacturer System Product
> > > > > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> > > > >
> > > > > What's next? Increase this value to 19?
> > > >
> > > > Yes, though increasing the value is a workaround so you may see the
> > > > warning again.
> > >
> > > Is there any sense in this WARNING if we would ignore it and every
> > > time increase the threshold value?
> >
> > Lockdep uses static allocated array to track lock holdings chains to
> > avoid dynmaic memory allocation in its own code. So if you see the
> > warning it means your test has more combination of lock holdings than
> > the array can record. In other words, you reach the resource limitation,
> > and in that sense it makes sense to just ignore it and increase the
> > value: you want to give lockdep enough resource to work, right?
> 
> It is needed for correct working btrfs. David, am I right?
> 

Lockdep is not needed for correct working btrfs in production. It's a
tool to help btrfs developers to find deadlocks in
development/test/debug environment. End users, i.e. the users of linux
kernel don't need it.

Regards,
Boqun

> >
> > > May Be set 99 right away? Or remove such a check condition?
> >
> > That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
> > chains array..
> >
> > However, a few other options we can try in lockdep are:
> >
> > *       warn but not turn off the lockdep: the lock holding chain is
> >         only a cache for what lock holding combination lockdep has ever
> >         see, we also record the dependency in the graph. Without the
> >         lock holding chain, lockdep can still work but just slower.
> >
> > *       allow dynmaic memory allocation in lockdep: I think this might
> >         be OK since we have lockdep_recursion to avoid lockdep code ->
> >         mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
> >         missing something. And even we allow it, the use of memory
> >         doesn't change, you will still need that amout of memory to
> >         track lock holding chains.
> >
> > I'm not sure whether these options are better than just increasing the
> > number, maybe to unblock your ASAP, you can try make it 30 and make sure
> > you have large enough memory to test.
> 
> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
> be done? In vanilla kernel on kernel.org? In a specific distribution?
> or the user must rebuild the kernel himself? Maybe increase
> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
> to distribute to end users because the meaning of using packaged
> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
> config and rebuild the kernel by yourself).
> 
> It would be great if the chosen value would simply work always
> everywhere. 30? ok! But as I understand, btrfs does not have any
> guarantees for this. David, am I right?
> 
> Anyway, thank you for keeping the conversation going.
> 
> -- 
> Best Regards,
> Mike Gavrilov.

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Boqun Feng]

On Fri, Jan 27, 2023 at 03:42:52AM +0500, Mikhail Gavrilov wrote:
> On Thu, Jan 26, 2023 at 10:39 PM Boqun Feng <boqun.feng@gmail.com> wrote:
> >
> > [Cc lock folks]
> >
> > On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
> > > On Wed, Jan 25, 2023 at 10:21 PM David Sterba <dsterba@suse.cz> wrote:
> > > >
> > > > On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > > > > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
> > > > > >
> > > > > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > > > > Hi guys.
> > > > > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > > > > >
> > > > > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > > > > tends to work.
> > > > >
> > > > > Hi,
> > > > > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > > > > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> > > > >
> > > > > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > > > > CONFIG_LOCKDEP_CHAINS_BITS=18
> > > > >
> > > > > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > > > > [88685.088124] turning off the locking correctness validator.
> > > > > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > > > > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G        W    L
> > > > >   -------  ---  6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > > > > [88685.088154] Hardware name: System manufacturer System Product
> > > > > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> > > > >
> > > > > What's next? Increase this value to 19?
> > > >
> > > > Yes, though increasing the value is a workaround so you may see the
> > > > warning again.
> > >
> > > Is there any sense in this WARNING if we would ignore it and every
> > > time increase the threshold value?
> >
> > Lockdep uses static allocated array to track lock holdings chains to
> > avoid dynmaic memory allocation in its own code. So if you see the
> > warning it means your test has more combination of lock holdings than
> > the array can record. In other words, you reach the resource limitation,
> > and in that sense it makes sense to just ignore it and increase the
> > value: you want to give lockdep enough resource to work, right?
> 
> It is needed for correct working btrfs. David, am I right?
> 
> >
> > > May Be set 99 right away? Or remove such a check condition?
> >
> > That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
> > chains array..
> >
> > However, a few other options we can try in lockdep are:
> >
> > *       warn but not turn off the lockdep: the lock holding chain is
> >         only a cache for what lock holding combination lockdep has ever
> >         see, we also record the dependency in the graph. Without the
> >         lock holding chain, lockdep can still work but just slower.
> >
> > *       allow dynmaic memory allocation in lockdep: I think this might
> >         be OK since we have lockdep_recursion to avoid lockdep code ->
> >         mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
> >         missing something. And even we allow it, the use of memory
> >         doesn't change, you will still need that amout of memory to
> >         track lock holding chains.
> >
> > I'm not sure whether these options are better than just increasing the
> > number, maybe to unblock your ASAP, you can try make it 30 and make sure
> > you have large enough memory to test.
> 
> About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
> be done? In vanilla kernel on kernel.org? In a specific distribution?
> or the user must rebuild the kernel himself? Maybe increase
> LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
> to distribute to end users because the meaning of using packaged
> distributions is lost (user should change LOCKDEP_CHAINS_BITS in
> config and rebuild the kernel by yourself).
> 

Lockdep is a dev tool to help finding out deadlocks, and it introduces
cost when enabled, although it's possible, I doubt no one will run
LOCKDEP enabled kernel in production environment. In other words, it's
a debug/test-kernel-only option.

Regards,
Boqun

> It would be great if the chosen value would simply work always
> everywhere. 30? ok! But as I understand, btrfs does not have any
> guarantees for this. David, am I right?
> 
> Anyway, thank you for keeping the conversation going.
> 
> -- 
> Best Regards,
> Mike Gavrilov.

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Mikhail Gavrilov]

On Thu, Jan 26, 2023 at 10:39 PM Boqun Feng <boqun.feng@gmail.com> wrote:
>
> [Cc lock folks]
>
> On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
> > On Wed, Jan 25, 2023 at 10:21 PM David Sterba <dsterba@suse.cz> wrote:
> > >
> > > On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > > > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
> > > > >
> > > > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > > > Hi guys.
> > > > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > > > >
> > > > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > > > tends to work.
> > > >
> > > > Hi,
> > > > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > > > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> > > >
> > > > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > > > CONFIG_LOCKDEP_CHAINS_BITS=18
> > > >
> > > > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > > > [88685.088124] turning off the locking correctness validator.
> > > > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > > > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G        W    L
> > > >   -------  ---  6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > > > [88685.088154] Hardware name: System manufacturer System Product
> > > > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> > > >
> > > > What's next? Increase this value to 19?
> > >
> > > Yes, though increasing the value is a workaround so you may see the
> > > warning again.
> >
> > Is there any sense in this WARNING if we would ignore it and every
> > time increase the threshold value?
>
> Lockdep uses static allocated array to track lock holdings chains to
> avoid dynmaic memory allocation in its own code. So if you see the
> warning it means your test has more combination of lock holdings than
> the array can record. In other words, you reach the resource limitation,
> and in that sense it makes sense to just ignore it and increase the
> value: you want to give lockdep enough resource to work, right?

It is needed for correct working btrfs. David, am I right?

>
> > May Be set 99 right away? Or remove such a check condition?
>
> That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
> chains array..
>
> However, a few other options we can try in lockdep are:
>
> *       warn but not turn off the lockdep: the lock holding chain is
>         only a cache for what lock holding combination lockdep has ever
>         see, we also record the dependency in the graph. Without the
>         lock holding chain, lockdep can still work but just slower.
>
> *       allow dynmaic memory allocation in lockdep: I think this might
>         be OK since we have lockdep_recursion to avoid lockdep code ->
>         mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
>         missing something. And even we allow it, the use of memory
>         doesn't change, you will still need that amout of memory to
>         track lock holding chains.
>
> I'm not sure whether these options are better than just increasing the
> number, maybe to unblock your ASAP, you can try make it 30 and make sure
> you have large enough memory to test.

About just to increase the LOCKDEP_CHAINS_BITS by 1. Where should this
be done? In vanilla kernel on kernel.org? In a specific distribution?
or the user must rebuild the kernel himself? Maybe increase
LOCKDEP_CHAINS_BITS by 1 is most reliable solution, but it difficult
to distribute to end users because the meaning of using packaged
distributions is lost (user should change LOCKDEP_CHAINS_BITS in
config and rebuild the kernel by yourself).

It would be great if the chosen value would simply work always
everywhere. 30? ok! But as I understand, btrfs does not have any
guarantees for this. David, am I right?

Anyway, thank you for keeping the conversation going.

-- 
Best Regards,
Mike Gavrilov.

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Waiman Long]

On 1/26/23 13:59, Boqun Feng wrote:
> On Thu, Jan 26, 2023 at 01:30:34PM -0500, Waiman Long wrote:
>> On 1/26/23 12:38, Boqun Feng wrote:
>>> [Cc lock folks]
>>>
>>> On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
>>>> On Wed, Jan 25, 2023 at 10:21 PM David Sterba <dsterba@suse.cz> wrote:
>>>>> On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
>>>>>> On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
>>>>>>> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
>>>>>>>> Hi guys.
>>>>>>>> Always with intensive writing on a btrfs volume, the message "BUG:
>>>>>>>> MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>>>>>>> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
>>>>>>> tends to work.
>>>>>> Hi,
>>>>>> Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
>>>>>> low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
>>>>>>
>>>>>> ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
>>>>>> CONFIG_LOCKDEP_CHAINS_BITS=18
>>>>>>
>>>>>> [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
>>>>>> [88685.088124] turning off the locking correctness validator.
>>>>>> [88685.088133] Please attach the output of /proc/lock_stat to the bug report
>>>>>> [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G        W    L
>>>>>>     -------  ---  6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
>>>>>> [88685.088154] Hardware name: System manufacturer System Product
>>>>>> Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
>>>>>>
>>>>>> What's next? Increase this value to 19?
>>>>> Yes, though increasing the value is a workaround so you may see the
>>>>> warning again.
>>>> Is there any sense in this WARNING if we would ignore it and every
>>>> time increase the threshold value?
>>> Lockdep uses static allocated array to track lock holdings chains to
>>> avoid dynmaic memory allocation in its own code. So if you see the
>>> warning it means your test has more combination of lock holdings than
>>> the array can record. In other words, you reach the resource limitation,
>>> and in that sense it makes sense to just ignore it and increase the
>>> value: you want to give lockdep enough resource to work, right?
>>>
>>>> May Be set 99 right away? Or remove such a check condition?
>>> That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
>>> chains array..
>> Note that every increment of LOCKDEP_CHAINS_BITS double the storage space.
>> With 99, that will likely exceed the total amount of memory you have in your
>> system.
>>
>> Boqun, where does the 5 figure come from. It is just a simple u16 array of
> 	#define MAX_LOCKDEP_CHAINS_BITS	CONFIG_LOCKDEP_CHAINS_BITS
> 	#define MAX_LOCKDEP_CHAINS	(1UL << MAX_LOCKDEP_CHAINS_BITS)
>
> 	#define MAX_LOCKDEP_CHAIN_HLOCKS (MAX_LOCKDEP_CHAINS*5)
>
> I think the last one means we think the average length of a lock chain
> is 5, in other words, in average, a task hold at most 5 locks. I don't
> know where the 5 came from either, but it's there ;-)

You are right. I missed that when I looked. So 5 is assumed to the 
average length of a lock chain.

Thanks,
Longman

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Boqun Feng]

On Thu, Jan 26, 2023 at 01:30:34PM -0500, Waiman Long wrote:
> On 1/26/23 12:38, Boqun Feng wrote:
> > [Cc lock folks]
> > 
> > On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
> > > On Wed, Jan 25, 2023 at 10:21 PM David Sterba <dsterba@suse.cz> wrote:
> > > > On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > > > > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
> > > > > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > > > > Hi guys.
> > > > > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > > > > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > > > > tends to work.
> > > > > Hi,
> > > > > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > > > > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> > > > > 
> > > > > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > > > > CONFIG_LOCKDEP_CHAINS_BITS=18
> > > > > 
> > > > > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > > > > [88685.088124] turning off the locking correctness validator.
> > > > > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > > > > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G        W    L
> > > > >    -------  ---  6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > > > > [88685.088154] Hardware name: System manufacturer System Product
> > > > > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> > > > > 
> > > > > What's next? Increase this value to 19?
> > > > Yes, though increasing the value is a workaround so you may see the
> > > > warning again.
> > > Is there any sense in this WARNING if we would ignore it and every
> > > time increase the threshold value?
> > Lockdep uses static allocated array to track lock holdings chains to
> > avoid dynmaic memory allocation in its own code. So if you see the
> > warning it means your test has more combination of lock holdings than
> > the array can record. In other words, you reach the resource limitation,
> > and in that sense it makes sense to just ignore it and increase the
> > value: you want to give lockdep enough resource to work, right?
> > 
> > > May Be set 99 right away? Or remove such a check condition?
> > That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
> > chains array..
> 
> Note that every increment of LOCKDEP_CHAINS_BITS double the storage space.
> With 99, that will likely exceed the total amount of memory you have in your
> system.
> 
> Boqun, where does the 5 figure come from. It is just a simple u16 array of

	#define MAX_LOCKDEP_CHAINS_BITS	CONFIG_LOCKDEP_CHAINS_BITS
	#define MAX_LOCKDEP_CHAINS	(1UL << MAX_LOCKDEP_CHAINS_BITS)

	#define MAX_LOCKDEP_CHAIN_HLOCKS (MAX_LOCKDEP_CHAINS*5)

I think the last one means we think the average length of a lock chain
is 5, in other words, in average, a task hold at most 5 locks. I don't
know where the 5 came from either, but it's there ;-)

Regards,
Boqun

> size MAX_LOCKDEP_CHAIN_HLOCKS. The chain_hlocks array stores the lock chains
> that show up in the lockdep splats and in the /proc/lockdep* files. Each
> chain is variable size. As we add new lock into the chain, we have to
> repeatedly deallocate and reallocate a larger chain buffer. That will cause
> fragmentation in the chain_hlocks[]. So if we have a very long lock chain,
> the allocation may fail because the largest free block is smaller than the
> requested chain length. There may be enough free space in chain_hlocks, but
> it is just too fragmented to be useful.
> 
> Maybe we should figure out a better way to handle this fragmentation. In the
> mean time, the easiest way forward is just to increase the
> LOCKDEP_CHAINS_BITS by 1.
> 
> > 
> > However, a few other options we can try in lockdep are:
> > 
> > *	warn but not turn off the lockdep: the lock holding chain is
> > 	only a cache for what lock holding combination lockdep has ever
> > 	see, we also record the dependency in the graph. Without the
> > 	lock holding chain, lockdep can still work but just slower.
> > 
> > *	allow dynmaic memory allocation in lockdep: I think this might
> > 	be OK since we have lockdep_recursion to avoid lockdep code ->
> > 	mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
> > 	missing something. And even we allow it, the use of memory
> > 	doesn't change, you will still need that amout of memory to
> > 	track lock holding chains.
> 
> It is not just the issue of calling the memory allocator. There is also the
> issue of copying data from old chain_hlocks to new one while the old one may
> be updated during the copying process unless we can freeze everything else.
> 
> Cheers,
> Longman
> 

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Waiman Long]

On 1/26/23 12:38, Boqun Feng wrote:
> [Cc lock folks]
>
> On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
>> On Wed, Jan 25, 2023 at 10:21 PM David Sterba <dsterba@suse.cz> wrote:
>>> On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
>>>> On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
>>>>> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
>>>>>> Hi guys.
>>>>>> Always with intensive writing on a btrfs volume, the message "BUG:
>>>>>> MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>>>>> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
>>>>> tends to work.
>>>> Hi,
>>>> Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
>>>> low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
>>>>
>>>> ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
>>>> CONFIG_LOCKDEP_CHAINS_BITS=18
>>>>
>>>> [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
>>>> [88685.088124] turning off the locking correctness validator.
>>>> [88685.088133] Please attach the output of /proc/lock_stat to the bug report
>>>> [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G        W    L
>>>>    -------  ---  6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
>>>> [88685.088154] Hardware name: System manufacturer System Product
>>>> Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
>>>>
>>>> What's next? Increase this value to 19?
>>> Yes, though increasing the value is a workaround so you may see the
>>> warning again.
>> Is there any sense in this WARNING if we would ignore it and every
>> time increase the threshold value?
> Lockdep uses static allocated array to track lock holdings chains to
> avoid dynmaic memory allocation in its own code. So if you see the
> warning it means your test has more combination of lock holdings than
> the array can record. In other words, you reach the resource limitation,
> and in that sense it makes sense to just ignore it and increase the
> value: you want to give lockdep enough resource to work, right?
>
>> May Be set 99 right away? Or remove such a check condition?
> That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
> chains array..

Note that every increment of LOCKDEP_CHAINS_BITS double the storage 
space. With 99, that will likely exceed the total amount of memory you 
have in your system.

Boqun, where does the 5 figure come from. It is just a simple u16 array 
of size MAX_LOCKDEP_CHAIN_HLOCKS. The chain_hlocks array stores the lock 
chains that show up in the lockdep splats and in the /proc/lockdep* 
files. Each chain is variable size. As we add new lock into the chain, 
we have to repeatedly deallocate and reallocate a larger chain buffer. 
That will cause fragmentation in the chain_hlocks[]. So if we have a 
very long lock chain, the allocation may fail because the largest free 
block is smaller than the requested chain length. There may be enough 
free space in chain_hlocks, but it is just too fragmented to be useful.

Maybe we should figure out a better way to handle this fragmentation. In 
the mean time, the easiest way forward is just to increase the 
LOCKDEP_CHAINS_BITS by 1.

>
> However, a few other options we can try in lockdep are:
>
> *	warn but not turn off the lockdep: the lock holding chain is
> 	only a cache for what lock holding combination lockdep has ever
> 	see, we also record the dependency in the graph. Without the
> 	lock holding chain, lockdep can still work but just slower.
>
> *	allow dynmaic memory allocation in lockdep: I think this might
> 	be OK since we have lockdep_recursion to avoid lockdep code ->
> 	mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
> 	missing something. And even we allow it, the use of memory
> 	doesn't change, you will still need that amout of memory to
> 	track lock holding chains.

It is not just the issue of calling the memory allocator. There is also 
the issue of copying data from old chain_hlocks to new one while the old 
one may be updated during the copying process unless we can freeze 
everything else.

Cheers,
Longman

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Boqun Feng]

[Cc lock folks]

On Thu, Jan 26, 2023 at 02:47:42PM +0500, Mikhail Gavrilov wrote:
> On Wed, Jan 25, 2023 at 10:21 PM David Sterba <dsterba@suse.cz> wrote:
> >
> > On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
> > > >
> > > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > > Hi guys.
> > > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > > >
> > > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > > tends to work.
> > >
> > > Hi,
> > > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> > >
> > > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > > CONFIG_LOCKDEP_CHAINS_BITS=18
> > >
> > > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > > [88685.088124] turning off the locking correctness validator.
> > > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G        W    L
> > >   -------  ---  6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > > [88685.088154] Hardware name: System manufacturer System Product
> > > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> > >
> > > What's next? Increase this value to 19?
> >
> > Yes, though increasing the value is a workaround so you may see the
> > warning again.
> 
> Is there any sense in this WARNING if we would ignore it and every
> time increase the threshold value?

Lockdep uses static allocated array to track lock holdings chains to
avoid dynmaic memory allocation in its own code. So if you see the
warning it means your test has more combination of lock holdings than
the array can record. In other words, you reach the resource limitation,
and in that sense it makes sense to just ignore it and increase the
value: you want to give lockdep enough resource to work, right?

> May Be set 99 right away? Or remove such a check condition?

That requires having 2^99 * 5 * sizeof(u16) memory for lock holding
chains array..

However, a few other options we can try in lockdep are:

*	warn but not turn off the lockdep: the lock holding chain is
	only a cache for what lock holding combination lockdep has ever
	see, we also record the dependency in the graph. Without the
	lock holding chain, lockdep can still work but just slower.

*	allow dynmaic memory allocation in lockdep: I think this might
	be OK since we have lockdep_recursion to avoid lockdep code ->
	mm code -> lockdep code -> mm code ... deadlock. But maybe I'm
	missing something. And even we allow it, the use of memory
	doesn't change, you will still need that amout of memory to
	track lock holding chains.

I'm not sure whether these options are better than just increasing the
number, maybe to unblock your ASAP, you can try make it 30 and make sure
you have large enough memory to test.

Regards,
Boqun

> 
> -- 
> Best Regards,
> Mike Gavrilov.

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Mikhail Gavrilov]

On Wed, Jan 25, 2023 at 10:21 PM David Sterba <dsterba@suse.cz> wrote:
>
> On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> > On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
> > >
> > > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > > Hi guys.
> > > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> > >
> > > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > > tends to work.
> >
> > Hi,
> > Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> > low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> >
> > ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> > CONFIG_LOCKDEP_CHAINS_BITS=18
> >
> > [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> > [88685.088124] turning off the locking correctness validator.
> > [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> > [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G        W    L
> >   -------  ---  6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> > [88685.088154] Hardware name: System manufacturer System Product
> > Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> >
> > What's next? Increase this value to 19?
>
> Yes, though increasing the value is a workaround so you may see the
> warning again.

Is there any sense in this WARNING if we would ignore it and every
time increase the threshold value?
May Be set 99 right away? Or remove such a check condition?

-- 
Best Regards,
Mike Gavrilov.

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[David Sterba]

On Wed, Jan 25, 2023 at 01:27:48AM +0500, Mikhail Gavrilov wrote:
> On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
> >
> > On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > > Hi guys.
> > > Always with intensive writing on a btrfs volume, the message "BUG:
> > > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> >
> > Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> > tends to work.
> 
> Hi,
> Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
> low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.
> 
> ❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
> CONFIG_LOCKDEP_CHAINS_BITS=18
> 
> [88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
> [88685.088124] turning off the locking correctness validator.
> [88685.088133] Please attach the output of /proc/lock_stat to the bug report
> [88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G        W    L
>   -------  ---  6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
> [88685.088154] Hardware name: System manufacturer System Product
> Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022
> 
> What's next? Increase this value to 19?

Yes, though increasing the value is a workaround so you may see the
warning again.

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Mikhail Gavrilov]

[-- Attachment #1: Type: text/plain, Size: 1220 bytes --]

On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
>
> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > Hi guys.
> > Always with intensive writing on a btrfs volume, the message "BUG:
> > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>
> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> tends to work.

Hi,
Today I was able to get the message "BUG: MAX_LOCKDEP_CHAIN_HLOCKS too
low!" again even with LOCKDEP_CHAINS_BITS=18 and kernel 6.2-rc5.

❯ cat /boot/config-`uname -r` | grep LOCKDEP_CHAINS_BITS
CONFIG_LOCKDEP_CHAINS_BITS=18

[88685.088099] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
[88685.088124] turning off the locking correctness validator.
[88685.088133] Please attach the output of /proc/lock_stat to the bug report
[88685.088142] CPU: 14 PID: 1749746 Comm: mv Tainted: G        W    L
  -------  ---  6.2.0-0.rc5.20230123git2475bf0250de.38.fc38.x86_64 #1
[88685.088154] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 4408 10/28/2022

What's next? Increase this value to 19?
Actual full kernel log and lock_stat attached below.

-- 
Best Regards,
Mike Gavrilov.

[-- Attachment #2: dmesg.tar.xz --]
[-- Type: application/x-xz, Size: 36752 bytes --]

[-- Attachment #3: lock_stat.tar.xz --]
[-- Type: application/x-xz, Size: 96920 bytes --]

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Tetsuo Handa]

On 2022/08/04 16:35, Mikhail Gavrilov wrote:
> On Thu, Aug 4, 2022 at 1:01 AM Chris Murphy <lists@colorremedies.com> wrote:
>>
>> This will be making it into Fedora debug kernels, which have lockdep enabled on them, starting with 5.20 series, which are now building in koji.
>> https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921
> 
> I saw this change, but it would be good if users of all other
> distributions will be happy too.
> 

I'm not a lockdep maintainer.

Please submit a patch to lockdep maintainers and persuade lockdep maintainers
to change the default value. ;-)

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Mikhail Gavrilov]

On Thu, Aug 4, 2022 at 1:01 AM Chris Murphy <lists@colorremedies.com> wrote:
>
> This will be making it into Fedora debug kernels, which have lockdep enabled on them, starting with 5.20 series, which are now building in koji.
> https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921

I saw this change, but it would be good if users of all other
distributions will be happy too.

-- 
Best Regards,
Mike Gavrilov.

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Chris Murphy]

On Wed, Aug 3, 2022, at 3:28 PM, Mikhail Gavrilov wrote:
> On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
>>
>> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
>> > Hi guys.
>> > Always with intensive writing on a btrfs volume, the message "BUG:
>> > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>>
>> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
>> tends to work.
>
> I confirm that after bumping LOCKDEP_CHAINS_BITS to 18 several days of
> continuous writing on the BTRFS partition with different files with a
> total size of 10Tb I didn't see this kernel bug message again.
> Tetsuo, I saw your commit 5dc33592e95534dc8455ce3e9baaaf3dae0fff82 [1]
> set for LOCKDEP_CHAINS_BITS default value 16.
> Why not increase LOCKDEP_CHAINS_BITS to 18 by default?

This will be making it into Fedora debug kernels, which have lockdep enabled on them, starting with 5.20 series, which are now building in koji.
https://gitlab.com/cki-project/kernel-ark/-/merge_requests/1921




-- 
Chris Murphy

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Mikhail Gavrilov]

On Tue, Jul 26, 2022 at 9:47 PM David Sterba <dsterba@suse.cz> wrote:
>
> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> > Hi guys.
> > Always with intensive writing on a btrfs volume, the message "BUG:
> > MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>
> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> tends to work.

I confirm that after bumping LOCKDEP_CHAINS_BITS to 18 several days of
continuous writing on the BTRFS partition with different files with a
total size of 10Tb I didn't see this kernel bug message again.
Tetsuo, I saw your commit 5dc33592e95534dc8455ce3e9baaaf3dae0fff82 [1]
set for LOCKDEP_CHAINS_BITS default value 16.
Why not increase LOCKDEP_CHAINS_BITS to 18 by default?
Thanks.


[1] https://github.com/torvalds/linux/blame/master/lib/Kconfig.debug#L1387

-- 
Best Regards,
Mike Gavrilov.

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[David Sterba]

On Tue, Jul 26, 2022 at 03:21:32PM -0400, Chris Murphy wrote:
> 
> 
> On Tue, Jul 26, 2022, at 3:19 PM, Chris Murphy wrote:
> > On Tue, Jul 26, 2022, at 12:42 PM, David Sterba wrote:
> >> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> >>> Hi guys.
> >>> Always with intensive writing on a btrfs volume, the message "BUG:
> >>> MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
> >>
> >> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> >> tends to work.
> >
> > Fedora is using 17. I'll make a request to bump it to 18. Thanks.
> 
> Should it be 18 across all archs? Or is it OK to only bump x86_64?

I think it applies to all achritectures equally but I'm no lockdep
expert.

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Chris Murphy]

On Tue, Jul 26, 2022, at 3:19 PM, Chris Murphy wrote:
> On Tue, Jul 26, 2022, at 12:42 PM, David Sterba wrote:
>> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
>>> Hi guys.
>>> Always with intensive writing on a btrfs volume, the message "BUG:
>>> MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>>
>> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
>> tends to work.
>
> Fedora is using 17. I'll make a request to bump it to 18. Thanks.

Should it be 18 across all archs? Or is it OK to only bump x86_64?

-- 
Chris Murphy

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Chris Murphy]

On Tue, Jul 26, 2022, at 12:42 PM, David Sterba wrote:
> On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
>> Hi guys.
>> Always with intensive writing on a btrfs volume, the message "BUG:
>> MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.
>
> Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
> tends to work.

Fedora is using 17. I'll make a request to bump it to 18. Thanks.

--
Chris Murphy

Re: BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[David Sterba]

On Tue, Jul 26, 2022 at 05:32:54PM +0500, Mikhail Gavrilov wrote:
> Hi guys.
> Always with intensive writing on a btrfs volume, the message "BUG:
> MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.

Increase the config value of LOCKDEP_CHAINS_BITS, default is 16, 18
tends to work.

BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Mikhail Gavrilov]

Hi guys.
Always with intensive writing on a btrfs volume, the message "BUG:
MAX_LOCKDEP_CHAIN_HLOCKS too low!" appears in the kernel logs.

[46729.134549] BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!
[46729.134557] turning off the locking correctness validator.
[46729.134559] Please attach the output of /proc/lock_stat to the bug report
[46729.134561] CPU: 22 PID: 166516 Comm: ThreadPoolForeg Tainted: G
    W    L   --------  ---
5.19.0-0.rc7.20220722git68e77ffbfd06.56.fc37.x86_64 #1
[46729.134566] Hardware name: System manufacturer System Product
Name/ROG STRIX X570-I GAMING, BIOS 4403 04/27/2022
[46729.134569] Call Trace:
[46729.134572]  <TASK>
[46729.134576]  dump_stack_lvl+0x5b/0x77
[46729.134583]  __lock_acquire.cold+0x167/0x29e
[46729.134594]  lock_acquire+0xce/0x2d0
[46729.134599]  ? btrfs_reserve_extent+0xbd/0x250
[46729.134606]  ? btrfs_get_alloc_profile+0x17e/0x240
[46729.134611]  btrfs_get_alloc_profile+0x19c/0x240
[46729.134614]  ? btrfs_reserve_extent+0xbd/0x250
[46729.134618]  btrfs_reserve_extent+0xbd/0x250
[46729.134629]  btrfs_alloc_tree_block+0xa3/0x510
[46729.134635]  ? release_extent_buffer+0xa7/0xe0
[46729.134643]  split_node+0x131/0x3d0
[46729.134652]  btrfs_search_slot+0x2f3/0xc30
[46729.134659]  ? btrfs_insert_inode_ref+0x50/0x3b0
[46729.134664]  btrfs_insert_empty_items+0x31/0x70
[46729.134669]  btrfs_insert_inode_ref+0x99/0x3b0
[46729.134678]  btrfs_rename2+0x317/0x1510
[46729.134690]  ? vfs_rename+0x49d/0xd20
[46729.134693]  ? btrfs_symlink+0x460/0x460
[46729.134696]  vfs_rename+0x49d/0xd20
[46729.134705]  ? do_renameat2+0x4a0/0x510
[46729.134709]  do_renameat2+0x4a0/0x510
[46729.134720]  __x64_sys_rename+0x3f/0x50
[46729.134724]  do_syscall_64+0x5b/0x80
[46729.134729]  ? memcg_slab_free_hook+0x1fd/0x2e0
[46729.134735]  ? do_faccessat+0x111/0x260
[46729.134739]  ? kmem_cache_free+0x379/0x3d0
[46729.134744]  ? lock_is_held_type+0xe8/0x140
[46729.134749]  ? do_syscall_64+0x67/0x80
[46729.134752]  ? lockdep_hardirqs_on+0x7d/0x100
[46729.134757]  ? do_syscall_64+0x67/0x80
[46729.134760]  ? asm_exc_page_fault+0x22/0x30
[46729.134764]  ? lockdep_hardirqs_on+0x7d/0x100
[46729.134768]  entry_SYSCALL_64_after_hwframe+0x63/0xcd
[46729.134773] RIP: 0033:0x7fd2a29b5afb
[46729.134798] Code: e8 7a 27 0a 00 f7 d8 19 c0 5b c3 0f 1f 40 00 b8
ff ff ff ff 5b c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa b8 52 00 00
00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 f1 82 17 00
f7 d8
[46729.134801] RSP: 002b:00007fd25b70a5a8 EFLAGS: 00000282 ORIG_RAX:
0000000000000052
[46729.134805] RAX: ffffffffffffffda RBX: 00007fd25b70a5e0 RCX: 00007fd2a29b5afb
[46729.134808] RDX: 0000000000000000 RSI: 00003ba01ef60820 RDI: 00003ba00e4b2da0
[46729.134810] RBP: 00007fd25b70a660 R08: 0000000000000000 R09: 00007fd25b70a570
[46729.134812] R10: 00007ffd36b1f080 R11: 0000000000000282 R12: 00007fd25b70a5b8
[46729.134815] R13: 00003ba00e4b2da0 R14: 00007fd25b70a6c4 R15: 00003ba01ef60820
[46729.134823]  </TASK>

In this regard, I want to ask, is this really a bug?
The kernel version is 5.19-rc7.

Here's the full kernel log: https://pastebin.com/hYWH7RHu
Here's /proc/lock_stat: https://pastebin.com/ex5w0QW9

-- 
Best Regards,
Mike Gavrilov.

BUG: MAX_LOCKDEP_CHAIN_HLOCKS too low!

[Dmitry Vyukov]

This stopped happening a while ago, let's close this to get
notifications about new instances.
One of likely candidates:

#syz fix: net: partially revert dynamic lockdep key changes