Re: [PATCH] lockdep: Introduce CONFIG_LOCKDEP_LARGE

From: Dmitry Vyukov <dvyukov@google.com>
To: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Cc: Peter Zijlstra <peterz@infradead.org>,
	Ingo Molnar <mingo@redhat.com>, Will Deacon <will@kernel.org>,
	LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] lockdep: Introduce CONFIG_LOCKDEP_LARGE
Date: Tue, 18 Aug 2020 11:57:01 +0200	[thread overview]
Message-ID: <CACT4Y+Y0ptJDj1F89jKQEKB_L8U2yFCUUZ7pwxh+fcE-ZpsBkg@mail.gmail.com> (raw)
In-Reply-To: <37b60b14-4eb6-36b3-1195-97c2d27b7ed8@i-love.sakura.ne.jp>

On Tue, Aug 4, 2020 at 4:36 AM Tetsuo Handa
<penguin-kernel@i-love.sakura.ne.jp> wrote:
>
> Hello, Peter, Ingo and Will.
>
> (Q1) Can we change the capacity using kernel config?
>
> (Q2) If we can change the capacity, is it OK to specify these constants
>      independently? (In other words, is there inter-dependency among
>      these constants?)

I think we should do this.
syzbot uses a very beefy kernel config and very broad load.
We are hitting "BUG: MAX_LOCKDEP_ENTRIES too low!" for the part 428
days and already hit it 96K times. It's just harming overall kernel
testing:
https://syzkaller.appspot.com/bug?id=3d97ba93fb3566000c1c59691ea427370d33ea1b

I think it's better if exact values are not hardcoded, but rather
specified in the config. Today we are switching from 4K to 8K, but as
we enable more configs and learn to reach more code, we may need 16K.

> (Q3) Do you think that we can extend lockdep to be used as a tool for auditing
>      locks held in kernel space and rebuilding lock dependency map in user space?

This looks like lots of work. Also unpleasant dependencies on
user-space. If there is a user-space component, it will need to be
deployed to _all_ of kernel testing systems and for all users of
syzkaller. And it will also be a dependency for reproducers. Currently
one can run a C reproducer and get the errors message from LOCKDEP. It
seems that a user-space component will make it significantly more
complicated.

> On 2020/07/25 14:23, Tetsuo Handa wrote:
> >> Also somebody may use it to _reduce_ size of the table for a smaller kernel.
> >
> > Maybe. But my feeling is that it is very rare that the kernel actually deadlocks
> > as soon as lockdep warned the possibility of deadlock.
> >
> > Since syzbot runs many instances in parallel, a lot of CPU resource is spent for
> > checking the same dependency tree. However, the possibility of deadlock can be
> > warned for only locks held within each kernel boot, and it is impossible to hold
> > all locks with one kernel boot.
> >
> > Then, it might be nice if lockdep can audit only "which lock was held from which
> > context and what backtrace" and export that log like KCOV data (instead of evaluating
> > the possibility of deadlock), and rebuild the whole dependency (and evaluate the
> > possibility of deadlock) across multiple kernel boots in userspace.
>