From: Dmitry Vyukov <email@example.com>
To: Steven Rostedt <firstname.lastname@example.org>
Cc: syzkaller <email@example.com>,
Andrew Morton <firstname.lastname@example.org>,
Jens Axboe <email@example.com>, Borislav Petkov <firstname.lastname@example.org>,
"H. Peter Anvin" <email@example.com>,
Jim Mattson <firstname.lastname@example.org>, Joerg Roedel <email@example.com>,
KVM list <firstname.lastname@example.org>,
Mark Rutland <email@example.com>,
firstname.lastname@example.org, Ingo Molnar <email@example.com>,
Paolo Bonzini <firstname.lastname@example.org>,
Peter Zijlstra <email@example.com>,
Sean Christopherson <firstname.lastname@example.org>,
Thomas Gleixner <email@example.com>,
Vitaly Kuznetsov <firstname.lastname@example.org>,
Wanpeng Li <email@example.com>, Will Deacon <firstname.lastname@example.org>,
"the arch/x86 maintainers" <email@example.com>
Subject: Re: Bisections with different bug manifestations
Date: Fri, 16 Apr 2021 15:48:55 +0200 [thread overview]
Message-ID: <CACT4Y+ZdDwSthQ4tWcoig3bKejgpEYDOo8QcSPjBkg3=b618Ww@mail.gmail.com> (raw)
On Fri, Apr 16, 2021 at 3:26 PM Dmitry Vyukov <firstname.lastname@example.org> wrote:
> On Fri, Apr 16, 2021 at 3:13 PM Steven Rostedt <email@example.com> wrote:
> > On Fri, 16 Apr 2021 09:51:45 +0200
> > Dmitry Vyukov <firstname.lastname@example.org> wrote:
> > >
> > > If you look at substantial base of bisection logs, you will find lots
> > > of cases where bug types, functions don't match. Kernel crashes
> > > differently even on the same revision. And obviously things change if
> > > you change revisions. Also if you see presumably a different bug, what
> > > does it say regarding the original bug.
> > Yes, but there are also several types of cases where the issue will be the
> > same. Namely lockdep. I agree that use after free warnings can have a side
> > effect, and may be more difficult.
> But how do we know it's lockdep, rather than a use-after-free
> manifested as lockdep?
> A Significant portion of kernel bugs are caused by concurrency and can
> manifest in different ways, e.g. these are not lockdep, or WARN, or
> use-after-free, but rather a race in nature.
> > But there's many other bugs that remain
> > consistent across kernels. And if you stumble on one of them, look for it
> > only.
> For example? Does not look to be true for WARN, BUG, KASAN,
> "inconsistent lock state".
> > And if you hit another bug, and if it doesn't crash, then ignore it (of
> > course this could be an issue if you have panic on warning set). But
> > otherwise, just skip it.
> It's not possible to skip, say, BUG.
> And if we skip, say, a use-after-free, how do we know we are not
> making things worse? Because now we are running on corrupted memory,
> so anything can happen. Definitely a stray lockdep report can happen,
> or other way around not happen when it should...
> > > I would very much like to improve automatic bisection quality, but it
> > > does not look trivial at all.
> > >
> > > Some random examples where, say, your hypothesis of WARN-to-WARN,
> > > BUG-to-BUG does not hold even on the same kernel revision (add to this
> > At least lockdep to lockdep, as when I do manual bisects, that's exactly
> > what I look for, and ignore all other warnings. And that has found the
> > problem commit pretty much every time.
> What lockdep bug types do you mean? All?
> In the examples above you can see at least "inconsistent lock state"
> mixed with 2 other completely different bug types.
I've looked for some examples.
Here a reliable deadlock in io_uring changes to reliable task hung in io_uring:
run #6: crashed: possible deadlock in io_poll_double_wake
all runs: crashed: INFO: task hung in io_ring_exit_work
Here deadlock is mixed with "lock held when returning to user space":
run #6: crashed: possible deadlock in inet_stream_connect
run #7: crashed: WARNING: lock held when returning to user space in
Here a reliable deadlock in n_tty_receive_buf_common changes to UAF in
all runs: crashed: possible deadlock in n_tty_receive_buf_common
all runs: crashed: KASAN: use-after-free Read in n_tty_receive_buf_common
Here deadlock changes to UAF and corrupted list:
run #4: crashed: possible deadlock in neigh_change_state
run #1: crashed: KASAN: use-after-free Read in neigh_mark_dead
run #2: crashed: BUG: corrupted list in ___neigh_create
run #3: crashed: BUG: corrupted list in neigh_mark_dead
But it also shows another interesting aspect, kernel output is
sometimes corrupted and kernel does not manage to print proper
complete oops message. In these cases we can't reliably say that it's
also a deadlock or not, we can't say anything about the crash, it's
unparsable. This problem also shows up here:
and in the second one also these 2 are mixed:
run #1: crashed: WARNING: locking bug in corrupted
run #2: crashed: WARNING: possible circular locking dependency detected
This one shows a related issue. Kernel crash output periodically
changes, and a testing system fails to parse it. This can happen if we
go back to older releases, or when we do fix bisection and test a very
new kernel with changed output:
It's also easy to imagine that a program that triggers 2 deadlocks on
different revisions, both are deadlocks, but completely different and
unrelated. And we can't look at function names as they change over
All these things are almost trivial when you do bisection manually.
But for an automated system, they are quite challenging to handle.
> > > different revisions and the fact that a different bug does not give
> > > info regarding the original bug):
> > >
> > Can you tell me that all these examples bisected to the commit that caused
> > the bug? Because if it did not, then you may have just proved my point ;-)
> I don't know now what was the result, but for a single run these were
> manifestations of the same root bug.
> E.g. see below, that's UAF in fuse_dev_do_read vs WARNING in
> request_end. request_end is also fuse. And you can see that a memory
> corruption causing a random bug type, in this case WARNING, but can as
> well be LOCKDEP.
> > > run #0: crashed: KASAN: use-after-free Read in fuse_dev_do_read
> > > run #1: crashed: WARNING in request_end
> > > run #2: crashed: KASAN: use-after-free Read in fuse_dev_do_read
> > > run #3: OK
> > > run #4: OK
next prev parent reply other threads:[~2021-04-16 13:49 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-13 17:33 [syzbot] possible deadlock in del_gendisk syzbot
2021-04-13 17:41 ` Steven Rostedt
2021-04-13 17:43 ` Steven Rostedt
2021-04-13 18:24 ` Dmitry Vyukov
2021-04-13 18:40 ` Steven Rostedt
2021-04-13 18:43 ` Steven Rostedt
2021-04-16 7:51 ` Bisections with different bug manifestations Dmitry Vyukov
2021-04-16 13:13 ` Steven Rostedt
2021-04-16 13:26 ` Dmitry Vyukov
2021-04-16 13:48 ` Dmitry Vyukov [this message]
2021-06-07 10:56 ` [syzbot] possible deadlock in del_gendisk Tetsuo Handa
2021-06-09 16:31 ` Tetsuo Handa
2021-06-09 16:46 ` Tyler Hicks
2021-06-10 13:37 ` Tetsuo Handa
2021-06-11 6:46 ` Tetsuo Handa
2021-06-11 15:14 ` [PATCH] loop: drop loop_ctl_mutex around del_gendisk() in loop_remove() Tetsuo Handa
2021-06-15 5:30 ` Tetsuo Handa
2021-06-15 5:31 ` syzbot
2021-06-15 9:04 ` Jan Kara
2021-06-11 14:47 ` [syzbot] possible deadlock in del_gendisk Tetsuo Handa
2021-06-11 15:11 ` Pavel Tatashin
2021-06-11 15:18 ` Pavel Tatashin
2021-06-11 15:49 ` Tetsuo Handa
2021-06-12 2:35 ` Tetsuo Handa
2021-06-13 11:01 ` Tetsuo Handa
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).