From: Ivan Babrou <ivan@cloudflare.com>
To: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>,
kernel-team <kernel-team@cloudflare.com>,
Ignat Korchagin <ignat@cloudflare.com>,
Hailong liu <liu.hailong6@zte.com.cn>,
Andrey Ryabinin <aryabinin@virtuozzo.com>,
Alexander Potapenko <glider@google.com>,
Dmitry Vyukov <dvyukov@google.com>,
Andrew Morton <akpm@linux-foundation.org>,
Thomas Gleixner <tglx@linutronix.de>,
Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>,
x86@kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
Miroslav Benes <mbenes@suse.cz>,
Julien Thierry <jthierry@redhat.com>,
Jiri Slaby <jirislaby@kernel.org>,
kasan-dev@googlegroups.com, linux-mm@kvack.org,
linux-kernel <linux-kernel@vger.kernel.org>,
Alasdair Kergon <agk@redhat.com>,
Mike Snitzer <snitzer@redhat.com>,
dm-devel@redhat.com,
"Steven Rostedt (VMware)" <rostedt@goodmis.org>,
Alexei Starovoitov <ast@kernel.org>,
Daniel Borkmann <daniel@iogearbox.net>,
Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>,
Yonghong Song <yhs@fb.com>, Andrii Nakryiko <andriin@fb.com>,
John Fastabend <john.fastabend@gmail.com>,
KP Singh <kpsingh@chromium.org>, Robert Richter <rric@kernel.org>,
"Joel Fernandes (Google)" <joel@joelfernandes.org>,
Mathieu Desnoyers <mathieu.desnoyers@efficios.com>,
Linux Kernel Network Developers <netdev@vger.kernel.org>,
bpf@vger.kernel.org, Alexey Kardashevskiy <aik@ozlabs.ru>
Subject: Re: BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1df5/0x2650
Date: Wed, 3 Feb 2021 16:52:42 -0800 [thread overview]
Message-ID: <CABWYdi0p91Y+TDUu38eey-p2GtxL6f=VHicTxS629VCMmrNLpQ@mail.gmail.com> (raw)
In-Reply-To: <20210204001700.ry6dpqvavcswyvy7@treble>
On Wed, Feb 3, 2021 at 4:17 PM Josh Poimboeuf <jpoimboe@redhat.com> wrote:
>
> On Wed, Feb 03, 2021 at 03:30:35PM -0800, Ivan Babrou wrote:
> > > > > Can you recreate with this patch, and add "unwind_debug" to the cmdline?
> > > > > It will spit out a bunch of stack data.
> > > >
> > > > Here's the three I'm building:
> > > >
> > > > * https://github.com/bobrik/linux/tree/ivan/static-call-5.9
> > > >
> > > > It contains:
> > > >
> > > > * v5.9 tag as the base
> > > > * static_call-2020-10-12 tag
> > > > * dm-crypt patches to reproduce the issue with KASAN
> > > > * x86/unwind: Add 'unwind_debug' cmdline option
> > > > * tracepoint: Fix race between tracing and removing tracepoint
> > > >
> > > > The very same issue can be reproduced on 5.10.11 with no patches,
> > > > but I'm going with 5.9, since it boils down to static call changes.
> > > >
> > > > Here's the decoded stack from the kernel with unwind debug enabled:
> > > >
> > > > * https://gist.github.com/bobrik/ed052ac0ae44c880f3170299ad4af56b
> > > >
> > > > See my first email for the exact commands that trigger this.
> > >
> > > Thanks. Do you happen to have the original dmesg, before running it
> > > through the post-processing script?
> >
> > Yes, here it is:
> >
> > * https://gist.github.com/bobrik/8c13e6a02555fb21cadabb74cdd6f9ab
>
> It appears the unwinder is getting lost in crypto code. No idea what
> this has to do with static calls though. Or maybe you're seeing
> multiple issues.
>
> Does this fix it?
It does for the dm-crypt case! But so does the following commit in
5.11 (and 5.10.12):
* https://github.com/torvalds/linux/commit/ce8f86ee94?w=1
The reason I stuck to dm-crypt reproduction is that it reproduces reliably.
We also have the following stack that doesn't touch any crypto:
* https://gist.github.com/bobrik/40e2559add2f0b26ae39da30dc451f1e
I cannot reproduce this one, and it took 2 days of uptime for it to
happen. Is there anything I can do to help diagnose it?
My goal is to enable multishot KASAN in our pre-production
environment, but currently it sometimes starves TX queues on the NIC
due to multiple reports in a row in an interrupt about
unwind_next_frame, which disables network interface, which is not
something we can tolerate.
next prev parent reply other threads:[~2021-02-04 0:54 UTC|newest]
Thread overview: 16+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <CABWYdi3HjduhY-nQXzy2ezGbiMB1Vk9cnhW2pMypUa+P1OjtzQ@mail.gmail.com>
2021-02-03 3:09 ` BUG: KASAN: stack-out-of-bounds in unwind_next_frame+0x1df5/0x2650 Ivan Babrou
2021-02-03 16:46 ` Peter Zijlstra
2021-02-03 17:46 ` Ivan Babrou
2021-02-03 19:05 ` Josh Poimboeuf
2021-02-03 22:41 ` Ivan Babrou
2021-02-03 23:27 ` Josh Poimboeuf
2021-02-03 23:30 ` Ivan Babrou
2021-02-04 0:17 ` Josh Poimboeuf
2021-02-04 0:52 ` Ivan Babrou [this message]
2021-02-04 2:37 ` Josh Poimboeuf
2021-02-04 19:51 ` Ivan Babrou
2021-02-04 20:22 ` Josh Poimboeuf
2021-02-04 9:22 ` Peter Zijlstra
2021-02-04 2:44 ` Steven Rostedt
2021-02-04 3:09 ` Josh Poimboeuf
2021-02-04 18:41 ` Ivan Babrou
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CABWYdi0p91Y+TDUu38eey-p2GtxL6f=VHicTxS629VCMmrNLpQ@mail.gmail.com' \
--to=ivan@cloudflare.com \
--cc=agk@redhat.com \
--cc=aik@ozlabs.ru \
--cc=akpm@linux-foundation.org \
--cc=andriin@fb.com \
--cc=aryabinin@virtuozzo.com \
--cc=ast@kernel.org \
--cc=bp@alien8.de \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=dm-devel@redhat.com \
--cc=dvyukov@google.com \
--cc=glider@google.com \
--cc=hpa@zytor.com \
--cc=ignat@cloudflare.com \
--cc=jirislaby@kernel.org \
--cc=joel@joelfernandes.org \
--cc=john.fastabend@gmail.com \
--cc=jpoimboe@redhat.com \
--cc=jthierry@redhat.com \
--cc=kafai@fb.com \
--cc=kasan-dev@googlegroups.com \
--cc=kernel-team@cloudflare.com \
--cc=kpsingh@chromium.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=liu.hailong6@zte.com.cn \
--cc=mathieu.desnoyers@efficios.com \
--cc=mbenes@suse.cz \
--cc=mingo@redhat.com \
--cc=netdev@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=rostedt@goodmis.org \
--cc=rric@kernel.org \
--cc=snitzer@redhat.com \
--cc=songliubraving@fb.com \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).