All of lore.kernel.org
 help / color / mirror / Atom feed
From: John Fastabend <john.fastabend@gmail.com>
To: Eric Biggers <ebiggers@kernel.org>, Al Viro <viro@zeniv.linux.org.uk>
Cc: Hillf Danton <hdanton@sina.com>,
	syzbot <syzbot+d88a977731a9888db7ba@syzkaller.appspotmail.com>,
	"linux-fsdevel@vger.kernel.org" <linux-fsdevel@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"syzkaller-bugs@googlegroups.com"
	<syzkaller-bugs@googlegroups.com>,
	Alexei Starovoitov <ast@kernel.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	Martin KaFai Lau <kafai@fb.com>, Song Liu <songliubraving@fb.com>,
	Yonghong Song <yhs@fb.com>,
	bpf@vger.kernel.org, Boris Pismenny <borisp@mellanox.com>,
	Aviad Yehezkel <aviadye@mellanox.com>,
	Dave Watson <davejwatson@fb.com>,
	John Fastabend <john.fastabend@gmail.com>
Subject: Re: kernel panic: corrupted stack end in dput
Date: Wed, 03 Jul 2019 09:14:46 -0700	[thread overview]
Message-ID: <5d1cd47644b45_8ce2b1bd49125c4ed@john-XPS-13-9370.notmuch> (raw)
In-Reply-To: <20190703154543.GA21629@sol.localdomain>

Eric Biggers wrote:
> [+bpf and tls maintainers]
> 
> On Wed, Jul 03, 2019 at 04:23:34PM +0100, Al Viro wrote:
> > On Wed, Jul 03, 2019 at 03:40:00PM +0100, Al Viro wrote:
> > > On Wed, Jul 03, 2019 at 02:43:07PM +0800, Hillf Danton wrote:
> > > 
> > > > > This is very much *NOT* fine.
> > > > > 	1) trylock can fail from any number of reasons, starting
> > > > > with "somebody is going through the hash chain doing a lookup on
> > > > > something completely unrelated"
> > > > 
> > > > They are also a red light that we need to bail out of spiraling up
> > > > the directory hierarchy imho.
> > > 
> > > Translation: "let's leak the reference to parent, shall we?"
> > > 
> > > > > 	2) whoever had been holding the lock and whatever they'd
> > > > > been doing might be over right after we get the return value from
> > > > > spin_trylock().
> > > > 
> > > > Or after we send a mail using git. I don't know.
> > > > 
> > > > > 	3) even had that been really somebody adding children in
> > > > > the same parent *AND* even if they really kept doing that, rather
> > > > > than unlocking and buggering off, would you care to explain why
> > > > > dentry_unlist() called by __dentry_kill() and removing the victim
> > > > > from the list of children would be safe to do in parallel with that?
> > > > >
> > > > My bad. I have to walk around that unsafety.
> > > 
> > > WHAT unsafety?  Can you explain what are you seeing and how to
> > > reproduce it, whatever it is?
> > 
> > BTW, what makes you think that it's something inside dput() itself?
> > All I see is that at some point in the beginning of the loop body
> > in dput() we observe a buggered stack.
> > 
> > Is that the first iteration through the loop?  IOW, is that just
> > the place where we first notice preexisting corruption, or is
> > that something the code called from that loop does?  If it's
> > a stack overflow, I would be very surprised to see it here -
> > dput() is iterative and it's called on a very shallow stack in
> > those traces.
> > 
> > What happens if you e.g. turn that
> > 	dput(dentry);
> > in __fput() into
> > 	rcu_read_lock(); rcu_read_unlock(); // trigger the check
> > 	dput(dentry);
> > 
> > and run your reporducer?
> > 
> 
> Please don't waste your time on this, it looks like just another report from the
> massive memory corruption in BPF and/or TLS.  Look at reproducer:
> 
> bpf$MAP_CREATE(0x0, &(0x7f0000000280)={0xf, 0x4, 0x4, 0x400, 0x0, 0x1}, 0x3c)
> socket$rxrpc(0x21, 0x2, 0x800000000a)
> r0 = socket$inet6_tcp(0xa, 0x1, 0x0)
> setsockopt$inet6_tcp_int(r0, 0x6, 0x13, &(0x7f00000000c0)=0x100000001, 0x1d4)
> connect$inet6(r0, &(0x7f0000000140), 0x1c)
> bpf$MAP_CREATE(0x0, &(0x7f0000000000)={0x5}, 0xfffffffffffffdcb)
> bpf$MAP_CREATE(0x2, &(0x7f0000003000)={0x3, 0x0, 0x77fffb, 0x0, 0x10020000000, 0x0}, 0x2c)
> setsockopt$inet6_tcp_TCP_ULP(r0, 0x6, 0x1f, &(0x7f0000000040)='tls\x00', 0x4)
> 
> It's the same as like 20 other syzbot reports.

There is a missing synchronize_rcu we need to add and we have a race
between map_free and tls close at the moment. The race cuases us to
incorrectly set the sk->prot pointers when tls socket is closed in
this case. I've added a hook to the ULP side now that should let
the map_free reset the saved sk->prot pointers on the TLS side and
am testing this now.

The 20 syzbot reports appear to all be due to these two issues.
This has nothing to do with dput().

Thanks,
John

> 
> - Eric



  reply	other threads:[~2019-07-03 16:14 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <20190703064307.13740-1-hdanton@sina.com>
2019-07-03 14:40 ` kernel panic: corrupted stack end in dput Al Viro
2019-07-03 15:23   ` Al Viro
2019-07-03 15:45     ` Eric Biggers
2019-07-03 16:14       ` John Fastabend [this message]
2019-07-01  8:27 syzbot
2019-07-01 12:14 ` syzbot
2019-07-02 13:21 ` Al Viro

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=5d1cd47644b45_8ce2b1bd49125c4ed@john-XPS-13-9370.notmuch \
    --to=john.fastabend@gmail.com \
    --cc=ast@kernel.org \
    --cc=aviadye@mellanox.com \
    --cc=borisp@mellanox.com \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=davejwatson@fb.com \
    --cc=ebiggers@kernel.org \
    --cc=hdanton@sina.com \
    --cc=kafai@fb.com \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=songliubraving@fb.com \
    --cc=syzbot+d88a977731a9888db7ba@syzkaller.appspotmail.com \
    --cc=syzkaller-bugs@googlegroups.com \
    --cc=viro@zeniv.linux.org.uk \
    --cc=yhs@fb.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.