BPF Archive on lore.kernel.org
 help / color / Atom feed
From: Alexei Starovoitov <alexei.starovoitov@gmail.com>
To: Roman Gushchin <guro@fb.com>
Cc: "tj@kernel.org" <tj@kernel.org>,
	"bpf@vger.kernel.org" <bpf@vger.kernel.org>,
	Alexei Starovoitov <ast@kernel.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Kernel Team <Kernel-team@fb.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: Re: [PATCH bpf] bpf: cgroup: prevent out-of-order release of cgroup bpf
Date: Mon, 6 Jan 2020 14:07:48 -0800
Message-ID: <20200106220746.fm3hp3zynaiaqgly@ast-mbp> (raw)
In-Reply-To: <20200104030041.GA12685@localhost.localdomain>

On Sat, Jan 04, 2020 at 03:00:46AM +0000, Roman Gushchin wrote:
> On Fri, Jan 03, 2020 at 06:31:14PM -0800, Alexei Starovoitov wrote:
> > On Sat, Jan 04, 2020 at 01:13:24AM +0000, Roman Gushchin wrote:
> > > On Fri, Jan 03, 2020 at 04:35:25PM -0800, Alexei Starovoitov wrote:
> > > > On Fri, Dec 27, 2019 at 01:50:34PM -0800, Roman Gushchin wrote:
> > > > > Before commit 4bfc0bb2c60e ("bpf: decouple the lifetime of cgroup_bpf
> > > > > from cgroup itself") cgroup bpf structures were released with
> > > > > corresponding cgroup structures. It guaranteed the hierarchical order
> > > > > of destruction: children were always first. It preserved attached
> > > > > programs from being released before their propagated copies.
> > > > > 
> > > > > But with cgroup auto-detachment there are no such guarantees anymore:
> > > > > cgroup bpf is released as soon as the cgroup is offline and there are
> > > > > no live associated sockets. It means that an attached program can be
> > > > > detached and released, while its propagated copy is still living
> > > > > in the cgroup subtree. This will obviously lead to an use-after-free
> > > > > bug.
> > > > ...
> > > > > @@ -65,6 +65,9 @@ static void cgroup_bpf_release(struct work_struct *work)
> > > > >  
> > > > >  	mutex_unlock(&cgroup_mutex);
> > > > >  
> > > > > +	for (p = cgroup_parent(cgrp); p; p = cgroup_parent(p))
> > > > > +		cgroup_bpf_put(p);
> > > > > +
> > > > 
> > > > The fix makes sense, but is it really safe to walk cgroup hierarchy
> > > > without holding cgroup_mutex?
> > > 
> > > It is, because we're holding a reference to the original cgroup and going
> > > towards the root. On each level the cgroup is protected by a reference
> > > from their child cgroup.
> > 
> > cgroup_bpf_put(p) can make bpf.refcnt zero which may call cgroup_bpf_release()
> > on another cpu which will do cgroup_put() and this cpu p = cgroup_parent(p)
> > would be use-after-free?
> > May be not due to the way work_queues are implemented.
> > But it feels dangerous to have such delicate release logic.
> 
> If I understand your concern correctly: you assume that parent's
> cgroup_bpf_release() can be finished prior to the child's one and
> the final cgroup_put() will release the parent?
> 
> If so, it's not possible, because the child hold a reference to the
> parent (independent to all cgroup bpf stuff), which exists at least
> until the final cgroup_put() in cgroup_bpf_release(). Please, look
> at css_free_rwork_fn() for details.
> 
> > Why not to move the loop under the mutex and make things obvious?
> 
> Traversing the cgroup tree to the root cgroup without additional
> locking seems pretty common to me. You can find a ton of examples in
> mm/memcontrol.c. So it doesn't look scary or adventurous to me.
> 
> I think it doesn't matter that much here, so I'm ok with putting it
> under the mutex, but IMO it won't make the code any safer.
> 
> 
> cc Tejun for the second opinion on cgroup locking

Checked with TJ offline. This seems fine.

I tweaked commit log:
- extra 'diff' lines were confusing 'git am'
- commit description shouldn't be split into multiline

And applied to bpf tree. Thanks

  reply index

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-27 21:50 Roman Gushchin
2020-01-03 15:30 ` Roman Gushchin
2020-01-03 17:47 ` Song Liu
2020-01-04  0:35 ` Alexei Starovoitov
2020-01-04  1:13   ` Roman Gushchin
2020-01-04  2:31     ` Alexei Starovoitov
2020-01-04  3:00       ` Roman Gushchin
2020-01-06 22:07         ` Alexei Starovoitov [this message]
2020-01-06 22:20           ` Roman Gushchin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200106220746.fm3hp3zynaiaqgly@ast-mbp \
    --to=alexei.starovoitov@gmail.com \
    --cc=Kernel-team@fb.com \
    --cc=ast@kernel.org \
    --cc=bpf@vger.kernel.org \
    --cc=daniel@iogearbox.net \
    --cc=guro@fb.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=netdev@vger.kernel.org \
    --cc=stable@vger.kernel.org \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

BPF Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/bpf/0 bpf/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 bpf bpf/ https://lore.kernel.org/bpf \
		bpf@vger.kernel.org
	public-inbox-index bpf

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.bpf


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git