Re: [PATCH bpf] bpf: cgroup: prevent out-of-order release of cgroup bpf

From: Roman Gushchin <guro@fb.com>
To: Alexei Starovoitov <alexei.starovoitov@gmail.com>,
	"tj@kernel.org" <tj@kernel.org>
Cc: "bpf@vger.kernel.org" <bpf@vger.kernel.org>,
	Alexei Starovoitov <ast@kernel.org>,
	Daniel Borkmann <daniel@iogearbox.net>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Kernel Team <Kernel-team@fb.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"stable@vger.kernel.org" <stable@vger.kernel.org>
Subject: Re: [PATCH bpf] bpf: cgroup: prevent out-of-order release of cgroup bpf
Date: Sat, 4 Jan 2020 03:00:46 +0000	[thread overview]
Message-ID: <20200104030041.GA12685@localhost.localdomain> (raw)
In-Reply-To: <20200104023112.6edfdvsff6cgsstn@ast-mbp>

On Fri, Jan 03, 2020 at 06:31:14PM -0800, Alexei Starovoitov wrote:
> On Sat, Jan 04, 2020 at 01:13:24AM +0000, Roman Gushchin wrote:
> > On Fri, Jan 03, 2020 at 04:35:25PM -0800, Alexei Starovoitov wrote:
> > > On Fri, Dec 27, 2019 at 01:50:34PM -0800, Roman Gushchin wrote:
> > > > Before commit 4bfc0bb2c60e ("bpf: decouple the lifetime of cgroup_bpf
> > > > from cgroup itself") cgroup bpf structures were released with
> > > > corresponding cgroup structures. It guaranteed the hierarchical order
> > > > of destruction: children were always first. It preserved attached
> > > > programs from being released before their propagated copies.
> > > > 
> > > > But with cgroup auto-detachment there are no such guarantees anymore:
> > > > cgroup bpf is released as soon as the cgroup is offline and there are
> > > > no live associated sockets. It means that an attached program can be
> > > > detached and released, while its propagated copy is still living
> > > > in the cgroup subtree. This will obviously lead to an use-after-free
> > > > bug.
> > > ...
> > > > @@ -65,6 +65,9 @@ static void cgroup_bpf_release(struct work_struct *work)
> > > >  
> > > >  	mutex_unlock(&cgroup_mutex);
> > > >  
> > > > +	for (p = cgroup_parent(cgrp); p; p = cgroup_parent(p))
> > > > +		cgroup_bpf_put(p);
> > > > +
> > > 
> > > The fix makes sense, but is it really safe to walk cgroup hierarchy
> > > without holding cgroup_mutex?
> > 
> > It is, because we're holding a reference to the original cgroup and going
> > towards the root. On each level the cgroup is protected by a reference
> > from their child cgroup.
> 
> cgroup_bpf_put(p) can make bpf.refcnt zero which may call cgroup_bpf_release()
> on another cpu which will do cgroup_put() and this cpu p = cgroup_parent(p)
> would be use-after-free?
> May be not due to the way work_queues are implemented.
> But it feels dangerous to have such delicate release logic.

If I understand your concern correctly: you assume that parent's
cgroup_bpf_release() can be finished prior to the child's one and
the final cgroup_put() will release the parent?

If so, it's not possible, because the child hold a reference to the
parent (independent to all cgroup bpf stuff), which exists at least
until the final cgroup_put() in cgroup_bpf_release(). Please, look
at css_free_rwork_fn() for details.

> Why not to move the loop under the mutex and make things obvious?

Traversing the cgroup tree to the root cgroup without additional
locking seems pretty common to me. You can find a ton of examples in
mm/memcontrol.c. So it doesn't look scary or adventurous to me.

I think it doesn't matter that much here, so I'm ok with putting it
under the mutex, but IMO it won't make the code any safer.

cc Tejun for the second opinion on cgroup locking