RE: [PATCH 01/23] net, sunrpc: convert rpc_cred.cr_count from atomic_t to refcount_t

From: "Reshetova, Elena" <elena.reshetova@intel.com>
To: Trond Myklebust <trondmy@primarydata.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"jlayton@poochiereds.net" <jlayton@poochiereds.net>
Cc: "herbert@gondor.apana.org.au" <herbert@gondor.apana.org.au>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"peterz@infradead.org" <peterz@infradead.org>,
	"ralf@linux-mips.org" <ralf@linux-mips.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	"ishkamiel@gmail.com" <ishkamiel@gmail.com>,
	"bfields@fieldses.org" <bfields@fieldses.org>,
	"steffen.klassert@secunet.com" <steffen.klassert@secunet.com>,
	"nhorman@tuxdriver.com" <nhorman@tuxdriver.com>,
	"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
	"jreuter@yaina.de" <jreuter@yaina.de>,
	"keescook@chromium.org" <keescook@chromium.org>,
	"linux-hams@vger.kernel.org" <linux-hams@vger.kernel.org>,
	"dwindsor@gmail.com" <dwindsor@gmail.com>,
	"zyan@redhat.com" <zyan@redhat.com>, sage@redhat.
Subject: RE: [PATCH 01/23] net, sunrpc: convert rpc_cred.cr_count from atomic_t to refcount_t
Date: Mon, 20 Mar 2017 16:15:37 +0000	[thread overview]
Message-ID: <2236FBA76BA1254E88B949DDB74E612B41C5B3C5@IRSMSX102.ger.corp.intel.com> (raw)
In-Reply-To: <1489760913.8441.1.camel@primarydata.com>

> On Fri, 2017-03-17 at 09:02 -0400, Jeff Layton wrote:
> > On Fri, 2017-03-17 at 12:50 +0000, Trond Myklebust wrote:
> > > On Fri, 2017-03-17 at 14:10 +0200, Elena Reshetova wrote:
> > > > refcount_t type and corresponding API should be
> > > > used instead of atomic_t when the variable is used as
> > > > a reference counter. This allows to avoid accidental
> > > > refcounter overflows that might lead to use-after-free
> > > > situations.
> > > >
> > > > Signed-off-by: Elena Reshetova <elena.reshetova@intel.com>
> > > > Signed-off-by: Hans Liljestrand <ishkamiel@gmail.com>
> > > > Signed-off-by: Kees Cook <keescook@chromium.org>
> > > > Signed-off-by: David Windsor <dwindsor@gmail.com>
> > > > ---
> > > >  include/linux/sunrpc/auth.h |  8 ++++----
> > > >  net/sunrpc/auth.c           | 12 ++++++------
> > > >  2 files changed, 10 insertions(+), 10 deletions(-)
> > > >
> > > > diff --git a/include/linux/sunrpc/auth.h
> > > > b/include/linux/sunrpc/auth.h
> > > > index b1bc62b..bd36e0b 100644
> > > > --- a/include/linux/sunrpc/auth.h
> > > > +++ b/include/linux/sunrpc/auth.h
> > > > @@ -15,7 +15,7 @@
> > > >  #include <linux/sunrpc/msg_prot.h>
> > > >  #include <linux/sunrpc/xdr.h>
> > > >
> > > > -#include <linux/atomic.h>
> > > > +#include <linux/refcount.h>
> > > >  #include <linux/rcupdate.h>
> > > >  #include <linux/uidgid.h>
> > > >  #include <linux/utsname.h>
> > > > @@ -68,7 +68,7 @@ struct rpc_cred {
> > > >  #endif
> > > >  	unsigned long		cr_expire;	/* when
> > > > to gc
> > > > */
> > > >  	unsigned long		cr_flags;	/* various
> > > > flags */
> > > > -	atomic_t		cr_count;	/* ref count */
> > > > +	refcount_t		cr_count;	/* ref count
> > > > */
> > > >
> > >
> > > NACK. That's going to be hitting
> > > WARN_ONCE(!refcount_inc_not_zero(r),
> > > "refcount_t: increment on 0; use-after-free.\n") like there's no
> > > tomorrow...
> > >
> > > Please stop with these automated conversions. They are going to
> > > cause a
> > > lot more bugs than they fix.
> > >
> >
> > Agreed. These patchsets are touching places where we've already
> > banged
> > out most of the refcounting bugs. I'm against doing large scale
> > conversions like this without a damned good reason.
> >
> > I think it may be best to do this sort of thing in a more piecemeal
> > fashion. Pick a subsystem or two and do the conversions there to
> > prove
> > that they're better than what we have. If the subsystem already has
> > problems with its refcounting, then so much the better. Point to bugs
> > that this new infrastructure helped find.
> >
> > Encourage people to adopt your new infrastructure as new refcounted
> > objects are introduced into the kernel. You might even consider a LWN
> > article about this.
> >
> > Eventually we'll get around to changing existing code to use it, once
> > there is a sufficient advantage to doing so. Most likely when we're
> > reworking the code for other reasons, or when we're chasing some
> > horrid
> > refcounting bug and think that this might help find it.
> 
> The main issue is that this "refcount_t" implementation appears to be
> assuming that there is one and only one model for refcounts (the one
> where a value of "0" means "free me immediately").
> 
> The kernel has a plethora of object caching implementations where this
> is simply not the case; the dcache is a prime example, and this cache
> is another. In both these implementation, the atomic_t variable is
> being used more as a semaphore-style lock that prevents freeing of the
> object while it is in active use as opposed to being freeable, but
> cached. This is why these automated conversions are a nuisance and a
> source of bugs.

Ok, in this particular patch I agree that we missed that object is being reused 
(and yes there are many parts in kernel where similar thing happens as we learned from this exercise). 
Note that refcount_t implementation is fine with you "correctly" reusing your object:
i.e. when counter reaches zero, you take the object away from active use, but it might still stay in cache. 
BUT when you get a new object from cache you should initialize refcounter properly: set it to one vs. just do a "inc" on it. 
Problem really comes from this "increment me from zero". 

And the goal with these conversions is to take a look broadly on the kernel source and determine (with the feedback from maintainers who know code best, like your feedback now) what can be converted already now. 
Maintainers know their code and their usage of counters, so if it doesn't make sense to do it in a particular place (because of errors or other reasons), then it doesn't. 
But more we cover with new refcount_t, less chances we have with ever hitting refcounter bugs anywhere in the future. 

Best Regards,
Elena.

> 
> --
> Trond Myklebust
> Linux NFS client maintainer, PrimaryData
> trond.myklebust@primarydata.com