From: "J. Bruce Fields" <bfields@fieldses.org>
To: Frank Sorenson <sorenson@redhat.com>
Cc: Roberto Bergantinos Corpas <rbergant@redhat.com>,
chuck.lever@oracle.com, trond.myklebust@hammerspace.com,
linux-nfs@vger.kernel.org
Subject: Re: [PATCH] sunrpc: expiry_time should be seconds not timeval
Date: Fri, 24 Jan 2020 16:23:06 -0500 [thread overview]
Message-ID: <20200124212306.GD26874@fieldses.org> (raw)
In-Reply-To: <438af54f-e1a8-467a-4ef1-821b67b7bb6c@redhat.com>
On Fri, Jan 24, 2020 at 10:53:19AM -0600, Frank Sorenson wrote:
> On 1/24/20 4:11 AM, Roberto Bergantinos Corpas wrote:
> > When upcalling gssproxy, cache_head.expiry_time is set as a
> > timeval, not seconds since boot. As such, RPC cache expiry
> > logic will not clean expired objects created under
> > auth.rpcsec.context cache.
Looks like expiration times have worked this way since 2010's
c5b29f885afe "sunrpc: use seconds since boot in expiry cache".
gss_proxy_save_rsc was added in 2012 with 030d794bf498 "SUNRPC: Use
gssproxy upcall for server RPCGSS authentication", so it's the gssproxy
code that introduced the bug. That's a while for this to lurk, but it
sounds like it required a bit of an extreme case to make it obvious.
Applying with a stable cc, Frank's Tested-by and a note on the above.
Thanks, everyone!
--b.
> >
> > This has proven to cause kernel memory leaks on field.
> >
> > Signed-off-by: Roberto Bergantinos Corpas <rbergant@redhat.com>
> > ---
> > net/sunrpc/auth_gss/svcauth_gss.c | 4 ++++
> > 1 file changed, 4 insertions(+)
> >
> > diff --git a/net/sunrpc/auth_gss/svcauth_gss.c b/net/sunrpc/auth_gss/svcauth_gss.c
> > index 8be2f209982b..725cf5b5ae40 100644
> > --- a/net/sunrpc/auth_gss/svcauth_gss.c
> > +++ b/net/sunrpc/auth_gss/svcauth_gss.c
> > @@ -1211,6 +1211,7 @@ static int gss_proxy_save_rsc(struct cache_detail *cd,
> > dprintk("RPC: No creds found!\n");
> > goto out;
> > } else {
> > + struct timespec boot;
> >
> > /* steal creds */
> > rsci.cred = ud->creds;
> > @@ -1231,6 +1232,9 @@ static int gss_proxy_save_rsc(struct cache_detail *cd,
> > &expiry, GFP_KERNEL);
> > if (status)
> > goto out;
> > +
> > + getboottime(&boot);
> > + expiry -= boot.tv_sec;
> > }
> >
> > rsci.h.expiry_time = expiry;
> >
>
> The accumulating become apparent when the client uses kerberos tickets
> with very short (10 seconds or fewer) lifetimes and renewable lifetimes:
>
> mount server:/exports /mnt/tmp -overs=4,sec=krb5p
> life="2s"
> rlife="3s"
> while true ; do
> while true ; do
> kinit -l $life -R >/dev/null 2>&1 && break
> echo 'PASSWORD' | kinit -l $life -r $rlife \
> >/dev/null 2>&1 && break
> done
> timeout -k 1 2 touch /mnt/tmp/foo
> echo -n .
> done
>
> Due to the entry expiration occurring 50 years in the future, the
> customer had accumulated in excess of 400,000 entries in the cache over
> about a month with just 6 nfs clients. The entries, with all the
> accompanying structs which had been allocated consumed over 2 GiB from
> various slab caches.
>
> A flush of the cache cleans everything out, however they will again
> accumulate afterward.
>
> This patch fixes the expiration issue.
>
> Tested-By: Frank Sorenson <sorenson@redhat.com>
>
>
> Frank
> --
> Frank Sorenson
> sorenson@redhat.com
> Principal Software Maintenance Engineer
> Global Support Services - filesystems
> Red Hat
>
>
>
>
>
>
>
next prev parent reply other threads:[~2020-01-24 21:23 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-24 10:11 [PATCH] sunrpc: expiry_time should be seconds not timeval Roberto Bergantinos Corpas
2020-01-24 16:53 ` Frank Sorenson
2020-01-24 21:23 ` J. Bruce Fields [this message]
2020-02-04 10:32 Roberto Bergantinos Corpas
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200124212306.GD26874@fieldses.org \
--to=bfields@fieldses.org \
--cc=chuck.lever@oracle.com \
--cc=linux-nfs@vger.kernel.org \
--cc=rbergant@redhat.com \
--cc=sorenson@redhat.com \
--cc=trond.myklebust@hammerspace.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).