[PATCH 0/1] SUNRPC handle EKEYEXPIRED in call_refreshresult

[PATCH 0/1] SUNRPC handle EKEYEXPIRED in call_refreshresult

[andros]

From: Andy Adamson <andros@netapp.com>

Without this patch attempting to access a Kerberos mount with expired or no
credentials resulted in the NFS client hanging while retrying to refresh creds
for ever.

I tested NFSv3/v4/v4.1 sec=krb5 mounts. With expired or non-existent user
Kerberos credentials, trying to ls the mountpoint, or cd into the mountpoint
resulted in three failed upcalls to gssd (due to tk_cred_retry being set to 2)
then the 'Operation not permitted' message is returned to the user.

I think this patch should go into the stable kernel.

Andy Adamson (1):
  SUNRPC handle EKEYEXPIRED in call_refreshresult

 fs/nfs/nfs4proc.c |    2 --
 net/sunrpc/clnt.c |    4 ++++
 2 files changed, 4 insertions(+), 2 deletions(-)

-- 
1.7.7.6

[PATCH 1/1] SUNRPC handle EKEYEXPIRED in call_refreshresult

[andros]

From: Andy Adamson <andros@netapp.com>

When an RPCSEC_GSS context has expired or is non-existent, and the user
(Kerberos) credentials have also expired or are non-existent, the client
retries to refresh the context for ever and the application
hangs. The user is not prompted to refresh/establish their credentials.

Move the -EKEYEXPIRED handling into the RPC layer. Try tk_cred_retry number
of times to refresh the gss_context, and then pass -EPERM to application.

Signed-off-by: Andy Adamson <andros@netapp.com>
---
 fs/nfs/nfs4proc.c |    2 --
 net/sunrpc/clnt.c |    4 ++++
 2 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 5a7b372..2f291b3 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -342,7 +342,6 @@ static int nfs4_handle_exception(struct nfs_server *server, int errorcode, struc
 			}
 		case -NFS4ERR_GRACE:
 		case -NFS4ERR_DELAY:
-		case -EKEYEXPIRED:
 			ret = nfs4_delay(server->client, &exception->timeout);
 			if (ret != 0)
 				break;
@@ -3939,7 +3938,6 @@ nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,
 		case -NFS4ERR_DELAY:
 			nfs_inc_server_stats(server, NFSIOS_DELAY);
 		case -NFS4ERR_GRACE:
-		case -EKEYEXPIRED:
 			rpc_delay(task, NFS4_POLL_RETRY_MAX);
 			task->tk_status = 0;
 			return -EAGAIN;
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index f56f045..a94fc0c 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -1334,8 +1334,12 @@ call_refreshresult(struct rpc_task *task)
 		return;
 	case -ETIMEDOUT:
 		rpc_delay(task, 3*HZ);
+	case -EKEYEXPIRED:
+		status = -EPERM;
+		goto cred_retry;
 	case -EAGAIN:
 		status = -EACCES;
+cred_retry:
 		if (!task->tk_cred_retry)
 			break;
 		task->tk_cred_retry--;
-- 
1.7.7.6

Re: [PATCH 0/1] SUNRPC handle EKEYEXPIRED in call_refreshresult

[Jeff Layton]

On Wed, 27 Jun 2012 16:05:34 -0400
andros@netapp.com wrote:

> From: Andy Adamson <andros@netapp.com>
> 
> Without this patch attempting to access a Kerberos mount with expired or no
> credentials resulted in the NFS client hanging while retrying to refresh creds
> for ever.
> 
> I tested NFSv3/v4/v4.1 sec=krb5 mounts. With expired or non-existent user
> Kerberos credentials, trying to ls the mountpoint, or cd into the mountpoint
> resulted in three failed upcalls to gssd (due to tk_cred_retry being set to 2)
> then the 'Operation not permitted' message is returned to the user.
> 
> I think this patch should go into the stable kernel.
> 
> Andy Adamson (1):
>   SUNRPC handle EKEYEXPIRED in call_refreshresult
> 
>  fs/nfs/nfs4proc.c |    2 --
>  net/sunrpc/clnt.c |    4 ++++
>  2 files changed, 4 insertions(+), 2 deletions(-)
> 

Wait...is this really the behavior you want here?

We had many complaints from users of krb5 mounts where long-running
jobs would routinely fail when the ticket expired.

The compromise behavior that we worked out at that time was to treat an
expired credcache differently from a "no credcache" situation. gssd would
return EKEYEXPIRED if the credcache existed but was expired, and
EACCES otherwise. The kernel would then treat those errors
differently:

    http://permalink.gmane.org/gmane.linux.nfsv4/11019

With EKEYEXPIRED, we'd want RPCs to hang indefinitely until the tickets
were renewed. With EACCES, the call would return an error. The idea
there is that the user would kdestroy if he needed to unwedge his krb5
mount.

This patch makes it sound like you're wanting to revert that behavior.
Is that the case? If so, what about people trying to run long-running
tasks on a kerberized mount? Are they just SOL if their ticket isn't
renewed in time?

-- 
Jeff Layton <jlayton@redhat.com>

Re: [PATCH 0/1] SUNRPC handle EKEYEXPIRED in call_refreshresult

[Adamson, Andy]

On Jun 28, 2012, at 11:43 AM, Jeff Layton wrote:

> On Wed, 27 Jun 2012 16:05:34 -0400
> andros@netapp.com wrote:
> 
>> From: Andy Adamson <andros@netapp.com>
>> 
>> Without this patch attempting to access a Kerberos mount with expired or no
>> credentials resulted in the NFS client hanging while retrying to refresh creds
>> for ever.
>> 
>> I tested NFSv3/v4/v4.1 sec=krb5 mounts. With expired or non-existent user
>> Kerberos credentials, trying to ls the mountpoint, or cd into the mountpoint
>> resulted in three failed upcalls to gssd (due to tk_cred_retry being set to 2)
>> then the 'Operation not permitted' message is returned to the user.
>> 
>> I think this patch should go into the stable kernel.
>> 
>> Andy Adamson (1):
>>  SUNRPC handle EKEYEXPIRED in call_refreshresult
>> 
>> fs/nfs/nfs4proc.c |    2 --
>> net/sunrpc/clnt.c |    4 ++++
>> 2 files changed, 4 insertions(+), 2 deletions(-)
>> 
> 
> Wait...is this really the behavior you want here?

Yes. Just having the client hang with no indication to the user is wrong.

> 
> We had many complaints from users of krb5 mounts where long-running
> jobs would routinely fail when the ticket expired.

That is a Kerberos ticket management issue, not an NFS kernel client issue.
You have long-running jobs, then kinit -l,  run krenew, or use a keytab with a cron job, 
or use some other credential management software package.


> The compromise behavior that we worked out at that time was to treat an
> expired credcache differently from a "no credcache" situation. gssd would
> return EKEYEXPIRED if the credcache existed but was expired, and
> EACCES otherwise. The kernel would then treat those errors
> differently:

In both cases, EPERM is the correct response from the Linux NFS client, as 
the user has no permissions to do anything in the file system.

> 
>    http://permalink.gmane.org/gmane.linux.nfsv4/11019
> 
> With EKEYEXPIRED, we'd want RPCs to hang indefinitely until the tickets
> were renewed.

Sounds like a good DOS attack.  Consider V4.1 and a multi-user machine. If a
users credentials expire during a heavy I/O run - that user could be using all of the
session slots, and no other user could make progress while the RPCs call rpc_delay 
and retry  indefinitely...


> With EACCES, the call would return an error. The idea
> there is that the user would kdestroy if he needed to unwedge his krb5
> mount.

Exactly how is the user supposed to know to kdestroy? All they see is a hung mount.
 
> 
> This patch makes it sound like you're wanting to revert that behavior.
> Is that the case?

Yes.

> If so, what about people trying to run long-running
> tasks on a kerberized mount? Are they just SOL if their ticket isn't
> renewed in time?

Yes - as with _any_ resource, you need to plan ahead.  As I said above, the administrator in such a situation
needs to setup krenew or the equivalent.

-->Andy


> 
> -- 
> Jeff Layton <jlayton@redhat.com>

Re: [PATCH 0/1] SUNRPC handle EKEYEXPIRED in call_refreshresult

[Jeff Layton]

On Thu, 28 Jun 2012 16:31:41 +0000
"Adamson, Andy" <William.Adamson@netapp.com> wrote:

> 
> On Jun 28, 2012, at 11:43 AM, Jeff Layton wrote:
> 
> > On Wed, 27 Jun 2012 16:05:34 -0400
> > andros@netapp.com wrote:
> > 
> >> From: Andy Adamson <andros@netapp.com>
> >> 
> >> Without this patch attempting to access a Kerberos mount with expired or no
> >> credentials resulted in the NFS client hanging while retrying to refresh creds
> >> for ever.
> >> 
> >> I tested NFSv3/v4/v4.1 sec=krb5 mounts. With expired or non-existent user
> >> Kerberos credentials, trying to ls the mountpoint, or cd into the mountpoint
> >> resulted in three failed upcalls to gssd (due to tk_cred_retry being set to 2)
> >> then the 'Operation not permitted' message is returned to the user.
> >> 
> >> I think this patch should go into the stable kernel.
> >> 
> >> Andy Adamson (1):
> >>  SUNRPC handle EKEYEXPIRED in call_refreshresult
> >> 
> >> fs/nfs/nfs4proc.c |    2 --
> >> net/sunrpc/clnt.c |    4 ++++
> >> 2 files changed, 4 insertions(+), 2 deletions(-)
> >> 
> > 
> > Wait...is this really the behavior you want here?
> 
> Yes. Just having the client hang with no indication to the user is wrong.
> 

I presume you mean to say that that behavior isn't ideal. I tend to
agree, but there's no good way to report that to the user who can do
anything about it. I'll also point out that this scheme doesn't really
help that either. The user will end up with a failing job, at which
point it's too late to do anything about it...

> > 
> > We had many complaints from users of krb5 mounts where long-running
> > jobs would routinely fail when the ticket expired.
> 
> That is a Kerberos ticket management issue, not an NFS kernel client issue.
> You have long-running jobs, then kinit -l,  run krenew, or use a keytab with a cron job, 
> or use some other credential management software package.
> 

Easy to say, far more difficult to do. Most of the people who
complained about the non-robustness of this were people who were
running jobs that took days or weeks. They were understandably upset
when that job failed just because the ticket expired.

> 
> > The compromise behavior that we worked out at that time was to treat an
> > expired credcache differently from a "no credcache" situation. gssd would
> > return EKEYEXPIRED if the credcache existed but was expired, and
> > EACCES otherwise. The kernel would then treat those errors
> > differently:
> 
> In both cases, EPERM is the correct response from the Linux NFS client, as 
> the user has no permissions to do anything in the file system.
> 

But, in the case of an expired ticket, it's quite likely that he had
permissions at some point in time. The rationale at the time was that
if that user could reacquire creds he could keep his job going.

> > 
> >    http://permalink.gmane.org/gmane.linux.nfsv4/11019
> > 
> > With EKEYEXPIRED, we'd want RPCs to hang indefinitely until the tickets
> > were renewed.
> 
> Sounds like a good DOS attack.  Consider V4.1 and a multi-user machine. If a
> users credentials expire during a heavy I/O run - that user could be using all of the
> session slots, and no other user could make progress while the RPCs call rpc_delay 
> and retry  indefinitely...
> 

Well, no. That was the main reason we handled this in the NFS layer and
not in sunrpc. The rpc_task would exit with EKEYEXPIRED and the NFS
code would treat that like an NFS4ERR_DELAY. Back off and try again
later. Once the task has exited, any resources held in the rpc layer
including the slot should be available.

> 
> > With EACCES, the call would return an error. The idea
> > there is that the user would kdestroy if he needed to unwedge his krb5
> > mount.
> 
> Exactly how is the user supposed to know to kdestroy? All they see is a hung mount.
>  

We do throw a warning when the state manager's ticket expires. Perhaps
we could do something similar from gssd for user tickets. The point is
though that the user has the ability to unwedge the mount without
reacquiring the ticket if he so chooses.

> > 
> > This patch makes it sound like you're wanting to revert that behavior.
> > Is that the case?
> 
> Yes.
> 
> > If so, what about people trying to run long-running
> > tasks on a kerberized mount? Are they just SOL if their ticket isn't
> > renewed in time?
> 
> Yes - as with _any_ resource, you need to plan ahead.  As I said above, the administrator in such a situation
> needs to setup krenew or the equivalent.
> 

That's not helpful. Everyone makes mistakes and you don't necessarily
want your job to fail simply due to that fact. But regardless, Trond
NAK'ed a similar idea not that long ago:

    http://marc.info/?l=linux-nfs&m=132161606503398&w=2

...you may want to read over that thread as I'm fairly certain what
you're proposing will have the same issues...

-- 
Jeff Layton <jlayton@redhat.com>

Re: [PATCH 0/1] SUNRPC handle EKEYEXPIRED in call_refreshresult

[Andy Adamson]

On Thu, Jun 28, 2012 at 2:03 PM, Jeff Layton <jlayton@redhat.com> wrote:
> On Thu, 28 Jun 2012 16:31:41 +0000
> "Adamson, Andy" <William.Adamson@netapp.com> wrote:
>
>>
>> On Jun 28, 2012, at 11:43 AM, Jeff Layton wrote:
>>
>> > On Wed, 27 Jun 2012 16:05:34 -0400
>> > andros@netapp.com wrote:
>> >
>> >> From: Andy Adamson <andros@netapp.com>
>> >>
>> >> Without this patch attempting to access a Kerberos mount with expired or no
>> >> credentials resulted in the NFS client hanging while retrying to refresh creds
>> >> for ever.
>> >>
>> >> I tested NFSv3/v4/v4.1 sec=krb5 mounts. With expired or non-existent user
>> >> Kerberos credentials, trying to ls the mountpoint, or cd into the mountpoint
>> >> resulted in three failed upcalls to gssd (due to tk_cred_retry being set to 2)
>> >> then the 'Operation not permitted' message is returned to the user.
>> >>
>> >> I think this patch should go into the stable kernel.
>> >>
>> >> Andy Adamson (1):
>> >>  SUNRPC handle EKEYEXPIRED in call_refreshresult
>> >>
>> >> fs/nfs/nfs4proc.c |    2 --
>> >> net/sunrpc/clnt.c |    4 ++++
>> >> 2 files changed, 4 insertions(+), 2 deletions(-)
>> >>
>> >
>> > Wait...is this really the behavior you want here?
>>
>> Yes. Just having the client hang with no indication to the user is wrong.
>>
>
> I presume you mean to say that that behavior isn't ideal. I tend to
> agree, but there's no good way to report that to the user who can do
> anything about it. I'll also point out that this scheme doesn't really
> help that either. The user will end up with a failing job, at which
> point it's too late to do anything about it...
>
>> >
>> > We had many complaints from users of krb5 mounts where long-running
>> > jobs would routinely fail when the ticket expired.
>>
>> That is a Kerberos ticket management issue, not an NFS kernel client issue.
>> You have long-running jobs, then kinit -l,  run krenew, or use a keytab with a cron job,
>> or use some other credential management software package.
>>
>
> Easy to say, far more difficult to do.

Here is how simple it is.

$ kinit
$ krenew  -K 30&

This simple krenew command says to renew the current users TGT every
30 minutes for
as long as the TGT is renewable. For my setup, the TGT lasts 10 days -
long enough for most long-running jobs. The kinit is to get a fresh
TGT prior to running renew so that you start with as much time as
possible.

I just tested it. My TGS expires in 30 minutes, my TGT in 1 hour. 3
hours later, I'm still able to access the kerberos mount.

I'm sure most users with long running jobs could handle this. If you
need longer running jobs, then you simply talk to the Kerberos admin
and either get an extended TGT lifetime, or get a keytab and run a
cron job (or use krenew to run a script).


>Most of the people who
> complained about the non-robustness of this were people who were
> running jobs that took days or weeks. They were understandably upset
> when that job failed just because the ticket expired.

Well, they get what they setup. If they didn't ensure a long enough
ticket lifetime, then
they shouldn't be surprised when the access to the file system fails.

>>
>> > The compromise behavior that we worked out at that time was to treat an
>> > expired credcache differently from a "no credcache" situation. gssd would
>> > return EKEYEXPIRED if the credcache existed but was expired, and
>> > EACCES otherwise. The kernel would then treat those errors
>> > differently:
>>
>> In both cases, EPERM is the correct response from the Linux NFS client, as
>> the user has no permissions to do anything in the file system.
>>
>
> But, in the case of an expired ticket, it's quite likely that he had
> permissions at some point in time. The rationale at the time was that
> if that user could reacquire creds he could keep his job going.

Exactly! It's just not the NFS clients job to arrange this! It's a
Kerberos Ticket Lifetime Issue. Use the Kerberos Ticket Lifetime Issue
Tools!

>
>> >
>> >    http://permalink.gmane.org/gmane.linux.nfsv4/11019
>> >
>> > With EKEYEXPIRED, we'd want RPCs to hang indefinitely until the tickets
>> > were renewed.
>>
>> Sounds like a good DOS attack.  Consider V4.1 and a multi-user machine. If a
>> users credentials expire during a heavy I/O run - that user could be using all of the
>> session slots, and no other user could make progress while the RPCs call rpc_delay
>> and retry  indefinitely...
>>
>
> Well, no. That was the main reason we handled this in the NFS layer and
> not in sunrpc. The rpc_task would exit with EKEYEXPIRED and the NFS
> code would treat that like an NFS4ERR_DELAY. Back off and try again
> later.
> Once the task has exited, any resources held in the rpc layer
> including the slot should be available.

Ah - nfs4_async_handle_error on EKEYEXPIRED calls rpc_delay then
returs with an EAGAIN error code - so it  doesn't call rpc_exit, but
the EKEYEXPIRED is handled  the rpc_call_done routine which frees the
slot allowing it to be reused.

Fair enough.

After the delay timer expires, the task goes into the rpc_prepare_task
state to get a new slot, e.g. put on the session slot_tbl_waitq in
heavy I/O situations where all slots are being used.

So in my example above with an expired user GSS context on a
multi-user client, I/O gets stacked up in the session slot_tbl_waitq -
which I've seen to grow to hundreds of RPC requests depending on the
number of session slots and the amount of I/O.  Each I/O will keep on
returning to the session slot_tbl_waitq after an up call to GSSD which
fails with -EKEYEXPIRED.  Gssd gets pounded. The session
slot_tbl_waitq is not drained of this users I/O.

Other users will feel pain. Especially if there is more than one user
on the client irresponsible enough to start a long-running job without
handling their Kerberos credentials.

>
>>
>> > With EACCES, the call would return an error. The idea
>> > there is that the user would kdestroy if he needed to unwedge his krb5
>> > mount.
>>
>> Exactly how is the user supposed to know to kdestroy? All they see is a hung mount.
>>
>
> We do throw a warning when the state manager's ticket expires. Perhaps
> we could do something similar from gssd for user tickets. The point is
> though that the user has the ability to unwedge the mount without
> reacquiring the ticket if he so chooses.

And my point is - this is wrong!  It is a key management issue!  What
if the user doesn't choose to 'unwed'  the mount? Does it hang
forever?

If the user is going to have to make a choice, let them make it on the
front end, and setup their Kerberos tickets according to their need.

>
>> >
>> > This patch makes it sound like you're wanting to revert that behavior.
>> > Is that the case?
>>
>> Yes.
>>
>> > If so, what about people trying to run long-running
>> > tasks on a kerberized mount? Are they just SOL if their ticket isn't
>> > renewed in time?
>>
>> Yes - as with _any_ resource, you need to plan ahead.  As I said above, the administrator in such a situation
>> needs to setup krenew or the equivalent.
>>
>
> That's not helpful. Everyone makes mistakes and you don't necessarily
> want your job to fail simply due to that fact.

Yes, WRT security, you actually do.  There is a reason Kerberos
tickets have a lifetime.
The user needs to be aware of it, and extend or cut-off credentials
according to what their use case is.

> But regardless, Trond
> NAK'ed a similar idea not that long ago:
>
>    http://marc.info/?l=linux-nfs&m=132161606503398&w=2
>
> ...you may want to read over that thread as I'm fairly certain what
> you're proposing will have the same issues...

Your'e referring to data corruption due to buffered writes or commits
with expired GSS credentials?
I'm preparing patches for this issue as well. I'll resend this patch
with the others that
handle the expired GSS creds.

-->Andy

>
> --
> Jeff Layton <jlayton@redhat.com>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/1] SUNRPC handle EKEYEXPIRED in call_refreshresult

[Steve Dickson]

On 06/28/2012 07:33 PM, Andy Adamson wrote:
> On Thu, Jun 28, 2012 at 2:03 PM, Jeff Layton <jlayton@redhat.com> wrote:
>> On Thu, 28 Jun 2012 16:31:41 +0000
>> "Adamson, Andy" <William.Adamson@netapp.com> wrote:
>>
>>>
>>> On Jun 28, 2012, at 11:43 AM, Jeff Layton wrote:
>>>
>>>> On Wed, 27 Jun 2012 16:05:34 -0400
>>>> andros@netapp.com wrote:
>>>>
>>>>> From: Andy Adamson <andros@netapp.com>
>>>>>
>>>>> Without this patch attempting to access a Kerberos mount with expired or no
>>>>> credentials resulted in the NFS client hanging while retrying to refresh creds
>>>>> for ever.
>>>>>
>>>>> I tested NFSv3/v4/v4.1 sec=krb5 mounts. With expired or non-existent user
>>>>> Kerberos credentials, trying to ls the mountpoint, or cd into the mountpoint
>>>>> resulted in three failed upcalls to gssd (due to tk_cred_retry being set to 2)
>>>>> then the 'Operation not permitted' message is returned to the user.
>>>>>
>>>>> I think this patch should go into the stable kernel.
>>>>>
>>>>> Andy Adamson (1):
>>>>>  SUNRPC handle EKEYEXPIRED in call_refreshresult
>>>>>
>>>>> fs/nfs/nfs4proc.c |    2 --
>>>>> net/sunrpc/clnt.c |    4 ++++
>>>>> 2 files changed, 4 insertions(+), 2 deletions(-)
>>>>>
>>>>
>>>> Wait...is this really the behavior you want here?
>>>
>>> Yes. Just having the client hang with no indication to the user is wrong.
+1 I completely agree with this... 
>>>
>>
>> I presume you mean to say that that behavior isn't ideal. I tend to
>> agree, but there's no good way to report that to the user who can do
>> anything about it. I'll also point out that this scheme doesn't really
>> help that either. The user will end up with a failing job, at which
>> point it's too late to do anything about it...
There are mechanisms (krenew,sssd) the user can use to stop the 
jobs from failing... 

>>
>>>>
>>>> We had many complaints from users of krb5 mounts where long-running
>>>> jobs would routinely fail when the ticket expired.
We also have complaints about things hanging too... ;-) 

>>>
>>> That is a Kerberos ticket management issue, not an NFS kernel client issue.
>>> You have long-running jobs, then kinit -l,  run krenew, or use a keytab with a cron job,
>>> or use some other credential management software package.
Right... Some early on preparations to ensure the tickets remain renewable is
the solution..,

>>>
>>
>> Easy to say, far more difficult to do.
> 
> Here is how simple it is.
> 
> $ kinit
> $ krenew  -K 30&
If this simple process cannot renew a ticket then its a bug
in the client... We have to let processes like this to renew
renewable tickets! 

I've never been a fan of this hang from the beginning but 
if its getting in the way of things renewing tickets then its
got to go... IMHO...

Lets push the error up to the app and then educate people
on how to avoid the error... 

steved.

> 
> This simple krenew command says to renew the current users TGT every
> 30 minutes for
> as long as the TGT is renewable. For my setup, the TGT lasts 10 days -
> long enough for most long-running jobs. The kinit is to get a fresh
> TGT prior to running renew so that you start with as much time as
> possible.
> 
> I just tested it. My TGS expires in 30 minutes, my TGT in 1 hour. 3
> hours later, I'm still able to access the kerberos mount.
> 
> I'm sure most users with long running jobs could handle this. If you
> need longer running jobs, then you simply talk to the Kerberos admin
> and either get an extended TGT lifetime, or get a keytab and run a
> cron job (or use krenew to run a script).
> 
> 
>> Most of the people who
>> complained about the non-robustness of this were people who were
>> running jobs that took days or weeks. They were understandably upset
>> when that job failed just because the ticket expired.
> 
> Well, they get what they setup. If they didn't ensure a long enough
> ticket lifetime, then
> they shouldn't be surprised when the access to the file system fails.
> 
>>>
>>>> The compromise behavior that we worked out at that time was to treat an
>>>> expired credcache differently from a "no credcache" situation. gssd would
>>>> return EKEYEXPIRED if the credcache existed but was expired, and
>>>> EACCES otherwise. The kernel would then treat those errors
>>>> differently:
>>>
>>> In both cases, EPERM is the correct response from the Linux NFS client, as
>>> the user has no permissions to do anything in the file system.
>>>
>>
>> But, in the case of an expired ticket, it's quite likely that he had
>> permissions at some point in time. The rationale at the time was that
>> if that user could reacquire creds he could keep his job going.
> 
> Exactly! It's just not the NFS clients job to arrange this! It's a
> Kerberos Ticket Lifetime Issue. Use the Kerberos Ticket Lifetime Issue
> Tools!
> 
>>
>>>>
>>>>    http://permalink.gmane.org/gmane.linux.nfsv4/11019
>>>>
>>>> With EKEYEXPIRED, we'd want RPCs to hang indefinitely until the tickets
>>>> were renewed.
>>>
>>> Sounds like a good DOS attack.  Consider V4.1 and a multi-user machine. If a
>>> users credentials expire during a heavy I/O run - that user could be using all of the
>>> session slots, and no other user could make progress while the RPCs call rpc_delay
>>> and retry  indefinitely...
>>>
>>
>> Well, no. That was the main reason we handled this in the NFS layer and
>> not in sunrpc. The rpc_task would exit with EKEYEXPIRED and the NFS
>> code would treat that like an NFS4ERR_DELAY. Back off and try again
>> later.
>> Once the task has exited, any resources held in the rpc layer
>> including the slot should be available.
> 
> Ah - nfs4_async_handle_error on EKEYEXPIRED calls rpc_delay then
> returs with an EAGAIN error code - so it  doesn't call rpc_exit, but
> the EKEYEXPIRED is handled  the rpc_call_done routine which frees the
> slot allowing it to be reused.
> 
> Fair enough.
> 
> After the delay timer expires, the task goes into the rpc_prepare_task
> state to get a new slot, e.g. put on the session slot_tbl_waitq in
> heavy I/O situations where all slots are being used.
> 
> So in my example above with an expired user GSS context on a
> multi-user client, I/O gets stacked up in the session slot_tbl_waitq -
> which I've seen to grow to hundreds of RPC requests depending on the
> number of session slots and the amount of I/O.  Each I/O will keep on
> returning to the session slot_tbl_waitq after an up call to GSSD which
> fails with -EKEYEXPIRED.  Gssd gets pounded. The session
> slot_tbl_waitq is not drained of this users I/O.
> 
> Other users will feel pain. Especially if there is more than one user
> on the client irresponsible enough to start a long-running job without
> handling their Kerberos credentials.
> 
>>
>>>
>>>> With EACCES, the call would return an error. The idea
>>>> there is that the user would kdestroy if he needed to unwedge his krb5
>>>> mount.
>>>
>>> Exactly how is the user supposed to know to kdestroy? All they see is a hung mount.
>>>
>>
>> We do throw a warning when the state manager's ticket expires. Perhaps
>> we could do something similar from gssd for user tickets. The point is
>> though that the user has the ability to unwedge the mount without
>> reacquiring the ticket if he so chooses.
> 
> And my point is - this is wrong!  It is a key management issue!  What
> if the user doesn't choose to 'unwed'  the mount? Does it hang
> forever?
> 
> If the user is going to have to make a choice, let them make it on the
> front end, and setup their Kerberos tickets according to their need.
> 
>>
>>>>
>>>> This patch makes it sound like you're wanting to revert that behavior.
>>>> Is that the case?
>>>
>>> Yes.
>>>
>>>> If so, what about people trying to run long-running
>>>> tasks on a kerberized mount? Are they just SOL if their ticket isn't
>>>> renewed in time?
>>>
>>> Yes - as with _any_ resource, you need to plan ahead.  As I said above, the administrator in such a situation
>>> needs to setup krenew or the equivalent.
>>>
>>
>> That's not helpful. Everyone makes mistakes and you don't necessarily
>> want your job to fail simply due to that fact.
> 
> Yes, WRT security, you actually do.  There is a reason Kerberos
> tickets have a lifetime.
> The user needs to be aware of it, and extend or cut-off credentials
> according to what their use case is.
> 
>> But regardless, Trond
>> NAK'ed a similar idea not that long ago:
>>
>>    http://marc.info/?l=linux-nfs&m=132161606503398&w=2
>>
>> ...you may want to read over that thread as I'm fairly certain what
>> you're proposing will have the same issues...
> 
> Your'e referring to data corruption due to buffered writes or commits
> with expired GSS credentials?
> I'm preparing patches for this issue as well. I'll resend this patch
> with the others that
> handle the expired GSS creds.
> 
> -->Andy
> 
>>
>> --
>> Jeff Layton <jlayton@redhat.com>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 0/1] SUNRPC handle EKEYEXPIRED in call_refreshresult

[Jeff Layton]

On Fri, 29 Jun 2012 16:43:45 -0400
Steve Dickson <SteveD@redhat.com> wrote:

> >>>
> >>
> >> Easy to say, far more difficult to do.
> > 
> > Here is how simple it is.
> > 
> > $ kinit
> > $ krenew  -K 30&
> If this simple process cannot renew a ticket then its a bug
> in the client... We have to let processes like this to renew
> renewable tickets! 
> 
> I've never been a fan of this hang from the beginning but 
> if its getting in the way of things renewing tickets then its
> got to go... IMHO...
> 
> Lets push the error up to the app and then educate people
> on how to avoid the error... 
> 

Fair enough. I'd prefer to see this handled better too. If Andy has a
solution for the writeback issues then I'm fine with reverting this
behavior.

If we have kerberized NFS howtos anywhere (do we?) then we probably
need to add some notes about krenewd or how to make sssd handle this.

-- 
Jeff Layton <jlayton@redhat.com>

Re: [PATCH 1/1] SUNRPC handle EKEYEXPIRED in call_refreshresult

[Myklebust, Trond]

T24gV2VkLCAyMDEyLTA2LTI3IGF0IDE2OjA1IC0wNDAwLCBhbmRyb3NAbmV0YXBwLmNvbSB3cm90
ZToNCj4gRnJvbTogQW5keSBBZGFtc29uIDxhbmRyb3NAbmV0YXBwLmNvbT4NCj4gDQo+IFdoZW4g
YW4gUlBDU0VDX0dTUyBjb250ZXh0IGhhcyBleHBpcmVkIG9yIGlzIG5vbi1leGlzdGVudCwgYW5k
IHRoZSB1c2VyDQo+IChLZXJiZXJvcykgY3JlZGVudGlhbHMgaGF2ZSBhbHNvIGV4cGlyZWQgb3Ig
YXJlIG5vbi1leGlzdGVudCwgdGhlIGNsaWVudA0KPiByZXRyaWVzIHRvIHJlZnJlc2ggdGhlIGNv
bnRleHQgZm9yIGV2ZXIgYW5kIHRoZSBhcHBsaWNhdGlvbg0KPiBoYW5ncy4gVGhlIHVzZXIgaXMg
bm90IHByb21wdGVkIHRvIHJlZnJlc2gvZXN0YWJsaXNoIHRoZWlyIGNyZWRlbnRpYWxzLg0KPiAN
Cj4gTW92ZSB0aGUgLUVLRVlFWFBJUkVEIGhhbmRsaW5nIGludG8gdGhlIFJQQyBsYXllci4gVHJ5
IHRrX2NyZWRfcmV0cnkgbnVtYmVyDQo+IG9mIHRpbWVzIHRvIHJlZnJlc2ggdGhlIGdzc19jb250
ZXh0LCBhbmQgdGhlbiBwYXNzIC1FUEVSTSB0byBhcHBsaWNhdGlvbi4NCj4gDQo+IFNpZ25lZC1v
ZmYtYnk6IEFuZHkgQWRhbXNvbiA8YW5kcm9zQG5ldGFwcC5jb20+DQo+IC0tLQ0KPiAgZnMvbmZz
L25mczRwcm9jLmMgfCAgICAyIC0tDQo+ICBuZXQvc3VucnBjL2NsbnQuYyB8ICAgIDQgKysrKw0K
PiAgMiBmaWxlcyBjaGFuZ2VkLCA0IGluc2VydGlvbnMoKyksIDIgZGVsZXRpb25zKC0pDQo+IA0K
PiBkaWZmIC0tZ2l0IGEvZnMvbmZzL25mczRwcm9jLmMgYi9mcy9uZnMvbmZzNHByb2MuYw0KPiBp
bmRleCA1YTdiMzcyLi4yZjI5MWIzIDEwMDY0NA0KPiAtLS0gYS9mcy9uZnMvbmZzNHByb2MuYw0K
PiArKysgYi9mcy9uZnMvbmZzNHByb2MuYw0KPiBAQCAtMzQyLDcgKzM0Miw2IEBAIHN0YXRpYyBp
bnQgbmZzNF9oYW5kbGVfZXhjZXB0aW9uKHN0cnVjdCBuZnNfc2VydmVyICpzZXJ2ZXIsIGludCBl
cnJvcmNvZGUsIHN0cnVjDQo+ICAJCQl9DQo+ICAJCWNhc2UgLU5GUzRFUlJfR1JBQ0U6DQo+ICAJ
CWNhc2UgLU5GUzRFUlJfREVMQVk6DQo+IC0JCWNhc2UgLUVLRVlFWFBJUkVEOg0KPiAgCQkJcmV0
ID0gbmZzNF9kZWxheShzZXJ2ZXItPmNsaWVudCwgJmV4Y2VwdGlvbi0+dGltZW91dCk7DQo+ICAJ
CQlpZiAocmV0ICE9IDApDQo+ICAJCQkJYnJlYWs7DQo+IEBAIC0zOTM5LDcgKzM5MzgsNiBAQCBu
ZnM0X2FzeW5jX2hhbmRsZV9lcnJvcihzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2ssIGNvbnN0IHN0cnVj
dCBuZnNfc2VydmVyICpzZXJ2ZXIsDQo+ICAJCWNhc2UgLU5GUzRFUlJfREVMQVk6DQo+ICAJCQlu
ZnNfaW5jX3NlcnZlcl9zdGF0cyhzZXJ2ZXIsIE5GU0lPU19ERUxBWSk7DQo+ICAJCWNhc2UgLU5G
UzRFUlJfR1JBQ0U6DQo+IC0JCWNhc2UgLUVLRVlFWFBJUkVEOg0KPiAgCQkJcnBjX2RlbGF5KHRh
c2ssIE5GUzRfUE9MTF9SRVRSWV9NQVgpOw0KPiAgCQkJdGFzay0+dGtfc3RhdHVzID0gMDsNCj4g
IAkJCXJldHVybiAtRUFHQUlOOw0KPiBkaWZmIC0tZ2l0IGEvbmV0L3N1bnJwYy9jbG50LmMgYi9u
ZXQvc3VucnBjL2NsbnQuYw0KPiBpbmRleCBmNTZmMDQ1Li5hOTRmYzBjIDEwMDY0NA0KPiAtLS0g
YS9uZXQvc3VucnBjL2NsbnQuYw0KPiArKysgYi9uZXQvc3VucnBjL2NsbnQuYw0KPiBAQCAtMTMz
NCw4ICsxMzM0LDEyIEBAIGNhbGxfcmVmcmVzaHJlc3VsdChzdHJ1Y3QgcnBjX3Rhc2sgKnRhc2sp
DQo+ICAJCXJldHVybjsNCj4gIAljYXNlIC1FVElNRURPVVQ6DQo+ICAJCXJwY19kZWxheSh0YXNr
LCAzKkhaKTsNCj4gKwljYXNlIC1FS0VZRVhQSVJFRDoNCj4gKwkJc3RhdHVzID0gLUVQRVJNOw0K
DQpUaGlzIG5lZWRzIHRvIGJlIEVBQ0NFUy4uLg0KDQo+ICsJCWdvdG8gY3JlZF9yZXRyeTsNCj4g
IAljYXNlIC1FQUdBSU46DQo+ICAJCXN0YXR1cyA9IC1FQUNDRVM7DQo+ICtjcmVkX3JldHJ5Og0K
PiAgCQlpZiAoIXRhc2stPnRrX2NyZWRfcmV0cnkpDQo+ICAJCQlicmVhazsNCj4gIAkJdGFzay0+
dGtfY3JlZF9yZXRyeS0tOw0KDQotLSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVu
dCBtYWludGFpbmVyDQoNCk5ldEFwcA0KVHJvbmQuTXlrbGVidXN0QG5ldGFwcC5jb20NCnd3dy5u
ZXRhcHAuY29tDQoNCg==

Re: [PATCH 1/1] SUNRPC handle EKEYEXPIRED in call_refreshresult

[Adamson, Andy]

On Jul 16, 2012, at 2:44 PM, Myklebust, Trond wrote:

> On Wed, 2012-06-27 at 16:05 -0400, andros@netapp.com wrote:
>> From: Andy Adamson <andros@netapp.com>
>> 
>> When an RPCSEC_GSS context has expired or is non-existent, and the user
>> (Kerberos) credentials have also expired or are non-existent, the client
>> retries to refresh the context for ever and the application
>> hangs. The user is not prompted to refresh/establish their credentials.
>> 
>> Move the -EKEYEXPIRED handling into the RPC layer. Try tk_cred_retry number
>> of times to refresh the gss_context, and then pass -EPERM to application.
>> 
>> Signed-off-by: Andy Adamson <andros@netapp.com>
>> ---
>> fs/nfs/nfs4proc.c |    2 --
>> net/sunrpc/clnt.c |    4 ++++
>> 2 files changed, 4 insertions(+), 2 deletions(-)
>> 
>> diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
>> index 5a7b372..2f291b3 100644
>> --- a/fs/nfs/nfs4proc.c
>> +++ b/fs/nfs/nfs4proc.c
>> @@ -342,7 +342,6 @@ static int nfs4_handle_exception(struct nfs_server *server, int errorcode, struc
>> 			}
>> 		case -NFS4ERR_GRACE:
>> 		case -NFS4ERR_DELAY:
>> -		case -EKEYEXPIRED:
>> 			ret = nfs4_delay(server->client, &exception->timeout);
>> 			if (ret != 0)
>> 				break;
>> @@ -3939,7 +3938,6 @@ nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server,
>> 		case -NFS4ERR_DELAY:
>> 			nfs_inc_server_stats(server, NFSIOS_DELAY);
>> 		case -NFS4ERR_GRACE:
>> -		case -EKEYEXPIRED:
>> 			rpc_delay(task, NFS4_POLL_RETRY_MAX);
>> 			task->tk_status = 0;
>> 			return -EAGAIN;
>> diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
>> index f56f045..a94fc0c 100644
>> --- a/net/sunrpc/clnt.c
>> +++ b/net/sunrpc/clnt.c
>> @@ -1334,8 +1334,12 @@ call_refreshresult(struct rpc_task *task)
>> 		return;
>> 	case -ETIMEDOUT:
>> 		rpc_delay(task, 3*HZ);
>> +	case -EKEYEXPIRED:
>> +		status = -EPERM;
> 
> This needs to be EACCES…

Agreed. I'll also remove all -EKEYEXPIRED handling from NFS.

-->Andy

> 
>> +		goto cred_retry;
>> 	case -EAGAIN:
>> 		status = -EACCES;
>> +cred_retry:
>> 		if (!task->tk_cred_retry)
>> 			break;
>> 		task->tk_cred_retry--;
> 
> -- 
> Trond Myklebust
> Linux NFS client maintainer
> 
> NetApp
> Trond.Myklebust@netapp.com
> www.netapp.com
>