4.0 client and server restart with decreased lease time

4.0 client and server restart with decreased lease time

[Donald Buczek]

The nfsd default lease time has been changed from 90 seconds to 45
seconds between Linux 4.14 and 4.19 by commit d6ebf5088f09 ("nfsd4:
return default lease period").

After we did an upgrade of a nfs server from 4.14 to 4.19, we noticed
a failing process and dmesg logs "NFS: nfs4_reclaim_open_state: Lock
reclaim failed!" on a client (Linux 4.14.87). The client had the
file system mounted with vers=4.0.

Network trace indicated, that the client continued to use the
90 seconds lease period of the previous server incarnation and
sent RENEWs every 60 seconds (2/3 of 90 seconds).  Sometimes the
server answered with NFS4ERR_EXPIRED.

When this happened, the client executed recovery (SETCLIENTID...) but
did non query the server for a new lease_time.  So the problem was
persistent even after the first failure.

As an experiment, I've also restarted a server with the lease time
decrement from 90 to 45 seconds, but the grace period fixed to
90 seconds. Now the client got NFS4ERR_STALE_CLIENTID  but still did
not query the server for a new lease_time and continued to send RENEWs
in 60 second intervals.

At least for the later case, the RFC say, a client should refetch
the lease_time:

> A server may, upon restart, establish a new value for the lease
> period.  Therefore, clients should, once a new client ID is
> established, refetch the lease_time attribute and use it as the basis
> for lease renewal for the lease associated with that server.
> However, the server must establish, for this restart event, a grace
> period at least as long as the lease period for the previous server
> instantiation.  This allows the client state obtained during the
> previous server instance to be reliably re-established.

[ https://tools.ietf.org/html/rfc7530 ]

I understand that a restart with a grace period smaller than the
previous lease time is never save.

Aside from that, is a server restart with a decreased lease time
supposed to be supported by the Linux client? If not, this is kind
of a trap for server upgrades when just relying on the defaults.

Best
   Donald

Re: 4.0 client and server restart with decreased lease time

[J. Bruce Fields]

On Thu, Feb 07, 2019 at 12:48:41PM +0100, Donald Buczek wrote:
> The nfsd default lease time has been changed from 90 seconds to 45
> seconds between Linux 4.14 and 4.19 by commit d6ebf5088f09 ("nfsd4:
> return default lease period").
> 
> After we did an upgrade of a nfs server from 4.14 to 4.19, we noticed
> a failing process and dmesg logs "NFS: nfs4_reclaim_open_state: Lock
> reclaim failed!" on a client (Linux 4.14.87). The client had the
> file system mounted with vers=4.0.

Argh.  My fault for changing that default without thinking of the impact
on a server upgrade on existing client mounts, sorry.

> Network trace indicated, that the client continued to use the
> 90 seconds lease period of the previous server incarnation and
> sent RENEWs every 60 seconds (2/3 of 90 seconds).  Sometimes the
> server answered with NFS4ERR_EXPIRED.
>
> When this happened, the client executed recovery (SETCLIENTID...) but
> did non query the server for a new lease_time.  So the problem was
> persistent even after the first failure.

That certainly sounds like a client bug, though.

> As an experiment, I've also restarted a server with the lease time
> decrement from 90 to 45 seconds, but the grace period fixed to
> 90 seconds. Now the client got NFS4ERR_STALE_CLIENTID  but still did
> not query the server for a new lease_time and continued to send RENEWs
> in 60 second intervals.
> 
> At least for the later case, the RFC say, a client should refetch
> the lease_time:
> 
> >A server may, upon restart, establish a new value for the lease
> >period.  Therefore, clients should, once a new client ID is
> >established, refetch the lease_time attribute and use it as the basis
> >for lease renewal for the lease associated with that server.
> >However, the server must establish, for this restart event, a grace
> >period at least as long as the lease period for the previous server
> >instantiation.  This allows the client state obtained during the
> >previous server instance to be reliably re-established.
> 
> [ https://tools.ietf.org/html/rfc7530 ]
> 
> I understand that a restart with a grace period smaller than the
> previous lease time is never save.
> 
> Aside from that, is a server restart with a decreased lease time
> supposed to be supported by the Linux client? If not, this is kind
> of a trap for server upgrades when just relying on the defaults.

The client should be following the RFC and requesting the lease period
again after restart.

With that fixed, it may still fail to reclaim locks in this case, since
the upgraded server isn't providing a 90-second grace period as it
should, but that should at least avoid problems caused by it continuing to
renew only at 60-second intervals.

The server shouldn't have changed the defaults.  The only safe way to
decrease the lease period is to do it from userspace (e.g., change
NFSD_V4_LEASE value in /etc/sysconfig/nfs, restart, wait at least 90
seconds for every client to have a chance to get the new lease time,
then change NFSD_V4_GRACE).  The kernel on its own can't do this safely
since it doesn't know how to find the old value of the lease time.

--b.

[PATCH] Revert "nfsd4: return default lease period"

[J. Bruce Fields]

From: "J. Bruce Fields" <bfields@redhat.com>

This reverts commit d6ebf5088f09472c1136cd506bdc27034a6763f8.

I forgot that the kernel's default lease period should never be
decreased!

After a kernel upgrade, the kernel has no way of knowing on its own what
the previous lease time was.  Unless userspace tells it otherwise, it
will assume the previous lease period was the same.

So if we decrease this value in a kernel upgrade, we end up enforcing a
grace period that's too short, and clients will fail to reclaim state in
time.  Symptoms may include EIO and log messages like "NFS:
nfs4_reclaim_open_state: Lock reclaim failed!"

There was no real justification for the lease period decrease anyway.

Reported-by: Donald Buczek <buczek@molgen.mpg.de>
Fixes: d6ebf5088f09 "nfsd4: return default lease period"
Cc: stable@vger.kernel.org
Signed-off-by: J. Bruce Fields <bfields@redhat.com>
---
 fs/nfsd/nfsctl.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

On Thu, Feb 07, 2019 at 09:03:22PM -0500, bfields wrote:
> On Thu, Feb 07, 2019 at 12:48:41PM +0100, Donald Buczek wrote:
> > As an experiment, I've also restarted a server with the lease time
> > decrement from 90 to 45 seconds, but the grace period fixed to
> > 90 seconds. Now the client got NFS4ERR_STALE_CLIENTID  but still did
> > not query the server for a new lease_time and continued to send RENEWs
> > in 60 second intervals.

So this doesn't fix that client bug, but it does fix the server.

diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c
index b33f9785b756..72a7681f4046 100644
--- a/fs/nfsd/nfsctl.c
+++ b/fs/nfsd/nfsctl.c
@@ -1239,8 +1239,8 @@ static __net_init int nfsd_init_net(struct net *net)
 	retval = nfsd_idmap_init(net);
 	if (retval)
 		goto out_idmap_error;
-	nn->nfsd4_lease = 45;	/* default lease time */
-	nn->nfsd4_grace = 45;
+	nn->nfsd4_lease = 90;	/* default lease time */
+	nn->nfsd4_grace = 90;
 	nn->somebody_reclaimed = false;
 	nn->clverifier_counter = prandom_u32();
 	nn->clientid_counter = prandom_u32();
-- 
2.20.1