NFS/TCP timeout sequence

NFS/TCP timeout sequence

[Max Matveev]

I've had to look at the way NFS/TCP does its timeouts and backoff
and it does not make a lot of sense to me: according to the
following paragram from nfs(5) on Fedora 14 (I'm using Fedora 14
because it has more text then the same page in nfs-utils):

      timeo=n    The time (in tenths of a second) the  NFS  client  waits
                 for a response before it retries an NFS request. If this
                 option is not specified, requests are retried  every  60
                 seconds  for NFS over TCP.  The NFS client does not per‐
                 form any kind of timeout backoff for NFS over TCP.

but if I try the mount with timeo=20,retrans=7 then I'm getting
retransmits which are 2, 4, 6, 8, 2, 4, 6, 8 seconds apart, i.e.
there is a) linear backoff and b) the backoff is not long enough to
let the complete sequence of 7 retransmits run its course.

This is happening because to_maxval for NFS_TCP is too short to
accomodate the linear backoff - we need to either increase the
to_maxval to something like:

--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -606,7 +606,8 @@ static void nfs_init_timeout_values(struct rpc_timeout *to, 
                if (to->to_initval > NFS_MAX_TCP_TIMEOUT)
                        to->to_initval = NFS_MAX_TCP_TIMEOUT;
                to->to_increment = to->to_initval;
-               to->to_maxval = to->to_initval + (to->to_increment * to->to_retries);
+               to->to_maxval = to->to_increment * (to->to_retries + 1) 
+                             * (to->to_retries + 2) / 2;
                if (to->to_maxval > NFS_MAX_TCP_TIMEOUT)
                        to->to_maxval = NFS_MAX_TCP_TIMEOUT;
                if (to->to_maxval < to->to_initval)

or don't do the linear backoff

--- a/net/sunrpc/xprt.c
+++ b/net/sunrpc/xprt.c
@@ -546,7 +546,7 @@ static void xprt_reset_majortimeo(struct rpc_rqst *req)
        if (to->to_exponential)
                req->rq_majortimeo <<= to->to_retries;
        else
-               req->rq_majortimeo += to->to_increment * to->to_retries;
+               req->rq_majortimeo += to->to_increment;
        if (req->rq_majortimeo > to->to_maxval || req->rq_majortimeo == 0)
                req->rq_majortimeo = to->to_maxval;
        req->rq_majortimeo += jiffies;

max

Re: NFS/TCP timeout sequence

[Trond Myklebust]

On Thu, 2011-07-07 at 18:11 +1000, Max Matveev wrote: 
> I've had to look at the way NFS/TCP does its timeouts and backoff
> and it does not make a lot of sense to me: according to the
> following paragram from nfs(5) on Fedora 14 (I'm using Fedora 14
> because it has more text then the same page in nfs-utils):
> 
>       timeo=n    The time (in tenths of a second) the  NFS  client  waits
>                  for a response before it retries an NFS request. If this
>                  option is not specified, requests are retried  every  60
>                  seconds  for NFS over TCP.  The NFS client does not per‐
>                  form any kind of timeout backoff for NFS over TCP.
> 
> but if I try the mount with timeo=20,retrans=7 then I'm getting
> retransmits which are 2, 4, 6, 8, 2, 4, 6, 8 seconds apart, i.e.
> there is a) linear backoff and b) the backoff is not long enough to
> let the complete sequence of 7 retransmits run its course.

Sigh... Firstly, 2 second timeouts are complete lunacy when using a
protocol that guarantees reliable delivery, such as TCP does. Anyone who
tries it deserves exactly what they get: poor unreliable performance.

Secondly, the _other_ fix for this problem is to fix the documentation.

Trond
-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: NFS/TCP timeout sequence

[Chuck Lever]

On Jul 7, 2011, at 9:47 AM, Trond Myklebust wrote:

> On Thu, 2011-07-07 at 18:11 +1000, Max Matveev wrote: 
>> I've had to look at the way NFS/TCP does its timeouts and backoff
>> and it does not make a lot of sense to me: according to the
>> following paragram from nfs(5) on Fedora 14 (I'm using Fedora 14
>> because it has more text then the same page in nfs-utils):
>> 
>>      timeo=n    The time (in tenths of a second) the  NFS  client  waits
>>                 for a response before it retries an NFS request. If this
>>                 option is not specified, requests are retried  every  60
>>                 seconds  for NFS over TCP.  The NFS client does not per‐
>>                 form any kind of timeout backoff for NFS over TCP.
>> 
>> but if I try the mount with timeo=20,retrans=7 then I'm getting
>> retransmits which are 2, 4, 6, 8, 2, 4, 6, 8 seconds apart, i.e.
>> there is a) linear backoff and b) the backoff is not long enough to
>> let the complete sequence of 7 retransmits run its course.
> 
> Sigh... Firstly, 2 second timeouts are complete lunacy when using a
> protocol that guarantees reliable delivery, such as TCP does. Anyone who
> tries it deserves exactly what they get: poor unreliable performance.

We shouldn't allow such low settings.

> Secondly, the _other_ fix for this problem is to fix the documentation.

How is the documentation incorrect?  We do not want any kind of back-off for stream transports.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: NFS/TCP timeout sequence

[Trond Myklebust]

On Thu, 2011-07-07 at 10:04 -0400, Chuck Lever wrote: 
> On Jul 7, 2011, at 9:47 AM, Trond Myklebust wrote:
> 
> > On Thu, 2011-07-07 at 18:11 +1000, Max Matveev wrote: 
> >> I've had to look at the way NFS/TCP does its timeouts and backoff
> >> and it does not make a lot of sense to me: according to the
> >> following paragram from nfs(5) on Fedora 14 (I'm using Fedora 14
> >> because it has more text then the same page in nfs-utils):
> >> 
> >>      timeo=n    The time (in tenths of a second) the  NFS  client  waits
> >>                 for a response before it retries an NFS request. If this
> >>                 option is not specified, requests are retried  every  60
> >>                 seconds  for NFS over TCP.  The NFS client does not per‐
> >>                 form any kind of timeout backoff for NFS over TCP.
> >> 
> >> but if I try the mount with timeo=20,retrans=7 then I'm getting
> >> retransmits which are 2, 4, 6, 8, 2, 4, 6, 8 seconds apart, i.e.
> >> there is a) linear backoff and b) the backoff is not long enough to
> >> let the complete sequence of 7 retransmits run its course.
> > 
> > Sigh... Firstly, 2 second timeouts are complete lunacy when using a
> > protocol that guarantees reliable delivery, such as TCP does. Anyone who
> > tries it deserves exactly what they get: poor unreliable performance.
> 
> We shouldn't allow such low settings.
> 
> > Secondly, the _other_ fix for this problem is to fix the documentation.
> 
> How is the documentation incorrect?  We do not want any kind of back-off for stream transports.

The documentation states that we don't do back off, but as Max points
out, in practice the kernel does a linear back off (and has always done
so).

Anyway, why shouldn't we back off if the server is failing to respond?

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: NFS/TCP timeout sequence

[Chuck Lever]

On Jul 7, 2011, at 10:16 AM, Trond Myklebust wrote:

> On Thu, 2011-07-07 at 10:04 -0400, Chuck Lever wrote: 
>> On Jul 7, 2011, at 9:47 AM, Trond Myklebust wrote:
>> 
>>> On Thu, 2011-07-07 at 18:11 +1000, Max Matveev wrote: 
>>>> I've had to look at the way NFS/TCP does its timeouts and backoff
>>>> and it does not make a lot of sense to me: according to the
>>>> following paragram from nfs(5) on Fedora 14 (I'm using Fedora 14
>>>> because it has more text then the same page in nfs-utils):
>>>> 
>>>>     timeo=n    The time (in tenths of a second) the  NFS  client  waits
>>>>                for a response before it retries an NFS request. If this
>>>>                option is not specified, requests are retried  every  60
>>>>                seconds  for NFS over TCP.  The NFS client does not per‐
>>>>                form any kind of timeout backoff for NFS over TCP.
>>>> 
>>>> but if I try the mount with timeo=20,retrans=7 then I'm getting
>>>> retransmits which are 2, 4, 6, 8, 2, 4, 6, 8 seconds apart, i.e.
>>>> there is a) linear backoff and b) the backoff is not long enough to
>>>> let the complete sequence of 7 retransmits run its course.
>>> 
>>> Sigh... Firstly, 2 second timeouts are complete lunacy when using a
>>> protocol that guarantees reliable delivery, such as TCP does. Anyone who
>>> tries it deserves exactly what they get: poor unreliable performance.
>> 
>> We shouldn't allow such low settings.
>> 
>>> Secondly, the _other_ fix for this problem is to fix the documentation.
>> 
>> How is the documentation incorrect?  We do not want any kind of back-off for stream transports.
> 
> The documentation states that we don't do back off, but as Max points
> out, in practice the kernel does a linear back off (and has always done
> so).

I question that parenthetical assertion.  When I've looked at this behavior in the past, it has not backed off.  It has retried every 60 seconds.  That's why I wrote that in nfs(5).  I've had many discussions about this with you in the past.  We agreed: no back-off for TCP.  The default settings for TCP transports are timeo=600,retrans=2, which means try three times at fixed 60 second intervals.

So it seems to me the kernel has diverged (perhaps long ago) from the documentation, not the other way around.

> Anyway, why shouldn't we back off if the server is failing to respond?

Because the Solaris NFS client behaves this way, and we want to keep the syntax and semantics of our admin interfaces aligned between these implementations unless there is a good reason not to, because these mount options are published in automounter maps.

More importantly, a 60 second wait is not an onerous workload for either the network or the server.  Back-offs are usually used to provide quick recovery but then reduce network traffic if the server is down for a long while.  If we start at 60 seconds, there's already no onerous workload; plus we already have a slow recovery anyway...

In fact, for a long time we've wanted to make server restart recovery _faster_ not slower.  Thus using back-off with already lengthy retransmit timeouts seems like a step in the wrong direction.  After a server restart, our users want the client talking to the server again as quickly as possible.  At a guess, quicker recovery time after a server reboot is probably the number one reason why people try using a smaller timeo= setting for TCP.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

Re: NFS/TCP timeout sequence

[Trond Myklebust]

On Thu, 2011-07-07 at 10:44 -0400, Chuck Lever wrote: 
> On Jul 7, 2011, at 10:16 AM, Trond Myklebust wrote:
> 
> > On Thu, 2011-07-07 at 10:04 -0400, Chuck Lever wrote: 
> >> On Jul 7, 2011, at 9:47 AM, Trond Myklebust wrote:
> >> 
> >>> On Thu, 2011-07-07 at 18:11 +1000, Max Matveev wrote: 
> >>>> I've had to look at the way NFS/TCP does its timeouts and backoff
> >>>> and it does not make a lot of sense to me: according to the
> >>>> following paragram from nfs(5) on Fedora 14 (I'm using Fedora 14
> >>>> because it has more text then the same page in nfs-utils):
> >>>> 
> >>>>     timeo=n    The time (in tenths of a second) the  NFS  client  waits
> >>>>                for a response before it retries an NFS request. If this
> >>>>                option is not specified, requests are retried  every  60
> >>>>                seconds  for NFS over TCP.  The NFS client does not per‐
> >>>>                form any kind of timeout backoff for NFS over TCP.
> >>>> 
> >>>> but if I try the mount with timeo=20,retrans=7 then I'm getting
> >>>> retransmits which are 2, 4, 6, 8, 2, 4, 6, 8 seconds apart, i.e.
> >>>> there is a) linear backoff and b) the backoff is not long enough to
> >>>> let the complete sequence of 7 retransmits run its course.
> >>> 
> >>> Sigh... Firstly, 2 second timeouts are complete lunacy when using a
> >>> protocol that guarantees reliable delivery, such as TCP does. Anyone who
> >>> tries it deserves exactly what they get: poor unreliable performance.
> >> 
> >> We shouldn't allow such low settings.
> >> 
> >>> Secondly, the _other_ fix for this problem is to fix the documentation.
> >> 
> >> How is the documentation incorrect?  We do not want any kind of back-off for stream transports.
> > 
> > The documentation states that we don't do back off, but as Max points
> > out, in practice the kernel does a linear back off (and has always done
> > so).
> 
> I question that parenthetical assertion.  When I've looked at this behavior in the past, it has not backed off.  It has retried every 60 seconds.  That's why I wrote that in nfs(5).  I've had many discussions about this with you in the past.  We agreed: no back-off for TCP.  The default settings for TCP transports are timeo=600,retrans=2, which means try three times at fixed 60 second intervals.

Looking at the code:

v2.6.0: exponential back off
v2.6.4: exponential back off
v2.6.9: exponential back off
v2.6.16: linear back off
v2.6.18: linear back off
v2.6.24: linear back off
v2.6.32: linear back off
....

So I've no idea what you were testing.

> So it seems to me the kernel has diverged (perhaps long ago) from the documentation, not the other way around.

Nope. The documentation has simply always been inaccurate afaics from
the above inspection.

> > Anyway, why shouldn't we back off if the server is failing to respond?
> 
> Because the Solaris NFS client behaves this way, and we want to keep the syntax and semantics of our admin interfaces aligned between these implementations unless there is a good reason not to, because these mount options are published in automounter maps.
> 
> More importantly, a 60 second wait is not an onerous workload for either the network or the server.  Back-offs are usually used to provide quick recovery but then reduce network traffic if the server is down for a long while.  If we start at 60 seconds, there's already no onerous workload; plus we already have a slow recovery anyway...
> 
> In fact, for a long time we've wanted to make server restart recovery _faster_ not slower.  Thus using back-off with already lengthy retransmit timeouts seems like a step in the wrong direction.  After a server restart, our users want the client talking to the server again as quickly as possible.  At a guess, quicker recovery time after a server reboot is probably the number one reason why people try using a smaller timeo= setting for TCP.

'retrans=1' will do this for you.

-- 
Trond Myklebust
Linux NFS client maintainer

NetApp
Trond.Myklebust@netapp.com
www.netapp.com

Re: NFS/TCP timeout sequence

[Max Matveev]

On Thu, 07 Jul 2011 09:47:19 -0400, Trond Myklebust wrote:

 TM> On Thu, 2011-07-07 at 18:11 +1000, Max Matveev wrote: 

 TM> Sigh... Firstly, 2 second timeouts are complete lunacy when using a
 TM> protocol that guarantees reliable delivery, such as TCP does. Anyone who
 TM> tries it deserves exactly what they get: poor unreliable performance.

2 seconds is besides the point - I'm not going to wait for 28 minutes
(timeout=600,retrans=7) when doing timeout testing just to prove that
it does not work.

 TM> Secondly, the _other_ fix for this problem is to fix the documentation.

I can live with that too.

max

Re: NFS/TCP timeout sequence

[Max Matveev]

On Thu, 07 Jul 2011 10:16:53 -0400, Trond Myklebust wrote:

> Anyway, why shouldn't we back off if the server is failing to respond?

Wasn't it the no-backoff/drop-connection approach what Mike Eisler was
advocating back in '06 during Connectathon?

http://www.connectathon.org/talks06/eisler.pdf

I think this was the trigger to go from exponential backoff to linear.

max

[PATCH] NFS: allow enough time for timeouts to run

[Max Matveev]

NFS/TCP uses linear backoff when retransmiting requests but
miscalculates the cut-off time for the timeout sequence resulting
in premature major timeouts.

The cutoff should be the sum of first retransmits+1 numbers
multiplied by the timeo, not the retransmits+1 multiplied by
timeo.

Signed-off-by: Max Matveev <makc@redhat.com>
---
 fs/nfs/client.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

diff --git a/fs/nfs/client.c b/fs/nfs/client.c
index 5833fbb..3d12d10 100644
--- a/fs/nfs/client.c
+++ b/fs/nfs/client.c
@@ -604,7 +604,8 @@ static void nfs_init_timeout_values(struct rpc_timeout *to, int proto,
 		if (to->to_initval > NFS_MAX_TCP_TIMEOUT)
 			to->to_initval = NFS_MAX_TCP_TIMEOUT;
 		to->to_increment = to->to_initval;
-		to->to_maxval = to->to_initval + (to->to_increment * to->to_retries);
+		to->to_maxval = (to->to_retries + 1) * (to->to_retries + 2) / 2;
+		to->to_maxval *= to->to_initval;
 		if (to->to_maxval > NFS_MAX_TCP_TIMEOUT)
 			to->to_maxval = NFS_MAX_TCP_TIMEOUT;
 		if (to->to_maxval < to->to_initval)
-- 
1.7.4.4

[PATCH] Update nfs(5) manpage - timeo for NFS/TCP

[Max Matveev]

NFS/TCP does linear backoff then retransmiting - the manpage
was mistakenly asserting the "no backoff" theory.

Signed-off-by: Max Matveev <makc@redhat.com>
---
 utils/mount/nfs.man |   14 +++++++++-----
 1 files changed, 9 insertions(+), 5 deletions(-)

diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
index be91a25..7681b10 100644
--- a/utils/mount/nfs.man
+++ b/utils/mount/nfs.man
@@ -114,11 +114,15 @@ option.
 .TP 1.5i
 .BI timeo= n
 The time (in tenths of a second) the NFS client waits for a
-response before it retries an NFS request. If this
-option is not specified, requests are retried every
-60 seconds for NFS over TCP.
-The NFS client does not perform any kind of timeout backoff
-for NFS over TCP.
+response before it retries an NFS request.
+.IP
+For NFS over TCP the default
+.B timeo
+value is 600 which equals to 60 seconds.
+The NFS client performs linear backoff  -
+after each retransmission the timeout is increased by
+.BR timeo /10
+seconds upto the maximum of 600 seconds.
 .IP
 However, for NFS over UDP, the client uses an adaptive
 algorithm to estimate an appropriate timeout value for frequently used
-- 
1.7.4.4

Re: NFS/TCP timeout sequence

[Max Matveev]

Resuming a conversation from last month - it fizzed out and died
without any resoltion...

On Thu, 07 Jul 2011 10:59:12 -0400, Trond Myklebust wrote:

 trond> Looking at the code:

 trond> v2.6.0: exponential back off
 trond> v2.6.4: exponential back off
 trond> v2.6.9: exponential back off
 trond> v2.6.16: linear back off
 trond> v2.6.18: linear back off
 trond> v2.6.24: linear back off
 trond> v2.6.32: linear back off
 trond> ....

 trond> So I've no idea what you were testing.

I'm going to assume that we're keeping linear backoff and
send a patch which correct calculation of maxint for this case.

 >> So it seems to me the kernel has diverged (perhaps long ago) from
 >> the documentation, not the other way around.

 trond> Nope. The documentation has simply always been inaccurate afaics from
 trond> the above inspection.

And another one to update the documentation..

max

Re: [PATCH] Update nfs(5) manpage - timeo for NFS/TCP

[Jim Rees]

Max Matveev wrote:

  NFS/TCP does linear backoff then retransmiting - the manpage
  was mistakenly asserting the "no backoff" theory.
  
  Signed-off-by: Max Matveev <makc@redhat.com>
  ---
   utils/mount/nfs.man |   14 +++++++++-----
   1 files changed, 9 insertions(+), 5 deletions(-)
  
  diff --git a/utils/mount/nfs.man b/utils/mount/nfs.man
  index be91a25..7681b10 100644
  --- a/utils/mount/nfs.man
  +++ b/utils/mount/nfs.man
  @@ -114,11 +114,15 @@ option.
   .TP 1.5i
   .BI timeo= n
   The time (in tenths of a second) the NFS client waits for a
  -response before it retries an NFS request. If this
  -option is not specified, requests are retried every
  -60 seconds for NFS over TCP.
  -The NFS client does not perform any kind of timeout backoff
  -for NFS over TCP.
  +response before it retries an NFS request.
  +.IP
  +For NFS over TCP the default
  +.B timeo
  +value is 600 which equals to 60 seconds.
  +The NFS client performs linear backoff  -
  +after each retransmission the timeout is increased by
  +.BR timeo /10
  +seconds upto the maximum of 600 seconds.

"upto" should be "up to".

I'm not crazy about the wording.  How about

  For NFS over TCP the default timeo is 600 (60 seconds).  The NFS client
  performs linear backoff.  After each retransmission the timeout is
  increased by timeo up to the maximum of 600 seconds.

Finally, I would call them "deciseconds".

timeo: The time in deciseconds (tenths of a second) ...

Re: [PATCH] Update nfs(5) manpage - timeo for NFS/TCP

[Max Matveev]

On Thu, 4 Aug 2011 08:04:01 -0400, Jim Rees wrote:

 rees> I'm not crazy about the wording.  How about

 rees>   For NFS over TCP the default timeo is 600 (60 seconds).  The NFS client
 rees>   performs linear backoff.  After each retransmission the timeout is
 rees>   increased by timeo up to the maximum of 600 seconds.

I'm fine with this except the third sentence is the explanation of
"linear backoff" - I'd rather keep them together, i.e.:

	The NFS client performs linear backoff: after each
	retransmission the timeout is increased by timeo up to the
	maximum of 600 seconds.

 rees> Finally, I would call them "deciseconds".

Secquipedalians'r'us.

 rees> timeo: The time in deciseconds (tenths of a second) ...
Will do.

Anything else before I send v2?

max

Re: [PATCH] Update nfs(5) manpage - timeo for NFS/TCP

[Jim Rees]

Max Matveev wrote:

  	The NFS client performs linear backoff: after each
  	retransmission the timeout is increased by timeo up to the
  	maximum of 600 seconds.

Good point about keeping this together.  Probably should be a semicolon, but
if you go with a colon, capitalize "after".

Re: [PATCH] Update nfs(5) manpage - timeo for NFS/TCP

[Max Matveev]

On Thu, 4 Aug 2011 21:39:36 -0400, Jim Rees wrote:

 rees> Max Matveev wrote:
 rees>   	The NFS client performs linear backoff: after each
 rees>   	retransmission the timeout is increased by timeo up to the
 rees>   	maximum of 600 seconds.

 rees> Good point about keeping this together.  Probably should be a
 rees> semicolon, but if you go with a colon, capitalize "after".

Will keep the colon and capitalize.

max