Re: NFS/TCP timeouts

[Trond Myklebust <trondmy@hammerspace.com>]

Hi Olga,

On Wed, 2019-12-11 at 15:36 -0500, Olga Kornievskaia wrote:
> Hi Trond,
> 
> I'd like to raise this once again. Is this true that setting a
> timeout
> limit (TCP_USER_TIMEOUT) is not user configurable (rather I'm pretty
> sure it is not) but my question is why shouldn't it be tied to the
> "timeo" mount option? Right now, only the sesson/lease manager thread
> sets it via rpc_set_connect_timeout() to be lease period related.
> 
> Is it the fact that we don't want to allow user to control TCP
> settings via the mount options? But somehow folks are expecting to be
> able to set low "timeo" value and have the (dead) connection to be
> considered dead earlier than for a rather long timeout period which
> is
> happening now.

In my mind, the two are correlated, but are not equivalent.

The 'timeo' value is basically a timeout for how long it takes for the
whole process of "send RPC call", "have it processed by the server" and
"receive reply".
IOW: 'timeo' is about how long it takes for an RPC call to execute end-
to-end.

The TCP_USER_TIMEOUT, is essentially a timeout for how long it takes
the server to ACK receipt of the RPC call once we've placed it in the
TCP socket.
IOW: it is a timeout for the networking part of an RPC call
transmission.

So, as I said, the two are correlated: if the server is down, then your
timeout is dominated by the fact that the network transmission never
completes. However if the server is up and congested, then the
"processing by the server" is likely to dominate.

The other thing to note is that if the TCP connection is unresponsive,
we may want to fail that much faster in order to give ourselves a
chance to close the connection, open a new one and retransmit the
requests from the old connection before the 'timeo' is triggered (since
in the case of a soft timeout, that could be a fatal error).

Does that make sense?

> 
> Thanks.
> 
> On Wed, Oct 3, 2018 at 3:06 PM Olga Kornievskaia <aglo@umich.edu>
> wrote:
> > On Wed, Oct 3, 2018 at 2:45 PM Trond Myklebust <
> > trondmy@hammerspace.com> wrote:
> > > On Wed, 2018-10-03 at 14:31 -0400, Olga Kornievskaia wrote:
> > > > Hi folks,
> > > > 
> > > > Is it true that NFS mount option "timeo" has nothing to do with
> > > > the
> > > > socket's setting of the user-specified timeout
> > > > TCP_USER_TIMEOUT.
> > > > Instead, when creating a TCP socket NFS uses either
> > > > default/hard
> > > > coded
> > > > value of 60s for v3 or for v4.x it's lease based. Is there no
> > > > value
> > > > is
> > > > having an adjustable TCP timeout value?
> > > > 
> > > 
> > > It is adjusted. Please see the calculation in
> > > xs_tcp_set_socket_timeouts().
> > 
> > but it's not user configurable, is it? I don't see a way to modify
> > v3's default 60s TCP timeout. and also in v4, the timeouts are set
> > from xs_tcp_set_connect_timeout() for the lease period but again
> > not
> > user configurable, as far as i can tell.
> > 
> > > --
> > > Trond Myklebust
> > > Linux NFS client maintainer, Hammerspace
> > > trond.myklebust@hammerspace.com
> > > 
> > > 
-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com