SOFT + NO_RETRANS_TIMEOUT semantics

SOFT + NO_RETRANS_TIMEOUT semantics

[Chuck Lever III]

Hi Trond-

I'm seeing some interesting client hangs that arise from a well-
timed server crash or network partition.

The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.

NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll
refer to it as NORTO) when creating a new rpc_clnt. The initial
rpc_ping() for that rpc_clnt is done before the logic that sets
cl_noretranstimeo, thus that ping works as expected (SOFT |
SOFTCONN) and can time out properly if the server isn't
responsive.

However, once that ping succeeds, cl_noretranstimeo is asserted,
and all subsequent RPC requests on that rpc_clnt are with NORTO
semantics.

When it comes time to destroy the GSS context for that rpc_clnt,
the NULL procedure with the GSS decorations is sent with SOFT |
SOFTCONN | NORTO. If the server isn't responding at that point,
the client continues to retransmit the GSS context destruction
request forever, and the xprt and possibly the nfs_client are
pinned.

The problem also arises for lease management operations such as
singleton SEQUENCE or RENEW requests. These are also done with
SOFT, as I recall they need to time out properly. But with
NORTO + SOFT, they will be retried until a connection loss that
might never come.

I've thought of some ways to modify the cl_noretranstimeo logic
such that it can be disabled for particular RPC tasks, though
none is really striking me as exceptionally clever:

 - Add a field to struct rpc_procinfo that contains a mask of
   RPC_TASK flags to clear for each procedure.
 - Add logic to rpc_task_set_client() that clears NORTO in
   some special cases.
 - Reverse the meaning of NORTO (e.g., make it
   RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
   for particular RPC tasks if the rpc_clnt-default behavior
   is NORTO.

Any thoughts?

--
Chuck Lever

Re: SOFT + NO_RETRANS_TIMEOUT semantics

[Trond Myklebust]

On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote:
> Hi Trond-
> 
> I'm seeing some interesting client hangs that arise from a well-
> timed server crash or network partition.
> 
> The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.
> 
> NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll
> refer to it as NORTO) when creating a new rpc_clnt. The initial
> rpc_ping() for that rpc_clnt is done before the logic that sets
> cl_noretranstimeo, thus that ping works as expected (SOFT |
> SOFTCONN) and can time out properly if the server isn't
> responsive.
> 
> However, once that ping succeeds, cl_noretranstimeo is asserted,
> and all subsequent RPC requests on that rpc_clnt are with NORTO
> semantics.
> 
> When it comes time to destroy the GSS context for that rpc_clnt,
> the NULL procedure with the GSS decorations is sent with SOFT |
> SOFTCONN | NORTO. If the server isn't responding at that point,
> the client continues to retransmit the GSS context destruction
> request forever, and the xprt and possibly the nfs_client are
> pinned.
> 
> The problem also arises for lease management operations such as
> singleton SEQUENCE or RENEW requests. These are also done with
> SOFT, as I recall they need to time out properly. But with
> NORTO + SOFT, they will be retried until a connection loss that
> might never come.
> 
> I've thought of some ways to modify the cl_noretranstimeo logic
> such that it can be disabled for particular RPC tasks, though
> none is really striking me as exceptionally clever:
> 
>  - Add a field to struct rpc_procinfo that contains a mask of
>    RPC_TASK flags to clear for each procedure.
>  - Add logic to rpc_task_set_client() that clears NORTO in
>    some special cases.
>  - Reverse the meaning of NORTO (e.g., make it
>    RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
>    for particular RPC tasks if the rpc_clnt-default behavior
>    is NORTO.
> 
> Any thoughts?
> 

Why would the connection not break when the server goes down? Aren't
the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they should?

Is this an RDMA problem?

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: SOFT + NO_RETRANS_TIMEOUT semantics

[Chuck Lever III]

> On Jul 12, 2021, at 1:36 PM, Trond Myklebust <trondmy@hammerspace.com> wrote:
> 
> On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote:
>> Hi Trond-
>> 
>> I'm seeing some interesting client hangs that arise from a well-
>> timed server crash or network partition.
>> 
>> The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.
>> 
>> NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll
>> refer to it as NORTO) when creating a new rpc_clnt. The initial
>> rpc_ping() for that rpc_clnt is done before the logic that sets
>> cl_noretranstimeo, thus that ping works as expected (SOFT |
>> SOFTCONN) and can time out properly if the server isn't
>> responsive.
>> 
>> However, once that ping succeeds, cl_noretranstimeo is asserted,
>> and all subsequent RPC requests on that rpc_clnt are with NORTO
>> semantics.
>> 
>> When it comes time to destroy the GSS context for that rpc_clnt,
>> the NULL procedure with the GSS decorations is sent with SOFT |
>> SOFTCONN | NORTO. If the server isn't responding at that point,
>> the client continues to retransmit the GSS context destruction
>> request forever, and the xprt and possibly the nfs_client are
>> pinned.
>> 
>> The problem also arises for lease management operations such as
>> singleton SEQUENCE or RENEW requests. These are also done with
>> SOFT, as I recall they need to time out properly. But with
>> NORTO + SOFT, they will be retried until a connection loss that
>> might never come.
>> 
>> I've thought of some ways to modify the cl_noretranstimeo logic
>> such that it can be disabled for particular RPC tasks, though
>> none is really striking me as exceptionally clever:
>> 
>>  - Add a field to struct rpc_procinfo that contains a mask of
>>    RPC_TASK flags to clear for each procedure.
>>  - Add logic to rpc_task_set_client() that clears NORTO in
>>    some special cases.
>>  - Reverse the meaning of NORTO (e.g., make it
>>    RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
>>    for particular RPC tasks if the rpc_clnt-default behavior
>>    is NORTO.
>> 
>> Any thoughts?
>> 
> 
> Why would the connection not break when the server goes down?

The server can't actively RST or FIN the connection if a network
partition occurs; and some servers might crash while their kernel
is still alive to respond to keep-alive.


> Aren't
> the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they should?

I don't see them kicking in, but I let the test run only for about
12 minutes. 


--
Chuck Lever

Re: SOFT + NO_RETRANS_TIMEOUT semantics

[Trond Myklebust]

On Mon, 2021-07-12 at 17:48 +0000, Chuck Lever III wrote:
> 
> 
> > On Jul 12, 2021, at 1:36 PM, Trond Myklebust
> > <trondmy@hammerspace.com> wrote:
> > 
> > On Mon, 2021-07-12 at 17:07 +0000, Chuck Lever III wrote:
> > > Hi Trond-
> > > 
> > > I'm seeing some interesting client hangs that arise from a well-
> > > timed server crash or network partition.
> > > 
> > > The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.
> > > 
> > > NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter
> > > I'll
> > > refer to it as NORTO) when creating a new rpc_clnt. The initial
> > > rpc_ping() for that rpc_clnt is done before the logic that sets
> > > cl_noretranstimeo, thus that ping works as expected (SOFT |
> > > SOFTCONN) and can time out properly if the server isn't
> > > responsive.
> > > 
> > > However, once that ping succeeds, cl_noretranstimeo is asserted,
> > > and all subsequent RPC requests on that rpc_clnt are with NORTO
> > > semantics.
> > > 
> > > When it comes time to destroy the GSS context for that rpc_clnt,
> > > the NULL procedure with the GSS decorations is sent with SOFT |
> > > SOFTCONN | NORTO. If the server isn't responding at that point,
> > > the client continues to retransmit the GSS context destruction
> > > request forever, and the xprt and possibly the nfs_client are
> > > pinned.
> > > 
> > > The problem also arises for lease management operations such as
> > > singleton SEQUENCE or RENEW requests. These are also done with
> > > SOFT, as I recall they need to time out properly. But with
> > > NORTO + SOFT, they will be retried until a connection loss that
> > > might never come.
> > > 
> > > I've thought of some ways to modify the cl_noretranstimeo logic
> > > such that it can be disabled for particular RPC tasks, though
> > > none is really striking me as exceptionally clever:
> > > 
> > >  - Add a field to struct rpc_procinfo that contains a mask of
> > >    RPC_TASK flags to clear for each procedure.
> > >  - Add logic to rpc_task_set_client() that clears NORTO in
> > >    some special cases.
> > >  - Reverse the meaning of NORTO (e.g., make it
> > >    RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
> > >    for particular RPC tasks if the rpc_clnt-default behavior
> > >    is NORTO.
> > > 
> > > Any thoughts?
> > > 
> > 
> > Why would the connection not break when the server goes down?
> 
> The server can't actively RST or FIN the connection if a network
> partition occurs; and some servers might crash while their kernel
> is still alive to respond to keep-alive.
> 
> 
> > Aren't
> > the TCP_USER_TIMEOUT or the TCP_KEEPALIVE kicking in as they
> > should?
> 
> I don't see them kicking in, but I let the test run only for about
> 12 minutes. 
> 

TCP_USER_TIMEOUT should kick in any time when the server is failing to
read the socket contents, and should close the connection.

A more likely scenario is that the server is actually reading the
socket, but is just dropping the requests on the floor. I agree that
needs to be handled correctly.
One way to do that could be to add a flag that says "don't apply any
other default task flags" for special cases like this one?

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com