SOFT + NO_RETRANS_TIMEOUT semantics

[Chuck Lever III <chuck.lever@oracle.com>]

Hi Trond-

I'm seeing some interesting client hangs that arise from a well-
timed server crash or network partition.

The easiest to see is gss_destroy() on an Kerberized NFSv4 mount.

NFSv4 asserts the RPC_TASK_NO_RETRANS_TIMEOUT flag (hereafter I'll
refer to it as NORTO) when creating a new rpc_clnt. The initial
rpc_ping() for that rpc_clnt is done before the logic that sets
cl_noretranstimeo, thus that ping works as expected (SOFT |
SOFTCONN) and can time out properly if the server isn't
responsive.

However, once that ping succeeds, cl_noretranstimeo is asserted,
and all subsequent RPC requests on that rpc_clnt are with NORTO
semantics.

When it comes time to destroy the GSS context for that rpc_clnt,
the NULL procedure with the GSS decorations is sent with SOFT |
SOFTCONN | NORTO. If the server isn't responding at that point,
the client continues to retransmit the GSS context destruction
request forever, and the xprt and possibly the nfs_client are
pinned.

The problem also arises for lease management operations such as
singleton SEQUENCE or RENEW requests. These are also done with
SOFT, as I recall they need to time out properly. But with
NORTO + SOFT, they will be retried until a connection loss that
might never come.

I've thought of some ways to modify the cl_noretranstimeo logic
such that it can be disabled for particular RPC tasks, though
none is really striking me as exceptionally clever:

 - Add a field to struct rpc_procinfo that contains a mask of
   RPC_TASK flags to clear for each procedure.
 - Add logic to rpc_task_set_client() that clears NORTO in
   some special cases.
 - Reverse the meaning of NORTO (e.g., make it
   RPC_TASK_RETRANS_TIMEOUT) so that it can be set by a caller
   for particular RPC tasks if the rpc_clnt-default behavior
   is NORTO.

Any thoughts?

--
Chuck Lever