Re: interrupted rpcs problem

[Olga Kornievskaia <aglo@umich.edu>]

On Fri, Jan 10, 2020 at 4:03 PM Trond Myklebust <trondmy@hammerspace.com> wrote:
>
> On Fri, 2020-01-10 at 14:29 -0500, Olga Kornievskaia wrote:
> > Hi folks,
> >
> > We are having an issue with an interrupted RPCs again. Here's what I
> > see when xfstests were ctrl-c-ed.
> >
> > frame 332 SETATTR call slot=0 seqid=0x000013ca (I'm assuming this is
> > interrupted and released)
> > frame 333 CLOSE call slot=0 seqid=0x000013cb  (only way the slot
> > could
> > be free before the reply if it was interrupted, right? Otherwise we
> > should never have the slot used by more than one outstanding RPC)
> > frame 334 reply to 333 with SEQ_MIS_ORDERED (I'm assuming server
> > received frame 333 before 332)
> > frame 336 CLOSE call slot=0 seqid=0x000013ca (??? why did we
> > decremented it. I mean I know why it's in the current code :-/ )
> > frame 337 reply to 336 SEQUENCE with ERR_DELAY
> > frame 339 reply to 332 SETATTR which nobody is waiting for
> > frame 543 CLOSE call slot=0 seqid=0x000013ca (retry after waiting for
> > err_delay)
> > frame 544 reply to 543 with SETATTR (out of the cache).
> >
> > What this leads to is: file is never closed on the server. Can't
> > remove it. Unmount fails with CLID_BUSY.
> >
> > I believe that's the result of commit
> > 3453d5708b33efe76f40eca1c0ed60923094b971.
> > We used to have code that bumped the sequence up when the slot was
> > interrupted but after the commit "NFSv4.1: Avoid false retries when
> > RPC calls are interrupted".
> >
> > Commit has this "The obvious fix is to bump the sequence number
> > pre-emptively if an
> >     RPC call is interrupted, but in order to deal with the corner
> > cases
> >     where the interrupted call is not actually received and processed
> > by
> >     the server, we need to interpret the error NFS4ERR_SEQ_MISORDERED
> >     as a sign that we need to either wait or locate a correct
> > sequence
> >     number that lies between the value we sent, and the last value
> > that
> >     was acked by a SEQUENCE call on that slot."
> >
> > If we can't no longer just bump the sequence up, I don't think the
> > correct action is to automatically bump it down (as per example
> > here)?
> > The commit doesn't describe the corner case where it was necessary to
> > bump the sequence up. I wonder if we can return the knowledge of the
> > interrupted slot and make a decision based on that as well as
> > whatever
> > the other corner case is.
> >
> > I guess what I'm getting is, can somebody (Trond) provide the info
> > for
> > the corner case for this that patch was created. I can see if I can
> > fix the "common" case which is now broken and not break the corner
> > case....
> >
>
> There is no pure client side solution for this problem.
>
> The change was made because if you have multiple interruptions of the
> RPC call, then the client has to somehow figure out what the correct
> slot number is. If it starts low, and then goes high, and the server is
> not caching the arguments for the RPC call that is in the session
> cache, then we will _always_ hit this bug because we will always hit
> the replay of the last entry.
>
> At least if we start high, and iterate by low, then we reduce the
> problem to being a race with the processing of the interrupted request
> as it is in this case.
>
> However, as I said, the real solution here has to involve the server.

Ok I see your point that if the server cached the arguments, then the
server would tell that 2nd rpc using the same slot+seqid has different
args and would not use the replay cache.

However, I wonder if the client can do better. Can't we be more aware
of when we are interrupting the rpc? For instance, if we are
interrupted after we started to wait on the RPC, doesn't it mean the
rpc is sent on the network and since network is reliable then server
must have consumed the seqid for that slot (in this case increment
seqid)? That's the case that's failing now.

I really don't understand where the process is interrupted so that the
client consumes the seqid but the rpc never gets on the network. I see
the only other place is in call_allocate(). If that's the case, then
we can distinguish those two places and treat them differently (in
this case decrement seqid)?

>
> --
> Trond Myklebust
> Linux NFS client maintainer, Hammerspace
> trond.myklebust@hammerspace.com
>
>