Re: how to properly handle failures during delegation recall process

From: Olga Kornievskaia <aglo@umich.edu>
To: Trond Myklebust <trond.myklebust@primarydata.com>
Cc: linux-nfs <linux-nfs@vger.kernel.org>
Subject: Re: how to properly handle failures during delegation recall process
Date: Mon, 13 Oct 2014 18:13:05 -0400	[thread overview]
Message-ID: <CAN-5tyGsopFNFjo=LBwYE9n6b7Emv73Rkw0H2wyq2vCjMUOftA@mail.gmail.com> (raw)
In-Reply-To: <CAHQdGtRo2uw5U1-w9O2U=4C8BwxCECAjjeC9SCdkyEP8FQ94Gw@mail.gmail.com>

On Mon, Oct 13, 2014 at 5:29 PM, Trond Myklebust
<trond.myklebust@primarydata.com> wrote:
> On Mon, Oct 13, 2014 at 5:12 PM, Trond Myklebust
> <trond.myklebust@primarydata.com> wrote:
>> On Mon, Oct 13, 2014 at 4:23 PM, Olga Kornievskaia <aglo@umich.edu> wrote:
>>> On Mon, Oct 13, 2014 at 3:53 PM, Trond Myklebust
>>> <trond.myklebust@primarydata.com> wrote:
>>>> On Mon, Oct 13, 2014 at 2:51 PM, Olga Kornievskaia <aglo@umich.edu> wrote:
>>>>> +               }
>>>>> +       }
>>>>>         return nfs4_handle_delegation_recall_error(server, state, stateid, err);
>>>>>  }
>>>>>
>>>>> @@ -5957,6 +5973,10 @@ int nfs4_lock_delegation_recall(struct
>>>>> file_lock *fl, struct nfs4_state *state,
>>>>>         err = nfs4_set_lock_state(state, fl);
>>>>>         if (err != 0)
>>>>>                 return err;
>>>>> +       if (test_bit(NFS_LOCK_LOST, &fl->fl_u.nfs4_fl.owner->ls_flags)) {
>>>>> +               pr_warn_ratelimited("NFS: %s: Lock reclaim failed!\n",
>>>>> __func__);
>>>>> +               return -EIO;
>>>>> +       }
>>>>>         err = _nfs4_do_setlk(state, F_SETLK, fl, NFS_LOCK_NEW);
>>>>
>>>> Note that there is a potential race here if the server is playing with
>>>> NFS4ERR_DELEG_REVOKED or NFS4ERR_ADMIN_REVOKED. Since we do not
>>>> present the delegation as part of the LOCK request, we have no way of
>>>> knowing if the delegation is still in effect. I guess we can check the
>>>> return value of DELEGRETURN, but if it returns NFS4ERR_DELEG_REVOKED
>>>> or NFS4ERR_ADMIN_REVOKED, then we still do not know whether or not the
>>>> LOCK is safe.
>>>
>>> I'm not following you. We send LOCK before we send DELEGRETURN? Also,
>>> we normally send in LOCK the open_stateid returned by the open with
>>> cur so do we know that delegation is still in effect.
>>
>> How so? The open stateid doesn't tell you that the delegation is still
>> in effect. If the DELEGRETURN returns NFS4ERR_DELEG_REVOKED, how can
>> you tell if the delegation was revoked before or after the LOCK
>> request was handled?
>
> Actually, let me answer that myself. You can sort of figure things out
> in NFSv4.1 if you look at the SEQ4_STATUS_RECALLABLE_STATE_REVOKED
> flag. If it is set, you should probably distrust the lock stateid that
> you just attempted to recover, since you now know that at least one
> delegation was just revoked.
>
> In that case, we probably also have to do a TEST_STATEID+FREE_STATEID
> on the delegation stateid.

I think we are mis-communicating here by talking about different
nuances. I agree with you that when an operation is sent there is no
way of knowing if in the mean while the server has decided to revoke
the delegation. However, this is not what I'm confused about regarding
your comment. I'm noticing that in the flow of operations, we send (1)
open with cur, then (2) lock, then (3) delegreturn. So I was confused
about how can we check return of delegreturn, step 3, if we are in
step 2.

I think the LOCK is safe if the reply to the LOCK is successful.

Let me just step back from this to note that your solution to "lost
locks during delegation" is to recognize the open with cure failure
and skip locking and delegreturn. I can work on a patch for that.

Do you agree that the state recovery should not be initiated in case
we get those errors?

>
> --
> Trond Myklebust
>
> Linux NFS client maintainer, PrimaryData
>
> trond.myklebust@primarydata.com