Re: Linux NFSv4 client uses returned delegation in subsequent READ resulting in hang (BAD_STATEID)

From: "Charles 'Boyo" <charlesboyo@gmail.com>
To: Chuck Lever <chuck.lever@oracle.com>
Cc: "Myklebust, Trond" <Trond.Myklebust@netapp.com>,
	"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>
Subject: Re: Linux NFSv4 client uses returned delegation in subsequent READ resulting in hang (BAD_STATEID)
Date: Mon, 2 Jul 2012 22:50:38 +0100	[thread overview]
Message-ID: <CAPD_G3KojVhnjtfjN0poJvpV0i+6JZeK54rm5Swm3MDKEXQ=Zw@mail.gmail.com> (raw)
In-Reply-To: <6C5408EB-668A-43C9-B3FE-9D66427B5BE0@oracle.com>

On Mon, Jul 2, 2012 at 10:19 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
>
> On Jul 2, 2012, at 5:13 PM, Myklebust, Trond wrote:
>
>> On Mon, 2012-07-02 at 16:35 -0400, Chuck Lever wrote:
>>> On Jul 2, 2012, at 4:22 PM, Charles 'Boyo wrote:
>>>
>>>> On Mon, Jul 2, 2012 at 3:09 PM, Chuck Lever <chuck.lever@oracle.com> wrote:
>>>>>
>>>>> Usually we see this behavior because of a race between an OPEN with delegation and a delegation recall.  In this case, however, the client is actively returning a READ
>>>>> delegation, then proceeding to use it anyway.  I don't see the server's recall callback, though, and there are other indications that this trace is not complete. So it's hard
>>>>> to be 100% confident.
>>>>>
>>>> The trace is not complete, it includes just enough information to
>>>> explain the problem.
>>>> However I can confirm the service did not send a recall callback, the
>>>> client returned the delegation of its own "free will".
>>>
>>> The callback would come on a separate TCP connection.  I can't think of a reason that a client would return a delegation by itself and then subsequently start to use it.
>>
>> I can: there are a number of servers out there that violate the spec by
>> returning a delegation as part of an OPEN(CLAIM_DELEGATE_CUR). Usually
>> those broken servers will send the exact same stateid as the delegation
>> that is being returned.
>
> The OPEN in frame 7 is a CLAIM_NULL OPEN, isn't it?
>
The OPEN in this case is a CLAIM_NULL and I have re-examined my
network dump, there was no call back from the server.
So why would the client returns a delegation voluntarily and then re-use it?

>>>> Is it possible is a scheduling issue of some sort, where the READ
>>>> should have been sent ahead of the DELEGRETURN but somehow got mixed
>>>> up?

>>> Or possibly that the DELEGRETURN doesn't actually remove the delegation state ID until the server has replied, and the READ request was sent before the DELEGRETURN
>>> reply arrived at the client.

Indeed, the READ was issued after the DELEGRETURN but before the
response to it. Is it possible to check if this is expected behavior?