knfsd server returns writeverf of all 0 bits (but was not rebooted)

knfsd server returns writeverf of all 0 bits (but was not rebooted)

[Rick Macklem]

Hi,

When testing against the knfsd in a Linux 5.15.1 kernel, I received a
write reply with FILE_SYNC and a writeverf of all 0 bytes.
(Previous write verifiers were not all 0 bytes.)

The server seemed to be functioning normally and had not rebooted.

Is this intended behaviour?

Normally I would not expect the writeverf in a Write operation reply
to change unless the server had rebooted, but I can see there might
be circumstances where the knfsd server wants all non-FILE_SYNC
writes to be redone by the client and would choose to change the
writeverf.
However, changing it to all 0 bytes seems particularly odd?

rick

Re: knfsd server returns writeverf of all 0 bits (but was not rebooted)

[Chuck Lever III]

Hi Rick-

> On Dec 9, 2021, at 6:15 PM, Rick Macklem <rmacklem@uoguelph.ca> wrote:
> 
> Hi,
> 
> When testing against the knfsd in a Linux 5.15.1 kernel, I received a
> write reply with FILE_SYNC and a writeverf of all 0 bytes.
> (Previous write verifiers were not all 0 bytes.)
> 
> The server seemed to be functioning normally and had not rebooted.
> 
> Is this intended behaviour?
> 
> Normally I would not expect the writeverf in a Write operation reply
> to change unless the server had rebooted, but I can see there might
> be circumstances where the knfsd server wants all non-FILE_SYNC
> writes to be redone by the client and would choose to change the
> writeverf.
> However, changing it to all 0 bytes seems particularly odd?

I don't immediately see a code path for WRITE or COMMIT that would
set the verifier to zeroes. When Linux NFSD resets its write verifier,
it sets it to the current time.

Do you have a reproducer you can share?


--
Chuck Lever

Re: knfsd server returns writeverf of all 0 bits (but was not rebooted)

[Trond Myklebust]

On Mon, 2021-12-13 at 19:42 +0000, Chuck Lever III wrote:
> Hi Rick-
> 
> > On Dec 9, 2021, at 6:15 PM, Rick Macklem <rmacklem@uoguelph.ca>
> > wrote:
> > 
> > Hi,
> > 
> > When testing against the knfsd in a Linux 5.15.1 kernel, I received
> > a
> > write reply with FILE_SYNC and a writeverf of all 0 bytes.
> > (Previous write verifiers were not all 0 bytes.)
> > 
> > The server seemed to be functioning normally and had not rebooted.
> > 
> > Is this intended behaviour?
> > 
> > Normally I would not expect the writeverf in a Write operation
> > reply
> > to change unless the server had rebooted, but I can see there might
> > be circumstances where the knfsd server wants all non-FILE_SYNC
> > writes to be redone by the client and would choose to change the
> > writeverf.
> > However, changing it to all 0 bytes seems particularly odd?
> 
> I don't immediately see a code path for WRITE or COMMIT that would
> set the verifier to zeroes. When Linux NFSD resets its write
> verifier,
> it sets it to the current time.
> 
> Do you have a reproducer you can share?
> 

Rick is using FILE_SYNC, whereas nfsd_vfs_write() only actually sets
the boot verifier if the write is unstable or DATA_SYNC.

-- 
Trond Myklebust
Linux NFS client maintainer, Hammerspace
trond.myklebust@hammerspace.com

Re: knfsd server returns writeverf of all 0 bits (but was not rebooted)

[Chuck Lever III]

> On Dec 13, 2021, at 2:54 PM, Trond Myklebust <trondmy@hammerspace.com> wrote:
> 
> On Mon, 2021-12-13 at 19:42 +0000, Chuck Lever III wrote:
>> Hi Rick-
>> 
>>> On Dec 9, 2021, at 6:15 PM, Rick Macklem <rmacklem@uoguelph.ca>
>>> wrote:
>>> 
>>> Hi,
>>> 
>>> When testing against the knfsd in a Linux 5.15.1 kernel, I received
>>> a
>>> write reply with FILE_SYNC and a writeverf of all 0 bytes.
>>> (Previous write verifiers were not all 0 bytes.)
>>> 
>>> The server seemed to be functioning normally and had not rebooted.
>>> 
>>> Is this intended behaviour?
>>> 
>>> Normally I would not expect the writeverf in a Write operation
>>> reply
>>> to change unless the server had rebooted, but I can see there might
>>> be circumstances where the knfsd server wants all non-FILE_SYNC
>>> writes to be redone by the client and would choose to change the
>>> writeverf.
>>> However, changing it to all 0 bytes seems particularly odd?
>> 
>> I don't immediately see a code path for WRITE or COMMIT that would
>> set the verifier to zeroes. When Linux NFSD resets its write
>> verifier,
>> it sets it to the current time.
>> 
>> Do you have a reproducer you can share?
>> 
> 
> Rick is using FILE_SYNC, whereas nfsd_vfs_write() only actually sets
> the boot verifier if the write is unstable or DATA_SYNC.

It wasn't clear from Rick's note that the verifier change he
observed was not permanent.

So then the answer to "Is this intended behavior?" is "Yes,
Linux NFSD returns an all-zero verifier for FILE_SYNC writes,
since the client does not use the verifier in this case. The
boot verifier for subsequent non-FILE_SYNC writes is not
altered."


--
Chuck Lever

Re: knfsd server returns writeverf of all 0 bits (but was not rebooted)

[Rick Macklem]

Chuck Lever III wrote:
>> On Dec 13, 2021, at 2:54 PM, Trond Myklebust <trondmy@hammerspace.com> wrote:
>>
>> On Mon, 2021-12-13 at 19:42 +0000, Chuck Lever III wrote:
>>> Hi Rick-
>>>
>>>> On Dec 9, 2021, at 6:15 PM, Rick Macklem <rmacklem@uoguelph.ca>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> When testing against the knfsd in a Linux 5.15.1 kernel, I received
>>>> a
>>>> write reply with FILE_SYNC and a writeverf of all 0 bytes.
>>>> (Previous write verifiers were not all 0 bytes.)
>>>>
>>>> The server seemed to be functioning normally and had not rebooted.
>>>>
>>>> Is this intended behaviour?
>>>>
>>>> Normally I would not expect the writeverf in a Write operation
>>>> reply
>>>> to change unless the server had rebooted, but I can see there might
>>>> be circumstances where the knfsd server wants all non-FILE_SYNC
>>>> writes to be redone by the client and would choose to change the
>>>> writeverf.
>>>> However, changing it to all 0 bytes seems particularly odd?
>>>
>>> I don't immediately see a code path for WRITE or COMMIT that would
>>> set the verifier to zeroes. When Linux NFSD resets its write
>>> verifier,
>>> it sets it to the current time.
>>>
>>> Do you have a reproducer you can share?
>>>
>>
>> Rick is using FILE_SYNC, whereas nfsd_vfs_write() only actually sets
>> the boot verifier if the write is unstable or DATA_SYNC.
>
>It wasn't clear from Rick's note that the verifier change he
>observed was not permanent.
>
>So then the answer to "Is this intended behavior?" is "Yes,
>Linux NFSD returns an all-zero verifier for FILE_SYNC writes,
>since the client does not use the verifier in this case. The
>boot verifier for subsequent non-FILE_SYNC writes is not
>altered."
Yes, that is what I observed.

The good news is that the FreeBSD client can easily ignore
the writeverf reply for FILESYNC writes (ones where the
write request specifies FILESYNC and the reply is expected
to be FILESYNC). I have already patched the client for this case.

Although I agree that a write reply that is FILESYNC does not
need to be redone, no matter what happens to writeverf, I
do not see anything in RFC8881 that suggests it should be
ignored in this case or that the value of all 0 bytes is special.

My interpretation of RFC8881 is that the writeverf is global
for the server and any change would indicate all writes that
did not reply FILESYNC and have not been committed, must
be redone.
--> There is the weird case w.r.t. File Layout pNFS when
      "commit through MDS" is indicated, where the writeverf
      appears to be "per file" and not "per server". And I'll admit
      I don't really understand this, despite having read it.

Does this need to be discussed on nfsv4@ietf.org?

In any case, the FreeBSD client never does FILESYNC writes
by default and is now fixed to ignore the writeverf reply in
this case.

And what about the case where the client does a write
with UNSTABLE, but the server chooses to reply FILESYNC?
--> Is the writeverf ever going to be zero for that case?

Thanks, rick


--
Chuck Lever