RE: client caching and locks

["inoguchi.yuki@fujitsu.com" <inoguchi.yuki@fujitsu.com>]

Hi Neil and Bruce,

Thank you for your comments.
Now I have understood that this behavior is by design.

> > With NFSv4 there is no atomicity guarantees relating to writes and
> > changeid.
> > There is provision for atomicity around directory operations, but not
> > around data operations.

So I feel like clients cannot always trust changeid in NFSv4. 
Should it be described in the spec?

For example, the following section refers about the usage of changeid:
https://datatracker.ietf.org/doc/html/draft-dnoveck-nfsv4-rfc5661bis#section-14.3.1

It says clients use change attribute " to ensure that the data for the OPENed file is still 
correctly reflected in the client's cache." But in fact, it could be wrong.

Yuki Inoguchi

> -----Original Message-----
> From: 'bfields@fieldses.org' <bfields@fieldses.org>
> Sent: Tuesday, January 4, 2022 1:21 AM
> To: NeilBrown <neilb@suse.de>
> Cc: Inoguchi, Yuki/井ノ口 雄生 <inoguchi.yuki@fujitsu.com>; 'Matt Benjamin'
> <mbenjami@redhat.com>; 'Trond Myklebust' <trondmy@hammerspace.com>;
> 'linux-nfs@vger.kernel.org' <linux-nfs@vger.kernel.org>
> Subject: Re: client caching and locks
> 
> On Tue, Dec 28, 2021 at 04:11:51PM +1100, NeilBrown wrote:
> > This is due to an (arguable) weakness in the NFSv4 protocol.
> > In NFSv3 the reply to the WRITE request had "wcc" data which would
> > report change information before and after the write and, if present, it
> > was required to be atomic.  So, providing timestamps had a high
> > resolution, the client0 would see change information from *before* the
> > write from client1 completed.  So it would know it needed to refresh
> > after that write became visible.
> >
> > With NFSv4 there is no atomicity guarantees relating to writes and
> > changeid.
> > There is provision for atomicity around directory operations, but not
> > around data operations.
> >
> > This means that if different clients access a file concurrently, then
> > their cache can become incorrect.  The only way to ensure uncorrupted
> > data is to use locking for ALL reads and writes.  The above 'od -i' does
> > not perform a locked read, so can give incorrect data.
> > If you got a whole-file lock before reading, then you should get correct
> > data.
> 
> You'd also have to get a whole-file write lock on every write, wouldn't
> you, to prevent your own write from obscuring the change-attribute
> update caused by a concurrent writer?
> 
> > You could argue that this requirement (always lock if there is any risk)
> > is by design, and so this aspect of the protocl is not a weakness.
> 
> The spec discussion of byte-range locking and caching is here:
> https://datatracker.ietf.org/doc/html/rfc7530#section-10.3.2
> 
> The nfs man page, under ac/noac, says "Using the noac option provides
> greater  cache  coherence among  NFS  clients accessing the same files,
> but it extracts a significant performance penalty.  As such,  judicious
> use of file locking is encouraged instead.  The DATA AND METADATA
> COHERENCE section contains a detailed discussion of these trade-offs."
> 
> That section does have a "Using file locks with NFS" subsection, but
> that subsection doesn't actually discuss the interaction of file locks
> with client caching.
> 
> It's a confusing and under-documented area.
> 
> --b.