Re: [TOPIC] Extending the filesystem crash recovery guaranties contract

["Darrick J. Wong" <darrick.wong@oracle.com>]

On Thu, May 09, 2019 at 12:02:17AM -0500, Vijay Chidambaram wrote:
> On Wed, May 8, 2019 at 9:30 PM Theodore Ts'o <tytso@mit.edu> wrote:
> >
> > On Thu, May 09, 2019 at 11:43:27AM +1000, Dave Chinner wrote:
> > >
> > > .... the whole point of SOMC is that allows filesystems to avoid
> > > dragging external metadata into fsync() operations /unless/ there's
> > > a user visible ordering dependency that must be maintained between
> > > objects.  If all you are doing is stabilising file data in a stable
> > > file/directory, then independent, incremental journaling of the
> > > fsync operations on that file fit the SOMC model just fine.
> >
> > Well, that's not what Vijay's crash consistency guarantees state.  It
> > guarantees quite a bit more than what you've written above.  Which is
> > my concern.
> 
> The intention is to capture Dave's SOMC semantics. We can re-iterate
> and re-phrase until we capture what Dave meant precisely. I am fairly
> confident we can do this, given that Dave himself is participating and
> helping us refine the text. So this doesn't seem like a reason not to
> have documentation at all to me.
> 
> As we have stated on multiple times on this and other threads, the
> intention is *not* to come up with one set of crash-recovery
> guarantees that every Linux file system must abide by forever. Ted,
> you keep repeating this, though we have never said this was our
> intention.

It might not be your intention, but I can definitely imagine others
using such a SOMC document as a cudgel to, uh, pressure other
filesystems into implementing the same semantics ("This isn't SOMC
compliant!" "We never said it was." "It has to be compliant!").  That's
fine for XFS because that's how it's supposed to work, but I wouldn't
want other projects to have to defend themselves for lack of XFSiness.

> The intention behind this effort is to simply document the
> crash-recovery guarantees provided today by different Linux file
> systems. Ted, you question why this is required at all, and why we
> simply can't use POSIX and man pages. The answer:
> 
> 1. POSIX is vague. Not persisting data to stable media on fsync is
> also allowed in POSIX (but no Linux file system actually does this),
> so its not very useful in terms of understanding what crash-recovery
> guarantees file systems actually provide. Given that all Linux file
> systems provide something more than POSIX, the natural question to ask
> is what do they provide? We understood this from working on
> CrashMonkey, and we wanted to document it.
> 2. Other parts of the Linux kernel have much better documentation,
> even though they similarly want to provide freedom for developers to
> optimize and change internal implementation. I don't think
> documentation and freedom to change internals are mutually exclusive.
> 3. XFS provides SOMC semantics, and btrfs developers have stated they
> want to provide SOMC as well. F2FS developers have a mode in which
> they seek to provide SOMC semantics. Given all this, it seemed prudent
> to document SOMC.

Point.  To further soften/undercut my earlier email, I think we can
document the filesystem behaviors that specific projects are willing to
endorse while still making it clear that YMMV and you had better test
your workload if you want clarity of behavior. :)

> 4. Apart from developers, a document like this would also help
> academic researchers understand the current state-of-the-art in
> crash-recovery guarantees and the different choices made by different
> file systems. It is non-trivial to understand this without
> documentation.
> 
> FWIW, I think the position of "if we don't write it down, application
> developers can't depend on it" is wrong. Even with nothing written
> down, developers noticed they could skip fsync() in ext3 when
> atomically updating files with rename(). This lead to the whole ext4
> rename-and-delayed-allocation problem. The much better path, IMO, is
> to document the current set of guarantees given by different file
> systems, and talk about what is intended and what is not. This would
> give application developers much better guidance in writing
> applications.
> 
> If ext4 wants to develop incremental fsync and introduce a new set of
> semantics that is different from SOMC and much closer to minimal
> POSIX, I don't think the documentation affects that at all. As Dave
> notes, diversity is good! Documentation is also good :)
> 
> That being said, I think I'll stop our push to get this documented
> inside the Linux kernel at this point. We got useful comments from
> Dave, Amir, and others, so we will incorporate those comments and put
> up the documentation on a University of Texas web page. If someone
> else wants to carry on and get this merged, you are welcome to do so
> :)

Aww, I was going to suggest merging it with an explicit warning that the
document *only* reflects those that have endorsed it, and a pileup at
the end:

Endorsed-by: fs/xfs
Endorsed-by: fs/btrfs

Rejected-by: fs/djwongcrazyfs

(I'm still ¾ tempted to just put the XFS parts in a text file and merge
it into fs/xfs/ if the broader effort doesn't succeed...)

--D