Re: [TOPIC] Extending the filesystem crash recovery guaranties contract

[Vijay Chidambaram <vijay@cs.utexas.edu>]

On Wed, May 8, 2019 at 9:30 PM Theodore Ts'o <tytso@mit.edu> wrote:
>
> On Thu, May 09, 2019 at 11:43:27AM +1000, Dave Chinner wrote:
> >
> > .... the whole point of SOMC is that allows filesystems to avoid
> > dragging external metadata into fsync() operations /unless/ there's
> > a user visible ordering dependency that must be maintained between
> > objects.  If all you are doing is stabilising file data in a stable
> > file/directory, then independent, incremental journaling of the
> > fsync operations on that file fit the SOMC model just fine.
>
> Well, that's not what Vijay's crash consistency guarantees state.  It
> guarantees quite a bit more than what you've written above.  Which is
> my concern.

The intention is to capture Dave's SOMC semantics. We can re-iterate
and re-phrase until we capture what Dave meant precisely. I am fairly
confident we can do this, given that Dave himself is participating and
helping us refine the text. So this doesn't seem like a reason not to
have documentation at all to me.

As we have stated on multiple times on this and other threads, the
intention is *not* to come up with one set of crash-recovery
guarantees that every Linux file system must abide by forever. Ted,
you keep repeating this, though we have never said this was our
intention.

The intention behind this effort is to simply document the
crash-recovery guarantees provided today by different Linux file
systems. Ted, you question why this is required at all, and why we
simply can't use POSIX and man pages. The answer:

1. POSIX is vague. Not persisting data to stable media on fsync is
also allowed in POSIX (but no Linux file system actually does this),
so its not very useful in terms of understanding what crash-recovery
guarantees file systems actually provide. Given that all Linux file
systems provide something more than POSIX, the natural question to ask
is what do they provide? We understood this from working on
CrashMonkey, and we wanted to document it.
2. Other parts of the Linux kernel have much better documentation,
even though they similarly want to provide freedom for developers to
optimize and change internal implementation. I don't think
documentation and freedom to change internals are mutually exclusive.
3. XFS provides SOMC semantics, and btrfs developers have stated they
want to provide SOMC as well. F2FS developers have a mode in which
they seek to provide SOMC semantics. Given all this, it seemed prudent
to document SOMC.
4. Apart from developers, a document like this would also help
academic researchers understand the current state-of-the-art in
crash-recovery guarantees and the different choices made by different
file systems. It is non-trivial to understand this without
documentation.

FWIW, I think the position of "if we don't write it down, application
developers can't depend on it" is wrong. Even with nothing written
down, developers noticed they could skip fsync() in ext3 when
atomically updating files with rename(). This lead to the whole ext4
rename-and-delayed-allocation problem. The much better path, IMO, is
to document the current set of guarantees given by different file
systems, and talk about what is intended and what is not. This would
give application developers much better guidance in writing
applications.

If ext4 wants to develop incremental fsync and introduce a new set of
semantics that is different from SOMC and much closer to minimal
POSIX, I don't think the documentation affects that at all. As Dave
notes, diversity is good! Documentation is also good :)

That being said, I think I'll stop our push to get this documented
inside the Linux kernel at this point. We got useful comments from
Dave, Amir, and others, so we will incorporate those comments and put
up the documentation on a University of Texas web page. If someone
else wants to carry on and get this merged, you are welcome to do so
:)