Re: [PATCH RFC] vfs: add a O_NOMTIME flag

[Dave Chinner <david@fromorbit.com>]

On Thu, May 07, 2015 at 09:23:24PM -0400, Trond Myklebust wrote:
> On Thu, May 7, 2015 at 9:01 PM, Sage Weil <sage@newdream.net> wrote:
> > On Thu, 7 May 2015, Zach Brown wrote:
> >> On Thu, May 07, 2015 at 10:26:17AM +1000, Dave Chinner wrote:
> >> > On Wed, May 06, 2015 at 03:00:12PM -0700, Zach Brown wrote:
> >> > > The criteria for using O_NOMTIME is the same as for using O_NOATIME:
> >> > > owning the file or having the CAP_FOWNER capability.  If we're not
> >> > > comfortable allowing owners to prevent mtime/ctime updates then we
> >> > > should add a tunable to allow O_NOMTIME.  Maybe a mount option?
> >> >
> >> > I dislike "turn off safety for performance" options because Joe
> >> > SpeedRacer will always select performance over safety.
> >>
> >> Well, for ceph there's no safety concern.  They never use cmtime in
> >> these files.
> >>
> >> So are you suggesting not implementing this and making them rework their
> >> IO paths to avoid the fs maintaining mtime so that we don't give Joe
> >> Speedracer more rope?  Or are we talking about adding some speed bumps
> >> that ceph can flip on that might give Joe Speedracer pause?
> >
> > I think this is the fundamental question: who do we give the ammunition
> > to, the user or app writer, or the sysadmin?
> >
> > One might argue that we gave the user a similar power with O_NOATIME (the
> > power to break applications that assume atime is accurate).  Here we give
> > developers/users the power to not update mtime and suffer the consequences
> > (like, obviously, breaking mtime-based backups).  It should be pretty
> > obvious to anyone using the flag what the consequences are.
> >
> > Note that we can suffer similar lapses in mtime with fdatasync followed by
> > a system crash.  And as Andy points out it's semi-broken for writable
> > mmap.  The crash case is obviously a slightly different thing, but the
> > idea that mtime can't always be trusted certainly isn't crazy talk.
> >
> > Or, we can be conservative and require a mount option so that the admin
> > has to explicitly allow behavior that might break some existing
> > assumptions about mtime/ctime ('-o user_noatime' I guess?).
> >
> > I'm happy either way, so long as in the end an unprivileged ceph daemon
> > avoids the useless work.  In our case we always own the entire mount/disk,
> > so a mount option is just fine.
> >
> 
> So, what is the expectation here for filesystems that cannot support
> this flag? NFSv3 in particular would break pretty catastrophically if
> someone decided on a whim to turn off mtime: they will have turned off
> the client's ability to detect cache incoherencies.

It's worse than that, now that I think about it. I think nomtime
will break nfsv4 as the I_VERSION check is done *after* the
NO[C]MTIME checks. e.g. the atomic change count used to detect file
changes is only updated during the mtime update on write() calls in
XFS. i.e. when the timestamp is changed, a transaction to change
mtime is run, and that transaction commit bumps the change count.

So cutting out mtime updates at the VFS will prevent XFS and other
I_VERSION aware filesystems from updating the change count that
NFSv4 clients rely on to detect foreign data changes in a file.

Not sure what to do here, because the current NOCMTIME
implementation intentionally cuts out the timestamp update because
it's usage is fully invisible IO. i.e. it is used by utilities like
xfs_fsr and HSMs to move data into and out of files without the
application being able to detect the data movement in any way. These
are not data modification operations, though - the file contents as
read by the application do not change despite the fact we are moving
data in and out of the file. In this case we don't want timestamps
or change counters to change on the data movement, so I think we've
actually got a difference in behaviour here between O_NOMTIME and
O_NOCMTIME, right?

i.e. for nfsv4 sanity O_NOMTIME still needs to bump I_VERSION on
write, just not modify the timestamp? In which case, not modifying
the timestamps gains us nothing, because the inode is still dirtied?

The list of caveats on O_NOMTIME seems to be growing...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com