Re: [PATCH v6 2/7] fuse: make DAX mount option a tri-state

[Ira Weiny <ira.weiny@intel.com>]

On Mon, Oct 25, 2021 at 02:12:10PM -0400, Vivek Goyal wrote:
> On Mon, Oct 25, 2021 at 10:52:51AM -0700, Ira Weiny wrote:
> > On Fri, Oct 22, 2021 at 02:54:03PM +0800, JeffleXu wrote:
> > > cc [Ira Weiny], author of per inode DAX on xfs/ext4
> > > 
> > > On 10/20/21 11:17 PM, Vivek Goyal wrote:
> > > > On Wed, Oct 20, 2021 at 10:52:38AM +0800, JeffleXu wrote:
> > > >>
> > > >>
> > > >> On 10/18/21 10:10 PM, Vivek Goyal wrote:
> > > >>> On Mon, Oct 11, 2021 at 11:00:47AM +0800, Jeffle Xu wrote:
> > > >>>> We add 'always', 'never', and 'inode' (default). '-o dax' continues to
> > > >>>> operate the same which is equivalent to 'always'. To be consistemt with
> > > >>>> ext4/xfs's tri-state mount option, when neither '-o dax' nor '-o dax='
> > > >>>> option is specified, the default behaviour is equal to 'inode'.
> > > >>>
> > > >>> Hi Jeffle,
> > > >>>
> > > >>> I am not sure when  -o "dax=inode"  is used as a default? If user
> > > >>> specifies, "-o dax" then it is equal to "-o dax=always", otherwise
> > > >>> user will explicitly specify "-o dax=always/never/inode". So when
> > > >>> is dax=inode is used as default?
> > > >>
> > > >> That means when neither '-o dax' nor '-o dax=always/never/inode' is
> > > >> specified, it is actually equal to '-o dax=inode', which is also how
> > > >> per-file DAX on ext4/xfs works.
> > > > 
> > 
> > It's been a while so I'm fuzzy on the details of the discussions but yes that
> > is the way things are now in the code.
> > 
> > > > [ CC dave chinner] 
> > > > 
> > > > Is it not change of default behavior for ext4/xfs as well. My
> > > > understanding is that prior to this new dax options, "-o dax" enabled
> > > > dax on filesystem and if user did not specify it, DAX is disbaled
> > > > by default.
> > 
> > Technically it does change default behavior...  However, NOT in a way which
> > breaks anything.  See below.
> > 
> > > > 
> > > > Now after introduction of "-o dax=always/never/inode", if suddenly
> > > > "-o dax=inode" became the default if user did not specify anything,
> > > > that's change of behavior.
> > 
> > Technically yes but not in a broken way.
> > 
> > > >
> > > > Is that intentional. If given a choice,
> > > > I would rather not change default and ask user to opt-in for
> > > > appropriate dax functionality.
> > 
> > There is no need for this.
> > 
> > > > 
> > > > Dave, you might have thoughts on this. It makes me uncomfortable to
> > > > change virtiofs dax default now just because other filesytems did it.
> > > > 
> > > 
> > > I can only find the following discussions about the earliest record on
> > > this tri-state mount option:
> > > 
> > > https://lore.kernel.org/lkml/20200316095509.GA13788@lst.de/
> > > https://lore.kernel.org/lkml/20200401040021.GC56958@magnolia/
> > > 
> > > 
> > > Hi, Ira Weiny,
> > > 
> > > Do you have any thought on this, i.e. why the default behavior has
> > > changed after introduction of per inode dax?
> > 
> > While this is 'technically' different behavior the end user does not see any
> > difference in behavior if they continue without software changes.  Specifically
> > specifying nothing continues to operate with all the files on the FS to be
> > '_not_ DAX'.  While specifying '-o dax' forces DAX on all files.
> > 
> > This expands the default behavior in a backwards compatible manner.
> 
> This is backward compatible in a sense that if somebody upgrades to new
> kernel, things will still be same. 
> 
> I think little problematic change is that say I bring in persistent
> memory from another system (which has FS_XFLAGS_DAX set on some inodes)
> and then mount it without andy of the dax mount options, then per
> inode dax will be enabled unexpectedly if I boot with newer kernels
> but it will be disable if I mount with older kernels. Do I understand it
> right.

Indeed that will happen.  However, wouldn't the users (software) of those files
have knowledge that those files were DAX and want to continue with them in that
mode?

> 
> > The user
> > can now enable DAX on some files.  But this is an opt-in on the part of the
> > user of the FS and again does not change with existing software/scripts/etc.
> 
> Don't understand this "opt-in" bit. If user mounts an fs without
> specifying any of the dax options, then per inode dax will still be
> enabled if inode has the correct flag set.

But only users who actually set that flag 'opt-in'.

> So is setting of flag being
> considered as opt-in (insted of mount option).

Yes.

> 
> If setting of flag is being considered as opt-in, that probably will not
> work very well with virtiofs. Because server can enforce a different
> policy for enabling per file dax (instead of FS_XFLAG_DAX).

I'm not sure I understand how this happens?  I think the server probably has to
enable per INODE by default to allow the client to do what the end users wants.

I agree that if the end user is expecting DAX and the server disables it then
that is a problem but couldn't that happen before?  Maybe I'm getting confused
because I'm not familiar enough with virtiofs.

> 
> And given there are two entities here (client and server), I think it
> will be good if if we give client a chance as well to decide whether
> it wants to enable per file dax or not. I know it can alwasy do 
> "dax=never" but it can still be broken if client software remains
> same but host/server software is upgraded or commnad line changed.

But the files are 'owned' by a single user or group of users who must have
placed the file in DAX mode at some point right?

> 
> So for virtiofs, I think better behavior is to continue to not enable
> any dax until and unless user opts-in using "-o dax=foo" options.

I'm not sure, maybe.

Ira