Re: Question about fuse dax support

["John Groves" <john@jagalactic.com>]

On 23/10/10 04:06PM, Miklos Szeredi wrote:
> On Fri, 6 Oct 2023 at 20:12, John Groves <john@jagalactic.com> wrote:
> >
> > I see that there is some limited support for dax mapping of fuse files, but
> > it seems to be specifically for virtiofs. I admit I barely understand that
> > use case, but there is another fuse/dax use case that I’d like to explore.
> > I would appreciate feedback on this, including pointers to RTFM material,
> > etc.
> >
> > I’m interested in creating a file system interface to fabric-attached shared
> > memory (cxl). Think of a fuse file system that receives metadata (how MD is
> > distributed is orthogonal) and instantiates files that are backed by dax
> > memory (S_DAX files), such that the same ‘data sets’ can be visible as
> > mmap-able files on more than one server. I’d like feedback as to whether
> > this is (or could be) doable via fuse.
> >
> > Here is the main rub though. For this to perform adequately, I don’t think
> > it would be acceptable for each fault to call up to user space to resolve
> > the dax device & offset. So the kernel side of fuse would need to cache a
> > dax extent list for each file to TLB/page-table misses.
> >
> > I would appreciate any questions, pointers or feedback.
> 
> I think the passthrough patches should take care of this use case as well:
> 
> https://lore.kernel.org/all/20230519125705.598234-1-amir73il@gmail.com/
> Thanks,
> Miklos

Thanks for the reply Miklos.

I've looked over that patch set, and I'm pretty sure it's not what is needed
for my use case. I can see how my statement above "backed by dax memory
(S_DAX files)" could have implied that there is an S_DAX backing file that
fuse could use - but there is not already a backing file, just a dax device.

So it is the fuse file that would need to have the S_DAX flag and handle
mapping to an extent list from the dax device (rather than referring to a
backing file that already does this).

This is a performance-sensitive use case - S_DAX files are for direct access
to memory (duh). Posix read/write are supported, but the main use case for
S_DAX files is performant mmap. So I think this can only be viable if:

1) The fuse kernel module supports files with the S_DAX flag, and performs the
   appropriate mapping in conjunction with the dax driver (iomap, etc.)
2) The fuse kernel module caches the extent list of the backing memory
   (extents of the form [offset, length] or [device, offset, length]) so that
   TLB/page-table faults could be resolved without calling out to the user
   space handler.

My naive reading of the existence of some sort of fuse/dax support for virtiofs
suggested that there might be a way of doing this - but I may be wrong about
that.

Please let me know your thoughts on this.

Also: I will be at Linux Plumbers, and I'm speaking about this use case in
the cxl microconference. If you will be there, perhaps we can discuss it.
Any others interested in discussing it, here or at Plumbers, please ping me.

Thanks,
John Groves
Micron