Re: [LSF/MM/BPF TOPIC] Netfs support library

Re: [LSF/MM/BPF TOPIC] Netfs support library

[Steve French]

On Wed, Mar 9, 2022 at 10:17 AM Chuck Lever III <chuck.lever@oracle.com> wrote:
>
> > On Mar 4, 2022, at 3:06 PM, Steve French <smfrench@gmail.com> wrote:
> >
> > On Tue, Feb 1, 2022 at 10:51 PM Chuck Lever III <chuck.lever@oracle.com> wrote:
> >>
> >>> On Jan 31, 2022, at 4:06 PM, David Howells <dhowells@redhat.com> wrote:
> >>>
> >>> I've been working on a library (in fs/netfs/) to provide network filesystem
> >>> support services, with help particularly from Jeff Layton.  The idea is to
> >>> move the common features of the VM interface, including request splitting,
> >>> operation retrying, local caching, content encryption, bounce buffering and
> >>> compression into one place so that various filesystems can share it.
> >>
> >> IIUC this suite of functions is beneficial mainly to clients,
> >> is that correct? I'd like to be clear about that, this is not
> >> an objection to the topic.
> >>
> >> I'm interested in discussing how folios might work for the
> >> NFS _server_, perhaps as a separate or adjunct conversation.
> >
> > That is an interesting point.   Would like to also discuss whether it
> > could help ksmbd,
>
> Agreed that ksmbd might also benefit.
>
> Of primary interest is how read and write operations move
> their payloads from the network transports directly to the
> page cache. For the most efficient operation, the transport
> layer would need to set up a receive buffer using page
> cache pages instead of what is currently done: receive
> buffers are constructed from anonymous pages and then these
> pages replace the file's page cache pages.
>
> RDMA transports make this straightforward, at least from
> the transport layer's perspective.
>
>
> > and would like to continue discussion of netfs improvements - especially am
> > interested in how we can improve throttling when network (or server)
> > is congested
> > (or as network adapters are added/removed and additional bandwidth is
> > available).
>
> As I understand it, the protocols themselves have flow control
> built in. At least SMB Direct and NFS/RDMA do, and NFSv4
> sessions provides a similar mechanism for transports that don't
> have a native flow control mechanism. I'm not sufficiently
> familiar with SMB to know how flow control works there.
>
> However, each of these mechanisms all have their own specific
> rules. I'm not sure if there's much that can be extracted into
> a common library, but can you be more specific about what is
> needed?

At a high level given network latencies, we want to make sure
a few requests are in process on each interface unless server
is throttling back (or client observers problems) so the
network or cluster fs probably needs to be able to set:
- optimal i/o size (larger is usually better, for most SMB3.1.1
cases it would typically be 1M or 4M, but server negotiates
this with client at mount time)
- typical number of I/Os in flight (e.g. for SMB3.1.1 this
might be 2 or 3 times the number of channels)

and then dynamically we need to be able to reduce
(and then increase back to the default) the number of i/o when two things
happen:
- low on credits (the server returns 'credits' available
on each response, so when server is busy this number
can go down to the point where we don't want to do
much readahead)
- network or server events that indicate temporarily stopping
readahead (e.g. reconnect, or errors indicating server
resource problems)
- server notification about resource movement (server or share move)
- server notification about adding or deleting a network interface
(this and the above are handled via a loosely related
protocol called the 'Witness protocol' which generates dynamic
server notification to the client about important events).
Samuel Cabrero might have additional ideas about new events
we could add that could help here.

There also may be low memory situations where we see a need to
reduce the amount of resources used by the client fs.
-- 
Thanks,

Steve