Re: [PATCH RFC 0/6] NFSD: Pull Read chunks in XDR decoders

[Chuck Lever III <chuck.lever@oracle.com>]

> On Aug 31, 2021, at 4:42 PM, J. Bruce Fields <bfields@fieldses.org> wrote:
> 
> On Tue, Aug 31, 2021 at 03:05:09PM -0400, Chuck Lever wrote:
>> Hi Bruce-
>> 
>> Here is part of what we discussed recently about trying to align
>> pages in NFS WRITE requests so that splice can be used. This
>> series updates server-side RPC and svcrdma to ensure that an
>> aligned xdr_buf::pages array is presented to NFSD, which is then
>> converted into an aligned rq_vec for the VFS layer.
> 
> Seems sensible to me.
> 
> Do you have a git tree?

I don't yet, but can set up a topic branch somewhere where we're
a little further along.


> It didn't apply cleanly to 5.14 when I tried,
> but I didn't stop to figure out why.

It might apply to 5.15-pre now that nfsd-5.15 has been merged.


>> The next step would be to look at how to make the best use of the
>> aligned rq_vec.
> 
> Have you done any performance comparison just with this?

Not yet. I just got it behaving correctly with the usual tests.
Baby steps, sir.


> Doesn't seem like it should make a significant difference, but it might
> be interesting to check anyway.

I expect it to add a small amount of latency to NFS WRITEs, since
RDMA Reads are done just a little later than before.


> --b.
> 
>> My naive thought is that where there is a PAGE_SIZE
>> entry in rq_vec and there is no page in the file's page cache at
>> that offset, the transport-provided page can be flipped into place.
>> Might work for replacing whole pages as well, but baby steps first.
>> 
>> This series has been exercised a bit with both TCP and RDMA, but no
>> guarantees that it is completely bug-free. NFSv4 compounds with
>> multiple WRITE payloads on RDMA are treated like TCP: the RPC
>> message is contained in an unstructured stream of unaligned pages.
>> 
>> Comments encouraged.
>> 
>> ---
>> 
>> Chuck Lever (6):
>>      SUNRPC: Capture value of xdr_buf::page_base
>>      SUNRPC: xdr_stream_subsegment() must handle non-zero page_bases
>>      NFSD: Have legacy NFSD WRITE decoders use xdr_stream_subsegment()
>>      SUNRPC: svc_fill_write_vector() must handle non-zero page_bases
>>      NFSD: Add a transport hook for pulling argument payloads
>>      svcrdma: Pull Read chunks in ->xpo_argument_payload
>> 
>> 
>> fs/nfsd/nfs3proc.c                       |   3 +-
>> fs/nfsd/nfs3xdr.c                        |  16 +--
>> fs/nfsd/nfs4proc.c                       |   3 +-
>> fs/nfsd/nfs4xdr.c                        |   6 +
>> fs/nfsd/nfsproc.c                        |   3 +-
>> fs/nfsd/nfsxdr.c                         |  13 +--
>> fs/nfsd/xdr.h                            |   2 +-
>> fs/nfsd/xdr3.h                           |   2 +-
>> include/linux/sunrpc/svc.h               |   6 +-
>> include/linux/sunrpc/svc_rdma.h          |   8 ++
>> include/linux/sunrpc/svc_xprt.h          |   3 +
>> include/trace/events/rpcrdma.h           |  26 +++++
>> include/trace/events/sunrpc.h            |  20 +++-
>> net/sunrpc/svc.c                         |  38 +++++--
>> net/sunrpc/svcsock.c                     |   8 ++
>> net/sunrpc/xdr.c                         |  32 +++---
>> net/sunrpc/xprtrdma/svc_rdma_recvfrom.c  |  37 +++++-
>> net/sunrpc/xprtrdma/svc_rdma_rw.c        | 139 ++++++++++++++++++++---
>> net/sunrpc/xprtrdma/svc_rdma_transport.c |   1 +
>> 19 files changed, 292 insertions(+), 74 deletions(-)
>> 
>> --
>> Chuck Lever

--
Chuck Lever