Re: [RFC 1/2] xprtrdma: xdr pad optimization revisted again

From: Chuck Lever III <chuck.lever@oracle.com>
To: Olga Kornievskaia <olga.kornievskaia@gmail.com>
Cc: Trond Myklebust <trondmy@hammerspace.com>,
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>,
	Anna Schumaker <anna.schumaker@netapp.com>
Subject: Re: [RFC 1/2] xprtrdma: xdr pad optimization revisted again
Date: Tue, 31 Aug 2021 16:02:17 +0000	[thread overview]
Message-ID: <CAF2348E-72EC-447F-A94D-C98D97D79EED@oracle.com> (raw)
In-Reply-To: <CAN-5tyFwxC3BtU7xQiaUKdCnBQg1hfKv6QJ-dnnBrLnP0P9kfg@mail.gmail.com>

> On Aug 31, 2021, at 11:54 AM, Olga Kornievskaia <olga.kornievskaia@gmail.com> wrote:
> 
>> However, if there is a problem, it would be simple to create a
>> persistently-registered memory region that is not part of any RPC
>> buffer that can be used to catch unused Write chunk XDR padding.
> 
> When I think of XDR padding, I'm thinking of a number of bytes less
> than 4. Perhaps I'm confused in my understanding.  Tail.iov_len is
> never a value less than 4. Tail.iov_len can contain bytes after an
> opaque element for which a read or write chunk was used (eg
> READ+VERIFY or WRITE+GETATTR). Thus when RDMA looks at tail.iov_len
> and allocates that much memory, that allocation does not represent
> just XDR padding. Since RDMA can't distinguish between
> padding+operation vs padding, can this even be solved?

Yes, I think it can be solved. xprtrdma was relying on the upper
layers to tell it how big the XDR pad was. Instead, it can either
provide a 4-byte pad segment unconditionally, or it can look at
the value of in the rq_rcv_buf::page_len.

The reason for all the current cleverness in there is because
adding the pad segment has a cost. If we make it no-cost, it can
be added unconditionally, and all the cleverness can be tossed
out.

--
Chuck Lever