Re: [PATCH v1 4/4] xprtrdma: Plant XID in on-the-wire RDMA offset (FRWR)

[Jason Gunthorpe <jgg@ziepe.ca>]

On Mon, Nov 19, 2018 at 09:45:24PM -0500, Tom Talpey wrote:
> On 11/19/2018 5:46 PM, Jason Gunthorpe wrote:
> > On Mon, Nov 19, 2018 at 09:42:56PM +0000, Mora, Jorge wrote:
> > > Hello Chuck,
> > > 
> > > I am confused, is it the whole purpose of RDMA is to place the data
> > > directly into the memory location given by the virtual address or
> > > offset? What you are saying is that this offset is not the actual
> > > memory address and so the driver must map this offset to the actual
> > > address?
> > 
> > All RDMA on the wire addresses are 'virtual' and always go through
> > translation hardware to reach a physical memory address. Even if you
> > have a virtual address that matches your physical address there will
> > still be translation hardware to enforce security.
> > 
> > The direct placement refers to not copying data, not to using physical
> > memory addresses throughout the protocol design.
> > 
> > So it is better to use a virtual address that doesn't disclose any
> > information about the host's configuration to the network, such as
> > physical locations of kernel memory.
> 
> Agreed. And, this is a very subtle but very powerful aspect of FRWR
> registration, that the high-order bits of the "virtual address" can
> basically be any value of the requestor's (client's) choice.
> 
> Note, the low-order bits, up to the maximum offset used by the upper
> layer (e.g. a few MB for NFS), must be reserved in order to allow RDMA
> segments to iterate over the range of bytes in the region. That's only
> 24 or so bits out of the available 64, here however.

Why does it depend on protocol? 

So long as the adaptor can compute VA + Offset = Physical the upper
bits can be changed. For many adaptors that means PAGE_SIZE or huge
page size level alignment..

Jason