Re: [PATCH v1 4/4] xprtrdma: Plant XID in on-the-wire RDMA offset (FRWR)

From: Tom Talpey <tom@talpey.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: "Mora, Jorge" <Jorge.Mora@netapp.com>,
	Chuck Lever <chuck.lever@oracle.com>,
	Olga Kornievskaia <aglo@umich.edu>,
	linux-rdma <linux-rdma@vger.kernel.org>,
	Linux NFS Mailing List <linux-nfs@vger.kernel.org>
Subject: Re: [PATCH v1 4/4] xprtrdma: Plant XID in on-the-wire RDMA offset (FRWR)
Date: Mon, 19 Nov 2018 22:25:37 -0500	[thread overview]
Message-ID: <8fd325fa-290a-6381-80d8-e6ac20acffe8@talpey.com> (raw)
In-Reply-To: <20181120030916.GQ4890@ziepe.ca>

On 11/19/2018 10:09 PM, Jason Gunthorpe wrote:
> On Mon, Nov 19, 2018 at 09:45:24PM -0500, Tom Talpey wrote:
>> On 11/19/2018 5:46 PM, Jason Gunthorpe wrote:
>>> On Mon, Nov 19, 2018 at 09:42:56PM +0000, Mora, Jorge wrote:
>>>> Hello Chuck,
>>>>
>>>> I am confused, is it the whole purpose of RDMA is to place the data
>>>> directly into the memory location given by the virtual address or
>>>> offset? What you are saying is that this offset is not the actual
>>>> memory address and so the driver must map this offset to the actual
>>>> address?
>>>
>>> All RDMA on the wire addresses are 'virtual' and always go through
>>> translation hardware to reach a physical memory address. Even if you
>>> have a virtual address that matches your physical address there will
>>> still be translation hardware to enforce security.
>>>
>>> The direct placement refers to not copying data, not to using physical
>>> memory addresses throughout the protocol design.
>>>
>>> So it is better to use a virtual address that doesn't disclose any
>>> information about the host's configuration to the network, such as
>>> physical locations of kernel memory.
>>
>> Agreed. And, this is a very subtle but very powerful aspect of FRWR
>> registration, that the high-order bits of the "virtual address" can
>> basically be any value of the requestor's (client's) choice.
>>
>> Note, the low-order bits, up to the maximum offset used by the upper
>> layer (e.g. a few MB for NFS), must be reserved in order to allow RDMA
>> segments to iterate over the range of bytes in the region. That's only
>> 24 or so bits out of the available 64, here however.
> 
> Why does it depend on protocol?

Because the maximum payload being RDMA's is an upper layer matter. The
idea here is to provide a certain number of "invariant" bits in the
offset / virtual address. Chuck proposes reserving just 32 bits, leaving
32 bits of range, quite comfy.

> So long as the adaptor can compute VA + Offset = Physical the upper
> bits can be changed.

Absolutely, but if the offset walks into the high-order invariant bits,
they won't be invariant any longer. It would still work, but it would
defeat the wireshark trick that Chuck wants to play.

> For many adaptors that means PAGE_SIZE or huge
> page size level alignment..

I'm not sure I understand this. The FRWR hides the native page size, by
providing the page frames in the local work request. Their size isn't
known to the protocol.

Tom.