IB_MR_TYPE_SG_GAPS support

IB_MR_TYPE_SG_GAPS support

[Chuck Lever III]

Hi-

I have a report that net/sunrpc/xprtrdma/frwr_ops.c::frwr_map()
is attempting to register a mix of GPU and CPU pages with a
single MR during an NFSv4 READ.

We think we understand where both pages are coming from, and
why frwr_map() is attempting to coalesce them (the verbs
provider happens to support IB_DEVICE_SG_GAPS_REG).

frwr_map() does this:

305         for (i = 0; i < nsegs;) {
306                 sg_set_page(&mr->mr_sg[i], seg->mr_page,
307                             seg->mr_len, seg->mr_offset);
308 
309                 ++seg;
310                 ++i;
311                 if (ep->re_mrtype == IB_MR_TYPE_SG_GAPS)
312                         continue;
313                 if ((i < nsegs && seg->mr_offset) ||
314                     offset_in_page((seg-1)->mr_offset + (seg-1)->mr_len))
315                         break;
316         }

If GPU and CPU pages are not considered contiguous, then some
additional loop termination logic is needed here so that at
a boundary between page types, xprtrdma will simply move
into a fresh MR instead of coalescing.

Does anyone have suggestions of where to look for an
appropriate check?

TIA

--
Chuck Lever

Re: IB_MR_TYPE_SG_GAPS support

[Jason Gunthorpe]

On Thu, Aug 26, 2021 at 05:26:11PM +0000, Chuck Lever III wrote:

> If GPU and CPU pages are not considered contiguous, then some
> additional loop termination logic is needed here so that at
> a boundary between page types, xprtrdma will simply move
> into a fresh MR instead of coalescing.

Upstream Linux does not have support for GPU pages in the RDMA stack.

Jason

Re: IB_MR_TYPE_SG_GAPS support

[Christoph Hellwig]

On Thu, Aug 26, 2021 at 05:26:11PM +0000, Chuck Lever III wrote:
> Hi-
> 
> I have a report that net/sunrpc/xprtrdma/frwr_ops.c::frwr_map()
> is attempting to register a mix of GPU and CPU pages with a
> single MR during an NFSv4 READ.

What GPU pages?

Re: IB_MR_TYPE_SG_GAPS support

[Max Gurtovoy]

On 8/27/2021 9:27 AM, Christoph Hellwig wrote:
> On Thu, Aug 26, 2021 at 05:26:11PM +0000, Chuck Lever III wrote:
>> Hi-
>>
>> I have a report that net/sunrpc/xprtrdma/frwr_ops.c::frwr_map()
>> is attempting to register a mix of GPU and CPU pages with a
>> single MR during an NFSv4 READ.
> What GPU pages?

I guess it's GPU device memory.

But as Jason mentioned, This is still not supported in RDMA. So there 
should be a bounce buffer in the IO path between GPU and CPU memory 
subsystems.

Re: IB_MR_TYPE_SG_GAPS support

[Christoph Hellwig]

On Sun, Aug 29, 2021 at 04:46:52PM +0300, Max Gurtovoy wrote:
> > > I have a report that net/sunrpc/xprtrdma/frwr_ops.c::frwr_map()
> > > is attempting to register a mix of GPU and CPU pages with a
> > > single MR during an NFSv4 READ.
> > What GPU pages?
> 
> I guess it's GPU device memory.
> 
> But as Jason mentioned, This is still not supported in RDMA. So there should
> be a bounce buffer in the IO path between GPU and CPU memory subsystems.

Well, the RDMA stack does support P2P pages, which can be exported by
GPUs.  It also supports device private pages by migrating them back to
the CPU through hmm_range_fault.  But none of this should require
SG_GAPS.