* IB_MR_TYPE_SG_GAPS support
@ 2021-08-26 17:26 Chuck Lever III
2021-08-26 23:45 ` Jason Gunthorpe
2021-08-27 6:27 ` Christoph Hellwig
0 siblings, 2 replies; 5+ messages in thread
From: Chuck Lever III @ 2021-08-26 17:26 UTC (permalink / raw)
To: linux-rdma; +Cc: Kornievskaia, Olga
Hi-
I have a report that net/sunrpc/xprtrdma/frwr_ops.c::frwr_map()
is attempting to register a mix of GPU and CPU pages with a
single MR during an NFSv4 READ.
We think we understand where both pages are coming from, and
why frwr_map() is attempting to coalesce them (the verbs
provider happens to support IB_DEVICE_SG_GAPS_REG).
frwr_map() does this:
305 for (i = 0; i < nsegs;) {
306 sg_set_page(&mr->mr_sg[i], seg->mr_page,
307 seg->mr_len, seg->mr_offset);
308
309 ++seg;
310 ++i;
311 if (ep->re_mrtype == IB_MR_TYPE_SG_GAPS)
312 continue;
313 if ((i < nsegs && seg->mr_offset) ||
314 offset_in_page((seg-1)->mr_offset + (seg-1)->mr_len))
315 break;
316 }
If GPU and CPU pages are not considered contiguous, then some
additional loop termination logic is needed here so that at
a boundary between page types, xprtrdma will simply move
into a fresh MR instead of coalescing.
Does anyone have suggestions of where to look for an
appropriate check?
TIA
--
Chuck Lever
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: IB_MR_TYPE_SG_GAPS support
2021-08-26 17:26 IB_MR_TYPE_SG_GAPS support Chuck Lever III
@ 2021-08-26 23:45 ` Jason Gunthorpe
2021-08-27 6:27 ` Christoph Hellwig
1 sibling, 0 replies; 5+ messages in thread
From: Jason Gunthorpe @ 2021-08-26 23:45 UTC (permalink / raw)
To: Chuck Lever III; +Cc: linux-rdma, Kornievskaia, Olga
On Thu, Aug 26, 2021 at 05:26:11PM +0000, Chuck Lever III wrote:
> If GPU and CPU pages are not considered contiguous, then some
> additional loop termination logic is needed here so that at
> a boundary between page types, xprtrdma will simply move
> into a fresh MR instead of coalescing.
Upstream Linux does not have support for GPU pages in the RDMA stack.
Jason
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: IB_MR_TYPE_SG_GAPS support
2021-08-26 17:26 IB_MR_TYPE_SG_GAPS support Chuck Lever III
2021-08-26 23:45 ` Jason Gunthorpe
@ 2021-08-27 6:27 ` Christoph Hellwig
2021-08-29 13:46 ` Max Gurtovoy
1 sibling, 1 reply; 5+ messages in thread
From: Christoph Hellwig @ 2021-08-27 6:27 UTC (permalink / raw)
To: Chuck Lever III; +Cc: linux-rdma, Kornievskaia, Olga
On Thu, Aug 26, 2021 at 05:26:11PM +0000, Chuck Lever III wrote:
> Hi-
>
> I have a report that net/sunrpc/xprtrdma/frwr_ops.c::frwr_map()
> is attempting to register a mix of GPU and CPU pages with a
> single MR during an NFSv4 READ.
What GPU pages?
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: IB_MR_TYPE_SG_GAPS support
2021-08-27 6:27 ` Christoph Hellwig
@ 2021-08-29 13:46 ` Max Gurtovoy
2021-08-30 8:29 ` Christoph Hellwig
0 siblings, 1 reply; 5+ messages in thread
From: Max Gurtovoy @ 2021-08-29 13:46 UTC (permalink / raw)
To: Christoph Hellwig, Chuck Lever III; +Cc: linux-rdma, Kornievskaia, Olga
On 8/27/2021 9:27 AM, Christoph Hellwig wrote:
> On Thu, Aug 26, 2021 at 05:26:11PM +0000, Chuck Lever III wrote:
>> Hi-
>>
>> I have a report that net/sunrpc/xprtrdma/frwr_ops.c::frwr_map()
>> is attempting to register a mix of GPU and CPU pages with a
>> single MR during an NFSv4 READ.
> What GPU pages?
I guess it's GPU device memory.
But as Jason mentioned, This is still not supported in RDMA. So there
should be a bounce buffer in the IO path between GPU and CPU memory
subsystems.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: IB_MR_TYPE_SG_GAPS support
2021-08-29 13:46 ` Max Gurtovoy
@ 2021-08-30 8:29 ` Christoph Hellwig
0 siblings, 0 replies; 5+ messages in thread
From: Christoph Hellwig @ 2021-08-30 8:29 UTC (permalink / raw)
To: Max Gurtovoy
Cc: Christoph Hellwig, Chuck Lever III, linux-rdma, Kornievskaia, Olga
On Sun, Aug 29, 2021 at 04:46:52PM +0300, Max Gurtovoy wrote:
> > > I have a report that net/sunrpc/xprtrdma/frwr_ops.c::frwr_map()
> > > is attempting to register a mix of GPU and CPU pages with a
> > > single MR during an NFSv4 READ.
> > What GPU pages?
>
> I guess it's GPU device memory.
>
> But as Jason mentioned, This is still not supported in RDMA. So there should
> be a bounce buffer in the IO path between GPU and CPU memory subsystems.
Well, the RDMA stack does support P2P pages, which can be exported by
GPUs. It also supports device private pages by migrating them back to
the CPU through hmm_range_fault. But none of this should require
SG_GAPS.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-08-30 8:30 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-26 17:26 IB_MR_TYPE_SG_GAPS support Chuck Lever III
2021-08-26 23:45 ` Jason Gunthorpe
2021-08-27 6:27 ` Christoph Hellwig
2021-08-29 13:46 ` Max Gurtovoy
2021-08-30 8:29 ` Christoph Hellwig
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.