From: John Hubbard <jhubbard@nvidia.com>
To: Daniel Vetter <daniel@ffwll.ch>, Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leon@kernel.org>,
linux-rdma <linux-rdma@vger.kernel.org>,
Maling list - DRI developers <dri-devel@lists.freedesktop.org>,
Doug Ledford <dledford@redhat.com>,
Daniel Vetter <daniel.vetter@intel.com>,
Christian Koenig <christian.koenig@amd.com>,
Jianxin Xiong <jianxin.xiong@intel.com>
Subject: Re: [PATCH v16 0/4] RDMA: Add dma-buf support
Date: Fri, 5 Feb 2021 12:24:38 -0800 [thread overview]
Message-ID: <4c339fc3-087b-1008-fb99-7117bf326470@nvidia.com> (raw)
In-Reply-To: <YB1p4Bpmz0yFcbEf@phenom.ffwll.local>
On 2/5/21 7:53 AM, Daniel Vetter wrote:
> On Fri, Feb 05, 2021 at 11:43:19AM -0400, Jason Gunthorpe wrote:
>> On Fri, Feb 05, 2021 at 04:39:47PM +0100, Daniel Vetter wrote:
>>
>>>> And again, for slightly older hardware, without pinning to VRAM there is
>>>> no way to use this solution here for peer-to-peer. So I'm glad to see that
>>>> so far you're not ruling out the pinning option.
>>>
>>> Since HMM and ZONE_DEVICE came up, I'm kinda tempted to make ZONE_DEVICE
>>> ZONE_MOVEABLE (at least if you don't have a pinned vram contigent in your
>>> cgroups) or something like that, so we could benefit from the work to make
>>> sure pin_user_pages and all these never end up in there?
>>
>> ZONE_DEVICE should already not be returned from GUP.
>>
>> I've understood in the hmm casse the idea was a CPU touch of some
>> ZONE_DEVICE pages would trigger a migration to CPU memory, GUP would
>> want to follow the same logic, presumably it comes for free with the
>> fault handler somehow
>
> Oh I didn't know this, I thought the proposed p2p direct i/o patches would
> just use the fact that underneath ZONE_DEVICE there's "normal" struct
> pages. And so I got worried that maybe also pin_user_pages can creep in.
> But I didn't read the patches in full detail:
>
> https://lore.kernel.org/linux-block/20201106170036.18713-12-logang@deltatee.com/
>
> But if you're saying that this all needs specific code and all the gup/pup
> code we have is excluded, I think we can make sure that we're not ever
> building features that requiring time-unlimited pinning of ZONE_DEVICE.
> Which I think we want.
>
From an HMM perspective, the above sounds about right. HMM relies on the
GPU/device memory being ZONE_DEVICE, *and* on that memory *not* being pinned.
(HMM's mmu notifier callbacks act as a sort of virtual pin, but not a refcount
pin.)
It's a nice clean design point that we need to preserve, and fortunately it
doesn't conflict with anything I'm seeing here. But I want to say this out
loud because I see some doubt about it creeping into the discussion.
thanks,
--
John Hubbard
NVIDIA
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
prev parent reply other threads:[~2021-02-05 20:24 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-15 21:27 [PATCH v16 0/4] RDMA: Add dma-buf support Jianxin Xiong
2020-12-15 21:27 ` [PATCH v16 1/4] RDMA/umem: Support importing dma-buf as user memory region Jianxin Xiong
2020-12-15 21:27 ` [PATCH v16 2/4] RDMA/core: Add device method for registering dma-buf based " Jianxin Xiong
2020-12-15 21:27 ` [PATCH v16 3/4] RDMA/uverbs: Add uverbs command for dma-buf based MR registration Jianxin Xiong
2020-12-15 21:27 ` [PATCH v16 4/4] RDMA/mlx5: Support dma-buf based userspace memory region Jianxin Xiong
2021-01-11 15:24 ` [PATCH v16 0/4] RDMA: Add dma-buf support Xiong, Jianxin
2021-01-11 15:42 ` Jason Gunthorpe
2021-01-11 17:44 ` Xiong, Jianxin
2021-01-11 17:47 ` Alex Deucher
2021-01-11 17:55 ` Xiong, Jianxin
2021-01-12 12:49 ` Yishai Hadas
2021-01-12 18:11 ` Xiong, Jianxin
2021-01-21 16:59 ` Jason Gunthorpe
2021-02-04 7:48 ` John Hubbard
2021-02-04 13:50 ` Alex Deucher
2021-02-04 18:29 ` Jason Gunthorpe
2021-02-04 18:44 ` Alex Deucher
2021-02-04 19:00 ` John Hubbard
2021-02-05 15:39 ` Daniel Vetter
2021-02-05 15:43 ` Jason Gunthorpe
2021-02-05 15:53 ` Daniel Vetter
2021-02-05 16:00 ` Jason Gunthorpe
2021-02-05 16:06 ` Daniel Vetter
2021-02-05 20:24 ` John Hubbard [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4c339fc3-087b-1008-fb99-7117bf326470@nvidia.com \
--to=jhubbard@nvidia.com \
--cc=christian.koenig@amd.com \
--cc=daniel.vetter@intel.com \
--cc=daniel@ffwll.ch \
--cc=dledford@redhat.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=jgg@nvidia.com \
--cc=jianxin.xiong@intel.com \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).