Re: [PATCH v16 0/4] RDMA: Add dma-buf support

From: Daniel Vetter <daniel@ffwll.ch>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: Leon Romanovsky <leon@kernel.org>,
	linux-rdma <linux-rdma@vger.kernel.org>,
	John Hubbard <jhubbard@nvidia.com>,
	Maling list - DRI developers <dri-devel@lists.freedesktop.org>,
	Doug Ledford <dledford@redhat.com>,
	Daniel Vetter <daniel.vetter@intel.com>,
	Christian Koenig <christian.koenig@amd.com>,
	Jianxin Xiong <jianxin.xiong@intel.com>
Subject: Re: [PATCH v16 0/4] RDMA: Add dma-buf support
Date: Fri, 5 Feb 2021 17:06:43 +0100	[thread overview]
Message-ID: <YB1tE0WKUOBaO03x@phenom.ffwll.local> (raw)
In-Reply-To: <20210205160003.GU4247@nvidia.com>

On Fri, Feb 05, 2021 at 12:00:03PM -0400, Jason Gunthorpe wrote:
> On Fri, Feb 05, 2021 at 04:53:04PM +0100, Daniel Vetter wrote:
> > On Fri, Feb 05, 2021 at 11:43:19AM -0400, Jason Gunthorpe wrote:
> > > On Fri, Feb 05, 2021 at 04:39:47PM +0100, Daniel Vetter wrote:
> > > 
> > > > > And again, for slightly older hardware, without pinning to VRAM there is
> > > > > no way to use this solution here for peer-to-peer. So I'm glad to see that
> > > > > so far you're not ruling out the pinning option.
> > > > 
> > > > Since HMM and ZONE_DEVICE came up, I'm kinda tempted to make ZONE_DEVICE
> > > > ZONE_MOVEABLE (at least if you don't have a pinned vram contigent in your
> > > > cgroups) or something like that, so we could benefit from the work to make
> > > > sure pin_user_pages and all these never end up in there?
> > > 
> > > ZONE_DEVICE should already not be returned from GUP.
> > > 
> > > I've understood in the hmm casse the idea was a CPU touch of some
> > > ZONE_DEVICE pages would trigger a migration to CPU memory, GUP would
> > > want to follow the same logic, presumably it comes for free with the
> > > fault handler somehow
> > 
> > Oh I didn't know this, I thought the proposed p2p direct i/o patches would
> > just use the fact that underneath ZONE_DEVICE there's "normal" struct
> > pages. 
> 
> So, if that every happens, it would be some special FOLL_ALLOW_P2P
> flag to get the behavior.
> 
> > And so I got worried that maybe also pin_user_pages can creep in.
> > But I didn't read the patches in full detail:
> 
> And yes, you might want to say that you can't longterm pin certain
> kinds of zone_device pages, but if that is the common operating mode
> then we'd probably never create a FOLL_ALLOW_P2P
> 
> > But if you're saying that this all needs specific code and all the gup/pup
> > code we have is excluded, I think we can make sure that we're not ever
> > building features that requiring time-unlimited pinning of
> > ZONE_DEVICE.
> 
> Well, it is certainly a useful idea of some uses of ZONE_DEVICE, GPU
> vram is not the whole world.

Yeah non-volatile RAM can probably pin whatever it wants :-)

From the other thread, I think if we can get some cgroups going for
accounting pinned memory, then pinning gpu memory should also not be any
real issue. Might be somewhat tricky to glue that into a FOLL_ALLOW_P2P
flag, maybe through zone-awareness or something like that. With the right
accounting in place I'm happy to let userspace pin whatever they want
really.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel