From: Jerome Glisse <jglisse@redhat.com>
To: Jason Gunthorpe <jgg@mellanox.com>
Cc: Joerg Roedel <jroedel@suse.de>,
"Rafael J . Wysocki" <rafael@kernel.org>,
Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
Felix Kuehling <Felix.Kuehling@amd.com>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"dri-devel@lists.freedesktop.org"
<dri-devel@lists.freedesktop.org>, Christoph Hellwig <hch@lst.de>,
"linux-mm@kvack.org" <linux-mm@kvack.org>,
"iommu@lists.linux-foundation.org"
<iommu@lists.linux-foundation.org>,
"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
Bjorn Helgaas <bhelgaas@google.com>,
Robin Murphy <robin.murphy@arm.com>,
Logan Gunthorpe <logang@deltatee.com>,
Christian Koenig <christian.koenig@amd.com>,
Marek Szyprowski <m.szyprowski@samsung.com>
Subject: Re: [RFC PATCH 3/5] mm/vma: add support for peer to peer to device vma
Date: Wed, 30 Jan 2019 16:45:25 -0500 [thread overview]
Message-ID: <20190130214525.GG5061@redhat.com> (raw)
In-Reply-To: <20190130204954.GI17080@mellanox.com>
On Wed, Jan 30, 2019 at 08:50:00PM +0000, Jason Gunthorpe wrote:
> On Wed, Jan 30, 2019 at 03:43:32PM -0500, Jerome Glisse wrote:
> > On Wed, Jan 30, 2019 at 08:11:19PM +0000, Jason Gunthorpe wrote:
> > > On Wed, Jan 30, 2019 at 01:00:02PM -0700, Logan Gunthorpe wrote:
> > >
> > > > We never changed SGLs. We still use them to pass p2pdma pages, only we
> > > > need to be a bit careful where we send the entire SGL. I see no reason
> > > > why we can't continue to be careful once their in userspace if there's
> > > > something in GUP to deny them.
> > > >
> > > > It would be nice to have heterogeneous SGLs and it is something we
> > > > should work toward but in practice they aren't really necessary at the
> > > > moment.
> > >
> > > RDMA generally cannot cope well with an API that requires homogeneous
> > > SGLs.. User space can construct complex MRs (particularly with the
> > > proposed SGL MR flow) and we must marshal that into a single SGL or
> > > the drivers fall apart.
> > >
> > > Jerome explained that GPU is worse, a single VMA may have a random mix
> > > of CPU or device pages..
> > >
> > > This is a pretty big blocker that would have to somehow be fixed.
> >
> > Note that HMM takes care of that RDMA ODP with my ODP to HMM patch,
> > so what you get for an ODP umem is just a list of dma address you
> > can program your device to. The aim is to avoid the driver to care
> > about that. The access policy when the UMEM object is created by
> > userspace through verbs API should however ascertain that for mmap
> > of device file it is only creating a UMEM that is fully covered by
> > one and only one vma. GPU device driver will have one vma per logical
> > GPU object. I expect other kind of device do that same so that they
> > can match a vma to a unique object in their driver.
>
> A one VMA rule is not really workable.
>
> With ODP VMA boundaries can move around across the lifetime of the MR
> and we have no obvious way to fail anything if userpace puts a VMA
> boundary in the middle of an existing ODP MR address range.
This is true only for vma that are not mmap of a device file. This is
what i was trying to get accross. An mmap of a file is never merge
so it can only get split/butcher by munmap/mremap but when that happen
you also need to reflect the virtual address space change to the
device ie any access to a now invalid range must trigger error.
>
> I think the HMM mirror API really needs to deal with this for the
> driver somehow.
Yes the HMM does deal with this for you, you do not have to worry about
it. Sorry if that was not clear. I just wanted to stress that vma that
are mmap of a file do not behave like other vma hence when you create
the UMEM you can check for those if you feel the need.
Cheers,
Jérôme
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
next prev parent reply other threads:[~2019-01-30 21:45 UTC|newest]
Thread overview: 95+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-01-29 17:47 [RFC PATCH 0/5] Device peer to peer (p2p) through vma jglisse
2019-01-29 17:47 ` [RFC PATCH 1/5] pci/p2p: add a function to test peer to peer capability jglisse
2019-01-29 18:24 ` Logan Gunthorpe
2019-01-29 19:44 ` Greg Kroah-Hartman
2019-01-29 19:53 ` Jerome Glisse
2019-01-29 20:44 ` Logan Gunthorpe
2019-01-29 21:00 ` Jerome Glisse
2019-01-29 19:56 ` Alex Deucher
2019-01-29 20:00 ` Jerome Glisse
2019-01-29 20:24 ` Logan Gunthorpe
2019-01-29 21:28 ` Alex Deucher
2019-01-30 10:25 ` Christian König
2019-01-29 17:47 ` [RFC PATCH 2/5] drivers/base: " jglisse
2019-01-29 18:26 ` Logan Gunthorpe
2019-01-29 19:54 ` Jerome Glisse
2019-01-29 19:46 ` Greg Kroah-Hartman
2019-01-29 19:56 ` Jerome Glisse
2019-01-29 17:47 ` [RFC PATCH 3/5] mm/vma: add support for peer to peer to device vma jglisse
2019-01-29 18:36 ` Logan Gunthorpe
2019-01-29 19:11 ` Jerome Glisse
2019-01-29 19:24 ` Logan Gunthorpe
2019-01-29 19:44 ` Jerome Glisse
2019-01-29 20:43 ` Logan Gunthorpe
2019-01-30 7:52 ` Christoph Hellwig
2019-01-29 19:32 ` Jason Gunthorpe
2019-01-29 19:50 ` Jerome Glisse
2019-01-29 20:24 ` Jason Gunthorpe
2019-01-29 20:44 ` Jerome Glisse
2019-01-29 23:02 ` Jason Gunthorpe
2019-01-30 0:08 ` Jerome Glisse
2019-01-30 4:30 ` Jason Gunthorpe
2019-01-30 15:43 ` Jerome Glisse
2019-01-29 20:39 ` Logan Gunthorpe
2019-01-29 20:57 ` Jerome Glisse
2019-01-29 21:30 ` Logan Gunthorpe
2019-01-29 21:50 ` Jerome Glisse
2019-01-29 22:58 ` Logan Gunthorpe
2019-01-29 23:47 ` Jerome Glisse
2019-01-30 1:17 ` Logan Gunthorpe
2019-01-30 2:48 ` Jerome Glisse
2019-01-30 4:18 ` Jason Gunthorpe
2019-01-30 8:00 ` Christoph Hellwig
2019-01-30 15:49 ` Jerome Glisse
2019-01-30 19:06 ` Jason Gunthorpe
[not found] ` <20190130190651.GC17080-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2019-01-30 19:45 ` Logan Gunthorpe
[not found] ` <840256f8-0714-5d7d-e5f5-c96aec5c2c05-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>
2019-01-30 19:59 ` Jason Gunthorpe
[not found] ` <20190130195900.GG17080-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2019-01-30 21:01 ` Logan Gunthorpe
[not found] ` <35bad6d5-c06b-f2a3-08e6-2ed0197c8691-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>
2019-01-30 21:50 ` Jason Gunthorpe
[not found] ` <20190130215019.GL17080-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2019-01-30 22:52 ` Logan Gunthorpe
2019-01-30 23:30 ` Jason Gunthorpe
2019-01-31 8:13 ` Christoph Hellwig
2019-01-31 15:37 ` Jerome Glisse
2019-01-31 19:02 ` Jason Gunthorpe
2019-01-31 19:19 ` Logan Gunthorpe
2019-01-31 19:54 ` Jason Gunthorpe
2019-01-31 19:35 ` Jerome Glisse
2019-01-31 19:44 ` Logan Gunthorpe
2019-01-31 19:58 ` Jason Gunthorpe
2019-01-30 17:17 ` Logan Gunthorpe
[not found] ` <bdf03cd5-f5b1-4b78-a40e-b24024ca8c9f-OTvnGxWRz7hWk0Htik3J/w@public.gmane.org>
2019-01-30 18:56 ` Jason Gunthorpe
[not found] ` <20190130185652.GB17080-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2019-01-30 19:22 ` Jerome Glisse
2019-01-30 19:38 ` Jason Gunthorpe
[not found] ` <20190130193759.GE17080-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2019-01-30 20:00 ` Logan Gunthorpe
2019-01-30 20:11 ` Jason Gunthorpe
2019-01-30 20:43 ` Jerome Glisse
[not found] ` <20190130204332.GF5061-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2019-01-30 20:50 ` Jason Gunthorpe
2019-01-30 21:45 ` Jerome Glisse [this message]
[not found] ` <20190130214525.GG5061-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2019-01-30 21:56 ` Jason Gunthorpe
2019-01-30 22:30 ` Jerome Glisse
[not found] ` <20190130223027.GH5061-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2019-01-30 22:33 ` Jason Gunthorpe
2019-01-30 22:47 ` Jerome Glisse
[not found] ` <20190130224705.GI5061-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2019-01-30 22:51 ` Jason Gunthorpe
2019-01-30 22:58 ` Jerome Glisse
[not found] ` <20190130192234.GD5061-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
2019-01-30 19:52 ` Logan Gunthorpe
2019-01-30 20:35 ` Jerome Glisse
2019-01-29 20:58 ` Jason Gunthorpe
2019-01-30 8:02 ` Christoph Hellwig
2019-01-30 10:33 ` Koenig, Christian
2019-01-30 15:55 ` Jerome Glisse
2019-01-30 17:26 ` Christoph Hellwig
2019-01-30 17:32 ` Logan Gunthorpe
2019-01-30 17:39 ` Jason Gunthorpe
2019-01-30 18:05 ` Jerome Glisse
2019-01-30 17:44 ` Jason Gunthorpe
2019-01-30 18:13 ` Logan Gunthorpe
2019-01-30 18:50 ` Jerome Glisse
2019-01-31 8:02 ` Christoph Hellwig
2019-01-31 15:03 ` Jerome Glisse
2019-01-30 19:19 ` Jason Gunthorpe
[not found] ` <20190130191946.GD17080-VPRAkNaXOzVWk0Htik3J/w@public.gmane.org>
2019-01-30 19:48 ` Logan Gunthorpe
2019-01-30 20:44 ` Jason Gunthorpe
2019-01-31 8:05 ` Christoph Hellwig
2019-01-31 15:11 ` Jerome Glisse
2019-01-29 17:47 ` [RFC PATCH 4/5] mm/hmm: add support for peer to peer to HMM device memory jglisse
2019-01-29 17:47 ` [RFC PATCH 5/5] mm/hmm: add support for peer to peer to special device vma jglisse
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190130214525.GG5061@redhat.com \
--to=jglisse@redhat.com \
--cc=Felix.Kuehling@amd.com \
--cc=bhelgaas@google.com \
--cc=christian.koenig@amd.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=gregkh@linuxfoundation.org \
--cc=hch@lst.de \
--cc=iommu@lists.linux-foundation.org \
--cc=jgg@mellanox.com \
--cc=jroedel@suse.de \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-pci@vger.kernel.org \
--cc=logang@deltatee.com \
--cc=m.szyprowski@samsung.com \
--cc=rafael@kernel.org \
--cc=robin.murphy@arm.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).