From: Oded Gabbay <oded.gabbay@gmail.com>
To: Jason Gunthorpe <jgg@ziepe.ca>
Cc: "Christian König" <ckoenig.leichtzumerken@gmail.com>,
"Gal Pressman" <galpress@amazon.com>,
sleybo@amazon.com, linux-rdma <linux-rdma@vger.kernel.org>,
"Oded Gabbay" <ogabbay@kernel.org>,
"Christoph Hellwig" <hch@lst.de>,
"Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
dri-devel <dri-devel@lists.freedesktop.org>,
"Christian König" <christian.koenig@amd.com>,
"moderated list:DMA BUFFER SHARING FRAMEWORK"
<linaro-mm-sig@lists.linaro.org>,
"Doug Ledford" <dledford@redhat.com>,
"Tomer Tayar" <ttayar@habana.ai>,
"amd-gfx list" <amd-gfx@lists.freedesktop.org>,
"Greg KH" <gregkh@linuxfoundation.org>,
"Alex Deucher" <alexander.deucher@amd.com>,
"Leon Romanovsky" <leonro@nvidia.com>,
"open list:DMA BUFFER SHARING FRAMEWORK"
<linux-media@vger.kernel.org>
Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF
Date: Tue, 22 Jun 2021 16:12:26 +0300 [thread overview]
Message-ID: <CAFCwf13BuS+U3Pko_62hFPuvZPG26HQXuu-cxPmcADNPO22g9g@mail.gmail.com> (raw)
In-Reply-To: <20210622121546.GN1096940@ziepe.ca>
On Tue, Jun 22, 2021 at 3:15 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
>
> On Tue, Jun 22, 2021 at 03:04:30PM +0300, Oded Gabbay wrote:
> > On Tue, Jun 22, 2021 at 3:01 PM Jason Gunthorpe <jgg@ziepe.ca> wrote:
> > >
> > > On Tue, Jun 22, 2021 at 11:42:27AM +0300, Oded Gabbay wrote:
> > > > On Tue, Jun 22, 2021 at 9:37 AM Christian König
> > > > <ckoenig.leichtzumerken@gmail.com> wrote:
> > > > >
> > > > > Am 22.06.21 um 01:29 schrieb Jason Gunthorpe:
> > > > > > On Mon, Jun 21, 2021 at 10:24:16PM +0300, Oded Gabbay wrote:
> > > > > >
> > > > > >> Another thing I want to emphasize is that we are doing p2p only
> > > > > >> through the export/import of the FD. We do *not* allow the user to
> > > > > >> mmap the dma-buf as we do not support direct IO. So there is no access
> > > > > >> to these pages through the userspace.
> > > > > > Arguably mmaping the memory is a better choice, and is the direction
> > > > > > that Logan's series goes in. Here the use of DMABUF was specifically
> > > > > > designed to allow hitless revokation of the memory, which this isn't
> > > > > > even using.
> > > > >
> > > > > The major problem with this approach is that DMA-buf is also used for
> > > > > memory which isn't CPU accessible.
> > >
> > > That isn't an issue here because the memory is only intended to be
> > > used with P2P transfers so it must be CPU accessible.
> > >
> > > > > That was one of the reasons we didn't even considered using the mapping
> > > > > memory approach for GPUs.
> > >
> > > Well, now we have DEVICE_PRIVATE memory that can meet this need
> > > too.. Just nobody has wired it up to hmm_range_fault()
> > >
> > > > > > So you are taking the hit of very limited hardware support and reduced
> > > > > > performance just to squeeze into DMABUF..
> > > >
> > > > Thanks Jason for the clarification, but I honestly prefer to use
> > > > DMA-BUF at the moment.
> > > > It gives us just what we need (even more than what we need as you
> > > > pointed out), it is *already* integrated and tested in the RDMA
> > > > subsystem, and I'm feeling comfortable using it as I'm somewhat
> > > > familiar with it from my AMD days.
> > >
> > > You still have the issue that this patch is doing all of this P2P
> > > stuff wrong - following the already NAK'd AMD approach.
> >
> > Could you please point me exactly to the lines of code that are wrong
> > in your opinion ?
>
> 1) Setting sg_page to NULL
> 2) 'mapping' pages for P2P DMA without going through the iommu
> 3) Allowing P2P DMA without using the p2p dma API to validate that it
> can work at all in the first place.
>
> All of these result in functional bugs in certain system
> configurations.
>
> Jason
Hi Jason,
Thanks for the feedback.
Regarding point 1, why is that a problem if we disable the option to
mmap the dma-buf from user-space ? We don't want to support CPU
fallback/Direct IO.
In addition, I didn't see any problem with sg_page being NULL in the
RDMA p2p dma-buf code. Did I miss something here ?
Regarding points 2 & 3, I want to examine them more closely in a KVM
virtual machine environment with IOMMU enabled.
I will take two GAUDI devices and use one as an exporter and one as an
importer. I want to see that the solution works end-to-end, with real
device DMA from importer to exporter.
I fear that the dummy importer I wrote is bypassing these two issues
you brought up.
So thanks again and I'll get back and update once I've finished testing it.
Oded
next prev parent reply other threads:[~2021-06-22 13:12 UTC|newest]
Thread overview: 50+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-06-18 12:36 [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF Oded Gabbay
2021-06-18 12:36 ` [PATCH v3 2/2] habanalabs: add support for dma-buf exporter Oded Gabbay
2021-06-21 12:28 ` [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF Daniel Vetter
2021-06-21 13:02 ` Greg KH
2021-06-21 14:12 ` Jason Gunthorpe
2021-06-21 16:26 ` Oded Gabbay
2021-06-21 17:55 ` Jason Gunthorpe
2021-06-21 18:27 ` Daniel Vetter
2021-06-21 19:24 ` Oded Gabbay
2021-06-21 23:29 ` Jason Gunthorpe
2021-06-22 6:37 ` [Linaro-mm-sig] " Christian König
2021-06-22 8:42 ` Oded Gabbay
2021-06-22 12:01 ` Jason Gunthorpe
2021-06-22 12:04 ` Oded Gabbay
2021-06-22 12:15 ` Jason Gunthorpe
2021-06-22 13:12 ` Oded Gabbay [this message]
2021-06-22 15:11 ` Jason Gunthorpe
2021-06-22 15:24 ` Christian König
2021-06-22 15:28 ` Jason Gunthorpe
2021-06-22 15:31 ` Oded Gabbay
2021-06-22 15:31 ` Christian König
2021-06-22 15:40 ` Oded Gabbay
2021-06-22 15:49 ` Christian König
2021-06-22 15:24 ` Oded Gabbay
2021-06-22 15:34 ` Jason Gunthorpe
2021-06-22 12:23 ` Christian König
2021-06-22 15:23 ` Jason Gunthorpe
2021-06-22 15:29 ` Christian König
2021-06-22 15:40 ` Jason Gunthorpe
2021-06-22 15:48 ` Christian König
2021-06-22 16:05 ` Jason Gunthorpe
2021-06-23 8:57 ` Christian König
2021-06-23 9:14 ` Oded Gabbay
2021-06-23 18:24 ` Jason Gunthorpe
2021-06-23 18:43 ` Oded Gabbay
2021-06-23 18:50 ` Jason Gunthorpe
2021-06-23 19:00 ` Oded Gabbay
2021-06-23 19:34 ` Jason Gunthorpe
2021-06-23 19:39 ` Oded Gabbay
2021-06-24 0:45 ` Jason Gunthorpe
2021-06-24 5:40 ` Christoph Hellwig
2021-06-24 5:34 ` Christoph Hellwig
2021-06-24 8:07 ` Christian König
2021-06-24 8:12 ` Christoph Hellwig
2021-06-24 9:52 ` Christian König
2021-06-24 13:22 ` Christoph Hellwig
2021-06-22 16:50 ` Felix Kuehling
2021-06-21 14:20 ` Daniel Vetter
2021-06-21 14:49 ` Jason Gunthorpe
2021-06-21 14:17 ` Jason Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAFCwf13BuS+U3Pko_62hFPuvZPG26HQXuu-cxPmcADNPO22g9g@mail.gmail.com \
--to=oded.gabbay@gmail.com \
--cc=alexander.deucher@amd.com \
--cc=amd-gfx@lists.freedesktop.org \
--cc=christian.koenig@amd.com \
--cc=ckoenig.leichtzumerken@gmail.com \
--cc=dledford@redhat.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=galpress@amazon.com \
--cc=gregkh@linuxfoundation.org \
--cc=hch@lst.de \
--cc=jgg@ziepe.ca \
--cc=leonro@nvidia.com \
--cc=linaro-mm-sig@lists.linaro.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=ogabbay@kernel.org \
--cc=sleybo@amazon.com \
--cc=ttayar@habana.ai \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).