From: Jason Gunthorpe <jgg@ziepe.ca>
To: Jianxin Xiong <jianxin.xiong@intel.com>
Cc: Leon Romanovsky <leon@kernel.org>,
linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org,
Doug Ledford <dledford@redhat.com>,
Daniel Vetter <daniel.vetter@intel.com>,
Christian Koenig <christian.koenig@amd.com>
Subject: Re: [RFC PATCH v3 1/4] RDMA/umem: Support importing dma-buf as user memory region
Date: Mon, 5 Oct 2020 10:13:02 -0300 [thread overview]
Message-ID: <20201005131302.GQ9916@ziepe.ca> (raw)
In-Reply-To: <1601838751-148544-2-git-send-email-jianxin.xiong@intel.com>
On Sun, Oct 04, 2020 at 12:12:28PM -0700, Jianxin Xiong wrote:
> Dma-buf is a standard cross-driver buffer sharing mechanism that can be
> used to support peer-to-peer access from RDMA devices.
>
> Device memory exported via dma-buf is associated with a file descriptor.
> This is passed to the user space as a property associated with the
> buffer allocation. When the buffer is registered as a memory region,
> the file descriptor is passed to the RDMA driver along with other
> parameters.
>
> Implement the common code for importing dma-buf object and mapping
> dma-buf pages.
>
> Signed-off-by: Jianxin Xiong <jianxin.xiong@intel.com>
> Reviewed-by: Sean Hefty <sean.hefty@intel.com>
> Acked-by: Michael J. Ruhl <michael.j.ruhl@intel.com>
> ---
> drivers/infiniband/core/Makefile | 2 +-
> drivers/infiniband/core/umem.c | 4 +
> drivers/infiniband/core/umem_dmabuf.c | 291 ++++++++++++++++++++++++++++++++++
> drivers/infiniband/core/umem_dmabuf.h | 14 ++
> drivers/infiniband/core/umem_odp.c | 12 ++
> include/rdma/ib_umem.h | 19 ++-
> 6 files changed, 340 insertions(+), 2 deletions(-)
> create mode 100644 drivers/infiniband/core/umem_dmabuf.c
> create mode 100644 drivers/infiniband/core/umem_dmabuf.h
I think this is using ODP too literally, dmabuf isn't going to need
fine grained page faults, and I'm not sure this locking scheme is OK -
ODP is horrifically complicated.
If this is the approach then I think we should make dmabuf its own
stand alone API, reg_user_mr_dmabuf()
The implementation in mlx5 will be much more understandable, it would
just do dma_buf_dynamic_attach() and program the XLT exactly the same
as a normal umem.
The move_notify() simply zap's the XLT and triggers a work to reload
it after the move. Locking is provided by the dma_resv_lock. Only a
small disruption to the page fault handler is needed.
> + dma_resv_lock(umem_dmabuf->attach->dmabuf->resv, NULL);
> + sgt = dma_buf_map_attachment(umem_dmabuf->attach,
> + DMA_BIDIRECTIONAL);
> + dma_resv_unlock(umem_dmabuf->attach->dmabuf->resv);
This doesn't look right, this lock has to be held up until the HW is
prorgammed
The use of atomic looks probably wrong as well.
> + k = 0;
> + total_pages = ib_umem_odp_num_pages(umem_odp);
> + for_each_sg(umem->sg_head.sgl, sg, umem->sg_head.nents, j) {
> + addr = sg_dma_address(sg);
> + pages = sg_dma_len(sg) >> page_shift;
> + while (pages > 0 && k < total_pages) {
> + umem_odp->dma_list[k++] = addr | access_mask;
> + umem_odp->npages++;
> + addr += page_size;
> + pages--;
This isn't fragmenting the sg into a page list properly, won't work
for unaligned things
And really we don't need the dma_list for this case, with a fixed
whole mapping DMA SGL a normal umem sgl is OK and the normal umem XLT
programming in mlx5 is fine.
Jason
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
next prev parent reply other threads:[~2020-10-06 7:32 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-04 19:12 [RFC PATCH v3 0/4] RDMA: Add dma-buf support Jianxin Xiong
2020-10-04 19:12 ` [RFC PATCH v3 1/4] RDMA/umem: Support importing dma-buf as user memory region Jianxin Xiong
2020-10-05 10:54 ` Christian König
2020-10-05 16:19 ` Xiong, Jianxin
2020-10-05 13:13 ` Jason Gunthorpe [this message]
2020-10-05 16:18 ` Xiong, Jianxin
2020-10-05 16:33 ` Jason Gunthorpe
2020-10-05 19:41 ` Xiong, Jianxin
2020-10-06 9:22 ` Daniel Vetter
2020-10-06 15:26 ` Xiong, Jianxin
2020-10-06 15:49 ` Jason Gunthorpe
2020-10-06 16:34 ` Daniel Vetter
2020-10-06 17:24 ` Daniel Vetter
2020-10-06 18:02 ` Jason Gunthorpe
2020-10-06 18:17 ` Daniel Vetter
2020-10-06 18:38 ` Jason Gunthorpe
2020-10-06 19:12 ` Daniel Vetter
2020-10-07 7:13 ` Christian König
2020-10-06 16:40 ` Daniel Vetter
2020-10-04 19:12 ` [RFC PATCH v3 2/4] RDMA: Expand driver memory registration methods to support dma-buf Jianxin Xiong
2020-10-04 19:12 ` [RFC PATCH v3 3/4] RDMA/mlx5: Support dma-buf based userspace memory region Jianxin Xiong
2020-10-04 19:12 ` [RFC PATCH v3 4/4] RDMA/uverbs: Add uverbs command for dma-buf based MR registration Jianxin Xiong
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201005131302.GQ9916@ziepe.ca \
--to=jgg@ziepe.ca \
--cc=christian.koenig@amd.com \
--cc=daniel.vetter@intel.com \
--cc=dledford@redhat.com \
--cc=dri-devel@lists.freedesktop.org \
--cc=jianxin.xiong@intel.com \
--cc=leon@kernel.org \
--cc=linux-rdma@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).