linux-rdma.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Daniel Vetter <daniel@ffwll.ch>
To: Oded Gabbay <ogabbay@kernel.org>
Cc: linux-kernel@vger.kernel.org, gregkh@linuxfoundation.org,
	jgg@ziepe.ca, christian.koenig@amd.com, daniel.vetter@ffwll.ch,
	galpress@amazon.com, sleybo@amazon.com,
	dri-devel@lists.freedesktop.org, linux-rdma@vger.kernel.org,
	linux-media@vger.kernel.org, dledford@redhat.com,
	airlied@gmail.com, alexander.deucher@amd.com, leonro@nvidia.com,
	hch@lst.de, amd-gfx@lists.freedesktop.org,
	linaro-mm-sig@lists.linaro.org
Subject: Re: [PATCH v6 0/2] Add p2p via dmabuf to habanalabs
Date: Tue, 14 Sep 2021 16:18:31 +0200	[thread overview]
Message-ID: <YUCvNzpyC091KeaJ@phenom.ffwll.local> (raw)
In-Reply-To: <20210912165309.98695-1-ogabbay@kernel.org>

On Sun, Sep 12, 2021 at 07:53:07PM +0300, Oded Gabbay wrote:
> Hi,
> Re-sending this patch-set following the release of our user-space TPC
> compiler and runtime library.
> 
> I would appreciate a review on this.

I think the big open we have is the entire revoke discussions. Having the
option to let dma-buf hang around which map to random local memory ranges,
without clear ownership link and a way to kill it sounds bad to me.

I think there's a few options:
- We require revoke support. But I've heard rdma really doesn't like that,
  I guess because taking out an MR while holding the dma_resv_lock would
  be an inversion, so can't be done. Jason, can you recap what exactly the
  hold-up was again that makes this a no-go?

- The other option I discussed is a bit more the exlusive device ownership
  model we've had for gpus in drm of the really old kind. Roughly this
  would work like this, in terms of drm_device:
  - Only the current owner (drm_master in current drm code, but should
    probably rename that to drm_owner) is allowed to use the accel driver.
    So all ioctl would fail if you're not drm_master.
  - On dropmaster/file close we'd revoke as much as possible, e.g.
    in-flight commands, mmaps, anything really that can be revoked.
  - For non-revokable things like these dma-buf we'd keep a drm_master
    reference around. This would prevent the next open to acquire
    ownership rights, which at least prevents all the nasty potential
    problems.
  - admin (or well container orchestrator) then has responsibility to
    shoot down all process until the problem goes away (i.e. until you hit
    the one with the rdma MR which keeps the dma-buf alive)

- Not sure there's another reasonable way to do this without inviting some
  problems once we get outside of the "single kernel instance per tenant"
  use-case.

Wrt implementation there's the trouble of this reinventing a bunch of drm
stuff and concepts, but that's maybe for after we've figured out
semantics.

Also would be great if you have a pull request for the userspace runtime
that shows a bit how this all gets used and tied together. Or maybe some
pointers, since I guess retconning a PR in github is maybe a bit much.

Cheers, Daniel

> 
> Thanks,
> Oded
> 
> Oded Gabbay (1):
>   habanalabs: define uAPI to export FD for DMA-BUF
> 
> Tomer Tayar (1):
>   habanalabs: add support for dma-buf exporter
> 
>  drivers/misc/habanalabs/Kconfig             |   1 +
>  drivers/misc/habanalabs/common/habanalabs.h |  22 +
>  drivers/misc/habanalabs/common/memory.c     | 522 +++++++++++++++++++-
>  drivers/misc/habanalabs/gaudi/gaudi.c       |   1 +
>  drivers/misc/habanalabs/goya/goya.c         |   1 +
>  include/uapi/misc/habanalabs.h              |  28 +-
>  6 files changed, 570 insertions(+), 5 deletions(-)
> 
> -- 
> 2.17.1
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

  parent reply	other threads:[~2021-09-14 14:18 UTC|newest]

Thread overview: 24+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-09-12 16:53 Oded Gabbay
2021-09-12 16:53 ` [PATCH v6 1/2] habanalabs: define uAPI to export FD for DMA-BUF Oded Gabbay
2021-09-28 17:13   ` Jason Gunthorpe
2021-09-28 19:12     ` Oded Gabbay
2021-09-12 16:53 ` [PATCH v6 2/2] habanalabs: add support for dma-buf exporter Oded Gabbay
2021-09-28 17:36   ` Jason Gunthorpe
2021-09-28 21:17     ` Oded Gabbay
2021-09-30 12:46       ` Oded Gabbay
2021-10-01 14:52         ` Jason Gunthorpe
2021-10-01 14:50       ` Jason Gunthorpe
2021-09-14 14:18 ` Daniel Vetter [this message]
2021-09-14 14:58   ` [PATCH v6 0/2] Add p2p via dmabuf to habanalabs Oded Gabbay
2021-09-14 16:12   ` Jason Gunthorpe
2021-09-15  7:45     ` Oded Gabbay
2021-09-16 12:31       ` Daniel Vetter
2021-09-16 12:44         ` Oded Gabbay
2021-09-16 13:16           ` Oded Gabbay
2021-09-17 12:25           ` Daniel Vetter
2021-09-16 13:10         ` Jason Gunthorpe
2021-09-17 12:30           ` Daniel Vetter
2021-09-18  8:38             ` Oded Gabbay
2021-09-23  9:22               ` Oded Gabbay
2021-09-28  7:04                 ` Oded Gabbay
2021-09-30  9:13                   ` Daniel Vetter

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=YUCvNzpyC091KeaJ@phenom.ffwll.local \
    --to=daniel@ffwll.ch \
    --cc=airlied@gmail.com \
    --cc=alexander.deucher@amd.com \
    --cc=amd-gfx@lists.freedesktop.org \
    --cc=christian.koenig@amd.com \
    --cc=daniel.vetter@ffwll.ch \
    --cc=dledford@redhat.com \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=galpress@amazon.com \
    --cc=gregkh@linuxfoundation.org \
    --cc=hch@lst.de \
    --cc=jgg@ziepe.ca \
    --cc=leonro@nvidia.com \
    --cc=linaro-mm-sig@lists.linaro.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-media@vger.kernel.org \
    --cc=linux-rdma@vger.kernel.org \
    --cc=ogabbay@kernel.org \
    --cc=sleybo@amazon.com \
    --subject='Re: [PATCH v6 0/2] Add p2p via dmabuf to habanalabs' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).