dri-devel Archive on lore.kernel.org
 help / color / Atom feed
From: "Christian König" <christian.koenig@amd.com>
To: Daniel Vetter <daniel@ffwll.ch>
Cc: Leon Romanovsky <leon@kernel.org>,
	"linux-rdma@vger.kernel.org" <linux-rdma@vger.kernel.org>,
	<dri-devel@lists.freedesktop.org>, Jason Gunthorpe <jgg@ziepe.ca>,
	Doug Ledford <dledford@redhat.com>,
	"Vetter, Daniel" <daniel.vetter@intel.com>,
	"Xiong, Jianxin" <jianxin.xiong@intel.com>
Subject: Re: [RFC PATCH v2 0/3] RDMA: add dma-buf support
Date: Wed, 8 Jul 2020 16:20:19 +0200
Message-ID: <14659513-8164-dcae-e4f9-f0a199aee542@amd.com> (raw)
In-Reply-To: <20200708094934.GI3278063@phenom.ffwll.local>

Am 08.07.20 um 11:49 schrieb Daniel Vetter:
> On Wed, Jul 08, 2020 at 11:38:31AM +0200, Christian König wrote:
>> Am 07.07.20 um 23:58 schrieb Xiong, Jianxin:
>>>> -----Original Message-----
>>>> From: Christian König <christian.koenig@amd.com>
>>>> Am 03.07.20 um 15:14 schrieb Jason Gunthorpe:
>>>>> On Fri, Jul 03, 2020 at 02:52:03PM +0200, Daniel Vetter wrote:
>>>>>> So maybe I'm just totally confused about the rdma model. I thought:
>>>>>> - you bind a pile of memory for various transactions, that might
>>>>>> happen whenever. Kernel driver doesn't have much if any insight into
>>>>>> when memory isn't needed anymore. I think in the rdma world that's
>>>>>> called registering memory, but not sure.
>>>>> Sure, but once registered the memory is able to be used at any moment
>>>>> with no visibilty from the kernel.
>>>>> Unlike GPU the transactions that trigger memory access do not go
>>>>> through the kernel - so there is no ability to interrupt a command
>>>>> flow and fiddle with mappings.
>>>> This is the same for GPUs with user space queues as well.
>>>> But we can still say for a process if that this process is using a DMA-buf which is moved out and so can't run any more unless the DMA-buf is
>>>> accessible again.
>>>> In other words you somehow need to make sure that the hardware is not accessing a piece of memory any more when you want to move it.
>>> While a process can be easily suspended, there is no way to tell the RDMA NIC not to process posted work requests that use specific memory regions (or with any other conditions).
>>> So far it appears to me that DMA-buf dynamic mapping for RDMA is only viable with ODP support. For NICs without ODP, a way to allow pinning the device memory is still needed.
>> And that's exactly the reason why I introduced explicit pin()/unpin()
>> functions into the DMA-buf API:
>> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Felixir.bootlin.com%2Flinux%2Flatest%2Fsource%2Fdrivers%2Fdma-buf%2Fdma-buf.c%23L811&amp;data=02%7C01%7Cchristian.koenig%40amd.com%7C6d785861acc542a2f53608d823243a7c%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637297985792135311&amp;sdata=bBrkDynlACE9DAIlGntxXhE1unr%2FBxw5IRTm6AtV6WQ%3D&amp;reserved=0
>> It's just that at least our devices drivers currently prevent P2P with
>> pinned DMA-buf's for two main reasons:
>> a) To prevent deny of service attacks because P2P BARs are a rather rare
>> resource.
>> b) To prevent failures in configuration where P2P is not always possible
>> between all devices which want to access a buffer.
> So the above is more or less the question in the cover letter (which
> didn't make it to dri-devel). Can we somehow throw that limitation out, or
> is that simply not a good idea?

At least for the AMD graphics drivers that's most certain not a good idea.

We do have an use case where buffers need to be in system memory because 
P2P doesn't work.

And by pinning them to VRAM you can create a really nice deny of service 
attack against the X system.

> Simply moving buffers to system memory when they're pinned does simplify a
> lot of headaches. For a specific custom built system we can avoid that
> maybe, but I think upstream is kinda a different thing.

Yes, agree completely on that. Customers which are willing to take the 
risk can easily do this themselves.

But that is not something we should probably do for upstream.


> Cheers, Daniel
>> Regards,
>> Christian.
>>> Jianxin
>>>> Christian.
>>>>> Jason

dri-devel mailing list

  reply index

Thread overview: 28+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <1593451903-30959-1-git-send-email-jianxin.xiong@intel.com>
     [not found] ` <20200629185152.GD25301@ziepe.ca>
     [not found]   ` <MW3PR11MB4555A99038FA0CFC3ED80D3DE56F0@MW3PR11MB4555.namprd11.prod.outlook.com>
     [not found]     ` <20200630173435.GK25301@ziepe.ca>
2020-06-30 18:46       ` Xiong, Jianxin
2020-06-30 19:17         ` Jason Gunthorpe
2020-06-30 20:08           ` Xiong, Jianxin
2020-07-02 12:27             ` Jason Gunthorpe
2020-07-01  9:03         ` Christian König
2020-07-01 12:07           ` Daniel Vetter
2020-07-01 12:14             ` Daniel Vetter
2020-07-01 12:39           ` Jason Gunthorpe
2020-07-01 12:55             ` Christian König
2020-07-01 15:42               ` Daniel Vetter
2020-07-01 17:15                 ` Jason Gunthorpe
2020-07-02 13:10                   ` Daniel Vetter
2020-07-02 13:29                     ` Jason Gunthorpe
2020-07-02 14:50                       ` Christian König
2020-07-02 18:15                         ` Daniel Vetter
2020-07-03 12:03                           ` Jason Gunthorpe
2020-07-03 12:52                             ` Daniel Vetter
2020-07-03 13:14                               ` Jason Gunthorpe
2020-07-03 13:21                                 ` Christian König
2020-07-07 21:58                                   ` Xiong, Jianxin
2020-07-08  9:38                                     ` Christian König
2020-07-08  9:49                                       ` Daniel Vetter
2020-07-08 14:20                                         ` Christian König [this message]
2020-07-08 14:33                                           ` Alex Deucher
2020-06-30 18:56 ` Xiong, Jianxin
     [not found] ` <1593451903-30959-2-git-send-email-jianxin.xiong@intel.com>
2020-06-30 19:04   ` [RFC PATCH v2 1/3] RDMA/umem: Support importing dma-buf as user memory region Xiong, Jianxin
     [not found] ` <1593451903-30959-3-git-send-email-jianxin.xiong@intel.com>
2020-06-30 19:04   ` [RFC PATCH v2 2/3] RDMA/core: Expand the driver method 'reg_user_mr' to support dma-buf Xiong, Jianxin
     [not found] ` <1593451903-30959-4-git-send-email-jianxin.xiong@intel.com>
2020-06-30 19:05   ` [RFC PATCH v2 3/3] RDMA/uverbs: Add uverbs command for dma-buf based MR registration Xiong, Jianxin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=14659513-8164-dcae-e4f9-f0a199aee542@amd.com \
    --to=christian.koenig@amd.com \
    --cc=daniel.vetter@intel.com \
    --cc=daniel@ffwll.ch \
    --cc=dledford@redhat.com \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=jgg@ziepe.ca \
    --cc=jianxin.xiong@intel.com \
    --cc=leon@kernel.org \
    --cc=linux-rdma@vger.kernel.org \


* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

dri-devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/dri-devel/0 dri-devel/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 dri-devel dri-devel/ https://lore.kernel.org/dri-devel \
	public-inbox-index dri-devel

Example config snippet for mirrors

Newsgroup available over NNTP:

AGPL code for this site: git clone https://public-inbox.org/public-inbox.git