From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.7 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 370FAC4320A for ; Wed, 1 Sep 2021 11:20:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 20A4161053 for ; Wed, 1 Sep 2021 11:20:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241833AbhIALV3 (ORCPT ); Wed, 1 Sep 2021 07:21:29 -0400 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:61197 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234678AbhIALV2 (ORCPT ); Wed, 1 Sep 2021 07:21:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1630495232; x=1662031232; h=subject:to:cc:references:from:message-id:date: mime-version:in-reply-to:content-transfer-encoding; bh=9GsEs5OGmH4RMxW3IdDWH0nI6ePaUnPFoWEE+mXeB7M=; b=s5R/L7K4AInholvS5Y0Taw8bHu/oroA4xB7F5dQyBpVnDHVBI8JOsM48 xx77sPq+T86e3HqfbcmXN82juedb89t+gH5X04Hcma6msQ2rW99c4YeVx 2JbQEzcdVurPGJ7excQjXpEFcL0F9+6s9ra0oiCiFHY3jdVV5soepd4On U=; X-IronPort-AV: E=Sophos;i="5.84,369,1620691200"; d="scan'208";a="136822473" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-1d-546beb46.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-6002.iad6.amazon.com with ESMTP; 01 Sep 2021 11:20:25 +0000 Received: from EX13D19EUB003.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-1d-546beb46.us-east-1.amazon.com (Postfix) with ESMTPS id 0D4F4A0962; Wed, 1 Sep 2021 11:20:19 +0000 (UTC) Received: from 8c85908914bf.ant.amazon.com (10.43.162.52) by EX13D19EUB003.ant.amazon.com (10.43.166.69) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Wed, 1 Sep 2021 11:20:11 +0000 Subject: Re: [RFC] Make use of non-dynamic dmabuf in RDMA To: Jason Gunthorpe , John Hubbard CC: =?UTF-8?Q?Christian_K=c3=b6nig?= , Daniel Vetter , Sumit Semwal , Doug Ledford , "open list:DMA BUFFER SHARING FRAMEWORK" , dri-devel , Linux Kernel Mailing List , linux-rdma , Oded Gabbay , Tomer Tayar , Yossi Leybovich , Alexander Matushevsky , Leon Romanovsky , Jianxin Xiong References: <20210819230602.GU543798@ziepe.ca> <20210820123316.GV543798@ziepe.ca> <0fc94ac0-2bb9-4835-62b8-ea14f85fe512@amazon.com> <20210820143248.GX543798@ziepe.ca> <6b819064-feda-b70b-ea69-eb0a4fca6c0c@amd.com> <20210824173228.GE543798@ziepe.ca> From: Gal Pressman Message-ID: Date: Wed, 1 Sep 2021 14:20:02 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: <20210824173228.GE543798@ziepe.ca> Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: 8bit X-Originating-IP: [10.43.162.52] X-ClientProxiedBy: EX13D27UWB003.ant.amazon.com (10.43.161.195) To EX13D19EUB003.ant.amazon.com (10.43.166.69) Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On 24/08/2021 20:32, Jason Gunthorpe wrote: > On Tue, Aug 24, 2021 at 10:27:23AM -0700, John Hubbard wrote: >> On 8/24/21 2:32 AM, Christian König wrote: >>> Am 24.08.21 um 11:06 schrieb Gal Pressman: >>>> On 23/08/2021 13:43, Christian König wrote: >>>>> Am 21.08.21 um 11:16 schrieb Gal Pressman: >>>>>> On 20/08/2021 17:32, Jason Gunthorpe wrote: >>>>>>> On Fri, Aug 20, 2021 at 03:58:33PM +0300, Gal Pressman wrote: >> ... >>>>>> IIUC, we're talking about three different exporter "types": >>>>>> - Dynamic with move_notify (requires ODP) >>>>>> - Dynamic with revoke_notify >>>>>> - Static >>>>>> >>>>>> Which changes do we need to make the third one work? >>>>> Basically none at all in the framework. >>>>> >>>>> You just need to properly use the dma_buf_pin() function when you start using a >>>>> buffer (e.g. before you create an attachment) and the dma_buf_unpin() function >>>>> after you are done with the DMA-buf. >>>> I replied to your previous mail, but I'll ask again. >>>> Doesn't the pin operation migrate the memory to host memory? >>> >>> Sorry missed your previous reply. >>> >>> And yes at least for the amdgpu driver we migrate the memory to host >>> memory as soon as it is pinned and I would expect that other GPU drivers >>> do something similar. >> >> Well...for many topologies, migrating to host memory will result in a >> dramatically slower p2p setup. For that reason, some GPU drivers may >> want to allow pinning of video memory in some situations. >> >> Ideally, you've got modern ODP devices and you don't even need to pin. >> But if not, and you still hope to do high performance p2p between a GPU >> and a non-ODP Infiniband device, then you would need to leave the pinned >> memory in vidmem. >> >> So I think we don't want to rule out that behavior, right? Or is the >> thinking more like, "you're lucky that this old non-ODP setup works at >> all, and we'll make it work by routing through host/cpu memory, but it >> will be slow"? > > I think it depends on the user, if the user creates memory which is > permanently located on the GPU then it should be pinnable in this way > without force migration. But if the memory is inherently migratable > then it just cannot be pinned in the GPU at all as we can't > indefinately block migration from happening eg if the CPU touches it > later or something. So are we OK with exporters implementing dma_buf_pin() without migrating the memory? If so, do we still want a move_notify callback for non-dynamic importers? A noop?