From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45D23C432BE for ; Thu, 2 Sep 2021 06:57:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 12B2A6101A for ; Thu, 2 Sep 2021 06:57:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242458AbhIBG6O (ORCPT ); Thu, 2 Sep 2021 02:58:14 -0400 Received: from smtp-fw-80007.amazon.com ([99.78.197.218]:35200 "EHLO smtp-fw-80007.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242135AbhIBG6O (ORCPT ); Thu, 2 Sep 2021 02:58:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1630565837; x=1662101837; h=subject:to:cc:references:from:message-id:date: mime-version:in-reply-to:content-transfer-encoding; bh=XBXHWrVSJO9nDb98gdSK+8+MX1E89cWhhkrCj5FfjoM=; b=sqbXDH4bk6CjE89I/6roygexM24wzPPTXoszYX76lAknjnBWhgZNR5GX S+EAQxTAwM87mymzQaEonParZr6aktdu2AixZ8JRGWEkLeGOYPn4kSWnE 9HQkNQa9HbYQ0FHdFhHGP69Pi+Fw1Eev+AokQcNVimCTPt8WTKgOSeuhe I=; X-IronPort-AV: E=Sophos;i="5.84,371,1620691200"; d="scan'208";a="23868928" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO email-inbound-relay-1e-42f764a0.us-east-1.amazon.com) ([10.25.36.210]) by smtp-border-fw-80007.pdx80.corp.amazon.com with ESMTP; 02 Sep 2021 06:57:09 +0000 Received: from EX13D19EUB003.ant.amazon.com (iad12-ws-svc-p26-lb9-vlan3.iad.amazon.com [10.40.163.38]) by email-inbound-relay-1e-42f764a0.us-east-1.amazon.com (Postfix) with ESMTPS id 68570C00DC; Thu, 2 Sep 2021 06:57:04 +0000 (UTC) Received: from 8c85908914bf.ant.amazon.com (10.43.162.216) by EX13D19EUB003.ant.amazon.com (10.43.166.69) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Thu, 2 Sep 2021 06:56:55 +0000 Subject: Re: [RFC] Make use of non-dynamic dmabuf in RDMA To: =?UTF-8?Q?Christian_K=c3=b6nig?= , Jason Gunthorpe , John Hubbard CC: Daniel Vetter , Sumit Semwal , Doug Ledford , "open list:DMA BUFFER SHARING FRAMEWORK" , dri-devel , Linux Kernel Mailing List , linux-rdma , Oded Gabbay , Tomer Tayar , Yossi Leybovich , Alexander Matushevsky , Leon Romanovsky , Jianxin Xiong References: <20210819230602.GU543798@ziepe.ca> <20210820123316.GV543798@ziepe.ca> <0fc94ac0-2bb9-4835-62b8-ea14f85fe512@amazon.com> <20210820143248.GX543798@ziepe.ca> <6b819064-feda-b70b-ea69-eb0a4fca6c0c@amd.com> <20210824173228.GE543798@ziepe.ca> <98463545-c27a-77e6-0a5c-a658743ce86e@amd.com> From: Gal Pressman Message-ID: Date: Thu, 2 Sep 2021 09:56:50 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: <98463545-c27a-77e6-0a5c-a658743ce86e@amd.com> Content-Type: text/plain; charset="utf-8" Content-Language: en-US Content-Transfer-Encoding: 8bit X-Originating-IP: [10.43.162.216] X-ClientProxiedBy: EX13D32UWA001.ant.amazon.com (10.43.160.4) To EX13D19EUB003.ant.amazon.com (10.43.166.69) Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On 01/09/2021 14:24, Christian König wrote: > > > Am 01.09.21 um 13:20 schrieb Gal Pressman: >> On 24/08/2021 20:32, Jason Gunthorpe wrote: >>> On Tue, Aug 24, 2021 at 10:27:23AM -0700, John Hubbard wrote: >>>> On 8/24/21 2:32 AM, Christian König wrote: >>>>> Am 24.08.21 um 11:06 schrieb Gal Pressman: >>>>>> On 23/08/2021 13:43, Christian König wrote: >>>>>>> Am 21.08.21 um 11:16 schrieb Gal Pressman: >>>>>>>> On 20/08/2021 17:32, Jason Gunthorpe wrote: >>>>>>>>> On Fri, Aug 20, 2021 at 03:58:33PM +0300, Gal Pressman wrote: >>>> ... >>>>>>>> IIUC, we're talking about three different exporter "types": >>>>>>>> - Dynamic with move_notify (requires ODP) >>>>>>>> - Dynamic with revoke_notify >>>>>>>> - Static >>>>>>>> >>>>>>>> Which changes do we need to make the third one work? >>>>>>> Basically none at all in the framework. >>>>>>> >>>>>>> You just need to properly use the dma_buf_pin() function when you start >>>>>>> using a >>>>>>> buffer (e.g. before you create an attachment) and the dma_buf_unpin() >>>>>>> function >>>>>>> after you are done with the DMA-buf. >>>>>> I replied to your previous mail, but I'll ask again. >>>>>> Doesn't the pin operation migrate the memory to host memory? >>>>> Sorry missed your previous reply. >>>>> >>>>> And yes at least for the amdgpu driver we migrate the memory to host >>>>> memory as soon as it is pinned and I would expect that other GPU drivers >>>>> do something similar. >>>> Well...for many topologies, migrating to host memory will result in a >>>> dramatically slower p2p setup. For that reason, some GPU drivers may >>>> want to allow pinning of video memory in some situations. >>>> >>>> Ideally, you've got modern ODP devices and you don't even need to pin. >>>> But if not, and you still hope to do high performance p2p between a GPU >>>> and a non-ODP Infiniband device, then you would need to leave the pinned >>>> memory in vidmem. >>>> >>>> So I think we don't want to rule out that behavior, right? Or is the >>>> thinking more like, "you're lucky that this old non-ODP setup works at >>>> all, and we'll make it work by routing through host/cpu memory, but it >>>> will be slow"? >>> I think it depends on the user, if the user creates memory which is >>> permanently located on the GPU then it should be pinnable in this way >>> without force migration. But if the memory is inherently migratable >>> then it just cannot be pinned in the GPU at all as we can't >>> indefinately block migration from happening eg if the CPU touches it >>> later or something. >> So are we OK with exporters implementing dma_buf_pin() without migrating the >> memory? > > I think so, yes. > >> If so, do we still want a move_notify callback for non-dynamic importers? A noop? > > Well we could make the move_notify callback optional, e.g. so that you get the > new locking approach but still pin the buffers manually with dma_buf_pin(). Thanks Christian! So the end result will look similar to the original patch I posted, where peer2peer can be enabled without providing move_notify, correct?