From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDE53C48BC2 for ; Thu, 24 Jun 2021 00:45:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A759A6135C for ; Thu, 24 Jun 2021 00:45:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229796AbhFXArc (ORCPT ); Wed, 23 Jun 2021 20:47:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229755AbhFXArc (ORCPT ); Wed, 23 Jun 2021 20:47:32 -0400 Received: from mail-qk1-x729.google.com (mail-qk1-x729.google.com [IPv6:2607:f8b0:4864:20::729]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D599C06175F for ; Wed, 23 Jun 2021 17:45:13 -0700 (PDT) Received: by mail-qk1-x729.google.com with SMTP id y29so8534236qky.12 for ; Wed, 23 Jun 2021 17:45:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=EFcFEiW34m53hHVRH/hYAOinlbr3mbVvwuYYaYtTuh8=; b=G1KDqzwpGU22cmRZBgDHPWa87kbMIhmgX8Prq4EL0fmRPLBrFBeajfqHskFpu8RKTX yPnWNNhZxtgtf1LNFARLTco5JxjiV2oulGE5y/6osdL4RwXiUVMTAG5ut6nrvgtCxcyW WNdQQ1Be+uGC+xnXPFnIEB+CnxBIJjQavpHiEs+mnCS/UyCcFsHD23X3fg/71TBLMoTI hf7TgXvMgDwaZel4MR99dGUXP8aAPP1F5KANaExTx3B1Gzybwmv9QfKVsJCcbxExhAHC HBloe8ZRqPH9mLrrauCiklIS4IB5euV3NwN6tbG+Ga5k1mdv/+36MmqpccO7p5tm0V0m ZRWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=EFcFEiW34m53hHVRH/hYAOinlbr3mbVvwuYYaYtTuh8=; b=OC1V1DhKvvbJF0ty5ESE/w4bIpPV18rLn1O0U/rTfzi3kZbcwNYCODB+UWqrzFcrm/ ZwXimdnWUiOe+P2lrnaVOndZlCC+xnGqIUK8Z5X7TWPcQ2+MLqWcITrOT9Jbu5Eq+I2d 2sxef8+R36ncxgIZNCYN+25S8O+pFWFM6BSx1THMqEf8hNmDtkv6SC0d2CBLKqg7DNEA d+3bRHhIfE5W+7pMHu6zRApR5n4l19gRcLbKquEaO0AYNOUYDyc5orxcPedPb8ERWIeA EhfD3NqCqTCdmqalQJUYpjyJHmY+zv+DWKUbv10y1sqNbWnJPzw2I7UoD5cteW85yab+ s0uQ== X-Gm-Message-State: AOAM532fwn++d24782fk20nmnX92QVX1F0Wglz/4GGiTr1NpvNlDDXrV jIUFx48qWnAVHgDcrAxoJXwYlQ== X-Google-Smtp-Source: ABdhPJyllJgYeDzAla5FLLIkN9PiWxiijQsV7UXnwLjZeIyw9AzKaK9eZnxQ5B0p88Zubn0YY0PqUA== X-Received: by 2002:a37:b205:: with SMTP id b5mr3007100qkf.208.1624495512436; Wed, 23 Jun 2021 17:45:12 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-47-55-113-94.dhcp-dynamic.fibreop.ns.bellaliant.net. [47.55.113.94]) by smtp.gmail.com with ESMTPSA id n207sm1169771qka.101.2021.06.23.17.45.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Jun 2021 17:45:11 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1lwDUV-00Bris-8h; Wed, 23 Jun 2021 21:45:11 -0300 Date: Wed, 23 Jun 2021 21:45:11 -0300 From: Jason Gunthorpe To: Oded Gabbay Cc: Christian =?utf-8?B?S8O2bmln?= , Christian =?utf-8?B?S8O2bmln?= , Gal Pressman , sleybo@amazon.com, linux-rdma , Oded Gabbay , Christoph Hellwig , Linux Kernel Mailing List , dri-devel , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Greg KH , Alex Deucher , Leon Romanovsky , "open list:DMA BUFFER SHARING FRAMEWORK" Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF Message-ID: <20210624004511.GA1096940@ziepe.ca> References: <20210622154027.GS1096940@ziepe.ca> <09df4a03-d99c-3949-05b2-8b49c71a109e@amd.com> <20210622160538.GT1096940@ziepe.ca> <20210623182435.GX1096940@ziepe.ca> <20210623185045.GY1096940@ziepe.ca> <20210623193456.GZ1096940@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Wed, Jun 23, 2021 at 10:39:48PM +0300, Oded Gabbay wrote: > On Wed, Jun 23, 2021 at 10:34 PM Jason Gunthorpe wrote: > > > > On Wed, Jun 23, 2021 at 10:00:29PM +0300, Oded Gabbay wrote: > > > On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe wrote: > > > > > > > > On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > > > > > > > > > Can you please explain why it is so important to (allow) access them > > > > > through the CPU ? > > > > > > > > It is not so much important, as it reflects significant design choices > > > > that are already tightly baked into alot of our stacks. > > > > > > > > A SGL is CPU accessible by design - that is baked into this thing and > > > > places all over the place assume it. Even in RDMA we have > > > > RXE/SWI/HFI1/qib that might want to use the CPU side (grep for sg_page > > > > to see) > > > > > > > > So, the thing at the top of the stack - in this case the gaudi driver > > > > - simply can't assume what the rest of the stack is going to do and > > > > omit the CPU side. It breaks everything. > > > > > > > > Logan's patch series is the most fully developed way out of this > > > > predicament so far. > > > > > > I understand the argument and I agree that for the generic case, the > > > top of the stack can't assume anything. > > > Having said that, in this case the SGL is encapsulated inside a dma-buf object. > > > > > > Maybe its a stupid/over-simplified suggestion, but can't we add a > > > property to the dma-buf object, > > > that will be set by the exporter, which will "tell" the importer it > > > can't use any CPU fallback ? Only "real" p2p ? > > > > The block stack has been trying to do something like this. > > > > The flag doesn't solve the DMA API/IOMMU problems though. > hmm, I thought using dma_map_resource will solve the IOMMU issues, > no ? dma_map_resource() will configure the IOMMU but it is not the correct API to use when building a SG list for DMA, that would be dma_map_sg or sgtable. So it works, but it is an API abuse to build things this way. > If I use dma_map_resource to set the addresses inside the SGL before I > export the dma-buf, and guarantee no one will use the SGL in the > dma-buf for any other purpose than device p2p, what else is needed ? You still have to check the p2p stuff to ensure that p2p is even possible And this approach is misusing all the APIs and has been NAK'd by Christoph, so up to Greg if he wants to take it or insist you work with Logan to get the proper generlized solution finished. Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA20CC48BC2 for ; Thu, 24 Jun 2021 00:45:14 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6568361358 for ; Thu, 24 Jun 2021 00:45:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6568361358 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D0ABC6E99E; Thu, 24 Jun 2021 00:45:13 +0000 (UTC) Received: from mail-qk1-x72a.google.com (mail-qk1-x72a.google.com [IPv6:2607:f8b0:4864:20::72a]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4DF566E9A0 for ; Thu, 24 Jun 2021 00:45:13 +0000 (UTC) Received: by mail-qk1-x72a.google.com with SMTP id bm25so10239313qkb.0 for ; Wed, 23 Jun 2021 17:45:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=EFcFEiW34m53hHVRH/hYAOinlbr3mbVvwuYYaYtTuh8=; b=G1KDqzwpGU22cmRZBgDHPWa87kbMIhmgX8Prq4EL0fmRPLBrFBeajfqHskFpu8RKTX yPnWNNhZxtgtf1LNFARLTco5JxjiV2oulGE5y/6osdL4RwXiUVMTAG5ut6nrvgtCxcyW WNdQQ1Be+uGC+xnXPFnIEB+CnxBIJjQavpHiEs+mnCS/UyCcFsHD23X3fg/71TBLMoTI hf7TgXvMgDwaZel4MR99dGUXP8aAPP1F5KANaExTx3B1Gzybwmv9QfKVsJCcbxExhAHC HBloe8ZRqPH9mLrrauCiklIS4IB5euV3NwN6tbG+Ga5k1mdv/+36MmqpccO7p5tm0V0m ZRWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=EFcFEiW34m53hHVRH/hYAOinlbr3mbVvwuYYaYtTuh8=; b=F56rxjeB01q6kVdVzYqTI94fVMvTET166Spv804KggaR+9GIUW8ZSMRBLLuJ+3AI3h bjq7lNbRnXCHoglzT51Gy3XG59Uq2b1Jy62tvNguN/Pi38sc8OKgAkxMHEPjaCL1jljI f++IXzlBhNo3hdyypHRZSwHR1meNufdB1d3+gTnnrGMp2WS0wGH7OPO84cmVmaJZSzd5 QeuSIgQ27/9LTniLjvTAdcP1C4/v5F19TffJIuTOommqNu0xq3+BUt8P+2Q/Ml7wPQtV 1r84ZE+ASOe4Nh68tXZh7jZBNwg46O5YeSn0jgY3iXBRuVfRbVIb1AvVE2JUhV28YBkX QvUg== X-Gm-Message-State: AOAM531G86bLOgiFzpQ5U1OUTcXfxoIMfy5PbMhc42xOeg5KRjFVXNf6 vrh9dL6hbTd5b1FdxQqXArN+6A== X-Google-Smtp-Source: ABdhPJyllJgYeDzAla5FLLIkN9PiWxiijQsV7UXnwLjZeIyw9AzKaK9eZnxQ5B0p88Zubn0YY0PqUA== X-Received: by 2002:a37:b205:: with SMTP id b5mr3007100qkf.208.1624495512436; Wed, 23 Jun 2021 17:45:12 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-47-55-113-94.dhcp-dynamic.fibreop.ns.bellaliant.net. [47.55.113.94]) by smtp.gmail.com with ESMTPSA id n207sm1169771qka.101.2021.06.23.17.45.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Jun 2021 17:45:11 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1lwDUV-00Bris-8h; Wed, 23 Jun 2021 21:45:11 -0300 Date: Wed, 23 Jun 2021 21:45:11 -0300 From: Jason Gunthorpe To: Oded Gabbay Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF Message-ID: <20210624004511.GA1096940@ziepe.ca> References: <20210622154027.GS1096940@ziepe.ca> <09df4a03-d99c-3949-05b2-8b49c71a109e@amd.com> <20210622160538.GT1096940@ziepe.ca> <20210623182435.GX1096940@ziepe.ca> <20210623185045.GY1096940@ziepe.ca> <20210623193456.GZ1096940@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-rdma , Christian =?utf-8?B?S8O2bmln?= , sleybo@amazon.com, Leon Romanovsky , Gal Pressman , dri-devel , Christian =?utf-8?B?S8O2bmln?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Greg KH , Alex Deucher , Christoph Hellwig , Oded Gabbay , Linux Kernel Mailing List , "open list:DMA BUFFER SHARING FRAMEWORK" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Wed, Jun 23, 2021 at 10:39:48PM +0300, Oded Gabbay wrote: > On Wed, Jun 23, 2021 at 10:34 PM Jason Gunthorpe wrote: > > > > On Wed, Jun 23, 2021 at 10:00:29PM +0300, Oded Gabbay wrote: > > > On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe wrote: > > > > > > > > On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > > > > > > > > > Can you please explain why it is so important to (allow) access them > > > > > through the CPU ? > > > > > > > > It is not so much important, as it reflects significant design choices > > > > that are already tightly baked into alot of our stacks. > > > > > > > > A SGL is CPU accessible by design - that is baked into this thing and > > > > places all over the place assume it. Even in RDMA we have > > > > RXE/SWI/HFI1/qib that might want to use the CPU side (grep for sg_page > > > > to see) > > > > > > > > So, the thing at the top of the stack - in this case the gaudi driver > > > > - simply can't assume what the rest of the stack is going to do and > > > > omit the CPU side. It breaks everything. > > > > > > > > Logan's patch series is the most fully developed way out of this > > > > predicament so far. > > > > > > I understand the argument and I agree that for the generic case, the > > > top of the stack can't assume anything. > > > Having said that, in this case the SGL is encapsulated inside a dma-buf object. > > > > > > Maybe its a stupid/over-simplified suggestion, but can't we add a > > > property to the dma-buf object, > > > that will be set by the exporter, which will "tell" the importer it > > > can't use any CPU fallback ? Only "real" p2p ? > > > > The block stack has been trying to do something like this. > > > > The flag doesn't solve the DMA API/IOMMU problems though. > hmm, I thought using dma_map_resource will solve the IOMMU issues, > no ? dma_map_resource() will configure the IOMMU but it is not the correct API to use when building a SG list for DMA, that would be dma_map_sg or sgtable. So it works, but it is an API abuse to build things this way. > If I use dma_map_resource to set the addresses inside the SGL before I > export the dma-buf, and guarantee no one will use the SGL in the > dma-buf for any other purpose than device p2p, what else is needed ? You still have to check the p2p stuff to ensure that p2p is even possible And this approach is misusing all the APIs and has been NAK'd by Christoph, so up to Greg if he wants to take it or insist you work with Logan to get the proper generlized solution finished. Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 986FBC49EA5 for ; Thu, 24 Jun 2021 07:06:30 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6973B613D3 for ; Thu, 24 Jun 2021 07:06:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6973B613D3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=ziepe.ca Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=amd-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 256BF6EA1C; Thu, 24 Jun 2021 07:06:30 +0000 (UTC) Received: from mail-qk1-x733.google.com (mail-qk1-x733.google.com [IPv6:2607:f8b0:4864:20::733]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4D31B6E99E for ; Thu, 24 Jun 2021 00:45:13 +0000 (UTC) Received: by mail-qk1-x733.google.com with SMTP id o6so10113018qkh.4 for ; Wed, 23 Jun 2021 17:45:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=EFcFEiW34m53hHVRH/hYAOinlbr3mbVvwuYYaYtTuh8=; b=G1KDqzwpGU22cmRZBgDHPWa87kbMIhmgX8Prq4EL0fmRPLBrFBeajfqHskFpu8RKTX yPnWNNhZxtgtf1LNFARLTco5JxjiV2oulGE5y/6osdL4RwXiUVMTAG5ut6nrvgtCxcyW WNdQQ1Be+uGC+xnXPFnIEB+CnxBIJjQavpHiEs+mnCS/UyCcFsHD23X3fg/71TBLMoTI hf7TgXvMgDwaZel4MR99dGUXP8aAPP1F5KANaExTx3B1Gzybwmv9QfKVsJCcbxExhAHC HBloe8ZRqPH9mLrrauCiklIS4IB5euV3NwN6tbG+Ga5k1mdv/+36MmqpccO7p5tm0V0m ZRWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=EFcFEiW34m53hHVRH/hYAOinlbr3mbVvwuYYaYtTuh8=; b=fNwnGQBDcn6SzIIFS9jemmmHdXX+CrGt9jHskcdnff+WgbIrVU1FRvXlhg3nbwEYVU aoB0Jhsx6d8MOp/WUQJ4QAk3cFTgx1YESEb3TXDv4a0/pixXEHvThD78qiZ6ExzC4pW0 24QeQDS1tSP0uXUs3oeZGjt51+ucN7M3LqbpzCuvT1bE9jwglH+FTnIVpLIaaMPv3Pqm CTIE0XjDmIxuvB76EadSly3qyzJ14Bn6puOQoJ7Rd2F5r0nIx8qTYXJYt1EwyVhc+DlS Hm++44meFvUjk0CQtwpQL7dNntARsidhdCFPnFsErW2M9o7x+0mYN29FtrMJBKrjFxtL Ps1w== X-Gm-Message-State: AOAM530dpeFbrNvCq7wRHuxOepNSN0FZDu6r3gSa6Axib2qqGzpDGjzA gUPyihxQuBWusVpMJBhMO0kWSA== X-Google-Smtp-Source: ABdhPJyllJgYeDzAla5FLLIkN9PiWxiijQsV7UXnwLjZeIyw9AzKaK9eZnxQ5B0p88Zubn0YY0PqUA== X-Received: by 2002:a37:b205:: with SMTP id b5mr3007100qkf.208.1624495512436; Wed, 23 Jun 2021 17:45:12 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-47-55-113-94.dhcp-dynamic.fibreop.ns.bellaliant.net. [47.55.113.94]) by smtp.gmail.com with ESMTPSA id n207sm1169771qka.101.2021.06.23.17.45.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Jun 2021 17:45:11 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1lwDUV-00Bris-8h; Wed, 23 Jun 2021 21:45:11 -0300 Date: Wed, 23 Jun 2021 21:45:11 -0300 From: Jason Gunthorpe To: Oded Gabbay Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF Message-ID: <20210624004511.GA1096940@ziepe.ca> References: <20210622154027.GS1096940@ziepe.ca> <09df4a03-d99c-3949-05b2-8b49c71a109e@amd.com> <20210622160538.GT1096940@ziepe.ca> <20210623182435.GX1096940@ziepe.ca> <20210623185045.GY1096940@ziepe.ca> <20210623193456.GZ1096940@ziepe.ca> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-Mailman-Approved-At: Thu, 24 Jun 2021 07:06:12 +0000 X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-rdma , Christian =?utf-8?B?S8O2bmln?= , sleybo@amazon.com, Leon Romanovsky , Gal Pressman , dri-devel , Christian =?utf-8?B?S8O2bmln?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Greg KH , Alex Deucher , Christoph Hellwig , Oded Gabbay , Linux Kernel Mailing List , "open list:DMA BUFFER SHARING FRAMEWORK" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" On Wed, Jun 23, 2021 at 10:39:48PM +0300, Oded Gabbay wrote: > On Wed, Jun 23, 2021 at 10:34 PM Jason Gunthorpe wrote: > > > > On Wed, Jun 23, 2021 at 10:00:29PM +0300, Oded Gabbay wrote: > > > On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe wrote: > > > > > > > > On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > > > > > > > > > Can you please explain why it is so important to (allow) access them > > > > > through the CPU ? > > > > > > > > It is not so much important, as it reflects significant design choices > > > > that are already tightly baked into alot of our stacks. > > > > > > > > A SGL is CPU accessible by design - that is baked into this thing and > > > > places all over the place assume it. Even in RDMA we have > > > > RXE/SWI/HFI1/qib that might want to use the CPU side (grep for sg_page > > > > to see) > > > > > > > > So, the thing at the top of the stack - in this case the gaudi driver > > > > - simply can't assume what the rest of the stack is going to do and > > > > omit the CPU side. It breaks everything. > > > > > > > > Logan's patch series is the most fully developed way out of this > > > > predicament so far. > > > > > > I understand the argument and I agree that for the generic case, the > > > top of the stack can't assume anything. > > > Having said that, in this case the SGL is encapsulated inside a dma-buf object. > > > > > > Maybe its a stupid/over-simplified suggestion, but can't we add a > > > property to the dma-buf object, > > > that will be set by the exporter, which will "tell" the importer it > > > can't use any CPU fallback ? Only "real" p2p ? > > > > The block stack has been trying to do something like this. > > > > The flag doesn't solve the DMA API/IOMMU problems though. > hmm, I thought using dma_map_resource will solve the IOMMU issues, > no ? dma_map_resource() will configure the IOMMU but it is not the correct API to use when building a SG list for DMA, that would be dma_map_sg or sgtable. So it works, but it is an API abuse to build things this way. > If I use dma_map_resource to set the addresses inside the SGL before I > export the dma-buf, and guarantee no one will use the SGL in the > dma-buf for any other purpose than device p2p, what else is needed ? You still have to check the p2p stuff to ensure that p2p is even possible And this approach is misusing all the APIs and has been NAK'd by Christoph, so up to Greg if he wants to take it or insist you work with Logan to get the proper generlized solution finished. Jason _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx