From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA3F5C4743C for ; Mon, 21 Jun 2021 16:55:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A78C660200 for ; Mon, 21 Jun 2021 16:55:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232743AbhFUQ5u (ORCPT ); Mon, 21 Jun 2021 12:57:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233499AbhFUQ4o (ORCPT ); Mon, 21 Jun 2021 12:56:44 -0400 Received: from mail-ot1-x329.google.com (mail-ot1-x329.google.com [IPv6:2607:f8b0:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8ABB5C09B05B; Mon, 21 Jun 2021 09:26:38 -0700 (PDT) Received: by mail-ot1-x329.google.com with SMTP id g19-20020a9d12930000b0290457fde18ad0so4417845otg.1; Mon, 21 Jun 2021 09:26:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=yoCwDUxZJP3Ut8z/hr4sZH2+48UBBB2zck9Vy5dLtgU=; b=pM+MoLSU0LSJh98DQrRsTPmTh+95pmfRHHbTmKcyhP9ovDKRhZcvBC02S/55ftG/og ruYrldynFgVUzXcJ5cpWd8IKQzedGdpH4I+1WfQpqdIufPWiWf72S0LDFysmflufYM4y 4wEreRKEOPdT1VnnTfvE5bFAmShhQkzdbAuUs7Mj25c7MYRwZ3OzmCVuPeHzDrwJTziw rDSCsQLsvjXiYpI4vGyZmIa5HkJhW6dIjnKxwkKEFk6v9qnpGL7nodqGKszpOHNAzcK7 CHo/Eex5i6JrEqmb0tCi7M7hZujczlekBuIR9iw4PZd+ucMGW5rl9ynK99eEaKLIYyZD Yulg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=yoCwDUxZJP3Ut8z/hr4sZH2+48UBBB2zck9Vy5dLtgU=; b=aZ3mam2bc6fZtI/oiZtfbZ08v2dsF0LIzdq/vTRGPbzu/RxfVvSwuiOsqpEPOCnWPu pgzEhKtbDGCX08hcrn/mfGKRvOfwhYZnzvcszOf9lWv+fjDAdLcw9PTx0+lyzvnCka6n 8JuEDGQ7AeNYU3EisxnJrjgtfBMfNdLpVICCiliLr/q7gYYzR+GbG3LpwZ92+LiVWVER bqbZGLsg5YIdtKWfdcvNZtmcunNCR1KGxtsZ/sOEOSwffUt0iV8szH8+l9FP9mW5iyPL TBFN579YUbQuzJuQCRKsx60XjaKkbcoEwXlWKsQyWqFhi+NLQhIEEx4aSjpkzlnxnhSD PDPQ== X-Gm-Message-State: AOAM533faXcsOwUwcsHz4H7u3JUbTbX/pEDDoZOC2sIqZdOOnY++cQQ5 4vEtU8CtFwDxeJ5wUvcKBosPTGG6z6jVO9r7aZ8= X-Google-Smtp-Source: ABdhPJyCAxXTkhs7AspbVL/ypAtCWxFLyf2+jHwkfD8UredO9wC4bcv1TQdaRkBRCIklexFbhcAJuyI6ee0+SeGt/NU= X-Received: by 2002:a9d:4581:: with SMTP id x1mr22307079ote.145.1624292797889; Mon, 21 Jun 2021 09:26:37 -0700 (PDT) MIME-Version: 1.0 References: <20210618123615.11456-1-ogabbay@kernel.org> <20210621141217.GE1096940@ziepe.ca> In-Reply-To: <20210621141217.GE1096940@ziepe.ca> From: Oded Gabbay Date: Mon, 21 Jun 2021 19:26:14 +0300 Message-ID: Subject: Re: [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF To: Jason Gunthorpe Cc: Greg KH , Daniel Vetter , Oded Gabbay , linux-rdma , "open list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , "airlied@gmail.com" , Linux Kernel Mailing List , Sumit Semwal , =?UTF-8?Q?Christian_K=C3=B6nig?= , Gal Pressman , sleybo@amazon.com, dri-devel , Tomer Tayar , "moderated list:DMA BUFFER SHARING FRAMEWORK" , amd-gfx list , Alex Deucher , Leon Romanovsky , Christoph Hellwig Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Mon, Jun 21, 2021 at 5:12 PM Jason Gunthorpe wrote: > > On Mon, Jun 21, 2021 at 03:02:10PM +0200, Greg KH wrote: > > On Mon, Jun 21, 2021 at 02:28:48PM +0200, Daniel Vetter wrote: > > > > Also I'm wondering which is the other driver that we share buffers > > > with. The gaudi stuff doesn't have real struct pages as backing > > > storage, it only fills out the dma_addr_t. That tends to blow up with > > > other drivers, and the only place where this is guaranteed to work is > > > if you have a dynamic importer which sets the allow_peer2peer flag. > > > Adding maintainers from other subsystems who might want to chime in > > > here. So even aside of the big question as-is this is broken. > > > > From what I can tell this driver is sending the buffers to other > > instances of the same hardware, > > A dmabuf is consumed by something else in the kernel calling > dma_buf_map_attachment() on the FD. > > What is the other side of this? I don't see any > dma_buf_map_attachment() calls in drivers/misc, or added in this patch > set. This patch-set is only to enable the support for the exporter side. The "other side" is any generic RDMA networking device that will want to perform p2p communication over PCIe with our GAUDI accelerator. An example is indeed the mlnx5 card which has already integrated support for being an "importer". This is *not* used for communication with another GAUDI device. If I want to communicate with another GAUDI device, our userspace communications library will use our internal network links, without any need for dma-buf. Oded > > AFAIK the only viable in-tree other side is in mlx5 (look in > umem_dmabuf.c) > > Though as we already talked habana has their own networking (out of > tree, presumably) so I suspect this is really to support some out of > tree stuff?? > > Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F4CCC49EA7 for ; Mon, 21 Jun 2021 16:26:40 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3A214613DA for ; Mon, 21 Jun 2021 16:26:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3A214613DA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 88B3F6E20F; Mon, 21 Jun 2021 16:26:39 +0000 (UTC) Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) by gabe.freedesktop.org (Postfix) with ESMTPS id 90D796E20F; Mon, 21 Jun 2021 16:26:38 +0000 (UTC) Received: by mail-ot1-x32f.google.com with SMTP id v22-20020a0568301416b029044e2d8e855eso9094942otp.8; Mon, 21 Jun 2021 09:26:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=yoCwDUxZJP3Ut8z/hr4sZH2+48UBBB2zck9Vy5dLtgU=; b=pM+MoLSU0LSJh98DQrRsTPmTh+95pmfRHHbTmKcyhP9ovDKRhZcvBC02S/55ftG/og ruYrldynFgVUzXcJ5cpWd8IKQzedGdpH4I+1WfQpqdIufPWiWf72S0LDFysmflufYM4y 4wEreRKEOPdT1VnnTfvE5bFAmShhQkzdbAuUs7Mj25c7MYRwZ3OzmCVuPeHzDrwJTziw rDSCsQLsvjXiYpI4vGyZmIa5HkJhW6dIjnKxwkKEFk6v9qnpGL7nodqGKszpOHNAzcK7 CHo/Eex5i6JrEqmb0tCi7M7hZujczlekBuIR9iw4PZd+ucMGW5rl9ynK99eEaKLIYyZD Yulg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=yoCwDUxZJP3Ut8z/hr4sZH2+48UBBB2zck9Vy5dLtgU=; b=rpLOWEW/hasjgA9Xm3B42x2aQQjYRtOnqRz1lT/0pKMUBX4YmwMXoTB9U3j58OtKGa 4RlmnSwSZHeEkoh5G6/Z/1TZVoosNkpLGrZ7fTbVC/53sK14Y7HpGdYMd54Kcp/JXKQl ATq7pdVXFp+gJMYvvXOQFVMOoPojlcmMKKEax5uCYbbCOtPHQSjKnLUuwpH2pndkaovF OuDdMrtdhHgucN2sD9MJTkLRikYyrkg2u1SPB/PgQRKayvnOIAC2GmzH4pveecaZR7Cu aVY+ya75D+UdaGxviYMA2anQhOezpCbji0k50m8MObkIxVCzYhhyMx73W+pCjMOjL7/I WeFg== X-Gm-Message-State: AOAM531fthivmoJdRgyFmv4XcQLN+y49SJ/IOy3C9eWoxU7OPf3D2Ovt jMXzTntplABUlDvFRD0SssCzUx1rIrTdZBbpEr4= X-Google-Smtp-Source: ABdhPJyCAxXTkhs7AspbVL/ypAtCWxFLyf2+jHwkfD8UredO9wC4bcv1TQdaRkBRCIklexFbhcAJuyI6ee0+SeGt/NU= X-Received: by 2002:a9d:4581:: with SMTP id x1mr22307079ote.145.1624292797889; Mon, 21 Jun 2021 09:26:37 -0700 (PDT) MIME-Version: 1.0 References: <20210618123615.11456-1-ogabbay@kernel.org> <20210621141217.GE1096940@ziepe.ca> In-Reply-To: <20210621141217.GE1096940@ziepe.ca> From: Oded Gabbay Date: Mon, 21 Jun 2021 19:26:14 +0300 Message-ID: Subject: Re: [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF To: Jason Gunthorpe Content-Type: text/plain; charset="UTF-8" X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Gal Pressman , sleybo@amazon.com, linux-rdma , Greg KH , Oded Gabbay , Christoph Hellwig , Linux Kernel Mailing List , dri-devel , =?UTF-8?Q?Christian_K=C3=B6nig?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Daniel Vetter , Alex Deucher , Leon Romanovsky , "open list:DMA BUFFER SHARING FRAMEWORK" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Mon, Jun 21, 2021 at 5:12 PM Jason Gunthorpe wrote: > > On Mon, Jun 21, 2021 at 03:02:10PM +0200, Greg KH wrote: > > On Mon, Jun 21, 2021 at 02:28:48PM +0200, Daniel Vetter wrote: > > > > Also I'm wondering which is the other driver that we share buffers > > > with. The gaudi stuff doesn't have real struct pages as backing > > > storage, it only fills out the dma_addr_t. That tends to blow up with > > > other drivers, and the only place where this is guaranteed to work is > > > if you have a dynamic importer which sets the allow_peer2peer flag. > > > Adding maintainers from other subsystems who might want to chime in > > > here. So even aside of the big question as-is this is broken. > > > > From what I can tell this driver is sending the buffers to other > > instances of the same hardware, > > A dmabuf is consumed by something else in the kernel calling > dma_buf_map_attachment() on the FD. > > What is the other side of this? I don't see any > dma_buf_map_attachment() calls in drivers/misc, or added in this patch > set. This patch-set is only to enable the support for the exporter side. The "other side" is any generic RDMA networking device that will want to perform p2p communication over PCIe with our GAUDI accelerator. An example is indeed the mlnx5 card which has already integrated support for being an "importer". This is *not* used for communication with another GAUDI device. If I want to communicate with another GAUDI device, our userspace communications library will use our internal network links, without any need for dma-buf. Oded > > AFAIK the only viable in-tree other side is in mlx5 (look in > umem_dmabuf.c) > > Though as we already talked habana has their own networking (out of > tree, presumably) so I suspect this is really to support some out of > tree stuff?? > > Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF7E4C49EA7 for ; Mon, 21 Jun 2021 16:26:43 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8C5CE61406 for ; Mon, 21 Jun 2021 16:26:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8C5CE61406 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=amd-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C9C106E210; Mon, 21 Jun 2021 16:26:39 +0000 (UTC) Received: from mail-ot1-x32f.google.com (mail-ot1-x32f.google.com [IPv6:2607:f8b0:4864:20::32f]) by gabe.freedesktop.org (Postfix) with ESMTPS id 90D796E20F; Mon, 21 Jun 2021 16:26:38 +0000 (UTC) Received: by mail-ot1-x32f.google.com with SMTP id v22-20020a0568301416b029044e2d8e855eso9094942otp.8; Mon, 21 Jun 2021 09:26:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=yoCwDUxZJP3Ut8z/hr4sZH2+48UBBB2zck9Vy5dLtgU=; b=pM+MoLSU0LSJh98DQrRsTPmTh+95pmfRHHbTmKcyhP9ovDKRhZcvBC02S/55ftG/og ruYrldynFgVUzXcJ5cpWd8IKQzedGdpH4I+1WfQpqdIufPWiWf72S0LDFysmflufYM4y 4wEreRKEOPdT1VnnTfvE5bFAmShhQkzdbAuUs7Mj25c7MYRwZ3OzmCVuPeHzDrwJTziw rDSCsQLsvjXiYpI4vGyZmIa5HkJhW6dIjnKxwkKEFk6v9qnpGL7nodqGKszpOHNAzcK7 CHo/Eex5i6JrEqmb0tCi7M7hZujczlekBuIR9iw4PZd+ucMGW5rl9ynK99eEaKLIYyZD Yulg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=yoCwDUxZJP3Ut8z/hr4sZH2+48UBBB2zck9Vy5dLtgU=; b=rpLOWEW/hasjgA9Xm3B42x2aQQjYRtOnqRz1lT/0pKMUBX4YmwMXoTB9U3j58OtKGa 4RlmnSwSZHeEkoh5G6/Z/1TZVoosNkpLGrZ7fTbVC/53sK14Y7HpGdYMd54Kcp/JXKQl ATq7pdVXFp+gJMYvvXOQFVMOoPojlcmMKKEax5uCYbbCOtPHQSjKnLUuwpH2pndkaovF OuDdMrtdhHgucN2sD9MJTkLRikYyrkg2u1SPB/PgQRKayvnOIAC2GmzH4pveecaZR7Cu aVY+ya75D+UdaGxviYMA2anQhOezpCbji0k50m8MObkIxVCzYhhyMx73W+pCjMOjL7/I WeFg== X-Gm-Message-State: AOAM531fthivmoJdRgyFmv4XcQLN+y49SJ/IOy3C9eWoxU7OPf3D2Ovt jMXzTntplABUlDvFRD0SssCzUx1rIrTdZBbpEr4= X-Google-Smtp-Source: ABdhPJyCAxXTkhs7AspbVL/ypAtCWxFLyf2+jHwkfD8UredO9wC4bcv1TQdaRkBRCIklexFbhcAJuyI6ee0+SeGt/NU= X-Received: by 2002:a9d:4581:: with SMTP id x1mr22307079ote.145.1624292797889; Mon, 21 Jun 2021 09:26:37 -0700 (PDT) MIME-Version: 1.0 References: <20210618123615.11456-1-ogabbay@kernel.org> <20210621141217.GE1096940@ziepe.ca> In-Reply-To: <20210621141217.GE1096940@ziepe.ca> From: Oded Gabbay Date: Mon, 21 Jun 2021 19:26:14 +0300 Message-ID: Subject: Re: [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF To: Jason Gunthorpe X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Gal Pressman , sleybo@amazon.com, linux-rdma , Greg KH , Oded Gabbay , Christoph Hellwig , Linux Kernel Mailing List , dri-devel , =?UTF-8?Q?Christian_K=C3=B6nig?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Daniel Vetter , Alex Deucher , "airlied@gmail.com" , Sumit Semwal , Leon Romanovsky , "open list:DMA BUFFER SHARING FRAMEWORK" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" On Mon, Jun 21, 2021 at 5:12 PM Jason Gunthorpe wrote: > > On Mon, Jun 21, 2021 at 03:02:10PM +0200, Greg KH wrote: > > On Mon, Jun 21, 2021 at 02:28:48PM +0200, Daniel Vetter wrote: > > > > Also I'm wondering which is the other driver that we share buffers > > > with. The gaudi stuff doesn't have real struct pages as backing > > > storage, it only fills out the dma_addr_t. That tends to blow up with > > > other drivers, and the only place where this is guaranteed to work is > > > if you have a dynamic importer which sets the allow_peer2peer flag. > > > Adding maintainers from other subsystems who might want to chime in > > > here. So even aside of the big question as-is this is broken. > > > > From what I can tell this driver is sending the buffers to other > > instances of the same hardware, > > A dmabuf is consumed by something else in the kernel calling > dma_buf_map_attachment() on the FD. > > What is the other side of this? I don't see any > dma_buf_map_attachment() calls in drivers/misc, or added in this patch > set. This patch-set is only to enable the support for the exporter side. The "other side" is any generic RDMA networking device that will want to perform p2p communication over PCIe with our GAUDI accelerator. An example is indeed the mlnx5 card which has already integrated support for being an "importer". This is *not* used for communication with another GAUDI device. If I want to communicate with another GAUDI device, our userspace communications library will use our internal network links, without any need for dma-buf. Oded > > AFAIK the only viable in-tree other side is in mlx5 (look in > umem_dmabuf.c) > > Though as we already talked habana has their own networking (out of > tree, presumably) so I suspect this is really to support some out of > tree stuff?? > > Jason _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx