From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19D6BC4743C for ; Mon, 21 Jun 2021 19:24:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0461160FEB for ; Mon, 21 Jun 2021 19:24:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230052AbhFUT07 (ORCPT ); Mon, 21 Jun 2021 15:26:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230397AbhFUT06 (ORCPT ); Mon, 21 Jun 2021 15:26:58 -0400 Received: from mail-ot1-x329.google.com (mail-ot1-x329.google.com [IPv6:2607:f8b0:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7913BC061756; Mon, 21 Jun 2021 12:24:44 -0700 (PDT) Received: by mail-ot1-x329.google.com with SMTP id w23-20020a9d5a970000b02903d0ef989477so18803522oth.9; Mon, 21 Jun 2021 12:24:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=dBJ5d1VMITTeIiGdzPkAixCEDqtOI+Of2asfGh+UAZ4=; b=poWdRMlEM2ZoLHlzus9H+sJm2Gyv4Ad5UYLCdoksxIn2aeAMLa+dKaekWP8LifxgyJ n2jVVjTuonYLSxe3ETtsIuLK6gvACWCDRPmuF5tYeQjeJRX7OY/u7vWBpL0kZJzHl5sj Ny/XMZJGPL6fogHwoJFWsiHZjsbH13H9+wxt/q/eKq5DtLZntN4WMvAkk8EZkafocBEK 6xqocdwV6n4EqRoI+ZBfp3axIcf4mR5Wn6oTb26BEs3yq9YTXbI3ZeEuQkoD5A+xRhEy Pckce+Y9OeKX2dihb865SJYvyNOs4TOMg3c3iU7fvPH++FHzFI5BYoTOBQOjmtC3BaTY AZFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dBJ5d1VMITTeIiGdzPkAixCEDqtOI+Of2asfGh+UAZ4=; b=Z6TcuANUJds16hTi8hffR8eVoxSYOIdorTOVAbVhPCLkRydNfApPbN9mHaU7J5vqaK HfI67UOliHEvqeaglP6/4ndtXqAz/c4/pgu+/UVaJMnM+mC4gk8YMcrOiCpXe2uZMNbs zLswkXMm135/eCoC1WHCquIaUa9C8zGO5Nqydds8QaZR1HB5EYJ8z8GY/S2c5L1XtlXM CDrVfHxB3uItyyBfg8SS5IgD9iIpi1ronHwg5AI1EjjQFVMLkr0NAEvszmLkaRXrx3mg bcfeW0gkbJllQQQzCoVfMljcvtnJA51O1bHJOXG2jPbcbZe/falexQnPp2vbu53yz9W0 LNUg== X-Gm-Message-State: AOAM533krmmjlEmIcP4Xtps9U6+5mR4rAaO4zjtzTGRO/qp2UEZNfFu4 JGSGXM7dcJhENC7/cmNmzwX9rxvS8vJV1tWMDEU= X-Google-Smtp-Source: ABdhPJxgnG68naVmqaj9VbYQpqtCzzRUWIKZ1A7mHfJtyBXgRWCjJn597bAS85CrpNuYd9PVyrIs804CEiOHGbIXZSQ= X-Received: by 2002:a9d:542:: with SMTP id 60mr12808otw.143.1624303483767; Mon, 21 Jun 2021 12:24:43 -0700 (PDT) MIME-Version: 1.0 References: <20210618123615.11456-1-ogabbay@kernel.org> <20210621141217.GE1096940@ziepe.ca> <20210621175511.GI1096940@ziepe.ca> In-Reply-To: From: Oded Gabbay Date: Mon, 21 Jun 2021 22:24:16 +0300 Message-ID: Subject: Re: [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF To: Daniel Vetter Cc: Jason Gunthorpe , Greg KH , Oded Gabbay , linux-rdma , "open list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , "airlied@gmail.com" , Linux Kernel Mailing List , Sumit Semwal , =?UTF-8?Q?Christian_K=C3=B6nig?= , Gal Pressman , sleybo@amazon.com, dri-devel , Tomer Tayar , "moderated list:DMA BUFFER SHARING FRAMEWORK" , amd-gfx list , Alex Deucher , Leon Romanovsky , Christoph Hellwig Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Mon, Jun 21, 2021 at 9:27 PM Daniel Vetter wrote: > > On Mon, Jun 21, 2021 at 7:55 PM Jason Gunthorpe wrote: > > On Mon, Jun 21, 2021 at 07:26:14PM +0300, Oded Gabbay wrote: > > > On Mon, Jun 21, 2021 at 5:12 PM Jason Gunthorpe wrote: > > > > > > > > On Mon, Jun 21, 2021 at 03:02:10PM +0200, Greg KH wrote: > > > > > On Mon, Jun 21, 2021 at 02:28:48PM +0200, Daniel Vetter wrote: > > > > > > > > > > Also I'm wondering which is the other driver that we share buffers > > > > > > with. The gaudi stuff doesn't have real struct pages as backing > > > > > > storage, it only fills out the dma_addr_t. That tends to blow up with > > > > > > other drivers, and the only place where this is guaranteed to work is > > > > > > if you have a dynamic importer which sets the allow_peer2peer flag. > > > > > > Adding maintainers from other subsystems who might want to chime in > > > > > > here. So even aside of the big question as-is this is broken. > > > > > > > > > > From what I can tell this driver is sending the buffers to other > > > > > instances of the same hardware, > > > > > > > > A dmabuf is consumed by something else in the kernel calling > > > > dma_buf_map_attachment() on the FD. > > > > > > > > What is the other side of this? I don't see any > > > > dma_buf_map_attachment() calls in drivers/misc, or added in this patch > > > > set. > > > > > > This patch-set is only to enable the support for the exporter side. > > > The "other side" is any generic RDMA networking device that will want > > > to perform p2p communication over PCIe with our GAUDI accelerator. > > > An example is indeed the mlnx5 card which has already integrated > > > support for being an "importer". > > > > It raises the question of how you are testing this if you aren't using > > it with the only intree driver: mlx5. > > For p2p dma-buf there's also amdgpu as a possible in-tree candiate > driver, that's why I added amdgpu folks. Otoh I'm not aware of AI+GPU > combos being much in use, at least with upstream gpu drivers (nvidia > blob is a different story ofc, but I don't care what they do in their > own world). > -Daniel > -- We have/are doing three things: 1. I wrote a simple "importer" driver that emulates an RDMA driver. It calls all the IB_UMEM_DMABUF functions, same as the mlnx5 driver does. And instead of using h/w, it accesses the bar directly. We wrote several tests that emulated the real application. i.e. asking the habanalabs driver to create dma-buf object and export its FD back to userspace. Then the userspace sends the FD to the "importer" driver, which attaches to it, get the SG list and accesses the memory on the GAUDI device. This gave me the confidence that how we integrated the exporter is basically correct/working. 2. We are trying to do a POC with a MLNX card we have, WIP. 3. We are working with another 3rd party RDMA device that its driver is now adding support for being an "importer". also WIP In both points 2&3 We haven't yet reached the actual stage of checking this feature. Another thing I want to emphasize is that we are doing p2p only through the export/import of the FD. We do *not* allow the user to mmap the dma-buf as we do not support direct IO. So there is no access to these pages through the userspace. Thanks, Oded From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 472F2C49EA2 for ; Mon, 21 Jun 2021 19:24:47 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0C4B361350 for ; Mon, 21 Jun 2021 19:24:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0C4B361350 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7810A6E3E5; Mon, 21 Jun 2021 19:24:46 +0000 (UTC) Received: from mail-ot1-x32a.google.com (mail-ot1-x32a.google.com [IPv6:2607:f8b0:4864:20::32a]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6B9F66E3E5; Mon, 21 Jun 2021 19:24:45 +0000 (UTC) Received: by mail-ot1-x32a.google.com with SMTP id 6-20020a9d07860000b02903e83bf8f8fcso18820972oto.12; Mon, 21 Jun 2021 12:24:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=dBJ5d1VMITTeIiGdzPkAixCEDqtOI+Of2asfGh+UAZ4=; b=poWdRMlEM2ZoLHlzus9H+sJm2Gyv4Ad5UYLCdoksxIn2aeAMLa+dKaekWP8LifxgyJ n2jVVjTuonYLSxe3ETtsIuLK6gvACWCDRPmuF5tYeQjeJRX7OY/u7vWBpL0kZJzHl5sj Ny/XMZJGPL6fogHwoJFWsiHZjsbH13H9+wxt/q/eKq5DtLZntN4WMvAkk8EZkafocBEK 6xqocdwV6n4EqRoI+ZBfp3axIcf4mR5Wn6oTb26BEs3yq9YTXbI3ZeEuQkoD5A+xRhEy Pckce+Y9OeKX2dihb865SJYvyNOs4TOMg3c3iU7fvPH++FHzFI5BYoTOBQOjmtC3BaTY AZFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dBJ5d1VMITTeIiGdzPkAixCEDqtOI+Of2asfGh+UAZ4=; b=I01HTtfmzmQP3MoEGDvrESgSBQ+kn6rVBhFonmcn5Y+ZZ+/VqbBcuTAShHr/lv5elA uPmgEVHM1JUFbjwY9sNnX45xteh56aLmGNlOeZUJwDw6sZMKzSrCbTyGQWOSQ4RXGhKo yrdDnRv3g3p6NpfGiDmWeGwOANS5Fgt+iVvvBjfIAh5wmA8JZdmYxBKeuE3wMNFBXZMq rnnE8DfEQKYajsL8u2EA0I5dNiIhBCPRLZv5hRcGc3YN6d3mCKUBvGn1cERvIntagzXv yw1nnyRC4fx2WqN20BuQpcqAmzO3mQvNezTIp3C/FMIdEd6pUBF19RG4vn9bVKX2vseN TUgA== X-Gm-Message-State: AOAM531E368gzRIuQqs2tljGdLkkgizJfQEcgbahMexIqi7xvgLeyvSo ss4EBe4brKabWaHrg2yaX945oUnp1HsWjOuiEoc= X-Google-Smtp-Source: ABdhPJxgnG68naVmqaj9VbYQpqtCzzRUWIKZ1A7mHfJtyBXgRWCjJn597bAS85CrpNuYd9PVyrIs804CEiOHGbIXZSQ= X-Received: by 2002:a9d:542:: with SMTP id 60mr12808otw.143.1624303483767; Mon, 21 Jun 2021 12:24:43 -0700 (PDT) MIME-Version: 1.0 References: <20210618123615.11456-1-ogabbay@kernel.org> <20210621141217.GE1096940@ziepe.ca> <20210621175511.GI1096940@ziepe.ca> In-Reply-To: From: Oded Gabbay Date: Mon, 21 Jun 2021 22:24:16 +0300 Message-ID: Subject: Re: [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF To: Daniel Vetter Content-Type: text/plain; charset="UTF-8" X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Gal Pressman , sleybo@amazon.com, linux-rdma , Greg KH , Oded Gabbay , Christoph Hellwig , Linux Kernel Mailing List , dri-devel , =?UTF-8?Q?Christian_K=C3=B6nig?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Jason Gunthorpe , Doug Ledford , Tomer Tayar , amd-gfx list , Alex Deucher , Leon Romanovsky , "open list:DMA BUFFER SHARING FRAMEWORK" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Mon, Jun 21, 2021 at 9:27 PM Daniel Vetter wrote: > > On Mon, Jun 21, 2021 at 7:55 PM Jason Gunthorpe wrote: > > On Mon, Jun 21, 2021 at 07:26:14PM +0300, Oded Gabbay wrote: > > > On Mon, Jun 21, 2021 at 5:12 PM Jason Gunthorpe wrote: > > > > > > > > On Mon, Jun 21, 2021 at 03:02:10PM +0200, Greg KH wrote: > > > > > On Mon, Jun 21, 2021 at 02:28:48PM +0200, Daniel Vetter wrote: > > > > > > > > > > Also I'm wondering which is the other driver that we share buffers > > > > > > with. The gaudi stuff doesn't have real struct pages as backing > > > > > > storage, it only fills out the dma_addr_t. That tends to blow up with > > > > > > other drivers, and the only place where this is guaranteed to work is > > > > > > if you have a dynamic importer which sets the allow_peer2peer flag. > > > > > > Adding maintainers from other subsystems who might want to chime in > > > > > > here. So even aside of the big question as-is this is broken. > > > > > > > > > > From what I can tell this driver is sending the buffers to other > > > > > instances of the same hardware, > > > > > > > > A dmabuf is consumed by something else in the kernel calling > > > > dma_buf_map_attachment() on the FD. > > > > > > > > What is the other side of this? I don't see any > > > > dma_buf_map_attachment() calls in drivers/misc, or added in this patch > > > > set. > > > > > > This patch-set is only to enable the support for the exporter side. > > > The "other side" is any generic RDMA networking device that will want > > > to perform p2p communication over PCIe with our GAUDI accelerator. > > > An example is indeed the mlnx5 card which has already integrated > > > support for being an "importer". > > > > It raises the question of how you are testing this if you aren't using > > it with the only intree driver: mlx5. > > For p2p dma-buf there's also amdgpu as a possible in-tree candiate > driver, that's why I added amdgpu folks. Otoh I'm not aware of AI+GPU > combos being much in use, at least with upstream gpu drivers (nvidia > blob is a different story ofc, but I don't care what they do in their > own world). > -Daniel > -- We have/are doing three things: 1. I wrote a simple "importer" driver that emulates an RDMA driver. It calls all the IB_UMEM_DMABUF functions, same as the mlnx5 driver does. And instead of using h/w, it accesses the bar directly. We wrote several tests that emulated the real application. i.e. asking the habanalabs driver to create dma-buf object and export its FD back to userspace. Then the userspace sends the FD to the "importer" driver, which attaches to it, get the SG list and accesses the memory on the GAUDI device. This gave me the confidence that how we integrated the exporter is basically correct/working. 2. We are trying to do a POC with a MLNX card we have, WIP. 3. We are working with another 3rd party RDMA device that its driver is now adding support for being an "importer". also WIP In both points 2&3 We haven't yet reached the actual stage of checking this feature. Another thing I want to emphasize is that we are doing p2p only through the export/import of the FD. We do *not* allow the user to mmap the dma-buf as we do not support direct IO. So there is no access to these pages through the userspace. Thanks, Oded From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8CA4C4743C for ; Mon, 21 Jun 2021 19:24:48 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 86B206128C for ; Mon, 21 Jun 2021 19:24:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 86B206128C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=amd-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 158356E3F4; Mon, 21 Jun 2021 19:24:47 +0000 (UTC) Received: from mail-ot1-x32a.google.com (mail-ot1-x32a.google.com [IPv6:2607:f8b0:4864:20::32a]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6B9F66E3E5; Mon, 21 Jun 2021 19:24:45 +0000 (UTC) Received: by mail-ot1-x32a.google.com with SMTP id 6-20020a9d07860000b02903e83bf8f8fcso18820972oto.12; Mon, 21 Jun 2021 12:24:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=dBJ5d1VMITTeIiGdzPkAixCEDqtOI+Of2asfGh+UAZ4=; b=poWdRMlEM2ZoLHlzus9H+sJm2Gyv4Ad5UYLCdoksxIn2aeAMLa+dKaekWP8LifxgyJ n2jVVjTuonYLSxe3ETtsIuLK6gvACWCDRPmuF5tYeQjeJRX7OY/u7vWBpL0kZJzHl5sj Ny/XMZJGPL6fogHwoJFWsiHZjsbH13H9+wxt/q/eKq5DtLZntN4WMvAkk8EZkafocBEK 6xqocdwV6n4EqRoI+ZBfp3axIcf4mR5Wn6oTb26BEs3yq9YTXbI3ZeEuQkoD5A+xRhEy Pckce+Y9OeKX2dihb865SJYvyNOs4TOMg3c3iU7fvPH++FHzFI5BYoTOBQOjmtC3BaTY AZFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dBJ5d1VMITTeIiGdzPkAixCEDqtOI+Of2asfGh+UAZ4=; b=I01HTtfmzmQP3MoEGDvrESgSBQ+kn6rVBhFonmcn5Y+ZZ+/VqbBcuTAShHr/lv5elA uPmgEVHM1JUFbjwY9sNnX45xteh56aLmGNlOeZUJwDw6sZMKzSrCbTyGQWOSQ4RXGhKo yrdDnRv3g3p6NpfGiDmWeGwOANS5Fgt+iVvvBjfIAh5wmA8JZdmYxBKeuE3wMNFBXZMq rnnE8DfEQKYajsL8u2EA0I5dNiIhBCPRLZv5hRcGc3YN6d3mCKUBvGn1cERvIntagzXv yw1nnyRC4fx2WqN20BuQpcqAmzO3mQvNezTIp3C/FMIdEd6pUBF19RG4vn9bVKX2vseN TUgA== X-Gm-Message-State: AOAM531E368gzRIuQqs2tljGdLkkgizJfQEcgbahMexIqi7xvgLeyvSo ss4EBe4brKabWaHrg2yaX945oUnp1HsWjOuiEoc= X-Google-Smtp-Source: ABdhPJxgnG68naVmqaj9VbYQpqtCzzRUWIKZ1A7mHfJtyBXgRWCjJn597bAS85CrpNuYd9PVyrIs804CEiOHGbIXZSQ= X-Received: by 2002:a9d:542:: with SMTP id 60mr12808otw.143.1624303483767; Mon, 21 Jun 2021 12:24:43 -0700 (PDT) MIME-Version: 1.0 References: <20210618123615.11456-1-ogabbay@kernel.org> <20210621141217.GE1096940@ziepe.ca> <20210621175511.GI1096940@ziepe.ca> In-Reply-To: From: Oded Gabbay Date: Mon, 21 Jun 2021 22:24:16 +0300 Message-ID: Subject: Re: [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF To: Daniel Vetter X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Gal Pressman , sleybo@amazon.com, linux-rdma , Greg KH , Oded Gabbay , Christoph Hellwig , Linux Kernel Mailing List , dri-devel , =?UTF-8?Q?Christian_K=C3=B6nig?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Jason Gunthorpe , Doug Ledford , Tomer Tayar , amd-gfx list , Alex Deucher , "airlied@gmail.com" , Sumit Semwal , Leon Romanovsky , "open list:DMA BUFFER SHARING FRAMEWORK" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" On Mon, Jun 21, 2021 at 9:27 PM Daniel Vetter wrote: > > On Mon, Jun 21, 2021 at 7:55 PM Jason Gunthorpe wrote: > > On Mon, Jun 21, 2021 at 07:26:14PM +0300, Oded Gabbay wrote: > > > On Mon, Jun 21, 2021 at 5:12 PM Jason Gunthorpe wrote: > > > > > > > > On Mon, Jun 21, 2021 at 03:02:10PM +0200, Greg KH wrote: > > > > > On Mon, Jun 21, 2021 at 02:28:48PM +0200, Daniel Vetter wrote: > > > > > > > > > > Also I'm wondering which is the other driver that we share buffers > > > > > > with. The gaudi stuff doesn't have real struct pages as backing > > > > > > storage, it only fills out the dma_addr_t. That tends to blow up with > > > > > > other drivers, and the only place where this is guaranteed to work is > > > > > > if you have a dynamic importer which sets the allow_peer2peer flag. > > > > > > Adding maintainers from other subsystems who might want to chime in > > > > > > here. So even aside of the big question as-is this is broken. > > > > > > > > > > From what I can tell this driver is sending the buffers to other > > > > > instances of the same hardware, > > > > > > > > A dmabuf is consumed by something else in the kernel calling > > > > dma_buf_map_attachment() on the FD. > > > > > > > > What is the other side of this? I don't see any > > > > dma_buf_map_attachment() calls in drivers/misc, or added in this patch > > > > set. > > > > > > This patch-set is only to enable the support for the exporter side. > > > The "other side" is any generic RDMA networking device that will want > > > to perform p2p communication over PCIe with our GAUDI accelerator. > > > An example is indeed the mlnx5 card which has already integrated > > > support for being an "importer". > > > > It raises the question of how you are testing this if you aren't using > > it with the only intree driver: mlx5. > > For p2p dma-buf there's also amdgpu as a possible in-tree candiate > driver, that's why I added amdgpu folks. Otoh I'm not aware of AI+GPU > combos being much in use, at least with upstream gpu drivers (nvidia > blob is a different story ofc, but I don't care what they do in their > own world). > -Daniel > -- We have/are doing three things: 1. I wrote a simple "importer" driver that emulates an RDMA driver. It calls all the IB_UMEM_DMABUF functions, same as the mlnx5 driver does. And instead of using h/w, it accesses the bar directly. We wrote several tests that emulated the real application. i.e. asking the habanalabs driver to create dma-buf object and export its FD back to userspace. Then the userspace sends the FD to the "importer" driver, which attaches to it, get the SG list and accesses the memory on the GAUDI device. This gave me the confidence that how we integrated the exporter is basically correct/working. 2. We are trying to do a POC with a MLNX card we have, WIP. 3. We are working with another 3rd party RDMA device that its driver is now adding support for being an "importer". also WIP In both points 2&3 We haven't yet reached the actual stage of checking this feature. Another thing I want to emphasize is that we are doing p2p only through the export/import of the FD. We do *not* allow the user to mmap the dma-buf as we do not support direct IO. So there is no access to these pages through the userspace. Thanks, Oded _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx