From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 586A3C48BE5 for ; Wed, 23 Jun 2021 19:40:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4096161075 for ; Wed, 23 Jun 2021 19:40:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230103AbhFWTmg (ORCPT ); Wed, 23 Jun 2021 15:42:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45150 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230124AbhFWTmf (ORCPT ); Wed, 23 Jun 2021 15:42:35 -0400 Received: from mail-ot1-x32c.google.com (mail-ot1-x32c.google.com [IPv6:2607:f8b0:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58A0AC0613A2; Wed, 23 Jun 2021 12:40:15 -0700 (PDT) Received: by mail-ot1-x32c.google.com with SMTP id 6-20020a9d07860000b02903e83bf8f8fcso3086987oto.12; Wed, 23 Jun 2021 12:40:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=iDjEA/DLHJuJqbx7wc+mRyI0RJczQytGp1V6E75bd6s=; b=njxfplOx17G0/91fOLaiwVofE0DeSdGcCG3lxRi9OYVY6KYvRDAEsYT4a2X08avjYz uk9LS4BPT8B0ygh92FmzwsjnilNP75l1/gXEVq89GseCAdSIpq6YYw8fjjAPPxr4o+PW L1d2m01Uw4KLPxZ66w09nOVAPTd8mJoSjCCMU66jx/8k6x/Lf49D21aPA/ifhKbt2vkW jY4n3+MG8/wALoxr2cftjzWWAHPFat3vVo01kBbQP+RM7GGT5W4PQR6uFllaJRjFkjzQ vwcmftcSUgYfe5LpftZyXNntJzQ+Wo2BZqKzqtT+Ch7LKM5E+/AWX9vBPa/rH66Sg51K GNSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=iDjEA/DLHJuJqbx7wc+mRyI0RJczQytGp1V6E75bd6s=; b=d7oDwWtBSFOBJ15jMYouUqLjlwSCasEdomDIFvWMcYICYo1E4noBb+G7rEoltqExPQ oK5LrZgJo4cLXUUjvL/wEqLO+qCGCV0QpU8v8NpVOQf0eOwggoCxY/bAEUONGNAg3G3E 1H2mW+y/uJ28qjcmw5zHA6ATCJbAqLysGLEJBHQ0Ta3lxQXPJXlxoSYIpZ1OEZRGJwSV Oolm3VK1vALZZ+0mOTV5sZYKcxZJSnpjpqFY9lAk6SJ5SIueSR/cEO5mXbocGz6oEhjK o5NEp+EuytaKgHvkx494ehwNg1qUgOjagS5TPAlb5ktYTOou7GGpZ2ZzFx8LmCRWPJ1M kMlw== X-Gm-Message-State: AOAM530ColhpSh+ZVgUlBXq4Z5EH8gbuSriJwiqCl1z9Zdsp/r3nn2Wx mf4p6FqWRx9GdaIsIHJXiCDSklS4gX9i2w4LYhI= X-Google-Smtp-Source: ABdhPJznDm09EdJ4nc7aCaNiJgX/OzhaJax6fBkCDboeJEjZGy4mnzO3nT99DVOufQL4W+3P5HvD7FiPj986tjjoFdA= X-Received: by 2002:a9d:542:: with SMTP id 60mr1364333otw.143.1624477214591; Wed, 23 Jun 2021 12:40:14 -0700 (PDT) MIME-Version: 1.0 References: <20210622152343.GO1096940@ziepe.ca> <3fabe8b7-7174-bf49-5ffe-26db30968a27@amd.com> <20210622154027.GS1096940@ziepe.ca> <09df4a03-d99c-3949-05b2-8b49c71a109e@amd.com> <20210622160538.GT1096940@ziepe.ca> <20210623182435.GX1096940@ziepe.ca> <20210623185045.GY1096940@ziepe.ca> <20210623193456.GZ1096940@ziepe.ca> In-Reply-To: <20210623193456.GZ1096940@ziepe.ca> From: Oded Gabbay Date: Wed, 23 Jun 2021 22:39:48 +0300 Message-ID: Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF To: Jason Gunthorpe Cc: =?UTF-8?Q?Christian_K=C3=B6nig?= , =?UTF-8?Q?Christian_K=C3=B6nig?= , Gal Pressman , sleybo@amazon.com, linux-rdma , Oded Gabbay , Christoph Hellwig , Linux Kernel Mailing List , dri-devel , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Greg KH , Alex Deucher , Leon Romanovsky , "open list:DMA BUFFER SHARING FRAMEWORK" Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On Wed, Jun 23, 2021 at 10:34 PM Jason Gunthorpe wrote: > > On Wed, Jun 23, 2021 at 10:00:29PM +0300, Oded Gabbay wrote: > > On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe wrote: > > > > > > On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > > > > > > > Can you please explain why it is so important to (allow) access them > > > > through the CPU ? > > > > > > It is not so much important, as it reflects significant design choices > > > that are already tightly baked into alot of our stacks. > > > > > > A SGL is CPU accessible by design - that is baked into this thing and > > > places all over the place assume it. Even in RDMA we have > > > RXE/SWI/HFI1/qib that might want to use the CPU side (grep for sg_page > > > to see) > > > > > > So, the thing at the top of the stack - in this case the gaudi driver > > > - simply can't assume what the rest of the stack is going to do and > > > omit the CPU side. It breaks everything. > > > > > > Logan's patch series is the most fully developed way out of this > > > predicament so far. > > > > I understand the argument and I agree that for the generic case, the > > top of the stack can't assume anything. > > Having said that, in this case the SGL is encapsulated inside a dma-buf object. > > > > Maybe its a stupid/over-simplified suggestion, but can't we add a > > property to the dma-buf object, > > that will be set by the exporter, which will "tell" the importer it > > can't use any CPU fallback ? Only "real" p2p ? > > The block stack has been trying to do something like this. > > The flag doesn't solve the DMA API/IOMMU problems though. hmm, I thought using dma_map_resource will solve the IOMMU issues, no ? We talked about it yesterday, and you said that it will "work" (although I noticed a tone of reluctance when you said that). If I use dma_map_resource to set the addresses inside the SGL before I export the dma-buf, and guarantee no one will use the SGL in the dma-buf for any other purpose than device p2p, what else is needed ? Oded > > Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DB70C4743C for ; Wed, 23 Jun 2021 19:40:17 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id DE63C6115A for ; Wed, 23 Jun 2021 19:40:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org DE63C6115A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5FAB36E98A; Wed, 23 Jun 2021 19:40:16 +0000 (UTC) Received: from mail-ot1-x32b.google.com (mail-ot1-x32b.google.com [IPv6:2607:f8b0:4864:20::32b]) by gabe.freedesktop.org (Postfix) with ESMTPS id 551786E98A; Wed, 23 Jun 2021 19:40:15 +0000 (UTC) Received: by mail-ot1-x32b.google.com with SMTP id v5-20020a0568301bc5b029045c06b14f83so3088577ota.13; Wed, 23 Jun 2021 12:40:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=iDjEA/DLHJuJqbx7wc+mRyI0RJczQytGp1V6E75bd6s=; b=njxfplOx17G0/91fOLaiwVofE0DeSdGcCG3lxRi9OYVY6KYvRDAEsYT4a2X08avjYz uk9LS4BPT8B0ygh92FmzwsjnilNP75l1/gXEVq89GseCAdSIpq6YYw8fjjAPPxr4o+PW L1d2m01Uw4KLPxZ66w09nOVAPTd8mJoSjCCMU66jx/8k6x/Lf49D21aPA/ifhKbt2vkW jY4n3+MG8/wALoxr2cftjzWWAHPFat3vVo01kBbQP+RM7GGT5W4PQR6uFllaJRjFkjzQ vwcmftcSUgYfe5LpftZyXNntJzQ+Wo2BZqKzqtT+Ch7LKM5E+/AWX9vBPa/rH66Sg51K GNSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=iDjEA/DLHJuJqbx7wc+mRyI0RJczQytGp1V6E75bd6s=; b=lu0nrWClQuj4N27dIZZDshhmTVymirb7us8By+6TyHIpCbcEBsBlgPlUscGpEo9ykp wRMMcatZPsdWGIt011da5Nxys/7o7Ux8BQJSd3fbErKT+e7vBTk9iXTi//e6nE+pJC/V HtwXu4G93CtSJQIEHq6SEtWrI3h0H5HdGoiCdnvmdvaYS5NwPqdjQIEhtvdYVyWRN0vg epXM33L0r14jQM+Fu6c3MuXUq+7yI2nbqTxQEJajVW1+m+XSgtK3+pTE8+YUnHKvyw1A x6lP+kbx0w0oBmm6GBc2i7f/KyeotmwICcbjMJTCe3bDE+vZ2BsyTlwnqHTtqXq7hNT7 clIg== X-Gm-Message-State: AOAM533SJpBeI4b1rd6Ld5rU/yo6a1Neg8Wm3qQJMSVt7d9DCeYUGdLb Etb945gO6FSiMNQZvDSwZhFdYHniGI7gEU/cYEE= X-Google-Smtp-Source: ABdhPJznDm09EdJ4nc7aCaNiJgX/OzhaJax6fBkCDboeJEjZGy4mnzO3nT99DVOufQL4W+3P5HvD7FiPj986tjjoFdA= X-Received: by 2002:a9d:542:: with SMTP id 60mr1364333otw.143.1624477214591; Wed, 23 Jun 2021 12:40:14 -0700 (PDT) MIME-Version: 1.0 References: <20210622152343.GO1096940@ziepe.ca> <3fabe8b7-7174-bf49-5ffe-26db30968a27@amd.com> <20210622154027.GS1096940@ziepe.ca> <09df4a03-d99c-3949-05b2-8b49c71a109e@amd.com> <20210622160538.GT1096940@ziepe.ca> <20210623182435.GX1096940@ziepe.ca> <20210623185045.GY1096940@ziepe.ca> <20210623193456.GZ1096940@ziepe.ca> In-Reply-To: <20210623193456.GZ1096940@ziepe.ca> From: Oded Gabbay Date: Wed, 23 Jun 2021 22:39:48 +0300 Message-ID: Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF To: Jason Gunthorpe Content-Type: text/plain; charset="UTF-8" X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-rdma , =?UTF-8?Q?Christian_K=C3=B6nig?= , sleybo@amazon.com, Leon Romanovsky , Gal Pressman , dri-devel , =?UTF-8?Q?Christian_K=C3=B6nig?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Greg KH , Alex Deucher , Christoph Hellwig , Oded Gabbay , Linux Kernel Mailing List , "open list:DMA BUFFER SHARING FRAMEWORK" Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Wed, Jun 23, 2021 at 10:34 PM Jason Gunthorpe wrote: > > On Wed, Jun 23, 2021 at 10:00:29PM +0300, Oded Gabbay wrote: > > On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe wrote: > > > > > > On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > > > > > > > Can you please explain why it is so important to (allow) access them > > > > through the CPU ? > > > > > > It is not so much important, as it reflects significant design choices > > > that are already tightly baked into alot of our stacks. > > > > > > A SGL is CPU accessible by design - that is baked into this thing and > > > places all over the place assume it. Even in RDMA we have > > > RXE/SWI/HFI1/qib that might want to use the CPU side (grep for sg_page > > > to see) > > > > > > So, the thing at the top of the stack - in this case the gaudi driver > > > - simply can't assume what the rest of the stack is going to do and > > > omit the CPU side. It breaks everything. > > > > > > Logan's patch series is the most fully developed way out of this > > > predicament so far. > > > > I understand the argument and I agree that for the generic case, the > > top of the stack can't assume anything. > > Having said that, in this case the SGL is encapsulated inside a dma-buf object. > > > > Maybe its a stupid/over-simplified suggestion, but can't we add a > > property to the dma-buf object, > > that will be set by the exporter, which will "tell" the importer it > > can't use any CPU fallback ? Only "real" p2p ? > > The block stack has been trying to do something like this. > > The flag doesn't solve the DMA API/IOMMU problems though. hmm, I thought using dma_map_resource will solve the IOMMU issues, no ? We talked about it yesterday, and you said that it will "work" (although I noticed a tone of reluctance when you said that). If I use dma_map_resource to set the addresses inside the SGL before I export the dma-buf, and guarantee no one will use the SGL in the dma-buf for any other purpose than device p2p, what else is needed ? Oded > > Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFDF1C4743C for ; Wed, 23 Jun 2021 19:40:19 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B884061075 for ; Wed, 23 Jun 2021 19:40:19 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B884061075 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=amd-gfx-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 168216E98D; Wed, 23 Jun 2021 19:40:17 +0000 (UTC) Received: from mail-ot1-x32b.google.com (mail-ot1-x32b.google.com [IPv6:2607:f8b0:4864:20::32b]) by gabe.freedesktop.org (Postfix) with ESMTPS id 551786E98A; Wed, 23 Jun 2021 19:40:15 +0000 (UTC) Received: by mail-ot1-x32b.google.com with SMTP id v5-20020a0568301bc5b029045c06b14f83so3088577ota.13; Wed, 23 Jun 2021 12:40:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=iDjEA/DLHJuJqbx7wc+mRyI0RJczQytGp1V6E75bd6s=; b=njxfplOx17G0/91fOLaiwVofE0DeSdGcCG3lxRi9OYVY6KYvRDAEsYT4a2X08avjYz uk9LS4BPT8B0ygh92FmzwsjnilNP75l1/gXEVq89GseCAdSIpq6YYw8fjjAPPxr4o+PW L1d2m01Uw4KLPxZ66w09nOVAPTd8mJoSjCCMU66jx/8k6x/Lf49D21aPA/ifhKbt2vkW jY4n3+MG8/wALoxr2cftjzWWAHPFat3vVo01kBbQP+RM7GGT5W4PQR6uFllaJRjFkjzQ vwcmftcSUgYfe5LpftZyXNntJzQ+Wo2BZqKzqtT+Ch7LKM5E+/AWX9vBPa/rH66Sg51K GNSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=iDjEA/DLHJuJqbx7wc+mRyI0RJczQytGp1V6E75bd6s=; b=lu0nrWClQuj4N27dIZZDshhmTVymirb7us8By+6TyHIpCbcEBsBlgPlUscGpEo9ykp wRMMcatZPsdWGIt011da5Nxys/7o7Ux8BQJSd3fbErKT+e7vBTk9iXTi//e6nE+pJC/V HtwXu4G93CtSJQIEHq6SEtWrI3h0H5HdGoiCdnvmdvaYS5NwPqdjQIEhtvdYVyWRN0vg epXM33L0r14jQM+Fu6c3MuXUq+7yI2nbqTxQEJajVW1+m+XSgtK3+pTE8+YUnHKvyw1A x6lP+kbx0w0oBmm6GBc2i7f/KyeotmwICcbjMJTCe3bDE+vZ2BsyTlwnqHTtqXq7hNT7 clIg== X-Gm-Message-State: AOAM533SJpBeI4b1rd6Ld5rU/yo6a1Neg8Wm3qQJMSVt7d9DCeYUGdLb Etb945gO6FSiMNQZvDSwZhFdYHniGI7gEU/cYEE= X-Google-Smtp-Source: ABdhPJznDm09EdJ4nc7aCaNiJgX/OzhaJax6fBkCDboeJEjZGy4mnzO3nT99DVOufQL4W+3P5HvD7FiPj986tjjoFdA= X-Received: by 2002:a9d:542:: with SMTP id 60mr1364333otw.143.1624477214591; Wed, 23 Jun 2021 12:40:14 -0700 (PDT) MIME-Version: 1.0 References: <20210622152343.GO1096940@ziepe.ca> <3fabe8b7-7174-bf49-5ffe-26db30968a27@amd.com> <20210622154027.GS1096940@ziepe.ca> <09df4a03-d99c-3949-05b2-8b49c71a109e@amd.com> <20210622160538.GT1096940@ziepe.ca> <20210623182435.GX1096940@ziepe.ca> <20210623185045.GY1096940@ziepe.ca> <20210623193456.GZ1096940@ziepe.ca> In-Reply-To: <20210623193456.GZ1096940@ziepe.ca> From: Oded Gabbay Date: Wed, 23 Jun 2021 22:39:48 +0300 Message-ID: Subject: Re: [Linaro-mm-sig] [PATCH v3 1/2] habanalabs: define uAPI to export FD for DMA-BUF To: Jason Gunthorpe X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-rdma , =?UTF-8?Q?Christian_K=C3=B6nig?= , sleybo@amazon.com, Leon Romanovsky , Gal Pressman , dri-devel , =?UTF-8?Q?Christian_K=C3=B6nig?= , "moderated list:DMA BUFFER SHARING FRAMEWORK" , Doug Ledford , Tomer Tayar , amd-gfx list , Greg KH , Alex Deucher , Christoph Hellwig , Oded Gabbay , Linux Kernel Mailing List , "open list:DMA BUFFER SHARING FRAMEWORK" Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" On Wed, Jun 23, 2021 at 10:34 PM Jason Gunthorpe wrote: > > On Wed, Jun 23, 2021 at 10:00:29PM +0300, Oded Gabbay wrote: > > On Wed, Jun 23, 2021 at 9:50 PM Jason Gunthorpe wrote: > > > > > > On Wed, Jun 23, 2021 at 09:43:04PM +0300, Oded Gabbay wrote: > > > > > > > Can you please explain why it is so important to (allow) access them > > > > through the CPU ? > > > > > > It is not so much important, as it reflects significant design choices > > > that are already tightly baked into alot of our stacks. > > > > > > A SGL is CPU accessible by design - that is baked into this thing and > > > places all over the place assume it. Even in RDMA we have > > > RXE/SWI/HFI1/qib that might want to use the CPU side (grep for sg_page > > > to see) > > > > > > So, the thing at the top of the stack - in this case the gaudi driver > > > - simply can't assume what the rest of the stack is going to do and > > > omit the CPU side. It breaks everything. > > > > > > Logan's patch series is the most fully developed way out of this > > > predicament so far. > > > > I understand the argument and I agree that for the generic case, the > > top of the stack can't assume anything. > > Having said that, in this case the SGL is encapsulated inside a dma-buf object. > > > > Maybe its a stupid/over-simplified suggestion, but can't we add a > > property to the dma-buf object, > > that will be set by the exporter, which will "tell" the importer it > > can't use any CPU fallback ? Only "real" p2p ? > > The block stack has been trying to do something like this. > > The flag doesn't solve the DMA API/IOMMU problems though. hmm, I thought using dma_map_resource will solve the IOMMU issues, no ? We talked about it yesterday, and you said that it will "work" (although I noticed a tone of reluctance when you said that). If I use dma_map_resource to set the addresses inside the SGL before I export the dma-buf, and guarantee no one will use the SGL in the dma-buf for any other purpose than device p2p, what else is needed ? Oded > > Jason _______________________________________________ amd-gfx mailing list amd-gfx@lists.freedesktop.org https://lists.freedesktop.org/mailman/listinfo/amd-gfx