From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-19.5 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3AADC4338F for ; Wed, 18 Aug 2021 07:46:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9A9726108B for ; Wed, 18 Aug 2021 07:46:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239358AbhHRHqh (ORCPT ); Wed, 18 Aug 2021 03:46:37 -0400 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:9843 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239463AbhHRHqJ (ORCPT ); Wed, 18 Aug 2021 03:46:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1629272736; x=1660808736; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=gea+CxC3NBudbiuJVzgvdOp/DWIOyvVEeeipiHZ4r9w=; b=m+V12C6jTODZP7SJ15QG+Yq6dAdn+wibur511I+xyjdnsKdtVcV1csnr OBw9CXG39v8LgRss0OJ2vrHBQ9k0V3maJV1CNXHmd0dup1ODE4X4Pg0zV MamArYW9DzygIhXz1uTTG2KhmcH/64JH28hshgif+WqcYw15hhZP+qlxm U=; X-IronPort-AV: E=Sophos;i="5.84,330,1620691200"; d="scan'208";a="153217483" Received: from pdx4-co-svc-p1-lb2-vlan3.amazon.com (HELO email-inbound-relay-1e-c7c08562.us-east-1.amazon.com) ([10.25.36.214]) by smtp-border-fw-9102.sea19.amazon.com with ESMTP; 18 Aug 2021 07:45:28 +0000 Received: from EX13D19EUA001.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1e-c7c08562.us-east-1.amazon.com (Postfix) with ESMTPS id AA82324B6B6; Wed, 18 Aug 2021 07:45:22 +0000 (UTC) Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D19EUA001.ant.amazon.com (10.43.165.74) with Microsoft SMTP Server (TLS) id 15.0.1497.23; Wed, 18 Aug 2021 07:45:21 +0000 Received: from 8c85908914bf.ant.amazon.com.com (10.1.213.15) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1497.23 via Frontend Transport; Wed, 18 Aug 2021 07:45:16 +0000 From: Gal Pressman To: Sumit Semwal , =?UTF-8?q?Christian=20K=C3=B6nig?= , Doug Ledford , Jason Gunthorpe CC: , , , , Oded Gabbay , Tomer Tayar , Yossi Leybovich , Alexander Matushevsky , Leon Romanovsky , Jianxin Xiong , Gal Pressman Subject: [RFC] Make use of non-dynamic dmabuf in RDMA Date: Wed, 18 Aug 2021 10:43:51 +0300 Message-ID: <20210818074352.29950-1-galpress@amazon.com> X-Mailer: git-send-email 2.32.0 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org Hey all, Currently, the RDMA subsystem can only work with dynamic dmabuf attachments, which requires the RDMA device to support on-demand-paging (ODP) which is not common on most devices (only supported by mlx5). While the dynamic requirement makes sense for certain GPUs, some devices (such as habanalabs) have device memory that is always "pinned" and do not need/use the move_notify operation. The motivation of this RFC is to use habanalabs as the dmabuf exporter, and EFA as the importer to allow for peer2peer access through libibverbs. This draft patch changes the dmabuf driver to differentiate between static/dynamic attachments by looking at the move_notify op instead of the importer_ops struct, and allowing the peer2peer flag to be enabled in case of a static exporter. Thanks Signed-off-by: Gal Pressman --- drivers/dma-buf/dma-buf.c | 5 +++-- drivers/infiniband/core/umem_dmabuf.c | 2 +- include/linux/dma-buf.h | 2 +- 3 files changed, 5 insertions(+), 4 deletions(-) diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c index 511fe0d217a0..e3faad8f492c 100644 --- a/drivers/dma-buf/dma-buf.c +++ b/drivers/dma-buf/dma-buf.c @@ -727,7 +727,8 @@ dma_buf_dynamic_attach(struct dma_buf *dmabuf, struct device *dev, if (WARN_ON(!dmabuf || !dev)) return ERR_PTR(-EINVAL); - if (WARN_ON(importer_ops && !importer_ops->move_notify)) + if (WARN_ON(importer_ops && !importer_ops->move_notify && + dma_buf_is_dynamic(attach->dmabuf))) return ERR_PTR(-EINVAL); attach = kzalloc(sizeof(*attach), GFP_KERNEL); @@ -1048,7 +1049,7 @@ void dma_buf_move_notify(struct dma_buf *dmabuf) dma_resv_assert_held(dmabuf->resv); list_for_each_entry(attach, &dmabuf->attachments, node) - if (attach->importer_ops) + if (attach->importer_ops && attach->importer_ops->move_notify) attach->importer_ops->move_notify(attach); } EXPORT_SYMBOL_GPL(dma_buf_move_notify); diff --git a/drivers/infiniband/core/umem_dmabuf.c b/drivers/infiniband/core/umem_dmabuf.c index c6e875619fac..c502ae828bd3 100644 --- a/drivers/infiniband/core/umem_dmabuf.c +++ b/drivers/infiniband/core/umem_dmabuf.c @@ -118,7 +118,7 @@ struct ib_umem_dmabuf *ib_umem_dmabuf_get(struct ib_device *device, if (check_add_overflow(offset, (unsigned long)size, &end)) return ret; - if (unlikely(!ops || !ops->move_notify)) + if (unlikely(!ops)) return ret; dmabuf = dma_buf_get(fd); diff --git a/include/linux/dma-buf.h b/include/linux/dma-buf.h index efdc56b9d95f..4b2e99012cb1 100644 --- a/include/linux/dma-buf.h +++ b/include/linux/dma-buf.h @@ -473,7 +473,7 @@ static inline bool dma_buf_is_dynamic(struct dma_buf *dmabuf) static inline bool dma_buf_attachment_is_dynamic(struct dma_buf_attachment *attach) { - return !!attach->importer_ops; + return !!attach->importer_ops->move_notify; } struct dma_buf_attachment *dma_buf_attach(struct dma_buf *dmabuf, -- 2.32.0