From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.8 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E66EEC7618F for ; Mon, 22 Jul 2019 23:10:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C97ED222FC for ; Mon, 22 Jul 2019 23:10:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730388AbfGVXKL (ORCPT ); Mon, 22 Jul 2019 19:10:11 -0400 Received: from ale.deltatee.com ([207.54.116.67]:40258 "EHLO ale.deltatee.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726130AbfGVXJN (ORCPT ); Mon, 22 Jul 2019 19:09:13 -0400 Received: from cgy1-donard.priv.deltatee.com ([172.16.1.31]) by ale.deltatee.com with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from ) id 1hphQb-0002ju-Dj; Mon, 22 Jul 2019 17:09:11 -0600 Received: from gunthorp by cgy1-donard.priv.deltatee.com with local (Exim 4.89) (envelope-from ) id 1hphQU-0001Qc-4e; Mon, 22 Jul 2019 17:09:02 -0600 From: Logan Gunthorpe To: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org Cc: Bjorn Helgaas , Christoph Hellwig , Christian Koenig , Jason Gunthorpe , Sagi Grimberg , Keith Busch , Jens Axboe , Dan Williams , Eric Pilmore , Stephen Bates , Logan Gunthorpe Date: Mon, 22 Jul 2019 17:08:45 -0600 Message-Id: <20190722230859.5436-1-logang@deltatee.com> X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-SA-Exim-Connect-IP: 172.16.1.31 X-SA-Exim-Rcpt-To: linux-nvme@lists.infradead.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org, bhelgaas@google.com, hch@lst.de, Christian.Koenig@amd.com, jgg@mellanox.com, sagi@grimberg.me, kbusch@kernel.org, axboe@fb.com, dan.j.williams@intel.com, epilmore@gigaio.com, sbates@raithlin.com, logang@deltatee.com X-SA-Exim-Mail-From: gunthorp@deltatee.com Subject: [PATCH 00/14] PCI/P2PDMA: Support transactions that hit the host bridge X-SA-Exim-Version: 4.2.1 (built Tue, 02 Aug 2016 21:08:31 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org As discussed on the list previously, in order to fully support the whitelist Christian added with the IOMMU, we must ensure that we map any buffer going through the IOMMU with an aprropriate dma_map call. This patchset accomplishes this by cleaning up the output of upstream_bridge_distance() to better indicate the mapping requirements, caching these requirements in an xarray, then looking them up at map time and applying the appropriate mapping method. After this patchset, it's possible to use the NVMe-of P2P support to transfer between devices without a switch on the whitelisted root complexes. A couple Intel device I have tested this on have also been added to the white list. Most of the changes are contained within the p2pdma.c, but there are a few minor touches to other subsystems, mostly to add support to call an unmap function. The final patch in this series demonstrates a possible pci_p2pdma_map_resource() function that I expect Christian will need but does not have any users at this time so I don't intend for it to be considered for merging. This patchset is based on 5.3-rc1 and a git branch is available here: https://github.com/sbates130272/linux-p2pmem/ p2pdma_rc_map_v1 -- Logan Gunthorpe (14): PCI/P2PDMA: Add constants for not-supported result upstream_bridge_distance() PCI/P2PDMA: Factor out __upstream_bridge_distance() PCI/P2PDMA: Apply host bridge white list for ACS PCI/P2PDMA: Cache the result of upstream_bridge_distance() PCI/P2PDMA: Factor out host_bridge_whitelist() PCI/P2PDMA: Add whitelist support for Intel Host Bridges PCI/P2PDMA: Add the provider's pci_dev to the dev_pgmap struct PCI/P2PDMA: Add attrs argument to pci_p2pdma_map_sg() PCI/P2PDMA: Introduce pci_p2pdma_unmap_sg() PCI/P2PDMA: Factor out __pci_p2pdma_map_sg() PCI/P2PDMA: dma_map P2PDMA map requests that traverse the host bridge PCI/P2PDMA: No longer require no-mmu for host bridge whitelist PCI/P2PDMA: Update documentation for pci_p2pdma_distance_many() PCI/P2PDMA: Introduce pci_p2pdma_[un]map_resource() drivers/infiniband/core/rw.c | 6 +- drivers/nvme/host/pci.c | 10 +- drivers/pci/p2pdma.c | 400 +++++++++++++++++++++++++++-------- include/linux/memremap.h | 1 + include/linux/pci-p2pdma.h | 28 ++- 5 files changed, 341 insertions(+), 104 deletions(-) -- 2.20.1 From mboxrd@z Thu Jan 1 00:00:00 1970 From: logang@deltatee.com (Logan Gunthorpe) Date: Mon, 22 Jul 2019 17:08:45 -0600 Subject: [PATCH 00/14] PCI/P2PDMA: Support transactions that hit the host bridge Message-ID: <20190722230859.5436-1-logang@deltatee.com> As discussed on the list previously, in order to fully support the whitelist Christian added with the IOMMU, we must ensure that we map any buffer going through the IOMMU with an aprropriate dma_map call. This patchset accomplishes this by cleaning up the output of upstream_bridge_distance() to better indicate the mapping requirements, caching these requirements in an xarray, then looking them up at map time and applying the appropriate mapping method. After this patchset, it's possible to use the NVMe-of P2P support to transfer between devices without a switch on the whitelisted root complexes. A couple Intel device I have tested this on have also been added to the white list. Most of the changes are contained within the p2pdma.c, but there are a few minor touches to other subsystems, mostly to add support to call an unmap function. The final patch in this series demonstrates a possible pci_p2pdma_map_resource() function that I expect Christian will need but does not have any users at this time so I don't intend for it to be considered for merging. This patchset is based on 5.3-rc1 and a git branch is available here: https://github.com/sbates130272/linux-p2pmem/ p2pdma_rc_map_v1 -- Logan Gunthorpe (14): PCI/P2PDMA: Add constants for not-supported result upstream_bridge_distance() PCI/P2PDMA: Factor out __upstream_bridge_distance() PCI/P2PDMA: Apply host bridge white list for ACS PCI/P2PDMA: Cache the result of upstream_bridge_distance() PCI/P2PDMA: Factor out host_bridge_whitelist() PCI/P2PDMA: Add whitelist support for Intel Host Bridges PCI/P2PDMA: Add the provider's pci_dev to the dev_pgmap struct PCI/P2PDMA: Add attrs argument to pci_p2pdma_map_sg() PCI/P2PDMA: Introduce pci_p2pdma_unmap_sg() PCI/P2PDMA: Factor out __pci_p2pdma_map_sg() PCI/P2PDMA: dma_map P2PDMA map requests that traverse the host bridge PCI/P2PDMA: No longer require no-mmu for host bridge whitelist PCI/P2PDMA: Update documentation for pci_p2pdma_distance_many() PCI/P2PDMA: Introduce pci_p2pdma_[un]map_resource() drivers/infiniband/core/rw.c | 6 +- drivers/nvme/host/pci.c | 10 +- drivers/pci/p2pdma.c | 400 +++++++++++++++++++++++++++-------- include/linux/memremap.h | 1 + include/linux/pci-p2pdma.h | 28 ++- 5 files changed, 341 insertions(+), 104 deletions(-) -- 2.20.1