From: Logan Gunthorpe <logang@deltatee.com> To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: "Stephen Bates" <sbates@raithlin.com>, "Christoph Hellwig" <hch@lst.de>, "Dan Williams" <dan.j.williams@intel.com>, "Jason Gunthorpe" <jgg@ziepe.ca>, "Christian König" <christian.koenig@amd.com>, "John Hubbard" <jhubbard@nvidia.com>, "Don Dutile" <ddutile@redhat.com>, "Matthew Wilcox" <willy@infradead.org>, "Daniel Vetter" <daniel.vetter@ffwll.ch>, "Jakowski Andrzej" <andrzej.jakowski@intel.com>, "Minturn Dave B" <dave.b.minturn@intel.com>, "Jason Ekstrand" <jason@jlekstrand.net>, "Dave Hansen" <dave.hansen@linux.intel.com>, "Xiong Jianxin" <jianxin.xiong@intel.com>, "Bjorn Helgaas" <helgaas@kernel.org>, "Ira Weiny" <ira.weiny@intel.com>, "Robin Murphy" <robin.murphy@arm.com>, "Martin Oliveira" <martin.oliveira@eideticom.com>, "Chaitanya Kulkarni" <ckulkarnilinux@gmail.com>, "Logan Gunthorpe" <logang@deltatee.com> Subject: [PATCH v4 00/23] Userspace P2PDMA with O_DIRECT NVMe devices Date: Wed, 17 Nov 2021 14:53:47 -0700 [thread overview] Message-ID: <20211117215410.3695-1-logang@deltatee.com> (raw) Hi, This patchset continues my work to add userspace P2PDMA access using O_DIRECT NVMe devices. This posting fixes a lot of issues that were addresed in the last posting, which is here[1]. The patchset enables userspace P2PDMA by allowing userspace to mmap() allocated chunks of the CMB. The resulting VMA can be passed only to O_DIRECT IO on NVMe backed files or block devices. A flag is added to GUP() in Patch <>, then Patches <> through <> wire this flag up based on whether the block queue indicates P2PDMA support. Patches <> through <> enable the CMB to be mapped into userspace by mmaping the nvme char device. This is relatively straightforward, however the one significant problem is that, presently, pci_p2pdma_map_sg() requires a homogeneous SGL with all P2PDMA pages or all regular pages. Enhancing GUP to support enforcing this rule would require a huge hack that I don't expect would be all that pallatable. So the first 13 patches add support for P2PDMA pages to dma_map_sg[table]() to the dma-direct and dma-iommu implementations. Thus systems without an IOMMU plus Intel and AMD IOMMUs are supported. (Other IOMMU implementations would then be unsupported, notably ARM and PowerPC but support would be added when they convert to dma-iommu). dma_map_sgtable() is preferred when dealing with P2PDMA memory as it will return -EREMOTEIO when the DMA device cannot map specific P2PDMA pages based on the existing rules in calc_map_type_and_dist(). The other issue is dma_unmap_sg() needs a flag to determine whether a given dma_addr_t was mapped regularly or as a PCI bus address. To allow this, a third flag is added to the page_link field in struct scatterlist. This effectively means support for P2PDMA will now depend on CONFIG_64BIT. Feedback welcome. This series is based on v5.16-rc1. A git branch is available here: https://github.com/sbates130272/linux-p2pmem/ p2pdma_user_cmb_v4 Thanks, Logan [1] https://lore.kernel.org/all/20210916234100.122368-1-logang@deltatee.com -- Changes since v3: - Add some comment and commit message cleanups I had missed for v3, also moved the prototypes for some of the p2pdma helpers to dma-map-ops.h (which I missed in v3 and was suggested in v2). - Add separate cleanup patch for scatterlist.h and change the macros to functions. (Suggested by Chaitanya and Jason, respectively) - Rename sg_dma_mark_pci_p2pdma() and sg_is_dma_pci_p2pdma() to sg_dma_mark_bus_address() and sg_is_dma_bus_address() which is a more generic name (As requested by Jason) - Fixes to some comments and commit messages as suggested by Bjorn and Jason. - Ensure swiotlb is not used with P2PDMA pages. (Per Jason) - The sgtable coversion in RDMA was split out and sent upstream separately, the new patch is only the removal. (Per Jason) - Moved the FOLL_PCI_P2PDMA check outside of get_dev_pagemap() as Jason suggested this will be removed in the near term. - Add two patches to ensure that zone device pages with different pgmaps are never merged in the block layer or sg_alloc_append_table_from_pages() (Per Jason) - Ensure synchronize_rcu() or call_rcu() is used before returning pages to the genalloc. (Jason pointed out that pages are not gauranteed to be unused in all architectures until at least after an RCU grace period, and that synchronize_rcu() was likely too slow to use in the vma close operation. - Collected Acks and Reviews by Bjorn, Jason and Max. Logan Gunthorpe (23): lib/scatterlist: cleanup macros into static inline functions lib/scatterlist: add flag for indicating P2PDMA segments in an SGL PCI/P2PDMA: Attempt to set map_type if it has not been set PCI/P2PDMA: Expose pci_p2pdma_map_type() PCI/P2PDMA: Introduce helpers for dma_map_sg implementations dma-mapping: allow EREMOTEIO return code for P2PDMA transfers dma-direct: support PCI P2PDMA pages in dma-direct map_sg dma-mapping: add flags to dma_map_ops to indicate PCI P2PDMA support iommu/dma: support PCI P2PDMA pages in dma-iommu map_sg nvme-pci: check DMA ops when indicating support for PCI P2PDMA nvme-pci: convert to using dma_map_sgtable() RDMA/core: introduce ib_dma_pci_p2p_dma_supported() RDMA/rw: drop pci_p2pdma_[un]map_sg() PCI/P2PDMA: Remove pci_p2pdma_[un]map_sg() mm: introduce FOLL_PCI_P2PDMA to gate getting PCI P2PDMA pages iov_iter: introduce iov_iter_get_pages_[alloc_]flags() block: add check when merging zone device pages lib/scatterlist: add check when merging zone device pages block: set FOLL_PCI_P2PDMA in __bio_iov_iter_get_pages() block: set FOLL_PCI_P2PDMA in bio_map_user_iov() mm: use custom page_free for P2PDMA pages PCI/P2PDMA: Introduce pci_mmap_p2pmem() nvme-pci: allow mmaping the CMB in userspace block/bio.c | 10 +- block/blk-map.c | 7 +- drivers/infiniband/core/rw.c | 45 +--- drivers/iommu/dma-iommu.c | 67 +++++- drivers/nvme/host/core.c | 18 +- drivers/nvme/host/nvme.h | 4 +- drivers/nvme/host/pci.c | 98 ++++---- drivers/nvme/target/rdma.c | 2 +- drivers/pci/Kconfig | 6 + drivers/pci/p2pdma.c | 441 ++++++++++++++++++++++++++++++----- include/linux/dma-map-ops.h | 76 ++++++ include/linux/dma-mapping.h | 5 + include/linux/mm.h | 25 ++ include/linux/pci-p2pdma.h | 38 +-- include/linux/scatterlist.h | 71 +++++- include/linux/uio.h | 21 +- include/rdma/ib_verbs.h | 11 + include/uapi/linux/magic.h | 1 + kernel/dma/Kconfig | 10 + kernel/dma/direct.c | 43 +++- kernel/dma/direct.h | 7 +- kernel/dma/mapping.c | 22 +- lib/iov_iter.c | 15 +- lib/scatterlist.c | 25 +- mm/gup.c | 22 +- mm/memremap.c | 12 +- 26 files changed, 882 insertions(+), 220 deletions(-) base-commit: fa55b7dcdc43c1aa1ba12bca9d2dd4318c2a0dbf -- 2.30.2
WARNING: multiple messages have this Message-ID (diff)
From: Logan Gunthorpe <logang@deltatee.com> To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: "Minturn Dave B" <dave.b.minturn@intel.com>, "Martin Oliveira" <martin.oliveira@eideticom.com>, "Ira Weiny" <ira.weiny@intel.com>, "John Hubbard" <jhubbard@nvidia.com>, "Dave Hansen" <dave.hansen@linux.intel.com>, "Robin Murphy" <robin.murphy@arm.com>, "Matthew Wilcox" <willy@infradead.org>, "Christian König" <christian.koenig@amd.com>, "Jason Gunthorpe" <jgg@ziepe.ca>, "Logan Gunthorpe" <logang@deltatee.com>, "Chaitanya Kulkarni" <ckulkarnilinux@gmail.com>, "Jason Ekstrand" <jason@jlekstrand.net>, "Daniel Vetter" <daniel.vetter@ffwll.ch>, "Bjorn Helgaas" <helgaas@kernel.org>, "Dan Williams" <dan.j.williams@intel.com>, "Stephen Bates" <sbates@raithlin.com>, "Jakowski Andrzej" <andrzej.jakowski@intel.com>, "Christoph Hellwig" <hch@lst.de>, "Xiong Jianxin" <jianxin.xiong@intel.com> Subject: [PATCH v4 00/23] Userspace P2PDMA with O_DIRECT NVMe devices Date: Wed, 17 Nov 2021 14:53:47 -0700 [thread overview] Message-ID: <20211117215410.3695-1-logang@deltatee.com> (raw) Hi, This patchset continues my work to add userspace P2PDMA access using O_DIRECT NVMe devices. This posting fixes a lot of issues that were addresed in the last posting, which is here[1]. The patchset enables userspace P2PDMA by allowing userspace to mmap() allocated chunks of the CMB. The resulting VMA can be passed only to O_DIRECT IO on NVMe backed files or block devices. A flag is added to GUP() in Patch <>, then Patches <> through <> wire this flag up based on whether the block queue indicates P2PDMA support. Patches <> through <> enable the CMB to be mapped into userspace by mmaping the nvme char device. This is relatively straightforward, however the one significant problem is that, presently, pci_p2pdma_map_sg() requires a homogeneous SGL with all P2PDMA pages or all regular pages. Enhancing GUP to support enforcing this rule would require a huge hack that I don't expect would be all that pallatable. So the first 13 patches add support for P2PDMA pages to dma_map_sg[table]() to the dma-direct and dma-iommu implementations. Thus systems without an IOMMU plus Intel and AMD IOMMUs are supported. (Other IOMMU implementations would then be unsupported, notably ARM and PowerPC but support would be added when they convert to dma-iommu). dma_map_sgtable() is preferred when dealing with P2PDMA memory as it will return -EREMOTEIO when the DMA device cannot map specific P2PDMA pages based on the existing rules in calc_map_type_and_dist(). The other issue is dma_unmap_sg() needs a flag to determine whether a given dma_addr_t was mapped regularly or as a PCI bus address. To allow this, a third flag is added to the page_link field in struct scatterlist. This effectively means support for P2PDMA will now depend on CONFIG_64BIT. Feedback welcome. This series is based on v5.16-rc1. A git branch is available here: https://github.com/sbates130272/linux-p2pmem/ p2pdma_user_cmb_v4 Thanks, Logan [1] https://lore.kernel.org/all/20210916234100.122368-1-logang@deltatee.com -- Changes since v3: - Add some comment and commit message cleanups I had missed for v3, also moved the prototypes for some of the p2pdma helpers to dma-map-ops.h (which I missed in v3 and was suggested in v2). - Add separate cleanup patch for scatterlist.h and change the macros to functions. (Suggested by Chaitanya and Jason, respectively) - Rename sg_dma_mark_pci_p2pdma() and sg_is_dma_pci_p2pdma() to sg_dma_mark_bus_address() and sg_is_dma_bus_address() which is a more generic name (As requested by Jason) - Fixes to some comments and commit messages as suggested by Bjorn and Jason. - Ensure swiotlb is not used with P2PDMA pages. (Per Jason) - The sgtable coversion in RDMA was split out and sent upstream separately, the new patch is only the removal. (Per Jason) - Moved the FOLL_PCI_P2PDMA check outside of get_dev_pagemap() as Jason suggested this will be removed in the near term. - Add two patches to ensure that zone device pages with different pgmaps are never merged in the block layer or sg_alloc_append_table_from_pages() (Per Jason) - Ensure synchronize_rcu() or call_rcu() is used before returning pages to the genalloc. (Jason pointed out that pages are not gauranteed to be unused in all architectures until at least after an RCU grace period, and that synchronize_rcu() was likely too slow to use in the vma close operation. - Collected Acks and Reviews by Bjorn, Jason and Max. Logan Gunthorpe (23): lib/scatterlist: cleanup macros into static inline functions lib/scatterlist: add flag for indicating P2PDMA segments in an SGL PCI/P2PDMA: Attempt to set map_type if it has not been set PCI/P2PDMA: Expose pci_p2pdma_map_type() PCI/P2PDMA: Introduce helpers for dma_map_sg implementations dma-mapping: allow EREMOTEIO return code for P2PDMA transfers dma-direct: support PCI P2PDMA pages in dma-direct map_sg dma-mapping: add flags to dma_map_ops to indicate PCI P2PDMA support iommu/dma: support PCI P2PDMA pages in dma-iommu map_sg nvme-pci: check DMA ops when indicating support for PCI P2PDMA nvme-pci: convert to using dma_map_sgtable() RDMA/core: introduce ib_dma_pci_p2p_dma_supported() RDMA/rw: drop pci_p2pdma_[un]map_sg() PCI/P2PDMA: Remove pci_p2pdma_[un]map_sg() mm: introduce FOLL_PCI_P2PDMA to gate getting PCI P2PDMA pages iov_iter: introduce iov_iter_get_pages_[alloc_]flags() block: add check when merging zone device pages lib/scatterlist: add check when merging zone device pages block: set FOLL_PCI_P2PDMA in __bio_iov_iter_get_pages() block: set FOLL_PCI_P2PDMA in bio_map_user_iov() mm: use custom page_free for P2PDMA pages PCI/P2PDMA: Introduce pci_mmap_p2pmem() nvme-pci: allow mmaping the CMB in userspace block/bio.c | 10 +- block/blk-map.c | 7 +- drivers/infiniband/core/rw.c | 45 +--- drivers/iommu/dma-iommu.c | 67 +++++- drivers/nvme/host/core.c | 18 +- drivers/nvme/host/nvme.h | 4 +- drivers/nvme/host/pci.c | 98 ++++---- drivers/nvme/target/rdma.c | 2 +- drivers/pci/Kconfig | 6 + drivers/pci/p2pdma.c | 441 ++++++++++++++++++++++++++++++----- include/linux/dma-map-ops.h | 76 ++++++ include/linux/dma-mapping.h | 5 + include/linux/mm.h | 25 ++ include/linux/pci-p2pdma.h | 38 +-- include/linux/scatterlist.h | 71 +++++- include/linux/uio.h | 21 +- include/rdma/ib_verbs.h | 11 + include/uapi/linux/magic.h | 1 + kernel/dma/Kconfig | 10 + kernel/dma/direct.c | 43 +++- kernel/dma/direct.h | 7 +- kernel/dma/mapping.c | 22 +- lib/iov_iter.c | 15 +- lib/scatterlist.c | 25 +- mm/gup.c | 22 +- mm/memremap.c | 12 +- 26 files changed, 882 insertions(+), 220 deletions(-) base-commit: fa55b7dcdc43c1aa1ba12bca9d2dd4318c2a0dbf -- 2.30.2 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
next reply other threads:[~2021-11-17 21:54 UTC|newest] Thread overview: 80+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-11-17 21:53 Logan Gunthorpe [this message] 2021-11-17 21:53 ` [PATCH v4 00/23] Userspace P2PDMA with O_DIRECT NVMe devices Logan Gunthorpe 2021-11-17 21:53 ` [PATCH v4 01/23] lib/scatterlist: cleanup macros into static inline functions Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-12-13 21:51 ` Chaitanya Kulkarni 2021-12-13 21:51 ` Chaitanya Kulkarni via iommu 2021-12-21 9:00 ` Christoph Hellwig 2021-12-21 9:00 ` Christoph Hellwig 2021-12-21 17:23 ` Logan Gunthorpe 2021-12-21 17:23 ` Logan Gunthorpe 2021-12-22 8:22 ` Christoph Hellwig 2021-12-22 8:22 ` Christoph Hellwig 2021-11-17 21:53 ` [PATCH v4 02/23] lib/scatterlist: add flag for indicating P2PDMA segments in an SGL Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-12-13 21:55 ` Chaitanya Kulkarni 2021-12-13 21:55 ` Chaitanya Kulkarni via iommu 2021-12-21 9:02 ` Christoph Hellwig 2021-12-21 9:02 ` Christoph Hellwig 2021-11-17 21:53 ` [PATCH v4 03/23] PCI/P2PDMA: Attempt to set map_type if it has not been set Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-12-13 22:00 ` Chaitanya Kulkarni 2021-12-13 22:00 ` Chaitanya Kulkarni via iommu 2021-11-17 21:53 ` [PATCH v4 04/23] PCI/P2PDMA: Expose pci_p2pdma_map_type() Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-12-13 22:05 ` Chaitanya Kulkarni 2021-12-13 22:05 ` Chaitanya Kulkarni via iommu 2021-11-17 21:53 ` [PATCH v4 05/23] PCI/P2PDMA: Introduce helpers for dma_map_sg implementations Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-11-17 21:53 ` [PATCH v4 06/23] dma-mapping: allow EREMOTEIO return code for P2PDMA transfers Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-11-17 21:53 ` [PATCH v4 07/23] dma-direct: support PCI P2PDMA pages in dma-direct map_sg Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-11-17 21:53 ` [PATCH v4 08/23] dma-mapping: add flags to dma_map_ops to indicate PCI P2PDMA support Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-11-17 21:53 ` [PATCH v4 09/23] iommu/dma: support PCI P2PDMA pages in dma-iommu map_sg Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-11-17 21:53 ` [PATCH v4 10/23] nvme-pci: check DMA ops when indicating support for PCI P2PDMA Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-12-13 22:10 ` Chaitanya Kulkarni 2021-12-13 22:10 ` Chaitanya Kulkarni via iommu 2021-11-17 21:53 ` [PATCH v4 11/23] nvme-pci: convert to using dma_map_sgtable() Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-12-13 22:21 ` Chaitanya Kulkarni 2021-12-13 22:21 ` Chaitanya Kulkarni via iommu 2021-12-13 22:28 ` Logan Gunthorpe 2021-12-13 22:28 ` Logan Gunthorpe 2021-11-17 21:53 ` [PATCH v4 12/23] RDMA/core: introduce ib_dma_pci_p2p_dma_supported() Logan Gunthorpe 2021-11-17 21:53 ` Logan Gunthorpe 2021-11-17 21:54 ` [PATCH v4 13/23] RDMA/rw: drop pci_p2pdma_[un]map_sg() Logan Gunthorpe 2021-11-17 21:54 ` Logan Gunthorpe 2021-11-17 21:54 ` [PATCH v4 14/23] PCI/P2PDMA: Remove pci_p2pdma_[un]map_sg() Logan Gunthorpe 2021-11-17 21:54 ` Logan Gunthorpe 2021-11-17 21:54 ` [PATCH v4 15/23] mm: introduce FOLL_PCI_P2PDMA to gate getting PCI P2PDMA pages Logan Gunthorpe 2021-11-17 21:54 ` Logan Gunthorpe 2021-11-17 21:54 ` [PATCH v4 16/23] iov_iter: introduce iov_iter_get_pages_[alloc_]flags() Logan Gunthorpe 2021-11-17 21:54 ` Logan Gunthorpe 2021-12-21 9:04 ` Christoph Hellwig 2021-12-21 9:04 ` Christoph Hellwig 2021-11-17 21:54 ` [PATCH v4 17/23] block: add check when merging zone device pages Logan Gunthorpe 2021-11-17 21:54 ` Logan Gunthorpe 2021-12-21 9:05 ` Christoph Hellwig 2021-12-21 9:05 ` Christoph Hellwig 2021-11-17 21:54 ` [PATCH v4 18/23] lib/scatterlist: " Logan Gunthorpe 2021-11-17 21:54 ` Logan Gunthorpe 2021-11-17 21:54 ` [PATCH v4 19/23] block: set FOLL_PCI_P2PDMA in __bio_iov_iter_get_pages() Logan Gunthorpe 2021-11-17 21:54 ` Logan Gunthorpe 2021-11-17 21:54 ` [PATCH v4 20/23] block: set FOLL_PCI_P2PDMA in bio_map_user_iov() Logan Gunthorpe 2021-11-17 21:54 ` Logan Gunthorpe 2021-11-17 21:54 ` [PATCH v4 21/23] mm: use custom page_free for P2PDMA pages Logan Gunthorpe 2021-11-17 21:54 ` Logan Gunthorpe 2021-12-21 9:06 ` Christoph Hellwig 2021-12-21 9:06 ` Christoph Hellwig 2021-12-21 17:27 ` Logan Gunthorpe 2021-12-21 17:27 ` Logan Gunthorpe 2021-11-17 21:54 ` [PATCH v4 22/23] PCI/P2PDMA: Introduce pci_mmap_p2pmem() Logan Gunthorpe 2021-11-17 21:54 ` Logan Gunthorpe 2021-11-17 21:54 ` [PATCH v4 23/23] nvme-pci: allow mmaping the CMB in userspace Logan Gunthorpe 2021-11-17 21:54 ` Logan Gunthorpe 2021-12-21 9:07 ` Christoph Hellwig 2021-12-21 9:07 ` Christoph Hellwig
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20211117215410.3695-1-logang@deltatee.com \ --to=logang@deltatee.com \ --cc=andrzej.jakowski@intel.com \ --cc=christian.koenig@amd.com \ --cc=ckulkarnilinux@gmail.com \ --cc=dan.j.williams@intel.com \ --cc=daniel.vetter@ffwll.ch \ --cc=dave.b.minturn@intel.com \ --cc=dave.hansen@linux.intel.com \ --cc=ddutile@redhat.com \ --cc=hch@lst.de \ --cc=helgaas@kernel.org \ --cc=iommu@lists.linux-foundation.org \ --cc=ira.weiny@intel.com \ --cc=jason@jlekstrand.net \ --cc=jgg@ziepe.ca \ --cc=jhubbard@nvidia.com \ --cc=jianxin.xiong@intel.com \ --cc=linux-block@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linux-nvme@lists.infradead.org \ --cc=linux-pci@vger.kernel.org \ --cc=martin.oliveira@eideticom.com \ --cc=robin.murphy@arm.com \ --cc=sbates@raithlin.com \ --cc=willy@infradead.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.