From: Logan Gunthorpe <logang@deltatee.com> To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: "Stephen Bates" <sbates@raithlin.com>, "Christoph Hellwig" <hch@lst.de>, "Dan Williams" <dan.j.williams@intel.com>, "Jason Gunthorpe" <jgg@ziepe.ca>, "Christian König" <christian.koenig@amd.com>, "John Hubbard" <jhubbard@nvidia.com>, "Don Dutile" <ddutile@redhat.com>, "Matthew Wilcox" <willy@infradead.org>, "Daniel Vetter" <daniel.vetter@ffwll.ch>, "Jakowski Andrzej" <andrzej.jakowski@intel.com>, "Minturn Dave B" <dave.b.minturn@intel.com>, "Jason Ekstrand" <jason@jlekstrand.net>, "Dave Hansen" <dave.hansen@linux.intel.com>, "Xiong Jianxin" <jianxin.xiong@intel.com>, "Bjorn Helgaas" <helgaas@kernel.org>, "Ira Weiny" <ira.weiny@intel.com>, "Robin Murphy" <robin.murphy@arm.com>, "Martin Oliveira" <martin.oliveira@eideticom.com>, "Chaitanya Kulkarni" <ckulkarnilinux@gmail.com>, "Ralph Campbell" <rcampbell@nvidia.com>, "Logan Gunthorpe" <logang@deltatee.com> Subject: [PATCH v5 08/24] dma-direct: support PCI P2PDMA pages in dma-direct map_sg Date: Thu, 27 Jan 2022 17:25:58 -0700 [thread overview] Message-ID: <20220128002614.6136-9-logang@deltatee.com> (raw) In-Reply-To: <20220128002614.6136-1-logang@deltatee.com> Add PCI P2PDMA support for dma_direct_map_sg() so that it can map PCI P2PDMA pages directly without a hack in the callers. This allows for heterogeneous SGLs that contain both P2PDMA and regular pages. A P2PDMA page may have three possible outcomes when being mapped: 1) If the data path between the two devices doesn't go through the root port, then it should be mapped with a PCI bus address 2) If the data path goes through the host bridge, it should be mapped normally, as though it were a CPU physical address 3) It is not possible for the two devices to communicate and thus the mapping operation should fail (and it will return -EREMOTEIO). SGL segments that contain PCI bus addresses are marked with sg_dma_mark_pci_p2pdma() and are ignored when unmapped. P2PDMA mappings are also failed if swiotlb needs to be used on the mapping. Signed-off-by: Logan Gunthorpe <logang@deltatee.com> --- kernel/dma/direct.c | 43 +++++++++++++++++++++++++++++++++++++------ kernel/dma/direct.h | 7 ++++++- 2 files changed, 43 insertions(+), 7 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 50f48e9e4598..975df5f3aaf9 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -461,29 +461,60 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, arch_sync_dma_for_cpu_all(); } +/* + * Unmaps segments, except for ones marked as pci_p2pdma which do not + * require any further action as they contain a bus address. + */ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { struct scatterlist *sg; int i; - for_each_sg(sgl, sg, nents, i) - dma_direct_unmap_page(dev, sg->dma_address, sg_dma_len(sg), dir, - attrs); + for_each_sg(sgl, sg, nents, i) { + if (sg_is_dma_bus_address(sg)) + sg_dma_unmark_bus_address(sg); + else + dma_direct_unmap_page(dev, sg->dma_address, + sg_dma_len(sg), dir, attrs); + } } #endif int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { - int i; + struct pci_p2pdma_map_state p2pdma_state = {}; + enum pci_p2pdma_map_type map; struct scatterlist *sg; + int i, ret; for_each_sg(sgl, sg, nents, i) { + if (is_pci_p2pdma_page(sg_page(sg))) { + map = pci_p2pdma_map_segment(&p2pdma_state, dev, sg); + switch (map) { + case PCI_P2PDMA_MAP_BUS_ADDR: + continue; + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + /* + * Any P2P mapping that traverses the PCI + * host bridge must be mapped with CPU physical + * address and not PCI bus addresses. This is + * done with dma_direct_map_page() below. + */ + break; + default: + ret = -EREMOTEIO; + goto out_unmap; + } + } + sg->dma_address = dma_direct_map_page(dev, sg_page(sg), sg->offset, sg->length, dir, attrs); - if (sg->dma_address == DMA_MAPPING_ERROR) + if (sg->dma_address == DMA_MAPPING_ERROR) { + ret = -EIO; goto out_unmap; + } sg_dma_len(sg) = sg->length; } @@ -491,7 +522,7 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, out_unmap: dma_direct_unmap_sg(dev, sgl, i, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC); - return -EIO; + return ret; } dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index 4632b0f4f72e..a33152d79069 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -87,10 +87,15 @@ static inline dma_addr_t dma_direct_map_page(struct device *dev, phys_addr_t phys = page_to_phys(page) + offset; dma_addr_t dma_addr = phys_to_dma(dev, phys); - if (is_swiotlb_force_bounce(dev)) + if (is_swiotlb_force_bounce(dev)) { + if (is_pci_p2pdma_page(page)) + return DMA_MAPPING_ERROR; return swiotlb_map(dev, phys, size, dir, attrs); + } if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (is_pci_p2pdma_page(page)) + return DMA_MAPPING_ERROR; if (swiotlb_force != SWIOTLB_NO_FORCE) return swiotlb_map(dev, phys, size, dir, attrs); -- 2.30.2
WARNING: multiple messages have this Message-ID (diff)
From: Logan Gunthorpe <logang@deltatee.com> To: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org Cc: "Minturn Dave B" <dave.b.minturn@intel.com>, "Martin Oliveira" <martin.oliveira@eideticom.com>, "Ralph Campbell" <rcampbell@nvidia.com>, "Ira Weiny" <ira.weiny@intel.com>, "John Hubbard" <jhubbard@nvidia.com>, "Dave Hansen" <dave.hansen@linux.intel.com>, "Robin Murphy" <robin.murphy@arm.com>, "Matthew Wilcox" <willy@infradead.org>, "Christian König" <christian.koenig@amd.com>, "Jason Gunthorpe" <jgg@ziepe.ca>, "Logan Gunthorpe" <logang@deltatee.com>, "Chaitanya Kulkarni" <ckulkarnilinux@gmail.com>, "Jason Ekstrand" <jason@jlekstrand.net>, "Daniel Vetter" <daniel.vetter@ffwll.ch>, "Bjorn Helgaas" <helgaas@kernel.org>, "Dan Williams" <dan.j.williams@intel.com>, "Stephen Bates" <sbates@raithlin.com>, "Jakowski Andrzej" <andrzej.jakowski@intel.com>, "Christoph Hellwig" <hch@lst.de>, "Xiong Jianxin" <jianxin.xiong@intel.com> Subject: [PATCH v5 08/24] dma-direct: support PCI P2PDMA pages in dma-direct map_sg Date: Thu, 27 Jan 2022 17:25:58 -0700 [thread overview] Message-ID: <20220128002614.6136-9-logang@deltatee.com> (raw) In-Reply-To: <20220128002614.6136-1-logang@deltatee.com> Add PCI P2PDMA support for dma_direct_map_sg() so that it can map PCI P2PDMA pages directly without a hack in the callers. This allows for heterogeneous SGLs that contain both P2PDMA and regular pages. A P2PDMA page may have three possible outcomes when being mapped: 1) If the data path between the two devices doesn't go through the root port, then it should be mapped with a PCI bus address 2) If the data path goes through the host bridge, it should be mapped normally, as though it were a CPU physical address 3) It is not possible for the two devices to communicate and thus the mapping operation should fail (and it will return -EREMOTEIO). SGL segments that contain PCI bus addresses are marked with sg_dma_mark_pci_p2pdma() and are ignored when unmapped. P2PDMA mappings are also failed if swiotlb needs to be used on the mapping. Signed-off-by: Logan Gunthorpe <logang@deltatee.com> --- kernel/dma/direct.c | 43 +++++++++++++++++++++++++++++++++++++------ kernel/dma/direct.h | 7 ++++++- 2 files changed, 43 insertions(+), 7 deletions(-) diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 50f48e9e4598..975df5f3aaf9 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -461,29 +461,60 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, arch_sync_dma_for_cpu_all(); } +/* + * Unmaps segments, except for ones marked as pci_p2pdma which do not + * require any further action as they contain a bus address. + */ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { struct scatterlist *sg; int i; - for_each_sg(sgl, sg, nents, i) - dma_direct_unmap_page(dev, sg->dma_address, sg_dma_len(sg), dir, - attrs); + for_each_sg(sgl, sg, nents, i) { + if (sg_is_dma_bus_address(sg)) + sg_dma_unmark_bus_address(sg); + else + dma_direct_unmap_page(dev, sg->dma_address, + sg_dma_len(sg), dir, attrs); + } } #endif int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { - int i; + struct pci_p2pdma_map_state p2pdma_state = {}; + enum pci_p2pdma_map_type map; struct scatterlist *sg; + int i, ret; for_each_sg(sgl, sg, nents, i) { + if (is_pci_p2pdma_page(sg_page(sg))) { + map = pci_p2pdma_map_segment(&p2pdma_state, dev, sg); + switch (map) { + case PCI_P2PDMA_MAP_BUS_ADDR: + continue; + case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE: + /* + * Any P2P mapping that traverses the PCI + * host bridge must be mapped with CPU physical + * address and not PCI bus addresses. This is + * done with dma_direct_map_page() below. + */ + break; + default: + ret = -EREMOTEIO; + goto out_unmap; + } + } + sg->dma_address = dma_direct_map_page(dev, sg_page(sg), sg->offset, sg->length, dir, attrs); - if (sg->dma_address == DMA_MAPPING_ERROR) + if (sg->dma_address == DMA_MAPPING_ERROR) { + ret = -EIO; goto out_unmap; + } sg_dma_len(sg) = sg->length; } @@ -491,7 +522,7 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, out_unmap: dma_direct_unmap_sg(dev, sgl, i, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC); - return -EIO; + return ret; } dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, diff --git a/kernel/dma/direct.h b/kernel/dma/direct.h index 4632b0f4f72e..a33152d79069 100644 --- a/kernel/dma/direct.h +++ b/kernel/dma/direct.h @@ -87,10 +87,15 @@ static inline dma_addr_t dma_direct_map_page(struct device *dev, phys_addr_t phys = page_to_phys(page) + offset; dma_addr_t dma_addr = phys_to_dma(dev, phys); - if (is_swiotlb_force_bounce(dev)) + if (is_swiotlb_force_bounce(dev)) { + if (is_pci_p2pdma_page(page)) + return DMA_MAPPING_ERROR; return swiotlb_map(dev, phys, size, dir, attrs); + } if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (is_pci_p2pdma_page(page)) + return DMA_MAPPING_ERROR; if (swiotlb_force != SWIOTLB_NO_FORCE) return swiotlb_map(dev, phys, size, dir, attrs); -- 2.30.2 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu
next prev parent reply other threads:[~2022-01-28 0:26 UTC|newest] Thread overview: 72+ messages / expand[flat|nested] mbox.gz Atom feed top 2022-01-28 0:25 [PATCH v5 00/24] Userspace P2PDMA with O_DIRECT NVMe devices Logan Gunthorpe 2022-01-28 0:25 ` Logan Gunthorpe 2022-01-28 0:25 ` [PATCH v5 01/24] ext4/xfs: add page refcount helper Logan Gunthorpe 2022-01-28 0:25 ` Logan Gunthorpe 2022-01-28 9:26 ` Chaitanya Kulkarni 2022-01-28 9:26 ` Chaitanya Kulkarni via iommu 2022-01-28 0:25 ` [PATCH v5 02/24] mm: remove extra ZONE_DEVICE struct page refcount Logan Gunthorpe 2022-01-28 0:25 ` Logan Gunthorpe 2022-01-28 14:21 ` Jason Gunthorpe 2022-01-28 14:21 ` Jason Gunthorpe 2022-01-28 16:51 ` Logan Gunthorpe 2022-01-28 16:51 ` Logan Gunthorpe 2022-01-28 17:16 ` Ralph Campbell 2022-01-28 17:16 ` Ralph Campbell via iommu 2022-01-28 0:25 ` [PATCH v5 03/24] lib/scatterlist: add flag for indicating P2PDMA segments in an SGL Logan Gunthorpe 2022-01-28 0:25 ` Logan Gunthorpe 2022-01-28 0:25 ` [PATCH v5 04/24] PCI/P2PDMA: Attempt to set map_type if it has not been set Logan Gunthorpe 2022-01-28 0:25 ` Logan Gunthorpe 2022-01-28 0:25 ` [PATCH v5 05/24] PCI/P2PDMA: Expose pci_p2pdma_map_type() Logan Gunthorpe 2022-01-28 0:25 ` Logan Gunthorpe 2022-01-28 0:25 ` [PATCH v5 06/24] PCI/P2PDMA: Introduce helpers for dma_map_sg implementations Logan Gunthorpe 2022-01-28 0:25 ` Logan Gunthorpe 2022-01-28 0:25 ` [PATCH v5 07/24] dma-mapping: allow EREMOTEIO return code for P2PDMA transfers Logan Gunthorpe 2022-01-28 0:25 ` Logan Gunthorpe 2022-01-28 0:25 ` Logan Gunthorpe [this message] 2022-01-28 0:25 ` [PATCH v5 08/24] dma-direct: support PCI P2PDMA pages in dma-direct map_sg Logan Gunthorpe 2022-02-01 20:53 ` Jonathan Derrick 2022-02-01 20:53 ` Jonathan Derrick 2022-02-01 20:57 ` Logan Gunthorpe 2022-02-01 20:57 ` Logan Gunthorpe 2022-01-28 0:25 ` [PATCH v5 09/24] dma-mapping: add flags to dma_map_ops to indicate PCI P2PDMA support Logan Gunthorpe 2022-01-28 0:25 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 10/24] iommu/dma: support PCI P2PDMA pages in dma-iommu map_sg Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 11/24] nvme-pci: check DMA ops when indicating support for PCI P2PDMA Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 12/24] nvme-pci: convert to using dma_map_sgtable() Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 9:13 ` Chaitanya Kulkarni 2022-01-28 9:13 ` Chaitanya Kulkarni via iommu 2022-01-28 0:26 ` [PATCH v5 13/24] RDMA/core: introduce ib_dma_pci_p2p_dma_supported() Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 14/24] RDMA/rw: drop pci_p2pdma_[un]map_sg() Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 15/24] PCI/P2PDMA: Remove pci_p2pdma_[un]map_sg() Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 16/24] mm: introduce FOLL_PCI_P2PDMA to gate getting PCI P2PDMA pages Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 17/24] iov_iter: introduce iov_iter_get_pages_[alloc_]flags() Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 18/24] block: add check when merging zone device pages Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 19/24] lib/scatterlist: " Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 20/24] block: set FOLL_PCI_P2PDMA in __bio_iov_iter_get_pages() Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 21/24] block: set FOLL_PCI_P2PDMA in bio_map_user_iov() Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 22/24] mm: use custom page_free for P2PDMA pages Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 14:22 ` Jason Gunthorpe 2022-01-28 14:22 ` Jason Gunthorpe 2022-01-28 16:52 ` Logan Gunthorpe 2022-01-28 16:52 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 23/24] PCI/P2PDMA: Introduce pci_mmap_p2pmem() Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-28 0:26 ` [PATCH v5 24/24] nvme-pci: allow mmaping the CMB in userspace Logan Gunthorpe 2022-01-28 0:26 ` Logan Gunthorpe 2022-01-31 18:56 ` [PATCH v5 00/24] Userspace P2PDMA with O_DIRECT NVMe devices Jonathan Derrick 2022-01-31 18:56 ` Jonathan Derrick 2022-01-31 19:00 ` Logan Gunthorpe 2022-01-31 19:00 ` Logan Gunthorpe
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20220128002614.6136-9-logang@deltatee.com \ --to=logang@deltatee.com \ --cc=andrzej.jakowski@intel.com \ --cc=christian.koenig@amd.com \ --cc=ckulkarnilinux@gmail.com \ --cc=dan.j.williams@intel.com \ --cc=daniel.vetter@ffwll.ch \ --cc=dave.b.minturn@intel.com \ --cc=dave.hansen@linux.intel.com \ --cc=ddutile@redhat.com \ --cc=hch@lst.de \ --cc=helgaas@kernel.org \ --cc=iommu@lists.linux-foundation.org \ --cc=ira.weiny@intel.com \ --cc=jason@jlekstrand.net \ --cc=jgg@ziepe.ca \ --cc=jhubbard@nvidia.com \ --cc=jianxin.xiong@intel.com \ --cc=linux-block@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linux-nvme@lists.infradead.org \ --cc=linux-pci@vger.kernel.org \ --cc=martin.oliveira@eideticom.com \ --cc=rcampbell@nvidia.com \ --cc=robin.murphy@arm.com \ --cc=sbates@raithlin.com \ --cc=willy@infradead.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.