From: Bjorn Helgaas <helgaas@kernel.org>
To: Logan Gunthorpe <logang@deltatee.com>
Cc: linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org,
linux-nvme@lists.infradead.org, linux-rdma@vger.kernel.org,
linux-nvdimm@lists.01.org, linux-block@vger.kernel.org,
"Stephen Bates" <sbates@raithlin.com>,
"Christoph Hellwig" <hch@lst.de>,
"Keith Busch" <keith.busch@intel.com>,
"Sagi Grimberg" <sagi@grimberg.me>,
"Bjorn Helgaas" <bhelgaas@google.com>,
"Jason Gunthorpe" <jgg@mellanox.com>,
"Max Gurtovoy" <maxg@mellanox.com>,
"Dan Williams" <dan.j.williams@intel.com>,
"Jérôme Glisse" <jglisse@redhat.com>,
"Benjamin Herrenschmidt" <benh@kernel.crashing.org>,
"Alex Williamson" <alex.williamson@redhat.com>,
"Christian König" <christian.koenig@amd.com>,
"Jens Axboe" <axboe@kernel.dk>
Subject: Re: [PATCH v6 03/13] PCI/P2PDMA: Add PCI p2pmem DMA mappings to adjust the bus offset
Date: Fri, 21 Sep 2018 08:15:50 -0500 [thread overview]
Message-ID: <20180921131550.GG224714@bhelgaas-glaptop.roam.corp.google.com> (raw)
In-Reply-To: <20180913001156.4115-4-logang@deltatee.com>
On Wed, Sep 12, 2018 at 06:11:46PM -0600, Logan Gunthorpe wrote:
> The DMA address used when mapping PCI P2P memory must be the PCI bus
> address. Thus, introduce pci_p2pmem_map_sg() to map the correct
> addresses when using P2P memory. Memory mapped in this way does not
> need to be unmapped.
I think the use of "map" in this context is slightly confusing because the
general expectation is that map/unmap must be balanced.
I assume it's because the "mapping" consumes no resources, e.g., requires
no page table entries. Possibly there's a better verb than "map", e.g.,
"convert", "convert_to_p2pdma", etc?
If you keep "map", maybe add a sentence or two about why there's no
corresponding unmap?
> For this, we assume that an SGL passed to these functions contain all
> P2P memory or no P2P memory.
>
> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
> ---
> drivers/pci/p2pdma.c | 43 ++++++++++++++++++++++++++++++++++++++
> include/linux/memremap.h | 1 +
> include/linux/pci-p2pdma.h | 7 +++++++
> 3 files changed, 51 insertions(+)
>
> diff --git a/drivers/pci/p2pdma.c b/drivers/pci/p2pdma.c
> index 67c1daf1189e..29bd40a87768 100644
> --- a/drivers/pci/p2pdma.c
> +++ b/drivers/pci/p2pdma.c
> @@ -191,6 +191,8 @@ int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar, size_t size,
> pgmap->res.flags = pci_resource_flags(pdev, bar);
> pgmap->ref = &pdev->p2pdma->devmap_ref;
> pgmap->type = MEMORY_DEVICE_PCI_P2PDMA;
> + pgmap->pci_p2pdma_bus_offset = pci_bus_address(pdev, bar) -
> + pci_resource_start(pdev, bar);
>
> addr = devm_memremap_pages(&pdev->dev, pgmap);
> if (IS_ERR(addr)) {
> @@ -813,3 +815,44 @@ void pci_p2pmem_publish(struct pci_dev *pdev, bool publish)
> pdev->p2pdma->p2pmem_published = publish;
> }
> EXPORT_SYMBOL_GPL(pci_p2pmem_publish);
> +
> +/**
> + * pci_p2pdma_map_sg - map a PCI peer-to-peer scatterlist for DMA
> + * @dev: device doing the DMA request
> + * @sg: scatter list to map
> + * @nents: elements in the scatterlist
> + * @dir: DMA direction
> + *
> + * Scatterlists mapped with this function should not be unmapped in any way.
> + *
> + * Returns the number of SG entries mapped or 0 on error.
> + */
> +int pci_p2pdma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
> + enum dma_data_direction dir)
> +{
> + struct dev_pagemap *pgmap;
> + struct scatterlist *s;
> + phys_addr_t paddr;
> + int i;
> +
> + /*
> + * p2pdma mappings are not compatible with devices that use
> + * dma_virt_ops. If the upper layers do the right thing
> + * this should never happen because it will be prevented
> + * by the check in pci_p2pdma_add_client()
> + */
> + if (WARN_ON_ONCE(IS_ENABLED(CONFIG_DMA_VIRT_OPS) &&
> + dev->dma_ops == &dma_virt_ops))
> + return 0;
> +
> + for_each_sg(sg, s, nents, i) {
> + pgmap = sg_page(s)->pgmap;
> + paddr = sg_phys(s);
> +
> + s->dma_address = paddr - pgmap->pci_p2pdma_bus_offset;
> + sg_dma_len(s) = s->length;
> + }
> +
> + return nents;
> +}
> +EXPORT_SYMBOL_GPL(pci_p2pdma_map_sg);
> diff --git a/include/linux/memremap.h b/include/linux/memremap.h
> index 9553370ebdad..0ac69ddf5fc4 100644
> --- a/include/linux/memremap.h
> +++ b/include/linux/memremap.h
> @@ -125,6 +125,7 @@ struct dev_pagemap {
> struct device *dev;
> void *data;
> enum memory_type type;
> + u64 pci_p2pdma_bus_offset;
> };
>
> #ifdef CONFIG_ZONE_DEVICE
> diff --git a/include/linux/pci-p2pdma.h b/include/linux/pci-p2pdma.h
> index 7b2b0f547528..2f03dbbf5af6 100644
> --- a/include/linux/pci-p2pdma.h
> +++ b/include/linux/pci-p2pdma.h
> @@ -36,6 +36,8 @@ struct scatterlist *pci_p2pmem_alloc_sgl(struct pci_dev *pdev,
> unsigned int *nents, u32 length);
> void pci_p2pmem_free_sgl(struct pci_dev *pdev, struct scatterlist *sgl);
> void pci_p2pmem_publish(struct pci_dev *pdev, bool publish);
> +int pci_p2pdma_map_sg(struct device *dev, struct scatterlist *sg, int nents,
> + enum dma_data_direction dir);
> #else /* CONFIG_PCI_P2PDMA */
> static inline int pci_p2pdma_add_resource(struct pci_dev *pdev, int bar,
> size_t size, u64 offset)
> @@ -98,5 +100,10 @@ static inline void pci_p2pmem_free_sgl(struct pci_dev *pdev,
> static inline void pci_p2pmem_publish(struct pci_dev *pdev, bool publish)
> {
> }
> +static inline int pci_p2pdma_map_sg(struct device *dev,
> + struct scatterlist *sg, int nents, enum dma_data_direction dir)
> +{
> + return 0;
> +}
> #endif /* CONFIG_PCI_P2PDMA */
> #endif /* _LINUX_PCI_P2P_H */
> --
> 2.19.0
>
next prev parent reply other threads:[~2018-09-21 13:15 UTC|newest]
Thread overview: 34+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-09-13 0:11 [PATCH v6 00/13] Copy Offload in NVMe Fabrics with P2P PCI Memory Logan Gunthorpe
2018-09-13 0:11 ` [PATCH v6 01/13] PCI/P2PDMA: Support peer-to-peer memory Logan Gunthorpe
2018-09-20 22:38 ` Bjorn Helgaas
2018-09-20 22:47 ` Logan Gunthorpe
2018-09-21 13:00 ` Bjorn Helgaas
2018-09-21 15:37 ` Logan Gunthorpe
2018-09-21 16:05 ` Christoph Hellwig
2018-09-13 0:11 ` [PATCH v6 02/13] PCI/P2PDMA: Add sysfs group to display p2pmem stats Logan Gunthorpe
2018-09-21 13:07 ` Bjorn Helgaas
2018-09-13 0:11 ` [PATCH v6 03/13] PCI/P2PDMA: Add PCI p2pmem DMA mappings to adjust the bus offset Logan Gunthorpe
2018-09-21 13:15 ` Bjorn Helgaas [this message]
2018-09-21 16:48 ` Bjorn Helgaas
2018-09-21 18:13 ` Logan Gunthorpe
2018-09-21 20:00 ` Bjorn Helgaas
2018-09-21 20:01 ` Logan Gunthorpe
2018-09-13 0:11 ` [PATCH v6 04/13] PCI/P2PDMA: Introduce configfs/sysfs enable attribute helpers Logan Gunthorpe
2018-09-21 16:18 ` Bjorn Helgaas
2018-09-21 19:44 ` Logan Gunthorpe
2018-09-21 21:12 ` Logan Gunthorpe
2018-09-24 22:39 ` Bjorn Helgaas
2018-09-13 0:11 ` [PATCH v6 05/13] docs-rst: Add a new directory for PCI documentation Logan Gunthorpe
2018-09-13 0:11 ` [PATCH v6 06/13] PCI/P2PDMA: Add P2P DMA driver writer's documentation Logan Gunthorpe
2018-09-21 16:41 ` Bjorn Helgaas
2018-09-21 18:03 ` Logan Gunthorpe
2018-09-21 19:47 ` Bjorn Helgaas
2018-09-13 0:11 ` [PATCH v6 07/13] block: Add PCI P2P flag for request queue and check support for requests Logan Gunthorpe
2018-09-13 0:28 ` Jens Axboe
2018-09-13 16:14 ` Logan Gunthorpe
2018-09-13 0:11 ` [PATCH v6 08/13] IB/core: Ensure we map P2P memory correctly in rdma_rw_ctx_[init|destroy]() Logan Gunthorpe
2018-09-13 0:11 ` [PATCH v6 09/13] nvme-pci: Use PCI p2pmem subsystem to manage the CMB Logan Gunthorpe
2018-09-13 0:11 ` [PATCH v6 10/13] nvme-pci: Add support for P2P memory in requests Logan Gunthorpe
2018-09-13 0:11 ` [PATCH v6 11/13] nvme-pci: Add a quirk for a pseudo CMB Logan Gunthorpe
2018-09-13 0:11 ` [PATCH v6 12/13] nvmet: Introduce helper functions to allocate and free request SGLs Logan Gunthorpe
2018-09-13 0:11 ` [PATCH v6 13/13] nvmet: Optionally use PCI P2P memory Logan Gunthorpe
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180921131550.GG224714@bhelgaas-glaptop.roam.corp.google.com \
--to=helgaas@kernel.org \
--cc=alex.williamson@redhat.com \
--cc=axboe@kernel.dk \
--cc=benh@kernel.crashing.org \
--cc=bhelgaas@google.com \
--cc=christian.koenig@amd.com \
--cc=dan.j.williams@intel.com \
--cc=hch@lst.de \
--cc=jgg@mellanox.com \
--cc=jglisse@redhat.com \
--cc=keith.busch@intel.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nvdimm@lists.01.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=linux-rdma@vger.kernel.org \
--cc=logang@deltatee.com \
--cc=maxg@mellanox.com \
--cc=sagi@grimberg.me \
--cc=sbates@raithlin.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).