From: John Hubbard <jhubbard@nvidia.com>
To: Logan Gunthorpe <logang@deltatee.com>,
<linux-kernel@vger.kernel.org>, <linux-nvme@lists.infradead.org>,
<linux-block@vger.kernel.org>, <linux-pci@vger.kernel.org>,
<linux-mm@kvack.org>, <iommu@lists.linux-foundation.org>
Cc: "Stephen Bates" <sbates@raithlin.com>,
"Christoph Hellwig" <hch@lst.de>,
"Dan Williams" <dan.j.williams@intel.com>,
"Jason Gunthorpe" <jgg@ziepe.ca>,
"Christian König" <christian.koenig@amd.com>,
"Don Dutile" <ddutile@redhat.com>,
"Matthew Wilcox" <willy@infradead.org>,
"Daniel Vetter" <daniel.vetter@ffwll.ch>,
"Jakowski Andrzej" <andrzej.jakowski@intel.com>,
"Minturn Dave B" <dave.b.minturn@intel.com>,
"Jason Ekstrand" <jason@jlekstrand.net>,
"Dave Hansen" <dave.hansen@linux.intel.com>,
"Xiong Jianxin" <jianxin.xiong@intel.com>,
"Bjorn Helgaas" <helgaas@kernel.org>,
"Ira Weiny" <ira.weiny@intel.com>,
"Robin Murphy" <robin.murphy@arm.com>
Subject: Re: [PATCH 05/16] dma-mapping: Introduce dma_map_sg_p2pdma()
Date: Sun, 2 May 2021 14:23:31 -0700 [thread overview]
Message-ID: <c050bcae-e223-bb41-021f-b1fda572647b@nvidia.com> (raw)
In-Reply-To: <20210408170123.8788-6-logang@deltatee.com>
On 4/8/21 10:01 AM, Logan Gunthorpe wrote:
> dma_map_sg() either returns a positive number indicating the number
> of entries mapped or zero indicating that resources were not available
> to create the mapping. When zero is returned, it is always safe to retry
> the mapping later once resources have been freed.
>
> Once P2PDMA pages are mixed into the SGL there may be pages that may
> never be successfully mapped with a given device because that device may
> not actually be able to access those pages. Thus, multiple error
> conditions will need to be distinguished to determine weather a retry
> is safe.
>
> Introduce dma_map_sg_p2pdma[_attrs]() with a different calling
> convention from dma_map_sg(). The function will return a positive
> integer on success or a negative errno on failure.
>
> ENOMEM will be used to indicate a resource failure and EREMOTEIO to
> indicate that a P2PDMA page is not mappable.
>
> The __DMA_ATTR_PCI_P2PDMA attribute is introduced to inform the lower
> level implementations that P2PDMA pages are allowed and to warn if a
> caller introduces them into the regular dma_map_sg() interface.
>
> Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
> ---
> include/linux/dma-mapping.h | 15 +++++++++++
> kernel/dma/mapping.c | 52 ++++++++++++++++++++++++++++++++-----
> 2 files changed, 61 insertions(+), 6 deletions(-)
>
> diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h
> index 2a984cb4d1e0..50b8f586cf59 100644
> --- a/include/linux/dma-mapping.h
> +++ b/include/linux/dma-mapping.h
> @@ -60,6 +60,12 @@
> * at least read-only at lesser-privileged levels).
> */
> #define DMA_ATTR_PRIVILEGED (1UL << 9)
> +/*
> + * __DMA_ATTR_PCI_P2PDMA: This should not be used directly, use
> + * dma_map_sg_p2pdma() instead. Used internally to indicate that the
> + * caller is using the dma_map_sg_p2pdma() interface.
> + */
> +#define __DMA_ATTR_PCI_P2PDMA (1UL << 10)
>
As mentioned near the top of this file,
Documentation/core-api/dma-attributes.rst also needs to be updated, for
this new item.
> /*
> * A dma_addr_t can hold any valid DMA or bus address for the platform. It can
> @@ -107,6 +113,8 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size,
> enum dma_data_direction dir, unsigned long attrs);
> int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents,
> enum dma_data_direction dir, unsigned long attrs);
> +int dma_map_sg_p2pdma_attrs(struct device *dev, struct scatterlist *sg,
> + int nents, enum dma_data_direction dir, unsigned long attrs);
> void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg,
> int nents, enum dma_data_direction dir,
> unsigned long attrs);
> @@ -160,6 +168,12 @@ static inline int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg,
> {
> return 0;
> }
> +static inline int dma_map_sg_p2pdma_attrs(struct device *dev,
> + struct scatterlist *sg, int nents, enum dma_data_direction dir,
> + unsigned long attrs)
> +{
> + return 0;
> +}
> static inline void dma_unmap_sg_attrs(struct device *dev,
> struct scatterlist *sg, int nents, enum dma_data_direction dir,
> unsigned long attrs)
> @@ -392,6 +406,7 @@ static inline void dma_sync_sgtable_for_device(struct device *dev,
> #define dma_map_single(d, a, s, r) dma_map_single_attrs(d, a, s, r, 0)
> #define dma_unmap_single(d, a, s, r) dma_unmap_single_attrs(d, a, s, r, 0)
> #define dma_map_sg(d, s, n, r) dma_map_sg_attrs(d, s, n, r, 0)
> +#define dma_map_sg_p2pdma(d, s, n, r) dma_map_sg_p2pdma_attrs(d, s, n, r, 0)
This hunk is fine, of course.
But, about pre-existing issues: note to self, or to anyone: send a patch to turn
these into inline functions. The macro redirection here is not adding value, but
it does make things just a little bit worse.
> #define dma_unmap_sg(d, s, n, r) dma_unmap_sg_attrs(d, s, n, r, 0)
> #define dma_map_page(d, p, o, s, r) dma_map_page_attrs(d, p, o, s, r, 0)
> #define dma_unmap_page(d, a, s, r) dma_unmap_page_attrs(d, a, s, r, 0)
> diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c
> index b6a633679933..923089c4267b 100644
> --- a/kernel/dma/mapping.c
> +++ b/kernel/dma/mapping.c
> @@ -177,12 +177,8 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size,
> }
> EXPORT_SYMBOL(dma_unmap_page_attrs);
>
> -/*
> - * dma_maps_sg_attrs returns 0 on error and > 0 on success.
> - * It should never return a value < 0.
> - */
It would be better to leave the comment in place, given the non-standard
return values. However, looking around here, it would be better if we go
with the standard -ERRNO for error, and >0 for sucess.
There are pre-existing BUG_ON() and WARN_ON_ONCE() items that are partly
an attempt to compensate for not being able to return proper -ERRNO
codes. For example, this:
BUG_ON(!valid_dma_direction(dir));
...arguably should be more like this:
if(WARN_ON_ONCE(!valid_dma_direction(dir)))
return -EINVAL;
> -int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents,
> - enum dma_data_direction dir, unsigned long attrs)
> +static int __dma_map_sg_attrs(struct device *dev, struct scatterlist *sg,
> + int nents, enum dma_data_direction dir, unsigned long attrs)
> {
> const struct dma_map_ops *ops = get_dma_ops(dev);
> int ents;
> @@ -197,6 +193,20 @@ int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents,
> ents = dma_direct_map_sg(dev, sg, nents, dir, attrs);
> else
> ents = ops->map_sg(dev, sg, nents, dir, attrs);
> +
> + return ents;
> +}
> +
> +/*
> + * dma_maps_sg_attrs returns 0 on error and > 0 on success.
> + * It should never return a value < 0.
> + */
> +int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents,
> + enum dma_data_direction dir, unsigned long attrs)
> +{
> + int ents;
Pre-existing note, feel free to ignore: the ents and nents in the same
routines together, are way too close to the each other in naming. Maybe
using "requested_nents", or "nents_arg", for the incoming value, would
help.
> +
> + ents = __dma_map_sg_attrs(dev, sg, nents, dir, attrs);
> BUG_ON(ents < 0);
> debug_dma_map_sg(dev, sg, nents, ents, dir);
>
> @@ -204,6 +214,36 @@ int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents,
> }
> EXPORT_SYMBOL(dma_map_sg_attrs);
>
> +/*
> + * like dma_map_sg_attrs, but returns a negative errno on error (and > 0
> + * on success). This function must be used if PCI P2PDMA pages might
> + * be in the scatterlist.
Let's turn this into a kernel doc comment block, seeing as how it clearly
wants to be--you're almost there already. You've even reinvented @Return,
below. :)
> + *
> + * On error this function may return:
> + * -ENOMEM indicating that there was not enough resources available and
> + * the transfer may be retried later
> + * -EREMOTEIO indicating that P2PDMA pages were included but cannot
> + * be mapped by the specified device, retries will always fail
> + *
> + * The scatterlist should be unmapped with the regular dma_unmap_sg[_attrs]().
How about:
"The scatterlist should be unmapped via dma_unmap_sg[_attrs]()."
> + */
> +int dma_map_sg_p2pdma_attrs(struct device *dev, struct scatterlist *sg,
> + int nents, enum dma_data_direction dir, unsigned long attrs)
> +{
> + int ents;
> +
> + ents = __dma_map_sg_attrs(dev, sg, nents, dir,
> + attrs | __DMA_ATTR_PCI_P2PDMA);
> + if (!ents)
> + ents = -ENOMEM;
> +
> + if (ents > 0)
> + debug_dma_map_sg(dev, sg, nents, ents, dir);
> +
> + return ents;
> +}
> +EXPORT_SYMBOL_GPL(dma_map_sg_p2pdma_attrs);
> +
> void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg,
> int nents, enum dma_data_direction dir,
> unsigned long attrs)
>
thanks,
--
John Hubbard
NVIDIA
next prev parent reply other threads:[~2021-05-02 21:23 UTC|newest]
Thread overview: 99+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-04-08 17:01 [PATCH 00/16] Add new DMA mapping operation for P2PDMA Logan Gunthorpe
2021-04-08 17:01 ` [PATCH 01/16] PCI/P2PDMA: Pass gfp_mask flags to upstream_bridge_distance_warn() Logan Gunthorpe
2021-05-02 3:58 ` John Hubbard
2021-05-03 15:57 ` Logan Gunthorpe
2021-05-03 18:17 ` John Hubbard
2021-05-03 18:20 ` Logan Gunthorpe
2021-05-03 18:23 ` John Hubbard
2021-05-03 18:24 ` Christoph Hellwig
2021-05-11 16:05 ` Don Dutile
2021-05-11 16:12 ` Logan Gunthorpe
2021-05-11 16:23 ` Don Dutile
2021-04-08 17:01 ` [PATCH 02/16] PCI/P2PDMA: Avoid pci_get_slot() which sleeps Logan Gunthorpe
2021-05-02 5:35 ` John Hubbard
2021-05-03 16:08 ` Logan Gunthorpe
2021-05-03 18:20 ` John Hubbard
2021-05-03 18:25 ` Christoph Hellwig
2021-05-11 16:05 ` Don Dutile
2021-05-11 16:16 ` Logan Gunthorpe
2021-05-11 16:05 ` Don Dutile
2021-05-11 16:14 ` Logan Gunthorpe
2021-04-08 17:01 ` [PATCH 03/16] PCI/P2PDMA: Attempt to set map_type if it has not been set Logan Gunthorpe
2021-05-02 19:58 ` John Hubbard
2021-05-03 16:17 ` Logan Gunthorpe
2021-05-03 18:22 ` John Hubbard
2021-05-03 18:35 ` Christoph Hellwig
2021-05-03 18:46 ` Logan Gunthorpe
2021-05-11 16:05 ` Don Dutile
2021-04-08 17:01 ` [PATCH 04/16] PCI/P2PDMA: Refactor pci_p2pdma_map_type() to take pagmap and device Logan Gunthorpe
2021-05-02 20:41 ` John Hubbard
2021-05-03 16:30 ` Logan Gunthorpe
2021-05-03 18:31 ` John Hubbard
2021-05-03 18:56 ` Logan Gunthorpe
2021-05-03 21:54 ` John Hubbard
2021-05-03 22:57 ` Jason Gunthorpe
2021-05-03 23:40 ` John Hubbard
2021-04-08 17:01 ` [PATCH 05/16] dma-mapping: Introduce dma_map_sg_p2pdma() Logan Gunthorpe
2021-04-27 19:22 ` Jason Gunthorpe
2021-04-27 22:49 ` Logan Gunthorpe
2021-04-27 19:31 ` Jason Gunthorpe
2021-04-27 22:55 ` Logan Gunthorpe
2021-04-27 23:01 ` Jason Gunthorpe
2021-05-03 18:28 ` Christoph Hellwig
2021-05-03 18:31 ` Logan Gunthorpe
2021-05-02 21:23 ` John Hubbard [this message]
2021-05-03 16:38 ` Logan Gunthorpe
2021-05-11 16:05 ` Don Dutile
2021-04-08 17:01 ` [PATCH 06/16] lib/scatterlist: Add flag for indicating P2PDMA segments in an SGL Logan Gunthorpe
2021-05-02 22:34 ` John Hubbard
2021-04-08 17:01 ` [PATCH 07/16] PCI/P2PDMA: Make pci_p2pdma_map_type() non-static Logan Gunthorpe
2021-05-02 22:44 ` John Hubbard
2021-05-03 16:39 ` Logan Gunthorpe
2021-05-11 16:06 ` Don Dutile
2021-05-11 16:17 ` Logan Gunthorpe
2021-04-08 17:01 ` [PATCH 08/16] PCI/P2PDMA: Introduce helpers for dma_map_sg implementations Logan Gunthorpe
2021-05-02 22:52 ` John Hubbard
2021-05-03 0:50 ` John Hubbard
2021-05-03 17:15 ` Logan Gunthorpe
2021-04-08 17:01 ` [PATCH 09/16] dma-direct: Support PCI P2PDMA pages in dma-direct map_sg Logan Gunthorpe
2021-04-27 19:33 ` Jason Gunthorpe
2021-04-27 19:40 ` Jason Gunthorpe
2021-04-27 22:56 ` Logan Gunthorpe
2021-05-02 23:28 ` John Hubbard
2021-05-02 23:32 ` John Hubbard
2021-05-03 17:06 ` Logan Gunthorpe
2021-05-03 16:55 ` Logan Gunthorpe
2021-05-04 0:12 ` John Hubbard
2021-05-03 17:04 ` Logan Gunthorpe
2021-05-04 0:01 ` John Hubbard
2021-04-08 17:01 ` [PATCH 10/16] dma-mapping: Add flags to dma_map_ops to indicate PCI P2PDMA support Logan Gunthorpe
2021-05-03 0:32 ` John Hubbard
2021-05-03 17:09 ` Logan Gunthorpe
2021-05-11 16:06 ` Don Dutile
2021-05-11 16:19 ` Logan Gunthorpe
2021-04-08 17:01 ` [PATCH 11/16] iommu/dma: Support PCI P2PDMA pages in dma-iommu map_sg Logan Gunthorpe
2021-04-27 19:43 ` Jason Gunthorpe
2021-04-27 22:59 ` Logan Gunthorpe
2021-05-03 1:14 ` John Hubbard
2021-05-06 23:59 ` Logan Gunthorpe
2021-05-11 16:06 ` Don Dutile
2021-05-11 16:35 ` Logan Gunthorpe
2021-04-08 17:01 ` [PATCH 12/16] nvme-pci: Check DMA ops when indicating support for PCI P2PDMA Logan Gunthorpe
2021-05-03 1:29 ` John Hubbard
2021-05-03 17:17 ` Logan Gunthorpe
2021-05-04 0:17 ` John Hubbard
2021-04-08 17:01 ` [PATCH 13/16] nvme-pci: Convert to using dma_map_sg_p2pdma for p2pdma pages Logan Gunthorpe
2021-05-03 1:34 ` John Hubbard
2021-05-03 17:19 ` Logan Gunthorpe
2021-05-04 0:26 ` John Hubbard
2021-04-08 17:01 ` [PATCH 14/16] nvme-rdma: Ensure dma support when using p2pdma Logan Gunthorpe
2021-04-27 19:47 ` Jason Gunthorpe
2021-04-27 22:59 ` Logan Gunthorpe
2021-05-03 1:37 ` John Hubbard
2021-04-08 17:01 ` [PATCH 15/16] RDMA/rw: use dma_map_sg_p2pdma() Logan Gunthorpe
2021-04-08 17:01 ` [PATCH 16/16] PCI/P2PDMA: Remove pci_p2pdma_[un]map_sg() Logan Gunthorpe
2021-04-27 19:28 ` [PATCH 00/16] Add new DMA mapping operation for P2PDMA Jason Gunthorpe
2021-04-27 20:21 ` John Hubbard
2021-04-27 20:48 ` Dan Williams
2021-05-02 1:22 ` John Hubbard
2021-05-11 16:05 ` Don Dutile
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c050bcae-e223-bb41-021f-b1fda572647b@nvidia.com \
--to=jhubbard@nvidia.com \
--cc=andrzej.jakowski@intel.com \
--cc=christian.koenig@amd.com \
--cc=dan.j.williams@intel.com \
--cc=daniel.vetter@ffwll.ch \
--cc=dave.b.minturn@intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=ddutile@redhat.com \
--cc=hch@lst.de \
--cc=helgaas@kernel.org \
--cc=iommu@lists.linux-foundation.org \
--cc=ira.weiny@intel.com \
--cc=jason@jlekstrand.net \
--cc=jgg@ziepe.ca \
--cc=jianxin.xiong@intel.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-nvme@lists.infradead.org \
--cc=linux-pci@vger.kernel.org \
--cc=logang@deltatee.com \
--cc=robin.murphy@arm.com \
--cc=sbates@raithlin.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).