* generic DMA bypass flag v3 @ 2020-04-14 12:25 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel Hi all, I've recently beeing chatting with Lu about using dma-iommu and per-device DMA ops in the intel IOMMU driver, and one missing feature in dma-iommu is a bypass mode where the direct mapping is used even when an iommu is attached to improve performance. The powerpc code already has a similar mode, so I'd like to move it to the core DMA mapping code. As part of that I noticed that the current powerpc code has a little bug in that it used the wrong check in the dma_sync_* routines to see if the direct mapping code is used. These two patches just add the generic code and move powerpc over, the intel IOMMU bits will require a separate discussion. The x86 AMD Gart code also has a bypass mode, but it is a lot strange, so I'm not going to touch it for now. Changes since v2: - move the dma mapping helpers out of line - check for possible direct mappings using the dma mask Changes since v1: - rebased to the current dma-mapping-for-next tree ^ permalink raw reply [flat|nested] 60+ messages in thread
* generic DMA bypass flag v3 @ 2020-04-14 12:25 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, linuxppc-dev Hi all, I've recently beeing chatting with Lu about using dma-iommu and per-device DMA ops in the intel IOMMU driver, and one missing feature in dma-iommu is a bypass mode where the direct mapping is used even when an iommu is attached to improve performance. The powerpc code already has a similar mode, so I'd like to move it to the core DMA mapping code. As part of that I noticed that the current powerpc code has a little bug in that it used the wrong check in the dma_sync_* routines to see if the direct mapping code is used. These two patches just add the generic code and move powerpc over, the intel IOMMU bits will require a separate discussion. The x86 AMD Gart code also has a bypass mode, but it is a lot strange, so I'm not going to touch it for now. Changes since v2: - move the dma mapping helpers out of line - check for possible direct mappings using the dma mask Changes since v1: - rebased to the current dma-mapping-for-next tree _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* generic DMA bypass flag v3 @ 2020-04-14 12:25 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, linuxppc-dev, Lu Baolu Hi all, I've recently beeing chatting with Lu about using dma-iommu and per-device DMA ops in the intel IOMMU driver, and one missing feature in dma-iommu is a bypass mode where the direct mapping is used even when an iommu is attached to improve performance. The powerpc code already has a similar mode, so I'd like to move it to the core DMA mapping code. As part of that I noticed that the current powerpc code has a little bug in that it used the wrong check in the dma_sync_* routines to see if the direct mapping code is used. These two patches just add the generic code and move powerpc over, the intel IOMMU bits will require a separate discussion. The x86 AMD Gart code also has a bypass mode, but it is a lot strange, so I'm not going to touch it for now. Changes since v2: - move the dma mapping helpers out of line - check for possible direct mappings using the dma mask Changes since v1: - rebased to the current dma-mapping-for-next tree ^ permalink raw reply [flat|nested] 60+ messages in thread
* [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line 2020-04-14 12:25 ` Christoph Hellwig (?) @ 2020-04-14 12:25 ` Christoph Hellwig -1 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel For a long time the DMA API has been implemented inline in dma-mapping.h, but the function bodies can be quite large. Move them all out of line. Signed-off-by: Christoph Hellwig <hch@lst.de> --- include/linux/dma-direct.h | 58 +++++++++ include/linux/dma-mapping.h | 247 ++++-------------------------------- kernel/dma/direct.c | 9 -- kernel/dma/mapping.c | 164 ++++++++++++++++++++++++ 4 files changed, 244 insertions(+), 234 deletions(-) diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index 24b8684aa21d..da689ad5fffd 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -85,4 +85,62 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma, void *cpu_addr, dma_addr_t dma_addr, size_t size, unsigned long attrs); int dma_direct_supported(struct device *dev, u64 mask); +dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, + unsigned long offset, size_t size, enum dma_data_direction dir, + unsigned long attrs); +int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, + enum dma_data_direction dir, unsigned long attrs); +dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, + size_t size, enum dma_data_direction dir, unsigned long attrs); + +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ + defined(CONFIG_SWIOTLB) +void dma_direct_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir); +void dma_direct_sync_sg_for_device(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir); +#else +static inline void dma_direct_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ +} +static inline void dma_direct_sync_sg_for_device(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir) +{ +} +#endif + +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ + defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ + defined(CONFIG_SWIOTLB) +void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir, unsigned long attrs); +void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, + int nents, enum dma_data_direction dir, unsigned long attrs); +void dma_direct_sync_single_for_cpu(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir); +void dma_direct_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir); +#else +static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ +} +static inline void dma_direct_unmap_sg(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir, + unsigned long attrs) +{ +} +static inline void dma_direct_sync_single_for_cpu(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ +} +static inline void dma_direct_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir) +{ +} +#endif + +size_t dma_direct_max_mapping_size(struct device *dev); + #endif /* _LINUX_DMA_DIRECT_H */ diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 330ad58fbf4d..793ad775cd54 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -188,73 +188,6 @@ static inline int dma_mmap_from_global_coherent(struct vm_area_struct *vma, } #endif /* CONFIG_DMA_DECLARE_COHERENT */ -static inline bool dma_is_direct(const struct dma_map_ops *ops) -{ - return likely(!ops); -} - -/* - * All the dma_direct_* declarations are here just for the indirect call bypass, - * and must not be used directly drivers! - */ -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs); -int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, - enum dma_data_direction dir, unsigned long attrs); -dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, - size_t size, enum dma_data_direction dir, unsigned long attrs); - -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ - defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); -void dma_direct_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir); -#else -static inline void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ -} -static inline void dma_direct_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ -} -#endif - -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ - defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ - defined(CONFIG_SWIOTLB) -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs); -void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, - int nents, enum dma_data_direction dir, unsigned long attrs); -void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); -void dma_direct_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir); -#else -static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ -} -static inline void dma_direct_unmap_sg(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir, - unsigned long attrs) -{ -} -static inline void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ -} -static inline void dma_direct_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ -} -#endif - -size_t dma_direct_max_mapping_size(struct device *dev); - #ifdef CONFIG_HAS_DMA #include <asm/dma-mapping.h> @@ -271,164 +204,6 @@ static inline void set_dma_ops(struct device *dev, dev->dma_ops = dma_ops; } -static inline dma_addr_t dma_map_page_attrs(struct device *dev, - struct page *page, size_t offset, size_t size, - enum dma_data_direction dir, unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - dma_addr_t addr; - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); - else - addr = ops->map_page(dev, page, offset, size, dir, attrs); - debug_dma_map_page(dev, page, offset, size, dir, addr); - - return addr; -} - -static inline void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_unmap_page(dev, addr, size, dir, attrs); - else if (ops->unmap_page) - ops->unmap_page(dev, addr, size, dir, attrs); - debug_dma_unmap_page(dev, addr, size, dir); -} - -/* - * dma_maps_sg_attrs returns 0 on error and > 0 on success. - * It should never return a value < 0. - */ -static inline int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - int ents; - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); - else - ents = ops->map_sg(dev, sg, nents, dir, attrs); - BUG_ON(ents < 0); - debug_dma_map_sg(dev, sg, nents, ents, dir); - - return ents; -} - -static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - debug_dma_unmap_sg(dev, sg, nents, dir); - if (dma_is_direct(ops)) - dma_direct_unmap_sg(dev, sg, nents, dir, attrs); - else if (ops->unmap_sg) - ops->unmap_sg(dev, sg, nents, dir, attrs); -} - -static inline dma_addr_t dma_map_resource(struct device *dev, - phys_addr_t phys_addr, - size_t size, - enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - dma_addr_t addr = DMA_MAPPING_ERROR; - - BUG_ON(!valid_dma_direction(dir)); - - /* Don't allow RAM to be mapped */ - if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) - return DMA_MAPPING_ERROR; - - if (dma_is_direct(ops)) - addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); - else if (ops->map_resource) - addr = ops->map_resource(dev, phys_addr, size, dir, attrs); - - debug_dma_map_resource(dev, phys_addr, size, dir, addr); - return addr; -} - -static inline void dma_unmap_resource(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (!dma_is_direct(ops) && ops->unmap_resource) - ops->unmap_resource(dev, addr, size, dir, attrs); - debug_dma_unmap_resource(dev, addr, size, dir); -} - -static inline void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, - size_t size, - enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_single_for_cpu(dev, addr, size, dir); - else if (ops->sync_single_for_cpu) - ops->sync_single_for_cpu(dev, addr, size, dir); - debug_dma_sync_single_for_cpu(dev, addr, size, dir); -} - -static inline void dma_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, - enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_single_for_device(dev, addr, size, dir); - else if (ops->sync_single_for_device) - ops->sync_single_for_device(dev, addr, size, dir); - debug_dma_sync_single_for_device(dev, addr, size, dir); -} - -static inline void -dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, - int nelems, enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); - else if (ops->sync_sg_for_cpu) - ops->sync_sg_for_cpu(dev, sg, nelems, dir); - debug_dma_sync_sg_for_cpu(dev, sg, nelems, dir); -} - -static inline void -dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, - int nelems, enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_sg_for_device(dev, sg, nelems, dir); - else if (ops->sync_sg_for_device) - ops->sync_sg_for_device(dev, sg, nelems, dir); - debug_dma_sync_sg_for_device(dev, sg, nelems, dir); - -} static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) { @@ -439,6 +214,28 @@ static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) return 0; } +dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs); +void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs); +int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, + enum dma_data_direction dir, unsigned long attrs); +void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, + int nents, enum dma_data_direction dir, + unsigned long attrs); +dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, + size_t size, enum dma_data_direction dir, unsigned long attrs); +void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs); +void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir); +void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir); +void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir); +void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir); void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag, unsigned long attrs); void dma_free_attrs(struct device *dev, size_t size, void *cpu_addr, diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 8f4bbdaf965e..f1a9099a4b5b 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -260,7 +260,6 @@ void dma_direct_sync_single_for_device(struct device *dev, if (!dev_is_dma_coherent(dev)) arch_sync_dma_for_device(paddr, size, dir); } -EXPORT_SYMBOL(dma_direct_sync_single_for_device); void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) @@ -280,7 +279,6 @@ void dma_direct_sync_sg_for_device(struct device *dev, dir); } } -EXPORT_SYMBOL(dma_direct_sync_sg_for_device); #endif #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ @@ -299,7 +297,6 @@ void dma_direct_sync_single_for_cpu(struct device *dev, if (unlikely(is_swiotlb_buffer(paddr))) swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_CPU); } -EXPORT_SYMBOL(dma_direct_sync_single_for_cpu); void dma_direct_sync_sg_for_cpu(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) @@ -321,7 +318,6 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, if (!dev_is_dma_coherent(dev)) arch_sync_dma_for_cpu_all(); } -EXPORT_SYMBOL(dma_direct_sync_sg_for_cpu); void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs) @@ -334,7 +330,6 @@ void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, if (unlikely(is_swiotlb_buffer(phys))) swiotlb_tbl_unmap_single(dev, phys, size, size, dir, attrs); } -EXPORT_SYMBOL(dma_direct_unmap_page); void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) @@ -346,7 +341,6 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, dma_direct_unmap_page(dev, sg->dma_address, sg_dma_len(sg), dir, attrs); } -EXPORT_SYMBOL(dma_direct_unmap_sg); #endif dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, @@ -373,7 +367,6 @@ dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, arch_sync_dma_for_device(phys, size, dir); return dma_addr; } -EXPORT_SYMBOL(dma_direct_map_page); int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) @@ -395,7 +388,6 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, dma_direct_unmap_sg(dev, sgl, i, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC); return 0; } -EXPORT_SYMBOL(dma_direct_map_sg); dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, size_t size, enum dma_data_direction dir, unsigned long attrs) @@ -412,7 +404,6 @@ dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, return dma_addr; } -EXPORT_SYMBOL(dma_direct_map_resource); int dma_direct_get_sgtable(struct device *dev, struct sg_table *sgt, void *cpu_addr, dma_addr_t dma_addr, size_t size, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 98e3d873792e..8e4155f9ee69 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -105,6 +105,170 @@ void *dmam_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, } EXPORT_SYMBOL(dmam_alloc_attrs); +static inline bool dma_is_direct(const struct dma_map_ops *ops) +{ + return likely(!ops); +} + +dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + dma_addr_t addr; + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); + else + addr = ops->map_page(dev, page, offset, size, dir, attrs); + debug_dma_map_page(dev, page, offset, size, dir, addr); + + return addr; +} +EXPORT_SYMBOL(dma_map_page_attrs); + +void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_unmap_page(dev, addr, size, dir, attrs); + else if (ops->unmap_page) + ops->unmap_page(dev, addr, size, dir, attrs); + debug_dma_unmap_page(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_unmap_page_attrs); + +/* + * dma_maps_sg_attrs returns 0 on error and > 0 on success. + * It should never return a value < 0. + */ +int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, + enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + int ents; + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); + else + ents = ops->map_sg(dev, sg, nents, dir, attrs); + BUG_ON(ents < 0); + debug_dma_map_sg(dev, sg, nents, ents, dir); + + return ents; +} +EXPORT_SYMBOL(dma_map_sg_attrs); + +void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, + int nents, enum dma_data_direction dir, + unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + debug_dma_unmap_sg(dev, sg, nents, dir); + if (dma_is_direct(ops)) + dma_direct_unmap_sg(dev, sg, nents, dir, attrs); + else if (ops->unmap_sg) + ops->unmap_sg(dev, sg, nents, dir, attrs); +} +EXPORT_SYMBOL(dma_unmap_sg_attrs); + +dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + dma_addr_t addr = DMA_MAPPING_ERROR; + + BUG_ON(!valid_dma_direction(dir)); + + /* Don't allow RAM to be mapped */ + if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) + return DMA_MAPPING_ERROR; + + if (dma_is_direct(ops)) + addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); + else if (ops->map_resource) + addr = ops->map_resource(dev, phys_addr, size, dir, attrs); + + debug_dma_map_resource(dev, phys_addr, size, dir, addr); + return addr; +} +EXPORT_SYMBOL(dma_map_resource); + +void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (!dma_is_direct(ops) && ops->unmap_resource) + ops->unmap_resource(dev, addr, size, dir, attrs); + debug_dma_unmap_resource(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_unmap_resource); + +void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_single_for_cpu(dev, addr, size, dir); + else if (ops->sync_single_for_cpu) + ops->sync_single_for_cpu(dev, addr, size, dir); + debug_dma_sync_single_for_cpu(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_sync_single_for_cpu); + +void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_single_for_device(dev, addr, size, dir); + else if (ops->sync_single_for_device) + ops->sync_single_for_device(dev, addr, size, dir); + debug_dma_sync_single_for_device(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_sync_single_for_device); + +void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); + else if (ops->sync_sg_for_cpu) + ops->sync_sg_for_cpu(dev, sg, nelems, dir); + debug_dma_sync_sg_for_cpu(dev, sg, nelems, dir); +} +EXPORT_SYMBOL(dma_sync_sg_for_cpu); + +void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_sg_for_device(dev, sg, nelems, dir); + else if (ops->sync_sg_for_device) + ops->sync_sg_for_device(dev, sg, nelems, dir); + debug_dma_sync_sg_for_device(dev, sg, nelems, dir); +} +EXPORT_SYMBOL(dma_sync_sg_for_device); + /* * Create scatter-list for the already allocated DMA buffer. */ -- 2.25.1 ^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-04-14 12:25 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, linuxppc-dev For a long time the DMA API has been implemented inline in dma-mapping.h, but the function bodies can be quite large. Move them all out of line. Signed-off-by: Christoph Hellwig <hch@lst.de> --- include/linux/dma-direct.h | 58 +++++++++ include/linux/dma-mapping.h | 247 ++++-------------------------------- kernel/dma/direct.c | 9 -- kernel/dma/mapping.c | 164 ++++++++++++++++++++++++ 4 files changed, 244 insertions(+), 234 deletions(-) diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index 24b8684aa21d..da689ad5fffd 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -85,4 +85,62 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma, void *cpu_addr, dma_addr_t dma_addr, size_t size, unsigned long attrs); int dma_direct_supported(struct device *dev, u64 mask); +dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, + unsigned long offset, size_t size, enum dma_data_direction dir, + unsigned long attrs); +int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, + enum dma_data_direction dir, unsigned long attrs); +dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, + size_t size, enum dma_data_direction dir, unsigned long attrs); + +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ + defined(CONFIG_SWIOTLB) +void dma_direct_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir); +void dma_direct_sync_sg_for_device(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir); +#else +static inline void dma_direct_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ +} +static inline void dma_direct_sync_sg_for_device(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir) +{ +} +#endif + +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ + defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ + defined(CONFIG_SWIOTLB) +void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir, unsigned long attrs); +void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, + int nents, enum dma_data_direction dir, unsigned long attrs); +void dma_direct_sync_single_for_cpu(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir); +void dma_direct_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir); +#else +static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ +} +static inline void dma_direct_unmap_sg(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir, + unsigned long attrs) +{ +} +static inline void dma_direct_sync_single_for_cpu(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ +} +static inline void dma_direct_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir) +{ +} +#endif + +size_t dma_direct_max_mapping_size(struct device *dev); + #endif /* _LINUX_DMA_DIRECT_H */ diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 330ad58fbf4d..793ad775cd54 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -188,73 +188,6 @@ static inline int dma_mmap_from_global_coherent(struct vm_area_struct *vma, } #endif /* CONFIG_DMA_DECLARE_COHERENT */ -static inline bool dma_is_direct(const struct dma_map_ops *ops) -{ - return likely(!ops); -} - -/* - * All the dma_direct_* declarations are here just for the indirect call bypass, - * and must not be used directly drivers! - */ -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs); -int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, - enum dma_data_direction dir, unsigned long attrs); -dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, - size_t size, enum dma_data_direction dir, unsigned long attrs); - -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ - defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); -void dma_direct_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir); -#else -static inline void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ -} -static inline void dma_direct_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ -} -#endif - -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ - defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ - defined(CONFIG_SWIOTLB) -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs); -void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, - int nents, enum dma_data_direction dir, unsigned long attrs); -void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); -void dma_direct_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir); -#else -static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ -} -static inline void dma_direct_unmap_sg(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir, - unsigned long attrs) -{ -} -static inline void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ -} -static inline void dma_direct_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ -} -#endif - -size_t dma_direct_max_mapping_size(struct device *dev); - #ifdef CONFIG_HAS_DMA #include <asm/dma-mapping.h> @@ -271,164 +204,6 @@ static inline void set_dma_ops(struct device *dev, dev->dma_ops = dma_ops; } -static inline dma_addr_t dma_map_page_attrs(struct device *dev, - struct page *page, size_t offset, size_t size, - enum dma_data_direction dir, unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - dma_addr_t addr; - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); - else - addr = ops->map_page(dev, page, offset, size, dir, attrs); - debug_dma_map_page(dev, page, offset, size, dir, addr); - - return addr; -} - -static inline void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_unmap_page(dev, addr, size, dir, attrs); - else if (ops->unmap_page) - ops->unmap_page(dev, addr, size, dir, attrs); - debug_dma_unmap_page(dev, addr, size, dir); -} - -/* - * dma_maps_sg_attrs returns 0 on error and > 0 on success. - * It should never return a value < 0. - */ -static inline int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - int ents; - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); - else - ents = ops->map_sg(dev, sg, nents, dir, attrs); - BUG_ON(ents < 0); - debug_dma_map_sg(dev, sg, nents, ents, dir); - - return ents; -} - -static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - debug_dma_unmap_sg(dev, sg, nents, dir); - if (dma_is_direct(ops)) - dma_direct_unmap_sg(dev, sg, nents, dir, attrs); - else if (ops->unmap_sg) - ops->unmap_sg(dev, sg, nents, dir, attrs); -} - -static inline dma_addr_t dma_map_resource(struct device *dev, - phys_addr_t phys_addr, - size_t size, - enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - dma_addr_t addr = DMA_MAPPING_ERROR; - - BUG_ON(!valid_dma_direction(dir)); - - /* Don't allow RAM to be mapped */ - if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) - return DMA_MAPPING_ERROR; - - if (dma_is_direct(ops)) - addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); - else if (ops->map_resource) - addr = ops->map_resource(dev, phys_addr, size, dir, attrs); - - debug_dma_map_resource(dev, phys_addr, size, dir, addr); - return addr; -} - -static inline void dma_unmap_resource(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (!dma_is_direct(ops) && ops->unmap_resource) - ops->unmap_resource(dev, addr, size, dir, attrs); - debug_dma_unmap_resource(dev, addr, size, dir); -} - -static inline void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, - size_t size, - enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_single_for_cpu(dev, addr, size, dir); - else if (ops->sync_single_for_cpu) - ops->sync_single_for_cpu(dev, addr, size, dir); - debug_dma_sync_single_for_cpu(dev, addr, size, dir); -} - -static inline void dma_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, - enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_single_for_device(dev, addr, size, dir); - else if (ops->sync_single_for_device) - ops->sync_single_for_device(dev, addr, size, dir); - debug_dma_sync_single_for_device(dev, addr, size, dir); -} - -static inline void -dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, - int nelems, enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); - else if (ops->sync_sg_for_cpu) - ops->sync_sg_for_cpu(dev, sg, nelems, dir); - debug_dma_sync_sg_for_cpu(dev, sg, nelems, dir); -} - -static inline void -dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, - int nelems, enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_sg_for_device(dev, sg, nelems, dir); - else if (ops->sync_sg_for_device) - ops->sync_sg_for_device(dev, sg, nelems, dir); - debug_dma_sync_sg_for_device(dev, sg, nelems, dir); - -} static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) { @@ -439,6 +214,28 @@ static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) return 0; } +dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs); +void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs); +int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, + enum dma_data_direction dir, unsigned long attrs); +void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, + int nents, enum dma_data_direction dir, + unsigned long attrs); +dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, + size_t size, enum dma_data_direction dir, unsigned long attrs); +void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs); +void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir); +void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir); +void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir); +void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir); void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag, unsigned long attrs); void dma_free_attrs(struct device *dev, size_t size, void *cpu_addr, diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 8f4bbdaf965e..f1a9099a4b5b 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -260,7 +260,6 @@ void dma_direct_sync_single_for_device(struct device *dev, if (!dev_is_dma_coherent(dev)) arch_sync_dma_for_device(paddr, size, dir); } -EXPORT_SYMBOL(dma_direct_sync_single_for_device); void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) @@ -280,7 +279,6 @@ void dma_direct_sync_sg_for_device(struct device *dev, dir); } } -EXPORT_SYMBOL(dma_direct_sync_sg_for_device); #endif #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ @@ -299,7 +297,6 @@ void dma_direct_sync_single_for_cpu(struct device *dev, if (unlikely(is_swiotlb_buffer(paddr))) swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_CPU); } -EXPORT_SYMBOL(dma_direct_sync_single_for_cpu); void dma_direct_sync_sg_for_cpu(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) @@ -321,7 +318,6 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, if (!dev_is_dma_coherent(dev)) arch_sync_dma_for_cpu_all(); } -EXPORT_SYMBOL(dma_direct_sync_sg_for_cpu); void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs) @@ -334,7 +330,6 @@ void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, if (unlikely(is_swiotlb_buffer(phys))) swiotlb_tbl_unmap_single(dev, phys, size, size, dir, attrs); } -EXPORT_SYMBOL(dma_direct_unmap_page); void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) @@ -346,7 +341,6 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, dma_direct_unmap_page(dev, sg->dma_address, sg_dma_len(sg), dir, attrs); } -EXPORT_SYMBOL(dma_direct_unmap_sg); #endif dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, @@ -373,7 +367,6 @@ dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, arch_sync_dma_for_device(phys, size, dir); return dma_addr; } -EXPORT_SYMBOL(dma_direct_map_page); int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) @@ -395,7 +388,6 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, dma_direct_unmap_sg(dev, sgl, i, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC); return 0; } -EXPORT_SYMBOL(dma_direct_map_sg); dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, size_t size, enum dma_data_direction dir, unsigned long attrs) @@ -412,7 +404,6 @@ dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, return dma_addr; } -EXPORT_SYMBOL(dma_direct_map_resource); int dma_direct_get_sgtable(struct device *dev, struct sg_table *sgt, void *cpu_addr, dma_addr_t dma_addr, size_t size, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 98e3d873792e..8e4155f9ee69 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -105,6 +105,170 @@ void *dmam_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, } EXPORT_SYMBOL(dmam_alloc_attrs); +static inline bool dma_is_direct(const struct dma_map_ops *ops) +{ + return likely(!ops); +} + +dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + dma_addr_t addr; + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); + else + addr = ops->map_page(dev, page, offset, size, dir, attrs); + debug_dma_map_page(dev, page, offset, size, dir, addr); + + return addr; +} +EXPORT_SYMBOL(dma_map_page_attrs); + +void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_unmap_page(dev, addr, size, dir, attrs); + else if (ops->unmap_page) + ops->unmap_page(dev, addr, size, dir, attrs); + debug_dma_unmap_page(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_unmap_page_attrs); + +/* + * dma_maps_sg_attrs returns 0 on error and > 0 on success. + * It should never return a value < 0. + */ +int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, + enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + int ents; + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); + else + ents = ops->map_sg(dev, sg, nents, dir, attrs); + BUG_ON(ents < 0); + debug_dma_map_sg(dev, sg, nents, ents, dir); + + return ents; +} +EXPORT_SYMBOL(dma_map_sg_attrs); + +void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, + int nents, enum dma_data_direction dir, + unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + debug_dma_unmap_sg(dev, sg, nents, dir); + if (dma_is_direct(ops)) + dma_direct_unmap_sg(dev, sg, nents, dir, attrs); + else if (ops->unmap_sg) + ops->unmap_sg(dev, sg, nents, dir, attrs); +} +EXPORT_SYMBOL(dma_unmap_sg_attrs); + +dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + dma_addr_t addr = DMA_MAPPING_ERROR; + + BUG_ON(!valid_dma_direction(dir)); + + /* Don't allow RAM to be mapped */ + if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) + return DMA_MAPPING_ERROR; + + if (dma_is_direct(ops)) + addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); + else if (ops->map_resource) + addr = ops->map_resource(dev, phys_addr, size, dir, attrs); + + debug_dma_map_resource(dev, phys_addr, size, dir, addr); + return addr; +} +EXPORT_SYMBOL(dma_map_resource); + +void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (!dma_is_direct(ops) && ops->unmap_resource) + ops->unmap_resource(dev, addr, size, dir, attrs); + debug_dma_unmap_resource(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_unmap_resource); + +void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_single_for_cpu(dev, addr, size, dir); + else if (ops->sync_single_for_cpu) + ops->sync_single_for_cpu(dev, addr, size, dir); + debug_dma_sync_single_for_cpu(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_sync_single_for_cpu); + +void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_single_for_device(dev, addr, size, dir); + else if (ops->sync_single_for_device) + ops->sync_single_for_device(dev, addr, size, dir); + debug_dma_sync_single_for_device(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_sync_single_for_device); + +void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); + else if (ops->sync_sg_for_cpu) + ops->sync_sg_for_cpu(dev, sg, nelems, dir); + debug_dma_sync_sg_for_cpu(dev, sg, nelems, dir); +} +EXPORT_SYMBOL(dma_sync_sg_for_cpu); + +void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_sg_for_device(dev, sg, nelems, dir); + else if (ops->sync_sg_for_device) + ops->sync_sg_for_device(dev, sg, nelems, dir); + debug_dma_sync_sg_for_device(dev, sg, nelems, dir); +} +EXPORT_SYMBOL(dma_sync_sg_for_device); + /* * Create scatter-list for the already allocated DMA buffer. */ -- 2.25.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-04-14 12:25 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, linuxppc-dev, Lu Baolu For a long time the DMA API has been implemented inline in dma-mapping.h, but the function bodies can be quite large. Move them all out of line. Signed-off-by: Christoph Hellwig <hch@lst.de> --- include/linux/dma-direct.h | 58 +++++++++ include/linux/dma-mapping.h | 247 ++++-------------------------------- kernel/dma/direct.c | 9 -- kernel/dma/mapping.c | 164 ++++++++++++++++++++++++ 4 files changed, 244 insertions(+), 234 deletions(-) diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index 24b8684aa21d..da689ad5fffd 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -85,4 +85,62 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma, void *cpu_addr, dma_addr_t dma_addr, size_t size, unsigned long attrs); int dma_direct_supported(struct device *dev, u64 mask); +dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, + unsigned long offset, size_t size, enum dma_data_direction dir, + unsigned long attrs); +int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, + enum dma_data_direction dir, unsigned long attrs); +dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, + size_t size, enum dma_data_direction dir, unsigned long attrs); + +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ + defined(CONFIG_SWIOTLB) +void dma_direct_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir); +void dma_direct_sync_sg_for_device(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir); +#else +static inline void dma_direct_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ +} +static inline void dma_direct_sync_sg_for_device(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir) +{ +} +#endif + +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ + defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ + defined(CONFIG_SWIOTLB) +void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir, unsigned long attrs); +void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, + int nents, enum dma_data_direction dir, unsigned long attrs); +void dma_direct_sync_single_for_cpu(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir); +void dma_direct_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir); +#else +static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ +} +static inline void dma_direct_unmap_sg(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir, + unsigned long attrs) +{ +} +static inline void dma_direct_sync_single_for_cpu(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ +} +static inline void dma_direct_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir) +{ +} +#endif + +size_t dma_direct_max_mapping_size(struct device *dev); + #endif /* _LINUX_DMA_DIRECT_H */ diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h index 330ad58fbf4d..793ad775cd54 100644 --- a/include/linux/dma-mapping.h +++ b/include/linux/dma-mapping.h @@ -188,73 +188,6 @@ static inline int dma_mmap_from_global_coherent(struct vm_area_struct *vma, } #endif /* CONFIG_DMA_DECLARE_COHERENT */ -static inline bool dma_is_direct(const struct dma_map_ops *ops) -{ - return likely(!ops); -} - -/* - * All the dma_direct_* declarations are here just for the indirect call bypass, - * and must not be used directly drivers! - */ -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs); -int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, - enum dma_data_direction dir, unsigned long attrs); -dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, - size_t size, enum dma_data_direction dir, unsigned long attrs); - -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ - defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); -void dma_direct_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir); -#else -static inline void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ -} -static inline void dma_direct_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ -} -#endif - -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ - defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ - defined(CONFIG_SWIOTLB) -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs); -void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, - int nents, enum dma_data_direction dir, unsigned long attrs); -void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); -void dma_direct_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir); -#else -static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ -} -static inline void dma_direct_unmap_sg(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir, - unsigned long attrs) -{ -} -static inline void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ -} -static inline void dma_direct_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ -} -#endif - -size_t dma_direct_max_mapping_size(struct device *dev); - #ifdef CONFIG_HAS_DMA #include <asm/dma-mapping.h> @@ -271,164 +204,6 @@ static inline void set_dma_ops(struct device *dev, dev->dma_ops = dma_ops; } -static inline dma_addr_t dma_map_page_attrs(struct device *dev, - struct page *page, size_t offset, size_t size, - enum dma_data_direction dir, unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - dma_addr_t addr; - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); - else - addr = ops->map_page(dev, page, offset, size, dir, attrs); - debug_dma_map_page(dev, page, offset, size, dir, addr); - - return addr; -} - -static inline void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_unmap_page(dev, addr, size, dir, attrs); - else if (ops->unmap_page) - ops->unmap_page(dev, addr, size, dir, attrs); - debug_dma_unmap_page(dev, addr, size, dir); -} - -/* - * dma_maps_sg_attrs returns 0 on error and > 0 on success. - * It should never return a value < 0. - */ -static inline int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - int ents; - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); - else - ents = ops->map_sg(dev, sg, nents, dir, attrs); - BUG_ON(ents < 0); - debug_dma_map_sg(dev, sg, nents, ents, dir); - - return ents; -} - -static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, - int nents, enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - debug_dma_unmap_sg(dev, sg, nents, dir); - if (dma_is_direct(ops)) - dma_direct_unmap_sg(dev, sg, nents, dir, attrs); - else if (ops->unmap_sg) - ops->unmap_sg(dev, sg, nents, dir, attrs); -} - -static inline dma_addr_t dma_map_resource(struct device *dev, - phys_addr_t phys_addr, - size_t size, - enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - dma_addr_t addr = DMA_MAPPING_ERROR; - - BUG_ON(!valid_dma_direction(dir)); - - /* Don't allow RAM to be mapped */ - if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) - return DMA_MAPPING_ERROR; - - if (dma_is_direct(ops)) - addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); - else if (ops->map_resource) - addr = ops->map_resource(dev, phys_addr, size, dir, attrs); - - debug_dma_map_resource(dev, phys_addr, size, dir, addr); - return addr; -} - -static inline void dma_unmap_resource(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (!dma_is_direct(ops) && ops->unmap_resource) - ops->unmap_resource(dev, addr, size, dir, attrs); - debug_dma_unmap_resource(dev, addr, size, dir); -} - -static inline void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, - size_t size, - enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_single_for_cpu(dev, addr, size, dir); - else if (ops->sync_single_for_cpu) - ops->sync_single_for_cpu(dev, addr, size, dir); - debug_dma_sync_single_for_cpu(dev, addr, size, dir); -} - -static inline void dma_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, - enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_single_for_device(dev, addr, size, dir); - else if (ops->sync_single_for_device) - ops->sync_single_for_device(dev, addr, size, dir); - debug_dma_sync_single_for_device(dev, addr, size, dir); -} - -static inline void -dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, - int nelems, enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); - else if (ops->sync_sg_for_cpu) - ops->sync_sg_for_cpu(dev, sg, nelems, dir); - debug_dma_sync_sg_for_cpu(dev, sg, nelems, dir); -} - -static inline void -dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, - int nelems, enum dma_data_direction dir) -{ - const struct dma_map_ops *ops = get_dma_ops(dev); - - BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) - dma_direct_sync_sg_for_device(dev, sg, nelems, dir); - else if (ops->sync_sg_for_device) - ops->sync_sg_for_device(dev, sg, nelems, dir); - debug_dma_sync_sg_for_device(dev, sg, nelems, dir); - -} static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) { @@ -439,6 +214,28 @@ static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) return 0; } +dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs); +void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs); +int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, + enum dma_data_direction dir, unsigned long attrs); +void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, + int nents, enum dma_data_direction dir, + unsigned long attrs); +dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, + size_t size, enum dma_data_direction dir, unsigned long attrs); +void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs); +void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir); +void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir); +void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir); +void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir); void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag, unsigned long attrs); void dma_free_attrs(struct device *dev, size_t size, void *cpu_addr, diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 8f4bbdaf965e..f1a9099a4b5b 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -260,7 +260,6 @@ void dma_direct_sync_single_for_device(struct device *dev, if (!dev_is_dma_coherent(dev)) arch_sync_dma_for_device(paddr, size, dir); } -EXPORT_SYMBOL(dma_direct_sync_single_for_device); void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) @@ -280,7 +279,6 @@ void dma_direct_sync_sg_for_device(struct device *dev, dir); } } -EXPORT_SYMBOL(dma_direct_sync_sg_for_device); #endif #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ @@ -299,7 +297,6 @@ void dma_direct_sync_single_for_cpu(struct device *dev, if (unlikely(is_swiotlb_buffer(paddr))) swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_CPU); } -EXPORT_SYMBOL(dma_direct_sync_single_for_cpu); void dma_direct_sync_sg_for_cpu(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) @@ -321,7 +318,6 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, if (!dev_is_dma_coherent(dev)) arch_sync_dma_for_cpu_all(); } -EXPORT_SYMBOL(dma_direct_sync_sg_for_cpu); void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir, unsigned long attrs) @@ -334,7 +330,6 @@ void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, if (unlikely(is_swiotlb_buffer(phys))) swiotlb_tbl_unmap_single(dev, phys, size, size, dir, attrs); } -EXPORT_SYMBOL(dma_direct_unmap_page); void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) @@ -346,7 +341,6 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, dma_direct_unmap_page(dev, sg->dma_address, sg_dma_len(sg), dir, attrs); } -EXPORT_SYMBOL(dma_direct_unmap_sg); #endif dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, @@ -373,7 +367,6 @@ dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, arch_sync_dma_for_device(phys, size, dir); return dma_addr; } -EXPORT_SYMBOL(dma_direct_map_page); int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) @@ -395,7 +388,6 @@ int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, dma_direct_unmap_sg(dev, sgl, i, dir, attrs | DMA_ATTR_SKIP_CPU_SYNC); return 0; } -EXPORT_SYMBOL(dma_direct_map_sg); dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, size_t size, enum dma_data_direction dir, unsigned long attrs) @@ -412,7 +404,6 @@ dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, return dma_addr; } -EXPORT_SYMBOL(dma_direct_map_resource); int dma_direct_get_sgtable(struct device *dev, struct sg_table *sgt, void *cpu_addr, dma_addr_t dma_addr, size_t size, diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 98e3d873792e..8e4155f9ee69 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -105,6 +105,170 @@ void *dmam_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, } EXPORT_SYMBOL(dmam_alloc_attrs); +static inline bool dma_is_direct(const struct dma_map_ops *ops) +{ + return likely(!ops); +} + +dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, + size_t offset, size_t size, enum dma_data_direction dir, + unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + dma_addr_t addr; + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); + else + addr = ops->map_page(dev, page, offset, size, dir, attrs); + debug_dma_map_page(dev, page, offset, size, dir, addr); + + return addr; +} +EXPORT_SYMBOL(dma_map_page_attrs); + +void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_unmap_page(dev, addr, size, dir, attrs); + else if (ops->unmap_page) + ops->unmap_page(dev, addr, size, dir, attrs); + debug_dma_unmap_page(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_unmap_page_attrs); + +/* + * dma_maps_sg_attrs returns 0 on error and > 0 on success. + * It should never return a value < 0. + */ +int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, + enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + int ents; + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); + else + ents = ops->map_sg(dev, sg, nents, dir, attrs); + BUG_ON(ents < 0); + debug_dma_map_sg(dev, sg, nents, ents, dir); + + return ents; +} +EXPORT_SYMBOL(dma_map_sg_attrs); + +void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, + int nents, enum dma_data_direction dir, + unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + debug_dma_unmap_sg(dev, sg, nents, dir); + if (dma_is_direct(ops)) + dma_direct_unmap_sg(dev, sg, nents, dir, attrs); + else if (ops->unmap_sg) + ops->unmap_sg(dev, sg, nents, dir, attrs); +} +EXPORT_SYMBOL(dma_unmap_sg_attrs); + +dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + dma_addr_t addr = DMA_MAPPING_ERROR; + + BUG_ON(!valid_dma_direction(dir)); + + /* Don't allow RAM to be mapped */ + if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) + return DMA_MAPPING_ERROR; + + if (dma_is_direct(ops)) + addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); + else if (ops->map_resource) + addr = ops->map_resource(dev, phys_addr, size, dir, attrs); + + debug_dma_map_resource(dev, phys_addr, size, dir, addr); + return addr; +} +EXPORT_SYMBOL(dma_map_resource); + +void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir, unsigned long attrs) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (!dma_is_direct(ops) && ops->unmap_resource) + ops->unmap_resource(dev, addr, size, dir, attrs); + debug_dma_unmap_resource(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_unmap_resource); + +void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, + enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_single_for_cpu(dev, addr, size, dir); + else if (ops->sync_single_for_cpu) + ops->sync_single_for_cpu(dev, addr, size, dir); + debug_dma_sync_single_for_cpu(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_sync_single_for_cpu); + +void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_single_for_device(dev, addr, size, dir); + else if (ops->sync_single_for_device) + ops->sync_single_for_device(dev, addr, size, dir); + debug_dma_sync_single_for_device(dev, addr, size, dir); +} +EXPORT_SYMBOL(dma_sync_single_for_device); + +void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); + else if (ops->sync_sg_for_cpu) + ops->sync_sg_for_cpu(dev, sg, nelems, dir); + debug_dma_sync_sg_for_cpu(dev, sg, nelems, dir); +} +EXPORT_SYMBOL(dma_sync_sg_for_cpu); + +void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, + int nelems, enum dma_data_direction dir) +{ + const struct dma_map_ops *ops = get_dma_ops(dev); + + BUG_ON(!valid_dma_direction(dir)); + if (dma_is_direct(ops)) + dma_direct_sync_sg_for_device(dev, sg, nelems, dir); + else if (ops->sync_sg_for_device) + ops->sync_sg_for_device(dev, sg, nelems, dir); + debug_dma_sync_sg_for_device(dev, sg, nelems, dir); +} +EXPORT_SYMBOL(dma_sync_sg_for_device); + /* * Create scatter-list for the already allocated DMA buffer. */ -- 2.25.1 ^ permalink raw reply related [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line 2020-04-14 12:25 ` Christoph Hellwig (?) @ 2020-04-15 2:26 ` Alexey Kardashevskiy -1 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-04-15 2:26 UTC (permalink / raw) To: Christoph Hellwig, iommu Cc: linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, Michael Ellerman On 14/04/2020 22:25, Christoph Hellwig wrote: > For a long time the DMA API has been implemented inline in dma-mapping.h, > but the function bodies can be quite large. Move them all out of line. > > Signed-off-by: Christoph Hellwig <hch@lst.de> > --- > include/linux/dma-direct.h | 58 +++++++++ > include/linux/dma-mapping.h | 247 ++++-------------------------------- > kernel/dma/direct.c | 9 -- > kernel/dma/mapping.c | 164 ++++++++++++++++++++++++ > 4 files changed, 244 insertions(+), 234 deletions(-) > > diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h > index 24b8684aa21d..da689ad5fffd 100644 > --- a/include/linux/dma-direct.h > +++ b/include/linux/dma-direct.h > @@ -85,4 +85,62 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma, > void *cpu_addr, dma_addr_t dma_addr, size_t size, > unsigned long attrs); > int dma_direct_supported(struct device *dev, u64 mask); > +dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, > + unsigned long offset, size_t size, enum dma_data_direction dir, > + unsigned long attrs); > +int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, > + enum dma_data_direction dir, unsigned long attrs); > +dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, > + size_t size, enum dma_data_direction dir, unsigned long attrs); > + > +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > + defined(CONFIG_SWIOTLB) > +void dma_direct_sync_single_for_device(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir); > +void dma_direct_sync_sg_for_device(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir); > +#else > +static inline void dma_direct_sync_single_for_device(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir) > +{ > +} > +static inline void dma_direct_sync_sg_for_device(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir) > +{ > +} > +#endif > + > +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ > + defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ > + defined(CONFIG_SWIOTLB) > +void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > + size_t size, enum dma_data_direction dir, unsigned long attrs); > +void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, > + int nents, enum dma_data_direction dir, unsigned long attrs); > +void dma_direct_sync_single_for_cpu(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir); > +void dma_direct_sync_sg_for_cpu(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir); > +#else > +static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > + size_t size, enum dma_data_direction dir, unsigned long attrs) > +{ > +} > +static inline void dma_direct_unmap_sg(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir, > + unsigned long attrs) > +{ > +} > +static inline void dma_direct_sync_single_for_cpu(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir) > +{ > +} > +static inline void dma_direct_sync_sg_for_cpu(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir) > +{ > +} > +#endif > + > +size_t dma_direct_max_mapping_size(struct device *dev); > + > #endif /* _LINUX_DMA_DIRECT_H */ > diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h > index 330ad58fbf4d..793ad775cd54 100644 > --- a/include/linux/dma-mapping.h > +++ b/include/linux/dma-mapping.h > @@ -188,73 +188,6 @@ static inline int dma_mmap_from_global_coherent(struct vm_area_struct *vma, > } > #endif /* CONFIG_DMA_DECLARE_COHERENT */ > > -static inline bool dma_is_direct(const struct dma_map_ops *ops) > -{ > - return likely(!ops); > -} > - > -/* > - * All the dma_direct_* declarations are here just for the indirect call bypass, > - * and must not be used directly drivers! > - */ > -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, > - unsigned long offset, size_t size, enum dma_data_direction dir, > - unsigned long attrs); > -int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, > - enum dma_data_direction dir, unsigned long attrs); > -dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, > - size_t size, enum dma_data_direction dir, unsigned long attrs); > - > -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > - defined(CONFIG_SWIOTLB) > -void dma_direct_sync_single_for_device(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir); > -void dma_direct_sync_sg_for_device(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir); > -#else > -static inline void dma_direct_sync_single_for_device(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir) > -{ > -} > -static inline void dma_direct_sync_sg_for_device(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir) > -{ > -} > -#endif > - > -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ > - defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ > - defined(CONFIG_SWIOTLB) > -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, unsigned long attrs); > -void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, > - int nents, enum dma_data_direction dir, unsigned long attrs); > -void dma_direct_sync_single_for_cpu(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir); > -void dma_direct_sync_sg_for_cpu(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir); > -#else > -static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, unsigned long attrs) > -{ > -} > -static inline void dma_direct_unmap_sg(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir, > - unsigned long attrs) > -{ > -} > -static inline void dma_direct_sync_single_for_cpu(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir) > -{ > -} > -static inline void dma_direct_sync_sg_for_cpu(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir) > -{ > -} > -#endif > - > -size_t dma_direct_max_mapping_size(struct device *dev); > - > #ifdef CONFIG_HAS_DMA > #include <asm/dma-mapping.h> > > @@ -271,164 +204,6 @@ static inline void set_dma_ops(struct device *dev, > dev->dma_ops = dma_ops; > } > > -static inline dma_addr_t dma_map_page_attrs(struct device *dev, > - struct page *page, size_t offset, size_t size, > - enum dma_data_direction dir, unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - dma_addr_t addr; > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); > - else > - addr = ops->map_page(dev, page, offset, size, dir, attrs); > - debug_dma_map_page(dev, page, offset, size, dir, addr); > - > - return addr; > -} > - > -static inline void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_unmap_page(dev, addr, size, dir, attrs); > - else if (ops->unmap_page) > - ops->unmap_page(dev, addr, size, dir, attrs); > - debug_dma_unmap_page(dev, addr, size, dir); > -} > - > -/* > - * dma_maps_sg_attrs returns 0 on error and > 0 on success. > - * It should never return a value < 0. > - */ > -static inline int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, > - int nents, enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - int ents; > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); > - else > - ents = ops->map_sg(dev, sg, nents, dir, attrs); > - BUG_ON(ents < 0); > - debug_dma_map_sg(dev, sg, nents, ents, dir); > - > - return ents; > -} > - > -static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, > - int nents, enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - debug_dma_unmap_sg(dev, sg, nents, dir); > - if (dma_is_direct(ops)) > - dma_direct_unmap_sg(dev, sg, nents, dir, attrs); > - else if (ops->unmap_sg) > - ops->unmap_sg(dev, sg, nents, dir, attrs); > -} > - > -static inline dma_addr_t dma_map_resource(struct device *dev, > - phys_addr_t phys_addr, > - size_t size, > - enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - dma_addr_t addr = DMA_MAPPING_ERROR; > - > - BUG_ON(!valid_dma_direction(dir)); > - > - /* Don't allow RAM to be mapped */ > - if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) > - return DMA_MAPPING_ERROR; > - > - if (dma_is_direct(ops)) > - addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); > - else if (ops->map_resource) > - addr = ops->map_resource(dev, phys_addr, size, dir, attrs); > - > - debug_dma_map_resource(dev, phys_addr, size, dir, addr); > - return addr; > -} > - > -static inline void dma_unmap_resource(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (!dma_is_direct(ops) && ops->unmap_resource) > - ops->unmap_resource(dev, addr, size, dir, attrs); > - debug_dma_unmap_resource(dev, addr, size, dir); > -} > - > -static inline void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, > - size_t size, > - enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_single_for_cpu(dev, addr, size, dir); > - else if (ops->sync_single_for_cpu) > - ops->sync_single_for_cpu(dev, addr, size, dir); > - debug_dma_sync_single_for_cpu(dev, addr, size, dir); > -} > - > -static inline void dma_sync_single_for_device(struct device *dev, > - dma_addr_t addr, size_t size, > - enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_single_for_device(dev, addr, size, dir); > - else if (ops->sync_single_for_device) > - ops->sync_single_for_device(dev, addr, size, dir); > - debug_dma_sync_single_for_device(dev, addr, size, dir); > -} > - > -static inline void > -dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, > - int nelems, enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); > - else if (ops->sync_sg_for_cpu) > - ops->sync_sg_for_cpu(dev, sg, nelems, dir); > - debug_dma_sync_sg_for_cpu(dev, sg, nelems, dir); > -} > - > -static inline void > -dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, > - int nelems, enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_sg_for_device(dev, sg, nelems, dir); > - else if (ops->sync_sg_for_device) > - ops->sync_sg_for_device(dev, sg, nelems, dir); > - debug_dma_sync_sg_for_device(dev, sg, nelems, dir); > - > -} > > static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) > { > @@ -439,6 +214,28 @@ static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) > return 0; > } > > +dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, > + size_t offset, size_t size, enum dma_data_direction dir, > + unsigned long attrs); > +void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, > + enum dma_data_direction dir, unsigned long attrs); > +int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, > + enum dma_data_direction dir, unsigned long attrs); > +void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, > + int nents, enum dma_data_direction dir, > + unsigned long attrs); > +dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, > + size_t size, enum dma_data_direction dir, unsigned long attrs); > +void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, > + enum dma_data_direction dir, unsigned long attrs); > +void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, > + enum dma_data_direction dir); > +void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, > + size_t size, enum dma_data_direction dir); > +void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, > + int nelems, enum dma_data_direction dir); > +void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, > + int nelems, enum dma_data_direction dir); > void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > gfp_t flag, unsigned long attrs); > void dma_free_attrs(struct device *dev, size_t size, void *cpu_addr, > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 8f4bbdaf965e..f1a9099a4b5b 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -260,7 +260,6 @@ void dma_direct_sync_single_for_device(struct device *dev, > if (!dev_is_dma_coherent(dev)) > arch_sync_dma_for_device(paddr, size, dir); > } > -EXPORT_SYMBOL(dma_direct_sync_single_for_device); May be this is correct and allowed (no idea) but removing exported symbols at least deserves a mention in the commit log, does not it? The rest of the series is fine and works. Thanks, -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-04-15 2:26 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-04-15 2:26 UTC (permalink / raw) To: Christoph Hellwig, iommu Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, Michael Ellerman, linuxppc-dev On 14/04/2020 22:25, Christoph Hellwig wrote: > For a long time the DMA API has been implemented inline in dma-mapping.h, > but the function bodies can be quite large. Move them all out of line. > > Signed-off-by: Christoph Hellwig <hch@lst.de> > --- > include/linux/dma-direct.h | 58 +++++++++ > include/linux/dma-mapping.h | 247 ++++-------------------------------- > kernel/dma/direct.c | 9 -- > kernel/dma/mapping.c | 164 ++++++++++++++++++++++++ > 4 files changed, 244 insertions(+), 234 deletions(-) > > diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h > index 24b8684aa21d..da689ad5fffd 100644 > --- a/include/linux/dma-direct.h > +++ b/include/linux/dma-direct.h > @@ -85,4 +85,62 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma, > void *cpu_addr, dma_addr_t dma_addr, size_t size, > unsigned long attrs); > int dma_direct_supported(struct device *dev, u64 mask); > +dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, > + unsigned long offset, size_t size, enum dma_data_direction dir, > + unsigned long attrs); > +int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, > + enum dma_data_direction dir, unsigned long attrs); > +dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, > + size_t size, enum dma_data_direction dir, unsigned long attrs); > + > +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > + defined(CONFIG_SWIOTLB) > +void dma_direct_sync_single_for_device(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir); > +void dma_direct_sync_sg_for_device(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir); > +#else > +static inline void dma_direct_sync_single_for_device(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir) > +{ > +} > +static inline void dma_direct_sync_sg_for_device(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir) > +{ > +} > +#endif > + > +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ > + defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ > + defined(CONFIG_SWIOTLB) > +void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > + size_t size, enum dma_data_direction dir, unsigned long attrs); > +void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, > + int nents, enum dma_data_direction dir, unsigned long attrs); > +void dma_direct_sync_single_for_cpu(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir); > +void dma_direct_sync_sg_for_cpu(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir); > +#else > +static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > + size_t size, enum dma_data_direction dir, unsigned long attrs) > +{ > +} > +static inline void dma_direct_unmap_sg(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir, > + unsigned long attrs) > +{ > +} > +static inline void dma_direct_sync_single_for_cpu(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir) > +{ > +} > +static inline void dma_direct_sync_sg_for_cpu(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir) > +{ > +} > +#endif > + > +size_t dma_direct_max_mapping_size(struct device *dev); > + > #endif /* _LINUX_DMA_DIRECT_H */ > diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h > index 330ad58fbf4d..793ad775cd54 100644 > --- a/include/linux/dma-mapping.h > +++ b/include/linux/dma-mapping.h > @@ -188,73 +188,6 @@ static inline int dma_mmap_from_global_coherent(struct vm_area_struct *vma, > } > #endif /* CONFIG_DMA_DECLARE_COHERENT */ > > -static inline bool dma_is_direct(const struct dma_map_ops *ops) > -{ > - return likely(!ops); > -} > - > -/* > - * All the dma_direct_* declarations are here just for the indirect call bypass, > - * and must not be used directly drivers! > - */ > -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, > - unsigned long offset, size_t size, enum dma_data_direction dir, > - unsigned long attrs); > -int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, > - enum dma_data_direction dir, unsigned long attrs); > -dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, > - size_t size, enum dma_data_direction dir, unsigned long attrs); > - > -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > - defined(CONFIG_SWIOTLB) > -void dma_direct_sync_single_for_device(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir); > -void dma_direct_sync_sg_for_device(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir); > -#else > -static inline void dma_direct_sync_single_for_device(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir) > -{ > -} > -static inline void dma_direct_sync_sg_for_device(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir) > -{ > -} > -#endif > - > -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ > - defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ > - defined(CONFIG_SWIOTLB) > -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, unsigned long attrs); > -void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, > - int nents, enum dma_data_direction dir, unsigned long attrs); > -void dma_direct_sync_single_for_cpu(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir); > -void dma_direct_sync_sg_for_cpu(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir); > -#else > -static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, unsigned long attrs) > -{ > -} > -static inline void dma_direct_unmap_sg(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir, > - unsigned long attrs) > -{ > -} > -static inline void dma_direct_sync_single_for_cpu(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir) > -{ > -} > -static inline void dma_direct_sync_sg_for_cpu(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir) > -{ > -} > -#endif > - > -size_t dma_direct_max_mapping_size(struct device *dev); > - > #ifdef CONFIG_HAS_DMA > #include <asm/dma-mapping.h> > > @@ -271,164 +204,6 @@ static inline void set_dma_ops(struct device *dev, > dev->dma_ops = dma_ops; > } > > -static inline dma_addr_t dma_map_page_attrs(struct device *dev, > - struct page *page, size_t offset, size_t size, > - enum dma_data_direction dir, unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - dma_addr_t addr; > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); > - else > - addr = ops->map_page(dev, page, offset, size, dir, attrs); > - debug_dma_map_page(dev, page, offset, size, dir, addr); > - > - return addr; > -} > - > -static inline void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_unmap_page(dev, addr, size, dir, attrs); > - else if (ops->unmap_page) > - ops->unmap_page(dev, addr, size, dir, attrs); > - debug_dma_unmap_page(dev, addr, size, dir); > -} > - > -/* > - * dma_maps_sg_attrs returns 0 on error and > 0 on success. > - * It should never return a value < 0. > - */ > -static inline int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, > - int nents, enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - int ents; > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); > - else > - ents = ops->map_sg(dev, sg, nents, dir, attrs); > - BUG_ON(ents < 0); > - debug_dma_map_sg(dev, sg, nents, ents, dir); > - > - return ents; > -} > - > -static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, > - int nents, enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - debug_dma_unmap_sg(dev, sg, nents, dir); > - if (dma_is_direct(ops)) > - dma_direct_unmap_sg(dev, sg, nents, dir, attrs); > - else if (ops->unmap_sg) > - ops->unmap_sg(dev, sg, nents, dir, attrs); > -} > - > -static inline dma_addr_t dma_map_resource(struct device *dev, > - phys_addr_t phys_addr, > - size_t size, > - enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - dma_addr_t addr = DMA_MAPPING_ERROR; > - > - BUG_ON(!valid_dma_direction(dir)); > - > - /* Don't allow RAM to be mapped */ > - if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) > - return DMA_MAPPING_ERROR; > - > - if (dma_is_direct(ops)) > - addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); > - else if (ops->map_resource) > - addr = ops->map_resource(dev, phys_addr, size, dir, attrs); > - > - debug_dma_map_resource(dev, phys_addr, size, dir, addr); > - return addr; > -} > - > -static inline void dma_unmap_resource(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (!dma_is_direct(ops) && ops->unmap_resource) > - ops->unmap_resource(dev, addr, size, dir, attrs); > - debug_dma_unmap_resource(dev, addr, size, dir); > -} > - > -static inline void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, > - size_t size, > - enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_single_for_cpu(dev, addr, size, dir); > - else if (ops->sync_single_for_cpu) > - ops->sync_single_for_cpu(dev, addr, size, dir); > - debug_dma_sync_single_for_cpu(dev, addr, size, dir); > -} > - > -static inline void dma_sync_single_for_device(struct device *dev, > - dma_addr_t addr, size_t size, > - enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_single_for_device(dev, addr, size, dir); > - else if (ops->sync_single_for_device) > - ops->sync_single_for_device(dev, addr, size, dir); > - debug_dma_sync_single_for_device(dev, addr, size, dir); > -} > - > -static inline void > -dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, > - int nelems, enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); > - else if (ops->sync_sg_for_cpu) > - ops->sync_sg_for_cpu(dev, sg, nelems, dir); > - debug_dma_sync_sg_for_cpu(dev, sg, nelems, dir); > -} > - > -static inline void > -dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, > - int nelems, enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_sg_for_device(dev, sg, nelems, dir); > - else if (ops->sync_sg_for_device) > - ops->sync_sg_for_device(dev, sg, nelems, dir); > - debug_dma_sync_sg_for_device(dev, sg, nelems, dir); > - > -} > > static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) > { > @@ -439,6 +214,28 @@ static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) > return 0; > } > > +dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, > + size_t offset, size_t size, enum dma_data_direction dir, > + unsigned long attrs); > +void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, > + enum dma_data_direction dir, unsigned long attrs); > +int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, > + enum dma_data_direction dir, unsigned long attrs); > +void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, > + int nents, enum dma_data_direction dir, > + unsigned long attrs); > +dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, > + size_t size, enum dma_data_direction dir, unsigned long attrs); > +void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, > + enum dma_data_direction dir, unsigned long attrs); > +void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, > + enum dma_data_direction dir); > +void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, > + size_t size, enum dma_data_direction dir); > +void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, > + int nelems, enum dma_data_direction dir); > +void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, > + int nelems, enum dma_data_direction dir); > void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > gfp_t flag, unsigned long attrs); > void dma_free_attrs(struct device *dev, size_t size, void *cpu_addr, > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 8f4bbdaf965e..f1a9099a4b5b 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -260,7 +260,6 @@ void dma_direct_sync_single_for_device(struct device *dev, > if (!dev_is_dma_coherent(dev)) > arch_sync_dma_for_device(paddr, size, dir); > } > -EXPORT_SYMBOL(dma_direct_sync_single_for_device); May be this is correct and allowed (no idea) but removing exported symbols at least deserves a mention in the commit log, does not it? The rest of the series is fine and works. Thanks, -- Alexey _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-04-15 2:26 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-04-15 2:26 UTC (permalink / raw) To: Christoph Hellwig, iommu Cc: Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, linuxppc-dev, Lu Baolu On 14/04/2020 22:25, Christoph Hellwig wrote: > For a long time the DMA API has been implemented inline in dma-mapping.h, > but the function bodies can be quite large. Move them all out of line. > > Signed-off-by: Christoph Hellwig <hch@lst.de> > --- > include/linux/dma-direct.h | 58 +++++++++ > include/linux/dma-mapping.h | 247 ++++-------------------------------- > kernel/dma/direct.c | 9 -- > kernel/dma/mapping.c | 164 ++++++++++++++++++++++++ > 4 files changed, 244 insertions(+), 234 deletions(-) > > diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h > index 24b8684aa21d..da689ad5fffd 100644 > --- a/include/linux/dma-direct.h > +++ b/include/linux/dma-direct.h > @@ -85,4 +85,62 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma, > void *cpu_addr, dma_addr_t dma_addr, size_t size, > unsigned long attrs); > int dma_direct_supported(struct device *dev, u64 mask); > +dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, > + unsigned long offset, size_t size, enum dma_data_direction dir, > + unsigned long attrs); > +int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, > + enum dma_data_direction dir, unsigned long attrs); > +dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, > + size_t size, enum dma_data_direction dir, unsigned long attrs); > + > +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > + defined(CONFIG_SWIOTLB) > +void dma_direct_sync_single_for_device(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir); > +void dma_direct_sync_sg_for_device(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir); > +#else > +static inline void dma_direct_sync_single_for_device(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir) > +{ > +} > +static inline void dma_direct_sync_sg_for_device(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir) > +{ > +} > +#endif > + > +#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ > + defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ > + defined(CONFIG_SWIOTLB) > +void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > + size_t size, enum dma_data_direction dir, unsigned long attrs); > +void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, > + int nents, enum dma_data_direction dir, unsigned long attrs); > +void dma_direct_sync_single_for_cpu(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir); > +void dma_direct_sync_sg_for_cpu(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir); > +#else > +static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > + size_t size, enum dma_data_direction dir, unsigned long attrs) > +{ > +} > +static inline void dma_direct_unmap_sg(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir, > + unsigned long attrs) > +{ > +} > +static inline void dma_direct_sync_single_for_cpu(struct device *dev, > + dma_addr_t addr, size_t size, enum dma_data_direction dir) > +{ > +} > +static inline void dma_direct_sync_sg_for_cpu(struct device *dev, > + struct scatterlist *sgl, int nents, enum dma_data_direction dir) > +{ > +} > +#endif > + > +size_t dma_direct_max_mapping_size(struct device *dev); > + > #endif /* _LINUX_DMA_DIRECT_H */ > diff --git a/include/linux/dma-mapping.h b/include/linux/dma-mapping.h > index 330ad58fbf4d..793ad775cd54 100644 > --- a/include/linux/dma-mapping.h > +++ b/include/linux/dma-mapping.h > @@ -188,73 +188,6 @@ static inline int dma_mmap_from_global_coherent(struct vm_area_struct *vma, > } > #endif /* CONFIG_DMA_DECLARE_COHERENT */ > > -static inline bool dma_is_direct(const struct dma_map_ops *ops) > -{ > - return likely(!ops); > -} > - > -/* > - * All the dma_direct_* declarations are here just for the indirect call bypass, > - * and must not be used directly drivers! > - */ > -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, > - unsigned long offset, size_t size, enum dma_data_direction dir, > - unsigned long attrs); > -int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, > - enum dma_data_direction dir, unsigned long attrs); > -dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, > - size_t size, enum dma_data_direction dir, unsigned long attrs); > - > -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ > - defined(CONFIG_SWIOTLB) > -void dma_direct_sync_single_for_device(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir); > -void dma_direct_sync_sg_for_device(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir); > -#else > -static inline void dma_direct_sync_single_for_device(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir) > -{ > -} > -static inline void dma_direct_sync_sg_for_device(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir) > -{ > -} > -#endif > - > -#if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ > - defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ > - defined(CONFIG_SWIOTLB) > -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, unsigned long attrs); > -void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, > - int nents, enum dma_data_direction dir, unsigned long attrs); > -void dma_direct_sync_single_for_cpu(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir); > -void dma_direct_sync_sg_for_cpu(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir); > -#else > -static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, unsigned long attrs) > -{ > -} > -static inline void dma_direct_unmap_sg(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir, > - unsigned long attrs) > -{ > -} > -static inline void dma_direct_sync_single_for_cpu(struct device *dev, > - dma_addr_t addr, size_t size, enum dma_data_direction dir) > -{ > -} > -static inline void dma_direct_sync_sg_for_cpu(struct device *dev, > - struct scatterlist *sgl, int nents, enum dma_data_direction dir) > -{ > -} > -#endif > - > -size_t dma_direct_max_mapping_size(struct device *dev); > - > #ifdef CONFIG_HAS_DMA > #include <asm/dma-mapping.h> > > @@ -271,164 +204,6 @@ static inline void set_dma_ops(struct device *dev, > dev->dma_ops = dma_ops; > } > > -static inline dma_addr_t dma_map_page_attrs(struct device *dev, > - struct page *page, size_t offset, size_t size, > - enum dma_data_direction dir, unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - dma_addr_t addr; > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); > - else > - addr = ops->map_page(dev, page, offset, size, dir, attrs); > - debug_dma_map_page(dev, page, offset, size, dir, addr); > - > - return addr; > -} > - > -static inline void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_unmap_page(dev, addr, size, dir, attrs); > - else if (ops->unmap_page) > - ops->unmap_page(dev, addr, size, dir, attrs); > - debug_dma_unmap_page(dev, addr, size, dir); > -} > - > -/* > - * dma_maps_sg_attrs returns 0 on error and > 0 on success. > - * It should never return a value < 0. > - */ > -static inline int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, > - int nents, enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - int ents; > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); > - else > - ents = ops->map_sg(dev, sg, nents, dir, attrs); > - BUG_ON(ents < 0); > - debug_dma_map_sg(dev, sg, nents, ents, dir); > - > - return ents; > -} > - > -static inline void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, > - int nents, enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - debug_dma_unmap_sg(dev, sg, nents, dir); > - if (dma_is_direct(ops)) > - dma_direct_unmap_sg(dev, sg, nents, dir, attrs); > - else if (ops->unmap_sg) > - ops->unmap_sg(dev, sg, nents, dir, attrs); > -} > - > -static inline dma_addr_t dma_map_resource(struct device *dev, > - phys_addr_t phys_addr, > - size_t size, > - enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - dma_addr_t addr = DMA_MAPPING_ERROR; > - > - BUG_ON(!valid_dma_direction(dir)); > - > - /* Don't allow RAM to be mapped */ > - if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) > - return DMA_MAPPING_ERROR; > - > - if (dma_is_direct(ops)) > - addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); > - else if (ops->map_resource) > - addr = ops->map_resource(dev, phys_addr, size, dir, attrs); > - > - debug_dma_map_resource(dev, phys_addr, size, dir, addr); > - return addr; > -} > - > -static inline void dma_unmap_resource(struct device *dev, dma_addr_t addr, > - size_t size, enum dma_data_direction dir, > - unsigned long attrs) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (!dma_is_direct(ops) && ops->unmap_resource) > - ops->unmap_resource(dev, addr, size, dir, attrs); > - debug_dma_unmap_resource(dev, addr, size, dir); > -} > - > -static inline void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, > - size_t size, > - enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_single_for_cpu(dev, addr, size, dir); > - else if (ops->sync_single_for_cpu) > - ops->sync_single_for_cpu(dev, addr, size, dir); > - debug_dma_sync_single_for_cpu(dev, addr, size, dir); > -} > - > -static inline void dma_sync_single_for_device(struct device *dev, > - dma_addr_t addr, size_t size, > - enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_single_for_device(dev, addr, size, dir); > - else if (ops->sync_single_for_device) > - ops->sync_single_for_device(dev, addr, size, dir); > - debug_dma_sync_single_for_device(dev, addr, size, dir); > -} > - > -static inline void > -dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, > - int nelems, enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); > - else if (ops->sync_sg_for_cpu) > - ops->sync_sg_for_cpu(dev, sg, nelems, dir); > - debug_dma_sync_sg_for_cpu(dev, sg, nelems, dir); > -} > - > -static inline void > -dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, > - int nelems, enum dma_data_direction dir) > -{ > - const struct dma_map_ops *ops = get_dma_ops(dev); > - > - BUG_ON(!valid_dma_direction(dir)); > - if (dma_is_direct(ops)) > - dma_direct_sync_sg_for_device(dev, sg, nelems, dir); > - else if (ops->sync_sg_for_device) > - ops->sync_sg_for_device(dev, sg, nelems, dir); > - debug_dma_sync_sg_for_device(dev, sg, nelems, dir); > - > -} > > static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) > { > @@ -439,6 +214,28 @@ static inline int dma_mapping_error(struct device *dev, dma_addr_t dma_addr) > return 0; > } > > +dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, > + size_t offset, size_t size, enum dma_data_direction dir, > + unsigned long attrs); > +void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, > + enum dma_data_direction dir, unsigned long attrs); > +int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, > + enum dma_data_direction dir, unsigned long attrs); > +void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, > + int nents, enum dma_data_direction dir, > + unsigned long attrs); > +dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, > + size_t size, enum dma_data_direction dir, unsigned long attrs); > +void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, > + enum dma_data_direction dir, unsigned long attrs); > +void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, > + enum dma_data_direction dir); > +void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, > + size_t size, enum dma_data_direction dir); > +void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, > + int nelems, enum dma_data_direction dir); > +void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, > + int nelems, enum dma_data_direction dir); > void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, > gfp_t flag, unsigned long attrs); > void dma_free_attrs(struct device *dev, size_t size, void *cpu_addr, > diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c > index 8f4bbdaf965e..f1a9099a4b5b 100644 > --- a/kernel/dma/direct.c > +++ b/kernel/dma/direct.c > @@ -260,7 +260,6 @@ void dma_direct_sync_single_for_device(struct device *dev, > if (!dev_is_dma_coherent(dev)) > arch_sync_dma_for_device(paddr, size, dir); > } > -EXPORT_SYMBOL(dma_direct_sync_single_for_device); May be this is correct and allowed (no idea) but removing exported symbols at least deserves a mention in the commit log, does not it? The rest of the series is fine and works. Thanks, -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line 2020-04-15 2:26 ` Alexey Kardashevskiy (?) @ 2020-04-15 6:18 ` Christoph Hellwig -1 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-15 6:18 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Christoph Hellwig, iommu, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, Michael Ellerman On Wed, Apr 15, 2020 at 12:26:04PM +1000, Alexey Kardashevskiy wrote: > May be this is correct and allowed (no idea) but removing exported > symbols at least deserves a mention in the commit log, does not it? > > The rest of the series is fine and works. Thanks, Maybe I can throw in a line, but the point is that dma_direct_* was exported as dma_* called them inline. Now dma_* is out of line and exported instead, which always was the actual API. ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-04-15 6:18 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-15 6:18 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, Michael Ellerman, linuxppc-dev, Christoph Hellwig On Wed, Apr 15, 2020 at 12:26:04PM +1000, Alexey Kardashevskiy wrote: > May be this is correct and allowed (no idea) but removing exported > symbols at least deserves a mention in the commit log, does not it? > > The rest of the series is fine and works. Thanks, Maybe I can throw in a line, but the point is that dma_direct_* was exported as dma_* called them inline. Now dma_* is out of line and exported instead, which always was the actual API. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-04-15 6:18 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-15 6:18 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, iommu, linuxppc-dev, Christoph Hellwig, Lu Baolu On Wed, Apr 15, 2020 at 12:26:04PM +1000, Alexey Kardashevskiy wrote: > May be this is correct and allowed (no idea) but removing exported > symbols at least deserves a mention in the commit log, does not it? > > The rest of the series is fine and works. Thanks, Maybe I can throw in a line, but the point is that dma_direct_* was exported as dma_* called them inline. Now dma_* is out of line and exported instead, which always was the actual API. ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line 2020-04-15 6:18 ` Christoph Hellwig (?) @ 2020-04-15 11:21 ` Alexey Kardashevskiy -1 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-04-15 11:21 UTC (permalink / raw) To: Christoph Hellwig Cc: iommu, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, Michael Ellerman On 15/04/2020 16:18, Christoph Hellwig wrote: > On Wed, Apr 15, 2020 at 12:26:04PM +1000, Alexey Kardashevskiy wrote: >> May be this is correct and allowed (no idea) but removing exported >> symbols at least deserves a mention in the commit log, does not it? >> >> The rest of the series is fine and works. Thanks, > > Maybe I can throw in a line, but the point is that dma_direct_* > was exported as dma_* called them inline. Now dma_* is out of line > and exported instead, which always was the actual API. They become inline in 2/4. And the fact they were exported leaves possibility that there is a driver somewhere relying on these symbols or distro kernel won't build because the symbol disappeared from exports (I do not know what KABI guarantees or if mainline kernel cares). I do not care in particular but some might, a line separated with empty lines in the commit log would do. -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-04-15 11:21 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-04-15 11:21 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, linuxppc-dev, linux-kernel, iommu, Michael Ellerman, Robin Murphy On 15/04/2020 16:18, Christoph Hellwig wrote: > On Wed, Apr 15, 2020 at 12:26:04PM +1000, Alexey Kardashevskiy wrote: >> May be this is correct and allowed (no idea) but removing exported >> symbols at least deserves a mention in the commit log, does not it? >> >> The rest of the series is fine and works. Thanks, > > Maybe I can throw in a line, but the point is that dma_direct_* > was exported as dma_* called them inline. Now dma_* is out of line > and exported instead, which always was the actual API. They become inline in 2/4. And the fact they were exported leaves possibility that there is a driver somewhere relying on these symbols or distro kernel won't build because the symbol disappeared from exports (I do not know what KABI guarantees or if mainline kernel cares). I do not care in particular but some might, a line separated with empty lines in the commit log would do. -- Alexey _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-04-15 11:21 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-04-15 11:21 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, Joerg Roedel, linuxppc-dev, linux-kernel, iommu, Robin Murphy, Lu Baolu On 15/04/2020 16:18, Christoph Hellwig wrote: > On Wed, Apr 15, 2020 at 12:26:04PM +1000, Alexey Kardashevskiy wrote: >> May be this is correct and allowed (no idea) but removing exported >> symbols at least deserves a mention in the commit log, does not it? >> >> The rest of the series is fine and works. Thanks, > > Maybe I can throw in a line, but the point is that dma_direct_* > was exported as dma_* called them inline. Now dma_* is out of line > and exported instead, which always was the actual API. They become inline in 2/4. And the fact they were exported leaves possibility that there is a driver somewhere relying on these symbols or distro kernel won't build because the symbol disappeared from exports (I do not know what KABI guarantees or if mainline kernel cares). I do not care in particular but some might, a line separated with empty lines in the commit log would do. -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line 2020-04-15 11:21 ` Alexey Kardashevskiy (?) @ 2020-04-17 7:58 ` Christoph Hellwig -1 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-17 7:58 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Christoph Hellwig, iommu, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, Michael Ellerman On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: > And the fact they were exported leaves possibility that there is a > driver somewhere relying on these symbols or distro kernel won't build > because the symbol disappeared from exports (I do not know what KABI > guarantees or if mainline kernel cares). We absolutely do not care. In fact for abuses of APIs that drivers should not use we almost care to make them private and break people abusing them. > I do not care in particular but > some might, a line separated with empty lines in the commit log would do. I'll add a blurb for the next version. ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-04-17 7:58 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-17 7:58 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, Michael Ellerman, linuxppc-dev, Christoph Hellwig On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: > And the fact they were exported leaves possibility that there is a > driver somewhere relying on these symbols or distro kernel won't build > because the symbol disappeared from exports (I do not know what KABI > guarantees or if mainline kernel cares). We absolutely do not care. In fact for abuses of APIs that drivers should not use we almost care to make them private and break people abusing them. > I do not care in particular but > some might, a line separated with empty lines in the commit log would do. I'll add a blurb for the next version. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-04-17 7:58 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-17 7:58 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, iommu, linuxppc-dev, Christoph Hellwig, Lu Baolu On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: > And the fact they were exported leaves possibility that there is a > driver somewhere relying on these symbols or distro kernel won't build > because the symbol disappeared from exports (I do not know what KABI > guarantees or if mainline kernel cares). We absolutely do not care. In fact for abuses of APIs that drivers should not use we almost care to make them private and break people abusing them. > I do not care in particular but > some might, a line separated with empty lines in the commit log would do. I'll add a blurb for the next version. ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line 2020-04-17 7:58 ` Christoph Hellwig (?) @ 2020-05-05 4:18 ` Alexey Kardashevskiy -1 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-05-05 4:18 UTC (permalink / raw) To: Christoph Hellwig Cc: iommu, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, Michael Ellerman On 17/04/2020 17:58, Christoph Hellwig wrote: > On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >> And the fact they were exported leaves possibility that there is a >> driver somewhere relying on these symbols or distro kernel won't build >> because the symbol disappeared from exports (I do not know what KABI >> guarantees or if mainline kernel cares). > > We absolutely do not care. In fact for abuses of APIs that drivers > should not use we almost care to make them private and break people > abusing them. ok :) >> I do not care in particular but >> some might, a line separated with empty lines in the commit log would do. > > I'll add a blurb for the next version. Has it gone anywhere? Thanks, -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-05-05 4:18 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-05-05 4:18 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, linuxppc-dev, linux-kernel, iommu, Michael Ellerman, Robin Murphy On 17/04/2020 17:58, Christoph Hellwig wrote: > On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >> And the fact they were exported leaves possibility that there is a >> driver somewhere relying on these symbols or distro kernel won't build >> because the symbol disappeared from exports (I do not know what KABI >> guarantees or if mainline kernel cares). > > We absolutely do not care. In fact for abuses of APIs that drivers > should not use we almost care to make them private and break people > abusing them. ok :) >> I do not care in particular but >> some might, a line separated with empty lines in the commit log would do. > > I'll add a blurb for the next version. Has it gone anywhere? Thanks, -- Alexey _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-05-05 4:18 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-05-05 4:18 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, Joerg Roedel, linuxppc-dev, linux-kernel, iommu, Robin Murphy, Lu Baolu On 17/04/2020 17:58, Christoph Hellwig wrote: > On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >> And the fact they were exported leaves possibility that there is a >> driver somewhere relying on these symbols or distro kernel won't build >> because the symbol disappeared from exports (I do not know what KABI >> guarantees or if mainline kernel cares). > > We absolutely do not care. In fact for abuses of APIs that drivers > should not use we almost care to make them private and break people > abusing them. ok :) >> I do not care in particular but >> some might, a line separated with empty lines in the commit log would do. > > I'll add a blurb for the next version. Has it gone anywhere? Thanks, -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line 2020-05-05 4:18 ` Alexey Kardashevskiy (?) @ 2020-05-09 8:19 ` Christoph Hellwig -1 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-05-09 8:19 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Christoph Hellwig, iommu, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, Michael Ellerman On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: > > > On 17/04/2020 17:58, Christoph Hellwig wrote: > > On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: > >> And the fact they were exported leaves possibility that there is a > >> driver somewhere relying on these symbols or distro kernel won't build > >> because the symbol disappeared from exports (I do not know what KABI > >> guarantees or if mainline kernel cares). > > > > We absolutely do not care. In fact for abuses of APIs that drivers > > should not use we almost care to make them private and break people > > abusing them. > > ok :) > > >> I do not care in particular but > >> some might, a line separated with empty lines in the commit log would do. > > > > I'll add a blurb for the next version. > > > Has it gone anywhere? Thanks, I've been hoping for the sg_buf helpers to land first, as they need backporting and would conflict. Do you urgently need the series? ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-05-09 8:19 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-05-09 8:19 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, Michael Ellerman, linuxppc-dev, Christoph Hellwig On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: > > > On 17/04/2020 17:58, Christoph Hellwig wrote: > > On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: > >> And the fact they were exported leaves possibility that there is a > >> driver somewhere relying on these symbols or distro kernel won't build > >> because the symbol disappeared from exports (I do not know what KABI > >> guarantees or if mainline kernel cares). > > > > We absolutely do not care. In fact for abuses of APIs that drivers > > should not use we almost care to make them private and break people > > abusing them. > > ok :) > > >> I do not care in particular but > >> some might, a line separated with empty lines in the commit log would do. > > > > I'll add a blurb for the next version. > > > Has it gone anywhere? Thanks, I've been hoping for the sg_buf helpers to land first, as they need backporting and would conflict. Do you urgently need the series? _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-05-09 8:19 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-05-09 8:19 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, iommu, linuxppc-dev, Christoph Hellwig, Lu Baolu On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: > > > On 17/04/2020 17:58, Christoph Hellwig wrote: > > On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: > >> And the fact they were exported leaves possibility that there is a > >> driver somewhere relying on these symbols or distro kernel won't build > >> because the symbol disappeared from exports (I do not know what KABI > >> guarantees or if mainline kernel cares). > > > > We absolutely do not care. In fact for abuses of APIs that drivers > > should not use we almost care to make them private and break people > > abusing them. > > ok :) > > >> I do not care in particular but > >> some might, a line separated with empty lines in the commit log would do. > > > > I'll add a blurb for the next version. > > > Has it gone anywhere? Thanks, I've been hoping for the sg_buf helpers to land first, as they need backporting and would conflict. Do you urgently need the series? ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line 2020-05-09 8:19 ` Christoph Hellwig (?) @ 2020-05-09 14:07 ` Alexey Kardashevskiy -1 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-05-09 14:07 UTC (permalink / raw) To: Christoph Hellwig Cc: iommu, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, Michael Ellerman On 09/05/2020 18:19, Christoph Hellwig wrote: > On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: >> >> >> On 17/04/2020 17:58, Christoph Hellwig wrote: >>> On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >>>> And the fact they were exported leaves possibility that there is a >>>> driver somewhere relying on these symbols or distro kernel won't build >>>> because the symbol disappeared from exports (I do not know what KABI >>>> guarantees or if mainline kernel cares). >>> >>> We absolutely do not care. In fact for abuses of APIs that drivers >>> should not use we almost care to make them private and break people >>> abusing them. >> >> ok :) >> >>>> I do not care in particular but >>>> some might, a line separated with empty lines in the commit log would do. >>> >>> I'll add a blurb for the next version. >> >> >> Has it gone anywhere? Thanks, > > I've been hoping for the sg_buf helpers to land first, as they need > backporting and would conflict. Do you urgently need the series? Nah, not that urgent. Thanks, -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-05-09 14:07 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-05-09 14:07 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, linuxppc-dev, linux-kernel, iommu, Michael Ellerman, Robin Murphy On 09/05/2020 18:19, Christoph Hellwig wrote: > On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: >> >> >> On 17/04/2020 17:58, Christoph Hellwig wrote: >>> On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >>>> And the fact they were exported leaves possibility that there is a >>>> driver somewhere relying on these symbols or distro kernel won't build >>>> because the symbol disappeared from exports (I do not know what KABI >>>> guarantees or if mainline kernel cares). >>> >>> We absolutely do not care. In fact for abuses of APIs that drivers >>> should not use we almost care to make them private and break people >>> abusing them. >> >> ok :) >> >>>> I do not care in particular but >>>> some might, a line separated with empty lines in the commit log would do. >>> >>> I'll add a blurb for the next version. >> >> >> Has it gone anywhere? Thanks, > > I've been hoping for the sg_buf helpers to land first, as they need > backporting and would conflict. Do you urgently need the series? Nah, not that urgent. Thanks, -- Alexey _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-05-09 14:07 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-05-09 14:07 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, Joerg Roedel, linuxppc-dev, linux-kernel, iommu, Robin Murphy, Lu Baolu On 09/05/2020 18:19, Christoph Hellwig wrote: > On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: >> >> >> On 17/04/2020 17:58, Christoph Hellwig wrote: >>> On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >>>> And the fact they were exported leaves possibility that there is a >>>> driver somewhere relying on these symbols or distro kernel won't build >>>> because the symbol disappeared from exports (I do not know what KABI >>>> guarantees or if mainline kernel cares). >>> >>> We absolutely do not care. In fact for abuses of APIs that drivers >>> should not use we almost care to make them private and break people >>> abusing them. >> >> ok :) >> >>>> I do not care in particular but >>>> some might, a line separated with empty lines in the commit log would do. >>> >>> I'll add a blurb for the next version. >> >> >> Has it gone anywhere? Thanks, > > I've been hoping for the sg_buf helpers to land first, as they need > backporting and would conflict. Do you urgently need the series? Nah, not that urgent. Thanks, -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line 2020-05-09 8:19 ` Christoph Hellwig (?) @ 2020-06-03 4:13 ` Alexey Kardashevskiy -1 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-06-03 4:13 UTC (permalink / raw) To: Christoph Hellwig Cc: iommu, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, Michael Ellerman On 09/05/2020 18:19, Christoph Hellwig wrote: > On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: >> >> >> On 17/04/2020 17:58, Christoph Hellwig wrote: >>> On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >>>> And the fact they were exported leaves possibility that there is a >>>> driver somewhere relying on these symbols or distro kernel won't build >>>> because the symbol disappeared from exports (I do not know what KABI >>>> guarantees or if mainline kernel cares). >>> >>> We absolutely do not care. In fact for abuses of APIs that drivers >>> should not use we almost care to make them private and break people >>> abusing them. >> >> ok :) >> >>>> I do not care in particular but >>>> some might, a line separated with empty lines in the commit log would do. >>> >>> I'll add a blurb for the next version. >> >> >> Has it gone anywhere? Thanks, > > I've been hoping for the sg_buf helpers to land first, as they need > backporting and would conflict. Do you urgently need the series? Any progress with sg_buf helpers stuff? Thanks, -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-06-03 4:13 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-06-03 4:13 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, linuxppc-dev, linux-kernel, iommu, Michael Ellerman, Robin Murphy On 09/05/2020 18:19, Christoph Hellwig wrote: > On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: >> >> >> On 17/04/2020 17:58, Christoph Hellwig wrote: >>> On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >>>> And the fact they were exported leaves possibility that there is a >>>> driver somewhere relying on these symbols or distro kernel won't build >>>> because the symbol disappeared from exports (I do not know what KABI >>>> guarantees or if mainline kernel cares). >>> >>> We absolutely do not care. In fact for abuses of APIs that drivers >>> should not use we almost care to make them private and break people >>> abusing them. >> >> ok :) >> >>>> I do not care in particular but >>>> some might, a line separated with empty lines in the commit log would do. >>> >>> I'll add a blurb for the next version. >> >> >> Has it gone anywhere? Thanks, > > I've been hoping for the sg_buf helpers to land first, as they need > backporting and would conflict. Do you urgently need the series? Any progress with sg_buf helpers stuff? Thanks, -- Alexey _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-06-03 4:13 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-06-03 4:13 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, Joerg Roedel, linuxppc-dev, linux-kernel, iommu, Robin Murphy, Lu Baolu On 09/05/2020 18:19, Christoph Hellwig wrote: > On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: >> >> >> On 17/04/2020 17:58, Christoph Hellwig wrote: >>> On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >>>> And the fact they were exported leaves possibility that there is a >>>> driver somewhere relying on these symbols or distro kernel won't build >>>> because the symbol disappeared from exports (I do not know what KABI >>>> guarantees or if mainline kernel cares). >>> >>> We absolutely do not care. In fact for abuses of APIs that drivers >>> should not use we almost care to make them private and break people >>> abusing them. >> >> ok :) >> >>>> I do not care in particular but >>>> some might, a line separated with empty lines in the commit log would do. >>> >>> I'll add a blurb for the next version. >> >> >> Has it gone anywhere? Thanks, > > I've been hoping for the sg_buf helpers to land first, as they need > backporting and would conflict. Do you urgently need the series? Any progress with sg_buf helpers stuff? Thanks, -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line 2020-06-03 4:13 ` Alexey Kardashevskiy (?) @ 2020-07-07 0:43 ` Alexey Kardashevskiy -1 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-07-07 0:43 UTC (permalink / raw) To: Christoph Hellwig Cc: iommu, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, Michael Ellerman On 03/06/2020 14:13, Alexey Kardashevskiy wrote: > > > On 09/05/2020 18:19, Christoph Hellwig wrote: >> On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: >>> >>> >>> On 17/04/2020 17:58, Christoph Hellwig wrote: >>>> On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >>>>> And the fact they were exported leaves possibility that there is a >>>>> driver somewhere relying on these symbols or distro kernel won't build >>>>> because the symbol disappeared from exports (I do not know what KABI >>>>> guarantees or if mainline kernel cares). >>>> >>>> We absolutely do not care. In fact for abuses of APIs that drivers >>>> should not use we almost care to make them private and break people >>>> abusing them. >>> >>> ok :) >>> >>>>> I do not care in particular but >>>>> some might, a line separated with empty lines in the commit log would do. >>>> >>>> I'll add a blurb for the next version. >>> >>> >>> Has it gone anywhere? Thanks, >> >> I've been hoping for the sg_buf helpers to land first, as they need >> backporting and would conflict. Do you urgently need the series? > > Any progress with sg_buf helpers stuff? Thanks, Any luck there? I'd really like to cross this off my todo list :) Thanks, -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-07-07 0:43 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-07-07 0:43 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, linuxppc-dev, linux-kernel, iommu, Michael Ellerman, Robin Murphy On 03/06/2020 14:13, Alexey Kardashevskiy wrote: > > > On 09/05/2020 18:19, Christoph Hellwig wrote: >> On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: >>> >>> >>> On 17/04/2020 17:58, Christoph Hellwig wrote: >>>> On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >>>>> And the fact they were exported leaves possibility that there is a >>>>> driver somewhere relying on these symbols or distro kernel won't build >>>>> because the symbol disappeared from exports (I do not know what KABI >>>>> guarantees or if mainline kernel cares). >>>> >>>> We absolutely do not care. In fact for abuses of APIs that drivers >>>> should not use we almost care to make them private and break people >>>> abusing them. >>> >>> ok :) >>> >>>>> I do not care in particular but >>>>> some might, a line separated with empty lines in the commit log would do. >>>> >>>> I'll add a blurb for the next version. >>> >>> >>> Has it gone anywhere? Thanks, >> >> I've been hoping for the sg_buf helpers to land first, as they need >> backporting and would conflict. Do you urgently need the series? > > Any progress with sg_buf helpers stuff? Thanks, Any luck there? I'd really like to cross this off my todo list :) Thanks, -- Alexey _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-07-07 0:43 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-07-07 0:43 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, Joerg Roedel, linuxppc-dev, linux-kernel, iommu, Robin Murphy, Lu Baolu On 03/06/2020 14:13, Alexey Kardashevskiy wrote: > > > On 09/05/2020 18:19, Christoph Hellwig wrote: >> On Tue, May 05, 2020 at 02:18:37PM +1000, Alexey Kardashevskiy wrote: >>> >>> >>> On 17/04/2020 17:58, Christoph Hellwig wrote: >>>> On Wed, Apr 15, 2020 at 09:21:37PM +1000, Alexey Kardashevskiy wrote: >>>>> And the fact they were exported leaves possibility that there is a >>>>> driver somewhere relying on these symbols or distro kernel won't build >>>>> because the symbol disappeared from exports (I do not know what KABI >>>>> guarantees or if mainline kernel cares). >>>> >>>> We absolutely do not care. In fact for abuses of APIs that drivers >>>> should not use we almost care to make them private and break people >>>> abusing them. >>> >>> ok :) >>> >>>>> I do not care in particular but >>>>> some might, a line separated with empty lines in the commit log would do. >>>> >>>> I'll add a blurb for the next version. >>> >>> >>> Has it gone anywhere? Thanks, >> >> I've been hoping for the sg_buf helpers to land first, as they need >> backporting and would conflict. Do you urgently need the series? > > Any progress with sg_buf helpers stuff? Thanks, Any luck there? I'd really like to cross this off my todo list :) Thanks, -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line 2020-07-07 0:43 ` Alexey Kardashevskiy (?) @ 2020-07-07 14:51 ` Christoph Hellwig -1 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-07-07 14:51 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Christoph Hellwig, iommu, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, Michael Ellerman On Tue, Jul 07, 2020 at 10:43:10AM +1000, Alexey Kardashevskiy wrote: > Any luck there? I'd really like to cross this off my todo list :) Thanks, We had another incident with new net code poking into dma internals blocking this series. That is now sorted out, so the series is back on track. ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-07-07 14:51 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-07-07 14:51 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, Michael Ellerman, linuxppc-dev, Christoph Hellwig On Tue, Jul 07, 2020 at 10:43:10AM +1000, Alexey Kardashevskiy wrote: > Any luck there? I'd really like to cross this off my todo list :) Thanks, We had another incident with new net code poking into dma internals blocking this series. That is now sorted out, so the series is back on track. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line @ 2020-07-07 14:51 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-07-07 14:51 UTC (permalink / raw) To: Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, iommu, linuxppc-dev, Christoph Hellwig, Lu Baolu On Tue, Jul 07, 2020 at 10:43:10AM +1000, Alexey Kardashevskiy wrote: > Any luck there? I'd really like to cross this off my todo list :) Thanks, We had another incident with new net code poking into dma internals blocking this series. That is now sorted out, so the series is back on track. ^ permalink raw reply [flat|nested] 60+ messages in thread
* [PATCH 2/4] dma-mapping: inline the fast path dma-direct calls 2020-04-14 12:25 ` Christoph Hellwig (?) @ 2020-04-14 12:25 ` Christoph Hellwig -1 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel Inline the single page map/unmap/sync dma-direct calls into the now out of line generic wrappers. This restores the behavior of a single function call that we had before moving the generic calls out of line. Besides the dma-mapping callers there are just a few callers in IOMMU drivers that have a bypass mode, and more of those are going to be switched to the generic bypass soon. Signed-off-by: Christoph Hellwig <hch@lst.de> --- include/linux/dma-direct.h | 88 ++++++++++++++++++++++++++++---------- kernel/dma/direct.c | 65 ---------------------------- 2 files changed, 65 insertions(+), 88 deletions(-) diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index da689ad5fffd..d653070d515b 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -3,8 +3,10 @@ #define _LINUX_DMA_DIRECT_H 1 #include <linux/dma-mapping.h> +#include <linux/dma-noncoherent.h> #include <linux/memblock.h> /* for min_low_pfn */ #include <linux/mem_encrypt.h> +#include <linux/swiotlb.h> extern unsigned int zone_dma_bits; @@ -85,25 +87,17 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma, void *cpu_addr, dma_addr_t dma_addr, size_t size, unsigned long attrs); int dma_direct_supported(struct device *dev, u64 mask); -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs); int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs); dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, size_t size, enum dma_data_direction dir, unsigned long attrs); +size_t dma_direct_max_mapping_size(struct device *dev); #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); -void dma_direct_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir); +void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, + int nents, enum dma_data_direction dir); #else -static inline void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ -} static inline void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) { @@ -113,34 +107,82 @@ static inline void dma_direct_sync_sg_for_device(struct device *dev, #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ defined(CONFIG_SWIOTLB) -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs); void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs); -void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); void dma_direct_sync_sg_for_cpu(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir); #else -static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ -} static inline void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { } +static inline void dma_direct_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir) +{ +} +#endif + +static inline void dma_direct_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ + phys_addr_t paddr = dma_to_phys(dev, addr); + + if (unlikely(is_swiotlb_buffer(paddr))) + swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_DEVICE); + + if (!dev_is_dma_coherent(dev)) + arch_sync_dma_for_device(paddr, size, dir); +} + static inline void dma_direct_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir) { + phys_addr_t paddr = dma_to_phys(dev, addr); + + if (!dev_is_dma_coherent(dev)) { + arch_sync_dma_for_cpu(paddr, size, dir); + arch_sync_dma_for_cpu_all(); + } + + if (unlikely(is_swiotlb_buffer(paddr))) + swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_CPU); } -static inline void dma_direct_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) + +static inline dma_addr_t dma_direct_map_page(struct device *dev, + struct page *page, unsigned long offset, size_t size, + enum dma_data_direction dir, unsigned long attrs) { + phys_addr_t phys = page_to_phys(page) + offset; + dma_addr_t dma_addr = phys_to_dma(dev, phys); + + if (unlikely(swiotlb_force == SWIOTLB_FORCE)) + return swiotlb_map(dev, phys, size, dir, attrs); + + if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (swiotlb_force != SWIOTLB_NO_FORCE) + return swiotlb_map(dev, phys, size, dir, attrs); + + dev_WARN_ONCE(dev, 1, + "DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n", + &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); + return DMA_MAPPING_ERROR; + } + + if (!dev_is_dma_coherent(dev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + arch_sync_dma_for_device(phys, size, dir); + return dma_addr; } -#endif -size_t dma_direct_max_mapping_size(struct device *dev); +static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + phys_addr_t phys = dma_to_phys(dev, addr); + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + dma_direct_sync_single_for_cpu(dev, addr, size, dir); + + if (unlikely(is_swiotlb_buffer(phys))) + swiotlb_tbl_unmap_single(dev, phys, size, size, dir, attrs); +} #endif /* _LINUX_DMA_DIRECT_H */ diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index f1a9099a4b5b..0a320f9b837f 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -10,11 +10,9 @@ #include <linux/dma-direct.h> #include <linux/scatterlist.h> #include <linux/dma-contiguous.h> -#include <linux/dma-noncoherent.h> #include <linux/pfn.h> #include <linux/vmalloc.h> #include <linux/set_memory.h> -#include <linux/swiotlb.h> /* * Most architectures use ZONE_DMA for the first 16 Megabytes, but some use it @@ -249,18 +247,6 @@ void dma_direct_free(struct device *dev, size_t size, #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ - phys_addr_t paddr = dma_to_phys(dev, addr); - - if (unlikely(is_swiotlb_buffer(paddr))) - swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_DEVICE); - - if (!dev_is_dma_coherent(dev)) - arch_sync_dma_for_device(paddr, size, dir); -} - void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) { @@ -284,20 +270,6 @@ void dma_direct_sync_sg_for_device(struct device *dev, #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ - phys_addr_t paddr = dma_to_phys(dev, addr); - - if (!dev_is_dma_coherent(dev)) { - arch_sync_dma_for_cpu(paddr, size, dir); - arch_sync_dma_for_cpu_all(); - } - - if (unlikely(is_swiotlb_buffer(paddr))) - swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_CPU); -} - void dma_direct_sync_sg_for_cpu(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) { @@ -319,18 +291,6 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, arch_sync_dma_for_cpu_all(); } -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ - phys_addr_t phys = dma_to_phys(dev, addr); - - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - dma_direct_sync_single_for_cpu(dev, addr, size, dir); - - if (unlikely(is_swiotlb_buffer(phys))) - swiotlb_tbl_unmap_single(dev, phys, size, size, dir, attrs); -} - void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { @@ -343,31 +303,6 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, } #endif -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - phys_addr_t phys = page_to_phys(page) + offset; - dma_addr_t dma_addr = phys_to_dma(dev, phys); - - if (unlikely(swiotlb_force == SWIOTLB_FORCE)) - return swiotlb_map(dev, phys, size, dir, attrs); - - if (unlikely(!dma_capable(dev, dma_addr, size, true))) { - if (swiotlb_force != SWIOTLB_NO_FORCE) - return swiotlb_map(dev, phys, size, dir, attrs); - - dev_WARN_ONCE(dev, 1, - "DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n", - &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); - return DMA_MAPPING_ERROR; - } - - if (!dev_is_dma_coherent(dev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - arch_sync_dma_for_device(phys, size, dir); - return dma_addr; -} - int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { -- 2.25.1 ^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 2/4] dma-mapping: inline the fast path dma-direct calls @ 2020-04-14 12:25 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, linuxppc-dev Inline the single page map/unmap/sync dma-direct calls into the now out of line generic wrappers. This restores the behavior of a single function call that we had before moving the generic calls out of line. Besides the dma-mapping callers there are just a few callers in IOMMU drivers that have a bypass mode, and more of those are going to be switched to the generic bypass soon. Signed-off-by: Christoph Hellwig <hch@lst.de> --- include/linux/dma-direct.h | 88 ++++++++++++++++++++++++++++---------- kernel/dma/direct.c | 65 ---------------------------- 2 files changed, 65 insertions(+), 88 deletions(-) diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index da689ad5fffd..d653070d515b 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -3,8 +3,10 @@ #define _LINUX_DMA_DIRECT_H 1 #include <linux/dma-mapping.h> +#include <linux/dma-noncoherent.h> #include <linux/memblock.h> /* for min_low_pfn */ #include <linux/mem_encrypt.h> +#include <linux/swiotlb.h> extern unsigned int zone_dma_bits; @@ -85,25 +87,17 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma, void *cpu_addr, dma_addr_t dma_addr, size_t size, unsigned long attrs); int dma_direct_supported(struct device *dev, u64 mask); -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs); int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs); dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, size_t size, enum dma_data_direction dir, unsigned long attrs); +size_t dma_direct_max_mapping_size(struct device *dev); #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); -void dma_direct_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir); +void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, + int nents, enum dma_data_direction dir); #else -static inline void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ -} static inline void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) { @@ -113,34 +107,82 @@ static inline void dma_direct_sync_sg_for_device(struct device *dev, #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ defined(CONFIG_SWIOTLB) -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs); void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs); -void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); void dma_direct_sync_sg_for_cpu(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir); #else -static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ -} static inline void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { } +static inline void dma_direct_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir) +{ +} +#endif + +static inline void dma_direct_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ + phys_addr_t paddr = dma_to_phys(dev, addr); + + if (unlikely(is_swiotlb_buffer(paddr))) + swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_DEVICE); + + if (!dev_is_dma_coherent(dev)) + arch_sync_dma_for_device(paddr, size, dir); +} + static inline void dma_direct_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir) { + phys_addr_t paddr = dma_to_phys(dev, addr); + + if (!dev_is_dma_coherent(dev)) { + arch_sync_dma_for_cpu(paddr, size, dir); + arch_sync_dma_for_cpu_all(); + } + + if (unlikely(is_swiotlb_buffer(paddr))) + swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_CPU); } -static inline void dma_direct_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) + +static inline dma_addr_t dma_direct_map_page(struct device *dev, + struct page *page, unsigned long offset, size_t size, + enum dma_data_direction dir, unsigned long attrs) { + phys_addr_t phys = page_to_phys(page) + offset; + dma_addr_t dma_addr = phys_to_dma(dev, phys); + + if (unlikely(swiotlb_force == SWIOTLB_FORCE)) + return swiotlb_map(dev, phys, size, dir, attrs); + + if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (swiotlb_force != SWIOTLB_NO_FORCE) + return swiotlb_map(dev, phys, size, dir, attrs); + + dev_WARN_ONCE(dev, 1, + "DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n", + &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); + return DMA_MAPPING_ERROR; + } + + if (!dev_is_dma_coherent(dev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + arch_sync_dma_for_device(phys, size, dir); + return dma_addr; } -#endif -size_t dma_direct_max_mapping_size(struct device *dev); +static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + phys_addr_t phys = dma_to_phys(dev, addr); + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + dma_direct_sync_single_for_cpu(dev, addr, size, dir); + + if (unlikely(is_swiotlb_buffer(phys))) + swiotlb_tbl_unmap_single(dev, phys, size, size, dir, attrs); +} #endif /* _LINUX_DMA_DIRECT_H */ diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index f1a9099a4b5b..0a320f9b837f 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -10,11 +10,9 @@ #include <linux/dma-direct.h> #include <linux/scatterlist.h> #include <linux/dma-contiguous.h> -#include <linux/dma-noncoherent.h> #include <linux/pfn.h> #include <linux/vmalloc.h> #include <linux/set_memory.h> -#include <linux/swiotlb.h> /* * Most architectures use ZONE_DMA for the first 16 Megabytes, but some use it @@ -249,18 +247,6 @@ void dma_direct_free(struct device *dev, size_t size, #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ - phys_addr_t paddr = dma_to_phys(dev, addr); - - if (unlikely(is_swiotlb_buffer(paddr))) - swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_DEVICE); - - if (!dev_is_dma_coherent(dev)) - arch_sync_dma_for_device(paddr, size, dir); -} - void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) { @@ -284,20 +270,6 @@ void dma_direct_sync_sg_for_device(struct device *dev, #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ - phys_addr_t paddr = dma_to_phys(dev, addr); - - if (!dev_is_dma_coherent(dev)) { - arch_sync_dma_for_cpu(paddr, size, dir); - arch_sync_dma_for_cpu_all(); - } - - if (unlikely(is_swiotlb_buffer(paddr))) - swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_CPU); -} - void dma_direct_sync_sg_for_cpu(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) { @@ -319,18 +291,6 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, arch_sync_dma_for_cpu_all(); } -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ - phys_addr_t phys = dma_to_phys(dev, addr); - - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - dma_direct_sync_single_for_cpu(dev, addr, size, dir); - - if (unlikely(is_swiotlb_buffer(phys))) - swiotlb_tbl_unmap_single(dev, phys, size, size, dir, attrs); -} - void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { @@ -343,31 +303,6 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, } #endif -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - phys_addr_t phys = page_to_phys(page) + offset; - dma_addr_t dma_addr = phys_to_dma(dev, phys); - - if (unlikely(swiotlb_force == SWIOTLB_FORCE)) - return swiotlb_map(dev, phys, size, dir, attrs); - - if (unlikely(!dma_capable(dev, dma_addr, size, true))) { - if (swiotlb_force != SWIOTLB_NO_FORCE) - return swiotlb_map(dev, phys, size, dir, attrs); - - dev_WARN_ONCE(dev, 1, - "DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n", - &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); - return DMA_MAPPING_ERROR; - } - - if (!dev_is_dma_coherent(dev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - arch_sync_dma_for_device(phys, size, dir); - return dma_addr; -} - int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { -- 2.25.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 2/4] dma-mapping: inline the fast path dma-direct calls @ 2020-04-14 12:25 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, linuxppc-dev, Lu Baolu Inline the single page map/unmap/sync dma-direct calls into the now out of line generic wrappers. This restores the behavior of a single function call that we had before moving the generic calls out of line. Besides the dma-mapping callers there are just a few callers in IOMMU drivers that have a bypass mode, and more of those are going to be switched to the generic bypass soon. Signed-off-by: Christoph Hellwig <hch@lst.de> --- include/linux/dma-direct.h | 88 ++++++++++++++++++++++++++++---------- kernel/dma/direct.c | 65 ---------------------------- 2 files changed, 65 insertions(+), 88 deletions(-) diff --git a/include/linux/dma-direct.h b/include/linux/dma-direct.h index da689ad5fffd..d653070d515b 100644 --- a/include/linux/dma-direct.h +++ b/include/linux/dma-direct.h @@ -3,8 +3,10 @@ #define _LINUX_DMA_DIRECT_H 1 #include <linux/dma-mapping.h> +#include <linux/dma-noncoherent.h> #include <linux/memblock.h> /* for min_low_pfn */ #include <linux/mem_encrypt.h> +#include <linux/swiotlb.h> extern unsigned int zone_dma_bits; @@ -85,25 +87,17 @@ int dma_direct_mmap(struct device *dev, struct vm_area_struct *vma, void *cpu_addr, dma_addr_t dma_addr, size_t size, unsigned long attrs); int dma_direct_supported(struct device *dev, u64 mask); -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs); int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs); dma_addr_t dma_direct_map_resource(struct device *dev, phys_addr_t paddr, size_t size, enum dma_data_direction dir, unsigned long attrs); +size_t dma_direct_max_mapping_size(struct device *dev); #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); -void dma_direct_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir); +void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, + int nents, enum dma_data_direction dir); #else -static inline void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ -} static inline void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) { @@ -113,34 +107,82 @@ static inline void dma_direct_sync_sg_for_device(struct device *dev, #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ defined(CONFIG_SWIOTLB) -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs); void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs); -void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir); void dma_direct_sync_sg_for_cpu(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir); #else -static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ -} static inline void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { } +static inline void dma_direct_sync_sg_for_cpu(struct device *dev, + struct scatterlist *sgl, int nents, enum dma_data_direction dir) +{ +} +#endif + +static inline void dma_direct_sync_single_for_device(struct device *dev, + dma_addr_t addr, size_t size, enum dma_data_direction dir) +{ + phys_addr_t paddr = dma_to_phys(dev, addr); + + if (unlikely(is_swiotlb_buffer(paddr))) + swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_DEVICE); + + if (!dev_is_dma_coherent(dev)) + arch_sync_dma_for_device(paddr, size, dir); +} + static inline void dma_direct_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, enum dma_data_direction dir) { + phys_addr_t paddr = dma_to_phys(dev, addr); + + if (!dev_is_dma_coherent(dev)) { + arch_sync_dma_for_cpu(paddr, size, dir); + arch_sync_dma_for_cpu_all(); + } + + if (unlikely(is_swiotlb_buffer(paddr))) + swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_CPU); } -static inline void dma_direct_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) + +static inline dma_addr_t dma_direct_map_page(struct device *dev, + struct page *page, unsigned long offset, size_t size, + enum dma_data_direction dir, unsigned long attrs) { + phys_addr_t phys = page_to_phys(page) + offset; + dma_addr_t dma_addr = phys_to_dma(dev, phys); + + if (unlikely(swiotlb_force == SWIOTLB_FORCE)) + return swiotlb_map(dev, phys, size, dir, attrs); + + if (unlikely(!dma_capable(dev, dma_addr, size, true))) { + if (swiotlb_force != SWIOTLB_NO_FORCE) + return swiotlb_map(dev, phys, size, dir, attrs); + + dev_WARN_ONCE(dev, 1, + "DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n", + &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); + return DMA_MAPPING_ERROR; + } + + if (!dev_is_dma_coherent(dev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + arch_sync_dma_for_device(phys, size, dir); + return dma_addr; } -#endif -size_t dma_direct_max_mapping_size(struct device *dev); +static inline void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, + size_t size, enum dma_data_direction dir, unsigned long attrs) +{ + phys_addr_t phys = dma_to_phys(dev, addr); + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) + dma_direct_sync_single_for_cpu(dev, addr, size, dir); + + if (unlikely(is_swiotlb_buffer(phys))) + swiotlb_tbl_unmap_single(dev, phys, size, size, dir, attrs); +} #endif /* _LINUX_DMA_DIRECT_H */ diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index f1a9099a4b5b..0a320f9b837f 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -10,11 +10,9 @@ #include <linux/dma-direct.h> #include <linux/scatterlist.h> #include <linux/dma-contiguous.h> -#include <linux/dma-noncoherent.h> #include <linux/pfn.h> #include <linux/vmalloc.h> #include <linux/set_memory.h> -#include <linux/swiotlb.h> /* * Most architectures use ZONE_DMA for the first 16 Megabytes, but some use it @@ -249,18 +247,6 @@ void dma_direct_free(struct device *dev, size_t size, #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_DEVICE) || \ defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_device(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ - phys_addr_t paddr = dma_to_phys(dev, addr); - - if (unlikely(is_swiotlb_buffer(paddr))) - swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_DEVICE); - - if (!dev_is_dma_coherent(dev)) - arch_sync_dma_for_device(paddr, size, dir); -} - void dma_direct_sync_sg_for_device(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) { @@ -284,20 +270,6 @@ void dma_direct_sync_sg_for_device(struct device *dev, #if defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU) || \ defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) || \ defined(CONFIG_SWIOTLB) -void dma_direct_sync_single_for_cpu(struct device *dev, - dma_addr_t addr, size_t size, enum dma_data_direction dir) -{ - phys_addr_t paddr = dma_to_phys(dev, addr); - - if (!dev_is_dma_coherent(dev)) { - arch_sync_dma_for_cpu(paddr, size, dir); - arch_sync_dma_for_cpu_all(); - } - - if (unlikely(is_swiotlb_buffer(paddr))) - swiotlb_tbl_sync_single(dev, paddr, size, dir, SYNC_FOR_CPU); -} - void dma_direct_sync_sg_for_cpu(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir) { @@ -319,18 +291,6 @@ void dma_direct_sync_sg_for_cpu(struct device *dev, arch_sync_dma_for_cpu_all(); } -void dma_direct_unmap_page(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir, unsigned long attrs) -{ - phys_addr_t phys = dma_to_phys(dev, addr); - - if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - dma_direct_sync_single_for_cpu(dev, addr, size, dir); - - if (unlikely(is_swiotlb_buffer(phys))) - swiotlb_tbl_unmap_single(dev, phys, size, size, dir, attrs); -} - void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { @@ -343,31 +303,6 @@ void dma_direct_unmap_sg(struct device *dev, struct scatterlist *sgl, } #endif -dma_addr_t dma_direct_map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, enum dma_data_direction dir, - unsigned long attrs) -{ - phys_addr_t phys = page_to_phys(page) + offset; - dma_addr_t dma_addr = phys_to_dma(dev, phys); - - if (unlikely(swiotlb_force == SWIOTLB_FORCE)) - return swiotlb_map(dev, phys, size, dir, attrs); - - if (unlikely(!dma_capable(dev, dma_addr, size, true))) { - if (swiotlb_force != SWIOTLB_NO_FORCE) - return swiotlb_map(dev, phys, size, dir, attrs); - - dev_WARN_ONCE(dev, 1, - "DMA addr %pad+%zu overflow (mask %llx, bus limit %llx).\n", - &dma_addr, size, *dev->dma_mask, dev->bus_dma_limit); - return DMA_MAPPING_ERROR; - } - - if (!dev_is_dma_coherent(dev) && !(attrs & DMA_ATTR_SKIP_CPU_SYNC)) - arch_sync_dma_for_device(phys, size, dir); - return dma_addr; -} - int dma_direct_map_sg(struct device *dev, struct scatterlist *sgl, int nents, enum dma_data_direction dir, unsigned long attrs) { -- 2.25.1 ^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device 2020-04-14 12:25 ` Christoph Hellwig (?) @ 2020-04-14 12:25 ` Christoph Hellwig -1 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel Several IOMMU drivers have a bypass mode where they can use a direct mapping if the devices DMA mask is large enough. Add generic support to the core dma-mapping code to do that to switch those drivers to a common solution. Signed-off-by: Christoph Hellwig <hch@lst.de> --- include/linux/device.h | 6 ++++ kernel/dma/mapping.c | 70 +++++++++++++++++++++++++++++------------- 2 files changed, 55 insertions(+), 21 deletions(-) diff --git a/include/linux/device.h b/include/linux/device.h index ac8e37cd716a..143c2e0edb19 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -523,6 +523,11 @@ struct dev_links_info { * sync_state() callback. * @dma_coherent: this particular device is dma coherent, even if the * architecture supports non-coherent devices. + * @dma_ops_bypass: If set to %true then the dma_ops are bypassed for the + * streaming DMA operations (->map_* / ->unmap_* / ->sync_*), + * and optionall (if the coherent mask is large enough) also + * for dma allocations. This flag is managed by the dma ops + * instance from ->dma_supported. * * At the lowest level, every device in a Linux system is represented by an * instance of struct device. The device structure contains the information @@ -622,6 +627,7 @@ struct device { defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) bool dma_coherent:1; #endif + bool dma_ops_bypass : 1; }; static inline struct device *kobj_to_dev(struct kobject *kobj) diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 8e4155f9ee69..d776fb9f5adb 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -105,9 +105,33 @@ void *dmam_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, } EXPORT_SYMBOL(dmam_alloc_attrs); -static inline bool dma_is_direct(const struct dma_map_ops *ops) +/* + * Check if the devices uses a direct mapping for streaming DMA operations. + * This allows IOMMU drivers to set a bypass mode if the DMA mask is large + * enough. + */ +static inline bool dma_alloc_direct(struct device *dev, + const struct dma_map_ops *ops) { - return likely(!ops); + if (likely(!ops)) + return true; + if (!dev->dma_ops_bypass) + return false; + + return min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit) >= + dma_direct_get_required_mask(dev); +} + +static inline bool dma_map_direct(struct device *dev, + const struct dma_map_ops *ops) +{ + if (likely(!ops)) + return true; + if (!dev->dma_ops_bypass) + return false; + + return min_not_zero(*dev->dma_mask, dev->bus_dma_limit) >= + dma_direct_get_required_mask(dev); } dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, @@ -118,7 +142,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, dma_addr_t addr; BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); else addr = ops->map_page(dev, page, offset, size, dir, attrs); @@ -134,7 +158,7 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_unmap_page(dev, addr, size, dir, attrs); else if (ops->unmap_page) ops->unmap_page(dev, addr, size, dir, attrs); @@ -153,7 +177,7 @@ int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, int ents; BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); else ents = ops->map_sg(dev, sg, nents, dir, attrs); @@ -172,7 +196,7 @@ void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, BUG_ON(!valid_dma_direction(dir)); debug_dma_unmap_sg(dev, sg, nents, dir); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_unmap_sg(dev, sg, nents, dir, attrs); else if (ops->unmap_sg) ops->unmap_sg(dev, sg, nents, dir, attrs); @@ -191,7 +215,7 @@ dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) return DMA_MAPPING_ERROR; - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); else if (ops->map_resource) addr = ops->map_resource(dev, phys_addr, size, dir, attrs); @@ -207,7 +231,7 @@ void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (!dma_is_direct(ops) && ops->unmap_resource) + if (!dma_map_direct(dev, ops) && ops->unmap_resource) ops->unmap_resource(dev, addr, size, dir, attrs); debug_dma_unmap_resource(dev, addr, size, dir); } @@ -219,7 +243,7 @@ void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_single_for_cpu(dev, addr, size, dir); else if (ops->sync_single_for_cpu) ops->sync_single_for_cpu(dev, addr, size, dir); @@ -233,7 +257,7 @@ void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_single_for_device(dev, addr, size, dir); else if (ops->sync_single_for_device) ops->sync_single_for_device(dev, addr, size, dir); @@ -247,7 +271,7 @@ void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); else if (ops->sync_sg_for_cpu) ops->sync_sg_for_cpu(dev, sg, nelems, dir); @@ -261,7 +285,7 @@ void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_sg_for_device(dev, sg, nelems, dir); else if (ops->sync_sg_for_device) ops->sync_sg_for_device(dev, sg, nelems, dir); @@ -302,7 +326,7 @@ int dma_get_sgtable_attrs(struct device *dev, struct sg_table *sgt, { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_get_sgtable(dev, sgt, cpu_addr, dma_addr, size, attrs); if (!ops->get_sgtable) @@ -372,7 +396,7 @@ bool dma_can_mmap(struct device *dev) { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_can_mmap(dev); return ops->mmap != NULL; } @@ -397,7 +421,7 @@ int dma_mmap_attrs(struct device *dev, struct vm_area_struct *vma, { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_mmap(dev, vma, cpu_addr, dma_addr, size, attrs); if (!ops->mmap) @@ -410,7 +434,7 @@ u64 dma_get_required_mask(struct device *dev) { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_get_required_mask(dev); if (ops->get_required_mask) return ops->get_required_mask(dev); @@ -441,7 +465,7 @@ void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, /* let the implementation decide on the zone to allocate from: */ flag &= ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) cpu_addr = dma_direct_alloc(dev, size, dma_handle, flag, attrs); else if (ops->alloc) cpu_addr = ops->alloc(dev, size, dma_handle, flag, attrs); @@ -473,7 +497,7 @@ void dma_free_attrs(struct device *dev, size_t size, void *cpu_addr, return; debug_dma_free_coherent(dev, size, cpu_addr, dma_handle); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) dma_direct_free(dev, size, cpu_addr, dma_handle, attrs); else if (ops->free) ops->free(dev, size, cpu_addr, dma_handle, attrs); @@ -484,7 +508,11 @@ int dma_supported(struct device *dev, u64 mask) { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + /* + * ->dma_supported sets the bypass flag, so we must always call + * into the method here unless the device is truly direct mapped. + */ + if (!ops) return dma_direct_supported(dev, mask); if (!ops->dma_supported) return 1; @@ -540,7 +568,7 @@ void dma_cache_sync(struct device *dev, void *vaddr, size_t size, BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) arch_dma_cache_sync(dev, vaddr, size, dir); else if (ops->cache_sync) ops->cache_sync(dev, vaddr, size, dir); @@ -552,7 +580,7 @@ size_t dma_max_mapping_size(struct device *dev) const struct dma_map_ops *ops = get_dma_ops(dev); size_t size = SIZE_MAX; - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) size = dma_direct_max_mapping_size(dev); else if (ops && ops->max_mapping_size) size = ops->max_mapping_size(dev); -- 2.25.1 ^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-14 12:25 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, linuxppc-dev Several IOMMU drivers have a bypass mode where they can use a direct mapping if the devices DMA mask is large enough. Add generic support to the core dma-mapping code to do that to switch those drivers to a common solution. Signed-off-by: Christoph Hellwig <hch@lst.de> --- include/linux/device.h | 6 ++++ kernel/dma/mapping.c | 70 +++++++++++++++++++++++++++++------------- 2 files changed, 55 insertions(+), 21 deletions(-) diff --git a/include/linux/device.h b/include/linux/device.h index ac8e37cd716a..143c2e0edb19 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -523,6 +523,11 @@ struct dev_links_info { * sync_state() callback. * @dma_coherent: this particular device is dma coherent, even if the * architecture supports non-coherent devices. + * @dma_ops_bypass: If set to %true then the dma_ops are bypassed for the + * streaming DMA operations (->map_* / ->unmap_* / ->sync_*), + * and optionall (if the coherent mask is large enough) also + * for dma allocations. This flag is managed by the dma ops + * instance from ->dma_supported. * * At the lowest level, every device in a Linux system is represented by an * instance of struct device. The device structure contains the information @@ -622,6 +627,7 @@ struct device { defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) bool dma_coherent:1; #endif + bool dma_ops_bypass : 1; }; static inline struct device *kobj_to_dev(struct kobject *kobj) diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 8e4155f9ee69..d776fb9f5adb 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -105,9 +105,33 @@ void *dmam_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, } EXPORT_SYMBOL(dmam_alloc_attrs); -static inline bool dma_is_direct(const struct dma_map_ops *ops) +/* + * Check if the devices uses a direct mapping for streaming DMA operations. + * This allows IOMMU drivers to set a bypass mode if the DMA mask is large + * enough. + */ +static inline bool dma_alloc_direct(struct device *dev, + const struct dma_map_ops *ops) { - return likely(!ops); + if (likely(!ops)) + return true; + if (!dev->dma_ops_bypass) + return false; + + return min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit) >= + dma_direct_get_required_mask(dev); +} + +static inline bool dma_map_direct(struct device *dev, + const struct dma_map_ops *ops) +{ + if (likely(!ops)) + return true; + if (!dev->dma_ops_bypass) + return false; + + return min_not_zero(*dev->dma_mask, dev->bus_dma_limit) >= + dma_direct_get_required_mask(dev); } dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, @@ -118,7 +142,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, dma_addr_t addr; BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); else addr = ops->map_page(dev, page, offset, size, dir, attrs); @@ -134,7 +158,7 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_unmap_page(dev, addr, size, dir, attrs); else if (ops->unmap_page) ops->unmap_page(dev, addr, size, dir, attrs); @@ -153,7 +177,7 @@ int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, int ents; BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); else ents = ops->map_sg(dev, sg, nents, dir, attrs); @@ -172,7 +196,7 @@ void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, BUG_ON(!valid_dma_direction(dir)); debug_dma_unmap_sg(dev, sg, nents, dir); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_unmap_sg(dev, sg, nents, dir, attrs); else if (ops->unmap_sg) ops->unmap_sg(dev, sg, nents, dir, attrs); @@ -191,7 +215,7 @@ dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) return DMA_MAPPING_ERROR; - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); else if (ops->map_resource) addr = ops->map_resource(dev, phys_addr, size, dir, attrs); @@ -207,7 +231,7 @@ void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (!dma_is_direct(ops) && ops->unmap_resource) + if (!dma_map_direct(dev, ops) && ops->unmap_resource) ops->unmap_resource(dev, addr, size, dir, attrs); debug_dma_unmap_resource(dev, addr, size, dir); } @@ -219,7 +243,7 @@ void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_single_for_cpu(dev, addr, size, dir); else if (ops->sync_single_for_cpu) ops->sync_single_for_cpu(dev, addr, size, dir); @@ -233,7 +257,7 @@ void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_single_for_device(dev, addr, size, dir); else if (ops->sync_single_for_device) ops->sync_single_for_device(dev, addr, size, dir); @@ -247,7 +271,7 @@ void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); else if (ops->sync_sg_for_cpu) ops->sync_sg_for_cpu(dev, sg, nelems, dir); @@ -261,7 +285,7 @@ void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_sg_for_device(dev, sg, nelems, dir); else if (ops->sync_sg_for_device) ops->sync_sg_for_device(dev, sg, nelems, dir); @@ -302,7 +326,7 @@ int dma_get_sgtable_attrs(struct device *dev, struct sg_table *sgt, { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_get_sgtable(dev, sgt, cpu_addr, dma_addr, size, attrs); if (!ops->get_sgtable) @@ -372,7 +396,7 @@ bool dma_can_mmap(struct device *dev) { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_can_mmap(dev); return ops->mmap != NULL; } @@ -397,7 +421,7 @@ int dma_mmap_attrs(struct device *dev, struct vm_area_struct *vma, { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_mmap(dev, vma, cpu_addr, dma_addr, size, attrs); if (!ops->mmap) @@ -410,7 +434,7 @@ u64 dma_get_required_mask(struct device *dev) { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_get_required_mask(dev); if (ops->get_required_mask) return ops->get_required_mask(dev); @@ -441,7 +465,7 @@ void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, /* let the implementation decide on the zone to allocate from: */ flag &= ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) cpu_addr = dma_direct_alloc(dev, size, dma_handle, flag, attrs); else if (ops->alloc) cpu_addr = ops->alloc(dev, size, dma_handle, flag, attrs); @@ -473,7 +497,7 @@ void dma_free_attrs(struct device *dev, size_t size, void *cpu_addr, return; debug_dma_free_coherent(dev, size, cpu_addr, dma_handle); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) dma_direct_free(dev, size, cpu_addr, dma_handle, attrs); else if (ops->free) ops->free(dev, size, cpu_addr, dma_handle, attrs); @@ -484,7 +508,11 @@ int dma_supported(struct device *dev, u64 mask) { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + /* + * ->dma_supported sets the bypass flag, so we must always call + * into the method here unless the device is truly direct mapped. + */ + if (!ops) return dma_direct_supported(dev, mask); if (!ops->dma_supported) return 1; @@ -540,7 +568,7 @@ void dma_cache_sync(struct device *dev, void *vaddr, size_t size, BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) arch_dma_cache_sync(dev, vaddr, size, dir); else if (ops->cache_sync) ops->cache_sync(dev, vaddr, size, dir); @@ -552,7 +580,7 @@ size_t dma_max_mapping_size(struct device *dev) const struct dma_map_ops *ops = get_dma_ops(dev); size_t size = SIZE_MAX; - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) size = dma_direct_max_mapping_size(dev); else if (ops && ops->max_mapping_size) size = ops->max_mapping_size(dev); -- 2.25.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-14 12:25 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, linuxppc-dev, Lu Baolu Several IOMMU drivers have a bypass mode where they can use a direct mapping if the devices DMA mask is large enough. Add generic support to the core dma-mapping code to do that to switch those drivers to a common solution. Signed-off-by: Christoph Hellwig <hch@lst.de> --- include/linux/device.h | 6 ++++ kernel/dma/mapping.c | 70 +++++++++++++++++++++++++++++------------- 2 files changed, 55 insertions(+), 21 deletions(-) diff --git a/include/linux/device.h b/include/linux/device.h index ac8e37cd716a..143c2e0edb19 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -523,6 +523,11 @@ struct dev_links_info { * sync_state() callback. * @dma_coherent: this particular device is dma coherent, even if the * architecture supports non-coherent devices. + * @dma_ops_bypass: If set to %true then the dma_ops are bypassed for the + * streaming DMA operations (->map_* / ->unmap_* / ->sync_*), + * and optionall (if the coherent mask is large enough) also + * for dma allocations. This flag is managed by the dma ops + * instance from ->dma_supported. * * At the lowest level, every device in a Linux system is represented by an * instance of struct device. The device structure contains the information @@ -622,6 +627,7 @@ struct device { defined(CONFIG_ARCH_HAS_SYNC_DMA_FOR_CPU_ALL) bool dma_coherent:1; #endif + bool dma_ops_bypass : 1; }; static inline struct device *kobj_to_dev(struct kobject *kobj) diff --git a/kernel/dma/mapping.c b/kernel/dma/mapping.c index 8e4155f9ee69..d776fb9f5adb 100644 --- a/kernel/dma/mapping.c +++ b/kernel/dma/mapping.c @@ -105,9 +105,33 @@ void *dmam_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, } EXPORT_SYMBOL(dmam_alloc_attrs); -static inline bool dma_is_direct(const struct dma_map_ops *ops) +/* + * Check if the devices uses a direct mapping for streaming DMA operations. + * This allows IOMMU drivers to set a bypass mode if the DMA mask is large + * enough. + */ +static inline bool dma_alloc_direct(struct device *dev, + const struct dma_map_ops *ops) { - return likely(!ops); + if (likely(!ops)) + return true; + if (!dev->dma_ops_bypass) + return false; + + return min_not_zero(dev->coherent_dma_mask, dev->bus_dma_limit) >= + dma_direct_get_required_mask(dev); +} + +static inline bool dma_map_direct(struct device *dev, + const struct dma_map_ops *ops) +{ + if (likely(!ops)) + return true; + if (!dev->dma_ops_bypass) + return false; + + return min_not_zero(*dev->dma_mask, dev->bus_dma_limit) >= + dma_direct_get_required_mask(dev); } dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, @@ -118,7 +142,7 @@ dma_addr_t dma_map_page_attrs(struct device *dev, struct page *page, dma_addr_t addr; BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) addr = dma_direct_map_page(dev, page, offset, size, dir, attrs); else addr = ops->map_page(dev, page, offset, size, dir, attrs); @@ -134,7 +158,7 @@ void dma_unmap_page_attrs(struct device *dev, dma_addr_t addr, size_t size, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_unmap_page(dev, addr, size, dir, attrs); else if (ops->unmap_page) ops->unmap_page(dev, addr, size, dir, attrs); @@ -153,7 +177,7 @@ int dma_map_sg_attrs(struct device *dev, struct scatterlist *sg, int nents, int ents; BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) ents = dma_direct_map_sg(dev, sg, nents, dir, attrs); else ents = ops->map_sg(dev, sg, nents, dir, attrs); @@ -172,7 +196,7 @@ void dma_unmap_sg_attrs(struct device *dev, struct scatterlist *sg, BUG_ON(!valid_dma_direction(dir)); debug_dma_unmap_sg(dev, sg, nents, dir); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_unmap_sg(dev, sg, nents, dir, attrs); else if (ops->unmap_sg) ops->unmap_sg(dev, sg, nents, dir, attrs); @@ -191,7 +215,7 @@ dma_addr_t dma_map_resource(struct device *dev, phys_addr_t phys_addr, if (WARN_ON_ONCE(pfn_valid(PHYS_PFN(phys_addr)))) return DMA_MAPPING_ERROR; - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) addr = dma_direct_map_resource(dev, phys_addr, size, dir, attrs); else if (ops->map_resource) addr = ops->map_resource(dev, phys_addr, size, dir, attrs); @@ -207,7 +231,7 @@ void dma_unmap_resource(struct device *dev, dma_addr_t addr, size_t size, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (!dma_is_direct(ops) && ops->unmap_resource) + if (!dma_map_direct(dev, ops) && ops->unmap_resource) ops->unmap_resource(dev, addr, size, dir, attrs); debug_dma_unmap_resource(dev, addr, size, dir); } @@ -219,7 +243,7 @@ void dma_sync_single_for_cpu(struct device *dev, dma_addr_t addr, size_t size, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_single_for_cpu(dev, addr, size, dir); else if (ops->sync_single_for_cpu) ops->sync_single_for_cpu(dev, addr, size, dir); @@ -233,7 +257,7 @@ void dma_sync_single_for_device(struct device *dev, dma_addr_t addr, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_single_for_device(dev, addr, size, dir); else if (ops->sync_single_for_device) ops->sync_single_for_device(dev, addr, size, dir); @@ -247,7 +271,7 @@ void dma_sync_sg_for_cpu(struct device *dev, struct scatterlist *sg, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_sg_for_cpu(dev, sg, nelems, dir); else if (ops->sync_sg_for_cpu) ops->sync_sg_for_cpu(dev, sg, nelems, dir); @@ -261,7 +285,7 @@ void dma_sync_sg_for_device(struct device *dev, struct scatterlist *sg, const struct dma_map_ops *ops = get_dma_ops(dev); BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) dma_direct_sync_sg_for_device(dev, sg, nelems, dir); else if (ops->sync_sg_for_device) ops->sync_sg_for_device(dev, sg, nelems, dir); @@ -302,7 +326,7 @@ int dma_get_sgtable_attrs(struct device *dev, struct sg_table *sgt, { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_get_sgtable(dev, sgt, cpu_addr, dma_addr, size, attrs); if (!ops->get_sgtable) @@ -372,7 +396,7 @@ bool dma_can_mmap(struct device *dev) { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_can_mmap(dev); return ops->mmap != NULL; } @@ -397,7 +421,7 @@ int dma_mmap_attrs(struct device *dev, struct vm_area_struct *vma, { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_mmap(dev, vma, cpu_addr, dma_addr, size, attrs); if (!ops->mmap) @@ -410,7 +434,7 @@ u64 dma_get_required_mask(struct device *dev) { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) return dma_direct_get_required_mask(dev); if (ops->get_required_mask) return ops->get_required_mask(dev); @@ -441,7 +465,7 @@ void *dma_alloc_attrs(struct device *dev, size_t size, dma_addr_t *dma_handle, /* let the implementation decide on the zone to allocate from: */ flag &= ~(__GFP_DMA | __GFP_DMA32 | __GFP_HIGHMEM); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) cpu_addr = dma_direct_alloc(dev, size, dma_handle, flag, attrs); else if (ops->alloc) cpu_addr = ops->alloc(dev, size, dma_handle, flag, attrs); @@ -473,7 +497,7 @@ void dma_free_attrs(struct device *dev, size_t size, void *cpu_addr, return; debug_dma_free_coherent(dev, size, cpu_addr, dma_handle); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) dma_direct_free(dev, size, cpu_addr, dma_handle, attrs); else if (ops->free) ops->free(dev, size, cpu_addr, dma_handle, attrs); @@ -484,7 +508,11 @@ int dma_supported(struct device *dev, u64 mask) { const struct dma_map_ops *ops = get_dma_ops(dev); - if (dma_is_direct(ops)) + /* + * ->dma_supported sets the bypass flag, so we must always call + * into the method here unless the device is truly direct mapped. + */ + if (!ops) return dma_direct_supported(dev, mask); if (!ops->dma_supported) return 1; @@ -540,7 +568,7 @@ void dma_cache_sync(struct device *dev, void *vaddr, size_t size, BUG_ON(!valid_dma_direction(dir)); - if (dma_is_direct(ops)) + if (dma_alloc_direct(dev, ops)) arch_dma_cache_sync(dev, vaddr, size, dir); else if (ops->cache_sync) ops->cache_sync(dev, vaddr, size, dir); @@ -552,7 +580,7 @@ size_t dma_max_mapping_size(struct device *dev) const struct dma_map_ops *ops = get_dma_ops(dev); size_t size = SIZE_MAX; - if (dma_is_direct(ops)) + if (dma_map_direct(dev, ops)) size = dma_direct_max_mapping_size(dev); else if (ops && ops->max_mapping_size) size = ops->max_mapping_size(dev); -- 2.25.1 ^ permalink raw reply related [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device 2020-04-14 12:25 ` Christoph Hellwig (?) @ 2020-04-14 12:47 ` Greg Kroah-Hartman -1 siblings, 0 replies; 60+ messages in thread From: Greg Kroah-Hartman @ 2020-04-14 12:47 UTC (permalink / raw) To: Christoph Hellwig Cc: iommu, Alexey Kardashevskiy, linuxppc-dev, Lu Baolu, Joerg Roedel, Robin Murphy, linux-kernel On Tue, Apr 14, 2020 at 02:25:05PM +0200, Christoph Hellwig wrote: > Several IOMMU drivers have a bypass mode where they can use a direct > mapping if the devices DMA mask is large enough. Add generic support > to the core dma-mapping code to do that to switch those drivers to > a common solution. > > Signed-off-by: Christoph Hellwig <hch@lst.de> > --- > include/linux/device.h | 6 ++++ > kernel/dma/mapping.c | 70 +++++++++++++++++++++++++++++------------- > 2 files changed, 55 insertions(+), 21 deletions(-) Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-14 12:47 ` Greg Kroah-Hartman 0 siblings, 0 replies; 60+ messages in thread From: Greg Kroah-Hartman @ 2020-04-14 12:47 UTC (permalink / raw) To: Christoph Hellwig; +Cc: linuxppc-dev, linux-kernel, iommu, Robin Murphy On Tue, Apr 14, 2020 at 02:25:05PM +0200, Christoph Hellwig wrote: > Several IOMMU drivers have a bypass mode where they can use a direct > mapping if the devices DMA mask is large enough. Add generic support > to the core dma-mapping code to do that to switch those drivers to > a common solution. > > Signed-off-by: Christoph Hellwig <hch@lst.de> > --- > include/linux/device.h | 6 ++++ > kernel/dma/mapping.c | 70 +++++++++++++++++++++++++++++------------- > 2 files changed, 55 insertions(+), 21 deletions(-) Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-14 12:47 ` Greg Kroah-Hartman 0 siblings, 0 replies; 60+ messages in thread From: Greg Kroah-Hartman @ 2020-04-14 12:47 UTC (permalink / raw) To: Christoph Hellwig Cc: Alexey Kardashevskiy, linuxppc-dev, Joerg Roedel, linux-kernel, iommu, Robin Murphy, Lu Baolu On Tue, Apr 14, 2020 at 02:25:05PM +0200, Christoph Hellwig wrote: > Several IOMMU drivers have a bypass mode where they can use a direct > mapping if the devices DMA mask is large enough. Add generic support > to the core dma-mapping code to do that to switch those drivers to > a common solution. > > Signed-off-by: Christoph Hellwig <hch@lst.de> > --- > include/linux/device.h | 6 ++++ > kernel/dma/mapping.c | 70 +++++++++++++++++++++++++++++------------- > 2 files changed, 55 insertions(+), 21 deletions(-) Reviewed-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device 2020-04-14 12:25 ` Christoph Hellwig (?) @ 2020-04-18 12:42 ` Joerg Roedel -1 siblings, 0 replies; 60+ messages in thread From: Joerg Roedel @ 2020-04-18 12:42 UTC (permalink / raw) To: Christoph Hellwig Cc: iommu, Alexey Kardashevskiy, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Robin Murphy, linux-kernel Hi Christoph, On Tue, Apr 14, 2020 at 02:25:05PM +0200, Christoph Hellwig wrote: > +static inline bool dma_map_direct(struct device *dev, > + const struct dma_map_ops *ops) > +{ > + if (likely(!ops)) > + return true; > + if (!dev->dma_ops_bypass) > + return false; > + > + return min_not_zero(*dev->dma_mask, dev->bus_dma_limit) >= > + dma_direct_get_required_mask(dev); Why is the dma-mask check done here? The dma-direct code handles memory outside of the devices dma-mask with swiotlb, no? I also don't quite get what the difference between setting the dma_ops_bypass flag non-zero and setting ops to NULL is. Joerg ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-18 12:42 ` Joerg Roedel 0 siblings, 0 replies; 60+ messages in thread From: Joerg Roedel @ 2020-04-18 12:42 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, linuxppc-dev Hi Christoph, On Tue, Apr 14, 2020 at 02:25:05PM +0200, Christoph Hellwig wrote: > +static inline bool dma_map_direct(struct device *dev, > + const struct dma_map_ops *ops) > +{ > + if (likely(!ops)) > + return true; > + if (!dev->dma_ops_bypass) > + return false; > + > + return min_not_zero(*dev->dma_mask, dev->bus_dma_limit) >= > + dma_direct_get_required_mask(dev); Why is the dma-mask check done here? The dma-direct code handles memory outside of the devices dma-mask with swiotlb, no? I also don't quite get what the difference between setting the dma_ops_bypass flag non-zero and setting ops to NULL is. Joerg _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-18 12:42 ` Joerg Roedel 0 siblings, 0 replies; 60+ messages in thread From: Joerg Roedel @ 2020-04-18 12:42 UTC (permalink / raw) To: Christoph Hellwig Cc: Alexey Kardashevskiy, Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, linuxppc-dev, Lu Baolu Hi Christoph, On Tue, Apr 14, 2020 at 02:25:05PM +0200, Christoph Hellwig wrote: > +static inline bool dma_map_direct(struct device *dev, > + const struct dma_map_ops *ops) > +{ > + if (likely(!ops)) > + return true; > + if (!dev->dma_ops_bypass) > + return false; > + > + return min_not_zero(*dev->dma_mask, dev->bus_dma_limit) >= > + dma_direct_get_required_mask(dev); Why is the dma-mask check done here? The dma-direct code handles memory outside of the devices dma-mask with swiotlb, no? I also don't quite get what the difference between setting the dma_ops_bypass flag non-zero and setting ops to NULL is. Joerg ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device 2020-04-18 12:42 ` Joerg Roedel (?) @ 2020-04-19 8:00 ` Christoph Hellwig -1 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-19 8:00 UTC (permalink / raw) To: Joerg Roedel Cc: Christoph Hellwig, iommu, Alexey Kardashevskiy, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Robin Murphy, linux-kernel On Sat, Apr 18, 2020 at 02:42:05PM +0200, Joerg Roedel wrote: > Hi Christoph, > > On Tue, Apr 14, 2020 at 02:25:05PM +0200, Christoph Hellwig wrote: > > +static inline bool dma_map_direct(struct device *dev, > > + const struct dma_map_ops *ops) > > +{ > > + if (likely(!ops)) > > + return true; > > + if (!dev->dma_ops_bypass) > > + return false; > > + > > + return min_not_zero(*dev->dma_mask, dev->bus_dma_limit) >= > > + dma_direct_get_required_mask(dev); > > Why is the dma-mask check done here? The dma-direct code handles memory > outside of the devices dma-mask with swiotlb, no? > > I also don't quite get what the difference between setting the > dma_ops_bypass flag non-zero and setting ops to NULL is. The difference is that NULL ops mean imply the direct mapping is always used, dma_ops_bypass means a direct mapping is used if no bounce buffering using swiotlb is needed, which should also answer your first question. The idea is to consolidate code in the core to use an opportunistic direct mapping instead of the dynamic iommu mapping. I though the cover letter and commit log explained this well enough, but maybe I need to do a better job. ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-19 8:00 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-19 8:00 UTC (permalink / raw) To: Joerg Roedel Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, linuxppc-dev, Christoph Hellwig On Sat, Apr 18, 2020 at 02:42:05PM +0200, Joerg Roedel wrote: > Hi Christoph, > > On Tue, Apr 14, 2020 at 02:25:05PM +0200, Christoph Hellwig wrote: > > +static inline bool dma_map_direct(struct device *dev, > > + const struct dma_map_ops *ops) > > +{ > > + if (likely(!ops)) > > + return true; > > + if (!dev->dma_ops_bypass) > > + return false; > > + > > + return min_not_zero(*dev->dma_mask, dev->bus_dma_limit) >= > > + dma_direct_get_required_mask(dev); > > Why is the dma-mask check done here? The dma-direct code handles memory > outside of the devices dma-mask with swiotlb, no? > > I also don't quite get what the difference between setting the > dma_ops_bypass flag non-zero and setting ops to NULL is. The difference is that NULL ops mean imply the direct mapping is always used, dma_ops_bypass means a direct mapping is used if no bounce buffering using swiotlb is needed, which should also answer your first question. The idea is to consolidate code in the core to use an opportunistic direct mapping instead of the dynamic iommu mapping. I though the cover letter and commit log explained this well enough, but maybe I need to do a better job. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-19 8:00 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-19 8:00 UTC (permalink / raw) To: Joerg Roedel Cc: Alexey Kardashevskiy, Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, linuxppc-dev, Christoph Hellwig, Lu Baolu On Sat, Apr 18, 2020 at 02:42:05PM +0200, Joerg Roedel wrote: > Hi Christoph, > > On Tue, Apr 14, 2020 at 02:25:05PM +0200, Christoph Hellwig wrote: > > +static inline bool dma_map_direct(struct device *dev, > > + const struct dma_map_ops *ops) > > +{ > > + if (likely(!ops)) > > + return true; > > + if (!dev->dma_ops_bypass) > > + return false; > > + > > + return min_not_zero(*dev->dma_mask, dev->bus_dma_limit) >= > > + dma_direct_get_required_mask(dev); > > Why is the dma-mask check done here? The dma-direct code handles memory > outside of the devices dma-mask with swiotlb, no? > > I also don't quite get what the difference between setting the > dma_ops_bypass flag non-zero and setting ops to NULL is. The difference is that NULL ops mean imply the direct mapping is always used, dma_ops_bypass means a direct mapping is used if no bounce buffering using swiotlb is needed, which should also answer your first question. The idea is to consolidate code in the core to use an opportunistic direct mapping instead of the dynamic iommu mapping. I though the cover letter and commit log explained this well enough, but maybe I need to do a better job. ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device 2020-04-19 8:00 ` Christoph Hellwig (?) @ 2020-04-19 12:25 ` Joerg Roedel -1 siblings, 0 replies; 60+ messages in thread From: Joerg Roedel @ 2020-04-19 12:25 UTC (permalink / raw) To: Christoph Hellwig Cc: iommu, Alexey Kardashevskiy, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Robin Murphy, linux-kernel On Sun, Apr 19, 2020 at 10:00:58AM +0200, Christoph Hellwig wrote: > The difference is that NULL ops mean imply the direct mapping is always > used, dma_ops_bypass means a direct mapping is used if no bounce buffering > using swiotlb is needed, which should also answer your first question. > The idea is to consolidate code in the core to use an opportunistic > direct mapping instead of the dynamic iommu mapping. I though the cover > letter and commit log explained this well enough, but maybe I need to > do a better job. Ah right, now I see it, when dma_ops_bypass is set it will only use direct mapping when the available memory fits into the device's dma_masks, and calls into dma_ops otherwise. I wonder how that will interact with an IOMMU driver, which has to make sure that the direct mapping is accessible for the device at all. It can either put the device into a passthrough domain for direct mapping or into a re-mapped domain, but then all DMA-API calls need to use dma-ops. When the dma_mask covers available memory but coherent_mask doesn't, the streaming calls will use dma-direct and alloc_coherent() calls into dma-ops. There is no way for the IOMMU driver to ensure both works. So what are the conditions under which an IOMMU driver would set dma_ops_bypass to 1 and get a different result as to when setting dev->dma_ops to NULL? Regards, Joerg ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-19 12:25 ` Joerg Roedel 0 siblings, 0 replies; 60+ messages in thread From: Joerg Roedel @ 2020-04-19 12:25 UTC (permalink / raw) To: Christoph Hellwig Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, linuxppc-dev On Sun, Apr 19, 2020 at 10:00:58AM +0200, Christoph Hellwig wrote: > The difference is that NULL ops mean imply the direct mapping is always > used, dma_ops_bypass means a direct mapping is used if no bounce buffering > using swiotlb is needed, which should also answer your first question. > The idea is to consolidate code in the core to use an opportunistic > direct mapping instead of the dynamic iommu mapping. I though the cover > letter and commit log explained this well enough, but maybe I need to > do a better job. Ah right, now I see it, when dma_ops_bypass is set it will only use direct mapping when the available memory fits into the device's dma_masks, and calls into dma_ops otherwise. I wonder how that will interact with an IOMMU driver, which has to make sure that the direct mapping is accessible for the device at all. It can either put the device into a passthrough domain for direct mapping or into a re-mapped domain, but then all DMA-API calls need to use dma-ops. When the dma_mask covers available memory but coherent_mask doesn't, the streaming calls will use dma-direct and alloc_coherent() calls into dma-ops. There is no way for the IOMMU driver to ensure both works. So what are the conditions under which an IOMMU driver would set dma_ops_bypass to 1 and get a different result as to when setting dev->dma_ops to NULL? Regards, Joerg _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-19 12:25 ` Joerg Roedel 0 siblings, 0 replies; 60+ messages in thread From: Joerg Roedel @ 2020-04-19 12:25 UTC (permalink / raw) To: Christoph Hellwig Cc: Alexey Kardashevskiy, Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, linuxppc-dev, Lu Baolu On Sun, Apr 19, 2020 at 10:00:58AM +0200, Christoph Hellwig wrote: > The difference is that NULL ops mean imply the direct mapping is always > used, dma_ops_bypass means a direct mapping is used if no bounce buffering > using swiotlb is needed, which should also answer your first question. > The idea is to consolidate code in the core to use an opportunistic > direct mapping instead of the dynamic iommu mapping. I though the cover > letter and commit log explained this well enough, but maybe I need to > do a better job. Ah right, now I see it, when dma_ops_bypass is set it will only use direct mapping when the available memory fits into the device's dma_masks, and calls into dma_ops otherwise. I wonder how that will interact with an IOMMU driver, which has to make sure that the direct mapping is accessible for the device at all. It can either put the device into a passthrough domain for direct mapping or into a re-mapped domain, but then all DMA-API calls need to use dma-ops. When the dma_mask covers available memory but coherent_mask doesn't, the streaming calls will use dma-direct and alloc_coherent() calls into dma-ops. There is no way for the IOMMU driver to ensure both works. So what are the conditions under which an IOMMU driver would set dma_ops_bypass to 1 and get a different result as to when setting dev->dma_ops to NULL? Regards, Joerg ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device 2020-04-19 12:25 ` Joerg Roedel (?) @ 2020-04-19 14:46 ` Alexey Kardashevskiy -1 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-04-19 14:46 UTC (permalink / raw) To: Joerg Roedel, Christoph Hellwig Cc: iommu, linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Robin Murphy, linux-kernel On 19/04/2020 22:25, Joerg Roedel wrote: > On Sun, Apr 19, 2020 at 10:00:58AM +0200, Christoph Hellwig wrote: >> The difference is that NULL ops mean imply the direct mapping is always >> used, dma_ops_bypass means a direct mapping is used if no bounce buffering >> using swiotlb is needed, which should also answer your first question. >> The idea is to consolidate code in the core to use an opportunistic >> direct mapping instead of the dynamic iommu mapping. I though the cover >> letter and commit log explained this well enough, but maybe I need to >> do a better job. > > Ah right, now I see it, when dma_ops_bypass is set it will only use > direct mapping when the available memory fits into the device's > dma_masks, and calls into dma_ops otherwise. > > I wonder how that will interact with an IOMMU driver, which has to make > sure that the direct mapping is accessible for the device at all. It > can either put the device into a passthrough domain for direct mapping > or into a re-mapped domain, but then all DMA-API calls need to use dma-ops. > When the dma_mask covers available memory but coherent_mask doesn't, > the streaming calls will use dma-direct and alloc_coherent() calls into > dma-ops. There is no way for the IOMMU driver to ensure both works. > > So what are the conditions under which an IOMMU driver would set > dma_ops_bypass to 1 and get a different result as to when setting > dev->dma_ops to NULL? One example is powerpc64/pseries (arch/powerpc/kernel/dma-iommu.c) where dma_iommu_ops::dma_iommu_dma_supported() (i.e. need ops) decides whether to set dma_ops_bypass to 1. It tries creating a DMA window with 1:1 mapping to fit maximum possible RAM address, if that works, then ops is not needed. -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-19 14:46 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-04-19 14:46 UTC (permalink / raw) To: Joerg Roedel, Christoph Hellwig Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, linuxppc-dev On 19/04/2020 22:25, Joerg Roedel wrote: > On Sun, Apr 19, 2020 at 10:00:58AM +0200, Christoph Hellwig wrote: >> The difference is that NULL ops mean imply the direct mapping is always >> used, dma_ops_bypass means a direct mapping is used if no bounce buffering >> using swiotlb is needed, which should also answer your first question. >> The idea is to consolidate code in the core to use an opportunistic >> direct mapping instead of the dynamic iommu mapping. I though the cover >> letter and commit log explained this well enough, but maybe I need to >> do a better job. > > Ah right, now I see it, when dma_ops_bypass is set it will only use > direct mapping when the available memory fits into the device's > dma_masks, and calls into dma_ops otherwise. > > I wonder how that will interact with an IOMMU driver, which has to make > sure that the direct mapping is accessible for the device at all. It > can either put the device into a passthrough domain for direct mapping > or into a re-mapped domain, but then all DMA-API calls need to use dma-ops. > When the dma_mask covers available memory but coherent_mask doesn't, > the streaming calls will use dma-direct and alloc_coherent() calls into > dma-ops. There is no way for the IOMMU driver to ensure both works. > > So what are the conditions under which an IOMMU driver would set > dma_ops_bypass to 1 and get a different result as to when setting > dev->dma_ops to NULL? One example is powerpc64/pseries (arch/powerpc/kernel/dma-iommu.c) where dma_iommu_ops::dma_iommu_dma_supported() (i.e. need ops) decides whether to set dma_ops_bypass to 1. It tries creating a DMA window with 1:1 mapping to fit maximum possible RAM address, if that works, then ops is not needed. -- Alexey _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 60+ messages in thread
* Re: [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device @ 2020-04-19 14:46 ` Alexey Kardashevskiy 0 siblings, 0 replies; 60+ messages in thread From: Alexey Kardashevskiy @ 2020-04-19 14:46 UTC (permalink / raw) To: Joerg Roedel, Christoph Hellwig Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, iommu, linuxppc-dev, Lu Baolu On 19/04/2020 22:25, Joerg Roedel wrote: > On Sun, Apr 19, 2020 at 10:00:58AM +0200, Christoph Hellwig wrote: >> The difference is that NULL ops mean imply the direct mapping is always >> used, dma_ops_bypass means a direct mapping is used if no bounce buffering >> using swiotlb is needed, which should also answer your first question. >> The idea is to consolidate code in the core to use an opportunistic >> direct mapping instead of the dynamic iommu mapping. I though the cover >> letter and commit log explained this well enough, but maybe I need to >> do a better job. > > Ah right, now I see it, when dma_ops_bypass is set it will only use > direct mapping when the available memory fits into the device's > dma_masks, and calls into dma_ops otherwise. > > I wonder how that will interact with an IOMMU driver, which has to make > sure that the direct mapping is accessible for the device at all. It > can either put the device into a passthrough domain for direct mapping > or into a re-mapped domain, but then all DMA-API calls need to use dma-ops. > When the dma_mask covers available memory but coherent_mask doesn't, > the streaming calls will use dma-direct and alloc_coherent() calls into > dma-ops. There is no way for the IOMMU driver to ensure both works. > > So what are the conditions under which an IOMMU driver would set > dma_ops_bypass to 1 and get a different result as to when setting > dev->dma_ops to NULL? One example is powerpc64/pseries (arch/powerpc/kernel/dma-iommu.c) where dma_iommu_ops::dma_iommu_dma_supported() (i.e. need ops) decides whether to set dma_ops_bypass to 1. It tries creating a DMA window with 1:1 mapping to fit maximum possible RAM address, if that works, then ops is not needed. -- Alexey ^ permalink raw reply [flat|nested] 60+ messages in thread
* [PATCH 4/4] powerpc: use the generic dma_ops_bypass mode 2020-04-14 12:25 ` Christoph Hellwig (?) @ 2020-04-14 12:25 ` Christoph Hellwig -1 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: linuxppc-dev, Lu Baolu, Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel Use the DMA API bypass mechanism for direct window mappings. This uses common code and speed up the direct mapping case by avoiding indirect calls just when not using dma ops at all. It also fixes a problem where the sync_* methods were using the bypass check for DMA allocations, but those are part of the streaming ops. Note that this patch loses the DMA_ATTR_WEAK_ORDERING override, which has never been well defined, as is only used by a few drivers, which IIRC never showed up in the typical Cell blade setups that are affected by the ordering workaround. Fixes: efd176a04bef ("powerpc/pseries/dma: Allow SWIOTLB") Signed-off-by: Christoph Hellwig <hch@lst.de> --- arch/powerpc/include/asm/device.h | 5 -- arch/powerpc/kernel/dma-iommu.c | 90 ++++--------------------------- 2 files changed, 9 insertions(+), 86 deletions(-) diff --git a/arch/powerpc/include/asm/device.h b/arch/powerpc/include/asm/device.h index 266542769e4b..452402215e12 100644 --- a/arch/powerpc/include/asm/device.h +++ b/arch/powerpc/include/asm/device.h @@ -18,11 +18,6 @@ struct iommu_table; * drivers/macintosh/macio_asic.c */ struct dev_archdata { - /* - * Set to %true if the dma_iommu_ops are requested to use a direct - * window instead of dynamically mapping memory. - */ - bool iommu_bypass : 1; /* * These two used to be a union. However, with the hybrid ops we need * both so here we store both a DMA offset for direct mappings and diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c index e486d1d78de2..569fecd7b5b2 100644 --- a/arch/powerpc/kernel/dma-iommu.c +++ b/arch/powerpc/kernel/dma-iommu.c @@ -14,23 +14,6 @@ * Generic iommu implementation */ -/* - * The coherent mask may be smaller than the real mask, check if we can - * really use a direct window. - */ -static inline bool dma_iommu_alloc_bypass(struct device *dev) -{ - return dev->archdata.iommu_bypass && !iommu_fixed_is_weak && - dma_direct_supported(dev, dev->coherent_dma_mask); -} - -static inline bool dma_iommu_map_bypass(struct device *dev, - unsigned long attrs) -{ - return dev->archdata.iommu_bypass && - (!iommu_fixed_is_weak || (attrs & DMA_ATTR_WEAK_ORDERING)); -} - /* Allocates a contiguous real buffer and creates mappings over it. * Returns the virtual address of the buffer and sets dma_handle * to the dma address (mapping) of the first page. @@ -39,8 +22,6 @@ static void *dma_iommu_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag, unsigned long attrs) { - if (dma_iommu_alloc_bypass(dev)) - return dma_direct_alloc(dev, size, dma_handle, flag, attrs); return iommu_alloc_coherent(dev, get_iommu_table_base(dev), size, dma_handle, dev->coherent_dma_mask, flag, dev_to_node(dev)); @@ -50,11 +31,7 @@ static void dma_iommu_free_coherent(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_handle, unsigned long attrs) { - if (dma_iommu_alloc_bypass(dev)) - dma_direct_free(dev, size, vaddr, dma_handle, attrs); - else - iommu_free_coherent(get_iommu_table_base(dev), size, vaddr, - dma_handle); + iommu_free_coherent(get_iommu_table_base(dev), size, vaddr, dma_handle); } /* Creates TCEs for a user provided buffer. The user buffer must be @@ -67,9 +44,6 @@ static dma_addr_t dma_iommu_map_page(struct device *dev, struct page *page, enum dma_data_direction direction, unsigned long attrs) { - if (dma_iommu_map_bypass(dev, attrs)) - return dma_direct_map_page(dev, page, offset, size, direction, - attrs); return iommu_map_page(dev, get_iommu_table_base(dev), page, offset, size, dma_get_mask(dev), direction, attrs); } @@ -79,11 +53,8 @@ static void dma_iommu_unmap_page(struct device *dev, dma_addr_t dma_handle, size_t size, enum dma_data_direction direction, unsigned long attrs) { - if (!dma_iommu_map_bypass(dev, attrs)) - iommu_unmap_page(get_iommu_table_base(dev), dma_handle, size, - direction, attrs); - else - dma_direct_unmap_page(dev, dma_handle, size, direction, attrs); + iommu_unmap_page(get_iommu_table_base(dev), dma_handle, size, direction, + attrs); } @@ -91,8 +62,6 @@ static int dma_iommu_map_sg(struct device *dev, struct scatterlist *sglist, int nelems, enum dma_data_direction direction, unsigned long attrs) { - if (dma_iommu_map_bypass(dev, attrs)) - return dma_direct_map_sg(dev, sglist, nelems, direction, attrs); return ppc_iommu_map_sg(dev, get_iommu_table_base(dev), sglist, nelems, dma_get_mask(dev), direction, attrs); } @@ -101,11 +70,8 @@ static void dma_iommu_unmap_sg(struct device *dev, struct scatterlist *sglist, int nelems, enum dma_data_direction direction, unsigned long attrs) { - if (!dma_iommu_map_bypass(dev, attrs)) - ppc_iommu_unmap_sg(get_iommu_table_base(dev), sglist, nelems, + ppc_iommu_unmap_sg(get_iommu_table_base(dev), sglist, nelems, direction, attrs); - else - dma_direct_unmap_sg(dev, sglist, nelems, direction, attrs); } static bool dma_iommu_bypass_supported(struct device *dev, u64 mask) @@ -113,8 +79,9 @@ static bool dma_iommu_bypass_supported(struct device *dev, u64 mask) struct pci_dev *pdev = to_pci_dev(dev); struct pci_controller *phb = pci_bus_to_host(pdev->bus); - return phb->controller_ops.iommu_bypass_supported && - phb->controller_ops.iommu_bypass_supported(pdev, mask); + if (iommu_fixed_is_weak || !phb->controller_ops.iommu_bypass_supported) + return false; + return phb->controller_ops.iommu_bypass_supported(pdev, mask); } /* We support DMA to/from any memory page via the iommu */ @@ -123,7 +90,7 @@ int dma_iommu_dma_supported(struct device *dev, u64 mask) struct iommu_table *tbl = get_iommu_table_base(dev); if (dev_is_pci(dev) && dma_iommu_bypass_supported(dev, mask)) { - dev->archdata.iommu_bypass = true; + dev->dma_ops_bypass = true; dev_dbg(dev, "iommu: 64-bit OK, using fixed ops\n"); return 1; } @@ -141,7 +108,7 @@ int dma_iommu_dma_supported(struct device *dev, u64 mask) } dev_dbg(dev, "iommu: not 64-bit, using default ops\n"); - dev->archdata.iommu_bypass = false; + dev->dma_ops_bypass = false; return 1; } @@ -153,47 +120,12 @@ u64 dma_iommu_get_required_mask(struct device *dev) if (!tbl) return 0; - if (dev_is_pci(dev)) { - u64 bypass_mask = dma_direct_get_required_mask(dev); - - if (dma_iommu_bypass_supported(dev, bypass_mask)) - return bypass_mask; - } - mask = 1ULL < (fls_long(tbl->it_offset + tbl->it_size) - 1); mask += mask - 1; return mask; } -static void dma_iommu_sync_for_cpu(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_single_for_cpu(dev, addr, size, dir); -} - -static void dma_iommu_sync_for_device(struct device *dev, dma_addr_t addr, - size_t sz, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_single_for_device(dev, addr, sz, dir); -} - -extern void dma_iommu_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_sg_for_cpu(dev, sgl, nents, dir); -} - -extern void dma_iommu_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_sg_for_device(dev, sgl, nents, dir); -} - const struct dma_map_ops dma_iommu_ops = { .alloc = dma_iommu_alloc_coherent, .free = dma_iommu_free_coherent, @@ -203,10 +135,6 @@ const struct dma_map_ops dma_iommu_ops = { .map_page = dma_iommu_map_page, .unmap_page = dma_iommu_unmap_page, .get_required_mask = dma_iommu_get_required_mask, - .sync_single_for_cpu = dma_iommu_sync_for_cpu, - .sync_single_for_device = dma_iommu_sync_for_device, - .sync_sg_for_cpu = dma_iommu_sync_sg_for_cpu, - .sync_sg_for_device = dma_iommu_sync_sg_for_device, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, }; -- 2.25.1 ^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 4/4] powerpc: use the generic dma_ops_bypass mode @ 2020-04-14 12:25 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Robin Murphy, linux-kernel, linuxppc-dev Use the DMA API bypass mechanism for direct window mappings. This uses common code and speed up the direct mapping case by avoiding indirect calls just when not using dma ops at all. It also fixes a problem where the sync_* methods were using the bypass check for DMA allocations, but those are part of the streaming ops. Note that this patch loses the DMA_ATTR_WEAK_ORDERING override, which has never been well defined, as is only used by a few drivers, which IIRC never showed up in the typical Cell blade setups that are affected by the ordering workaround. Fixes: efd176a04bef ("powerpc/pseries/dma: Allow SWIOTLB") Signed-off-by: Christoph Hellwig <hch@lst.de> --- arch/powerpc/include/asm/device.h | 5 -- arch/powerpc/kernel/dma-iommu.c | 90 ++++--------------------------- 2 files changed, 9 insertions(+), 86 deletions(-) diff --git a/arch/powerpc/include/asm/device.h b/arch/powerpc/include/asm/device.h index 266542769e4b..452402215e12 100644 --- a/arch/powerpc/include/asm/device.h +++ b/arch/powerpc/include/asm/device.h @@ -18,11 +18,6 @@ struct iommu_table; * drivers/macintosh/macio_asic.c */ struct dev_archdata { - /* - * Set to %true if the dma_iommu_ops are requested to use a direct - * window instead of dynamically mapping memory. - */ - bool iommu_bypass : 1; /* * These two used to be a union. However, with the hybrid ops we need * both so here we store both a DMA offset for direct mappings and diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c index e486d1d78de2..569fecd7b5b2 100644 --- a/arch/powerpc/kernel/dma-iommu.c +++ b/arch/powerpc/kernel/dma-iommu.c @@ -14,23 +14,6 @@ * Generic iommu implementation */ -/* - * The coherent mask may be smaller than the real mask, check if we can - * really use a direct window. - */ -static inline bool dma_iommu_alloc_bypass(struct device *dev) -{ - return dev->archdata.iommu_bypass && !iommu_fixed_is_weak && - dma_direct_supported(dev, dev->coherent_dma_mask); -} - -static inline bool dma_iommu_map_bypass(struct device *dev, - unsigned long attrs) -{ - return dev->archdata.iommu_bypass && - (!iommu_fixed_is_weak || (attrs & DMA_ATTR_WEAK_ORDERING)); -} - /* Allocates a contiguous real buffer and creates mappings over it. * Returns the virtual address of the buffer and sets dma_handle * to the dma address (mapping) of the first page. @@ -39,8 +22,6 @@ static void *dma_iommu_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag, unsigned long attrs) { - if (dma_iommu_alloc_bypass(dev)) - return dma_direct_alloc(dev, size, dma_handle, flag, attrs); return iommu_alloc_coherent(dev, get_iommu_table_base(dev), size, dma_handle, dev->coherent_dma_mask, flag, dev_to_node(dev)); @@ -50,11 +31,7 @@ static void dma_iommu_free_coherent(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_handle, unsigned long attrs) { - if (dma_iommu_alloc_bypass(dev)) - dma_direct_free(dev, size, vaddr, dma_handle, attrs); - else - iommu_free_coherent(get_iommu_table_base(dev), size, vaddr, - dma_handle); + iommu_free_coherent(get_iommu_table_base(dev), size, vaddr, dma_handle); } /* Creates TCEs for a user provided buffer. The user buffer must be @@ -67,9 +44,6 @@ static dma_addr_t dma_iommu_map_page(struct device *dev, struct page *page, enum dma_data_direction direction, unsigned long attrs) { - if (dma_iommu_map_bypass(dev, attrs)) - return dma_direct_map_page(dev, page, offset, size, direction, - attrs); return iommu_map_page(dev, get_iommu_table_base(dev), page, offset, size, dma_get_mask(dev), direction, attrs); } @@ -79,11 +53,8 @@ static void dma_iommu_unmap_page(struct device *dev, dma_addr_t dma_handle, size_t size, enum dma_data_direction direction, unsigned long attrs) { - if (!dma_iommu_map_bypass(dev, attrs)) - iommu_unmap_page(get_iommu_table_base(dev), dma_handle, size, - direction, attrs); - else - dma_direct_unmap_page(dev, dma_handle, size, direction, attrs); + iommu_unmap_page(get_iommu_table_base(dev), dma_handle, size, direction, + attrs); } @@ -91,8 +62,6 @@ static int dma_iommu_map_sg(struct device *dev, struct scatterlist *sglist, int nelems, enum dma_data_direction direction, unsigned long attrs) { - if (dma_iommu_map_bypass(dev, attrs)) - return dma_direct_map_sg(dev, sglist, nelems, direction, attrs); return ppc_iommu_map_sg(dev, get_iommu_table_base(dev), sglist, nelems, dma_get_mask(dev), direction, attrs); } @@ -101,11 +70,8 @@ static void dma_iommu_unmap_sg(struct device *dev, struct scatterlist *sglist, int nelems, enum dma_data_direction direction, unsigned long attrs) { - if (!dma_iommu_map_bypass(dev, attrs)) - ppc_iommu_unmap_sg(get_iommu_table_base(dev), sglist, nelems, + ppc_iommu_unmap_sg(get_iommu_table_base(dev), sglist, nelems, direction, attrs); - else - dma_direct_unmap_sg(dev, sglist, nelems, direction, attrs); } static bool dma_iommu_bypass_supported(struct device *dev, u64 mask) @@ -113,8 +79,9 @@ static bool dma_iommu_bypass_supported(struct device *dev, u64 mask) struct pci_dev *pdev = to_pci_dev(dev); struct pci_controller *phb = pci_bus_to_host(pdev->bus); - return phb->controller_ops.iommu_bypass_supported && - phb->controller_ops.iommu_bypass_supported(pdev, mask); + if (iommu_fixed_is_weak || !phb->controller_ops.iommu_bypass_supported) + return false; + return phb->controller_ops.iommu_bypass_supported(pdev, mask); } /* We support DMA to/from any memory page via the iommu */ @@ -123,7 +90,7 @@ int dma_iommu_dma_supported(struct device *dev, u64 mask) struct iommu_table *tbl = get_iommu_table_base(dev); if (dev_is_pci(dev) && dma_iommu_bypass_supported(dev, mask)) { - dev->archdata.iommu_bypass = true; + dev->dma_ops_bypass = true; dev_dbg(dev, "iommu: 64-bit OK, using fixed ops\n"); return 1; } @@ -141,7 +108,7 @@ int dma_iommu_dma_supported(struct device *dev, u64 mask) } dev_dbg(dev, "iommu: not 64-bit, using default ops\n"); - dev->archdata.iommu_bypass = false; + dev->dma_ops_bypass = false; return 1; } @@ -153,47 +120,12 @@ u64 dma_iommu_get_required_mask(struct device *dev) if (!tbl) return 0; - if (dev_is_pci(dev)) { - u64 bypass_mask = dma_direct_get_required_mask(dev); - - if (dma_iommu_bypass_supported(dev, bypass_mask)) - return bypass_mask; - } - mask = 1ULL < (fls_long(tbl->it_offset + tbl->it_size) - 1); mask += mask - 1; return mask; } -static void dma_iommu_sync_for_cpu(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_single_for_cpu(dev, addr, size, dir); -} - -static void dma_iommu_sync_for_device(struct device *dev, dma_addr_t addr, - size_t sz, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_single_for_device(dev, addr, sz, dir); -} - -extern void dma_iommu_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_sg_for_cpu(dev, sgl, nents, dir); -} - -extern void dma_iommu_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_sg_for_device(dev, sgl, nents, dir); -} - const struct dma_map_ops dma_iommu_ops = { .alloc = dma_iommu_alloc_coherent, .free = dma_iommu_free_coherent, @@ -203,10 +135,6 @@ const struct dma_map_ops dma_iommu_ops = { .map_page = dma_iommu_map_page, .unmap_page = dma_iommu_unmap_page, .get_required_mask = dma_iommu_get_required_mask, - .sync_single_for_cpu = dma_iommu_sync_for_cpu, - .sync_single_for_device = dma_iommu_sync_for_device, - .sync_sg_for_cpu = dma_iommu_sync_sg_for_cpu, - .sync_sg_for_device = dma_iommu_sync_sg_for_device, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, }; -- 2.25.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 60+ messages in thread
* [PATCH 4/4] powerpc: use the generic dma_ops_bypass mode @ 2020-04-14 12:25 ` Christoph Hellwig 0 siblings, 0 replies; 60+ messages in thread From: Christoph Hellwig @ 2020-04-14 12:25 UTC (permalink / raw) To: iommu, Alexey Kardashevskiy Cc: Greg Kroah-Hartman, Joerg Roedel, Robin Murphy, linux-kernel, linuxppc-dev, Lu Baolu Use the DMA API bypass mechanism for direct window mappings. This uses common code and speed up the direct mapping case by avoiding indirect calls just when not using dma ops at all. It also fixes a problem where the sync_* methods were using the bypass check for DMA allocations, but those are part of the streaming ops. Note that this patch loses the DMA_ATTR_WEAK_ORDERING override, which has never been well defined, as is only used by a few drivers, which IIRC never showed up in the typical Cell blade setups that are affected by the ordering workaround. Fixes: efd176a04bef ("powerpc/pseries/dma: Allow SWIOTLB") Signed-off-by: Christoph Hellwig <hch@lst.de> --- arch/powerpc/include/asm/device.h | 5 -- arch/powerpc/kernel/dma-iommu.c | 90 ++++--------------------------- 2 files changed, 9 insertions(+), 86 deletions(-) diff --git a/arch/powerpc/include/asm/device.h b/arch/powerpc/include/asm/device.h index 266542769e4b..452402215e12 100644 --- a/arch/powerpc/include/asm/device.h +++ b/arch/powerpc/include/asm/device.h @@ -18,11 +18,6 @@ struct iommu_table; * drivers/macintosh/macio_asic.c */ struct dev_archdata { - /* - * Set to %true if the dma_iommu_ops are requested to use a direct - * window instead of dynamically mapping memory. - */ - bool iommu_bypass : 1; /* * These two used to be a union. However, with the hybrid ops we need * both so here we store both a DMA offset for direct mappings and diff --git a/arch/powerpc/kernel/dma-iommu.c b/arch/powerpc/kernel/dma-iommu.c index e486d1d78de2..569fecd7b5b2 100644 --- a/arch/powerpc/kernel/dma-iommu.c +++ b/arch/powerpc/kernel/dma-iommu.c @@ -14,23 +14,6 @@ * Generic iommu implementation */ -/* - * The coherent mask may be smaller than the real mask, check if we can - * really use a direct window. - */ -static inline bool dma_iommu_alloc_bypass(struct device *dev) -{ - return dev->archdata.iommu_bypass && !iommu_fixed_is_weak && - dma_direct_supported(dev, dev->coherent_dma_mask); -} - -static inline bool dma_iommu_map_bypass(struct device *dev, - unsigned long attrs) -{ - return dev->archdata.iommu_bypass && - (!iommu_fixed_is_weak || (attrs & DMA_ATTR_WEAK_ORDERING)); -} - /* Allocates a contiguous real buffer and creates mappings over it. * Returns the virtual address of the buffer and sets dma_handle * to the dma address (mapping) of the first page. @@ -39,8 +22,6 @@ static void *dma_iommu_alloc_coherent(struct device *dev, size_t size, dma_addr_t *dma_handle, gfp_t flag, unsigned long attrs) { - if (dma_iommu_alloc_bypass(dev)) - return dma_direct_alloc(dev, size, dma_handle, flag, attrs); return iommu_alloc_coherent(dev, get_iommu_table_base(dev), size, dma_handle, dev->coherent_dma_mask, flag, dev_to_node(dev)); @@ -50,11 +31,7 @@ static void dma_iommu_free_coherent(struct device *dev, size_t size, void *vaddr, dma_addr_t dma_handle, unsigned long attrs) { - if (dma_iommu_alloc_bypass(dev)) - dma_direct_free(dev, size, vaddr, dma_handle, attrs); - else - iommu_free_coherent(get_iommu_table_base(dev), size, vaddr, - dma_handle); + iommu_free_coherent(get_iommu_table_base(dev), size, vaddr, dma_handle); } /* Creates TCEs for a user provided buffer. The user buffer must be @@ -67,9 +44,6 @@ static dma_addr_t dma_iommu_map_page(struct device *dev, struct page *page, enum dma_data_direction direction, unsigned long attrs) { - if (dma_iommu_map_bypass(dev, attrs)) - return dma_direct_map_page(dev, page, offset, size, direction, - attrs); return iommu_map_page(dev, get_iommu_table_base(dev), page, offset, size, dma_get_mask(dev), direction, attrs); } @@ -79,11 +53,8 @@ static void dma_iommu_unmap_page(struct device *dev, dma_addr_t dma_handle, size_t size, enum dma_data_direction direction, unsigned long attrs) { - if (!dma_iommu_map_bypass(dev, attrs)) - iommu_unmap_page(get_iommu_table_base(dev), dma_handle, size, - direction, attrs); - else - dma_direct_unmap_page(dev, dma_handle, size, direction, attrs); + iommu_unmap_page(get_iommu_table_base(dev), dma_handle, size, direction, + attrs); } @@ -91,8 +62,6 @@ static int dma_iommu_map_sg(struct device *dev, struct scatterlist *sglist, int nelems, enum dma_data_direction direction, unsigned long attrs) { - if (dma_iommu_map_bypass(dev, attrs)) - return dma_direct_map_sg(dev, sglist, nelems, direction, attrs); return ppc_iommu_map_sg(dev, get_iommu_table_base(dev), sglist, nelems, dma_get_mask(dev), direction, attrs); } @@ -101,11 +70,8 @@ static void dma_iommu_unmap_sg(struct device *dev, struct scatterlist *sglist, int nelems, enum dma_data_direction direction, unsigned long attrs) { - if (!dma_iommu_map_bypass(dev, attrs)) - ppc_iommu_unmap_sg(get_iommu_table_base(dev), sglist, nelems, + ppc_iommu_unmap_sg(get_iommu_table_base(dev), sglist, nelems, direction, attrs); - else - dma_direct_unmap_sg(dev, sglist, nelems, direction, attrs); } static bool dma_iommu_bypass_supported(struct device *dev, u64 mask) @@ -113,8 +79,9 @@ static bool dma_iommu_bypass_supported(struct device *dev, u64 mask) struct pci_dev *pdev = to_pci_dev(dev); struct pci_controller *phb = pci_bus_to_host(pdev->bus); - return phb->controller_ops.iommu_bypass_supported && - phb->controller_ops.iommu_bypass_supported(pdev, mask); + if (iommu_fixed_is_weak || !phb->controller_ops.iommu_bypass_supported) + return false; + return phb->controller_ops.iommu_bypass_supported(pdev, mask); } /* We support DMA to/from any memory page via the iommu */ @@ -123,7 +90,7 @@ int dma_iommu_dma_supported(struct device *dev, u64 mask) struct iommu_table *tbl = get_iommu_table_base(dev); if (dev_is_pci(dev) && dma_iommu_bypass_supported(dev, mask)) { - dev->archdata.iommu_bypass = true; + dev->dma_ops_bypass = true; dev_dbg(dev, "iommu: 64-bit OK, using fixed ops\n"); return 1; } @@ -141,7 +108,7 @@ int dma_iommu_dma_supported(struct device *dev, u64 mask) } dev_dbg(dev, "iommu: not 64-bit, using default ops\n"); - dev->archdata.iommu_bypass = false; + dev->dma_ops_bypass = false; return 1; } @@ -153,47 +120,12 @@ u64 dma_iommu_get_required_mask(struct device *dev) if (!tbl) return 0; - if (dev_is_pci(dev)) { - u64 bypass_mask = dma_direct_get_required_mask(dev); - - if (dma_iommu_bypass_supported(dev, bypass_mask)) - return bypass_mask; - } - mask = 1ULL < (fls_long(tbl->it_offset + tbl->it_size) - 1); mask += mask - 1; return mask; } -static void dma_iommu_sync_for_cpu(struct device *dev, dma_addr_t addr, - size_t size, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_single_for_cpu(dev, addr, size, dir); -} - -static void dma_iommu_sync_for_device(struct device *dev, dma_addr_t addr, - size_t sz, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_single_for_device(dev, addr, sz, dir); -} - -extern void dma_iommu_sync_sg_for_cpu(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_sg_for_cpu(dev, sgl, nents, dir); -} - -extern void dma_iommu_sync_sg_for_device(struct device *dev, - struct scatterlist *sgl, int nents, enum dma_data_direction dir) -{ - if (dma_iommu_alloc_bypass(dev)) - dma_direct_sync_sg_for_device(dev, sgl, nents, dir); -} - const struct dma_map_ops dma_iommu_ops = { .alloc = dma_iommu_alloc_coherent, .free = dma_iommu_free_coherent, @@ -203,10 +135,6 @@ const struct dma_map_ops dma_iommu_ops = { .map_page = dma_iommu_map_page, .unmap_page = dma_iommu_unmap_page, .get_required_mask = dma_iommu_get_required_mask, - .sync_single_for_cpu = dma_iommu_sync_for_cpu, - .sync_single_for_device = dma_iommu_sync_for_device, - .sync_sg_for_cpu = dma_iommu_sync_sg_for_cpu, - .sync_sg_for_device = dma_iommu_sync_sg_for_device, .mmap = dma_common_mmap, .get_sgtable = dma_common_get_sgtable, }; -- 2.25.1 ^ permalink raw reply related [flat|nested] 60+ messages in thread
end of thread, other threads:[~2020-07-07 15:16 UTC | newest] Thread overview: 60+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-04-14 12:25 generic DMA bypass flag v3 Christoph Hellwig 2020-04-14 12:25 ` Christoph Hellwig 2020-04-14 12:25 ` Christoph Hellwig 2020-04-14 12:25 ` [PATCH 1/4] dma-mapping: move the remaining DMA API calls out of line Christoph Hellwig 2020-04-14 12:25 ` Christoph Hellwig 2020-04-14 12:25 ` Christoph Hellwig 2020-04-15 2:26 ` Alexey Kardashevskiy 2020-04-15 2:26 ` Alexey Kardashevskiy 2020-04-15 2:26 ` Alexey Kardashevskiy 2020-04-15 6:18 ` Christoph Hellwig 2020-04-15 6:18 ` Christoph Hellwig 2020-04-15 6:18 ` Christoph Hellwig 2020-04-15 11:21 ` Alexey Kardashevskiy 2020-04-15 11:21 ` Alexey Kardashevskiy 2020-04-15 11:21 ` Alexey Kardashevskiy 2020-04-17 7:58 ` Christoph Hellwig 2020-04-17 7:58 ` Christoph Hellwig 2020-04-17 7:58 ` Christoph Hellwig 2020-05-05 4:18 ` Alexey Kardashevskiy 2020-05-05 4:18 ` Alexey Kardashevskiy 2020-05-05 4:18 ` Alexey Kardashevskiy 2020-05-09 8:19 ` Christoph Hellwig 2020-05-09 8:19 ` Christoph Hellwig 2020-05-09 8:19 ` Christoph Hellwig 2020-05-09 14:07 ` Alexey Kardashevskiy 2020-05-09 14:07 ` Alexey Kardashevskiy 2020-05-09 14:07 ` Alexey Kardashevskiy 2020-06-03 4:13 ` Alexey Kardashevskiy 2020-06-03 4:13 ` Alexey Kardashevskiy 2020-06-03 4:13 ` Alexey Kardashevskiy 2020-07-07 0:43 ` Alexey Kardashevskiy 2020-07-07 0:43 ` Alexey Kardashevskiy 2020-07-07 0:43 ` Alexey Kardashevskiy 2020-07-07 14:51 ` Christoph Hellwig 2020-07-07 14:51 ` Christoph Hellwig 2020-07-07 14:51 ` Christoph Hellwig 2020-04-14 12:25 ` [PATCH 2/4] dma-mapping: inline the fast path dma-direct calls Christoph Hellwig 2020-04-14 12:25 ` Christoph Hellwig 2020-04-14 12:25 ` Christoph Hellwig 2020-04-14 12:25 ` [PATCH 3/4] dma-mapping: add a dma_ops_bypass flag to struct device Christoph Hellwig 2020-04-14 12:25 ` Christoph Hellwig 2020-04-14 12:25 ` Christoph Hellwig 2020-04-14 12:47 ` Greg Kroah-Hartman 2020-04-14 12:47 ` Greg Kroah-Hartman 2020-04-14 12:47 ` Greg Kroah-Hartman 2020-04-18 12:42 ` Joerg Roedel 2020-04-18 12:42 ` Joerg Roedel 2020-04-18 12:42 ` Joerg Roedel 2020-04-19 8:00 ` Christoph Hellwig 2020-04-19 8:00 ` Christoph Hellwig 2020-04-19 8:00 ` Christoph Hellwig 2020-04-19 12:25 ` Joerg Roedel 2020-04-19 12:25 ` Joerg Roedel 2020-04-19 12:25 ` Joerg Roedel 2020-04-19 14:46 ` Alexey Kardashevskiy 2020-04-19 14:46 ` Alexey Kardashevskiy 2020-04-19 14:46 ` Alexey Kardashevskiy 2020-04-14 12:25 ` [PATCH 4/4] powerpc: use the generic dma_ops_bypass mode Christoph Hellwig 2020-04-14 12:25 ` Christoph Hellwig 2020-04-14 12:25 ` Christoph Hellwig
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.