* [PATCH V5 0/5] iommu/amd: Convert the AMD iommu driver to the dma-iommu api @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA Cc: Heiko Stuebner, virtualization-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, linux-tegra-u79uwXL29TY76Z2rM5mHXA, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc-u79uwXL29TY76Z2rM5mHXA, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, Andy Gross, Gerald Schaefer, linux-s390-u79uwXL29TY76Z2rM5mHXA, linux-arm-msm-u79uwXL29TY76Z2rM5mHXA, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, Matthias Brugger, linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, David Woodhouse, linux-kernel-u79uwXL29TY76Z2rM5mHXA, Tom Murphy, Kukjin Kim, Robin Murphy Convert the AMD iommu driver to the dma-iommu api. Remove the iova handling and reserve region code from the AMD iommu driver. Change-log: V5: -Rebase on top of linux-next V4: -Rebase on top of linux-next -Split the removing of the unnecessary locking in the amd iommu driver into a seperate patch -refactor the "iommu/dma-iommu: Handle deferred devices" patch and address comments v3: -rename dma_limit to dma_mask -exit handle_deferred_device early if (!is_kdump_kernel()) -remove pointless calls to handle_deferred_device v2: -Rebase on top of this series: http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/dma-iommu-ops.3 -Add a gfp_t parameter to the iommu_ops::map function. -Made use of the reserve region code inside the dma-iommu api Tom Murphy (5): iommu/amd: Remove unnecessary locking from AMD iommu driver iommu: Add gfp parameter to iommu_ops::map iommu/dma-iommu: Handle deferred devices iommu/dma-iommu: Use the dev->coherent_dma_mask iommu/amd: Convert AMD iommu driver to the dma-iommu api drivers/iommu/Kconfig | 1 + drivers/iommu/amd_iommu.c | 690 ++++---------------------------- drivers/iommu/amd_iommu_types.h | 1 - drivers/iommu/arm-smmu-v3.c | 2 +- drivers/iommu/arm-smmu.c | 2 +- drivers/iommu/dma-iommu.c | 43 +- drivers/iommu/exynos-iommu.c | 2 +- drivers/iommu/intel-iommu.c | 2 +- drivers/iommu/iommu.c | 43 +- drivers/iommu/ipmmu-vmsa.c | 2 +- drivers/iommu/msm_iommu.c | 2 +- drivers/iommu/mtk_iommu.c | 2 +- drivers/iommu/mtk_iommu_v1.c | 2 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 2 +- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 2 +- drivers/iommu/tegra-gart.c | 2 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/iommu/virtio-iommu.c | 2 +- include/linux/iommu.h | 21 +- 21 files changed, 178 insertions(+), 651 deletions(-) -- 2.20.1 ^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V5 0/5] iommu/amd: Convert the AMD iommu driver to the dma-iommu api @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Rob Clark, Kukjin Kim, Robin Murphy Convert the AMD iommu driver to the dma-iommu api. Remove the iova handling and reserve region code from the AMD iommu driver. Change-log: V5: -Rebase on top of linux-next V4: -Rebase on top of linux-next -Split the removing of the unnecessary locking in the amd iommu driver into a seperate patch -refactor the "iommu/dma-iommu: Handle deferred devices" patch and address comments v3: -rename dma_limit to dma_mask -exit handle_deferred_device early if (!is_kdump_kernel()) -remove pointless calls to handle_deferred_device v2: -Rebase on top of this series: http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/dma-iommu-ops.3 -Add a gfp_t parameter to the iommu_ops::map function. -Made use of the reserve region code inside the dma-iommu api Tom Murphy (5): iommu/amd: Remove unnecessary locking from AMD iommu driver iommu: Add gfp parameter to iommu_ops::map iommu/dma-iommu: Handle deferred devices iommu/dma-iommu: Use the dev->coherent_dma_mask iommu/amd: Convert AMD iommu driver to the dma-iommu api drivers/iommu/Kconfig | 1 + drivers/iommu/amd_iommu.c | 690 ++++---------------------------- drivers/iommu/amd_iommu_types.h | 1 - drivers/iommu/arm-smmu-v3.c | 2 +- drivers/iommu/arm-smmu.c | 2 +- drivers/iommu/dma-iommu.c | 43 +- drivers/iommu/exynos-iommu.c | 2 +- drivers/iommu/intel-iommu.c | 2 +- drivers/iommu/iommu.c | 43 +- drivers/iommu/ipmmu-vmsa.c | 2 +- drivers/iommu/msm_iommu.c | 2 +- drivers/iommu/mtk_iommu.c | 2 +- drivers/iommu/mtk_iommu_v1.c | 2 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 2 +- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 2 +- drivers/iommu/tegra-gart.c | 2 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/iommu/virtio-iommu.c | 2 +- include/linux/iommu.h | 21 +- 21 files changed, 178 insertions(+), 651 deletions(-) -- 2.20.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V5 0/5] iommu/amd: Convert the AMD iommu driver to the dma-iommu api @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Kukjin Kim, Robin Murphy Convert the AMD iommu driver to the dma-iommu api. Remove the iova handling and reserve region code from the AMD iommu driver. Change-log: V5: -Rebase on top of linux-next V4: -Rebase on top of linux-next -Split the removing of the unnecessary locking in the amd iommu driver into a seperate patch -refactor the "iommu/dma-iommu: Handle deferred devices" patch and address comments v3: -rename dma_limit to dma_mask -exit handle_deferred_device early if (!is_kdump_kernel()) -remove pointless calls to handle_deferred_device v2: -Rebase on top of this series: http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/dma-iommu-ops.3 -Add a gfp_t parameter to the iommu_ops::map function. -Made use of the reserve region code inside the dma-iommu api Tom Murphy (5): iommu/amd: Remove unnecessary locking from AMD iommu driver iommu: Add gfp parameter to iommu_ops::map iommu/dma-iommu: Handle deferred devices iommu/dma-iommu: Use the dev->coherent_dma_mask iommu/amd: Convert AMD iommu driver to the dma-iommu api drivers/iommu/Kconfig | 1 + drivers/iommu/amd_iommu.c | 690 ++++---------------------------- drivers/iommu/amd_iommu_types.h | 1 - drivers/iommu/arm-smmu-v3.c | 2 +- drivers/iommu/arm-smmu.c | 2 +- drivers/iommu/dma-iommu.c | 43 +- drivers/iommu/exynos-iommu.c | 2 +- drivers/iommu/intel-iommu.c | 2 +- drivers/iommu/iommu.c | 43 +- drivers/iommu/ipmmu-vmsa.c | 2 +- drivers/iommu/msm_iommu.c | 2 +- drivers/iommu/mtk_iommu.c | 2 +- drivers/iommu/mtk_iommu_v1.c | 2 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 2 +- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 2 +- drivers/iommu/tegra-gart.c | 2 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/iommu/virtio-iommu.c | 2 +- include/linux/iommu.h | 21 +- 21 files changed, 178 insertions(+), 651 deletions(-) -- 2.20.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V5 0/5] iommu/amd: Convert the AMD iommu driver to the dma-iommu api @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Tom Murphy, Joerg Roedel, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390, linux-tegra, virtualization Convert the AMD iommu driver to the dma-iommu api. Remove the iova handling and reserve region code from the AMD iommu driver. Change-log: V5: -Rebase on top of linux-next V4: -Rebase on top of linux-next -Split the removing of the unnecessary locking in the amd iommu driver into a seperate patch -refactor the "iommu/dma-iommu: Handle deferred devices" patch and address comments v3: -rename dma_limit to dma_mask -exit handle_deferred_device early if (!is_kdump_kernel()) -remove pointless calls to handle_deferred_device v2: -Rebase on top of this series: http://git.infradead.org/users/hch/misc.git/shortlog/refs/heads/dma-iommu-ops.3 -Add a gfp_t parameter to the iommu_ops::map function. -Made use of the reserve region code inside the dma-iommu api Tom Murphy (5): iommu/amd: Remove unnecessary locking from AMD iommu driver iommu: Add gfp parameter to iommu_ops::map iommu/dma-iommu: Handle deferred devices iommu/dma-iommu: Use the dev->coherent_dma_mask iommu/amd: Convert AMD iommu driver to the dma-iommu api drivers/iommu/Kconfig | 1 + drivers/iommu/amd_iommu.c | 690 ++++---------------------------- drivers/iommu/amd_iommu_types.h | 1 - drivers/iommu/arm-smmu-v3.c | 2 +- drivers/iommu/arm-smmu.c | 2 +- drivers/iommu/dma-iommu.c | 43 +- drivers/iommu/exynos-iommu.c | 2 +- drivers/iommu/intel-iommu.c | 2 +- drivers/iommu/iommu.c | 43 +- drivers/iommu/ipmmu-vmsa.c | 2 +- drivers/iommu/msm_iommu.c | 2 +- drivers/iommu/mtk_iommu.c | 2 +- drivers/iommu/mtk_iommu_v1.c | 2 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 2 +- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 2 +- drivers/iommu/tegra-gart.c | 2 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/iommu/virtio-iommu.c | 2 +- include/linux/iommu.h | 21 +- 21 files changed, 178 insertions(+), 651 deletions(-) -- 2.20.1 ^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver 2019-08-15 11:09 ` Tom Murphy ` (2 preceding siblings ...) (?) @ 2019-08-15 11:09 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Rob Clark, Kukjin Kim, Robin We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. iommu_map doesn’t lock while mapping and so no two calls should touch the same iova range. The AMD driver already handles the page table page allocations without locks so we can safely remove the locks. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/amd_iommu.c | 10 +--------- drivers/iommu/amd_iommu_types.h | 1 - 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 008da21a2592..1948be7ac8f8 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -2858,7 +2858,6 @@ static void protection_domain_free(struct protection_domain *domain) static int protection_domain_init(struct protection_domain *domain) { spin_lock_init(&domain->lock); - mutex_init(&domain->api_lock); domain->id = domain_id_alloc(); if (!domain->id) return -ENOMEM; @@ -3045,9 +3044,7 @@ static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, if (iommu_prot & IOMMU_WRITE) prot |= IOMMU_PROT_IW; - mutex_lock(&domain->api_lock); ret = iommu_map_page(domain, iova, paddr, page_size, prot, GFP_KERNEL); - mutex_unlock(&domain->api_lock); domain_flush_np_cache(domain, iova, page_size); @@ -3058,16 +3055,11 @@ static size_t amd_iommu_unmap(struct iommu_domain *dom, unsigned long iova, size_t page_size) { struct protection_domain *domain = to_pdomain(dom); - size_t unmap_size; if (domain->mode == PAGE_MODE_NONE) return 0; - mutex_lock(&domain->api_lock); - unmap_size = iommu_unmap_page(domain, iova, page_size); - mutex_unlock(&domain->api_lock); - - return unmap_size; + return iommu_unmap_page(domain, iova, page_size); } static phys_addr_t amd_iommu_iova_to_phys(struct iommu_domain *dom, diff --git a/drivers/iommu/amd_iommu_types.h b/drivers/iommu/amd_iommu_types.h index 9ac229e92b07..b764e1a73dcf 100644 --- a/drivers/iommu/amd_iommu_types.h +++ b/drivers/iommu/amd_iommu_types.h @@ -468,7 +468,6 @@ struct protection_domain { struct iommu_domain domain; /* generic domain handle used by iommu core code */ spinlock_t lock; /* mostly used to lock the page table*/ - struct mutex api_lock; /* protect page tables in the iommu-api path */ u16 id; /* the domain id written to the device table */ int mode; /* paging mode (0-6 levels) */ u64 *pt_root; /* page table root pointer */ -- 2.20.1 _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-15 11:09 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Tom Murphy, Joerg Roedel, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. iommu_map doesn’t lock while mapping and so no two calls should touch the same iova range. The AMD driver already handles the page table page allocations without locks so we can safely remove the locks. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/amd_iommu.c | 10 +--------- drivers/iommu/amd_iommu_types.h | 1 - 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 008da21a2592..1948be7ac8f8 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -2858,7 +2858,6 @@ static void protection_domain_free(struct protection_domain *domain) static int protection_domain_init(struct protection_domain *domain) { spin_lock_init(&domain->lock); - mutex_init(&domain->api_lock); domain->id = domain_id_alloc(); if (!domain->id) return -ENOMEM; @@ -3045,9 +3044,7 @@ static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, if (iommu_prot & IOMMU_WRITE) prot |= IOMMU_PROT_IW; - mutex_lock(&domain->api_lock); ret = iommu_map_page(domain, iova, paddr, page_size, prot, GFP_KERNEL); - mutex_unlock(&domain->api_lock); domain_flush_np_cache(domain, iova, page_size); @@ -3058,16 +3055,11 @@ static size_t amd_iommu_unmap(struct iommu_domain *dom, unsigned long iova, size_t page_size) { struct protection_domain *domain = to_pdomain(dom); - size_t unmap_size; if (domain->mode == PAGE_MODE_NONE) return 0; - mutex_lock(&domain->api_lock); - unmap_size = iommu_unmap_page(domain, iova, page_size); - mutex_unlock(&domain->api_lock); - - return unmap_size; + return iommu_unmap_page(domain, iova, page_size); } static phys_addr_t amd_iommu_iova_to_phys(struct iommu_domain *dom, diff --git a/drivers/iommu/amd_iommu_types.h b/drivers/iommu/amd_iommu_types.h index 9ac229e92b07..b764e1a73dcf 100644 --- a/drivers/iommu/amd_iommu_types.h +++ b/drivers/iommu/amd_iommu_types.h @@ -468,7 +468,6 @@ struct protection_domain { struct iommu_domain domain; /* generic domain handle used by iommu core code */ spinlock_t lock; /* mostly used to lock the page table*/ - struct mutex api_lock; /* protect page tables in the iommu-api path */ u16 id; /* the domain id written to the device table */ int mode; /* paging mode (0-6 levels) */ u64 *pt_root; /* page table root pointer */ -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Rob Clark, Kukjin Kim, Robin Murphy We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. iommu_map doesn’t lock while mapping and so no two calls should touch the same iova range. The AMD driver already handles the page table page allocations without locks so we can safely remove the locks. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/amd_iommu.c | 10 +--------- drivers/iommu/amd_iommu_types.h | 1 - 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 008da21a2592..1948be7ac8f8 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -2858,7 +2858,6 @@ static void protection_domain_free(struct protection_domain *domain) static int protection_domain_init(struct protection_domain *domain) { spin_lock_init(&domain->lock); - mutex_init(&domain->api_lock); domain->id = domain_id_alloc(); if (!domain->id) return -ENOMEM; @@ -3045,9 +3044,7 @@ static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, if (iommu_prot & IOMMU_WRITE) prot |= IOMMU_PROT_IW; - mutex_lock(&domain->api_lock); ret = iommu_map_page(domain, iova, paddr, page_size, prot, GFP_KERNEL); - mutex_unlock(&domain->api_lock); domain_flush_np_cache(domain, iova, page_size); @@ -3058,16 +3055,11 @@ static size_t amd_iommu_unmap(struct iommu_domain *dom, unsigned long iova, size_t page_size) { struct protection_domain *domain = to_pdomain(dom); - size_t unmap_size; if (domain->mode == PAGE_MODE_NONE) return 0; - mutex_lock(&domain->api_lock); - unmap_size = iommu_unmap_page(domain, iova, page_size); - mutex_unlock(&domain->api_lock); - - return unmap_size; + return iommu_unmap_page(domain, iova, page_size); } static phys_addr_t amd_iommu_iova_to_phys(struct iommu_domain *dom, diff --git a/drivers/iommu/amd_iommu_types.h b/drivers/iommu/amd_iommu_types.h index 9ac229e92b07..b764e1a73dcf 100644 --- a/drivers/iommu/amd_iommu_types.h +++ b/drivers/iommu/amd_iommu_types.h @@ -468,7 +468,6 @@ struct protection_domain { struct iommu_domain domain; /* generic domain handle used by iommu core code */ spinlock_t lock; /* mostly used to lock the page table*/ - struct mutex api_lock; /* protect page tables in the iommu-api path */ u16 id; /* the domain id written to the device table */ int mode; /* paging mode (0-6 levels) */ u64 *pt_root; /* page table root pointer */ -- 2.20.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Kukjin Kim, Robin Murphy We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. iommu_map doesn’t lock while mapping and so no two calls should touch the same iova range. The AMD driver already handles the page table page allocations without locks so we can safely remove the locks. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/amd_iommu.c | 10 +--------- drivers/iommu/amd_iommu_types.h | 1 - 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 008da21a2592..1948be7ac8f8 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -2858,7 +2858,6 @@ static void protection_domain_free(struct protection_domain *domain) static int protection_domain_init(struct protection_domain *domain) { spin_lock_init(&domain->lock); - mutex_init(&domain->api_lock); domain->id = domain_id_alloc(); if (!domain->id) return -ENOMEM; @@ -3045,9 +3044,7 @@ static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, if (iommu_prot & IOMMU_WRITE) prot |= IOMMU_PROT_IW; - mutex_lock(&domain->api_lock); ret = iommu_map_page(domain, iova, paddr, page_size, prot, GFP_KERNEL); - mutex_unlock(&domain->api_lock); domain_flush_np_cache(domain, iova, page_size); @@ -3058,16 +3055,11 @@ static size_t amd_iommu_unmap(struct iommu_domain *dom, unsigned long iova, size_t page_size) { struct protection_domain *domain = to_pdomain(dom); - size_t unmap_size; if (domain->mode == PAGE_MODE_NONE) return 0; - mutex_lock(&domain->api_lock); - unmap_size = iommu_unmap_page(domain, iova, page_size); - mutex_unlock(&domain->api_lock); - - return unmap_size; + return iommu_unmap_page(domain, iova, page_size); } static phys_addr_t amd_iommu_iova_to_phys(struct iommu_domain *dom, diff --git a/drivers/iommu/amd_iommu_types.h b/drivers/iommu/amd_iommu_types.h index 9ac229e92b07..b764e1a73dcf 100644 --- a/drivers/iommu/amd_iommu_types.h +++ b/drivers/iommu/amd_iommu_types.h @@ -468,7 +468,6 @@ struct protection_domain { struct iommu_domain domain; /* generic domain handle used by iommu core code */ spinlock_t lock; /* mostly used to lock the page table*/ - struct mutex api_lock; /* protect page tables in the iommu-api path */ u16 id; /* the domain id written to the device table */ int mode; /* paging mode (0-6 levels) */ u64 *pt_root; /* page table root pointer */ -- 2.20.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Tom Murphy, Joerg Roedel, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390, linux-tegra, virtualization We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. iommu_map doesn’t lock while mapping and so no two calls should touch the same iova range. The AMD driver already handles the page table page allocations without locks so we can safely remove the locks. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/amd_iommu.c | 10 +--------- drivers/iommu/amd_iommu_types.h | 1 - 2 files changed, 1 insertion(+), 10 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 008da21a2592..1948be7ac8f8 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -2858,7 +2858,6 @@ static void protection_domain_free(struct protection_domain *domain) static int protection_domain_init(struct protection_domain *domain) { spin_lock_init(&domain->lock); - mutex_init(&domain->api_lock); domain->id = domain_id_alloc(); if (!domain->id) return -ENOMEM; @@ -3045,9 +3044,7 @@ static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, if (iommu_prot & IOMMU_WRITE) prot |= IOMMU_PROT_IW; - mutex_lock(&domain->api_lock); ret = iommu_map_page(domain, iova, paddr, page_size, prot, GFP_KERNEL); - mutex_unlock(&domain->api_lock); domain_flush_np_cache(domain, iova, page_size); @@ -3058,16 +3055,11 @@ static size_t amd_iommu_unmap(struct iommu_domain *dom, unsigned long iova, size_t page_size) { struct protection_domain *domain = to_pdomain(dom); - size_t unmap_size; if (domain->mode == PAGE_MODE_NONE) return 0; - mutex_lock(&domain->api_lock); - unmap_size = iommu_unmap_page(domain, iova, page_size); - mutex_unlock(&domain->api_lock); - - return unmap_size; + return iommu_unmap_page(domain, iova, page_size); } static phys_addr_t amd_iommu_iova_to_phys(struct iommu_domain *dom, diff --git a/drivers/iommu/amd_iommu_types.h b/drivers/iommu/amd_iommu_types.h index 9ac229e92b07..b764e1a73dcf 100644 --- a/drivers/iommu/amd_iommu_types.h +++ b/drivers/iommu/amd_iommu_types.h @@ -468,7 +468,6 @@ struct protection_domain { struct iommu_domain domain; /* generic domain handle used by iommu core code */ spinlock_t lock; /* mostly used to lock the page table*/ - struct mutex api_lock; /* protect page tables in the iommu-api path */ u16 id; /* the domain id written to the device table */ int mode; /* paging mode (0-6 levels) */ u64 *pt_root; /* page table root pointer */ -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver 2019-08-15 11:09 ` Tom Murphy ` (2 preceding siblings ...) (?) @ 2019-08-20 9:41 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:41 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy On Thu, Aug 15, 2019 at 12:09:39PM +0100, Tom Murphy wrote: > We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. > iommu_map doesn’t lock while mapping and so no two calls should touch > the same iova range. The AMD driver already handles the page table page > allocations without locks so we can safely remove the locks. I've been looking over the code and trying to understand how the synchronization works. I gues we the cmpxchg64 in free_clear_pte is the important point here? I have to admit I don't fully understand the concurrency issues here, but neither do I understand what the mutex you removed might have helped to start with. _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-20 9:41 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:41 UTC (permalink / raw) To: Tom Murphy Cc: iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Kukjin Kim On Thu, Aug 15, 2019 at 12:09:39PM +0100, Tom Murphy wrote: > We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. > iommu_map doesn’t lock while mapping and so no two calls should touch > the same iova range. The AMD driver already handles the page table page > allocations without locks so we can safely remove the locks. I've been looking over the code and trying to understand how the synchronization works. I gues we the cmpxchg64 in free_clear_pte is the important point here? I have to admit I don't fully understand the concurrency issues here, but neither do I understand what the mutex you removed might have helped to start with. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-20 9:41 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:41 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy On Thu, Aug 15, 2019 at 12:09:39PM +0100, Tom Murphy wrote: > We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. > iommu_map doesn’t lock while mapping and so no two calls should touch > the same iova range. The AMD driver already handles the page table page > allocations without locks so we can safely remove the locks. I've been looking over the code and trying to understand how the synchronization works. I gues we the cmpxchg64 in free_clear_pte is the important point here? I have to admit I don't fully understand the concurrency issues here, but neither do I understand what the mutex you removed might have helped to start with. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-20 9:41 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:41 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy On Thu, Aug 15, 2019 at 12:09:39PM +0100, Tom Murphy wrote: > We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. > iommu_map doesn’t lock while mapping and so no two calls should touch > the same iova range. The AMD driver already handles the page table page > allocations without locks so we can safely remove the locks. I've been looking over the code and trying to understand how the synchronization works. I gues we the cmpxchg64 in free_clear_pte is the important point here? I have to admit I don't fully understand the concurrency issues here, but neither do I understand what the mutex you removed might have helped to start with. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-20 9:41 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:41 UTC (permalink / raw) To: Tom Murphy Cc: iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Kukjin Kim, Robin Murphy On Thu, Aug 15, 2019 at 12:09:39PM +0100, Tom Murphy wrote: > We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. > iommu_map doesn’t lock while mapping and so no two calls should touch > the same iova range. The AMD driver already handles the page table page > allocations without locks so we can safely remove the locks. I've been looking over the code and trying to understand how the synchronization works. I gues we the cmpxchg64 in free_clear_pte is the important point here? I have to admit I don't fully understand the concurrency issues here, but neither do I understand what the mutex you removed might have helped to start with. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver 2019-08-20 9:41 ` Christoph Hellwig ` (2 preceding siblings ...) (?) @ 2019-08-24 7:56 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-24 7:56 UTC (permalink / raw) To: Christoph Hellwig Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, Linux Kernel Mailing List, iommu, Robin Murphy >I have to admit I don't fully understand the concurrency issues here, but neither do I understand what the mutex you removed might have helped to start with. Each range in the page tables is protected by the IO virtual address allocator. The iommu driver allocates an IOVA range using locks before it writes to a page table range. The IOVA allocator acts like a lock on a specific range of the page tables. So we can handle most of the concurrency issues in the IOVA allocator and avoid locking while writing to a range in the page tables. However because we have multiple levels of pages we might have to allocate a middle page (a PMD) which covers more than the IOVA range we have allocated. To solve this we could use locks: //pseudo code lock_page_table() if (we need to allocate middle pages) { //allocate the page //set the PMD value } unlock_page_table() but we can actually avoid having any locking by doing the following: //pseudo code if (we need to allocate middle pages) { //allocate the page //cmpxchg64 to set the PMD if it wasn't already set since we last checked if (the PMD was set while since we last checked) //free the page we just allocated } In this case we can end up doing a pointless page allocate and free but it's worth it to avoid using locks You can see this in the intel iommu code here: https://github.com/torvalds/linux/blob/9140d8bdd4c5a04abe181bb300378355d56990a4/drivers/iommu/intel-iommu.c#L904 >what the mutex you removed might have helped to start with. The mutex I removed is arguably completely useless. In the dma ops path we handle the IOVA allocations in the driver so we can be sure a certain range is protected by the IOVA allocator. Because the iommu ops path doesn't handle the IOVA allocations it seems reasonable to lock the page tables to avoid two writers writing to the same range at the same time. Without the lock it's complete chaos and all writers can be writing to the same range at the same time resulting in complete garbage. BUT the locking doesn't actually make any real difference. Even with locking we still have a race condition if two writers want to write to the same range at the same time, the race is just whoever gets the lock first, we still can't be sure what the result will be. So the result is still garbage, just slightly more usable garbage because at least the range is correct for one writer. It just makes no sense to ever have two writers writing to the same range and adding a lock doesn't fix that. Already the Intel iommu ops path doesn't use locks for it's page table so this isn't a new idea I'm just doing the same for the AMD iommu driver Does all that make sense? On Tue, 20 Aug 2019 at 10:41, Christoph Hellwig <hch@infradead.org> wrote: > > On Thu, Aug 15, 2019 at 12:09:39PM +0100, Tom Murphy wrote: > > We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. > > iommu_map doesn’t lock while mapping and so no two calls should touch > > the same iova range. The AMD driver already handles the page table page > > allocations without locks so we can safely remove the locks. > > I've been looking over the code and trying to understand how the > synchronization works. I gues we the cmpxchg64 in free_clear_pte > is the important point here? I have to admit I don't fully understand > the concurrency issues here, but neither do I understand what the > mutex you removed might have helped to start with. _______________________________________________ Virtualization mailing list Virtualization@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/virtualization ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver 2019-08-20 9:41 ` Christoph Hellwig (?) (?) @ 2019-08-24 7:56 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-24 7:56 UTC (permalink / raw) To: Christoph Hellwig Cc: iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, Linux Kernel Mailing List >I have to admit I don't fully understand the concurrency issues here, but neither do I understand what the mutex you removed might have helped to start with. Each range in the page tables is protected by the IO virtual address allocator. The iommu driver allocates an IOVA range using locks before it writes to a page table range. The IOVA allocator acts like a lock on a specific range of the page tables. So we can handle most of the concurrency issues in the IOVA allocator and avoid locking while writing to a range in the page tables. However because we have multiple levels of pages we might have to allocate a middle page (a PMD) which covers more than the IOVA range we have allocated. To solve this we could use locks: //pseudo code lock_page_table() if (we need to allocate middle pages) { //allocate the page //set the PMD value } unlock_page_table() but we can actually avoid having any locking by doing the following: //pseudo code if (we need to allocate middle pages) { //allocate the page //cmpxchg64 to set the PMD if it wasn't already set since we last checked if (the PMD was set while since we last checked) //free the page we just allocated } In this case we can end up doing a pointless page allocate and free but it's worth it to avoid using locks You can see this in the intel iommu code here: https://github.com/torvalds/linux/blob/9140d8bdd4c5a04abe181bb300378355d56990a4/drivers/iommu/intel-iommu.c#L904 >what the mutex you removed might have helped to start with. The mutex I removed is arguably completely useless. In the dma ops path we handle the IOVA allocations in the driver so we can be sure a certain range is protected by the IOVA allocator. Because the iommu ops path doesn't handle the IOVA allocations it seems reasonable to lock the page tables to avoid two writers writing to the same range at the same time. Without the lock it's complete chaos and all writers can be writing to the same range at the same time resulting in complete garbage. BUT the locking doesn't actually make any real difference. Even with locking we still have a race condition if two writers want to write to the same range at the same time, the race is just whoever gets the lock first, we still can't be sure what the result will be. So the result is still garbage, just slightly more usable garbage because at least the range is correct for one writer. It just makes no sense to ever have two writers writing to the same range and adding a lock doesn't fix that. Already the Intel iommu ops path doesn't use locks for it's page table so this isn't a new idea I'm just doing the same for the AMD iommu driver Does all that make sense? On Tue, 20 Aug 2019 at 10:41, Christoph Hellwig <hch@infradead.org> wrote: > > On Thu, Aug 15, 2019 at 12:09:39PM +0100, Tom Murphy wrote: > > We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. > > iommu_map doesn’t lock while mapping and so no two calls should touch > > the same iova range. The AMD driver already handles the page table page > > allocations without locks so we can safely remove the locks. > > I've been looking over the code and trying to understand how the > synchronization works. I gues we the cmpxchg64 in free_clear_pte > is the important point here? I have to admit I don't fully understand > the concurrency issues here, but neither do I understand what the > mutex you removed might have helped to start with. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-24 7:56 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-24 7:56 UTC (permalink / raw) To: Christoph Hellwig Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, Linux Kernel Mailing List, iommu, Robin Murphy >I have to admit I don't fully understand the concurrency issues here, but neither do I understand what the mutex you removed might have helped to start with. Each range in the page tables is protected by the IO virtual address allocator. The iommu driver allocates an IOVA range using locks before it writes to a page table range. The IOVA allocator acts like a lock on a specific range of the page tables. So we can handle most of the concurrency issues in the IOVA allocator and avoid locking while writing to a range in the page tables. However because we have multiple levels of pages we might have to allocate a middle page (a PMD) which covers more than the IOVA range we have allocated. To solve this we could use locks: //pseudo code lock_page_table() if (we need to allocate middle pages) { //allocate the page //set the PMD value } unlock_page_table() but we can actually avoid having any locking by doing the following: //pseudo code if (we need to allocate middle pages) { //allocate the page //cmpxchg64 to set the PMD if it wasn't already set since we last checked if (the PMD was set while since we last checked) //free the page we just allocated } In this case we can end up doing a pointless page allocate and free but it's worth it to avoid using locks You can see this in the intel iommu code here: https://github.com/torvalds/linux/blob/9140d8bdd4c5a04abe181bb300378355d56990a4/drivers/iommu/intel-iommu.c#L904 >what the mutex you removed might have helped to start with. The mutex I removed is arguably completely useless. In the dma ops path we handle the IOVA allocations in the driver so we can be sure a certain range is protected by the IOVA allocator. Because the iommu ops path doesn't handle the IOVA allocations it seems reasonable to lock the page tables to avoid two writers writing to the same range at the same time. Without the lock it's complete chaos and all writers can be writing to the same range at the same time resulting in complete garbage. BUT the locking doesn't actually make any real difference. Even with locking we still have a race condition if two writers want to write to the same range at the same time, the race is just whoever gets the lock first, we still can't be sure what the result will be. So the result is still garbage, just slightly more usable garbage because at least the range is correct for one writer. It just makes no sense to ever have two writers writing to the same range and adding a lock doesn't fix that. Already the Intel iommu ops path doesn't use locks for it's page table so this isn't a new idea I'm just doing the same for the AMD iommu driver Does all that make sense? On Tue, 20 Aug 2019 at 10:41, Christoph Hellwig <hch@infradead.org> wrote: > > On Thu, Aug 15, 2019 at 12:09:39PM +0100, Tom Murphy wrote: > > We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. > > iommu_map doesn’t lock while mapping and so no two calls should touch > > the same iova range. The AMD driver already handles the page table page > > allocations without locks so we can safely remove the locks. > > I've been looking over the code and trying to understand how the > synchronization works. I gues we the cmpxchg64 in free_clear_pte > is the important point here? I have to admit I don't fully understand > the concurrency issues here, but neither do I understand what the > mutex you removed might have helped to start with. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-24 7:56 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-24 7:56 UTC (permalink / raw) To: Christoph Hellwig Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, Linux Kernel Mailing List, iommu, Robin Murphy >I have to admit I don't fully understand the concurrency issues here, but neither do I understand what the mutex you removed might have helped to start with. Each range in the page tables is protected by the IO virtual address allocator. The iommu driver allocates an IOVA range using locks before it writes to a page table range. The IOVA allocator acts like a lock on a specific range of the page tables. So we can handle most of the concurrency issues in the IOVA allocator and avoid locking while writing to a range in the page tables. However because we have multiple levels of pages we might have to allocate a middle page (a PMD) which covers more than the IOVA range we have allocated. To solve this we could use locks: //pseudo code lock_page_table() if (we need to allocate middle pages) { //allocate the page //set the PMD value } unlock_page_table() but we can actually avoid having any locking by doing the following: //pseudo code if (we need to allocate middle pages) { //allocate the page //cmpxchg64 to set the PMD if it wasn't already set since we last checked if (the PMD was set while since we last checked) //free the page we just allocated } In this case we can end up doing a pointless page allocate and free but it's worth it to avoid using locks You can see this in the intel iommu code here: https://github.com/torvalds/linux/blob/9140d8bdd4c5a04abe181bb300378355d56990a4/drivers/iommu/intel-iommu.c#L904 >what the mutex you removed might have helped to start with. The mutex I removed is arguably completely useless. In the dma ops path we handle the IOVA allocations in the driver so we can be sure a certain range is protected by the IOVA allocator. Because the iommu ops path doesn't handle the IOVA allocations it seems reasonable to lock the page tables to avoid two writers writing to the same range at the same time. Without the lock it's complete chaos and all writers can be writing to the same range at the same time resulting in complete garbage. BUT the locking doesn't actually make any real difference. Even with locking we still have a race condition if two writers want to write to the same range at the same time, the race is just whoever gets the lock first, we still can't be sure what the result will be. So the result is still garbage, just slightly more usable garbage because at least the range is correct for one writer. It just makes no sense to ever have two writers writing to the same range and adding a lock doesn't fix that. Already the Intel iommu ops path doesn't use locks for it's page table so this isn't a new idea I'm just doing the same for the AMD iommu driver Does all that make sense? On Tue, 20 Aug 2019 at 10:41, Christoph Hellwig <hch@infradead.org> wrote: > > On Thu, Aug 15, 2019 at 12:09:39PM +0100, Tom Murphy wrote: > > We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. > > iommu_map doesn’t lock while mapping and so no two calls should touch > > the same iova range. The AMD driver already handles the page table page > > allocations without locks so we can safely remove the locks. > > I've been looking over the code and trying to understand how the > synchronization works. I gues we the cmpxchg64 in free_clear_pte > is the important point here? I have to admit I don't fully understand > the concurrency issues here, but neither do I understand what the > mutex you removed might have helped to start with. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-24 7:56 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-24 7:56 UTC (permalink / raw) To: Christoph Hellwig Cc: iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, Linux Kernel Mailing List, Kukjin Kim, Robin Murphy >I have to admit I don't fully understand the concurrency issues here, but neither do I understand what the mutex you removed might have helped to start with. Each range in the page tables is protected by the IO virtual address allocator. The iommu driver allocates an IOVA range using locks before it writes to a page table range. The IOVA allocator acts like a lock on a specific range of the page tables. So we can handle most of the concurrency issues in the IOVA allocator and avoid locking while writing to a range in the page tables. However because we have multiple levels of pages we might have to allocate a middle page (a PMD) which covers more than the IOVA range we have allocated. To solve this we could use locks: //pseudo code lock_page_table() if (we need to allocate middle pages) { //allocate the page //set the PMD value } unlock_page_table() but we can actually avoid having any locking by doing the following: //pseudo code if (we need to allocate middle pages) { //allocate the page //cmpxchg64 to set the PMD if it wasn't already set since we last checked if (the PMD was set while since we last checked) //free the page we just allocated } In this case we can end up doing a pointless page allocate and free but it's worth it to avoid using locks You can see this in the intel iommu code here: https://github.com/torvalds/linux/blob/9140d8bdd4c5a04abe181bb300378355d56990a4/drivers/iommu/intel-iommu.c#L904 >what the mutex you removed might have helped to start with. The mutex I removed is arguably completely useless. In the dma ops path we handle the IOVA allocations in the driver so we can be sure a certain range is protected by the IOVA allocator. Because the iommu ops path doesn't handle the IOVA allocations it seems reasonable to lock the page tables to avoid two writers writing to the same range at the same time. Without the lock it's complete chaos and all writers can be writing to the same range at the same time resulting in complete garbage. BUT the locking doesn't actually make any real difference. Even with locking we still have a race condition if two writers want to write to the same range at the same time, the race is just whoever gets the lock first, we still can't be sure what the result will be. So the result is still garbage, just slightly more usable garbage because at least the range is correct for one writer. It just makes no sense to ever have two writers writing to the same range and adding a lock doesn't fix that. Already the Intel iommu ops path doesn't use locks for it's page table so this isn't a new idea I'm just doing the same for the AMD iommu driver Does all that make sense? On Tue, 20 Aug 2019 at 10:41, Christoph Hellwig <hch@infradead.org> wrote: > > On Thu, Aug 15, 2019 at 12:09:39PM +0100, Tom Murphy wrote: > > We can remove the mutex lock from amd_iommu_map and amd_iommu_unmap. > > iommu_map doesn’t lock while mapping and so no two calls should touch > > the same iova range. The AMD driver already handles the page table page > > allocations without locks so we can safely remove the locks. > > I've been looking over the code and trying to understand how the > synchronization works. I gues we the cmpxchg64 in free_clear_pte > is the important point here? I have to admit I don't fully understand > the concurrency issues here, but neither do I understand what the > mutex you removed might have helped to start with. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver 2019-08-24 7:56 ` Tom Murphy ` (2 preceding siblings ...) (?) @ 2019-08-24 22:41 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-24 22:41 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, Christoph Hellwig, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, Robin Murphy, Linux Kernel Mailing List, iommu, David Woodhouse Thank for the explanation Tom. It might make sense to add a condensed version of it to commit log for the next iteration. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver 2019-08-24 7:56 ` Tom Murphy (?) (?) @ 2019-08-24 22:41 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-24 22:41 UTC (permalink / raw) To: Tom Murphy Cc: Christoph Hellwig, iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, Linux Thank for the explanation Tom. It might make sense to add a condensed version of it to commit log for the next iteration. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-24 22:41 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-24 22:41 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, Christoph Hellwig, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, Robin Murphy, Linux Kernel Mailing List, iommu, David Woodhouse Thank for the explanation Tom. It might make sense to add a condensed version of it to commit log for the next iteration. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-24 22:41 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-24 22:41 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, Robin Murphy, Linux Kernel Mailing List, iommu, David Woodhouse Thank for the explanation Tom. It might make sense to add a condensed version of it to commit log for the next iteration. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver @ 2019-08-24 22:41 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-24 22:41 UTC (permalink / raw) To: Tom Murphy Cc: Christoph Hellwig, iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, Linux Kernel Mailing List, Kukjin Kim, Robin Murphy Thank for the explanation Tom. It might make sense to add a condensed version of it to commit log for the next iteration. ^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-15 11:09 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Tom Murphy, Joerg Roedel, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek Add a gfp_t parameter to the iommu_ops::map function. Remove the needless locking in the AMD iommu driver. The iommu_ops::map function (or the iommu_map function which calls it) was always supposed to be sleepable (according to Joerg's comment in this thread: https://lore.kernel.org/patchwork/patch/977520/ ) and so should probably have had a "might_sleep()" since it was written. However currently the dma-iommu api can call iommu_map in an atomic context, which it shouldn't do. This doesn't cause any problems because any iommu driver which uses the dma-iommu api uses gfp_atomic in it's iommu_ops::map function. But doing this wastes the memory allocators atomic pools. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/amd_iommu.c | 3 ++- drivers/iommu/arm-smmu-v3.c | 2 +- drivers/iommu/arm-smmu.c | 2 +- drivers/iommu/dma-iommu.c | 6 ++--- drivers/iommu/exynos-iommu.c | 2 +- drivers/iommu/intel-iommu.c | 2 +- drivers/iommu/iommu.c | 43 +++++++++++++++++++++++++++++----- drivers/iommu/ipmmu-vmsa.c | 2 +- drivers/iommu/msm_iommu.c | 2 +- drivers/iommu/mtk_iommu.c | 2 +- drivers/iommu/mtk_iommu_v1.c | 2 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 2 +- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 2 +- drivers/iommu/tegra-gart.c | 2 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/iommu/virtio-iommu.c | 2 +- include/linux/iommu.h | 21 ++++++++++++++++- 19 files changed, 77 insertions(+), 26 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 1948be7ac8f8..0e53f9bd2be7 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -3030,7 +3030,8 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, } static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, - phys_addr_t paddr, size_t page_size, int iommu_prot) + phys_addr_t paddr, size_t page_size, int iommu_prot, + gfp_t gfp) { struct protection_domain *domain = to_pdomain(dom); int prot = 0; diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index e7f49fd1a7ba..acc0eae7963f 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -1975,7 +1975,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c index aa06498f291d..05f42bdee494 100644 --- a/drivers/iommu/arm-smmu.c +++ b/drivers/iommu/arm-smmu.c @@ -1284,7 +1284,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index d991d40f797f..2712fbc68b28 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -469,7 +469,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, if (!iova) return DMA_MAPPING_ERROR; - if (iommu_map(domain, iova, phys - iova_off, size, prot)) { + if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) { iommu_dma_free_iova(cookie, iova, size); return DMA_MAPPING_ERROR; } @@ -613,7 +613,7 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, arch_dma_prep_coherent(sg_page(sg), sg->length); } - if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) + if (iommu_map_sg_atomic(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) < size) goto out_free_sg; @@ -873,7 +873,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, * We'll leave any physical concatenation to the IOMMU driver's * implementation - it knows better than we do. */ - if (iommu_map_sg(domain, iova, sg, nents, prot) < iova_len) + if (iommu_map_sg_atomic(domain, iova, sg, nents, prot) < iova_len) goto out_free_iova; return __finalise_sg(dev, sg, nents, iova); diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c index 1934c16a5abc..b7dd46884692 100644 --- a/drivers/iommu/exynos-iommu.c +++ b/drivers/iommu/exynos-iommu.c @@ -1073,7 +1073,7 @@ static int lv2set_page(sysmmu_pte_t *pent, phys_addr_t paddr, size_t size, */ static int exynos_iommu_map(struct iommu_domain *iommu_domain, unsigned long l_iova, phys_addr_t paddr, size_t size, - int prot) + int prot, gfp_t gfp) { struct exynos_iommu_domain *domain = to_exynos_domain(iommu_domain); sysmmu_pte_t *entry; diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 12d094d08c0a..0e7cb0cbbd46 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -5113,7 +5113,7 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain, static int intel_iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t hpa, - size_t size, int iommu_prot) + size_t size, int iommu_prot, gfp_t gfp) { struct dmar_domain *dmar_domain = to_dmar_domain(domain); u64 max_addr; diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 0c674d80c37f..133ec8e00a3a 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1799,8 +1799,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, return pgsize; } -int iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) +int __iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { const struct iommu_ops *ops = domain->ops; unsigned long orig_iova = iova; @@ -1837,8 +1837,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", iova, &paddr, pgsize); + ret = ops->map(domain, iova, paddr, pgsize, prot, gfp); - ret = ops->map(domain, iova, paddr, pgsize, prot); if (ret) break; @@ -1858,8 +1858,22 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, return ret; } + +int iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot) +{ + might_sleep(); + return __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL); +} EXPORT_SYMBOL_GPL(iommu_map); +int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot) +{ + return __iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC); +} +EXPORT_SYMBOL_GPL(iommu_map_atomic); + static size_t __iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size, bool sync) @@ -1934,8 +1948,9 @@ size_t iommu_unmap_fast(struct iommu_domain *domain, } EXPORT_SYMBOL_GPL(iommu_unmap_fast); -size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, - struct scatterlist *sg, unsigned int nents, int prot) +size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot, + gfp_t gfp) { size_t len = 0, mapped = 0; phys_addr_t start; @@ -1946,7 +1961,9 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, phys_addr_t s_phys = sg_phys(sg); if (len && s_phys != start + len) { - ret = iommu_map(domain, iova + mapped, start, len, prot); + ret = __iommu_map(domain, iova + mapped, start, + len, prot, gfp); + if (ret) goto out_err; @@ -1974,8 +1991,22 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, return 0; } + +size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + might_sleep(); + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_KERNEL); +} EXPORT_SYMBOL_GPL(iommu_map_sg); +size_t iommu_map_sg_atomic(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_ATOMIC); +} +EXPORT_SYMBOL_GPL(iommu_map_sg_atomic); + int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, phys_addr_t paddr, u64 size, int prot) { diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c index ad0098c0c87c..41572b40b844 100644 --- a/drivers/iommu/ipmmu-vmsa.c +++ b/drivers/iommu/ipmmu-vmsa.c @@ -722,7 +722,7 @@ static void ipmmu_detach_device(struct iommu_domain *io_domain, } static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain); diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c index 3df9266abe65..5fa915c5d541 100644 --- a/drivers/iommu/msm_iommu.c +++ b/drivers/iommu/msm_iommu.c @@ -495,7 +495,7 @@ static void msm_iommu_detach_dev(struct iommu_domain *domain, } static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t pa, size_t len, int prot) + phys_addr_t pa, size_t len, int prot, gfp_t gfp) { struct msm_priv *priv = to_msm_priv(domain); unsigned long flags; diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c index 82e4be4dfdaf..53371b9e9af5 100644 --- a/drivers/iommu/mtk_iommu.c +++ b/drivers/iommu/mtk_iommu.c @@ -356,7 +356,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, } static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct mtk_iommu_domain *dom = to_mtk_domain(domain); unsigned long flags; diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c index abeeac488372..453e704bee3d 100644 --- a/drivers/iommu/mtk_iommu_v1.c +++ b/drivers/iommu/mtk_iommu_v1.c @@ -295,7 +295,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, } static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct mtk_iommu_domain *dom = to_mtk_domain(domain); unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT; diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c index 99a9ff3e7f71..3c530768200e 100644 --- a/drivers/iommu/omap-iommu.c +++ b/drivers/iommu/omap-iommu.c @@ -1339,7 +1339,7 @@ static u32 iotlb_init_entry(struct iotlb_entry *e, u32 da, u32 pa, int pgsz) } static int omap_iommu_map(struct iommu_domain *domain, unsigned long da, - phys_addr_t pa, size_t bytes, int prot) + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) { struct omap_iommu_domain *omap_domain = to_omap_domain(domain); struct device *dev = omap_domain->dev; diff --git a/drivers/iommu/qcom_iommu.c b/drivers/iommu/qcom_iommu.c index fb45486c6d14..b373c0f18ad5 100644 --- a/drivers/iommu/qcom_iommu.c +++ b/drivers/iommu/qcom_iommu.c @@ -400,7 +400,7 @@ static void qcom_iommu_detach_dev(struct iommu_domain *domain, struct device *de } static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { int ret; unsigned long flags; diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c index dc26d74d79c2..b06d2986b52e 100644 --- a/drivers/iommu/rockchip-iommu.c +++ b/drivers/iommu/rockchip-iommu.c @@ -757,7 +757,7 @@ static int rk_iommu_map_iova(struct rk_iommu_domain *rk_domain, u32 *pte_addr, } static int rk_iommu_map(struct iommu_domain *domain, unsigned long _iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct rk_iommu_domain *rk_domain = to_rk_domain(domain); unsigned long flags; diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c index 22d4db302c1c..efa6aa68521d 100644 --- a/drivers/iommu/s390-iommu.c +++ b/drivers/iommu/s390-iommu.c @@ -265,7 +265,7 @@ static int s390_iommu_update_trans(struct s390_domain *s390_domain, } static int s390_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct s390_domain *s390_domain = to_s390_domain(domain); int flags = ZPCI_PTE_VALID, rc = 0; diff --git a/drivers/iommu/tegra-gart.c b/drivers/iommu/tegra-gart.c index 6d40bc1b38bf..43c725243e8a 100644 --- a/drivers/iommu/tegra-gart.c +++ b/drivers/iommu/tegra-gart.c @@ -178,7 +178,7 @@ static inline int __gart_iommu_map(struct gart_device *gart, unsigned long iova, } static int gart_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t pa, size_t bytes, int prot) + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) { struct gart_device *gart = gart_handle; int ret; diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c index c4a652b227f8..6ec58a2bcc13 100644 --- a/drivers/iommu/tegra-smmu.c +++ b/drivers/iommu/tegra-smmu.c @@ -650,7 +650,7 @@ static void tegra_smmu_set_pte(struct tegra_smmu_as *as, unsigned long iova, } static int tegra_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct tegra_smmu_as *as = to_smmu_as(domain); dma_addr_t pte_dma; diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c index 80a740df0737..15443457f3fc 100644 --- a/drivers/iommu/virtio-iommu.c +++ b/drivers/iommu/virtio-iommu.c @@ -713,7 +713,7 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int viommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { int ret; u32 flags; diff --git a/include/linux/iommu.h b/include/linux/iommu.h index fdc355ccc570..89af6271c506 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -240,7 +240,7 @@ struct iommu_ops { int (*attach_dev)(struct iommu_domain *domain, struct device *dev); void (*detach_dev)(struct iommu_domain *domain, struct device *dev); int (*map)(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot); + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); size_t (*unmap)(struct iommu_domain *domain, unsigned long iova, size_t size); void (*flush_iotlb_all)(struct iommu_domain *domain); @@ -399,12 +399,17 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); extern int iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t paddr, size_t size, int prot); +extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot); extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_unmap_fast(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, struct scatterlist *sg,unsigned int nents, int prot); +extern size_t iommu_map_sg_atomic(struct iommu_domain *domain, + unsigned long iova, struct scatterlist *sg, + unsigned int nents, int prot); extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova); extern void iommu_set_fault_handler(struct iommu_domain *domain, iommu_fault_handler_t handler, void *token); @@ -614,6 +619,13 @@ static inline int iommu_map(struct iommu_domain *domain, unsigned long iova, return -ENODEV; } +static inline int iommu_map_atomic(struct iommu_domain *domain, + unsigned long iova, phys_addr_t paddr, + size_t size, int prot) +{ + return -ENODEV; +} + static inline size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size) { @@ -633,6 +645,13 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain, return 0; } +static inline size_t iommu_map_sg_atomic(struct iommu_domain *domain, + unsigned long iova, struct scatterlist *sg, + unsigned int nents, int prot) +{ + return 0; +} + static inline void iommu_flush_tlb_all(struct iommu_domain *domain) { } -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Rob Clark, Kukjin Kim, Robin Murphy Add a gfp_t parameter to the iommu_ops::map function. Remove the needless locking in the AMD iommu driver. The iommu_ops::map function (or the iommu_map function which calls it) was always supposed to be sleepable (according to Joerg's comment in this thread: https://lore.kernel.org/patchwork/patch/977520/ ) and so should probably have had a "might_sleep()" since it was written. However currently the dma-iommu api can call iommu_map in an atomic context, which it shouldn't do. This doesn't cause any problems because any iommu driver which uses the dma-iommu api uses gfp_atomic in it's iommu_ops::map function. But doing this wastes the memory allocators atomic pools. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/amd_iommu.c | 3 ++- drivers/iommu/arm-smmu-v3.c | 2 +- drivers/iommu/arm-smmu.c | 2 +- drivers/iommu/dma-iommu.c | 6 ++--- drivers/iommu/exynos-iommu.c | 2 +- drivers/iommu/intel-iommu.c | 2 +- drivers/iommu/iommu.c | 43 +++++++++++++++++++++++++++++----- drivers/iommu/ipmmu-vmsa.c | 2 +- drivers/iommu/msm_iommu.c | 2 +- drivers/iommu/mtk_iommu.c | 2 +- drivers/iommu/mtk_iommu_v1.c | 2 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 2 +- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 2 +- drivers/iommu/tegra-gart.c | 2 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/iommu/virtio-iommu.c | 2 +- include/linux/iommu.h | 21 ++++++++++++++++- 19 files changed, 77 insertions(+), 26 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 1948be7ac8f8..0e53f9bd2be7 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -3030,7 +3030,8 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, } static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, - phys_addr_t paddr, size_t page_size, int iommu_prot) + phys_addr_t paddr, size_t page_size, int iommu_prot, + gfp_t gfp) { struct protection_domain *domain = to_pdomain(dom); int prot = 0; diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index e7f49fd1a7ba..acc0eae7963f 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -1975,7 +1975,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c index aa06498f291d..05f42bdee494 100644 --- a/drivers/iommu/arm-smmu.c +++ b/drivers/iommu/arm-smmu.c @@ -1284,7 +1284,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index d991d40f797f..2712fbc68b28 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -469,7 +469,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, if (!iova) return DMA_MAPPING_ERROR; - if (iommu_map(domain, iova, phys - iova_off, size, prot)) { + if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) { iommu_dma_free_iova(cookie, iova, size); return DMA_MAPPING_ERROR; } @@ -613,7 +613,7 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, arch_dma_prep_coherent(sg_page(sg), sg->length); } - if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) + if (iommu_map_sg_atomic(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) < size) goto out_free_sg; @@ -873,7 +873,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, * We'll leave any physical concatenation to the IOMMU driver's * implementation - it knows better than we do. */ - if (iommu_map_sg(domain, iova, sg, nents, prot) < iova_len) + if (iommu_map_sg_atomic(domain, iova, sg, nents, prot) < iova_len) goto out_free_iova; return __finalise_sg(dev, sg, nents, iova); diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c index 1934c16a5abc..b7dd46884692 100644 --- a/drivers/iommu/exynos-iommu.c +++ b/drivers/iommu/exynos-iommu.c @@ -1073,7 +1073,7 @@ static int lv2set_page(sysmmu_pte_t *pent, phys_addr_t paddr, size_t size, */ static int exynos_iommu_map(struct iommu_domain *iommu_domain, unsigned long l_iova, phys_addr_t paddr, size_t size, - int prot) + int prot, gfp_t gfp) { struct exynos_iommu_domain *domain = to_exynos_domain(iommu_domain); sysmmu_pte_t *entry; diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 12d094d08c0a..0e7cb0cbbd46 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -5113,7 +5113,7 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain, static int intel_iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t hpa, - size_t size, int iommu_prot) + size_t size, int iommu_prot, gfp_t gfp) { struct dmar_domain *dmar_domain = to_dmar_domain(domain); u64 max_addr; diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 0c674d80c37f..133ec8e00a3a 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1799,8 +1799,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, return pgsize; } -int iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) +int __iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { const struct iommu_ops *ops = domain->ops; unsigned long orig_iova = iova; @@ -1837,8 +1837,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", iova, &paddr, pgsize); + ret = ops->map(domain, iova, paddr, pgsize, prot, gfp); - ret = ops->map(domain, iova, paddr, pgsize, prot); if (ret) break; @@ -1858,8 +1858,22 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, return ret; } + +int iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot) +{ + might_sleep(); + return __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL); +} EXPORT_SYMBOL_GPL(iommu_map); +int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot) +{ + return __iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC); +} +EXPORT_SYMBOL_GPL(iommu_map_atomic); + static size_t __iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size, bool sync) @@ -1934,8 +1948,9 @@ size_t iommu_unmap_fast(struct iommu_domain *domain, } EXPORT_SYMBOL_GPL(iommu_unmap_fast); -size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, - struct scatterlist *sg, unsigned int nents, int prot) +size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot, + gfp_t gfp) { size_t len = 0, mapped = 0; phys_addr_t start; @@ -1946,7 +1961,9 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, phys_addr_t s_phys = sg_phys(sg); if (len && s_phys != start + len) { - ret = iommu_map(domain, iova + mapped, start, len, prot); + ret = __iommu_map(domain, iova + mapped, start, + len, prot, gfp); + if (ret) goto out_err; @@ -1974,8 +1991,22 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, return 0; } + +size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + might_sleep(); + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_KERNEL); +} EXPORT_SYMBOL_GPL(iommu_map_sg); +size_t iommu_map_sg_atomic(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_ATOMIC); +} +EXPORT_SYMBOL_GPL(iommu_map_sg_atomic); + int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, phys_addr_t paddr, u64 size, int prot) { diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c index ad0098c0c87c..41572b40b844 100644 --- a/drivers/iommu/ipmmu-vmsa.c +++ b/drivers/iommu/ipmmu-vmsa.c @@ -722,7 +722,7 @@ static void ipmmu_detach_device(struct iommu_domain *io_domain, } static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain); diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c index 3df9266abe65..5fa915c5d541 100644 --- a/drivers/iommu/msm_iommu.c +++ b/drivers/iommu/msm_iommu.c @@ -495,7 +495,7 @@ static void msm_iommu_detach_dev(struct iommu_domain *domain, } static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t pa, size_t len, int prot) + phys_addr_t pa, size_t len, int prot, gfp_t gfp) { struct msm_priv *priv = to_msm_priv(domain); unsigned long flags; diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c index 82e4be4dfdaf..53371b9e9af5 100644 --- a/drivers/iommu/mtk_iommu.c +++ b/drivers/iommu/mtk_iommu.c @@ -356,7 +356,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, } static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct mtk_iommu_domain *dom = to_mtk_domain(domain); unsigned long flags; diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c index abeeac488372..453e704bee3d 100644 --- a/drivers/iommu/mtk_iommu_v1.c +++ b/drivers/iommu/mtk_iommu_v1.c @@ -295,7 +295,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, } static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct mtk_iommu_domain *dom = to_mtk_domain(domain); unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT; diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c index 99a9ff3e7f71..3c530768200e 100644 --- a/drivers/iommu/omap-iommu.c +++ b/drivers/iommu/omap-iommu.c @@ -1339,7 +1339,7 @@ static u32 iotlb_init_entry(struct iotlb_entry *e, u32 da, u32 pa, int pgsz) } static int omap_iommu_map(struct iommu_domain *domain, unsigned long da, - phys_addr_t pa, size_t bytes, int prot) + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) { struct omap_iommu_domain *omap_domain = to_omap_domain(domain); struct device *dev = omap_domain->dev; diff --git a/drivers/iommu/qcom_iommu.c b/drivers/iommu/qcom_iommu.c index fb45486c6d14..b373c0f18ad5 100644 --- a/drivers/iommu/qcom_iommu.c +++ b/drivers/iommu/qcom_iommu.c @@ -400,7 +400,7 @@ static void qcom_iommu_detach_dev(struct iommu_domain *domain, struct device *de } static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { int ret; unsigned long flags; diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c index dc26d74d79c2..b06d2986b52e 100644 --- a/drivers/iommu/rockchip-iommu.c +++ b/drivers/iommu/rockchip-iommu.c @@ -757,7 +757,7 @@ static int rk_iommu_map_iova(struct rk_iommu_domain *rk_domain, u32 *pte_addr, } static int rk_iommu_map(struct iommu_domain *domain, unsigned long _iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct rk_iommu_domain *rk_domain = to_rk_domain(domain); unsigned long flags; diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c index 22d4db302c1c..efa6aa68521d 100644 --- a/drivers/iommu/s390-iommu.c +++ b/drivers/iommu/s390-iommu.c @@ -265,7 +265,7 @@ static int s390_iommu_update_trans(struct s390_domain *s390_domain, } static int s390_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct s390_domain *s390_domain = to_s390_domain(domain); int flags = ZPCI_PTE_VALID, rc = 0; diff --git a/drivers/iommu/tegra-gart.c b/drivers/iommu/tegra-gart.c index 6d40bc1b38bf..43c725243e8a 100644 --- a/drivers/iommu/tegra-gart.c +++ b/drivers/iommu/tegra-gart.c @@ -178,7 +178,7 @@ static inline int __gart_iommu_map(struct gart_device *gart, unsigned long iova, } static int gart_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t pa, size_t bytes, int prot) + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) { struct gart_device *gart = gart_handle; int ret; diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c index c4a652b227f8..6ec58a2bcc13 100644 --- a/drivers/iommu/tegra-smmu.c +++ b/drivers/iommu/tegra-smmu.c @@ -650,7 +650,7 @@ static void tegra_smmu_set_pte(struct tegra_smmu_as *as, unsigned long iova, } static int tegra_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct tegra_smmu_as *as = to_smmu_as(domain); dma_addr_t pte_dma; diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c index 80a740df0737..15443457f3fc 100644 --- a/drivers/iommu/virtio-iommu.c +++ b/drivers/iommu/virtio-iommu.c @@ -713,7 +713,7 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int viommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { int ret; u32 flags; diff --git a/include/linux/iommu.h b/include/linux/iommu.h index fdc355ccc570..89af6271c506 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -240,7 +240,7 @@ struct iommu_ops { int (*attach_dev)(struct iommu_domain *domain, struct device *dev); void (*detach_dev)(struct iommu_domain *domain, struct device *dev); int (*map)(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot); + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); size_t (*unmap)(struct iommu_domain *domain, unsigned long iova, size_t size); void (*flush_iotlb_all)(struct iommu_domain *domain); @@ -399,12 +399,17 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); extern int iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t paddr, size_t size, int prot); +extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot); extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_unmap_fast(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, struct scatterlist *sg,unsigned int nents, int prot); +extern size_t iommu_map_sg_atomic(struct iommu_domain *domain, + unsigned long iova, struct scatterlist *sg, + unsigned int nents, int prot); extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova); extern void iommu_set_fault_handler(struct iommu_domain *domain, iommu_fault_handler_t handler, void *token); @@ -614,6 +619,13 @@ static inline int iommu_map(struct iommu_domain *domain, unsigned long iova, return -ENODEV; } +static inline int iommu_map_atomic(struct iommu_domain *domain, + unsigned long iova, phys_addr_t paddr, + size_t size, int prot) +{ + return -ENODEV; +} + static inline size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size) { @@ -633,6 +645,13 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain, return 0; } +static inline size_t iommu_map_sg_atomic(struct iommu_domain *domain, + unsigned long iova, struct scatterlist *sg, + unsigned int nents, int prot) +{ + return 0; +} + static inline void iommu_flush_tlb_all(struct iommu_domain *domain) { } -- 2.20.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Kukjin Kim, Robin Murphy Add a gfp_t parameter to the iommu_ops::map function. Remove the needless locking in the AMD iommu driver. The iommu_ops::map function (or the iommu_map function which calls it) was always supposed to be sleepable (according to Joerg's comment in this thread: https://lore.kernel.org/patchwork/patch/977520/ ) and so should probably have had a "might_sleep()" since it was written. However currently the dma-iommu api can call iommu_map in an atomic context, which it shouldn't do. This doesn't cause any problems because any iommu driver which uses the dma-iommu api uses gfp_atomic in it's iommu_ops::map function. But doing this wastes the memory allocators atomic pools. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/amd_iommu.c | 3 ++- drivers/iommu/arm-smmu-v3.c | 2 +- drivers/iommu/arm-smmu.c | 2 +- drivers/iommu/dma-iommu.c | 6 ++--- drivers/iommu/exynos-iommu.c | 2 +- drivers/iommu/intel-iommu.c | 2 +- drivers/iommu/iommu.c | 43 +++++++++++++++++++++++++++++----- drivers/iommu/ipmmu-vmsa.c | 2 +- drivers/iommu/msm_iommu.c | 2 +- drivers/iommu/mtk_iommu.c | 2 +- drivers/iommu/mtk_iommu_v1.c | 2 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 2 +- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 2 +- drivers/iommu/tegra-gart.c | 2 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/iommu/virtio-iommu.c | 2 +- include/linux/iommu.h | 21 ++++++++++++++++- 19 files changed, 77 insertions(+), 26 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 1948be7ac8f8..0e53f9bd2be7 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -3030,7 +3030,8 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, } static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, - phys_addr_t paddr, size_t page_size, int iommu_prot) + phys_addr_t paddr, size_t page_size, int iommu_prot, + gfp_t gfp) { struct protection_domain *domain = to_pdomain(dom); int prot = 0; diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index e7f49fd1a7ba..acc0eae7963f 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -1975,7 +1975,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c index aa06498f291d..05f42bdee494 100644 --- a/drivers/iommu/arm-smmu.c +++ b/drivers/iommu/arm-smmu.c @@ -1284,7 +1284,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index d991d40f797f..2712fbc68b28 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -469,7 +469,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, if (!iova) return DMA_MAPPING_ERROR; - if (iommu_map(domain, iova, phys - iova_off, size, prot)) { + if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) { iommu_dma_free_iova(cookie, iova, size); return DMA_MAPPING_ERROR; } @@ -613,7 +613,7 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, arch_dma_prep_coherent(sg_page(sg), sg->length); } - if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) + if (iommu_map_sg_atomic(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) < size) goto out_free_sg; @@ -873,7 +873,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, * We'll leave any physical concatenation to the IOMMU driver's * implementation - it knows better than we do. */ - if (iommu_map_sg(domain, iova, sg, nents, prot) < iova_len) + if (iommu_map_sg_atomic(domain, iova, sg, nents, prot) < iova_len) goto out_free_iova; return __finalise_sg(dev, sg, nents, iova); diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c index 1934c16a5abc..b7dd46884692 100644 --- a/drivers/iommu/exynos-iommu.c +++ b/drivers/iommu/exynos-iommu.c @@ -1073,7 +1073,7 @@ static int lv2set_page(sysmmu_pte_t *pent, phys_addr_t paddr, size_t size, */ static int exynos_iommu_map(struct iommu_domain *iommu_domain, unsigned long l_iova, phys_addr_t paddr, size_t size, - int prot) + int prot, gfp_t gfp) { struct exynos_iommu_domain *domain = to_exynos_domain(iommu_domain); sysmmu_pte_t *entry; diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 12d094d08c0a..0e7cb0cbbd46 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -5113,7 +5113,7 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain, static int intel_iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t hpa, - size_t size, int iommu_prot) + size_t size, int iommu_prot, gfp_t gfp) { struct dmar_domain *dmar_domain = to_dmar_domain(domain); u64 max_addr; diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 0c674d80c37f..133ec8e00a3a 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1799,8 +1799,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, return pgsize; } -int iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) +int __iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { const struct iommu_ops *ops = domain->ops; unsigned long orig_iova = iova; @@ -1837,8 +1837,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", iova, &paddr, pgsize); + ret = ops->map(domain, iova, paddr, pgsize, prot, gfp); - ret = ops->map(domain, iova, paddr, pgsize, prot); if (ret) break; @@ -1858,8 +1858,22 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, return ret; } + +int iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot) +{ + might_sleep(); + return __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL); +} EXPORT_SYMBOL_GPL(iommu_map); +int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot) +{ + return __iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC); +} +EXPORT_SYMBOL_GPL(iommu_map_atomic); + static size_t __iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size, bool sync) @@ -1934,8 +1948,9 @@ size_t iommu_unmap_fast(struct iommu_domain *domain, } EXPORT_SYMBOL_GPL(iommu_unmap_fast); -size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, - struct scatterlist *sg, unsigned int nents, int prot) +size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot, + gfp_t gfp) { size_t len = 0, mapped = 0; phys_addr_t start; @@ -1946,7 +1961,9 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, phys_addr_t s_phys = sg_phys(sg); if (len && s_phys != start + len) { - ret = iommu_map(domain, iova + mapped, start, len, prot); + ret = __iommu_map(domain, iova + mapped, start, + len, prot, gfp); + if (ret) goto out_err; @@ -1974,8 +1991,22 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, return 0; } + +size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + might_sleep(); + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_KERNEL); +} EXPORT_SYMBOL_GPL(iommu_map_sg); +size_t iommu_map_sg_atomic(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_ATOMIC); +} +EXPORT_SYMBOL_GPL(iommu_map_sg_atomic); + int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, phys_addr_t paddr, u64 size, int prot) { diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c index ad0098c0c87c..41572b40b844 100644 --- a/drivers/iommu/ipmmu-vmsa.c +++ b/drivers/iommu/ipmmu-vmsa.c @@ -722,7 +722,7 @@ static void ipmmu_detach_device(struct iommu_domain *io_domain, } static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain); diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c index 3df9266abe65..5fa915c5d541 100644 --- a/drivers/iommu/msm_iommu.c +++ b/drivers/iommu/msm_iommu.c @@ -495,7 +495,7 @@ static void msm_iommu_detach_dev(struct iommu_domain *domain, } static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t pa, size_t len, int prot) + phys_addr_t pa, size_t len, int prot, gfp_t gfp) { struct msm_priv *priv = to_msm_priv(domain); unsigned long flags; diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c index 82e4be4dfdaf..53371b9e9af5 100644 --- a/drivers/iommu/mtk_iommu.c +++ b/drivers/iommu/mtk_iommu.c @@ -356,7 +356,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, } static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct mtk_iommu_domain *dom = to_mtk_domain(domain); unsigned long flags; diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c index abeeac488372..453e704bee3d 100644 --- a/drivers/iommu/mtk_iommu_v1.c +++ b/drivers/iommu/mtk_iommu_v1.c @@ -295,7 +295,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, } static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct mtk_iommu_domain *dom = to_mtk_domain(domain); unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT; diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c index 99a9ff3e7f71..3c530768200e 100644 --- a/drivers/iommu/omap-iommu.c +++ b/drivers/iommu/omap-iommu.c @@ -1339,7 +1339,7 @@ static u32 iotlb_init_entry(struct iotlb_entry *e, u32 da, u32 pa, int pgsz) } static int omap_iommu_map(struct iommu_domain *domain, unsigned long da, - phys_addr_t pa, size_t bytes, int prot) + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) { struct omap_iommu_domain *omap_domain = to_omap_domain(domain); struct device *dev = omap_domain->dev; diff --git a/drivers/iommu/qcom_iommu.c b/drivers/iommu/qcom_iommu.c index fb45486c6d14..b373c0f18ad5 100644 --- a/drivers/iommu/qcom_iommu.c +++ b/drivers/iommu/qcom_iommu.c @@ -400,7 +400,7 @@ static void qcom_iommu_detach_dev(struct iommu_domain *domain, struct device *de } static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { int ret; unsigned long flags; diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c index dc26d74d79c2..b06d2986b52e 100644 --- a/drivers/iommu/rockchip-iommu.c +++ b/drivers/iommu/rockchip-iommu.c @@ -757,7 +757,7 @@ static int rk_iommu_map_iova(struct rk_iommu_domain *rk_domain, u32 *pte_addr, } static int rk_iommu_map(struct iommu_domain *domain, unsigned long _iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct rk_iommu_domain *rk_domain = to_rk_domain(domain); unsigned long flags; diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c index 22d4db302c1c..efa6aa68521d 100644 --- a/drivers/iommu/s390-iommu.c +++ b/drivers/iommu/s390-iommu.c @@ -265,7 +265,7 @@ static int s390_iommu_update_trans(struct s390_domain *s390_domain, } static int s390_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct s390_domain *s390_domain = to_s390_domain(domain); int flags = ZPCI_PTE_VALID, rc = 0; diff --git a/drivers/iommu/tegra-gart.c b/drivers/iommu/tegra-gart.c index 6d40bc1b38bf..43c725243e8a 100644 --- a/drivers/iommu/tegra-gart.c +++ b/drivers/iommu/tegra-gart.c @@ -178,7 +178,7 @@ static inline int __gart_iommu_map(struct gart_device *gart, unsigned long iova, } static int gart_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t pa, size_t bytes, int prot) + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) { struct gart_device *gart = gart_handle; int ret; diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c index c4a652b227f8..6ec58a2bcc13 100644 --- a/drivers/iommu/tegra-smmu.c +++ b/drivers/iommu/tegra-smmu.c @@ -650,7 +650,7 @@ static void tegra_smmu_set_pte(struct tegra_smmu_as *as, unsigned long iova, } static int tegra_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct tegra_smmu_as *as = to_smmu_as(domain); dma_addr_t pte_dma; diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c index 80a740df0737..15443457f3fc 100644 --- a/drivers/iommu/virtio-iommu.c +++ b/drivers/iommu/virtio-iommu.c @@ -713,7 +713,7 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int viommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { int ret; u32 flags; diff --git a/include/linux/iommu.h b/include/linux/iommu.h index fdc355ccc570..89af6271c506 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -240,7 +240,7 @@ struct iommu_ops { int (*attach_dev)(struct iommu_domain *domain, struct device *dev); void (*detach_dev)(struct iommu_domain *domain, struct device *dev); int (*map)(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot); + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); size_t (*unmap)(struct iommu_domain *domain, unsigned long iova, size_t size); void (*flush_iotlb_all)(struct iommu_domain *domain); @@ -399,12 +399,17 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); extern int iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t paddr, size_t size, int prot); +extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot); extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_unmap_fast(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, struct scatterlist *sg,unsigned int nents, int prot); +extern size_t iommu_map_sg_atomic(struct iommu_domain *domain, + unsigned long iova, struct scatterlist *sg, + unsigned int nents, int prot); extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova); extern void iommu_set_fault_handler(struct iommu_domain *domain, iommu_fault_handler_t handler, void *token); @@ -614,6 +619,13 @@ static inline int iommu_map(struct iommu_domain *domain, unsigned long iova, return -ENODEV; } +static inline int iommu_map_atomic(struct iommu_domain *domain, + unsigned long iova, phys_addr_t paddr, + size_t size, int prot) +{ + return -ENODEV; +} + static inline size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size) { @@ -633,6 +645,13 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain, return 0; } +static inline size_t iommu_map_sg_atomic(struct iommu_domain *domain, + unsigned long iova, struct scatterlist *sg, + unsigned int nents, int prot) +{ + return 0; +} + static inline void iommu_flush_tlb_all(struct iommu_domain *domain) { } -- 2.20.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Tom Murphy, Joerg Roedel, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390, linux-tegra, virtualization Add a gfp_t parameter to the iommu_ops::map function. Remove the needless locking in the AMD iommu driver. The iommu_ops::map function (or the iommu_map function which calls it) was always supposed to be sleepable (according to Joerg's comment in this thread: https://lore.kernel.org/patchwork/patch/977520/ ) and so should probably have had a "might_sleep()" since it was written. However currently the dma-iommu api can call iommu_map in an atomic context, which it shouldn't do. This doesn't cause any problems because any iommu driver which uses the dma-iommu api uses gfp_atomic in it's iommu_ops::map function. But doing this wastes the memory allocators atomic pools. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/amd_iommu.c | 3 ++- drivers/iommu/arm-smmu-v3.c | 2 +- drivers/iommu/arm-smmu.c | 2 +- drivers/iommu/dma-iommu.c | 6 ++--- drivers/iommu/exynos-iommu.c | 2 +- drivers/iommu/intel-iommu.c | 2 +- drivers/iommu/iommu.c | 43 +++++++++++++++++++++++++++++----- drivers/iommu/ipmmu-vmsa.c | 2 +- drivers/iommu/msm_iommu.c | 2 +- drivers/iommu/mtk_iommu.c | 2 +- drivers/iommu/mtk_iommu_v1.c | 2 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 2 +- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 2 +- drivers/iommu/tegra-gart.c | 2 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/iommu/virtio-iommu.c | 2 +- include/linux/iommu.h | 21 ++++++++++++++++- 19 files changed, 77 insertions(+), 26 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 1948be7ac8f8..0e53f9bd2be7 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -3030,7 +3030,8 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, } static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, - phys_addr_t paddr, size_t page_size, int iommu_prot) + phys_addr_t paddr, size_t page_size, int iommu_prot, + gfp_t gfp) { struct protection_domain *domain = to_pdomain(dom); int prot = 0; diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index e7f49fd1a7ba..acc0eae7963f 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -1975,7 +1975,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c index aa06498f291d..05f42bdee494 100644 --- a/drivers/iommu/arm-smmu.c +++ b/drivers/iommu/arm-smmu.c @@ -1284,7 +1284,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index d991d40f797f..2712fbc68b28 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -469,7 +469,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, if (!iova) return DMA_MAPPING_ERROR; - if (iommu_map(domain, iova, phys - iova_off, size, prot)) { + if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) { iommu_dma_free_iova(cookie, iova, size); return DMA_MAPPING_ERROR; } @@ -613,7 +613,7 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, arch_dma_prep_coherent(sg_page(sg), sg->length); } - if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) + if (iommu_map_sg_atomic(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) < size) goto out_free_sg; @@ -873,7 +873,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, * We'll leave any physical concatenation to the IOMMU driver's * implementation - it knows better than we do. */ - if (iommu_map_sg(domain, iova, sg, nents, prot) < iova_len) + if (iommu_map_sg_atomic(domain, iova, sg, nents, prot) < iova_len) goto out_free_iova; return __finalise_sg(dev, sg, nents, iova); diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c index 1934c16a5abc..b7dd46884692 100644 --- a/drivers/iommu/exynos-iommu.c +++ b/drivers/iommu/exynos-iommu.c @@ -1073,7 +1073,7 @@ static int lv2set_page(sysmmu_pte_t *pent, phys_addr_t paddr, size_t size, */ static int exynos_iommu_map(struct iommu_domain *iommu_domain, unsigned long l_iova, phys_addr_t paddr, size_t size, - int prot) + int prot, gfp_t gfp) { struct exynos_iommu_domain *domain = to_exynos_domain(iommu_domain); sysmmu_pte_t *entry; diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 12d094d08c0a..0e7cb0cbbd46 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -5113,7 +5113,7 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain, static int intel_iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t hpa, - size_t size, int iommu_prot) + size_t size, int iommu_prot, gfp_t gfp) { struct dmar_domain *dmar_domain = to_dmar_domain(domain); u64 max_addr; diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 0c674d80c37f..133ec8e00a3a 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1799,8 +1799,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, return pgsize; } -int iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) +int __iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { const struct iommu_ops *ops = domain->ops; unsigned long orig_iova = iova; @@ -1837,8 +1837,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", iova, &paddr, pgsize); + ret = ops->map(domain, iova, paddr, pgsize, prot, gfp); - ret = ops->map(domain, iova, paddr, pgsize, prot); if (ret) break; @@ -1858,8 +1858,22 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, return ret; } + +int iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot) +{ + might_sleep(); + return __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL); +} EXPORT_SYMBOL_GPL(iommu_map); +int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot) +{ + return __iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC); +} +EXPORT_SYMBOL_GPL(iommu_map_atomic); + static size_t __iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size, bool sync) @@ -1934,8 +1948,9 @@ size_t iommu_unmap_fast(struct iommu_domain *domain, } EXPORT_SYMBOL_GPL(iommu_unmap_fast); -size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, - struct scatterlist *sg, unsigned int nents, int prot) +size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot, + gfp_t gfp) { size_t len = 0, mapped = 0; phys_addr_t start; @@ -1946,7 +1961,9 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, phys_addr_t s_phys = sg_phys(sg); if (len && s_phys != start + len) { - ret = iommu_map(domain, iova + mapped, start, len, prot); + ret = __iommu_map(domain, iova + mapped, start, + len, prot, gfp); + if (ret) goto out_err; @@ -1974,8 +1991,22 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, return 0; } + +size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + might_sleep(); + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_KERNEL); +} EXPORT_SYMBOL_GPL(iommu_map_sg); +size_t iommu_map_sg_atomic(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_ATOMIC); +} +EXPORT_SYMBOL_GPL(iommu_map_sg_atomic); + int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, phys_addr_t paddr, u64 size, int prot) { diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c index ad0098c0c87c..41572b40b844 100644 --- a/drivers/iommu/ipmmu-vmsa.c +++ b/drivers/iommu/ipmmu-vmsa.c @@ -722,7 +722,7 @@ static void ipmmu_detach_device(struct iommu_domain *io_domain, } static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain); diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c index 3df9266abe65..5fa915c5d541 100644 --- a/drivers/iommu/msm_iommu.c +++ b/drivers/iommu/msm_iommu.c @@ -495,7 +495,7 @@ static void msm_iommu_detach_dev(struct iommu_domain *domain, } static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t pa, size_t len, int prot) + phys_addr_t pa, size_t len, int prot, gfp_t gfp) { struct msm_priv *priv = to_msm_priv(domain); unsigned long flags; diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c index 82e4be4dfdaf..53371b9e9af5 100644 --- a/drivers/iommu/mtk_iommu.c +++ b/drivers/iommu/mtk_iommu.c @@ -356,7 +356,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, } static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct mtk_iommu_domain *dom = to_mtk_domain(domain); unsigned long flags; diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c index abeeac488372..453e704bee3d 100644 --- a/drivers/iommu/mtk_iommu_v1.c +++ b/drivers/iommu/mtk_iommu_v1.c @@ -295,7 +295,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, } static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct mtk_iommu_domain *dom = to_mtk_domain(domain); unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT; diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c index 99a9ff3e7f71..3c530768200e 100644 --- a/drivers/iommu/omap-iommu.c +++ b/drivers/iommu/omap-iommu.c @@ -1339,7 +1339,7 @@ static u32 iotlb_init_entry(struct iotlb_entry *e, u32 da, u32 pa, int pgsz) } static int omap_iommu_map(struct iommu_domain *domain, unsigned long da, - phys_addr_t pa, size_t bytes, int prot) + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) { struct omap_iommu_domain *omap_domain = to_omap_domain(domain); struct device *dev = omap_domain->dev; diff --git a/drivers/iommu/qcom_iommu.c b/drivers/iommu/qcom_iommu.c index fb45486c6d14..b373c0f18ad5 100644 --- a/drivers/iommu/qcom_iommu.c +++ b/drivers/iommu/qcom_iommu.c @@ -400,7 +400,7 @@ static void qcom_iommu_detach_dev(struct iommu_domain *domain, struct device *de } static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { int ret; unsigned long flags; diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c index dc26d74d79c2..b06d2986b52e 100644 --- a/drivers/iommu/rockchip-iommu.c +++ b/drivers/iommu/rockchip-iommu.c @@ -757,7 +757,7 @@ static int rk_iommu_map_iova(struct rk_iommu_domain *rk_domain, u32 *pte_addr, } static int rk_iommu_map(struct iommu_domain *domain, unsigned long _iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct rk_iommu_domain *rk_domain = to_rk_domain(domain); unsigned long flags; diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c index 22d4db302c1c..efa6aa68521d 100644 --- a/drivers/iommu/s390-iommu.c +++ b/drivers/iommu/s390-iommu.c @@ -265,7 +265,7 @@ static int s390_iommu_update_trans(struct s390_domain *s390_domain, } static int s390_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct s390_domain *s390_domain = to_s390_domain(domain); int flags = ZPCI_PTE_VALID, rc = 0; diff --git a/drivers/iommu/tegra-gart.c b/drivers/iommu/tegra-gart.c index 6d40bc1b38bf..43c725243e8a 100644 --- a/drivers/iommu/tegra-gart.c +++ b/drivers/iommu/tegra-gart.c @@ -178,7 +178,7 @@ static inline int __gart_iommu_map(struct gart_device *gart, unsigned long iova, } static int gart_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t pa, size_t bytes, int prot) + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) { struct gart_device *gart = gart_handle; int ret; diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c index c4a652b227f8..6ec58a2bcc13 100644 --- a/drivers/iommu/tegra-smmu.c +++ b/drivers/iommu/tegra-smmu.c @@ -650,7 +650,7 @@ static void tegra_smmu_set_pte(struct tegra_smmu_as *as, unsigned long iova, } static int tegra_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct tegra_smmu_as *as = to_smmu_as(domain); dma_addr_t pte_dma; diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c index 80a740df0737..15443457f3fc 100644 --- a/drivers/iommu/virtio-iommu.c +++ b/drivers/iommu/virtio-iommu.c @@ -713,7 +713,7 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int viommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { int ret; u32 flags; diff --git a/include/linux/iommu.h b/include/linux/iommu.h index fdc355ccc570..89af6271c506 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -240,7 +240,7 @@ struct iommu_ops { int (*attach_dev)(struct iommu_domain *domain, struct device *dev); void (*detach_dev)(struct iommu_domain *domain, struct device *dev); int (*map)(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot); + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); size_t (*unmap)(struct iommu_domain *domain, unsigned long iova, size_t size); void (*flush_iotlb_all)(struct iommu_domain *domain); @@ -399,12 +399,17 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); extern int iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t paddr, size_t size, int prot); +extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot); extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_unmap_fast(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, struct scatterlist *sg,unsigned int nents, int prot); +extern size_t iommu_map_sg_atomic(struct iommu_domain *domain, + unsigned long iova, struct scatterlist *sg, + unsigned int nents, int prot); extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova); extern void iommu_set_fault_handler(struct iommu_domain *domain, iommu_fault_handler_t handler, void *token); @@ -614,6 +619,13 @@ static inline int iommu_map(struct iommu_domain *domain, unsigned long iova, return -ENODEV; } +static inline int iommu_map_atomic(struct iommu_domain *domain, + unsigned long iova, phys_addr_t paddr, + size_t size, int prot) +{ + return -ENODEV; +} + static inline size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size) { @@ -633,6 +645,13 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain, return 0; } +static inline size_t iommu_map_sg_atomic(struct iommu_domain *domain, + unsigned long iova, struct scatterlist *sg, + unsigned int nents, int prot) +{ + return 0; +} + static inline void iommu_flush_tlb_all(struct iommu_domain *domain) { } -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-19 18:23 ` Robin Murphy -1 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:23 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Joerg Roedel, Will Deacon, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390 On 15/08/2019 12:09, Tom Murphy wrote: > Add a gfp_t parameter to the iommu_ops::map function. > Remove the needless locking in the AMD iommu driver. > > The iommu_ops::map function (or the iommu_map function which calls it) > was always supposed to be sleepable (according to Joerg's comment in > this thread: https://lore.kernel.org/patchwork/patch/977520/ ) and so > should probably have had a "might_sleep()" since it was written. However > currently the dma-iommu api can call iommu_map in an atomic context, > which it shouldn't do. This doesn't cause any problems because any iommu > driver which uses the dma-iommu api uses gfp_atomic in it's > iommu_ops::map function. But doing this wastes the memory allocators > atomic pools. Looks reasonable to me - once we get the merges sorted out I'll take a look at propagating the flags through to io-pgtable for the SMMU drivers and friends. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/amd_iommu.c | 3 ++- > drivers/iommu/arm-smmu-v3.c | 2 +- > drivers/iommu/arm-smmu.c | 2 +- > drivers/iommu/dma-iommu.c | 6 ++--- > drivers/iommu/exynos-iommu.c | 2 +- > drivers/iommu/intel-iommu.c | 2 +- > drivers/iommu/iommu.c | 43 +++++++++++++++++++++++++++++----- > drivers/iommu/ipmmu-vmsa.c | 2 +- > drivers/iommu/msm_iommu.c | 2 +- > drivers/iommu/mtk_iommu.c | 2 +- > drivers/iommu/mtk_iommu_v1.c | 2 +- > drivers/iommu/omap-iommu.c | 2 +- > drivers/iommu/qcom_iommu.c | 2 +- > drivers/iommu/rockchip-iommu.c | 2 +- > drivers/iommu/s390-iommu.c | 2 +- > drivers/iommu/tegra-gart.c | 2 +- > drivers/iommu/tegra-smmu.c | 2 +- > drivers/iommu/virtio-iommu.c | 2 +- > include/linux/iommu.h | 21 ++++++++++++++++- > 19 files changed, 77 insertions(+), 26 deletions(-) > > diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c > index 1948be7ac8f8..0e53f9bd2be7 100644 > --- a/drivers/iommu/amd_iommu.c > +++ b/drivers/iommu/amd_iommu.c > @@ -3030,7 +3030,8 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, > } > > static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, > - phys_addr_t paddr, size_t page_size, int iommu_prot) > + phys_addr_t paddr, size_t page_size, int iommu_prot, > + gfp_t gfp) > { > struct protection_domain *domain = to_pdomain(dom); > int prot = 0; > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c > index e7f49fd1a7ba..acc0eae7963f 100644 > --- a/drivers/iommu/arm-smmu-v3.c > +++ b/drivers/iommu/arm-smmu-v3.c > @@ -1975,7 +1975,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > index aa06498f291d..05f42bdee494 100644 > --- a/drivers/iommu/arm-smmu.c > +++ b/drivers/iommu/arm-smmu.c > @@ -1284,7 +1284,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index d991d40f797f..2712fbc68b28 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -469,7 +469,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > if (!iova) > return DMA_MAPPING_ERROR; > > - if (iommu_map(domain, iova, phys - iova_off, size, prot)) { > + if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) { > iommu_dma_free_iova(cookie, iova, size); > return DMA_MAPPING_ERROR; > } > @@ -613,7 +613,7 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, > arch_dma_prep_coherent(sg_page(sg), sg->length); > } > > - if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) > + if (iommu_map_sg_atomic(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) > < size) > goto out_free_sg; > > @@ -873,7 +873,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, > * We'll leave any physical concatenation to the IOMMU driver's > * implementation - it knows better than we do. > */ > - if (iommu_map_sg(domain, iova, sg, nents, prot) < iova_len) > + if (iommu_map_sg_atomic(domain, iova, sg, nents, prot) < iova_len) > goto out_free_iova; > > return __finalise_sg(dev, sg, nents, iova); > diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c > index 1934c16a5abc..b7dd46884692 100644 > --- a/drivers/iommu/exynos-iommu.c > +++ b/drivers/iommu/exynos-iommu.c > @@ -1073,7 +1073,7 @@ static int lv2set_page(sysmmu_pte_t *pent, phys_addr_t paddr, size_t size, > */ > static int exynos_iommu_map(struct iommu_domain *iommu_domain, > unsigned long l_iova, phys_addr_t paddr, size_t size, > - int prot) > + int prot, gfp_t gfp) > { > struct exynos_iommu_domain *domain = to_exynos_domain(iommu_domain); > sysmmu_pte_t *entry; > diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c > index 12d094d08c0a..0e7cb0cbbd46 100644 > --- a/drivers/iommu/intel-iommu.c > +++ b/drivers/iommu/intel-iommu.c > @@ -5113,7 +5113,7 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain, > > static int intel_iommu_map(struct iommu_domain *domain, > unsigned long iova, phys_addr_t hpa, > - size_t size, int iommu_prot) > + size_t size, int iommu_prot, gfp_t gfp) > { > struct dmar_domain *dmar_domain = to_dmar_domain(domain); > u64 max_addr; > diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c > index 0c674d80c37f..133ec8e00a3a 100644 > --- a/drivers/iommu/iommu.c > +++ b/drivers/iommu/iommu.c > @@ -1799,8 +1799,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, > return pgsize; > } > > -int iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > +int __iommu_map(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > const struct iommu_ops *ops = domain->ops; > unsigned long orig_iova = iova; > @@ -1837,8 +1837,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, > > pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", > iova, &paddr, pgsize); > + ret = ops->map(domain, iova, paddr, pgsize, prot, gfp); > > - ret = ops->map(domain, iova, paddr, pgsize, prot); > if (ret) > break; > > @@ -1858,8 +1858,22 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, > > return ret; > } > + > +int iommu_map(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot) > +{ > + might_sleep(); > + return __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL); > +} > EXPORT_SYMBOL_GPL(iommu_map); > > +int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot) > +{ > + return __iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC); > +} > +EXPORT_SYMBOL_GPL(iommu_map_atomic); > + > static size_t __iommu_unmap(struct iommu_domain *domain, > unsigned long iova, size_t size, > bool sync) > @@ -1934,8 +1948,9 @@ size_t iommu_unmap_fast(struct iommu_domain *domain, > } > EXPORT_SYMBOL_GPL(iommu_unmap_fast); > > -size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > - struct scatterlist *sg, unsigned int nents, int prot) > +size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot, > + gfp_t gfp) > { > size_t len = 0, mapped = 0; > phys_addr_t start; > @@ -1946,7 +1961,9 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > phys_addr_t s_phys = sg_phys(sg); > > if (len && s_phys != start + len) { > - ret = iommu_map(domain, iova + mapped, start, len, prot); > + ret = __iommu_map(domain, iova + mapped, start, > + len, prot, gfp); > + > if (ret) > goto out_err; > > @@ -1974,8 +1991,22 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > return 0; > > } > + > +size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot) > +{ > + might_sleep(); > + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_KERNEL); > +} > EXPORT_SYMBOL_GPL(iommu_map_sg); > > +size_t iommu_map_sg_atomic(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot) > +{ > + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_ATOMIC); > +} > +EXPORT_SYMBOL_GPL(iommu_map_sg_atomic); > + > int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, > phys_addr_t paddr, u64 size, int prot) > { > diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c > index ad0098c0c87c..41572b40b844 100644 > --- a/drivers/iommu/ipmmu-vmsa.c > +++ b/drivers/iommu/ipmmu-vmsa.c > @@ -722,7 +722,7 @@ static void ipmmu_detach_device(struct iommu_domain *io_domain, > } > > static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain); > > diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c > index 3df9266abe65..5fa915c5d541 100644 > --- a/drivers/iommu/msm_iommu.c > +++ b/drivers/iommu/msm_iommu.c > @@ -495,7 +495,7 @@ static void msm_iommu_detach_dev(struct iommu_domain *domain, > } > > static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t pa, size_t len, int prot) > + phys_addr_t pa, size_t len, int prot, gfp_t gfp) > { > struct msm_priv *priv = to_msm_priv(domain); > unsigned long flags; > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c > index 82e4be4dfdaf..53371b9e9af5 100644 > --- a/drivers/iommu/mtk_iommu.c > +++ b/drivers/iommu/mtk_iommu.c > @@ -356,7 +356,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, > } > > static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct mtk_iommu_domain *dom = to_mtk_domain(domain); > unsigned long flags; > diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c > index abeeac488372..453e704bee3d 100644 > --- a/drivers/iommu/mtk_iommu_v1.c > +++ b/drivers/iommu/mtk_iommu_v1.c > @@ -295,7 +295,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, > } > > static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct mtk_iommu_domain *dom = to_mtk_domain(domain); > unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT; > diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c > index 99a9ff3e7f71..3c530768200e 100644 > --- a/drivers/iommu/omap-iommu.c > +++ b/drivers/iommu/omap-iommu.c > @@ -1339,7 +1339,7 @@ static u32 iotlb_init_entry(struct iotlb_entry *e, u32 da, u32 pa, int pgsz) > } > > static int omap_iommu_map(struct iommu_domain *domain, unsigned long da, > - phys_addr_t pa, size_t bytes, int prot) > + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) > { > struct omap_iommu_domain *omap_domain = to_omap_domain(domain); > struct device *dev = omap_domain->dev; > diff --git a/drivers/iommu/qcom_iommu.c b/drivers/iommu/qcom_iommu.c > index fb45486c6d14..b373c0f18ad5 100644 > --- a/drivers/iommu/qcom_iommu.c > +++ b/drivers/iommu/qcom_iommu.c > @@ -400,7 +400,7 @@ static void qcom_iommu_detach_dev(struct iommu_domain *domain, struct device *de > } > > static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > int ret; > unsigned long flags; > diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c > index dc26d74d79c2..b06d2986b52e 100644 > --- a/drivers/iommu/rockchip-iommu.c > +++ b/drivers/iommu/rockchip-iommu.c > @@ -757,7 +757,7 @@ static int rk_iommu_map_iova(struct rk_iommu_domain *rk_domain, u32 *pte_addr, > } > > static int rk_iommu_map(struct iommu_domain *domain, unsigned long _iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct rk_iommu_domain *rk_domain = to_rk_domain(domain); > unsigned long flags; > diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c > index 22d4db302c1c..efa6aa68521d 100644 > --- a/drivers/iommu/s390-iommu.c > +++ b/drivers/iommu/s390-iommu.c > @@ -265,7 +265,7 @@ static int s390_iommu_update_trans(struct s390_domain *s390_domain, > } > > static int s390_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct s390_domain *s390_domain = to_s390_domain(domain); > int flags = ZPCI_PTE_VALID, rc = 0; > diff --git a/drivers/iommu/tegra-gart.c b/drivers/iommu/tegra-gart.c > index 6d40bc1b38bf..43c725243e8a 100644 > --- a/drivers/iommu/tegra-gart.c > +++ b/drivers/iommu/tegra-gart.c > @@ -178,7 +178,7 @@ static inline int __gart_iommu_map(struct gart_device *gart, unsigned long iova, > } > > static int gart_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t pa, size_t bytes, int prot) > + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) > { > struct gart_device *gart = gart_handle; > int ret; > diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c > index c4a652b227f8..6ec58a2bcc13 100644 > --- a/drivers/iommu/tegra-smmu.c > +++ b/drivers/iommu/tegra-smmu.c > @@ -650,7 +650,7 @@ static void tegra_smmu_set_pte(struct tegra_smmu_as *as, unsigned long iova, > } > > static int tegra_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct tegra_smmu_as *as = to_smmu_as(domain); > dma_addr_t pte_dma; > diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c > index 80a740df0737..15443457f3fc 100644 > --- a/drivers/iommu/virtio-iommu.c > +++ b/drivers/iommu/virtio-iommu.c > @@ -713,7 +713,7 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int viommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > int ret; > u32 flags; > diff --git a/include/linux/iommu.h b/include/linux/iommu.h > index fdc355ccc570..89af6271c506 100644 > --- a/include/linux/iommu.h > +++ b/include/linux/iommu.h > @@ -240,7 +240,7 @@ struct iommu_ops { > int (*attach_dev)(struct iommu_domain *domain, struct device *dev); > void (*detach_dev)(struct iommu_domain *domain, struct device *dev); > int (*map)(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot); > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); > size_t (*unmap)(struct iommu_domain *domain, unsigned long iova, > size_t size); > void (*flush_iotlb_all)(struct iommu_domain *domain); > @@ -399,12 +399,17 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); > extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); > extern int iommu_map(struct iommu_domain *domain, unsigned long iova, > phys_addr_t paddr, size_t size, int prot); > +extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot); > extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, > size_t size); > extern size_t iommu_unmap_fast(struct iommu_domain *domain, > unsigned long iova, size_t size); > extern size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > struct scatterlist *sg,unsigned int nents, int prot); > +extern size_t iommu_map_sg_atomic(struct iommu_domain *domain, > + unsigned long iova, struct scatterlist *sg, > + unsigned int nents, int prot); > extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova); > extern void iommu_set_fault_handler(struct iommu_domain *domain, > iommu_fault_handler_t handler, void *token); > @@ -614,6 +619,13 @@ static inline int iommu_map(struct iommu_domain *domain, unsigned long iova, > return -ENODEV; > } > > +static inline int iommu_map_atomic(struct iommu_domain *domain, > + unsigned long iova, phys_addr_t paddr, > + size_t size, int prot) > +{ > + return -ENODEV; > +} > + > static inline size_t iommu_unmap(struct iommu_domain *domain, > unsigned long iova, size_t size) > { > @@ -633,6 +645,13 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain, > return 0; > } > > +static inline size_t iommu_map_sg_atomic(struct iommu_domain *domain, > + unsigned long iova, struct scatterlist *sg, > + unsigned int nents, int prot) > +{ > + return 0; > +} > + > static inline void iommu_flush_tlb_all(struct iommu_domain *domain) > { > } > ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map @ 2019-08-19 18:23 ` Robin Murphy 0 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:23 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, linux-kernel, Rob Clark, Kukjin Kim, David Woodhouse On 15/08/2019 12:09, Tom Murphy wrote: > Add a gfp_t parameter to the iommu_ops::map function. > Remove the needless locking in the AMD iommu driver. > > The iommu_ops::map function (or the iommu_map function which calls it) > was always supposed to be sleepable (according to Joerg's comment in > this thread: https://lore.kernel.org/patchwork/patch/977520/ ) and so > should probably have had a "might_sleep()" since it was written. However > currently the dma-iommu api can call iommu_map in an atomic context, > which it shouldn't do. This doesn't cause any problems because any iommu > driver which uses the dma-iommu api uses gfp_atomic in it's > iommu_ops::map function. But doing this wastes the memory allocators > atomic pools. Looks reasonable to me - once we get the merges sorted out I'll take a look at propagating the flags through to io-pgtable for the SMMU drivers and friends. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/amd_iommu.c | 3 ++- > drivers/iommu/arm-smmu-v3.c | 2 +- > drivers/iommu/arm-smmu.c | 2 +- > drivers/iommu/dma-iommu.c | 6 ++--- > drivers/iommu/exynos-iommu.c | 2 +- > drivers/iommu/intel-iommu.c | 2 +- > drivers/iommu/iommu.c | 43 +++++++++++++++++++++++++++++----- > drivers/iommu/ipmmu-vmsa.c | 2 +- > drivers/iommu/msm_iommu.c | 2 +- > drivers/iommu/mtk_iommu.c | 2 +- > drivers/iommu/mtk_iommu_v1.c | 2 +- > drivers/iommu/omap-iommu.c | 2 +- > drivers/iommu/qcom_iommu.c | 2 +- > drivers/iommu/rockchip-iommu.c | 2 +- > drivers/iommu/s390-iommu.c | 2 +- > drivers/iommu/tegra-gart.c | 2 +- > drivers/iommu/tegra-smmu.c | 2 +- > drivers/iommu/virtio-iommu.c | 2 +- > include/linux/iommu.h | 21 ++++++++++++++++- > 19 files changed, 77 insertions(+), 26 deletions(-) > > diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c > index 1948be7ac8f8..0e53f9bd2be7 100644 > --- a/drivers/iommu/amd_iommu.c > +++ b/drivers/iommu/amd_iommu.c > @@ -3030,7 +3030,8 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, > } > > static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, > - phys_addr_t paddr, size_t page_size, int iommu_prot) > + phys_addr_t paddr, size_t page_size, int iommu_prot, > + gfp_t gfp) > { > struct protection_domain *domain = to_pdomain(dom); > int prot = 0; > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c > index e7f49fd1a7ba..acc0eae7963f 100644 > --- a/drivers/iommu/arm-smmu-v3.c > +++ b/drivers/iommu/arm-smmu-v3.c > @@ -1975,7 +1975,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > index aa06498f291d..05f42bdee494 100644 > --- a/drivers/iommu/arm-smmu.c > +++ b/drivers/iommu/arm-smmu.c > @@ -1284,7 +1284,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index d991d40f797f..2712fbc68b28 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -469,7 +469,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > if (!iova) > return DMA_MAPPING_ERROR; > > - if (iommu_map(domain, iova, phys - iova_off, size, prot)) { > + if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) { > iommu_dma_free_iova(cookie, iova, size); > return DMA_MAPPING_ERROR; > } > @@ -613,7 +613,7 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, > arch_dma_prep_coherent(sg_page(sg), sg->length); > } > > - if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) > + if (iommu_map_sg_atomic(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) > < size) > goto out_free_sg; > > @@ -873,7 +873,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, > * We'll leave any physical concatenation to the IOMMU driver's > * implementation - it knows better than we do. > */ > - if (iommu_map_sg(domain, iova, sg, nents, prot) < iova_len) > + if (iommu_map_sg_atomic(domain, iova, sg, nents, prot) < iova_len) > goto out_free_iova; > > return __finalise_sg(dev, sg, nents, iova); > diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c > index 1934c16a5abc..b7dd46884692 100644 > --- a/drivers/iommu/exynos-iommu.c > +++ b/drivers/iommu/exynos-iommu.c > @@ -1073,7 +1073,7 @@ static int lv2set_page(sysmmu_pte_t *pent, phys_addr_t paddr, size_t size, > */ > static int exynos_iommu_map(struct iommu_domain *iommu_domain, > unsigned long l_iova, phys_addr_t paddr, size_t size, > - int prot) > + int prot, gfp_t gfp) > { > struct exynos_iommu_domain *domain = to_exynos_domain(iommu_domain); > sysmmu_pte_t *entry; > diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c > index 12d094d08c0a..0e7cb0cbbd46 100644 > --- a/drivers/iommu/intel-iommu.c > +++ b/drivers/iommu/intel-iommu.c > @@ -5113,7 +5113,7 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain, > > static int intel_iommu_map(struct iommu_domain *domain, > unsigned long iova, phys_addr_t hpa, > - size_t size, int iommu_prot) > + size_t size, int iommu_prot, gfp_t gfp) > { > struct dmar_domain *dmar_domain = to_dmar_domain(domain); > u64 max_addr; > diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c > index 0c674d80c37f..133ec8e00a3a 100644 > --- a/drivers/iommu/iommu.c > +++ b/drivers/iommu/iommu.c > @@ -1799,8 +1799,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, > return pgsize; > } > > -int iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > +int __iommu_map(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > const struct iommu_ops *ops = domain->ops; > unsigned long orig_iova = iova; > @@ -1837,8 +1837,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, > > pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", > iova, &paddr, pgsize); > + ret = ops->map(domain, iova, paddr, pgsize, prot, gfp); > > - ret = ops->map(domain, iova, paddr, pgsize, prot); > if (ret) > break; > > @@ -1858,8 +1858,22 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, > > return ret; > } > + > +int iommu_map(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot) > +{ > + might_sleep(); > + return __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL); > +} > EXPORT_SYMBOL_GPL(iommu_map); > > +int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot) > +{ > + return __iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC); > +} > +EXPORT_SYMBOL_GPL(iommu_map_atomic); > + > static size_t __iommu_unmap(struct iommu_domain *domain, > unsigned long iova, size_t size, > bool sync) > @@ -1934,8 +1948,9 @@ size_t iommu_unmap_fast(struct iommu_domain *domain, > } > EXPORT_SYMBOL_GPL(iommu_unmap_fast); > > -size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > - struct scatterlist *sg, unsigned int nents, int prot) > +size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot, > + gfp_t gfp) > { > size_t len = 0, mapped = 0; > phys_addr_t start; > @@ -1946,7 +1961,9 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > phys_addr_t s_phys = sg_phys(sg); > > if (len && s_phys != start + len) { > - ret = iommu_map(domain, iova + mapped, start, len, prot); > + ret = __iommu_map(domain, iova + mapped, start, > + len, prot, gfp); > + > if (ret) > goto out_err; > > @@ -1974,8 +1991,22 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > return 0; > > } > + > +size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot) > +{ > + might_sleep(); > + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_KERNEL); > +} > EXPORT_SYMBOL_GPL(iommu_map_sg); > > +size_t iommu_map_sg_atomic(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot) > +{ > + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_ATOMIC); > +} > +EXPORT_SYMBOL_GPL(iommu_map_sg_atomic); > + > int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, > phys_addr_t paddr, u64 size, int prot) > { > diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c > index ad0098c0c87c..41572b40b844 100644 > --- a/drivers/iommu/ipmmu-vmsa.c > +++ b/drivers/iommu/ipmmu-vmsa.c > @@ -722,7 +722,7 @@ static void ipmmu_detach_device(struct iommu_domain *io_domain, > } > > static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain); > > diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c > index 3df9266abe65..5fa915c5d541 100644 > --- a/drivers/iommu/msm_iommu.c > +++ b/drivers/iommu/msm_iommu.c > @@ -495,7 +495,7 @@ static void msm_iommu_detach_dev(struct iommu_domain *domain, > } > > static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t pa, size_t len, int prot) > + phys_addr_t pa, size_t len, int prot, gfp_t gfp) > { > struct msm_priv *priv = to_msm_priv(domain); > unsigned long flags; > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c > index 82e4be4dfdaf..53371b9e9af5 100644 > --- a/drivers/iommu/mtk_iommu.c > +++ b/drivers/iommu/mtk_iommu.c > @@ -356,7 +356,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, > } > > static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct mtk_iommu_domain *dom = to_mtk_domain(domain); > unsigned long flags; > diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c > index abeeac488372..453e704bee3d 100644 > --- a/drivers/iommu/mtk_iommu_v1.c > +++ b/drivers/iommu/mtk_iommu_v1.c > @@ -295,7 +295,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, > } > > static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct mtk_iommu_domain *dom = to_mtk_domain(domain); > unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT; > diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c > index 99a9ff3e7f71..3c530768200e 100644 > --- a/drivers/iommu/omap-iommu.c > +++ b/drivers/iommu/omap-iommu.c > @@ -1339,7 +1339,7 @@ static u32 iotlb_init_entry(struct iotlb_entry *e, u32 da, u32 pa, int pgsz) > } > > static int omap_iommu_map(struct iommu_domain *domain, unsigned long da, > - phys_addr_t pa, size_t bytes, int prot) > + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) > { > struct omap_iommu_domain *omap_domain = to_omap_domain(domain); > struct device *dev = omap_domain->dev; > diff --git a/drivers/iommu/qcom_iommu.c b/drivers/iommu/qcom_iommu.c > index fb45486c6d14..b373c0f18ad5 100644 > --- a/drivers/iommu/qcom_iommu.c > +++ b/drivers/iommu/qcom_iommu.c > @@ -400,7 +400,7 @@ static void qcom_iommu_detach_dev(struct iommu_domain *domain, struct device *de > } > > static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > int ret; > unsigned long flags; > diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c > index dc26d74d79c2..b06d2986b52e 100644 > --- a/drivers/iommu/rockchip-iommu.c > +++ b/drivers/iommu/rockchip-iommu.c > @@ -757,7 +757,7 @@ static int rk_iommu_map_iova(struct rk_iommu_domain *rk_domain, u32 *pte_addr, > } > > static int rk_iommu_map(struct iommu_domain *domain, unsigned long _iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct rk_iommu_domain *rk_domain = to_rk_domain(domain); > unsigned long flags; > diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c > index 22d4db302c1c..efa6aa68521d 100644 > --- a/drivers/iommu/s390-iommu.c > +++ b/drivers/iommu/s390-iommu.c > @@ -265,7 +265,7 @@ static int s390_iommu_update_trans(struct s390_domain *s390_domain, > } > > static int s390_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct s390_domain *s390_domain = to_s390_domain(domain); > int flags = ZPCI_PTE_VALID, rc = 0; > diff --git a/drivers/iommu/tegra-gart.c b/drivers/iommu/tegra-gart.c > index 6d40bc1b38bf..43c725243e8a 100644 > --- a/drivers/iommu/tegra-gart.c > +++ b/drivers/iommu/tegra-gart.c > @@ -178,7 +178,7 @@ static inline int __gart_iommu_map(struct gart_device *gart, unsigned long iova, > } > > static int gart_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t pa, size_t bytes, int prot) > + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) > { > struct gart_device *gart = gart_handle; > int ret; > diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c > index c4a652b227f8..6ec58a2bcc13 100644 > --- a/drivers/iommu/tegra-smmu.c > +++ b/drivers/iommu/tegra-smmu.c > @@ -650,7 +650,7 @@ static void tegra_smmu_set_pte(struct tegra_smmu_as *as, unsigned long iova, > } > > static int tegra_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct tegra_smmu_as *as = to_smmu_as(domain); > dma_addr_t pte_dma; > diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c > index 80a740df0737..15443457f3fc 100644 > --- a/drivers/iommu/virtio-iommu.c > +++ b/drivers/iommu/virtio-iommu.c > @@ -713,7 +713,7 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int viommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > int ret; > u32 flags; > diff --git a/include/linux/iommu.h b/include/linux/iommu.h > index fdc355ccc570..89af6271c506 100644 > --- a/include/linux/iommu.h > +++ b/include/linux/iommu.h > @@ -240,7 +240,7 @@ struct iommu_ops { > int (*attach_dev)(struct iommu_domain *domain, struct device *dev); > void (*detach_dev)(struct iommu_domain *domain, struct device *dev); > int (*map)(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot); > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); > size_t (*unmap)(struct iommu_domain *domain, unsigned long iova, > size_t size); > void (*flush_iotlb_all)(struct iommu_domain *domain); > @@ -399,12 +399,17 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); > extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); > extern int iommu_map(struct iommu_domain *domain, unsigned long iova, > phys_addr_t paddr, size_t size, int prot); > +extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot); > extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, > size_t size); > extern size_t iommu_unmap_fast(struct iommu_domain *domain, > unsigned long iova, size_t size); > extern size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > struct scatterlist *sg,unsigned int nents, int prot); > +extern size_t iommu_map_sg_atomic(struct iommu_domain *domain, > + unsigned long iova, struct scatterlist *sg, > + unsigned int nents, int prot); > extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova); > extern void iommu_set_fault_handler(struct iommu_domain *domain, > iommu_fault_handler_t handler, void *token); > @@ -614,6 +619,13 @@ static inline int iommu_map(struct iommu_domain *domain, unsigned long iova, > return -ENODEV; > } > > +static inline int iommu_map_atomic(struct iommu_domain *domain, > + unsigned long iova, phys_addr_t paddr, > + size_t size, int prot) > +{ > + return -ENODEV; > +} > + > static inline size_t iommu_unmap(struct iommu_domain *domain, > unsigned long iova, size_t size) > { > @@ -633,6 +645,13 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain, > return 0; > } > > +static inline size_t iommu_map_sg_atomic(struct iommu_domain *domain, > + unsigned long iova, struct scatterlist *sg, > + unsigned int nents, int prot) > +{ > + return 0; > +} > + > static inline void iommu_flush_tlb_all(struct iommu_domain *domain) > { > } > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map @ 2019-08-19 18:23 ` Robin Murphy 0 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:23 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, linux-kernel, Kukjin Kim, David Woodhouse On 15/08/2019 12:09, Tom Murphy wrote: > Add a gfp_t parameter to the iommu_ops::map function. > Remove the needless locking in the AMD iommu driver. > > The iommu_ops::map function (or the iommu_map function which calls it) > was always supposed to be sleepable (according to Joerg's comment in > this thread: https://lore.kernel.org/patchwork/patch/977520/ ) and so > should probably have had a "might_sleep()" since it was written. However > currently the dma-iommu api can call iommu_map in an atomic context, > which it shouldn't do. This doesn't cause any problems because any iommu > driver which uses the dma-iommu api uses gfp_atomic in it's > iommu_ops::map function. But doing this wastes the memory allocators > atomic pools. Looks reasonable to me - once we get the merges sorted out I'll take a look at propagating the flags through to io-pgtable for the SMMU drivers and friends. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/amd_iommu.c | 3 ++- > drivers/iommu/arm-smmu-v3.c | 2 +- > drivers/iommu/arm-smmu.c | 2 +- > drivers/iommu/dma-iommu.c | 6 ++--- > drivers/iommu/exynos-iommu.c | 2 +- > drivers/iommu/intel-iommu.c | 2 +- > drivers/iommu/iommu.c | 43 +++++++++++++++++++++++++++++----- > drivers/iommu/ipmmu-vmsa.c | 2 +- > drivers/iommu/msm_iommu.c | 2 +- > drivers/iommu/mtk_iommu.c | 2 +- > drivers/iommu/mtk_iommu_v1.c | 2 +- > drivers/iommu/omap-iommu.c | 2 +- > drivers/iommu/qcom_iommu.c | 2 +- > drivers/iommu/rockchip-iommu.c | 2 +- > drivers/iommu/s390-iommu.c | 2 +- > drivers/iommu/tegra-gart.c | 2 +- > drivers/iommu/tegra-smmu.c | 2 +- > drivers/iommu/virtio-iommu.c | 2 +- > include/linux/iommu.h | 21 ++++++++++++++++- > 19 files changed, 77 insertions(+), 26 deletions(-) > > diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c > index 1948be7ac8f8..0e53f9bd2be7 100644 > --- a/drivers/iommu/amd_iommu.c > +++ b/drivers/iommu/amd_iommu.c > @@ -3030,7 +3030,8 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, > } > > static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, > - phys_addr_t paddr, size_t page_size, int iommu_prot) > + phys_addr_t paddr, size_t page_size, int iommu_prot, > + gfp_t gfp) > { > struct protection_domain *domain = to_pdomain(dom); > int prot = 0; > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c > index e7f49fd1a7ba..acc0eae7963f 100644 > --- a/drivers/iommu/arm-smmu-v3.c > +++ b/drivers/iommu/arm-smmu-v3.c > @@ -1975,7 +1975,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > index aa06498f291d..05f42bdee494 100644 > --- a/drivers/iommu/arm-smmu.c > +++ b/drivers/iommu/arm-smmu.c > @@ -1284,7 +1284,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index d991d40f797f..2712fbc68b28 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -469,7 +469,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > if (!iova) > return DMA_MAPPING_ERROR; > > - if (iommu_map(domain, iova, phys - iova_off, size, prot)) { > + if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) { > iommu_dma_free_iova(cookie, iova, size); > return DMA_MAPPING_ERROR; > } > @@ -613,7 +613,7 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, > arch_dma_prep_coherent(sg_page(sg), sg->length); > } > > - if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) > + if (iommu_map_sg_atomic(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) > < size) > goto out_free_sg; > > @@ -873,7 +873,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, > * We'll leave any physical concatenation to the IOMMU driver's > * implementation - it knows better than we do. > */ > - if (iommu_map_sg(domain, iova, sg, nents, prot) < iova_len) > + if (iommu_map_sg_atomic(domain, iova, sg, nents, prot) < iova_len) > goto out_free_iova; > > return __finalise_sg(dev, sg, nents, iova); > diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c > index 1934c16a5abc..b7dd46884692 100644 > --- a/drivers/iommu/exynos-iommu.c > +++ b/drivers/iommu/exynos-iommu.c > @@ -1073,7 +1073,7 @@ static int lv2set_page(sysmmu_pte_t *pent, phys_addr_t paddr, size_t size, > */ > static int exynos_iommu_map(struct iommu_domain *iommu_domain, > unsigned long l_iova, phys_addr_t paddr, size_t size, > - int prot) > + int prot, gfp_t gfp) > { > struct exynos_iommu_domain *domain = to_exynos_domain(iommu_domain); > sysmmu_pte_t *entry; > diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c > index 12d094d08c0a..0e7cb0cbbd46 100644 > --- a/drivers/iommu/intel-iommu.c > +++ b/drivers/iommu/intel-iommu.c > @@ -5113,7 +5113,7 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain, > > static int intel_iommu_map(struct iommu_domain *domain, > unsigned long iova, phys_addr_t hpa, > - size_t size, int iommu_prot) > + size_t size, int iommu_prot, gfp_t gfp) > { > struct dmar_domain *dmar_domain = to_dmar_domain(domain); > u64 max_addr; > diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c > index 0c674d80c37f..133ec8e00a3a 100644 > --- a/drivers/iommu/iommu.c > +++ b/drivers/iommu/iommu.c > @@ -1799,8 +1799,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, > return pgsize; > } > > -int iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > +int __iommu_map(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > const struct iommu_ops *ops = domain->ops; > unsigned long orig_iova = iova; > @@ -1837,8 +1837,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, > > pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", > iova, &paddr, pgsize); > + ret = ops->map(domain, iova, paddr, pgsize, prot, gfp); > > - ret = ops->map(domain, iova, paddr, pgsize, prot); > if (ret) > break; > > @@ -1858,8 +1858,22 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, > > return ret; > } > + > +int iommu_map(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot) > +{ > + might_sleep(); > + return __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL); > +} > EXPORT_SYMBOL_GPL(iommu_map); > > +int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot) > +{ > + return __iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC); > +} > +EXPORT_SYMBOL_GPL(iommu_map_atomic); > + > static size_t __iommu_unmap(struct iommu_domain *domain, > unsigned long iova, size_t size, > bool sync) > @@ -1934,8 +1948,9 @@ size_t iommu_unmap_fast(struct iommu_domain *domain, > } > EXPORT_SYMBOL_GPL(iommu_unmap_fast); > > -size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > - struct scatterlist *sg, unsigned int nents, int prot) > +size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot, > + gfp_t gfp) > { > size_t len = 0, mapped = 0; > phys_addr_t start; > @@ -1946,7 +1961,9 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > phys_addr_t s_phys = sg_phys(sg); > > if (len && s_phys != start + len) { > - ret = iommu_map(domain, iova + mapped, start, len, prot); > + ret = __iommu_map(domain, iova + mapped, start, > + len, prot, gfp); > + > if (ret) > goto out_err; > > @@ -1974,8 +1991,22 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > return 0; > > } > + > +size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot) > +{ > + might_sleep(); > + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_KERNEL); > +} > EXPORT_SYMBOL_GPL(iommu_map_sg); > > +size_t iommu_map_sg_atomic(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot) > +{ > + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_ATOMIC); > +} > +EXPORT_SYMBOL_GPL(iommu_map_sg_atomic); > + > int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, > phys_addr_t paddr, u64 size, int prot) > { > diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c > index ad0098c0c87c..41572b40b844 100644 > --- a/drivers/iommu/ipmmu-vmsa.c > +++ b/drivers/iommu/ipmmu-vmsa.c > @@ -722,7 +722,7 @@ static void ipmmu_detach_device(struct iommu_domain *io_domain, > } > > static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain); > > diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c > index 3df9266abe65..5fa915c5d541 100644 > --- a/drivers/iommu/msm_iommu.c > +++ b/drivers/iommu/msm_iommu.c > @@ -495,7 +495,7 @@ static void msm_iommu_detach_dev(struct iommu_domain *domain, > } > > static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t pa, size_t len, int prot) > + phys_addr_t pa, size_t len, int prot, gfp_t gfp) > { > struct msm_priv *priv = to_msm_priv(domain); > unsigned long flags; > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c > index 82e4be4dfdaf..53371b9e9af5 100644 > --- a/drivers/iommu/mtk_iommu.c > +++ b/drivers/iommu/mtk_iommu.c > @@ -356,7 +356,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, > } > > static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct mtk_iommu_domain *dom = to_mtk_domain(domain); > unsigned long flags; > diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c > index abeeac488372..453e704bee3d 100644 > --- a/drivers/iommu/mtk_iommu_v1.c > +++ b/drivers/iommu/mtk_iommu_v1.c > @@ -295,7 +295,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, > } > > static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct mtk_iommu_domain *dom = to_mtk_domain(domain); > unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT; > diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c > index 99a9ff3e7f71..3c530768200e 100644 > --- a/drivers/iommu/omap-iommu.c > +++ b/drivers/iommu/omap-iommu.c > @@ -1339,7 +1339,7 @@ static u32 iotlb_init_entry(struct iotlb_entry *e, u32 da, u32 pa, int pgsz) > } > > static int omap_iommu_map(struct iommu_domain *domain, unsigned long da, > - phys_addr_t pa, size_t bytes, int prot) > + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) > { > struct omap_iommu_domain *omap_domain = to_omap_domain(domain); > struct device *dev = omap_domain->dev; > diff --git a/drivers/iommu/qcom_iommu.c b/drivers/iommu/qcom_iommu.c > index fb45486c6d14..b373c0f18ad5 100644 > --- a/drivers/iommu/qcom_iommu.c > +++ b/drivers/iommu/qcom_iommu.c > @@ -400,7 +400,7 @@ static void qcom_iommu_detach_dev(struct iommu_domain *domain, struct device *de > } > > static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > int ret; > unsigned long flags; > diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c > index dc26d74d79c2..b06d2986b52e 100644 > --- a/drivers/iommu/rockchip-iommu.c > +++ b/drivers/iommu/rockchip-iommu.c > @@ -757,7 +757,7 @@ static int rk_iommu_map_iova(struct rk_iommu_domain *rk_domain, u32 *pte_addr, > } > > static int rk_iommu_map(struct iommu_domain *domain, unsigned long _iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct rk_iommu_domain *rk_domain = to_rk_domain(domain); > unsigned long flags; > diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c > index 22d4db302c1c..efa6aa68521d 100644 > --- a/drivers/iommu/s390-iommu.c > +++ b/drivers/iommu/s390-iommu.c > @@ -265,7 +265,7 @@ static int s390_iommu_update_trans(struct s390_domain *s390_domain, > } > > static int s390_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct s390_domain *s390_domain = to_s390_domain(domain); > int flags = ZPCI_PTE_VALID, rc = 0; > diff --git a/drivers/iommu/tegra-gart.c b/drivers/iommu/tegra-gart.c > index 6d40bc1b38bf..43c725243e8a 100644 > --- a/drivers/iommu/tegra-gart.c > +++ b/drivers/iommu/tegra-gart.c > @@ -178,7 +178,7 @@ static inline int __gart_iommu_map(struct gart_device *gart, unsigned long iova, > } > > static int gart_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t pa, size_t bytes, int prot) > + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) > { > struct gart_device *gart = gart_handle; > int ret; > diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c > index c4a652b227f8..6ec58a2bcc13 100644 > --- a/drivers/iommu/tegra-smmu.c > +++ b/drivers/iommu/tegra-smmu.c > @@ -650,7 +650,7 @@ static void tegra_smmu_set_pte(struct tegra_smmu_as *as, unsigned long iova, > } > > static int tegra_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct tegra_smmu_as *as = to_smmu_as(domain); > dma_addr_t pte_dma; > diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c > index 80a740df0737..15443457f3fc 100644 > --- a/drivers/iommu/virtio-iommu.c > +++ b/drivers/iommu/virtio-iommu.c > @@ -713,7 +713,7 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int viommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > int ret; > u32 flags; > diff --git a/include/linux/iommu.h b/include/linux/iommu.h > index fdc355ccc570..89af6271c506 100644 > --- a/include/linux/iommu.h > +++ b/include/linux/iommu.h > @@ -240,7 +240,7 @@ struct iommu_ops { > int (*attach_dev)(struct iommu_domain *domain, struct device *dev); > void (*detach_dev)(struct iommu_domain *domain, struct device *dev); > int (*map)(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot); > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); > size_t (*unmap)(struct iommu_domain *domain, unsigned long iova, > size_t size); > void (*flush_iotlb_all)(struct iommu_domain *domain); > @@ -399,12 +399,17 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); > extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); > extern int iommu_map(struct iommu_domain *domain, unsigned long iova, > phys_addr_t paddr, size_t size, int prot); > +extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot); > extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, > size_t size); > extern size_t iommu_unmap_fast(struct iommu_domain *domain, > unsigned long iova, size_t size); > extern size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > struct scatterlist *sg,unsigned int nents, int prot); > +extern size_t iommu_map_sg_atomic(struct iommu_domain *domain, > + unsigned long iova, struct scatterlist *sg, > + unsigned int nents, int prot); > extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova); > extern void iommu_set_fault_handler(struct iommu_domain *domain, > iommu_fault_handler_t handler, void *token); > @@ -614,6 +619,13 @@ static inline int iommu_map(struct iommu_domain *domain, unsigned long iova, > return -ENODEV; > } > > +static inline int iommu_map_atomic(struct iommu_domain *domain, > + unsigned long iova, phys_addr_t paddr, > + size_t size, int prot) > +{ > + return -ENODEV; > +} > + > static inline size_t iommu_unmap(struct iommu_domain *domain, > unsigned long iova, size_t size) > { > @@ -633,6 +645,13 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain, > return 0; > } > > +static inline size_t iommu_map_sg_atomic(struct iommu_domain *domain, > + unsigned long iova, struct scatterlist *sg, > + unsigned int nents, int prot) > +{ > + return 0; > +} > + > static inline void iommu_flush_tlb_all(struct iommu_domain *domain) > { > } > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map @ 2019-08-19 18:23 ` Robin Murphy 0 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:23 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Joerg Roedel, Will Deacon, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390, linux-tegra, virtualization On 15/08/2019 12:09, Tom Murphy wrote: > Add a gfp_t parameter to the iommu_ops::map function. > Remove the needless locking in the AMD iommu driver. > > The iommu_ops::map function (or the iommu_map function which calls it) > was always supposed to be sleepable (according to Joerg's comment in > this thread: https://lore.kernel.org/patchwork/patch/977520/ ) and so > should probably have had a "might_sleep()" since it was written. However > currently the dma-iommu api can call iommu_map in an atomic context, > which it shouldn't do. This doesn't cause any problems because any iommu > driver which uses the dma-iommu api uses gfp_atomic in it's > iommu_ops::map function. But doing this wastes the memory allocators > atomic pools. Looks reasonable to me - once we get the merges sorted out I'll take a look at propagating the flags through to io-pgtable for the SMMU drivers and friends. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/amd_iommu.c | 3 ++- > drivers/iommu/arm-smmu-v3.c | 2 +- > drivers/iommu/arm-smmu.c | 2 +- > drivers/iommu/dma-iommu.c | 6 ++--- > drivers/iommu/exynos-iommu.c | 2 +- > drivers/iommu/intel-iommu.c | 2 +- > drivers/iommu/iommu.c | 43 +++++++++++++++++++++++++++++----- > drivers/iommu/ipmmu-vmsa.c | 2 +- > drivers/iommu/msm_iommu.c | 2 +- > drivers/iommu/mtk_iommu.c | 2 +- > drivers/iommu/mtk_iommu_v1.c | 2 +- > drivers/iommu/omap-iommu.c | 2 +- > drivers/iommu/qcom_iommu.c | 2 +- > drivers/iommu/rockchip-iommu.c | 2 +- > drivers/iommu/s390-iommu.c | 2 +- > drivers/iommu/tegra-gart.c | 2 +- > drivers/iommu/tegra-smmu.c | 2 +- > drivers/iommu/virtio-iommu.c | 2 +- > include/linux/iommu.h | 21 ++++++++++++++++- > 19 files changed, 77 insertions(+), 26 deletions(-) > > diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c > index 1948be7ac8f8..0e53f9bd2be7 100644 > --- a/drivers/iommu/amd_iommu.c > +++ b/drivers/iommu/amd_iommu.c > @@ -3030,7 +3030,8 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, > } > > static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, > - phys_addr_t paddr, size_t page_size, int iommu_prot) > + phys_addr_t paddr, size_t page_size, int iommu_prot, > + gfp_t gfp) > { > struct protection_domain *domain = to_pdomain(dom); > int prot = 0; > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c > index e7f49fd1a7ba..acc0eae7963f 100644 > --- a/drivers/iommu/arm-smmu-v3.c > +++ b/drivers/iommu/arm-smmu-v3.c > @@ -1975,7 +1975,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > index aa06498f291d..05f42bdee494 100644 > --- a/drivers/iommu/arm-smmu.c > +++ b/drivers/iommu/arm-smmu.c > @@ -1284,7 +1284,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index d991d40f797f..2712fbc68b28 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -469,7 +469,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > if (!iova) > return DMA_MAPPING_ERROR; > > - if (iommu_map(domain, iova, phys - iova_off, size, prot)) { > + if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) { > iommu_dma_free_iova(cookie, iova, size); > return DMA_MAPPING_ERROR; > } > @@ -613,7 +613,7 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, > arch_dma_prep_coherent(sg_page(sg), sg->length); > } > > - if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) > + if (iommu_map_sg_atomic(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) > < size) > goto out_free_sg; > > @@ -873,7 +873,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, > * We'll leave any physical concatenation to the IOMMU driver's > * implementation - it knows better than we do. > */ > - if (iommu_map_sg(domain, iova, sg, nents, prot) < iova_len) > + if (iommu_map_sg_atomic(domain, iova, sg, nents, prot) < iova_len) > goto out_free_iova; > > return __finalise_sg(dev, sg, nents, iova); > diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c > index 1934c16a5abc..b7dd46884692 100644 > --- a/drivers/iommu/exynos-iommu.c > +++ b/drivers/iommu/exynos-iommu.c > @@ -1073,7 +1073,7 @@ static int lv2set_page(sysmmu_pte_t *pent, phys_addr_t paddr, size_t size, > */ > static int exynos_iommu_map(struct iommu_domain *iommu_domain, > unsigned long l_iova, phys_addr_t paddr, size_t size, > - int prot) > + int prot, gfp_t gfp) > { > struct exynos_iommu_domain *domain = to_exynos_domain(iommu_domain); > sysmmu_pte_t *entry; > diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c > index 12d094d08c0a..0e7cb0cbbd46 100644 > --- a/drivers/iommu/intel-iommu.c > +++ b/drivers/iommu/intel-iommu.c > @@ -5113,7 +5113,7 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain, > > static int intel_iommu_map(struct iommu_domain *domain, > unsigned long iova, phys_addr_t hpa, > - size_t size, int iommu_prot) > + size_t size, int iommu_prot, gfp_t gfp) > { > struct dmar_domain *dmar_domain = to_dmar_domain(domain); > u64 max_addr; > diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c > index 0c674d80c37f..133ec8e00a3a 100644 > --- a/drivers/iommu/iommu.c > +++ b/drivers/iommu/iommu.c > @@ -1799,8 +1799,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, > return pgsize; > } > > -int iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > +int __iommu_map(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > const struct iommu_ops *ops = domain->ops; > unsigned long orig_iova = iova; > @@ -1837,8 +1837,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, > > pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", > iova, &paddr, pgsize); > + ret = ops->map(domain, iova, paddr, pgsize, prot, gfp); > > - ret = ops->map(domain, iova, paddr, pgsize, prot); > if (ret) > break; > > @@ -1858,8 +1858,22 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, > > return ret; > } > + > +int iommu_map(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot) > +{ > + might_sleep(); > + return __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL); > +} > EXPORT_SYMBOL_GPL(iommu_map); > > +int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot) > +{ > + return __iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC); > +} > +EXPORT_SYMBOL_GPL(iommu_map_atomic); > + > static size_t __iommu_unmap(struct iommu_domain *domain, > unsigned long iova, size_t size, > bool sync) > @@ -1934,8 +1948,9 @@ size_t iommu_unmap_fast(struct iommu_domain *domain, > } > EXPORT_SYMBOL_GPL(iommu_unmap_fast); > > -size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > - struct scatterlist *sg, unsigned int nents, int prot) > +size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot, > + gfp_t gfp) > { > size_t len = 0, mapped = 0; > phys_addr_t start; > @@ -1946,7 +1961,9 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > phys_addr_t s_phys = sg_phys(sg); > > if (len && s_phys != start + len) { > - ret = iommu_map(domain, iova + mapped, start, len, prot); > + ret = __iommu_map(domain, iova + mapped, start, > + len, prot, gfp); > + > if (ret) > goto out_err; > > @@ -1974,8 +1991,22 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > return 0; > > } > + > +size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot) > +{ > + might_sleep(); > + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_KERNEL); > +} > EXPORT_SYMBOL_GPL(iommu_map_sg); > > +size_t iommu_map_sg_atomic(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot) > +{ > + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_ATOMIC); > +} > +EXPORT_SYMBOL_GPL(iommu_map_sg_atomic); > + > int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, > phys_addr_t paddr, u64 size, int prot) > { > diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c > index ad0098c0c87c..41572b40b844 100644 > --- a/drivers/iommu/ipmmu-vmsa.c > +++ b/drivers/iommu/ipmmu-vmsa.c > @@ -722,7 +722,7 @@ static void ipmmu_detach_device(struct iommu_domain *io_domain, > } > > static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain); > > diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c > index 3df9266abe65..5fa915c5d541 100644 > --- a/drivers/iommu/msm_iommu.c > +++ b/drivers/iommu/msm_iommu.c > @@ -495,7 +495,7 @@ static void msm_iommu_detach_dev(struct iommu_domain *domain, > } > > static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t pa, size_t len, int prot) > + phys_addr_t pa, size_t len, int prot, gfp_t gfp) > { > struct msm_priv *priv = to_msm_priv(domain); > unsigned long flags; > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c > index 82e4be4dfdaf..53371b9e9af5 100644 > --- a/drivers/iommu/mtk_iommu.c > +++ b/drivers/iommu/mtk_iommu.c > @@ -356,7 +356,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, > } > > static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct mtk_iommu_domain *dom = to_mtk_domain(domain); > unsigned long flags; > diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c > index abeeac488372..453e704bee3d 100644 > --- a/drivers/iommu/mtk_iommu_v1.c > +++ b/drivers/iommu/mtk_iommu_v1.c > @@ -295,7 +295,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, > } > > static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct mtk_iommu_domain *dom = to_mtk_domain(domain); > unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT; > diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c > index 99a9ff3e7f71..3c530768200e 100644 > --- a/drivers/iommu/omap-iommu.c > +++ b/drivers/iommu/omap-iommu.c > @@ -1339,7 +1339,7 @@ static u32 iotlb_init_entry(struct iotlb_entry *e, u32 da, u32 pa, int pgsz) > } > > static int omap_iommu_map(struct iommu_domain *domain, unsigned long da, > - phys_addr_t pa, size_t bytes, int prot) > + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) > { > struct omap_iommu_domain *omap_domain = to_omap_domain(domain); > struct device *dev = omap_domain->dev; > diff --git a/drivers/iommu/qcom_iommu.c b/drivers/iommu/qcom_iommu.c > index fb45486c6d14..b373c0f18ad5 100644 > --- a/drivers/iommu/qcom_iommu.c > +++ b/drivers/iommu/qcom_iommu.c > @@ -400,7 +400,7 @@ static void qcom_iommu_detach_dev(struct iommu_domain *domain, struct device *de > } > > static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > int ret; > unsigned long flags; > diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c > index dc26d74d79c2..b06d2986b52e 100644 > --- a/drivers/iommu/rockchip-iommu.c > +++ b/drivers/iommu/rockchip-iommu.c > @@ -757,7 +757,7 @@ static int rk_iommu_map_iova(struct rk_iommu_domain *rk_domain, u32 *pte_addr, > } > > static int rk_iommu_map(struct iommu_domain *domain, unsigned long _iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct rk_iommu_domain *rk_domain = to_rk_domain(domain); > unsigned long flags; > diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c > index 22d4db302c1c..efa6aa68521d 100644 > --- a/drivers/iommu/s390-iommu.c > +++ b/drivers/iommu/s390-iommu.c > @@ -265,7 +265,7 @@ static int s390_iommu_update_trans(struct s390_domain *s390_domain, > } > > static int s390_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct s390_domain *s390_domain = to_s390_domain(domain); > int flags = ZPCI_PTE_VALID, rc = 0; > diff --git a/drivers/iommu/tegra-gart.c b/drivers/iommu/tegra-gart.c > index 6d40bc1b38bf..43c725243e8a 100644 > --- a/drivers/iommu/tegra-gart.c > +++ b/drivers/iommu/tegra-gart.c > @@ -178,7 +178,7 @@ static inline int __gart_iommu_map(struct gart_device *gart, unsigned long iova, > } > > static int gart_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t pa, size_t bytes, int prot) > + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) > { > struct gart_device *gart = gart_handle; > int ret; > diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c > index c4a652b227f8..6ec58a2bcc13 100644 > --- a/drivers/iommu/tegra-smmu.c > +++ b/drivers/iommu/tegra-smmu.c > @@ -650,7 +650,7 @@ static void tegra_smmu_set_pte(struct tegra_smmu_as *as, unsigned long iova, > } > > static int tegra_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct tegra_smmu_as *as = to_smmu_as(domain); > dma_addr_t pte_dma; > diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c > index 80a740df0737..15443457f3fc 100644 > --- a/drivers/iommu/virtio-iommu.c > +++ b/drivers/iommu/virtio-iommu.c > @@ -713,7 +713,7 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int viommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > int ret; > u32 flags; > diff --git a/include/linux/iommu.h b/include/linux/iommu.h > index fdc355ccc570..89af6271c506 100644 > --- a/include/linux/iommu.h > +++ b/include/linux/iommu.h > @@ -240,7 +240,7 @@ struct iommu_ops { > int (*attach_dev)(struct iommu_domain *domain, struct device *dev); > void (*detach_dev)(struct iommu_domain *domain, struct device *dev); > int (*map)(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot); > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); > size_t (*unmap)(struct iommu_domain *domain, unsigned long iova, > size_t size); > void (*flush_iotlb_all)(struct iommu_domain *domain); > @@ -399,12 +399,17 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); > extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); > extern int iommu_map(struct iommu_domain *domain, unsigned long iova, > phys_addr_t paddr, size_t size, int prot); > +extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot); > extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, > size_t size); > extern size_t iommu_unmap_fast(struct iommu_domain *domain, > unsigned long iova, size_t size); > extern size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > struct scatterlist *sg,unsigned int nents, int prot); > +extern size_t iommu_map_sg_atomic(struct iommu_domain *domain, > + unsigned long iova, struct scatterlist *sg, > + unsigned int nents, int prot); > extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova); > extern void iommu_set_fault_handler(struct iommu_domain *domain, > iommu_fault_handler_t handler, void *token); > @@ -614,6 +619,13 @@ static inline int iommu_map(struct iommu_domain *domain, unsigned long iova, > return -ENODEV; > } > > +static inline int iommu_map_atomic(struct iommu_domain *domain, > + unsigned long iova, phys_addr_t paddr, > + size_t size, int prot) > +{ > + return -ENODEV; > +} > + > static inline size_t iommu_unmap(struct iommu_domain *domain, > unsigned long iova, size_t size) > { > @@ -633,6 +645,13 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain, > return 0; > } > > +static inline size_t iommu_map_sg_atomic(struct iommu_domain *domain, > + unsigned long iova, struct scatterlist *sg, > + unsigned int nents, int prot) > +{ > + return 0; > +} > + > static inline void iommu_flush_tlb_all(struct iommu_domain *domain) > { > } > ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map 2019-08-15 11:09 ` Tom Murphy ` (3 preceding siblings ...) (?) @ 2019-08-19 18:23 ` Robin Murphy -1 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:23 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, linux-kernel, Rob Clark, Kukjin Kim, David Woodhouse On 15/08/2019 12:09, Tom Murphy wrote: > Add a gfp_t parameter to the iommu_ops::map function. > Remove the needless locking in the AMD iommu driver. > > The iommu_ops::map function (or the iommu_map function which calls it) > was always supposed to be sleepable (according to Joerg's comment in > this thread: https://lore.kernel.org/patchwork/patch/977520/ ) and so > should probably have had a "might_sleep()" since it was written. However > currently the dma-iommu api can call iommu_map in an atomic context, > which it shouldn't do. This doesn't cause any problems because any iommu > driver which uses the dma-iommu api uses gfp_atomic in it's > iommu_ops::map function. But doing this wastes the memory allocators > atomic pools. Looks reasonable to me - once we get the merges sorted out I'll take a look at propagating the flags through to io-pgtable for the SMMU drivers and friends. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/amd_iommu.c | 3 ++- > drivers/iommu/arm-smmu-v3.c | 2 +- > drivers/iommu/arm-smmu.c | 2 +- > drivers/iommu/dma-iommu.c | 6 ++--- > drivers/iommu/exynos-iommu.c | 2 +- > drivers/iommu/intel-iommu.c | 2 +- > drivers/iommu/iommu.c | 43 +++++++++++++++++++++++++++++----- > drivers/iommu/ipmmu-vmsa.c | 2 +- > drivers/iommu/msm_iommu.c | 2 +- > drivers/iommu/mtk_iommu.c | 2 +- > drivers/iommu/mtk_iommu_v1.c | 2 +- > drivers/iommu/omap-iommu.c | 2 +- > drivers/iommu/qcom_iommu.c | 2 +- > drivers/iommu/rockchip-iommu.c | 2 +- > drivers/iommu/s390-iommu.c | 2 +- > drivers/iommu/tegra-gart.c | 2 +- > drivers/iommu/tegra-smmu.c | 2 +- > drivers/iommu/virtio-iommu.c | 2 +- > include/linux/iommu.h | 21 ++++++++++++++++- > 19 files changed, 77 insertions(+), 26 deletions(-) > > diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c > index 1948be7ac8f8..0e53f9bd2be7 100644 > --- a/drivers/iommu/amd_iommu.c > +++ b/drivers/iommu/amd_iommu.c > @@ -3030,7 +3030,8 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, > } > > static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, > - phys_addr_t paddr, size_t page_size, int iommu_prot) > + phys_addr_t paddr, size_t page_size, int iommu_prot, > + gfp_t gfp) > { > struct protection_domain *domain = to_pdomain(dom); > int prot = 0; > diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c > index e7f49fd1a7ba..acc0eae7963f 100644 > --- a/drivers/iommu/arm-smmu-v3.c > +++ b/drivers/iommu/arm-smmu-v3.c > @@ -1975,7 +1975,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > > diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c > index aa06498f291d..05f42bdee494 100644 > --- a/drivers/iommu/arm-smmu.c > +++ b/drivers/iommu/arm-smmu.c > @@ -1284,7 +1284,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; > struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index d991d40f797f..2712fbc68b28 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -469,7 +469,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > if (!iova) > return DMA_MAPPING_ERROR; > > - if (iommu_map(domain, iova, phys - iova_off, size, prot)) { > + if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) { > iommu_dma_free_iova(cookie, iova, size); > return DMA_MAPPING_ERROR; > } > @@ -613,7 +613,7 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, > arch_dma_prep_coherent(sg_page(sg), sg->length); > } > > - if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) > + if (iommu_map_sg_atomic(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) > < size) > goto out_free_sg; > > @@ -873,7 +873,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, > * We'll leave any physical concatenation to the IOMMU driver's > * implementation - it knows better than we do. > */ > - if (iommu_map_sg(domain, iova, sg, nents, prot) < iova_len) > + if (iommu_map_sg_atomic(domain, iova, sg, nents, prot) < iova_len) > goto out_free_iova; > > return __finalise_sg(dev, sg, nents, iova); > diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c > index 1934c16a5abc..b7dd46884692 100644 > --- a/drivers/iommu/exynos-iommu.c > +++ b/drivers/iommu/exynos-iommu.c > @@ -1073,7 +1073,7 @@ static int lv2set_page(sysmmu_pte_t *pent, phys_addr_t paddr, size_t size, > */ > static int exynos_iommu_map(struct iommu_domain *iommu_domain, > unsigned long l_iova, phys_addr_t paddr, size_t size, > - int prot) > + int prot, gfp_t gfp) > { > struct exynos_iommu_domain *domain = to_exynos_domain(iommu_domain); > sysmmu_pte_t *entry; > diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c > index 12d094d08c0a..0e7cb0cbbd46 100644 > --- a/drivers/iommu/intel-iommu.c > +++ b/drivers/iommu/intel-iommu.c > @@ -5113,7 +5113,7 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain, > > static int intel_iommu_map(struct iommu_domain *domain, > unsigned long iova, phys_addr_t hpa, > - size_t size, int iommu_prot) > + size_t size, int iommu_prot, gfp_t gfp) > { > struct dmar_domain *dmar_domain = to_dmar_domain(domain); > u64 max_addr; > diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c > index 0c674d80c37f..133ec8e00a3a 100644 > --- a/drivers/iommu/iommu.c > +++ b/drivers/iommu/iommu.c > @@ -1799,8 +1799,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, > return pgsize; > } > > -int iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > +int __iommu_map(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > const struct iommu_ops *ops = domain->ops; > unsigned long orig_iova = iova; > @@ -1837,8 +1837,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, > > pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", > iova, &paddr, pgsize); > + ret = ops->map(domain, iova, paddr, pgsize, prot, gfp); > > - ret = ops->map(domain, iova, paddr, pgsize, prot); > if (ret) > break; > > @@ -1858,8 +1858,22 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, > > return ret; > } > + > +int iommu_map(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot) > +{ > + might_sleep(); > + return __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL); > +} > EXPORT_SYMBOL_GPL(iommu_map); > > +int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot) > +{ > + return __iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC); > +} > +EXPORT_SYMBOL_GPL(iommu_map_atomic); > + > static size_t __iommu_unmap(struct iommu_domain *domain, > unsigned long iova, size_t size, > bool sync) > @@ -1934,8 +1948,9 @@ size_t iommu_unmap_fast(struct iommu_domain *domain, > } > EXPORT_SYMBOL_GPL(iommu_unmap_fast); > > -size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > - struct scatterlist *sg, unsigned int nents, int prot) > +size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot, > + gfp_t gfp) > { > size_t len = 0, mapped = 0; > phys_addr_t start; > @@ -1946,7 +1961,9 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > phys_addr_t s_phys = sg_phys(sg); > > if (len && s_phys != start + len) { > - ret = iommu_map(domain, iova + mapped, start, len, prot); > + ret = __iommu_map(domain, iova + mapped, start, > + len, prot, gfp); > + > if (ret) > goto out_err; > > @@ -1974,8 +1991,22 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > return 0; > > } > + > +size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot) > +{ > + might_sleep(); > + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_KERNEL); > +} > EXPORT_SYMBOL_GPL(iommu_map_sg); > > +size_t iommu_map_sg_atomic(struct iommu_domain *domain, unsigned long iova, > + struct scatterlist *sg, unsigned int nents, int prot) > +{ > + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_ATOMIC); > +} > +EXPORT_SYMBOL_GPL(iommu_map_sg_atomic); > + > int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, > phys_addr_t paddr, u64 size, int prot) > { > diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c > index ad0098c0c87c..41572b40b844 100644 > --- a/drivers/iommu/ipmmu-vmsa.c > +++ b/drivers/iommu/ipmmu-vmsa.c > @@ -722,7 +722,7 @@ static void ipmmu_detach_device(struct iommu_domain *io_domain, > } > > static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain); > > diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c > index 3df9266abe65..5fa915c5d541 100644 > --- a/drivers/iommu/msm_iommu.c > +++ b/drivers/iommu/msm_iommu.c > @@ -495,7 +495,7 @@ static void msm_iommu_detach_dev(struct iommu_domain *domain, > } > > static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t pa, size_t len, int prot) > + phys_addr_t pa, size_t len, int prot, gfp_t gfp) > { > struct msm_priv *priv = to_msm_priv(domain); > unsigned long flags; > diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c > index 82e4be4dfdaf..53371b9e9af5 100644 > --- a/drivers/iommu/mtk_iommu.c > +++ b/drivers/iommu/mtk_iommu.c > @@ -356,7 +356,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, > } > > static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct mtk_iommu_domain *dom = to_mtk_domain(domain); > unsigned long flags; > diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c > index abeeac488372..453e704bee3d 100644 > --- a/drivers/iommu/mtk_iommu_v1.c > +++ b/drivers/iommu/mtk_iommu_v1.c > @@ -295,7 +295,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, > } > > static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct mtk_iommu_domain *dom = to_mtk_domain(domain); > unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT; > diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c > index 99a9ff3e7f71..3c530768200e 100644 > --- a/drivers/iommu/omap-iommu.c > +++ b/drivers/iommu/omap-iommu.c > @@ -1339,7 +1339,7 @@ static u32 iotlb_init_entry(struct iotlb_entry *e, u32 da, u32 pa, int pgsz) > } > > static int omap_iommu_map(struct iommu_domain *domain, unsigned long da, > - phys_addr_t pa, size_t bytes, int prot) > + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) > { > struct omap_iommu_domain *omap_domain = to_omap_domain(domain); > struct device *dev = omap_domain->dev; > diff --git a/drivers/iommu/qcom_iommu.c b/drivers/iommu/qcom_iommu.c > index fb45486c6d14..b373c0f18ad5 100644 > --- a/drivers/iommu/qcom_iommu.c > +++ b/drivers/iommu/qcom_iommu.c > @@ -400,7 +400,7 @@ static void qcom_iommu_detach_dev(struct iommu_domain *domain, struct device *de > } > > static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > int ret; > unsigned long flags; > diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c > index dc26d74d79c2..b06d2986b52e 100644 > --- a/drivers/iommu/rockchip-iommu.c > +++ b/drivers/iommu/rockchip-iommu.c > @@ -757,7 +757,7 @@ static int rk_iommu_map_iova(struct rk_iommu_domain *rk_domain, u32 *pte_addr, > } > > static int rk_iommu_map(struct iommu_domain *domain, unsigned long _iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct rk_iommu_domain *rk_domain = to_rk_domain(domain); > unsigned long flags; > diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c > index 22d4db302c1c..efa6aa68521d 100644 > --- a/drivers/iommu/s390-iommu.c > +++ b/drivers/iommu/s390-iommu.c > @@ -265,7 +265,7 @@ static int s390_iommu_update_trans(struct s390_domain *s390_domain, > } > > static int s390_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct s390_domain *s390_domain = to_s390_domain(domain); > int flags = ZPCI_PTE_VALID, rc = 0; > diff --git a/drivers/iommu/tegra-gart.c b/drivers/iommu/tegra-gart.c > index 6d40bc1b38bf..43c725243e8a 100644 > --- a/drivers/iommu/tegra-gart.c > +++ b/drivers/iommu/tegra-gart.c > @@ -178,7 +178,7 @@ static inline int __gart_iommu_map(struct gart_device *gart, unsigned long iova, > } > > static int gart_iommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t pa, size_t bytes, int prot) > + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) > { > struct gart_device *gart = gart_handle; > int ret; > diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c > index c4a652b227f8..6ec58a2bcc13 100644 > --- a/drivers/iommu/tegra-smmu.c > +++ b/drivers/iommu/tegra-smmu.c > @@ -650,7 +650,7 @@ static void tegra_smmu_set_pte(struct tegra_smmu_as *as, unsigned long iova, > } > > static int tegra_smmu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > struct tegra_smmu_as *as = to_smmu_as(domain); > dma_addr_t pte_dma; > diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c > index 80a740df0737..15443457f3fc 100644 > --- a/drivers/iommu/virtio-iommu.c > +++ b/drivers/iommu/virtio-iommu.c > @@ -713,7 +713,7 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev) > } > > static int viommu_map(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot) > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) > { > int ret; > u32 flags; > diff --git a/include/linux/iommu.h b/include/linux/iommu.h > index fdc355ccc570..89af6271c506 100644 > --- a/include/linux/iommu.h > +++ b/include/linux/iommu.h > @@ -240,7 +240,7 @@ struct iommu_ops { > int (*attach_dev)(struct iommu_domain *domain, struct device *dev); > void (*detach_dev)(struct iommu_domain *domain, struct device *dev); > int (*map)(struct iommu_domain *domain, unsigned long iova, > - phys_addr_t paddr, size_t size, int prot); > + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); > size_t (*unmap)(struct iommu_domain *domain, unsigned long iova, > size_t size); > void (*flush_iotlb_all)(struct iommu_domain *domain); > @@ -399,12 +399,17 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); > extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); > extern int iommu_map(struct iommu_domain *domain, unsigned long iova, > phys_addr_t paddr, size_t size, int prot); > +extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, > + phys_addr_t paddr, size_t size, int prot); > extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, > size_t size); > extern size_t iommu_unmap_fast(struct iommu_domain *domain, > unsigned long iova, size_t size); > extern size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, > struct scatterlist *sg,unsigned int nents, int prot); > +extern size_t iommu_map_sg_atomic(struct iommu_domain *domain, > + unsigned long iova, struct scatterlist *sg, > + unsigned int nents, int prot); > extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova); > extern void iommu_set_fault_handler(struct iommu_domain *domain, > iommu_fault_handler_t handler, void *token); > @@ -614,6 +619,13 @@ static inline int iommu_map(struct iommu_domain *domain, unsigned long iova, > return -ENODEV; > } > > +static inline int iommu_map_atomic(struct iommu_domain *domain, > + unsigned long iova, phys_addr_t paddr, > + size_t size, int prot) > +{ > + return -ENODEV; > +} > + > static inline size_t iommu_unmap(struct iommu_domain *domain, > unsigned long iova, size_t size) > { > @@ -633,6 +645,13 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain, > return 0; > } > > +static inline size_t iommu_map_sg_atomic(struct iommu_domain *domain, > + unsigned long iova, struct scatterlist *sg, > + unsigned int nents, int prot) > +{ > + return 0; > +} > + > static inline void iommu_flush_tlb_all(struct iommu_domain *domain) > { > } > ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map 2019-08-15 11:09 ` Tom Murphy ` (4 preceding siblings ...) (?) @ 2019-08-20 9:41 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:41 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy Looks good, Reviewed-by: Christoph Hellwig <hch@lst.de> ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-20 9:41 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:41 UTC (permalink / raw) To: Tom Murphy Cc: iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Kukjin Kim Looks good, Reviewed-by: Christoph Hellwig <hch@lst.de> ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map @ 2019-08-20 9:41 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:41 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy Looks good, Reviewed-by: Christoph Hellwig <hch@lst.de> _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map @ 2019-08-20 9:41 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:41 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy Looks good, Reviewed-by: Christoph Hellwig <hch@lst.de> _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map @ 2019-08-20 9:41 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:41 UTC (permalink / raw) To: Tom Murphy Cc: iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Kukjin Kim, Robin Murphy Looks good, Reviewed-by: Christoph Hellwig <hch@lst.de> ^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map 2019-08-15 11:09 ` Tom Murphy ` (5 preceding siblings ...) (?) @ 2019-08-15 11:09 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Rob Clark, Kukjin Kim, Robin Add a gfp_t parameter to the iommu_ops::map function. Remove the needless locking in the AMD iommu driver. The iommu_ops::map function (or the iommu_map function which calls it) was always supposed to be sleepable (according to Joerg's comment in this thread: https://lore.kernel.org/patchwork/patch/977520/ ) and so should probably have had a "might_sleep()" since it was written. However currently the dma-iommu api can call iommu_map in an atomic context, which it shouldn't do. This doesn't cause any problems because any iommu driver which uses the dma-iommu api uses gfp_atomic in it's iommu_ops::map function. But doing this wastes the memory allocators atomic pools. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/amd_iommu.c | 3 ++- drivers/iommu/arm-smmu-v3.c | 2 +- drivers/iommu/arm-smmu.c | 2 +- drivers/iommu/dma-iommu.c | 6 ++--- drivers/iommu/exynos-iommu.c | 2 +- drivers/iommu/intel-iommu.c | 2 +- drivers/iommu/iommu.c | 43 +++++++++++++++++++++++++++++----- drivers/iommu/ipmmu-vmsa.c | 2 +- drivers/iommu/msm_iommu.c | 2 +- drivers/iommu/mtk_iommu.c | 2 +- drivers/iommu/mtk_iommu_v1.c | 2 +- drivers/iommu/omap-iommu.c | 2 +- drivers/iommu/qcom_iommu.c | 2 +- drivers/iommu/rockchip-iommu.c | 2 +- drivers/iommu/s390-iommu.c | 2 +- drivers/iommu/tegra-gart.c | 2 +- drivers/iommu/tegra-smmu.c | 2 +- drivers/iommu/virtio-iommu.c | 2 +- include/linux/iommu.h | 21 ++++++++++++++++- 19 files changed, 77 insertions(+), 26 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 1948be7ac8f8..0e53f9bd2be7 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -3030,7 +3030,8 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, } static int amd_iommu_map(struct iommu_domain *dom, unsigned long iova, - phys_addr_t paddr, size_t page_size, int iommu_prot) + phys_addr_t paddr, size_t page_size, int iommu_prot, + gfp_t gfp) { struct protection_domain *domain = to_pdomain(dom); int prot = 0; diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index e7f49fd1a7ba..acc0eae7963f 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -1975,7 +1975,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; diff --git a/drivers/iommu/arm-smmu.c b/drivers/iommu/arm-smmu.c index aa06498f291d..05f42bdee494 100644 --- a/drivers/iommu/arm-smmu.c +++ b/drivers/iommu/arm-smmu.c @@ -1284,7 +1284,7 @@ static int arm_smmu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int arm_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct io_pgtable_ops *ops = to_smmu_domain(domain)->pgtbl_ops; struct arm_smmu_device *smmu = to_smmu_domain(domain)->smmu; diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index d991d40f797f..2712fbc68b28 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -469,7 +469,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, if (!iova) return DMA_MAPPING_ERROR; - if (iommu_map(domain, iova, phys - iova_off, size, prot)) { + if (iommu_map_atomic(domain, iova, phys - iova_off, size, prot)) { iommu_dma_free_iova(cookie, iova, size); return DMA_MAPPING_ERROR; } @@ -613,7 +613,7 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, arch_dma_prep_coherent(sg_page(sg), sg->length); } - if (iommu_map_sg(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) + if (iommu_map_sg_atomic(domain, iova, sgt.sgl, sgt.orig_nents, ioprot) < size) goto out_free_sg; @@ -873,7 +873,7 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, * We'll leave any physical concatenation to the IOMMU driver's * implementation - it knows better than we do. */ - if (iommu_map_sg(domain, iova, sg, nents, prot) < iova_len) + if (iommu_map_sg_atomic(domain, iova, sg, nents, prot) < iova_len) goto out_free_iova; return __finalise_sg(dev, sg, nents, iova); diff --git a/drivers/iommu/exynos-iommu.c b/drivers/iommu/exynos-iommu.c index 1934c16a5abc..b7dd46884692 100644 --- a/drivers/iommu/exynos-iommu.c +++ b/drivers/iommu/exynos-iommu.c @@ -1073,7 +1073,7 @@ static int lv2set_page(sysmmu_pte_t *pent, phys_addr_t paddr, size_t size, */ static int exynos_iommu_map(struct iommu_domain *iommu_domain, unsigned long l_iova, phys_addr_t paddr, size_t size, - int prot) + int prot, gfp_t gfp) { struct exynos_iommu_domain *domain = to_exynos_domain(iommu_domain); sysmmu_pte_t *entry; diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 12d094d08c0a..0e7cb0cbbd46 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -5113,7 +5113,7 @@ static void intel_iommu_aux_detach_device(struct iommu_domain *domain, static int intel_iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t hpa, - size_t size, int iommu_prot) + size_t size, int iommu_prot, gfp_t gfp) { struct dmar_domain *dmar_domain = to_dmar_domain(domain); u64 max_addr; diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 0c674d80c37f..133ec8e00a3a 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -1799,8 +1799,8 @@ static size_t iommu_pgsize(struct iommu_domain *domain, return pgsize; } -int iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) +int __iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { const struct iommu_ops *ops = domain->ops; unsigned long orig_iova = iova; @@ -1837,8 +1837,8 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, pr_debug("mapping: iova 0x%lx pa %pa pgsize 0x%zx\n", iova, &paddr, pgsize); + ret = ops->map(domain, iova, paddr, pgsize, prot, gfp); - ret = ops->map(domain, iova, paddr, pgsize, prot); if (ret) break; @@ -1858,8 +1858,22 @@ int iommu_map(struct iommu_domain *domain, unsigned long iova, return ret; } + +int iommu_map(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot) +{ + might_sleep(); + return __iommu_map(domain, iova, paddr, size, prot, GFP_KERNEL); +} EXPORT_SYMBOL_GPL(iommu_map); +int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot) +{ + return __iommu_map(domain, iova, paddr, size, prot, GFP_ATOMIC); +} +EXPORT_SYMBOL_GPL(iommu_map_atomic); + static size_t __iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size, bool sync) @@ -1934,8 +1948,9 @@ size_t iommu_unmap_fast(struct iommu_domain *domain, } EXPORT_SYMBOL_GPL(iommu_unmap_fast); -size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, - struct scatterlist *sg, unsigned int nents, int prot) +size_t __iommu_map_sg(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot, + gfp_t gfp) { size_t len = 0, mapped = 0; phys_addr_t start; @@ -1946,7 +1961,9 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, phys_addr_t s_phys = sg_phys(sg); if (len && s_phys != start + len) { - ret = iommu_map(domain, iova + mapped, start, len, prot); + ret = __iommu_map(domain, iova + mapped, start, + len, prot, gfp); + if (ret) goto out_err; @@ -1974,8 +1991,22 @@ size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, return 0; } + +size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + might_sleep(); + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_KERNEL); +} EXPORT_SYMBOL_GPL(iommu_map_sg); +size_t iommu_map_sg_atomic(struct iommu_domain *domain, unsigned long iova, + struct scatterlist *sg, unsigned int nents, int prot) +{ + return __iommu_map_sg(domain, iova, sg, nents, prot, GFP_ATOMIC); +} +EXPORT_SYMBOL_GPL(iommu_map_sg_atomic); + int iommu_domain_window_enable(struct iommu_domain *domain, u32 wnd_nr, phys_addr_t paddr, u64 size, int prot) { diff --git a/drivers/iommu/ipmmu-vmsa.c b/drivers/iommu/ipmmu-vmsa.c index ad0098c0c87c..41572b40b844 100644 --- a/drivers/iommu/ipmmu-vmsa.c +++ b/drivers/iommu/ipmmu-vmsa.c @@ -722,7 +722,7 @@ static void ipmmu_detach_device(struct iommu_domain *io_domain, } static int ipmmu_map(struct iommu_domain *io_domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct ipmmu_vmsa_domain *domain = to_vmsa_domain(io_domain); diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c index 3df9266abe65..5fa915c5d541 100644 --- a/drivers/iommu/msm_iommu.c +++ b/drivers/iommu/msm_iommu.c @@ -495,7 +495,7 @@ static void msm_iommu_detach_dev(struct iommu_domain *domain, } static int msm_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t pa, size_t len, int prot) + phys_addr_t pa, size_t len, int prot, gfp_t gfp) { struct msm_priv *priv = to_msm_priv(domain); unsigned long flags; diff --git a/drivers/iommu/mtk_iommu.c b/drivers/iommu/mtk_iommu.c index 82e4be4dfdaf..53371b9e9af5 100644 --- a/drivers/iommu/mtk_iommu.c +++ b/drivers/iommu/mtk_iommu.c @@ -356,7 +356,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, } static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct mtk_iommu_domain *dom = to_mtk_domain(domain); unsigned long flags; diff --git a/drivers/iommu/mtk_iommu_v1.c b/drivers/iommu/mtk_iommu_v1.c index abeeac488372..453e704bee3d 100644 --- a/drivers/iommu/mtk_iommu_v1.c +++ b/drivers/iommu/mtk_iommu_v1.c @@ -295,7 +295,7 @@ static void mtk_iommu_detach_device(struct iommu_domain *domain, } static int mtk_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct mtk_iommu_domain *dom = to_mtk_domain(domain); unsigned int page_num = size >> MT2701_IOMMU_PAGE_SHIFT; diff --git a/drivers/iommu/omap-iommu.c b/drivers/iommu/omap-iommu.c index 99a9ff3e7f71..3c530768200e 100644 --- a/drivers/iommu/omap-iommu.c +++ b/drivers/iommu/omap-iommu.c @@ -1339,7 +1339,7 @@ static u32 iotlb_init_entry(struct iotlb_entry *e, u32 da, u32 pa, int pgsz) } static int omap_iommu_map(struct iommu_domain *domain, unsigned long da, - phys_addr_t pa, size_t bytes, int prot) + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) { struct omap_iommu_domain *omap_domain = to_omap_domain(domain); struct device *dev = omap_domain->dev; diff --git a/drivers/iommu/qcom_iommu.c b/drivers/iommu/qcom_iommu.c index fb45486c6d14..b373c0f18ad5 100644 --- a/drivers/iommu/qcom_iommu.c +++ b/drivers/iommu/qcom_iommu.c @@ -400,7 +400,7 @@ static void qcom_iommu_detach_dev(struct iommu_domain *domain, struct device *de } static int qcom_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { int ret; unsigned long flags; diff --git a/drivers/iommu/rockchip-iommu.c b/drivers/iommu/rockchip-iommu.c index dc26d74d79c2..b06d2986b52e 100644 --- a/drivers/iommu/rockchip-iommu.c +++ b/drivers/iommu/rockchip-iommu.c @@ -757,7 +757,7 @@ static int rk_iommu_map_iova(struct rk_iommu_domain *rk_domain, u32 *pte_addr, } static int rk_iommu_map(struct iommu_domain *domain, unsigned long _iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct rk_iommu_domain *rk_domain = to_rk_domain(domain); unsigned long flags; diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c index 22d4db302c1c..efa6aa68521d 100644 --- a/drivers/iommu/s390-iommu.c +++ b/drivers/iommu/s390-iommu.c @@ -265,7 +265,7 @@ static int s390_iommu_update_trans(struct s390_domain *s390_domain, } static int s390_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct s390_domain *s390_domain = to_s390_domain(domain); int flags = ZPCI_PTE_VALID, rc = 0; diff --git a/drivers/iommu/tegra-gart.c b/drivers/iommu/tegra-gart.c index 6d40bc1b38bf..43c725243e8a 100644 --- a/drivers/iommu/tegra-gart.c +++ b/drivers/iommu/tegra-gart.c @@ -178,7 +178,7 @@ static inline int __gart_iommu_map(struct gart_device *gart, unsigned long iova, } static int gart_iommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t pa, size_t bytes, int prot) + phys_addr_t pa, size_t bytes, int prot, gfp_t gfp) { struct gart_device *gart = gart_handle; int ret; diff --git a/drivers/iommu/tegra-smmu.c b/drivers/iommu/tegra-smmu.c index c4a652b227f8..6ec58a2bcc13 100644 --- a/drivers/iommu/tegra-smmu.c +++ b/drivers/iommu/tegra-smmu.c @@ -650,7 +650,7 @@ static void tegra_smmu_set_pte(struct tegra_smmu_as *as, unsigned long iova, } static int tegra_smmu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { struct tegra_smmu_as *as = to_smmu_as(domain); dma_addr_t pte_dma; diff --git a/drivers/iommu/virtio-iommu.c b/drivers/iommu/virtio-iommu.c index 80a740df0737..15443457f3fc 100644 --- a/drivers/iommu/virtio-iommu.c +++ b/drivers/iommu/virtio-iommu.c @@ -713,7 +713,7 @@ static int viommu_attach_dev(struct iommu_domain *domain, struct device *dev) } static int viommu_map(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot) + phys_addr_t paddr, size_t size, int prot, gfp_t gfp) { int ret; u32 flags; diff --git a/include/linux/iommu.h b/include/linux/iommu.h index fdc355ccc570..89af6271c506 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -240,7 +240,7 @@ struct iommu_ops { int (*attach_dev)(struct iommu_domain *domain, struct device *dev); void (*detach_dev)(struct iommu_domain *domain, struct device *dev); int (*map)(struct iommu_domain *domain, unsigned long iova, - phys_addr_t paddr, size_t size, int prot); + phys_addr_t paddr, size_t size, int prot, gfp_t gfp); size_t (*unmap)(struct iommu_domain *domain, unsigned long iova, size_t size); void (*flush_iotlb_all)(struct iommu_domain *domain); @@ -399,12 +399,17 @@ extern struct iommu_domain *iommu_get_domain_for_dev(struct device *dev); extern struct iommu_domain *iommu_get_dma_domain(struct device *dev); extern int iommu_map(struct iommu_domain *domain, unsigned long iova, phys_addr_t paddr, size_t size, int prot); +extern int iommu_map_atomic(struct iommu_domain *domain, unsigned long iova, + phys_addr_t paddr, size_t size, int prot); extern size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_unmap_fast(struct iommu_domain *domain, unsigned long iova, size_t size); extern size_t iommu_map_sg(struct iommu_domain *domain, unsigned long iova, struct scatterlist *sg,unsigned int nents, int prot); +extern size_t iommu_map_sg_atomic(struct iommu_domain *domain, + unsigned long iova, struct scatterlist *sg, + unsigned int nents, int prot); extern phys_addr_t iommu_iova_to_phys(struct iommu_domain *domain, dma_addr_t iova); extern void iommu_set_fault_handler(struct iommu_domain *domain, iommu_fault_handler_t handler, void *token); @@ -614,6 +619,13 @@ static inline int iommu_map(struct iommu_domain *domain, unsigned long iova, return -ENODEV; } +static inline int iommu_map_atomic(struct iommu_domain *domain, + unsigned long iova, phys_addr_t paddr, + size_t size, int prot) +{ + return -ENODEV; +} + static inline size_t iommu_unmap(struct iommu_domain *domain, unsigned long iova, size_t size) { @@ -633,6 +645,13 @@ static inline size_t iommu_map_sg(struct iommu_domain *domain, return 0; } +static inline size_t iommu_map_sg_atomic(struct iommu_domain *domain, + unsigned long iova, struct scatterlist *sg, + unsigned int nents, int prot) +{ + return 0; +} + static inline void iommu_flush_tlb_all(struct iommu_domain *domain) { } -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices 2019-08-15 11:09 ` Tom Murphy ` (6 preceding siblings ...) (?) @ 2019-08-15 11:09 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Rob Clark, Kukjin Kim, Robin Handle devices which defer their attach to the iommu in the dma-iommu api Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 2712fbc68b28..906b7fa14d3c 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -22,6 +22,7 @@ #include <linux/pci.h> #include <linux/scatterlist.h> #include <linux/vmalloc.h> +#include <linux/crash_dump.h> struct iommu_dma_msi_page { struct list_head list; @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, return iova_reserve_iommu_regions(dev, domain); } +static int handle_deferred_device(struct device *dev, + struct iommu_domain *domain) +{ + const struct iommu_ops *ops = domain->ops; + + if (!is_kdump_kernel()) + return 0; + + if (unlikely(ops->is_attach_deferred && + ops->is_attach_deferred(domain, dev))) + return iommu_attach_device(domain, dev); + + return 0; +} + /** * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API * page flags. @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size_t iova_off = iova_offset(iovad, phys); dma_addr_t iova; + if (unlikely(handle_deferred_device(dev, domain))) + return DMA_MAPPING_ERROR; + size = iova_align(iovad, size + iova_off); iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); @@ -581,6 +600,9 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, *dma_handle = DMA_MAPPING_ERROR; + if (unlikely(handle_deferred_device(dev, domain))) + return NULL; + min_size = alloc_sizes & -alloc_sizes; if (min_size < PAGE_SIZE) { min_size = PAGE_SIZE; @@ -713,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, int prot = dma_info_to_prot(dir, coherent, attrs); dma_addr_t dma_handle; - dma_handle =__iommu_dma_map(dev, phys, size, prot); + dma_handle = __iommu_dma_map(dev, phys, size, prot); if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && dma_handle != DMA_MAPPING_ERROR) arch_sync_dma_for_device(dev, phys, size, dir); @@ -823,6 +845,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, unsigned long mask = dma_get_seg_boundary(dev); int i; + if (unlikely(handle_deferred_device(dev, domain))) + return 0; + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) iommu_dma_sync_sg_for_device(dev, sg, nents, dir); -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-15 11:09 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Tom Murphy, Joerg Roedel, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek Handle devices which defer their attach to the iommu in the dma-iommu api Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 2712fbc68b28..906b7fa14d3c 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -22,6 +22,7 @@ #include <linux/pci.h> #include <linux/scatterlist.h> #include <linux/vmalloc.h> +#include <linux/crash_dump.h> struct iommu_dma_msi_page { struct list_head list; @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, return iova_reserve_iommu_regions(dev, domain); } +static int handle_deferred_device(struct device *dev, + struct iommu_domain *domain) +{ + const struct iommu_ops *ops = domain->ops; + + if (!is_kdump_kernel()) + return 0; + + if (unlikely(ops->is_attach_deferred && + ops->is_attach_deferred(domain, dev))) + return iommu_attach_device(domain, dev); + + return 0; +} + /** * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API * page flags. @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size_t iova_off = iova_offset(iovad, phys); dma_addr_t iova; + if (unlikely(handle_deferred_device(dev, domain))) + return DMA_MAPPING_ERROR; + size = iova_align(iovad, size + iova_off); iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); @@ -581,6 +600,9 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, *dma_handle = DMA_MAPPING_ERROR; + if (unlikely(handle_deferred_device(dev, domain))) + return NULL; + min_size = alloc_sizes & -alloc_sizes; if (min_size < PAGE_SIZE) { min_size = PAGE_SIZE; @@ -713,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, int prot = dma_info_to_prot(dir, coherent, attrs); dma_addr_t dma_handle; - dma_handle =__iommu_dma_map(dev, phys, size, prot); + dma_handle = __iommu_dma_map(dev, phys, size, prot); if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && dma_handle != DMA_MAPPING_ERROR) arch_sync_dma_for_device(dev, phys, size, dir); @@ -823,6 +845,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, unsigned long mask = dma_get_seg_boundary(dev); int i; + if (unlikely(handle_deferred_device(dev, domain))) + return 0; + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) iommu_dma_sync_sg_for_device(dev, sg, nents, dir); -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Rob Clark, Kukjin Kim, Robin Murphy Handle devices which defer their attach to the iommu in the dma-iommu api Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 2712fbc68b28..906b7fa14d3c 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -22,6 +22,7 @@ #include <linux/pci.h> #include <linux/scatterlist.h> #include <linux/vmalloc.h> +#include <linux/crash_dump.h> struct iommu_dma_msi_page { struct list_head list; @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, return iova_reserve_iommu_regions(dev, domain); } +static int handle_deferred_device(struct device *dev, + struct iommu_domain *domain) +{ + const struct iommu_ops *ops = domain->ops; + + if (!is_kdump_kernel()) + return 0; + + if (unlikely(ops->is_attach_deferred && + ops->is_attach_deferred(domain, dev))) + return iommu_attach_device(domain, dev); + + return 0; +} + /** * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API * page flags. @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size_t iova_off = iova_offset(iovad, phys); dma_addr_t iova; + if (unlikely(handle_deferred_device(dev, domain))) + return DMA_MAPPING_ERROR; + size = iova_align(iovad, size + iova_off); iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); @@ -581,6 +600,9 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, *dma_handle = DMA_MAPPING_ERROR; + if (unlikely(handle_deferred_device(dev, domain))) + return NULL; + min_size = alloc_sizes & -alloc_sizes; if (min_size < PAGE_SIZE) { min_size = PAGE_SIZE; @@ -713,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, int prot = dma_info_to_prot(dir, coherent, attrs); dma_addr_t dma_handle; - dma_handle =__iommu_dma_map(dev, phys, size, prot); + dma_handle = __iommu_dma_map(dev, phys, size, prot); if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && dma_handle != DMA_MAPPING_ERROR) arch_sync_dma_for_device(dev, phys, size, dir); @@ -823,6 +845,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, unsigned long mask = dma_get_seg_boundary(dev); int i; + if (unlikely(handle_deferred_device(dev, domain))) + return 0; + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) iommu_dma_sync_sg_for_device(dev, sg, nents, dir); -- 2.20.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Kukjin Kim, Robin Murphy Handle devices which defer their attach to the iommu in the dma-iommu api Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 2712fbc68b28..906b7fa14d3c 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -22,6 +22,7 @@ #include <linux/pci.h> #include <linux/scatterlist.h> #include <linux/vmalloc.h> +#include <linux/crash_dump.h> struct iommu_dma_msi_page { struct list_head list; @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, return iova_reserve_iommu_regions(dev, domain); } +static int handle_deferred_device(struct device *dev, + struct iommu_domain *domain) +{ + const struct iommu_ops *ops = domain->ops; + + if (!is_kdump_kernel()) + return 0; + + if (unlikely(ops->is_attach_deferred && + ops->is_attach_deferred(domain, dev))) + return iommu_attach_device(domain, dev); + + return 0; +} + /** * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API * page flags. @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size_t iova_off = iova_offset(iovad, phys); dma_addr_t iova; + if (unlikely(handle_deferred_device(dev, domain))) + return DMA_MAPPING_ERROR; + size = iova_align(iovad, size + iova_off); iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); @@ -581,6 +600,9 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, *dma_handle = DMA_MAPPING_ERROR; + if (unlikely(handle_deferred_device(dev, domain))) + return NULL; + min_size = alloc_sizes & -alloc_sizes; if (min_size < PAGE_SIZE) { min_size = PAGE_SIZE; @@ -713,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, int prot = dma_info_to_prot(dir, coherent, attrs); dma_addr_t dma_handle; - dma_handle =__iommu_dma_map(dev, phys, size, prot); + dma_handle = __iommu_dma_map(dev, phys, size, prot); if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && dma_handle != DMA_MAPPING_ERROR) arch_sync_dma_for_device(dev, phys, size, dir); @@ -823,6 +845,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, unsigned long mask = dma_get_seg_boundary(dev); int i; + if (unlikely(handle_deferred_device(dev, domain))) + return 0; + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) iommu_dma_sync_sg_for_device(dev, sg, nents, dir); -- 2.20.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Tom Murphy, Joerg Roedel, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390, linux-tegra, virtualization Handle devices which defer their attach to the iommu in the dma-iommu api Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- 1 file changed, 26 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 2712fbc68b28..906b7fa14d3c 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -22,6 +22,7 @@ #include <linux/pci.h> #include <linux/scatterlist.h> #include <linux/vmalloc.h> +#include <linux/crash_dump.h> struct iommu_dma_msi_page { struct list_head list; @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, return iova_reserve_iommu_regions(dev, domain); } +static int handle_deferred_device(struct device *dev, + struct iommu_domain *domain) +{ + const struct iommu_ops *ops = domain->ops; + + if (!is_kdump_kernel()) + return 0; + + if (unlikely(ops->is_attach_deferred && + ops->is_attach_deferred(domain, dev))) + return iommu_attach_device(domain, dev); + + return 0; +} + /** * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API * page flags. @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size_t iova_off = iova_offset(iovad, phys); dma_addr_t iova; + if (unlikely(handle_deferred_device(dev, domain))) + return DMA_MAPPING_ERROR; + size = iova_align(iovad, size + iova_off); iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); @@ -581,6 +600,9 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, *dma_handle = DMA_MAPPING_ERROR; + if (unlikely(handle_deferred_device(dev, domain))) + return NULL; + min_size = alloc_sizes & -alloc_sizes; if (min_size < PAGE_SIZE) { min_size = PAGE_SIZE; @@ -713,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, int prot = dma_info_to_prot(dir, coherent, attrs); dma_addr_t dma_handle; - dma_handle =__iommu_dma_map(dev, phys, size, prot); + dma_handle = __iommu_dma_map(dev, phys, size, prot); if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && dma_handle != DMA_MAPPING_ERROR) arch_sync_dma_for_device(dev, phys, size, dir); @@ -823,6 +845,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, unsigned long mask = dma_get_seg_boundary(dev); int i; + if (unlikely(handle_deferred_device(dev, domain))) + return 0; + if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) iommu_dma_sync_sg_for_device(dev, sg, nents, dir); -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-19 18:26 ` Robin Murphy -1 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:26 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Joerg Roedel, Will Deacon, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390 On 15/08/2019 12:09, Tom Murphy wrote: > Handle devices which defer their attach to the iommu in the dma-iommu api Other than nitpicking the name (I'd lean towards something like iommu_dma_deferred_attach), Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > 1 file changed, 26 insertions(+), 1 deletion(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 2712fbc68b28..906b7fa14d3c 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -22,6 +22,7 @@ > #include <linux/pci.h> > #include <linux/scatterlist.h> > #include <linux/vmalloc.h> > +#include <linux/crash_dump.h> > > struct iommu_dma_msi_page { > struct list_head list; > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > return iova_reserve_iommu_regions(dev, domain); > } > > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) > +{ > + const struct iommu_ops *ops = domain->ops; > + > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); > + > + return 0; > +} > + > /** > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > * page flags. > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > size_t iova_off = iova_offset(iovad, phys); > dma_addr_t iova; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return DMA_MAPPING_ERROR; > + > size = iova_align(iovad, size + iova_off); > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > @@ -581,6 +600,9 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, > > *dma_handle = DMA_MAPPING_ERROR; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return NULL; > + > min_size = alloc_sizes & -alloc_sizes; > if (min_size < PAGE_SIZE) { > min_size = PAGE_SIZE; > @@ -713,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, > int prot = dma_info_to_prot(dir, coherent, attrs); > dma_addr_t dma_handle; > > - dma_handle =__iommu_dma_map(dev, phys, size, prot); > + dma_handle = __iommu_dma_map(dev, phys, size, prot); > if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && > dma_handle != DMA_MAPPING_ERROR) > arch_sync_dma_for_device(dev, phys, size, dir); > @@ -823,6 +845,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, > unsigned long mask = dma_get_seg_boundary(dev); > int i; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return 0; > + > if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) > iommu_dma_sync_sg_for_device(dev, sg, nents, dir); > > ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-19 18:26 ` Robin Murphy 0 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:26 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, linux-kernel, Rob Clark, Kukjin Kim, David Woodhouse On 15/08/2019 12:09, Tom Murphy wrote: > Handle devices which defer their attach to the iommu in the dma-iommu api Other than nitpicking the name (I'd lean towards something like iommu_dma_deferred_attach), Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > 1 file changed, 26 insertions(+), 1 deletion(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 2712fbc68b28..906b7fa14d3c 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -22,6 +22,7 @@ > #include <linux/pci.h> > #include <linux/scatterlist.h> > #include <linux/vmalloc.h> > +#include <linux/crash_dump.h> > > struct iommu_dma_msi_page { > struct list_head list; > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > return iova_reserve_iommu_regions(dev, domain); > } > > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) > +{ > + const struct iommu_ops *ops = domain->ops; > + > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); > + > + return 0; > +} > + > /** > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > * page flags. > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > size_t iova_off = iova_offset(iovad, phys); > dma_addr_t iova; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return DMA_MAPPING_ERROR; > + > size = iova_align(iovad, size + iova_off); > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > @@ -581,6 +600,9 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, > > *dma_handle = DMA_MAPPING_ERROR; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return NULL; > + > min_size = alloc_sizes & -alloc_sizes; > if (min_size < PAGE_SIZE) { > min_size = PAGE_SIZE; > @@ -713,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, > int prot = dma_info_to_prot(dir, coherent, attrs); > dma_addr_t dma_handle; > > - dma_handle =__iommu_dma_map(dev, phys, size, prot); > + dma_handle = __iommu_dma_map(dev, phys, size, prot); > if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && > dma_handle != DMA_MAPPING_ERROR) > arch_sync_dma_for_device(dev, phys, size, dir); > @@ -823,6 +845,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, > unsigned long mask = dma_get_seg_boundary(dev); > int i; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return 0; > + > if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) > iommu_dma_sync_sg_for_device(dev, sg, nents, dir); > > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-19 18:26 ` Robin Murphy 0 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:26 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, linux-kernel, Kukjin Kim, David Woodhouse On 15/08/2019 12:09, Tom Murphy wrote: > Handle devices which defer their attach to the iommu in the dma-iommu api Other than nitpicking the name (I'd lean towards something like iommu_dma_deferred_attach), Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > 1 file changed, 26 insertions(+), 1 deletion(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 2712fbc68b28..906b7fa14d3c 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -22,6 +22,7 @@ > #include <linux/pci.h> > #include <linux/scatterlist.h> > #include <linux/vmalloc.h> > +#include <linux/crash_dump.h> > > struct iommu_dma_msi_page { > struct list_head list; > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > return iova_reserve_iommu_regions(dev, domain); > } > > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) > +{ > + const struct iommu_ops *ops = domain->ops; > + > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); > + > + return 0; > +} > + > /** > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > * page flags. > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > size_t iova_off = iova_offset(iovad, phys); > dma_addr_t iova; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return DMA_MAPPING_ERROR; > + > size = iova_align(iovad, size + iova_off); > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > @@ -581,6 +600,9 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, > > *dma_handle = DMA_MAPPING_ERROR; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return NULL; > + > min_size = alloc_sizes & -alloc_sizes; > if (min_size < PAGE_SIZE) { > min_size = PAGE_SIZE; > @@ -713,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, > int prot = dma_info_to_prot(dir, coherent, attrs); > dma_addr_t dma_handle; > > - dma_handle =__iommu_dma_map(dev, phys, size, prot); > + dma_handle = __iommu_dma_map(dev, phys, size, prot); > if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && > dma_handle != DMA_MAPPING_ERROR) > arch_sync_dma_for_device(dev, phys, size, dir); > @@ -823,6 +845,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, > unsigned long mask = dma_get_seg_boundary(dev); > int i; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return 0; > + > if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) > iommu_dma_sync_sg_for_device(dev, sg, nents, dir); > > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-19 18:26 ` Robin Murphy 0 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:26 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Joerg Roedel, Will Deacon, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390, linux-tegra, virtualization On 15/08/2019 12:09, Tom Murphy wrote: > Handle devices which defer their attach to the iommu in the dma-iommu api Other than nitpicking the name (I'd lean towards something like iommu_dma_deferred_attach), Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > 1 file changed, 26 insertions(+), 1 deletion(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 2712fbc68b28..906b7fa14d3c 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -22,6 +22,7 @@ > #include <linux/pci.h> > #include <linux/scatterlist.h> > #include <linux/vmalloc.h> > +#include <linux/crash_dump.h> > > struct iommu_dma_msi_page { > struct list_head list; > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > return iova_reserve_iommu_regions(dev, domain); > } > > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) > +{ > + const struct iommu_ops *ops = domain->ops; > + > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); > + > + return 0; > +} > + > /** > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > * page flags. > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > size_t iova_off = iova_offset(iovad, phys); > dma_addr_t iova; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return DMA_MAPPING_ERROR; > + > size = iova_align(iovad, size + iova_off); > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > @@ -581,6 +600,9 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, > > *dma_handle = DMA_MAPPING_ERROR; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return NULL; > + > min_size = alloc_sizes & -alloc_sizes; > if (min_size < PAGE_SIZE) { > min_size = PAGE_SIZE; > @@ -713,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, > int prot = dma_info_to_prot(dir, coherent, attrs); > dma_addr_t dma_handle; > > - dma_handle =__iommu_dma_map(dev, phys, size, prot); > + dma_handle = __iommu_dma_map(dev, phys, size, prot); > if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && > dma_handle != DMA_MAPPING_ERROR) > arch_sync_dma_for_device(dev, phys, size, dir); > @@ -823,6 +845,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, > unsigned long mask = dma_get_seg_boundary(dev); > int i; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return 0; > + > if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) > iommu_dma_sync_sg_for_device(dev, sg, nents, dir); > > ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices 2019-08-15 11:09 ` Tom Murphy ` (3 preceding siblings ...) (?) @ 2019-08-19 18:26 ` Robin Murphy -1 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:26 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, linux-kernel, Rob Clark, Kukjin Kim, David Woodhouse On 15/08/2019 12:09, Tom Murphy wrote: > Handle devices which defer their attach to the iommu in the dma-iommu api Other than nitpicking the name (I'd lean towards something like iommu_dma_deferred_attach), Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > 1 file changed, 26 insertions(+), 1 deletion(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 2712fbc68b28..906b7fa14d3c 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -22,6 +22,7 @@ > #include <linux/pci.h> > #include <linux/scatterlist.h> > #include <linux/vmalloc.h> > +#include <linux/crash_dump.h> > > struct iommu_dma_msi_page { > struct list_head list; > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > return iova_reserve_iommu_regions(dev, domain); > } > > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) > +{ > + const struct iommu_ops *ops = domain->ops; > + > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); > + > + return 0; > +} > + > /** > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > * page flags. > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > size_t iova_off = iova_offset(iovad, phys); > dma_addr_t iova; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return DMA_MAPPING_ERROR; > + > size = iova_align(iovad, size + iova_off); > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > @@ -581,6 +600,9 @@ static void *iommu_dma_alloc_remap(struct device *dev, size_t size, > > *dma_handle = DMA_MAPPING_ERROR; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return NULL; > + > min_size = alloc_sizes & -alloc_sizes; > if (min_size < PAGE_SIZE) { > min_size = PAGE_SIZE; > @@ -713,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, > int prot = dma_info_to_prot(dir, coherent, attrs); > dma_addr_t dma_handle; > > - dma_handle =__iommu_dma_map(dev, phys, size, prot); > + dma_handle = __iommu_dma_map(dev, phys, size, prot); > if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && > dma_handle != DMA_MAPPING_ERROR) > arch_sync_dma_for_device(dev, phys, size, dir); > @@ -823,6 +845,9 @@ static int iommu_dma_map_sg(struct device *dev, struct scatterlist *sg, > unsigned long mask = dma_get_seg_boundary(dev); > int i; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return 0; > + > if (!(attrs & DMA_ATTR_SKIP_CPU_SYNC)) > iommu_dma_sync_sg_for_device(dev, sg, nents, dir); > > ^ permalink raw reply [flat|nested] 87+ messages in thread
[parent not found: <20190815110944.3579-4-murphyt7-/Zw1syy4LHg@public.gmane.org>]
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-20 9:43 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:43 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc-u79uwXL29TY76Z2rM5mHXA, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, Kukjin Kim, Andy Gross, linux-s390-u79uwXL29TY76Z2rM5mHXA, Gerald Schaefer, linux-arm-msm-u79uwXL29TY76Z2rM5mHXA, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, linux-tegra-u79uwXL29TY76Z2rM5mHXA, linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, David Woodhouse, linux-kernel-u79uwXL29TY76Z2rM5mHXA, iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, Robin Murphy > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) Nitick: we usually use double tab indents (or indents to after the opening brace) for multi-line prototyped. > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); And for multi-line conditionals we also use two-tab indents. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-20 9:43 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:43 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) Nitick: we usually use double tab indents (or indents to after the opening brace) for multi-line prototyped. > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); And for multi-line conditionals we also use two-tab indents. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-20 9:43 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:43 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) Nitick: we usually use double tab indents (or indents to after the opening brace) for multi-line prototyped. > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); And for multi-line conditionals we also use two-tab indents. _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-20 9:43 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:43 UTC (permalink / raw) To: Tom Murphy Cc: iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Kukjin Kim, Robin Murphy > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) Nitick: we usually use double tab indents (or indents to after the opening brace) for multi-line prototyped. > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); And for multi-line conditionals we also use two-tab indents. ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices 2019-08-15 11:09 ` Tom Murphy ` (5 preceding siblings ...) (?) @ 2019-08-20 9:43 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:43 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) Nitick: we usually use double tab indents (or indents to after the opening brace) for multi-line prototyped. > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); And for multi-line conditionals we also use two-tab indents. ^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask 2019-08-15 11:09 ` Tom Murphy ` (8 preceding siblings ...) (?) @ 2019-08-15 11:09 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Rob Clark, Kukjin Kim, Robin Use the dev->coherent_dma_mask when allocating in the dma-iommu ops api. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/dma-iommu.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 906b7fa14d3c..b9a3ab02434b 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -471,7 +471,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, } static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, - size_t size, int prot) + size_t size, int prot, dma_addr_t dma_mask) { struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; @@ -484,7 +484,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size = iova_align(iovad, size + iova_off); - iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); + iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev); if (!iova) return DMA_MAPPING_ERROR; @@ -735,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, int prot = dma_info_to_prot(dir, coherent, attrs); dma_addr_t dma_handle; - dma_handle = __iommu_dma_map(dev, phys, size, prot); + dma_handle = __iommu_dma_map(dev, phys, size, prot, dma_get_mask(dev)); if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && dma_handle != DMA_MAPPING_ERROR) arch_sync_dma_for_device(dev, phys, size, dir); @@ -938,7 +938,8 @@ static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs) { return __iommu_dma_map(dev, phys, size, - dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO); + dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO, + dma_get_mask(dev)); } static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, @@ -1041,7 +1042,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, if (!cpu_addr) return NULL; - *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot, + dev->coherent_dma_mask); if (*handle == DMA_MAPPING_ERROR) { __iommu_dma_free(dev, size, cpu_addr); return NULL; -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-15 11:09 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Tom Murphy, Joerg Roedel, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek Use the dev->coherent_dma_mask when allocating in the dma-iommu ops api. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/dma-iommu.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 906b7fa14d3c..b9a3ab02434b 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -471,7 +471,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, } static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, - size_t size, int prot) + size_t size, int prot, dma_addr_t dma_mask) { struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; @@ -484,7 +484,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size = iova_align(iovad, size + iova_off); - iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); + iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev); if (!iova) return DMA_MAPPING_ERROR; @@ -735,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, int prot = dma_info_to_prot(dir, coherent, attrs); dma_addr_t dma_handle; - dma_handle = __iommu_dma_map(dev, phys, size, prot); + dma_handle = __iommu_dma_map(dev, phys, size, prot, dma_get_mask(dev)); if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && dma_handle != DMA_MAPPING_ERROR) arch_sync_dma_for_device(dev, phys, size, dir); @@ -938,7 +938,8 @@ static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs) { return __iommu_dma_map(dev, phys, size, - dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO); + dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO, + dma_get_mask(dev)); } static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, @@ -1041,7 +1042,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, if (!cpu_addr) return NULL; - *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot, + dev->coherent_dma_mask); if (*handle == DMA_MAPPING_ERROR) { __iommu_dma_free(dev, size, cpu_addr); return NULL; -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Rob Clark, Kukjin Kim, Robin Murphy Use the dev->coherent_dma_mask when allocating in the dma-iommu ops api. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/dma-iommu.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 906b7fa14d3c..b9a3ab02434b 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -471,7 +471,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, } static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, - size_t size, int prot) + size_t size, int prot, dma_addr_t dma_mask) { struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; @@ -484,7 +484,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size = iova_align(iovad, size + iova_off); - iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); + iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev); if (!iova) return DMA_MAPPING_ERROR; @@ -735,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, int prot = dma_info_to_prot(dir, coherent, attrs); dma_addr_t dma_handle; - dma_handle = __iommu_dma_map(dev, phys, size, prot); + dma_handle = __iommu_dma_map(dev, phys, size, prot, dma_get_mask(dev)); if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && dma_handle != DMA_MAPPING_ERROR) arch_sync_dma_for_device(dev, phys, size, dir); @@ -938,7 +938,8 @@ static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs) { return __iommu_dma_map(dev, phys, size, - dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO); + dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO, + dma_get_mask(dev)); } static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, @@ -1041,7 +1042,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, if (!cpu_addr) return NULL; - *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot, + dev->coherent_dma_mask); if (*handle == DMA_MAPPING_ERROR) { __iommu_dma_free(dev, size, cpu_addr); return NULL; -- 2.20.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Kukjin Kim, Robin Murphy Use the dev->coherent_dma_mask when allocating in the dma-iommu ops api. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/dma-iommu.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 906b7fa14d3c..b9a3ab02434b 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -471,7 +471,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, } static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, - size_t size, int prot) + size_t size, int prot, dma_addr_t dma_mask) { struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; @@ -484,7 +484,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size = iova_align(iovad, size + iova_off); - iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); + iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev); if (!iova) return DMA_MAPPING_ERROR; @@ -735,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, int prot = dma_info_to_prot(dir, coherent, attrs); dma_addr_t dma_handle; - dma_handle = __iommu_dma_map(dev, phys, size, prot); + dma_handle = __iommu_dma_map(dev, phys, size, prot, dma_get_mask(dev)); if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && dma_handle != DMA_MAPPING_ERROR) arch_sync_dma_for_device(dev, phys, size, dir); @@ -938,7 +938,8 @@ static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs) { return __iommu_dma_map(dev, phys, size, - dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO); + dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO, + dma_get_mask(dev)); } static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, @@ -1041,7 +1042,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, if (!cpu_addr) return NULL; - *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot, + dev->coherent_dma_mask); if (*handle == DMA_MAPPING_ERROR) { __iommu_dma_free(dev, size, cpu_addr); return NULL; -- 2.20.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Tom Murphy, Joerg Roedel, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390, linux-tegra, virtualization Use the dev->coherent_dma_mask when allocating in the dma-iommu ops api. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/dma-iommu.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 906b7fa14d3c..b9a3ab02434b 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -471,7 +471,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, } static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, - size_t size, int prot) + size_t size, int prot, dma_addr_t dma_mask) { struct iommu_domain *domain = iommu_get_dma_domain(dev); struct iommu_dma_cookie *cookie = domain->iova_cookie; @@ -484,7 +484,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, size = iova_align(iovad, size + iova_off); - iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); + iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev); if (!iova) return DMA_MAPPING_ERROR; @@ -735,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, int prot = dma_info_to_prot(dir, coherent, attrs); dma_addr_t dma_handle; - dma_handle = __iommu_dma_map(dev, phys, size, prot); + dma_handle = __iommu_dma_map(dev, phys, size, prot, dma_get_mask(dev)); if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && dma_handle != DMA_MAPPING_ERROR) arch_sync_dma_for_device(dev, phys, size, dir); @@ -938,7 +938,8 @@ static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, size_t size, enum dma_data_direction dir, unsigned long attrs) { return __iommu_dma_map(dev, phys, size, - dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO); + dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO, + dma_get_mask(dev)); } static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, @@ -1041,7 +1042,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, if (!cpu_addr) return NULL; - *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot, + dev->coherent_dma_mask); if (*handle == DMA_MAPPING_ERROR) { __iommu_dma_free(dev, size, cpu_addr); return NULL; -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-19 18:39 ` Robin Murphy -1 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:39 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Joerg Roedel, Will Deacon, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390 On 15/08/2019 12:09, Tom Murphy wrote: > Use the dev->coherent_dma_mask when allocating in the dma-iommu ops api. Oops... I suppose technically that's my latent bug, but since we've all missed it so far, I doubt arm64 systems ever see any devices which actually have different masks. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 12 +++++++----- > 1 file changed, 7 insertions(+), 5 deletions(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 906b7fa14d3c..b9a3ab02434b 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -471,7 +471,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, > } > > static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > - size_t size, int prot) > + size_t size, int prot, dma_addr_t dma_mask) > { > struct iommu_domain *domain = iommu_get_dma_domain(dev); > struct iommu_dma_cookie *cookie = domain->iova_cookie; > @@ -484,7 +484,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > > size = iova_align(iovad, size + iova_off); > > - iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > + iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev); > if (!iova) > return DMA_MAPPING_ERROR; > > @@ -735,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, > int prot = dma_info_to_prot(dir, coherent, attrs); > dma_addr_t dma_handle; > > - dma_handle = __iommu_dma_map(dev, phys, size, prot); > + dma_handle = __iommu_dma_map(dev, phys, size, prot, dma_get_mask(dev)); > if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && > dma_handle != DMA_MAPPING_ERROR) > arch_sync_dma_for_device(dev, phys, size, dir); > @@ -938,7 +938,8 @@ static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, > size_t size, enum dma_data_direction dir, unsigned long attrs) > { > return __iommu_dma_map(dev, phys, size, > - dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO); > + dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO, > + dma_get_mask(dev)); > } > > static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, > @@ -1041,7 +1042,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, > if (!cpu_addr) > return NULL; > > - *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); > + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot, > + dev->coherent_dma_mask); > if (*handle == DMA_MAPPING_ERROR) { > __iommu_dma_free(dev, size, cpu_addr); > return NULL; > ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask @ 2019-08-19 18:39 ` Robin Murphy 0 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:39 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, linux-kernel, Rob Clark, Kukjin Kim, David Woodhouse On 15/08/2019 12:09, Tom Murphy wrote: > Use the dev->coherent_dma_mask when allocating in the dma-iommu ops api. Oops... I suppose technically that's my latent bug, but since we've all missed it so far, I doubt arm64 systems ever see any devices which actually have different masks. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 12 +++++++----- > 1 file changed, 7 insertions(+), 5 deletions(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 906b7fa14d3c..b9a3ab02434b 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -471,7 +471,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, > } > > static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > - size_t size, int prot) > + size_t size, int prot, dma_addr_t dma_mask) > { > struct iommu_domain *domain = iommu_get_dma_domain(dev); > struct iommu_dma_cookie *cookie = domain->iova_cookie; > @@ -484,7 +484,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > > size = iova_align(iovad, size + iova_off); > > - iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > + iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev); > if (!iova) > return DMA_MAPPING_ERROR; > > @@ -735,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, > int prot = dma_info_to_prot(dir, coherent, attrs); > dma_addr_t dma_handle; > > - dma_handle = __iommu_dma_map(dev, phys, size, prot); > + dma_handle = __iommu_dma_map(dev, phys, size, prot, dma_get_mask(dev)); > if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && > dma_handle != DMA_MAPPING_ERROR) > arch_sync_dma_for_device(dev, phys, size, dir); > @@ -938,7 +938,8 @@ static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, > size_t size, enum dma_data_direction dir, unsigned long attrs) > { > return __iommu_dma_map(dev, phys, size, > - dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO); > + dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO, > + dma_get_mask(dev)); > } > > static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, > @@ -1041,7 +1042,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, > if (!cpu_addr) > return NULL; > > - *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); > + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot, > + dev->coherent_dma_mask); > if (*handle == DMA_MAPPING_ERROR) { > __iommu_dma_free(dev, size, cpu_addr); > return NULL; > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask @ 2019-08-19 18:39 ` Robin Murphy 0 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:39 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, linux-kernel, Kukjin Kim, David Woodhouse On 15/08/2019 12:09, Tom Murphy wrote: > Use the dev->coherent_dma_mask when allocating in the dma-iommu ops api. Oops... I suppose technically that's my latent bug, but since we've all missed it so far, I doubt arm64 systems ever see any devices which actually have different masks. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 12 +++++++----- > 1 file changed, 7 insertions(+), 5 deletions(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 906b7fa14d3c..b9a3ab02434b 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -471,7 +471,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, > } > > static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > - size_t size, int prot) > + size_t size, int prot, dma_addr_t dma_mask) > { > struct iommu_domain *domain = iommu_get_dma_domain(dev); > struct iommu_dma_cookie *cookie = domain->iova_cookie; > @@ -484,7 +484,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > > size = iova_align(iovad, size + iova_off); > > - iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > + iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev); > if (!iova) > return DMA_MAPPING_ERROR; > > @@ -735,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, > int prot = dma_info_to_prot(dir, coherent, attrs); > dma_addr_t dma_handle; > > - dma_handle = __iommu_dma_map(dev, phys, size, prot); > + dma_handle = __iommu_dma_map(dev, phys, size, prot, dma_get_mask(dev)); > if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && > dma_handle != DMA_MAPPING_ERROR) > arch_sync_dma_for_device(dev, phys, size, dir); > @@ -938,7 +938,8 @@ static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, > size_t size, enum dma_data_direction dir, unsigned long attrs) > { > return __iommu_dma_map(dev, phys, size, > - dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO); > + dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO, > + dma_get_mask(dev)); > } > > static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, > @@ -1041,7 +1042,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, > if (!cpu_addr) > return NULL; > > - *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); > + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot, > + dev->coherent_dma_mask); > if (*handle == DMA_MAPPING_ERROR) { > __iommu_dma_free(dev, size, cpu_addr); > return NULL; > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask @ 2019-08-19 18:39 ` Robin Murphy 0 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:39 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Joerg Roedel, Will Deacon, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390, linux-tegra, virtualization On 15/08/2019 12:09, Tom Murphy wrote: > Use the dev->coherent_dma_mask when allocating in the dma-iommu ops api. Oops... I suppose technically that's my latent bug, but since we've all missed it so far, I doubt arm64 systems ever see any devices which actually have different masks. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 12 +++++++----- > 1 file changed, 7 insertions(+), 5 deletions(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 906b7fa14d3c..b9a3ab02434b 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -471,7 +471,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, > } > > static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > - size_t size, int prot) > + size_t size, int prot, dma_addr_t dma_mask) > { > struct iommu_domain *domain = iommu_get_dma_domain(dev); > struct iommu_dma_cookie *cookie = domain->iova_cookie; > @@ -484,7 +484,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > > size = iova_align(iovad, size + iova_off); > > - iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > + iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev); > if (!iova) > return DMA_MAPPING_ERROR; > > @@ -735,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, > int prot = dma_info_to_prot(dir, coherent, attrs); > dma_addr_t dma_handle; > > - dma_handle = __iommu_dma_map(dev, phys, size, prot); > + dma_handle = __iommu_dma_map(dev, phys, size, prot, dma_get_mask(dev)); > if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && > dma_handle != DMA_MAPPING_ERROR) > arch_sync_dma_for_device(dev, phys, size, dir); > @@ -938,7 +938,8 @@ static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, > size_t size, enum dma_data_direction dir, unsigned long attrs) > { > return __iommu_dma_map(dev, phys, size, > - dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO); > + dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO, > + dma_get_mask(dev)); > } > > static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, > @@ -1041,7 +1042,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, > if (!cpu_addr) > return NULL; > > - *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); > + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot, > + dev->coherent_dma_mask); > if (*handle == DMA_MAPPING_ERROR) { > __iommu_dma_free(dev, size, cpu_addr); > return NULL; > ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask 2019-08-15 11:09 ` Tom Murphy ` (3 preceding siblings ...) (?) @ 2019-08-19 18:39 ` Robin Murphy -1 siblings, 0 replies; 87+ messages in thread From: Robin Murphy @ 2019-08-19 18:39 UTC (permalink / raw) To: Tom Murphy, iommu Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, linux-kernel, Rob Clark, Kukjin Kim, David Woodhouse On 15/08/2019 12:09, Tom Murphy wrote: > Use the dev->coherent_dma_mask when allocating in the dma-iommu ops api. Oops... I suppose technically that's my latent bug, but since we've all missed it so far, I doubt arm64 systems ever see any devices which actually have different masks. Reviewed-by: Robin Murphy <robin.murphy@arm.com> > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 12 +++++++----- > 1 file changed, 7 insertions(+), 5 deletions(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 906b7fa14d3c..b9a3ab02434b 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -471,7 +471,7 @@ static void __iommu_dma_unmap(struct device *dev, dma_addr_t dma_addr, > } > > static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > - size_t size, int prot) > + size_t size, int prot, dma_addr_t dma_mask) > { > struct iommu_domain *domain = iommu_get_dma_domain(dev); > struct iommu_dma_cookie *cookie = domain->iova_cookie; > @@ -484,7 +484,7 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > > size = iova_align(iovad, size + iova_off); > > - iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > + iova = iommu_dma_alloc_iova(domain, size, dma_mask, dev); > if (!iova) > return DMA_MAPPING_ERROR; > > @@ -735,7 +735,7 @@ static dma_addr_t iommu_dma_map_page(struct device *dev, struct page *page, > int prot = dma_info_to_prot(dir, coherent, attrs); > dma_addr_t dma_handle; > > - dma_handle = __iommu_dma_map(dev, phys, size, prot); > + dma_handle = __iommu_dma_map(dev, phys, size, prot, dma_get_mask(dev)); > if (!coherent && !(attrs & DMA_ATTR_SKIP_CPU_SYNC) && > dma_handle != DMA_MAPPING_ERROR) > arch_sync_dma_for_device(dev, phys, size, dir); > @@ -938,7 +938,8 @@ static dma_addr_t iommu_dma_map_resource(struct device *dev, phys_addr_t phys, > size_t size, enum dma_data_direction dir, unsigned long attrs) > { > return __iommu_dma_map(dev, phys, size, > - dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO); > + dma_info_to_prot(dir, false, attrs) | IOMMU_MMIO, > + dma_get_mask(dev)); > } > > static void iommu_dma_unmap_resource(struct device *dev, dma_addr_t handle, > @@ -1041,7 +1042,8 @@ static void *iommu_dma_alloc(struct device *dev, size_t size, > if (!cpu_addr) > return NULL; > > - *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot); > + *handle = __iommu_dma_map(dev, page_to_phys(page), size, ioprot, > + dev->coherent_dma_mask); > if (*handle == DMA_MAPPING_ERROR) { > __iommu_dma_free(dev, size, cpu_addr); > return NULL; > ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask 2019-08-15 11:09 ` Tom Murphy ` (4 preceding siblings ...) (?) @ 2019-08-20 9:43 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:43 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy Looks good, and should probably be queued up asap as a bug fix: Reviewed-by: Christoph Hellwig <hch@lst.de> ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-20 9:43 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:43 UTC (permalink / raw) To: Tom Murphy Cc: iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Kukjin Kim Looks good, and should probably be queued up asap as a bug fix: Reviewed-by: Christoph Hellwig <hch@lst.de> ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask @ 2019-08-20 9:43 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:43 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy Looks good, and should probably be queued up asap as a bug fix: Reviewed-by: Christoph Hellwig <hch@lst.de> _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask @ 2019-08-20 9:43 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:43 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy Looks good, and should probably be queued up asap as a bug fix: Reviewed-by: Christoph Hellwig <hch@lst.de> _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask @ 2019-08-20 9:43 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-08-20 9:43 UTC (permalink / raw) To: Tom Murphy Cc: iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Kukjin Kim, Robin Murphy Looks good, and should probably be queued up asap as a bug fix: Reviewed-by: Christoph Hellwig <hch@lst.de> ^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH V5 5/5] iommu/amd: Convert AMD iommu driver to the dma-iommu api 2019-08-15 11:09 ` Tom Murphy ` (10 preceding siblings ...) (?) @ 2019-08-15 11:09 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Rob Clark, Kukjin Kim, Robin Convert the AMD iommu driver to the dma-iommu api. Remove the iova handling and reserve region code from the AMD iommu driver. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/Kconfig | 1 + drivers/iommu/amd_iommu.c | 677 ++++---------------------------------- 2 files changed, 68 insertions(+), 610 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index e15cdcd8cb3c..437428571512 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -138,6 +138,7 @@ config AMD_IOMMU select PCI_PASID select IOMMU_API select IOMMU_IOVA + select IOMMU_DMA depends on X86_64 && PCI && ACPI ---help--- With this option you can enable support for AMD IOMMU hardware in diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 0e53f9bd2be7..eb4801031a99 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -20,6 +20,7 @@ #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/dma-direct.h> +#include <linux/dma-iommu.h> #include <linux/iommu-helper.h> #include <linux/iommu.h> #include <linux/delay.h> @@ -89,8 +90,6 @@ const struct iommu_ops amd_iommu_ops; static ATOMIC_NOTIFIER_HEAD(ppr_notifier); int amd_iommu_max_glx_val = -1; -static const struct dma_map_ops amd_iommu_dma_ops; - /* * general struct to manage commands send to an IOMMU */ @@ -103,21 +102,6 @@ struct kmem_cache *amd_iommu_irq_cache; static void update_domain(struct protection_domain *domain); static int protection_domain_init(struct protection_domain *domain); static void detach_device(struct device *dev); -static void iova_domain_flush_tlb(struct iova_domain *iovad); - -/* - * Data container for a dma_ops specific protection domain - */ -struct dma_ops_domain { - /* generic protection domain information */ - struct protection_domain domain; - - /* IOVA RB-Tree */ - struct iova_domain iovad; -}; - -static struct iova_domain reserved_iova_ranges; -static struct lock_class_key reserved_rbtree_key; /**************************************************************************** * @@ -188,12 +172,6 @@ static struct protection_domain *to_pdomain(struct iommu_domain *dom) return container_of(dom, struct protection_domain, domain); } -static struct dma_ops_domain* to_dma_ops_domain(struct protection_domain *domain) -{ - BUG_ON(domain->flags != PD_DMA_OPS_MASK); - return container_of(domain, struct dma_ops_domain, domain); -} - static struct iommu_dev_data *alloc_dev_data(u16 devid) { struct iommu_dev_data *dev_data; @@ -1267,12 +1245,6 @@ static void domain_flush_pages(struct protection_domain *domain, __domain_flush_pages(domain, address, size, 0); } -/* Flush the whole IO/TLB for a given protection domain */ -static void domain_flush_tlb(struct protection_domain *domain) -{ - __domain_flush_pages(domain, 0, CMD_INV_IOMMU_ALL_PAGES_ADDRESS, 0); -} - /* Flush the whole IO/TLB for a given protection domain - including PDE */ static void domain_flush_tlb_pde(struct protection_domain *domain) { @@ -1674,43 +1646,6 @@ static unsigned long iommu_unmap_page(struct protection_domain *dom, return unmapped; } -/**************************************************************************** - * - * The next functions belong to the address allocator for the dma_ops - * interface functions. - * - ****************************************************************************/ - - -static unsigned long dma_ops_alloc_iova(struct device *dev, - struct dma_ops_domain *dma_dom, - unsigned int pages, u64 dma_mask) -{ - unsigned long pfn = 0; - - pages = __roundup_pow_of_two(pages); - - if (dma_mask > DMA_BIT_MASK(32)) - pfn = alloc_iova_fast(&dma_dom->iovad, pages, - IOVA_PFN(DMA_BIT_MASK(32)), false); - - if (!pfn) - pfn = alloc_iova_fast(&dma_dom->iovad, pages, - IOVA_PFN(dma_mask), true); - - return (pfn << PAGE_SHIFT); -} - -static void dma_ops_free_iova(struct dma_ops_domain *dma_dom, - unsigned long address, - unsigned int pages) -{ - pages = __roundup_pow_of_two(pages); - address >>= PAGE_SHIFT; - - free_iova_fast(&dma_dom->iovad, address, pages); -} - /**************************************************************************** * * The next functions belong to the domain allocation. A domain is @@ -1787,38 +1722,23 @@ static void free_gcr3_table(struct protection_domain *domain) free_page((unsigned long)domain->gcr3_tbl); } -static void dma_ops_domain_flush_tlb(struct dma_ops_domain *dom) -{ - domain_flush_tlb(&dom->domain); - domain_flush_complete(&dom->domain); -} - -static void iova_domain_flush_tlb(struct iova_domain *iovad) -{ - struct dma_ops_domain *dom; - - dom = container_of(iovad, struct dma_ops_domain, iovad); - - dma_ops_domain_flush_tlb(dom); -} - /* * Free a domain, only used if something went wrong in the * allocation path and we need to free an already allocated page table */ -static void dma_ops_domain_free(struct dma_ops_domain *dom) +static void dma_ops_domain_free(struct protection_domain *domain) { - if (!dom) + if (!domain) return; - put_iova_domain(&dom->iovad); + iommu_put_dma_cookie(&domain->domain); - free_pagetable(&dom->domain); + free_pagetable(domain); - if (dom->domain.id) - domain_id_free(dom->domain.id); + if (domain->id) + domain_id_free(domain->id); - kfree(dom); + kfree(domain); } /* @@ -1826,35 +1746,30 @@ static void dma_ops_domain_free(struct dma_ops_domain *dom) * It also initializes the page table and the address allocator data * structures required for the dma_ops interface */ -static struct dma_ops_domain *dma_ops_domain_alloc(void) +static struct protection_domain *dma_ops_domain_alloc(void) { - struct dma_ops_domain *dma_dom; + struct protection_domain *domain; - dma_dom = kzalloc(sizeof(struct dma_ops_domain), GFP_KERNEL); - if (!dma_dom) + domain = kzalloc(sizeof(struct protection_domain), GFP_KERNEL); + if (!domain) return NULL; - if (protection_domain_init(&dma_dom->domain)) - goto free_dma_dom; - - dma_dom->domain.mode = PAGE_MODE_3_LEVEL; - dma_dom->domain.pt_root = (void *)get_zeroed_page(GFP_KERNEL); - dma_dom->domain.flags = PD_DMA_OPS_MASK; - if (!dma_dom->domain.pt_root) - goto free_dma_dom; - - init_iova_domain(&dma_dom->iovad, PAGE_SIZE, IOVA_START_PFN); + if (protection_domain_init(domain)) + goto free_domain; - if (init_iova_flush_queue(&dma_dom->iovad, iova_domain_flush_tlb, NULL)) - goto free_dma_dom; + domain->mode = PAGE_MODE_3_LEVEL; + domain->pt_root = (void *)get_zeroed_page(GFP_KERNEL); + domain->flags = PD_DMA_OPS_MASK; + if (!domain->pt_root) + goto free_domain; - /* Initialize reserved ranges */ - copy_reserved_iova(&reserved_iova_ranges, &dma_dom->iovad); + if (iommu_get_dma_cookie(&domain->domain) == -ENOMEM) + goto free_domain; - return dma_dom; + return domain; -free_dma_dom: - dma_ops_domain_free(dma_dom); +free_domain: + dma_ops_domain_free(domain); return NULL; } @@ -2233,8 +2148,8 @@ static int amd_iommu_add_device(struct device *dev) domain = iommu_get_domain_for_dev(dev); if (domain->type == IOMMU_DOMAIN_IDENTITY) dev_data->passthrough = true; - else - dev->dma_ops = &amd_iommu_dma_ops; + else if (domain->type == IOMMU_DOMAIN_DMA) + iommu_setup_dma_ops(dev, IOVA_START_PFN << PAGE_SHIFT, 0); out: iommu_completion_wait(iommu); @@ -2268,43 +2183,32 @@ static struct iommu_group *amd_iommu_device_group(struct device *dev) return acpihid_device_group(dev); } +static int amd_iommu_domain_get_attr(struct iommu_domain *domain, + enum iommu_attr attr, void *data) +{ + switch (domain->type) { + case IOMMU_DOMAIN_UNMANAGED: + return -ENODEV; + case IOMMU_DOMAIN_DMA: + switch (attr) { + case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE: + *(int *)data = !amd_iommu_unmap_flush; + return 0; + default: + return -ENODEV; + } + break; + default: + return -EINVAL; + } +} + /***************************************************************************** * * The next functions belong to the dma_ops mapping/unmapping code. * *****************************************************************************/ -/* - * In the dma_ops path we only have the struct device. This function - * finds the corresponding IOMMU, the protection domain and the - * requestor id for a given device. - * If the device is not yet associated with a domain this is also done - * in this function. - */ -static struct protection_domain *get_domain(struct device *dev) -{ - struct protection_domain *domain; - struct iommu_domain *io_domain; - - if (!check_device(dev)) - return ERR_PTR(-EINVAL); - - domain = get_dev_data(dev)->domain; - if (domain == NULL && get_dev_data(dev)->defer_attach) { - get_dev_data(dev)->defer_attach = false; - io_domain = iommu_get_domain_for_dev(dev); - domain = to_pdomain(io_domain); - attach_device(dev, domain); - } - if (domain == NULL) - return ERR_PTR(-EBUSY); - - if (!dma_ops_domain(domain)) - return ERR_PTR(-EBUSY); - - return domain; -} - static void update_device_table(struct protection_domain *domain) { struct iommu_dev_data *dev_data; @@ -2335,447 +2239,6 @@ static void update_domain(struct protection_domain *domain) domain->updated = false; } -static int dir2prot(enum dma_data_direction direction) -{ - if (direction == DMA_TO_DEVICE) - return IOMMU_PROT_IR; - else if (direction == DMA_FROM_DEVICE) - return IOMMU_PROT_IW; - else if (direction == DMA_BIDIRECTIONAL) - return IOMMU_PROT_IW | IOMMU_PROT_IR; - else - return 0; -} - -/* - * This function contains common code for mapping of a physically - * contiguous memory region into DMA address space. It is used by all - * mapping functions provided with this IOMMU driver. - * Must be called with the domain lock held. - */ -static dma_addr_t __map_single(struct device *dev, - struct dma_ops_domain *dma_dom, - phys_addr_t paddr, - size_t size, - enum dma_data_direction direction, - u64 dma_mask) -{ - dma_addr_t offset = paddr & ~PAGE_MASK; - dma_addr_t address, start, ret; - unsigned int pages; - int prot = 0; - int i; - - pages = iommu_num_pages(paddr, size, PAGE_SIZE); - paddr &= PAGE_MASK; - - address = dma_ops_alloc_iova(dev, dma_dom, pages, dma_mask); - if (!address) - goto out; - - prot = dir2prot(direction); - - start = address; - for (i = 0; i < pages; ++i) { - ret = iommu_map_page(&dma_dom->domain, start, paddr, - PAGE_SIZE, prot, GFP_ATOMIC); - if (ret) - goto out_unmap; - - paddr += PAGE_SIZE; - start += PAGE_SIZE; - } - address += offset; - - domain_flush_np_cache(&dma_dom->domain, address, size); - -out: - return address; - -out_unmap: - - for (--i; i >= 0; --i) { - start -= PAGE_SIZE; - iommu_unmap_page(&dma_dom->domain, start, PAGE_SIZE); - } - - domain_flush_tlb(&dma_dom->domain); - domain_flush_complete(&dma_dom->domain); - - dma_ops_free_iova(dma_dom, address, pages); - - return DMA_MAPPING_ERROR; -} - -/* - * Does the reverse of the __map_single function. Must be called with - * the domain lock held too - */ -static void __unmap_single(struct dma_ops_domain *dma_dom, - dma_addr_t dma_addr, - size_t size, - int dir) -{ - dma_addr_t i, start; - unsigned int pages; - - pages = iommu_num_pages(dma_addr, size, PAGE_SIZE); - dma_addr &= PAGE_MASK; - start = dma_addr; - - for (i = 0; i < pages; ++i) { - iommu_unmap_page(&dma_dom->domain, start, PAGE_SIZE); - start += PAGE_SIZE; - } - - if (amd_iommu_unmap_flush) { - domain_flush_tlb(&dma_dom->domain); - domain_flush_complete(&dma_dom->domain); - dma_ops_free_iova(dma_dom, dma_addr, pages); - } else { - pages = __roundup_pow_of_two(pages); - queue_iova(&dma_dom->iovad, dma_addr >> PAGE_SHIFT, pages, 0); - } -} - -/* - * The exported map_single function for dma_ops. - */ -static dma_addr_t map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, - enum dma_data_direction dir, - unsigned long attrs) -{ - phys_addr_t paddr = page_to_phys(page) + offset; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - u64 dma_mask; - - domain = get_domain(dev); - if (PTR_ERR(domain) == -EINVAL) - return (dma_addr_t)paddr; - else if (IS_ERR(domain)) - return DMA_MAPPING_ERROR; - - dma_mask = *dev->dma_mask; - dma_dom = to_dma_ops_domain(domain); - - return __map_single(dev, dma_dom, paddr, size, dir, dma_mask); -} - -/* - * The exported unmap_single function for dma_ops. - */ -static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size, - enum dma_data_direction dir, unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return; - - dma_dom = to_dma_ops_domain(domain); - - __unmap_single(dma_dom, dma_addr, size, dir); -} - -static int sg_num_pages(struct device *dev, - struct scatterlist *sglist, - int nelems) -{ - unsigned long mask, boundary_size; - struct scatterlist *s; - int i, npages = 0; - - mask = dma_get_seg_boundary(dev); - boundary_size = mask + 1 ? ALIGN(mask + 1, PAGE_SIZE) >> PAGE_SHIFT : - 1UL << (BITS_PER_LONG - PAGE_SHIFT); - - for_each_sg(sglist, s, nelems, i) { - int p, n; - - s->dma_address = npages << PAGE_SHIFT; - p = npages % boundary_size; - n = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - if (p + n > boundary_size) - npages += boundary_size - p; - npages += n; - } - - return npages; -} - -/* - * The exported map_sg function for dma_ops (handles scatter-gather - * lists). - */ -static int map_sg(struct device *dev, struct scatterlist *sglist, - int nelems, enum dma_data_direction direction, - unsigned long attrs) -{ - int mapped_pages = 0, npages = 0, prot = 0, i; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct scatterlist *s; - unsigned long address; - u64 dma_mask; - int ret; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return 0; - - dma_dom = to_dma_ops_domain(domain); - dma_mask = *dev->dma_mask; - - npages = sg_num_pages(dev, sglist, nelems); - - address = dma_ops_alloc_iova(dev, dma_dom, npages, dma_mask); - if (!address) - goto out_err; - - prot = dir2prot(direction); - - /* Map all sg entries */ - for_each_sg(sglist, s, nelems, i) { - int j, pages = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - - for (j = 0; j < pages; ++j) { - unsigned long bus_addr, phys_addr; - - bus_addr = address + s->dma_address + (j << PAGE_SHIFT); - phys_addr = (sg_phys(s) & PAGE_MASK) + (j << PAGE_SHIFT); - ret = iommu_map_page(domain, bus_addr, phys_addr, PAGE_SIZE, prot, GFP_ATOMIC); - if (ret) - goto out_unmap; - - mapped_pages += 1; - } - } - - /* Everything is mapped - write the right values into s->dma_address */ - for_each_sg(sglist, s, nelems, i) { - /* - * Add in the remaining piece of the scatter-gather offset that - * was masked out when we were determining the physical address - * via (sg_phys(s) & PAGE_MASK) earlier. - */ - s->dma_address += address + (s->offset & ~PAGE_MASK); - s->dma_length = s->length; - } - - if (s) - domain_flush_np_cache(domain, s->dma_address, s->dma_length); - - return nelems; - -out_unmap: - dev_err(dev, "IOMMU mapping error in map_sg (io-pages: %d reason: %d)\n", - npages, ret); - - for_each_sg(sglist, s, nelems, i) { - int j, pages = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - - for (j = 0; j < pages; ++j) { - unsigned long bus_addr; - - bus_addr = address + s->dma_address + (j << PAGE_SHIFT); - iommu_unmap_page(domain, bus_addr, PAGE_SIZE); - - if (--mapped_pages == 0) - goto out_free_iova; - } - } - -out_free_iova: - free_iova_fast(&dma_dom->iovad, address >> PAGE_SHIFT, npages); - -out_err: - return 0; -} - -/* - * The exported map_sg function for dma_ops (handles scatter-gather - * lists). - */ -static void unmap_sg(struct device *dev, struct scatterlist *sglist, - int nelems, enum dma_data_direction dir, - unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - unsigned long startaddr; - int npages; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return; - - startaddr = sg_dma_address(sglist) & PAGE_MASK; - dma_dom = to_dma_ops_domain(domain); - npages = sg_num_pages(dev, sglist, nelems); - - __unmap_single(dma_dom, startaddr, npages << PAGE_SHIFT, dir); -} - -/* - * The exported alloc_coherent function for dma_ops. - */ -static void *alloc_coherent(struct device *dev, size_t size, - dma_addr_t *dma_addr, gfp_t flag, - unsigned long attrs) -{ - u64 dma_mask = dev->coherent_dma_mask; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct page *page; - - domain = get_domain(dev); - if (PTR_ERR(domain) == -EINVAL) { - page = alloc_pages(flag, get_order(size)); - *dma_addr = page_to_phys(page); - return page_address(page); - } else if (IS_ERR(domain)) - return NULL; - - dma_dom = to_dma_ops_domain(domain); - size = PAGE_ALIGN(size); - dma_mask = dev->coherent_dma_mask; - flag &= ~(__GFP_DMA | __GFP_HIGHMEM | __GFP_DMA32); - flag |= __GFP_ZERO; - - page = alloc_pages(flag | __GFP_NOWARN, get_order(size)); - if (!page) { - if (!gfpflags_allow_blocking(flag)) - return NULL; - - page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, - get_order(size), flag & __GFP_NOWARN); - if (!page) - return NULL; - } - - if (!dma_mask) - dma_mask = *dev->dma_mask; - - *dma_addr = __map_single(dev, dma_dom, page_to_phys(page), - size, DMA_BIDIRECTIONAL, dma_mask); - - if (*dma_addr == DMA_MAPPING_ERROR) - goto out_free; - - return page_address(page); - -out_free: - - if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) - __free_pages(page, get_order(size)); - - return NULL; -} - -/* - * The exported free_coherent function for dma_ops. - */ -static void free_coherent(struct device *dev, size_t size, - void *virt_addr, dma_addr_t dma_addr, - unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct page *page; - - page = virt_to_page(virt_addr); - size = PAGE_ALIGN(size); - - domain = get_domain(dev); - if (IS_ERR(domain)) - goto free_mem; - - dma_dom = to_dma_ops_domain(domain); - - __unmap_single(dma_dom, dma_addr, size, DMA_BIDIRECTIONAL); - -free_mem: - if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) - __free_pages(page, get_order(size)); -} - -/* - * This function is called by the DMA layer to find out if we can handle a - * particular device. It is part of the dma_ops. - */ -static int amd_iommu_dma_supported(struct device *dev, u64 mask) -{ - if (!dma_direct_supported(dev, mask)) - return 0; - return check_device(dev); -} - -static const struct dma_map_ops amd_iommu_dma_ops = { - .alloc = alloc_coherent, - .free = free_coherent, - .map_page = map_page, - .unmap_page = unmap_page, - .map_sg = map_sg, - .unmap_sg = unmap_sg, - .dma_supported = amd_iommu_dma_supported, -}; - -static int init_reserved_iova_ranges(void) -{ - struct pci_dev *pdev = NULL; - struct iova *val; - - init_iova_domain(&reserved_iova_ranges, PAGE_SIZE, IOVA_START_PFN); - - lockdep_set_class(&reserved_iova_ranges.iova_rbtree_lock, - &reserved_rbtree_key); - - /* MSI memory range */ - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(MSI_RANGE_START), IOVA_PFN(MSI_RANGE_END)); - if (!val) { - pr_err("Reserving MSI range failed\n"); - return -ENOMEM; - } - - /* HT memory range */ - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(HT_RANGE_START), IOVA_PFN(HT_RANGE_END)); - if (!val) { - pr_err("Reserving HT range failed\n"); - return -ENOMEM; - } - - /* - * Memory used for PCI resources - * FIXME: Check whether we can reserve the PCI-hole completly - */ - for_each_pci_dev(pdev) { - int i; - - for (i = 0; i < PCI_NUM_RESOURCES; ++i) { - struct resource *r = &pdev->resource[i]; - - if (!(r->flags & IORESOURCE_MEM)) - continue; - - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(r->start), - IOVA_PFN(r->end)); - if (!val) { - pci_err(pdev, "Reserve pci-resource range %pR failed\n", r); - return -ENOMEM; - } - } - } - - return 0; -} - int __init amd_iommu_init_api(void) { int ret, err = 0; @@ -2784,10 +2247,6 @@ int __init amd_iommu_init_api(void) if (ret) return ret; - ret = init_reserved_iova_ranges(); - if (ret) - return ret; - err = bus_set_iommu(&pci_bus_type, &amd_iommu_ops); if (err) return err; @@ -2888,7 +2347,6 @@ static struct protection_domain *protection_domain_alloc(void) static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) { struct protection_domain *pdomain; - struct dma_ops_domain *dma_domain; switch (type) { case IOMMU_DOMAIN_UNMANAGED: @@ -2909,12 +2367,11 @@ static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) break; case IOMMU_DOMAIN_DMA: - dma_domain = dma_ops_domain_alloc(); - if (!dma_domain) { + pdomain = dma_ops_domain_alloc(); + if (!pdomain) { pr_err("Failed to allocate\n"); return NULL; } - pdomain = &dma_domain->domain; break; case IOMMU_DOMAIN_IDENTITY: pdomain = protection_domain_alloc(); @@ -2933,7 +2390,6 @@ static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) static void amd_iommu_domain_free(struct iommu_domain *dom) { struct protection_domain *domain; - struct dma_ops_domain *dma_dom; domain = to_pdomain(dom); @@ -2948,8 +2404,7 @@ static void amd_iommu_domain_free(struct iommu_domain *dom) switch (dom->type) { case IOMMU_DOMAIN_DMA: /* Now release the domain */ - dma_dom = to_dma_ops_domain(domain); - dma_ops_domain_free(dma_dom); + dma_ops_domain_free(domain); break; default: if (domain->mode != PAGE_MODE_NONE) @@ -3005,6 +2460,7 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, return -EINVAL; dev_data = dev->archdata.iommu; + dev_data->defer_attach = false; iommu = amd_iommu_rlookup_table[dev_data->devid]; if (!iommu) @@ -3161,19 +2617,6 @@ static void amd_iommu_put_resv_regions(struct device *dev, kfree(entry); } -static void amd_iommu_apply_resv_region(struct device *dev, - struct iommu_domain *domain, - struct iommu_resv_region *region) -{ - struct dma_ops_domain *dma_dom = to_dma_ops_domain(to_pdomain(domain)); - unsigned long start, end; - - start = IOVA_PFN(region->start); - end = IOVA_PFN(region->start + region->length - 1); - - WARN_ON_ONCE(reserve_iova(&dma_dom->iovad, start, end) == NULL); -} - static bool amd_iommu_is_attach_deferred(struct iommu_domain *domain, struct device *dev) { @@ -3206,9 +2649,9 @@ const struct iommu_ops amd_iommu_ops = { .add_device = amd_iommu_add_device, .remove_device = amd_iommu_remove_device, .device_group = amd_iommu_device_group, + .domain_get_attr = amd_iommu_domain_get_attr, .get_resv_regions = amd_iommu_get_resv_regions, .put_resv_regions = amd_iommu_put_resv_regions, - .apply_resv_region = amd_iommu_apply_resv_region, .is_attach_deferred = amd_iommu_is_attach_deferred, .pgsize_bitmap = AMD_IOMMU_PGSIZES, .flush_iotlb_all = amd_iommu_flush_iotlb_all, @@ -3523,9 +2966,23 @@ EXPORT_SYMBOL(amd_iommu_complete_ppr); struct iommu_domain *amd_iommu_get_v2_domain(struct pci_dev *pdev) { struct protection_domain *pdomain; + struct iommu_domain *io_domain; + struct device *dev = &pdev->dev; + + if (!check_device(dev)) + return NULL; + + pdomain = get_dev_data(dev)->domain; + if (pdomain == NULL && get_dev_data(dev)->defer_attach) { + get_dev_data(dev)->defer_attach = false; + io_domain = iommu_get_domain_for_dev(dev); + pdomain = to_pdomain(io_domain); + attach_device(dev, pdomain); + } + if (pdomain == NULL) + return NULL; - pdomain = get_domain(&pdev->dev); - if (IS_ERR(pdomain)) + if (!dma_ops_domain(pdomain)) return NULL; /* Only return IOMMUv2 domains */ -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 5/5] iommu/amd: Convert AMD iommu driver to the dma-iommu api 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-08-15 11:09 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Tom Murphy, Joerg Roedel, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek Convert the AMD iommu driver to the dma-iommu api. Remove the iova handling and reserve region code from the AMD iommu driver. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/Kconfig | 1 + drivers/iommu/amd_iommu.c | 677 ++++---------------------------------- 2 files changed, 68 insertions(+), 610 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index e15cdcd8cb3c..437428571512 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -138,6 +138,7 @@ config AMD_IOMMU select PCI_PASID select IOMMU_API select IOMMU_IOVA + select IOMMU_DMA depends on X86_64 && PCI && ACPI ---help--- With this option you can enable support for AMD IOMMU hardware in diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 0e53f9bd2be7..eb4801031a99 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -20,6 +20,7 @@ #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/dma-direct.h> +#include <linux/dma-iommu.h> #include <linux/iommu-helper.h> #include <linux/iommu.h> #include <linux/delay.h> @@ -89,8 +90,6 @@ const struct iommu_ops amd_iommu_ops; static ATOMIC_NOTIFIER_HEAD(ppr_notifier); int amd_iommu_max_glx_val = -1; -static const struct dma_map_ops amd_iommu_dma_ops; - /* * general struct to manage commands send to an IOMMU */ @@ -103,21 +102,6 @@ struct kmem_cache *amd_iommu_irq_cache; static void update_domain(struct protection_domain *domain); static int protection_domain_init(struct protection_domain *domain); static void detach_device(struct device *dev); -static void iova_domain_flush_tlb(struct iova_domain *iovad); - -/* - * Data container for a dma_ops specific protection domain - */ -struct dma_ops_domain { - /* generic protection domain information */ - struct protection_domain domain; - - /* IOVA RB-Tree */ - struct iova_domain iovad; -}; - -static struct iova_domain reserved_iova_ranges; -static struct lock_class_key reserved_rbtree_key; /**************************************************************************** * @@ -188,12 +172,6 @@ static struct protection_domain *to_pdomain(struct iommu_domain *dom) return container_of(dom, struct protection_domain, domain); } -static struct dma_ops_domain* to_dma_ops_domain(struct protection_domain *domain) -{ - BUG_ON(domain->flags != PD_DMA_OPS_MASK); - return container_of(domain, struct dma_ops_domain, domain); -} - static struct iommu_dev_data *alloc_dev_data(u16 devid) { struct iommu_dev_data *dev_data; @@ -1267,12 +1245,6 @@ static void domain_flush_pages(struct protection_domain *domain, __domain_flush_pages(domain, address, size, 0); } -/* Flush the whole IO/TLB for a given protection domain */ -static void domain_flush_tlb(struct protection_domain *domain) -{ - __domain_flush_pages(domain, 0, CMD_INV_IOMMU_ALL_PAGES_ADDRESS, 0); -} - /* Flush the whole IO/TLB for a given protection domain - including PDE */ static void domain_flush_tlb_pde(struct protection_domain *domain) { @@ -1674,43 +1646,6 @@ static unsigned long iommu_unmap_page(struct protection_domain *dom, return unmapped; } -/**************************************************************************** - * - * The next functions belong to the address allocator for the dma_ops - * interface functions. - * - ****************************************************************************/ - - -static unsigned long dma_ops_alloc_iova(struct device *dev, - struct dma_ops_domain *dma_dom, - unsigned int pages, u64 dma_mask) -{ - unsigned long pfn = 0; - - pages = __roundup_pow_of_two(pages); - - if (dma_mask > DMA_BIT_MASK(32)) - pfn = alloc_iova_fast(&dma_dom->iovad, pages, - IOVA_PFN(DMA_BIT_MASK(32)), false); - - if (!pfn) - pfn = alloc_iova_fast(&dma_dom->iovad, pages, - IOVA_PFN(dma_mask), true); - - return (pfn << PAGE_SHIFT); -} - -static void dma_ops_free_iova(struct dma_ops_domain *dma_dom, - unsigned long address, - unsigned int pages) -{ - pages = __roundup_pow_of_two(pages); - address >>= PAGE_SHIFT; - - free_iova_fast(&dma_dom->iovad, address, pages); -} - /**************************************************************************** * * The next functions belong to the domain allocation. A domain is @@ -1787,38 +1722,23 @@ static void free_gcr3_table(struct protection_domain *domain) free_page((unsigned long)domain->gcr3_tbl); } -static void dma_ops_domain_flush_tlb(struct dma_ops_domain *dom) -{ - domain_flush_tlb(&dom->domain); - domain_flush_complete(&dom->domain); -} - -static void iova_domain_flush_tlb(struct iova_domain *iovad) -{ - struct dma_ops_domain *dom; - - dom = container_of(iovad, struct dma_ops_domain, iovad); - - dma_ops_domain_flush_tlb(dom); -} - /* * Free a domain, only used if something went wrong in the * allocation path and we need to free an already allocated page table */ -static void dma_ops_domain_free(struct dma_ops_domain *dom) +static void dma_ops_domain_free(struct protection_domain *domain) { - if (!dom) + if (!domain) return; - put_iova_domain(&dom->iovad); + iommu_put_dma_cookie(&domain->domain); - free_pagetable(&dom->domain); + free_pagetable(domain); - if (dom->domain.id) - domain_id_free(dom->domain.id); + if (domain->id) + domain_id_free(domain->id); - kfree(dom); + kfree(domain); } /* @@ -1826,35 +1746,30 @@ static void dma_ops_domain_free(struct dma_ops_domain *dom) * It also initializes the page table and the address allocator data * structures required for the dma_ops interface */ -static struct dma_ops_domain *dma_ops_domain_alloc(void) +static struct protection_domain *dma_ops_domain_alloc(void) { - struct dma_ops_domain *dma_dom; + struct protection_domain *domain; - dma_dom = kzalloc(sizeof(struct dma_ops_domain), GFP_KERNEL); - if (!dma_dom) + domain = kzalloc(sizeof(struct protection_domain), GFP_KERNEL); + if (!domain) return NULL; - if (protection_domain_init(&dma_dom->domain)) - goto free_dma_dom; - - dma_dom->domain.mode = PAGE_MODE_3_LEVEL; - dma_dom->domain.pt_root = (void *)get_zeroed_page(GFP_KERNEL); - dma_dom->domain.flags = PD_DMA_OPS_MASK; - if (!dma_dom->domain.pt_root) - goto free_dma_dom; - - init_iova_domain(&dma_dom->iovad, PAGE_SIZE, IOVA_START_PFN); + if (protection_domain_init(domain)) + goto free_domain; - if (init_iova_flush_queue(&dma_dom->iovad, iova_domain_flush_tlb, NULL)) - goto free_dma_dom; + domain->mode = PAGE_MODE_3_LEVEL; + domain->pt_root = (void *)get_zeroed_page(GFP_KERNEL); + domain->flags = PD_DMA_OPS_MASK; + if (!domain->pt_root) + goto free_domain; - /* Initialize reserved ranges */ - copy_reserved_iova(&reserved_iova_ranges, &dma_dom->iovad); + if (iommu_get_dma_cookie(&domain->domain) == -ENOMEM) + goto free_domain; - return dma_dom; + return domain; -free_dma_dom: - dma_ops_domain_free(dma_dom); +free_domain: + dma_ops_domain_free(domain); return NULL; } @@ -2233,8 +2148,8 @@ static int amd_iommu_add_device(struct device *dev) domain = iommu_get_domain_for_dev(dev); if (domain->type == IOMMU_DOMAIN_IDENTITY) dev_data->passthrough = true; - else - dev->dma_ops = &amd_iommu_dma_ops; + else if (domain->type == IOMMU_DOMAIN_DMA) + iommu_setup_dma_ops(dev, IOVA_START_PFN << PAGE_SHIFT, 0); out: iommu_completion_wait(iommu); @@ -2268,43 +2183,32 @@ static struct iommu_group *amd_iommu_device_group(struct device *dev) return acpihid_device_group(dev); } +static int amd_iommu_domain_get_attr(struct iommu_domain *domain, + enum iommu_attr attr, void *data) +{ + switch (domain->type) { + case IOMMU_DOMAIN_UNMANAGED: + return -ENODEV; + case IOMMU_DOMAIN_DMA: + switch (attr) { + case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE: + *(int *)data = !amd_iommu_unmap_flush; + return 0; + default: + return -ENODEV; + } + break; + default: + return -EINVAL; + } +} + /***************************************************************************** * * The next functions belong to the dma_ops mapping/unmapping code. * *****************************************************************************/ -/* - * In the dma_ops path we only have the struct device. This function - * finds the corresponding IOMMU, the protection domain and the - * requestor id for a given device. - * If the device is not yet associated with a domain this is also done - * in this function. - */ -static struct protection_domain *get_domain(struct device *dev) -{ - struct protection_domain *domain; - struct iommu_domain *io_domain; - - if (!check_device(dev)) - return ERR_PTR(-EINVAL); - - domain = get_dev_data(dev)->domain; - if (domain == NULL && get_dev_data(dev)->defer_attach) { - get_dev_data(dev)->defer_attach = false; - io_domain = iommu_get_domain_for_dev(dev); - domain = to_pdomain(io_domain); - attach_device(dev, domain); - } - if (domain == NULL) - return ERR_PTR(-EBUSY); - - if (!dma_ops_domain(domain)) - return ERR_PTR(-EBUSY); - - return domain; -} - static void update_device_table(struct protection_domain *domain) { struct iommu_dev_data *dev_data; @@ -2335,447 +2239,6 @@ static void update_domain(struct protection_domain *domain) domain->updated = false; } -static int dir2prot(enum dma_data_direction direction) -{ - if (direction == DMA_TO_DEVICE) - return IOMMU_PROT_IR; - else if (direction == DMA_FROM_DEVICE) - return IOMMU_PROT_IW; - else if (direction == DMA_BIDIRECTIONAL) - return IOMMU_PROT_IW | IOMMU_PROT_IR; - else - return 0; -} - -/* - * This function contains common code for mapping of a physically - * contiguous memory region into DMA address space. It is used by all - * mapping functions provided with this IOMMU driver. - * Must be called with the domain lock held. - */ -static dma_addr_t __map_single(struct device *dev, - struct dma_ops_domain *dma_dom, - phys_addr_t paddr, - size_t size, - enum dma_data_direction direction, - u64 dma_mask) -{ - dma_addr_t offset = paddr & ~PAGE_MASK; - dma_addr_t address, start, ret; - unsigned int pages; - int prot = 0; - int i; - - pages = iommu_num_pages(paddr, size, PAGE_SIZE); - paddr &= PAGE_MASK; - - address = dma_ops_alloc_iova(dev, dma_dom, pages, dma_mask); - if (!address) - goto out; - - prot = dir2prot(direction); - - start = address; - for (i = 0; i < pages; ++i) { - ret = iommu_map_page(&dma_dom->domain, start, paddr, - PAGE_SIZE, prot, GFP_ATOMIC); - if (ret) - goto out_unmap; - - paddr += PAGE_SIZE; - start += PAGE_SIZE; - } - address += offset; - - domain_flush_np_cache(&dma_dom->domain, address, size); - -out: - return address; - -out_unmap: - - for (--i; i >= 0; --i) { - start -= PAGE_SIZE; - iommu_unmap_page(&dma_dom->domain, start, PAGE_SIZE); - } - - domain_flush_tlb(&dma_dom->domain); - domain_flush_complete(&dma_dom->domain); - - dma_ops_free_iova(dma_dom, address, pages); - - return DMA_MAPPING_ERROR; -} - -/* - * Does the reverse of the __map_single function. Must be called with - * the domain lock held too - */ -static void __unmap_single(struct dma_ops_domain *dma_dom, - dma_addr_t dma_addr, - size_t size, - int dir) -{ - dma_addr_t i, start; - unsigned int pages; - - pages = iommu_num_pages(dma_addr, size, PAGE_SIZE); - dma_addr &= PAGE_MASK; - start = dma_addr; - - for (i = 0; i < pages; ++i) { - iommu_unmap_page(&dma_dom->domain, start, PAGE_SIZE); - start += PAGE_SIZE; - } - - if (amd_iommu_unmap_flush) { - domain_flush_tlb(&dma_dom->domain); - domain_flush_complete(&dma_dom->domain); - dma_ops_free_iova(dma_dom, dma_addr, pages); - } else { - pages = __roundup_pow_of_two(pages); - queue_iova(&dma_dom->iovad, dma_addr >> PAGE_SHIFT, pages, 0); - } -} - -/* - * The exported map_single function for dma_ops. - */ -static dma_addr_t map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, - enum dma_data_direction dir, - unsigned long attrs) -{ - phys_addr_t paddr = page_to_phys(page) + offset; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - u64 dma_mask; - - domain = get_domain(dev); - if (PTR_ERR(domain) == -EINVAL) - return (dma_addr_t)paddr; - else if (IS_ERR(domain)) - return DMA_MAPPING_ERROR; - - dma_mask = *dev->dma_mask; - dma_dom = to_dma_ops_domain(domain); - - return __map_single(dev, dma_dom, paddr, size, dir, dma_mask); -} - -/* - * The exported unmap_single function for dma_ops. - */ -static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size, - enum dma_data_direction dir, unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return; - - dma_dom = to_dma_ops_domain(domain); - - __unmap_single(dma_dom, dma_addr, size, dir); -} - -static int sg_num_pages(struct device *dev, - struct scatterlist *sglist, - int nelems) -{ - unsigned long mask, boundary_size; - struct scatterlist *s; - int i, npages = 0; - - mask = dma_get_seg_boundary(dev); - boundary_size = mask + 1 ? ALIGN(mask + 1, PAGE_SIZE) >> PAGE_SHIFT : - 1UL << (BITS_PER_LONG - PAGE_SHIFT); - - for_each_sg(sglist, s, nelems, i) { - int p, n; - - s->dma_address = npages << PAGE_SHIFT; - p = npages % boundary_size; - n = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - if (p + n > boundary_size) - npages += boundary_size - p; - npages += n; - } - - return npages; -} - -/* - * The exported map_sg function for dma_ops (handles scatter-gather - * lists). - */ -static int map_sg(struct device *dev, struct scatterlist *sglist, - int nelems, enum dma_data_direction direction, - unsigned long attrs) -{ - int mapped_pages = 0, npages = 0, prot = 0, i; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct scatterlist *s; - unsigned long address; - u64 dma_mask; - int ret; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return 0; - - dma_dom = to_dma_ops_domain(domain); - dma_mask = *dev->dma_mask; - - npages = sg_num_pages(dev, sglist, nelems); - - address = dma_ops_alloc_iova(dev, dma_dom, npages, dma_mask); - if (!address) - goto out_err; - - prot = dir2prot(direction); - - /* Map all sg entries */ - for_each_sg(sglist, s, nelems, i) { - int j, pages = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - - for (j = 0; j < pages; ++j) { - unsigned long bus_addr, phys_addr; - - bus_addr = address + s->dma_address + (j << PAGE_SHIFT); - phys_addr = (sg_phys(s) & PAGE_MASK) + (j << PAGE_SHIFT); - ret = iommu_map_page(domain, bus_addr, phys_addr, PAGE_SIZE, prot, GFP_ATOMIC); - if (ret) - goto out_unmap; - - mapped_pages += 1; - } - } - - /* Everything is mapped - write the right values into s->dma_address */ - for_each_sg(sglist, s, nelems, i) { - /* - * Add in the remaining piece of the scatter-gather offset that - * was masked out when we were determining the physical address - * via (sg_phys(s) & PAGE_MASK) earlier. - */ - s->dma_address += address + (s->offset & ~PAGE_MASK); - s->dma_length = s->length; - } - - if (s) - domain_flush_np_cache(domain, s->dma_address, s->dma_length); - - return nelems; - -out_unmap: - dev_err(dev, "IOMMU mapping error in map_sg (io-pages: %d reason: %d)\n", - npages, ret); - - for_each_sg(sglist, s, nelems, i) { - int j, pages = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - - for (j = 0; j < pages; ++j) { - unsigned long bus_addr; - - bus_addr = address + s->dma_address + (j << PAGE_SHIFT); - iommu_unmap_page(domain, bus_addr, PAGE_SIZE); - - if (--mapped_pages == 0) - goto out_free_iova; - } - } - -out_free_iova: - free_iova_fast(&dma_dom->iovad, address >> PAGE_SHIFT, npages); - -out_err: - return 0; -} - -/* - * The exported map_sg function for dma_ops (handles scatter-gather - * lists). - */ -static void unmap_sg(struct device *dev, struct scatterlist *sglist, - int nelems, enum dma_data_direction dir, - unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - unsigned long startaddr; - int npages; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return; - - startaddr = sg_dma_address(sglist) & PAGE_MASK; - dma_dom = to_dma_ops_domain(domain); - npages = sg_num_pages(dev, sglist, nelems); - - __unmap_single(dma_dom, startaddr, npages << PAGE_SHIFT, dir); -} - -/* - * The exported alloc_coherent function for dma_ops. - */ -static void *alloc_coherent(struct device *dev, size_t size, - dma_addr_t *dma_addr, gfp_t flag, - unsigned long attrs) -{ - u64 dma_mask = dev->coherent_dma_mask; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct page *page; - - domain = get_domain(dev); - if (PTR_ERR(domain) == -EINVAL) { - page = alloc_pages(flag, get_order(size)); - *dma_addr = page_to_phys(page); - return page_address(page); - } else if (IS_ERR(domain)) - return NULL; - - dma_dom = to_dma_ops_domain(domain); - size = PAGE_ALIGN(size); - dma_mask = dev->coherent_dma_mask; - flag &= ~(__GFP_DMA | __GFP_HIGHMEM | __GFP_DMA32); - flag |= __GFP_ZERO; - - page = alloc_pages(flag | __GFP_NOWARN, get_order(size)); - if (!page) { - if (!gfpflags_allow_blocking(flag)) - return NULL; - - page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, - get_order(size), flag & __GFP_NOWARN); - if (!page) - return NULL; - } - - if (!dma_mask) - dma_mask = *dev->dma_mask; - - *dma_addr = __map_single(dev, dma_dom, page_to_phys(page), - size, DMA_BIDIRECTIONAL, dma_mask); - - if (*dma_addr == DMA_MAPPING_ERROR) - goto out_free; - - return page_address(page); - -out_free: - - if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) - __free_pages(page, get_order(size)); - - return NULL; -} - -/* - * The exported free_coherent function for dma_ops. - */ -static void free_coherent(struct device *dev, size_t size, - void *virt_addr, dma_addr_t dma_addr, - unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct page *page; - - page = virt_to_page(virt_addr); - size = PAGE_ALIGN(size); - - domain = get_domain(dev); - if (IS_ERR(domain)) - goto free_mem; - - dma_dom = to_dma_ops_domain(domain); - - __unmap_single(dma_dom, dma_addr, size, DMA_BIDIRECTIONAL); - -free_mem: - if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) - __free_pages(page, get_order(size)); -} - -/* - * This function is called by the DMA layer to find out if we can handle a - * particular device. It is part of the dma_ops. - */ -static int amd_iommu_dma_supported(struct device *dev, u64 mask) -{ - if (!dma_direct_supported(dev, mask)) - return 0; - return check_device(dev); -} - -static const struct dma_map_ops amd_iommu_dma_ops = { - .alloc = alloc_coherent, - .free = free_coherent, - .map_page = map_page, - .unmap_page = unmap_page, - .map_sg = map_sg, - .unmap_sg = unmap_sg, - .dma_supported = amd_iommu_dma_supported, -}; - -static int init_reserved_iova_ranges(void) -{ - struct pci_dev *pdev = NULL; - struct iova *val; - - init_iova_domain(&reserved_iova_ranges, PAGE_SIZE, IOVA_START_PFN); - - lockdep_set_class(&reserved_iova_ranges.iova_rbtree_lock, - &reserved_rbtree_key); - - /* MSI memory range */ - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(MSI_RANGE_START), IOVA_PFN(MSI_RANGE_END)); - if (!val) { - pr_err("Reserving MSI range failed\n"); - return -ENOMEM; - } - - /* HT memory range */ - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(HT_RANGE_START), IOVA_PFN(HT_RANGE_END)); - if (!val) { - pr_err("Reserving HT range failed\n"); - return -ENOMEM; - } - - /* - * Memory used for PCI resources - * FIXME: Check whether we can reserve the PCI-hole completly - */ - for_each_pci_dev(pdev) { - int i; - - for (i = 0; i < PCI_NUM_RESOURCES; ++i) { - struct resource *r = &pdev->resource[i]; - - if (!(r->flags & IORESOURCE_MEM)) - continue; - - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(r->start), - IOVA_PFN(r->end)); - if (!val) { - pci_err(pdev, "Reserve pci-resource range %pR failed\n", r); - return -ENOMEM; - } - } - } - - return 0; -} - int __init amd_iommu_init_api(void) { int ret, err = 0; @@ -2784,10 +2247,6 @@ int __init amd_iommu_init_api(void) if (ret) return ret; - ret = init_reserved_iova_ranges(); - if (ret) - return ret; - err = bus_set_iommu(&pci_bus_type, &amd_iommu_ops); if (err) return err; @@ -2888,7 +2347,6 @@ static struct protection_domain *protection_domain_alloc(void) static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) { struct protection_domain *pdomain; - struct dma_ops_domain *dma_domain; switch (type) { case IOMMU_DOMAIN_UNMANAGED: @@ -2909,12 +2367,11 @@ static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) break; case IOMMU_DOMAIN_DMA: - dma_domain = dma_ops_domain_alloc(); - if (!dma_domain) { + pdomain = dma_ops_domain_alloc(); + if (!pdomain) { pr_err("Failed to allocate\n"); return NULL; } - pdomain = &dma_domain->domain; break; case IOMMU_DOMAIN_IDENTITY: pdomain = protection_domain_alloc(); @@ -2933,7 +2390,6 @@ static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) static void amd_iommu_domain_free(struct iommu_domain *dom) { struct protection_domain *domain; - struct dma_ops_domain *dma_dom; domain = to_pdomain(dom); @@ -2948,8 +2404,7 @@ static void amd_iommu_domain_free(struct iommu_domain *dom) switch (dom->type) { case IOMMU_DOMAIN_DMA: /* Now release the domain */ - dma_dom = to_dma_ops_domain(domain); - dma_ops_domain_free(dma_dom); + dma_ops_domain_free(domain); break; default: if (domain->mode != PAGE_MODE_NONE) @@ -3005,6 +2460,7 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, return -EINVAL; dev_data = dev->archdata.iommu; + dev_data->defer_attach = false; iommu = amd_iommu_rlookup_table[dev_data->devid]; if (!iommu) @@ -3161,19 +2617,6 @@ static void amd_iommu_put_resv_regions(struct device *dev, kfree(entry); } -static void amd_iommu_apply_resv_region(struct device *dev, - struct iommu_domain *domain, - struct iommu_resv_region *region) -{ - struct dma_ops_domain *dma_dom = to_dma_ops_domain(to_pdomain(domain)); - unsigned long start, end; - - start = IOVA_PFN(region->start); - end = IOVA_PFN(region->start + region->length - 1); - - WARN_ON_ONCE(reserve_iova(&dma_dom->iovad, start, end) == NULL); -} - static bool amd_iommu_is_attach_deferred(struct iommu_domain *domain, struct device *dev) { @@ -3206,9 +2649,9 @@ const struct iommu_ops amd_iommu_ops = { .add_device = amd_iommu_add_device, .remove_device = amd_iommu_remove_device, .device_group = amd_iommu_device_group, + .domain_get_attr = amd_iommu_domain_get_attr, .get_resv_regions = amd_iommu_get_resv_regions, .put_resv_regions = amd_iommu_put_resv_regions, - .apply_resv_region = amd_iommu_apply_resv_region, .is_attach_deferred = amd_iommu_is_attach_deferred, .pgsize_bitmap = AMD_IOMMU_PGSIZES, .flush_iotlb_all = amd_iommu_flush_iotlb_all, @@ -3523,9 +2966,23 @@ EXPORT_SYMBOL(amd_iommu_complete_ppr); struct iommu_domain *amd_iommu_get_v2_domain(struct pci_dev *pdev) { struct protection_domain *pdomain; + struct iommu_domain *io_domain; + struct device *dev = &pdev->dev; + + if (!check_device(dev)) + return NULL; + + pdomain = get_dev_data(dev)->domain; + if (pdomain == NULL && get_dev_data(dev)->defer_attach) { + get_dev_data(dev)->defer_attach = false; + io_domain = iommu_get_domain_for_dev(dev); + pdomain = to_pdomain(io_domain); + attach_device(dev, pdomain); + } + if (pdomain == NULL) + return NULL; - pdomain = get_domain(&pdev->dev); - if (IS_ERR(pdomain)) + if (!dma_ops_domain(pdomain)) return NULL; /* Only return IOMMUv2 domains */ -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 5/5] iommu/amd: Convert AMD iommu driver to the dma-iommu api @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Rob Clark, Kukjin Kim, Robin Murphy Convert the AMD iommu driver to the dma-iommu api. Remove the iova handling and reserve region code from the AMD iommu driver. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/Kconfig | 1 + drivers/iommu/amd_iommu.c | 677 ++++---------------------------------- 2 files changed, 68 insertions(+), 610 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index e15cdcd8cb3c..437428571512 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -138,6 +138,7 @@ config AMD_IOMMU select PCI_PASID select IOMMU_API select IOMMU_IOVA + select IOMMU_DMA depends on X86_64 && PCI && ACPI ---help--- With this option you can enable support for AMD IOMMU hardware in diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 0e53f9bd2be7..eb4801031a99 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -20,6 +20,7 @@ #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/dma-direct.h> +#include <linux/dma-iommu.h> #include <linux/iommu-helper.h> #include <linux/iommu.h> #include <linux/delay.h> @@ -89,8 +90,6 @@ const struct iommu_ops amd_iommu_ops; static ATOMIC_NOTIFIER_HEAD(ppr_notifier); int amd_iommu_max_glx_val = -1; -static const struct dma_map_ops amd_iommu_dma_ops; - /* * general struct to manage commands send to an IOMMU */ @@ -103,21 +102,6 @@ struct kmem_cache *amd_iommu_irq_cache; static void update_domain(struct protection_domain *domain); static int protection_domain_init(struct protection_domain *domain); static void detach_device(struct device *dev); -static void iova_domain_flush_tlb(struct iova_domain *iovad); - -/* - * Data container for a dma_ops specific protection domain - */ -struct dma_ops_domain { - /* generic protection domain information */ - struct protection_domain domain; - - /* IOVA RB-Tree */ - struct iova_domain iovad; -}; - -static struct iova_domain reserved_iova_ranges; -static struct lock_class_key reserved_rbtree_key; /**************************************************************************** * @@ -188,12 +172,6 @@ static struct protection_domain *to_pdomain(struct iommu_domain *dom) return container_of(dom, struct protection_domain, domain); } -static struct dma_ops_domain* to_dma_ops_domain(struct protection_domain *domain) -{ - BUG_ON(domain->flags != PD_DMA_OPS_MASK); - return container_of(domain, struct dma_ops_domain, domain); -} - static struct iommu_dev_data *alloc_dev_data(u16 devid) { struct iommu_dev_data *dev_data; @@ -1267,12 +1245,6 @@ static void domain_flush_pages(struct protection_domain *domain, __domain_flush_pages(domain, address, size, 0); } -/* Flush the whole IO/TLB for a given protection domain */ -static void domain_flush_tlb(struct protection_domain *domain) -{ - __domain_flush_pages(domain, 0, CMD_INV_IOMMU_ALL_PAGES_ADDRESS, 0); -} - /* Flush the whole IO/TLB for a given protection domain - including PDE */ static void domain_flush_tlb_pde(struct protection_domain *domain) { @@ -1674,43 +1646,6 @@ static unsigned long iommu_unmap_page(struct protection_domain *dom, return unmapped; } -/**************************************************************************** - * - * The next functions belong to the address allocator for the dma_ops - * interface functions. - * - ****************************************************************************/ - - -static unsigned long dma_ops_alloc_iova(struct device *dev, - struct dma_ops_domain *dma_dom, - unsigned int pages, u64 dma_mask) -{ - unsigned long pfn = 0; - - pages = __roundup_pow_of_two(pages); - - if (dma_mask > DMA_BIT_MASK(32)) - pfn = alloc_iova_fast(&dma_dom->iovad, pages, - IOVA_PFN(DMA_BIT_MASK(32)), false); - - if (!pfn) - pfn = alloc_iova_fast(&dma_dom->iovad, pages, - IOVA_PFN(dma_mask), true); - - return (pfn << PAGE_SHIFT); -} - -static void dma_ops_free_iova(struct dma_ops_domain *dma_dom, - unsigned long address, - unsigned int pages) -{ - pages = __roundup_pow_of_two(pages); - address >>= PAGE_SHIFT; - - free_iova_fast(&dma_dom->iovad, address, pages); -} - /**************************************************************************** * * The next functions belong to the domain allocation. A domain is @@ -1787,38 +1722,23 @@ static void free_gcr3_table(struct protection_domain *domain) free_page((unsigned long)domain->gcr3_tbl); } -static void dma_ops_domain_flush_tlb(struct dma_ops_domain *dom) -{ - domain_flush_tlb(&dom->domain); - domain_flush_complete(&dom->domain); -} - -static void iova_domain_flush_tlb(struct iova_domain *iovad) -{ - struct dma_ops_domain *dom; - - dom = container_of(iovad, struct dma_ops_domain, iovad); - - dma_ops_domain_flush_tlb(dom); -} - /* * Free a domain, only used if something went wrong in the * allocation path and we need to free an already allocated page table */ -static void dma_ops_domain_free(struct dma_ops_domain *dom) +static void dma_ops_domain_free(struct protection_domain *domain) { - if (!dom) + if (!domain) return; - put_iova_domain(&dom->iovad); + iommu_put_dma_cookie(&domain->domain); - free_pagetable(&dom->domain); + free_pagetable(domain); - if (dom->domain.id) - domain_id_free(dom->domain.id); + if (domain->id) + domain_id_free(domain->id); - kfree(dom); + kfree(domain); } /* @@ -1826,35 +1746,30 @@ static void dma_ops_domain_free(struct dma_ops_domain *dom) * It also initializes the page table and the address allocator data * structures required for the dma_ops interface */ -static struct dma_ops_domain *dma_ops_domain_alloc(void) +static struct protection_domain *dma_ops_domain_alloc(void) { - struct dma_ops_domain *dma_dom; + struct protection_domain *domain; - dma_dom = kzalloc(sizeof(struct dma_ops_domain), GFP_KERNEL); - if (!dma_dom) + domain = kzalloc(sizeof(struct protection_domain), GFP_KERNEL); + if (!domain) return NULL; - if (protection_domain_init(&dma_dom->domain)) - goto free_dma_dom; - - dma_dom->domain.mode = PAGE_MODE_3_LEVEL; - dma_dom->domain.pt_root = (void *)get_zeroed_page(GFP_KERNEL); - dma_dom->domain.flags = PD_DMA_OPS_MASK; - if (!dma_dom->domain.pt_root) - goto free_dma_dom; - - init_iova_domain(&dma_dom->iovad, PAGE_SIZE, IOVA_START_PFN); + if (protection_domain_init(domain)) + goto free_domain; - if (init_iova_flush_queue(&dma_dom->iovad, iova_domain_flush_tlb, NULL)) - goto free_dma_dom; + domain->mode = PAGE_MODE_3_LEVEL; + domain->pt_root = (void *)get_zeroed_page(GFP_KERNEL); + domain->flags = PD_DMA_OPS_MASK; + if (!domain->pt_root) + goto free_domain; - /* Initialize reserved ranges */ - copy_reserved_iova(&reserved_iova_ranges, &dma_dom->iovad); + if (iommu_get_dma_cookie(&domain->domain) == -ENOMEM) + goto free_domain; - return dma_dom; + return domain; -free_dma_dom: - dma_ops_domain_free(dma_dom); +free_domain: + dma_ops_domain_free(domain); return NULL; } @@ -2233,8 +2148,8 @@ static int amd_iommu_add_device(struct device *dev) domain = iommu_get_domain_for_dev(dev); if (domain->type == IOMMU_DOMAIN_IDENTITY) dev_data->passthrough = true; - else - dev->dma_ops = &amd_iommu_dma_ops; + else if (domain->type == IOMMU_DOMAIN_DMA) + iommu_setup_dma_ops(dev, IOVA_START_PFN << PAGE_SHIFT, 0); out: iommu_completion_wait(iommu); @@ -2268,43 +2183,32 @@ static struct iommu_group *amd_iommu_device_group(struct device *dev) return acpihid_device_group(dev); } +static int amd_iommu_domain_get_attr(struct iommu_domain *domain, + enum iommu_attr attr, void *data) +{ + switch (domain->type) { + case IOMMU_DOMAIN_UNMANAGED: + return -ENODEV; + case IOMMU_DOMAIN_DMA: + switch (attr) { + case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE: + *(int *)data = !amd_iommu_unmap_flush; + return 0; + default: + return -ENODEV; + } + break; + default: + return -EINVAL; + } +} + /***************************************************************************** * * The next functions belong to the dma_ops mapping/unmapping code. * *****************************************************************************/ -/* - * In the dma_ops path we only have the struct device. This function - * finds the corresponding IOMMU, the protection domain and the - * requestor id for a given device. - * If the device is not yet associated with a domain this is also done - * in this function. - */ -static struct protection_domain *get_domain(struct device *dev) -{ - struct protection_domain *domain; - struct iommu_domain *io_domain; - - if (!check_device(dev)) - return ERR_PTR(-EINVAL); - - domain = get_dev_data(dev)->domain; - if (domain == NULL && get_dev_data(dev)->defer_attach) { - get_dev_data(dev)->defer_attach = false; - io_domain = iommu_get_domain_for_dev(dev); - domain = to_pdomain(io_domain); - attach_device(dev, domain); - } - if (domain == NULL) - return ERR_PTR(-EBUSY); - - if (!dma_ops_domain(domain)) - return ERR_PTR(-EBUSY); - - return domain; -} - static void update_device_table(struct protection_domain *domain) { struct iommu_dev_data *dev_data; @@ -2335,447 +2239,6 @@ static void update_domain(struct protection_domain *domain) domain->updated = false; } -static int dir2prot(enum dma_data_direction direction) -{ - if (direction == DMA_TO_DEVICE) - return IOMMU_PROT_IR; - else if (direction == DMA_FROM_DEVICE) - return IOMMU_PROT_IW; - else if (direction == DMA_BIDIRECTIONAL) - return IOMMU_PROT_IW | IOMMU_PROT_IR; - else - return 0; -} - -/* - * This function contains common code for mapping of a physically - * contiguous memory region into DMA address space. It is used by all - * mapping functions provided with this IOMMU driver. - * Must be called with the domain lock held. - */ -static dma_addr_t __map_single(struct device *dev, - struct dma_ops_domain *dma_dom, - phys_addr_t paddr, - size_t size, - enum dma_data_direction direction, - u64 dma_mask) -{ - dma_addr_t offset = paddr & ~PAGE_MASK; - dma_addr_t address, start, ret; - unsigned int pages; - int prot = 0; - int i; - - pages = iommu_num_pages(paddr, size, PAGE_SIZE); - paddr &= PAGE_MASK; - - address = dma_ops_alloc_iova(dev, dma_dom, pages, dma_mask); - if (!address) - goto out; - - prot = dir2prot(direction); - - start = address; - for (i = 0; i < pages; ++i) { - ret = iommu_map_page(&dma_dom->domain, start, paddr, - PAGE_SIZE, prot, GFP_ATOMIC); - if (ret) - goto out_unmap; - - paddr += PAGE_SIZE; - start += PAGE_SIZE; - } - address += offset; - - domain_flush_np_cache(&dma_dom->domain, address, size); - -out: - return address; - -out_unmap: - - for (--i; i >= 0; --i) { - start -= PAGE_SIZE; - iommu_unmap_page(&dma_dom->domain, start, PAGE_SIZE); - } - - domain_flush_tlb(&dma_dom->domain); - domain_flush_complete(&dma_dom->domain); - - dma_ops_free_iova(dma_dom, address, pages); - - return DMA_MAPPING_ERROR; -} - -/* - * Does the reverse of the __map_single function. Must be called with - * the domain lock held too - */ -static void __unmap_single(struct dma_ops_domain *dma_dom, - dma_addr_t dma_addr, - size_t size, - int dir) -{ - dma_addr_t i, start; - unsigned int pages; - - pages = iommu_num_pages(dma_addr, size, PAGE_SIZE); - dma_addr &= PAGE_MASK; - start = dma_addr; - - for (i = 0; i < pages; ++i) { - iommu_unmap_page(&dma_dom->domain, start, PAGE_SIZE); - start += PAGE_SIZE; - } - - if (amd_iommu_unmap_flush) { - domain_flush_tlb(&dma_dom->domain); - domain_flush_complete(&dma_dom->domain); - dma_ops_free_iova(dma_dom, dma_addr, pages); - } else { - pages = __roundup_pow_of_two(pages); - queue_iova(&dma_dom->iovad, dma_addr >> PAGE_SHIFT, pages, 0); - } -} - -/* - * The exported map_single function for dma_ops. - */ -static dma_addr_t map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, - enum dma_data_direction dir, - unsigned long attrs) -{ - phys_addr_t paddr = page_to_phys(page) + offset; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - u64 dma_mask; - - domain = get_domain(dev); - if (PTR_ERR(domain) == -EINVAL) - return (dma_addr_t)paddr; - else if (IS_ERR(domain)) - return DMA_MAPPING_ERROR; - - dma_mask = *dev->dma_mask; - dma_dom = to_dma_ops_domain(domain); - - return __map_single(dev, dma_dom, paddr, size, dir, dma_mask); -} - -/* - * The exported unmap_single function for dma_ops. - */ -static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size, - enum dma_data_direction dir, unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return; - - dma_dom = to_dma_ops_domain(domain); - - __unmap_single(dma_dom, dma_addr, size, dir); -} - -static int sg_num_pages(struct device *dev, - struct scatterlist *sglist, - int nelems) -{ - unsigned long mask, boundary_size; - struct scatterlist *s; - int i, npages = 0; - - mask = dma_get_seg_boundary(dev); - boundary_size = mask + 1 ? ALIGN(mask + 1, PAGE_SIZE) >> PAGE_SHIFT : - 1UL << (BITS_PER_LONG - PAGE_SHIFT); - - for_each_sg(sglist, s, nelems, i) { - int p, n; - - s->dma_address = npages << PAGE_SHIFT; - p = npages % boundary_size; - n = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - if (p + n > boundary_size) - npages += boundary_size - p; - npages += n; - } - - return npages; -} - -/* - * The exported map_sg function for dma_ops (handles scatter-gather - * lists). - */ -static int map_sg(struct device *dev, struct scatterlist *sglist, - int nelems, enum dma_data_direction direction, - unsigned long attrs) -{ - int mapped_pages = 0, npages = 0, prot = 0, i; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct scatterlist *s; - unsigned long address; - u64 dma_mask; - int ret; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return 0; - - dma_dom = to_dma_ops_domain(domain); - dma_mask = *dev->dma_mask; - - npages = sg_num_pages(dev, sglist, nelems); - - address = dma_ops_alloc_iova(dev, dma_dom, npages, dma_mask); - if (!address) - goto out_err; - - prot = dir2prot(direction); - - /* Map all sg entries */ - for_each_sg(sglist, s, nelems, i) { - int j, pages = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - - for (j = 0; j < pages; ++j) { - unsigned long bus_addr, phys_addr; - - bus_addr = address + s->dma_address + (j << PAGE_SHIFT); - phys_addr = (sg_phys(s) & PAGE_MASK) + (j << PAGE_SHIFT); - ret = iommu_map_page(domain, bus_addr, phys_addr, PAGE_SIZE, prot, GFP_ATOMIC); - if (ret) - goto out_unmap; - - mapped_pages += 1; - } - } - - /* Everything is mapped - write the right values into s->dma_address */ - for_each_sg(sglist, s, nelems, i) { - /* - * Add in the remaining piece of the scatter-gather offset that - * was masked out when we were determining the physical address - * via (sg_phys(s) & PAGE_MASK) earlier. - */ - s->dma_address += address + (s->offset & ~PAGE_MASK); - s->dma_length = s->length; - } - - if (s) - domain_flush_np_cache(domain, s->dma_address, s->dma_length); - - return nelems; - -out_unmap: - dev_err(dev, "IOMMU mapping error in map_sg (io-pages: %d reason: %d)\n", - npages, ret); - - for_each_sg(sglist, s, nelems, i) { - int j, pages = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - - for (j = 0; j < pages; ++j) { - unsigned long bus_addr; - - bus_addr = address + s->dma_address + (j << PAGE_SHIFT); - iommu_unmap_page(domain, bus_addr, PAGE_SIZE); - - if (--mapped_pages == 0) - goto out_free_iova; - } - } - -out_free_iova: - free_iova_fast(&dma_dom->iovad, address >> PAGE_SHIFT, npages); - -out_err: - return 0; -} - -/* - * The exported map_sg function for dma_ops (handles scatter-gather - * lists). - */ -static void unmap_sg(struct device *dev, struct scatterlist *sglist, - int nelems, enum dma_data_direction dir, - unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - unsigned long startaddr; - int npages; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return; - - startaddr = sg_dma_address(sglist) & PAGE_MASK; - dma_dom = to_dma_ops_domain(domain); - npages = sg_num_pages(dev, sglist, nelems); - - __unmap_single(dma_dom, startaddr, npages << PAGE_SHIFT, dir); -} - -/* - * The exported alloc_coherent function for dma_ops. - */ -static void *alloc_coherent(struct device *dev, size_t size, - dma_addr_t *dma_addr, gfp_t flag, - unsigned long attrs) -{ - u64 dma_mask = dev->coherent_dma_mask; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct page *page; - - domain = get_domain(dev); - if (PTR_ERR(domain) == -EINVAL) { - page = alloc_pages(flag, get_order(size)); - *dma_addr = page_to_phys(page); - return page_address(page); - } else if (IS_ERR(domain)) - return NULL; - - dma_dom = to_dma_ops_domain(domain); - size = PAGE_ALIGN(size); - dma_mask = dev->coherent_dma_mask; - flag &= ~(__GFP_DMA | __GFP_HIGHMEM | __GFP_DMA32); - flag |= __GFP_ZERO; - - page = alloc_pages(flag | __GFP_NOWARN, get_order(size)); - if (!page) { - if (!gfpflags_allow_blocking(flag)) - return NULL; - - page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, - get_order(size), flag & __GFP_NOWARN); - if (!page) - return NULL; - } - - if (!dma_mask) - dma_mask = *dev->dma_mask; - - *dma_addr = __map_single(dev, dma_dom, page_to_phys(page), - size, DMA_BIDIRECTIONAL, dma_mask); - - if (*dma_addr == DMA_MAPPING_ERROR) - goto out_free; - - return page_address(page); - -out_free: - - if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) - __free_pages(page, get_order(size)); - - return NULL; -} - -/* - * The exported free_coherent function for dma_ops. - */ -static void free_coherent(struct device *dev, size_t size, - void *virt_addr, dma_addr_t dma_addr, - unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct page *page; - - page = virt_to_page(virt_addr); - size = PAGE_ALIGN(size); - - domain = get_domain(dev); - if (IS_ERR(domain)) - goto free_mem; - - dma_dom = to_dma_ops_domain(domain); - - __unmap_single(dma_dom, dma_addr, size, DMA_BIDIRECTIONAL); - -free_mem: - if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) - __free_pages(page, get_order(size)); -} - -/* - * This function is called by the DMA layer to find out if we can handle a - * particular device. It is part of the dma_ops. - */ -static int amd_iommu_dma_supported(struct device *dev, u64 mask) -{ - if (!dma_direct_supported(dev, mask)) - return 0; - return check_device(dev); -} - -static const struct dma_map_ops amd_iommu_dma_ops = { - .alloc = alloc_coherent, - .free = free_coherent, - .map_page = map_page, - .unmap_page = unmap_page, - .map_sg = map_sg, - .unmap_sg = unmap_sg, - .dma_supported = amd_iommu_dma_supported, -}; - -static int init_reserved_iova_ranges(void) -{ - struct pci_dev *pdev = NULL; - struct iova *val; - - init_iova_domain(&reserved_iova_ranges, PAGE_SIZE, IOVA_START_PFN); - - lockdep_set_class(&reserved_iova_ranges.iova_rbtree_lock, - &reserved_rbtree_key); - - /* MSI memory range */ - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(MSI_RANGE_START), IOVA_PFN(MSI_RANGE_END)); - if (!val) { - pr_err("Reserving MSI range failed\n"); - return -ENOMEM; - } - - /* HT memory range */ - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(HT_RANGE_START), IOVA_PFN(HT_RANGE_END)); - if (!val) { - pr_err("Reserving HT range failed\n"); - return -ENOMEM; - } - - /* - * Memory used for PCI resources - * FIXME: Check whether we can reserve the PCI-hole completly - */ - for_each_pci_dev(pdev) { - int i; - - for (i = 0; i < PCI_NUM_RESOURCES; ++i) { - struct resource *r = &pdev->resource[i]; - - if (!(r->flags & IORESOURCE_MEM)) - continue; - - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(r->start), - IOVA_PFN(r->end)); - if (!val) { - pci_err(pdev, "Reserve pci-resource range %pR failed\n", r); - return -ENOMEM; - } - } - } - - return 0; -} - int __init amd_iommu_init_api(void) { int ret, err = 0; @@ -2784,10 +2247,6 @@ int __init amd_iommu_init_api(void) if (ret) return ret; - ret = init_reserved_iova_ranges(); - if (ret) - return ret; - err = bus_set_iommu(&pci_bus_type, &amd_iommu_ops); if (err) return err; @@ -2888,7 +2347,6 @@ static struct protection_domain *protection_domain_alloc(void) static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) { struct protection_domain *pdomain; - struct dma_ops_domain *dma_domain; switch (type) { case IOMMU_DOMAIN_UNMANAGED: @@ -2909,12 +2367,11 @@ static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) break; case IOMMU_DOMAIN_DMA: - dma_domain = dma_ops_domain_alloc(); - if (!dma_domain) { + pdomain = dma_ops_domain_alloc(); + if (!pdomain) { pr_err("Failed to allocate\n"); return NULL; } - pdomain = &dma_domain->domain; break; case IOMMU_DOMAIN_IDENTITY: pdomain = protection_domain_alloc(); @@ -2933,7 +2390,6 @@ static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) static void amd_iommu_domain_free(struct iommu_domain *dom) { struct protection_domain *domain; - struct dma_ops_domain *dma_dom; domain = to_pdomain(dom); @@ -2948,8 +2404,7 @@ static void amd_iommu_domain_free(struct iommu_domain *dom) switch (dom->type) { case IOMMU_DOMAIN_DMA: /* Now release the domain */ - dma_dom = to_dma_ops_domain(domain); - dma_ops_domain_free(dma_dom); + dma_ops_domain_free(domain); break; default: if (domain->mode != PAGE_MODE_NONE) @@ -3005,6 +2460,7 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, return -EINVAL; dev_data = dev->archdata.iommu; + dev_data->defer_attach = false; iommu = amd_iommu_rlookup_table[dev_data->devid]; if (!iommu) @@ -3161,19 +2617,6 @@ static void amd_iommu_put_resv_regions(struct device *dev, kfree(entry); } -static void amd_iommu_apply_resv_region(struct device *dev, - struct iommu_domain *domain, - struct iommu_resv_region *region) -{ - struct dma_ops_domain *dma_dom = to_dma_ops_domain(to_pdomain(domain)); - unsigned long start, end; - - start = IOVA_PFN(region->start); - end = IOVA_PFN(region->start + region->length - 1); - - WARN_ON_ONCE(reserve_iova(&dma_dom->iovad, start, end) == NULL); -} - static bool amd_iommu_is_attach_deferred(struct iommu_domain *domain, struct device *dev) { @@ -3206,9 +2649,9 @@ const struct iommu_ops amd_iommu_ops = { .add_device = amd_iommu_add_device, .remove_device = amd_iommu_remove_device, .device_group = amd_iommu_device_group, + .domain_get_attr = amd_iommu_domain_get_attr, .get_resv_regions = amd_iommu_get_resv_regions, .put_resv_regions = amd_iommu_put_resv_regions, - .apply_resv_region = amd_iommu_apply_resv_region, .is_attach_deferred = amd_iommu_is_attach_deferred, .pgsize_bitmap = AMD_IOMMU_PGSIZES, .flush_iotlb_all = amd_iommu_flush_iotlb_all, @@ -3523,9 +2966,23 @@ EXPORT_SYMBOL(amd_iommu_complete_ppr); struct iommu_domain *amd_iommu_get_v2_domain(struct pci_dev *pdev) { struct protection_domain *pdomain; + struct iommu_domain *io_domain; + struct device *dev = &pdev->dev; + + if (!check_device(dev)) + return NULL; + + pdomain = get_dev_data(dev)->domain; + if (pdomain == NULL && get_dev_data(dev)->defer_attach) { + get_dev_data(dev)->defer_attach = false; + io_domain = iommu_get_domain_for_dev(dev); + pdomain = to_pdomain(io_domain); + attach_device(dev, pdomain); + } + if (pdomain == NULL) + return NULL; - pdomain = get_domain(&pdev->dev); - if (IS_ERR(pdomain)) + if (!dma_ops_domain(pdomain)) return NULL; /* Only return IOMMUv2 domains */ -- 2.20.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 5/5] iommu/amd: Convert AMD iommu driver to the dma-iommu api @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Tom Murphy, Kukjin Kim, Robin Murphy Convert the AMD iommu driver to the dma-iommu api. Remove the iova handling and reserve region code from the AMD iommu driver. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/Kconfig | 1 + drivers/iommu/amd_iommu.c | 677 ++++---------------------------------- 2 files changed, 68 insertions(+), 610 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index e15cdcd8cb3c..437428571512 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -138,6 +138,7 @@ config AMD_IOMMU select PCI_PASID select IOMMU_API select IOMMU_IOVA + select IOMMU_DMA depends on X86_64 && PCI && ACPI ---help--- With this option you can enable support for AMD IOMMU hardware in diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 0e53f9bd2be7..eb4801031a99 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -20,6 +20,7 @@ #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/dma-direct.h> +#include <linux/dma-iommu.h> #include <linux/iommu-helper.h> #include <linux/iommu.h> #include <linux/delay.h> @@ -89,8 +90,6 @@ const struct iommu_ops amd_iommu_ops; static ATOMIC_NOTIFIER_HEAD(ppr_notifier); int amd_iommu_max_glx_val = -1; -static const struct dma_map_ops amd_iommu_dma_ops; - /* * general struct to manage commands send to an IOMMU */ @@ -103,21 +102,6 @@ struct kmem_cache *amd_iommu_irq_cache; static void update_domain(struct protection_domain *domain); static int protection_domain_init(struct protection_domain *domain); static void detach_device(struct device *dev); -static void iova_domain_flush_tlb(struct iova_domain *iovad); - -/* - * Data container for a dma_ops specific protection domain - */ -struct dma_ops_domain { - /* generic protection domain information */ - struct protection_domain domain; - - /* IOVA RB-Tree */ - struct iova_domain iovad; -}; - -static struct iova_domain reserved_iova_ranges; -static struct lock_class_key reserved_rbtree_key; /**************************************************************************** * @@ -188,12 +172,6 @@ static struct protection_domain *to_pdomain(struct iommu_domain *dom) return container_of(dom, struct protection_domain, domain); } -static struct dma_ops_domain* to_dma_ops_domain(struct protection_domain *domain) -{ - BUG_ON(domain->flags != PD_DMA_OPS_MASK); - return container_of(domain, struct dma_ops_domain, domain); -} - static struct iommu_dev_data *alloc_dev_data(u16 devid) { struct iommu_dev_data *dev_data; @@ -1267,12 +1245,6 @@ static void domain_flush_pages(struct protection_domain *domain, __domain_flush_pages(domain, address, size, 0); } -/* Flush the whole IO/TLB for a given protection domain */ -static void domain_flush_tlb(struct protection_domain *domain) -{ - __domain_flush_pages(domain, 0, CMD_INV_IOMMU_ALL_PAGES_ADDRESS, 0); -} - /* Flush the whole IO/TLB for a given protection domain - including PDE */ static void domain_flush_tlb_pde(struct protection_domain *domain) { @@ -1674,43 +1646,6 @@ static unsigned long iommu_unmap_page(struct protection_domain *dom, return unmapped; } -/**************************************************************************** - * - * The next functions belong to the address allocator for the dma_ops - * interface functions. - * - ****************************************************************************/ - - -static unsigned long dma_ops_alloc_iova(struct device *dev, - struct dma_ops_domain *dma_dom, - unsigned int pages, u64 dma_mask) -{ - unsigned long pfn = 0; - - pages = __roundup_pow_of_two(pages); - - if (dma_mask > DMA_BIT_MASK(32)) - pfn = alloc_iova_fast(&dma_dom->iovad, pages, - IOVA_PFN(DMA_BIT_MASK(32)), false); - - if (!pfn) - pfn = alloc_iova_fast(&dma_dom->iovad, pages, - IOVA_PFN(dma_mask), true); - - return (pfn << PAGE_SHIFT); -} - -static void dma_ops_free_iova(struct dma_ops_domain *dma_dom, - unsigned long address, - unsigned int pages) -{ - pages = __roundup_pow_of_two(pages); - address >>= PAGE_SHIFT; - - free_iova_fast(&dma_dom->iovad, address, pages); -} - /**************************************************************************** * * The next functions belong to the domain allocation. A domain is @@ -1787,38 +1722,23 @@ static void free_gcr3_table(struct protection_domain *domain) free_page((unsigned long)domain->gcr3_tbl); } -static void dma_ops_domain_flush_tlb(struct dma_ops_domain *dom) -{ - domain_flush_tlb(&dom->domain); - domain_flush_complete(&dom->domain); -} - -static void iova_domain_flush_tlb(struct iova_domain *iovad) -{ - struct dma_ops_domain *dom; - - dom = container_of(iovad, struct dma_ops_domain, iovad); - - dma_ops_domain_flush_tlb(dom); -} - /* * Free a domain, only used if something went wrong in the * allocation path and we need to free an already allocated page table */ -static void dma_ops_domain_free(struct dma_ops_domain *dom) +static void dma_ops_domain_free(struct protection_domain *domain) { - if (!dom) + if (!domain) return; - put_iova_domain(&dom->iovad); + iommu_put_dma_cookie(&domain->domain); - free_pagetable(&dom->domain); + free_pagetable(domain); - if (dom->domain.id) - domain_id_free(dom->domain.id); + if (domain->id) + domain_id_free(domain->id); - kfree(dom); + kfree(domain); } /* @@ -1826,35 +1746,30 @@ static void dma_ops_domain_free(struct dma_ops_domain *dom) * It also initializes the page table and the address allocator data * structures required for the dma_ops interface */ -static struct dma_ops_domain *dma_ops_domain_alloc(void) +static struct protection_domain *dma_ops_domain_alloc(void) { - struct dma_ops_domain *dma_dom; + struct protection_domain *domain; - dma_dom = kzalloc(sizeof(struct dma_ops_domain), GFP_KERNEL); - if (!dma_dom) + domain = kzalloc(sizeof(struct protection_domain), GFP_KERNEL); + if (!domain) return NULL; - if (protection_domain_init(&dma_dom->domain)) - goto free_dma_dom; - - dma_dom->domain.mode = PAGE_MODE_3_LEVEL; - dma_dom->domain.pt_root = (void *)get_zeroed_page(GFP_KERNEL); - dma_dom->domain.flags = PD_DMA_OPS_MASK; - if (!dma_dom->domain.pt_root) - goto free_dma_dom; - - init_iova_domain(&dma_dom->iovad, PAGE_SIZE, IOVA_START_PFN); + if (protection_domain_init(domain)) + goto free_domain; - if (init_iova_flush_queue(&dma_dom->iovad, iova_domain_flush_tlb, NULL)) - goto free_dma_dom; + domain->mode = PAGE_MODE_3_LEVEL; + domain->pt_root = (void *)get_zeroed_page(GFP_KERNEL); + domain->flags = PD_DMA_OPS_MASK; + if (!domain->pt_root) + goto free_domain; - /* Initialize reserved ranges */ - copy_reserved_iova(&reserved_iova_ranges, &dma_dom->iovad); + if (iommu_get_dma_cookie(&domain->domain) == -ENOMEM) + goto free_domain; - return dma_dom; + return domain; -free_dma_dom: - dma_ops_domain_free(dma_dom); +free_domain: + dma_ops_domain_free(domain); return NULL; } @@ -2233,8 +2148,8 @@ static int amd_iommu_add_device(struct device *dev) domain = iommu_get_domain_for_dev(dev); if (domain->type == IOMMU_DOMAIN_IDENTITY) dev_data->passthrough = true; - else - dev->dma_ops = &amd_iommu_dma_ops; + else if (domain->type == IOMMU_DOMAIN_DMA) + iommu_setup_dma_ops(dev, IOVA_START_PFN << PAGE_SHIFT, 0); out: iommu_completion_wait(iommu); @@ -2268,43 +2183,32 @@ static struct iommu_group *amd_iommu_device_group(struct device *dev) return acpihid_device_group(dev); } +static int amd_iommu_domain_get_attr(struct iommu_domain *domain, + enum iommu_attr attr, void *data) +{ + switch (domain->type) { + case IOMMU_DOMAIN_UNMANAGED: + return -ENODEV; + case IOMMU_DOMAIN_DMA: + switch (attr) { + case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE: + *(int *)data = !amd_iommu_unmap_flush; + return 0; + default: + return -ENODEV; + } + break; + default: + return -EINVAL; + } +} + /***************************************************************************** * * The next functions belong to the dma_ops mapping/unmapping code. * *****************************************************************************/ -/* - * In the dma_ops path we only have the struct device. This function - * finds the corresponding IOMMU, the protection domain and the - * requestor id for a given device. - * If the device is not yet associated with a domain this is also done - * in this function. - */ -static struct protection_domain *get_domain(struct device *dev) -{ - struct protection_domain *domain; - struct iommu_domain *io_domain; - - if (!check_device(dev)) - return ERR_PTR(-EINVAL); - - domain = get_dev_data(dev)->domain; - if (domain == NULL && get_dev_data(dev)->defer_attach) { - get_dev_data(dev)->defer_attach = false; - io_domain = iommu_get_domain_for_dev(dev); - domain = to_pdomain(io_domain); - attach_device(dev, domain); - } - if (domain == NULL) - return ERR_PTR(-EBUSY); - - if (!dma_ops_domain(domain)) - return ERR_PTR(-EBUSY); - - return domain; -} - static void update_device_table(struct protection_domain *domain) { struct iommu_dev_data *dev_data; @@ -2335,447 +2239,6 @@ static void update_domain(struct protection_domain *domain) domain->updated = false; } -static int dir2prot(enum dma_data_direction direction) -{ - if (direction == DMA_TO_DEVICE) - return IOMMU_PROT_IR; - else if (direction == DMA_FROM_DEVICE) - return IOMMU_PROT_IW; - else if (direction == DMA_BIDIRECTIONAL) - return IOMMU_PROT_IW | IOMMU_PROT_IR; - else - return 0; -} - -/* - * This function contains common code for mapping of a physically - * contiguous memory region into DMA address space. It is used by all - * mapping functions provided with this IOMMU driver. - * Must be called with the domain lock held. - */ -static dma_addr_t __map_single(struct device *dev, - struct dma_ops_domain *dma_dom, - phys_addr_t paddr, - size_t size, - enum dma_data_direction direction, - u64 dma_mask) -{ - dma_addr_t offset = paddr & ~PAGE_MASK; - dma_addr_t address, start, ret; - unsigned int pages; - int prot = 0; - int i; - - pages = iommu_num_pages(paddr, size, PAGE_SIZE); - paddr &= PAGE_MASK; - - address = dma_ops_alloc_iova(dev, dma_dom, pages, dma_mask); - if (!address) - goto out; - - prot = dir2prot(direction); - - start = address; - for (i = 0; i < pages; ++i) { - ret = iommu_map_page(&dma_dom->domain, start, paddr, - PAGE_SIZE, prot, GFP_ATOMIC); - if (ret) - goto out_unmap; - - paddr += PAGE_SIZE; - start += PAGE_SIZE; - } - address += offset; - - domain_flush_np_cache(&dma_dom->domain, address, size); - -out: - return address; - -out_unmap: - - for (--i; i >= 0; --i) { - start -= PAGE_SIZE; - iommu_unmap_page(&dma_dom->domain, start, PAGE_SIZE); - } - - domain_flush_tlb(&dma_dom->domain); - domain_flush_complete(&dma_dom->domain); - - dma_ops_free_iova(dma_dom, address, pages); - - return DMA_MAPPING_ERROR; -} - -/* - * Does the reverse of the __map_single function. Must be called with - * the domain lock held too - */ -static void __unmap_single(struct dma_ops_domain *dma_dom, - dma_addr_t dma_addr, - size_t size, - int dir) -{ - dma_addr_t i, start; - unsigned int pages; - - pages = iommu_num_pages(dma_addr, size, PAGE_SIZE); - dma_addr &= PAGE_MASK; - start = dma_addr; - - for (i = 0; i < pages; ++i) { - iommu_unmap_page(&dma_dom->domain, start, PAGE_SIZE); - start += PAGE_SIZE; - } - - if (amd_iommu_unmap_flush) { - domain_flush_tlb(&dma_dom->domain); - domain_flush_complete(&dma_dom->domain); - dma_ops_free_iova(dma_dom, dma_addr, pages); - } else { - pages = __roundup_pow_of_two(pages); - queue_iova(&dma_dom->iovad, dma_addr >> PAGE_SHIFT, pages, 0); - } -} - -/* - * The exported map_single function for dma_ops. - */ -static dma_addr_t map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, - enum dma_data_direction dir, - unsigned long attrs) -{ - phys_addr_t paddr = page_to_phys(page) + offset; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - u64 dma_mask; - - domain = get_domain(dev); - if (PTR_ERR(domain) == -EINVAL) - return (dma_addr_t)paddr; - else if (IS_ERR(domain)) - return DMA_MAPPING_ERROR; - - dma_mask = *dev->dma_mask; - dma_dom = to_dma_ops_domain(domain); - - return __map_single(dev, dma_dom, paddr, size, dir, dma_mask); -} - -/* - * The exported unmap_single function for dma_ops. - */ -static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size, - enum dma_data_direction dir, unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return; - - dma_dom = to_dma_ops_domain(domain); - - __unmap_single(dma_dom, dma_addr, size, dir); -} - -static int sg_num_pages(struct device *dev, - struct scatterlist *sglist, - int nelems) -{ - unsigned long mask, boundary_size; - struct scatterlist *s; - int i, npages = 0; - - mask = dma_get_seg_boundary(dev); - boundary_size = mask + 1 ? ALIGN(mask + 1, PAGE_SIZE) >> PAGE_SHIFT : - 1UL << (BITS_PER_LONG - PAGE_SHIFT); - - for_each_sg(sglist, s, nelems, i) { - int p, n; - - s->dma_address = npages << PAGE_SHIFT; - p = npages % boundary_size; - n = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - if (p + n > boundary_size) - npages += boundary_size - p; - npages += n; - } - - return npages; -} - -/* - * The exported map_sg function for dma_ops (handles scatter-gather - * lists). - */ -static int map_sg(struct device *dev, struct scatterlist *sglist, - int nelems, enum dma_data_direction direction, - unsigned long attrs) -{ - int mapped_pages = 0, npages = 0, prot = 0, i; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct scatterlist *s; - unsigned long address; - u64 dma_mask; - int ret; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return 0; - - dma_dom = to_dma_ops_domain(domain); - dma_mask = *dev->dma_mask; - - npages = sg_num_pages(dev, sglist, nelems); - - address = dma_ops_alloc_iova(dev, dma_dom, npages, dma_mask); - if (!address) - goto out_err; - - prot = dir2prot(direction); - - /* Map all sg entries */ - for_each_sg(sglist, s, nelems, i) { - int j, pages = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - - for (j = 0; j < pages; ++j) { - unsigned long bus_addr, phys_addr; - - bus_addr = address + s->dma_address + (j << PAGE_SHIFT); - phys_addr = (sg_phys(s) & PAGE_MASK) + (j << PAGE_SHIFT); - ret = iommu_map_page(domain, bus_addr, phys_addr, PAGE_SIZE, prot, GFP_ATOMIC); - if (ret) - goto out_unmap; - - mapped_pages += 1; - } - } - - /* Everything is mapped - write the right values into s->dma_address */ - for_each_sg(sglist, s, nelems, i) { - /* - * Add in the remaining piece of the scatter-gather offset that - * was masked out when we were determining the physical address - * via (sg_phys(s) & PAGE_MASK) earlier. - */ - s->dma_address += address + (s->offset & ~PAGE_MASK); - s->dma_length = s->length; - } - - if (s) - domain_flush_np_cache(domain, s->dma_address, s->dma_length); - - return nelems; - -out_unmap: - dev_err(dev, "IOMMU mapping error in map_sg (io-pages: %d reason: %d)\n", - npages, ret); - - for_each_sg(sglist, s, nelems, i) { - int j, pages = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - - for (j = 0; j < pages; ++j) { - unsigned long bus_addr; - - bus_addr = address + s->dma_address + (j << PAGE_SHIFT); - iommu_unmap_page(domain, bus_addr, PAGE_SIZE); - - if (--mapped_pages == 0) - goto out_free_iova; - } - } - -out_free_iova: - free_iova_fast(&dma_dom->iovad, address >> PAGE_SHIFT, npages); - -out_err: - return 0; -} - -/* - * The exported map_sg function for dma_ops (handles scatter-gather - * lists). - */ -static void unmap_sg(struct device *dev, struct scatterlist *sglist, - int nelems, enum dma_data_direction dir, - unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - unsigned long startaddr; - int npages; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return; - - startaddr = sg_dma_address(sglist) & PAGE_MASK; - dma_dom = to_dma_ops_domain(domain); - npages = sg_num_pages(dev, sglist, nelems); - - __unmap_single(dma_dom, startaddr, npages << PAGE_SHIFT, dir); -} - -/* - * The exported alloc_coherent function for dma_ops. - */ -static void *alloc_coherent(struct device *dev, size_t size, - dma_addr_t *dma_addr, gfp_t flag, - unsigned long attrs) -{ - u64 dma_mask = dev->coherent_dma_mask; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct page *page; - - domain = get_domain(dev); - if (PTR_ERR(domain) == -EINVAL) { - page = alloc_pages(flag, get_order(size)); - *dma_addr = page_to_phys(page); - return page_address(page); - } else if (IS_ERR(domain)) - return NULL; - - dma_dom = to_dma_ops_domain(domain); - size = PAGE_ALIGN(size); - dma_mask = dev->coherent_dma_mask; - flag &= ~(__GFP_DMA | __GFP_HIGHMEM | __GFP_DMA32); - flag |= __GFP_ZERO; - - page = alloc_pages(flag | __GFP_NOWARN, get_order(size)); - if (!page) { - if (!gfpflags_allow_blocking(flag)) - return NULL; - - page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, - get_order(size), flag & __GFP_NOWARN); - if (!page) - return NULL; - } - - if (!dma_mask) - dma_mask = *dev->dma_mask; - - *dma_addr = __map_single(dev, dma_dom, page_to_phys(page), - size, DMA_BIDIRECTIONAL, dma_mask); - - if (*dma_addr == DMA_MAPPING_ERROR) - goto out_free; - - return page_address(page); - -out_free: - - if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) - __free_pages(page, get_order(size)); - - return NULL; -} - -/* - * The exported free_coherent function for dma_ops. - */ -static void free_coherent(struct device *dev, size_t size, - void *virt_addr, dma_addr_t dma_addr, - unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct page *page; - - page = virt_to_page(virt_addr); - size = PAGE_ALIGN(size); - - domain = get_domain(dev); - if (IS_ERR(domain)) - goto free_mem; - - dma_dom = to_dma_ops_domain(domain); - - __unmap_single(dma_dom, dma_addr, size, DMA_BIDIRECTIONAL); - -free_mem: - if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) - __free_pages(page, get_order(size)); -} - -/* - * This function is called by the DMA layer to find out if we can handle a - * particular device. It is part of the dma_ops. - */ -static int amd_iommu_dma_supported(struct device *dev, u64 mask) -{ - if (!dma_direct_supported(dev, mask)) - return 0; - return check_device(dev); -} - -static const struct dma_map_ops amd_iommu_dma_ops = { - .alloc = alloc_coherent, - .free = free_coherent, - .map_page = map_page, - .unmap_page = unmap_page, - .map_sg = map_sg, - .unmap_sg = unmap_sg, - .dma_supported = amd_iommu_dma_supported, -}; - -static int init_reserved_iova_ranges(void) -{ - struct pci_dev *pdev = NULL; - struct iova *val; - - init_iova_domain(&reserved_iova_ranges, PAGE_SIZE, IOVA_START_PFN); - - lockdep_set_class(&reserved_iova_ranges.iova_rbtree_lock, - &reserved_rbtree_key); - - /* MSI memory range */ - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(MSI_RANGE_START), IOVA_PFN(MSI_RANGE_END)); - if (!val) { - pr_err("Reserving MSI range failed\n"); - return -ENOMEM; - } - - /* HT memory range */ - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(HT_RANGE_START), IOVA_PFN(HT_RANGE_END)); - if (!val) { - pr_err("Reserving HT range failed\n"); - return -ENOMEM; - } - - /* - * Memory used for PCI resources - * FIXME: Check whether we can reserve the PCI-hole completly - */ - for_each_pci_dev(pdev) { - int i; - - for (i = 0; i < PCI_NUM_RESOURCES; ++i) { - struct resource *r = &pdev->resource[i]; - - if (!(r->flags & IORESOURCE_MEM)) - continue; - - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(r->start), - IOVA_PFN(r->end)); - if (!val) { - pci_err(pdev, "Reserve pci-resource range %pR failed\n", r); - return -ENOMEM; - } - } - } - - return 0; -} - int __init amd_iommu_init_api(void) { int ret, err = 0; @@ -2784,10 +2247,6 @@ int __init amd_iommu_init_api(void) if (ret) return ret; - ret = init_reserved_iova_ranges(); - if (ret) - return ret; - err = bus_set_iommu(&pci_bus_type, &amd_iommu_ops); if (err) return err; @@ -2888,7 +2347,6 @@ static struct protection_domain *protection_domain_alloc(void) static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) { struct protection_domain *pdomain; - struct dma_ops_domain *dma_domain; switch (type) { case IOMMU_DOMAIN_UNMANAGED: @@ -2909,12 +2367,11 @@ static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) break; case IOMMU_DOMAIN_DMA: - dma_domain = dma_ops_domain_alloc(); - if (!dma_domain) { + pdomain = dma_ops_domain_alloc(); + if (!pdomain) { pr_err("Failed to allocate\n"); return NULL; } - pdomain = &dma_domain->domain; break; case IOMMU_DOMAIN_IDENTITY: pdomain = protection_domain_alloc(); @@ -2933,7 +2390,6 @@ static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) static void amd_iommu_domain_free(struct iommu_domain *dom) { struct protection_domain *domain; - struct dma_ops_domain *dma_dom; domain = to_pdomain(dom); @@ -2948,8 +2404,7 @@ static void amd_iommu_domain_free(struct iommu_domain *dom) switch (dom->type) { case IOMMU_DOMAIN_DMA: /* Now release the domain */ - dma_dom = to_dma_ops_domain(domain); - dma_ops_domain_free(dma_dom); + dma_ops_domain_free(domain); break; default: if (domain->mode != PAGE_MODE_NONE) @@ -3005,6 +2460,7 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, return -EINVAL; dev_data = dev->archdata.iommu; + dev_data->defer_attach = false; iommu = amd_iommu_rlookup_table[dev_data->devid]; if (!iommu) @@ -3161,19 +2617,6 @@ static void amd_iommu_put_resv_regions(struct device *dev, kfree(entry); } -static void amd_iommu_apply_resv_region(struct device *dev, - struct iommu_domain *domain, - struct iommu_resv_region *region) -{ - struct dma_ops_domain *dma_dom = to_dma_ops_domain(to_pdomain(domain)); - unsigned long start, end; - - start = IOVA_PFN(region->start); - end = IOVA_PFN(region->start + region->length - 1); - - WARN_ON_ONCE(reserve_iova(&dma_dom->iovad, start, end) == NULL); -} - static bool amd_iommu_is_attach_deferred(struct iommu_domain *domain, struct device *dev) { @@ -3206,9 +2649,9 @@ const struct iommu_ops amd_iommu_ops = { .add_device = amd_iommu_add_device, .remove_device = amd_iommu_remove_device, .device_group = amd_iommu_device_group, + .domain_get_attr = amd_iommu_domain_get_attr, .get_resv_regions = amd_iommu_get_resv_regions, .put_resv_regions = amd_iommu_put_resv_regions, - .apply_resv_region = amd_iommu_apply_resv_region, .is_attach_deferred = amd_iommu_is_attach_deferred, .pgsize_bitmap = AMD_IOMMU_PGSIZES, .flush_iotlb_all = amd_iommu_flush_iotlb_all, @@ -3523,9 +2966,23 @@ EXPORT_SYMBOL(amd_iommu_complete_ppr); struct iommu_domain *amd_iommu_get_v2_domain(struct pci_dev *pdev) { struct protection_domain *pdomain; + struct iommu_domain *io_domain; + struct device *dev = &pdev->dev; + + if (!check_device(dev)) + return NULL; + + pdomain = get_dev_data(dev)->domain; + if (pdomain == NULL && get_dev_data(dev)->defer_attach) { + get_dev_data(dev)->defer_attach = false; + io_domain = iommu_get_domain_for_dev(dev); + pdomain = to_pdomain(io_domain); + attach_device(dev, pdomain); + } + if (pdomain == NULL) + return NULL; - pdomain = get_domain(&pdev->dev); - if (IS_ERR(pdomain)) + if (!dma_ops_domain(pdomain)) return NULL; /* Only return IOMMUv2 domains */ -- 2.20.1 _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH V5 5/5] iommu/amd: Convert AMD iommu driver to the dma-iommu api @ 2019-08-15 11:09 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-15 11:09 UTC (permalink / raw) To: iommu Cc: Tom Murphy, Joerg Roedel, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, linux-kernel, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390, linux-tegra, virtualization Convert the AMD iommu driver to the dma-iommu api. Remove the iova handling and reserve region code from the AMD iommu driver. Signed-off-by: Tom Murphy <murphyt7@tcd.ie> --- drivers/iommu/Kconfig | 1 + drivers/iommu/amd_iommu.c | 677 ++++---------------------------------- 2 files changed, 68 insertions(+), 610 deletions(-) diff --git a/drivers/iommu/Kconfig b/drivers/iommu/Kconfig index e15cdcd8cb3c..437428571512 100644 --- a/drivers/iommu/Kconfig +++ b/drivers/iommu/Kconfig @@ -138,6 +138,7 @@ config AMD_IOMMU select PCI_PASID select IOMMU_API select IOMMU_IOVA + select IOMMU_DMA depends on X86_64 && PCI && ACPI ---help--- With this option you can enable support for AMD IOMMU hardware in diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 0e53f9bd2be7..eb4801031a99 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -20,6 +20,7 @@ #include <linux/scatterlist.h> #include <linux/dma-mapping.h> #include <linux/dma-direct.h> +#include <linux/dma-iommu.h> #include <linux/iommu-helper.h> #include <linux/iommu.h> #include <linux/delay.h> @@ -89,8 +90,6 @@ const struct iommu_ops amd_iommu_ops; static ATOMIC_NOTIFIER_HEAD(ppr_notifier); int amd_iommu_max_glx_val = -1; -static const struct dma_map_ops amd_iommu_dma_ops; - /* * general struct to manage commands send to an IOMMU */ @@ -103,21 +102,6 @@ struct kmem_cache *amd_iommu_irq_cache; static void update_domain(struct protection_domain *domain); static int protection_domain_init(struct protection_domain *domain); static void detach_device(struct device *dev); -static void iova_domain_flush_tlb(struct iova_domain *iovad); - -/* - * Data container for a dma_ops specific protection domain - */ -struct dma_ops_domain { - /* generic protection domain information */ - struct protection_domain domain; - - /* IOVA RB-Tree */ - struct iova_domain iovad; -}; - -static struct iova_domain reserved_iova_ranges; -static struct lock_class_key reserved_rbtree_key; /**************************************************************************** * @@ -188,12 +172,6 @@ static struct protection_domain *to_pdomain(struct iommu_domain *dom) return container_of(dom, struct protection_domain, domain); } -static struct dma_ops_domain* to_dma_ops_domain(struct protection_domain *domain) -{ - BUG_ON(domain->flags != PD_DMA_OPS_MASK); - return container_of(domain, struct dma_ops_domain, domain); -} - static struct iommu_dev_data *alloc_dev_data(u16 devid) { struct iommu_dev_data *dev_data; @@ -1267,12 +1245,6 @@ static void domain_flush_pages(struct protection_domain *domain, __domain_flush_pages(domain, address, size, 0); } -/* Flush the whole IO/TLB for a given protection domain */ -static void domain_flush_tlb(struct protection_domain *domain) -{ - __domain_flush_pages(domain, 0, CMD_INV_IOMMU_ALL_PAGES_ADDRESS, 0); -} - /* Flush the whole IO/TLB for a given protection domain - including PDE */ static void domain_flush_tlb_pde(struct protection_domain *domain) { @@ -1674,43 +1646,6 @@ static unsigned long iommu_unmap_page(struct protection_domain *dom, return unmapped; } -/**************************************************************************** - * - * The next functions belong to the address allocator for the dma_ops - * interface functions. - * - ****************************************************************************/ - - -static unsigned long dma_ops_alloc_iova(struct device *dev, - struct dma_ops_domain *dma_dom, - unsigned int pages, u64 dma_mask) -{ - unsigned long pfn = 0; - - pages = __roundup_pow_of_two(pages); - - if (dma_mask > DMA_BIT_MASK(32)) - pfn = alloc_iova_fast(&dma_dom->iovad, pages, - IOVA_PFN(DMA_BIT_MASK(32)), false); - - if (!pfn) - pfn = alloc_iova_fast(&dma_dom->iovad, pages, - IOVA_PFN(dma_mask), true); - - return (pfn << PAGE_SHIFT); -} - -static void dma_ops_free_iova(struct dma_ops_domain *dma_dom, - unsigned long address, - unsigned int pages) -{ - pages = __roundup_pow_of_two(pages); - address >>= PAGE_SHIFT; - - free_iova_fast(&dma_dom->iovad, address, pages); -} - /**************************************************************************** * * The next functions belong to the domain allocation. A domain is @@ -1787,38 +1722,23 @@ static void free_gcr3_table(struct protection_domain *domain) free_page((unsigned long)domain->gcr3_tbl); } -static void dma_ops_domain_flush_tlb(struct dma_ops_domain *dom) -{ - domain_flush_tlb(&dom->domain); - domain_flush_complete(&dom->domain); -} - -static void iova_domain_flush_tlb(struct iova_domain *iovad) -{ - struct dma_ops_domain *dom; - - dom = container_of(iovad, struct dma_ops_domain, iovad); - - dma_ops_domain_flush_tlb(dom); -} - /* * Free a domain, only used if something went wrong in the * allocation path and we need to free an already allocated page table */ -static void dma_ops_domain_free(struct dma_ops_domain *dom) +static void dma_ops_domain_free(struct protection_domain *domain) { - if (!dom) + if (!domain) return; - put_iova_domain(&dom->iovad); + iommu_put_dma_cookie(&domain->domain); - free_pagetable(&dom->domain); + free_pagetable(domain); - if (dom->domain.id) - domain_id_free(dom->domain.id); + if (domain->id) + domain_id_free(domain->id); - kfree(dom); + kfree(domain); } /* @@ -1826,35 +1746,30 @@ static void dma_ops_domain_free(struct dma_ops_domain *dom) * It also initializes the page table and the address allocator data * structures required for the dma_ops interface */ -static struct dma_ops_domain *dma_ops_domain_alloc(void) +static struct protection_domain *dma_ops_domain_alloc(void) { - struct dma_ops_domain *dma_dom; + struct protection_domain *domain; - dma_dom = kzalloc(sizeof(struct dma_ops_domain), GFP_KERNEL); - if (!dma_dom) + domain = kzalloc(sizeof(struct protection_domain), GFP_KERNEL); + if (!domain) return NULL; - if (protection_domain_init(&dma_dom->domain)) - goto free_dma_dom; - - dma_dom->domain.mode = PAGE_MODE_3_LEVEL; - dma_dom->domain.pt_root = (void *)get_zeroed_page(GFP_KERNEL); - dma_dom->domain.flags = PD_DMA_OPS_MASK; - if (!dma_dom->domain.pt_root) - goto free_dma_dom; - - init_iova_domain(&dma_dom->iovad, PAGE_SIZE, IOVA_START_PFN); + if (protection_domain_init(domain)) + goto free_domain; - if (init_iova_flush_queue(&dma_dom->iovad, iova_domain_flush_tlb, NULL)) - goto free_dma_dom; + domain->mode = PAGE_MODE_3_LEVEL; + domain->pt_root = (void *)get_zeroed_page(GFP_KERNEL); + domain->flags = PD_DMA_OPS_MASK; + if (!domain->pt_root) + goto free_domain; - /* Initialize reserved ranges */ - copy_reserved_iova(&reserved_iova_ranges, &dma_dom->iovad); + if (iommu_get_dma_cookie(&domain->domain) == -ENOMEM) + goto free_domain; - return dma_dom; + return domain; -free_dma_dom: - dma_ops_domain_free(dma_dom); +free_domain: + dma_ops_domain_free(domain); return NULL; } @@ -2233,8 +2148,8 @@ static int amd_iommu_add_device(struct device *dev) domain = iommu_get_domain_for_dev(dev); if (domain->type == IOMMU_DOMAIN_IDENTITY) dev_data->passthrough = true; - else - dev->dma_ops = &amd_iommu_dma_ops; + else if (domain->type == IOMMU_DOMAIN_DMA) + iommu_setup_dma_ops(dev, IOVA_START_PFN << PAGE_SHIFT, 0); out: iommu_completion_wait(iommu); @@ -2268,43 +2183,32 @@ static struct iommu_group *amd_iommu_device_group(struct device *dev) return acpihid_device_group(dev); } +static int amd_iommu_domain_get_attr(struct iommu_domain *domain, + enum iommu_attr attr, void *data) +{ + switch (domain->type) { + case IOMMU_DOMAIN_UNMANAGED: + return -ENODEV; + case IOMMU_DOMAIN_DMA: + switch (attr) { + case DOMAIN_ATTR_DMA_USE_FLUSH_QUEUE: + *(int *)data = !amd_iommu_unmap_flush; + return 0; + default: + return -ENODEV; + } + break; + default: + return -EINVAL; + } +} + /***************************************************************************** * * The next functions belong to the dma_ops mapping/unmapping code. * *****************************************************************************/ -/* - * In the dma_ops path we only have the struct device. This function - * finds the corresponding IOMMU, the protection domain and the - * requestor id for a given device. - * If the device is not yet associated with a domain this is also done - * in this function. - */ -static struct protection_domain *get_domain(struct device *dev) -{ - struct protection_domain *domain; - struct iommu_domain *io_domain; - - if (!check_device(dev)) - return ERR_PTR(-EINVAL); - - domain = get_dev_data(dev)->domain; - if (domain == NULL && get_dev_data(dev)->defer_attach) { - get_dev_data(dev)->defer_attach = false; - io_domain = iommu_get_domain_for_dev(dev); - domain = to_pdomain(io_domain); - attach_device(dev, domain); - } - if (domain == NULL) - return ERR_PTR(-EBUSY); - - if (!dma_ops_domain(domain)) - return ERR_PTR(-EBUSY); - - return domain; -} - static void update_device_table(struct protection_domain *domain) { struct iommu_dev_data *dev_data; @@ -2335,447 +2239,6 @@ static void update_domain(struct protection_domain *domain) domain->updated = false; } -static int dir2prot(enum dma_data_direction direction) -{ - if (direction == DMA_TO_DEVICE) - return IOMMU_PROT_IR; - else if (direction == DMA_FROM_DEVICE) - return IOMMU_PROT_IW; - else if (direction == DMA_BIDIRECTIONAL) - return IOMMU_PROT_IW | IOMMU_PROT_IR; - else - return 0; -} - -/* - * This function contains common code for mapping of a physically - * contiguous memory region into DMA address space. It is used by all - * mapping functions provided with this IOMMU driver. - * Must be called with the domain lock held. - */ -static dma_addr_t __map_single(struct device *dev, - struct dma_ops_domain *dma_dom, - phys_addr_t paddr, - size_t size, - enum dma_data_direction direction, - u64 dma_mask) -{ - dma_addr_t offset = paddr & ~PAGE_MASK; - dma_addr_t address, start, ret; - unsigned int pages; - int prot = 0; - int i; - - pages = iommu_num_pages(paddr, size, PAGE_SIZE); - paddr &= PAGE_MASK; - - address = dma_ops_alloc_iova(dev, dma_dom, pages, dma_mask); - if (!address) - goto out; - - prot = dir2prot(direction); - - start = address; - for (i = 0; i < pages; ++i) { - ret = iommu_map_page(&dma_dom->domain, start, paddr, - PAGE_SIZE, prot, GFP_ATOMIC); - if (ret) - goto out_unmap; - - paddr += PAGE_SIZE; - start += PAGE_SIZE; - } - address += offset; - - domain_flush_np_cache(&dma_dom->domain, address, size); - -out: - return address; - -out_unmap: - - for (--i; i >= 0; --i) { - start -= PAGE_SIZE; - iommu_unmap_page(&dma_dom->domain, start, PAGE_SIZE); - } - - domain_flush_tlb(&dma_dom->domain); - domain_flush_complete(&dma_dom->domain); - - dma_ops_free_iova(dma_dom, address, pages); - - return DMA_MAPPING_ERROR; -} - -/* - * Does the reverse of the __map_single function. Must be called with - * the domain lock held too - */ -static void __unmap_single(struct dma_ops_domain *dma_dom, - dma_addr_t dma_addr, - size_t size, - int dir) -{ - dma_addr_t i, start; - unsigned int pages; - - pages = iommu_num_pages(dma_addr, size, PAGE_SIZE); - dma_addr &= PAGE_MASK; - start = dma_addr; - - for (i = 0; i < pages; ++i) { - iommu_unmap_page(&dma_dom->domain, start, PAGE_SIZE); - start += PAGE_SIZE; - } - - if (amd_iommu_unmap_flush) { - domain_flush_tlb(&dma_dom->domain); - domain_flush_complete(&dma_dom->domain); - dma_ops_free_iova(dma_dom, dma_addr, pages); - } else { - pages = __roundup_pow_of_two(pages); - queue_iova(&dma_dom->iovad, dma_addr >> PAGE_SHIFT, pages, 0); - } -} - -/* - * The exported map_single function for dma_ops. - */ -static dma_addr_t map_page(struct device *dev, struct page *page, - unsigned long offset, size_t size, - enum dma_data_direction dir, - unsigned long attrs) -{ - phys_addr_t paddr = page_to_phys(page) + offset; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - u64 dma_mask; - - domain = get_domain(dev); - if (PTR_ERR(domain) == -EINVAL) - return (dma_addr_t)paddr; - else if (IS_ERR(domain)) - return DMA_MAPPING_ERROR; - - dma_mask = *dev->dma_mask; - dma_dom = to_dma_ops_domain(domain); - - return __map_single(dev, dma_dom, paddr, size, dir, dma_mask); -} - -/* - * The exported unmap_single function for dma_ops. - */ -static void unmap_page(struct device *dev, dma_addr_t dma_addr, size_t size, - enum dma_data_direction dir, unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return; - - dma_dom = to_dma_ops_domain(domain); - - __unmap_single(dma_dom, dma_addr, size, dir); -} - -static int sg_num_pages(struct device *dev, - struct scatterlist *sglist, - int nelems) -{ - unsigned long mask, boundary_size; - struct scatterlist *s; - int i, npages = 0; - - mask = dma_get_seg_boundary(dev); - boundary_size = mask + 1 ? ALIGN(mask + 1, PAGE_SIZE) >> PAGE_SHIFT : - 1UL << (BITS_PER_LONG - PAGE_SHIFT); - - for_each_sg(sglist, s, nelems, i) { - int p, n; - - s->dma_address = npages << PAGE_SHIFT; - p = npages % boundary_size; - n = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - if (p + n > boundary_size) - npages += boundary_size - p; - npages += n; - } - - return npages; -} - -/* - * The exported map_sg function for dma_ops (handles scatter-gather - * lists). - */ -static int map_sg(struct device *dev, struct scatterlist *sglist, - int nelems, enum dma_data_direction direction, - unsigned long attrs) -{ - int mapped_pages = 0, npages = 0, prot = 0, i; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct scatterlist *s; - unsigned long address; - u64 dma_mask; - int ret; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return 0; - - dma_dom = to_dma_ops_domain(domain); - dma_mask = *dev->dma_mask; - - npages = sg_num_pages(dev, sglist, nelems); - - address = dma_ops_alloc_iova(dev, dma_dom, npages, dma_mask); - if (!address) - goto out_err; - - prot = dir2prot(direction); - - /* Map all sg entries */ - for_each_sg(sglist, s, nelems, i) { - int j, pages = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - - for (j = 0; j < pages; ++j) { - unsigned long bus_addr, phys_addr; - - bus_addr = address + s->dma_address + (j << PAGE_SHIFT); - phys_addr = (sg_phys(s) & PAGE_MASK) + (j << PAGE_SHIFT); - ret = iommu_map_page(domain, bus_addr, phys_addr, PAGE_SIZE, prot, GFP_ATOMIC); - if (ret) - goto out_unmap; - - mapped_pages += 1; - } - } - - /* Everything is mapped - write the right values into s->dma_address */ - for_each_sg(sglist, s, nelems, i) { - /* - * Add in the remaining piece of the scatter-gather offset that - * was masked out when we were determining the physical address - * via (sg_phys(s) & PAGE_MASK) earlier. - */ - s->dma_address += address + (s->offset & ~PAGE_MASK); - s->dma_length = s->length; - } - - if (s) - domain_flush_np_cache(domain, s->dma_address, s->dma_length); - - return nelems; - -out_unmap: - dev_err(dev, "IOMMU mapping error in map_sg (io-pages: %d reason: %d)\n", - npages, ret); - - for_each_sg(sglist, s, nelems, i) { - int j, pages = iommu_num_pages(sg_phys(s), s->length, PAGE_SIZE); - - for (j = 0; j < pages; ++j) { - unsigned long bus_addr; - - bus_addr = address + s->dma_address + (j << PAGE_SHIFT); - iommu_unmap_page(domain, bus_addr, PAGE_SIZE); - - if (--mapped_pages == 0) - goto out_free_iova; - } - } - -out_free_iova: - free_iova_fast(&dma_dom->iovad, address >> PAGE_SHIFT, npages); - -out_err: - return 0; -} - -/* - * The exported map_sg function for dma_ops (handles scatter-gather - * lists). - */ -static void unmap_sg(struct device *dev, struct scatterlist *sglist, - int nelems, enum dma_data_direction dir, - unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - unsigned long startaddr; - int npages; - - domain = get_domain(dev); - if (IS_ERR(domain)) - return; - - startaddr = sg_dma_address(sglist) & PAGE_MASK; - dma_dom = to_dma_ops_domain(domain); - npages = sg_num_pages(dev, sglist, nelems); - - __unmap_single(dma_dom, startaddr, npages << PAGE_SHIFT, dir); -} - -/* - * The exported alloc_coherent function for dma_ops. - */ -static void *alloc_coherent(struct device *dev, size_t size, - dma_addr_t *dma_addr, gfp_t flag, - unsigned long attrs) -{ - u64 dma_mask = dev->coherent_dma_mask; - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct page *page; - - domain = get_domain(dev); - if (PTR_ERR(domain) == -EINVAL) { - page = alloc_pages(flag, get_order(size)); - *dma_addr = page_to_phys(page); - return page_address(page); - } else if (IS_ERR(domain)) - return NULL; - - dma_dom = to_dma_ops_domain(domain); - size = PAGE_ALIGN(size); - dma_mask = dev->coherent_dma_mask; - flag &= ~(__GFP_DMA | __GFP_HIGHMEM | __GFP_DMA32); - flag |= __GFP_ZERO; - - page = alloc_pages(flag | __GFP_NOWARN, get_order(size)); - if (!page) { - if (!gfpflags_allow_blocking(flag)) - return NULL; - - page = dma_alloc_from_contiguous(dev, size >> PAGE_SHIFT, - get_order(size), flag & __GFP_NOWARN); - if (!page) - return NULL; - } - - if (!dma_mask) - dma_mask = *dev->dma_mask; - - *dma_addr = __map_single(dev, dma_dom, page_to_phys(page), - size, DMA_BIDIRECTIONAL, dma_mask); - - if (*dma_addr == DMA_MAPPING_ERROR) - goto out_free; - - return page_address(page); - -out_free: - - if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) - __free_pages(page, get_order(size)); - - return NULL; -} - -/* - * The exported free_coherent function for dma_ops. - */ -static void free_coherent(struct device *dev, size_t size, - void *virt_addr, dma_addr_t dma_addr, - unsigned long attrs) -{ - struct protection_domain *domain; - struct dma_ops_domain *dma_dom; - struct page *page; - - page = virt_to_page(virt_addr); - size = PAGE_ALIGN(size); - - domain = get_domain(dev); - if (IS_ERR(domain)) - goto free_mem; - - dma_dom = to_dma_ops_domain(domain); - - __unmap_single(dma_dom, dma_addr, size, DMA_BIDIRECTIONAL); - -free_mem: - if (!dma_release_from_contiguous(dev, page, size >> PAGE_SHIFT)) - __free_pages(page, get_order(size)); -} - -/* - * This function is called by the DMA layer to find out if we can handle a - * particular device. It is part of the dma_ops. - */ -static int amd_iommu_dma_supported(struct device *dev, u64 mask) -{ - if (!dma_direct_supported(dev, mask)) - return 0; - return check_device(dev); -} - -static const struct dma_map_ops amd_iommu_dma_ops = { - .alloc = alloc_coherent, - .free = free_coherent, - .map_page = map_page, - .unmap_page = unmap_page, - .map_sg = map_sg, - .unmap_sg = unmap_sg, - .dma_supported = amd_iommu_dma_supported, -}; - -static int init_reserved_iova_ranges(void) -{ - struct pci_dev *pdev = NULL; - struct iova *val; - - init_iova_domain(&reserved_iova_ranges, PAGE_SIZE, IOVA_START_PFN); - - lockdep_set_class(&reserved_iova_ranges.iova_rbtree_lock, - &reserved_rbtree_key); - - /* MSI memory range */ - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(MSI_RANGE_START), IOVA_PFN(MSI_RANGE_END)); - if (!val) { - pr_err("Reserving MSI range failed\n"); - return -ENOMEM; - } - - /* HT memory range */ - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(HT_RANGE_START), IOVA_PFN(HT_RANGE_END)); - if (!val) { - pr_err("Reserving HT range failed\n"); - return -ENOMEM; - } - - /* - * Memory used for PCI resources - * FIXME: Check whether we can reserve the PCI-hole completly - */ - for_each_pci_dev(pdev) { - int i; - - for (i = 0; i < PCI_NUM_RESOURCES; ++i) { - struct resource *r = &pdev->resource[i]; - - if (!(r->flags & IORESOURCE_MEM)) - continue; - - val = reserve_iova(&reserved_iova_ranges, - IOVA_PFN(r->start), - IOVA_PFN(r->end)); - if (!val) { - pci_err(pdev, "Reserve pci-resource range %pR failed\n", r); - return -ENOMEM; - } - } - } - - return 0; -} - int __init amd_iommu_init_api(void) { int ret, err = 0; @@ -2784,10 +2247,6 @@ int __init amd_iommu_init_api(void) if (ret) return ret; - ret = init_reserved_iova_ranges(); - if (ret) - return ret; - err = bus_set_iommu(&pci_bus_type, &amd_iommu_ops); if (err) return err; @@ -2888,7 +2347,6 @@ static struct protection_domain *protection_domain_alloc(void) static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) { struct protection_domain *pdomain; - struct dma_ops_domain *dma_domain; switch (type) { case IOMMU_DOMAIN_UNMANAGED: @@ -2909,12 +2367,11 @@ static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) break; case IOMMU_DOMAIN_DMA: - dma_domain = dma_ops_domain_alloc(); - if (!dma_domain) { + pdomain = dma_ops_domain_alloc(); + if (!pdomain) { pr_err("Failed to allocate\n"); return NULL; } - pdomain = &dma_domain->domain; break; case IOMMU_DOMAIN_IDENTITY: pdomain = protection_domain_alloc(); @@ -2933,7 +2390,6 @@ static struct iommu_domain *amd_iommu_domain_alloc(unsigned type) static void amd_iommu_domain_free(struct iommu_domain *dom) { struct protection_domain *domain; - struct dma_ops_domain *dma_dom; domain = to_pdomain(dom); @@ -2948,8 +2404,7 @@ static void amd_iommu_domain_free(struct iommu_domain *dom) switch (dom->type) { case IOMMU_DOMAIN_DMA: /* Now release the domain */ - dma_dom = to_dma_ops_domain(domain); - dma_ops_domain_free(dma_dom); + dma_ops_domain_free(domain); break; default: if (domain->mode != PAGE_MODE_NONE) @@ -3005,6 +2460,7 @@ static int amd_iommu_attach_device(struct iommu_domain *dom, return -EINVAL; dev_data = dev->archdata.iommu; + dev_data->defer_attach = false; iommu = amd_iommu_rlookup_table[dev_data->devid]; if (!iommu) @@ -3161,19 +2617,6 @@ static void amd_iommu_put_resv_regions(struct device *dev, kfree(entry); } -static void amd_iommu_apply_resv_region(struct device *dev, - struct iommu_domain *domain, - struct iommu_resv_region *region) -{ - struct dma_ops_domain *dma_dom = to_dma_ops_domain(to_pdomain(domain)); - unsigned long start, end; - - start = IOVA_PFN(region->start); - end = IOVA_PFN(region->start + region->length - 1); - - WARN_ON_ONCE(reserve_iova(&dma_dom->iovad, start, end) == NULL); -} - static bool amd_iommu_is_attach_deferred(struct iommu_domain *domain, struct device *dev) { @@ -3206,9 +2649,9 @@ const struct iommu_ops amd_iommu_ops = { .add_device = amd_iommu_add_device, .remove_device = amd_iommu_remove_device, .device_group = amd_iommu_device_group, + .domain_get_attr = amd_iommu_domain_get_attr, .get_resv_regions = amd_iommu_get_resv_regions, .put_resv_regions = amd_iommu_put_resv_regions, - .apply_resv_region = amd_iommu_apply_resv_region, .is_attach_deferred = amd_iommu_is_attach_deferred, .pgsize_bitmap = AMD_IOMMU_PGSIZES, .flush_iotlb_all = amd_iommu_flush_iotlb_all, @@ -3523,9 +2966,23 @@ EXPORT_SYMBOL(amd_iommu_complete_ppr); struct iommu_domain *amd_iommu_get_v2_domain(struct pci_dev *pdev) { struct protection_domain *pdomain; + struct iommu_domain *io_domain; + struct device *dev = &pdev->dev; + + if (!check_device(dev)) + return NULL; + + pdomain = get_dev_data(dev)->domain; + if (pdomain == NULL && get_dev_data(dev)->defer_attach) { + get_dev_data(dev)->defer_attach = false; + io_domain = iommu_get_domain_for_dev(dev); + pdomain = to_pdomain(io_domain); + attach_device(dev, pdomain); + } + if (pdomain == NULL) + return NULL; - pdomain = get_domain(&pdev->dev); - if (IS_ERR(pdomain)) + if (!dma_ops_domain(pdomain)) return NULL; /* Only return IOMMUv2 domains */ -- 2.20.1 ^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices 2019-08-15 11:09 ` Tom Murphy (?) @ 2019-08-17 3:39 ` Hillf Danton -1 siblings, 0 replies; 87+ messages in thread From: Hillf Danton @ 2019-08-17 3:39 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, iommu, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Rob Clark, Kukjin Kim On Thu, 15 Aug 2019 12:09:41 +0100 Tom Murphy wrote: > > Handle devices which defer their attach to the iommu in the dma-iommu api > > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > 1 file changed, 26 insertions(+), 1 deletion(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 2712fbc68b28..906b7fa14d3c 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -22,6 +22,7 @@ > #include <linux/pci.h> > #include <linux/scatterlist.h> > #include <linux/vmalloc.h> > +#include <linux/crash_dump.h> > > struct iommu_dma_msi_page { > struct list_head list; > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > return iova_reserve_iommu_regions(dev, domain); > } > > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) > +{ > + const struct iommu_ops *ops = domain->ops; > + > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); > + > + return 0; > +} > + > /** > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > * page flags. > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > size_t iova_off = iova_offset(iovad, phys); > dma_addr_t iova; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return DMA_MAPPING_ERROR; > + > size = iova_align(iovad, size + iova_off); > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); iommu_map_atomic() is applied to __iommu_dma_map() in 2/5. Is it an atomic context currently given the mutex_lock() in iommu_attach_device()? ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-17 3:39 ` Hillf Danton 0 siblings, 0 replies; 87+ messages in thread From: Hillf Danton @ 2019-08-17 3:39 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, iommu, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Rob Clark, Kukjin Kim, Robin Murphy On Thu, 15 Aug 2019 12:09:41 +0100 Tom Murphy wrote: > > Handle devices which defer their attach to the iommu in the dma-iommu api > > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > 1 file changed, 26 insertions(+), 1 deletion(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 2712fbc68b28..906b7fa14d3c 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -22,6 +22,7 @@ > #include <linux/pci.h> > #include <linux/scatterlist.h> > #include <linux/vmalloc.h> > +#include <linux/crash_dump.h> > > struct iommu_dma_msi_page { > struct list_head list; > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > return iova_reserve_iommu_regions(dev, domain); > } > > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) > +{ > + const struct iommu_ops *ops = domain->ops; > + > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); > + > + return 0; > +} > + > /** > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > * page flags. > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > size_t iova_off = iova_offset(iovad, phys); > dma_addr_t iova; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return DMA_MAPPING_ERROR; > + > size = iova_align(iovad, size + iova_off); > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); iommu_map_atomic() is applied to __iommu_dma_map() in 2/5. Is it an atomic context currently given the mutex_lock() in iommu_attach_device()? _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-17 3:39 ` Hillf Danton 0 siblings, 0 replies; 87+ messages in thread From: Hillf Danton @ 2019-08-17 3:39 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, iommu, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Kukjin Kim, Robin Murphy On Thu, 15 Aug 2019 12:09:41 +0100 Tom Murphy wrote: > > Handle devices which defer their attach to the iommu in the dma-iommu api > > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > --- > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > 1 file changed, 26 insertions(+), 1 deletion(-) > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > index 2712fbc68b28..906b7fa14d3c 100644 > --- a/drivers/iommu/dma-iommu.c > +++ b/drivers/iommu/dma-iommu.c > @@ -22,6 +22,7 @@ > #include <linux/pci.h> > #include <linux/scatterlist.h> > #include <linux/vmalloc.h> > +#include <linux/crash_dump.h> > > struct iommu_dma_msi_page { > struct list_head list; > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > return iova_reserve_iommu_regions(dev, domain); > } > > +static int handle_deferred_device(struct device *dev, > + struct iommu_domain *domain) > +{ > + const struct iommu_ops *ops = domain->ops; > + > + if (!is_kdump_kernel()) > + return 0; > + > + if (unlikely(ops->is_attach_deferred && > + ops->is_attach_deferred(domain, dev))) > + return iommu_attach_device(domain, dev); > + > + return 0; > +} > + > /** > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > * page flags. > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > size_t iova_off = iova_offset(iovad, phys); > dma_addr_t iova; > > + if (unlikely(handle_deferred_device(dev, domain))) > + return DMA_MAPPING_ERROR; > + > size = iova_align(iovad, size + iova_off); > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); iommu_map_atomic() is applied to __iommu_dma_map() in 2/5. Is it an atomic context currently given the mutex_lock() in iommu_attach_device()? _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices 2019-08-17 3:39 ` Hillf Danton (?) (?) @ 2019-08-17 7:19 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-17 7:19 UTC (permalink / raw) To: Hillf Danton Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, iommu, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, Linux Kernel Mailing List, Rob Clark, Kukji On Sat, 17 Aug 2019 at 04:39, Hillf Danton <hdanton@sina.com> wrote: > > > On Thu, 15 Aug 2019 12:09:41 +0100 Tom Murphy wrote: > > > > Handle devices which defer their attach to the iommu in the dma-iommu api > > > > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > > --- > > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > > 1 file changed, 26 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > > index 2712fbc68b28..906b7fa14d3c 100644 > > --- a/drivers/iommu/dma-iommu.c > > +++ b/drivers/iommu/dma-iommu.c > > @@ -22,6 +22,7 @@ > > #include <linux/pci.h> > > #include <linux/scatterlist.h> > > #include <linux/vmalloc.h> > > +#include <linux/crash_dump.h> > > > > struct iommu_dma_msi_page { > > struct list_head list; > > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > > return iova_reserve_iommu_regions(dev, domain); > > } > > > > +static int handle_deferred_device(struct device *dev, > > + struct iommu_domain *domain) > > +{ > > + const struct iommu_ops *ops = domain->ops; > > + > > + if (!is_kdump_kernel()) > > + return 0; > > + > > + if (unlikely(ops->is_attach_deferred && > > + ops->is_attach_deferred(domain, dev))) > > + return iommu_attach_device(domain, dev); > > + > > + return 0; > > +} > > + > > /** > > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > > * page flags. > > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > > size_t iova_off = iova_offset(iovad, phys); > > dma_addr_t iova; > > > > + if (unlikely(handle_deferred_device(dev, domain))) > > + return DMA_MAPPING_ERROR; > > + > > size = iova_align(iovad, size + iova_off); > > > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > > iommu_map_atomic() is applied to __iommu_dma_map() in 2/5. > Is it an atomic context currently given the mutex_lock() in > iommu_attach_device()? I don't see your point here. __iommu_dma_map isn't called from iommu_attach_device, why would we care about a mutex in iommu_attach_device? __iommu_dma_map can be called from an atomic context (it isn't always but it does happen). __iommu_dma_map is called by iommu_dma_alloc which implements the iommu_dma_ops::alloc function which by design needs to be callable from an atomic context. Does that answer your question? > ^ permalink raw reply [flat|nested] 87+ messages in thread
[parent not found: <20190817033914.4812-1-hdanton-k+cT0dCbe1g@public.gmane.org>]
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices 2019-08-17 3:39 ` Hillf Danton (?) (?) @ 2019-08-17 7:19 ` Tom Murphy -1 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-17 7:19 UTC (permalink / raw) To: Hillf Danton Cc: Heiko Stuebner, virtualization-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, linux-tegra-u79uwXL29TY76Z2rM5mHXA, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc-u79uwXL29TY76Z2rM5mHXA, iommu-cunTk1MwBs9QetFLy7KEm3xJsTq8ys+cHZ5vskTnxNA, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, Andy Gross, Gerald Schaefer, linux-s390-u79uwXL29TY76Z2rM5mHXA, linux-arm-msm-u79uwXL29TY76Z2rM5mHXA, linux-mediatek-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, Matthias Brugger, linux-arm-kernel-IAPFreCvJWM7uuMidbF8XUB+6BGkLq7r, David Woodhouse, Linux Kernel Mailing List, Kukjin Kim, Robin Murphy On Sat, 17 Aug 2019 at 04:39, Hillf Danton <hdanton-k+cT0dCbe1g@public.gmane.org> wrote: > > > On Thu, 15 Aug 2019 12:09:41 +0100 Tom Murphy wrote: > > > > Handle devices which defer their attach to the iommu in the dma-iommu api > > > > Signed-off-by: Tom Murphy <murphyt7-/Zw1syy4LHg@public.gmane.org> > > --- > > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > > 1 file changed, 26 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > > index 2712fbc68b28..906b7fa14d3c 100644 > > --- a/drivers/iommu/dma-iommu.c > > +++ b/drivers/iommu/dma-iommu.c > > @@ -22,6 +22,7 @@ > > #include <linux/pci.h> > > #include <linux/scatterlist.h> > > #include <linux/vmalloc.h> > > +#include <linux/crash_dump.h> > > > > struct iommu_dma_msi_page { > > struct list_head list; > > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > > return iova_reserve_iommu_regions(dev, domain); > > } > > > > +static int handle_deferred_device(struct device *dev, > > + struct iommu_domain *domain) > > +{ > > + const struct iommu_ops *ops = domain->ops; > > + > > + if (!is_kdump_kernel()) > > + return 0; > > + > > + if (unlikely(ops->is_attach_deferred && > > + ops->is_attach_deferred(domain, dev))) > > + return iommu_attach_device(domain, dev); > > + > > + return 0; > > +} > > + > > /** > > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > > * page flags. > > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > > size_t iova_off = iova_offset(iovad, phys); > > dma_addr_t iova; > > > > + if (unlikely(handle_deferred_device(dev, domain))) > > + return DMA_MAPPING_ERROR; > > + > > size = iova_align(iovad, size + iova_off); > > > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > > iommu_map_atomic() is applied to __iommu_dma_map() in 2/5. > Is it an atomic context currently given the mutex_lock() in > iommu_attach_device()? I don't see your point here. __iommu_dma_map isn't called from iommu_attach_device, why would we care about a mutex in iommu_attach_device? __iommu_dma_map can be called from an atomic context (it isn't always but it does happen). __iommu_dma_map is called by iommu_dma_alloc which implements the iommu_dma_ops::alloc function which by design needs to be callable from an atomic context. Does that answer your question? > ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-17 7:19 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-17 7:19 UTC (permalink / raw) To: Hillf Danton Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Marek Szyprowski, Jean-Philippe Brucker, linux-samsung-soc, Joerg Roedel, iommu, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, Linux Kernel Mailing List, Rob Clark, Kukjin Kim, Robin Murphy On Sat, 17 Aug 2019 at 04:39, Hillf Danton <hdanton@sina.com> wrote: > > > On Thu, 15 Aug 2019 12:09:41 +0100 Tom Murphy wrote: > > > > Handle devices which defer their attach to the iommu in the dma-iommu api > > > > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > > --- > > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > > 1 file changed, 26 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > > index 2712fbc68b28..906b7fa14d3c 100644 > > --- a/drivers/iommu/dma-iommu.c > > +++ b/drivers/iommu/dma-iommu.c > > @@ -22,6 +22,7 @@ > > #include <linux/pci.h> > > #include <linux/scatterlist.h> > > #include <linux/vmalloc.h> > > +#include <linux/crash_dump.h> > > > > struct iommu_dma_msi_page { > > struct list_head list; > > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > > return iova_reserve_iommu_regions(dev, domain); > > } > > > > +static int handle_deferred_device(struct device *dev, > > + struct iommu_domain *domain) > > +{ > > + const struct iommu_ops *ops = domain->ops; > > + > > + if (!is_kdump_kernel()) > > + return 0; > > + > > + if (unlikely(ops->is_attach_deferred && > > + ops->is_attach_deferred(domain, dev))) > > + return iommu_attach_device(domain, dev); > > + > > + return 0; > > +} > > + > > /** > > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > > * page flags. > > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > > size_t iova_off = iova_offset(iovad, phys); > > dma_addr_t iova; > > > > + if (unlikely(handle_deferred_device(dev, domain))) > > + return DMA_MAPPING_ERROR; > > + > > size = iova_align(iovad, size + iova_off); > > > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > > iommu_map_atomic() is applied to __iommu_dma_map() in 2/5. > Is it an atomic context currently given the mutex_lock() in > iommu_attach_device()? I don't see your point here. __iommu_dma_map isn't called from iommu_attach_device, why would we care about a mutex in iommu_attach_device? __iommu_dma_map can be called from an atomic context (it isn't always but it does happen). __iommu_dma_map is called by iommu_dma_alloc which implements the iommu_dma_ops::alloc function which by design needs to be callable from an atomic context. Does that answer your question? > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-17 7:19 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-17 7:19 UTC (permalink / raw) To: Hillf Danton Cc: Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, iommu, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, Linux Kernel Mailing List, Kukjin Kim, Robin Murphy On Sat, 17 Aug 2019 at 04:39, Hillf Danton <hdanton@sina.com> wrote: > > > On Thu, 15 Aug 2019 12:09:41 +0100 Tom Murphy wrote: > > > > Handle devices which defer their attach to the iommu in the dma-iommu api > > > > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > > --- > > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > > 1 file changed, 26 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > > index 2712fbc68b28..906b7fa14d3c 100644 > > --- a/drivers/iommu/dma-iommu.c > > +++ b/drivers/iommu/dma-iommu.c > > @@ -22,6 +22,7 @@ > > #include <linux/pci.h> > > #include <linux/scatterlist.h> > > #include <linux/vmalloc.h> > > +#include <linux/crash_dump.h> > > > > struct iommu_dma_msi_page { > > struct list_head list; > > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > > return iova_reserve_iommu_regions(dev, domain); > > } > > > > +static int handle_deferred_device(struct device *dev, > > + struct iommu_domain *domain) > > +{ > > + const struct iommu_ops *ops = domain->ops; > > + > > + if (!is_kdump_kernel()) > > + return 0; > > + > > + if (unlikely(ops->is_attach_deferred && > > + ops->is_attach_deferred(domain, dev))) > > + return iommu_attach_device(domain, dev); > > + > > + return 0; > > +} > > + > > /** > > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > > * page flags. > > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > > size_t iova_off = iova_offset(iovad, phys); > > dma_addr_t iova; > > > > + if (unlikely(handle_deferred_device(dev, domain))) > > + return DMA_MAPPING_ERROR; > > + > > size = iova_align(iovad, size + iova_off); > > > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > > iommu_map_atomic() is applied to __iommu_dma_map() in 2/5. > Is it an atomic context currently given the mutex_lock() in > iommu_attach_device()? I don't see your point here. __iommu_dma_map isn't called from iommu_attach_device, why would we care about a mutex in iommu_attach_device? __iommu_dma_map can be called from an atomic context (it isn't always but it does happen). __iommu_dma_map is called by iommu_dma_alloc which implements the iommu_dma_ops::alloc function which by design needs to be callable from an atomic context. Does that answer your question? > _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices @ 2019-08-17 7:19 ` Tom Murphy 0 siblings, 0 replies; 87+ messages in thread From: Tom Murphy @ 2019-08-17 7:19 UTC (permalink / raw) To: Hillf Danton Cc: Joerg Roedel, iommu, Will Deacon, Robin Murphy, Marek Szyprowski, Kukjin Kim, Krzysztof Kozlowski, David Woodhouse, Andy Gross, Matthias Brugger, Rob Clark, Heiko Stuebner, Gerald Schaefer, Thierry Reding, Jonathan Hunter, Jean-Philippe Brucker, Linux Kernel Mailing List, linux-arm-kernel, linux-samsung-soc, linux-arm-msm, linux-mediatek, linux-rockchip, linux-s390, linux-tegra, virtualization On Sat, 17 Aug 2019 at 04:39, Hillf Danton <hdanton@sina.com> wrote: > > > On Thu, 15 Aug 2019 12:09:41 +0100 Tom Murphy wrote: > > > > Handle devices which defer their attach to the iommu in the dma-iommu api > > > > Signed-off-by: Tom Murphy <murphyt7@tcd.ie> > > --- > > drivers/iommu/dma-iommu.c | 27 ++++++++++++++++++++++++++- > > 1 file changed, 26 insertions(+), 1 deletion(-) > > > > diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c > > index 2712fbc68b28..906b7fa14d3c 100644 > > --- a/drivers/iommu/dma-iommu.c > > +++ b/drivers/iommu/dma-iommu.c > > @@ -22,6 +22,7 @@ > > #include <linux/pci.h> > > #include <linux/scatterlist.h> > > #include <linux/vmalloc.h> > > +#include <linux/crash_dump.h> > > > > struct iommu_dma_msi_page { > > struct list_head list; > > @@ -351,6 +352,21 @@ static int iommu_dma_init_domain(struct iommu_domain *domain, dma_addr_t base, > > return iova_reserve_iommu_regions(dev, domain); > > } > > > > +static int handle_deferred_device(struct device *dev, > > + struct iommu_domain *domain) > > +{ > > + const struct iommu_ops *ops = domain->ops; > > + > > + if (!is_kdump_kernel()) > > + return 0; > > + > > + if (unlikely(ops->is_attach_deferred && > > + ops->is_attach_deferred(domain, dev))) > > + return iommu_attach_device(domain, dev); > > + > > + return 0; > > +} > > + > > /** > > * dma_info_to_prot - Translate DMA API directions and attributes to IOMMU API > > * page flags. > > @@ -463,6 +479,9 @@ static dma_addr_t __iommu_dma_map(struct device *dev, phys_addr_t phys, > > size_t iova_off = iova_offset(iovad, phys); > > dma_addr_t iova; > > > > + if (unlikely(handle_deferred_device(dev, domain))) > > + return DMA_MAPPING_ERROR; > > + > > size = iova_align(iovad, size + iova_off); > > > > iova = iommu_dma_alloc_iova(domain, size, dma_get_mask(dev), dev); > > iommu_map_atomic() is applied to __iommu_dma_map() in 2/5. > Is it an atomic context currently given the mutex_lock() in > iommu_attach_device()? I don't see your point here. __iommu_dma_map isn't called from iommu_attach_device, why would we care about a mutex in iommu_attach_device? __iommu_dma_map can be called from an atomic context (it isn't always but it does happen). __iommu_dma_map is called by iommu_dma_alloc which implements the iommu_dma_ops::alloc function which by design needs to be callable from an atomic context. Does that answer your question? > ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 0/5] iommu/amd: Convert the AMD iommu driver to the dma-iommu api 2019-08-15 11:09 ` Tom Murphy ` (13 preceding siblings ...) (?) @ 2019-09-05 6:18 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-09-05 6:18 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy Dave, Joerg, Robin: is there any chance we could at least pick up patches 2 and 4 ASAP as they are clearly fixes for current deficits, even without the amd conversion? ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 0/5] iommu/amd: Convert the AMD iommu driver to the dma-iommu api 2019-08-15 11:09 ` Tom Murphy (?) (?) @ 2019-09-05 6:18 ` Christoph Hellwig -1 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-09-05 6:18 UTC (permalink / raw) To: Tom Murphy Cc: iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Kukjin Kim Dave, Joerg, Robin: is there any chance we could at least pick up patches 2 and 4 ASAP as they are clearly fixes for current deficits, even without the amd conversion? ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 0/5] iommu/amd: Convert the AMD iommu driver to the dma-iommu api @ 2019-09-05 6:18 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-09-05 6:18 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy Dave, Joerg, Robin: is there any chance we could at least pick up patches 2 and 4 ASAP as they are clearly fixes for current deficits, even without the amd conversion? _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 0/5] iommu/amd: Convert the AMD iommu driver to the dma-iommu api @ 2019-09-05 6:18 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-09-05 6:18 UTC (permalink / raw) To: Tom Murphy Cc: Heiko Stuebner, virtualization, Matthias Brugger, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Kukjin Kim, Andy Gross, linux-s390, Gerald Schaefer, linux-arm-msm, linux-mediatek, linux-tegra, linux-arm-kernel, David Woodhouse, linux-kernel, iommu, Robin Murphy Dave, Joerg, Robin: is there any chance we could at least pick up patches 2 and 4 ASAP as they are clearly fixes for current deficits, even without the amd conversion? _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu ^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH V5 0/5] iommu/amd: Convert the AMD iommu driver to the dma-iommu api @ 2019-09-05 6:18 ` Christoph Hellwig 0 siblings, 0 replies; 87+ messages in thread From: Christoph Hellwig @ 2019-09-05 6:18 UTC (permalink / raw) To: Tom Murphy Cc: iommu, Heiko Stuebner, virtualization, linux-tegra, Thierry Reding, Will Deacon, Jean-Philippe Brucker, linux-samsung-soc, Krzysztof Kozlowski, Jonathan Hunter, linux-rockchip, Andy Gross, Gerald Schaefer, linux-s390, linux-arm-msm, linux-mediatek, Matthias Brugger, linux-arm-kernel, David Woodhouse, linux-kernel, Kukjin Kim, Robin Murphy Dave, Joerg, Robin: is there any chance we could at least pick up patches 2 and 4 ASAP as they are clearly fixes for current deficits, even without the amd conversion? ^ permalink raw reply [flat|nested] 87+ messages in thread
end of thread, other threads:[~2019-09-05 6:18 UTC | newest]
Thread overview: 87+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-08-15 11:09 [PATCH V5 0/5] iommu/amd: Convert the AMD iommu driver to the dma-iommu api Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` [PATCH V5 1/5] iommu/amd: Remove unnecessary locking from AMD iommu driver Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-20 9:41 ` Christoph Hellwig
2019-08-20 9:41 ` Christoph Hellwig
2019-08-20 9:41 ` Christoph Hellwig
2019-08-20 9:41 ` Christoph Hellwig
2019-08-20 9:41 ` Christoph Hellwig
2019-08-24 7:56 ` Tom Murphy
2019-08-24 7:56 ` Tom Murphy
2019-08-24 7:56 ` Tom Murphy
2019-08-24 7:56 ` Tom Murphy
2019-08-24 7:56 ` Tom Murphy
2019-08-24 22:41 ` Christoph Hellwig
2019-08-24 22:41 ` Christoph Hellwig
2019-08-24 22:41 ` Christoph Hellwig
2019-08-24 22:41 ` Christoph Hellwig
2019-08-24 22:41 ` Christoph Hellwig
2019-08-15 11:09 ` [PATCH V5 2/5] iommu: Add gfp parameter to iommu_ops::map Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-19 18:23 ` Robin Murphy
2019-08-19 18:23 ` Robin Murphy
2019-08-19 18:23 ` Robin Murphy
2019-08-19 18:23 ` Robin Murphy
2019-08-19 18:23 ` Robin Murphy
2019-08-20 9:41 ` Christoph Hellwig
2019-08-20 9:41 ` Christoph Hellwig
2019-08-20 9:41 ` Christoph Hellwig
2019-08-20 9:41 ` Christoph Hellwig
2019-08-20 9:41 ` Christoph Hellwig
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-19 18:26 ` Robin Murphy
2019-08-19 18:26 ` Robin Murphy
2019-08-19 18:26 ` Robin Murphy
2019-08-19 18:26 ` Robin Murphy
2019-08-19 18:26 ` Robin Murphy
[not found] ` <20190815110944.3579-4-murphyt7-/Zw1syy4LHg@public.gmane.org>
2019-08-20 9:43 ` Christoph Hellwig
2019-08-20 9:43 ` Christoph Hellwig
2019-08-20 9:43 ` Christoph Hellwig
2019-08-20 9:43 ` Christoph Hellwig
2019-08-20 9:43 ` Christoph Hellwig
2019-08-15 11:09 ` [PATCH V5 4/5] iommu/dma-iommu: Use the dev->coherent_dma_mask Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-19 18:39 ` Robin Murphy
2019-08-19 18:39 ` Robin Murphy
2019-08-19 18:39 ` Robin Murphy
2019-08-19 18:39 ` Robin Murphy
2019-08-19 18:39 ` Robin Murphy
2019-08-20 9:43 ` Christoph Hellwig
2019-08-20 9:43 ` Christoph Hellwig
2019-08-20 9:43 ` Christoph Hellwig
2019-08-20 9:43 ` Christoph Hellwig
2019-08-20 9:43 ` Christoph Hellwig
2019-08-15 11:09 ` [PATCH V5 5/5] iommu/amd: Convert AMD iommu driver to the dma-iommu api Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-15 11:09 ` Tom Murphy
2019-08-17 3:39 ` [PATCH V5 3/5] iommu/dma-iommu: Handle deferred devices Hillf Danton
2019-08-17 3:39 ` Hillf Danton
2019-08-17 3:39 ` Hillf Danton
2019-08-17 7:19 ` Tom Murphy
[not found] ` <20190817033914.4812-1-hdanton-k+cT0dCbe1g@public.gmane.org>
2019-08-17 7:19 ` Tom Murphy
2019-08-17 7:19 ` Tom Murphy
2019-08-17 7:19 ` Tom Murphy
2019-08-17 7:19 ` Tom Murphy
2019-09-05 6:18 ` [PATCH V5 0/5] iommu/amd: Convert the AMD iommu driver to the dma-iommu api Christoph Hellwig
2019-09-05 6:18 ` Christoph Hellwig
2019-09-05 6:18 ` Christoph Hellwig
2019-09-05 6:18 ` Christoph Hellwig
2019-09-05 6:18 ` Christoph Hellwig
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.