[PATCH][v2] iommu/iova: Consider NUMA affinity when allocating per-CPU iova_magazine

[PATCH][v2] iommu/iova: Consider NUMA affinity when allocating per-CPU iova_magazine

[Li RongQing]

per-CPU iova_magazine are dominantly accessed from their own local CPUs,
so allocate them node-local for performance reason.

Signed-off-by: Li RongQing <lirongqing@baidu.com>
---
diff with v1: not consider num node in  __iova_rcache_insert 

 drivers/iommu/iova.c | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c
index d59d0ea..68697cc 100644
--- a/drivers/iommu/iova.c
+++ b/drivers/iommu/iova.c
@@ -597,17 +597,23 @@ unsigned long iova_rcache_range(void)
 	return PAGE_SIZE << (IOVA_RANGE_CACHE_MAX_SIZE - 1);
 }
 
-static struct iova_magazine *iova_magazine_alloc(gfp_t flags)
+static struct iova_magazine *iova_magazine_alloc_node(gfp_t flags, int node)
 {
 	struct iova_magazine *mag;
 
-	mag = kmem_cache_alloc(iova_magazine_cache, flags);
+	mag = kmem_cache_alloc_node(iova_magazine_cache, flags, node);
+
 	if (mag)
 		mag->size = 0;
 
 	return mag;
 }
 
+static inline struct iova_magazine *iova_magazine_alloc(gfp_t flags)
+{
+	return iova_magazine_alloc_node(flags, NUMA_NO_NODE);
+}
+
 static void iova_magazine_free(struct iova_magazine *mag)
 {
 	kmem_cache_free(iova_magazine_cache, mag);
@@ -707,7 +713,7 @@ static void iova_depot_work_func(struct work_struct *work)
 int iova_domain_init_rcaches(struct iova_domain *iovad)
 {
 	unsigned int cpu;
-	int i, ret;
+	int i, ret, nid;
 
 	iovad->rcaches = kcalloc(IOVA_RANGE_CACHE_MAX_SIZE,
 				 sizeof(struct iova_rcache),
@@ -731,10 +737,11 @@ int iova_domain_init_rcaches(struct iova_domain *iovad)
 		}
 		for_each_possible_cpu(cpu) {
 			cpu_rcache = per_cpu_ptr(rcache->cpu_rcaches, cpu);
+			nid = cpu_to_node(cpu);
 
 			spin_lock_init(&cpu_rcache->lock);
-			cpu_rcache->loaded = iova_magazine_alloc(GFP_KERNEL);
-			cpu_rcache->prev = iova_magazine_alloc(GFP_KERNEL);
+			cpu_rcache->loaded = iova_magazine_alloc_node(GFP_KERNEL, nid);
+			cpu_rcache->prev = iova_magazine_alloc_node(GFP_KERNEL, nid);
 			if (!cpu_rcache->loaded || !cpu_rcache->prev) {
 				ret = -ENOMEM;
 				goto out_err;
-- 
2.9.4

Re: [PATCH][v2] iommu/iova: Consider NUMA affinity when allocating per-CPU iova_magazine

[Will Deacon]

On Fri, Apr 26, 2024 at 01:16:19PM +0800, Li RongQing wrote:
> per-CPU iova_magazine are dominantly accessed from their own local CPUs,
> so allocate them node-local for performance reason.
> 
> Signed-off-by: Li RongQing <lirongqing@baidu.com>
> ---
> diff with v1: not consider num node in  __iova_rcache_insert 
> 
>  drivers/iommu/iova.c | 17 ++++++++++++-----
>  1 file changed, 12 insertions(+), 5 deletions(-)

Do you have any numbers showing that this is worthwhile?

Will