linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2
@ 2022-04-14  8:57 Hyeonggon Yoo
  2022-04-14  8:57 ` [PATCH v2 01/23] mm/slab: move NUMA-related code to __do_cache_alloc() Hyeonggon Yoo
                   ` (23 more replies)
  0 siblings, 24 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Changes since v1:
- The code is basically same, but split some patches to make it
easier to review.
- Adjusted comments and added Reviewed-by from Vlastimil. Thanks!

Hello, this series is cleanup of kmalloc code. After this series,
kmalloc subsystem is perfectly generalized on SLAB and SLUB.

This series is not small, but each patch is easy to review and
I bet you will like this :)

Many thanks to Matthew, Marco, Vlastimil who gave comments
in previous series.

Any feedbacks will be appreciated.
Thanks!

======== series description =======

patch 1-2 make slab_alloc_node() in SLAB available for non-NUMA
configurations for further cleanup.

patch 3-12 remove duplicate code in common kmalloc functions.

patch 13 makes SLAB pass requests larger than order-1 page to page
allocator. This is useful for further generalization.

patch 14-16 unify tracepoints. after them, we use only
kmem_cache_alloc and kmem_cache_free tracepoints in slab allocators.

patch 17-18 generalize whole kmalloc subsystem on SLAB/SLUB.

patch 19-20 factor out common code of __kmalloc_node()
and __kmalloc_node_track_caller().

patch 21 removes now unnecessary kmem_cache_alloc_node_trace().

patch 22-23 is small improvements of __ksize(). They are not part of
generalization but depends on this series.

Hyeonggon Yoo (23):
  mm/slab: move NUMA-related code to __do_cache_alloc()
  mm/slab: cleanup slab_alloc() and slab_alloc_node()
  mm/slab_common: remove CONFIG_NUMA ifdefs for common kmalloc functions
  mm/slab_common: cleanup kmalloc_track_caller()
  mm/slab_common: cleanup __kmalloc()
  mm/sl[auo]b: fold kmalloc_order_trace() into kmalloc_large()
  mm/slub: move kmalloc_large_node() to slab_common.c
  mm/slab_common: make kmalloc_large_node() consistent with
    kmalloc_large()
  mm/slab_common: cleanup kmalloc_large()
  mm/slab_common: cleanup kmem_cache_alloc{,node,lru}
  mm/slab_common: kmalloc_node: pass large requests to page allocator
  mm/slab_common: cleanup kmalloc()
  mm/slab: kmalloc: pass requests larger than order-1 page to page
    allocator
  mm/slab_common: print cache name in tracepoints
  mm/slab_common: use same tracepoint in kmalloc and normal caches
  mm/slab_common: rename tracepoint
  mm/slab_common: implement __kmem_cache_free()
  mm/sl[au]b: generalize kmalloc subsystem
  mm/slab_common: add kasan_kmalloc() in __kmalloc_node_track_caller()
  mm/slab_common: factor out __do_kmalloc_node()
  mm/sl[au]b: remove kmem_cache_alloc_node_trace()
  mm/sl[auo]b: move definition of __ksize() to mm/slab.h
  mm/sl[au]b: check if large object is valid in __ksize()

 include/linux/slab.h        | 245 ++++++++++++-------------
 include/trace/events/kmem.h | 109 ++---------
 mm/slab.c                   | 354 ++++++------------------------------
 mm/slab.h                   |   9 +
 mm/slab_common.c            | 146 ++++++++++++---
 mm/slob.c                   |  78 +++-----
 mm/slub.c                   | 264 ++-------------------------
 7 files changed, 350 insertions(+), 855 deletions(-)

-- 
2.32.0


^ permalink raw reply	[flat|nested] 51+ messages in thread

* [PATCH v2 01/23] mm/slab: move NUMA-related code to __do_cache_alloc()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-22 18:04   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 02/23] mm/slab: cleanup slab_alloc() and slab_alloc_node() Hyeonggon Yoo
                   ` (22 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

To implement slab_alloc_node() independent of NUMA configuration,
move NUMA fallback/alternate allocation code into __do_cache_alloc().

One functional change here is not to check availability of node
when allocating from local node.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
Changes from v1:
- Undo removing path to alternate_node_alloc code when node id is not specified (which
  was mistake.)

 mm/slab.c | 68 +++++++++++++++++++++++++------------------------------
 1 file changed, 31 insertions(+), 37 deletions(-)

diff --git a/mm/slab.c b/mm/slab.c
index e882657c1494..d854c24d5f5a 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3187,13 +3187,14 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags,
 	return obj ? obj : fallback_alloc(cachep, flags);
 }
 
+static void *__do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid);
+
 static __always_inline void *
 slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_size,
 		   unsigned long caller)
 {
 	unsigned long save_flags;
 	void *ptr;
-	int slab_node = numa_mem_id();
 	struct obj_cgroup *objcg = NULL;
 	bool init = false;
 
@@ -3208,30 +3209,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_
 
 	cache_alloc_debugcheck_before(cachep, flags);
 	local_irq_save(save_flags);
-
-	if (nodeid == NUMA_NO_NODE)
-		nodeid = slab_node;
-
-	if (unlikely(!get_node(cachep, nodeid))) {
-		/* Node not bootstrapped yet */
-		ptr = fallback_alloc(cachep, flags);
-		goto out;
-	}
-
-	if (nodeid == slab_node) {
-		/*
-		 * Use the locally cached objects if possible.
-		 * However ____cache_alloc does not allow fallback
-		 * to other nodes. It may fail while we still have
-		 * objects on other nodes available.
-		 */
-		ptr = ____cache_alloc(cachep, flags);
-		if (ptr)
-			goto out;
-	}
-	/* ___cache_alloc_node can fall back to other nodes */
-	ptr = ____cache_alloc_node(cachep, flags, nodeid);
-  out:
+	ptr = __do_cache_alloc(cachep, flags, nodeid);
 	local_irq_restore(save_flags);
 	ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller);
 	init = slab_want_init_on_alloc(flags, cachep);
@@ -3242,31 +3220,46 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_
 }
 
 static __always_inline void *
-__do_cache_alloc(struct kmem_cache *cache, gfp_t flags)
+__do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid)
 {
 	void *objp;
+	int slab_node = numa_mem_id();
 
-	if (current->mempolicy || cpuset_do_slab_mem_spread()) {
-		objp = alternate_node_alloc(cache, flags);
-		if (objp)
-			goto out;
+	if (nodeid == NUMA_NO_NODE) {
+		if (current->mempolicy || cpuset_do_slab_mem_spread()) {
+			objp = alternate_node_alloc(cachep, flags);
+			if (objp)
+				goto out;
+		}
+		/*
+		 * Use the locally cached objects if possible.
+		 * However ____cache_alloc does not allow fallback
+		 * to other nodes. It may fail while we still have
+		 * objects on other nodes available.
+		 */
+		objp = ____cache_alloc(cachep, flags);
+		nodeid = slab_node;
+	} else if (nodeid == slab_node) {
+		objp = ____cache_alloc(cachep, flags);
+	} else if (!get_node(cachep, nodeid)) {
+		/* Node not bootstrapped yet */
+		objp = fallback_alloc(cachep, flags);
+		goto out;
 	}
-	objp = ____cache_alloc(cache, flags);
 
 	/*
 	 * We may just have run out of memory on the local node.
 	 * ____cache_alloc_node() knows how to locate memory on other nodes
 	 */
 	if (!objp)
-		objp = ____cache_alloc_node(cache, flags, numa_mem_id());
-
-  out:
+		objp = ____cache_alloc_node(cachep, flags, nodeid);
+out:
 	return objp;
 }
 #else
 
 static __always_inline void *
-__do_cache_alloc(struct kmem_cache *cachep, gfp_t flags)
+__do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid __maybe_unused)
 {
 	return ____cache_alloc(cachep, flags);
 }
@@ -3293,7 +3286,7 @@ slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags,
 
 	cache_alloc_debugcheck_before(cachep, flags);
 	local_irq_save(save_flags);
-	objp = __do_cache_alloc(cachep, flags);
+	objp = __do_cache_alloc(cachep, flags, NUMA_NO_NODE);
 	local_irq_restore(save_flags);
 	objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller);
 	prefetchw(objp);
@@ -3532,7 +3525,8 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
 
 	local_irq_disable();
 	for (i = 0; i < size; i++) {
-		void *objp = kfence_alloc(s, s->object_size, flags) ?: __do_cache_alloc(s, flags);
+		void *objp = kfence_alloc(s, s->object_size, flags) ?:
+			     __do_cache_alloc(s, flags, NUMA_NO_NODE);
 
 		if (unlikely(!objp))
 			goto error;
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 02/23] mm/slab: cleanup slab_alloc() and slab_alloc_node()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
  2022-04-14  8:57 ` [PATCH v2 01/23] mm/slab: move NUMA-related code to __do_cache_alloc() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-25 14:05   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 03/23] mm/slab_common: remove CONFIG_NUMA ifdefs for common kmalloc functions Hyeonggon Yoo
                   ` (21 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Make slab_alloc_node() available even when CONFIG_NUMA=n and
make slab_alloc() wrapper of slab_alloc_node().

This is necessary for further cleanup.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/slab.c | 50 +++++++++++++-------------------------------------
 1 file changed, 13 insertions(+), 37 deletions(-)

diff --git a/mm/slab.c b/mm/slab.c
index d854c24d5f5a..f033d5b4fefb 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3187,38 +3187,6 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags,
 	return obj ? obj : fallback_alloc(cachep, flags);
 }
 
-static void *__do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid);
-
-static __always_inline void *
-slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_size,
-		   unsigned long caller)
-{
-	unsigned long save_flags;
-	void *ptr;
-	struct obj_cgroup *objcg = NULL;
-	bool init = false;
-
-	flags &= gfp_allowed_mask;
-	cachep = slab_pre_alloc_hook(cachep, NULL, &objcg, 1, flags);
-	if (unlikely(!cachep))
-		return NULL;
-
-	ptr = kfence_alloc(cachep, orig_size, flags);
-	if (unlikely(ptr))
-		goto out_hooks;
-
-	cache_alloc_debugcheck_before(cachep, flags);
-	local_irq_save(save_flags);
-	ptr = __do_cache_alloc(cachep, flags, nodeid);
-	local_irq_restore(save_flags);
-	ptr = cache_alloc_debugcheck_after(cachep, flags, ptr, caller);
-	init = slab_want_init_on_alloc(flags, cachep);
-
-out_hooks:
-	slab_post_alloc_hook(cachep, objcg, flags, 1, &ptr, init);
-	return ptr;
-}
-
 static __always_inline void *
 __do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid)
 {
@@ -3267,8 +3235,8 @@ __do_cache_alloc(struct kmem_cache *cachep, gfp_t flags, int nodeid __maybe_unus
 #endif /* CONFIG_NUMA */
 
 static __always_inline void *
-slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags,
-	   size_t orig_size, unsigned long caller)
+slab_alloc_node(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags,
+		int nodeid, size_t orig_size, unsigned long caller)
 {
 	unsigned long save_flags;
 	void *objp;
@@ -3286,7 +3254,7 @@ slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags,
 
 	cache_alloc_debugcheck_before(cachep, flags);
 	local_irq_save(save_flags);
-	objp = __do_cache_alloc(cachep, flags, NUMA_NO_NODE);
+	objp = __do_cache_alloc(cachep, flags, nodeid);
 	local_irq_restore(save_flags);
 	objp = cache_alloc_debugcheck_after(cachep, flags, objp, caller);
 	prefetchw(objp);
@@ -3297,6 +3265,14 @@ slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags,
 	return objp;
 }
 
+static __always_inline void *
+slab_alloc(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags,
+	   size_t orig_size, unsigned long caller)
+{
+	return slab_alloc_node(cachep, lru, flags, NUMA_NO_NODE, orig_size,
+			       caller);
+}
+
 /*
  * Caller needs to acquire correct kmem_cache_node's list_lock
  * @list: List of detached free slabs should be freed by caller
@@ -3585,7 +3561,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_trace);
  */
 void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid)
 {
-	void *ret = slab_alloc_node(cachep, flags, nodeid, cachep->object_size, _RET_IP_);
+	void *ret = slab_alloc_node(cachep, NULL, flags, nodeid, cachep->object_size, _RET_IP_);
 
 	trace_kmem_cache_alloc_node(_RET_IP_, ret,
 				    cachep->object_size, cachep->size,
@@ -3603,7 +3579,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
 {
 	void *ret;
 
-	ret = slab_alloc_node(cachep, flags, nodeid, size, _RET_IP_);
+	ret = slab_alloc_node(cachep, NULL, flags, nodeid, size, _RET_IP_);
 
 	ret = kasan_kmalloc(cachep, ret, size, flags);
 	trace_kmalloc_node(_RET_IP_, ret,
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 03/23] mm/slab_common: remove CONFIG_NUMA ifdefs for common kmalloc functions
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
  2022-04-14  8:57 ` [PATCH v2 01/23] mm/slab: move NUMA-related code to __do_cache_alloc() Hyeonggon Yoo
  2022-04-14  8:57 ` [PATCH v2 02/23] mm/slab: cleanup slab_alloc() and slab_alloc_node() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-25 14:41   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 04/23] mm/slab_common: cleanup kmalloc_track_caller() Hyeonggon Yoo
                   ` (20 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Now that slab_alloc_node() is available for SLAB when CONFIG_NUMA=n,
remove CONFIG_NUMA ifdefs for common kmalloc functions.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/linux/slab.h | 28 ----------------------------
 mm/slab.c            |  2 --
 mm/slob.c            |  5 +----
 mm/slub.c            |  6 ------
 4 files changed, 1 insertion(+), 40 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 11ceddcae9f4..a3b9d4c20d7e 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -444,38 +444,18 @@ static __always_inline void kfree_bulk(size_t size, void **p)
 	kmem_cache_free_bulk(NULL, size, p);
 }
 
-#ifdef CONFIG_NUMA
 void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment
 							 __alloc_size(1);
 void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment
 									 __malloc;
-#else
-static __always_inline __alloc_size(1) void *__kmalloc_node(size_t size, gfp_t flags, int node)
-{
-	return __kmalloc(size, flags);
-}
-
-static __always_inline void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node)
-{
-	return kmem_cache_alloc(s, flags);
-}
-#endif
 
 #ifdef CONFIG_TRACING
 extern void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t flags, size_t size)
 				   __assume_slab_alignment __alloc_size(3);
 
-#ifdef CONFIG_NUMA
 extern void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags,
 					 int node, size_t size) __assume_slab_alignment
 								__alloc_size(4);
-#else
-static __always_inline __alloc_size(4) void *kmem_cache_alloc_node_trace(struct kmem_cache *s,
-						 gfp_t gfpflags, int node, size_t size)
-{
-	return kmem_cache_alloc_trace(s, gfpflags, size);
-}
-#endif /* CONFIG_NUMA */
 
 #else /* CONFIG_TRACING */
 static __always_inline __alloc_size(3) void *kmem_cache_alloc_trace(struct kmem_cache *s,
@@ -689,20 +669,12 @@ static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t
 }
 
 
-#ifdef CONFIG_NUMA
 extern void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node,
 					 unsigned long caller) __alloc_size(1);
 #define kmalloc_node_track_caller(size, flags, node) \
 	__kmalloc_node_track_caller(size, flags, node, \
 			_RET_IP_)
 
-#else /* CONFIG_NUMA */
-
-#define kmalloc_node_track_caller(size, flags, node) \
-	kmalloc_track_caller(size, flags)
-
-#endif /* CONFIG_NUMA */
-
 /*
  * Shortcuts
  */
diff --git a/mm/slab.c b/mm/slab.c
index f033d5b4fefb..5ad55ca96ab6 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3545,7 +3545,6 @@ kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size)
 EXPORT_SYMBOL(kmem_cache_alloc_trace);
 #endif
 
-#ifdef CONFIG_NUMA
 /**
  * kmem_cache_alloc_node - Allocate an object on the specified node
  * @cachep: The cache to allocate from.
@@ -3619,7 +3618,6 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t flags,
 	return __do_kmalloc_node(size, flags, node, caller);
 }
 EXPORT_SYMBOL(__kmalloc_node_track_caller);
-#endif /* CONFIG_NUMA */
 
 #ifdef CONFIG_PRINTK
 void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
diff --git a/mm/slob.c b/mm/slob.c
index dfa6808dff36..c8c3b5662edf 100644
--- a/mm/slob.c
+++ b/mm/slob.c
@@ -534,14 +534,12 @@ void *__kmalloc_track_caller(size_t size, gfp_t gfp, unsigned long caller)
 }
 EXPORT_SYMBOL(__kmalloc_track_caller);
 
-#ifdef CONFIG_NUMA
 void *__kmalloc_node_track_caller(size_t size, gfp_t gfp,
 					int node, unsigned long caller)
 {
 	return __do_kmalloc_node(size, gfp, node, caller);
 }
 EXPORT_SYMBOL(__kmalloc_node_track_caller);
-#endif
 
 void kfree(const void *block)
 {
@@ -641,7 +639,7 @@ void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, gfp_
 	return slob_alloc_node(cachep, flags, NUMA_NO_NODE);
 }
 EXPORT_SYMBOL(kmem_cache_alloc_lru);
-#ifdef CONFIG_NUMA
+
 void *__kmalloc_node(size_t size, gfp_t gfp, int node)
 {
 	return __do_kmalloc_node(size, gfp, node, _RET_IP_);
@@ -653,7 +651,6 @@ void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t gfp, int node)
 	return slob_alloc_node(cachep, gfp, node);
 }
 EXPORT_SYMBOL(kmem_cache_alloc_node);
-#endif
 
 static void __kmem_cache_free(void *b, int size)
 {
diff --git a/mm/slub.c b/mm/slub.c
index d7e8355b2f08..e36c148e5069 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3260,7 +3260,6 @@ void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size)
 EXPORT_SYMBOL(kmem_cache_alloc_trace);
 #endif
 
-#ifdef CONFIG_NUMA
 void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node)
 {
 	void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size);
@@ -3287,7 +3286,6 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *s,
 }
 EXPORT_SYMBOL(kmem_cache_alloc_node_trace);
 #endif
-#endif	/* CONFIG_NUMA */
 
 /*
  * Slow path handling. This may still be called frequently since objects
@@ -4424,7 +4422,6 @@ void *__kmalloc(size_t size, gfp_t flags)
 }
 EXPORT_SYMBOL(__kmalloc);
 
-#ifdef CONFIG_NUMA
 static void *kmalloc_large_node(size_t size, gfp_t flags, int node)
 {
 	struct page *page;
@@ -4471,7 +4468,6 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
 	return ret;
 }
 EXPORT_SYMBOL(__kmalloc_node);
-#endif	/* CONFIG_NUMA */
 
 #ifdef CONFIG_HARDENED_USERCOPY
 /*
@@ -4929,7 +4925,6 @@ void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller)
 }
 EXPORT_SYMBOL(__kmalloc_track_caller);
 
-#ifdef CONFIG_NUMA
 void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 					int node, unsigned long caller)
 {
@@ -4959,7 +4954,6 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 	return ret;
 }
 EXPORT_SYMBOL(__kmalloc_node_track_caller);
-#endif
 
 #ifdef CONFIG_SYSFS
 static int count_inuse(struct slab *slab)
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 04/23] mm/slab_common: cleanup kmalloc_track_caller()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (2 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 03/23] mm/slab_common: remove CONFIG_NUMA ifdefs for common kmalloc functions Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-25 15:05   ` Vlastimil Babka
  2022-04-26 15:49   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 05/23] mm/slab_common: cleanup __kmalloc() Hyeonggon Yoo
                   ` (19 subsequent siblings)
  23 siblings, 2 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Make kmalloc_track_caller() wrapper of kmalloc_node_track_caller().

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/linux/slab.h | 17 ++++++++---------
 mm/slab.c            |  6 ------
 mm/slob.c            |  6 ------
 mm/slub.c            | 22 ----------------------
 4 files changed, 8 insertions(+), 43 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index a3b9d4c20d7e..acdb4b7428f9 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -639,6 +639,12 @@ static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flag
 	return kmalloc_array(n, size, flags | __GFP_ZERO);
 }
 
+extern void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node,
+					 unsigned long caller) __alloc_size(1);
+#define kmalloc_node_track_caller(size, flags, node) \
+	__kmalloc_node_track_caller(size, flags, node, \
+				    _RET_IP_)
+
 /*
  * kmalloc_track_caller is a special version of kmalloc that records the
  * calling function of the routine calling it for slab leak tracking instead
@@ -647,9 +653,9 @@ static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flag
  * allocator where we care about the real place the memory allocation
  * request comes from.
  */
-extern void *__kmalloc_track_caller(size_t size, gfp_t flags, unsigned long caller);
 #define kmalloc_track_caller(size, flags) \
-	__kmalloc_track_caller(size, flags, _RET_IP_)
+	__kmalloc_node_track_caller(size, flags, \
+				    NUMA_NO_NODE, _RET_IP_)
 
 static inline __alloc_size(1, 2) void *kmalloc_array_node(size_t n, size_t size, gfp_t flags,
 							  int node)
@@ -668,13 +674,6 @@ static inline __alloc_size(1, 2) void *kcalloc_node(size_t n, size_t size, gfp_t
 	return kmalloc_array_node(n, size, flags | __GFP_ZERO, node);
 }
 
-
-extern void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node,
-					 unsigned long caller) __alloc_size(1);
-#define kmalloc_node_track_caller(size, flags, node) \
-	__kmalloc_node_track_caller(size, flags, node, \
-			_RET_IP_)
-
 /*
  * Shortcuts
  */
diff --git a/mm/slab.c b/mm/slab.c
index 5ad55ca96ab6..5f20efc7a330 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3675,12 +3675,6 @@ void *__kmalloc(size_t size, gfp_t flags)
 }
 EXPORT_SYMBOL(__kmalloc);
 
-void *__kmalloc_track_caller(size_t size, gfp_t flags, unsigned long caller)
-{
-	return __do_kmalloc(size, flags, caller);
-}
-EXPORT_SYMBOL(__kmalloc_track_caller);
-
 /**
  * kmem_cache_free - Deallocate an object
  * @cachep: The cache the allocation was from.
diff --git a/mm/slob.c b/mm/slob.c
index c8c3b5662edf..6d0fc6ad1413 100644
--- a/mm/slob.c
+++ b/mm/slob.c
@@ -528,12 +528,6 @@ void *__kmalloc(size_t size, gfp_t gfp)
 }
 EXPORT_SYMBOL(__kmalloc);
 
-void *__kmalloc_track_caller(size_t size, gfp_t gfp, unsigned long caller)
-{
-	return __do_kmalloc_node(size, gfp, NUMA_NO_NODE, caller);
-}
-EXPORT_SYMBOL(__kmalloc_track_caller);
-
 void *__kmalloc_node_track_caller(size_t size, gfp_t gfp,
 					int node, unsigned long caller)
 {
diff --git a/mm/slub.c b/mm/slub.c
index e36c148e5069..e425c5c372de 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4903,28 +4903,6 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags)
 	return 0;
 }
 
-void *__kmalloc_track_caller(size_t size, gfp_t gfpflags, unsigned long caller)
-{
-	struct kmem_cache *s;
-	void *ret;
-
-	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
-		return kmalloc_large(size, gfpflags);
-
-	s = kmalloc_slab(size, gfpflags);
-
-	if (unlikely(ZERO_OR_NULL_PTR(s)))
-		return s;
-
-	ret = slab_alloc(s, NULL, gfpflags, caller, size);
-
-	/* Honor the call site pointer we received. */
-	trace_kmalloc(caller, ret, size, s->size, gfpflags);
-
-	return ret;
-}
-EXPORT_SYMBOL(__kmalloc_track_caller);
-
 void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 					int node, unsigned long caller)
 {
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 05/23] mm/slab_common: cleanup __kmalloc()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (3 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 04/23] mm/slab_common: cleanup kmalloc_track_caller() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-26 16:02   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 06/23] mm/sl[auo]b: fold kmalloc_order_trace() into kmalloc_large() Hyeonggon Yoo
                   ` (18 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Make __kmalloc() wrapper of __kmalloc_node().

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/linux/slab.h | 13 ++++++++++---
 mm/slab.c            | 34 ----------------------------------
 mm/slob.c            |  6 ------
 mm/slub.c            | 23 -----------------------
 4 files changed, 10 insertions(+), 66 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index acdb4b7428f9..4c06d15f731c 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -419,7 +419,16 @@ static __always_inline unsigned int __kmalloc_index(size_t size,
 #define kmalloc_index(s) __kmalloc_index(s, true)
 #endif /* !CONFIG_SLOB */
 
-void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1);
+extern void *__kmalloc_node(size_t size, gfp_t flags, int node)
+			    __assume_kmalloc_alignment
+			    __alloc_size(1);
+
+static __always_inline __alloc_size(1) __assume_kmalloc_alignment
+void *__kmalloc(size_t size, gfp_t flags)
+{
+	return __kmalloc_node(size, flags, NUMA_NO_NODE);
+}
+
 void *kmem_cache_alloc(struct kmem_cache *s, gfp_t flags) __assume_slab_alignment __malloc;
 void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
 			   gfp_t gfpflags) __assume_slab_alignment __malloc;
@@ -444,8 +453,6 @@ static __always_inline void kfree_bulk(size_t size, void **p)
 	kmem_cache_free_bulk(NULL, size, p);
 }
 
-void *__kmalloc_node(size_t size, gfp_t flags, int node) __assume_kmalloc_alignment
-							 __alloc_size(1);
 void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment
 									 __malloc;
 
diff --git a/mm/slab.c b/mm/slab.c
index 5f20efc7a330..db7eab9e2e9f 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3641,40 +3641,6 @@ void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
 }
 #endif
 
-/**
- * __do_kmalloc - allocate memory
- * @size: how many bytes of memory are required.
- * @flags: the type of memory to allocate (see kmalloc).
- * @caller: function caller for debug tracking of the caller
- *
- * Return: pointer to the allocated memory or %NULL in case of error
- */
-static __always_inline void *__do_kmalloc(size_t size, gfp_t flags,
-					  unsigned long caller)
-{
-	struct kmem_cache *cachep;
-	void *ret;
-
-	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
-		return NULL;
-	cachep = kmalloc_slab(size, flags);
-	if (unlikely(ZERO_OR_NULL_PTR(cachep)))
-		return cachep;
-	ret = slab_alloc(cachep, NULL, flags, size, caller);
-
-	ret = kasan_kmalloc(cachep, ret, size, flags);
-	trace_kmalloc(caller, ret,
-		      size, cachep->size, flags);
-
-	return ret;
-}
-
-void *__kmalloc(size_t size, gfp_t flags)
-{
-	return __do_kmalloc(size, flags, _RET_IP_);
-}
-EXPORT_SYMBOL(__kmalloc);
-
 /**
  * kmem_cache_free - Deallocate an object
  * @cachep: The cache the allocation was from.
diff --git a/mm/slob.c b/mm/slob.c
index 6d0fc6ad1413..ab67c8219e8d 100644
--- a/mm/slob.c
+++ b/mm/slob.c
@@ -522,12 +522,6 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller)
 	return ret;
 }
 
-void *__kmalloc(size_t size, gfp_t gfp)
-{
-	return __do_kmalloc_node(size, gfp, NUMA_NO_NODE, _RET_IP_);
-}
-EXPORT_SYMBOL(__kmalloc);
-
 void *__kmalloc_node_track_caller(size_t size, gfp_t gfp,
 					int node, unsigned long caller)
 {
diff --git a/mm/slub.c b/mm/slub.c
index e425c5c372de..44170b4f084b 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4399,29 +4399,6 @@ static int __init setup_slub_min_objects(char *str)
 
 __setup("slub_min_objects=", setup_slub_min_objects);
 
-void *__kmalloc(size_t size, gfp_t flags)
-{
-	struct kmem_cache *s;
-	void *ret;
-
-	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
-		return kmalloc_large(size, flags);
-
-	s = kmalloc_slab(size, flags);
-
-	if (unlikely(ZERO_OR_NULL_PTR(s)))
-		return s;
-
-	ret = slab_alloc(s, NULL, flags, _RET_IP_, size);
-
-	trace_kmalloc(_RET_IP_, ret, size, s->size, flags);
-
-	ret = kasan_kmalloc(s, ret, size, flags);
-
-	return ret;
-}
-EXPORT_SYMBOL(__kmalloc);
-
 static void *kmalloc_large_node(size_t size, gfp_t flags, int node)
 {
 	struct page *page;
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 06/23] mm/sl[auo]b: fold kmalloc_order_trace() into kmalloc_large()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (4 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 05/23] mm/slab_common: cleanup __kmalloc() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-26 16:09   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 07/23] mm/slub: move kmalloc_large_node() to slab_common.c Hyeonggon Yoo
                   ` (17 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

There is no caller of kmalloc_order_trace() except kmalloc_large().
Fold it into kmalloc_large() and remove kmalloc_order{,_trace}().

Also add tracepoint in kmalloc_large() that was previously
in kmalloc_order_trace().

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
---
Changes from v1:
- updated some changelog (kmalloc_order() -> kmalloc_order_trace())

 include/linux/slab.h | 22 ++--------------------
 mm/slab_common.c     | 14 +++-----------
 2 files changed, 5 insertions(+), 31 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 4c06d15f731c..6f6e22959b39 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -484,26 +484,8 @@ static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, g
 }
 #endif /* CONFIG_TRACING */
 
-extern void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) __assume_page_alignment
-									 __alloc_size(1);
-
-#ifdef CONFIG_TRACING
-extern void *kmalloc_order_trace(size_t size, gfp_t flags, unsigned int order)
-				__assume_page_alignment __alloc_size(1);
-#else
-static __always_inline __alloc_size(1) void *kmalloc_order_trace(size_t size, gfp_t flags,
-								 unsigned int order)
-{
-	return kmalloc_order(size, flags, order);
-}
-#endif
-
-static __always_inline __alloc_size(1) void *kmalloc_large(size_t size, gfp_t flags)
-{
-	unsigned int order = get_order(size);
-	return kmalloc_order_trace(size, flags, order);
-}
-
+extern void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment
+						     __alloc_size(1);
 /**
  * kmalloc - allocate memory
  * @size: how many bytes of memory are required.
diff --git a/mm/slab_common.c b/mm/slab_common.c
index c4d63f2c78b8..308cd5449285 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -925,10 +925,11 @@ gfp_t kmalloc_fix_flags(gfp_t flags)
  * directly to the page allocator. We use __GFP_COMP, because we will need to
  * know the allocation order to free the pages properly in kfree.
  */
-void *kmalloc_order(size_t size, gfp_t flags, unsigned int order)
+void *kmalloc_large(size_t size, gfp_t flags)
 {
 	void *ret = NULL;
 	struct page *page;
+	unsigned int order = get_order(size);
 
 	if (unlikely(flags & GFP_SLAB_BUG_MASK))
 		flags = kmalloc_fix_flags(flags);
@@ -943,19 +944,10 @@ void *kmalloc_order(size_t size, gfp_t flags, unsigned int order)
 	ret = kasan_kmalloc_large(ret, size, flags);
 	/* As ret might get tagged, call kmemleak hook after KASAN. */
 	kmemleak_alloc(ret, size, 1, flags);
-	return ret;
-}
-EXPORT_SYMBOL(kmalloc_order);
-
-#ifdef CONFIG_TRACING
-void *kmalloc_order_trace(size_t size, gfp_t flags, unsigned int order)
-{
-	void *ret = kmalloc_order(size, flags, order);
 	trace_kmalloc(_RET_IP_, ret, size, PAGE_SIZE << order, flags);
 	return ret;
 }
-EXPORT_SYMBOL(kmalloc_order_trace);
-#endif
+EXPORT_SYMBOL(kmalloc_large);
 
 #ifdef CONFIG_SLAB_FREELIST_RANDOM
 /* Randomize a generic freelist */
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 07/23] mm/slub: move kmalloc_large_node() to slab_common.c
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (5 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 06/23] mm/sl[auo]b: fold kmalloc_order_trace() into kmalloc_large() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-26 16:13   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 08/23] mm/slab_common: make kmalloc_large_node() consistent with kmalloc_large() Hyeonggon Yoo
                   ` (16 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

In later patch SLAB will also pass requests larger than order-1 page
to page allocator. Move kmalloc_large_node() to slab_common.c.

Fold kmalloc_large_node_hook() into kmalloc_large_node() as there is
no other caller.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/linux/slab.h |  3 +++
 mm/slab_common.c     | 22 ++++++++++++++++++++++
 mm/slub.c            | 25 -------------------------
 3 files changed, 25 insertions(+), 25 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 6f6e22959b39..97336acbebbf 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -486,6 +486,9 @@ static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, g
 
 extern void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment
 						     __alloc_size(1);
+
+extern void *kmalloc_large_node(size_t size, gfp_t flags, int node)
+				__assume_page_alignment __alloc_size(1);
 /**
  * kmalloc - allocate memory
  * @size: how many bytes of memory are required.
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 308cd5449285..e72089515030 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -949,6 +949,28 @@ void *kmalloc_large(size_t size, gfp_t flags)
 }
 EXPORT_SYMBOL(kmalloc_large);
 
+void *kmalloc_large_node(size_t size, gfp_t flags, int node)
+{
+	struct page *page;
+	void *ptr = NULL;
+	unsigned int order = get_order(size);
+
+	flags |= __GFP_COMP;
+	page = alloc_pages_node(node, flags, order);
+	if (page) {
+		ptr = page_address(page);
+		mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B,
+				      PAGE_SIZE << order);
+	}
+
+	ptr = kasan_kmalloc_large(ptr, size, flags);
+	/* As ptr might get tagged, call kmemleak hook after KASAN. */
+	kmemleak_alloc(ptr, size, 1, flags);
+
+	return ptr;
+}
+EXPORT_SYMBOL(kmalloc_large_node);
+
 #ifdef CONFIG_SLAB_FREELIST_RANDOM
 /* Randomize a generic freelist */
 static void freelist_randomize(struct rnd_state *state, unsigned int *list,
diff --git a/mm/slub.c b/mm/slub.c
index 44170b4f084b..640712706f2b 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -1679,14 +1679,6 @@ static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab,
  * Hooks for other subsystems that check memory allocations. In a typical
  * production configuration these hooks all should produce no code at all.
  */
-static inline void *kmalloc_large_node_hook(void *ptr, size_t size, gfp_t flags)
-{
-	ptr = kasan_kmalloc_large(ptr, size, flags);
-	/* As ptr might get tagged, call kmemleak hook after KASAN. */
-	kmemleak_alloc(ptr, size, 1, flags);
-	return ptr;
-}
-
 static __always_inline void kfree_hook(void *x)
 {
 	kmemleak_free(x);
@@ -4399,23 +4391,6 @@ static int __init setup_slub_min_objects(char *str)
 
 __setup("slub_min_objects=", setup_slub_min_objects);
 
-static void *kmalloc_large_node(size_t size, gfp_t flags, int node)
-{
-	struct page *page;
-	void *ptr = NULL;
-	unsigned int order = get_order(size);
-
-	flags |= __GFP_COMP;
-	page = alloc_pages_node(node, flags, order);
-	if (page) {
-		ptr = page_address(page);
-		mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B,
-				      PAGE_SIZE << order);
-	}
-
-	return kmalloc_large_node_hook(ptr, size, flags);
-}
-
 void *__kmalloc_node(size_t size, gfp_t flags, int node)
 {
 	struct kmem_cache *s;
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 08/23] mm/slab_common: make kmalloc_large_node() consistent with kmalloc_large()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (6 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 07/23] mm/slub: move kmalloc_large_node() to slab_common.c Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-26 17:15   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 09/23] mm/slab_common: cleanup kmalloc_large() Hyeonggon Yoo
                   ` (15 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Move tracepoints into kmalloc_large_node() and add missing flag fix code.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/slab_common.c |  6 ++++++
 mm/slub.c        | 22 ++++------------------
 2 files changed, 10 insertions(+), 18 deletions(-)

diff --git a/mm/slab_common.c b/mm/slab_common.c
index e72089515030..cf17be8cd9ad 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -955,6 +955,9 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node)
 	void *ptr = NULL;
 	unsigned int order = get_order(size);
 
+	if (unlikely(flags & GFP_SLAB_BUG_MASK))
+		flags = kmalloc_fix_flags(flags);
+
 	flags |= __GFP_COMP;
 	page = alloc_pages_node(node, flags, order);
 	if (page) {
@@ -966,6 +969,9 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node)
 	ptr = kasan_kmalloc_large(ptr, size, flags);
 	/* As ptr might get tagged, call kmemleak hook after KASAN. */
 	kmemleak_alloc(ptr, size, 1, flags);
+	trace_kmalloc_node(_RET_IP_, ptr,
+			   size, PAGE_SIZE << order,
+			   flags, node);
 
 	return ptr;
 }
diff --git a/mm/slub.c b/mm/slub.c
index 640712706f2b..f10a892f1772 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4396,15 +4396,8 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
 	struct kmem_cache *s;
 	void *ret;
 
-	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
-		ret = kmalloc_large_node(size, flags, node);
-
-		trace_kmalloc_node(_RET_IP_, ret,
-				   size, PAGE_SIZE << get_order(size),
-				   flags, node);
-
-		return ret;
-	}
+	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
+		return kmalloc_large_node(size, flags, node);
 
 	s = kmalloc_slab(size, flags);
 
@@ -4861,15 +4854,8 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 	struct kmem_cache *s;
 	void *ret;
 
-	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
-		ret = kmalloc_large_node(size, gfpflags, node);
-
-		trace_kmalloc_node(caller, ret,
-				   size, PAGE_SIZE << get_order(size),
-				   gfpflags, node);
-
-		return ret;
-	}
+	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
+		return kmalloc_large_node(size, gfpflags, node);
 
 	s = kmalloc_slab(size, gfpflags);
 
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 09/23] mm/slab_common: cleanup kmalloc_large()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (7 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 08/23] mm/slab_common: make kmalloc_large_node() consistent with kmalloc_large() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-26 17:18   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 10/23] mm/slab_common: cleanup kmem_cache_alloc{,node,lru} Hyeonggon Yoo
                   ` (14 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Now that kmalloc_large() and kmalloc_large_node() do same job,
make kmalloc_large() wrapper of kmalloc_large_node().

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/linux/slab.h |  9 ++++++---
 mm/slab_common.c     | 24 ------------------------
 2 files changed, 6 insertions(+), 27 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 97336acbebbf..143830f57a7f 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -484,11 +484,14 @@ static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, g
 }
 #endif /* CONFIG_TRACING */
 
-extern void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment
-						     __alloc_size(1);
-
 extern void *kmalloc_large_node(size_t size, gfp_t flags, int node)
 				__assume_page_alignment __alloc_size(1);
+
+static __always_inline void *kmalloc_large(size_t size, gfp_t flags)
+{
+	return kmalloc_large_node(size, flags, NUMA_NO_NODE);
+}
+
 /**
  * kmalloc - allocate memory
  * @size: how many bytes of memory are required.
diff --git a/mm/slab_common.c b/mm/slab_common.c
index cf17be8cd9ad..30684efc89d7 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -925,30 +925,6 @@ gfp_t kmalloc_fix_flags(gfp_t flags)
  * directly to the page allocator. We use __GFP_COMP, because we will need to
  * know the allocation order to free the pages properly in kfree.
  */
-void *kmalloc_large(size_t size, gfp_t flags)
-{
-	void *ret = NULL;
-	struct page *page;
-	unsigned int order = get_order(size);
-
-	if (unlikely(flags & GFP_SLAB_BUG_MASK))
-		flags = kmalloc_fix_flags(flags);
-
-	flags |= __GFP_COMP;
-	page = alloc_pages(flags, order);
-	if (likely(page)) {
-		ret = page_address(page);
-		mod_lruvec_page_state(page, NR_SLAB_UNRECLAIMABLE_B,
-				      PAGE_SIZE << order);
-	}
-	ret = kasan_kmalloc_large(ret, size, flags);
-	/* As ret might get tagged, call kmemleak hook after KASAN. */
-	kmemleak_alloc(ret, size, 1, flags);
-	trace_kmalloc(_RET_IP_, ret, size, PAGE_SIZE << order, flags);
-	return ret;
-}
-EXPORT_SYMBOL(kmalloc_large);
-
 void *kmalloc_large_node(size_t size, gfp_t flags, int node)
 {
 	struct page *page;
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 10/23] mm/slab_common: cleanup kmem_cache_alloc{,node,lru}
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (8 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 09/23] mm/slab_common: cleanup kmalloc_large() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-26 18:01   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 11/23] mm/slab_common: kmalloc_node: pass large requests to page allocator Hyeonggon Yoo
                   ` (13 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Implement only __kmem_cache_alloc_node() in slab allocators and make
kmem_cache_alloc{,node,lru} wrapper of it.

Now that kmem_cache_alloc{,node,lru} is inline function, we should
use _THIS_IP_ instead of _RET_IP_ for consistency.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/linux/slab.h | 52 ++++++++++++++++++++++++++++++++-----
 mm/slab.c            | 61 +++++---------------------------------------
 mm/slob.c            | 27 ++++++--------------
 mm/slub.c            | 35 +++++--------------------
 4 files changed, 67 insertions(+), 108 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 143830f57a7f..1b5bdcb0fd31 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -429,9 +429,52 @@ void *__kmalloc(size_t size, gfp_t flags)
 	return __kmalloc_node(size, flags, NUMA_NO_NODE);
 }
 
-void *kmem_cache_alloc(struct kmem_cache *s, gfp_t flags) __assume_slab_alignment __malloc;
-void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
-			   gfp_t gfpflags) __assume_slab_alignment __malloc;
+
+void *__kmem_cache_alloc_node(struct kmem_cache *s, struct list_lru *lru,
+			   gfp_t gfpflags, int node, unsigned long caller __maybe_unused)
+			    __assume_slab_alignment __malloc;
+
+/**
+ * kmem_cache_alloc - Allocate an object
+ * @cachep: The cache to allocate from.
+ * @flags: See kmalloc().
+ *
+ * Allocate an object from this cache.  The flags are only relevant
+ * if the cache has no available objects.
+ *
+ * Return: pointer to the new object or %NULL in case of error
+ */
+static __always_inline __malloc
+void *kmem_cache_alloc(struct kmem_cache *s, gfp_t flags)
+{
+	return __kmem_cache_alloc_node(s, NULL, flags, NUMA_NO_NODE, _THIS_IP_);
+}
+
+/**
+ * kmem_cache_alloc_node - Allocate an object on the specified node
+ * @s: The cache to allocate from.
+ * @flags: See kmalloc().
+ * @node: node number of the target node.
+ *
+ * Identical to kmem_cache_alloc but it will allocate memory on the given
+ * node, which can improve the performance for cpu bound structures.
+ *
+ * Fallback to other node is possible if __GFP_THISNODE is not set.
+ *
+ * Return: pointer to the new object or %NULL in case of error
+ */
+static __always_inline __malloc
+void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node)
+{
+	return __kmem_cache_alloc_node(s, NULL, flags, node, _THIS_IP_);
+}
+
+static __always_inline __malloc
+void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags)
+{
+	return __kmem_cache_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, _THIS_IP_);
+}
+
 void kmem_cache_free(struct kmem_cache *s, void *objp);
 
 /*
@@ -453,9 +496,6 @@ static __always_inline void kfree_bulk(size_t size, void **p)
 	kmem_cache_free_bulk(NULL, size, p);
 }
 
-void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t flags, int node) __assume_slab_alignment
-									 __malloc;
-
 #ifdef CONFIG_TRACING
 extern void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t flags, size_t size)
 				   __assume_slab_alignment __alloc_size(3);
diff --git a/mm/slab.c b/mm/slab.c
index db7eab9e2e9f..c5ffe54c207a 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3442,40 +3442,18 @@ void ___cache_free(struct kmem_cache *cachep, void *objp,
 	__free_one(ac, objp);
 }
 
-static __always_inline
-void *__kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru,
-			     gfp_t flags)
+void *__kmem_cache_alloc_node(struct kmem_cache *cachep, struct list_lru *lru,
+			      gfp_t flags, int nodeid, unsigned long caller)
 {
-	void *ret = slab_alloc(cachep, lru, flags, cachep->object_size, _RET_IP_);
+	void *ret = slab_alloc_node(cachep, lru, flags, nodeid,
+				    cachep->object_size, caller);
 
-	trace_kmem_cache_alloc(_RET_IP_, ret,
-			       cachep->object_size, cachep->size, flags);
+	trace_kmem_cache_alloc_node(caller, ret, cachep->object_size,
+				    cachep->size, flags, nodeid);
 
 	return ret;
 }
-
-/**
- * kmem_cache_alloc - Allocate an object
- * @cachep: The cache to allocate from.
- * @flags: See kmalloc().
- *
- * Allocate an object from this cache.  The flags are only relevant
- * if the cache has no available objects.
- *
- * Return: pointer to the new object or %NULL in case of error
- */
-void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags)
-{
-	return __kmem_cache_alloc_lru(cachep, NULL, flags);
-}
-EXPORT_SYMBOL(kmem_cache_alloc);
-
-void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru,
-			   gfp_t flags)
-{
-	return __kmem_cache_alloc_lru(cachep, lru, flags);
-}
-EXPORT_SYMBOL(kmem_cache_alloc_lru);
+EXPORT_SYMBOL(__kmem_cache_alloc_node);
 
 static __always_inline void
 cache_alloc_debugcheck_after_bulk(struct kmem_cache *s, gfp_t flags,
@@ -3545,31 +3523,6 @@ kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size)
 EXPORT_SYMBOL(kmem_cache_alloc_trace);
 #endif
 
-/**
- * kmem_cache_alloc_node - Allocate an object on the specified node
- * @cachep: The cache to allocate from.
- * @flags: See kmalloc().
- * @nodeid: node number of the target node.
- *
- * Identical to kmem_cache_alloc but it will allocate memory on the given
- * node, which can improve the performance for cpu bound structures.
- *
- * Fallback to other node is possible if __GFP_THISNODE is not set.
- *
- * Return: pointer to the new object or %NULL in case of error
- */
-void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid)
-{
-	void *ret = slab_alloc_node(cachep, NULL, flags, nodeid, cachep->object_size, _RET_IP_);
-
-	trace_kmem_cache_alloc_node(_RET_IP_, ret,
-				    cachep->object_size, cachep->size,
-				    flags, nodeid);
-
-	return ret;
-}
-EXPORT_SYMBOL(kmem_cache_alloc_node);
-
 #ifdef CONFIG_TRACING
 void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
 				  gfp_t flags,
diff --git a/mm/slob.c b/mm/slob.c
index ab67c8219e8d..6c7c30845056 100644
--- a/mm/slob.c
+++ b/mm/slob.c
@@ -586,7 +586,8 @@ int __kmem_cache_create(struct kmem_cache *c, slab_flags_t flags)
 	return 0;
 }
 
-static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node)
+static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node,
+			     unsigned long caller)
 {
 	void *b;
 
@@ -596,12 +597,12 @@ static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node)
 
 	if (c->size < PAGE_SIZE) {
 		b = slob_alloc(c->size, flags, c->align, node, 0);
-		trace_kmem_cache_alloc_node(_RET_IP_, b, c->object_size,
+		trace_kmem_cache_alloc_node(caller, b, c->object_size,
 					    SLOB_UNITS(c->size) * SLOB_UNIT,
 					    flags, node);
 	} else {
 		b = slob_new_pages(flags, get_order(c->size), node);
-		trace_kmem_cache_alloc_node(_RET_IP_, b, c->object_size,
+		trace_kmem_cache_alloc_node(caller, b, c->object_size,
 					    PAGE_SIZE << get_order(c->size),
 					    flags, node);
 	}
@@ -615,30 +616,18 @@ static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node)
 	return b;
 }
 
-void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags)
-{
-	return slob_alloc_node(cachep, flags, NUMA_NO_NODE);
-}
-EXPORT_SYMBOL(kmem_cache_alloc);
-
-
-void *kmem_cache_alloc_lru(struct kmem_cache *cachep, struct list_lru *lru, gfp_t flags)
-{
-	return slob_alloc_node(cachep, flags, NUMA_NO_NODE);
-}
-EXPORT_SYMBOL(kmem_cache_alloc_lru);
-
 void *__kmalloc_node(size_t size, gfp_t gfp, int node)
 {
 	return __do_kmalloc_node(size, gfp, node, _RET_IP_);
 }
 EXPORT_SYMBOL(__kmalloc_node);
 
-void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t gfp, int node)
+void *__kmem_cache_alloc_node(struct kmem_cache *cachep, struct list_lru *lru __maybe_unused,
+			      gfp_t gfp, int node, unsigned long caller __maybe_unused)
 {
-	return slob_alloc_node(cachep, gfp, node);
+	return slob_alloc_node(cachep, gfp, node, caller);
 }
-EXPORT_SYMBOL(kmem_cache_alloc_node);
+EXPORT_SYMBOL(__kmem_cache_alloc_node);
 
 static void __kmem_cache_free(void *b, int size)
 {
diff --git a/mm/slub.c b/mm/slub.c
index f10a892f1772..2a2be2a8a5d0 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3216,30 +3216,6 @@ static __always_inline void *slab_alloc(struct kmem_cache *s, struct list_lru *l
 	return slab_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, addr, orig_size);
 }
 
-static __always_inline
-void *__kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
-			     gfp_t gfpflags)
-{
-	void *ret = slab_alloc(s, lru, gfpflags, _RET_IP_, s->object_size);
-
-	trace_kmem_cache_alloc(_RET_IP_, ret, s->object_size,
-				s->size, gfpflags);
-
-	return ret;
-}
-
-void *kmem_cache_alloc(struct kmem_cache *s, gfp_t gfpflags)
-{
-	return __kmem_cache_alloc_lru(s, NULL, gfpflags);
-}
-EXPORT_SYMBOL(kmem_cache_alloc);
-
-void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
-			   gfp_t gfpflags)
-{
-	return __kmem_cache_alloc_lru(s, lru, gfpflags);
-}
-EXPORT_SYMBOL(kmem_cache_alloc_lru);
 
 #ifdef CONFIG_TRACING
 void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size)
@@ -3252,16 +3228,17 @@ void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size)
 EXPORT_SYMBOL(kmem_cache_alloc_trace);
 #endif
 
-void *kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, int node)
+void *__kmem_cache_alloc_node(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags,
+			      int node, unsigned long caller __maybe_unused)
 {
-	void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, s->object_size);
+	void *ret = slab_alloc_node(s, lru, gfpflags, node, caller, s->object_size);
 
-	trace_kmem_cache_alloc_node(_RET_IP_, ret,
-				    s->object_size, s->size, gfpflags, node);
+	trace_kmem_cache_alloc_node(caller, ret, s->object_size,
+				    s->size, gfpflags, node);
 
 	return ret;
 }
-EXPORT_SYMBOL(kmem_cache_alloc_node);
+EXPORT_SYMBOL(__kmem_cache_alloc_node);
 
 #ifdef CONFIG_TRACING
 void *kmem_cache_alloc_node_trace(struct kmem_cache *s,
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 11/23] mm/slab_common: kmalloc_node: pass large requests to page allocator
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (9 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 10/23] mm/slab_common: cleanup kmem_cache_alloc{,node,lru} Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-14  8:57 ` [PATCH v2 12/23] mm/slab_common: cleanup kmalloc() Hyeonggon Yoo
                   ` (12 subsequent siblings)
  23 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Now that kmalloc_large_node() is in common code, pass large requests
to page allocator in kmalloc_node() using kmalloc_large_node().

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
---
 include/linux/slab.h | 26 +++++++++++++++++++-------
 1 file changed, 19 insertions(+), 7 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 1b5bdcb0fd31..eb457f20f415 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -608,23 +608,35 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags)
 	return __kmalloc(size, flags);
 }
 
+#ifndef CONFIG_SLOB
 static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
 {
-#ifndef CONFIG_SLOB
-	if (__builtin_constant_p(size) &&
-		size <= KMALLOC_MAX_CACHE_SIZE) {
-		unsigned int i = kmalloc_index(size);
+	if (__builtin_constant_p(size)) {
+		unsigned int index;
 
-		if (!i)
+		if (size > KMALLOC_MAX_CACHE_SIZE)
+			return kmalloc_large_node(size, flags, node);
+
+		index = kmalloc_index(size);
+
+		if (!index)
 			return ZERO_SIZE_PTR;
 
 		return kmem_cache_alloc_node_trace(
-				kmalloc_caches[kmalloc_type(flags)][i],
+				kmalloc_caches[kmalloc_type(flags)][index],
 						flags, node, size);
 	}
-#endif
 	return __kmalloc_node(size, flags, node);
 }
+#else
+static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
+{
+	if (__builtin_constant_p(size) && size > KMALLOC_MAX_CACHE_SIZE)
+		return kmalloc_large_node(size, flags, node);
+
+	return __kmalloc_node(size, flags, node);
+}
+#endif
 
 /**
  * kmalloc_array - allocate memory for an array.
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 12/23] mm/slab_common: cleanup kmalloc()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (10 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 11/23] mm/slab_common: kmalloc_node: pass large requests to page allocator Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-26 18:00   ` Joe Perches
  2022-04-27  7:50   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 13/23] mm/slab: kmalloc: pass requests larger than order-1 page to page allocator Hyeonggon Yoo
                   ` (11 subsequent siblings)
  23 siblings, 2 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Now that kmalloc() and kmalloc_node() do same job, make kmalloc()
wrapper of kmalloc_node().

Remove kmem_cache_alloc_trace() that is now unused.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/linux/slab.h | 93 +++++++++++++++-----------------------------
 mm/slab.c            | 16 --------
 mm/slub.c            | 12 ------
 3 files changed, 32 insertions(+), 89 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index eb457f20f415..ea168f8a248d 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -497,23 +497,10 @@ static __always_inline void kfree_bulk(size_t size, void **p)
 }
 
 #ifdef CONFIG_TRACING
-extern void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t flags, size_t size)
-				   __assume_slab_alignment __alloc_size(3);
-
 extern void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags,
 					 int node, size_t size) __assume_slab_alignment
 								__alloc_size(4);
-
 #else /* CONFIG_TRACING */
-static __always_inline __alloc_size(3) void *kmem_cache_alloc_trace(struct kmem_cache *s,
-								    gfp_t flags, size_t size)
-{
-	void *ret = kmem_cache_alloc(s, flags);
-
-	ret = kasan_kmalloc(s, ret, size, flags);
-	return ret;
-}
-
 static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags,
 							 int node, size_t size)
 {
@@ -532,6 +519,37 @@ static __always_inline void *kmalloc_large(size_t size, gfp_t flags)
 	return kmalloc_large_node(size, flags, NUMA_NO_NODE);
 }
 
+#ifndef CONFIG_SLOB
+static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
+{
+	if (__builtin_constant_p(size)) {
+		unsigned int index;
+
+		if (size > KMALLOC_MAX_CACHE_SIZE)
+			return kmalloc_large_node(size, flags, node);
+
+		index = kmalloc_index(size);
+
+		if (!index)
+			return ZERO_SIZE_PTR;
+
+		return kmem_cache_alloc_node_trace(
+				kmalloc_caches[kmalloc_type(flags)][index],
+						flags, node, size);
+	}
+	return __kmalloc_node(size, flags, node);
+}
+#else
+static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
+{
+	if (__builtin_constant_p(size) && size > KMALLOC_MAX_CACHE_SIZE)
+		return kmalloc_large_node(size, flags, node);
+
+	return __kmalloc_node(size, flags, node);
+}
+#endif
+
+
 /**
  * kmalloc - allocate memory
  * @size: how many bytes of memory are required.
@@ -588,55 +606,8 @@ static __always_inline void *kmalloc_large(size_t size, gfp_t flags)
  */
 static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags)
 {
-	if (__builtin_constant_p(size)) {
-#ifndef CONFIG_SLOB
-		unsigned int index;
-#endif
-		if (size > KMALLOC_MAX_CACHE_SIZE)
-			return kmalloc_large(size, flags);
-#ifndef CONFIG_SLOB
-		index = kmalloc_index(size);
-
-		if (!index)
-			return ZERO_SIZE_PTR;
-
-		return kmem_cache_alloc_trace(
-				kmalloc_caches[kmalloc_type(flags)][index],
-				flags, size);
-#endif
-	}
-	return __kmalloc(size, flags);
-}
-
-#ifndef CONFIG_SLOB
-static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
-{
-	if (__builtin_constant_p(size)) {
-		unsigned int index;
-
-		if (size > KMALLOC_MAX_CACHE_SIZE)
-			return kmalloc_large_node(size, flags, node);
-
-		index = kmalloc_index(size);
-
-		if (!index)
-			return ZERO_SIZE_PTR;
-
-		return kmem_cache_alloc_node_trace(
-				kmalloc_caches[kmalloc_type(flags)][index],
-						flags, node, size);
-	}
-	return __kmalloc_node(size, flags, node);
-}
-#else
-static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
-{
-	if (__builtin_constant_p(size) && size > KMALLOC_MAX_CACHE_SIZE)
-		return kmalloc_large_node(size, flags, node);
-
-	return __kmalloc_node(size, flags, node);
+	return kmalloc_node(size, flags, NUMA_NO_NODE);
 }
-#endif
 
 /**
  * kmalloc_array - allocate memory for an array.
diff --git a/mm/slab.c b/mm/slab.c
index c5ffe54c207a..b0aaca017f42 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3507,22 +3507,6 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
 }
 EXPORT_SYMBOL(kmem_cache_alloc_bulk);
 
-#ifdef CONFIG_TRACING
-void *
-kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size)
-{
-	void *ret;
-
-	ret = slab_alloc(cachep, NULL, flags, size, _RET_IP_);
-
-	ret = kasan_kmalloc(cachep, ret, size, flags);
-	trace_kmalloc(_RET_IP_, ret,
-		      size, cachep->size, flags);
-	return ret;
-}
-EXPORT_SYMBOL(kmem_cache_alloc_trace);
-#endif
-
 #ifdef CONFIG_TRACING
 void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
 				  gfp_t flags,
diff --git a/mm/slub.c b/mm/slub.c
index 2a2be2a8a5d0..892988990da7 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3216,18 +3216,6 @@ static __always_inline void *slab_alloc(struct kmem_cache *s, struct list_lru *l
 	return slab_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, addr, orig_size);
 }
 
-
-#ifdef CONFIG_TRACING
-void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size)
-{
-	void *ret = slab_alloc(s, NULL, gfpflags, _RET_IP_, size);
-	trace_kmalloc(_RET_IP_, ret, size, s->size, gfpflags);
-	ret = kasan_kmalloc(s, ret, size, gfpflags);
-	return ret;
-}
-EXPORT_SYMBOL(kmem_cache_alloc_trace);
-#endif
-
 void *__kmem_cache_alloc_node(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags,
 			      int node, unsigned long caller __maybe_unused)
 {
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 13/23] mm/slab: kmalloc: pass requests larger than order-1 page to page allocator
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (11 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 12/23] mm/slab_common: cleanup kmalloc() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-27  8:10   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 14/23] mm/slab_common: print cache name in tracepoints Hyeonggon Yoo
                   ` (10 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

There is not much benefit for serving large objects in kmalloc().
Let's pass large requests to page allocator like SLUB for better
maintenance of common code.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---

Changes from previous series (Thanks to Vlastimil):
- Disable/Enable irqs around free_large_kmalloc()
- Do not lose NUMA locality in __do_kmalloc
- Some style fixes (use slab->slab_cache instead of virt_to_cache)
- Remove unsupported sizes in __kmalloc_index

Changes from v1:
- instead of defining varaible x, just casting to (void *) while calling free_large_kmalloc().


 include/linux/slab.h | 23 +++++------------------
 mm/slab.c            | 44 ++++++++++++++++++++++++++++++--------------
 mm/slab.h            |  3 +++
 mm/slab_common.c     | 25 ++++++++++++++++++-------
 mm/slub.c            | 19 -------------------
 5 files changed, 56 insertions(+), 58 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index ea168f8a248d..c8c82087c3f9 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -231,27 +231,17 @@ void kmem_dump_obj(void *object);
 
 #ifdef CONFIG_SLAB
 /*
- * The largest kmalloc size supported by the SLAB allocators is
- * 32 megabyte (2^25) or the maximum allocatable page order if that is
- * less than 32 MB.
- *
- * WARNING: Its not easy to increase this value since the allocators have
- * to do various tricks to work around compiler limitations in order to
- * ensure proper constant folding.
+ * SLAB and SLUB directly allocates requests fitting in to an order-1 page
+ * (PAGE_SIZE*2).  Larger requests are passed to the page allocator.
  */
-#define KMALLOC_SHIFT_HIGH	((MAX_ORDER + PAGE_SHIFT - 1) <= 25 ? \
-				(MAX_ORDER + PAGE_SHIFT - 1) : 25)
-#define KMALLOC_SHIFT_MAX	KMALLOC_SHIFT_HIGH
+#define KMALLOC_SHIFT_HIGH	(PAGE_SHIFT + 1)
+#define KMALLOC_SHIFT_MAX	(MAX_ORDER + PAGE_SHIFT - 1)
 #ifndef KMALLOC_SHIFT_LOW
 #define KMALLOC_SHIFT_LOW	5
 #endif
 #endif
 
 #ifdef CONFIG_SLUB
-/*
- * SLUB directly allocates requests fitting in to an order-1 page
- * (PAGE_SIZE*2).  Larger requests are passed to the page allocator.
- */
 #define KMALLOC_SHIFT_HIGH	(PAGE_SHIFT + 1)
 #define KMALLOC_SHIFT_MAX	(MAX_ORDER + PAGE_SHIFT - 1)
 #ifndef KMALLOC_SHIFT_LOW
@@ -403,10 +393,6 @@ static __always_inline unsigned int __kmalloc_index(size_t size,
 	if (size <= 512 * 1024) return 19;
 	if (size <= 1024 * 1024) return 20;
 	if (size <=  2 * 1024 * 1024) return 21;
-	if (size <=  4 * 1024 * 1024) return 22;
-	if (size <=  8 * 1024 * 1024) return 23;
-	if (size <=  16 * 1024 * 1024) return 24;
-	if (size <=  32 * 1024 * 1024) return 25;
 
 	if (!IS_ENABLED(CONFIG_PROFILE_ALL_BRANCHES) && size_is_constant)
 		BUILD_BUG_ON_MSG(1, "unexpected size in kmalloc_index()");
@@ -416,6 +402,7 @@ static __always_inline unsigned int __kmalloc_index(size_t size,
 	/* Will never be reached. Needed because the compiler may complain */
 	return -1;
 }
+static_assert(PAGE_SHIFT <= 20);
 #define kmalloc_index(s) __kmalloc_index(s, true)
 #endif /* !CONFIG_SLOB */
 
diff --git a/mm/slab.c b/mm/slab.c
index b0aaca017f42..1dfe0f9d5882 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3533,7 +3533,7 @@ __do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller)
 	void *ret;
 
 	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
-		return NULL;
+		return kmalloc_large_node(size, flags, node);
 	cachep = kmalloc_slab(size, flags);
 	if (unlikely(ZERO_OR_NULL_PTR(cachep)))
 		return cachep;
@@ -3607,15 +3607,25 @@ void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p)
 {
 	struct kmem_cache *s;
 	size_t i;
+	struct folio *folio;
 
 	local_irq_disable();
 	for (i = 0; i < size; i++) {
 		void *objp = p[i];
 
-		if (!orig_s) /* called via kfree_bulk */
-			s = virt_to_cache(objp);
-		else
+		if (!orig_s) {
+			folio = virt_to_folio(objp);
+			/* called via kfree_bulk */
+			if (!folio_test_slab(folio)) {
+				local_irq_enable();
+				free_large_kmalloc(folio, objp);
+				local_irq_disable();
+				continue;
+			}
+			s = folio_slab(folio)->slab_cache;
+		} else
 			s = cache_from_obj(orig_s, objp);
+
 		if (!s)
 			continue;
 
@@ -3644,20 +3654,24 @@ void kfree(const void *objp)
 {
 	struct kmem_cache *c;
 	unsigned long flags;
+	struct folio *folio;
 
 	trace_kfree(_RET_IP_, objp);
 
 	if (unlikely(ZERO_OR_NULL_PTR(objp)))
 		return;
-	local_irq_save(flags);
-	kfree_debugcheck(objp);
-	c = virt_to_cache(objp);
-	if (!c) {
-		local_irq_restore(flags);
+
+	folio = virt_to_folio(objp);
+	if (!folio_test_slab(folio)) {
+		free_large_kmalloc(folio, (void *)objp);
 		return;
 	}
-	debug_check_no_locks_freed(objp, c->object_size);
 
+	c = folio_slab(folio)->slab_cache;
+
+	local_irq_save(flags);
+	kfree_debugcheck(objp);
+	debug_check_no_locks_freed(objp, c->object_size);
 	debug_check_no_obj_freed(objp, c->object_size);
 	__cache_free(c, (void *)objp, _RET_IP_);
 	local_irq_restore(flags);
@@ -4079,15 +4093,17 @@ void __check_heap_object(const void *ptr, unsigned long n,
 size_t __ksize(const void *objp)
 {
 	struct kmem_cache *c;
-	size_t size;
+	struct folio *folio;
 
 	BUG_ON(!objp);
 	if (unlikely(objp == ZERO_SIZE_PTR))
 		return 0;
 
-	c = virt_to_cache(objp);
-	size = c ? c->object_size : 0;
+	folio = virt_to_folio(objp);
+	if (!folio_test_slab(folio))
+		return folio_size(folio);
 
-	return size;
+	c = folio_slab(folio)->slab_cache;
+	return c->object_size;
 }
 EXPORT_SYMBOL(__ksize);
diff --git a/mm/slab.h b/mm/slab.h
index f7d018100994..b864c5bc4c25 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -681,6 +681,9 @@ static inline struct kmem_cache *cache_from_obj(struct kmem_cache *s, void *x)
 		print_tracking(cachep, x);
 	return cachep;
 }
+
+void free_large_kmalloc(struct folio *folio, void *object);
+
 #endif /* CONFIG_SLOB */
 
 static inline size_t slab_ksize(const struct kmem_cache *s)
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 30684efc89d7..960cc07c3a91 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -764,8 +764,8 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags)
 
 /*
  * kmalloc_info[] is to make slub_debug=,kmalloc-xx option work at boot time.
- * kmalloc_index() supports up to 2^25=32MB, so the final entry of the table is
- * kmalloc-32M.
+ * kmalloc_index() supports up to 2^21=2MB, so the final entry of the table is
+ * kmalloc-2M.
  */
 const struct kmalloc_info_struct kmalloc_info[] __initconst = {
 	INIT_KMALLOC_INFO(0, 0),
@@ -789,11 +789,7 @@ const struct kmalloc_info_struct kmalloc_info[] __initconst = {
 	INIT_KMALLOC_INFO(262144, 256k),
 	INIT_KMALLOC_INFO(524288, 512k),
 	INIT_KMALLOC_INFO(1048576, 1M),
-	INIT_KMALLOC_INFO(2097152, 2M),
-	INIT_KMALLOC_INFO(4194304, 4M),
-	INIT_KMALLOC_INFO(8388608, 8M),
-	INIT_KMALLOC_INFO(16777216, 16M),
-	INIT_KMALLOC_INFO(33554432, 32M)
+	INIT_KMALLOC_INFO(2097152, 2M)
 };
 
 /*
@@ -906,6 +902,21 @@ void __init create_kmalloc_caches(slab_flags_t flags)
 	/* Kmalloc array is now usable */
 	slab_state = UP;
 }
+
+void free_large_kmalloc(struct folio *folio, void *object)
+{
+	unsigned int order = folio_order(folio);
+
+	if (WARN_ON_ONCE(order == 0))
+		pr_warn_once("object pointer: 0x%p\n", object);
+
+	kmemleak_free(object);
+	kasan_kfree_large(object);
+
+	mod_lruvec_page_state(folio_page(folio, 0), NR_SLAB_UNRECLAIMABLE_B,
+			      -(PAGE_SIZE << order));
+	__free_pages(folio_page(folio, 0), order);
+}
 #endif /* !CONFIG_SLOB */
 
 gfp_t kmalloc_fix_flags(gfp_t flags)
diff --git a/mm/slub.c b/mm/slub.c
index 892988990da7..1dc9e8eebb62 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -1679,12 +1679,6 @@ static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab,
  * Hooks for other subsystems that check memory allocations. In a typical
  * production configuration these hooks all should produce no code at all.
  */
-static __always_inline void kfree_hook(void *x)
-{
-	kmemleak_free(x);
-	kasan_kfree_large(x);
-}
-
 static __always_inline bool slab_free_hook(struct kmem_cache *s,
 						void *x, bool init)
 {
@@ -3490,19 +3484,6 @@ struct detached_freelist {
 	struct kmem_cache *s;
 };
 
-static inline void free_large_kmalloc(struct folio *folio, void *object)
-{
-	unsigned int order = folio_order(folio);
-
-	if (WARN_ON_ONCE(order == 0))
-		pr_warn_once("object pointer: 0x%p\n", object);
-
-	kfree_hook(object);
-	mod_lruvec_page_state(folio_page(folio, 0), NR_SLAB_UNRECLAIMABLE_B,
-			      -(PAGE_SIZE << order));
-	__free_pages(folio_page(folio, 0), order);
-}
-
 /*
  * This function progressively scans the array with free objects (with
  * a limited look ahead) and extract objects belonging to the same
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 14/23] mm/slab_common: print cache name in tracepoints
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (12 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 13/23] mm/slab: kmalloc: pass requests larger than order-1 page to page allocator Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-29 14:05   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 15/23] mm/slab_common: use same tracepoint in kmalloc and normal caches Hyeonggon Yoo
                   ` (9 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Print cache name in tracepoints. If there is no corresponding cache
(kmalloc in SLOB or kmalloc_large_node), use KMALLOC_{,LARGE_}NAME
macro.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/trace/events/kmem.h | 34 +++++++++++++++++++---------------
 mm/slab.c                   |  9 +++++----
 mm/slab.h                   |  4 ++++
 mm/slab_common.c            |  6 ++----
 mm/slob.c                   | 10 +++++-----
 mm/slub.c                   | 10 +++++-----
 6 files changed, 40 insertions(+), 33 deletions(-)

diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h
index ddc8c944f417..35e6887c6101 100644
--- a/include/trace/events/kmem.h
+++ b/include/trace/events/kmem.h
@@ -61,16 +61,18 @@ DEFINE_EVENT(kmem_alloc, kmem_cache_alloc,
 
 DECLARE_EVENT_CLASS(kmem_alloc_node,
 
-	TP_PROTO(unsigned long call_site,
+	TP_PROTO(const char *name,
+		 unsigned long call_site,
 		 const void *ptr,
 		 size_t bytes_req,
 		 size_t bytes_alloc,
 		 gfp_t gfp_flags,
 		 int node),
 
-	TP_ARGS(call_site, ptr, bytes_req, bytes_alloc, gfp_flags, node),
+	TP_ARGS(name, call_site, ptr, bytes_req, bytes_alloc, gfp_flags, node),
 
 	TP_STRUCT__entry(
+		__string(	name,		name		)
 		__field(	unsigned long,	call_site	)
 		__field(	const void *,	ptr		)
 		__field(	size_t,		bytes_req	)
@@ -80,6 +82,7 @@ DECLARE_EVENT_CLASS(kmem_alloc_node,
 	),
 
 	TP_fast_assign(
+		__assign_str(name, name);
 		__entry->call_site	= call_site;
 		__entry->ptr		= ptr;
 		__entry->bytes_req	= bytes_req;
@@ -88,7 +91,8 @@ DECLARE_EVENT_CLASS(kmem_alloc_node,
 		__entry->node		= node;
 	),
 
-	TP_printk("call_site=%pS ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s node=%d",
+	TP_printk("name=%s call_site=%pS ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s node=%d",
+		__get_str(name),
 		(void *)__entry->call_site,
 		__entry->ptr,
 		__entry->bytes_req,
@@ -99,20 +103,20 @@ DECLARE_EVENT_CLASS(kmem_alloc_node,
 
 DEFINE_EVENT(kmem_alloc_node, kmalloc_node,
 
-	TP_PROTO(unsigned long call_site, const void *ptr,
-		 size_t bytes_req, size_t bytes_alloc,
+	TP_PROTO(const char *name, unsigned long call_site,
+		 const void *ptr, size_t bytes_req, size_t bytes_alloc,
 		 gfp_t gfp_flags, int node),
 
-	TP_ARGS(call_site, ptr, bytes_req, bytes_alloc, gfp_flags, node)
+	TP_ARGS(name, call_site, ptr, bytes_req, bytes_alloc, gfp_flags, node)
 );
 
 DEFINE_EVENT(kmem_alloc_node, kmem_cache_alloc_node,
 
-	TP_PROTO(unsigned long call_site, const void *ptr,
-		 size_t bytes_req, size_t bytes_alloc,
+	TP_PROTO(const char *name, unsigned long call_site,
+		 const void *ptr, size_t bytes_req, size_t bytes_alloc,
 		 gfp_t gfp_flags, int node),
 
-	TP_ARGS(call_site, ptr, bytes_req, bytes_alloc, gfp_flags, node)
+	TP_ARGS(name, call_site, ptr, bytes_req, bytes_alloc, gfp_flags, node)
 );
 
 TRACE_EVENT(kfree,
@@ -137,24 +141,24 @@ TRACE_EVENT(kfree,
 
 TRACE_EVENT(kmem_cache_free,
 
-	TP_PROTO(unsigned long call_site, const void *ptr, const char *name),
+	TP_PROTO(const char *name, unsigned long call_site, const void *ptr),
 
-	TP_ARGS(call_site, ptr, name),
+	TP_ARGS(name, call_site, ptr),
 
 	TP_STRUCT__entry(
+		__string(	name,	name	)
 		__field(	unsigned long,	call_site	)
 		__field(	const void *,	ptr		)
-		__string(	name,	name	)
 	),
 
 	TP_fast_assign(
+		__assign_str(name, name);
 		__entry->call_site	= call_site;
 		__entry->ptr		= ptr;
-		__assign_str(name, name);
 	),
 
-	TP_printk("call_site=%pS ptr=%p name=%s",
-		  (void *)__entry->call_site, __entry->ptr, __get_str(name))
+	TP_printk("name=%s call_site=%pS ptr=%p",
+		  __get_str(name), (void *)__entry->call_site, __entry->ptr)
 );
 
 TRACE_EVENT(mm_page_free,
diff --git a/mm/slab.c b/mm/slab.c
index 1dfe0f9d5882..3c47d0979706 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3448,8 +3448,9 @@ void *__kmem_cache_alloc_node(struct kmem_cache *cachep, struct list_lru *lru,
 	void *ret = slab_alloc_node(cachep, lru, flags, nodeid,
 				    cachep->object_size, caller);
 
-	trace_kmem_cache_alloc_node(caller, ret, cachep->object_size,
-				    cachep->size, flags, nodeid);
+	trace_kmem_cache_alloc_node(cachep->name, caller, ret,
+				    cachep->object_size, cachep->size,
+				    flags, nodeid);
 
 	return ret;
 }
@@ -3518,7 +3519,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
 	ret = slab_alloc_node(cachep, NULL, flags, nodeid, size, _RET_IP_);
 
 	ret = kasan_kmalloc(cachep, ret, size, flags);
-	trace_kmalloc_node(_RET_IP_, ret,
+	trace_kmalloc_node(cachep->name, _RET_IP_, ret,
 			   size, cachep->size,
 			   flags, nodeid);
 	return ret;
@@ -3593,7 +3594,7 @@ void kmem_cache_free(struct kmem_cache *cachep, void *objp)
 	if (!cachep)
 		return;
 
-	trace_kmem_cache_free(_RET_IP_, objp, cachep->name);
+	trace_kmem_cache_free(cachep->name, _RET_IP_, objp);
 	local_irq_save(flags);
 	debug_check_no_locks_freed(objp, cachep->object_size);
 	if (!(cachep->flags & SLAB_DEBUG_OBJECTS))
diff --git a/mm/slab.h b/mm/slab.h
index b864c5bc4c25..45ddb19df319 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -275,6 +275,10 @@ void create_kmalloc_caches(slab_flags_t);
 struct kmem_cache *kmalloc_slab(size_t, gfp_t);
 #endif
 
+/* cache names for tracepoints where it has no corresponding cache  */
+#define KMALLOC_LARGE_NAME "kmalloc_large_node"
+#define KMALLOC_NAME "kmalloc_node"
+
 gfp_t kmalloc_fix_flags(gfp_t flags);
 
 /* Functions provided by the slab allocators */
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 960cc07c3a91..416f0a1f17a6 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -956,10 +956,8 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node)
 	ptr = kasan_kmalloc_large(ptr, size, flags);
 	/* As ptr might get tagged, call kmemleak hook after KASAN. */
 	kmemleak_alloc(ptr, size, 1, flags);
-	trace_kmalloc_node(_RET_IP_, ptr,
-			   size, PAGE_SIZE << order,
-			   flags, node);
-
+	trace_kmalloc_node(KMALLOC_LARGE_NAME, _RET_IP_, ptr, size,
+			   PAGE_SIZE << order, flags, node);
 	return ptr;
 }
 EXPORT_SYMBOL(kmalloc_large_node);
diff --git a/mm/slob.c b/mm/slob.c
index 6c7c30845056..8abde6037d95 100644
--- a/mm/slob.c
+++ b/mm/slob.c
@@ -505,7 +505,7 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller)
 		*m = size;
 		ret = (void *)m + minalign;
 
-		trace_kmalloc_node(caller, ret,
+		trace_kmalloc_node(KMALLOC_NAME, caller, ret,
 				   size, size + minalign, gfp, node);
 	} else {
 		unsigned int order = get_order(size);
@@ -514,7 +514,7 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller)
 			gfp |= __GFP_COMP;
 		ret = slob_new_pages(gfp, order, node);
 
-		trace_kmalloc_node(caller, ret,
+		trace_kmalloc_node(KMALLOC_LARGE_NAME, caller, ret,
 				   size, PAGE_SIZE << order, gfp, node);
 	}
 
@@ -597,12 +597,12 @@ static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node,
 
 	if (c->size < PAGE_SIZE) {
 		b = slob_alloc(c->size, flags, c->align, node, 0);
-		trace_kmem_cache_alloc_node(caller, b, c->object_size,
+		trace_kmem_cache_alloc_node(c->name, caller, b, c->object_size,
 					    SLOB_UNITS(c->size) * SLOB_UNIT,
 					    flags, node);
 	} else {
 		b = slob_new_pages(flags, get_order(c->size), node);
-		trace_kmem_cache_alloc_node(caller, b, c->object_size,
+		trace_kmem_cache_alloc_node(c->name, caller, b, c->object_size,
 					    PAGE_SIZE << get_order(c->size),
 					    flags, node);
 	}
@@ -648,7 +648,7 @@ static void kmem_rcu_free(struct rcu_head *head)
 void kmem_cache_free(struct kmem_cache *c, void *b)
 {
 	kmemleak_free_recursive(b, c->flags);
-	trace_kmem_cache_free(_RET_IP_, b, c->name);
+	trace_kmem_cache_free(c->name, _RET_IP_, b);
 	if (unlikely(c->flags & SLAB_TYPESAFE_BY_RCU)) {
 		struct slob_rcu *slob_rcu;
 		slob_rcu = b + (c->size - sizeof(struct slob_rcu));
diff --git a/mm/slub.c b/mm/slub.c
index 1dc9e8eebb62..de03fa1f5667 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3215,7 +3215,7 @@ void *__kmem_cache_alloc_node(struct kmem_cache *s, struct list_lru *lru, gfp_t
 {
 	void *ret = slab_alloc_node(s, lru, gfpflags, node, caller, s->object_size);
 
-	trace_kmem_cache_alloc_node(caller, ret, s->object_size,
+	trace_kmem_cache_alloc_node(s->name, caller, ret, s->object_size,
 				    s->size, gfpflags, node);
 
 	return ret;
@@ -3229,7 +3229,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *s,
 {
 	void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size);
 
-	trace_kmalloc_node(_RET_IP_, ret,
+	trace_kmalloc_node(s->name, _RET_IP_, ret,
 			   size, s->size, gfpflags, node);
 
 	ret = kasan_kmalloc(s, ret, size, gfpflags);
@@ -3471,7 +3471,7 @@ void kmem_cache_free(struct kmem_cache *s, void *x)
 	s = cache_from_obj(s, x);
 	if (!s)
 		return;
-	trace_kmem_cache_free(_RET_IP_, x, s->name);
+	trace_kmem_cache_free(s->name, _RET_IP_, x);
 	slab_free(s, virt_to_slab(x), x, NULL, 1, _RET_IP_);
 }
 EXPORT_SYMBOL(kmem_cache_free);
@@ -4352,7 +4352,7 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
 
 	ret = slab_alloc_node(s, NULL, flags, node, _RET_IP_, size);
 
-	trace_kmalloc_node(_RET_IP_, ret, size, s->size, flags, node);
+	trace_kmalloc_node(s->name, _RET_IP_, ret, size, s->size, flags, node);
 
 	ret = kasan_kmalloc(s, ret, size, flags);
 
@@ -4811,7 +4811,7 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 	ret = slab_alloc_node(s, NULL, gfpflags, node, caller, size);
 
 	/* Honor the call site pointer we received. */
-	trace_kmalloc_node(caller, ret, size, s->size, gfpflags, node);
+	trace_kmalloc_node(s->name, caller, ret, size, s->size, gfpflags, node);
 
 	return ret;
 }
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 15/23] mm/slab_common: use same tracepoint in kmalloc and normal caches
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (13 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 14/23] mm/slab_common: print cache name in tracepoints Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-14  8:57 ` [PATCH v2 16/23] mm/slab_common: rename tracepoint Hyeonggon Yoo
                   ` (8 subsequent siblings)
  23 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Now that tracepoints print cache names, we can distinguish kmalloc and
normal cache allocations.

Use same tracepoint in kmalloc and normal caches. After this patch,
there is only two tracepoints in slab allocators: kmem_cache_alloc_node
and kmem_cache_free.

Remove all unused tracepoints.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/trace/events/kmem.h | 79 -------------------------------------
 mm/slab.c                   |  8 ++--
 mm/slab_common.c            |  9 ++---
 mm/slob.c                   | 14 ++++---
 mm/slub.c                   | 19 +++++----
 5 files changed, 27 insertions(+), 102 deletions(-)

diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h
index 35e6887c6101..ca67ba5fd76a 100644
--- a/include/trace/events/kmem.h
+++ b/include/trace/events/kmem.h
@@ -9,56 +9,6 @@
 #include <linux/tracepoint.h>
 #include <trace/events/mmflags.h>
 
-DECLARE_EVENT_CLASS(kmem_alloc,
-
-	TP_PROTO(unsigned long call_site,
-		 const void *ptr,
-		 size_t bytes_req,
-		 size_t bytes_alloc,
-		 gfp_t gfp_flags),
-
-	TP_ARGS(call_site, ptr, bytes_req, bytes_alloc, gfp_flags),
-
-	TP_STRUCT__entry(
-		__field(	unsigned long,	call_site	)
-		__field(	const void *,	ptr		)
-		__field(	size_t,		bytes_req	)
-		__field(	size_t,		bytes_alloc	)
-		__field(	gfp_t,		gfp_flags	)
-	),
-
-	TP_fast_assign(
-		__entry->call_site	= call_site;
-		__entry->ptr		= ptr;
-		__entry->bytes_req	= bytes_req;
-		__entry->bytes_alloc	= bytes_alloc;
-		__entry->gfp_flags	= gfp_flags;
-	),
-
-	TP_printk("call_site=%pS ptr=%p bytes_req=%zu bytes_alloc=%zu gfp_flags=%s",
-		(void *)__entry->call_site,
-		__entry->ptr,
-		__entry->bytes_req,
-		__entry->bytes_alloc,
-		show_gfp_flags(__entry->gfp_flags))
-);
-
-DEFINE_EVENT(kmem_alloc, kmalloc,
-
-	TP_PROTO(unsigned long call_site, const void *ptr,
-		 size_t bytes_req, size_t bytes_alloc, gfp_t gfp_flags),
-
-	TP_ARGS(call_site, ptr, bytes_req, bytes_alloc, gfp_flags)
-);
-
-DEFINE_EVENT(kmem_alloc, kmem_cache_alloc,
-
-	TP_PROTO(unsigned long call_site, const void *ptr,
-		 size_t bytes_req, size_t bytes_alloc, gfp_t gfp_flags),
-
-	TP_ARGS(call_site, ptr, bytes_req, bytes_alloc, gfp_flags)
-);
-
 DECLARE_EVENT_CLASS(kmem_alloc_node,
 
 	TP_PROTO(const char *name,
@@ -101,15 +51,6 @@ DECLARE_EVENT_CLASS(kmem_alloc_node,
 		__entry->node)
 );
 
-DEFINE_EVENT(kmem_alloc_node, kmalloc_node,
-
-	TP_PROTO(const char *name, unsigned long call_site,
-		 const void *ptr, size_t bytes_req, size_t bytes_alloc,
-		 gfp_t gfp_flags, int node),
-
-	TP_ARGS(name, call_site, ptr, bytes_req, bytes_alloc, gfp_flags, node)
-);
-
 DEFINE_EVENT(kmem_alloc_node, kmem_cache_alloc_node,
 
 	TP_PROTO(const char *name, unsigned long call_site,
@@ -119,26 +60,6 @@ DEFINE_EVENT(kmem_alloc_node, kmem_cache_alloc_node,
 	TP_ARGS(name, call_site, ptr, bytes_req, bytes_alloc, gfp_flags, node)
 );
 
-TRACE_EVENT(kfree,
-
-	TP_PROTO(unsigned long call_site, const void *ptr),
-
-	TP_ARGS(call_site, ptr),
-
-	TP_STRUCT__entry(
-		__field(	unsigned long,	call_site	)
-		__field(	const void *,	ptr		)
-	),
-
-	TP_fast_assign(
-		__entry->call_site	= call_site;
-		__entry->ptr		= ptr;
-	),
-
-	TP_printk("call_site=%pS ptr=%p",
-		  (void *)__entry->call_site, __entry->ptr)
-);
-
 TRACE_EVENT(kmem_cache_free,
 
 	TP_PROTO(const char *name, unsigned long call_site, const void *ptr),
diff --git a/mm/slab.c b/mm/slab.c
index 3c47d0979706..b9959a6b5c48 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3519,9 +3519,9 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
 	ret = slab_alloc_node(cachep, NULL, flags, nodeid, size, _RET_IP_);
 
 	ret = kasan_kmalloc(cachep, ret, size, flags);
-	trace_kmalloc_node(cachep->name, _RET_IP_, ret,
-			   size, cachep->size,
-			   flags, nodeid);
+	trace_kmem_cache_alloc_node(cachep->name, _RET_IP_, ret,
+				    size, cachep->size,
+				    flags, nodeid);
 	return ret;
 }
 EXPORT_SYMBOL(kmem_cache_alloc_node_trace);
@@ -3657,7 +3657,6 @@ void kfree(const void *objp)
 	unsigned long flags;
 	struct folio *folio;
 
-	trace_kfree(_RET_IP_, objp);
 
 	if (unlikely(ZERO_OR_NULL_PTR(objp)))
 		return;
@@ -3669,6 +3668,7 @@ void kfree(const void *objp)
 	}
 
 	c = folio_slab(folio)->slab_cache;
+	trace_kmem_cache_free(c->name, _RET_IP_, objp);
 
 	local_irq_save(flags);
 	kfree_debugcheck(objp);
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 416f0a1f17a6..3d1569085c54 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -910,6 +910,7 @@ void free_large_kmalloc(struct folio *folio, void *object)
 	if (WARN_ON_ONCE(order == 0))
 		pr_warn_once("object pointer: 0x%p\n", object);
 
+	trace_kmem_cache_free(KMALLOC_LARGE_NAME, _RET_IP_, object);
 	kmemleak_free(object);
 	kasan_kfree_large(object);
 
@@ -956,8 +957,8 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node)
 	ptr = kasan_kmalloc_large(ptr, size, flags);
 	/* As ptr might get tagged, call kmemleak hook after KASAN. */
 	kmemleak_alloc(ptr, size, 1, flags);
-	trace_kmalloc_node(KMALLOC_LARGE_NAME, _RET_IP_, ptr, size,
-			   PAGE_SIZE << order, flags, node);
+	trace_kmem_cache_alloc_node(KMALLOC_LARGE_NAME, _RET_IP_, ptr, size,
+				    PAGE_SIZE << order, flags, node);
 	return ptr;
 }
 EXPORT_SYMBOL(kmalloc_large_node);
@@ -1290,11 +1291,7 @@ size_t ksize(const void *objp)
 EXPORT_SYMBOL(ksize);
 
 /* Tracepoints definitions. */
-EXPORT_TRACEPOINT_SYMBOL(kmalloc);
-EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
-EXPORT_TRACEPOINT_SYMBOL(kmalloc_node);
 EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc_node);
-EXPORT_TRACEPOINT_SYMBOL(kfree);
 EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free);
 
 int should_failslab(struct kmem_cache *s, gfp_t gfpflags)
diff --git a/mm/slob.c b/mm/slob.c
index 8abde6037d95..b1f291128e94 100644
--- a/mm/slob.c
+++ b/mm/slob.c
@@ -505,8 +505,8 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller)
 		*m = size;
 		ret = (void *)m + minalign;
 
-		trace_kmalloc_node(KMALLOC_NAME, caller, ret,
-				   size, size + minalign, gfp, node);
+		trace_kmem_cache_alloc_node(KMALLOC_NAME, caller, ret,
+					    size, size + minalign, gfp, node);
 	} else {
 		unsigned int order = get_order(size);
 
@@ -514,8 +514,9 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller)
 			gfp |= __GFP_COMP;
 		ret = slob_new_pages(gfp, order, node);
 
-		trace_kmalloc_node(KMALLOC_LARGE_NAME, caller, ret,
-				   size, PAGE_SIZE << order, gfp, node);
+		trace_kmem_cache_alloc_node(KMALLOC_LARGE_NAME, caller,
+					    ret, size, PAGE_SIZE << order,
+					    gfp, node);
 	}
 
 	kmemleak_alloc(ret, size, 1, gfp);
@@ -533,8 +534,6 @@ void kfree(const void *block)
 {
 	struct folio *sp;
 
-	trace_kfree(_RET_IP_, block);
-
 	if (unlikely(ZERO_OR_NULL_PTR(block)))
 		return;
 	kmemleak_free(block);
@@ -543,10 +542,13 @@ void kfree(const void *block)
 	if (folio_test_slab(sp)) {
 		int align = max_t(size_t, ARCH_KMALLOC_MINALIGN, ARCH_SLAB_MINALIGN);
 		unsigned int *m = (unsigned int *)(block - align);
+
+		trace_kmem_cache_free(KMALLOC_LARGE_NAME, _RET_IP_, block);
 		slob_free(m, *m + align);
 	} else {
 		unsigned int order = folio_order(sp);
 
+		trace_kmem_cache_free(KMALLOC_NAME, _RET_IP_, block);
 		mod_node_page_state(folio_pgdat(sp), NR_SLAB_UNRECLAIMABLE_B,
 				    -(PAGE_SIZE << order));
 		__free_pages(folio_page(sp, 0), order);
diff --git a/mm/slub.c b/mm/slub.c
index de03fa1f5667..d53e9e22d67e 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3229,8 +3229,8 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *s,
 {
 	void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size);
 
-	trace_kmalloc_node(s->name, _RET_IP_, ret,
-			   size, s->size, gfpflags, node);
+	trace_kmem_cache_alloc_node(s->name, _RET_IP_, ret,
+				    size, s->size, gfpflags, node);
 
 	ret = kasan_kmalloc(s, ret, size, gfpflags);
 	return ret;
@@ -4352,7 +4352,8 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
 
 	ret = slab_alloc_node(s, NULL, flags, node, _RET_IP_, size);
 
-	trace_kmalloc_node(s->name, _RET_IP_, ret, size, s->size, flags, node);
+	trace_kmem_cache_alloc_node(s->name, _RET_IP_, ret, size,
+				    s->size, flags, node);
 
 	ret = kasan_kmalloc(s, ret, size, flags);
 
@@ -4431,8 +4432,7 @@ void kfree(const void *x)
 	struct folio *folio;
 	struct slab *slab;
 	void *object = (void *)x;
-
-	trace_kfree(_RET_IP_, x);
+	struct kmem_cache *s;
 
 	if (unlikely(ZERO_OR_NULL_PTR(x)))
 		return;
@@ -4442,8 +4442,12 @@ void kfree(const void *x)
 		free_large_kmalloc(folio, object);
 		return;
 	}
+
 	slab = folio_slab(folio);
-	slab_free(slab->slab_cache, slab, object, NULL, 1, _RET_IP_);
+	s = slab->slab_cache;
+
+	trace_kmem_cache_free(s->name, _RET_IP_, x);
+	slab_free(s, slab, object, NULL, 1, _RET_IP_);
 }
 EXPORT_SYMBOL(kfree);
 
@@ -4811,7 +4815,8 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 	ret = slab_alloc_node(s, NULL, gfpflags, node, caller, size);
 
 	/* Honor the call site pointer we received. */
-	trace_kmalloc_node(s->name, caller, ret, size, s->size, gfpflags, node);
+	trace_kmem_cache_alloc_node(s->name, caller, ret, size,
+				    s->size, gfpflags, node);
 
 	return ret;
 }
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 16/23] mm/slab_common: rename tracepoint
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (14 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 15/23] mm/slab_common: use same tracepoint in kmalloc and normal caches Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-14  8:57 ` [PATCH v2 17/23] mm/slab_common: implement __kmem_cache_free() Hyeonggon Yoo
                   ` (7 subsequent siblings)
  23 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

To reduce overhead of printing tracepoint name,
rename trace_kmem_cache_alloc_node to kmem_cache_alloc.

Suggested-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/trace/events/kmem.h |  4 ++--
 mm/slab.c                   |  8 ++++----
 mm/slab_common.c            |  6 +++---
 mm/slob.c                   | 22 +++++++++++-----------
 mm/slub.c                   | 16 ++++++++--------
 5 files changed, 28 insertions(+), 28 deletions(-)

diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h
index ca67ba5fd76a..58edb2e3e5a4 100644
--- a/include/trace/events/kmem.h
+++ b/include/trace/events/kmem.h
@@ -9,7 +9,7 @@
 #include <linux/tracepoint.h>
 #include <trace/events/mmflags.h>
 
-DECLARE_EVENT_CLASS(kmem_alloc_node,
+DECLARE_EVENT_CLASS(kmem_alloc,
 
 	TP_PROTO(const char *name,
 		 unsigned long call_site,
@@ -51,7 +51,7 @@ DECLARE_EVENT_CLASS(kmem_alloc_node,
 		__entry->node)
 );
 
-DEFINE_EVENT(kmem_alloc_node, kmem_cache_alloc_node,
+DEFINE_EVENT(kmem_alloc, kmem_cache_alloc,
 
 	TP_PROTO(const char *name, unsigned long call_site,
 		 const void *ptr, size_t bytes_req, size_t bytes_alloc,
diff --git a/mm/slab.c b/mm/slab.c
index b9959a6b5c48..424168b96790 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3448,7 +3448,7 @@ void *__kmem_cache_alloc_node(struct kmem_cache *cachep, struct list_lru *lru,
 	void *ret = slab_alloc_node(cachep, lru, flags, nodeid,
 				    cachep->object_size, caller);
 
-	trace_kmem_cache_alloc_node(cachep->name, caller, ret,
+	trace_kmem_cache_alloc(cachep->name, caller, ret,
 				    cachep->object_size, cachep->size,
 				    flags, nodeid);
 
@@ -3519,9 +3519,9 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
 	ret = slab_alloc_node(cachep, NULL, flags, nodeid, size, _RET_IP_);
 
 	ret = kasan_kmalloc(cachep, ret, size, flags);
-	trace_kmem_cache_alloc_node(cachep->name, _RET_IP_, ret,
-				    size, cachep->size,
-				    flags, nodeid);
+	trace_kmem_cache_alloc(cachep->name, _RET_IP_, ret,
+			       size, cachep->size,
+			       flags, nodeid);
 	return ret;
 }
 EXPORT_SYMBOL(kmem_cache_alloc_node_trace);
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 3d1569085c54..3cd5d7a47ec7 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -957,8 +957,8 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node)
 	ptr = kasan_kmalloc_large(ptr, size, flags);
 	/* As ptr might get tagged, call kmemleak hook after KASAN. */
 	kmemleak_alloc(ptr, size, 1, flags);
-	trace_kmem_cache_alloc_node(KMALLOC_LARGE_NAME, _RET_IP_, ptr, size,
-				    PAGE_SIZE << order, flags, node);
+	trace_kmem_cache_alloc(KMALLOC_LARGE_NAME, _RET_IP_, ptr, size,
+			       PAGE_SIZE << order, flags, node);
 	return ptr;
 }
 EXPORT_SYMBOL(kmalloc_large_node);
@@ -1291,7 +1291,7 @@ size_t ksize(const void *objp)
 EXPORT_SYMBOL(ksize);
 
 /* Tracepoints definitions. */
-EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc_node);
+EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc);
 EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free);
 
 int should_failslab(struct kmem_cache *s, gfp_t gfpflags)
diff --git a/mm/slob.c b/mm/slob.c
index b1f291128e94..1bb4c577b908 100644
--- a/mm/slob.c
+++ b/mm/slob.c
@@ -505,8 +505,8 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller)
 		*m = size;
 		ret = (void *)m + minalign;
 
-		trace_kmem_cache_alloc_node(KMALLOC_NAME, caller, ret,
-					    size, size + minalign, gfp, node);
+		trace_kmem_cache_alloc(KMALLOC_NAME, caller, ret,
+				       size, size + minalign, gfp, node);
 	} else {
 		unsigned int order = get_order(size);
 
@@ -514,9 +514,9 @@ __do_kmalloc_node(size_t size, gfp_t gfp, int node, unsigned long caller)
 			gfp |= __GFP_COMP;
 		ret = slob_new_pages(gfp, order, node);
 
-		trace_kmem_cache_alloc_node(KMALLOC_LARGE_NAME, caller,
-					    ret, size, PAGE_SIZE << order,
-					    gfp, node);
+		trace_kmem_cache_alloc(KMALLOC_LARGE_NAME, caller,
+				       ret, size, PAGE_SIZE << order,
+				       gfp, node);
 	}
 
 	kmemleak_alloc(ret, size, 1, gfp);
@@ -599,14 +599,14 @@ static void *slob_alloc_node(struct kmem_cache *c, gfp_t flags, int node,
 
 	if (c->size < PAGE_SIZE) {
 		b = slob_alloc(c->size, flags, c->align, node, 0);
-		trace_kmem_cache_alloc_node(c->name, caller, b, c->object_size,
-					    SLOB_UNITS(c->size) * SLOB_UNIT,
-					    flags, node);
+		trace_kmem_cache_alloc(c->name, caller, b, c->object_size,
+				       SLOB_UNITS(c->size) * SLOB_UNIT,
+				       flags, node);
 	} else {
 		b = slob_new_pages(flags, get_order(c->size), node);
-		trace_kmem_cache_alloc_node(c->name, caller, b, c->object_size,
-					    PAGE_SIZE << get_order(c->size),
-					    flags, node);
+		trace_kmem_cache_alloc(c->name, caller, b, c->object_size,
+				       PAGE_SIZE << get_order(c->size),
+				       flags, node);
 	}
 
 	if (b && c->ctor) {
diff --git a/mm/slub.c b/mm/slub.c
index d53e9e22d67e..a088d4fa1062 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3215,8 +3215,8 @@ void *__kmem_cache_alloc_node(struct kmem_cache *s, struct list_lru *lru, gfp_t
 {
 	void *ret = slab_alloc_node(s, lru, gfpflags, node, caller, s->object_size);
 
-	trace_kmem_cache_alloc_node(s->name, caller, ret, s->object_size,
-				    s->size, gfpflags, node);
+	trace_kmem_cache_alloc(s->name, caller, ret, s->object_size,
+			       s->size, gfpflags, node);
 
 	return ret;
 }
@@ -3229,8 +3229,8 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *s,
 {
 	void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size);
 
-	trace_kmem_cache_alloc_node(s->name, _RET_IP_, ret,
-				    size, s->size, gfpflags, node);
+	trace_kmem_cache_alloc(s->name, _RET_IP_, ret,
+			       size, s->size, gfpflags, node);
 
 	ret = kasan_kmalloc(s, ret, size, gfpflags);
 	return ret;
@@ -4352,8 +4352,8 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
 
 	ret = slab_alloc_node(s, NULL, flags, node, _RET_IP_, size);
 
-	trace_kmem_cache_alloc_node(s->name, _RET_IP_, ret, size,
-				    s->size, flags, node);
+	trace_kmem_cache_alloc(s->name, _RET_IP_, ret, size,
+			       s->size, flags, node);
 
 	ret = kasan_kmalloc(s, ret, size, flags);
 
@@ -4815,8 +4815,8 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 	ret = slab_alloc_node(s, NULL, gfpflags, node, caller, size);
 
 	/* Honor the call site pointer we received. */
-	trace_kmem_cache_alloc_node(s->name, caller, ret, size,
-				    s->size, gfpflags, node);
+	trace_kmem_cache_alloc(s->name, caller, ret, size,
+			       s->size, gfpflags, node);
 
 	return ret;
 }
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 17/23] mm/slab_common: implement __kmem_cache_free()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (15 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 16/23] mm/slab_common: rename tracepoint Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-14  8:57 ` [PATCH v2 18/23] mm/sl[au]b: generalize kmalloc subsystem Hyeonggon Yoo
                   ` (6 subsequent siblings)
  23 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

To generalize kfree in later patch, implement __kmem_cache_free() that
takes caller address and make kmem_cache_free() wrapper of it.

Now that kmem_cache_free() is inline function, we should
use _THIS_IP_ instead of _RET_IP_ for consistency.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/linux/slab.h | 16 +++++++++++++++-
 mm/slab.c            | 17 +++++------------
 mm/slob.c            | 13 +++++++------
 mm/slub.c            |  9 +++++----
 4 files changed, 32 insertions(+), 23 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index c8c82087c3f9..0630c37ee630 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -462,7 +462,21 @@ void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru, gfp_t gfp
 	return __kmem_cache_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, _THIS_IP_);
 }
 
-void kmem_cache_free(struct kmem_cache *s, void *objp);
+void __kmem_cache_free(struct kmem_cache *s, void *objp, unsigned long caller __maybe_unused);
+
+/**
+ * kmem_cache_free - Deallocate an object
+ * @s: The cache the allocation was from.
+ * @objp: The previously allocated object.
+ *
+ * Free an object which was previously allocated from this
+ * cache.
+ */
+static __always_inline void kmem_cache_free(struct kmem_cache *s, void *objp)
+{
+	__kmem_cache_free(s, objp, _THIS_IP_);
+}
+
 
 /*
  * Bulk allocation and freeing operations. These are accelerated in an
diff --git a/mm/slab.c b/mm/slab.c
index 424168b96790..d35873da5572 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3579,30 +3579,23 @@ void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
 }
 #endif
 
-/**
- * kmem_cache_free - Deallocate an object
- * @cachep: The cache the allocation was from.
- * @objp: The previously allocated object.
- *
- * Free an object which was previously allocated from this
- * cache.
- */
-void kmem_cache_free(struct kmem_cache *cachep, void *objp)
+void __kmem_cache_free(struct kmem_cache *cachep, void *objp,
+		       unsigned long caller __maybe_unused)
 {
 	unsigned long flags;
 	cachep = cache_from_obj(cachep, objp);
 	if (!cachep)
 		return;
 
-	trace_kmem_cache_free(cachep->name, _RET_IP_, objp);
+	trace_kmem_cache_free(cachep->name, caller, objp);
 	local_irq_save(flags);
 	debug_check_no_locks_freed(objp, cachep->object_size);
 	if (!(cachep->flags & SLAB_DEBUG_OBJECTS))
 		debug_check_no_obj_freed(objp, cachep->object_size);
-	__cache_free(cachep, objp, _RET_IP_);
+	__cache_free(cachep, objp, caller);
 	local_irq_restore(flags);
 }
-EXPORT_SYMBOL(kmem_cache_free);
+EXPORT_SYMBOL(__kmem_cache_free);
 
 void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p)
 {
diff --git a/mm/slob.c b/mm/slob.c
index 1bb4c577b908..e893d182d471 100644
--- a/mm/slob.c
+++ b/mm/slob.c
@@ -631,7 +631,7 @@ void *__kmem_cache_alloc_node(struct kmem_cache *cachep, struct list_lru *lru __
 }
 EXPORT_SYMBOL(__kmem_cache_alloc_node);
 
-static void __kmem_cache_free(void *b, int size)
+static void ____kmem_cache_free(void *b, int size)
 {
 	if (size < PAGE_SIZE)
 		slob_free(b, size);
@@ -644,23 +644,24 @@ static void kmem_rcu_free(struct rcu_head *head)
 	struct slob_rcu *slob_rcu = (struct slob_rcu *)head;
 	void *b = (void *)slob_rcu - (slob_rcu->size - sizeof(struct slob_rcu));
 
-	__kmem_cache_free(b, slob_rcu->size);
+	____kmem_cache_free(b, slob_rcu->size);
 }
 
-void kmem_cache_free(struct kmem_cache *c, void *b)
+void __kmem_cache_free(struct kmem_cache *c, void *b,
+		       unsigned long caller __maybe_unused)
 {
 	kmemleak_free_recursive(b, c->flags);
-	trace_kmem_cache_free(c->name, _RET_IP_, b);
+	trace_kmem_cache_free(c->name, caller, b);
 	if (unlikely(c->flags & SLAB_TYPESAFE_BY_RCU)) {
 		struct slob_rcu *slob_rcu;
 		slob_rcu = b + (c->size - sizeof(struct slob_rcu));
 		slob_rcu->size = c->size;
 		call_rcu(&slob_rcu->head, kmem_rcu_free);
 	} else {
-		__kmem_cache_free(b, c->size);
+		____kmem_cache_free(b, c->size);
 	}
 }
-EXPORT_SYMBOL(kmem_cache_free);
+EXPORT_SYMBOL(__kmem_cache_free);
 
 void kmem_cache_free_bulk(struct kmem_cache *s, size_t size, void **p)
 {
diff --git a/mm/slub.c b/mm/slub.c
index a088d4fa1062..a72a2d08e793 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3466,15 +3466,16 @@ void ___cache_free(struct kmem_cache *cache, void *x, unsigned long addr)
 }
 #endif
 
-void kmem_cache_free(struct kmem_cache *s, void *x)
+void __kmem_cache_free(struct kmem_cache *s, void *x,
+		       unsigned long caller __maybe_unused)
 {
 	s = cache_from_obj(s, x);
 	if (!s)
 		return;
-	trace_kmem_cache_free(s->name, _RET_IP_, x);
-	slab_free(s, virt_to_slab(x), x, NULL, 1, _RET_IP_);
+	trace_kmem_cache_free(s->name, caller, x);
+	slab_free(s, virt_to_slab(x), x, NULL, 1, caller);
 }
-EXPORT_SYMBOL(kmem_cache_free);
+EXPORT_SYMBOL(__kmem_cache_free);
 
 struct detached_freelist {
 	struct slab *slab;
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 18/23] mm/sl[au]b: generalize kmalloc subsystem
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (16 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 17/23] mm/slab_common: implement __kmem_cache_free() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-29 14:30   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 19/23] mm/slab_common: add kasan_kmalloc() in __kmalloc_node_track_caller() Hyeonggon Yoo
                   ` (5 subsequent siblings)
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Now everything in kmalloc subsystem can be generalized.
Let's do it!

Generalize __kmalloc_node_track_caller(), kfree(), __ksize(),
__kmalloc_node() and move them to slab_common.c.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/slab.c        | 94 -----------------------------------------------
 mm/slab_common.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++++
 mm/slub.c        | 88 --------------------------------------------
 3 files changed, 95 insertions(+), 182 deletions(-)

diff --git a/mm/slab.c b/mm/slab.c
index d35873da5572..fc00aca62ae3 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3527,36 +3527,6 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
 EXPORT_SYMBOL(kmem_cache_alloc_node_trace);
 #endif
 
-static __always_inline void *
-__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller)
-{
-	struct kmem_cache *cachep;
-	void *ret;
-
-	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
-		return kmalloc_large_node(size, flags, node);
-	cachep = kmalloc_slab(size, flags);
-	if (unlikely(ZERO_OR_NULL_PTR(cachep)))
-		return cachep;
-	ret = kmem_cache_alloc_node_trace(cachep, flags, node, size);
-	ret = kasan_kmalloc(cachep, ret, size, flags);
-
-	return ret;
-}
-
-void *__kmalloc_node(size_t size, gfp_t flags, int node)
-{
-	return __do_kmalloc_node(size, flags, node, _RET_IP_);
-}
-EXPORT_SYMBOL(__kmalloc_node);
-
-void *__kmalloc_node_track_caller(size_t size, gfp_t flags,
-		int node, unsigned long caller)
-{
-	return __do_kmalloc_node(size, flags, node, caller);
-}
-EXPORT_SYMBOL(__kmalloc_node_track_caller);
-
 #ifdef CONFIG_PRINTK
 void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
 {
@@ -3635,43 +3605,6 @@ void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p)
 }
 EXPORT_SYMBOL(kmem_cache_free_bulk);
 
-/**
- * kfree - free previously allocated memory
- * @objp: pointer returned by kmalloc.
- *
- * If @objp is NULL, no operation is performed.
- *
- * Don't free memory not originally allocated by kmalloc()
- * or you will run into trouble.
- */
-void kfree(const void *objp)
-{
-	struct kmem_cache *c;
-	unsigned long flags;
-	struct folio *folio;
-
-
-	if (unlikely(ZERO_OR_NULL_PTR(objp)))
-		return;
-
-	folio = virt_to_folio(objp);
-	if (!folio_test_slab(folio)) {
-		free_large_kmalloc(folio, (void *)objp);
-		return;
-	}
-
-	c = folio_slab(folio)->slab_cache;
-	trace_kmem_cache_free(c->name, _RET_IP_, objp);
-
-	local_irq_save(flags);
-	kfree_debugcheck(objp);
-	debug_check_no_locks_freed(objp, c->object_size);
-	debug_check_no_obj_freed(objp, c->object_size);
-	__cache_free(c, (void *)objp, _RET_IP_);
-	local_irq_restore(flags);
-}
-EXPORT_SYMBOL(kfree);
-
 /*
  * This initializes kmem_cache_node or resizes various caches for all nodes.
  */
@@ -4074,30 +4007,3 @@ void __check_heap_object(const void *ptr, unsigned long n,
 	usercopy_abort("SLAB object", cachep->name, to_user, offset, n);
 }
 #endif /* CONFIG_HARDENED_USERCOPY */
-
-/**
- * __ksize -- Uninstrumented ksize.
- * @objp: pointer to the object
- *
- * Unlike ksize(), __ksize() is uninstrumented, and does not provide the same
- * safety checks as ksize() with KASAN instrumentation enabled.
- *
- * Return: size of the actual memory used by @objp in bytes
- */
-size_t __ksize(const void *objp)
-{
-	struct kmem_cache *c;
-	struct folio *folio;
-
-	BUG_ON(!objp);
-	if (unlikely(objp == ZERO_SIZE_PTR))
-		return 0;
-
-	folio = virt_to_folio(objp);
-	if (!folio_test_slab(folio))
-		return folio_size(folio);
-
-	c = folio_slab(folio)->slab_cache;
-	return c->object_size;
-}
-EXPORT_SYMBOL(__ksize);
diff --git a/mm/slab_common.c b/mm/slab_common.c
index 3cd5d7a47ec7..daf626e25e12 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -918,6 +918,101 @@ void free_large_kmalloc(struct folio *folio, void *object)
 			      -(PAGE_SIZE << order));
 	__free_pages(folio_page(folio, 0), order);
 }
+
+void *__kmalloc_node(size_t size, gfp_t flags, int node)
+{
+	struct kmem_cache *s;
+	void *ret;
+
+	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
+		return kmalloc_large_node(size, flags, node);
+
+	s = kmalloc_slab(size, flags);
+
+	if (unlikely(ZERO_OR_NULL_PTR(s)))
+		return s;
+
+	ret = __kmem_cache_alloc_node(s, NULL, flags, node, _RET_IP_);
+	ret = kasan_kmalloc(s, ret, size, flags);
+
+	return ret;
+}
+EXPORT_SYMBOL(__kmalloc_node);
+
+void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
+					int node, unsigned long caller)
+{
+	struct kmem_cache *s;
+	void *ret;
+
+	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
+		return kmalloc_large_node(size, gfpflags, node);
+
+	s = kmalloc_slab(size, gfpflags);
+
+	if (unlikely(ZERO_OR_NULL_PTR(s)))
+		return s;
+
+	ret = __kmem_cache_alloc_node(s, NULL, gfpflags, node, caller);
+
+	return ret;
+}
+EXPORT_SYMBOL(__kmalloc_node_track_caller);
+
+/**
+ * kfree - free previously allocated memory
+ * @objp: pointer returned by kmalloc.
+ *
+ * If @objp is NULL, no operation is performed.
+ *
+ * Don't free memory not originally allocated by kmalloc()
+ * or you will run into trouble.
+ */
+void kfree(const void *object)
+{
+	struct folio *folio;
+	struct slab *slab;
+	struct kmem_cache *s;
+
+	if (unlikely(ZERO_OR_NULL_PTR(object)))
+		return;
+
+	folio = virt_to_folio(object);
+	if (unlikely(!folio_test_slab(folio))) {
+		free_large_kmalloc(folio, (void *)object);
+		return;
+	}
+
+	slab = folio_slab(folio);
+	s = slab->slab_cache;
+	__kmem_cache_free(s, object, _RET_IP_);
+}
+EXPORT_SYMBOL(kfree);
+
+/**
+ * __ksize -- Uninstrumented ksize.
+ * @objp: pointer to the object
+ *
+ * Unlike ksize(), __ksize() is uninstrumented, and does not provide the same
+ * safety checks as ksize() with KASAN instrumentation enabled.
+ *
+ * Return: size of the actual memory used by @objp in bytes
+ */
+size_t __ksize(const void *object)
+{
+	struct folio *folio;
+
+	if (unlikely(object == ZERO_SIZE_PTR))
+		return 0;
+
+	folio = virt_to_folio(object);
+
+	if (unlikely(!folio_test_slab(folio)))
+		return folio_size(folio);
+
+	return slab_ksize(folio_slab(folio)->slab_cache);
+}
+EXPORT_SYMBOL(__ksize);
 #endif /* !CONFIG_SLOB */
 
 gfp_t kmalloc_fix_flags(gfp_t flags)
diff --git a/mm/slub.c b/mm/slub.c
index a72a2d08e793..bc9c96ce0521 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -4338,30 +4338,6 @@ static int __init setup_slub_min_objects(char *str)
 
 __setup("slub_min_objects=", setup_slub_min_objects);
 
-void *__kmalloc_node(size_t size, gfp_t flags, int node)
-{
-	struct kmem_cache *s;
-	void *ret;
-
-	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
-		return kmalloc_large_node(size, flags, node);
-
-	s = kmalloc_slab(size, flags);
-
-	if (unlikely(ZERO_OR_NULL_PTR(s)))
-		return s;
-
-	ret = slab_alloc_node(s, NULL, flags, node, _RET_IP_, size);
-
-	trace_kmem_cache_alloc(s->name, _RET_IP_, ret, size,
-			       s->size, flags, node);
-
-	ret = kasan_kmalloc(s, ret, size, flags);
-
-	return ret;
-}
-EXPORT_SYMBOL(__kmalloc_node);
-
 #ifdef CONFIG_HARDENED_USERCOPY
 /*
  * Rejects incorrectly sized objects and objects that are to be copied
@@ -4412,46 +4388,6 @@ void __check_heap_object(const void *ptr, unsigned long n,
 }
 #endif /* CONFIG_HARDENED_USERCOPY */
 
-size_t __ksize(const void *object)
-{
-	struct folio *folio;
-
-	if (unlikely(object == ZERO_SIZE_PTR))
-		return 0;
-
-	folio = virt_to_folio(object);
-
-	if (unlikely(!folio_test_slab(folio)))
-		return folio_size(folio);
-
-	return slab_ksize(folio_slab(folio)->slab_cache);
-}
-EXPORT_SYMBOL(__ksize);
-
-void kfree(const void *x)
-{
-	struct folio *folio;
-	struct slab *slab;
-	void *object = (void *)x;
-	struct kmem_cache *s;
-
-	if (unlikely(ZERO_OR_NULL_PTR(x)))
-		return;
-
-	folio = virt_to_folio(x);
-	if (unlikely(!folio_test_slab(folio))) {
-		free_large_kmalloc(folio, object);
-		return;
-	}
-
-	slab = folio_slab(folio);
-	s = slab->slab_cache;
-
-	trace_kmem_cache_free(s->name, _RET_IP_, x);
-	slab_free(s, slab, object, NULL, 1, _RET_IP_);
-}
-EXPORT_SYMBOL(kfree);
-
 #define SHRINK_PROMOTE_MAX 32
 
 /*
@@ -4799,30 +4735,6 @@ int __kmem_cache_create(struct kmem_cache *s, slab_flags_t flags)
 	return 0;
 }
 
-void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
-					int node, unsigned long caller)
-{
-	struct kmem_cache *s;
-	void *ret;
-
-	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
-		return kmalloc_large_node(size, gfpflags, node);
-
-	s = kmalloc_slab(size, gfpflags);
-
-	if (unlikely(ZERO_OR_NULL_PTR(s)))
-		return s;
-
-	ret = slab_alloc_node(s, NULL, gfpflags, node, caller, size);
-
-	/* Honor the call site pointer we received. */
-	trace_kmem_cache_alloc(s->name, caller, ret, size,
-			       s->size, gfpflags, node);
-
-	return ret;
-}
-EXPORT_SYMBOL(__kmalloc_node_track_caller);
-
 #ifdef CONFIG_SYSFS
 static int count_inuse(struct slab *slab)
 {
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 19/23] mm/slab_common: add kasan_kmalloc() in __kmalloc_node_track_caller()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (17 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 18/23] mm/sl[au]b: generalize kmalloc subsystem Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-14  8:57 ` [PATCH v2 20/23] mm/slab_common: factor out __do_kmalloc_node() Hyeonggon Yoo
                   ` (4 subsequent siblings)
  23 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Add missing kasan_kmalloc() in __kmalloc_node_track_caller().

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/slab_common.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/slab_common.c b/mm/slab_common.c
index a3eb66e5180c..6abe7f61c197 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -954,6 +954,7 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 		return s;
 
 	ret = __kmem_cache_alloc_node(s, NULL, gfpflags, node, caller);
+	ret = kasan_kmalloc(s, ret, size, gfpflags);
 
 	return ret;
 }
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 20/23] mm/slab_common: factor out __do_kmalloc_node()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (18 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 19/23] mm/slab_common: add kasan_kmalloc() in __kmalloc_node_track_caller() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-14 11:45   ` Hyeonggon Yoo
  2022-04-29 14:48   ` Vlastimil Babka
  2022-04-14  8:57 ` [PATCH v2 21/23] mm/sl[au]b: remove kmem_cache_alloc_node_trace() Hyeonggon Yoo
                   ` (3 subsequent siblings)
  23 siblings, 2 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

Factor out common code into __do_kmalloc_node().

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/slab_common.c | 27 ++++++++++-----------------
 1 file changed, 10 insertions(+), 17 deletions(-)

diff --git a/mm/slab_common.c b/mm/slab_common.c
index 6abe7f61c197..af563e64e8aa 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -919,7 +919,9 @@ void free_large_kmalloc(struct folio *folio, void *object)
 	__free_pages(folio_page(folio, 0), order);
 }
 
-void *__kmalloc_node(size_t size, gfp_t flags, int node)
+static __always_inline
+void *__do_kmalloc_node(size_t size, gfp_t flags, int node,
+			unsigned long caller __maybe_unused)
 {
 	struct kmem_cache *s;
 	void *ret;
@@ -932,31 +934,22 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
 	if (unlikely(ZERO_OR_NULL_PTR(s)))
 		return s;
 
-	ret = __kmem_cache_alloc_node(s, NULL, flags, node, _RET_IP_);
+	ret = __kmem_cache_alloc_node(s, NULL, flags, node, caller);
 	ret = kasan_kmalloc(s, ret, size, flags);
 
 	return ret;
 }
+
+void *__kmalloc_node(size_t size, gfp_t flags, int node)
+{
+	return __do_kmalloc_node(size, flags, node, _RET_IP_);
+}
 EXPORT_SYMBOL(__kmalloc_node);
 
 void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
 					int node, unsigned long caller)
 {
-	struct kmem_cache *s;
-	void *ret;
-
-	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
-		return kmalloc_large_node(size, gfpflags, node);
-
-	s = kmalloc_slab(size, gfpflags);
-
-	if (unlikely(ZERO_OR_NULL_PTR(s)))
-		return s;
-
-	ret = __kmem_cache_alloc_node(s, NULL, gfpflags, node, caller);
-	ret = kasan_kmalloc(s, ret, size, gfpflags);
-
-	return ret;
+	return __do_kmalloc_node(size, flags, node, caller);
 }
 EXPORT_SYMBOL(__kmalloc_node_track_caller);
 
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 21/23] mm/sl[au]b: remove kmem_cache_alloc_node_trace()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (19 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 20/23] mm/slab_common: factor out __do_kmalloc_node() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-14  8:57 ` [PATCH v2 22/23] mm/sl[auo]b: move definition of __ksize() to mm/slab.h Hyeonggon Yoo
                   ` (2 subsequent siblings)
  23 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

kmem_cache_alloc_node_trace() was introduced by commit 4a92379bdfb4
("slub tracing: move trace calls out of always inlined functions to reduce
kernel code size") to avoid inlining tracepoints for inlined kmalloc
function calls.

Now that we use same tracepoint in kmalloc and normal caches,
kmem_cache_alloc_node_trace() can be replaced with
__kmem_cache_alloc_node() and kasan_kmalloc().

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 include/linux/slab.h | 26 ++++++++------------------
 mm/slab.c            | 19 -------------------
 mm/slub.c            | 16 ----------------
 3 files changed, 8 insertions(+), 53 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index 0630c37ee630..c1aed9d97cf2 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -497,21 +497,6 @@ static __always_inline void kfree_bulk(size_t size, void **p)
 	kmem_cache_free_bulk(NULL, size, p);
 }
 
-#ifdef CONFIG_TRACING
-extern void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags,
-					 int node, size_t size) __assume_slab_alignment
-								__alloc_size(4);
-#else /* CONFIG_TRACING */
-static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags,
-							 int node, size_t size)
-{
-	void *ret = kmem_cache_alloc_node(s, gfpflags, node);
-
-	ret = kasan_kmalloc(s, ret, size, gfpflags);
-	return ret;
-}
-#endif /* CONFIG_TRACING */
-
 extern void *kmalloc_large_node(size_t size, gfp_t flags, int node)
 				__assume_page_alignment __alloc_size(1);
 
@@ -523,6 +508,9 @@ static __always_inline void *kmalloc_large(size_t size, gfp_t flags)
 #ifndef CONFIG_SLOB
 static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
 {
+	struct kmem_cache *s;
+	void *objp;
+
 	if (__builtin_constant_p(size)) {
 		unsigned int index;
 
@@ -534,9 +522,11 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla
 		if (!index)
 			return ZERO_SIZE_PTR;
 
-		return kmem_cache_alloc_node_trace(
-				kmalloc_caches[kmalloc_type(flags)][index],
-						flags, node, size);
+		s = kmalloc_caches[kmalloc_type(flags)][index];
+
+		objp = __kmem_cache_alloc_node(s, NULL, flags, node, _RET_IP_);
+		objp = kasan_kmalloc(s, objp, size, flags);
+		return objp;
 	}
 	return __kmalloc_node(size, flags, node);
 }
diff --git a/mm/slab.c b/mm/slab.c
index fc00aca62ae3..24010e72f603 100644
--- a/mm/slab.c
+++ b/mm/slab.c
@@ -3508,25 +3508,6 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
 }
 EXPORT_SYMBOL(kmem_cache_alloc_bulk);
 
-#ifdef CONFIG_TRACING
-void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
-				  gfp_t flags,
-				  int nodeid,
-				  size_t size)
-{
-	void *ret;
-
-	ret = slab_alloc_node(cachep, NULL, flags, nodeid, size, _RET_IP_);
-
-	ret = kasan_kmalloc(cachep, ret, size, flags);
-	trace_kmem_cache_alloc(cachep->name, _RET_IP_, ret,
-			       size, cachep->size,
-			       flags, nodeid);
-	return ret;
-}
-EXPORT_SYMBOL(kmem_cache_alloc_node_trace);
-#endif
-
 #ifdef CONFIG_PRINTK
 void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
 {
diff --git a/mm/slub.c b/mm/slub.c
index bc9c96ce0521..1899c7e1de10 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -3222,22 +3222,6 @@ void *__kmem_cache_alloc_node(struct kmem_cache *s, struct list_lru *lru, gfp_t
 }
 EXPORT_SYMBOL(__kmem_cache_alloc_node);
 
-#ifdef CONFIG_TRACING
-void *kmem_cache_alloc_node_trace(struct kmem_cache *s,
-				    gfp_t gfpflags,
-				    int node, size_t size)
-{
-	void *ret = slab_alloc_node(s, NULL, gfpflags, node, _RET_IP_, size);
-
-	trace_kmem_cache_alloc(s->name, _RET_IP_, ret,
-			       size, s->size, gfpflags, node);
-
-	ret = kasan_kmalloc(s, ret, size, gfpflags);
-	return ret;
-}
-EXPORT_SYMBOL(kmem_cache_alloc_node_trace);
-#endif
-
 /*
  * Slow path handling. This may still be called frequently since objects
  * have a longer lifetime than the cpu slabs in most processing loads.
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 22/23] mm/sl[auo]b: move definition of __ksize() to mm/slab.h
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (20 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 21/23] mm/sl[au]b: remove kmem_cache_alloc_node_trace() Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-14  8:57 ` [PATCH v2 23/23] mm/sl[au]b: check if large object is valid in __ksize() Hyeonggon Yoo
  2022-04-14 12:36 ` [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
  23 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

__ksize() is only called by KASAN. Remove export symbol and move
definition to mm/slab.h as we don't want to grow its callers.

Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
---
 include/linux/slab.h |  1 -
 mm/slab.h            |  2 ++
 mm/slab_common.c     | 11 +----------
 mm/slob.c            |  1 -
 4 files changed, 3 insertions(+), 12 deletions(-)

diff --git a/include/linux/slab.h b/include/linux/slab.h
index c1aed9d97cf2..e30c0675d6b2 100644
--- a/include/linux/slab.h
+++ b/include/linux/slab.h
@@ -187,7 +187,6 @@ int kmem_cache_shrink(struct kmem_cache *s);
 void * __must_check krealloc(const void *objp, size_t new_size, gfp_t flags) __alloc_size(2);
 void kfree(const void *objp);
 void kfree_sensitive(const void *objp);
-size_t __ksize(const void *objp);
 size_t ksize(const void *objp);
 #ifdef CONFIG_PRINTK
 bool kmem_valid_obj(void *object);
diff --git a/mm/slab.h b/mm/slab.h
index 45ddb19df319..5a500894317b 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -690,6 +690,8 @@ void free_large_kmalloc(struct folio *folio, void *object);
 
 #endif /* CONFIG_SLOB */
 
+size_t __ksize(const void *objp);
+
 static inline size_t slab_ksize(const struct kmem_cache *s)
 {
 #ifndef CONFIG_SLUB
diff --git a/mm/slab_common.c b/mm/slab_common.c
index af563e64e8aa..8facade42bdd 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -984,15 +984,7 @@ void kfree(const void *x)
 }
 EXPORT_SYMBOL(kfree);
 
-/**
- * __ksize -- Uninstrumented ksize.
- * @objp: pointer to the object
- *
- * Unlike ksize(), __ksize() is uninstrumented, and does not provide the same
- * safety checks as ksize() with KASAN instrumentation enabled.
- *
- * Return: size of the actual memory used by @objp in bytes
- */
+/* Uninstrumented ksize. Only called by KASAN. */
 size_t __ksize(const void *object)
 {
 	struct folio *folio;
@@ -1007,7 +999,6 @@ size_t __ksize(const void *object)
 
 	return slab_ksize(folio_slab(folio)->slab_cache);
 }
-EXPORT_SYMBOL(__ksize);
 #endif /* !CONFIG_SLOB */
 
 gfp_t kmalloc_fix_flags(gfp_t flags)
diff --git a/mm/slob.c b/mm/slob.c
index e893d182d471..adf794d58eb5 100644
--- a/mm/slob.c
+++ b/mm/slob.c
@@ -576,7 +576,6 @@ size_t __ksize(const void *block)
 	m = (unsigned int *)(block - align);
 	return SLOB_UNITS(*m) * SLOB_UNIT;
 }
-EXPORT_SYMBOL(__ksize);
 
 int __kmem_cache_create(struct kmem_cache *c, slab_flags_t flags)
 {
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* [PATCH v2 23/23] mm/sl[au]b: check if large object is valid in __ksize()
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (21 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 22/23] mm/sl[auo]b: move definition of __ksize() to mm/slab.h Hyeonggon Yoo
@ 2022-04-14  8:57 ` Hyeonggon Yoo
  2022-04-14  9:58   ` Christoph Lameter
  2022-04-14 12:36 ` [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
  23 siblings, 1 reply; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14  8:57 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Hyeonggon Yoo,
	Roman Gushchin, linux-mm, linux-kernel

__ksize() returns size of objects allocated from slab allocator.
When invalid object is passed to __ksize(), returning zero
prevents further memory corruption and makes caller be able to
check if there is an error.

If address of large object is not beginning of folio or size of
the folio is too small, it must be invalid. Return zero in such cases.

Suggested-by: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
---
 mm/slab_common.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/mm/slab_common.c b/mm/slab_common.c
index 8facade42bdd..a14f9990b159 100644
--- a/mm/slab_common.c
+++ b/mm/slab_common.c
@@ -994,8 +994,12 @@ size_t __ksize(const void *object)
 
 	folio = virt_to_folio(object);
 
-	if (unlikely(!folio_test_slab(folio)))
+	if (unlikely(!folio_test_slab(folio))) {
+		if (object != folio_address(folio) ||
+				folio_size(folio) <= KMALLOC_MAX_CACHE_SIZE)
+			return 0;
 		return folio_size(folio);
+	}
 
 	return slab_ksize(folio_slab(folio)->slab_cache);
 }
-- 
2.32.0


^ permalink raw reply related	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 23/23] mm/sl[au]b: check if large object is valid in __ksize()
  2022-04-14  8:57 ` [PATCH v2 23/23] mm/sl[au]b: check if large object is valid in __ksize() Hyeonggon Yoo
@ 2022-04-14  9:58   ` Christoph Lameter
  2022-04-14 11:46     ` Hyeonggon Yoo
  0 siblings, 1 reply; 51+ messages in thread
From: Christoph Lameter @ 2022-04-14  9:58 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Vlastimil Babka, Marco Elver, Matthew WilCox, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On Thu, 14 Apr 2022, Hyeonggon Yoo wrote:

> If address of large object is not beginning of folio or size of
> the folio is too small, it must be invalid. Return zero in such cases.

This looks like a slab usage error on the part of the caller that would
benefit from a kernel log entry. A warning or an error?

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 20/23] mm/slab_common: factor out __do_kmalloc_node()
  2022-04-14  8:57 ` [PATCH v2 20/23] mm/slab_common: factor out __do_kmalloc_node() Hyeonggon Yoo
@ 2022-04-14 11:45   ` Hyeonggon Yoo
  2022-04-29 14:48   ` Vlastimil Babka
  1 sibling, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14 11:45 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On Thu, Apr 14, 2022 at 05:57:24PM +0900, Hyeonggon Yoo wrote:
> Factor out common code into __do_kmalloc_node().
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> ---
>  mm/slab_common.c | 27 ++++++++++-----------------
>  1 file changed, 10 insertions(+), 17 deletions(-)
> 
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 6abe7f61c197..af563e64e8aa 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -919,7 +919,9 @@ void free_large_kmalloc(struct folio *folio, void *object)
>  	__free_pages(folio_page(folio, 0), order);
>  }
>  
> -void *__kmalloc_node(size_t size, gfp_t flags, int node)
> +static __always_inline
> +void *__do_kmalloc_node(size_t size, gfp_t flags, int node,
> +			unsigned long caller __maybe_unused)
>  {
>  	struct kmem_cache *s;
>  	void *ret;
> @@ -932,31 +934,22 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
>  	if (unlikely(ZERO_OR_NULL_PTR(s)))
>  		return s;
>  
> -	ret = __kmem_cache_alloc_node(s, NULL, flags, node, _RET_IP_);
> +	ret = __kmem_cache_alloc_node(s, NULL, flags, node, caller);
>  	ret = kasan_kmalloc(s, ret, size, flags);
>  
>  	return ret;
>  }
> +
> +void *__kmalloc_node(size_t size, gfp_t flags, int node)
> +{
> +	return __do_kmalloc_node(size, flags, node, _RET_IP_);
> +}
>  EXPORT_SYMBOL(__kmalloc_node);
>  
>  void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
>  					int node, unsigned long caller)
>  {
> -	struct kmem_cache *s;
> -	void *ret;
> -
> -	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
> -		return kmalloc_large_node(size, gfpflags, node);
> -
> -	s = kmalloc_slab(size, gfpflags);
> -
> -	if (unlikely(ZERO_OR_NULL_PTR(s)))
> -		return s;
> -
> -	ret = __kmem_cache_alloc_node(s, NULL, gfpflags, node, caller);
> -	ret = kasan_kmalloc(s, ret, size, gfpflags);
> -
> -	return ret;
> +	return __do_kmalloc_node(size, flags, node, caller);

Ah, 'flags' is undefined here. (it's gfpflags here)
Seems like I sent a version of this patch before fixing build error.

I'll send updated version when sending v3 :)


>  }
>  EXPORT_SYMBOL(__kmalloc_node_track_caller);
>  
> -- 
> 2.32.0
> 

-- 
Thanks,
Hyeonggon

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 23/23] mm/sl[au]b: check if large object is valid in __ksize()
  2022-04-14  9:58   ` Christoph Lameter
@ 2022-04-14 11:46     ` Hyeonggon Yoo
  0 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14 11:46 UTC (permalink / raw)
  To: Christoph Lameter
  Cc: Vlastimil Babka, Marco Elver, Matthew WilCox, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On Thu, Apr 14, 2022 at 11:58:01AM +0200, Christoph Lameter wrote:
> On Thu, 14 Apr 2022, Hyeonggon Yoo wrote:
> 
> > If address of large object is not beginning of folio or size of
> > the folio is too small, it must be invalid. Return zero in such cases.
> 
> This looks like a slab usage error on the part of the caller that would
> benefit from a kernel log entry. A warning or an error?


I think a WARN_ON() would be sufficient. here, will do in v3.
Thanks!

-- 
Thanks,
Hyeonggon

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2
  2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
                   ` (22 preceding siblings ...)
  2022-04-14  8:57 ` [PATCH v2 23/23] mm/sl[au]b: check if large object is valid in __ksize() Hyeonggon Yoo
@ 2022-04-14 12:36 ` Hyeonggon Yoo
  23 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-14 12:36 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On Thu, Apr 14, 2022 at 05:57:04PM +0900, Hyeonggon Yoo wrote:
> Changes since v1:
> - The code is basically same, but split some patches to make it
> easier to review.
> - Adjusted comments and added Reviewed-by from Vlastimil. Thanks!
> 
> Hello, this series is cleanup of kmalloc code. After this series,
> kmalloc subsystem is perfectly generalized on SLAB and SLUB.
> 
> This series is not small, but each patch is easy to review and
> I bet you will like this :)
> 
> Many thanks to Matthew, Marco, Vlastimil who gave comments
> in previous series.
> 
> Any feedbacks will be appreciated.
> Thanks!

I mistakenly sent a wrong version of series, before fixing a build bug
and a build warning. The latest version is available at:

https://github.com/hygoni/linux/tree/slab-common-v5.19-v1r1

And below is difference between the series in this thread
and tree above is:

diff --git a/0018-mm-sl-au-b-generalize-kmalloc-subsystem.patch b/0018-mm-sl-au-b-generalize-kmalloc-subsystem-v2.patch
index 2389975..510f7fd 100644
--- a/0018-mm-sl-au-b-generalize-kmalloc-subsystem.patch
+++ b/0018-mm-sl-au-b-generalize-kmalloc-subsystem-v2.patch
@@ -207,7 +207,7 @@ index 3cd5d7a47ec7..daf626e25e12 100644
 +
 +      slab = folio_slab(folio);
 +      s = slab->slab_cache;
-+      __kmem_cache_free(s, object, _RET_IP_);
++      __kmem_cache_free(s, (void *)object, _RET_IP_);
 +}
 +EXPORT_SYMBOL(kfree);
 +

Fixed a warning here

diff --git a/0020-mm-slab_common-factor-out-__do_kmalloc_node.patch b/0020-mm-slab_common-factor-out-__do_kmalloc_node-v2.patch
index 133d293..30d9ca2 100644
--- a/0020-mm-slab_common-factor-out-__do_kmalloc_node.patch
+++ b/0020-mm-slab_common-factor-out-__do_kmalloc_node-v2.patch
@@ -60,7 +60,7 @@ index 6abe7f61c197..af563e64e8aa 100644
 -      ret = kasan_kmalloc(s, ret, size, gfpflags);
 -
 -      return ret;
-+      return __do_kmalloc_node(size, flags, node, caller);
++      return __do_kmalloc_node(size, gfpflags, node, caller);
  }
  EXPORT_SYMBOL(__kmalloc_node_track_caller);

And a build bug here.

Anything else looks just fine.
Thanks!

-- 
Thanks,
Hyeonggon

^ permalink raw reply related	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 01/23] mm/slab: move NUMA-related code to __do_cache_alloc()
  2022-04-14  8:57 ` [PATCH v2 01/23] mm/slab: move NUMA-related code to __do_cache_alloc() Hyeonggon Yoo
@ 2022-04-22 18:04   ` Vlastimil Babka
  0 siblings, 0 replies; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-22 18:04 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> To implement slab_alloc_node() independent of NUMA configuration,
> move NUMA fallback/alternate allocation code into __do_cache_alloc().
> 
> One functional change here is not to check availability of node
> when allocating from local node.
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>

Thanks.

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 02/23] mm/slab: cleanup slab_alloc() and slab_alloc_node()
  2022-04-14  8:57 ` [PATCH v2 02/23] mm/slab: cleanup slab_alloc() and slab_alloc_node() Hyeonggon Yoo
@ 2022-04-25 14:05   ` Vlastimil Babka
  0 siblings, 0 replies; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-25 14:05 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Make slab_alloc_node() available even when CONFIG_NUMA=n and
> make slab_alloc() wrapper of slab_alloc_node().
> 
> This is necessary for further cleanup.

It also removes a lot of nearly-duplicite code.

> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 03/23] mm/slab_common: remove CONFIG_NUMA ifdefs for common kmalloc functions
  2022-04-14  8:57 ` [PATCH v2 03/23] mm/slab_common: remove CONFIG_NUMA ifdefs for common kmalloc functions Hyeonggon Yoo
@ 2022-04-25 14:41   ` Vlastimil Babka
  0 siblings, 0 replies; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-25 14:41 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Now that slab_alloc_node() is available for SLAB when CONFIG_NUMA=n,
> remove CONFIG_NUMA ifdefs for common kmalloc functions.
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>


^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 04/23] mm/slab_common: cleanup kmalloc_track_caller()
  2022-04-14  8:57 ` [PATCH v2 04/23] mm/slab_common: cleanup kmalloc_track_caller() Hyeonggon Yoo
@ 2022-04-25 15:05   ` Vlastimil Babka
  2022-04-26 15:49   ` Vlastimil Babka
  1 sibling, 0 replies; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-25 15:05 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Make kmalloc_track_caller() wrapper of kmalloc_node_track_caller().

Looks like there's an immediate benefit of SLUB not throwing away the caller
pointer anymore in case of kmalloc_large().

> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>


^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 04/23] mm/slab_common: cleanup kmalloc_track_caller()
  2022-04-14  8:57 ` [PATCH v2 04/23] mm/slab_common: cleanup kmalloc_track_caller() Hyeonggon Yoo
  2022-04-25 15:05   ` Vlastimil Babka
@ 2022-04-26 15:49   ` Vlastimil Babka
  2022-04-30 11:44     ` Hyeonggon Yoo
  1 sibling, 1 reply; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-26 15:49 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Make kmalloc_track_caller() wrapper of kmalloc_node_track_caller().
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> ---
>  include/linux/slab.h | 17 ++++++++---------
>  mm/slab.c            |  6 ------
>  mm/slob.c            |  6 ------
>  mm/slub.c            | 22 ----------------------
>  4 files changed, 8 insertions(+), 43 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index a3b9d4c20d7e..acdb4b7428f9 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -639,6 +639,12 @@ static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flag
>  	return kmalloc_array(n, size, flags | __GFP_ZERO);
>  }
>  
> +extern void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node,
> +					 unsigned long caller) __alloc_size(1);

Update: in v3 remove the extern, it's not necessary and the kernel is slowly
getting rid of them.

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 05/23] mm/slab_common: cleanup __kmalloc()
  2022-04-14  8:57 ` [PATCH v2 05/23] mm/slab_common: cleanup __kmalloc() Hyeonggon Yoo
@ 2022-04-26 16:02   ` Vlastimil Babka
  0 siblings, 0 replies; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-26 16:02 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Make __kmalloc() wrapper of __kmalloc_node().

Again/similarly, looks like this will make SLUB not miss trace for
kmalloc_large() anymore.

> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>

Nit below:

> ---
>  include/linux/slab.h | 13 ++++++++++---
>  mm/slab.c            | 34 ----------------------------------
>  mm/slob.c            |  6 ------
>  mm/slub.c            | 23 -----------------------
>  4 files changed, 10 insertions(+), 66 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index acdb4b7428f9..4c06d15f731c 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -419,7 +419,16 @@ static __always_inline unsigned int __kmalloc_index(size_t size,
>  #define kmalloc_index(s) __kmalloc_index(s, true)
>  #endif /* !CONFIG_SLOB */
>  
> -void *__kmalloc(size_t size, gfp_t flags) __assume_kmalloc_alignment __alloc_size(1);
> +extern void *__kmalloc_node(size_t size, gfp_t flags, int node)
> +			    __assume_kmalloc_alignment
> +			    __alloc_size(1);
> +

Again, no 'extern' please.

> +static __always_inline __alloc_size(1) __assume_kmalloc_alignment
> +void *__kmalloc(size_t size, gfp_t flags)
> +{
> +	return __kmalloc_node(size, flags, NUMA_NO_NODE);
> +}
> +
>  void *kmem_cache_alloc(struct kmem_cache *s, gfp_t flags) __assume_slab_alignment __malloc;
>  void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
>  			   gfp_t gfpflags) __assume_slab_alignment __malloc;

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 06/23] mm/sl[auo]b: fold kmalloc_order_trace() into kmalloc_large()
  2022-04-14  8:57 ` [PATCH v2 06/23] mm/sl[auo]b: fold kmalloc_order_trace() into kmalloc_large() Hyeonggon Yoo
@ 2022-04-26 16:09   ` Vlastimil Babka
  0 siblings, 0 replies; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-26 16:09 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> There is no caller of kmalloc_order_trace() except kmalloc_large().
> Fold it into kmalloc_large() and remove kmalloc_order{,_trace}().
> 
> Also add tracepoint in kmalloc_large() that was previously
> in kmalloc_order_trace().
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
> ---
> Changes from v1:
> - updated some changelog (kmalloc_order() -> kmalloc_order_trace())
> 
>  include/linux/slab.h | 22 ++--------------------
>  mm/slab_common.c     | 14 +++-----------
>  2 files changed, 5 insertions(+), 31 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 4c06d15f731c..6f6e22959b39 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -484,26 +484,8 @@ static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, g
>  }
>  #endif /* CONFIG_TRACING */
>  
> -extern void *kmalloc_order(size_t size, gfp_t flags, unsigned int order) __assume_page_alignment
> -									 __alloc_size(1);
> -
> -#ifdef CONFIG_TRACING
> -extern void *kmalloc_order_trace(size_t size, gfp_t flags, unsigned int order)
> -				__assume_page_alignment __alloc_size(1);
> -#else
> -static __always_inline __alloc_size(1) void *kmalloc_order_trace(size_t size, gfp_t flags,
> -								 unsigned int order)
> -{
> -	return kmalloc_order(size, flags, order);
> -}
> -#endif
> -
> -static __always_inline __alloc_size(1) void *kmalloc_large(size_t size, gfp_t flags)
> -{
> -	unsigned int order = get_order(size);
> -	return kmalloc_order_trace(size, flags, order);
> -}
> -
> +extern void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment
> +						     __alloc_size(1);

I missed this extern in v1, please drop.

>  /**
>   * kmalloc - allocate memory
>   * @size: how many bytes of memory are required.
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index c4d63f2c78b8..308cd5449285 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -925,10 +925,11 @@ gfp_t kmalloc_fix_flags(gfp_t flags)
>   * directly to the page allocator. We use __GFP_COMP, because we will need to
>   * know the allocation order to free the pages properly in kfree.
>   */
> -void *kmalloc_order(size_t size, gfp_t flags, unsigned int order)
> +void *kmalloc_large(size_t size, gfp_t flags)
>  {
>  	void *ret = NULL;
>  	struct page *page;
> +	unsigned int order = get_order(size);
>  
>  	if (unlikely(flags & GFP_SLAB_BUG_MASK))
>  		flags = kmalloc_fix_flags(flags);
> @@ -943,19 +944,10 @@ void *kmalloc_order(size_t size, gfp_t flags, unsigned int order)
>  	ret = kasan_kmalloc_large(ret, size, flags);
>  	/* As ret might get tagged, call kmemleak hook after KASAN. */
>  	kmemleak_alloc(ret, size, 1, flags);
> -	return ret;
> -}
> -EXPORT_SYMBOL(kmalloc_order);
> -
> -#ifdef CONFIG_TRACING
> -void *kmalloc_order_trace(size_t size, gfp_t flags, unsigned int order)
> -{
> -	void *ret = kmalloc_order(size, flags, order);
>  	trace_kmalloc(_RET_IP_, ret, size, PAGE_SIZE << order, flags);
>  	return ret;
>  }
> -EXPORT_SYMBOL(kmalloc_order_trace);
> -#endif
> +EXPORT_SYMBOL(kmalloc_large);
>  
>  #ifdef CONFIG_SLAB_FREELIST_RANDOM
>  /* Randomize a generic freelist */


^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 07/23] mm/slub: move kmalloc_large_node() to slab_common.c
  2022-04-14  8:57 ` [PATCH v2 07/23] mm/slub: move kmalloc_large_node() to slab_common.c Hyeonggon Yoo
@ 2022-04-26 16:13   ` Vlastimil Babka
  0 siblings, 0 replies; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-26 16:13 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> In later patch SLAB will also pass requests larger than order-1 page
> to page allocator. Move kmalloc_large_node() to slab_common.c.
> 
> Fold kmalloc_large_node_hook() into kmalloc_large_node() as there is
> no other caller.
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>

> ---
>  include/linux/slab.h |  3 +++
>  mm/slab_common.c     | 22 ++++++++++++++++++++++
>  mm/slub.c            | 25 -------------------------
>  3 files changed, 25 insertions(+), 25 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 6f6e22959b39..97336acbebbf 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -486,6 +486,9 @@ static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, g
>  
>  extern void *kmalloc_large(size_t size, gfp_t flags) __assume_page_alignment
>  						     __alloc_size(1);
> +
> +extern void *kmalloc_large_node(size_t size, gfp_t flags, int node)
> +				__assume_page_alignment __alloc_size(1);

The usual :)


^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 08/23] mm/slab_common: make kmalloc_large_node() consistent with kmalloc_large()
  2022-04-14  8:57 ` [PATCH v2 08/23] mm/slab_common: make kmalloc_large_node() consistent with kmalloc_large() Hyeonggon Yoo
@ 2022-04-26 17:15   ` Vlastimil Babka
  2022-04-28  6:35     ` Hyeonggon Yoo
  0 siblings, 1 reply; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-26 17:15 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Move tracepoints into kmalloc_large_node() and add missing flag fix code.
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Hm so there's a problem with the tracepoint's caller.

kmalloc_large() is only called from kmalloc() which is an inline  thus the
callsite of kmalloc() calls directly kmalloc_large(). So when
kmalloc_large() does "trace_kmalloc(_RET_IP_, ...)" the _RET_IP_ is the
callsite of kmalloc(), which is what we want.

But with kmalloc_large_node()...

> ---
>  mm/slab_common.c |  6 ++++++
>  mm/slub.c        | 22 ++++------------------
>  2 files changed, 10 insertions(+), 18 deletions(-)
> 
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index e72089515030..cf17be8cd9ad 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -955,6 +955,9 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node)
>  	void *ptr = NULL;
>  	unsigned int order = get_order(size);
>  
> +	if (unlikely(flags & GFP_SLAB_BUG_MASK))
> +		flags = kmalloc_fix_flags(flags);
> +
>  	flags |= __GFP_COMP;
>  	page = alloc_pages_node(node, flags, order);
>  	if (page) {
> @@ -966,6 +969,9 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node)
>  	ptr = kasan_kmalloc_large(ptr, size, flags);
>  	/* As ptr might get tagged, call kmemleak hook after KASAN. */
>  	kmemleak_alloc(ptr, size, 1, flags);
> +	trace_kmalloc_node(_RET_IP_, ptr,
> +			   size, PAGE_SIZE << order,
> +			   flags, node);

... the _RET_IP_ here would be __kmalloc_node() which is not useful.

>  	return ptr;
>  }
> diff --git a/mm/slub.c b/mm/slub.c
> index 640712706f2b..f10a892f1772 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -4396,15 +4396,8 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
>  	struct kmem_cache *s;
>  	void *ret;
>  
> -	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
> -		ret = kmalloc_large_node(size, flags, node);
> -
> -		trace_kmalloc_node(_RET_IP_, ret,
> -				   size, PAGE_SIZE << get_order(size),
> -				   flags, node);

Here it was OK because __kmalloc_node is expanded from something inline
coming from slab.h.

> -
> -		return ret;
> -	}
> +	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
> +		return kmalloc_large_node(size, flags, node);
>  
>  	s = kmalloc_slab(size, flags);
>  
> @@ -4861,15 +4854,8 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
>  	struct kmem_cache *s;
>  	void *ret;
>  
> -	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
> -		ret = kmalloc_large_node(size, gfpflags, node);
> -
> -		trace_kmalloc_node(caller, ret,
> -				   size, PAGE_SIZE << get_order(size),
> -				   gfpflags, node);
> -
> -		return ret;
> -	}
> +	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
> +		return kmalloc_large_node(size, gfpflags, node);

And here it even forgets the 'caller'.

>  
>  	s = kmalloc_slab(size, gfpflags);
>  


^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 09/23] mm/slab_common: cleanup kmalloc_large()
  2022-04-14  8:57 ` [PATCH v2 09/23] mm/slab_common: cleanup kmalloc_large() Hyeonggon Yoo
@ 2022-04-26 17:18   ` Vlastimil Babka
  0 siblings, 0 replies; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-26 17:18 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Now that kmalloc_large() and kmalloc_large_node() do same job,
> make kmalloc_large() wrapper of kmalloc_large_node().
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

This would be fine, but depends on how the previous patch can be fixed up...

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 12/23] mm/slab_common: cleanup kmalloc()
  2022-04-14  8:57 ` [PATCH v2 12/23] mm/slab_common: cleanup kmalloc() Hyeonggon Yoo
@ 2022-04-26 18:00   ` Joe Perches
  2022-04-28 11:30     ` Hyeonggon Yoo
  2022-04-27  7:50   ` Vlastimil Babka
  1 sibling, 1 reply; 51+ messages in thread
From: Joe Perches @ 2022-04-26 18:00 UTC (permalink / raw)
  To: Hyeonggon Yoo, Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On Thu, 2022-04-14 at 17:57 +0900, Hyeonggon Yoo wrote:
> Now that kmalloc() and kmalloc_node() do same job, make kmalloc()
> wrapper of kmalloc_node().

Why make every kmalloc larger object code size ?

What is the overall object size change in a typical defconfig ?



^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 10/23] mm/slab_common: cleanup kmem_cache_alloc{,node,lru}
  2022-04-14  8:57 ` [PATCH v2 10/23] mm/slab_common: cleanup kmem_cache_alloc{,node,lru} Hyeonggon Yoo
@ 2022-04-26 18:01   ` Vlastimil Babka
  2022-04-30 11:48     ` Hyeonggon Yoo
  0 siblings, 1 reply; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-26 18:01 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Implement only __kmem_cache_alloc_node() in slab allocators and make
> kmem_cache_alloc{,node,lru} wrapper of it.
> 
> Now that kmem_cache_alloc{,node,lru} is inline function, we should
> use _THIS_IP_ instead of _RET_IP_ for consistency.

Hm yeah looks like this actually fixes some damage of obscured actual
__RET_IP_ by the recent addition and wrapping of __kmem_cache_alloc_lru().

> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>

Some nits:

> ---
>  include/linux/slab.h | 52 ++++++++++++++++++++++++++++++++-----
>  mm/slab.c            | 61 +++++---------------------------------------
>  mm/slob.c            | 27 ++++++--------------
>  mm/slub.c            | 35 +++++--------------------
>  4 files changed, 67 insertions(+), 108 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index 143830f57a7f..1b5bdcb0fd31 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -429,9 +429,52 @@ void *__kmalloc(size_t size, gfp_t flags)
>  	return __kmalloc_node(size, flags, NUMA_NO_NODE);
>  }
>  
> -void *kmem_cache_alloc(struct kmem_cache *s, gfp_t flags) __assume_slab_alignment __malloc;
> -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
> -			   gfp_t gfpflags) __assume_slab_alignment __malloc;
> +
> +void *__kmem_cache_alloc_node(struct kmem_cache *s, struct list_lru *lru,
> +			   gfp_t gfpflags, int node, unsigned long caller __maybe_unused)
> +			    __assume_slab_alignment __malloc;

I don't think caller needs to be __maybe_unused in the declaration nor any
of the implementations of __kmem_cache_alloc_node(), all actually pass it on?

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 12/23] mm/slab_common: cleanup kmalloc()
  2022-04-14  8:57 ` [PATCH v2 12/23] mm/slab_common: cleanup kmalloc() Hyeonggon Yoo
  2022-04-26 18:00   ` Joe Perches
@ 2022-04-27  7:50   ` Vlastimil Babka
  1 sibling, 0 replies; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-27  7:50 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel, Joe Perches

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Now that kmalloc() and kmalloc_node() do same job, make kmalloc()
> wrapper of kmalloc_node().
> 
> Remove kmem_cache_alloc_trace() that is now unused.
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

From correctness point of view:

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>

But yeah, impact of requiring NUMA_NO_NODE parameter should be evaluated. If
it's significant I believe we should be still able to implement the common
kmalloc, but keep separate kmalloc and kmalloc_node entry points.

> ---
>  include/linux/slab.h | 93 +++++++++++++++-----------------------------
>  mm/slab.c            | 16 --------
>  mm/slub.c            | 12 ------
>  3 files changed, 32 insertions(+), 89 deletions(-)
> 
> diff --git a/include/linux/slab.h b/include/linux/slab.h
> index eb457f20f415..ea168f8a248d 100644
> --- a/include/linux/slab.h
> +++ b/include/linux/slab.h
> @@ -497,23 +497,10 @@ static __always_inline void kfree_bulk(size_t size, void **p)
>  }
>  
>  #ifdef CONFIG_TRACING
> -extern void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t flags, size_t size)
> -				   __assume_slab_alignment __alloc_size(3);
> -
>  extern void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags,
>  					 int node, size_t size) __assume_slab_alignment
>  								__alloc_size(4);
> -
>  #else /* CONFIG_TRACING */
> -static __always_inline __alloc_size(3) void *kmem_cache_alloc_trace(struct kmem_cache *s,
> -								    gfp_t flags, size_t size)
> -{
> -	void *ret = kmem_cache_alloc(s, flags);
> -
> -	ret = kasan_kmalloc(s, ret, size, flags);
> -	return ret;
> -}
> -
>  static __always_inline void *kmem_cache_alloc_node_trace(struct kmem_cache *s, gfp_t gfpflags,
>  							 int node, size_t size)
>  {
> @@ -532,6 +519,37 @@ static __always_inline void *kmalloc_large(size_t size, gfp_t flags)
>  	return kmalloc_large_node(size, flags, NUMA_NO_NODE);
>  }
>  
> +#ifndef CONFIG_SLOB
> +static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
> +{
> +	if (__builtin_constant_p(size)) {
> +		unsigned int index;
> +
> +		if (size > KMALLOC_MAX_CACHE_SIZE)
> +			return kmalloc_large_node(size, flags, node);
> +
> +		index = kmalloc_index(size);
> +
> +		if (!index)
> +			return ZERO_SIZE_PTR;
> +
> +		return kmem_cache_alloc_node_trace(
> +				kmalloc_caches[kmalloc_type(flags)][index],
> +						flags, node, size);
> +	}
> +	return __kmalloc_node(size, flags, node);
> +}
> +#else
> +static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
> +{
> +	if (__builtin_constant_p(size) && size > KMALLOC_MAX_CACHE_SIZE)
> +		return kmalloc_large_node(size, flags, node);
> +
> +	return __kmalloc_node(size, flags, node);
> +}
> +#endif
> +
> +
>  /**
>   * kmalloc - allocate memory
>   * @size: how many bytes of memory are required.
> @@ -588,55 +606,8 @@ static __always_inline void *kmalloc_large(size_t size, gfp_t flags)
>   */
>  static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags)
>  {
> -	if (__builtin_constant_p(size)) {
> -#ifndef CONFIG_SLOB
> -		unsigned int index;
> -#endif
> -		if (size > KMALLOC_MAX_CACHE_SIZE)
> -			return kmalloc_large(size, flags);
> -#ifndef CONFIG_SLOB
> -		index = kmalloc_index(size);
> -
> -		if (!index)
> -			return ZERO_SIZE_PTR;
> -
> -		return kmem_cache_alloc_trace(
> -				kmalloc_caches[kmalloc_type(flags)][index],
> -				flags, size);
> -#endif
> -	}
> -	return __kmalloc(size, flags);
> -}
> -
> -#ifndef CONFIG_SLOB
> -static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
> -{
> -	if (__builtin_constant_p(size)) {
> -		unsigned int index;
> -
> -		if (size > KMALLOC_MAX_CACHE_SIZE)
> -			return kmalloc_large_node(size, flags, node);
> -
> -		index = kmalloc_index(size);
> -
> -		if (!index)
> -			return ZERO_SIZE_PTR;
> -
> -		return kmem_cache_alloc_node_trace(
> -				kmalloc_caches[kmalloc_type(flags)][index],
> -						flags, node, size);
> -	}
> -	return __kmalloc_node(size, flags, node);
> -}
> -#else
> -static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node)
> -{
> -	if (__builtin_constant_p(size) && size > KMALLOC_MAX_CACHE_SIZE)
> -		return kmalloc_large_node(size, flags, node);
> -
> -	return __kmalloc_node(size, flags, node);
> +	return kmalloc_node(size, flags, NUMA_NO_NODE);
>  }
> -#endif
>  
>  /**
>   * kmalloc_array - allocate memory for an array.
> diff --git a/mm/slab.c b/mm/slab.c
> index c5ffe54c207a..b0aaca017f42 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -3507,22 +3507,6 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size,
>  }
>  EXPORT_SYMBOL(kmem_cache_alloc_bulk);
>  
> -#ifdef CONFIG_TRACING
> -void *
> -kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size)
> -{
> -	void *ret;
> -
> -	ret = slab_alloc(cachep, NULL, flags, size, _RET_IP_);
> -
> -	ret = kasan_kmalloc(cachep, ret, size, flags);
> -	trace_kmalloc(_RET_IP_, ret,
> -		      size, cachep->size, flags);
> -	return ret;
> -}
> -EXPORT_SYMBOL(kmem_cache_alloc_trace);
> -#endif
> -
>  #ifdef CONFIG_TRACING
>  void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
>  				  gfp_t flags,
> diff --git a/mm/slub.c b/mm/slub.c
> index 2a2be2a8a5d0..892988990da7 100644
> --- a/mm/slub.c
> +++ b/mm/slub.c
> @@ -3216,18 +3216,6 @@ static __always_inline void *slab_alloc(struct kmem_cache *s, struct list_lru *l
>  	return slab_alloc_node(s, lru, gfpflags, NUMA_NO_NODE, addr, orig_size);
>  }
>  
> -
> -#ifdef CONFIG_TRACING
> -void *kmem_cache_alloc_trace(struct kmem_cache *s, gfp_t gfpflags, size_t size)
> -{
> -	void *ret = slab_alloc(s, NULL, gfpflags, _RET_IP_, size);
> -	trace_kmalloc(_RET_IP_, ret, size, s->size, gfpflags);
> -	ret = kasan_kmalloc(s, ret, size, gfpflags);
> -	return ret;
> -}
> -EXPORT_SYMBOL(kmem_cache_alloc_trace);
> -#endif
> -
>  void *__kmem_cache_alloc_node(struct kmem_cache *s, struct list_lru *lru, gfp_t gfpflags,
>  			      int node, unsigned long caller __maybe_unused)
>  {


^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 13/23] mm/slab: kmalloc: pass requests larger than order-1 page to page allocator
  2022-04-14  8:57 ` [PATCH v2 13/23] mm/slab: kmalloc: pass requests larger than order-1 page to page allocator Hyeonggon Yoo
@ 2022-04-27  8:10   ` Vlastimil Babka
  2022-04-30 11:50     ` Hyeonggon Yoo
  0 siblings, 1 reply; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-27  8:10 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> There is not much benefit for serving large objects in kmalloc().
> Let's pass large requests to page allocator like SLUB for better
> maintenance of common code.
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Reviewed-by: Vlastimil Babka <vbabka@suse.cz>

Some nits:

> @@ -3607,15 +3607,25 @@ void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p)
>  {
>  	struct kmem_cache *s;
>  	size_t i;
> +	struct folio *folio;
>  
>  	local_irq_disable();
>  	for (i = 0; i < size; i++) {
>  		void *objp = p[i];

folio can be declared here
could probably move 's' too, and 'i' to the for () thanks to gnu11

>  
> -		if (!orig_s) /* called via kfree_bulk */
> -			s = virt_to_cache(objp);
> -		else
> +		if (!orig_s) {
> +			folio = virt_to_folio(objp);
> +			/* called via kfree_bulk */
> +			if (!folio_test_slab(folio)) {
> +				local_irq_enable();
> +				free_large_kmalloc(folio, objp);
> +				local_irq_disable();
> +				continue;
> +			}
> +			s = folio_slab(folio)->slab_cache;
> +		} else
>  			s = cache_from_obj(orig_s, objp);

This should now use { } brackets per kernel style.

> +
>  		if (!s)
>  			continue;
>  

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 08/23] mm/slab_common: make kmalloc_large_node() consistent with kmalloc_large()
  2022-04-26 17:15   ` Vlastimil Babka
@ 2022-04-28  6:35     ` Hyeonggon Yoo
  0 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-28  6:35 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On Tue, Apr 26, 2022 at 07:15:06PM +0200, Vlastimil Babka wrote:
> On 4/14/22 10:57, Hyeonggon Yoo wrote:
> > Move tracepoints into kmalloc_large_node() and add missing flag fix code.
> > 
> > Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> 

Hello Vlastimil, thanks for review! ;-)

> Hm so there's a problem with the tracepoint's caller.
> 
> kmalloc_large() is only called from kmalloc() which is an inline  thus the
> callsite of kmalloc() calls directly kmalloc_large().
> So when kmalloc_large() does "trace_kmalloc(_RET_IP_, ...)" the _RET_IP_ is the
> callsite of kmalloc(), which is what we want.

kmalloc_large() had the exact problem before my series when called from __kmalloc().

On top of current slab/for-next:
  [000] .....    43.172574: kmalloc: call_site=__kmalloc+0x2aa/0x300 ptr=ffff88e2183a0000 bytes_req=12368 bytes_alloc=16384 gfp_flags=GFP_KERNEL

Considering different usecases of kmalloc_large_node() (called from kmalloc_node() or __kmalloc_node()),
I think we need trace/notrace version of kmalloc_large_node().

> 
> But with kmalloc_large_node()...
> 
> > ---
> >  mm/slab_common.c |  6 ++++++
> >  mm/slub.c        | 22 ++++------------------
> >  2 files changed, 10 insertions(+), 18 deletions(-)
> > 
> > diff --git a/mm/slab_common.c b/mm/slab_common.c
> > index e72089515030..cf17be8cd9ad 100644
> > --- a/mm/slab_common.c
> > +++ b/mm/slab_common.c
> > @@ -955,6 +955,9 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node)
> >  	void *ptr = NULL;
> >  	unsigned int order = get_order(size);
> >  
> > +	if (unlikely(flags & GFP_SLAB_BUG_MASK))
> > +		flags = kmalloc_fix_flags(flags);
> > +
> >  	flags |= __GFP_COMP;
> >  	page = alloc_pages_node(node, flags, order);
> >  	if (page) {
> > @@ -966,6 +969,9 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node)
> >  	ptr = kasan_kmalloc_large(ptr, size, flags);
> >  	/* As ptr might get tagged, call kmemleak hook after KASAN. */
> >  	kmemleak_alloc(ptr, size, 1, flags);
> > +	trace_kmalloc_node(_RET_IP_, ptr,
> > +			   size, PAGE_SIZE << order,
> > +			   flags, node);
> 
> ... the _RET_IP_ here would be __kmalloc_node() which is not useful.
> 
> >  	return ptr;
> >  }
> > diff --git a/mm/slub.c b/mm/slub.c
> > index 640712706f2b..f10a892f1772 100644
> > --- a/mm/slub.c
> > +++ b/mm/slub.c
> > @@ -4396,15 +4396,8 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
> >  	struct kmem_cache *s;
> >  	void *ret;
> >  
> > -	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
> > -		ret = kmalloc_large_node(size, flags, node);
> > -
> > -		trace_kmalloc_node(_RET_IP_, ret,
> > -				   size, PAGE_SIZE << get_order(size),
> > -				   flags, node);
> 
> Here it was OK because __kmalloc_node is expanded from something inline
> coming from slab.h.
> 
> > -
> > -		return ret;
> > -	}
> > +	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
> > +		return kmalloc_large_node(size, flags, node);
> >  
> >  	s = kmalloc_slab(size, flags);
> >  
> > @@ -4861,15 +4854,8 @@ void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
> >  	struct kmem_cache *s;
> >  	void *ret;
> >  
> > -	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
> > -		ret = kmalloc_large_node(size, gfpflags, node);
> > -
> > -		trace_kmalloc_node(caller, ret,
> > -				   size, PAGE_SIZE << get_order(size),
> > -				   gfpflags, node);
> > -
> > -		return ret;
> > -	}
> > +	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
> > +		return kmalloc_large_node(size, gfpflags, node);
> 
> And here it even forgets the 'caller'.
>

Thanks for catching this.
I think notrace version + tracepoint would fit here.

> >  
> >  	s = kmalloc_slab(size, gfpflags);
> >  
> 

-- 
Thanks,
Hyeonggon

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 12/23] mm/slab_common: cleanup kmalloc()
  2022-04-26 18:00   ` Joe Perches
@ 2022-04-28 11:30     ` Hyeonggon Yoo
  0 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-28 11:30 UTC (permalink / raw)
  To: Joe Perches
  Cc: Vlastimil Babka, Marco Elver, Matthew WilCox, Christoph Lameter,
	Pekka Enberg, David Rientjes, Joonsoo Kim, Andrew Morton,
	Roman Gushchin, linux-mm, linux-kernel

On Tue, Apr 26, 2022 at 11:00:32AM -0700, Joe Perches wrote:
> On Thu, 2022-04-14 at 17:57 +0900, Hyeonggon Yoo wrote:
> > Now that kmalloc() and kmalloc_node() do same job, make kmalloc()
> > wrapper of kmalloc_node().
> 
> Why make every kmalloc larger object code size ?
> 
> What is the overall object size change in a typical defconfig ?

Thank you for pointing this.

It increases every call to kmalloc() is increased (+5 bytes) due to
passing NUMA_NO_NODE. on x86_64's defconfig, the difference of vmlinux size is not that
visible because this patch also removes tracepoint.

But yeah, increasing 5 bytes for every caller seems questionable.
I'll think about this for a bit :)

Thanks!

-- 
Thanks,
Hyeonggon

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 14/23] mm/slab_common: print cache name in tracepoints
  2022-04-14  8:57 ` [PATCH v2 14/23] mm/slab_common: print cache name in tracepoints Hyeonggon Yoo
@ 2022-04-29 14:05   ` Vlastimil Babka
  2022-04-30 14:06     ` Hyeonggon Yoo
  0 siblings, 1 reply; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-29 14:05 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Print cache name in tracepoints. If there is no corresponding cache
> (kmalloc in SLOB or kmalloc_large_node), use KMALLOC_{,LARGE_}NAME
> macro.
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Regarding tracepoints, I'm not sure it's a good idea to unify kmalloc and
kmem_cache_alloc. I think the common use case is to trace kmalloc as there
are many different callers, and then I'm not interested in kmem_cache_alloc
callers much.

What I would suggest instead is:
- drop the _node versions, add node to normal versions
- drop the kmem_alloc EVENT_CLASS, as kmalloc is different enough from
kmem_cache_alloc (see next points), define separately as TRACE_EVENT().
- printing cache_name makes sense to add for kmem_cache_alloc (also allows
filtering events) but not for kmalloc.
- kmem_cache_alloc with name can then drop the bytes_req, bytes_alloc as
they are fixed for given name (and can be read from slabinfo).

Not using a common tracepoint will prevent some later unifications/cleanup
(patch 21?), but hopefully not too much?

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 18/23] mm/sl[au]b: generalize kmalloc subsystem
  2022-04-14  8:57 ` [PATCH v2 18/23] mm/sl[au]b: generalize kmalloc subsystem Hyeonggon Yoo
@ 2022-04-29 14:30   ` Vlastimil Babka
  0 siblings, 0 replies; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-29 14:30 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Now everything in kmalloc subsystem can be generalized.
> Let's do it!
> 
> Generalize __kmalloc_node_track_caller(), kfree(), __ksize(),
> __kmalloc_node() and move them to slab_common.c.
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> ---
>  mm/slab.c        | 94 -----------------------------------------------
>  mm/slab_common.c | 95 ++++++++++++++++++++++++++++++++++++++++++++++++
>  mm/slub.c        | 88 --------------------------------------------
>  3 files changed, 95 insertions(+), 182 deletions(-)
> 
> diff --git a/mm/slab.c b/mm/slab.c
> index d35873da5572..fc00aca62ae3 100644
> --- a/mm/slab.c
> +++ b/mm/slab.c
> @@ -3527,36 +3527,6 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep,
>  EXPORT_SYMBOL(kmem_cache_alloc_node_trace);
>  #endif
>  
> -static __always_inline void *
> -__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller)
> -{
> -	struct kmem_cache *cachep;
> -	void *ret;
> -
> -	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
> -		return kmalloc_large_node(size, flags, node);
> -	cachep = kmalloc_slab(size, flags);
> -	if (unlikely(ZERO_OR_NULL_PTR(cachep)))
> -		return cachep;
> -	ret = kmem_cache_alloc_node_trace(cachep, flags, node, size);
> -	ret = kasan_kmalloc(cachep, ret, size, flags);

SLAB did kasan_kmalloc() when called from __kmalloc_node_track_caller(), and
will not do after this patch, until the next patch 19/23. So I would just
fold it to this patch.

> -
> -	return ret;
> -}
> -
> -void *__kmalloc_node(size_t size, gfp_t flags, int node)
> -{
> -	return __do_kmalloc_node(size, flags, node, _RET_IP_);
> -}
> -EXPORT_SYMBOL(__kmalloc_node);
> -
> -void *__kmalloc_node_track_caller(size_t size, gfp_t flags,
> -		int node, unsigned long caller)
> -{
> -	return __do_kmalloc_node(size, flags, node, caller);
> -}
> -EXPORT_SYMBOL(__kmalloc_node_track_caller);
> -
>  #ifdef CONFIG_PRINTK
>  void kmem_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *slab)
>  {

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 20/23] mm/slab_common: factor out __do_kmalloc_node()
  2022-04-14  8:57 ` [PATCH v2 20/23] mm/slab_common: factor out __do_kmalloc_node() Hyeonggon Yoo
  2022-04-14 11:45   ` Hyeonggon Yoo
@ 2022-04-29 14:48   ` Vlastimil Babka
  1 sibling, 0 replies; 51+ messages in thread
From: Vlastimil Babka @ 2022-04-29 14:48 UTC (permalink / raw)
  To: Hyeonggon Yoo
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On 4/14/22 10:57, Hyeonggon Yoo wrote:
> Factor out common code into __do_kmalloc_node().
> 
> Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>

Looks good but let's see how things look like after changes to previous patches.

> ---
>  mm/slab_common.c | 27 ++++++++++-----------------
>  1 file changed, 10 insertions(+), 17 deletions(-)
> 
> diff --git a/mm/slab_common.c b/mm/slab_common.c
> index 6abe7f61c197..af563e64e8aa 100644
> --- a/mm/slab_common.c
> +++ b/mm/slab_common.c
> @@ -919,7 +919,9 @@ void free_large_kmalloc(struct folio *folio, void *object)
>  	__free_pages(folio_page(folio, 0), order);
>  }
>  
> -void *__kmalloc_node(size_t size, gfp_t flags, int node)
> +static __always_inline
> +void *__do_kmalloc_node(size_t size, gfp_t flags, int node,
> +			unsigned long caller __maybe_unused)
>  {
>  	struct kmem_cache *s;
>  	void *ret;
> @@ -932,31 +934,22 @@ void *__kmalloc_node(size_t size, gfp_t flags, int node)
>  	if (unlikely(ZERO_OR_NULL_PTR(s)))
>  		return s;
>  
> -	ret = __kmem_cache_alloc_node(s, NULL, flags, node, _RET_IP_);
> +	ret = __kmem_cache_alloc_node(s, NULL, flags, node, caller);
>  	ret = kasan_kmalloc(s, ret, size, flags);
>  
>  	return ret;
>  }
> +
> +void *__kmalloc_node(size_t size, gfp_t flags, int node)
> +{
> +	return __do_kmalloc_node(size, flags, node, _RET_IP_);
> +}
>  EXPORT_SYMBOL(__kmalloc_node);
>  
>  void *__kmalloc_node_track_caller(size_t size, gfp_t gfpflags,
>  					int node, unsigned long caller)
>  {
> -	struct kmem_cache *s;
> -	void *ret;
> -
> -	if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
> -		return kmalloc_large_node(size, gfpflags, node);
> -
> -	s = kmalloc_slab(size, gfpflags);
> -
> -	if (unlikely(ZERO_OR_NULL_PTR(s)))
> -		return s;
> -
> -	ret = __kmem_cache_alloc_node(s, NULL, gfpflags, node, caller);
> -	ret = kasan_kmalloc(s, ret, size, gfpflags);
> -
> -	return ret;
> +	return __do_kmalloc_node(size, flags, node, caller);
>  }
>  EXPORT_SYMBOL(__kmalloc_node_track_caller);
>  


^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 04/23] mm/slab_common: cleanup kmalloc_track_caller()
  2022-04-26 15:49   ` Vlastimil Babka
@ 2022-04-30 11:44     ` Hyeonggon Yoo
  0 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-30 11:44 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On Tue, Apr 26, 2022 at 05:49:08PM +0200, Vlastimil Babka wrote:
> On 4/14/22 10:57, Hyeonggon Yoo wrote:
> > Make kmalloc_track_caller() wrapper of kmalloc_node_track_caller().
> > 
> > Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> > ---
> >  include/linux/slab.h | 17 ++++++++---------
> >  mm/slab.c            |  6 ------
> >  mm/slob.c            |  6 ------
> >  mm/slub.c            | 22 ----------------------
> >  4 files changed, 8 insertions(+), 43 deletions(-)
> > 
> > diff --git a/include/linux/slab.h b/include/linux/slab.h
> > index a3b9d4c20d7e..acdb4b7428f9 100644
> > --- a/include/linux/slab.h
> > +++ b/include/linux/slab.h
> > @@ -639,6 +639,12 @@ static inline __alloc_size(1, 2) void *kcalloc(size_t n, size_t size, gfp_t flag
> >  	return kmalloc_array(n, size, flags | __GFP_ZERO);
> >  }
> >  
> > +extern void *__kmalloc_node_track_caller(size_t size, gfp_t flags, int node,
> > +					 unsigned long caller) __alloc_size(1);
> 
> Update: in v3 remove the extern, it's not necessary and the kernel is slowly
> getting rid of them.

Will do in v3. Thanks!

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 10/23] mm/slab_common: cleanup kmem_cache_alloc{,node,lru}
  2022-04-26 18:01   ` Vlastimil Babka
@ 2022-04-30 11:48     ` Hyeonggon Yoo
  0 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-30 11:48 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On Tue, Apr 26, 2022 at 08:01:27PM +0200, Vlastimil Babka wrote:
> On 4/14/22 10:57, Hyeonggon Yoo wrote:
> > Implement only __kmem_cache_alloc_node() in slab allocators and make
> > kmem_cache_alloc{,node,lru} wrapper of it.
> > 
> > Now that kmem_cache_alloc{,node,lru} is inline function, we should
> > use _THIS_IP_ instead of _RET_IP_ for consistency.
> 
> Hm yeah looks like this actually fixes some damage of obscured actual
> __RET_IP_ by the recent addition and wrapping of __kmem_cache_alloc_lru().
> 
> > Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> 
> Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
> 
> Some nits:
> 
> > ---
> >  include/linux/slab.h | 52 ++++++++++++++++++++++++++++++++-----
> >  mm/slab.c            | 61 +++++---------------------------------------
> >  mm/slob.c            | 27 ++++++--------------
> >  mm/slub.c            | 35 +++++--------------------
> >  4 files changed, 67 insertions(+), 108 deletions(-)
> > 
> > diff --git a/include/linux/slab.h b/include/linux/slab.h
> > index 143830f57a7f..1b5bdcb0fd31 100644
> > --- a/include/linux/slab.h
> > +++ b/include/linux/slab.h
> > @@ -429,9 +429,52 @@ void *__kmalloc(size_t size, gfp_t flags)
> >  	return __kmalloc_node(size, flags, NUMA_NO_NODE);
> >  }
> >  
> > -void *kmem_cache_alloc(struct kmem_cache *s, gfp_t flags) __assume_slab_alignment __malloc;
> > -void *kmem_cache_alloc_lru(struct kmem_cache *s, struct list_lru *lru,
> > -			   gfp_t gfpflags) __assume_slab_alignment __malloc;
> > +
> > +void *__kmem_cache_alloc_node(struct kmem_cache *s, struct list_lru *lru,
> > +			   gfp_t gfpflags, int node, unsigned long caller __maybe_unused)
> > +			    __assume_slab_alignment __malloc;
> 
> I don't think caller needs to be __maybe_unused in the declaration nor any
> of the implementations of __kmem_cache_alloc_node(), all actually pass it on?

My intention was to give hints to compilers when CONFIG_TRACING=n.
I'll check if the compiler just optimizes them without __maybe_unused.

Thanks!

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 13/23] mm/slab: kmalloc: pass requests larger than order-1 page to page allocator
  2022-04-27  8:10   ` Vlastimil Babka
@ 2022-04-30 11:50     ` Hyeonggon Yoo
  0 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-30 11:50 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On Wed, Apr 27, 2022 at 10:10:00AM +0200, Vlastimil Babka wrote:
> On 4/14/22 10:57, Hyeonggon Yoo wrote:
> > There is not much benefit for serving large objects in kmalloc().
> > Let's pass large requests to page allocator like SLUB for better
> > maintenance of common code.
> > 
> > Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> 
> Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
> 
> Some nits:
> 
> > @@ -3607,15 +3607,25 @@ void kmem_cache_free_bulk(struct kmem_cache *orig_s, size_t size, void **p)
> >  {
> >  	struct kmem_cache *s;
> >  	size_t i;
> > +	struct folio *folio;
> >  
> >  	local_irq_disable();
> >  	for (i = 0; i < size; i++) {
> >  		void *objp = p[i];
> 
> folio can be declared here
> could probably move 's' too, and 'i' to the for () thanks to gnu11
>

Right!

> >  
> > -		if (!orig_s) /* called via kfree_bulk */
> > -			s = virt_to_cache(objp);
> > -		else
> > +		if (!orig_s) {
> > +			folio = virt_to_folio(objp);
> > +			/* called via kfree_bulk */
> > +			if (!folio_test_slab(folio)) {
> > +				local_irq_enable();
> > +				free_large_kmalloc(folio, objp);
> > +				local_irq_disable();
> > +				continue;
> > +			}
> > +			s = folio_slab(folio)->slab_cache;
> > +		} else
> >  			s = cache_from_obj(orig_s, objp);
> 
> This should now use { } brackets per kernel style.
> 

Yes. will do both in v3.

> > +
> >  		if (!s)
> >  			continue;
> >  

^ permalink raw reply	[flat|nested] 51+ messages in thread

* Re: [PATCH v2 14/23] mm/slab_common: print cache name in tracepoints
  2022-04-29 14:05   ` Vlastimil Babka
@ 2022-04-30 14:06     ` Hyeonggon Yoo
  0 siblings, 0 replies; 51+ messages in thread
From: Hyeonggon Yoo @ 2022-04-30 14:06 UTC (permalink / raw)
  To: Vlastimil Babka
  Cc: Marco Elver, Matthew WilCox, Christoph Lameter, Pekka Enberg,
	David Rientjes, Joonsoo Kim, Andrew Morton, Roman Gushchin,
	linux-mm, linux-kernel

On Fri, Apr 29, 2022 at 04:05:14PM +0200, Vlastimil Babka wrote:
> On 4/14/22 10:57, Hyeonggon Yoo wrote:
> > Print cache name in tracepoints. If there is no corresponding cache
> > (kmalloc in SLOB or kmalloc_large_node), use KMALLOC_{,LARGE_}NAME
> > macro.
> > 
> > Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com>
> 
> Regarding tracepoints, I'm not sure it's a good idea to unify kmalloc and
> kmem_cache_alloc. I think the common use case is to trace kmalloc as there
> are many different callers, and then I'm not interested in kmem_cache_alloc
> callers much.

Make sense.

> What I would suggest instead is:
> - drop the _node versions, add node to normal versions
> - drop the kmem_alloc EVENT_CLASS, as kmalloc is different enough from
> kmem_cache_alloc (see next points), define separately as TRACE_EVENT().
> - printing cache_name makes sense to add for kmem_cache_alloc (also allows
> filtering events) but not for kmalloc.
> - kmem_cache_alloc with name can then drop the bytes_req, bytes_alloc as
> they are fixed for given name (and can be read from slabinfo).

Those suggestions makes sense to me.
kmalloc and kmem_cache_alloc are a bit different.

> Not using a common tracepoint will prevent some later unifications/cleanup
> (patch 21?), but hopefully not too much?

In v2 unification of __kmalloc_node()/kfree() somewhat depends on unified tracepoint.
I'll try to unify them with this approach in v3.

^ permalink raw reply	[flat|nested] 51+ messages in thread

end of thread, other threads:[~2022-04-30 14:07 UTC | newest]

Thread overview: 51+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-14  8:57 [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 01/23] mm/slab: move NUMA-related code to __do_cache_alloc() Hyeonggon Yoo
2022-04-22 18:04   ` Vlastimil Babka
2022-04-14  8:57 ` [PATCH v2 02/23] mm/slab: cleanup slab_alloc() and slab_alloc_node() Hyeonggon Yoo
2022-04-25 14:05   ` Vlastimil Babka
2022-04-14  8:57 ` [PATCH v2 03/23] mm/slab_common: remove CONFIG_NUMA ifdefs for common kmalloc functions Hyeonggon Yoo
2022-04-25 14:41   ` Vlastimil Babka
2022-04-14  8:57 ` [PATCH v2 04/23] mm/slab_common: cleanup kmalloc_track_caller() Hyeonggon Yoo
2022-04-25 15:05   ` Vlastimil Babka
2022-04-26 15:49   ` Vlastimil Babka
2022-04-30 11:44     ` Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 05/23] mm/slab_common: cleanup __kmalloc() Hyeonggon Yoo
2022-04-26 16:02   ` Vlastimil Babka
2022-04-14  8:57 ` [PATCH v2 06/23] mm/sl[auo]b: fold kmalloc_order_trace() into kmalloc_large() Hyeonggon Yoo
2022-04-26 16:09   ` Vlastimil Babka
2022-04-14  8:57 ` [PATCH v2 07/23] mm/slub: move kmalloc_large_node() to slab_common.c Hyeonggon Yoo
2022-04-26 16:13   ` Vlastimil Babka
2022-04-14  8:57 ` [PATCH v2 08/23] mm/slab_common: make kmalloc_large_node() consistent with kmalloc_large() Hyeonggon Yoo
2022-04-26 17:15   ` Vlastimil Babka
2022-04-28  6:35     ` Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 09/23] mm/slab_common: cleanup kmalloc_large() Hyeonggon Yoo
2022-04-26 17:18   ` Vlastimil Babka
2022-04-14  8:57 ` [PATCH v2 10/23] mm/slab_common: cleanup kmem_cache_alloc{,node,lru} Hyeonggon Yoo
2022-04-26 18:01   ` Vlastimil Babka
2022-04-30 11:48     ` Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 11/23] mm/slab_common: kmalloc_node: pass large requests to page allocator Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 12/23] mm/slab_common: cleanup kmalloc() Hyeonggon Yoo
2022-04-26 18:00   ` Joe Perches
2022-04-28 11:30     ` Hyeonggon Yoo
2022-04-27  7:50   ` Vlastimil Babka
2022-04-14  8:57 ` [PATCH v2 13/23] mm/slab: kmalloc: pass requests larger than order-1 page to page allocator Hyeonggon Yoo
2022-04-27  8:10   ` Vlastimil Babka
2022-04-30 11:50     ` Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 14/23] mm/slab_common: print cache name in tracepoints Hyeonggon Yoo
2022-04-29 14:05   ` Vlastimil Babka
2022-04-30 14:06     ` Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 15/23] mm/slab_common: use same tracepoint in kmalloc and normal caches Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 16/23] mm/slab_common: rename tracepoint Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 17/23] mm/slab_common: implement __kmem_cache_free() Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 18/23] mm/sl[au]b: generalize kmalloc subsystem Hyeonggon Yoo
2022-04-29 14:30   ` Vlastimil Babka
2022-04-14  8:57 ` [PATCH v2 19/23] mm/slab_common: add kasan_kmalloc() in __kmalloc_node_track_caller() Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 20/23] mm/slab_common: factor out __do_kmalloc_node() Hyeonggon Yoo
2022-04-14 11:45   ` Hyeonggon Yoo
2022-04-29 14:48   ` Vlastimil Babka
2022-04-14  8:57 ` [PATCH v2 21/23] mm/sl[au]b: remove kmem_cache_alloc_node_trace() Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 22/23] mm/sl[auo]b: move definition of __ksize() to mm/slab.h Hyeonggon Yoo
2022-04-14  8:57 ` [PATCH v2 23/23] mm/sl[au]b: check if large object is valid in __ksize() Hyeonggon Yoo
2022-04-14  9:58   ` Christoph Lameter
2022-04-14 11:46     ` Hyeonggon Yoo
2022-04-14 12:36 ` [PATCH v2 00/23] common kmalloc for SLUB and SLAB v2 Hyeonggon Yoo

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).