* CK5 [01/18] Use correct cpu_slab on dead cpu
[not found] <20121101214538.971500204@linux.com>
2012-11-01 21:46 ` CK5 [02/18] slab: Simplify bootstrap Christoph Lameter
@ 2012-11-01 21:46 ` Christoph Lameter
2012-11-02 20:11 ` David Rientjes
2012-11-01 21:47 ` CK5 [05/18] slab: Use the new create_boot_cache function to simplify bootstrap Christoph Lameter
` (15 subsequent siblings)
17 siblings, 1 reply; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:46 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Pass a kmem_cache_cpu pointer into unfreeze partials so that a different
kmem_cache_cpu structure than the local one can be specified.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/mm/slub.c
===================================================================
--- linux.orig/mm/slub.c 2012-11-01 10:10:05.073716747 -0500
+++ linux/mm/slub.c 2012-11-01 10:10:06.173734998 -0500
@@ -1871,10 +1871,10 @@ redo:
*
* This function must be called with interrupt disabled.
*/
-static void unfreeze_partials(struct kmem_cache *s)
+static void unfreeze_partials(struct kmem_cache *s,
+ struct kmem_cache_cpu *c)
{
struct kmem_cache_node *n = NULL, *n2 = NULL;
- struct kmem_cache_cpu *c = this_cpu_ptr(s->cpu_slab);
struct page *page, *discard_page = NULL;
while ((page = c->partial)) {
@@ -1960,7 +1960,7 @@ static int put_cpu_partial(struct kmem_c
* set to the per node partial list.
*/
local_irq_save(flags);
- unfreeze_partials(s);
+ unfreeze_partials(s, this_cpu_ptr(s->cpu_slab));
local_irq_restore(flags);
oldpage = NULL;
pobjects = 0;
@@ -2003,7 +2003,7 @@ static inline void __flush_cpu_slab(stru
if (c->page)
flush_slab(s, c);
- unfreeze_partials(s);
+ unfreeze_partials(s, c);
}
}
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [02/18] slab: Simplify bootstrap
[not found] <20121101214538.971500204@linux.com>
@ 2012-11-01 21:46 ` Christoph Lameter
2012-11-02 20:22 ` David Rientjes
2012-11-01 21:46 ` CK5 [01/18] Use correct cpu_slab on dead cpu Christoph Lameter
` (16 subsequent siblings)
17 siblings, 1 reply; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:46 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
The nodelists field in kmem_cache is pointing to the first unused
object in the array field when bootstrap is complete.
A problem with the current approach is that the statically sized
kmem_cache structure use on boot can only contain NR_CPUS entries.
If the number of nodes plus the number of cpus is greater then we
would overwrite memory following the kmem_cache_boot definition.
Increase the size of the array field to ensure that also the node
pointers fit into the array field.
Once we do that we no longer need the kmem_cache_nodelists
array and we can then also use that structure elsewhere.
V1->V2:
- No need to zap kmem_cache->nodelists since it is allocated
with kmem_cache_zalloc() [glommer]
Acked-by: Glauber Costa <glommer@parallels.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
---
include/linux/slab_def.h | 2 +-
mm/slab.c | 18 +++++++++++++-----
2 files changed, 14 insertions(+), 6 deletions(-)
Index: linux/include/linux/slab_def.h
===================================================================
--- linux.orig/include/linux/slab_def.h 2012-11-01 10:09:47.073417947 -0500
+++ linux/include/linux/slab_def.h 2012-11-01 10:09:55.357555494 -0500
@@ -91,7 +91,7 @@ struct kmem_cache {
* is statically defined, so we reserve the max number of cpus.
*/
struct kmem_list3 **nodelists;
- struct array_cache *array[NR_CPUS];
+ struct array_cache *array[NR_CPUS + MAX_NUMNODES];
/*
* Do not add fields after array[]
*/
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 10:09:47.073417947 -0500
+++ linux/mm/slab.c 2012-11-01 10:09:55.361555562 -0500
@@ -553,9 +553,7 @@ static struct arraycache_init initarray_
{ {0, BOOT_CPUCACHE_ENTRIES, 1, 0} };
/* internal cache of cache description objs */
-static struct kmem_list3 *kmem_cache_nodelists[MAX_NUMNODES];
static struct kmem_cache kmem_cache_boot = {
- .nodelists = kmem_cache_nodelists,
.batchcount = 1,
.limit = BOOT_CPUCACHE_ENTRIES,
.shared = 1,
@@ -1560,6 +1558,15 @@ static void __init set_up_list3s(struct
}
/*
+ * The memory after the last cpu cache pointer is used for the
+ * the nodelists pointer.
+ */
+static void setup_nodelists_pointer(struct kmem_cache *s)
+{
+ s->nodelists = (struct kmem_list3 **)&s->array[nr_cpu_ids];
+}
+
+/*
* Initialisation. Called after the page allocator have been initialised and
* before smp_init().
*/
@@ -1573,15 +1580,14 @@ void __init kmem_cache_init(void)
int node;
kmem_cache = &kmem_cache_boot;
+ setup_nodelists_pointer(kmem_cache);
if (num_possible_nodes() == 1)
use_alien_caches = 0;
- for (i = 0; i < NUM_INIT_LISTS; i++) {
+ for (i = 0; i < NUM_INIT_LISTS; i++)
kmem_list3_init(&initkmem_list3[i]);
- if (i < MAX_NUMNODES)
- kmem_cache->nodelists[i] = NULL;
- }
+
set_up_list3s(kmem_cache, CACHE_CACHE);
/*
@@ -1619,7 +1625,6 @@ void __init kmem_cache_init(void)
list_add(&kmem_cache->list, &slab_caches);
kmem_cache->colour_off = cache_line_size();
kmem_cache->array[smp_processor_id()] = &initarray_cache.cache;
- kmem_cache->nodelists[node] = &initkmem_list3[CACHE_CACHE + node];
/*
* struct kmem_cache size depends on nr_node_ids & nr_cpu_ids
@@ -2425,7 +2430,7 @@ __kmem_cache_create (struct kmem_cache *
else
gfp = GFP_NOWAIT;
- cachep->nodelists = (struct kmem_list3 **)&cachep->array[nr_cpu_ids];
+ setup_nodelists_pointer(cachep);
#if DEBUG
/*
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [04/18] slub: Use a statically allocated kmem_cache boot structure for bootstrap
[not found] <20121101214538.971500204@linux.com>
` (2 preceding siblings ...)
2012-11-01 21:47 ` CK5 [05/18] slab: Use the new create_boot_cache function to simplify bootstrap Christoph Lameter
@ 2012-11-01 21:47 ` Christoph Lameter
2012-11-01 21:47 ` CK5 [03/18] create common functions for boot slab creation Christoph Lameter
` (13 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:47 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Simplify bootstrap by statically allocated two kmem_cache structures. These are
freed after bootup is complete. Allows us to no longer worry about calculations
of sizes of kmem_cache structures during bootstrap.
V1->V2:
- Use kmem_cache_zalloc to properly zero structures.
- Simplify setup by introducing a new boottime
function "bootstrap()".
V2->V3:
- bootstrap function adds slab to list of slab caches
*after* performing relocation.
- Get of of kmem_size.
- Make static boot structures local to kmem_cache_init
Reviewed-by: Glauber Costa <glommer@parallels.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
---
mm/slub.c | 41 +++++++++++------------------------------
1 file changed, 11 insertions(+), 30 deletions(-)
Index: linux/mm/slub.c
===================================================================
--- linux.orig/mm/slub.c 2012-11-01 16:08:07.489743348 -0500
+++ linux/mm/slub.c 2012-11-01 16:11:31.068603160 -0500
@@ -176,8 +176,6 @@ static inline int kmem_cache_debug(struc
#define __OBJECT_POISON 0x80000000UL /* Poison object */
#define __CMPXCHG_DOUBLE 0x40000000UL /* Use cmpxchg_double */
-static int kmem_size = sizeof(struct kmem_cache);
-
#ifdef CONFIG_SMP
static struct notifier_block slab_notifier;
#endif
@@ -3632,15 +3630,16 @@ static int slab_memory_callback(struct n
/*
* Used for early kmem_cache structures that were allocated using
- * the page allocator
+ * the page allocator. Allocate them properly then fix up the pointers
+ * that may be pointing to the wrong kmem_cache structure.
*/
-static void __init kmem_cache_bootstrap_fixup(struct kmem_cache *s)
+static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache)
{
int node;
+ struct kmem_cache *s = kmem_cache_zalloc(kmem_cache, GFP_NOWAIT);
- list_add(&s->list, &slab_caches);
- s->refcount = -1;
+ memcpy(s, static_cache, kmem_cache->object_size);
for_each_node_state(node, N_NORMAL_MEMORY) {
struct kmem_cache_node *n = get_node(s, node);
@@ -3656,70 +3655,44 @@ static void __init kmem_cache_bootstrap_
#endif
}
}
+ list_add(&s->list, &slab_caches);
+ return s;
}
void __init kmem_cache_init(void)
{
+ static __initdata struct kmem_cache boot_kmem_cache,
+ boot_kmem_cache_node;
int i;
- int caches = 0;
- struct kmem_cache *temp_kmem_cache;
- int order;
- struct kmem_cache *temp_kmem_cache_node;
- unsigned long kmalloc_size;
+ int caches = 2;
if (debug_guardpage_minorder())
slub_max_order = 0;
- kmem_size = offsetof(struct kmem_cache, node) +
- nr_node_ids * sizeof(struct kmem_cache_node *);
+ kmem_cache_node = &boot_kmem_cache_node;
+ kmem_cache = &boot_kmem_cache;
- /* Allocate two kmem_caches from the page allocator */
- kmalloc_size = ALIGN(kmem_size, cache_line_size());
- order = get_order(2 * kmalloc_size);
- kmem_cache = (void *)__get_free_pages(GFP_NOWAIT | __GFP_ZERO, order);
-
- /*
- * Must first have the slab cache available for the allocations of the
- * struct kmem_cache_node's. There is special bootstrap code in
- * kmem_cache_open for slab_state == DOWN.
- */
- kmem_cache_node = (void *)kmem_cache + kmalloc_size;
-
- kmem_cache_node->name = "kmem_cache_node";
- kmem_cache_node->size = kmem_cache_node->object_size =
- sizeof(struct kmem_cache_node);
- kmem_cache_open(kmem_cache_node, SLAB_HWCACHE_ALIGN | SLAB_PANIC);
+ create_boot_cache(kmem_cache_node, "kmem_cache_node",
+ sizeof(struct kmem_cache_node), SLAB_HWCACHE_ALIGN);
hotplug_memory_notifier(slab_memory_callback, SLAB_CALLBACK_PRI);
/* Able to allocate the per node structures */
slab_state = PARTIAL;
- temp_kmem_cache = kmem_cache;
- kmem_cache->name = "kmem_cache";
- kmem_cache->size = kmem_cache->object_size = kmem_size;
- kmem_cache_open(kmem_cache, SLAB_HWCACHE_ALIGN | SLAB_PANIC);
+ create_boot_cache(kmem_cache, "kmem_cache",
+ offsetof(struct kmem_cache, node) +
+ nr_node_ids * sizeof(struct kmem_cache_node *),
+ SLAB_HWCACHE_ALIGN);
- kmem_cache = kmem_cache_alloc(kmem_cache, GFP_NOWAIT);
- memcpy(kmem_cache, temp_kmem_cache, kmem_size);
+ kmem_cache = bootstrap(&boot_kmem_cache);
/*
* Allocate kmem_cache_node properly from the kmem_cache slab.
* kmem_cache_node is separately allocated so no need to
* update any list pointers.
*/
- temp_kmem_cache_node = kmem_cache_node;
-
- kmem_cache_node = kmem_cache_alloc(kmem_cache, GFP_NOWAIT);
- memcpy(kmem_cache_node, temp_kmem_cache_node, kmem_size);
-
- kmem_cache_bootstrap_fixup(kmem_cache_node);
-
- caches++;
- kmem_cache_bootstrap_fixup(kmem_cache);
- caches++;
- /* Free temporary boot structure */
- free_pages((unsigned long)temp_kmem_cache, order);
+ kmem_cache_node = bootstrap(&boot_kmem_cache_node);
/* Now we can use the kmem_cache to allocate kmalloc slabs */
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [03/18] create common functions for boot slab creation
[not found] <20121101214538.971500204@linux.com>
` (3 preceding siblings ...)
2012-11-01 21:47 ` CK5 [04/18] slub: Use a statically allocated kmem_cache boot structure for bootstrap Christoph Lameter
@ 2012-11-01 21:47 ` Christoph Lameter
2012-11-02 20:34 ` David Rientjes
2012-11-01 21:47 ` CK5 [12/18] Common constants for kmalloc boundaries Christoph Lameter
` (12 subsequent siblings)
17 siblings, 1 reply; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:47 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Use a special function to create kmalloc caches and use that function in
SLAB and SLUB.
V1->V2:
Do check for slasb state in slub's __kmem_cache_create to avoid
unlocking a lock that was not taken
V2->V3:
Remove slab_state check from sysfs_slab_add(). [Joonsoo]
V3->V4:
- Use %zd instead of %td for size info.
- Do not add slab caches to the list of slab caches during early
boot.
Acked-by: Joonsoo Kim <js1304@gmail.com>
Reviewed-by: Glauber Costa <glommer@parallels.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
---
mm/slab.c | 48 ++++++++++++++----------------------------------
mm/slab.h | 5 +++++
mm/slab_common.c | 32 ++++++++++++++++++++++++++++++++
mm/slub.c | 36 +++---------------------------------
4 files changed, 54 insertions(+), 67 deletions(-)
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 15:55:09.330845609 -0500
+++ linux/mm/slab.c 2012-11-01 15:55:09.342845776 -0500
@@ -1659,23 +1659,13 @@ void __init kmem_cache_init(void)
* bug.
*/
- sizes[INDEX_AC].cs_cachep = kmem_cache_zalloc(kmem_cache, GFP_NOWAIT);
- sizes[INDEX_AC].cs_cachep->name = names[INDEX_AC].name;
- sizes[INDEX_AC].cs_cachep->size = sizes[INDEX_AC].cs_size;
- sizes[INDEX_AC].cs_cachep->object_size = sizes[INDEX_AC].cs_size;
- sizes[INDEX_AC].cs_cachep->align = ARCH_KMALLOC_MINALIGN;
- __kmem_cache_create(sizes[INDEX_AC].cs_cachep, ARCH_KMALLOC_FLAGS|SLAB_PANIC);
- list_add(&sizes[INDEX_AC].cs_cachep->list, &slab_caches);
+ sizes[INDEX_AC].cs_cachep = create_kmalloc_cache(names[INDEX_AC].name,
+ sizes[INDEX_AC].cs_size, ARCH_KMALLOC_FLAGS);
- if (INDEX_AC != INDEX_L3) {
- sizes[INDEX_L3].cs_cachep = kmem_cache_zalloc(kmem_cache, GFP_NOWAIT);
- sizes[INDEX_L3].cs_cachep->name = names[INDEX_L3].name;
- sizes[INDEX_L3].cs_cachep->size = sizes[INDEX_L3].cs_size;
- sizes[INDEX_L3].cs_cachep->object_size = sizes[INDEX_L3].cs_size;
- sizes[INDEX_L3].cs_cachep->align = ARCH_KMALLOC_MINALIGN;
- __kmem_cache_create(sizes[INDEX_L3].cs_cachep, ARCH_KMALLOC_FLAGS|SLAB_PANIC);
- list_add(&sizes[INDEX_L3].cs_cachep->list, &slab_caches);
- }
+ if (INDEX_AC != INDEX_L3)
+ sizes[INDEX_L3].cs_cachep =
+ create_kmalloc_cache(names[INDEX_L3].name,
+ sizes[INDEX_L3].cs_size, ARCH_KMALLOC_FLAGS);
slab_early_init = 0;
@@ -1687,24 +1677,14 @@ void __init kmem_cache_init(void)
* Note for systems short on memory removing the alignment will
* allow tighter packing of the smaller caches.
*/
- if (!sizes->cs_cachep) {
- sizes->cs_cachep = kmem_cache_zalloc(kmem_cache, GFP_NOWAIT);
- sizes->cs_cachep->name = names->name;
- sizes->cs_cachep->size = sizes->cs_size;
- sizes->cs_cachep->object_size = sizes->cs_size;
- sizes->cs_cachep->align = ARCH_KMALLOC_MINALIGN;
- __kmem_cache_create(sizes->cs_cachep, ARCH_KMALLOC_FLAGS|SLAB_PANIC);
- list_add(&sizes->cs_cachep->list, &slab_caches);
- }
+ if (!sizes->cs_cachep)
+ sizes->cs_cachep = create_kmalloc_cache(names->name,
+ sizes->cs_size, ARCH_KMALLOC_FLAGS);
+
#ifdef CONFIG_ZONE_DMA
- sizes->cs_dmacachep = kmem_cache_zalloc(kmem_cache, GFP_NOWAIT);
- sizes->cs_dmacachep->name = names->name_dma;
- sizes->cs_dmacachep->size = sizes->cs_size;
- sizes->cs_dmacachep->object_size = sizes->cs_size;
- sizes->cs_dmacachep->align = ARCH_KMALLOC_MINALIGN;
- __kmem_cache_create(sizes->cs_dmacachep,
- ARCH_KMALLOC_FLAGS|SLAB_CACHE_DMA| SLAB_PANIC);
- list_add(&sizes->cs_dmacachep->list, &slab_caches);
+ sizes->cs_dmacachep = create_kmalloc_cache(
+ names->name_dma, sizes->cs_size,
+ SLAB_CACHE_DMA|ARCH_KMALLOC_FLAGS);
#endif
sizes++;
names++;
Index: linux/mm/slab.h
===================================================================
--- linux.orig/mm/slab.h 2012-11-01 15:54:17.926126373 -0500
+++ linux/mm/slab.h 2012-11-01 15:55:09.342845776 -0500
@@ -35,6 +35,11 @@ extern struct kmem_cache *kmem_cache;
/* Functions provided by the slab allocators */
extern int __kmem_cache_create(struct kmem_cache *, unsigned long flags);
+extern struct kmem_cache *create_kmalloc_cache(const char *name, size_t size,
+ unsigned long flags);
+extern void create_boot_cache(struct kmem_cache *, const char *name,
+ size_t size, unsigned long flags);
+
#ifdef CONFIG_SLUB
struct kmem_cache *__kmem_cache_alias(const char *name, size_t size,
size_t align, unsigned long flags, void (*ctor)(void *));
Index: linux/mm/slab_common.c
===================================================================
--- linux.orig/mm/slab_common.c 2012-11-01 15:54:17.926126373 -0500
+++ linux/mm/slab_common.c 2012-11-01 15:59:30.182494197 -0500
@@ -202,6 +202,42 @@ int slab_is_available(void)
return slab_state >= UP;
}
+#ifndef CONFIG_SLOB
+/* Create a cache during boot when no slab services are available yet */
+void __init create_boot_cache(struct kmem_cache *s, const char *name, size_t size,
+ unsigned long flags)
+{
+ int err;
+
+ s->name = name;
+ s->size = s->object_size = size;
+ s->align = ARCH_KMALLOC_MINALIGN;
+ err = __kmem_cache_create(s, flags);
+
+ if (err)
+ panic("Creation of kmalloc slab %s size=%zd failed. Reason %d\n",
+ name, size, err);
+
+ s->refcount = -1; /* Exempt from merging for now */
+}
+
+struct kmem_cache *__init create_kmalloc_cache(const char *name, size_t size,
+ unsigned long flags)
+{
+ struct kmem_cache *s = kmem_cache_zalloc(kmem_cache, GFP_NOWAIT);
+
+ if (!s)
+ panic("Out of memory when creating slab %s\n", name);
+
+ create_boot_cache(s, name, size, flags);
+ list_add(&s->list, &slab_caches);
+ s->refcount = 1;
+ return s;
+}
+
+#endif /* !CONFIG_SLOB */
+
+
#ifdef CONFIG_SLABINFO
static void print_slabinfo_header(struct seq_file *m)
{
Index: linux/mm/slub.c
===================================================================
--- linux.orig/mm/slub.c 2012-11-01 15:54:17.926126373 -0500
+++ linux/mm/slub.c 2012-11-01 15:58:33.293698642 -0500
@@ -3243,32 +3243,6 @@ static int __init setup_slub_nomerge(cha
__setup("slub_nomerge", setup_slub_nomerge);
-static struct kmem_cache *__init create_kmalloc_cache(const char *name,
- int size, unsigned int flags)
-{
- struct kmem_cache *s;
-
- s = kmem_cache_zalloc(kmem_cache, GFP_NOWAIT);
-
- s->name = name;
- s->size = s->object_size = size;
- s->align = ARCH_KMALLOC_MINALIGN;
-
- /*
- * This function is called with IRQs disabled during early-boot on
- * single CPU so there's no need to take slab_mutex here.
- */
- if (kmem_cache_open(s, flags))
- goto panic;
-
- list_add(&s->list, &slab_caches);
- return s;
-
-panic:
- panic("Creation of kmalloc slab %s size=%d failed.\n", name, size);
- return NULL;
-}
-
/*
* Conversion table for small slabs sizes / 8 to the index in the
* kmalloc array. This is necessary for slabs < 192 since we have non power
@@ -3946,6 +3920,10 @@ int __kmem_cache_create(struct kmem_cach
if (err)
return err;
+ /* Mutex is not taken during early boot */
+ if (slab_state <= UP)
+ return 0;
+
mutex_unlock(&slab_mutex);
err = sysfs_slab_add(s);
mutex_lock(&slab_mutex);
@@ -5247,13 +5225,8 @@ static int sysfs_slab_add(struct kmem_ca
{
int err;
const char *name;
- int unmergeable;
-
- if (slab_state < FULL)
- /* Defer until later */
- return 0;
+ int unmergeable = slab_unmergeable(s);
- unmergeable = slab_unmergeable(s);
if (unmergeable) {
/*
* Slabcache can never be merged so we can use the name proper.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [05/18] slab: Use the new create_boot_cache function to simplify bootstrap
[not found] <20121101214538.971500204@linux.com>
2012-11-01 21:46 ` CK5 [02/18] slab: Simplify bootstrap Christoph Lameter
2012-11-01 21:46 ` CK5 [01/18] Use correct cpu_slab on dead cpu Christoph Lameter
@ 2012-11-01 21:47 ` Christoph Lameter
2012-11-01 21:47 ` CK5 [04/18] slub: Use a statically allocated kmem_cache boot structure for bootstrap Christoph Lameter
` (14 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:47 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Simplify setup and reduce code in kmem_cache_init(). This allows us to
get rid of initarray_cache as well as the manual setup code for
the kmem_cache and kmem_cache_node arrays during bootstrap.
We introduce a new bootstrap state "PARTIAL" for slab that signals the
creation of a kmem_cache boot cache.
V1->V2: Get rid of initarray_cache as well.
V2->V3: Drop the setting of slab_state to PARTIAL [glommer]
Signed-off-by: Christoph Lameter <cl@linux.com>
---
mm/slab.c | 51 ++++++++++++++++++---------------------------------
1 file changed, 18 insertions(+), 33 deletions(-)
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 16:33:23.943008612 -0500
+++ linux/mm/slab.c 2012-11-01 16:39:57.844520127 -0500
@@ -547,8 +547,6 @@ static struct cache_names __initdata cac
#undef CACHE
};
-static struct arraycache_init initarray_cache __initdata =
- { {0, BOOT_CPUCACHE_ENTRIES, 1, 0} };
static struct arraycache_init initarray_generic =
{ {0, BOOT_CPUCACHE_ENTRIES, 1, 0} };
@@ -1572,12 +1570,9 @@ static void setup_nodelists_pointer(stru
*/
void __init kmem_cache_init(void)
{
- size_t left_over;
struct cache_sizes *sizes;
struct cache_names *names;
int i;
- int order;
- int node;
kmem_cache = &kmem_cache_boot;
setup_nodelists_pointer(kmem_cache);
@@ -1618,36 +1613,16 @@ void __init kmem_cache_init(void)
* 6) Resize the head arrays of the kmalloc caches to their final sizes.
*/
- node = numa_mem_id();
-
/* 1) create the kmem_cache */
- INIT_LIST_HEAD(&slab_caches);
- list_add(&kmem_cache->list, &slab_caches);
- kmem_cache->colour_off = cache_line_size();
- kmem_cache->array[smp_processor_id()] = &initarray_cache.cache;
/*
* struct kmem_cache size depends on nr_node_ids & nr_cpu_ids
*/
- kmem_cache->size = offsetof(struct kmem_cache, array[nr_cpu_ids]) +
- nr_node_ids * sizeof(struct kmem_list3 *);
- kmem_cache->object_size = kmem_cache->size;
- kmem_cache->size = ALIGN(kmem_cache->object_size,
- cache_line_size());
- kmem_cache->reciprocal_buffer_size =
- reciprocal_value(kmem_cache->size);
-
- for (order = 0; order < MAX_ORDER; order++) {
- cache_estimate(order, kmem_cache->size,
- cache_line_size(), 0, &left_over, &kmem_cache->num);
- if (kmem_cache->num)
- break;
- }
- BUG_ON(!kmem_cache->num);
- kmem_cache->gfporder = order;
- kmem_cache->colour = left_over / kmem_cache->colour_off;
- kmem_cache->slab_size = ALIGN(kmem_cache->num * sizeof(kmem_bufctl_t) +
- sizeof(struct slab), cache_line_size());
+ create_boot_cache(kmem_cache, "kmem_cache",
+ offsetof(struct kmem_cache, array[nr_cpu_ids]) +
+ nr_node_ids * sizeof(struct kmem_list3 *),
+ SLAB_HWCACHE_ALIGN);
+ list_add(&kmem_cache->list, &slab_caches);
/* 2+3) create the kmalloc caches */
sizes = malloc_sizes;
@@ -1695,7 +1670,6 @@ void __init kmem_cache_init(void)
ptr = kmalloc(sizeof(struct arraycache_init), GFP_NOWAIT);
- BUG_ON(cpu_cache_get(kmem_cache) != &initarray_cache.cache);
memcpy(ptr, cpu_cache_get(kmem_cache),
sizeof(struct arraycache_init));
/*
@@ -2250,7 +2224,15 @@ static int __init_refok setup_cpu_cache(
if (slab_state == DOWN) {
/*
- * Note: the first kmem_cache_create must create the cache
+ * Note: Creation of first cache (kmem_cache).
+ * The setup_list3s is taken care
+ * of by the caller of __kmem_cache_create
+ */
+ cachep->array[smp_processor_id()] = &initarray_generic.cache;
+ slab_state = PARTIAL;
+ } else if (slab_state == PARTIAL) {
+ /*
+ * Note: the second kmem_cache_create must create the cache
* that's used by kmalloc(24), otherwise the creation of
* further caches will BUG().
*/
@@ -2258,7 +2240,7 @@ static int __init_refok setup_cpu_cache(
/*
* If the cache that's used by kmalloc(sizeof(kmem_list3)) is
- * the first cache, then we need to set up all its list3s,
+ * the second cache, then we need to set up all its list3s,
* otherwise the creation of further caches will BUG().
*/
set_up_list3s(cachep, SIZE_AC);
@@ -2267,6 +2249,7 @@ static int __init_refok setup_cpu_cache(
else
slab_state = PARTIAL_ARRAYCACHE;
} else {
+ /* Remaining boot caches */
cachep->array[smp_processor_id()] =
kmalloc(sizeof(struct arraycache_init), gfp);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [07/18] Move kmalloc related function defs
[not found] <20121101214538.971500204@linux.com>
` (6 preceding siblings ...)
2012-11-01 21:47 ` CK5 [06/18] Common alignment code Christoph Lameter
@ 2012-11-01 21:47 ` Christoph Lameter
2012-11-01 21:48 ` CK5 [15/18] stat: Use size_t for sizes instead of unsigned Christoph Lameter
` (9 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:47 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Move these functions higher up in slab.h so that they are grouped with other
generic kmalloc related definitions.
Acked-by: Glauber Costa <glommer@parallels.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/slab.h
===================================================================
--- linux.orig/include/linux/slab.h 2012-11-01 10:09:46.745412498 -0500
+++ linux/include/linux/slab.h 2012-11-01 10:10:24.838044511 -0500
@@ -142,6 +142,15 @@ void kmem_cache_free(struct kmem_cache *
(__flags), NULL)
/*
+ * Common kmalloc functions provided by all allocators
+ */
+void * __must_check __krealloc(const void *, size_t, gfp_t);
+void * __must_check krealloc(const void *, size_t, gfp_t);
+void kfree(const void *);
+void kzfree(const void *);
+size_t ksize(const void *);
+
+/*
* The largest kmalloc size supported by the slab allocators is
* 32 megabyte (2^25) or the maximum allocatable page order if that is
* less than 32 MB.
@@ -177,15 +186,6 @@ void kmem_cache_free(struct kmem_cache *
#endif
/*
- * Common kmalloc functions provided by all allocators
- */
-void * __must_check __krealloc(const void *, size_t, gfp_t);
-void * __must_check krealloc(const void *, size_t, gfp_t);
-void kfree(const void *);
-void kzfree(const void *);
-size_t ksize(const void *);
-
-/*
* Allocator specific definitions. These are mainly used to establish optimized
* ways to convert kmalloc() calls to kmem_cache_alloc() invocations by
* selecting the appropriate general cache at compile time.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [12/18] Common constants for kmalloc boundaries
[not found] <20121101214538.971500204@linux.com>
` (4 preceding siblings ...)
2012-11-01 21:47 ` CK5 [03/18] create common functions for boot slab creation Christoph Lameter
@ 2012-11-01 21:47 ` Christoph Lameter
2012-11-01 21:47 ` CK5 [06/18] Common alignment code Christoph Lameter
` (11 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:47 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Standardize the constants that describe the smallest and largest
object kept in the kmalloc arrays for SLAB and SLUB.
Differentiate between the maximum size for which a slab cache is used
(KMALLOC_MAX_CACHE_SIZE) and the maximum allocatable size
(KMALLOC_MAX_SIZE, KMALLOC_MAX_ORDER).
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/slab.h
===================================================================
--- linux.orig/include/linux/slab.h 2012-11-01 16:12:19.909289011 -0500
+++ linux/include/linux/slab.h 2012-11-01 16:12:25.145362535 -0500
@@ -156,7 +156,12 @@ struct kmem_cache {
#else /* CONFIG_SLOB */
/*
- * The largest kmalloc size supported by the slab allocators is
+ * Kmalloc array related definitions
+ */
+
+#ifdef CONFIG_SLAB
+/*
+ * The largest kmalloc size supported by the SLAB allocators is
* 32 megabyte (2^25) or the maximum allocatable page order if that is
* less than 32 MB.
*
@@ -166,9 +171,24 @@ struct kmem_cache {
*/
#define KMALLOC_SHIFT_HIGH ((MAX_ORDER + PAGE_SHIFT - 1) <= 25 ? \
(MAX_ORDER + PAGE_SHIFT - 1) : 25)
+#define KMALLOC_SHIFT_MAX KMALLOC_SHIFT_HIGH
+#define KMALLOC_SHIFT_LOW 5
+#else
+/*
+ * SLUB allocates up to order 2 pages directly and otherwise
+ * passes the request to the page allocator.
+ */
+#define KMALLOC_SHIFT_HIGH (PAGE_SHIFT + 1)
+#define KMALLOC_SHIFT_MAX (MAX_ORDER + PAGE_SHIFT)
+#define KMALLOC_SHIFT_LOW 3
+#endif
-#define KMALLOC_MAX_SIZE (1UL << KMALLOC_SHIFT_HIGH)
-#define KMALLOC_MAX_ORDER (KMALLOC_SHIFT_HIGH - PAGE_SHIFT)
+/* Maximum allocatable size */
+#define KMALLOC_MAX_SIZE (1UL << KMALLOC_SHIFT_MAX)
+/* Maximum size for which we actually use a slab cache */
+#define KMALLOC_MAX_CACHE_SIZE (1UL << KMALLOC_SHIFT_HIGH)
+/* Maximum order allocatable via the slab allocagtor */
+#define KMALLOC_MAX_ORDER (KMALLOC_SHIFT_MAX - PAGE_SHIFT)
/*
* Kmalloc subsystem.
@@ -176,15 +196,9 @@ struct kmem_cache {
#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
#else
-#ifdef CONFIG_SLAB
-#define KMALLOC_MIN_SIZE 32
-#else
-#define KMALLOC_MIN_SIZE 8
-#endif
+#define KMALLOC_MIN_SIZE (1 << KMALLOC_SHIFT_LOW)
#endif
-#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
-
/*
* Figure out which kmalloc slab an allocation of a certain size
* belongs to.
Index: linux/include/linux/slub_def.h
===================================================================
--- linux.orig/include/linux/slub_def.h 2012-11-01 16:12:19.909289011 -0500
+++ linux/include/linux/slub_def.h 2012-11-01 16:12:25.145362535 -0500
@@ -111,19 +111,6 @@ struct kmem_cache {
struct kmem_cache_node *node[MAX_NUMNODES];
};
-/*
- * Maximum kmalloc object size handled by SLUB. Larger object allocations
- * are passed through to the page allocator. The page allocator "fastpath"
- * is relatively slow so we need this value sufficiently high so that
- * performance critical objects are allocated through the SLUB fastpath.
- *
- * This should be dropped to PAGE_SIZE / 2 once the page allocator
- * "fastpath" becomes competitive with the slab allocator fastpaths.
- */
-#define SLUB_MAX_SIZE (2 * PAGE_SIZE)
-
-#define SLUB_PAGE_SHIFT (PAGE_SHIFT + 2)
-
#ifdef CONFIG_ZONE_DMA
#define SLUB_DMA __GFP_DMA
#else
@@ -135,7 +122,7 @@ struct kmem_cache {
* We keep the general caches in an array of slab caches that are used for
* 2^x bytes of allocations.
*/
-extern struct kmem_cache *kmalloc_caches[SLUB_PAGE_SHIFT];
+extern struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1];
/*
* Find the slab cache for a given combination of allocation flags and size.
@@ -204,7 +191,7 @@ static __always_inline void *kmalloc_lar
static __always_inline void *kmalloc(size_t size, gfp_t flags)
{
if (__builtin_constant_p(size)) {
- if (size > SLUB_MAX_SIZE)
+ if (size > KMALLOC_MAX_CACHE_SIZE)
return kmalloc_large(size, flags);
if (!(flags & SLUB_DMA)) {
@@ -240,7 +227,7 @@ kmem_cache_alloc_node_trace(struct kmem_
static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node)
{
if (__builtin_constant_p(size) &&
- size <= SLUB_MAX_SIZE && !(flags & SLUB_DMA)) {
+ size <= KMALLOC_MAX_CACHE_SIZE && !(flags & SLUB_DMA)) {
struct kmem_cache *s = kmalloc_slab(size);
if (!s)
Index: linux/mm/slub.c
===================================================================
--- linux.orig/mm/slub.c 2012-11-01 16:12:12.705187852 -0500
+++ linux/mm/slub.c 2012-11-01 16:12:25.145362535 -0500
@@ -2774,7 +2774,7 @@ init_kmem_cache_node(struct kmem_cache_n
static inline int alloc_kmem_cache_cpus(struct kmem_cache *s)
{
BUILD_BUG_ON(PERCPU_DYNAMIC_EARLY_SIZE <
- SLUB_PAGE_SHIFT * sizeof(struct kmem_cache_cpu));
+ KMALLOC_SHIFT_HIGH * sizeof(struct kmem_cache_cpu));
/*
* Must align to double word boundary for the double cmpxchg
@@ -3162,11 +3162,11 @@ int __kmem_cache_shutdown(struct kmem_ca
* Kmalloc subsystem
*******************************************************************/
-struct kmem_cache *kmalloc_caches[SLUB_PAGE_SHIFT];
+struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1];
EXPORT_SYMBOL(kmalloc_caches);
#ifdef CONFIG_ZONE_DMA
-static struct kmem_cache *kmalloc_dma_caches[SLUB_PAGE_SHIFT];
+static struct kmem_cache *kmalloc_dma_caches[KMALLOC_SHIFT_HIGH + 1];
#endif
static int __init setup_slub_min_order(char *str)
@@ -3268,7 +3268,7 @@ void *__kmalloc(size_t size, gfp_t flags
struct kmem_cache *s;
void *ret;
- if (unlikely(size > SLUB_MAX_SIZE))
+ if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
return kmalloc_large(size, flags);
s = get_slab(size, flags);
@@ -3304,7 +3304,7 @@ void *__kmalloc_node(size_t size, gfp_t
struct kmem_cache *s;
void *ret;
- if (unlikely(size > SLUB_MAX_SIZE)) {
+ if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
ret = kmalloc_large_node(size, flags, node);
trace_kmalloc_node(_RET_IP_, ret,
@@ -3709,7 +3709,7 @@ void __init kmem_cache_init(void)
caches++;
}
- for (i = KMALLOC_SHIFT_LOW; i < SLUB_PAGE_SHIFT; i++) {
+ for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) {
kmalloc_caches[i] = create_kmalloc_cache("kmalloc", 1 << i, 0);
caches++;
}
@@ -3727,7 +3727,7 @@ void __init kmem_cache_init(void)
BUG_ON(!kmalloc_caches[2]->name);
}
- for (i = KMALLOC_SHIFT_LOW; i < SLUB_PAGE_SHIFT; i++) {
+ for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) {
char *s = kasprintf(GFP_NOWAIT, "kmalloc-%d", 1 << i);
BUG_ON(!s);
@@ -3739,7 +3739,7 @@ void __init kmem_cache_init(void)
#endif
#ifdef CONFIG_ZONE_DMA
- for (i = 0; i < SLUB_PAGE_SHIFT; i++) {
+ for (i = 0; i <= KMALLOC_SHIFT_HIGH; i++) {
struct kmem_cache *s = kmalloc_caches[i];
if (s && s->size) {
@@ -3913,7 +3913,7 @@ void *__kmalloc_track_caller(size_t size
struct kmem_cache *s;
void *ret;
- if (unlikely(size > SLUB_MAX_SIZE))
+ if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
return kmalloc_large(size, gfpflags);
s = get_slab(size, gfpflags);
@@ -3936,7 +3936,7 @@ void *__kmalloc_node_track_caller(size_t
struct kmem_cache *s;
void *ret;
- if (unlikely(size > SLUB_MAX_SIZE)) {
+ if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) {
ret = kmalloc_large_node(size, gfpflags, node);
trace_kmalloc_node(caller, ret,
@@ -4295,7 +4295,7 @@ static void resiliency_test(void)
{
u8 *p;
- BUILD_BUG_ON(KMALLOC_MIN_SIZE > 16 || SLUB_PAGE_SHIFT < 10);
+ BUILD_BUG_ON(KMALLOC_MIN_SIZE > 16 || KMALLOC_SHIFT_HIGH < 10);
printk(KERN_ERR "SLUB resiliency testing\n");
printk(KERN_ERR "-----------------------\n");
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [06/18] Common alignment code
[not found] <20121101214538.971500204@linux.com>
` (5 preceding siblings ...)
2012-11-01 21:47 ` CK5 [12/18] Common constants for kmalloc boundaries Christoph Lameter
@ 2012-11-01 21:47 ` Christoph Lameter
2012-11-01 21:47 ` CK5 [07/18] Move kmalloc related function defs Christoph Lameter
` (10 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:47 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Extract the code to do object alignment from the allocators.
Do the alignment calculations in slab_common so that the
__kmem_cache_create functions of the allocators do not have
to deal with alignment.
Signed-off-by: Christoph Lameter <cl@linux.com>
---
mm/slab.c | 20 --------------------
mm/slab.h | 3 +++
mm/slab_common.c | 32 ++++++++++++++++++++++++++++++--
mm/slob.c | 10 ----------
mm/slub.c | 38 +-------------------------------------
5 files changed, 34 insertions(+), 69 deletions(-)
Index: linux/mm/slab.h
===================================================================
--- linux.orig/mm/slab.h 2012-11-01 16:09:25.762843113 -0500
+++ linux/mm/slab.h 2012-11-01 16:12:12.705187852 -0500
@@ -32,6 +32,9 @@ extern struct list_head slab_caches;
/* The slab cache that manages slab cache information */
extern struct kmem_cache *kmem_cache;
+unsigned long calculate_alignment(unsigned long flags,
+ unsigned long align, unsigned long size);
+
/* Functions provided by the slab allocators */
extern int __kmem_cache_create(struct kmem_cache *, unsigned long flags);
Index: linux/mm/slab_common.c
===================================================================
--- linux.orig/mm/slab_common.c 2012-11-01 16:09:25.762843113 -0500
+++ linux/mm/slab_common.c 2012-11-01 16:12:12.705187852 -0500
@@ -73,6 +73,34 @@ static inline int kmem_cache_sanity_chec
#endif
/*
+ * Figure out what the alignment of the objects will be given a set of
+ * flags, a user specified alignment and the size of the objects.
+ */
+unsigned long calculate_alignment(unsigned long flags,
+ unsigned long align, unsigned long size)
+{
+ /*
+ * If the user wants hardware cache aligned objects then follow that
+ * suggestion if the object is sufficiently large.
+ *
+ * The hardware cache alignment cannot override the specified
+ * alignment though. If that is greater then use it.
+ */
+ if (flags & SLAB_HWCACHE_ALIGN) {
+ unsigned long ralign = cache_line_size();
+ while (size <= ralign / 2)
+ ralign /= 2;
+ align = max(align, ralign);
+ }
+
+ if (align < ARCH_SLAB_MINALIGN)
+ align = ARCH_SLAB_MINALIGN;
+
+ return ALIGN(align, sizeof(void *));
+}
+
+
+/*
* kmem_cache_create - Create a cache.
* @name: A string which is used in /proc/slabinfo to identify this cache.
* @size: The size of objects to be created in this cache.
@@ -124,7 +152,7 @@ struct kmem_cache *kmem_cache_create(con
s = kmem_cache_zalloc(kmem_cache, GFP_KERNEL);
if (s) {
s->object_size = s->size = size;
- s->align = align;
+ s->align = calculate_alignment(flags, align, size);
s->ctor = ctor;
s->name = kstrdup(name, GFP_KERNEL);
if (!s->name) {
@@ -211,7 +239,7 @@ void __init create_boot_cache(struct kme
s->name = name;
s->size = s->object_size = size;
- s->align = ARCH_KMALLOC_MINALIGN;
+ s->align = calculate_alignment(flags, ARCH_KMALLOC_MINALIGN, size);
err = __kmem_cache_create(s, flags);
if (err)
Index: linux/mm/slob.c
===================================================================
--- linux.orig/mm/slob.c 2012-11-01 16:09:25.762843113 -0500
+++ linux/mm/slob.c 2012-11-01 16:12:12.705187852 -0500
@@ -123,7 +123,6 @@ static inline void clear_slob_page_free(
#define SLOB_UNIT sizeof(slob_t)
#define SLOB_UNITS(size) (((size) + SLOB_UNIT - 1)/SLOB_UNIT)
-#define SLOB_ALIGN L1_CACHE_BYTES
/*
* struct slob_rcu is inserted at the tail of allocated slob blocks, which
@@ -527,20 +526,11 @@ EXPORT_SYMBOL(ksize);
int __kmem_cache_create(struct kmem_cache *c, unsigned long flags)
{
- size_t align = c->size;
-
if (flags & SLAB_DESTROY_BY_RCU) {
/* leave room for rcu footer at the end of object */
c->size += sizeof(struct slob_rcu);
}
c->flags = flags;
- /* ignore alignment unless it's forced */
- c->align = (flags & SLAB_HWCACHE_ALIGN) ? SLOB_ALIGN : 0;
- if (c->align < ARCH_SLAB_MINALIGN)
- c->align = ARCH_SLAB_MINALIGN;
- if (c->align < align)
- c->align = align;
-
return 0;
}
Index: linux/mm/slub.c
===================================================================
--- linux.orig/mm/slub.c 2012-11-01 16:11:31.068603160 -0500
+++ linux/mm/slub.c 2012-11-01 16:12:12.705187852 -0500
@@ -2758,32 +2758,6 @@ static inline int calculate_order(int si
return -ENOSYS;
}
-/*
- * Figure out what the alignment of the objects will be.
- */
-static unsigned long calculate_alignment(unsigned long flags,
- unsigned long align, unsigned long size)
-{
- /*
- * If the user wants hardware cache aligned objects then follow that
- * suggestion if the object is sufficiently large.
- *
- * The hardware cache alignment cannot override the specified
- * alignment though. If that is greater then use it.
- */
- if (flags & SLAB_HWCACHE_ALIGN) {
- unsigned long ralign = cache_line_size();
- while (size <= ralign / 2)
- ralign /= 2;
- align = max(align, ralign);
- }
-
- if (align < ARCH_SLAB_MINALIGN)
- align = ARCH_SLAB_MINALIGN;
-
- return ALIGN(align, sizeof(void *));
-}
-
static void
init_kmem_cache_node(struct kmem_cache_node *n)
{
@@ -2917,7 +2891,6 @@ static int calculate_sizes(struct kmem_c
{
unsigned long flags = s->flags;
unsigned long size = s->object_size;
- unsigned long align = s->align;
int order;
/*
@@ -2989,19 +2962,11 @@ static int calculate_sizes(struct kmem_c
#endif
/*
- * Determine the alignment based on various parameters that the
- * user specified and the dynamic determination of cache line size
- * on bootup.
- */
- align = calculate_alignment(flags, align, s->object_size);
- s->align = align;
-
- /*
* SLUB stores one object immediately after another beginning from
* offset 0. In order to align the objects we have to simply size
* each object to conform to the alignment.
*/
- size = ALIGN(size, align);
+ size = ALIGN(size, s->align);
s->size = size;
if (forced_order >= 0)
order = forced_order;
@@ -3030,7 +2995,6 @@ static int calculate_sizes(struct kmem_c
s->max = s->oo;
return !!oo_objects(s->oo);
-
}
static int kmem_cache_open(struct kmem_cache *s, unsigned long flags)
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 16:12:10.601158311 -0500
+++ linux/mm/slab.c 2012-11-01 16:12:12.705187852 -0500
@@ -2339,22 +2339,6 @@ __kmem_cache_create (struct kmem_cache *
size &= ~(BYTES_PER_WORD - 1);
}
- /* calculate the final buffer alignment: */
-
- /* 1) arch recommendation: can be overridden for debug */
- if (flags & SLAB_HWCACHE_ALIGN) {
- /*
- * Default alignment: as specified by the arch code. Except if
- * an object is really small, then squeeze multiple objects into
- * one cacheline.
- */
- ralign = cache_line_size();
- while (size <= ralign / 2)
- ralign /= 2;
- } else {
- ralign = BYTES_PER_WORD;
- }
-
/*
* Redzoning and user store require word alignment or possibly larger.
* Note this will be overridden by architecture or caller mandated
@@ -2371,10 +2355,6 @@ __kmem_cache_create (struct kmem_cache *
size &= ~(REDZONE_ALIGN - 1);
}
- /* 2) arch mandated alignment */
- if (ralign < ARCH_SLAB_MINALIGN) {
- ralign = ARCH_SLAB_MINALIGN;
- }
/* 3) caller mandated alignment */
if (ralign < cachep->align) {
ralign = cachep->align;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [15/18] stat: Use size_t for sizes instead of unsigned
[not found] <20121101214538.971500204@linux.com>
` (7 preceding siblings ...)
2012-11-01 21:47 ` CK5 [07/18] Move kmalloc related function defs Christoph Lameter
@ 2012-11-01 21:48 ` Christoph Lameter
2012-11-01 21:48 ` CK5 [11/18] slab: rename nodelists to node Christoph Lameter
` (8 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:48 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
On some platforms (such as IA64) the large page size may results in
slab allocations to be allowed of numbers that do not fit in 32 bit.
Acked-by: Glauber Costa <glommer@parallels.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/fs/proc/stat.c
===================================================================
--- linux.orig/fs/proc/stat.c 2012-11-01 10:09:46.221403795 -0500
+++ linux/fs/proc/stat.c 2012-11-01 11:19:37.164785019 -0500
@@ -184,7 +184,7 @@ static int show_stat(struct seq_file *p,
static int stat_open(struct inode *inode, struct file *file)
{
- unsigned size = 1024 + 128 * num_possible_cpus();
+ size_t size = 1024 + 128 * num_possible_cpus();
char *buf;
struct seq_file *m;
int res;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [13/18] Common definition for the array of kmalloc caches
[not found] <20121101214538.971500204@linux.com>
` (9 preceding siblings ...)
2012-11-01 21:48 ` CK5 [11/18] slab: rename nodelists to node Christoph Lameter
@ 2012-11-01 21:48 ` Christoph Lameter
2012-11-01 21:48 ` CK5 [14/18] Common function to create the kmalloc array Christoph Lameter
` (6 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:48 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Have a common definition fo the kmalloc cache arrays in
SLAB and SLUB
Acked-by: Glauber Costa <glommer@parallels.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/mm/slab_common.c
===================================================================
--- linux.orig/mm/slab_common.c 2012-11-01 10:10:21.801994185 -0500
+++ linux/mm/slab_common.c 2012-11-01 10:10:42.822342471 -0500
@@ -263,6 +263,14 @@ struct kmem_cache *__init create_kmalloc
return s;
}
+struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1];
+EXPORT_SYMBOL(kmalloc_caches);
+
+#ifdef CONFIG_ZONE_DMA
+struct kmem_cache *kmalloc_dma_caches[KMALLOC_SHIFT_HIGH + 1];
+EXPORT_SYMBOL(kmalloc_dma_caches);
+#endif
+
#endif /* !CONFIG_SLOB */
Index: linux/include/linux/slub_def.h
===================================================================
--- linux.orig/include/linux/slub_def.h 2012-11-01 10:10:39.866293513 -0500
+++ linux/include/linux/slub_def.h 2012-11-01 10:10:42.826342537 -0500
@@ -119,12 +119,6 @@ struct kmem_cache {
#endif
/*
- * We keep the general caches in an array of slab caches that are used for
- * 2^x bytes of allocations.
- */
-extern struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1];
-
-/*
* Find the slab cache for a given combination of allocation flags and size.
*
* This ought to end up with a global pointer to the right cache
Index: linux/mm/slub.c
===================================================================
--- linux.orig/mm/slub.c 2012-11-01 10:10:39.866293513 -0500
+++ linux/mm/slub.c 2012-11-01 10:10:42.826342537 -0500
@@ -3164,13 +3164,6 @@ int __kmem_cache_shutdown(struct kmem_ca
* Kmalloc subsystem
*******************************************************************/
-struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1];
-EXPORT_SYMBOL(kmalloc_caches);
-
-#ifdef CONFIG_ZONE_DMA
-static struct kmem_cache *kmalloc_dma_caches[KMALLOC_SHIFT_HIGH + 1];
-#endif
-
static int __init setup_slub_min_order(char *str)
{
get_option(&str, &slub_min_order);
Index: linux/include/linux/slab.h
===================================================================
--- linux.orig/include/linux/slab.h 2012-11-01 10:10:39.866293513 -0500
+++ linux/include/linux/slab.h 2012-11-01 10:10:42.826342537 -0500
@@ -199,6 +199,11 @@ struct kmem_cache {
#define KMALLOC_MIN_SIZE (1 << KMALLOC_SHIFT_LOW)
#endif
+extern struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1];
+#ifdef CONFIG_ZONE_DMA
+extern struct kmem_cache *kmalloc_dma_caches[KMALLOC_SHIFT_HIGH + 1];
+#endif
+
/*
* Figure out which kmalloc slab an allocation of a certain size
* belongs to.
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 10:10:36.938245015 -0500
+++ linux/mm/slab.c 2012-11-01 10:10:42.826342537 -0500
@@ -317,14 +317,6 @@ static void free_block(struct kmem_cache
static int enable_cpucache(struct kmem_cache *cachep, gfp_t gfp);
static void cache_reap(struct work_struct *unused);
-struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1];
-EXPORT_SYMBOL(kmalloc_caches);
-
-#ifdef CONFIG_ZONE_DMA
-struct kmem_cache *kmalloc_dma_caches[KMALLOC_SHIFT_HIGH + 1];
-EXPORT_SYMBOL(kmalloc_dma_caches);
-#endif
-
static int slab_early_init = 1;
#define INDEX_AC kmalloc_index(sizeof(struct arraycache_init))
Index: linux/include/linux/slab_def.h
===================================================================
--- linux.orig/include/linux/slab_def.h 2012-11-01 10:10:36.938245015 -0500
+++ linux/include/linux/slab_def.h 2012-11-01 10:10:42.826342537 -0500
@@ -95,9 +95,6 @@ struct kmem_cache {
*/
};
-extern struct kmem_cache *kmalloc_caches[PAGE_SHIFT + MAX_ORDER];
-extern struct kmem_cache *kmalloc_dma_caches[PAGE_SHIFT + MAX_ORDER];
-
void *kmem_cache_alloc(struct kmem_cache *, gfp_t);
void *__kmalloc(size_t size, gfp_t flags);
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [11/18] slab: rename nodelists to node
[not found] <20121101214538.971500204@linux.com>
` (8 preceding siblings ...)
2012-11-01 21:48 ` CK5 [15/18] stat: Use size_t for sizes instead of unsigned Christoph Lameter
@ 2012-11-01 21:48 ` Christoph Lameter
2012-11-01 21:48 ` CK5 [13/18] Common definition for the array of kmalloc caches Christoph Lameter
` (7 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:48 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Have a common naming between both slab caches for future changes.
Acked-by: Glauber Costa <glommer@parallels.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/slab_def.h
===================================================================
--- linux.orig/include/linux/slab_def.h 2012-11-01 10:10:33.446187162 -0500
+++ linux/include/linux/slab_def.h 2012-11-01 10:10:36.938245015 -0500
@@ -88,7 +88,7 @@ struct kmem_cache {
* We still use [NR_CPUS] and not [1] or [0] because cache_cache
* is statically defined, so we reserve the max number of cpus.
*/
- struct kmem_cache_node **nodelists;
+ struct kmem_cache_node **node;
struct array_cache *array[NR_CPUS + MAX_NUMNODES];
/*
* Do not add fields after array[]
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 10:10:33.450187230 -0500
+++ linux/mm/slab.c 2012-11-01 10:10:36.938245015 -0500
@@ -346,7 +346,7 @@ static void kmem_list3_init(struct kmem_
#define MAKE_LIST(cachep, listp, slab, nodeid) \
do { \
INIT_LIST_HEAD(listp); \
- list_splice(&(cachep->nodelists[nodeid]->slab), listp); \
+ list_splice(&(cachep->node[nodeid]->slab), listp); \
} while (0)
#define MAKE_ALL_LISTS(cachep, ptr, nodeid) \
@@ -548,7 +548,7 @@ static void slab_set_lock_classes(struct
struct kmem_cache_node *l3;
int r;
- l3 = cachep->nodelists[q];
+ l3 = cachep->node[q];
if (!l3)
return;
@@ -596,7 +596,7 @@ static void init_node_lock_keys(int q)
if (!cache)
continue;
- l3 = cache->nodelists[q];
+ l3 = cache->node[q];
if (!l3 || OFF_SLAB(cache))
continue;
@@ -872,7 +872,7 @@ static inline bool is_slab_pfmemalloc(st
static void recheck_pfmemalloc_active(struct kmem_cache *cachep,
struct array_cache *ac)
{
- struct kmem_cache_node *l3 = cachep->nodelists[numa_mem_id()];
+ struct kmem_cache_node *l3 = cachep->node[numa_mem_id()];
struct slab *slabp;
unsigned long flags;
@@ -927,7 +927,7 @@ static void *__ac_get_obj(struct kmem_ca
* If there are empty slabs on the slabs_free list and we are
* being forced to refill the cache, mark this one !pfmemalloc.
*/
- l3 = cachep->nodelists[numa_mem_id()];
+ l3 = cachep->node[numa_mem_id()];
if (!list_empty(&l3->slabs_free) && force_refill) {
struct slab *slabp = virt_to_slab(objp);
ClearPageSlabPfmemalloc(virt_to_head_page(slabp->s_mem));
@@ -1077,7 +1077,7 @@ static void free_alien_cache(struct arra
static void __drain_alien_cache(struct kmem_cache *cachep,
struct array_cache *ac, int node)
{
- struct kmem_cache_node *rl3 = cachep->nodelists[node];
+ struct kmem_cache_node *rl3 = cachep->node[node];
if (ac->avail) {
spin_lock(&rl3->list_lock);
@@ -1146,7 +1146,7 @@ static inline int cache_free_alien(struc
if (likely(slabp->nodeid == node))
return 0;
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
STATS_INC_NODEFREES(cachep);
if (l3->alien && l3->alien[nodeid]) {
alien = l3->alien[nodeid];
@@ -1158,24 +1158,24 @@ static inline int cache_free_alien(struc
ac_put_obj(cachep, alien, objp);
spin_unlock(&alien->lock);
} else {
- spin_lock(&(cachep->nodelists[nodeid])->list_lock);
+ spin_lock(&(cachep->node[nodeid])->list_lock);
free_block(cachep, &objp, 1, nodeid);
- spin_unlock(&(cachep->nodelists[nodeid])->list_lock);
+ spin_unlock(&(cachep->node[nodeid])->list_lock);
}
return 1;
}
#endif
/*
- * Allocates and initializes nodelists for a node on each slab cache, used for
+ * Allocates and initializes node for a node on each slab cache, used for
* either memory or cpu hotplug. If memory is being hot-added, the kmem_list3
* will be allocated off-node since memory is not yet online for the new node.
- * When hotplugging memory or a cpu, existing nodelists are not replaced if
+ * When hotplugging memory or a cpu, existing node are not replaced if
* already in use.
*
* Must hold slab_mutex.
*/
-static int init_cache_nodelists_node(int node)
+static int init_cache_node_node(int node)
{
struct kmem_cache *cachep;
struct kmem_cache_node *l3;
@@ -1187,7 +1187,7 @@ static int init_cache_nodelists_node(int
* begin anything. Make sure some other cpu on this
* node has not already allocated this
*/
- if (!cachep->nodelists[node]) {
+ if (!cachep->node[node]) {
l3 = kmalloc_node(memsize, GFP_KERNEL, node);
if (!l3)
return -ENOMEM;
@@ -1200,14 +1200,14 @@ static int init_cache_nodelists_node(int
* go. slab_mutex is sufficient
* protection here.
*/
- cachep->nodelists[node] = l3;
+ cachep->node[node] = l3;
}
- spin_lock_irq(&cachep->nodelists[node]->list_lock);
- cachep->nodelists[node]->free_limit =
+ spin_lock_irq(&cachep->node[node]->list_lock);
+ cachep->node[node]->free_limit =
(1 + nr_cpus_node(node)) *
cachep->batchcount + cachep->num;
- spin_unlock_irq(&cachep->nodelists[node]->list_lock);
+ spin_unlock_irq(&cachep->node[node]->list_lock);
}
return 0;
}
@@ -1227,7 +1227,7 @@ static void __cpuinit cpuup_canceled(lon
/* cpu is dead; no one can alloc from it. */
nc = cachep->array[cpu];
cachep->array[cpu] = NULL;
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
if (!l3)
goto free_array_cache;
@@ -1270,7 +1270,7 @@ free_array_cache:
* shrink each nodelist to its limit.
*/
list_for_each_entry(cachep, &slab_caches, list) {
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
if (!l3)
continue;
drain_freelist(cachep, l3, l3->free_objects);
@@ -1290,7 +1290,7 @@ static int __cpuinit cpuup_prepare(long
* kmalloc_node allows us to add the slab to the right
* kmem_list3 and not this cpu's kmem_list3
*/
- err = init_cache_nodelists_node(node);
+ err = init_cache_node_node(node);
if (err < 0)
goto bad;
@@ -1325,7 +1325,7 @@ static int __cpuinit cpuup_prepare(long
}
}
cachep->array[cpu] = nc;
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
BUG_ON(!l3);
spin_lock_irq(&l3->list_lock);
@@ -1425,7 +1425,7 @@ static struct notifier_block __cpuinitda
*
* Must hold slab_mutex.
*/
-static int __meminit drain_cache_nodelists_node(int node)
+static int __meminit drain_cache_node_node(int node)
{
struct kmem_cache *cachep;
int ret = 0;
@@ -1433,7 +1433,7 @@ static int __meminit drain_cache_nodelis
list_for_each_entry(cachep, &slab_caches, list) {
struct kmem_cache_node *l3;
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
if (!l3)
continue;
@@ -1462,12 +1462,12 @@ static int __meminit slab_memory_callbac
switch (action) {
case MEM_GOING_ONLINE:
mutex_lock(&slab_mutex);
- ret = init_cache_nodelists_node(nid);
+ ret = init_cache_node_node(nid);
mutex_unlock(&slab_mutex);
break;
case MEM_GOING_OFFLINE:
mutex_lock(&slab_mutex);
- ret = drain_cache_nodelists_node(nid);
+ ret = drain_cache_node_node(nid);
mutex_unlock(&slab_mutex);
break;
case MEM_ONLINE:
@@ -1499,7 +1499,7 @@ static void __init init_list(struct kmem
spin_lock_init(&ptr->list_lock);
MAKE_ALL_LISTS(cachep, ptr, nodeid);
- cachep->nodelists[nodeid] = ptr;
+ cachep->node[nodeid] = ptr;
}
/*
@@ -1511,8 +1511,8 @@ static void __init set_up_list3s(struct
int node;
for_each_online_node(node) {
- cachep->nodelists[node] = &initkmem_list3[index + node];
- cachep->nodelists[node]->next_reap = jiffies +
+ cachep->node[node] = &initkmem_list3[index + node];
+ cachep->node[node]->next_reap = jiffies +
REAPTIMEOUT_LIST3 +
((unsigned long)cachep) % REAPTIMEOUT_LIST3;
}
@@ -1520,11 +1520,11 @@ static void __init set_up_list3s(struct
/*
* The memory after the last cpu cache pointer is used for the
- * the nodelists pointer.
+ * the node pointer.
*/
-static void setup_nodelists_pointer(struct kmem_cache *s)
+static void setup_node_pointer(struct kmem_cache *s)
{
- s->nodelists = (struct kmem_cache_node **)&s->array[nr_cpu_ids];
+ s->node = (struct kmem_cache_node **)&s->array[nr_cpu_ids];
}
/*
@@ -1536,7 +1536,7 @@ void __init kmem_cache_init(void)
int i;
kmem_cache = &kmem_cache_boot;
- setup_nodelists_pointer(kmem_cache);
+ setup_node_pointer(kmem_cache);
if (num_possible_nodes() == 1)
use_alien_caches = 0;
@@ -1724,7 +1724,7 @@ void __init kmem_cache_init_late(void)
#ifdef CONFIG_NUMA
/*
* Register a memory hotplug callback that initializes and frees
- * nodelists.
+ * node.
*/
hotplug_memory_notifier(slab_memory_callback, SLAB_CALLBACK_PRI);
#endif
@@ -1769,7 +1769,7 @@ slab_out_of_memory(struct kmem_cache *ca
unsigned long active_objs = 0, num_objs = 0, free_objects = 0;
unsigned long active_slabs = 0, num_slabs = 0;
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
if (!l3)
continue;
@@ -2242,15 +2242,15 @@ static int __init_refok setup_cpu_cache(
} else {
int node;
for_each_online_node(node) {
- cachep->nodelists[node] =
+ cachep->node[node] =
kmalloc_node(sizeof(struct kmem_cache_node),
gfp, node);
- BUG_ON(!cachep->nodelists[node]);
- kmem_list3_init(cachep->nodelists[node]);
+ BUG_ON(!cachep->node[node]);
+ kmem_list3_init(cachep->node[node]);
}
}
}
- cachep->nodelists[numa_mem_id()]->next_reap =
+ cachep->node[numa_mem_id()]->next_reap =
jiffies + REAPTIMEOUT_LIST3 +
((unsigned long)cachep) % REAPTIMEOUT_LIST3;
@@ -2356,7 +2356,7 @@ __kmem_cache_create (struct kmem_cache *
else
gfp = GFP_NOWAIT;
- setup_nodelists_pointer(cachep);
+ setup_node_pointer(cachep);
#if DEBUG
/*
@@ -2494,7 +2494,7 @@ static void check_spinlock_acquired(stru
{
#ifdef CONFIG_SMP
check_irq_off();
- assert_spin_locked(&cachep->nodelists[numa_mem_id()]->list_lock);
+ assert_spin_locked(&cachep->node[numa_mem_id()]->list_lock);
#endif
}
@@ -2502,7 +2502,7 @@ static void check_spinlock_acquired_node
{
#ifdef CONFIG_SMP
check_irq_off();
- assert_spin_locked(&cachep->nodelists[node]->list_lock);
+ assert_spin_locked(&cachep->node[node]->list_lock);
#endif
}
@@ -2525,9 +2525,9 @@ static void do_drain(void *arg)
check_irq_off();
ac = cpu_cache_get(cachep);
- spin_lock(&cachep->nodelists[node]->list_lock);
+ spin_lock(&cachep->node[node]->list_lock);
free_block(cachep, ac->entry, ac->avail, node);
- spin_unlock(&cachep->nodelists[node]->list_lock);
+ spin_unlock(&cachep->node[node]->list_lock);
ac->avail = 0;
}
@@ -2539,13 +2539,13 @@ static void drain_cpu_caches(struct kmem
on_each_cpu(do_drain, cachep, 1);
check_irq_on();
for_each_online_node(node) {
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
if (l3 && l3->alien)
drain_alien_cache(cachep, l3->alien);
}
for_each_online_node(node) {
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
if (l3)
drain_array(cachep, l3, l3->shared, 1, node);
}
@@ -2602,7 +2602,7 @@ static int __cache_shrink(struct kmem_ca
check_irq_on();
for_each_online_node(i) {
- l3 = cachep->nodelists[i];
+ l3 = cachep->node[i];
if (!l3)
continue;
@@ -2649,7 +2649,7 @@ int __kmem_cache_shutdown(struct kmem_ca
/* NUMA: free the list3 structures */
for_each_online_node(i) {
- l3 = cachep->nodelists[i];
+ l3 = cachep->node[i];
if (l3) {
kfree(l3->shared);
free_alien_cache(l3->alien);
@@ -2846,7 +2846,7 @@ static int cache_grow(struct kmem_cache
/* Take the l3 list lock to change the colour_next on this node */
check_irq_off();
- l3 = cachep->nodelists[nodeid];
+ l3 = cachep->node[nodeid];
spin_lock(&l3->list_lock);
/* Get colour for the slab, and cal the next value. */
@@ -3044,7 +3044,7 @@ retry:
*/
batchcount = BATCHREFILL_LIMIT;
}
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
BUG_ON(ac->avail > 0 || !l3);
spin_lock(&l3->list_lock);
@@ -3266,7 +3266,7 @@ static void *alternate_node_alloc(struct
/*
* Fallback function if there was no memory available and no objects on a
* certain node and fall back is permitted. First we scan all the
- * available nodelists for available objects. If that fails then we
+ * available node for available objects. If that fails then we
* perform an allocation without specifying a node. This allows the page
* allocator to do its reclaim / fallback magic. We then insert the
* slab into the proper nodelist and then allocate from it.
@@ -3300,8 +3300,8 @@ retry:
nid = zone_to_nid(zone);
if (cpuset_zone_allowed_hardwall(zone, flags) &&
- cache->nodelists[nid] &&
- cache->nodelists[nid]->free_objects) {
+ cache->node[nid] &&
+ cache->node[nid]->free_objects) {
obj = ____cache_alloc_node(cache,
flags | GFP_THISNODE, nid);
if (obj)
@@ -3361,7 +3361,7 @@ static void *____cache_alloc_node(struct
void *obj;
int x;
- l3 = cachep->nodelists[nodeid];
+ l3 = cachep->node[nodeid];
BUG_ON(!l3);
retry:
@@ -3444,7 +3444,7 @@ slab_alloc_node(struct kmem_cache *cache
if (nodeid == NUMA_NO_NODE)
nodeid = slab_node;
- if (unlikely(!cachep->nodelists[nodeid])) {
+ if (unlikely(!cachep->node[nodeid])) {
/* Node not bootstrapped yet */
ptr = fallback_alloc(cachep, flags);
goto out;
@@ -3558,7 +3558,7 @@ static void free_block(struct kmem_cache
objp = objpp[i];
slabp = virt_to_slab(objp);
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
list_del(&slabp->list);
check_spinlock_acquired_node(cachep, node);
check_slabp(cachep, slabp);
@@ -3602,7 +3602,7 @@ static void cache_flusharray(struct kmem
BUG_ON(!batchcount || batchcount > ac->avail);
#endif
check_irq_off();
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
spin_lock(&l3->list_lock);
if (l3->shared) {
struct array_cache *shared_array = l3->shared;
@@ -3906,7 +3906,7 @@ static int alloc_kmemlist(struct kmem_ca
}
}
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
if (l3) {
struct array_cache *shared = l3->shared;
@@ -3942,7 +3942,7 @@ static int alloc_kmemlist(struct kmem_ca
l3->alien = new_alien;
l3->free_limit = (1 + nr_cpus_node(node)) *
cachep->batchcount + cachep->num;
- cachep->nodelists[node] = l3;
+ cachep->node[node] = l3;
}
return 0;
@@ -3951,13 +3951,13 @@ fail:
/* Cache is not active yet. Roll back what we did */
node--;
while (node >= 0) {
- if (cachep->nodelists[node]) {
- l3 = cachep->nodelists[node];
+ if (cachep->node[node]) {
+ l3 = cachep->node[node];
kfree(l3->shared);
free_alien_cache(l3->alien);
kfree(l3);
- cachep->nodelists[node] = NULL;
+ cachep->node[node] = NULL;
}
node--;
}
@@ -4017,9 +4017,9 @@ static int do_tune_cpucache(struct kmem_
struct array_cache *ccold = new->new[i];
if (!ccold)
continue;
- spin_lock_irq(&cachep->nodelists[cpu_to_mem(i)]->list_lock);
+ spin_lock_irq(&cachep->node[cpu_to_mem(i)]->list_lock);
free_block(cachep, ccold->entry, ccold->avail, cpu_to_mem(i));
- spin_unlock_irq(&cachep->nodelists[cpu_to_mem(i)]->list_lock);
+ spin_unlock_irq(&cachep->node[cpu_to_mem(i)]->list_lock);
kfree(ccold);
}
kfree(new);
@@ -4140,7 +4140,7 @@ static void cache_reap(struct work_struc
* have established with reasonable certainty that
* we can do some work if the lock was obtained.
*/
- l3 = searchp->nodelists[node];
+ l3 = searchp->node[node];
reap_alien(searchp, l3);
@@ -4193,7 +4193,7 @@ void get_slabinfo(struct kmem_cache *cac
active_objs = 0;
num_slabs = 0;
for_each_online_node(node) {
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
if (!l3)
continue;
@@ -4418,7 +4418,7 @@ static int leaks_show(struct seq_file *m
n[1] = 0;
for_each_online_node(node) {
- l3 = cachep->nodelists[node];
+ l3 = cachep->node[node];
if (!l3)
continue;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [14/18] Common function to create the kmalloc array
[not found] <20121101214538.971500204@linux.com>
` (10 preceding siblings ...)
2012-11-01 21:48 ` CK5 [13/18] Common definition for the array of kmalloc caches Christoph Lameter
@ 2012-11-01 21:48 ` Christoph Lameter
2012-11-01 21:48 ` CK5 [16/18] Common Kmalloc cache determination Christoph Lameter
` (5 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:48 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
The kmalloc array is created in similar ways in both SLAB
and SLUB. Create a common function and have both allocators
call that function.
V1->V2:
Whitespace cleanup
Reviewed-by: Glauber Costa <glommer@parallels.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 16:12:30.457437122 -0500
+++ linux/mm/slab.c 2012-11-01 16:12:34.137488791 -0500
@@ -1594,30 +1594,6 @@ void __init kmem_cache_init(void)
slab_early_init = 0;
- for (i = 1; i < PAGE_SHIFT + MAX_ORDER; i++) {
- size_t cs_size = kmalloc_size(i);
-
- if (cs_size < KMALLOC_MIN_SIZE)
- continue;
-
- if (!kmalloc_caches[i]) {
- /*
- * For performance, all the general caches are L1 aligned.
- * This should be particularly beneficial on SMP boxes, as it
- * eliminates "false sharing".
- * Note for systems short on memory removing the alignment will
- * allow tighter packing of the smaller caches.
- */
- kmalloc_caches[i] = create_kmalloc_cache("kmalloc",
- cs_size, ARCH_KMALLOC_FLAGS);
- }
-
-#ifdef CONFIG_ZONE_DMA
- kmalloc_dma_caches[i] = create_kmalloc_cache(
- "kmalloc-dma", cs_size,
- SLAB_CACHE_DMA|ARCH_KMALLOC_FLAGS);
-#endif
- }
/* 4) Replace the bootstrap head arrays */
{
struct array_cache *ptr;
@@ -1663,29 +1639,7 @@ void __init kmem_cache_init(void)
}
}
- slab_state = UP;
-
- /* Create the proper names */
- for (i = 1; i < PAGE_SHIFT + MAX_ORDER; i++) {
- char *s;
- struct kmem_cache *c = kmalloc_caches[i];
-
- if (!c)
- continue;
-
- s = kasprintf(GFP_NOWAIT, "kmalloc-%d", kmalloc_size(i));
-
- BUG_ON(!s);
- c->name = s;
-
-#ifdef CONFIG_ZONE_DMA
- c = kmalloc_dma_caches[i];
- BUG_ON(!c);
- s = kasprintf(GFP_NOWAIT, "dma-kmalloc-%d", kmalloc_size(i));
- BUG_ON(!s);
- c->name = s;
-#endif
- }
+ create_kmalloc_caches(ARCH_KMALLOC_FLAGS);
}
void __init kmem_cache_init_late(void)
Index: linux/mm/slab.h
===================================================================
--- linux.orig/mm/slab.h 2012-11-01 16:12:12.705187852 -0500
+++ linux/mm/slab.h 2012-11-01 16:12:34.137488791 -0500
@@ -35,6 +35,12 @@ extern struct kmem_cache *kmem_cache;
unsigned long calculate_alignment(unsigned long flags,
unsigned long align, unsigned long size);
+#ifndef CONFIG_SLOB
+/* Kmalloc array related functions */
+void create_kmalloc_caches(unsigned long);
+#endif
+
+
/* Functions provided by the slab allocators */
extern int __kmem_cache_create(struct kmem_cache *, unsigned long flags);
Index: linux/mm/slab_common.c
===================================================================
--- linux.orig/mm/slab_common.c 2012-11-01 16:12:30.457437122 -0500
+++ linux/mm/slab_common.c 2012-11-01 16:12:34.137488791 -0500
@@ -271,6 +271,60 @@ struct kmem_cache *kmalloc_dma_caches[KM
EXPORT_SYMBOL(kmalloc_dma_caches);
#endif
+/*
+ * Create the kmalloc array. Some of the regular kmalloc arrays
+ * may already have been created because they were needed to
+ * enable allocations for slab creation.
+ */
+void __init create_kmalloc_caches(unsigned long flags)
+{
+ int i;
+
+ /* Caches that are not of the two-to-the-power-of size */
+ if (KMALLOC_MIN_SIZE <= 32 && !kmalloc_caches[1])
+ kmalloc_caches[1] = create_kmalloc_cache(NULL, 96, flags);
+
+ if (KMALLOC_MIN_SIZE <= 64 && !kmalloc_caches[2])
+ kmalloc_caches[2] = create_kmalloc_cache(NULL, 192, flags);
+
+ for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++)
+ if (!kmalloc_caches[i])
+ kmalloc_caches[i] = create_kmalloc_cache(NULL,
+ 1 << i, flags);
+
+ /* Kmalloc array is now usable */
+ slab_state = UP;
+
+ for (i = 0; i <= KMALLOC_SHIFT_HIGH; i++) {
+ struct kmem_cache *s = kmalloc_caches[i];
+ char *n;
+
+ if (s) {
+ n = kasprintf(GFP_NOWAIT, "kmalloc-%d", kmalloc_size(i));
+
+ BUG_ON(!n);
+ s->name = n;
+ }
+ }
+
+#ifdef CONFIG_ZONE_DMA
+ for (i = 0; i <= KMALLOC_SHIFT_HIGH; i++) {
+ struct kmem_cache *s = kmalloc_caches[i];
+
+ if (s) {
+ int size = kmalloc_size(i);
+ char *n = kasprintf(GFP_NOWAIT,
+ "dma-kmalloc-%d", size);
+
+ BUG_ON(!n);
+ kmalloc_dma_caches[i] = create_kmalloc_cache(n,
+ size, SLAB_CACHE_DMA | flags);
+ }
+ }
+#endif
+}
+
+
#endif /* !CONFIG_SLOB */
Index: linux/mm/slub.c
===================================================================
--- linux.orig/mm/slub.c 2012-11-01 16:12:30.457437122 -0500
+++ linux/mm/slub.c 2012-11-01 16:12:34.137488791 -0500
@@ -3621,7 +3621,6 @@ void __init kmem_cache_init(void)
static __initdata struct kmem_cache boot_kmem_cache,
boot_kmem_cache_node;
int i;
- int caches = 2;
if (debug_guardpage_minorder())
slub_max_order = 0;
@@ -3691,64 +3690,16 @@ void __init kmem_cache_init(void)
size_index[size_index_elem(i)] = 8;
}
- /* Caches that are not of the two-to-the-power-of size */
- if (KMALLOC_MIN_SIZE <= 32) {
- kmalloc_caches[1] = create_kmalloc_cache("kmalloc-96", 96, 0);
- caches++;
- }
-
- if (KMALLOC_MIN_SIZE <= 64) {
- kmalloc_caches[2] = create_kmalloc_cache("kmalloc-192", 192, 0);
- caches++;
- }
-
- for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) {
- kmalloc_caches[i] = create_kmalloc_cache("kmalloc", 1 << i, 0);
- caches++;
- }
-
- slab_state = UP;
-
- /* Provide the correct kmalloc names now that the caches are up */
- if (KMALLOC_MIN_SIZE <= 32) {
- kmalloc_caches[1]->name = kstrdup(kmalloc_caches[1]->name, GFP_NOWAIT);
- BUG_ON(!kmalloc_caches[1]->name);
- }
-
- if (KMALLOC_MIN_SIZE <= 64) {
- kmalloc_caches[2]->name = kstrdup(kmalloc_caches[2]->name, GFP_NOWAIT);
- BUG_ON(!kmalloc_caches[2]->name);
- }
-
- for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++) {
- char *s = kasprintf(GFP_NOWAIT, "kmalloc-%d", 1 << i);
-
- BUG_ON(!s);
- kmalloc_caches[i]->name = s;
- }
+ create_kmalloc_caches(0);
#ifdef CONFIG_SMP
register_cpu_notifier(&slab_notifier);
#endif
-#ifdef CONFIG_ZONE_DMA
- for (i = 0; i <= KMALLOC_SHIFT_HIGH; i++) {
- struct kmem_cache *s = kmalloc_caches[i];
-
- if (s && s->size) {
- char *name = kasprintf(GFP_NOWAIT,
- "dma-kmalloc-%d", s->object_size);
-
- BUG_ON(!name);
- kmalloc_dma_caches[i] = create_kmalloc_cache(name,
- s->object_size, SLAB_CACHE_DMA);
- }
- }
-#endif
printk(KERN_INFO
- "SLUB: Genslabs=%d, HWalign=%d, Order=%d-%d, MinObjects=%d,"
+ "SLUB: HWalign=%d, Order=%d-%d, MinObjects=%d,"
" CPUs=%d, Nodes=%d\n",
- caches, cache_line_size(),
+ cache_line_size(),
slub_min_order, slub_max_order, slub_min_objects,
nr_cpu_ids, nr_node_ids);
}
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [16/18] Common Kmalloc cache determination
[not found] <20121101214538.971500204@linux.com>
` (11 preceding siblings ...)
2012-11-01 21:48 ` CK5 [14/18] Common function to create the kmalloc array Christoph Lameter
@ 2012-11-01 21:48 ` Christoph Lameter
2012-11-01 21:48 ` CK5 [08/18] Common kmalloc slab index determination Christoph Lameter
` (4 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:48 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Extract the optimized lookup functions from slub and put them into
slab_common.c. Then make slab use these functions as well.
Joonsoo notes that this fixes some issues with constant folding which
also reduces the code size for slub.
https://lkml.org/lkml/2012/10/20/82
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 16:12:34.137488791 -0500
+++ linux/mm/slab.c 2012-11-01 16:12:48.217686485 -0500
@@ -629,40 +629,6 @@ static inline struct array_cache *cpu_ca
return cachep->array[smp_processor_id()];
}
-static inline struct kmem_cache *__find_general_cachep(size_t size,
- gfp_t gfpflags)
-{
- int i;
-
-#if DEBUG
- /* This happens if someone tries to call
- * kmem_cache_create(), or __kmalloc(), before
- * the generic caches are initialized.
- */
- BUG_ON(kmalloc_caches[INDEX_AC] == NULL);
-#endif
- if (!size)
- return ZERO_SIZE_PTR;
-
- i = kmalloc_index(size);
-
- /*
- * Really subtle: The last entry with cs->cs_size==ULONG_MAX
- * has cs_{dma,}cachep==NULL. Thus no special case
- * for large kmalloc calls required.
- */
-#ifdef CONFIG_ZONE_DMA
- if (unlikely(gfpflags & GFP_DMA))
- return kmalloc_dma_caches[i];
-#endif
- return kmalloc_caches[i];
-}
-
-static struct kmem_cache *kmem_find_general_cachep(size_t size, gfp_t gfpflags)
-{
- return __find_general_cachep(size, gfpflags);
-}
-
static size_t slab_mgmt_size(size_t nr_objs, size_t align)
{
return ALIGN(sizeof(struct slab)+nr_objs*sizeof(kmem_bufctl_t), align);
@@ -2395,7 +2361,7 @@ __kmem_cache_create (struct kmem_cache *
cachep->reciprocal_buffer_size = reciprocal_value(size);
if (flags & CFLGS_OFF_SLAB) {
- cachep->slabp_cache = kmem_find_general_cachep(slab_size, 0u);
+ cachep->slabp_cache = kmalloc_slab(slab_size, 0u);
/*
* This is a possibility for one of the malloc_sizes caches.
* But since we go off slab only for object size greater than
@@ -3693,7 +3659,7 @@ __do_kmalloc_node(size_t size, gfp_t fla
{
struct kmem_cache *cachep;
- cachep = kmem_find_general_cachep(size, flags);
+ cachep = kmalloc_slab(size, flags);
if (unlikely(ZERO_OR_NULL_PTR(cachep)))
return cachep;
return kmem_cache_alloc_node_trace(cachep, flags, node, size);
@@ -3738,7 +3704,7 @@ static __always_inline void *__do_kmallo
* Then kmalloc uses the uninlined functions instead of the inline
* functions.
*/
- cachep = __find_general_cachep(size, flags);
+ cachep = kmalloc_slab(size, flags);
if (unlikely(ZERO_OR_NULL_PTR(cachep)))
return cachep;
ret = slab_alloc(cachep, flags, caller);
Index: linux/mm/slab.h
===================================================================
--- linux.orig/mm/slab.h 2012-11-01 16:12:34.000000000 -0500
+++ linux/mm/slab.h 2012-11-01 16:12:48.217686485 -0500
@@ -38,6 +38,9 @@ unsigned long calculate_alignment(unsign
#ifndef CONFIG_SLOB
/* Kmalloc array related functions */
void create_kmalloc_caches(unsigned long);
+
+/* Find the kmalloc slab corresponding for a certain size */
+struct kmem_cache *kmalloc_slab(size_t, gfp_t);
#endif
Index: linux/mm/slab_common.c
===================================================================
--- linux.orig/mm/slab_common.c 2012-11-01 16:12:34.000000000 -0500
+++ linux/mm/slab_common.c 2012-11-01 16:12:48.217686485 -0500
@@ -272,6 +272,68 @@ EXPORT_SYMBOL(kmalloc_dma_caches);
#endif
/*
+ * Conversion table for small slabs sizes / 8 to the index in the
+ * kmalloc array. This is necessary for slabs < 192 since we have non power
+ * of two cache sizes there. The size of larger slabs can be determined using
+ * fls.
+ */
+static s8 size_index[24] = {
+ 3, /* 8 */
+ 4, /* 16 */
+ 5, /* 24 */
+ 5, /* 32 */
+ 6, /* 40 */
+ 6, /* 48 */
+ 6, /* 56 */
+ 6, /* 64 */
+ 1, /* 72 */
+ 1, /* 80 */
+ 1, /* 88 */
+ 1, /* 96 */
+ 7, /* 104 */
+ 7, /* 112 */
+ 7, /* 120 */
+ 7, /* 128 */
+ 2, /* 136 */
+ 2, /* 144 */
+ 2, /* 152 */
+ 2, /* 160 */
+ 2, /* 168 */
+ 2, /* 176 */
+ 2, /* 184 */
+ 2 /* 192 */
+};
+
+static inline int size_index_elem(size_t bytes)
+{
+ return (bytes - 1) / 8;
+}
+
+/*
+ * Find the kmem_cache structure that serves a given size of
+ * allocation
+ */
+struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags)
+{
+ int index;
+
+ if (size <= 192) {
+ if (!size)
+ return ZERO_SIZE_PTR;
+
+ index = size_index[size_index_elem(size)];
+ } else
+ index = fls(size - 1);
+
+#ifdef CONFIG_ZONE_DMA
+ if (unlikely((flags & SLAB_CACHE_DMA)))
+ return kmalloc_dma_caches[index];
+
+#endif
+ return kmalloc_caches[index];
+}
+
+/*
* Create the kmalloc array. Some of the regular kmalloc arrays
* may already have been created because they were needed to
* enable allocations for slab creation.
@@ -280,6 +342,47 @@ void __init create_kmalloc_caches(unsign
{
int i;
+ /*
+ * Patch up the size_index table if we have strange large alignment
+ * requirements for the kmalloc array. This is only the case for
+ * MIPS it seems. The standard arches will not generate any code here.
+ *
+ * Largest permitted alignment is 256 bytes due to the way we
+ * handle the index determination for the smaller caches.
+ *
+ * Make sure that nothing crazy happens if someone starts tinkering
+ * around with ARCH_KMALLOC_MINALIGN
+ */
+ BUILD_BUG_ON(KMALLOC_MIN_SIZE > 256 ||
+ (KMALLOC_MIN_SIZE & (KMALLOC_MIN_SIZE - 1)));
+
+ for (i = 8; i < KMALLOC_MIN_SIZE; i += 8) {
+ int elem = size_index_elem(i);
+
+ if (elem >= ARRAY_SIZE(size_index))
+ break;
+ size_index[elem] = KMALLOC_SHIFT_LOW;
+ }
+
+ if (KMALLOC_MIN_SIZE >= 64) {
+ /*
+ * The 96 byte size cache is not used if the alignment
+ * is 64 byte.
+ */
+ for (i = 64 + 8; i <= 96; i += 8)
+ size_index[size_index_elem(i)] = 7;
+
+ }
+
+ if (KMALLOC_MIN_SIZE >= 128) {
+ /*
+ * The 192 byte sized cache is not used if the alignment
+ * is 128 byte. Redirect kmalloc to use the 256 byte cache
+ * instead.
+ */
+ for (i = 128 + 8; i <= 192; i += 8)
+ size_index[size_index_elem(i)] = 8;
+ }
/* Caches that are not of the two-to-the-power-of size */
if (KMALLOC_MIN_SIZE <= 32 && !kmalloc_caches[1])
kmalloc_caches[1] = create_kmalloc_cache(NULL, 96, flags);
@@ -323,8 +426,6 @@ void __init create_kmalloc_caches(unsign
}
#endif
}
-
-
#endif /* !CONFIG_SLOB */
Index: linux/mm/slub.c
===================================================================
--- linux.orig/mm/slub.c 2012-11-01 16:12:34.137488791 -0500
+++ linux/mm/slub.c 2012-11-01 16:13:27.382236339 -0500
@@ -2981,7 +2981,7 @@ static int calculate_sizes(struct kmem_c
s->allocflags |= __GFP_COMP;
if (s->flags & SLAB_CACHE_DMA)
- s->allocflags |= SLUB_DMA;
+ s->allocflags |= GFP_DMA;
if (s->flags & SLAB_RECLAIM_ACCOUNT)
s->allocflags |= __GFP_RECLAIMABLE;
@@ -3198,64 +3198,6 @@ static int __init setup_slub_nomerge(cha
__setup("slub_nomerge", setup_slub_nomerge);
-/*
- * Conversion table for small slabs sizes / 8 to the index in the
- * kmalloc array. This is necessary for slabs < 192 since we have non power
- * of two cache sizes there. The size of larger slabs can be determined using
- * fls.
- */
-static s8 size_index[24] = {
- 3, /* 8 */
- 4, /* 16 */
- 5, /* 24 */
- 5, /* 32 */
- 6, /* 40 */
- 6, /* 48 */
- 6, /* 56 */
- 6, /* 64 */
- 1, /* 72 */
- 1, /* 80 */
- 1, /* 88 */
- 1, /* 96 */
- 7, /* 104 */
- 7, /* 112 */
- 7, /* 120 */
- 7, /* 128 */
- 2, /* 136 */
- 2, /* 144 */
- 2, /* 152 */
- 2, /* 160 */
- 2, /* 168 */
- 2, /* 176 */
- 2, /* 184 */
- 2 /* 192 */
-};
-
-static inline int size_index_elem(size_t bytes)
-{
- return (bytes - 1) / 8;
-}
-
-static struct kmem_cache *get_slab(size_t size, gfp_t flags)
-{
- int index;
-
- if (size <= 192) {
- if (!size)
- return ZERO_SIZE_PTR;
-
- index = size_index[size_index_elem(size)];
- } else
- index = fls(size - 1);
-
-#ifdef CONFIG_ZONE_DMA
- if (unlikely((flags & SLUB_DMA)))
- return kmalloc_dma_caches[index];
-
-#endif
- return kmalloc_caches[index];
-}
-
void *__kmalloc(size_t size, gfp_t flags)
{
struct kmem_cache *s;
@@ -3264,7 +3206,7 @@ void *__kmalloc(size_t size, gfp_t flags
if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
return kmalloc_large(size, flags);
- s = get_slab(size, flags);
+ s = kmalloc_slab(size, flags);
if (unlikely(ZERO_OR_NULL_PTR(s)))
return s;
@@ -3307,7 +3249,7 @@ void *__kmalloc_node(size_t size, gfp_t
return ret;
}
- s = get_slab(size, flags);
+ s = kmalloc_slab(size, flags);
if (unlikely(ZERO_OR_NULL_PTR(s)))
return s;
@@ -3620,7 +3562,6 @@ void __init kmem_cache_init(void)
{
static __initdata struct kmem_cache boot_kmem_cache,
boot_kmem_cache_node;
- int i;
if (debug_guardpage_minorder())
slub_max_order = 0;
@@ -3651,45 +3592,6 @@ void __init kmem_cache_init(void)
kmem_cache_node = bootstrap(&boot_kmem_cache_node);
/* Now we can use the kmem_cache to allocate kmalloc slabs */
-
- /*
- * Patch up the size_index table if we have strange large alignment
- * requirements for the kmalloc array. This is only the case for
- * MIPS it seems. The standard arches will not generate any code here.
- *
- * Largest permitted alignment is 256 bytes due to the way we
- * handle the index determination for the smaller caches.
- *
- * Make sure that nothing crazy happens if someone starts tinkering
- * around with ARCH_KMALLOC_MINALIGN
- */
- BUILD_BUG_ON(KMALLOC_MIN_SIZE > 256 ||
- (KMALLOC_MIN_SIZE & (KMALLOC_MIN_SIZE - 1)));
-
- for (i = 8; i < KMALLOC_MIN_SIZE; i += 8) {
- int elem = size_index_elem(i);
- if (elem >= ARRAY_SIZE(size_index))
- break;
- size_index[elem] = KMALLOC_SHIFT_LOW;
- }
-
- if (KMALLOC_MIN_SIZE == 64) {
- /*
- * The 96 byte size cache is not used if the alignment
- * is 64 byte.
- */
- for (i = 64 + 8; i <= 96; i += 8)
- size_index[size_index_elem(i)] = 7;
- } else if (KMALLOC_MIN_SIZE == 128) {
- /*
- * The 192 byte sized cache is not used if the alignment
- * is 128 byte. Redirect kmalloc to use the 256 byte cache
- * instead.
- */
- for (i = 128 + 8; i <= 192; i += 8)
- size_index[size_index_elem(i)] = 8;
- }
-
create_kmalloc_caches(0);
#ifdef CONFIG_SMP
@@ -3860,7 +3762,7 @@ void *__kmalloc_track_caller(size_t size
if (unlikely(size > KMALLOC_MAX_CACHE_SIZE))
return kmalloc_large(size, gfpflags);
- s = get_slab(size, gfpflags);
+ s = kmalloc_slab(size, gfpflags);
if (unlikely(ZERO_OR_NULL_PTR(s)))
return s;
@@ -3890,7 +3792,7 @@ void *__kmalloc_node_track_caller(size_t
return ret;
}
- s = get_slab(size, gfpflags);
+ s = kmalloc_slab(size, gfpflags);
if (unlikely(ZERO_OR_NULL_PTR(s)))
return s;
Index: linux/include/linux/slub_def.h
===================================================================
--- linux.orig/include/linux/slub_def.h 2012-11-01 16:12:30.457437122 -0500
+++ linux/include/linux/slub_def.h 2012-11-01 16:12:48.217686485 -0500
@@ -111,29 +111,6 @@ struct kmem_cache {
struct kmem_cache_node *node[MAX_NUMNODES];
};
-#ifdef CONFIG_ZONE_DMA
-#define SLUB_DMA __GFP_DMA
-#else
-/* Disable DMA functionality */
-#define SLUB_DMA (__force gfp_t)0
-#endif
-
-/*
- * Find the slab cache for a given combination of allocation flags and size.
- *
- * This ought to end up with a global pointer to the right cache
- * in kmalloc_caches.
- */
-static __always_inline struct kmem_cache *kmalloc_slab(size_t size)
-{
- int index = kmalloc_index(size);
-
- if (index == 0)
- return NULL;
-
- return kmalloc_caches[index];
-}
-
void *kmem_cache_alloc(struct kmem_cache *, gfp_t);
void *__kmalloc(size_t size, gfp_t flags);
@@ -188,13 +165,14 @@ static __always_inline void *kmalloc(siz
if (size > KMALLOC_MAX_CACHE_SIZE)
return kmalloc_large(size, flags);
- if (!(flags & SLUB_DMA)) {
- struct kmem_cache *s = kmalloc_slab(size);
+ if (!(flags & GFP_DMA)) {
+ int index = kmalloc_index(size);
- if (!s)
+ if (!index)
return ZERO_SIZE_PTR;
- return kmem_cache_alloc_trace(s, flags, size);
+ return kmem_cache_alloc_trace(kmalloc_caches[index],
+ flags, size);
}
}
return __kmalloc(size, flags);
@@ -221,13 +199,14 @@ kmem_cache_alloc_node_trace(struct kmem_
static __always_inline void *kmalloc_node(size_t size, gfp_t flags, int node)
{
if (__builtin_constant_p(size) &&
- size <= KMALLOC_MAX_CACHE_SIZE && !(flags & SLUB_DMA)) {
- struct kmem_cache *s = kmalloc_slab(size);
+ size <= KMALLOC_MAX_CACHE_SIZE && !(flags & GFP_DMA)) {
+ int index = kmalloc_index(size);
- if (!s)
+ if (!index)
return ZERO_SIZE_PTR;
- return kmem_cache_alloc_node_trace(s, flags, node, size);
+ return kmem_cache_alloc_node_trace(kmalloc_caches[index],
+ flags, node, size);
}
return __kmalloc_node(size, flags, node);
}
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [08/18] Common kmalloc slab index determination
[not found] <20121101214538.971500204@linux.com>
` (12 preceding siblings ...)
2012-11-01 21:48 ` CK5 [16/18] Common Kmalloc cache determination Christoph Lameter
@ 2012-11-01 21:48 ` Christoph Lameter
2012-11-01 21:48 ` CK5 [17/18] slab: Rename list3/l3 to node Christoph Lameter
` (3 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:48 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Extract the function to determine the index of the slab within
the array of kmalloc caches as well as a function to determine
maximum object size from the nr of the kmalloc slab.
This is used here only to simplify slub bootstrap but will
be used later also for SLAB.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/slab.h
===================================================================
--- linux.orig/include/linux/slab.h 2012-11-01 10:10:24.838044511 -0500
+++ linux/include/linux/slab.h 2012-11-01 10:10:27.774093173 -0500
@@ -93,30 +93,6 @@
(unsigned long)ZERO_SIZE_PTR)
/*
- * Common fields provided in kmem_cache by all slab allocators
- * This struct is either used directly by the allocator (SLOB)
- * or the allocator must include definitions for all fields
- * provided in kmem_cache_common in their definition of kmem_cache.
- *
- * Once we can do anonymous structs (C11 standard) we could put a
- * anonymous struct definition in these allocators so that the
- * separate allocations in the kmem_cache structure of SLAB and
- * SLUB is no longer needed.
- */
-#ifdef CONFIG_SLOB
-struct kmem_cache {
- unsigned int object_size;/* The original size of the object */
- unsigned int size; /* The aligned/padded/added on size */
- unsigned int align; /* Alignment as calculated */
- unsigned long flags; /* Active flags on the slab */
- const char *name; /* Slab name for sysfs */
- int refcount; /* Use counter */
- void (*ctor)(void *); /* Called on object slot creation */
- struct list_head list; /* List of all slab caches on the system */
-};
-#endif
-
-/*
* struct kmem_cache related prototypes
*/
void __init kmem_cache_init(void);
@@ -150,6 +126,35 @@ void kfree(const void *);
void kzfree(const void *);
size_t ksize(const void *);
+#ifdef CONFIG_SLOB
+/*
+ * Common fields provided in kmem_cache by all slab allocators
+ * This struct is either used directly by the allocator (SLOB)
+ * or the allocator must include definitions for all fields
+ * provided in kmem_cache_common in their definition of kmem_cache.
+ *
+ * Once we can do anonymous structs (C11 standard) we could put a
+ * anonymous struct definition in these allocators so that the
+ * separate allocations in the kmem_cache structure of SLAB and
+ * SLUB is no longer needed.
+ */
+struct kmem_cache {
+ unsigned int object_size;/* The original size of the object */
+ unsigned int size; /* The aligned/padded/added on size */
+ unsigned int align; /* Alignment as calculated */
+ unsigned long flags; /* Active flags on the slab */
+ const char *name; /* Slab name for sysfs */
+ int refcount; /* Use counter */
+ void (*ctor)(void *); /* Called on object slot creation */
+ struct list_head list; /* List of all slab caches on the system */
+};
+
+#define KMALLOC_MAX_SIZE (1UL << 30)
+
+#include <linux/slob_def.h>
+
+#else /* CONFIG_SLOB */
+
/*
* The largest kmalloc size supported by the slab allocators is
* 32 megabyte (2^25) or the maximum allocatable page order if that is
@@ -166,6 +171,99 @@ size_t ksize(const void *);
#define KMALLOC_MAX_ORDER (KMALLOC_SHIFT_HIGH - PAGE_SHIFT)
/*
+ * Kmalloc subsystem.
+ */
+#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
+#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
+#else
+#ifdef CONFIG_SLAB
+#define KMALLOC_MIN_SIZE 32
+#else
+#define KMALLOC_MIN_SIZE 8
+#endif
+#endif
+
+#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
+
+/*
+ * Figure out which kmalloc slab an allocation of a certain size
+ * belongs to.
+ * 0 = zero alloc
+ * 1 = 65 .. 96 bytes
+ * 2 = 120 .. 192 bytes
+ * n = 2^(n-1) .. 2^n -1
+ */
+static __always_inline int kmalloc_index(size_t size)
+{
+ if (!size)
+ return 0;
+
+ if (size <= KMALLOC_MIN_SIZE)
+ return KMALLOC_SHIFT_LOW;
+
+ if (KMALLOC_MIN_SIZE <= 32 && size > 64 && size <= 96)
+ return 1;
+ if (KMALLOC_MIN_SIZE <= 64 && size > 128 && size <= 192)
+ return 2;
+ if (size <= 8) return 3;
+ if (size <= 16) return 4;
+ if (size <= 32) return 5;
+ if (size <= 64) return 6;
+ if (size <= 128) return 7;
+ if (size <= 256) return 8;
+ if (size <= 512) return 9;
+ if (size <= 1024) return 10;
+ if (size <= 2 * 1024) return 11;
+ if (size <= 4 * 1024) return 12;
+ if (size <= 8 * 1024) return 13;
+ if (size <= 16 * 1024) return 14;
+ if (size <= 32 * 1024) return 15;
+ if (size <= 64 * 1024) return 16;
+ if (size <= 128 * 1024) return 17;
+ if (size <= 256 * 1024) return 18;
+ if (size <= 512 * 1024) return 19;
+ if (size <= 1024 * 1024) return 20;
+ if (size <= 2 * 1024 * 1024) return 21;
+ if (size <= 4 * 1024 * 1024) return 22;
+ if (size <= 8 * 1024 * 1024) return 23;
+ if (size <= 16 * 1024 * 1024) return 24;
+ if (size <= 32 * 1024 * 1024) return 25;
+ if (size <= 64 * 1024 * 1024) return 26;
+ BUG();
+
+ /* Will never be reached. Needed because the compiler may complain */
+ return -1;
+}
+
+#ifdef CONFIG_SLAB
+#include <linux/slab_def.h>
+#elif defined(CONFIG_SLUB)
+#include <linux/slub_def.h>
+#else
+#error "Unknown slab allocator"
+#endif
+
+/*
+ * Determine size used for the nth kmalloc cache.
+ * return size or 0 if a kmalloc cache for that
+ * size does not exist
+ */
+static __always_inline int kmalloc_size(int n)
+{
+ if (n > 2)
+ return 1 << n;
+
+ if (n == 1 && KMALLOC_MIN_SIZE <= 32)
+ return 96;
+
+ if (n == 2 && KMALLOC_MIN_SIZE <= 64)
+ return 192;
+
+ return 0;
+}
+#endif /* !CONFIG_SLOB */
+
+/*
* Some archs want to perform DMA into kmalloc caches and need a guaranteed
* alignment larger than the alignment of a 64-bit integer.
* Setting ARCH_KMALLOC_MINALIGN in arch headers allows that.
@@ -185,33 +283,6 @@ size_t ksize(const void *);
#define ARCH_SLAB_MINALIGN __alignof__(unsigned long long)
#endif
-/*
- * Allocator specific definitions. These are mainly used to establish optimized
- * ways to convert kmalloc() calls to kmem_cache_alloc() invocations by
- * selecting the appropriate general cache at compile time.
- *
- * Allocators must define at least:
- *
- * kmem_cache_alloc()
- * __kmalloc()
- * kmalloc()
- *
- * Those wishing to support NUMA must also define:
- *
- * kmem_cache_alloc_node()
- * kmalloc_node()
- *
- * See each allocator definition file for additional comments and
- * implementation notes.
- */
-#ifdef CONFIG_SLUB
-#include <linux/slub_def.h>
-#elif defined(CONFIG_SLOB)
-#include <linux/slob_def.h>
-#else
-#include <linux/slab_def.h>
-#endif
-
/**
* kmalloc_array - allocate memory for an array.
* @n: number of elements.
Index: linux/include/linux/slub_def.h
===================================================================
--- linux.orig/include/linux/slub_def.h 2012-11-01 10:09:46.681411435 -0500
+++ linux/include/linux/slub_def.h 2012-11-01 10:10:27.778093241 -0500
@@ -112,17 +112,6 @@ struct kmem_cache {
};
/*
- * Kmalloc subsystem.
- */
-#if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8
-#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN
-#else
-#define KMALLOC_MIN_SIZE 8
-#endif
-
-#define KMALLOC_SHIFT_LOW ilog2(KMALLOC_MIN_SIZE)
-
-/*
* Maximum kmalloc object size handled by SLUB. Larger object allocations
* are passed through to the page allocator. The page allocator "fastpath"
* is relatively slow so we need this value sufficiently high so that
@@ -149,58 +138,6 @@ struct kmem_cache {
extern struct kmem_cache *kmalloc_caches[SLUB_PAGE_SHIFT];
/*
- * Sorry that the following has to be that ugly but some versions of GCC
- * have trouble with constant propagation and loops.
- */
-static __always_inline int kmalloc_index(size_t size)
-{
- if (!size)
- return 0;
-
- if (size <= KMALLOC_MIN_SIZE)
- return KMALLOC_SHIFT_LOW;
-
- if (KMALLOC_MIN_SIZE <= 32 && size > 64 && size <= 96)
- return 1;
- if (KMALLOC_MIN_SIZE <= 64 && size > 128 && size <= 192)
- return 2;
- if (size <= 8) return 3;
- if (size <= 16) return 4;
- if (size <= 32) return 5;
- if (size <= 64) return 6;
- if (size <= 128) return 7;
- if (size <= 256) return 8;
- if (size <= 512) return 9;
- if (size <= 1024) return 10;
- if (size <= 2 * 1024) return 11;
- if (size <= 4 * 1024) return 12;
-/*
- * The following is only needed to support architectures with a larger page
- * size than 4k. We need to support 2 * PAGE_SIZE here. So for a 64k page
- * size we would have to go up to 128k.
- */
- if (size <= 8 * 1024) return 13;
- if (size <= 16 * 1024) return 14;
- if (size <= 32 * 1024) return 15;
- if (size <= 64 * 1024) return 16;
- if (size <= 128 * 1024) return 17;
- if (size <= 256 * 1024) return 18;
- if (size <= 512 * 1024) return 19;
- if (size <= 1024 * 1024) return 20;
- if (size <= 2 * 1024 * 1024) return 21;
- BUG();
- return -1; /* Will never be reached */
-
-/*
- * What we really wanted to do and cannot do because of compiler issues is:
- * int i;
- * for (i = KMALLOC_SHIFT_LOW; i <= KMALLOC_SHIFT_HIGH; i++)
- * if (size <= (1 << i))
- * return i;
- */
-}
-
-/*
* Find the slab cache for a given combination of allocation flags and size.
*
* This ought to end up with a global pointer to the right cache
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [18/18] Common definition for kmem_cache_node
[not found] <20121101214538.971500204@linux.com>
` (15 preceding siblings ...)
2012-11-01 21:48 ` CK5 [09/18] slab: Use common kmalloc_index/kmalloc_size functions Christoph Lameter
@ 2012-11-01 21:48 ` Christoph Lameter
2012-11-01 21:55 ` CK5 [10/18] slab: Common name for the per node structures Christoph Lameter
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:48 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Put the definitions for the kmem_cache_node structures together so that
we have one structure. That will allow us to create more common fields in
the future which could yield more opportunities to share code.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 16:13:57.222655245 -0500
+++ linux/mm/slab.c 2012-11-01 16:15:21.371836357 -0500
@@ -285,23 +285,6 @@ struct arraycache_init {
};
/*
- * The slab lists for all objects.
- */
-struct kmem_cache_node {
- struct list_head slabs_partial; /* partial list first, better asm code */
- struct list_head slabs_full;
- struct list_head slabs_free;
- unsigned long free_objects;
- unsigned int free_limit;
- unsigned int colour_next; /* Per-node cache coloring */
- spinlock_t list_lock;
- struct array_cache *shared; /* shared per node */
- struct array_cache **alien; /* on other nodes */
- unsigned long next_reap; /* updated without locking */
- int free_touched; /* updated without locking */
-};
-
-/*
* Need this for bootstrapping a per node allocator.
*/
#define NUM_INIT_LISTS (3 * MAX_NUMNODES)
Index: linux/mm/slab.h
===================================================================
--- linux.orig/mm/slab.h 2012-11-01 16:13:47.466518289 -0500
+++ linux/mm/slab.h 2012-11-01 16:15:21.371836357 -0500
@@ -110,3 +110,36 @@ void slabinfo_show_stats(struct seq_file
ssize_t slabinfo_write(struct file *file, const char __user *buffer,
size_t count, loff_t *ppos);
#endif
+
+
+/*
+ * The slab lists for all objects.
+ */
+struct kmem_cache_node {
+ spinlock_t list_lock;
+
+#ifdef CONFIG_SLAB
+ struct list_head slabs_partial; /* partial list first, better asm code */
+ struct list_head slabs_full;
+ struct list_head slabs_free;
+ unsigned long free_objects;
+ unsigned int free_limit;
+ unsigned int colour_next; /* Per-node cache coloring */
+ struct array_cache *shared; /* shared per node */
+ struct array_cache **alien; /* on other nodes */
+ unsigned long next_reap; /* updated without locking */
+ int free_touched; /* updated without locking */
+#endif
+
+#ifdef CONFIG_SLUB
+ unsigned long nr_partial;
+ struct list_head partial;
+#ifdef CONFIG_SLUB_DEBUG
+ atomic_long_t nr_slabs;
+ atomic_long_t total_objects;
+ struct list_head full;
+#endif
+#endif
+
+};
+
Index: linux/include/linux/slub_def.h
===================================================================
--- linux.orig/include/linux/slub_def.h 2012-11-01 16:12:48.217686485 -0500
+++ linux/include/linux/slub_def.h 2012-11-01 16:15:21.371836357 -0500
@@ -53,17 +53,6 @@ struct kmem_cache_cpu {
#endif
};
-struct kmem_cache_node {
- spinlock_t list_lock; /* Protect partial list and nr_partial */
- unsigned long nr_partial;
- struct list_head partial;
-#ifdef CONFIG_SLUB_DEBUG
- atomic_long_t nr_slabs;
- atomic_long_t total_objects;
- struct list_head full;
-#endif
-};
-
/*
* Word size structure that can be atomically updated or read and that
* contains both the order and the number of objects that a slab of the
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [09/18] slab: Use common kmalloc_index/kmalloc_size functions
[not found] <20121101214538.971500204@linux.com>
` (14 preceding siblings ...)
2012-11-01 21:48 ` CK5 [17/18] slab: Rename list3/l3 to node Christoph Lameter
@ 2012-11-01 21:48 ` Christoph Lameter
2012-11-01 21:48 ` CK5 [18/18] Common definition for kmem_cache_node Christoph Lameter
2012-11-01 21:55 ` CK5 [10/18] slab: Common name for the per node structures Christoph Lameter
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:48 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Make slab use the common functions. We can get rid of a lot
of old ugly stuff as a results. Among them the sizes
array and the weird include/linux/kmalloc_sizes file and
some pretty bad #include statements in slab_def.h.
The one thing that is different in slab is that the 32 byte
cache will also be created for arches that have page sizes
larger than 4K. There are numerous smaller allocations that
SLOB and SLUB can handle better because of their support for
smaller allocation sizes so lets keep the 32 byte slab also
for arches with > 4K pages.
Reviewed-by: Glauber Costa <glommer@parallels.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 10:10:21.801994185 -0500
+++ linux/mm/slab.c 2012-11-01 10:10:30.522138714 -0500
@@ -317,34 +317,18 @@ static void free_block(struct kmem_cache
static int enable_cpucache(struct kmem_cache *cachep, gfp_t gfp);
static void cache_reap(struct work_struct *unused);
-/*
- * This function must be completely optimized away if a constant is passed to
- * it. Mostly the same as what is in linux/slab.h except it returns an index.
- */
-static __always_inline int index_of(const size_t size)
-{
- extern void __bad_size(void);
+struct kmem_cache *kmalloc_caches[KMALLOC_SHIFT_HIGH + 1];
+EXPORT_SYMBOL(kmalloc_caches);
- if (__builtin_constant_p(size)) {
- int i = 0;
-
-#define CACHE(x) \
- if (size <=x) \
- return i; \
- else \
- i++;
-#include <linux/kmalloc_sizes.h>
-#undef CACHE
- __bad_size();
- } else
- __bad_size();
- return 0;
-}
+#ifdef CONFIG_ZONE_DMA
+struct kmem_cache *kmalloc_dma_caches[KMALLOC_SHIFT_HIGH + 1];
+EXPORT_SYMBOL(kmalloc_dma_caches);
+#endif
static int slab_early_init = 1;
-#define INDEX_AC index_of(sizeof(struct arraycache_init))
-#define INDEX_L3 index_of(sizeof(struct kmem_list3))
+#define INDEX_AC kmalloc_index(sizeof(struct arraycache_init))
+#define INDEX_L3 kmalloc_index(sizeof(struct kmem_list3))
static void kmem_list3_init(struct kmem_list3 *parent)
{
@@ -523,30 +507,6 @@ static inline unsigned int obj_to_index(
return reciprocal_divide(offset, cache->reciprocal_buffer_size);
}
-/*
- * These are the default caches for kmalloc. Custom caches can have other sizes.
- */
-struct cache_sizes malloc_sizes[] = {
-#define CACHE(x) { .cs_size = (x) },
-#include <linux/kmalloc_sizes.h>
- CACHE(ULONG_MAX)
-#undef CACHE
-};
-EXPORT_SYMBOL(malloc_sizes);
-
-/* Must match cache_sizes above. Out of line to keep cache footprint low. */
-struct cache_names {
- char *name;
- char *name_dma;
-};
-
-static struct cache_names __initdata cache_names[] = {
-#define CACHE(x) { .name = "size-" #x, .name_dma = "size-" #x "(DMA)" },
-#include <linux/kmalloc_sizes.h>
- {NULL,}
-#undef CACHE
-};
-
static struct arraycache_init initarray_generic =
{ {0, BOOT_CPUCACHE_ENTRIES, 1, 0} };
@@ -624,19 +584,23 @@ static void slab_set_debugobj_lock_class
static void init_node_lock_keys(int q)
{
- struct cache_sizes *s = malloc_sizes;
+ int i;
if (slab_state < UP)
return;
- for (s = malloc_sizes; s->cs_size != ULONG_MAX; s++) {
+ for (i = 1; i < PAGE_SHIFT + MAX_ORDER; i++) {
struct kmem_list3 *l3;
+ struct kmem_cache *cache = kmalloc_caches[i];
+
+ if (!cache)
+ continue;
- l3 = s->cs_cachep->nodelists[q];
- if (!l3 || OFF_SLAB(s->cs_cachep))
+ l3 = cache->nodelists[q];
+ if (!l3 || OFF_SLAB(cache))
continue;
- slab_set_lock_classes(s->cs_cachep, &on_slab_l3_key,
+ slab_set_lock_classes(cache, &on_slab_l3_key,
&on_slab_alc_key, q);
}
}
@@ -676,20 +640,19 @@ static inline struct array_cache *cpu_ca
static inline struct kmem_cache *__find_general_cachep(size_t size,
gfp_t gfpflags)
{
- struct cache_sizes *csizep = malloc_sizes;
+ int i;
#if DEBUG
/* This happens if someone tries to call
* kmem_cache_create(), or __kmalloc(), before
* the generic caches are initialized.
*/
- BUG_ON(malloc_sizes[INDEX_AC].cs_cachep == NULL);
+ BUG_ON(kmalloc_caches[INDEX_AC] == NULL);
#endif
if (!size)
return ZERO_SIZE_PTR;
- while (size > csizep->cs_size)
- csizep++;
+ i = kmalloc_index(size);
/*
* Really subtle: The last entry with cs->cs_size==ULONG_MAX
@@ -698,9 +661,9 @@ static inline struct kmem_cache *__find_
*/
#ifdef CONFIG_ZONE_DMA
if (unlikely(gfpflags & GFP_DMA))
- return csizep->cs_dmacachep;
+ return kmalloc_dma_caches[i];
#endif
- return csizep->cs_cachep;
+ return kmalloc_caches[i];
}
static struct kmem_cache *kmem_find_general_cachep(size_t size, gfp_t gfpflags)
@@ -1570,8 +1533,6 @@ static void setup_nodelists_pointer(stru
*/
void __init kmem_cache_init(void)
{
- struct cache_sizes *sizes;
- struct cache_names *names;
int i;
kmem_cache = &kmem_cache_boot;
@@ -1624,8 +1585,6 @@ void __init kmem_cache_init(void)
SLAB_HWCACHE_ALIGN);
/* 2+3) create the kmalloc caches */
- sizes = malloc_sizes;
- names = cache_names;
/*
* Initialize the caches that provide memory for the array cache and the
@@ -1633,35 +1592,39 @@ void __init kmem_cache_init(void)
* bug.
*/
- sizes[INDEX_AC].cs_cachep = create_kmalloc_cache(names[INDEX_AC].name,
- sizes[INDEX_AC].cs_size, ARCH_KMALLOC_FLAGS);
+ kmalloc_caches[INDEX_AC] = create_kmalloc_cache("kmalloc-ac",
+ kmalloc_size(INDEX_AC), ARCH_KMALLOC_FLAGS);
if (INDEX_AC != INDEX_L3)
- sizes[INDEX_L3].cs_cachep =
- create_kmalloc_cache(names[INDEX_L3].name,
- sizes[INDEX_L3].cs_size, ARCH_KMALLOC_FLAGS);
+ kmalloc_caches[INDEX_L3] =
+ create_kmalloc_cache("kmalloc-l3",
+ kmalloc_size(INDEX_L3), ARCH_KMALLOC_FLAGS);
slab_early_init = 0;
- while (sizes->cs_size != ULONG_MAX) {
- /*
- * For performance, all the general caches are L1 aligned.
- * This should be particularly beneficial on SMP boxes, as it
- * eliminates "false sharing".
- * Note for systems short on memory removing the alignment will
- * allow tighter packing of the smaller caches.
- */
- if (!sizes->cs_cachep)
- sizes->cs_cachep = create_kmalloc_cache(names->name,
- sizes->cs_size, ARCH_KMALLOC_FLAGS);
+ for (i = 1; i < PAGE_SHIFT + MAX_ORDER; i++) {
+ size_t cs_size = kmalloc_size(i);
+
+ if (cs_size < KMALLOC_MIN_SIZE)
+ continue;
+
+ if (!kmalloc_caches[i]) {
+ /*
+ * For performance, all the general caches are L1 aligned.
+ * This should be particularly beneficial on SMP boxes, as it
+ * eliminates "false sharing".
+ * Note for systems short on memory removing the alignment will
+ * allow tighter packing of the smaller caches.
+ */
+ kmalloc_caches[i] = create_kmalloc_cache("kmalloc",
+ cs_size, ARCH_KMALLOC_FLAGS);
+ }
#ifdef CONFIG_ZONE_DMA
- sizes->cs_dmacachep = create_kmalloc_cache(
- names->name_dma, sizes->cs_size,
+ kmalloc_dma_caches[i] = create_kmalloc_cache(
+ "kmalloc-dma", cs_size,
SLAB_CACHE_DMA|ARCH_KMALLOC_FLAGS);
#endif
- sizes++;
- names++;
}
/* 4) Replace the bootstrap head arrays */
{
@@ -1680,17 +1643,16 @@ void __init kmem_cache_init(void)
ptr = kmalloc(sizeof(struct arraycache_init), GFP_NOWAIT);
- BUG_ON(cpu_cache_get(malloc_sizes[INDEX_AC].cs_cachep)
+ BUG_ON(cpu_cache_get(kmalloc_caches[INDEX_AC])
!= &initarray_generic.cache);
- memcpy(ptr, cpu_cache_get(malloc_sizes[INDEX_AC].cs_cachep),
+ memcpy(ptr, cpu_cache_get(kmalloc_caches[INDEX_AC]),
sizeof(struct arraycache_init));
/*
* Do not assume that spinlocks can be initialized via memcpy:
*/
spin_lock_init(&ptr->lock);
- malloc_sizes[INDEX_AC].cs_cachep->array[smp_processor_id()] =
- ptr;
+ kmalloc_caches[INDEX_AC]->array[smp_processor_id()] = ptr;
}
/* 5) Replace the bootstrap kmem_list3's */
{
@@ -1699,17 +1661,39 @@ void __init kmem_cache_init(void)
for_each_online_node(nid) {
init_list(kmem_cache, &initkmem_list3[CACHE_CACHE + nid], nid);
- init_list(malloc_sizes[INDEX_AC].cs_cachep,
+ init_list(kmalloc_caches[INDEX_AC],
&initkmem_list3[SIZE_AC + nid], nid);
if (INDEX_AC != INDEX_L3) {
- init_list(malloc_sizes[INDEX_L3].cs_cachep,
+ init_list(kmalloc_caches[INDEX_L3],
&initkmem_list3[SIZE_L3 + nid], nid);
}
}
}
slab_state = UP;
+
+ /* Create the proper names */
+ for (i = 1; i < PAGE_SHIFT + MAX_ORDER; i++) {
+ char *s;
+ struct kmem_cache *c = kmalloc_caches[i];
+
+ if (!c)
+ continue;
+
+ s = kasprintf(GFP_NOWAIT, "kmalloc-%d", kmalloc_size(i));
+
+ BUG_ON(!s);
+ c->name = s;
+
+#ifdef CONFIG_ZONE_DMA
+ c = kmalloc_dma_caches[i];
+ BUG_ON(!c);
+ s = kasprintf(GFP_NOWAIT, "dma-kmalloc-%d", kmalloc_size(i));
+ BUG_ON(!s);
+ c->name = s;
+#endif
+ }
}
void __init kmem_cache_init_late(void)
@@ -2395,10 +2379,9 @@ __kmem_cache_create (struct kmem_cache *
size += BYTES_PER_WORD;
}
#if FORCED_DEBUG && defined(CONFIG_DEBUG_PAGEALLOC)
- if (size >= malloc_sizes[INDEX_L3 + 1].cs_size
- && cachep->object_size > cache_line_size()
- && ALIGN(size, cachep->align) < PAGE_SIZE) {
- cachep->obj_offset += PAGE_SIZE - ALIGN(size, cachep->align);
+ if (size >= kmalloc_size(INDEX_L3 + 1)
+ && cachep->object_size > cache_line_size() && ALIGN(size, align) < PAGE_SIZE) {
+ cachep->obj_offset += PAGE_SIZE - ALIGN(size, align);
size = PAGE_SIZE;
}
#endif
Index: linux/include/linux/kmalloc_sizes.h
===================================================================
--- linux.orig/include/linux/kmalloc_sizes.h 2012-11-01 10:09:46.621410440 -0500
+++ /dev/null 1970-01-01 00:00:00.000000000 +0000
@@ -1,45 +0,0 @@
-#if (PAGE_SIZE == 4096)
- CACHE(32)
-#endif
- CACHE(64)
-#if L1_CACHE_BYTES < 64
- CACHE(96)
-#endif
- CACHE(128)
-#if L1_CACHE_BYTES < 128
- CACHE(192)
-#endif
- CACHE(256)
- CACHE(512)
- CACHE(1024)
- CACHE(2048)
- CACHE(4096)
- CACHE(8192)
- CACHE(16384)
- CACHE(32768)
- CACHE(65536)
- CACHE(131072)
-#if KMALLOC_MAX_SIZE >= 262144
- CACHE(262144)
-#endif
-#if KMALLOC_MAX_SIZE >= 524288
- CACHE(524288)
-#endif
-#if KMALLOC_MAX_SIZE >= 1048576
- CACHE(1048576)
-#endif
-#if KMALLOC_MAX_SIZE >= 2097152
- CACHE(2097152)
-#endif
-#if KMALLOC_MAX_SIZE >= 4194304
- CACHE(4194304)
-#endif
-#if KMALLOC_MAX_SIZE >= 8388608
- CACHE(8388608)
-#endif
-#if KMALLOC_MAX_SIZE >= 16777216
- CACHE(16777216)
-#endif
-#if KMALLOC_MAX_SIZE >= 33554432
- CACHE(33554432)
-#endif
Index: linux/include/linux/slab_def.h
===================================================================
--- linux.orig/include/linux/slab_def.h 2012-11-01 10:10:11.005815158 -0500
+++ linux/include/linux/slab_def.h 2012-11-01 10:10:30.522138714 -0500
@@ -11,8 +11,6 @@
*/
#include <linux/init.h>
-#include <asm/page.h> /* kmalloc_sizes.h needs PAGE_SIZE */
-#include <asm/cache.h> /* kmalloc_sizes.h needs L1_CACHE_BYTES */
#include <linux/compiler.h>
/*
@@ -97,15 +95,8 @@ struct kmem_cache {
*/
};
-/* Size description struct for general caches. */
-struct cache_sizes {
- size_t cs_size;
- struct kmem_cache *cs_cachep;
-#ifdef CONFIG_ZONE_DMA
- struct kmem_cache *cs_dmacachep;
-#endif
-};
-extern struct cache_sizes malloc_sizes[];
+extern struct kmem_cache *kmalloc_caches[PAGE_SHIFT + MAX_ORDER];
+extern struct kmem_cache *kmalloc_dma_caches[PAGE_SHIFT + MAX_ORDER];
void *kmem_cache_alloc(struct kmem_cache *, gfp_t);
void *__kmalloc(size_t size, gfp_t flags);
@@ -126,26 +117,19 @@ static __always_inline void *kmalloc(siz
void *ret;
if (__builtin_constant_p(size)) {
- int i = 0;
+ int i;
if (!size)
return ZERO_SIZE_PTR;
-#define CACHE(x) \
- if (size <= x) \
- goto found; \
- else \
- i++;
-#include <linux/kmalloc_sizes.h>
-#undef CACHE
- return NULL;
-found:
+ i = kmalloc_index(size);
+
#ifdef CONFIG_ZONE_DMA
if (flags & GFP_DMA)
- cachep = malloc_sizes[i].cs_dmacachep;
+ cachep = kmalloc_dma_caches[i];
else
#endif
- cachep = malloc_sizes[i].cs_cachep;
+ cachep = kmalloc_caches[i];
ret = kmem_cache_alloc_trace(cachep, flags, size);
@@ -179,26 +163,19 @@ static __always_inline void *kmalloc_nod
struct kmem_cache *cachep;
if (__builtin_constant_p(size)) {
- int i = 0;
+ int i;
if (!size)
return ZERO_SIZE_PTR;
-#define CACHE(x) \
- if (size <= x) \
- goto found; \
- else \
- i++;
-#include <linux/kmalloc_sizes.h>
-#undef CACHE
- return NULL;
-found:
+ i = kmalloc_index(size);
+
#ifdef CONFIG_ZONE_DMA
if (flags & GFP_DMA)
- cachep = malloc_sizes[i].cs_dmacachep;
+ cachep = kmalloc_dma_caches[i];
else
#endif
- cachep = malloc_sizes[i].cs_cachep;
+ cachep = kmalloc_caches[i];
return kmem_cache_alloc_node_trace(cachep, flags, node, size);
}
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [17/18] slab: Rename list3/l3 to node
[not found] <20121101214538.971500204@linux.com>
` (13 preceding siblings ...)
2012-11-01 21:48 ` CK5 [08/18] Common kmalloc slab index determination Christoph Lameter
@ 2012-11-01 21:48 ` Christoph Lameter
2012-11-01 21:48 ` CK5 [09/18] slab: Use common kmalloc_index/kmalloc_size functions Christoph Lameter
` (2 subsequent siblings)
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:48 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
The list3 or l3 pointers are pointing to per node structures. Reflect
that in the names of variables used.
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 16:12:48.217686485 -0500
+++ linux/mm/slab.c 2012-11-01 16:13:57.222655245 -0500
@@ -305,13 +305,13 @@ struct kmem_cache_node {
* Need this for bootstrapping a per node allocator.
*/
#define NUM_INIT_LISTS (3 * MAX_NUMNODES)
-static struct kmem_cache_node __initdata initkmem_list3[NUM_INIT_LISTS];
+static struct kmem_cache_node __initdata init_kmem_cache_node[NUM_INIT_LISTS];
#define CACHE_CACHE 0
#define SIZE_AC MAX_NUMNODES
-#define SIZE_L3 (2 * MAX_NUMNODES)
+#define SIZE_NODE (2 * MAX_NUMNODES)
static int drain_freelist(struct kmem_cache *cache,
- struct kmem_cache_node *l3, int tofree);
+ struct kmem_cache_node *n, int tofree);
static void free_block(struct kmem_cache *cachep, void **objpp, int len,
int node);
static int enable_cpucache(struct kmem_cache *cachep, gfp_t gfp);
@@ -320,9 +320,9 @@ static void cache_reap(struct work_struc
static int slab_early_init = 1;
#define INDEX_AC kmalloc_index(sizeof(struct arraycache_init))
-#define INDEX_L3 kmalloc_index(sizeof(struct kmem_cache_node))
+#define INDEX_NODE kmalloc_index(sizeof(struct kmem_cache_node))
-static void kmem_list3_init(struct kmem_cache_node *parent)
+static void kmem_cache_node_init(struct kmem_cache_node *parent)
{
INIT_LIST_HEAD(&parent->slabs_full);
INIT_LIST_HEAD(&parent->slabs_partial);
@@ -537,15 +537,15 @@ static void slab_set_lock_classes(struct
int q)
{
struct array_cache **alc;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
int r;
- l3 = cachep->node[q];
- if (!l3)
+ n = cachep->node[q];
+ if (!n)
return;
- lockdep_set_class(&l3->list_lock, l3_key);
- alc = l3->alien;
+ lockdep_set_class(&n->list_lock, l3_key);
+ alc = n->alien;
/*
* FIXME: This check for BAD_ALIEN_MAGIC
* should go away when common slab code is taught to
@@ -582,14 +582,14 @@ static void init_node_lock_keys(int q)
return;
for (i = 1; i < PAGE_SHIFT + MAX_ORDER; i++) {
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
struct kmem_cache *cache = kmalloc_caches[i];
if (!cache)
continue;
- l3 = cache->node[q];
- if (!l3 || OFF_SLAB(cache))
+ n = cache->node[q];
+ if (!n || OFF_SLAB(cache))
continue;
slab_set_lock_classes(cache, &on_slab_l3_key,
@@ -830,29 +830,29 @@ static inline bool is_slab_pfmemalloc(st
static void recheck_pfmemalloc_active(struct kmem_cache *cachep,
struct array_cache *ac)
{
- struct kmem_cache_node *l3 = cachep->node[numa_mem_id()];
+ struct kmem_cache_node *n = cachep->node[numa_mem_id()];
struct slab *slabp;
unsigned long flags;
if (!pfmemalloc_active)
return;
- spin_lock_irqsave(&l3->list_lock, flags);
- list_for_each_entry(slabp, &l3->slabs_full, list)
+ spin_lock_irqsave(&n->list_lock, flags);
+ list_for_each_entry(slabp, &n->slabs_full, list)
if (is_slab_pfmemalloc(slabp))
goto out;
- list_for_each_entry(slabp, &l3->slabs_partial, list)
+ list_for_each_entry(slabp, &n->slabs_partial, list)
if (is_slab_pfmemalloc(slabp))
goto out;
- list_for_each_entry(slabp, &l3->slabs_free, list)
+ list_for_each_entry(slabp, &n->slabs_free, list)
if (is_slab_pfmemalloc(slabp))
goto out;
pfmemalloc_active = false;
out:
- spin_unlock_irqrestore(&l3->list_lock, flags);
+ spin_unlock_irqrestore(&n->list_lock, flags);
}
static void *__ac_get_obj(struct kmem_cache *cachep, struct array_cache *ac,
@@ -863,7 +863,7 @@ static void *__ac_get_obj(struct kmem_ca
/* Ensure the caller is allowed to use objects from PFMEMALLOC slab */
if (unlikely(is_obj_pfmemalloc(objp))) {
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
if (gfp_pfmemalloc_allowed(flags)) {
clear_obj_pfmemalloc(&objp);
@@ -885,8 +885,8 @@ static void *__ac_get_obj(struct kmem_ca
* If there are empty slabs on the slabs_free list and we are
* being forced to refill the cache, mark this one !pfmemalloc.
*/
- l3 = cachep->node[numa_mem_id()];
- if (!list_empty(&l3->slabs_free) && force_refill) {
+ n = cachep->node[numa_mem_id()];
+ if (!list_empty(&n->slabs_free) && force_refill) {
struct slab *slabp = virt_to_slab(objp);
ClearPageSlabPfmemalloc(virt_to_head_page(slabp->s_mem));
clear_obj_pfmemalloc(&objp);
@@ -963,7 +963,7 @@ static int transfer_objects(struct array
#ifndef CONFIG_NUMA
#define drain_alien_cache(cachep, alien) do { } while (0)
-#define reap_alien(cachep, l3) do { } while (0)
+#define reap_alien(cachep, n) do { } while (0)
static inline struct array_cache **alloc_alien_cache(int node, int limit, gfp_t gfp)
{
@@ -1035,33 +1035,33 @@ static void free_alien_cache(struct arra
static void __drain_alien_cache(struct kmem_cache *cachep,
struct array_cache *ac, int node)
{
- struct kmem_cache_node *rl3 = cachep->node[node];
+ struct kmem_cache_node *n = cachep->node[node];
if (ac->avail) {
- spin_lock(&rl3->list_lock);
+ spin_lock(&n->list_lock);
/*
* Stuff objects into the remote nodes shared array first.
* That way we could avoid the overhead of putting the objects
* into the free lists and getting them back later.
*/
- if (rl3->shared)
- transfer_objects(rl3->shared, ac, ac->limit);
+ if (n->shared)
+ transfer_objects(n->shared, ac, ac->limit);
free_block(cachep, ac->entry, ac->avail, node);
ac->avail = 0;
- spin_unlock(&rl3->list_lock);
+ spin_unlock(&n->list_lock);
}
}
/*
* Called from cache_reap() to regularly drain alien caches round robin.
*/
-static void reap_alien(struct kmem_cache *cachep, struct kmem_cache_node *l3)
+static void reap_alien(struct kmem_cache *cachep, struct kmem_cache_node *n)
{
int node = __this_cpu_read(slab_reap_node);
- if (l3->alien) {
- struct array_cache *ac = l3->alien[node];
+ if (n->alien) {
+ struct array_cache *ac = n->alien[node];
if (ac && ac->avail && spin_trylock_irq(&ac->lock)) {
__drain_alien_cache(cachep, ac, node);
@@ -1091,7 +1091,7 @@ static inline int cache_free_alien(struc
{
struct slab *slabp = virt_to_slab(objp);
int nodeid = slabp->nodeid;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
struct array_cache *alien = NULL;
int node;
@@ -1104,10 +1104,10 @@ static inline int cache_free_alien(struc
if (likely(slabp->nodeid == node))
return 0;
- l3 = cachep->node[node];
+ n = cachep->node[node];
STATS_INC_NODEFREES(cachep);
- if (l3->alien && l3->alien[nodeid]) {
- alien = l3->alien[nodeid];
+ if (n->alien && n->alien[nodeid]) {
+ alien = n->alien[nodeid];
spin_lock(&alien->lock);
if (unlikely(alien->avail == alien->limit)) {
STATS_INC_ACOVERFLOW(cachep);
@@ -1126,7 +1126,7 @@ static inline int cache_free_alien(struc
/*
* Allocates and initializes node for a node on each slab cache, used for
- * either memory or cpu hotplug. If memory is being hot-added, the kmem_list3
+ * either memory or cpu hotplug. If memory is being hot-added, the kmem_cache_node
* will be allocated off-node since memory is not yet online for the new node.
* When hotplugging memory or a cpu, existing node are not replaced if
* already in use.
@@ -1136,7 +1136,7 @@ static inline int cache_free_alien(struc
static int init_cache_node_node(int node)
{
struct kmem_cache *cachep;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
const int memsize = sizeof(struct kmem_cache_node);
list_for_each_entry(cachep, &slab_caches, list) {
@@ -1146,11 +1146,11 @@ static int init_cache_node_node(int node
* node has not already allocated this
*/
if (!cachep->node[node]) {
- l3 = kmalloc_node(memsize, GFP_KERNEL, node);
- if (!l3)
+ n = kmalloc_node(memsize, GFP_KERNEL, node);
+ if (!n)
return -ENOMEM;
- kmem_list3_init(l3);
- l3->next_reap = jiffies + REAPTIMEOUT_LIST3 +
+ kmem_cache_node_init(n);
+ n->next_reap = jiffies + REAPTIMEOUT_LIST3 +
((unsigned long)cachep) % REAPTIMEOUT_LIST3;
/*
@@ -1158,7 +1158,7 @@ static int init_cache_node_node(int node
* go. slab_mutex is sufficient
* protection here.
*/
- cachep->node[node] = l3;
+ cachep->node[node] = n;
}
spin_lock_irq(&cachep->node[node]->list_lock);
@@ -1173,7 +1173,7 @@ static int init_cache_node_node(int node
static void __cpuinit cpuup_canceled(long cpu)
{
struct kmem_cache *cachep;
- struct kmem_cache_node *l3 = NULL;
+ struct kmem_cache_node *n = NULL;
int node = cpu_to_mem(cpu);
const struct cpumask *mask = cpumask_of_node(node);
@@ -1185,34 +1185,34 @@ static void __cpuinit cpuup_canceled(lon
/* cpu is dead; no one can alloc from it. */
nc = cachep->array[cpu];
cachep->array[cpu] = NULL;
- l3 = cachep->node[node];
+ n = cachep->node[node];
- if (!l3)
+ if (!n)
goto free_array_cache;
- spin_lock_irq(&l3->list_lock);
+ spin_lock_irq(&n->list_lock);
- /* Free limit for this kmem_list3 */
- l3->free_limit -= cachep->batchcount;
+ /* Free limit for this kmem_cache_node */
+ n->free_limit -= cachep->batchcount;
if (nc)
free_block(cachep, nc->entry, nc->avail, node);
if (!cpumask_empty(mask)) {
- spin_unlock_irq(&l3->list_lock);
+ spin_unlock_irq(&n->list_lock);
goto free_array_cache;
}
- shared = l3->shared;
+ shared = n->shared;
if (shared) {
free_block(cachep, shared->entry,
shared->avail, node);
- l3->shared = NULL;
+ n->shared = NULL;
}
- alien = l3->alien;
- l3->alien = NULL;
+ alien = n->alien;
+ n->alien = NULL;
- spin_unlock_irq(&l3->list_lock);
+ spin_unlock_irq(&n->list_lock);
kfree(shared);
if (alien) {
@@ -1228,17 +1228,17 @@ free_array_cache:
* shrink each nodelist to its limit.
*/
list_for_each_entry(cachep, &slab_caches, list) {
- l3 = cachep->node[node];
- if (!l3)
+ n = cachep->node[node];
+ if (!n)
continue;
- drain_freelist(cachep, l3, l3->free_objects);
+ drain_freelist(cachep, n, n->free_objects);
}
}
static int __cpuinit cpuup_prepare(long cpu)
{
struct kmem_cache *cachep;
- struct kmem_cache_node *l3 = NULL;
+ struct kmem_cache_node *n = NULL;
int node = cpu_to_mem(cpu);
int err;
@@ -1246,7 +1246,7 @@ static int __cpuinit cpuup_prepare(long
* We need to do this right in the beginning since
* alloc_arraycache's are going to use this list.
* kmalloc_node allows us to add the slab to the right
- * kmem_list3 and not this cpu's kmem_list3
+ * kmem_cache_node and not this cpu's kmem_cache_node
*/
err = init_cache_node_node(node);
if (err < 0)
@@ -1283,25 +1283,25 @@ static int __cpuinit cpuup_prepare(long
}
}
cachep->array[cpu] = nc;
- l3 = cachep->node[node];
- BUG_ON(!l3);
+ n = cachep->node[node];
+ BUG_ON(!n);
- spin_lock_irq(&l3->list_lock);
- if (!l3->shared) {
+ spin_lock_irq(&n->list_lock);
+ if (!n->shared) {
/*
* We are serialised from CPU_DEAD or
* CPU_UP_CANCELLED by the cpucontrol lock
*/
- l3->shared = shared;
+ n->shared = shared;
shared = NULL;
}
#ifdef CONFIG_NUMA
- if (!l3->alien) {
- l3->alien = alien;
+ if (!n->alien) {
+ n->alien = alien;
alien = NULL;
}
#endif
- spin_unlock_irq(&l3->list_lock);
+ spin_unlock_irq(&n->list_lock);
kfree(shared);
free_alien_cache(alien);
if (cachep->flags & SLAB_DEBUG_OBJECTS)
@@ -1353,9 +1353,9 @@ static int __cpuinit cpuup_callback(stru
case CPU_DEAD_FROZEN:
/*
* Even if all the cpus of a node are down, we don't free the
- * kmem_list3 of any cache. This to avoid a race between
+ * kmem_cache_node of any cache. This to avoid a race between
* cpu_down, and a kmalloc allocation from another cpu for
- * memory from the node of the cpu going down. The list3
+ * memory from the node of the cpu going down. The node
* structure is usually allocated from kmem_cache_create() and
* gets destroyed at kmem_cache_destroy().
*/
@@ -1389,16 +1389,16 @@ static int __meminit drain_cache_node_no
int ret = 0;
list_for_each_entry(cachep, &slab_caches, list) {
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
- l3 = cachep->node[node];
- if (!l3)
+ n = cachep->node[node];
+ if (!n)
continue;
- drain_freelist(cachep, l3, l3->free_objects);
+ drain_freelist(cachep, n, n->free_objects);
- if (!list_empty(&l3->slabs_full) ||
- !list_empty(&l3->slabs_partial)) {
+ if (!list_empty(&n->slabs_full) ||
+ !list_empty(&n->slabs_partial)) {
ret = -EBUSY;
break;
}
@@ -1440,7 +1440,7 @@ out:
#endif /* CONFIG_NUMA && CONFIG_MEMORY_HOTPLUG */
/*
- * swap the static kmem_list3 with kmalloced memory
+ * swap the static kmem_cache_node with kmalloced memory
*/
static void __init init_list(struct kmem_cache *cachep, struct kmem_cache_node *list,
int nodeid)
@@ -1461,15 +1461,15 @@ static void __init init_list(struct kmem
}
/*
- * For setting up all the kmem_list3s for cache whose buffer_size is same as
- * size of kmem_list3.
+ * For setting up all the kmem_cache_node for cache whose buffer_size is same as
+ * size of kmem_cache_node.
*/
-static void __init set_up_list3s(struct kmem_cache *cachep, int index)
+static void __init set_up_node(struct kmem_cache *cachep, int index)
{
int node;
for_each_online_node(node) {
- cachep->node[node] = &initkmem_list3[index + node];
+ cachep->node[node] = &init_kmem_cache_node[index + node];
cachep->node[node]->next_reap = jiffies +
REAPTIMEOUT_LIST3 +
((unsigned long)cachep) % REAPTIMEOUT_LIST3;
@@ -1500,9 +1500,9 @@ void __init kmem_cache_init(void)
use_alien_caches = 0;
for (i = 0; i < NUM_INIT_LISTS; i++)
- kmem_list3_init(&initkmem_list3[i]);
+ kmem_cache_node_init(&init_kmem_cache_node[i]);
- set_up_list3s(kmem_cache, CACHE_CACHE);
+ set_up_node(kmem_cache, CACHE_CACHE);
/*
* Fragmentation resistance on low memory - only use bigger
@@ -1518,7 +1518,7 @@ void __init kmem_cache_init(void)
* kmem_cache structures of all caches, except kmem_cache itself:
* kmem_cache is statically allocated.
* Initially an __init data area is used for the head array and the
- * kmem_list3 structures, it's replaced with a kmalloc allocated
+ * kmem_cache_node structures, it's replaced with a kmalloc allocated
* array at the end of the bootstrap.
* 2) Create the first kmalloc cache.
* The struct kmem_cache for the new cache is allocated normally.
@@ -1527,7 +1527,7 @@ void __init kmem_cache_init(void)
* head arrays.
* 4) Replace the __init data head arrays for kmem_cache and the first
* kmalloc cache with kmalloc allocated arrays.
- * 5) Replace the __init data for kmem_list3 for kmem_cache and
+ * 5) Replace the __init data for kmem_cache_node for kmem_cache and
* the other cache's with kmalloc allocated memory.
* 6) Resize the head arrays of the kmalloc caches to their final sizes.
*/
@@ -1546,17 +1546,17 @@ void __init kmem_cache_init(void)
/*
* Initialize the caches that provide memory for the array cache and the
- * kmem_list3 structures first. Without this, further allocations will
+ * kmem_cache_node structures first. Without this, further allocations will
* bug.
*/
kmalloc_caches[INDEX_AC] = create_kmalloc_cache("kmalloc-ac",
kmalloc_size(INDEX_AC), ARCH_KMALLOC_FLAGS);
- if (INDEX_AC != INDEX_L3)
- kmalloc_caches[INDEX_L3] =
- create_kmalloc_cache("kmalloc-l3",
- kmalloc_size(INDEX_L3), ARCH_KMALLOC_FLAGS);
+ if (INDEX_AC != INDEX_NODE)
+ kmalloc_caches[INDEX_NODE] =
+ create_kmalloc_cache("kmalloc-node",
+ kmalloc_size(INDEX_NODE), ARCH_KMALLOC_FLAGS);
slab_early_init = 0;
@@ -1588,19 +1588,19 @@ void __init kmem_cache_init(void)
kmalloc_caches[INDEX_AC]->array[smp_processor_id()] = ptr;
}
- /* 5) Replace the bootstrap kmem_list3's */
+ /* 5) Replace the bootstrap kmem_cache_node */
{
int nid;
for_each_online_node(nid) {
- init_list(kmem_cache, &initkmem_list3[CACHE_CACHE + nid], nid);
+ init_list(kmem_cache, &init_kmem_cache_node[CACHE_CACHE + nid], nid);
init_list(kmalloc_caches[INDEX_AC],
- &initkmem_list3[SIZE_AC + nid], nid);
+ &init_kmem_cache_node[SIZE_AC + nid], nid);
- if (INDEX_AC != INDEX_L3) {
- init_list(kmalloc_caches[INDEX_L3],
- &initkmem_list3[SIZE_L3 + nid], nid);
+ if (INDEX_AC != INDEX_NODE) {
+ init_list(kmalloc_caches[INDEX_NODE],
+ &init_kmem_cache_node[SIZE_NODE + nid], nid);
}
}
}
@@ -1666,7 +1666,7 @@ __initcall(cpucache_init);
static noinline void
slab_out_of_memory(struct kmem_cache *cachep, gfp_t gfpflags, int nodeid)
{
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
struct slab *slabp;
unsigned long flags;
int node;
@@ -1681,24 +1681,24 @@ slab_out_of_memory(struct kmem_cache *ca
unsigned long active_objs = 0, num_objs = 0, free_objects = 0;
unsigned long active_slabs = 0, num_slabs = 0;
- l3 = cachep->node[node];
- if (!l3)
+ n = cachep->node[node];
+ if (!n)
continue;
- spin_lock_irqsave(&l3->list_lock, flags);
- list_for_each_entry(slabp, &l3->slabs_full, list) {
+ spin_lock_irqsave(&n->list_lock, flags);
+ list_for_each_entry(slabp, &n->slabs_full, list) {
active_objs += cachep->num;
active_slabs++;
}
- list_for_each_entry(slabp, &l3->slabs_partial, list) {
+ list_for_each_entry(slabp, &n->slabs_partial, list) {
active_objs += slabp->inuse;
active_slabs++;
}
- list_for_each_entry(slabp, &l3->slabs_free, list)
+ list_for_each_entry(slabp, &n->slabs_free, list)
num_slabs++;
- free_objects += l3->free_objects;
- spin_unlock_irqrestore(&l3->list_lock, flags);
+ free_objects += n->free_objects;
+ spin_unlock_irqrestore(&n->list_lock, flags);
num_slabs += active_slabs;
num_objs = num_slabs * cachep->num;
@@ -2120,7 +2120,7 @@ static int __init_refok setup_cpu_cache(
if (slab_state == DOWN) {
/*
* Note: Creation of first cache (kmem_cache).
- * The setup_list3s is taken care
+ * The setup_node is taken care
* of by the caller of __kmem_cache_create
*/
cachep->array[smp_processor_id()] = &initarray_generic.cache;
@@ -2134,13 +2134,13 @@ static int __init_refok setup_cpu_cache(
cachep->array[smp_processor_id()] = &initarray_generic.cache;
/*
- * If the cache that's used by kmalloc(sizeof(kmem_list3)) is
- * the second cache, then we need to set up all its list3s,
+ * If the cache that's used by kmalloc(sizeof(kmem_cache_node)) is
+ * the second cache, then we need to set up all its node/,
* otherwise the creation of further caches will BUG().
*/
- set_up_list3s(cachep, SIZE_AC);
- if (INDEX_AC == INDEX_L3)
- slab_state = PARTIAL_L3;
+ set_up_node(cachep, SIZE_AC);
+ if (INDEX_AC == INDEX_NODE)
+ slab_state = PARTIAL_NODE;
else
slab_state = PARTIAL_ARRAYCACHE;
} else {
@@ -2149,8 +2149,8 @@ static int __init_refok setup_cpu_cache(
kmalloc(sizeof(struct arraycache_init), gfp);
if (slab_state == PARTIAL_ARRAYCACHE) {
- set_up_list3s(cachep, SIZE_L3);
- slab_state = PARTIAL_L3;
+ set_up_node(cachep, SIZE_NODE);
+ slab_state = PARTIAL_NODE;
} else {
int node;
for_each_online_node(node) {
@@ -2158,7 +2158,7 @@ static int __init_refok setup_cpu_cache(
kmalloc_node(sizeof(struct kmem_cache_node),
gfp, node);
BUG_ON(!cachep->node[node]);
- kmem_list3_init(cachep->node[node]);
+ kmem_cache_node_init(cachep->node[node]);
}
}
}
@@ -2291,7 +2291,7 @@ __kmem_cache_create (struct kmem_cache *
size += BYTES_PER_WORD;
}
#if FORCED_DEBUG && defined(CONFIG_DEBUG_PAGEALLOC)
- if (size >= kmalloc_size(INDEX_L3 + 1)
+ if (size >= kmalloc_size(INDEX_NODE + 1)
&& cachep->object_size > cache_line_size() && ALIGN(size, align) < PAGE_SIZE) {
cachep->obj_offset += PAGE_SIZE - ALIGN(size, align);
size = PAGE_SIZE;
@@ -2425,7 +2425,7 @@ static void check_spinlock_acquired_node
#define check_spinlock_acquired_node(x, y) do { } while(0)
#endif
-static void drain_array(struct kmem_cache *cachep, struct kmem_cache_node *l3,
+static void drain_array(struct kmem_cache *cachep, struct kmem_cache_node *n,
struct array_cache *ac,
int force, int node);
@@ -2445,21 +2445,21 @@ static void do_drain(void *arg)
static void drain_cpu_caches(struct kmem_cache *cachep)
{
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
int node;
on_each_cpu(do_drain, cachep, 1);
check_irq_on();
for_each_online_node(node) {
- l3 = cachep->node[node];
- if (l3 && l3->alien)
- drain_alien_cache(cachep, l3->alien);
+ n = cachep->node[node];
+ if (n && n->alien)
+ drain_alien_cache(cachep, n->alien);
}
for_each_online_node(node) {
- l3 = cachep->node[node];
- if (l3)
- drain_array(cachep, l3, l3->shared, 1, node);
+ n = cachep->node[node];
+ if (n)
+ drain_array(cachep, n, n->shared, 1, node);
}
}
@@ -2470,19 +2470,19 @@ static void drain_cpu_caches(struct kmem
* Returns the actual number of slabs released.
*/
static int drain_freelist(struct kmem_cache *cache,
- struct kmem_cache_node *l3, int tofree)
+ struct kmem_cache_node *n, int tofree)
{
struct list_head *p;
int nr_freed;
struct slab *slabp;
nr_freed = 0;
- while (nr_freed < tofree && !list_empty(&l3->slabs_free)) {
+ while (nr_freed < tofree && !list_empty(&n->slabs_free)) {
- spin_lock_irq(&l3->list_lock);
- p = l3->slabs_free.prev;
- if (p == &l3->slabs_free) {
- spin_unlock_irq(&l3->list_lock);
+ spin_lock_irq(&n->list_lock);
+ p = n->slabs_free.prev;
+ if (p == &n->slabs_free) {
+ spin_unlock_irq(&n->list_lock);
goto out;
}
@@ -2495,8 +2495,8 @@ static int drain_freelist(struct kmem_ca
* Safe to drop the lock. The slab is no longer linked
* to the cache.
*/
- l3->free_objects -= cache->num;
- spin_unlock_irq(&l3->list_lock);
+ n->free_objects -= cache->num;
+ spin_unlock_irq(&n->list_lock);
slab_destroy(cache, slabp);
nr_freed++;
}
@@ -2508,20 +2508,20 @@ out:
static int __cache_shrink(struct kmem_cache *cachep)
{
int ret = 0, i = 0;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
drain_cpu_caches(cachep);
check_irq_on();
for_each_online_node(i) {
- l3 = cachep->node[i];
- if (!l3)
+ n = cachep->node[i];
+ if (!n)
continue;
- drain_freelist(cachep, l3, l3->free_objects);
+ drain_freelist(cachep, n, n->free_objects);
- ret += !list_empty(&l3->slabs_full) ||
- !list_empty(&l3->slabs_partial);
+ ret += !list_empty(&n->slabs_full) ||
+ !list_empty(&n->slabs_partial);
}
return (ret ? 1 : 0);
}
@@ -2550,7 +2550,7 @@ EXPORT_SYMBOL(kmem_cache_shrink);
int __kmem_cache_shutdown(struct kmem_cache *cachep)
{
int i;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
int rc = __cache_shrink(cachep);
if (rc)
@@ -2559,13 +2559,13 @@ int __kmem_cache_shutdown(struct kmem_ca
for_each_online_cpu(i)
kfree(cachep->array[i]);
- /* NUMA: free the list3 structures */
+ /* NUMA: free the node structures */
for_each_online_node(i) {
- l3 = cachep->node[i];
- if (l3) {
- kfree(l3->shared);
- free_alien_cache(l3->alien);
- kfree(l3);
+ n = cachep->node[i];
+ if (n) {
+ kfree(n->shared);
+ free_alien_cache(n->alien);
+ kfree(n);
}
}
return 0;
@@ -2747,7 +2747,7 @@ static int cache_grow(struct kmem_cache
struct slab *slabp;
size_t offset;
gfp_t local_flags;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
/*
* Be lazy and only check for valid flags here, keeping it out of the
@@ -2756,17 +2756,17 @@ static int cache_grow(struct kmem_cache
BUG_ON(flags & GFP_SLAB_BUG_MASK);
local_flags = flags & (GFP_CONSTRAINT_MASK|GFP_RECLAIM_MASK);
- /* Take the l3 list lock to change the colour_next on this node */
+ /* Take the node list lock to change the colour_next on this node */
check_irq_off();
- l3 = cachep->node[nodeid];
- spin_lock(&l3->list_lock);
+ n = cachep->node[nodeid];
+ spin_lock(&n->list_lock);
/* Get colour for the slab, and cal the next value. */
- offset = l3->colour_next;
- l3->colour_next++;
- if (l3->colour_next >= cachep->colour)
- l3->colour_next = 0;
- spin_unlock(&l3->list_lock);
+ offset = n->colour_next;
+ n->colour_next++;
+ if (n->colour_next >= cachep->colour)
+ n->colour_next = 0;
+ spin_unlock(&n->list_lock);
offset *= cachep->colour_off;
@@ -2803,13 +2803,13 @@ static int cache_grow(struct kmem_cache
if (local_flags & __GFP_WAIT)
local_irq_disable();
check_irq_off();
- spin_lock(&l3->list_lock);
+ spin_lock(&n->list_lock);
/* Make slab active. */
- list_add_tail(&slabp->list, &(l3->slabs_free));
+ list_add_tail(&slabp->list, &(n->slabs_free));
STATS_INC_GROWN(cachep);
- l3->free_objects += cachep->num;
- spin_unlock(&l3->list_lock);
+ n->free_objects += cachep->num;
+ spin_unlock(&n->list_lock);
return 1;
opps1:
kmem_freepages(cachep, objp);
@@ -2937,7 +2937,7 @@ static void *cache_alloc_refill(struct k
bool force_refill)
{
int batchcount;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
struct array_cache *ac;
int node;
@@ -2956,14 +2956,14 @@ retry:
*/
batchcount = BATCHREFILL_LIMIT;
}
- l3 = cachep->node[node];
+ n = cachep->node[node];
- BUG_ON(ac->avail > 0 || !l3);
- spin_lock(&l3->list_lock);
+ BUG_ON(ac->avail > 0 || !n);
+ spin_lock(&n->list_lock);
/* See if we can refill from the shared array */
- if (l3->shared && transfer_objects(ac, l3->shared, batchcount)) {
- l3->shared->touched = 1;
+ if (n->shared && transfer_objects(ac, n->shared, batchcount)) {
+ n->shared->touched = 1;
goto alloc_done;
}
@@ -2971,11 +2971,11 @@ retry:
struct list_head *entry;
struct slab *slabp;
/* Get slab alloc is to come from. */
- entry = l3->slabs_partial.next;
- if (entry == &l3->slabs_partial) {
- l3->free_touched = 1;
- entry = l3->slabs_free.next;
- if (entry == &l3->slabs_free)
+ entry = n->slabs_partial.next;
+ if (entry == &n->slabs_partial) {
+ n->free_touched = 1;
+ entry = n->slabs_free.next;
+ if (entry == &n->slabs_free)
goto must_grow;
}
@@ -3003,15 +3003,15 @@ retry:
/* move slabp to correct slabp list: */
list_del(&slabp->list);
if (slabp->free == BUFCTL_END)
- list_add(&slabp->list, &l3->slabs_full);
+ list_add(&slabp->list, &n->slabs_full);
else
- list_add(&slabp->list, &l3->slabs_partial);
+ list_add(&slabp->list, &n->slabs_partial);
}
must_grow:
- l3->free_objects -= ac->avail;
+ n->free_objects -= ac->avail;
alloc_done:
- spin_unlock(&l3->list_lock);
+ spin_unlock(&n->list_lock);
if (unlikely(!ac->avail)) {
int x;
@@ -3269,21 +3269,21 @@ static void *____cache_alloc_node(struct
{
struct list_head *entry;
struct slab *slabp;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
void *obj;
int x;
- l3 = cachep->node[nodeid];
- BUG_ON(!l3);
+ n = cachep->node[nodeid];
+ BUG_ON(!n);
retry:
check_irq_off();
- spin_lock(&l3->list_lock);
- entry = l3->slabs_partial.next;
- if (entry == &l3->slabs_partial) {
- l3->free_touched = 1;
- entry = l3->slabs_free.next;
- if (entry == &l3->slabs_free)
+ spin_lock(&n->list_lock);
+ entry = n->slabs_partial.next;
+ if (entry == &n->slabs_partial) {
+ n->free_touched = 1;
+ entry = n->slabs_free.next;
+ if (entry == &n->slabs_free)
goto must_grow;
}
@@ -3299,20 +3299,20 @@ retry:
obj = slab_get_obj(cachep, slabp, nodeid);
check_slabp(cachep, slabp);
- l3->free_objects--;
+ n->free_objects--;
/* move slabp to correct slabp list: */
list_del(&slabp->list);
if (slabp->free == BUFCTL_END)
- list_add(&slabp->list, &l3->slabs_full);
+ list_add(&slabp->list, &n->slabs_full);
else
- list_add(&slabp->list, &l3->slabs_partial);
+ list_add(&slabp->list, &n->slabs_partial);
- spin_unlock(&l3->list_lock);
+ spin_unlock(&n->list_lock);
goto done;
must_grow:
- spin_unlock(&l3->list_lock);
+ spin_unlock(&n->list_lock);
x = cache_grow(cachep, flags | GFP_THISNODE, nodeid, NULL);
if (x)
goto retry;
@@ -3460,7 +3460,7 @@ static void free_block(struct kmem_cache
int node)
{
int i;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
for (i = 0; i < nr_objects; i++) {
void *objp;
@@ -3470,19 +3470,19 @@ static void free_block(struct kmem_cache
objp = objpp[i];
slabp = virt_to_slab(objp);
- l3 = cachep->node[node];
+ n = cachep->node[node];
list_del(&slabp->list);
check_spinlock_acquired_node(cachep, node);
check_slabp(cachep, slabp);
slab_put_obj(cachep, slabp, objp, node);
STATS_DEC_ACTIVE(cachep);
- l3->free_objects++;
+ n->free_objects++;
check_slabp(cachep, slabp);
/* fixup slab chains */
if (slabp->inuse == 0) {
- if (l3->free_objects > l3->free_limit) {
- l3->free_objects -= cachep->num;
+ if (n->free_objects > n->free_limit) {
+ n->free_objects -= cachep->num;
/* No need to drop any previously held
* lock here, even if we have a off-slab slab
* descriptor it is guaranteed to come from
@@ -3491,14 +3491,14 @@ static void free_block(struct kmem_cache
*/
slab_destroy(cachep, slabp);
} else {
- list_add(&slabp->list, &l3->slabs_free);
+ list_add(&slabp->list, &n->slabs_free);
}
} else {
/* Unconditionally move a slab to the end of the
* partial list on free - maximum time for the
* other objects to be freed, too.
*/
- list_add_tail(&slabp->list, &l3->slabs_partial);
+ list_add_tail(&slabp->list, &n->slabs_partial);
}
}
}
@@ -3506,7 +3506,7 @@ static void free_block(struct kmem_cache
static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac)
{
int batchcount;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
int node = numa_mem_id();
batchcount = ac->batchcount;
@@ -3514,10 +3514,10 @@ static void cache_flusharray(struct kmem
BUG_ON(!batchcount || batchcount > ac->avail);
#endif
check_irq_off();
- l3 = cachep->node[node];
- spin_lock(&l3->list_lock);
- if (l3->shared) {
- struct array_cache *shared_array = l3->shared;
+ n = cachep->node[node];
+ spin_lock(&n->list_lock);
+ if (n->shared) {
+ struct array_cache *shared_array = n->shared;
int max = shared_array->limit - shared_array->avail;
if (max) {
if (batchcount > max)
@@ -3536,8 +3536,8 @@ free_done:
int i = 0;
struct list_head *p;
- p = l3->slabs_free.next;
- while (p != &(l3->slabs_free)) {
+ p = n->slabs_free.next;
+ while (p != &(n->slabs_free)) {
struct slab *slabp;
slabp = list_entry(p, struct slab, list);
@@ -3549,7 +3549,7 @@ free_done:
STATS_SET_FREEABLE(cachep, i);
}
#endif
- spin_unlock(&l3->list_lock);
+ spin_unlock(&n->list_lock);
ac->avail -= batchcount;
memmove(ac->entry, &(ac->entry[batchcount]), sizeof(void *)*ac->avail);
}
@@ -3790,12 +3790,12 @@ void kfree(const void *objp)
EXPORT_SYMBOL(kfree);
/*
- * This initializes kmem_list3 or resizes various caches for all nodes.
+ * This initializes kmem_cache_node or resizes various caches for all nodes.
*/
static int alloc_kmemlist(struct kmem_cache *cachep, gfp_t gfp)
{
int node;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
struct array_cache *new_shared;
struct array_cache **new_alien = NULL;
@@ -3818,43 +3818,43 @@ static int alloc_kmemlist(struct kmem_ca
}
}
- l3 = cachep->node[node];
- if (l3) {
- struct array_cache *shared = l3->shared;
+ n = cachep->node[node];
+ if (n) {
+ struct array_cache *shared = n->shared;
- spin_lock_irq(&l3->list_lock);
+ spin_lock_irq(&n->list_lock);
if (shared)
free_block(cachep, shared->entry,
shared->avail, node);
- l3->shared = new_shared;
- if (!l3->alien) {
- l3->alien = new_alien;
+ n->shared = new_shared;
+ if (!n->alien) {
+ n->alien = new_alien;
new_alien = NULL;
}
- l3->free_limit = (1 + nr_cpus_node(node)) *
+ n->free_limit = (1 + nr_cpus_node(node)) *
cachep->batchcount + cachep->num;
- spin_unlock_irq(&l3->list_lock);
+ spin_unlock_irq(&n->list_lock);
kfree(shared);
free_alien_cache(new_alien);
continue;
}
- l3 = kmalloc_node(sizeof(struct kmem_cache_node), gfp, node);
- if (!l3) {
+ n = kmalloc_node(sizeof(struct kmem_cache_node), gfp, node);
+ if (!n) {
free_alien_cache(new_alien);
kfree(new_shared);
goto fail;
}
- kmem_list3_init(l3);
- l3->next_reap = jiffies + REAPTIMEOUT_LIST3 +
+ kmem_cache_node_init(n);
+ n->next_reap = jiffies + REAPTIMEOUT_LIST3 +
((unsigned long)cachep) % REAPTIMEOUT_LIST3;
- l3->shared = new_shared;
- l3->alien = new_alien;
- l3->free_limit = (1 + nr_cpus_node(node)) *
+ n->shared = new_shared;
+ n->alien = new_alien;
+ n->free_limit = (1 + nr_cpus_node(node)) *
cachep->batchcount + cachep->num;
- cachep->node[node] = l3;
+ cachep->node[node] = n;
}
return 0;
@@ -3864,11 +3864,11 @@ fail:
node--;
while (node >= 0) {
if (cachep->node[node]) {
- l3 = cachep->node[node];
+ n = cachep->node[node];
- kfree(l3->shared);
- free_alien_cache(l3->alien);
- kfree(l3);
+ kfree(n->shared);
+ free_alien_cache(n->alien);
+ kfree(n);
cachep->node[node] = NULL;
}
node--;
@@ -3993,11 +3993,11 @@ static int enable_cpucache(struct kmem_c
}
/*
- * Drain an array if it contains any elements taking the l3 lock only if
- * necessary. Note that the l3 listlock also protects the array_cache
+ * Drain an array if it contains any elements taking the node lock only if
+ * necessary. Note that the node listlock also protects the array_cache
* if drain_array() is used on the shared array.
*/
-static void drain_array(struct kmem_cache *cachep, struct kmem_cache_node *l3,
+static void drain_array(struct kmem_cache *cachep, struct kmem_cache_node *n,
struct array_cache *ac, int force, int node)
{
int tofree;
@@ -4007,7 +4007,7 @@ static void drain_array(struct kmem_cach
if (ac->touched && !force) {
ac->touched = 0;
} else {
- spin_lock_irq(&l3->list_lock);
+ spin_lock_irq(&n->list_lock);
if (ac->avail) {
tofree = force ? ac->avail : (ac->limit + 4) / 5;
if (tofree > ac->avail)
@@ -4017,7 +4017,7 @@ static void drain_array(struct kmem_cach
memmove(ac->entry, &(ac->entry[tofree]),
sizeof(void *) * ac->avail);
}
- spin_unlock_irq(&l3->list_lock);
+ spin_unlock_irq(&n->list_lock);
}
}
@@ -4036,7 +4036,7 @@ static void drain_array(struct kmem_cach
static void cache_reap(struct work_struct *w)
{
struct kmem_cache *searchp;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
int node = numa_mem_id();
struct delayed_work *work = to_delayed_work(w);
@@ -4048,33 +4048,33 @@ static void cache_reap(struct work_struc
check_irq_on();
/*
- * We only take the l3 lock if absolutely necessary and we
+ * We only take the node lock if absolutely necessary and we
* have established with reasonable certainty that
* we can do some work if the lock was obtained.
*/
- l3 = searchp->node[node];
+ n = searchp->node[node];
- reap_alien(searchp, l3);
+ reap_alien(searchp, n);
- drain_array(searchp, l3, cpu_cache_get(searchp), 0, node);
+ drain_array(searchp, n, cpu_cache_get(searchp), 0, node);
/*
* These are racy checks but it does not matter
* if we skip one check or scan twice.
*/
- if (time_after(l3->next_reap, jiffies))
+ if (time_after(n->next_reap, jiffies))
goto next;
- l3->next_reap = jiffies + REAPTIMEOUT_LIST3;
+ n->next_reap = jiffies + REAPTIMEOUT_LIST3;
- drain_array(searchp, l3, l3->shared, 0, node);
+ drain_array(searchp, n, n->shared, 0, node);
- if (l3->free_touched)
- l3->free_touched = 0;
+ if (n->free_touched)
+ n->free_touched = 0;
else {
int freed;
- freed = drain_freelist(searchp, l3, (l3->free_limit +
+ freed = drain_freelist(searchp, n, (n->free_limit +
5 * searchp->num - 1) / (5 * searchp->num));
STATS_ADD_REAPED(searchp, freed);
}
@@ -4100,25 +4100,25 @@ void get_slabinfo(struct kmem_cache *cac
const char *name;
char *error = NULL;
int node;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
active_objs = 0;
num_slabs = 0;
for_each_online_node(node) {
- l3 = cachep->node[node];
- if (!l3)
+ n = cachep->node[node];
+ if (!n)
continue;
check_irq_on();
- spin_lock_irq(&l3->list_lock);
+ spin_lock_irq(&n->list_lock);
- list_for_each_entry(slabp, &l3->slabs_full, list) {
+ list_for_each_entry(slabp, &n->slabs_full, list) {
if (slabp->inuse != cachep->num && !error)
error = "slabs_full accounting error";
active_objs += cachep->num;
active_slabs++;
}
- list_for_each_entry(slabp, &l3->slabs_partial, list) {
+ list_for_each_entry(slabp, &n->slabs_partial, list) {
if (slabp->inuse == cachep->num && !error)
error = "slabs_partial inuse accounting error";
if (!slabp->inuse && !error)
@@ -4126,16 +4126,16 @@ void get_slabinfo(struct kmem_cache *cac
active_objs += slabp->inuse;
active_slabs++;
}
- list_for_each_entry(slabp, &l3->slabs_free, list) {
+ list_for_each_entry(slabp, &n->slabs_free, list) {
if (slabp->inuse && !error)
error = "slabs_free/inuse accounting error";
num_slabs++;
}
- free_objects += l3->free_objects;
- if (l3->shared)
- shared_avail += l3->shared->avail;
+ free_objects += n->free_objects;
+ if (n->shared)
+ shared_avail += n->shared->avail;
- spin_unlock_irq(&l3->list_lock);
+ spin_unlock_irq(&n->list_lock);
}
num_slabs += active_slabs;
num_objs = num_slabs * cachep->num;
@@ -4161,7 +4161,7 @@ void get_slabinfo(struct kmem_cache *cac
void slabinfo_show_stats(struct seq_file *m, struct kmem_cache *cachep)
{
#if STATS
- { /* list3 stats */
+ { /* node stats */
unsigned long high = cachep->high_mark;
unsigned long allocs = cachep->num_allocations;
unsigned long grown = cachep->grown;
@@ -4314,7 +4314,7 @@ static int leaks_show(struct seq_file *m
{
struct kmem_cache *cachep = list_entry(p, struct kmem_cache, list);
struct slab *slabp;
- struct kmem_cache_node *l3;
+ struct kmem_cache_node *n;
const char *name;
unsigned long *n = m->private;
int node;
@@ -4330,18 +4330,18 @@ static int leaks_show(struct seq_file *m
n[1] = 0;
for_each_online_node(node) {
- l3 = cachep->node[node];
- if (!l3)
+ n = cachep->node[node];
+ if (!n)
continue;
check_irq_on();
- spin_lock_irq(&l3->list_lock);
+ spin_lock_irq(&n->list_lock);
- list_for_each_entry(slabp, &l3->slabs_full, list)
+ list_for_each_entry(slabp, &n->slabs_full, list)
handle_slab(n, cachep, slabp);
- list_for_each_entry(slabp, &l3->slabs_partial, list)
+ list_for_each_entry(slabp, &n->slabs_partial, list)
handle_slab(n, cachep, slabp);
- spin_unlock_irq(&l3->list_lock);
+ spin_unlock_irq(&n->list_lock);
}
name = cachep->name;
if (n[0] == n[1]) {
Index: linux/mm/slab.h
===================================================================
--- linux.orig/mm/slab.h 2012-11-01 16:12:48.217686485 -0500
+++ linux/mm/slab.h 2012-11-01 16:13:47.466518289 -0500
@@ -16,7 +16,7 @@ enum slab_state {
DOWN, /* No slab functionality yet */
PARTIAL, /* SLUB: kmem_cache_node available */
PARTIAL_ARRAYCACHE, /* SLAB: kmalloc size for arraycache available */
- PARTIAL_L3, /* SLAB: kmalloc size for l3 struct available */
+ PARTIAL_NODE, /* SLAB: kmalloc size for node struct available */
UP, /* Slab caches usable but not all extras yet */
FULL /* Everything is working */
};
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* CK5 [10/18] slab: Common name for the per node structures
[not found] <20121101214538.971500204@linux.com>
` (16 preceding siblings ...)
2012-11-01 21:48 ` CK5 [18/18] Common definition for kmem_cache_node Christoph Lameter
@ 2012-11-01 21:55 ` Christoph Lameter
17 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-01 21:55 UTC (permalink / raw)
To: Pekka Enberg
Cc: Joonsoo Kim, Glauber Costa, linux-mm, David Rientjes, elezegarcia
Rename the structure used for the per node structures in slab
to have a name that expresses that fact.
Acked-by: Glauber Costa <glommer@parallels.com>
Signed-off-by: Christoph Lameter <cl@linux.com>
Index: linux/include/linux/slab_def.h
===================================================================
--- linux.orig/include/linux/slab_def.h 2012-11-01 10:10:30.522138714 -0500
+++ linux/include/linux/slab_def.h 2012-11-01 10:10:33.446187162 -0500
@@ -88,7 +88,7 @@ struct kmem_cache {
* We still use [NR_CPUS] and not [1] or [0] because cache_cache
* is statically defined, so we reserve the max number of cpus.
*/
- struct kmem_list3 **nodelists;
+ struct kmem_cache_node **nodelists;
struct array_cache *array[NR_CPUS + MAX_NUMNODES];
/*
* Do not add fields after array[]
Index: linux/mm/slab.c
===================================================================
--- linux.orig/mm/slab.c 2012-11-01 10:10:30.522138714 -0500
+++ linux/mm/slab.c 2012-11-01 10:10:33.450187230 -0500
@@ -287,7 +287,7 @@ struct arraycache_init {
/*
* The slab lists for all objects.
*/
-struct kmem_list3 {
+struct kmem_cache_node {
struct list_head slabs_partial; /* partial list first, better asm code */
struct list_head slabs_full;
struct list_head slabs_free;
@@ -305,13 +305,13 @@ struct kmem_list3 {
* Need this for bootstrapping a per node allocator.
*/
#define NUM_INIT_LISTS (3 * MAX_NUMNODES)
-static struct kmem_list3 __initdata initkmem_list3[NUM_INIT_LISTS];
+static struct kmem_cache_node __initdata initkmem_list3[NUM_INIT_LISTS];
#define CACHE_CACHE 0
#define SIZE_AC MAX_NUMNODES
#define SIZE_L3 (2 * MAX_NUMNODES)
static int drain_freelist(struct kmem_cache *cache,
- struct kmem_list3 *l3, int tofree);
+ struct kmem_cache_node *l3, int tofree);
static void free_block(struct kmem_cache *cachep, void **objpp, int len,
int node);
static int enable_cpucache(struct kmem_cache *cachep, gfp_t gfp);
@@ -328,9 +328,9 @@ EXPORT_SYMBOL(kmalloc_dma_caches);
static int slab_early_init = 1;
#define INDEX_AC kmalloc_index(sizeof(struct arraycache_init))
-#define INDEX_L3 kmalloc_index(sizeof(struct kmem_list3))
+#define INDEX_L3 kmalloc_index(sizeof(struct kmem_cache_node))
-static void kmem_list3_init(struct kmem_list3 *parent)
+static void kmem_list3_init(struct kmem_cache_node *parent)
{
INIT_LIST_HEAD(&parent->slabs_full);
INIT_LIST_HEAD(&parent->slabs_partial);
@@ -545,7 +545,7 @@ static void slab_set_lock_classes(struct
int q)
{
struct array_cache **alc;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
int r;
l3 = cachep->nodelists[q];
@@ -590,7 +590,7 @@ static void init_node_lock_keys(int q)
return;
for (i = 1; i < PAGE_SHIFT + MAX_ORDER; i++) {
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
struct kmem_cache *cache = kmalloc_caches[i];
if (!cache)
@@ -872,7 +872,7 @@ static inline bool is_slab_pfmemalloc(st
static void recheck_pfmemalloc_active(struct kmem_cache *cachep,
struct array_cache *ac)
{
- struct kmem_list3 *l3 = cachep->nodelists[numa_mem_id()];
+ struct kmem_cache_node *l3 = cachep->nodelists[numa_mem_id()];
struct slab *slabp;
unsigned long flags;
@@ -905,7 +905,7 @@ static void *__ac_get_obj(struct kmem_ca
/* Ensure the caller is allowed to use objects from PFMEMALLOC slab */
if (unlikely(is_obj_pfmemalloc(objp))) {
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
if (gfp_pfmemalloc_allowed(flags)) {
clear_obj_pfmemalloc(&objp);
@@ -1077,7 +1077,7 @@ static void free_alien_cache(struct arra
static void __drain_alien_cache(struct kmem_cache *cachep,
struct array_cache *ac, int node)
{
- struct kmem_list3 *rl3 = cachep->nodelists[node];
+ struct kmem_cache_node *rl3 = cachep->nodelists[node];
if (ac->avail) {
spin_lock(&rl3->list_lock);
@@ -1098,7 +1098,7 @@ static void __drain_alien_cache(struct k
/*
* Called from cache_reap() to regularly drain alien caches round robin.
*/
-static void reap_alien(struct kmem_cache *cachep, struct kmem_list3 *l3)
+static void reap_alien(struct kmem_cache *cachep, struct kmem_cache_node *l3)
{
int node = __this_cpu_read(slab_reap_node);
@@ -1133,7 +1133,7 @@ static inline int cache_free_alien(struc
{
struct slab *slabp = virt_to_slab(objp);
int nodeid = slabp->nodeid;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
struct array_cache *alien = NULL;
int node;
@@ -1178,8 +1178,8 @@ static inline int cache_free_alien(struc
static int init_cache_nodelists_node(int node)
{
struct kmem_cache *cachep;
- struct kmem_list3 *l3;
- const int memsize = sizeof(struct kmem_list3);
+ struct kmem_cache_node *l3;
+ const int memsize = sizeof(struct kmem_cache_node);
list_for_each_entry(cachep, &slab_caches, list) {
/*
@@ -1215,7 +1215,7 @@ static int init_cache_nodelists_node(int
static void __cpuinit cpuup_canceled(long cpu)
{
struct kmem_cache *cachep;
- struct kmem_list3 *l3 = NULL;
+ struct kmem_cache_node *l3 = NULL;
int node = cpu_to_mem(cpu);
const struct cpumask *mask = cpumask_of_node(node);
@@ -1280,7 +1280,7 @@ free_array_cache:
static int __cpuinit cpuup_prepare(long cpu)
{
struct kmem_cache *cachep;
- struct kmem_list3 *l3 = NULL;
+ struct kmem_cache_node *l3 = NULL;
int node = cpu_to_mem(cpu);
int err;
@@ -1431,7 +1431,7 @@ static int __meminit drain_cache_nodelis
int ret = 0;
list_for_each_entry(cachep, &slab_caches, list) {
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
l3 = cachep->nodelists[node];
if (!l3)
@@ -1484,15 +1484,15 @@ out:
/*
* swap the static kmem_list3 with kmalloced memory
*/
-static void __init init_list(struct kmem_cache *cachep, struct kmem_list3 *list,
+static void __init init_list(struct kmem_cache *cachep, struct kmem_cache_node *list,
int nodeid)
{
- struct kmem_list3 *ptr;
+ struct kmem_cache_node *ptr;
- ptr = kmalloc_node(sizeof(struct kmem_list3), GFP_NOWAIT, nodeid);
+ ptr = kmalloc_node(sizeof(struct kmem_cache_node), GFP_NOWAIT, nodeid);
BUG_ON(!ptr);
- memcpy(ptr, list, sizeof(struct kmem_list3));
+ memcpy(ptr, list, sizeof(struct kmem_cache_node));
/*
* Do not assume that spinlocks can be initialized via memcpy:
*/
@@ -1524,7 +1524,7 @@ static void __init set_up_list3s(struct
*/
static void setup_nodelists_pointer(struct kmem_cache *s)
{
- s->nodelists = (struct kmem_list3 **)&s->array[nr_cpu_ids];
+ s->nodelists = (struct kmem_cache_node **)&s->array[nr_cpu_ids];
}
/*
@@ -1581,7 +1581,7 @@ void __init kmem_cache_init(void)
*/
create_boot_cache(kmem_cache, "kmem_cache",
offsetof(struct kmem_cache, array[nr_cpu_ids]) +
- nr_node_ids * sizeof(struct kmem_list3 *),
+ nr_node_ids * sizeof(struct kmem_cache_node *),
SLAB_HWCACHE_ALIGN);
/* 2+3) create the kmalloc caches */
@@ -1754,7 +1754,7 @@ __initcall(cpucache_init);
static noinline void
slab_out_of_memory(struct kmem_cache *cachep, gfp_t gfpflags, int nodeid)
{
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
struct slab *slabp;
unsigned long flags;
int node;
@@ -2243,7 +2243,7 @@ static int __init_refok setup_cpu_cache(
int node;
for_each_online_node(node) {
cachep->nodelists[node] =
- kmalloc_node(sizeof(struct kmem_list3),
+ kmalloc_node(sizeof(struct kmem_cache_node),
gfp, node);
BUG_ON(!cachep->nodelists[node]);
kmem_list3_init(cachep->nodelists[node]);
@@ -2513,7 +2513,7 @@ static void check_spinlock_acquired_node
#define check_spinlock_acquired_node(x, y) do { } while(0)
#endif
-static void drain_array(struct kmem_cache *cachep, struct kmem_list3 *l3,
+static void drain_array(struct kmem_cache *cachep, struct kmem_cache_node *l3,
struct array_cache *ac,
int force, int node);
@@ -2533,7 +2533,7 @@ static void do_drain(void *arg)
static void drain_cpu_caches(struct kmem_cache *cachep)
{
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
int node;
on_each_cpu(do_drain, cachep, 1);
@@ -2558,7 +2558,7 @@ static void drain_cpu_caches(struct kmem
* Returns the actual number of slabs released.
*/
static int drain_freelist(struct kmem_cache *cache,
- struct kmem_list3 *l3, int tofree)
+ struct kmem_cache_node *l3, int tofree)
{
struct list_head *p;
int nr_freed;
@@ -2596,7 +2596,7 @@ out:
static int __cache_shrink(struct kmem_cache *cachep)
{
int ret = 0, i = 0;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
drain_cpu_caches(cachep);
@@ -2638,7 +2638,7 @@ EXPORT_SYMBOL(kmem_cache_shrink);
int __kmem_cache_shutdown(struct kmem_cache *cachep)
{
int i;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
int rc = __cache_shrink(cachep);
if (rc)
@@ -2835,7 +2835,7 @@ static int cache_grow(struct kmem_cache
struct slab *slabp;
size_t offset;
gfp_t local_flags;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
/*
* Be lazy and only check for valid flags here, keeping it out of the
@@ -3025,7 +3025,7 @@ static void *cache_alloc_refill(struct k
bool force_refill)
{
int batchcount;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
struct array_cache *ac;
int node;
@@ -3357,7 +3357,7 @@ static void *____cache_alloc_node(struct
{
struct list_head *entry;
struct slab *slabp;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
void *obj;
int x;
@@ -3548,7 +3548,7 @@ static void free_block(struct kmem_cache
int node)
{
int i;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
for (i = 0; i < nr_objects; i++) {
void *objp;
@@ -3594,7 +3594,7 @@ static void free_block(struct kmem_cache
static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac)
{
int batchcount;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
int node = numa_mem_id();
batchcount = ac->batchcount;
@@ -3883,7 +3883,7 @@ EXPORT_SYMBOL(kfree);
static int alloc_kmemlist(struct kmem_cache *cachep, gfp_t gfp)
{
int node;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
struct array_cache *new_shared;
struct array_cache **new_alien = NULL;
@@ -3928,7 +3928,7 @@ static int alloc_kmemlist(struct kmem_ca
free_alien_cache(new_alien);
continue;
}
- l3 = kmalloc_node(sizeof(struct kmem_list3), gfp, node);
+ l3 = kmalloc_node(sizeof(struct kmem_cache_node), gfp, node);
if (!l3) {
free_alien_cache(new_alien);
kfree(new_shared);
@@ -4085,7 +4085,7 @@ static int enable_cpucache(struct kmem_c
* necessary. Note that the l3 listlock also protects the array_cache
* if drain_array() is used on the shared array.
*/
-static void drain_array(struct kmem_cache *cachep, struct kmem_list3 *l3,
+static void drain_array(struct kmem_cache *cachep, struct kmem_cache_node *l3,
struct array_cache *ac, int force, int node)
{
int tofree;
@@ -4124,7 +4124,7 @@ static void drain_array(struct kmem_cach
static void cache_reap(struct work_struct *w)
{
struct kmem_cache *searchp;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
int node = numa_mem_id();
struct delayed_work *work = to_delayed_work(w);
@@ -4188,7 +4188,7 @@ void get_slabinfo(struct kmem_cache *cac
const char *name;
char *error = NULL;
int node;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
active_objs = 0;
num_slabs = 0;
@@ -4402,7 +4402,7 @@ static int leaks_show(struct seq_file *m
{
struct kmem_cache *cachep = list_entry(p, struct kmem_cache, list);
struct slab *slabp;
- struct kmem_list3 *l3;
+ struct kmem_cache_node *l3;
const char *name;
unsigned long *n = m->private;
int node;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: CK5 [01/18] Use correct cpu_slab on dead cpu
2012-11-01 21:46 ` CK5 [01/18] Use correct cpu_slab on dead cpu Christoph Lameter
@ 2012-11-02 20:11 ` David Rientjes
0 siblings, 0 replies; 24+ messages in thread
From: David Rientjes @ 2012-11-02 20:11 UTC (permalink / raw)
To: Christoph Lameter
Cc: Pekka Enberg, Joonsoo Kim, Glauber Costa, linux-mm, elezegarcia
On Thu, 1 Nov 2012, Christoph Lameter wrote:
> Pass a kmem_cache_cpu pointer into unfreeze partials so that a different
> kmem_cache_cpu structure than the local one can be specified.
>
> Signed-off-by: Christoph Lameter <cl@linux.com>
Acked-by: David Rientjes <rientjes@google.com>
>
> Index: linux/mm/slub.c
> ===================================================================
> --- linux.orig/mm/slub.c 2012-11-01 10:10:05.073716747 -0500
> +++ linux/mm/slub.c 2012-11-01 10:10:06.173734998 -0500
> @@ -1871,10 +1871,10 @@ redo:
> *
> * This function must be called with interrupt disabled.
Should probably say interrupts disabled for c's cpu.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: CK5 [02/18] slab: Simplify bootstrap
2012-11-01 21:46 ` CK5 [02/18] slab: Simplify bootstrap Christoph Lameter
@ 2012-11-02 20:22 ` David Rientjes
2012-11-05 15:11 ` Christoph Lameter
0 siblings, 1 reply; 24+ messages in thread
From: David Rientjes @ 2012-11-02 20:22 UTC (permalink / raw)
To: Christoph Lameter
Cc: Pekka Enberg, Joonsoo Kim, Glauber Costa, linux-mm, elezegarcia
On Thu, 1 Nov 2012, Christoph Lameter wrote:
> The nodelists field in kmem_cache is pointing to the first unused
> object in the array field when bootstrap is complete.
>
> A problem with the current approach is that the statically sized
> kmem_cache structure use on boot can only contain NR_CPUS entries.
> If the number of nodes plus the number of cpus is greater then we
> would overwrite memory following the kmem_cache_boot definition.
>
> Increase the size of the array field to ensure that also the node
> pointers fit into the array field.
>
> Once we do that we no longer need the kmem_cache_nodelists
> array and we can then also use that structure elsewhere.
>
> V1->V2:
> - No need to zap kmem_cache->nodelists since it is allocated
> with kmem_cache_zalloc() [glommer]
>
> Acked-by: Glauber Costa <glommer@parallels.com>
> Signed-off-by: Christoph Lameter <cl@linux.com>
> ---
> include/linux/slab_def.h | 2 +-
> mm/slab.c | 18 +++++++++++++-----
> 2 files changed, 14 insertions(+), 6 deletions(-)
>
> Index: linux/include/linux/slab_def.h
> ===================================================================
> --- linux.orig/include/linux/slab_def.h 2012-11-01 10:09:47.073417947 -0500
> +++ linux/include/linux/slab_def.h 2012-11-01 10:09:55.357555494 -0500
> @@ -91,7 +91,7 @@ struct kmem_cache {
> * is statically defined, so we reserve the max number of cpus.
> */
> struct kmem_list3 **nodelists;
> - struct array_cache *array[NR_CPUS];
> + struct array_cache *array[NR_CPUS + MAX_NUMNODES];
Needs to update the comment which specifies this is only sized to NR_CPUS.
> /*
> * Do not add fields after array[]
> */
> Index: linux/mm/slab.c
> ===================================================================
> --- linux.orig/mm/slab.c 2012-11-01 10:09:47.073417947 -0500
> +++ linux/mm/slab.c 2012-11-01 10:09:55.361555562 -0500
> @@ -553,9 +553,7 @@ static struct arraycache_init initarray_
> { {0, BOOT_CPUCACHE_ENTRIES, 1, 0} };
>
> /* internal cache of cache description objs */
> -static struct kmem_list3 *kmem_cache_nodelists[MAX_NUMNODES];
> static struct kmem_cache kmem_cache_boot = {
> - .nodelists = kmem_cache_nodelists,
> .batchcount = 1,
> .limit = BOOT_CPUCACHE_ENTRIES,
> .shared = 1,
> @@ -1560,6 +1558,15 @@ static void __init set_up_list3s(struct
> }
>
> /*
> + * The memory after the last cpu cache pointer is used for the
> + * the nodelists pointer.
> + */
> +static void setup_nodelists_pointer(struct kmem_cache *s)
cachep
> +{
> + s->nodelists = (struct kmem_list3 **)&s->array[nr_cpu_ids];
> +}
> +
> +/*
> * Initialisation. Called after the page allocator have been initialised and
> * before smp_init().
> */
> @@ -1573,15 +1580,14 @@ void __init kmem_cache_init(void)
> int node;
>
> kmem_cache = &kmem_cache_boot;
> + setup_nodelists_pointer(kmem_cache);
>
> if (num_possible_nodes() == 1)
> use_alien_caches = 0;
>
> - for (i = 0; i < NUM_INIT_LISTS; i++) {
> + for (i = 0; i < NUM_INIT_LISTS; i++)
> kmem_list3_init(&initkmem_list3[i]);
> - if (i < MAX_NUMNODES)
> - kmem_cache->nodelists[i] = NULL;
> - }
> +
> set_up_list3s(kmem_cache, CACHE_CACHE);
>
> /*
> @@ -1619,7 +1625,6 @@ void __init kmem_cache_init(void)
> list_add(&kmem_cache->list, &slab_caches);
> kmem_cache->colour_off = cache_line_size();
> kmem_cache->array[smp_processor_id()] = &initarray_cache.cache;
> - kmem_cache->nodelists[node] = &initkmem_list3[CACHE_CACHE + node];
>
> /*
> * struct kmem_cache size depends on nr_node_ids & nr_cpu_ids
> @@ -2425,7 +2430,7 @@ __kmem_cache_create (struct kmem_cache *
> else
> gfp = GFP_NOWAIT;
>
> - cachep->nodelists = (struct kmem_list3 **)&cachep->array[nr_cpu_ids];
> + setup_nodelists_pointer(cachep);
> #if DEBUG
>
> /*
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: CK5 [03/18] create common functions for boot slab creation
2012-11-01 21:47 ` CK5 [03/18] create common functions for boot slab creation Christoph Lameter
@ 2012-11-02 20:34 ` David Rientjes
2012-11-05 15:15 ` Christoph Lameter
0 siblings, 1 reply; 24+ messages in thread
From: David Rientjes @ 2012-11-02 20:34 UTC (permalink / raw)
To: Christoph Lameter
Cc: Pekka Enberg, Joonsoo Kim, Glauber Costa, linux-mm, elezegarcia
On Thu, 1 Nov 2012, Christoph Lameter wrote:
> Use a special function to create kmalloc caches and use that function in
> SLAB and SLUB.
>
> V1->V2:
> Do check for slasb state in slub's __kmem_cache_create to avoid
> unlocking a lock that was not taken
> V2->V3:
> Remove slab_state check from sysfs_slab_add(). [Joonsoo]
>
> V3->V4:
> - Use %zd instead of %td for size info.
> - Do not add slab caches to the list of slab caches during early
> boot.
>
> Acked-by: Joonsoo Kim <js1304@gmail.com>
> Reviewed-by: Glauber Costa <glommer@parallels.com>
> Signed-off-by: Christoph Lameter <cl@linux.com>
Acked-by: David Rientjes <rientjes@google.com>
Eek, the calls to __kmem_cache_create() in the boot path as it sits in
slab/next right now are ignoring SLAB_PANIC.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: CK5 [02/18] slab: Simplify bootstrap
2012-11-02 20:22 ` David Rientjes
@ 2012-11-05 15:11 ` Christoph Lameter
0 siblings, 0 replies; 24+ messages in thread
From: Christoph Lameter @ 2012-11-05 15:11 UTC (permalink / raw)
To: David Rientjes
Cc: Pekka Enberg, Joonsoo Kim, Glauber Costa, linux-mm, elezegarcia
On Fri, 2 Nov 2012, David Rientjes wrote:
> Needs to update the comment which specifies this is only sized to NR_CPUS.
Ok.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: CK5 [03/18] create common functions for boot slab creation
2012-11-02 20:34 ` David Rientjes
@ 2012-11-05 15:15 ` Christoph Lameter
2012-11-05 21:52 ` David Rientjes
0 siblings, 1 reply; 24+ messages in thread
From: Christoph Lameter @ 2012-11-05 15:15 UTC (permalink / raw)
To: David Rientjes
Cc: Pekka Enberg, Joonsoo Kim, Glauber Costa, linux-mm, elezegarcia
On Fri, 2 Nov 2012, David Rientjes wrote:
> Eek, the calls to __kmem_cache_create() in the boot path as it sits in
> slab/next right now are ignoring SLAB_PANIC.
Any failure to create a slab cache during early boot is fatal and we panic
unconditionally. Like before as far as I can tell but without the use of
SLAB_PANIC.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
* Re: CK5 [03/18] create common functions for boot slab creation
2012-11-05 15:15 ` Christoph Lameter
@ 2012-11-05 21:52 ` David Rientjes
0 siblings, 0 replies; 24+ messages in thread
From: David Rientjes @ 2012-11-05 21:52 UTC (permalink / raw)
To: Christoph Lameter
Cc: Pekka Enberg, Joonsoo Kim, Glauber Costa, linux-mm, elezegarcia
On Mon, 5 Nov 2012, Christoph Lameter wrote:
> > Eek, the calls to __kmem_cache_create() in the boot path as it sits in
> > slab/next right now are ignoring SLAB_PANIC.
>
> Any failure to create a slab cache during early boot is fatal and we panic
> unconditionally. Like before as far as I can tell but without the use of
> SLAB_PANIC.
>
With your patch, yeah, but right now mm/slab.c calls directly into
__kmem_cache_create() with SLAB_PANIC which never gets respected during
bootstrap since it is handled in kmem_cache_create() in slab/next. So
this patch could actually be marketed as a bugfix :)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
^ permalink raw reply [flat|nested] 24+ messages in thread
end of thread, other threads:[~2012-11-05 21:52 UTC | newest]
Thread overview: 24+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
[not found] <20121101214538.971500204@linux.com>
2012-11-01 21:46 ` CK5 [02/18] slab: Simplify bootstrap Christoph Lameter
2012-11-02 20:22 ` David Rientjes
2012-11-05 15:11 ` Christoph Lameter
2012-11-01 21:46 ` CK5 [01/18] Use correct cpu_slab on dead cpu Christoph Lameter
2012-11-02 20:11 ` David Rientjes
2012-11-01 21:47 ` CK5 [05/18] slab: Use the new create_boot_cache function to simplify bootstrap Christoph Lameter
2012-11-01 21:47 ` CK5 [04/18] slub: Use a statically allocated kmem_cache boot structure for bootstrap Christoph Lameter
2012-11-01 21:47 ` CK5 [03/18] create common functions for boot slab creation Christoph Lameter
2012-11-02 20:34 ` David Rientjes
2012-11-05 15:15 ` Christoph Lameter
2012-11-05 21:52 ` David Rientjes
2012-11-01 21:47 ` CK5 [12/18] Common constants for kmalloc boundaries Christoph Lameter
2012-11-01 21:47 ` CK5 [06/18] Common alignment code Christoph Lameter
2012-11-01 21:47 ` CK5 [07/18] Move kmalloc related function defs Christoph Lameter
2012-11-01 21:48 ` CK5 [15/18] stat: Use size_t for sizes instead of unsigned Christoph Lameter
2012-11-01 21:48 ` CK5 [11/18] slab: rename nodelists to node Christoph Lameter
2012-11-01 21:48 ` CK5 [13/18] Common definition for the array of kmalloc caches Christoph Lameter
2012-11-01 21:48 ` CK5 [14/18] Common function to create the kmalloc array Christoph Lameter
2012-11-01 21:48 ` CK5 [16/18] Common Kmalloc cache determination Christoph Lameter
2012-11-01 21:48 ` CK5 [08/18] Common kmalloc slab index determination Christoph Lameter
2012-11-01 21:48 ` CK5 [17/18] slab: Rename list3/l3 to node Christoph Lameter
2012-11-01 21:48 ` CK5 [09/18] slab: Use common kmalloc_index/kmalloc_size functions Christoph Lameter
2012-11-01 21:48 ` CK5 [18/18] Common definition for kmem_cache_node Christoph Lameter
2012-11-01 21:55 ` CK5 [10/18] slab: Common name for the per node structures Christoph Lameter
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.