From: Vladimir Davydov <vdavydov@parallels.com> To: <akpm@linux-foundation.org> Cc: <cl@linux.com>, <hannes@cmpxchg.org>, <mhocko@suse.cz>, <linux-kernel@vger.kernel.org>, <linux-mm@kvack.org> Subject: [PATCH -mm 4/8] slub: never fail kmem_cache_shrink Date: Fri, 30 May 2014 17:51:07 +0400 [thread overview] Message-ID: <ac8907cace921c3209aa821649349106f4f70b34.1401457502.git.vdavydov@parallels.com> (raw) In-Reply-To: <cover.1401457502.git.vdavydov@parallels.com> SLUB's kmem_cache_shrink not only removes empty slabs from the cache, but also sorts slabs by the number of objects in-use to cope with fragmentation. To achieve that, it tries to allocate a temporary array. If it fails, it will abort the whole procedure. This is unacceptable for kmemcg, where we want to be sure that all empty slabs are removed from the cache on memcg offline, so let's just skip the de-fragmentation step if the allocation fails, but still get rid of empty slabs. Signed-off-by: Vladimir Davydov <vdavydov@parallels.com> --- mm/slub.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index d96faa2464c3..d9976ea93710 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3404,12 +3404,16 @@ int __kmem_cache_shrink(struct kmem_cache *s) struct page *page; struct page *t; int objects = oo_objects(s->max); - struct list_head *slabs_by_inuse = - kmalloc(sizeof(struct list_head) * objects, GFP_KERNEL); + LIST_HEAD(empty_slabs); + struct list_head *slabs_by_inuse; unsigned long flags; - if (!slabs_by_inuse) - return -ENOMEM; + slabs_by_inuse = kcalloc(objects - 1, sizeof(struct list_head), + GFP_KERNEL); + if (slabs_by_inuse) { + for (i = 0; i < objects - 1; i++) + INIT_LIST_HEAD(slabs_by_inuse + i); + } flush_all(s); for_each_node_state(node, N_NORMAL_MEMORY) { @@ -3418,9 +3422,6 @@ int __kmem_cache_shrink(struct kmem_cache *s) if (!n->nr_partial) continue; - for (i = 0; i < objects; i++) - INIT_LIST_HEAD(slabs_by_inuse + i); - spin_lock_irqsave(&n->list_lock, flags); /* @@ -3430,22 +3431,28 @@ int __kmem_cache_shrink(struct kmem_cache *s) * list_lock. page->inuse here is the upper limit. */ list_for_each_entry_safe(page, t, &n->partial, lru) { - list_move(&page->lru, slabs_by_inuse + page->inuse); - if (!page->inuse) + if (!page->inuse) { + list_move(&page->lru, &empty_slabs); n->nr_partial--; + } else if (slabs_by_inuse) + list_move(&page->lru, + slabs_by_inuse + page->inuse - 1); } /* * Rebuild the partial list with the slabs filled up most * first and the least used slabs at the end. */ - for (i = objects - 1; i > 0; i--) - list_splice(slabs_by_inuse + i, n->partial.prev); + if (slabs_by_inuse) { + for (i = objects - 2; i >= 0; i--) + list_splice_tail_init(slabs_by_inuse + i, + &n->partial); + } spin_unlock_irqrestore(&n->list_lock, flags); /* Release empty slabs */ - list_for_each_entry_safe(page, t, slabs_by_inuse, lru) + list_for_each_entry_safe(page, t, &empty_slabs, lru) discard_slab(s, page); } @@ -4780,13 +4787,9 @@ static ssize_t shrink_show(struct kmem_cache *s, char *buf) static ssize_t shrink_store(struct kmem_cache *s, const char *buf, size_t length) { - if (buf[0] == '1') { - int rc = kmem_cache_shrink(s); - - if (rc) - return rc; - } else + if (buf[0] != '1') return -EINVAL; + kmem_cache_shrink(s); return length; } SLAB_ATTR(shrink); -- 1.7.10.4
WARNING: multiple messages have this Message-ID (diff)
From: Vladimir Davydov <vdavydov@parallels.com> To: akpm@linux-foundation.org Cc: cl@linux.com, hannes@cmpxchg.org, mhocko@suse.cz, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH -mm 4/8] slub: never fail kmem_cache_shrink Date: Fri, 30 May 2014 17:51:07 +0400 [thread overview] Message-ID: <ac8907cace921c3209aa821649349106f4f70b34.1401457502.git.vdavydov@parallels.com> (raw) In-Reply-To: <cover.1401457502.git.vdavydov@parallels.com> SLUB's kmem_cache_shrink not only removes empty slabs from the cache, but also sorts slabs by the number of objects in-use to cope with fragmentation. To achieve that, it tries to allocate a temporary array. If it fails, it will abort the whole procedure. This is unacceptable for kmemcg, where we want to be sure that all empty slabs are removed from the cache on memcg offline, so let's just skip the de-fragmentation step if the allocation fails, but still get rid of empty slabs. Signed-off-by: Vladimir Davydov <vdavydov@parallels.com> --- mm/slub.c | 39 +++++++++++++++++++++------------------ 1 file changed, 21 insertions(+), 18 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index d96faa2464c3..d9976ea93710 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -3404,12 +3404,16 @@ int __kmem_cache_shrink(struct kmem_cache *s) struct page *page; struct page *t; int objects = oo_objects(s->max); - struct list_head *slabs_by_inuse = - kmalloc(sizeof(struct list_head) * objects, GFP_KERNEL); + LIST_HEAD(empty_slabs); + struct list_head *slabs_by_inuse; unsigned long flags; - if (!slabs_by_inuse) - return -ENOMEM; + slabs_by_inuse = kcalloc(objects - 1, sizeof(struct list_head), + GFP_KERNEL); + if (slabs_by_inuse) { + for (i = 0; i < objects - 1; i++) + INIT_LIST_HEAD(slabs_by_inuse + i); + } flush_all(s); for_each_node_state(node, N_NORMAL_MEMORY) { @@ -3418,9 +3422,6 @@ int __kmem_cache_shrink(struct kmem_cache *s) if (!n->nr_partial) continue; - for (i = 0; i < objects; i++) - INIT_LIST_HEAD(slabs_by_inuse + i); - spin_lock_irqsave(&n->list_lock, flags); /* @@ -3430,22 +3431,28 @@ int __kmem_cache_shrink(struct kmem_cache *s) * list_lock. page->inuse here is the upper limit. */ list_for_each_entry_safe(page, t, &n->partial, lru) { - list_move(&page->lru, slabs_by_inuse + page->inuse); - if (!page->inuse) + if (!page->inuse) { + list_move(&page->lru, &empty_slabs); n->nr_partial--; + } else if (slabs_by_inuse) + list_move(&page->lru, + slabs_by_inuse + page->inuse - 1); } /* * Rebuild the partial list with the slabs filled up most * first and the least used slabs at the end. */ - for (i = objects - 1; i > 0; i--) - list_splice(slabs_by_inuse + i, n->partial.prev); + if (slabs_by_inuse) { + for (i = objects - 2; i >= 0; i--) + list_splice_tail_init(slabs_by_inuse + i, + &n->partial); + } spin_unlock_irqrestore(&n->list_lock, flags); /* Release empty slabs */ - list_for_each_entry_safe(page, t, slabs_by_inuse, lru) + list_for_each_entry_safe(page, t, &empty_slabs, lru) discard_slab(s, page); } @@ -4780,13 +4787,9 @@ static ssize_t shrink_show(struct kmem_cache *s, char *buf) static ssize_t shrink_store(struct kmem_cache *s, const char *buf, size_t length) { - if (buf[0] == '1') { - int rc = kmem_cache_shrink(s); - - if (rc) - return rc; - } else + if (buf[0] != '1') return -EINVAL; + kmem_cache_shrink(s); return length; } SLAB_ATTR(shrink); -- 1.7.10.4 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2014-05-30 13:51 UTC|newest] Thread overview: 76+ messages / expand[flat|nested] mbox.gz Atom feed top 2014-05-30 13:51 [PATCH -mm 0/8] memcg/slab: reintroduce dead cache self-destruction Vladimir Davydov 2014-05-30 13:51 ` Vladimir Davydov 2014-05-30 13:51 ` [PATCH -mm 1/8] memcg: cleanup memcg_cache_params refcnt usage Vladimir Davydov 2014-05-30 13:51 ` Vladimir Davydov 2014-05-30 14:31 ` Christoph Lameter 2014-05-30 14:31 ` Christoph Lameter 2014-05-30 13:51 ` [PATCH -mm 2/8] memcg: destroy kmem caches when last slab is freed Vladimir Davydov 2014-05-30 13:51 ` Vladimir Davydov 2014-05-30 14:32 ` Christoph Lameter 2014-05-30 14:32 ` Christoph Lameter 2014-05-30 13:51 ` [PATCH -mm 3/8] memcg: mark caches that belong to offline memcgs as dead Vladimir Davydov 2014-05-30 13:51 ` Vladimir Davydov 2014-05-30 14:33 ` Christoph Lameter 2014-05-30 14:33 ` Christoph Lameter 2014-05-30 13:51 ` Vladimir Davydov [this message] 2014-05-30 13:51 ` [PATCH -mm 4/8] slub: never fail kmem_cache_shrink Vladimir Davydov 2014-05-30 14:46 ` Christoph Lameter 2014-05-30 14:46 ` Christoph Lameter 2014-05-31 10:18 ` Vladimir Davydov 2014-05-31 10:18 ` Vladimir Davydov 2014-06-02 15:13 ` Christoph Lameter 2014-06-02 15:13 ` Christoph Lameter 2014-05-30 13:51 ` [PATCH -mm 5/8] slab: remove kmem_cache_shrink retval Vladimir Davydov 2014-05-30 13:51 ` Vladimir Davydov 2014-05-30 14:49 ` Christoph Lameter 2014-05-30 14:49 ` Christoph Lameter 2014-05-31 10:27 ` Vladimir Davydov 2014-05-31 10:27 ` Vladimir Davydov 2014-06-02 15:16 ` Christoph Lameter 2014-06-02 15:16 ` Christoph Lameter 2014-06-03 9:06 ` Vladimir Davydov 2014-06-03 9:06 ` Vladimir Davydov 2014-06-03 14:48 ` Christoph Lameter 2014-06-03 14:48 ` Christoph Lameter 2014-06-03 19:00 ` Vladimir Davydov 2014-06-03 19:00 ` Vladimir Davydov 2014-05-30 13:51 ` [PATCH -mm 6/8] slub: do not use cmpxchg for adding cpu partials when irqs disabled Vladimir Davydov 2014-05-30 13:51 ` Vladimir Davydov 2014-05-30 13:51 ` [PATCH -mm 7/8] slub: make dead caches discard free slabs immediately Vladimir Davydov 2014-05-30 13:51 ` Vladimir Davydov 2014-05-30 14:57 ` Christoph Lameter 2014-05-30 14:57 ` Christoph Lameter 2014-05-31 11:04 ` Vladimir Davydov 2014-05-31 11:04 ` Vladimir Davydov 2014-06-02 4:24 ` Joonsoo Kim 2014-06-02 4:24 ` Joonsoo Kim 2014-06-02 11:47 ` Vladimir Davydov 2014-06-02 11:47 ` Vladimir Davydov 2014-06-02 14:03 ` Joonsoo Kim 2014-06-02 14:03 ` Joonsoo Kim 2014-06-02 15:17 ` Christoph Lameter 2014-06-02 15:17 ` Christoph Lameter 2014-06-03 8:16 ` Vladimir Davydov 2014-06-03 8:16 ` Vladimir Davydov 2014-06-04 8:53 ` Joonsoo Kim 2014-06-04 8:53 ` Joonsoo Kim 2014-06-04 9:47 ` Vladimir Davydov 2014-06-04 9:47 ` Vladimir Davydov 2014-05-30 13:51 ` [PATCH -mm 8/8] slab: reap dead memcg caches aggressively Vladimir Davydov 2014-05-30 13:51 ` Vladimir Davydov 2014-05-30 15:01 ` Christoph Lameter 2014-05-30 15:01 ` Christoph Lameter 2014-05-31 11:19 ` Vladimir Davydov 2014-05-31 11:19 ` Vladimir Davydov 2014-06-02 15:24 ` Christoph Lameter 2014-06-02 15:24 ` Christoph Lameter 2014-06-03 20:18 ` Vladimir Davydov 2014-06-03 20:18 ` Vladimir Davydov 2014-06-02 4:41 ` Joonsoo Kim 2014-06-02 4:41 ` Joonsoo Kim 2014-06-02 12:10 ` Vladimir Davydov 2014-06-02 12:10 ` Vladimir Davydov 2014-06-02 14:01 ` Joonsoo Kim 2014-06-02 14:01 ` Joonsoo Kim 2014-06-03 8:21 ` Vladimir Davydov 2014-06-03 8:21 ` Vladimir Davydov
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=ac8907cace921c3209aa821649349106f4f70b34.1401457502.git.vdavydov@parallels.com \ --to=vdavydov@parallels.com \ --cc=akpm@linux-foundation.org \ --cc=cl@linux.com \ --cc=hannes@cmpxchg.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mhocko@suse.cz \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.