From: Vladimir Davydov <vdavydov@parallels.com>
To: Joonsoo Kim <iamjoonsoo.kim@lge.com>
Cc: <akpm@linux-foundation.org>, <cl@linux.com>,
<rientjes@google.com>, <penberg@kernel.org>, <hannes@cmpxchg.org>,
<mhocko@suse.cz>, <linux-kernel@vger.kernel.org>,
<linux-mm@kvack.org>
Subject: Re: [PATCH -mm v3 8/8] slab: do not keep free objects/slabs on dead memcg caches
Date: Wed, 25 Jun 2014 17:45:45 +0400 [thread overview]
Message-ID: <20140625134545.GB22340@esperanza> (raw)
In-Reply-To: <20140624073840.GC4836@js1304-P5Q-DELUXE>
On Tue, Jun 24, 2014 at 04:38:41PM +0900, Joonsoo Kim wrote:
> On Fri, Jun 13, 2014 at 12:38:22AM +0400, Vladimir Davydov wrote:
> And, you said that this way of implementation would be slow because
> there could be many object in dead caches and this implementation
> needs node spin_lock on each object freeing. Is it no problem now?
>
> If you have any performance data about this implementation and
> alternative one, could you share it?
I ran some tests on a 2 CPU x 6 core x 2 HT box. The kernel was compiled
with a config taken from a popular distro, so it had most of debug
options turned off.
---
TEST #1: Each logical CPU executes a task that frees 1M objects
allocated from the same cache. All frees are node-local.
RESULTS:
objsize (bytes) | cache is dead? | objects free time (ms)
----------------+----------------+-----------------------
64 | - | 373 +- 5
- | + | 1300 +- 6
| |
128 | - | 387 +- 6
- | + | 1337 +- 6
| |
256 | - | 484 +- 4
- | + | 1407 +- 6
| |
512 | - | 686 +- 5
- | + | 1561 +- 18
| |
1024 | - | 1073 +- 11
- | + | 1897 +- 12
TEST #2: Each logical CPU executes a task that removes 1M empty files
from its own RAMFS mount. All frees are node-local.
RESULTS:
cache is dead? | files removal time (s)
----------------+----------------------------------
- | 15.57 +- 0.55 (base)
+ | 16.80 +- 0.62 (base + 8%)
---
So, according to TEST #1 the relative slowdown introduced by zapping per
cpu arrays is really dreadful - it can be up to 4x! However, the
absolute numbers aren't that huge - ~1 second for 24 million objects.
If we do something else except kfree the slowdown shouldn't be that
visible IMO.
TEST #2 is an attempt to estimate how zapping of per cpu arrays will
affect FS objects destruction, which is the most common case of dead
caches usage. To avoid disk-bound operations it uses RAMFS. From the
test results it follows that the relative slowdown of massive file
deletion is within 2 stdev, which looks decent.
Anyway, the alternative approach (reaping dead caches periodically)
won't have this kfree slowdown at all. However, periodic reaping can
become a real disaster as the system evolves and the number of dead
caches grows. Currently I don't know how we can estimate real life
effects of this. If you have any ideas, please let me know.
Thanks.
next prev parent reply other threads:[~2014-06-25 13:46 UTC|newest]
Thread overview: 25+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-06-12 20:38 [PATCH -mm v3 0/8] memcg/slab: reintroduce dead cache self-destruction Vladimir Davydov
2014-06-12 20:38 ` [PATCH -mm v3 1/8] memcg: cleanup memcg_cache_params refcnt usage Vladimir Davydov
2014-06-12 20:38 ` [PATCH -mm v3 2/8] memcg: destroy kmem caches when last slab is freed Vladimir Davydov
2014-06-12 20:38 ` [PATCH -mm v3 3/8] memcg: mark caches that belong to offline memcgs as dead Vladimir Davydov
2014-06-12 20:38 ` [PATCH -mm v3 4/8] slub: don't fail kmem_cache_shrink if slab placement optimization fails Vladimir Davydov
2014-06-12 20:38 ` [PATCH -mm v3 5/8] slub: make slab_free non-preemptable Vladimir Davydov
2014-06-12 20:38 ` [PATCH -mm v3 6/8] memcg: wait for kfree's to finish before destroying cache Vladimir Davydov
2014-06-12 20:38 ` [PATCH -mm v3 7/8] slub: make dead memcg caches discard free slabs immediately Vladimir Davydov
2014-06-13 16:54 ` Christoph Lameter
2014-06-24 7:50 ` Joonsoo Kim
2014-06-24 8:25 ` Vladimir Davydov
2014-06-24 9:42 ` [PATCH -mm] slub: kmem_cache_shrink: check if partial list is empty under list_lock Vladimir Davydov
2014-06-12 20:38 ` [PATCH -mm v3 8/8] slab: do not keep free objects/slabs on dead memcg caches Vladimir Davydov
2014-06-12 20:41 ` Vladimir Davydov
2014-06-24 7:25 ` Joonsoo Kim
2014-06-24 7:42 ` Vladimir Davydov
2014-06-24 12:28 ` [PATCH -mm] slab: set free_limit for dead caches to 0 Vladimir Davydov
2014-06-24 7:38 ` [PATCH -mm v3 8/8] slab: do not keep free objects/slabs on dead memcg caches Joonsoo Kim
2014-06-24 7:48 ` Vladimir Davydov
2014-06-25 13:45 ` Vladimir Davydov [this message]
2014-06-27 6:05 ` Joonsoo Kim
2014-06-30 15:49 ` Christoph Lameter
2014-07-01 7:46 ` Vladimir Davydov
2014-06-25 14:39 ` [PATCH] slab: document why cache can have no per cpu array on kfree Vladimir Davydov
2014-06-25 16:19 ` Christoph Lameter
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20140625134545.GB22340@esperanza \
--to=vdavydov@parallels.com \
--cc=akpm@linux-foundation.org \
--cc=cl@linux.com \
--cc=hannes@cmpxchg.org \
--cc=iamjoonsoo.kim@lge.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@suse.cz \
--cc=penberg@kernel.org \
--cc=rientjes@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).