From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.0 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A, SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45ADDC433DF for ; Thu, 6 Aug 2020 12:42:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EA8B523106 for ; Thu, 6 Aug 2020 12:42:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EA8B523106 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 02FCF6B0002; Thu, 6 Aug 2020 08:42:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F22666B0005; Thu, 6 Aug 2020 08:42:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E11286B0006; Thu, 6 Aug 2020 08:42:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0215.hostedemail.com [216.40.44.215]) by kanga.kvack.org (Postfix) with ESMTP id C80276B0002 for ; Thu, 6 Aug 2020 08:42:46 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 32F93180AD807 for ; Thu, 6 Aug 2020 12:42:46 +0000 (UTC) X-FDA: 77120107932.07.fog50_13141bb26fb7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 037461803F9A1 for ; Thu, 6 Aug 2020 12:42:45 +0000 (UTC) X-HE-Tag: fog50_13141bb26fb7 X-Filterd-Recvd-Size: 7736 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Thu, 6 Aug 2020 12:42:45 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 74907AFCD; Thu, 6 Aug 2020 12:43:01 +0000 (UTC) Subject: Re: [PATCH 1/2] mm/slub: Introduce two counters for the partial objects To: Xunlei Pang , Christoph Lameter , Andrew Morton , Wen Yang , Yang Shi , Roman Gushchin Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Konstantin Khlebnikov , David Rientjes References: <1593678728-128358-1-git-send-email-xlpang@linux.alibaba.com> From: Vlastimil Babka Message-ID: Date: Thu, 6 Aug 2020 14:42:44 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <1593678728-128358-1-git-send-email-xlpang@linux.alibaba.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 037461803F9A1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 7/2/20 10:32 AM, Xunlei Pang wrote: > The node list_lock in count_partial() spend long time iterating > in case of large amount of partial page lists, which can cause > thunder herd effect to the list_lock contention, e.g. it cause > business response-time jitters when accessing "/proc/slabinfo" > in our production environments. > > This patch introduces two counters to maintain the actual number > of partial objects dynamically instead of iterating the partial > page lists with list_lock held. > > New counters of kmem_cache_node are: pfree_objects, ptotal_objects. > The main operations are under list_lock in slow path, its performance > impact is minimal. > > Co-developed-by: Wen Yang > Signed-off-by: Xunlei Pang This or similar things seem to be reported every few months now, last time was here [1] AFAIK. The solution was to just stop counting at some point. Shall we perhaps add these counters under CONFIG_SLUB_DEBUG then and be done with it? If anyone needs the extreme performance and builds without CONFIG_SLUB_DEBUG, I'd assume they also don't have userspace programs reading /proc/slabinfo periodically anyway? [1] https://lore.kernel.org/linux-mm/158860845968.33385.4165926113074799048.stgit@buzz/ > --- > mm/slab.h | 2 ++ > mm/slub.c | 38 +++++++++++++++++++++++++++++++++++++- > 2 files changed, 39 insertions(+), 1 deletion(-) > > diff --git a/mm/slab.h b/mm/slab.h > index 7e94700..5935749 100644 > --- a/mm/slab.h > +++ b/mm/slab.h > @@ -616,6 +616,8 @@ struct kmem_cache_node { > #ifdef CONFIG_SLUB > unsigned long nr_partial; > struct list_head partial; > + atomic_long_t pfree_objects; /* partial free objects */ > + atomic_long_t ptotal_objects; /* partial total objects */ > #ifdef CONFIG_SLUB_DEBUG > atomic_long_t nr_slabs; > atomic_long_t total_objects; > diff --git a/mm/slub.c b/mm/slub.c > index 6589b41..53890f3 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -1775,10 +1775,24 @@ static void discard_slab(struct kmem_cache *s, struct page *page) > /* > * Management of partially allocated slabs. > */ > + > +static inline void > +__update_partial_free(struct kmem_cache_node *n, long delta) > +{ > + atomic_long_add(delta, &n->pfree_objects); > +} > + > +static inline void > +__update_partial_total(struct kmem_cache_node *n, long delta) > +{ > + atomic_long_add(delta, &n->ptotal_objects); > +} > + > static inline void > __add_partial(struct kmem_cache_node *n, struct page *page, int tail) > { > n->nr_partial++; > + __update_partial_total(n, page->objects); > if (tail == DEACTIVATE_TO_TAIL) > list_add_tail(&page->slab_list, &n->partial); > else > @@ -1798,6 +1812,7 @@ static inline void remove_partial(struct kmem_cache_node *n, > lockdep_assert_held(&n->list_lock); > list_del(&page->slab_list); > n->nr_partial--; > + __update_partial_total(n, -page->objects); > } > > /* > @@ -1842,6 +1857,7 @@ static inline void *acquire_slab(struct kmem_cache *s, > return NULL; > > remove_partial(n, page); > + __update_partial_free(n, -*objects); > WARN_ON(!freelist); > return freelist; > } > @@ -2174,8 +2190,11 @@ static void deactivate_slab(struct kmem_cache *s, struct page *page, > "unfreezing slab")) > goto redo; > > - if (lock) > + if (lock) { > + if (m == M_PARTIAL) > + __update_partial_free(n, page->objects - page->inuse); > spin_unlock(&n->list_lock); > + } > > if (m == M_PARTIAL) > stat(s, tail); > @@ -2241,6 +2260,7 @@ static void unfreeze_partials(struct kmem_cache *s, > discard_page = page; > } else { > add_partial(n, page, DEACTIVATE_TO_TAIL); > + __update_partial_free(n, page->objects - page->inuse); > stat(s, FREE_ADD_PARTIAL); > } > } > @@ -2915,6 +2935,14 @@ static void __slab_free(struct kmem_cache *s, struct page *page, > head, new.counters, > "__slab_free")); > > + if (!was_frozen && prior) { > + if (n) > + __update_partial_free(n, cnt); > + else > + __update_partial_free(get_node(s, page_to_nid(page)), > + cnt); > + } > + > if (likely(!n)) { > > /* > @@ -2944,6 +2972,7 @@ static void __slab_free(struct kmem_cache *s, struct page *page, > if (!kmem_cache_has_cpu_partial(s) && unlikely(!prior)) { > remove_full(s, n, page); > add_partial(n, page, DEACTIVATE_TO_TAIL); > + __update_partial_free(n, page->objects - page->inuse); > stat(s, FREE_ADD_PARTIAL); > } > spin_unlock_irqrestore(&n->list_lock, flags); > @@ -2955,6 +2984,7 @@ static void __slab_free(struct kmem_cache *s, struct page *page, > * Slab on the partial list. > */ > remove_partial(n, page); > + __update_partial_free(n, page->inuse - page->objects); > stat(s, FREE_REMOVE_PARTIAL); > } else { > /* Slab must be on the full list */ > @@ -3364,6 +3394,8 @@ static inline int calculate_order(unsigned int size) > n->nr_partial = 0; > spin_lock_init(&n->list_lock); > INIT_LIST_HEAD(&n->partial); > + atomic_long_set(&n->pfree_objects, 0); > + atomic_long_set(&n->ptotal_objects, 0); > #ifdef CONFIG_SLUB_DEBUG > atomic_long_set(&n->nr_slabs, 0); > atomic_long_set(&n->total_objects, 0); > @@ -3437,6 +3469,7 @@ static void early_kmem_cache_node_alloc(int node) > * initialized and there is no concurrent access. > */ > __add_partial(n, page, DEACTIVATE_TO_HEAD); > + __update_partial_free(n, page->objects - page->inuse); > } > > static void free_kmem_cache_nodes(struct kmem_cache *s) > @@ -3747,6 +3780,7 @@ static void free_partial(struct kmem_cache *s, struct kmem_cache_node *n) > list_for_each_entry_safe(page, h, &n->partial, slab_list) { > if (!page->inuse) { > remove_partial(n, page); > + __update_partial_free(n, page->objects - page->inuse); > list_add(&page->slab_list, &discard); > } else { > list_slab_objects(s, page, > @@ -4045,6 +4079,8 @@ int __kmem_cache_shrink(struct kmem_cache *s) > if (free == page->objects) { > list_move(&page->slab_list, &discard); > n->nr_partial--; > + __update_partial_free(n, -free); > + __update_partial_total(n, -free); > } else if (free <= SHRINK_PROMOTE_MAX) > list_move(&page->slab_list, promote + free - 1); > } >