All of lore.kernel.org
 help / color / mirror / Atom feed
From: Roman Gushchin <guro@fb.com>
To: Muchun Song <songmuchun@bytedance.com>
Cc: <willy@infradead.org>, <akpm@linux-foundation.org>,
	<hannes@cmpxchg.org>, <mhocko@kernel.org>,
	<vdavydov.dev@gmail.com>, <shakeelb@google.com>,
	<shy828301@gmail.com>, <alexs@kernel.org>,
	<richard.weiyang@gmail.com>, <david@fromorbit.com>,
	<trond.myklebust@hammerspace.com>, <anna.schumaker@netapp.com>,
	<jaegeuk@kernel.org>, <chao@kernel.org>,
	<kari.argillander@gmail.com>, <linux-fsdevel@vger.kernel.org>,
	<linux-kernel@vger.kernel.org>, <linux-mm@kvack.org>,
	<linux-nfs@vger.kernel.org>, <zhengqi.arch@bytedance.com>,
	<duanxiongchun@bytedance.com>, <fam.zheng@bytedance.com>,
	<smuchun@gmail.com>
Subject: Re: [PATCH v5 10/16] mm: list_lru: allocate list_lru_one only when needed
Date: Tue, 11 Jan 2022 12:00:25 -0800	[thread overview]
Message-ID: <Yd3h2YwGIZs1A+2s@carbon.dhcp.thefacebook.com> (raw)
In-Reply-To: <20211220085649.8196-11-songmuchun@bytedance.com>

On Mon, Dec 20, 2021 at 04:56:43PM +0800, Muchun Song wrote:
> In our server, we found a suspected memory leak problem. The kmalloc-32
> consumes more than 6GB of memory. Other kmem_caches consume less than
> 2GB memory.
> 
> After our in-depth analysis, the memory consumption of kmalloc-32 slab
> cache is the cause of list_lru_one allocation.
> 
>   crash> p memcg_nr_cache_ids
>   memcg_nr_cache_ids = $2 = 24574
> 
> memcg_nr_cache_ids is very large and memory consumption of each list_lru
> can be calculated with the following formula.
> 
>   num_numa_node * memcg_nr_cache_ids * 32 (kmalloc-32)
> 
> There are 4 numa nodes in our system, so each list_lru consumes ~3MB.
> 
>   crash> list super_blocks | wc -l
>   952
> 
> Every mount will register 2 list lrus, one is for inode, another is for
> dentry. There are 952 super_blocks. So the total memory is 952 * 2 * 3
> MB (~5.6GB). But the number of memory cgroup is less than 500. So I
> guess more than 12286 containers have been deployed on this machine (I
> do not know why there are so many containers, it may be a user's bug or
> the user really want to do that). And memcg_nr_cache_ids has not been
> reduced to a suitable value. This can waste a lot of memory.

But on the other side you increase the size of struct list_lru_per_memcg,
so if number of cgroups is close to memcg_nr_cache_ids, we can actually
waste more memory. I'm not saying the change is not worth it, but would be
nice to add some real-world numbers.

Or it's all irrelevant and is done as a preparation to the conversion to xarray?
If so, please, make it clear.

Thanks!

  parent reply	other threads:[~2022-01-11 20:00 UTC|newest]

Thread overview: 45+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-12-20  8:56 [PATCH v5 00/16] Optimize list lru memory consumption Muchun Song
2021-12-20  8:56 ` [PATCH v5 01/16] mm: list_lru: optimize memory consumption of arrays of per cgroup lists Muchun Song
2022-01-07  0:05   ` Roman Gushchin
2022-01-09  4:49     ` Muchun Song
2022-01-10 18:42       ` Roman Gushchin
2022-01-11  3:19         ` Muchun Song
2021-12-20  8:56 ` [PATCH v5 02/16] mm: introduce kmem_cache_alloc_lru Muchun Song
2022-01-07  3:04   ` Roman Gushchin
2022-01-09  6:21     ` Muchun Song
2022-01-10 18:47       ` Roman Gushchin
2022-01-11 15:41         ` Vlastimil Babka
2022-01-11 17:54           ` Roman Gushchin
2021-12-20  8:56 ` [PATCH v5 03/16] fs: introduce alloc_inode_sb() to allocate filesystems specific inode Muchun Song
2022-01-11 18:55   ` Roman Gushchin
2022-01-12  2:54     ` Muchun Song
2021-12-20  8:56 ` [PATCH v5 04/16] fs: allocate inode by using alloc_inode_sb() Muchun Song
2022-01-11 18:58   ` Roman Gushchin
2022-01-12  2:55     ` Muchun Song
2021-12-20  8:56 ` [PATCH v5 05/16] f2fs: " Muchun Song
2022-01-11 19:03   ` Roman Gushchin
2021-12-20  8:56 ` [PATCH v5 06/16] nfs42: use a specific kmem_cache to allocate nfs4_xattr_entry Muchun Song
2021-12-20  8:56 ` [PATCH v5 07/16] mm: dcache: use kmem_cache_alloc_lru() to allocate dentry Muchun Song
2022-01-11 19:05   ` Roman Gushchin
2021-12-20  8:56 ` [PATCH v5 08/16] xarray: use kmem_cache_alloc_lru to allocate xa_node Muchun Song
2022-01-11 19:14   ` Roman Gushchin
2021-12-20  8:56 ` [PATCH v5 09/16] mm: memcontrol: move memcg_online_kmem() to mem_cgroup_css_online() Muchun Song
2022-01-11 19:17   ` Roman Gushchin
2021-12-20  8:56 ` [PATCH v5 10/16] mm: list_lru: allocate list_lru_one only when needed Muchun Song
2022-01-06 11:00   ` Michal Koutný
2022-01-12 13:22     ` Muchun Song
2022-01-13 13:32       ` Michal Koutný
2022-01-18 12:05         ` Muchun Song
2022-01-19  9:33           ` Michal Koutný
2022-01-21  5:28             ` Muchun Song
2022-01-11 20:00   ` Roman Gushchin [this message]
2022-01-12  4:48     ` Muchun Song
2021-12-20  8:56 ` [PATCH v5 11/16] mm: list_lru: rename memcg_drain_all_list_lrus to memcg_reparent_list_lrus Muchun Song
2021-12-20  8:56 ` [PATCH v5 12/16] mm: list_lru: replace linear array with xarray Muchun Song
2021-12-20  8:56 ` [PATCH v5 13/16] mm: memcontrol: reuse memory cgroup ID for kmem ID Muchun Song
2021-12-20  9:27   ` Mika Penttilä
2022-01-05 17:03   ` Michal Koutný
2022-01-06  3:34     ` Muchun Song
2021-12-20  8:56 ` [PATCH v5 14/16] mm: memcontrol: fix cannot alloc the maximum memcg ID Muchun Song
2021-12-20  8:56 ` [PATCH v5 15/16] mm: list_lru: rename list_lru_per_memcg to list_lru_memcg Muchun Song
2021-12-20  8:56 ` [PATCH v5 16/16] mm: memcontrol: rename memcg_cache_id to memcg_kmem_id Muchun Song

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=Yd3h2YwGIZs1A+2s@carbon.dhcp.thefacebook.com \
    --to=guro@fb.com \
    --cc=akpm@linux-foundation.org \
    --cc=alexs@kernel.org \
    --cc=anna.schumaker@netapp.com \
    --cc=chao@kernel.org \
    --cc=david@fromorbit.com \
    --cc=duanxiongchun@bytedance.com \
    --cc=fam.zheng@bytedance.com \
    --cc=hannes@cmpxchg.org \
    --cc=jaegeuk@kernel.org \
    --cc=kari.argillander@gmail.com \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=linux-nfs@vger.kernel.org \
    --cc=mhocko@kernel.org \
    --cc=richard.weiyang@gmail.com \
    --cc=shakeelb@google.com \
    --cc=shy828301@gmail.com \
    --cc=smuchun@gmail.com \
    --cc=songmuchun@bytedance.com \
    --cc=trond.myklebust@hammerspace.com \
    --cc=vdavydov.dev@gmail.com \
    --cc=willy@infradead.org \
    --cc=zhengqi.arch@bytedance.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.