Re: PROBLEM: cgroup cost too much memory when transfer small files to tmpfs

From: Roman Gushchin <guro@fb.com>
To: Johannes Weiner <hannes@cmpxchg.org>
Cc: jingrui <jingrui@huawei.com>, "tj@kernel.org" <tj@kernel.org>,
	Lizefan <lizefan@huawei.com>,
	"mhocko@kernel.org" <mhocko@kernel.org>,
	"vdavydov.dev@gmail.com" <vdavydov.dev@gmail.com>,
	"akpm@linux-foundation.org" <akpm@linux-foundation.org>,
	"linux-mm@kvack.org" <linux-mm@kvack.org>,
	"cgroups@vger.kernel.org" <cgroups@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	caihaomin <caihaomin@huawei.com>,
	"Weiwei (N)" <wick.wei@huawei.com>, <guro@cmpxchg.org>
Subject: Re: PROBLEM: cgroup cost too much memory when transfer small files to tmpfs
Date: Tue, 21 Jul 2020 11:49:59 -0700	[thread overview]
Message-ID: <20200721184959.GA8266@carbon.DHCP.thefacebook.com> (raw)
In-Reply-To: <20200721174126.GA271870@cmpxchg.org>

On Tue, Jul 21, 2020 at 01:41:26PM -0400, Johannes Weiner wrote:
> On Tue, Jul 21, 2020 at 11:19:52AM +0000, jingrui wrote:
> > Cc: Johannes Weiner <hannes@cmpxchg.org> ; Michal Hocko <mhocko@kernel.org>; Vladimir Davydov <vdavydov.dev@gmail.com>
> > 
> > Thanks.
> > 
> > ---
> > PROBLEM: cgroup cost too much memory when transfer small files to tmpfs.
> > 
> > keywords: cgroup PERCPU/memory cost too much.
> > 
> > description:
> > 
> > We send small files from node-A to node-B tmpfs /tmp directory using sftp. On
> > node-B the systemd configured with pam on like below.
> > 
> > cat /etc/pam.d/password-auth | grep systemd
> > -session     optional      pam_systemd.so
> > 
> > So when transfer a file, a systemd session is created, that means a cgroup is
> > created, then file saved at /tmp will associated with a cgroup object. After
> > file transferred, session and cgroup-dir will be removed, but the file in /tmp
> > still associated with the cgroup object. The PERCPU memory in cgroup/css object
> > cost a lot(about 0.5MB/per-cgroup-object) on 200/cpus machine.
> 
> CC Roman who had a patch series to free all this extended (percpu)
> memory upon cgroup deletion:
> 
> https://lore.kernel.org/patchwork/cover/1050508/
> 
> It looks like it never got merged for some reason.

The mentioned patchset can make the problem less noticeable, but can't solve it completely.
It has never been merged, because the dying cgroup problem was mostly solved by other methods:
slab memory reparenting and various reclaim fixes. So there was no more reason to complicate
the code to release the memcg memory early.

The overhead of creating and destroying a new memory cgroup for a transfer of a small
file will be noticeable anyway. So IMO the solution is to use a single cgroup for all
transfers. I don't know if systemd supports such mode out of the box, but it shouldn't
be hard to add it.

But also I wonder if we need a special tmpfs mount option, something like "noaccount".
Not only for this specific case, but also for the case when tmpfs is extensively
shared between multiple cgroups or if it's used to pass some data from one cgroup
to another, or if we care about the performance more than about the accounting;
in other words for cases where the accounting makes more harm than good.

Thanks!