Re: [memcg] 0f12156dff: will-it-scale.per_process_ops -33.6% regression

From: Jens Axboe <axboe@kernel.dk>
To: Shakeel Butt <shakeelb@google.com>
Cc: kernel test robot <oliver.sang@intel.com>,
	Vasily Averin <vvs@virtuozzo.com>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Alexander Viro <viro@zeniv.linux.org.uk>,
	Alexey Dobriyan <adobriyan@gmail.com>,
	Andrei Vagin <avagin@gmail.com>, Borislav Petkov <bp@alien8.de>,
	Borislav Petkov <bp@suse.de>,
	Christian Brauner <christian.brauner@ubuntu.com>,
	Dmitry Safonov <0x7f454c46@gmail.com>,
	"Eric W. Biederman" <ebiederm@xmission.com>,
	Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	"H. Peter Anvin" <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>,
	"J. Bruce Fields" <bfields@fieldses.org>,
	Jeff Layton <jlayton@kernel.org>,
	Jiri Slaby <jirislaby@kernel.org>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Kirill Tkhai <ktkhai@virtuozzo.com>,
	Michal Hocko <mhocko@kernel.org>, Oleg Nesterov <oleg@redhat.com>,
	Roman Gushchin <guro@fb.com>, Serge Hallyn <serge@hallyn.com>,
	Tejun Heo <tj@kernel.org>, Thomas Gleixner <tglx@linutronix.de>,
	Vladimir Davydov <vdavydov.dev@gmail.com>,
	Yutian Yang <nglaive@gmail.com>,
	Zefan Li <lizefan.x@bytedance.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	LKML <linux-kernel@vger.kernel.org>,
	lkp@lists.01.org, kernel test robot <lkp@intel.com>,
	Huang Ying <ying.huang@intel.com>,
	Feng Tang <feng.tang@intel.com>,
	Xing Zhengjun <zhengjun.xing@linux.intel.com>
Subject: Re: [memcg] 0f12156dff: will-it-scale.per_process_ops -33.6% regression
Date: Tue, 7 Sep 2021 10:14:22 -0600	[thread overview]
Message-ID: <27face50-0e55-ad2b-ebb7-2fe48aee8374@kernel.dk> (raw)
In-Reply-To: <CALvZod6S+NmG8+-ftS-w3t3zJg_kbr3-F9h-n4jwSnqh0jTUGQ@mail.gmail.com>

On 9/7/21 9:57 AM, Shakeel Butt wrote:
> On Tue, Sep 7, 2021 at 8:46 AM Jens Axboe <axboe@kernel.dk> wrote:
>>
>> On 9/7/21 9:07 AM, kernel test robot wrote:
>>>
>>>
>>> Greeting,
>>>
>>> FYI, we noticed a -33.6% regression of will-it-scale.per_process_ops due to commit:
>>>
>>>
>>> commit: 0f12156dff2862ac54235fc72703f18770769042 ("memcg: enable accounting for file lock caches")
>>> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
>>
>> Are we at all worried about these? There's been a number of them
>> reported, basically for all the accounting enablements that have been
>> done in this merge window.
>>
>> When io_uring was switched to use accounted memory, we did a bunch of
>> work to ameliorate the inevitable slowdowns that happen if you do
>> repeated allocs and/or frees and have memcg accounting enabled.
>>
> 
> I think these are important and we should aim to continuously improve
> performance with memcg accounting. I would like to know more about the
> io_uring work done to improve memcg accounting. Maybe we can
> generalize it to others as well.

It's pretty basic and may not be applicable to all cases, we simply hang
on to our allocations for longer periods and reuse them. Hence instead
of always going through alloc+free to each "unit", they are recycled and
reused until no longer needed.

Now this is more efficient in general for us, as we can have a very high
rate of requests (and hence allocs+frees). I suspect most use cases
would benefit from simply having a cache in front of memcg slabs, but
that seems like solving the issue at the wrong layer. IMHO it'd be
better to have the memcg accounting be done in batches, eg have some
notion of deferred frees. If someone allocates before the deferred frees
are accounted, then that would have saved two pieces of accounting.

It is of course possible that a lot of these regressions are simply
accounting the alloc, in which case it seems like accounting in batches
might help there. All depends on the slack that is acceptable for memcg.

-- 
Jens Axboe