All of lore.kernel.org
 help / color / mirror / Atom feed
From: Vaibhav Rustagi <vaibhavrustagi@google.com>
To: Greg KH <gregkh@linuxfoundation.org>
Cc: stable@vger.kernel.org, hannes@cmpxchg.org, tj@kernel.org,
	mhocko@suse.com, vdavydov.dev@gmail.com, guro@fb.com,
	riel@surriel.com, sfr@canb.auug.org.au,
	akpm@linux-foundation.org, torvalds@linux-foundation.org,
	Aditya Kali <adityakali@google.com>
Subject: Re: [For Stable] mm: memcontrol: fix excessive complexity in memory.stat reporting
Date: Wed, 24 Apr 2019 10:35:51 -0700	[thread overview]
Message-ID: <CAMVonLhmPhcTVSAbZzCbmYQxRECWK+6bychxFzg232dtAXeqEA@mail.gmail.com> (raw)
In-Reply-To: <20190424165009.GE21916@kroah.com>

Apologies for sending a non-plain text e-mail previously.

This issue is encountered in the actual production environment by our
customers where they are constantly creating containers
and tearing them down (using kubernetes for the workload).  Kubernetes
constantly reads the memory.stat file for accounting memory
information and over time (around a week) the memcg's got accumulated
and the response time for reading memory.stat increases and
customer applications get affected.

The repro steps mentioned previously was just used for testing the
patches locally.

Yes, we are moving to 4.19 but are also supporting 4.14 till Jan 2020
(so production environment will still contain 4.14 kernel)

Let me know your thoughts on this.

Thanks,
Vaibhav


On Wed, Apr 24, 2019 at 9:50 AM Greg KH <gregkh@linuxfoundation.org> wrote:
>
> On Mon, Apr 01, 2019 at 10:35:59PM -0700, Vaibhav Rustagi wrote:
> > In linux stable kernel (tested on 4.14), reading memory.stat in case
> > of tens of thousands of ghost cgroups pinned by lingering page cache
> > takes up to 100 ms ~ 700 ms to complete the reading.
>
> Great, don't do that :)
>
> > Repro steps (tested on 4.14 kernel):
> >
> > $ cat /tmp/make_zombies
> >
> > mkdir /tmp/fs
> > mount -t tmpfs nodev /tmp/fs
> > for i in {1..10000}; do
> >    mkdir /sys/fs/cgroup/memory/z$i
> >    (echo $BASHPID >> /sys/fs/cgroup/memory/z$i/cgroup.procs && echo $i
> > > /tmp/fs/$i)
> >  done
> >
> > # establish baseline
> > $ perf stat -r3 cat /sys/fs/cgroup/memory/memory.stat > /dev/null
> > 0.011642670 seconds time elapsed
> >
> > $ bash /tmp/make_zombies
> > $ perf stat -r3 cat /sys/fs/cgroup/memory/memory.stat > /dev/null
> > 0.134939281 seconds time elapsed
> >
> > $ rmdir /sys/fs/cgroup/memory/z*
> > $ perf stat -r3 cat /sys/fs/cgroup/memory/memory.stat > /dev/null
> > 0.135323145 seconds time elapsed
> > # even after rmdir we have zombies, so still slow.
> >
> > The fix is already present in linux master (since 4.16) by following commits:
> >
> > c9019e9bf42e66d028d70d2da6206cad4dd9250d mm: memcontrol: eliminate raw
> > access to stat and event counters
> > 284542656e22c43fdada8c8cc0ca9ede8453eed7  mm: memcontrol: implement
> > lruvec stat functions on top of each other
> > a983b5ebee57209c99f68c8327072f25e0e6e3da  mm: memcontrol: fix
> > excessive complexity in memory.stat reporting
> > c3cc39118c3610eb6ab4711bc624af7fc48a35fe  mm: memcontrol: fix
> > NR_WRITEBACK leak in memcg and system stats
> > e27be240df53f1a20c659168e722b5d9f16cc7f4  mm: memcg: make sure
> > memory.events is uptodate when waking pollers
> >
> > I would like to request cherry-picking the above commits to
> > linux-stable branch - 4.14.
>
> What's wrong with just moving to a newer kernel, like 4.19.y, if you
> have this issue?  That's a much better thing to do than to backport the
> above patches, right?
>
> As this is just an "annoyance", on the old kernel, I don't really see
> why it's needed to be backported, it can't cause any problems overall,
> right?
>
> thanks,
>
> greg k-h

  reply	other threads:[~2019-04-24 17:42 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-04-02  5:35 [For Stable] mm: memcontrol: fix excessive complexity in memory.stat reporting Vaibhav Rustagi
2019-04-24 16:50 ` Greg KH
2019-04-24 17:35   ` Vaibhav Rustagi [this message]
2019-04-24 18:34     ` Greg KH
2019-04-30 20:41       ` Vaibhav Rustagi
2019-05-01  7:08         ` Greg KH
  -- strict thread matches above, loose matches on Subject: below --
2019-04-01 20:34 Vaibhav Rustagi
2019-04-02  5:24 ` Greg KH

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAMVonLhmPhcTVSAbZzCbmYQxRECWK+6bychxFzg232dtAXeqEA@mail.gmail.com \
    --to=vaibhavrustagi@google.com \
    --cc=adityakali@google.com \
    --cc=akpm@linux-foundation.org \
    --cc=gregkh@linuxfoundation.org \
    --cc=guro@fb.com \
    --cc=hannes@cmpxchg.org \
    --cc=mhocko@suse.com \
    --cc=riel@surriel.com \
    --cc=sfr@canb.auug.org.au \
    --cc=stable@vger.kernel.org \
    --cc=tj@kernel.org \
    --cc=torvalds@linux-foundation.org \
    --cc=vdavydov.dev@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.