From: Shakeel Butt <shakeelb@google.com>
To: Yang Shi <yang.shi@linux.alibaba.com>
Cc: Kirill Tkhai <ktkhai@virtuozzo.com>,
Vladimir Davydov <vdavydov.dev@gmail.com>,
Hugh Dickins <hughd@google.com>, Michal Hocko <mhocko@suse.com>,
Johannes Weiner <hannes@cmpxchg.org>,
Roman Gushchin <guro@fb.com>, Qian Cai <cai@lca.pw>,
"Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>,
Andrew Morton <akpm@linux-foundation.org>,
Linux MM <linux-mm@kvack.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: list corruption in deferred_split_scan()
Date: Wed, 17 Jul 2019 10:02:11 -0700 [thread overview]
Message-ID: <CALvZod7Je+gekSGR61LMeHdYoC_PJune_0qGNiDfNH2=oNeOgw@mail.gmail.com> (raw)
In-Reply-To: <50f57bf8-a71a-c61f-74f7-31fb7bfe3253@linux.alibaba.com>
On Tue, Jul 16, 2019 at 5:12 PM Yang Shi <yang.shi@linux.alibaba.com> wrote:
>
>
>
> On 7/16/19 4:36 PM, Shakeel Butt wrote:
> > Adding related people.
> >
> > The thread starts at:
> > http://lkml.kernel.org/r/1562795006.8510.19.camel@lca.pw
> >
> > On Mon, Jul 15, 2019 at 8:01 PM Yang Shi <yang.shi@linux.alibaba.com> wrote:
> >>
> >>
> >> On 7/15/19 6:36 PM, Qian Cai wrote:
> >>>> On Jul 15, 2019, at 8:22 PM, Yang Shi <yang.shi@linux.alibaba.com> wrote:
> >>>>
> >>>>
> >>>>
> >>>> On 7/15/19 2:23 PM, Qian Cai wrote:
> >>>>> On Fri, 2019-07-12 at 12:12 -0700, Yang Shi wrote:
> >>>>>>> Another possible lead is that without reverting the those commits below,
> >>>>>>> kdump
> >>>>>>> kernel would always also crash in shrink_slab_memcg() at this line,
> >>>>>>>
> >>>>>>> map = rcu_dereference_protected(memcg->nodeinfo[nid]->shrinker_map, true);
> >>>>>> This looks a little bit weird. It seems nodeinfo[nid] is NULL? I didn't
> >>>>>> think of where nodeinfo was freed but memcg was still online. Maybe a
> >>>>>> check is needed:
> >>>>> Actually, "memcg" is NULL.
> >>>> It sounds weird. shrink_slab() is called in mem_cgroup_iter which does pin the memcg. So, the memcg should not go away.
> >>> Well, the commit “mm: shrinker: make shrinker not depend on memcg kmem” changed this line in shrink_slab_memcg(),
> >>>
> >>> - if (!memcg_kmem_enabled() || !mem_cgroup_online(memcg))
> >>> + if (!mem_cgroup_online(memcg))
> >>> return 0;
> >>>
> >>> Since the kdump kernel has the parameter “cgroup_disable=memory”, shrink_slab_memcg() will no longer be able to handle NULL memcg from mem_cgroup_iter() as,
> >>>
> >>> if (mem_cgroup_disabled())
> >>> return NULL;
> >> Aha, yes. memcg_kmem_enabled() implicitly checks !mem_cgroup_disabled().
> >> Thanks for figuring this out. I think we need add mem_cgroup_dsiabled()
> >> check before calling shrink_slab_memcg() as below:
> >>
> >> diff --git a/mm/vmscan.c b/mm/vmscan.c
> >> index a0301ed..2f03c61 100644
> >> --- a/mm/vmscan.c
> >> +++ b/mm/vmscan.c
> >> @@ -701,7 +701,7 @@ static unsigned long shrink_slab(gfp_t gfp_mask, int
> >> nid,
> >> unsigned long ret, freed = 0;
> >> struct shrinker *shrinker;
> >>
> >> - if (!mem_cgroup_is_root(memcg))
> >> + if (!mem_cgroup_disabled() && !mem_cgroup_is_root(memcg))
> >> return shrink_slab_memcg(gfp_mask, nid, memcg, priority);
> >>
> >> if (!down_read_trylock(&shrinker_rwsem))
> >>
> > We were seeing unneeded oom-kills on kernels with
> > "cgroup_disabled=memory" and Yang's patch series basically expose the
> > bug to crash. I think the commit aeed1d325d42 ("mm/vmscan.c:
> > generalize shrink_slab() calls in shrink_node()") missed the case for
> > "cgroup_disabled=memory". However I am surprised that root_mem_cgroup
> > is allocated even for "cgroup_disabled=memory" and it seems like
> > css_alloc() is called even before checking if the corresponding
> > controller is disabled.
>
> I'm surprised too. A quick test with drgn shows root memcg is definitely
> allocated:
>
> >>> prog['root_mem_cgroup']
> *(struct mem_cgroup *)0xffff8902cf058000 = {
> [snip]
>
> But, isn't this a bug?
It can be treated as a bug as this is not expected but we can discuss
and take care of it later. I think we need your patch urgently as
memory reclaim and /proc/sys/vm/drop_caches is broken for
"cgroup_disabled=memory" kernel. So, please send your patch asap.
thanks,
Shakeel
next prev parent reply other threads:[~2019-07-17 17:02 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-07-10 21:43 list corruption in deferred_split_scan() Qian Cai
2019-07-11 0:16 ` Yang Shi
2019-07-11 21:07 ` Qian Cai
2019-07-12 19:12 ` Yang Shi
2019-07-13 4:41 ` Yang Shi
2019-07-15 21:23 ` Qian Cai
2019-07-16 0:22 ` Yang Shi
2019-07-16 1:36 ` Qian Cai
2019-07-16 3:00 ` Yang Shi
2019-07-16 23:36 ` Shakeel Butt
2019-07-17 0:12 ` Yang Shi
2019-07-17 17:02 ` Shakeel Butt [this message]
2019-07-17 17:09 ` Yang Shi
2019-07-19 0:54 ` Qian Cai
2019-07-19 0:59 ` Yang Shi
2019-07-24 18:10 ` Qian Cai
2019-07-14 3:53 ` Hillf Danton
2019-07-15 4:52 ` Yang Shi
2019-07-24 21:13 ` Qian Cai
2019-07-25 21:46 ` Yang Shi
2019-08-05 22:15 ` Yang Shi
2019-08-06 1:05 ` Qian Cai
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CALvZod7Je+gekSGR61LMeHdYoC_PJune_0qGNiDfNH2=oNeOgw@mail.gmail.com' \
--to=shakeelb@google.com \
--cc=akpm@linux-foundation.org \
--cc=cai@lca.pw \
--cc=guro@fb.com \
--cc=hannes@cmpxchg.org \
--cc=hughd@google.com \
--cc=kirill.shutemov@linux.intel.com \
--cc=ktkhai@virtuozzo.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@suse.com \
--cc=vdavydov.dev@gmail.com \
--cc=yang.shi@linux.alibaba.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).