LKML Archive on lore.kernel.org
 help / Atom feed
* [PATCH] mm: memcg: fix use after free in mem_cgroup_iter()
@ 2018-07-19 10:06 Jing Xia
  2018-07-19 10:43 ` Michal Hocko
  0 siblings, 1 reply; 6+ messages in thread
From: Jing Xia @ 2018-07-19 10:06 UTC (permalink / raw)
  To: hannes, mhocko, vdavydov.dev
  Cc: chunyan.zhang, cgroups, linux-mm, linux-kernel

It was reported that a kernel crash happened in mem_cgroup_iter(),
which can be triggered if the legacy cgroup-v1 non-hierarchical
mode is used.

Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
......
Call trace:
  mem_cgroup_iter+0x2e0/0x6d4
  shrink_zone+0x8c/0x324
  balance_pgdat+0x450/0x640
  kswapd+0x130/0x4b8
  kthread+0xe8/0xfc
  ret_from_fork+0x10/0x20

  mem_cgroup_iter():
      ......
      if (css_tryget(css))    <-- crash here
	    break;
      ......

The crashing reason is that mem_cgroup_iter() uses the memcg object
whose pointer is stored in iter->position, which has been freed before
and filled with POISON_FREE(0x6b).

And the root cause of the use-after-free issue is that
invalidate_reclaim_iterators() fails to reset the value of
iter->position to NULL when the css of the memcg is released in non-
hierarchical mode.

Signed-off-by: Jing Xia <jing.xia.mail@gmail.com>
---
 mm/memcontrol.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index e6f0d5e..8c0280b 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -850,7 +850,7 @@ static void invalidate_reclaim_iterators(struct mem_cgroup *dead_memcg)
 	int nid;
 	int i;
 
-	while ((memcg = parent_mem_cgroup(memcg))) {
+	for (; memcg; memcg = parent_mem_cgroup(memcg)) {
 		for_each_node(nid) {
 			mz = mem_cgroup_nodeinfo(memcg, nid);
 			for (i = 0; i <= DEF_PRIORITY; i++) {
-- 
1.9.1


^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] mm: memcg: fix use after free in mem_cgroup_iter()
  2018-07-19 10:06 [PATCH] mm: memcg: fix use after free in mem_cgroup_iter() Jing Xia
@ 2018-07-19 10:43 ` Michal Hocko
  2018-07-19 16:23   ` Shakeel Butt
  0 siblings, 1 reply; 6+ messages in thread
From: Michal Hocko @ 2018-07-19 10:43 UTC (permalink / raw)
  To: Jing Xia
  Cc: hannes, vdavydov.dev, chunyan.zhang, cgroups, linux-mm,
	linux-kernel, Andrew Morton

[CC Andrew]

On Thu 19-07-18 18:06:47, Jing Xia wrote:
> It was reported that a kernel crash happened in mem_cgroup_iter(),
> which can be triggered if the legacy cgroup-v1 non-hierarchical
> mode is used.
> 
> Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
> ......
> Call trace:
>   mem_cgroup_iter+0x2e0/0x6d4
>   shrink_zone+0x8c/0x324
>   balance_pgdat+0x450/0x640
>   kswapd+0x130/0x4b8
>   kthread+0xe8/0xfc
>   ret_from_fork+0x10/0x20
> 
>   mem_cgroup_iter():
>       ......
>       if (css_tryget(css))    <-- crash here
> 	    break;
>       ......
> 
> The crashing reason is that mem_cgroup_iter() uses the memcg object
> whose pointer is stored in iter->position, which has been freed before
> and filled with POISON_FREE(0x6b).
> 
> And the root cause of the use-after-free issue is that
> invalidate_reclaim_iterators() fails to reset the value of
> iter->position to NULL when the css of the memcg is released in non-
> hierarchical mode.

Well, spotted!

I suspect
Fixes: 6df38689e0e9 ("mm: memcontrol: fix possible memcg leak due to interrupted reclaim")

but maybe it goes further into past. I also suggest
Cc: stable

even though the non-hierarchical mode is strongly discouraged. A lack of
reports for 3 years is encouraging that not many people really use this
mode.

> Signed-off-by: Jing Xia <jing.xia.mail@gmail.com>

Acked-by: Michal Hocko <mhocko@suse.com>

Thanks!
> ---
>  mm/memcontrol.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index e6f0d5e..8c0280b 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -850,7 +850,7 @@ static void invalidate_reclaim_iterators(struct mem_cgroup *dead_memcg)
>  	int nid;
>  	int i;
>  
> -	while ((memcg = parent_mem_cgroup(memcg))) {
> +	for (; memcg; memcg = parent_mem_cgroup(memcg)) {
>  		for_each_node(nid) {
>  			mz = mem_cgroup_nodeinfo(memcg, nid);
>  			for (i = 0; i <= DEF_PRIORITY; i++) {
> -- 
> 1.9.1

-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] mm: memcg: fix use after free in mem_cgroup_iter()
  2018-07-19 10:43 ` Michal Hocko
@ 2018-07-19 16:23   ` Shakeel Butt
  2018-07-23  6:44     ` Michal Hocko
  0 siblings, 1 reply; 6+ messages in thread
From: Shakeel Butt @ 2018-07-19 16:23 UTC (permalink / raw)
  To: Michal Hocko
  Cc: jing.xia.mail, Johannes Weiner, Vladimir Davydov, chunyan.zhang,
	Cgroups, Linux MM, LKML, Andrew Morton

On Thu, Jul 19, 2018 at 3:43 AM Michal Hocko <mhocko@kernel.org> wrote:
>
> [CC Andrew]
>
> On Thu 19-07-18 18:06:47, Jing Xia wrote:
> > It was reported that a kernel crash happened in mem_cgroup_iter(),
> > which can be triggered if the legacy cgroup-v1 non-hierarchical
> > mode is used.
> >
> > Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
> > ......
> > Call trace:
> >   mem_cgroup_iter+0x2e0/0x6d4
> >   shrink_zone+0x8c/0x324
> >   balance_pgdat+0x450/0x640
> >   kswapd+0x130/0x4b8
> >   kthread+0xe8/0xfc
> >   ret_from_fork+0x10/0x20
> >
> >   mem_cgroup_iter():
> >       ......
> >       if (css_tryget(css))    <-- crash here
> >           break;
> >       ......
> >
> > The crashing reason is that mem_cgroup_iter() uses the memcg object
> > whose pointer is stored in iter->position, which has been freed before
> > and filled with POISON_FREE(0x6b).
> >
> > And the root cause of the use-after-free issue is that
> > invalidate_reclaim_iterators() fails to reset the value of
> > iter->position to NULL when the css of the memcg is released in non-
> > hierarchical mode.
>
> Well, spotted!
>
> I suspect
> Fixes: 6df38689e0e9 ("mm: memcontrol: fix possible memcg leak due to interrupted reclaim")
>
> but maybe it goes further into past. I also suggest
> Cc: stable
>
> even though the non-hierarchical mode is strongly discouraged.

Why not set root_mem_cgroup's use_hierarchy to true by default on
init? If someone wants non-hierarchical mode, they can explicitly set
it to false.

> A lack of
> reports for 3 years is encouraging that not many people really use this
> mode.
>
> > Signed-off-by: Jing Xia <jing.xia.mail@gmail.com>
>
> Acked-by: Michal Hocko <mhocko@suse.com>
>
> Thanks!
> > ---
> >  mm/memcontrol.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index e6f0d5e..8c0280b 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -850,7 +850,7 @@ static void invalidate_reclaim_iterators(struct mem_cgroup *dead_memcg)
> >       int nid;
> >       int i;
> >
> > -     while ((memcg = parent_mem_cgroup(memcg))) {
> > +     for (; memcg; memcg = parent_mem_cgroup(memcg)) {
> >               for_each_node(nid) {
> >                       mz = mem_cgroup_nodeinfo(memcg, nid);
> >                       for (i = 0; i <= DEF_PRIORITY; i++) {
> > --
> > 1.9.1
>
> --
> Michal Hocko
> SUSE Labs
> --
> To unsubscribe from this list: send the line "unsubscribe cgroups" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] mm: memcg: fix use after free in mem_cgroup_iter()
  2018-07-19 16:23   ` Shakeel Butt
@ 2018-07-23  6:44     ` Michal Hocko
  2018-07-23 16:17       ` Shakeel Butt
  0 siblings, 1 reply; 6+ messages in thread
From: Michal Hocko @ 2018-07-23  6:44 UTC (permalink / raw)
  To: Shakeel Butt
  Cc: jing.xia.mail, Johannes Weiner, Vladimir Davydov, chunyan.zhang,
	Cgroups, Linux MM, LKML, Andrew Morton

On Thu 19-07-18 09:23:10, Shakeel Butt wrote:
> On Thu, Jul 19, 2018 at 3:43 AM Michal Hocko <mhocko@kernel.org> wrote:
> >
> > [CC Andrew]
> >
> > On Thu 19-07-18 18:06:47, Jing Xia wrote:
> > > It was reported that a kernel crash happened in mem_cgroup_iter(),
> > > which can be triggered if the legacy cgroup-v1 non-hierarchical
> > > mode is used.
> > >
> > > Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
> > > ......
> > > Call trace:
> > >   mem_cgroup_iter+0x2e0/0x6d4
> > >   shrink_zone+0x8c/0x324
> > >   balance_pgdat+0x450/0x640
> > >   kswapd+0x130/0x4b8
> > >   kthread+0xe8/0xfc
> > >   ret_from_fork+0x10/0x20
> > >
> > >   mem_cgroup_iter():
> > >       ......
> > >       if (css_tryget(css))    <-- crash here
> > >           break;
> > >       ......
> > >
> > > The crashing reason is that mem_cgroup_iter() uses the memcg object
> > > whose pointer is stored in iter->position, which has been freed before
> > > and filled with POISON_FREE(0x6b).
> > >
> > > And the root cause of the use-after-free issue is that
> > > invalidate_reclaim_iterators() fails to reset the value of
> > > iter->position to NULL when the css of the memcg is released in non-
> > > hierarchical mode.
> >
> > Well, spotted!
> >
> > I suspect
> > Fixes: 6df38689e0e9 ("mm: memcontrol: fix possible memcg leak due to interrupted reclaim")
> >
> > but maybe it goes further into past. I also suggest
> > Cc: stable
> >
> > even though the non-hierarchical mode is strongly discouraged.
> 
> Why not set root_mem_cgroup's use_hierarchy to true by default on
> init? If someone wants non-hierarchical mode, they can explicitly set
> it to false.

We do not change defaults under users feet usually.
-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] mm: memcg: fix use after free in mem_cgroup_iter()
  2018-07-23  6:44     ` Michal Hocko
@ 2018-07-23 16:17       ` Shakeel Butt
  2018-07-24  7:28         ` Michal Hocko
  0 siblings, 1 reply; 6+ messages in thread
From: Shakeel Butt @ 2018-07-23 16:17 UTC (permalink / raw)
  To: Michal Hocko
  Cc: jing.xia.mail, Johannes Weiner, Vladimir Davydov, chunyan.zhang,
	Cgroups, Linux MM, LKML, Andrew Morton

On Sun, Jul 22, 2018 at 11:44 PM Michal Hocko <mhocko@kernel.org> wrote:
>
> On Thu 19-07-18 09:23:10, Shakeel Butt wrote:
> > On Thu, Jul 19, 2018 at 3:43 AM Michal Hocko <mhocko@kernel.org> wrote:
> > >
> > > [CC Andrew]
> > >
> > > On Thu 19-07-18 18:06:47, Jing Xia wrote:
> > > > It was reported that a kernel crash happened in mem_cgroup_iter(),
> > > > which can be triggered if the legacy cgroup-v1 non-hierarchical
> > > > mode is used.
> > > >
> > > > Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
> > > > ......
> > > > Call trace:
> > > >   mem_cgroup_iter+0x2e0/0x6d4
> > > >   shrink_zone+0x8c/0x324
> > > >   balance_pgdat+0x450/0x640
> > > >   kswapd+0x130/0x4b8
> > > >   kthread+0xe8/0xfc
> > > >   ret_from_fork+0x10/0x20
> > > >
> > > >   mem_cgroup_iter():
> > > >       ......
> > > >       if (css_tryget(css))    <-- crash here
> > > >           break;
> > > >       ......
> > > >
> > > > The crashing reason is that mem_cgroup_iter() uses the memcg object
> > > > whose pointer is stored in iter->position, which has been freed before
> > > > and filled with POISON_FREE(0x6b).
> > > >
> > > > And the root cause of the use-after-free issue is that
> > > > invalidate_reclaim_iterators() fails to reset the value of
> > > > iter->position to NULL when the css of the memcg is released in non-
> > > > hierarchical mode.
> > >
> > > Well, spotted!
> > >
> > > I suspect
> > > Fixes: 6df38689e0e9 ("mm: memcontrol: fix possible memcg leak due to interrupted reclaim")
> > >
> > > but maybe it goes further into past. I also suggest
> > > Cc: stable
> > >
> > > even though the non-hierarchical mode is strongly discouraged.
> >
> > Why not set root_mem_cgroup's use_hierarchy to true by default on
> > init? If someone wants non-hierarchical mode, they can explicitly set
> > it to false.
>
> We do not change defaults under users feet usually.

Then how non-hierarchical mode is being discouraged currently? I don't
see any comments in the docs.

Shakeel

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] mm: memcg: fix use after free in mem_cgroup_iter()
  2018-07-23 16:17       ` Shakeel Butt
@ 2018-07-24  7:28         ` Michal Hocko
  0 siblings, 0 replies; 6+ messages in thread
From: Michal Hocko @ 2018-07-24  7:28 UTC (permalink / raw)
  To: Shakeel Butt
  Cc: jing.xia.mail, Johannes Weiner, Vladimir Davydov, chunyan.zhang,
	Cgroups, Linux MM, LKML, Andrew Morton

On Mon 23-07-18 09:17:28, Shakeel Butt wrote:
> On Sun, Jul 22, 2018 at 11:44 PM Michal Hocko <mhocko@kernel.org> wrote:
> >
> > On Thu 19-07-18 09:23:10, Shakeel Butt wrote:
> > > On Thu, Jul 19, 2018 at 3:43 AM Michal Hocko <mhocko@kernel.org> wrote:
> > > >
> > > > [CC Andrew]
> > > >
> > > > On Thu 19-07-18 18:06:47, Jing Xia wrote:
> > > > > It was reported that a kernel crash happened in mem_cgroup_iter(),
> > > > > which can be triggered if the legacy cgroup-v1 non-hierarchical
> > > > > mode is used.
> > > > >
> > > > > Unable to handle kernel paging request at virtual address 6b6b6b6b6b6b8f
> > > > > ......
> > > > > Call trace:
> > > > >   mem_cgroup_iter+0x2e0/0x6d4
> > > > >   shrink_zone+0x8c/0x324
> > > > >   balance_pgdat+0x450/0x640
> > > > >   kswapd+0x130/0x4b8
> > > > >   kthread+0xe8/0xfc
> > > > >   ret_from_fork+0x10/0x20
> > > > >
> > > > >   mem_cgroup_iter():
> > > > >       ......
> > > > >       if (css_tryget(css))    <-- crash here
> > > > >           break;
> > > > >       ......
> > > > >
> > > > > The crashing reason is that mem_cgroup_iter() uses the memcg object
> > > > > whose pointer is stored in iter->position, which has been freed before
> > > > > and filled with POISON_FREE(0x6b).
> > > > >
> > > > > And the root cause of the use-after-free issue is that
> > > > > invalidate_reclaim_iterators() fails to reset the value of
> > > > > iter->position to NULL when the css of the memcg is released in non-
> > > > > hierarchical mode.
> > > >
> > > > Well, spotted!
> > > >
> > > > I suspect
> > > > Fixes: 6df38689e0e9 ("mm: memcontrol: fix possible memcg leak due to interrupted reclaim")
> > > >
> > > > but maybe it goes further into past. I also suggest
> > > > Cc: stable
> > > >
> > > > even though the non-hierarchical mode is strongly discouraged.
> > >
> > > Why not set root_mem_cgroup's use_hierarchy to true by default on
> > > init? If someone wants non-hierarchical mode, they can explicitly set
> > > it to false.
> >
> > We do not change defaults under users feet usually.
> 
> Then how non-hierarchical mode is being discouraged currently? I don't
> see any comments in the docs.

css_create warns about non-hierarchical hierarchies. We've been running
with a similar warning in (even older) SLES kernels for years now and
quite some tools have been updated because they simply didn't know they
are doing something wrong.
-- 
Michal Hocko
SUSE Labs

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, back to index

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-07-19 10:06 [PATCH] mm: memcg: fix use after free in mem_cgroup_iter() Jing Xia
2018-07-19 10:43 ` Michal Hocko
2018-07-19 16:23   ` Shakeel Butt
2018-07-23  6:44     ` Michal Hocko
2018-07-23 16:17       ` Shakeel Butt
2018-07-24  7:28         ` Michal Hocko

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org linux-kernel@archiver.kernel.org
	public-inbox-index lkml


Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/ public-inbox