All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] mm, memcg: show memcg min setting in oom messages
@ 2019-11-20  8:53 Yafang Shao
  2019-11-20 10:21 ` Michal Hocko
  0 siblings, 1 reply; 10+ messages in thread
From: Yafang Shao @ 2019-11-20  8:53 UTC (permalink / raw)
  To: mhocko, hannes, vdavydov.dev, akpm; +Cc: linux-mm, Yafang Shao

A task running in a memcg may OOM because of the memory.min settings of his
slibing and parent. If this happens, the current oom messages can't show
why file page cache can't be reclaimed. So it is better to show the memcg
min settings.
Let's take an example.
      bar    bar/memory.max = 1200M memory.min=800M
     /  \
   barA barB barA/memory.min = 800M memory.current=1G (file page cache)
             barB/memory.min = 0 (process in this memcg is allocating page)

The process will do memcg reclaim if the bar/memory.max is reached. Once
the barA/memory.min is reached it will stop reclaiming file page caches in
barA, and if there is no reclaimable pages in bar and bar/barB it will
enter memcg OOM then.
After this pacch, bellow messages will be show then (only includeing the
relevant messages here). The lines begin with '#' are newly added info (the
'#' symbol is not in the original messages).
	memory: usage 1228800kB, limit 1228800kB, failcnt 18337
	...
	# Memory cgroup min setting:
	# /bar: min 819200KB emin 0KB
	# /bar/barA: min 819200KB emin 819200KB
	# /bar/barB: min 0KB emin 0KB
	...
	Memory cgroup stats for /bar:
	anon 418328576
	file 835756032
	...
	unevictable 0
	...
	oom-kill:constraint=CONSTRAINT_MEMCG..oom_memcg=/bar,task_memcg=/bar/barB

With the new added information, we can find the memory.min in bar/barA is
reached and the processes in bar/barB can't reclaim file page cache from
bar/barA any more. While without this new added information we don't know
why the file page cache in bar can't be reclaimed.

Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
---
 mm/memcontrol.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1c4c08b..97fdd93 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1519,16 +1519,27 @@ void mem_cgroup_print_oom_context(struct mem_cgroup *memcg, struct task_struct *
  */
 void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg)
 {
+	struct mem_cgroup *iter;
 	char *buf;
 
 	pr_info("memory: usage %llukB, limit %llukB, failcnt %lu\n",
 		K((u64)page_counter_read(&memcg->memory)),
 		K((u64)memcg->memory.max), memcg->memory.failcnt);
-	if (cgroup_subsys_on_dfl(memory_cgrp_subsys))
+	if (cgroup_subsys_on_dfl(memory_cgrp_subsys)) {
 		pr_info("swap: usage %llukB, limit %llukB, failcnt %lu\n",
 			K((u64)page_counter_read(&memcg->swap)),
 			K((u64)memcg->swap.max), memcg->swap.failcnt);
-	else {
+
+		pr_info("Memory cgroup min setting:\n");
+		for_each_mem_cgroup_tree(iter, memcg) {
+			pr_cont_cgroup_path(iter->css.cgroup);
+			pr_cont(":");
+
+			pr_cont(" min %lluKB emin %lluKB\n",
+				K((u64)iter->memory.min),
+				K((u64)iter->memory.emin));
+		}
+	} else {
 		pr_info("memory+swap: usage %llukB, limit %llukB, failcnt %lu\n",
 			K((u64)page_counter_read(&memcg->memsw)),
 			K((u64)memcg->memsw.max), memcg->memsw.failcnt);
-- 
1.8.3.1



^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [PATCH] mm, memcg: show memcg min setting in oom messages
  2019-11-20  8:53 [PATCH] mm, memcg: show memcg min setting in oom messages Yafang Shao
@ 2019-11-20 10:21 ` Michal Hocko
  2019-11-20 10:53   ` Yafang Shao
  0 siblings, 1 reply; 10+ messages in thread
From: Michal Hocko @ 2019-11-20 10:21 UTC (permalink / raw)
  To: Yafang Shao; +Cc: hannes, vdavydov.dev, akpm, linux-mm

On Wed 20-11-19 03:53:05, Yafang Shao wrote:
> A task running in a memcg may OOM because of the memory.min settings of his
> slibing and parent. If this happens, the current oom messages can't show
> why file page cache can't be reclaimed.

min limit is not the only way to protect memory from being reclaim. The
memory might be pinned or unreclaimable for other reasons (e.g. swap
quota exceeded for memcg). Besides that, there is the very same problem
with the global OOM killer, right? And I do not expect we want to print
all memcgs in the system (this might be hundreds).

> So it is better to show the memcg
> min settings.
> Let's take an example.
>       bar    bar/memory.max = 1200M memory.min=800M
>      /  \
>    barA barB barA/memory.min = 800M memory.current=1G (file page cache)
>              barB/memory.min = 0 (process in this memcg is allocating page)
> 
> The process will do memcg reclaim if the bar/memory.max is reached. Once
> the barA/memory.min is reached it will stop reclaiming file page caches in
> barA, and if there is no reclaimable pages in bar and bar/barB it will
> enter memcg OOM then.
> After this pacch, bellow messages will be show then (only includeing the
> relevant messages here). The lines begin with '#' are newly added info (the
> '#' symbol is not in the original messages).
> 	memory: usage 1228800kB, limit 1228800kB, failcnt 18337
> 	...
> 	# Memory cgroup min setting:
> 	# /bar: min 819200KB emin 0KB
> 	# /bar/barA: min 819200KB emin 819200KB
> 	# /bar/barB: min 0KB emin 0KB
> 	...
> 	Memory cgroup stats for /bar:
> 	anon 418328576
> 	file 835756032
> 	...
> 	unevictable 0
> 	...
> 	oom-kill:constraint=CONSTRAINT_MEMCG..oom_memcg=/bar,task_memcg=/bar/barB
> 
> With the new added information, we can find the memory.min in bar/barA is
> reached and the processes in bar/barB can't reclaim file page cache from
> bar/barA any more. While without this new added information we don't know
> why the file page cache in bar can't be reclaimed.

Well, I am not sure this is really usefull enough TBH. It doesn't give
you the whole picture and it potentially generates a lot of output in
the oom report. FYI we used to have a more precise break down of
counters in memcg hierarchy, see 58cf188ed649 ("memcg, oom: provide more
precise dump info while memcg oom happening") which later got rewritten
by c8713d0b2312 ("mm: memcontrol: dump memory.stat during cgroup OOM")

Could you be more specific why do you really need this piece of
information?

> Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
> ---
>  mm/memcontrol.c | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] mm, memcg: show memcg min setting in oom messages
  2019-11-20 10:21 ` Michal Hocko
@ 2019-11-20 10:53   ` Yafang Shao
  2019-11-20 11:40     ` Michal Hocko
  0 siblings, 1 reply; 10+ messages in thread
From: Yafang Shao @ 2019-11-20 10:53 UTC (permalink / raw)
  To: Michal Hocko; +Cc: Johannes Weiner, Vladimir Davydov, Andrew Morton, Linux MM

On Wed, Nov 20, 2019 at 6:22 PM Michal Hocko <mhocko@kernel.org> wrote:
>
> On Wed 20-11-19 03:53:05, Yafang Shao wrote:
> > A task running in a memcg may OOM because of the memory.min settings of his
> > slibing and parent. If this happens, the current oom messages can't show
> > why file page cache can't be reclaimed.
>
> min limit is not the only way to protect memory from being reclaim. The
> memory might be pinned or unreclaimable for other reasons (e.g. swap
> quota exceeded for memcg).

Both swap or unreclaimabed (unevicteable) is printed in OOM messages.
If something else can prevent the file cache being reclaimed, we'd
better show them as well.

> Besides that, there is the very same problem
> with the global OOM killer, right? And I do not expect we want to print
> all memcgs in the system (this might be hundreds).
>

I forgot the global oom...

Why not just print the memcgs which are under memory.min protection or
something like a total number of min protected memory ?

> > So it is better to show the memcg
> > min settings.
> > Let's take an example.
> >       bar    bar/memory.max = 1200M memory.min=800M
> >      /  \
> >    barA barB barA/memory.min = 800M memory.current=1G (file page cache)
> >              barB/memory.min = 0 (process in this memcg is allocating page)
> >
> > The process will do memcg reclaim if the bar/memory.max is reached. Once
> > the barA/memory.min is reached it will stop reclaiming file page caches in
> > barA, and if there is no reclaimable pages in bar and bar/barB it will
> > enter memcg OOM then.
> > After this pacch, bellow messages will be show then (only includeing the
> > relevant messages here). The lines begin with '#' are newly added info (the
> > '#' symbol is not in the original messages).
> >       memory: usage 1228800kB, limit 1228800kB, failcnt 18337
> >       ...
> >       # Memory cgroup min setting:
> >       # /bar: min 819200KB emin 0KB
> >       # /bar/barA: min 819200KB emin 819200KB
> >       # /bar/barB: min 0KB emin 0KB
> >       ...
> >       Memory cgroup stats for /bar:
> >       anon 418328576
> >       file 835756032
> >       ...
> >       unevictable 0
> >       ...
> >       oom-kill:constraint=CONSTRAINT_MEMCG..oom_memcg=/bar,task_memcg=/bar/barB
> >
> > With the new added information, we can find the memory.min in bar/barA is
> > reached and the processes in bar/barB can't reclaim file page cache from
> > bar/barA any more. While without this new added information we don't know
> > why the file page cache in bar can't be reclaimed.
>
> Well, I am not sure this is really usefull enough TBH. It doesn't give
> you the whole picture and it potentially generates a lot of output in
> the oom report. FYI we used to have a more precise break down of
> counters in memcg hierarchy, see 58cf188ed649 ("memcg, oom: provide more
> precise dump info while memcg oom happening") which later got rewritten
> by c8713d0b2312 ("mm: memcontrol: dump memory.stat during cgroup OOM")
>

At least we'd better print a total protected memory in the oom messages.

> Could you be more specific why do you really need this piece of
> information?

I have said in the commit log, that we don't know why the file cache
can't be reclaimed (when evictable is 0 and dirty is 0 as well.)

>
> > Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
> > ---
> >  mm/memcontrol.c | 15 +++++++++++++--
> >  1 file changed, 13 insertions(+), 2 deletions(-)
> --
> Michal Hocko
> SUSE Labs


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] mm, memcg: show memcg min setting in oom messages
  2019-11-20 10:53   ` Yafang Shao
@ 2019-11-20 11:40     ` Michal Hocko
  2019-11-20 12:23       ` Yafang Shao
  0 siblings, 1 reply; 10+ messages in thread
From: Michal Hocko @ 2019-11-20 11:40 UTC (permalink / raw)
  To: Yafang Shao; +Cc: Johannes Weiner, Vladimir Davydov, Andrew Morton, Linux MM

On Wed 20-11-19 18:53:44, Yafang Shao wrote:
> On Wed, Nov 20, 2019 at 6:22 PM Michal Hocko <mhocko@kernel.org> wrote:
> >
> > On Wed 20-11-19 03:53:05, Yafang Shao wrote:
> > > A task running in a memcg may OOM because of the memory.min settings of his
> > > slibing and parent. If this happens, the current oom messages can't show
> > > why file page cache can't be reclaimed.
> >
> > min limit is not the only way to protect memory from being reclaim. The
> > memory might be pinned or unreclaimable for other reasons (e.g. swap
> > quota exceeded for memcg).
> 
> Both swap or unreclaimabed (unevicteable) is printed in OOM messages.

Not really. Consider a memcg which has reached it's swap limit. The
anonymous memory is not really reclaimable even when there is a lot of
swap space available.

> If something else can prevent the file cache being reclaimed, we'd
> better show them as well.

How are you going to do that? How do you track pins on pages?

> > Besides that, there is the very same problem
> > with the global OOM killer, right? And I do not expect we want to print
> > all memcgs in the system (this might be hundreds).
> >
> 
> I forgot the global oom...
> 
> Why not just print the memcgs which are under memory.min protection or
> something like a total number of min protected memory ?

Yes, this would likely help. But the main question really reamains, is
this really worth it?

> > > So it is better to show the memcg
> > > min settings.
> > > Let's take an example.
> > >       bar    bar/memory.max = 1200M memory.min=800M
> > >      /  \
> > >    barA barB barA/memory.min = 800M memory.current=1G (file page cache)
> > >              barB/memory.min = 0 (process in this memcg is allocating page)
> > >
> > > The process will do memcg reclaim if the bar/memory.max is reached. Once
> > > the barA/memory.min is reached it will stop reclaiming file page caches in
> > > barA, and if there is no reclaimable pages in bar and bar/barB it will
> > > enter memcg OOM then.
> > > After this pacch, bellow messages will be show then (only includeing the
> > > relevant messages here). The lines begin with '#' are newly added info (the
> > > '#' symbol is not in the original messages).
> > >       memory: usage 1228800kB, limit 1228800kB, failcnt 18337
> > >       ...
> > >       # Memory cgroup min setting:
> > >       # /bar: min 819200KB emin 0KB
> > >       # /bar/barA: min 819200KB emin 819200KB
> > >       # /bar/barB: min 0KB emin 0KB
> > >       ...
> > >       Memory cgroup stats for /bar:
> > >       anon 418328576
> > >       file 835756032
> > >       ...
> > >       unevictable 0
> > >       ...
> > >       oom-kill:constraint=CONSTRAINT_MEMCG..oom_memcg=/bar,task_memcg=/bar/barB
> > >
> > > With the new added information, we can find the memory.min in bar/barA is
> > > reached and the processes in bar/barB can't reclaim file page cache from
> > > bar/barA any more. While without this new added information we don't know
> > > why the file page cache in bar can't be reclaimed.
> >
> > Well, I am not sure this is really usefull enough TBH. It doesn't give
> > you the whole picture and it potentially generates a lot of output in
> > the oom report. FYI we used to have a more precise break down of
> > counters in memcg hierarchy, see 58cf188ed649 ("memcg, oom: provide more
> > precise dump info while memcg oom happening") which later got rewritten
> > by c8713d0b2312 ("mm: memcontrol: dump memory.stat during cgroup OOM")
> >
> 
> At least we'd better print a total protected memory in the oom messages.
> 
> > Could you be more specific why do you really need this piece of
> > information?
> 
> I have said in the commit log, that we don't know why the file cache
> can't be reclaimed (when evictable is 0 and dirty is 0 as well.)

And the counter argument is that this will not help you there much in
many large and much more common cases.

I argue, and I might be wrong here so feel free to correct me, that the
reclaim protection guarantee (min) is something to be under admins
control. It shouldn't really happen nilly-willy because it has really
large consequences, the OOM including. So if there is a suspicious
amount of memory that could be reclaimed normally then the reclaim
protection is really the first suspect to go after.
-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] mm, memcg: show memcg min setting in oom messages
  2019-11-20 11:40     ` Michal Hocko
@ 2019-11-20 12:23       ` Yafang Shao
  2019-11-22 10:28         ` Michal Hocko
  0 siblings, 1 reply; 10+ messages in thread
From: Yafang Shao @ 2019-11-20 12:23 UTC (permalink / raw)
  To: Michal Hocko; +Cc: Johannes Weiner, Vladimir Davydov, Andrew Morton, Linux MM

On Wed, Nov 20, 2019 at 7:40 PM Michal Hocko <mhocko@kernel.org> wrote:
>
> On Wed 20-11-19 18:53:44, Yafang Shao wrote:
> > On Wed, Nov 20, 2019 at 6:22 PM Michal Hocko <mhocko@kernel.org> wrote:
> > >
> > > On Wed 20-11-19 03:53:05, Yafang Shao wrote:
> > > > A task running in a memcg may OOM because of the memory.min settings of his
> > > > slibing and parent. If this happens, the current oom messages can't show
> > > > why file page cache can't be reclaimed.
> > >
> > > min limit is not the only way to protect memory from being reclaim. The
> > > memory might be pinned or unreclaimable for other reasons (e.g. swap
> > > quota exceeded for memcg).
> >
> > Both swap or unreclaimabed (unevicteable) is printed in OOM messages.
>
> Not really. Consider a memcg which has reached it's swap limit. The
> anonymous memory is not really reclaimable even when there is a lot of
> swap space available.
>

The memcg swap limit is already printed in oom messages, see bellow,

[  141.721625] memory: usage 1228800kB, limit 1228800kB, failcnt 18337
[  141.721958] swap: usage 0kB, limit 9007199254740988kB, failcnt 0

> > If something else can prevent the file cache being reclaimed, we'd
> > better show them as well.
>
> How are you going to do that? How do you track pins on pages?
>

Actually I don't have a clear idea how to track pins on pages yet.


> > > Besides that, there is the very same problem
> > > with the global OOM killer, right? And I do not expect we want to print
> > > all memcgs in the system (this might be hundreds).
> > >
> >
> > I forgot the global oom...
> >
> > Why not just print the memcgs which are under memory.min protection or
> > something like a total number of min protected memory ?
>
> Yes, this would likely help. But the main question really reamains, is
> this really worth it?
>

If it doesn't cost too much, I think it is worth to do it.
As the oom path is not the critical path, so adding some print info
should not add much overhead.

> > > > So it is better to show the memcg
> > > > min settings.
> > > > Let's take an example.
> > > >       bar    bar/memory.max = 1200M memory.min=800M
> > > >      /  \
> > > >    barA barB barA/memory.min = 800M memory.current=1G (file page cache)
> > > >              barB/memory.min = 0 (process in this memcg is allocating page)
> > > >
> > > > The process will do memcg reclaim if the bar/memory.max is reached. Once
> > > > the barA/memory.min is reached it will stop reclaiming file page caches in
> > > > barA, and if there is no reclaimable pages in bar and bar/barB it will
> > > > enter memcg OOM then.
> > > > After this pacch, bellow messages will be show then (only includeing the
> > > > relevant messages here). The lines begin with '#' are newly added info (the
> > > > '#' symbol is not in the original messages).
> > > >       memory: usage 1228800kB, limit 1228800kB, failcnt 18337
> > > >       ...
> > > >       # Memory cgroup min setting:
> > > >       # /bar: min 819200KB emin 0KB
> > > >       # /bar/barA: min 819200KB emin 819200KB
> > > >       # /bar/barB: min 0KB emin 0KB
> > > >       ...
> > > >       Memory cgroup stats for /bar:
> > > >       anon 418328576
> > > >       file 835756032
> > > >       ...
> > > >       unevictable 0
> > > >       ...
> > > >       oom-kill:constraint=CONSTRAINT_MEMCG..oom_memcg=/bar,task_memcg=/bar/barB
> > > >
> > > > With the new added information, we can find the memory.min in bar/barA is
> > > > reached and the processes in bar/barB can't reclaim file page cache from
> > > > bar/barA any more. While without this new added information we don't know
> > > > why the file page cache in bar can't be reclaimed.
> > >
> > > Well, I am not sure this is really usefull enough TBH. It doesn't give
> > > you the whole picture and it potentially generates a lot of output in
> > > the oom report. FYI we used to have a more precise break down of
> > > counters in memcg hierarchy, see 58cf188ed649 ("memcg, oom: provide more
> > > precise dump info while memcg oom happening") which later got rewritten
> > > by c8713d0b2312 ("mm: memcontrol: dump memory.stat during cgroup OOM")
> > >
> >
> > At least we'd better print a total protected memory in the oom messages.
> >
> > > Could you be more specific why do you really need this piece of
> > > information?
> >
> > I have said in the commit log, that we don't know why the file cache
> > can't be reclaimed (when evictable is 0 and dirty is 0 as well.)
>
> And the counter argument is that this will not help you there much in
> many large and much more common cases.
>
> I argue, and I might be wrong here so feel free to correct me, that the
> reclaim protection guarantee (min) is something to be under admins
> control. It shouldn't really happen nilly-willy because it has really
> large consequences, the OOM including. So if there is a suspicious
> amount of memory that could be reclaimed normally then the reclaim
> protection is really the first suspect to go after.
> --

I don't know whether it happens nilly-willy or not.
But if we all know that it may cause OOMs and it don't take too much
effort to show it in the OOM messages,
we'd better show it. That can help the admins and the admins don't
need to suspect it any more.

Thanks
Yafang


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] mm, memcg: show memcg min setting in oom messages
  2019-11-20 12:23       ` Yafang Shao
@ 2019-11-22 10:28         ` Michal Hocko
  2019-11-23  5:52           ` Yafang Shao
  0 siblings, 1 reply; 10+ messages in thread
From: Michal Hocko @ 2019-11-22 10:28 UTC (permalink / raw)
  To: Yafang Shao; +Cc: Johannes Weiner, Vladimir Davydov, Andrew Morton, Linux MM

On Wed 20-11-19 20:23:54, Yafang Shao wrote:
> On Wed, Nov 20, 2019 at 7:40 PM Michal Hocko <mhocko@kernel.org> wrote:
> >
> > On Wed 20-11-19 18:53:44, Yafang Shao wrote:
> > > On Wed, Nov 20, 2019 at 6:22 PM Michal Hocko <mhocko@kernel.org> wrote:
> > > >
> > > > On Wed 20-11-19 03:53:05, Yafang Shao wrote:
> > > > > A task running in a memcg may OOM because of the memory.min settings of his
> > > > > slibing and parent. If this happens, the current oom messages can't show
> > > > > why file page cache can't be reclaimed.
> > > >
> > > > min limit is not the only way to protect memory from being reclaim. The
> > > > memory might be pinned or unreclaimable for other reasons (e.g. swap
> > > > quota exceeded for memcg).
> > >
> > > Both swap or unreclaimabed (unevicteable) is printed in OOM messages.
> >
> > Not really. Consider a memcg which has reached it's swap limit. The
> > anonymous memory is not really reclaimable even when there is a lot of
> > swap space available.
> >
> 
> The memcg swap limit is already printed in oom messages, see bellow,
> 
> [  141.721625] memory: usage 1228800kB, limit 1228800kB, failcnt 18337
> [  141.721958] swap: usage 0kB, limit 9007199254740988kB, failcnt 0

But you do not have any insight on the swap limit down the oom
hierarchy, do you?

> > > Why not just print the memcgs which are under memory.min protection or
> > > something like a total number of min protected memory ?
> >
> > Yes, this would likely help. But the main question really reamains, is
> > this really worth it?
> >
> 
> If it doesn't cost too much, I think it is worth to do it.
> As the oom path is not the critical path, so adding some print info
> should not add much overhead.

Generating a lot of output for the oom reports has been a real problem
in many deployments.
[...]
> > > I have said in the commit log, that we don't know why the file cache
> > > can't be reclaimed (when evictable is 0 and dirty is 0 as well.)
> >
> > And the counter argument is that this will not help you there much in
> > many large and much more common cases.
> >
> > I argue, and I might be wrong here so feel free to correct me, that the
> > reclaim protection guarantee (min) is something to be under admins
> > control. It shouldn't really happen nilly-willy because it has really
> > large consequences, the OOM including. So if there is a suspicious
> > amount of memory that could be reclaimed normally then the reclaim
> > protection is really the first suspect to go after.
> > --
> 
> I don't know whether it happens nilly-willy or not.

It is a reclaim protection guarantee (so essentially an mlock like
thing) so it better have to be properly considered when used.

> But if we all know that it may cause OOMs and it don't take too much
> effort to show it in the OOM messages,

I do not think we are in agreement here. As mentioned above the oom
report is quite heavy already. So it should be other way around. There
should be a strong reason to add something more. A real use case where
not having that information is making debugging ooms considerably much
harder.

-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] mm, memcg: show memcg min setting in oom messages
  2019-11-22 10:28         ` Michal Hocko
@ 2019-11-23  5:52           ` Yafang Shao
  2019-11-25  8:20             ` Michal Hocko
  0 siblings, 1 reply; 10+ messages in thread
From: Yafang Shao @ 2019-11-23  5:52 UTC (permalink / raw)
  To: Michal Hocko; +Cc: Johannes Weiner, Vladimir Davydov, Andrew Morton, Linux MM

On Fri, Nov 22, 2019 at 6:28 PM Michal Hocko <mhocko@kernel.org> wrote:
>
> On Wed 20-11-19 20:23:54, Yafang Shao wrote:
> > On Wed, Nov 20, 2019 at 7:40 PM Michal Hocko <mhocko@kernel.org> wrote:
> > >
> > > On Wed 20-11-19 18:53:44, Yafang Shao wrote:
> > > > On Wed, Nov 20, 2019 at 6:22 PM Michal Hocko <mhocko@kernel.org> wrote:
> > > > >
> > > > > On Wed 20-11-19 03:53:05, Yafang Shao wrote:
> > > > > > A task running in a memcg may OOM because of the memory.min settings of his
> > > > > > slibing and parent. If this happens, the current oom messages can't show
> > > > > > why file page cache can't be reclaimed.
> > > > >
> > > > > min limit is not the only way to protect memory from being reclaim. The
> > > > > memory might be pinned or unreclaimable for other reasons (e.g. swap
> > > > > quota exceeded for memcg).
> > > >
> > > > Both swap or unreclaimabed (unevicteable) is printed in OOM messages.
> > >
> > > Not really. Consider a memcg which has reached it's swap limit. The
> > > anonymous memory is not really reclaimable even when there is a lot of
> > > swap space available.
> > >
> >
> > The memcg swap limit is already printed in oom messages, see bellow,
> >
> > [  141.721625] memory: usage 1228800kB, limit 1228800kB, failcnt 18337
> > [  141.721958] swap: usage 0kB, limit 9007199254740988kB, failcnt 0
>
> But you do not have any insight on the swap limit down the oom
> hierarchy, do you?
>
> > > > Why not just print the memcgs which are under memory.min protection or
> > > > something like a total number of min protected memory ?
> > >
> > > Yes, this would likely help. But the main question really reamains, is
> > > this really worth it?
> > >
> >
> > If it doesn't cost too much, I think it is worth to do it.
> > As the oom path is not the critical path, so adding some print info
> > should not add much overhead.
>
> Generating a lot of output for the oom reports has been a real problem
> in many deployments.

So why not only print non-zero counters ?
If some counters are 0, we don't print them, that can reduce the oom reports.

Something like "isolated_file:0 unevictable:0 dirty:0 writeback:0
unstable:0" can all be removed,
and we consider them as zero by default.
I mean we can optimze the OOM reports and only print the useful
information to make it not be a problem in many deployments.

> [...]
> > > > I have said in the commit log, that we don't know why the file cache
> > > > can't be reclaimed (when evictable is 0 and dirty is 0 as well.)
> > >
> > > And the counter argument is that this will not help you there much in
> > > many large and much more common cases.
> > >
> > > I argue, and I might be wrong here so feel free to correct me, that the
> > > reclaim protection guarantee (min) is something to be under admins
> > > control. It shouldn't really happen nilly-willy because it has really
> > > large consequences, the OOM including. So if there is a suspicious
> > > amount of memory that could be reclaimed normally then the reclaim
> > > protection is really the first suspect to go after.
> > > --
> >
> > I don't know whether it happens nilly-willy or not.
>
> It is a reclaim protection guarantee (so essentially an mlock like
> thing) so it better have to be properly considered when used.
>
> > But if we all know that it may cause OOMs and it don't take too much
> > effort to show it in the OOM messages,
>
> I do not think we are in agreement here. As mentioned above the oom
> report is quite heavy already. So it should be other way around. There
> should be a strong reason to add something more. A real use case where
> not having that information is making debugging ooms considerably much
> harder.
>
> --
> Michal Hocko
> SUSE Labs


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] mm, memcg: show memcg min setting in oom messages
  2019-11-23  5:52           ` Yafang Shao
@ 2019-11-25  8:20             ` Michal Hocko
  2019-11-25  9:12               ` Yafang Shao
  0 siblings, 1 reply; 10+ messages in thread
From: Michal Hocko @ 2019-11-25  8:20 UTC (permalink / raw)
  To: Yafang Shao; +Cc: Johannes Weiner, Vladimir Davydov, Andrew Morton, Linux MM

On Sat 23-11-19 13:52:57, Yafang Shao wrote:
> On Fri, Nov 22, 2019 at 6:28 PM Michal Hocko <mhocko@kernel.org> wrote:
> >
> > On Wed 20-11-19 20:23:54, Yafang Shao wrote:
> > > On Wed, Nov 20, 2019 at 7:40 PM Michal Hocko <mhocko@kernel.org> wrote:
> > > >
> > > > On Wed 20-11-19 18:53:44, Yafang Shao wrote:
> > > > > On Wed, Nov 20, 2019 at 6:22 PM Michal Hocko <mhocko@kernel.org> wrote:
> > > > > >
> > > > > > On Wed 20-11-19 03:53:05, Yafang Shao wrote:
> > > > > > > A task running in a memcg may OOM because of the memory.min settings of his
> > > > > > > slibing and parent. If this happens, the current oom messages can't show
> > > > > > > why file page cache can't be reclaimed.
> > > > > >
> > > > > > min limit is not the only way to protect memory from being reclaim. The
> > > > > > memory might be pinned or unreclaimable for other reasons (e.g. swap
> > > > > > quota exceeded for memcg).
> > > > >
> > > > > Both swap or unreclaimabed (unevicteable) is printed in OOM messages.
> > > >
> > > > Not really. Consider a memcg which has reached it's swap limit. The
> > > > anonymous memory is not really reclaimable even when there is a lot of
> > > > swap space available.
> > > >
> > >
> > > The memcg swap limit is already printed in oom messages, see bellow,
> > >
> > > [  141.721625] memory: usage 1228800kB, limit 1228800kB, failcnt 18337
> > > [  141.721958] swap: usage 0kB, limit 9007199254740988kB, failcnt 0
> >
> > But you do not have any insight on the swap limit down the oom
> > hierarchy, do you?
> >
> > > > > Why not just print the memcgs which are under memory.min protection or
> > > > > something like a total number of min protected memory ?
> > > >
> > > > Yes, this would likely help. But the main question really reamains, is
> > > > this really worth it?
> > > >
> > >
> > > If it doesn't cost too much, I think it is worth to do it.
> > > As the oom path is not the critical path, so adding some print info
> > > should not add much overhead.
> >
> > Generating a lot of output for the oom reports has been a real problem
> > in many deployments.
> 
> So why not only print non-zero counters ?
> If some counters are 0, we don't print them, that can reduce the oom reports.
> 
> Something like "isolated_file:0 unevictable:0 dirty:0 writeback:0
> unstable:0" can all be removed,
> and we consider them as zero by default.

because that would make parsing more complex.

> I mean we can optimze the OOM reports and only print the useful
> information to make it not be a problem in many deployments.

We can, but it would be great to have it backed by som real usecase to
change the current behavior. I haven't heard anything so far. It is all
about "this would be nice" without a strong justification.
-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] mm, memcg: show memcg min setting in oom messages
  2019-11-25  8:20             ` Michal Hocko
@ 2019-11-25  9:12               ` Yafang Shao
  2019-11-25  9:27                 ` Michal Hocko
  0 siblings, 1 reply; 10+ messages in thread
From: Yafang Shao @ 2019-11-25  9:12 UTC (permalink / raw)
  To: Michal Hocko; +Cc: Johannes Weiner, Vladimir Davydov, Andrew Morton, Linux MM

On Mon, Nov 25, 2019 at 4:20 PM Michal Hocko <mhocko@kernel.org> wrote:
>
> On Sat 23-11-19 13:52:57, Yafang Shao wrote:
> > On Fri, Nov 22, 2019 at 6:28 PM Michal Hocko <mhocko@kernel.org> wrote:
> > >
> > > On Wed 20-11-19 20:23:54, Yafang Shao wrote:
> > > > On Wed, Nov 20, 2019 at 7:40 PM Michal Hocko <mhocko@kernel.org> wrote:
> > > > >
> > > > > On Wed 20-11-19 18:53:44, Yafang Shao wrote:
> > > > > > On Wed, Nov 20, 2019 at 6:22 PM Michal Hocko <mhocko@kernel.org> wrote:
> > > > > > >
> > > > > > > On Wed 20-11-19 03:53:05, Yafang Shao wrote:
> > > > > > > > A task running in a memcg may OOM because of the memory.min settings of his
> > > > > > > > slibing and parent. If this happens, the current oom messages can't show
> > > > > > > > why file page cache can't be reclaimed.
> > > > > > >
> > > > > > > min limit is not the only way to protect memory from being reclaim. The
> > > > > > > memory might be pinned or unreclaimable for other reasons (e.g. swap
> > > > > > > quota exceeded for memcg).
> > > > > >
> > > > > > Both swap or unreclaimabed (unevicteable) is printed in OOM messages.
> > > > >
> > > > > Not really. Consider a memcg which has reached it's swap limit. The
> > > > > anonymous memory is not really reclaimable even when there is a lot of
> > > > > swap space available.
> > > > >
> > > >
> > > > The memcg swap limit is already printed in oom messages, see bellow,
> > > >
> > > > [  141.721625] memory: usage 1228800kB, limit 1228800kB, failcnt 18337
> > > > [  141.721958] swap: usage 0kB, limit 9007199254740988kB, failcnt 0
> > >
> > > But you do not have any insight on the swap limit down the oom
> > > hierarchy, do you?
> > >
> > > > > > Why not just print the memcgs which are under memory.min protection or
> > > > > > something like a total number of min protected memory ?
> > > > >
> > > > > Yes, this would likely help. But the main question really reamains, is
> > > > > this really worth it?
> > > > >
> > > >
> > > > If it doesn't cost too much, I think it is worth to do it.
> > > > As the oom path is not the critical path, so adding some print info
> > > > should not add much overhead.
> > >
> > > Generating a lot of output for the oom reports has been a real problem
> > > in many deployments.
> >
> > So why not only print non-zero counters ?
> > If some counters are 0, we don't print them, that can reduce the oom reports.
> >
> > Something like "isolated_file:0 unevictable:0 dirty:0 writeback:0
> > unstable:0" can all be removed,
> > and we consider them as zero by default.
>
> because that would make parsing more complex.
>
> > I mean we can optimze the OOM reports and only print the useful
> > information to make it not be a problem in many deployments.
>
> We can, but it would be great to have it backed by som real usecase to
> change the current behavior. I haven't heard anything so far. It is all
> about "this would be nice" without a strong justification.

Because I was told by you that "Generating a lot of output for the oom
reports has been a real problem in many deployments.".
Maybe I misunderstood you : (

Thanks
Yafang


^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] mm, memcg: show memcg min setting in oom messages
  2019-11-25  9:12               ` Yafang Shao
@ 2019-11-25  9:27                 ` Michal Hocko
  0 siblings, 0 replies; 10+ messages in thread
From: Michal Hocko @ 2019-11-25  9:27 UTC (permalink / raw)
  To: Yafang Shao; +Cc: Johannes Weiner, Vladimir Davydov, Andrew Morton, Linux MM

On Mon 25-11-19 17:12:54, Yafang Shao wrote:
> On Mon, Nov 25, 2019 at 4:20 PM Michal Hocko <mhocko@kernel.org> wrote:
> >
> > On Sat 23-11-19 13:52:57, Yafang Shao wrote:
> > > On Fri, Nov 22, 2019 at 6:28 PM Michal Hocko <mhocko@kernel.org> wrote:
> > > >
> > > > On Wed 20-11-19 20:23:54, Yafang Shao wrote:
> > > > > On Wed, Nov 20, 2019 at 7:40 PM Michal Hocko <mhocko@kernel.org> wrote:
> > > > > >
> > > > > > On Wed 20-11-19 18:53:44, Yafang Shao wrote:
> > > > > > > On Wed, Nov 20, 2019 at 6:22 PM Michal Hocko <mhocko@kernel.org> wrote:
> > > > > > > >
> > > > > > > > On Wed 20-11-19 03:53:05, Yafang Shao wrote:
> > > > > > > > > A task running in a memcg may OOM because of the memory.min settings of his
> > > > > > > > > slibing and parent. If this happens, the current oom messages can't show
> > > > > > > > > why file page cache can't be reclaimed.
> > > > > > > >
> > > > > > > > min limit is not the only way to protect memory from being reclaim. The
> > > > > > > > memory might be pinned or unreclaimable for other reasons (e.g. swap
> > > > > > > > quota exceeded for memcg).
> > > > > > >
> > > > > > > Both swap or unreclaimabed (unevicteable) is printed in OOM messages.
> > > > > >
> > > > > > Not really. Consider a memcg which has reached it's swap limit. The
> > > > > > anonymous memory is not really reclaimable even when there is a lot of
> > > > > > swap space available.
> > > > > >
> > > > >
> > > > > The memcg swap limit is already printed in oom messages, see bellow,
> > > > >
> > > > > [  141.721625] memory: usage 1228800kB, limit 1228800kB, failcnt 18337
> > > > > [  141.721958] swap: usage 0kB, limit 9007199254740988kB, failcnt 0
> > > >
> > > > But you do not have any insight on the swap limit down the oom
> > > > hierarchy, do you?
> > > >
> > > > > > > Why not just print the memcgs which are under memory.min protection or
> > > > > > > something like a total number of min protected memory ?
> > > > > >
> > > > > > Yes, this would likely help. But the main question really reamains, is
> > > > > > this really worth it?
> > > > > >
> > > > >
> > > > > If it doesn't cost too much, I think it is worth to do it.
> > > > > As the oom path is not the critical path, so adding some print info
> > > > > should not add much overhead.
> > > >
> > > > Generating a lot of output for the oom reports has been a real problem
> > > > in many deployments.
> > >
> > > So why not only print non-zero counters ?
> > > If some counters are 0, we don't print them, that can reduce the oom reports.
> > >
> > > Something like "isolated_file:0 unevictable:0 dirty:0 writeback:0
> > > unstable:0" can all be removed,
> > > and we consider them as zero by default.
> >
> > because that would make parsing more complex.
> >
> > > I mean we can optimze the OOM reports and only print the useful
> > > information to make it not be a problem in many deployments.
> >
> > We can, but it would be great to have it backed by som real usecase to
> > change the current behavior. I haven't heard anything so far. It is all
> > about "this would be nice" without a strong justification.
> 
> Because I was told by you that "Generating a lot of output for the oom
> reports has been a real problem in many deployments.".
> Maybe I misunderstood you : (

I meant to say that we do not want to add more output unless there is a
real need. Changing the existing output is usually tricky because there
are existing parsers that might break on that change. Not that we do
guarantee the output format officially but we try hard to preserve as
much as we can if that is possible.

That being said maybe the lack of reclaim guarantee information is a
more wide spread problem and more people are suffering from it. If that
is the case then the justification would be much easier to formulate.
-- 
Michal Hocko
SUSE Labs


^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2019-11-25  9:27 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-11-20  8:53 [PATCH] mm, memcg: show memcg min setting in oom messages Yafang Shao
2019-11-20 10:21 ` Michal Hocko
2019-11-20 10:53   ` Yafang Shao
2019-11-20 11:40     ` Michal Hocko
2019-11-20 12:23       ` Yafang Shao
2019-11-22 10:28         ` Michal Hocko
2019-11-23  5:52           ` Yafang Shao
2019-11-25  8:20             ` Michal Hocko
2019-11-25  9:12               ` Yafang Shao
2019-11-25  9:27                 ` Michal Hocko

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.