OOM notification for cgroupsv1 broken in 4.19

OOM notification for cgroupsv1 broken in 4.19

[Burt Holzman]

Hi,

This patch: 29ef680ae7c21110af8e6416d84d8a72fc147b14
[PATCH] memcg, oom: move out_of_memory back to the charge path

has broken the eventfd notification for cgroups-v1. This is because 
mem_cgroup_oom_notify() is called only in mem_cgroup_oom_synchronize and 
not with the new, additional call to mem_cgroup_out_of_memory in the 
charge path.

- B

Re: OOM notification for cgroupsv1 broken in 4.19

[Michal Hocko]

On Fri 21-12-18 14:49:38, Burt Holzman wrote:
> Hi,
> 
> This patch: 29ef680ae7c21110af8e6416d84d8a72fc147b14
> [PATCH] memcg, oom: move out_of_memory back to the charge path
> 
> has broken the eventfd notification for cgroups-v1. This is because 
> mem_cgroup_oom_notify() is called only in mem_cgroup_oom_synchronize and 
> not with the new, additional call to mem_cgroup_out_of_memory in the 
> charge path.

Yes, you are right and this is a clear regression. Does the following
patch fixes the issue for you? I am not super happy about the code
duplication but I wasn't able to separate this out from
mem_cgroup_oom_synchronize because that one has to handle the oom_killer
disabled case which is not the case in the charge path because we simply
back off and hand over to mem_cgroup_oom_synchronize in that case.
---
>From 51633f683173013741f4d0ab3e31bae575341c55 Mon Sep 17 00:00:00 2001
From: Michal Hocko <mhocko@suse.com>
Date: Fri, 21 Dec 2018 16:28:29 +0100
Subject: [PATCH] memcg, oom: notify on oom killer invocation from the charge
 path

Burt Holzman has noticed that memcg v1 doesn't notify about OOM events
via eventfd anymore. The reason is that 29ef680ae7c2 ("memcg, oom: move
out_of_memory back to the charge path") has moved the oom handling back
to the charge path. While doing so the notification was left behind in
mem_cgroup_oom_synchronize.

Fix the issue by replicating the oom hierarchy locking and the
notification.

Reported-by: Burt Holzman <burt@fnal.gov>
Fixes: 29ef680ae7c2 ("memcg, oom: move out_of_memory back to the charge path")
Cc: stable # 4.19+
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/memcontrol.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6e1469b80cb7..7e6bf74ddb1e 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1666,6 +1666,9 @@ enum oom_status {
 
 static enum oom_status mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order)
 {
+	enum oom_status ret;
+	bool locked;
+
 	if (order > PAGE_ALLOC_COSTLY_ORDER)
 		return OOM_SKIPPED;
 
@@ -1700,10 +1703,23 @@ static enum oom_status mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int
 		return OOM_ASYNC;
 	}
 
+	mem_cgroup_mark_under_oom(memcg);
+
+	locked = mem_cgroup_oom_trylock(memcg);
+
+	if (locked)
+		mem_cgroup_oom_notify(memcg);
+
+	mem_cgroup_unmark_under_oom(memcg);
 	if (mem_cgroup_out_of_memory(memcg, mask, order))
-		return OOM_SUCCESS;
+		ret = OOM_SUCCESS;
+	else
+		ret = OOM_FAILED;
 
-	return OOM_FAILED;
+	if (locked)
+		mem_cgroup_oom_unlock(memcg);
+
+	return ret;
 }
 
 /**
-- 
2.19.2

-- 
Michal Hocko
SUSE Labs

Re: OOM notification for cgroupsv1 broken in 4.19

[Michal Hocko]

On Fri 21-12-18 14:49:38, Burt Holzman wrote:
> Hi,
> 
> This patch: 29ef680ae7c21110af8e6416d84d8a72fc147b14
> [PATCH] memcg, oom: move out_of_memory back to the charge path
> 
> has broken the eventfd notification for cgroups-v1. This is because 
> mem_cgroup_oom_notify() is called only in mem_cgroup_oom_synchronize and 
> not with the new, additional call to mem_cgroup_out_of_memory in the 
> charge path.

Yes, you are right and this is a clear regression. Does the following
patch fixes the issue for you? I am not super happy about the code
duplication but I wasn't able to separate this out from
mem_cgroup_oom_synchronize because that one has to handle the oom_killer
disabled case which is not the case in the charge path because we simply
back off and hand over to mem_cgroup_oom_synchronize in that case.
---
From 51633f683173013741f4d0ab3e31bae575341c55 Mon Sep 17 00:00:00 2001
From: Michal Hocko <mhocko@suse.com>
Date: Fri, 21 Dec 2018 16:28:29 +0100
Subject: [PATCH] memcg, oom: notify on oom killer invocation from the charge
 path

Burt Holzman has noticed that memcg v1 doesn't notify about OOM events
via eventfd anymore. The reason is that 29ef680ae7c2 ("memcg, oom: move
out_of_memory back to the charge path") has moved the oom handling back
to the charge path. While doing so the notification was left behind in
mem_cgroup_oom_synchronize.

Fix the issue by replicating the oom hierarchy locking and the
notification.

Reported-by: Burt Holzman <burt@fnal.gov>
Fixes: 29ef680ae7c2 ("memcg, oom: move out_of_memory back to the charge path")
Cc: stable # 4.19+
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
 mm/memcontrol.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6e1469b80cb7..7e6bf74ddb1e 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1666,6 +1666,9 @@ enum oom_status {
 
 static enum oom_status mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order)
 {
+	enum oom_status ret;
+	bool locked;
+
 	if (order > PAGE_ALLOC_COSTLY_ORDER)
 		return OOM_SKIPPED;
 
@@ -1700,10 +1703,23 @@ static enum oom_status mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int
 		return OOM_ASYNC;
 	}
 
+	mem_cgroup_mark_under_oom(memcg);
+
+	locked = mem_cgroup_oom_trylock(memcg);
+
+	if (locked)
+		mem_cgroup_oom_notify(memcg);
+
+	mem_cgroup_unmark_under_oom(memcg);
 	if (mem_cgroup_out_of_memory(memcg, mask, order))
-		return OOM_SUCCESS;
+		ret = OOM_SUCCESS;
+	else
+		ret = OOM_FAILED;
 
-	return OOM_FAILED;
+	if (locked)
+		mem_cgroup_oom_unlock(memcg);
+
+	return ret;
 }
 
 /**
-- 
2.19.2

-- 
Michal Hocko
SUSE Labs

Re: OOM notification for cgroupsv1 broken in 4.19

[Burt Holzman]

> On Dec 21, 2018, at 9:33 AM, Michal Hocko <mhocko@kernel.org> wrote:
> 
> On Fri 21-12-18 14:49:38, Burt Holzman wrote:
>> Hi,
>> 
>> This patch: 29ef680ae7c21110af8e6416d84d8a72fc147b14
>> [PATCH] memcg, oom: move out_of_memory back to the charge path
>> 
>> has broken the eventfd notification for cgroups-v1. This is because 
>> mem_cgroup_oom_notify() is called only in mem_cgroup_oom_synchronize and 
>> not with the new, additional call to mem_cgroup_out_of_memory in the 
>> charge path.
> 
> Yes, you are right and this is a clear regression. Does the following
> patch fixes the issue for you? I am not super happy about the code
> duplication but I wasn't able to separate this out from
> mem_cgroup_oom_synchronize because that one has to handle the oom_killer
> disabled case which is not the case in the charge path because we simply
> back off and hand over to mem_cgroup_oom_synchronize in that case.

Hi Michal,

Thanks for the quick response & patch. I can confirm that with this patch the notification is working for my sample test case.

- B

Re: OOM notification for cgroupsv1 broken in 4.19

[Johannes Weiner]

On Fri, Dec 21, 2018 at 04:33:02PM +0100, Michal Hocko wrote:
> From 51633f683173013741f4d0ab3e31bae575341c55 Mon Sep 17 00:00:00 2001
> From: Michal Hocko <mhocko@suse.com>
> Date: Fri, 21 Dec 2018 16:28:29 +0100
> Subject: [PATCH] memcg, oom: notify on oom killer invocation from the charge
>  path
> 
> Burt Holzman has noticed that memcg v1 doesn't notify about OOM events
> via eventfd anymore. The reason is that 29ef680ae7c2 ("memcg, oom: move
> out_of_memory back to the charge path") has moved the oom handling back
> to the charge path. While doing so the notification was left behind in
> mem_cgroup_oom_synchronize.
> 
> Fix the issue by replicating the oom hierarchy locking and the
> notification.
> 
> Reported-by: Burt Holzman <burt@fnal.gov>
> Fixes: 29ef680ae7c2 ("memcg, oom: move out_of_memory back to the charge path")
> Cc: stable # 4.19+
> Signed-off-by: Michal Hocko <mhocko@suse.com>

Looks good to me. The async side really does too much other stuff to
cleanly share code between them, so I don't mind separate code even if
it means they both have to do the mark, lock, notify dance.

Acked-by: Johannes Weiner <hannes@cmpxchg.org>

[PATCH] memcg, oom: notify on oom killer invocation from the charge path

[Michal Hocko]

From: Michal Hocko <mhocko@suse.com>

Burt Holzman has noticed that memcg v1 doesn't notify about OOM events
via eventfd anymore. The reason is that 29ef680ae7c2 ("memcg, oom: move
out_of_memory back to the charge path") has moved the oom handling back
to the charge path. While doing so the notification was left behind in
mem_cgroup_oom_synchronize.

Fix the issue by replicating the oom hierarchy locking and the
notification.

Reported-by: Burt Holzman <burt@fnal.gov>
Fixes: 29ef680ae7c2 ("memcg, oom: move out_of_memory back to the charge path")
Cc: stable # 4.19+
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
Hi Andrew,
I forgot to CC you on the patch sent as a reply to the original bug
report [1] so I am reposting with Ack from Johannes. Burt has confirmed
this is resolving the regression for him [2]. 4.20 is out but I have
marked the patch for stable so it should hit both 4.19 and 4.20.

[1] http://lkml.kernel.org/r/20181221153302.GB6410@dhcp22.suse.cz
[2] http://lkml.kernel.org/r/96D4815C-420F-41B7-B1E9-A741E7523596@services.fnal.gov

 mm/memcontrol.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6e1469b80cb7..7e6bf74ddb1e 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1666,6 +1666,9 @@ enum oom_status {
 
 static enum oom_status mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order)
 {
+	enum oom_status ret;
+	bool locked;
+
 	if (order > PAGE_ALLOC_COSTLY_ORDER)
 		return OOM_SKIPPED;
 
@@ -1700,10 +1703,23 @@ static enum oom_status mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int
 		return OOM_ASYNC;
 	}
 
+	mem_cgroup_mark_under_oom(memcg);
+
+	locked = mem_cgroup_oom_trylock(memcg);
+
+	if (locked)
+		mem_cgroup_oom_notify(memcg);
+
+	mem_cgroup_unmark_under_oom(memcg);
 	if (mem_cgroup_out_of_memory(memcg, mask, order))
-		return OOM_SUCCESS;
+		ret = OOM_SUCCESS;
+	else
+		ret = OOM_FAILED;
 
-	return OOM_FAILED;
+	if (locked)
+		mem_cgroup_oom_unlock(memcg);
+
+	return ret;
 }
 
 /**
-- 
2.19.2

[PATCH] memcg, oom: notify on oom killer invocation from the charge path

[Michal Hocko]

From: Michal Hocko <mhocko@suse.com>

Burt Holzman has noticed that memcg v1 doesn't notify about OOM events
via eventfd anymore. The reason is that 29ef680ae7c2 ("memcg, oom: move
out_of_memory back to the charge path") has moved the oom handling back
to the charge path. While doing so the notification was left behind in
mem_cgroup_oom_synchronize.

Fix the issue by replicating the oom hierarchy locking and the
notification.

Reported-by: Burt Holzman <burt@fnal.gov>
Fixes: 29ef680ae7c2 ("memcg, oom: move out_of_memory back to the charge path")
Cc: stable # 4.19+
Acked-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Michal Hocko <mhocko@suse.com>
---
Hi Andrew,
I forgot to CC you on the patch sent as a reply to the original bug
report [1] so I am reposting with Ack from Johannes. Burt has confirmed
this is resolving the regression for him [2]. 4.20 is out but I have
marked the patch for stable so it should hit both 4.19 and 4.20.

[1] http://lkml.kernel.org/r/20181221153302.GB6410@dhcp22.suse.cz
[2] http://lkml.kernel.org/r/96D4815C-420F-41B7-B1E9-A741E7523596@services.fnal.gov

 mm/memcontrol.c | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 6e1469b80cb7..7e6bf74ddb1e 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1666,6 +1666,9 @@ enum oom_status {
 
 static enum oom_status mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int order)
 {
+	enum oom_status ret;
+	bool locked;
+
 	if (order > PAGE_ALLOC_COSTLY_ORDER)
 		return OOM_SKIPPED;
 
@@ -1700,10 +1703,23 @@ static enum oom_status mem_cgroup_oom(struct mem_cgroup *memcg, gfp_t mask, int
 		return OOM_ASYNC;
 	}
 
+	mem_cgroup_mark_under_oom(memcg);
+
+	locked = mem_cgroup_oom_trylock(memcg);
+
+	if (locked)
+		mem_cgroup_oom_notify(memcg);
+
+	mem_cgroup_unmark_under_oom(memcg);
 	if (mem_cgroup_out_of_memory(memcg, mask, order))
-		return OOM_SUCCESS;
+		ret = OOM_SUCCESS;
+	else
+		ret = OOM_FAILED;
 
-	return OOM_FAILED;
+	if (locked)
+		mem_cgroup_oom_unlock(memcg);
+
+	return ret;
 }
 
 /**
-- 
2.19.2