[PATCH] mm, oom: check memcg margin for parallel oom

[Yafang Shao <laoar.shao@gmail.com>]

The commit 7775face2079 ("memcg: killed threads should not invoke memcg OOM
killer") resolves the problem that different threads in a multi-threaded
task doing parallel memcg oom, but it doesn't solve the problem that
different tasks doing parallel memcg oom.

It may happens that many different tasks in the same memcg are waiting
oom_lock at the same time, if one of them has already made progress and
freed enough available memory, the others don't need to trigger the oom
killer again. By checking memcg margin after hold oom_lock can help
achieve it.

Suggested-by: Michal Hocko <mhocko@kernel.org>
Signed-off-by: Yafang Shao <laoar.shao@gmail.com>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Cc: David Rientjes <rientjes@google.com>
---
 mm/memcontrol.c | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1962232..df141e1 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1560,16 +1560,31 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask,
 		.gfp_mask = gfp_mask,
 		.order = order,
 	};
-	bool ret;
+	bool ret = true;
 
 	if (mutex_lock_killable(&oom_lock))
 		return true;
+
 	/*
 	 * A few threads which were not waiting at mutex_lock_killable() can
 	 * fail to bail out. Therefore, check again after holding oom_lock.
 	 */
-	ret = should_force_charge() || out_of_memory(&oc);
+	if (should_force_charge())
+		goto out;
+
+	/*
+	 * Different tasks may be doing parallel oom, so after hold the
+	 * oom_lock the task should check the memcg margin again to check
+	 * whether other task has already made progress.
+	 */
+	if (mem_cgroup_margin(memcg) >= (1 << order))
+		goto out;
+
+	ret = out_of_memory(&oc);
+
+out:
 	mutex_unlock(&oom_lock);
+
 	return ret;
 }
 
-- 
1.8.3.1