cgroup-aware OOM killer, how to move forward

[Roman Gushchin <guro@fb.com>]

Hello!

I was thinking on how to move forward with the cgroup-aware OOM killer.
It looks to me, that we all agree on the "cleanup" part of the patchset:
it's a nice feature to be able to kill all tasks in the cgroup
to guarantee the consistent state of the workload.
All our disagreements are related to the victim selection algorithm.

So, I wonder, if the right thing to do is to split the problem.
We can agree on the "cleanup" part, which is useful by itself,
merge it upstream, and then return to the victim selection
algorithm.

So, here is my proposal:
let's introduce the memory.group_oom knob with the following semantics:
if the knob is set, the OOM killer can kill either none, either all
tasks in the cgroup*.
It can perfectly work with the current OOM killer (as a "cleanup" option),
and allows _any_ further approach on the OOM victim selection.
It also doesn't require any mount/boot/tree-wide options.

How does it sound?

If we can agree on this, I will prepare the patchset.
It's quite small and straightforward in comparison to the current version.

Thanks!


* More precisely: if the OOM killer kills a task,
it will traverse the cgroup tree up to the OOM domain (OOMing memcg or root),
looking for the highest-level cgroup with group_oom set. Then it will
kill all tasks in such cgroup, if it does exist.