Re: [PATCH] mm,oom: Bring OOM notifier callbacks to outside of OOM killer.

[Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>]

On 2018/06/20 20:55, Michal Hocko wrote:
> On Wed 20-06-18 20:20:38, Tetsuo Handa wrote:
>> Sleeping with oom_lock held can cause AB-BA lockup bug because
>> __alloc_pages_may_oom() does not wait for oom_lock. Since
>> blocking_notifier_call_chain() in out_of_memory() might sleep, sleeping
>> with oom_lock held is currently an unavoidable problem.
> 
> Could you be more specific about the potential deadlock? Sleeping while
> holding oom lock is certainly not nice but I do not see how that would
> result in a deadlock assuming that the sleeping context doesn't sleep on
> the memory allocation obviously.

"A" is "owns oom_lock" and "B" is "owns CPU resources". It was demonstrated
at "mm,oom: Don't call schedule_timeout_killable() with oom_lock held." proposal.

But since you don't accept preserving the short sleep which is a heuristic for
reducing the possibility of AB-BA lockup, the only way we would accept will be
wait for the owner of oom_lock (e.g. by s/mutex_trylock/mutex_lock/ or whatever)
which is free of heuristic and free of AB-BA lockup.

> 
>> As a preparation for not to sleep with oom_lock held, this patch brings
>> OOM notifier callbacks to outside of OOM killer, with two small behavior
>> changes explained below.
> 
> Can we just eliminate this ugliness and remove it altogether? We do not
> have that many notifiers. Is there anything fundamental that would
> prevent us from moving them to shrinkers instead?
> 

For long term, it would be possible. But not within this patch. For example,
I think that virtio_balloon wants to release memory only when we have no
choice but OOM kill. If virtio_balloon trivially releases memory, it will
increase the risk of killing the entire guest by OOM-killer from the host
side.