From: "Michael S. Tsirkin" <mst@redhat.com>
To: Michal Hocko <mhocko@kernel.org>
Cc: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>, linux-mm@kvack.org
Subject: Re: [RFC] [PATCH] mm,oom: Offload OOM notify callback to a kernel thread.
Date: Mon, 9 Oct 2017 17:24:55 +0300 [thread overview]
Message-ID: <20171009172446-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <20171009133734.u544mwyjipamc566@dhcp22.suse.cz>
On Mon, Oct 09, 2017 at 03:37:34PM +0200, Michal Hocko wrote:
> On Mon 09-10-17 22:31:18, Tetsuo Handa wrote:
> > Michal Hocko wrote:
> > > On Mon 09-10-17 17:06:51, Tetsuo Handa wrote:
> > > > Michal Hocko wrote:
> > > > > On Sat 07-10-17 20:30:19, Tetsuo Handa wrote:
> > > > > [...]
> > > > > > >From 6a0fd8a5e013ac63a6bcd06bd2ae6fdb25a4f3de Mon Sep 17 00:00:00 2001
> > > > > > From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> > > > > > Date: Sat, 7 Oct 2017 19:29:21 +0900
> > > > > > Subject: [PATCH] virtio: avoid possible OOM lockup at virtballoon_oom_notify()
> > > > > >
> > > > > > In leak_balloon(), mutex_lock(&vb->balloon_lock) is called in order to
> > > > > > serialize against fill_balloon(). But in fill_balloon(),
> > > > > > alloc_page(GFP_HIGHUSER[_MOVABLE] | __GFP_NOMEMALLOC | __GFP_NORETRY) is
> > > > > > called with vb->balloon_lock mutex held. Since GFP_HIGHUSER[_MOVABLE]
> > > > > > implies __GFP_DIRECT_RECLAIM | __GFP_IO | __GFP_FS, despite __GFP_NORETRY
> > > > > > is specified, this allocation attempt might depend on somebody else's
> > > > > > __GFP_DIRECT_RECLAIM memory allocation.
> > > > >
> > > > > How would that dependency look like? Is the holder of the lock doing
> > > > > only __GFP_NORETRY?
> > > >
> > > > __GFP_NORETRY makes difference only after reclaim attempt failed.
> > > >
> > > > Reclaim attempt of __GFP_DIRECT_RECLAIM | __GFP_IO | __GFP_FS request can
> > > > indirectly wait for somebody else's GFP_NOFS and/or GFP_NOIO request (e.g.
> > > > blocked on filesystem's fs lock). And such indirect GFP_NOFS and/or
> > > > GFP_NOIO request can reach __alloc_pages_may_oom() unless they also have
> > > > __GFP_NORETRY. And such indirect GFP_NOFS and/or GFP_NOIO request can call
> > > > OOM notifier callback and try to hold balloon_lock at leak_balloon() which
> > > > fill_balloon() has already held before doing
> > > > GFP_HIGHUSER[_MOVABLE] | __GFP_NOMEMALLOC | __GFP_NORETRY request.
> > >
> > > OK, so let me decipher.
> > > Thread1 Thread2 Thread3
> > > alloc_pages(GFP_KERNEL) fill_balloon fs_lock #1
> > > out_of_memory balloon_lock #2 alloc_page(GFP_NOFS)
> > > blocking_notifier_call_chain balloon_page_enqueue # keep retrying
> > > leak_balloon alloc_page(GFP_HIGHUSER_MOVABLE)
> > > balloon_lock #2 direct_reclaim (__GFP_FS context)
> > > fs_lock #1
> > >
> > > in other words, let's make the description understandable even for
> > > somebody not really familiar with the allocation&reclaim internals.
> > > The whole point is that the dependency is indirect and it requires
> > > more actors and an example call grapg should be easier to follow.
> >
> >
> > Yes. But it is more simple. Only two threads are needed.
> >
> > Thread1 Thread2
> > fill_balloon
> > balloon_lock #1
> > balloon_page_enqueue
> > alloc_page(GFP_HIGHUSER_MOVABLE)
> > direct reclaim (__GFP_FS context) fs lock #2
> > fs lock #2 alloc_page(GFP_NOFS)
> > __alloc_pages_may_oom()
> > oom_lock
> > out_of_memory()
> > blocking_notifier_call_chain()
> > leak_balloon
> > balloon_lock #1 # dead lock
>
> Oh, right. I forgot we are allowed oom notifiers from NOFS context.
>
> > And other __GFP_DIRECT_RECLAIM && !__GFP_NORETRY allocations (if any) will keep
> > retrying forever because oom_lock is held by Thread2.
> >
> > >
> > > One more nit. If there is a way to estimate how much memory could be
> > > freed by the notifier when the trylock would succeed I would print that
> > > value for debugging purposes.
> >
> > I don't know internal of virtio-balloon.
>
> Maybe Michael can help here.
num_pages is a way to estimate that.
> --
> Michal Hocko
> SUSE Labs
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2017-10-09 14:25 UTC|newest]
Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-09-11 10:27 mm, virtio: possible OOM lockup at virtballoon_oom_notify() Tetsuo Handa
2017-09-29 4:00 ` Michael S. Tsirkin
2017-09-29 4:00 ` Michael S. Tsirkin
2017-09-29 4:44 ` Tetsuo Handa
2017-09-29 4:44 ` Tetsuo Handa
2017-10-01 5:44 ` [RFC] [PATCH] mm, oom: Offload OOM notify callback to a kernel thread Tetsuo Handa
2017-10-02 3:59 ` [RFC] [PATCH] mm,oom: " Michael S. Tsirkin
2017-10-02 3:59 ` [RFC] [PATCH] mm, oom: " Michael S. Tsirkin
2017-10-02 9:06 ` [RFC] [PATCH] mm,oom: " Michal Hocko
2017-10-02 9:06 ` Michal Hocko
2017-10-02 11:33 ` [RFC] [PATCH] mm, oom: " Tetsuo Handa
2017-10-02 11:50 ` Michal Hocko
2017-10-02 13:05 ` [RFC] [PATCH] mm,oom: " Tetsuo Handa
2017-10-02 13:13 ` Michal Hocko
2017-10-02 13:52 ` Tetsuo Handa
2017-10-02 14:23 ` Michael S. Tsirkin
2017-10-02 14:44 ` Tetsuo Handa
2017-10-07 11:30 ` Tetsuo Handa
2017-10-09 7:46 ` Michal Hocko
2017-10-09 8:06 ` Tetsuo Handa
2017-10-09 12:28 ` Michal Hocko
2017-10-09 13:31 ` Tetsuo Handa
2017-10-09 13:37 ` Michal Hocko
2017-10-09 14:24 ` Michael S. Tsirkin [this message]
2017-10-02 14:15 ` Michael S. Tsirkin
2017-10-02 14:11 ` Michael S. Tsirkin
2017-10-02 14:11 ` Michael S. Tsirkin
2017-10-02 14:11 ` [RFC] [PATCH] mm, oom: " Michael S. Tsirkin
2017-10-02 14:19 ` [RFC] [PATCH] mm,oom: " Michal Hocko
2017-10-02 14:19 ` [RFC] [PATCH] mm, oom: " Michal Hocko
2017-10-02 14:29 ` [RFC] [PATCH] mm,oom: " Michael S. Tsirkin
2017-10-02 14:29 ` Michael S. Tsirkin
2017-10-02 14:29 ` [RFC] [PATCH] mm, oom: " Michael S. Tsirkin
2017-10-02 14:31 ` [RFC] [PATCH] mm,oom: " Michal Hocko
2017-10-02 14:31 ` Michal Hocko
2017-10-02 14:19 ` Michal Hocko
2017-10-02 11:50 ` Michal Hocko
2017-10-02 3:59 ` Michael S. Tsirkin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20171009172446-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=penguin-kernel@I-love.SAKURA.ne.jp \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.