From: Mel Gorman <mgorman-IBi9RG/b67k@public.gmane.org> To: NeilBrown <neilb-l3A5Bk7waGM@public.gmane.org> Cc: Trond Myklebust <trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org>, Junxiao Bi <junxiao.bi-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org>, linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org Subject: Re: [PATCH v2 1/2] SUNRPC: Fix memory reclaim deadlocks in rpciod Date: Tue, 26 Aug 2014 11:53:04 +0100 [thread overview] Message-ID: <20140826105304.GT17696@novell.com> (raw) In-Reply-To: <20140825164852.50723141-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org> On Mon, Aug 25, 2014 at 04:48:52PM +1000, NeilBrown wrote: > On Fri, 22 Aug 2014 18:49:31 -0400 Trond Myklebust > <trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org> wrote: > > > Junxiao Bi reports seeing the following deadlock: > > > > @ crash> bt 1539 > > @ PID: 1539 TASK: ffff88178f64a040 CPU: 1 COMMAND: "rpciod/1" > > @ #0 [ffff88178f64d2c0] schedule at ffffffff8145833a > > @ #1 [ffff88178f64d348] io_schedule at ffffffff8145842c > > @ #2 [ffff88178f64d368] sync_page at ffffffff810d8161 > > @ #3 [ffff88178f64d378] __wait_on_bit at ffffffff8145895b > > @ #4 [ffff88178f64d3b8] wait_on_page_bit at ffffffff810d82fe > > @ #5 [ffff88178f64d418] wait_on_page_writeback at ffffffff810e2a1a > > @ #6 [ffff88178f64d438] shrink_page_list at ffffffff810e34e1 > > @ #7 [ffff88178f64d588] shrink_list at ffffffff810e3dbe > > @ #8 [ffff88178f64d6f8] shrink_zone at ffffffff810e425e > > @ #9 [ffff88178f64d7b8] do_try_to_free_pages at ffffffff810e4978 > > @ #10 [ffff88178f64d828] try_to_free_pages at ffffffff810e4c31 > > @ #11 [ffff88178f64d8c8] __alloc_pages_nodemask at ffffffff810de370 > > This stack trace (from 2.6.32) cannot happen in mainline, though it took me a > while to remember/discover exactly why. > > try_to_free_pages() creates a 'struct scan_control' with ->target_mem_cgroup > set to NULL. > shrink_page_list() checks ->target_mem_cgroup using global_reclaim() and if > it is NULL, wait_on_page_writeback is *not* called. > wait_on_page_writeback has a host of other damage associated with it which is why we don't do it from reclaim any more. If the storage is very slow then a process can be stalled by unrelated IO to slow storage. If the storage is broken and the writeback can never complete then it causes other issues. That kind of thing. > So we can only hit this deadlock if mem-cgroup limits are imposed on a > process which is using NFS - which is quite possible but probably not common. > > The fact that a dead-lock can happen only when memcg limits are imposed seems > very fragile. People aren't going to test that case much so there could well > be other deadlock possibilities lurking. > memcgs still can call wait_on_page_writeback and this is known to be a hand-grenade to the memcg people but I've never heard of them trying to tackle the problem. > Mel: might there be some other way we could get out of this deadlock? > Could the wait_on_page_writeback() in shrink_page_list() be made a timed-out > wait or something? Any other wait out of this deadlock other than setting > PF_MEMALLOC_NOIO everywhere? > I don't have the full thread as it was not cc'd to lkml so I don't know what circumstances reached this deadlock in the first place. If this is on 2.6.32 and the deadline cannot happen during reclaim in mainline then why is mainline being patched? Do not alter wait_on_page_writeback() to timeout as it will blow up spectacularly -- swap unuse races, data would not longer be synced correctly to disk, sync IO would be flaky, stable page writes would be fired out the window etc. -- Mel Gorman SUSE Labs -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: Mel Gorman <mgorman@suse.com> To: NeilBrown <neilb@suse.de> Cc: Trond Myklebust <trond.myklebust@primarydata.com>, Junxiao Bi <junxiao.bi@oracle.com>, linux-nfs@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: Re: [PATCH v2 1/2] SUNRPC: Fix memory reclaim deadlocks in rpciod Date: Tue, 26 Aug 2014 11:53:04 +0100 [thread overview] Message-ID: <20140826105304.GT17696@novell.com> (raw) In-Reply-To: <20140825164852.50723141@notabene.brown> On Mon, Aug 25, 2014 at 04:48:52PM +1000, NeilBrown wrote: > On Fri, 22 Aug 2014 18:49:31 -0400 Trond Myklebust > <trond.myklebust@primarydata.com> wrote: > > > Junxiao Bi reports seeing the following deadlock: > > > > @ crash> bt 1539 > > @ PID: 1539 TASK: ffff88178f64a040 CPU: 1 COMMAND: "rpciod/1" > > @ #0 [ffff88178f64d2c0] schedule at ffffffff8145833a > > @ #1 [ffff88178f64d348] io_schedule at ffffffff8145842c > > @ #2 [ffff88178f64d368] sync_page at ffffffff810d8161 > > @ #3 [ffff88178f64d378] __wait_on_bit at ffffffff8145895b > > @ #4 [ffff88178f64d3b8] wait_on_page_bit at ffffffff810d82fe > > @ #5 [ffff88178f64d418] wait_on_page_writeback at ffffffff810e2a1a > > @ #6 [ffff88178f64d438] shrink_page_list at ffffffff810e34e1 > > @ #7 [ffff88178f64d588] shrink_list at ffffffff810e3dbe > > @ #8 [ffff88178f64d6f8] shrink_zone at ffffffff810e425e > > @ #9 [ffff88178f64d7b8] do_try_to_free_pages at ffffffff810e4978 > > @ #10 [ffff88178f64d828] try_to_free_pages at ffffffff810e4c31 > > @ #11 [ffff88178f64d8c8] __alloc_pages_nodemask at ffffffff810de370 > > This stack trace (from 2.6.32) cannot happen in mainline, though it took me a > while to remember/discover exactly why. > > try_to_free_pages() creates a 'struct scan_control' with ->target_mem_cgroup > set to NULL. > shrink_page_list() checks ->target_mem_cgroup using global_reclaim() and if > it is NULL, wait_on_page_writeback is *not* called. > wait_on_page_writeback has a host of other damage associated with it which is why we don't do it from reclaim any more. If the storage is very slow then a process can be stalled by unrelated IO to slow storage. If the storage is broken and the writeback can never complete then it causes other issues. That kind of thing. > So we can only hit this deadlock if mem-cgroup limits are imposed on a > process which is using NFS - which is quite possible but probably not common. > > The fact that a dead-lock can happen only when memcg limits are imposed seems > very fragile. People aren't going to test that case much so there could well > be other deadlock possibilities lurking. > memcgs still can call wait_on_page_writeback and this is known to be a hand-grenade to the memcg people but I've never heard of them trying to tackle the problem. > Mel: might there be some other way we could get out of this deadlock? > Could the wait_on_page_writeback() in shrink_page_list() be made a timed-out > wait or something? Any other wait out of this deadlock other than setting > PF_MEMALLOC_NOIO everywhere? > I don't have the full thread as it was not cc'd to lkml so I don't know what circumstances reached this deadlock in the first place. If this is on 2.6.32 and the deadline cannot happen during reclaim in mainline then why is mainline being patched? Do not alter wait_on_page_writeback() to timeout as it will blow up spectacularly -- swap unuse races, data would not longer be synced correctly to disk, sync IO would be flaky, stable page writes would be fired out the window etc. -- Mel Gorman SUSE Labs
next prev parent reply other threads:[~2014-08-26 10:53 UTC|newest] Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top 2014-08-22 7:55 rpciod deadlock issue Junxiao Bi [not found] ` <53F6F772.6020708-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-08-22 22:49 ` [PATCH v2 1/2] SUNRPC: Fix memory reclaim deadlocks in rpciod Trond Myklebust 2014-08-22 22:49 ` Trond Myklebust 2014-08-22 22:49 ` [PATCH v2 2/2] NFS: Ensure that rpciod does not trigger reclaim writebacks Trond Myklebust [not found] ` <1408747772-37938-1-git-send-email-trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org> 2014-08-25 5:34 ` [PATCH v2 1/2] SUNRPC: Fix memory reclaim deadlocks in rpciod Junxiao Bi 2014-08-25 5:34 ` Junxiao Bi 2014-08-25 6:48 ` NeilBrown 2014-08-25 6:48 ` NeilBrown [not found] ` <20140825164852.50723141-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org> 2014-08-26 5:43 ` Junxiao Bi 2014-08-26 5:43 ` Junxiao Bi 2014-08-26 6:21 ` NeilBrown 2014-08-26 6:49 ` Junxiao Bi 2014-08-26 7:04 ` NeilBrown [not found] ` <20140826170410.20560764-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org> 2014-08-26 7:23 ` Junxiao Bi 2014-08-26 7:23 ` Junxiao Bi 2014-08-26 10:53 ` Mel Gorman [this message] 2014-08-26 10:53 ` Mel Gorman 2014-08-26 12:58 ` Trond Myklebust 2014-08-26 13:26 ` Mel Gorman [not found] ` <20140826132624.GU17696-Et1tbQHTxzrQT0dZR+AlfA@public.gmane.org> 2014-08-26 23:19 ` Johannes Weiner 2014-08-26 23:19 ` Johannes Weiner [not found] ` <20140826231938.GA13889-druUgvl0LCNAfugRpC6u6w@public.gmane.org> 2014-08-26 23:51 ` Trond Myklebust 2014-08-26 23:51 ` Trond Myklebust [not found] ` <CAHQdGtRPsVFVfph5OcsZk_+WYPPJ-MpE2myZfXAb3jq6fuM4zw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2014-08-27 0:00 ` Trond Myklebust 2014-08-27 0:00 ` Trond Myklebust 2014-08-27 15:36 ` Mel Gorman [not found] ` <20140827153644.GF12374-Et1tbQHTxzrQT0dZR+AlfA@public.gmane.org> 2014-08-27 16:15 ` Trond Myklebust 2014-08-27 16:15 ` Trond Myklebust 2014-08-28 8:30 ` Mel Gorman [not found] ` <20140828083053.GJ12374-Et1tbQHTxzrQT0dZR+AlfA@public.gmane.org> 2014-08-28 8:49 ` Junxiao Bi 2014-08-28 8:49 ` Junxiao Bi 2014-08-28 9:25 ` Mel Gorman 2014-09-04 13:54 ` Michal Hocko 2014-09-04 13:54 ` Michal Hocko 2014-09-09 2:33 ` NeilBrown 2014-09-10 13:48 ` Michal Hocko [not found] ` <20140910134842.GG25219-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org> 2014-09-10 23:57 ` NeilBrown 2014-09-10 23:57 ` NeilBrown [not found] ` <20140911095743.1ed87519-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org> 2014-09-11 8:50 ` Michal Hocko 2014-09-11 8:50 ` Michal Hocko [not found] ` <20140911085046.GC22042-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org> 2014-09-11 10:53 ` NeilBrown 2014-09-11 10:53 ` NeilBrown 2014-08-27 1:43 ` NeilBrown 2014-08-27 1:43 ` NeilBrown 2014-08-25 6:05 ` rpciod deadlock issue NeilBrown 2014-08-25 6:05 ` NeilBrown [not found] ` <20140825160501.433b3e9e-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org> 2014-08-25 6:15 ` NeilBrown 2014-08-25 6:15 ` NeilBrown
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20140826105304.GT17696@novell.com \ --to=mgorman-ibi9rg/b67k@public.gmane.org \ --cc=junxiao.bi-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org \ --cc=linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=neilb-l3A5Bk7waGM@public.gmane.org \ --cc=trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.