From: Junxiao Bi <junxiao.bi-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> To: NeilBrown <neilb-l3A5Bk7waGM@public.gmane.org> Cc: Trond Myklebust <trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org>, linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org, Mel Gorman <mgorman-IBi9RG/b67k@public.gmane.org> Subject: Re: [PATCH v2 1/2] SUNRPC: Fix memory reclaim deadlocks in rpciod Date: Tue, 26 Aug 2014 15:23:44 +0800 [thread overview] Message-ID: <53FC3600.9050202@oracle.com> (raw) In-Reply-To: <20140826170410.20560764-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org> On 08/26/2014 03:04 PM, NeilBrown wrote: > On Tue, 26 Aug 2014 14:49:01 +0800 Junxiao Bi <junxiao.bi-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote: > >> On 08/26/2014 02:21 PM, NeilBrown wrote: >>> On Tue, 26 Aug 2014 13:43:47 +0800 Junxiao Bi <junxiao.bi-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote: >>> >>>> On 08/25/2014 02:48 PM, NeilBrown wrote: >>>>> On Fri, 22 Aug 2014 18:49:31 -0400 Trond Myklebust >>>>> <trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org> wrote: >>>>> >>>>>> Junxiao Bi reports seeing the following deadlock: >>>>>> >>>>>> @ crash> bt 1539 >>>>>> @ PID: 1539 TASK: ffff88178f64a040 CPU: 1 COMMAND: "rpciod/1" >>>>>> @ #0 [ffff88178f64d2c0] schedule at ffffffff8145833a >>>>>> @ #1 [ffff88178f64d348] io_schedule at ffffffff8145842c >>>>>> @ #2 [ffff88178f64d368] sync_page at ffffffff810d8161 >>>>>> @ #3 [ffff88178f64d378] __wait_on_bit at ffffffff8145895b >>>>>> @ #4 [ffff88178f64d3b8] wait_on_page_bit at ffffffff810d82fe >>>>>> @ #5 [ffff88178f64d418] wait_on_page_writeback at ffffffff810e2a1a >>>>>> @ #6 [ffff88178f64d438] shrink_page_list at ffffffff810e34e1 >>>>>> @ #7 [ffff88178f64d588] shrink_list at ffffffff810e3dbe >>>>>> @ #8 [ffff88178f64d6f8] shrink_zone at ffffffff810e425e >>>>>> @ #9 [ffff88178f64d7b8] do_try_to_free_pages at ffffffff810e4978 >>>>>> @ #10 [ffff88178f64d828] try_to_free_pages at ffffffff810e4c31 >>>>>> @ #11 [ffff88178f64d8c8] __alloc_pages_nodemask at ffffffff810de370 >>>>> >>>>> This stack trace (from 2.6.32) cannot happen in mainline, though it took me a >>>>> while to remember/discover exactly why. >>>>> >>>>> try_to_free_pages() creates a 'struct scan_control' with ->target_mem_cgroup >>>>> set to NULL. >>>>> shrink_page_list() checks ->target_mem_cgroup using global_reclaim() and if >>>>> it is NULL, wait_on_page_writeback is *not* called. >>>>> >>>>> So we can only hit this deadlock if mem-cgroup limits are imposed on a >>>>> process which is using NFS - which is quite possible but probably not common. >>>>> >>>>> The fact that a dead-lock can happen only when memcg limits are imposed seems >>>>> very fragile. People aren't going to test that case much so there could well >>>>> be other deadlock possibilities lurking. >>>>> >>>>> Mel: might there be some other way we could get out of this deadlock? >>>>> Could the wait_on_page_writeback() in shrink_page_list() be made a timed-out >>>>> wait or something? Any other wait out of this deadlock other than setting >>>>> PF_MEMALLOC_NOIO everywhere? >>>> >>>> Not only the wait_on_page_writeback() cause the deadlock but also the >>>> next pageout()-> (mapping->a_ops->writepage), Trond's second patch fix >>>> this. So fix the wait_on_page_writeback is not enough to fix deadlock. >>> >>> Shortly before the only place that pageout() is called there is this code: >>> >>> if (page_is_file_cache(page) && >>> (!current_is_kswapd() || >>> !zone_is_reclaim_dirty(zone))) { >>> ..... >>> goto keep_locked; >>> >>> >>> So pageout() only gets called by kswapd() .... or for swap. swap-over-NFS is >>> already very cautious about memory allocations, and uses nfs_direct_IO, not >>> nfs_writepage. >>> >>> So nfs_writepage will never get called during direct reclaim. There is no >>> memory-allocate deadlock risk there. >> Yes, thanks for explaining this. >> But is it possible rpciod blocked somewhere by memory allocation using >> GFP_KERNEL and kswapd is trying to pageout nfs dirty pages and blocked >> by rpciod? > > I don't think so, no. > > Only 40% of memory (/proc/sys/vm/dirty_ratio) can be dirty. The direct > reclaim procedure will eventually find some non-dirty memory it can use. > If it cannot, and cannot write anything out to swap either, it will > eventually trigger the OOM killer. > > Direct reclaim shouldn't ever block indefinitely. It will sometimes wait for > a short while (e.g. congestion_wait()) but it should then push on until it > finds something it can do: free a clean page, write something to swap, or > kill a memory-hog with the OOM killer. That makes sense. Thanks. Junxiao. > > NeilBrown > -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org More majordomo info at http://vger.kernel.org/majordomo-info.html
WARNING: multiple messages have this Message-ID (diff)
From: Junxiao Bi <junxiao.bi@oracle.com> To: NeilBrown <neilb@suse.de> Cc: Trond Myklebust <trond.myklebust@primarydata.com>, linux-nfs@vger.kernel.org, linux-fsdevel@vger.kernel.org, Mel Gorman <mgorman@suse.com> Subject: Re: [PATCH v2 1/2] SUNRPC: Fix memory reclaim deadlocks in rpciod Date: Tue, 26 Aug 2014 15:23:44 +0800 [thread overview] Message-ID: <53FC3600.9050202@oracle.com> (raw) In-Reply-To: <20140826170410.20560764@notabene.brown> On 08/26/2014 03:04 PM, NeilBrown wrote: > On Tue, 26 Aug 2014 14:49:01 +0800 Junxiao Bi <junxiao.bi@oracle.com> wrote: > >> On 08/26/2014 02:21 PM, NeilBrown wrote: >>> On Tue, 26 Aug 2014 13:43:47 +0800 Junxiao Bi <junxiao.bi@oracle.com> wrote: >>> >>>> On 08/25/2014 02:48 PM, NeilBrown wrote: >>>>> On Fri, 22 Aug 2014 18:49:31 -0400 Trond Myklebust >>>>> <trond.myklebust@primarydata.com> wrote: >>>>> >>>>>> Junxiao Bi reports seeing the following deadlock: >>>>>> >>>>>> @ crash> bt 1539 >>>>>> @ PID: 1539 TASK: ffff88178f64a040 CPU: 1 COMMAND: "rpciod/1" >>>>>> @ #0 [ffff88178f64d2c0] schedule at ffffffff8145833a >>>>>> @ #1 [ffff88178f64d348] io_schedule at ffffffff8145842c >>>>>> @ #2 [ffff88178f64d368] sync_page at ffffffff810d8161 >>>>>> @ #3 [ffff88178f64d378] __wait_on_bit at ffffffff8145895b >>>>>> @ #4 [ffff88178f64d3b8] wait_on_page_bit at ffffffff810d82fe >>>>>> @ #5 [ffff88178f64d418] wait_on_page_writeback at ffffffff810e2a1a >>>>>> @ #6 [ffff88178f64d438] shrink_page_list at ffffffff810e34e1 >>>>>> @ #7 [ffff88178f64d588] shrink_list at ffffffff810e3dbe >>>>>> @ #8 [ffff88178f64d6f8] shrink_zone at ffffffff810e425e >>>>>> @ #9 [ffff88178f64d7b8] do_try_to_free_pages at ffffffff810e4978 >>>>>> @ #10 [ffff88178f64d828] try_to_free_pages at ffffffff810e4c31 >>>>>> @ #11 [ffff88178f64d8c8] __alloc_pages_nodemask at ffffffff810de370 >>>>> >>>>> This stack trace (from 2.6.32) cannot happen in mainline, though it took me a >>>>> while to remember/discover exactly why. >>>>> >>>>> try_to_free_pages() creates a 'struct scan_control' with ->target_mem_cgroup >>>>> set to NULL. >>>>> shrink_page_list() checks ->target_mem_cgroup using global_reclaim() and if >>>>> it is NULL, wait_on_page_writeback is *not* called. >>>>> >>>>> So we can only hit this deadlock if mem-cgroup limits are imposed on a >>>>> process which is using NFS - which is quite possible but probably not common. >>>>> >>>>> The fact that a dead-lock can happen only when memcg limits are imposed seems >>>>> very fragile. People aren't going to test that case much so there could well >>>>> be other deadlock possibilities lurking. >>>>> >>>>> Mel: might there be some other way we could get out of this deadlock? >>>>> Could the wait_on_page_writeback() in shrink_page_list() be made a timed-out >>>>> wait or something? Any other wait out of this deadlock other than setting >>>>> PF_MEMALLOC_NOIO everywhere? >>>> >>>> Not only the wait_on_page_writeback() cause the deadlock but also the >>>> next pageout()-> (mapping->a_ops->writepage), Trond's second patch fix >>>> this. So fix the wait_on_page_writeback is not enough to fix deadlock. >>> >>> Shortly before the only place that pageout() is called there is this code: >>> >>> if (page_is_file_cache(page) && >>> (!current_is_kswapd() || >>> !zone_is_reclaim_dirty(zone))) { >>> ..... >>> goto keep_locked; >>> >>> >>> So pageout() only gets called by kswapd() .... or for swap. swap-over-NFS is >>> already very cautious about memory allocations, and uses nfs_direct_IO, not >>> nfs_writepage. >>> >>> So nfs_writepage will never get called during direct reclaim. There is no >>> memory-allocate deadlock risk there. >> Yes, thanks for explaining this. >> But is it possible rpciod blocked somewhere by memory allocation using >> GFP_KERNEL and kswapd is trying to pageout nfs dirty pages and blocked >> by rpciod? > > I don't think so, no. > > Only 40% of memory (/proc/sys/vm/dirty_ratio) can be dirty. The direct > reclaim procedure will eventually find some non-dirty memory it can use. > If it cannot, and cannot write anything out to swap either, it will > eventually trigger the OOM killer. > > Direct reclaim shouldn't ever block indefinitely. It will sometimes wait for > a short while (e.g. congestion_wait()) but it should then push on until it > finds something it can do: free a clean page, write something to swap, or > kill a memory-hog with the OOM killer. That makes sense. Thanks. Junxiao. > > NeilBrown >
next prev parent reply other threads:[~2014-08-26 7:23 UTC|newest] Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top 2014-08-22 7:55 rpciod deadlock issue Junxiao Bi [not found] ` <53F6F772.6020708-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> 2014-08-22 22:49 ` [PATCH v2 1/2] SUNRPC: Fix memory reclaim deadlocks in rpciod Trond Myklebust 2014-08-22 22:49 ` Trond Myklebust 2014-08-22 22:49 ` [PATCH v2 2/2] NFS: Ensure that rpciod does not trigger reclaim writebacks Trond Myklebust [not found] ` <1408747772-37938-1-git-send-email-trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org> 2014-08-25 5:34 ` [PATCH v2 1/2] SUNRPC: Fix memory reclaim deadlocks in rpciod Junxiao Bi 2014-08-25 5:34 ` Junxiao Bi 2014-08-25 6:48 ` NeilBrown 2014-08-25 6:48 ` NeilBrown [not found] ` <20140825164852.50723141-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org> 2014-08-26 5:43 ` Junxiao Bi 2014-08-26 5:43 ` Junxiao Bi 2014-08-26 6:21 ` NeilBrown 2014-08-26 6:49 ` Junxiao Bi 2014-08-26 7:04 ` NeilBrown [not found] ` <20140826170410.20560764-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org> 2014-08-26 7:23 ` Junxiao Bi [this message] 2014-08-26 7:23 ` Junxiao Bi 2014-08-26 10:53 ` Mel Gorman 2014-08-26 10:53 ` Mel Gorman 2014-08-26 12:58 ` Trond Myklebust 2014-08-26 13:26 ` Mel Gorman [not found] ` <20140826132624.GU17696-Et1tbQHTxzrQT0dZR+AlfA@public.gmane.org> 2014-08-26 23:19 ` Johannes Weiner 2014-08-26 23:19 ` Johannes Weiner [not found] ` <20140826231938.GA13889-druUgvl0LCNAfugRpC6u6w@public.gmane.org> 2014-08-26 23:51 ` Trond Myklebust 2014-08-26 23:51 ` Trond Myklebust [not found] ` <CAHQdGtRPsVFVfph5OcsZk_+WYPPJ-MpE2myZfXAb3jq6fuM4zw-JsoAwUIsXosN+BqQ9rBEUg@public.gmane.org> 2014-08-27 0:00 ` Trond Myklebust 2014-08-27 0:00 ` Trond Myklebust 2014-08-27 15:36 ` Mel Gorman [not found] ` <20140827153644.GF12374-Et1tbQHTxzrQT0dZR+AlfA@public.gmane.org> 2014-08-27 16:15 ` Trond Myklebust 2014-08-27 16:15 ` Trond Myklebust 2014-08-28 8:30 ` Mel Gorman [not found] ` <20140828083053.GJ12374-Et1tbQHTxzrQT0dZR+AlfA@public.gmane.org> 2014-08-28 8:49 ` Junxiao Bi 2014-08-28 8:49 ` Junxiao Bi 2014-08-28 9:25 ` Mel Gorman 2014-09-04 13:54 ` Michal Hocko 2014-09-04 13:54 ` Michal Hocko 2014-09-09 2:33 ` NeilBrown 2014-09-10 13:48 ` Michal Hocko [not found] ` <20140910134842.GG25219-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org> 2014-09-10 23:57 ` NeilBrown 2014-09-10 23:57 ` NeilBrown [not found] ` <20140911095743.1ed87519-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org> 2014-09-11 8:50 ` Michal Hocko 2014-09-11 8:50 ` Michal Hocko [not found] ` <20140911085046.GC22042-2MMpYkNvuYDjFM9bn6wA6Q@public.gmane.org> 2014-09-11 10:53 ` NeilBrown 2014-09-11 10:53 ` NeilBrown 2014-08-27 1:43 ` NeilBrown 2014-08-27 1:43 ` NeilBrown 2014-08-25 6:05 ` rpciod deadlock issue NeilBrown 2014-08-25 6:05 ` NeilBrown [not found] ` <20140825160501.433b3e9e-wvvUuzkyo1EYVZTmpyfIwg@public.gmane.org> 2014-08-25 6:15 ` NeilBrown 2014-08-25 6:15 ` NeilBrown
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=53FC3600.9050202@oracle.com \ --to=junxiao.bi-qhclzuegtsvqt0dzr+alfa@public.gmane.org \ --cc=linux-fsdevel-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=linux-nfs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org \ --cc=mgorman-IBi9RG/b67k@public.gmane.org \ --cc=neilb-l3A5Bk7waGM@public.gmane.org \ --cc=trond.myklebust-7I+n7zu2hftEKMMhf/gKZA@public.gmane.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.