From: Andrew Morton <akpm@linux-foundation.org> To: Neil Brown <neilb@suse.de> Cc: Wu Fengguang <fengguang.wu@intel.com>, Rik van Riel <riel@redhat.com>, KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>, KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "linux-mm@kvack.org" <linux-mm@kvack.org>, "Li, Shaohua" <shaohua.li@intel.com> Subject: Re: Deadlock possibly caused by too_many_isolated. Date: Mon, 18 Oct 2010 15:41:37 -0700 [thread overview] Message-ID: <20101018154137.90f5325f.akpm@linux-foundation.org> (raw) In-Reply-To: <20101019093142.509d6947@notabene> On Tue, 19 Oct 2010 09:31:42 +1100 Neil Brown <neilb@suse.de> wrote: > On Mon, 18 Oct 2010 14:58:59 -0700 > Andrew Morton <akpm@linux-foundation.org> wrote: > > > On Tue, 19 Oct 2010 00:15:04 +0800 > > Wu Fengguang <fengguang.wu@intel.com> wrote: > > > > > Neil find that if too_many_isolated() returns true while performing > > > direct reclaim we can end up waiting for other threads to complete their > > > direct reclaim. If those threads are allowed to enter the FS or IO to > > > free memory, but this thread is not, then it is possible that those > > > threads will be waiting on this thread and so we get a circular > > > deadlock. > > > > > > some task enters direct reclaim with GFP_KERNEL > > > => too_many_isolated() false > > > => vmscan and run into dirty pages > > > => pageout() > > > => take some FS lock > > > => fs/block code does GFP_NOIO allocation > > > => enter direct reclaim again > > > => too_many_isolated() true > > > => waiting for others to progress, however the other > > > tasks may be circular waiting for the FS lock.. I'm assuming that the last four "=>"'s here should have been indented another stop. > > > The fix is to let !__GFP_IO and !__GFP_FS direct reclaims enjoy higher > > > priority than normal ones, by honouring them higher throttle threshold. > > > > > > Now !GFP_IOFS reclaims won't be waiting for GFP_IOFS reclaims to > > > progress. They will be blocked only when there are too many concurrent > > > !GFP_IOFS reclaims, however that's very unlikely because the IO-less > > > direct reclaims is able to progress much more faster, and they won't > > > deadlock each other. The threshold is raised high enough for them, so > > > that there can be sufficient parallel progress of !GFP_IOFS reclaims. > > > > I'm not sure that this is really a full fix. Torsten's analysis does > > appear to point at the real bug: raid1 has code paths which allocate > > more than a single element from a mempool without starting IO against > > previous elements. > > ... point at "a" real bug. > > I think there are two bugs here. > The raid1 bug that Torsten mentions is certainly real (and has been around > for an embarrassingly long time). > The bug that I identified in too_many_isolated is also a real bug and can be > triggered without md/raid1 in the mix. > So this is not a 'full fix' for every bug in the kernel :-), but it could > well be a full fix for this particular bug. > Can we just delete the too_many_isolated() logic? (Crappy comment describes what the code does but not why it does it).
WARNING: multiple messages have this Message-ID (diff)
From: Andrew Morton <akpm@linux-foundation.org> To: Neil Brown <neilb@suse.de> Cc: Wu Fengguang <fengguang.wu@intel.com>, Rik van Riel <riel@redhat.com>, KOSAKI Motohiro <kosaki.motohiro@jp.fujitsu.com>, KAMEZAWA Hiroyuki <kamezawa.hiroyu@jp.fujitsu.com>, "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>, "linux-mm@kvack.org" <linux-mm@kvack.org>, "Li, Shaohua" <shaohua.li@intel.com> Subject: Re: Deadlock possibly caused by too_many_isolated. Date: Mon, 18 Oct 2010 15:41:37 -0700 [thread overview] Message-ID: <20101018154137.90f5325f.akpm@linux-foundation.org> (raw) In-Reply-To: <20101019093142.509d6947@notabene> On Tue, 19 Oct 2010 09:31:42 +1100 Neil Brown <neilb@suse.de> wrote: > On Mon, 18 Oct 2010 14:58:59 -0700 > Andrew Morton <akpm@linux-foundation.org> wrote: > > > On Tue, 19 Oct 2010 00:15:04 +0800 > > Wu Fengguang <fengguang.wu@intel.com> wrote: > > > > > Neil find that if too_many_isolated() returns true while performing > > > direct reclaim we can end up waiting for other threads to complete their > > > direct reclaim. If those threads are allowed to enter the FS or IO to > > > free memory, but this thread is not, then it is possible that those > > > threads will be waiting on this thread and so we get a circular > > > deadlock. > > > > > > some task enters direct reclaim with GFP_KERNEL > > > => too_many_isolated() false > > > => vmscan and run into dirty pages > > > => pageout() > > > => take some FS lock > > > => fs/block code does GFP_NOIO allocation > > > => enter direct reclaim again > > > => too_many_isolated() true > > > => waiting for others to progress, however the other > > > tasks may be circular waiting for the FS lock.. I'm assuming that the last four "=>"'s here should have been indented another stop. > > > The fix is to let !__GFP_IO and !__GFP_FS direct reclaims enjoy higher > > > priority than normal ones, by honouring them higher throttle threshold. > > > > > > Now !GFP_IOFS reclaims won't be waiting for GFP_IOFS reclaims to > > > progress. They will be blocked only when there are too many concurrent > > > !GFP_IOFS reclaims, however that's very unlikely because the IO-less > > > direct reclaims is able to progress much more faster, and they won't > > > deadlock each other. The threshold is raised high enough for them, so > > > that there can be sufficient parallel progress of !GFP_IOFS reclaims. > > > > I'm not sure that this is really a full fix. Torsten's analysis does > > appear to point at the real bug: raid1 has code paths which allocate > > more than a single element from a mempool without starting IO against > > previous elements. > > ... point at "a" real bug. > > I think there are two bugs here. > The raid1 bug that Torsten mentions is certainly real (and has been around > for an embarrassingly long time). > The bug that I identified in too_many_isolated is also a real bug and can be > triggered without md/raid1 in the mix. > So this is not a 'full fix' for every bug in the kernel :-), but it could > well be a full fix for this particular bug. > Can we just delete the too_many_isolated() logic? (Crappy comment describes what the code does but not why it does it). -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2010-10-18 22:42 UTC|newest] Thread overview: 116+ messages / expand[flat|nested] mbox.gz Atom feed top 2010-09-14 23:11 Deadlock possibly caused by too_many_isolated Neil Brown 2010-09-14 23:11 ` Neil Brown 2010-09-15 0:30 ` Rik van Riel 2010-09-15 0:30 ` Rik van Riel 2010-09-15 2:23 ` Neil Brown 2010-09-15 2:23 ` Neil Brown 2010-09-15 2:37 ` Wu Fengguang 2010-09-15 2:37 ` Wu Fengguang 2010-09-15 2:54 ` Wu Fengguang 2010-09-15 2:54 ` Wu Fengguang 2010-09-15 3:06 ` Wu Fengguang 2010-09-15 3:06 ` Wu Fengguang 2010-09-15 3:13 ` Wu Fengguang 2010-09-15 3:13 ` Wu Fengguang 2010-09-15 3:18 ` Shaohua Li 2010-09-15 3:18 ` Shaohua Li 2010-09-15 3:31 ` Wu Fengguang 2010-09-15 3:31 ` Wu Fengguang 2010-09-15 3:17 ` Neil Brown 2010-09-15 3:17 ` Neil Brown 2010-09-15 3:47 ` Wu Fengguang 2010-09-15 3:47 ` Wu Fengguang 2010-09-15 8:28 ` Wu Fengguang 2010-09-15 8:28 ` Wu Fengguang 2010-09-15 8:44 ` Neil Brown 2010-09-15 8:44 ` Neil Brown 2010-10-18 4:14 ` Neil Brown 2010-10-18 4:14 ` Neil Brown 2010-10-18 5:04 ` KOSAKI Motohiro 2010-10-18 5:04 ` KOSAKI Motohiro 2010-10-18 10:58 ` Torsten Kaiser 2010-10-18 10:58 ` Torsten Kaiser 2010-10-18 23:11 ` Neil Brown 2010-10-18 23:11 ` Neil Brown 2010-10-19 8:43 ` Torsten Kaiser 2010-10-19 8:43 ` Torsten Kaiser 2010-10-19 10:06 ` Torsten Kaiser 2010-10-19 10:06 ` Torsten Kaiser 2010-10-20 5:57 ` Wu Fengguang 2010-10-20 5:57 ` Wu Fengguang 2010-10-20 7:05 ` KOSAKI Motohiro 2010-10-20 7:05 ` KOSAKI Motohiro 2010-10-20 9:27 ` Wu Fengguang 2010-10-20 9:27 ` Wu Fengguang 2010-10-20 13:03 ` Jens Axboe 2010-10-20 13:03 ` Jens Axboe 2010-10-22 5:37 ` Wu Fengguang 2010-10-22 5:37 ` Wu Fengguang 2010-10-22 8:07 ` Wu Fengguang 2010-10-22 8:07 ` Wu Fengguang 2010-10-22 8:09 ` Jens Axboe 2010-10-22 8:09 ` Jens Axboe 2010-10-24 16:52 ` Wu Fengguang 2010-10-24 16:52 ` Wu Fengguang 2010-10-25 6:40 ` Neil Brown 2010-10-25 6:40 ` Neil Brown 2010-10-25 7:26 ` Wu Fengguang 2010-10-25 7:26 ` Wu Fengguang 2010-10-20 7:25 ` Torsten Kaiser 2010-10-20 7:25 ` Torsten Kaiser 2010-10-20 9:01 ` Wu Fengguang 2010-10-20 9:01 ` Wu Fengguang 2010-10-20 10:07 ` Torsten Kaiser 2010-10-20 10:07 ` Torsten Kaiser 2010-10-20 14:23 ` Minchan Kim 2010-10-20 14:23 ` Minchan Kim 2010-10-20 15:35 ` Torsten Kaiser 2010-10-20 15:35 ` Torsten Kaiser 2010-10-20 23:31 ` Minchan Kim 2010-10-20 23:31 ` Minchan Kim 2010-10-18 16:15 ` Wu Fengguang 2010-10-18 16:15 ` Wu Fengguang 2010-10-18 21:58 ` Andrew Morton 2010-10-18 21:58 ` Andrew Morton 2010-10-18 22:31 ` Neil Brown 2010-10-18 22:31 ` Neil Brown 2010-10-18 22:41 ` Andrew Morton [this message] 2010-10-18 22:41 ` Andrew Morton 2010-10-19 0:57 ` KOSAKI Motohiro 2010-10-19 0:57 ` KOSAKI Motohiro 2010-10-19 1:15 ` Minchan Kim 2010-10-19 1:15 ` Minchan Kim 2010-10-19 1:21 ` KOSAKI Motohiro 2010-10-19 1:21 ` KOSAKI Motohiro 2010-10-19 1:32 ` Minchan Kim 2010-10-19 1:32 ` Minchan Kim 2010-10-19 2:03 ` KOSAKI Motohiro 2010-10-19 2:03 ` KOSAKI Motohiro 2010-10-19 2:16 ` Minchan Kim 2010-10-19 2:16 ` Minchan Kim 2010-10-19 2:54 ` KOSAKI Motohiro 2010-10-19 2:54 ` KOSAKI Motohiro 2010-10-19 2:35 ` Wu Fengguang 2010-10-19 2:35 ` Wu Fengguang 2010-10-19 2:52 ` Minchan Kim 2010-10-19 2:52 ` Minchan Kim 2010-10-19 3:05 ` Wu Fengguang 2010-10-19 3:05 ` Wu Fengguang 2010-10-19 3:09 ` Minchan Kim 2010-10-19 3:09 ` Minchan Kim 2010-10-19 3:13 ` KOSAKI Motohiro 2010-10-19 3:13 ` KOSAKI Motohiro 2010-10-19 5:11 ` Minchan Kim 2010-10-19 5:11 ` Minchan Kim 2010-10-19 3:21 ` Shaohua Li 2010-10-19 3:21 ` Shaohua Li 2010-10-19 7:15 ` Shaohua Li 2010-10-19 7:15 ` Shaohua Li 2010-10-19 7:34 ` Minchan Kim 2010-10-19 7:34 ` Minchan Kim 2010-10-19 2:24 ` Wu Fengguang 2010-10-19 2:24 ` Wu Fengguang 2010-10-19 2:37 ` KOSAKI Motohiro 2010-10-19 2:37 ` KOSAKI Motohiro 2010-10-19 2:37 ` Minchan Kim 2010-10-19 2:37 ` Minchan Kim
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20101018154137.90f5325f.akpm@linux-foundation.org \ --to=akpm@linux-foundation.org \ --cc=fengguang.wu@intel.com \ --cc=kamezawa.hiroyu@jp.fujitsu.com \ --cc=kosaki.motohiro@jp.fujitsu.com \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=neilb@suse.de \ --cc=riel@redhat.com \ --cc=shaohua.li@intel.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.