From: Dave Chinner <david@fromorbit.com> To: "Theodore Ts'o" <tytso@mit.edu>, Jan Kara <jack@suse.cz>, Mel Gorman <mgorman@suse.de>, linux-ext4@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>, Linux-MM <linux-mm@kvack.org>, Jiri Slaby <jslaby@suse.cz> Subject: Re: Excessive stall times on ext4 in 3.9-rc2 Date: Sat, 13 Apr 2013 11:23:41 +1000 [thread overview] Message-ID: <20130413012341.GJ30622@dastard> (raw) In-Reply-To: <20130412151952.GA4944@thunk.org> On Fri, Apr 12, 2013 at 11:19:52AM -0400, Theodore Ts'o wrote: > On Fri, Apr 12, 2013 at 02:50:42PM +1000, Dave Chinner wrote: > > > If that is the case, one possible solution that comes to mind would be > > > to mark buffer_heads that contain metadata with a flag, so that the > > > flusher thread can write them back at the same priority as reads. > > > > Ext4 is already using REQ_META for this purpose. > > We're using REQ_META | REQ_PRIO for reads, not writes. > > > I'm surprised that no-one has suggested "change the IO elevator" > > yet..... > > Well, testing to see if the stalls go away with the noop schedule is a > good thing to try just to validate the theory. Exactly. > The thing is, we do want to make ext4 work well with cfq, and > prioritizing non-readahead read requests ahead of data writeback does > make sense. The issue is with is that metadata writes going through > the block device could in some cases effectively cause a priority > inversion when what had previously been an asynchronous writeback > starts blocking a foreground, user-visible process. Here's the historic problem with CFQ: it's scheduling algorithms change from release to release, and so what you tune the filesystem to for this release is likely to cause different behaviour in a few releases time. We've had this problem time and time again with CFQ+XFS, so we stopped trying to "tune" to a particular elevator long ago. The best you can do it tag the Io as appropriately as possible (e.g. metadata with REQ_META, sync IO with ?_SYNC, etc), and then hope CFQ hasn't been broken since the last release.... > At least, that's the theory; we should confirm that this is indeed > what is causing the data stalls which Mel is reporting on HDD's before > we start figuring out how to fix this problem. *nod*. Cheers, Dave. -- Dave Chinner david@fromorbit.com
WARNING: multiple messages have this Message-ID (diff)
From: Dave Chinner <david@fromorbit.com> To: Theodore Ts'o <tytso@mit.edu>, Jan Kara <jack@suse.cz>, Mel Gorman <mgorman@suse.de>, linux-ext4@vger.kernel.org, LKML <linux-kernel@vger.kernel.org>, Linux-MM <linux-mm@kvack.org>, Jiri Slaby <jslaby@suse.cz> Subject: Re: Excessive stall times on ext4 in 3.9-rc2 Date: Sat, 13 Apr 2013 11:23:41 +1000 [thread overview] Message-ID: <20130413012341.GJ30622@dastard> (raw) In-Reply-To: <20130412151952.GA4944@thunk.org> On Fri, Apr 12, 2013 at 11:19:52AM -0400, Theodore Ts'o wrote: > On Fri, Apr 12, 2013 at 02:50:42PM +1000, Dave Chinner wrote: > > > If that is the case, one possible solution that comes to mind would be > > > to mark buffer_heads that contain metadata with a flag, so that the > > > flusher thread can write them back at the same priority as reads. > > > > Ext4 is already using REQ_META for this purpose. > > We're using REQ_META | REQ_PRIO for reads, not writes. > > > I'm surprised that no-one has suggested "change the IO elevator" > > yet..... > > Well, testing to see if the stalls go away with the noop schedule is a > good thing to try just to validate the theory. Exactly. > The thing is, we do want to make ext4 work well with cfq, and > prioritizing non-readahead read requests ahead of data writeback does > make sense. The issue is with is that metadata writes going through > the block device could in some cases effectively cause a priority > inversion when what had previously been an asynchronous writeback > starts blocking a foreground, user-visible process. Here's the historic problem with CFQ: it's scheduling algorithms change from release to release, and so what you tune the filesystem to for this release is likely to cause different behaviour in a few releases time. We've had this problem time and time again with CFQ+XFS, so we stopped trying to "tune" to a particular elevator long ago. The best you can do it tag the Io as appropriately as possible (e.g. metadata with REQ_META, sync IO with ?_SYNC, etc), and then hope CFQ hasn't been broken since the last release.... > At least, that's the theory; we should confirm that this is indeed > what is causing the data stalls which Mel is reporting on HDD's before > we start figuring out how to fix this problem. *nod*. Cheers, Dave. -- Dave Chinner david@fromorbit.com -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>
next prev parent reply other threads:[~2013-04-13 1:23 UTC|newest] Thread overview: 105+ messages / expand[flat|nested] mbox.gz Atom feed top 2013-04-02 14:27 Excessive stall times on ext4 in 3.9-rc2 Mel Gorman 2013-04-02 14:27 ` Mel Gorman 2013-04-02 15:00 ` Jiri Slaby 2013-04-02 15:00 ` Jiri Slaby 2013-04-02 15:03 ` Zheng Liu 2013-04-02 15:03 ` Zheng Liu 2013-04-02 15:15 ` Mel Gorman 2013-04-02 15:15 ` Mel Gorman 2013-04-02 15:06 ` Theodore Ts'o 2013-04-02 15:06 ` Theodore Ts'o 2013-04-02 15:14 ` Theodore Ts'o 2013-04-02 15:14 ` Theodore Ts'o 2013-04-02 18:19 ` Theodore Ts'o 2013-04-02 18:19 ` Theodore Ts'o 2013-04-07 21:59 ` Frank Ch. Eigler 2013-04-07 21:59 ` Frank Ch. Eigler 2013-04-08 8:36 ` Mel Gorman 2013-04-08 8:36 ` Mel Gorman 2013-04-08 10:52 ` Frank Ch. Eigler 2013-04-08 10:52 ` Frank Ch. Eigler 2013-04-08 11:01 ` Theodore Ts'o 2013-04-08 11:01 ` Theodore Ts'o 2013-04-03 10:19 ` Mel Gorman 2013-04-03 10:19 ` Mel Gorman 2013-04-03 12:05 ` Theodore Ts'o 2013-04-03 12:05 ` Theodore Ts'o 2013-04-03 15:15 ` Mel Gorman 2013-04-05 22:18 ` Jiri Slaby 2013-04-05 22:18 ` Jiri Slaby 2013-04-05 23:16 ` Theodore Ts'o 2013-04-05 23:16 ` Theodore Ts'o 2013-04-06 7:29 ` Jiri Slaby 2013-04-06 7:29 ` Jiri Slaby 2013-04-06 7:37 ` Jiri Slaby 2013-04-06 7:37 ` Jiri Slaby 2013-04-06 8:19 ` Jiri Slaby 2013-04-06 13:15 ` Theodore Ts'o 2013-04-06 13:15 ` Theodore Ts'o 2013-04-10 10:56 ` Mel Gorman 2013-04-10 10:56 ` Mel Gorman 2013-04-10 13:12 ` Theodore Ts'o 2013-04-10 13:12 ` Theodore Ts'o 2013-04-11 17:04 ` Mel Gorman 2013-04-11 17:04 ` Mel Gorman 2013-04-11 18:35 ` Theodore Ts'o 2013-04-11 18:35 ` Theodore Ts'o 2013-04-11 21:33 ` Jan Kara 2013-04-11 21:33 ` Jan Kara 2013-04-12 2:57 ` Theodore Ts'o 2013-04-12 2:57 ` Theodore Ts'o 2013-04-12 4:50 ` Dave Chinner 2013-04-12 4:50 ` Dave Chinner 2013-04-12 15:19 ` Theodore Ts'o 2013-04-12 15:19 ` Theodore Ts'o 2013-04-13 1:23 ` Dave Chinner [this message] 2013-04-13 1:23 ` Dave Chinner 2013-04-22 14:38 ` Mel Gorman 2013-04-22 14:38 ` Mel Gorman 2013-04-22 22:42 ` Jeff Moyer 2013-04-22 22:42 ` Jeff Moyer 2013-04-23 0:02 ` Theodore Ts'o 2013-04-23 0:02 ` Theodore Ts'o 2013-04-23 9:31 ` Jan Kara 2013-04-23 9:31 ` Jan Kara 2013-04-23 14:01 ` Mel Gorman 2013-04-23 14:01 ` Mel Gorman 2013-04-24 19:09 ` Jeff Moyer 2013-04-24 19:09 ` Jeff Moyer 2013-04-25 12:21 ` Mel Gorman 2013-04-25 12:21 ` Mel Gorman 2013-04-12 9:47 ` Mel Gorman 2013-04-12 9:47 ` Mel Gorman 2013-04-21 0:05 ` Theodore Ts'o 2013-04-21 0:05 ` Theodore Ts'o 2013-04-21 0:07 ` [PATCH 1/3] ext4: mark all metadata I/O with REQ_META Theodore Ts'o 2013-04-21 0:07 ` Theodore Ts'o 2013-04-21 0:07 ` [PATCH 2/3] buffer: add BH_Prio and BH_Meta flags Theodore Ts'o 2013-04-21 0:07 ` Theodore Ts'o 2013-04-21 0:07 ` [PATCH 3/3] ext4: mark metadata blocks using bh flags Theodore Ts'o 2013-04-21 0:07 ` Theodore Ts'o 2013-04-21 6:09 ` Jiri Slaby 2013-04-21 6:09 ` Jiri Slaby 2013-04-21 6:09 ` Jiri Slaby 2013-04-21 19:55 ` Theodore Ts'o 2013-04-21 19:55 ` Theodore Ts'o 2013-04-21 19:55 ` Theodore Ts'o 2013-04-21 20:48 ` [PATCH 3/3 -v2] " Theodore Ts'o 2013-04-21 20:48 ` Theodore Ts'o 2013-04-21 20:48 ` Theodore Ts'o 2013-04-22 12:06 ` [PATCH 1/3] ext4: mark all metadata I/O with REQ_META Zheng Liu 2013-04-22 12:06 ` Zheng Liu 2013-04-23 15:33 ` Excessive stall times on ext4 in 3.9-rc2 Mel Gorman 2013-04-23 15:33 ` Mel Gorman 2013-04-23 15:50 ` Theodore Ts'o 2013-04-23 15:50 ` Theodore Ts'o 2013-04-23 16:13 ` Mel Gorman 2013-04-23 16:13 ` Mel Gorman 2013-04-12 10:18 ` Tvrtko Ursulin 2013-04-12 10:18 ` Tvrtko Ursulin 2013-04-12 9:45 ` Mel Gorman 2013-04-12 9:45 ` Mel Gorman 2013-04-02 23:16 ` Theodore Ts'o 2013-04-02 23:16 ` Theodore Ts'o 2013-04-03 15:22 ` Mel Gorman 2013-04-03 15:22 ` Mel Gorman
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20130413012341.GJ30622@dastard \ --to=david@fromorbit.com \ --cc=jack@suse.cz \ --cc=jslaby@suse.cz \ --cc=linux-ext4@vger.kernel.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mgorman@suse.de \ --cc=tytso@mit.edu \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.