From: Dave Chinner <david@fromorbit.com>
To: Vivek Goyal <vgoyal@redhat.com>
Cc: linux-kernel@vger.kernel.org, jaxboe@fusionio.com,
linux-fsdevel@vger.kernel.org, andrea@betterlinux.com,
linux-ext4@vger.kernel.org
Subject: Re: fsync serialization on ext4 with blkio throttling (Was: Re: [PATCH 0/8][V2] blk-throttle: Throttle buffered WRITEs in balance_dirty_pages())
Date: Fri, 1 Jul 2011 10:16:25 +1000 [thread overview]
Message-ID: <20110701001625.GL561@dastard> (raw)
In-Reply-To: <20110630204432.GL27889@redhat.com>
On Thu, Jun 30, 2011 at 04:44:32PM -0400, Vivek Goyal wrote:
> On Thu, Jun 30, 2011 at 04:04:59PM -0400, Vivek Goyal wrote:
>
> [..]
> > Dave,
> >
> > Just another example where serialization is taking place with ext4.
> >
> > I created a group with 1MB/s write limit and ran tedso's fsync tester
> > program with little modification. I used write() system call instead
> > of pwrite() so that file size grows. This program basically writes
> > 1MB of data and then fsync's it and then measures the fsync time.
> >
> > I ran two instances of prgram in two groups on two separate files. One
> > instances is throttled to 1MB/s and other is in root group unthrottled.
> >
> > Unthrottled program gets serialized behind throttled one. Following
> > are fsync times.
> >
> > Throttled instance Unthrottled Instance
> > ------------------ --------------------
> > fsync time: 1.0051 fsync time: 1.0067
> > fsync time: 1.0049 fsync time: 1.0075
> > fsync time: 1.0048 fsync time: 1.0063
> > fsync time: 1.0073 fsync time: 1.0062
> > fsync time: 1.0070 fsync time: 1.0078
> > fsync time: 1.0032 fsync time: 1.0049
> > fsync time: 0.0154 fsync time: 1.0068
> > fsync time: 0.0137 fsync time: 1.0048
> >
> > Without any throttling both the instances do fine
> > -------------------------------------------------
> > Throttled instance Unthrottled Instance
> > ------------------ --------------------
> > fsync time: 0.0139 fsync time: 0.0162
> > fsync time: 0.0132 fsync time: 0.0156
> > fsync time: 0.0149 fsync time: 0.0169
> > fsync time: 0.0165 fsync time: 0.0152
> > fsync time: 0.0188 fsync time: 0.0135
> > fsync time: 0.0137 fsync time: 0.0142
> > fsync time: 0.0148 fsync time: 0.0149
> > fsync time: 0.0168 fsync time: 0.0163
> > fsync time: 0.0153 fsync time: 0.0143
> >
> > So when we are inreasing the size of file and fsyncing it, other
> > unthrottled instances of similar activities will get throttled
> > behind it.
> >
> > IMHO, this is a problem and should be fixed. If filesystem can fix it great.
> > But if not, then we should consider the option of throttling buffered writes
> > in balance_dirty_pages().
>
> XFS seems to be doing well for this particular test. Unthrottled
> fsyncer does not get serialized behind throttled one.
>
> Throttled instance Unthrottled Instance
> ------------------ --------------------
> fsync time: 1.0511 fsync time: 0.0204
> fsync time: 1.0486 fsync time: 0.0260
> fsync time: 1.0445 fsync time: 0.0260
> fsync time: 1.0485 fsync time: 0.0260
> fsync time: 1.0446 fsync time: 0.0260
> fsync time: 1.2157 fsync time: 0.0260
> fsync time: 1.0446 fsync time: 0.0300
> fsync time: 1.0484 fsync time: 0.0340
> fsync time: 1.0446 fsync time: 0.0221
> fsync time: 1.0486 fsync time: 0.0340
> fsync time: 1.0406 fsync time: 0.0340
And you've just illustrated my point better than I did - that
different filesytsems will suffer from different problems, but some
filesytsems will work better than others out of the box. :)
Of course, there's no guarantee XFS will remain this way - if you
want us to care about regressions of this sort at all, you need to
encapsulate all this behaviour in a set of automated tests.
Preferrably within the xfstests infrastructure because that now has
fairly wide usage within the fs dev community....
Cheers,
Dave.
--
Dave Chinner
david@fromorbit.com
prev parent reply other threads:[~2011-07-01 0:16 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-06-28 15:35 [PATCH 0/8][V2] blk-throttle: Throttle buffered WRITEs in balance_dirty_pages() Vivek Goyal
2011-06-28 15:35 ` [PATCH 1/8] blk-throttle: convert wait routines to return jiffies to wait Vivek Goyal
2011-06-28 15:35 ` [PATCH 2/8] blk-throttle: do not enforce first queued bio check in tg_wait_dispatch Vivek Goyal
2011-06-28 15:35 ` [PATCH 3/8] blk-throttle: use io size and direction as parameters to wait routines Vivek Goyal
2011-06-28 15:35 ` [PATCH 4/8] blk-throttle: specify number of ios during dispatch update Vivek Goyal
2011-06-28 15:35 ` [PATCH 5/8] blk-throttle: get rid of extend slice trace message Vivek Goyal
2011-06-28 15:35 ` [PATCH 6/8] blk-throttle: core logic to throttle task while dirtying pages Vivek Goyal
2011-06-29 9:30 ` Andrea Righi
2011-06-29 15:25 ` Andrea Righi
2011-06-29 20:03 ` Vivek Goyal
2011-06-28 15:35 ` [PATCH 7/8] blk-throttle: do not throttle writes at device level except direct io Vivek Goyal
2011-06-28 15:35 ` [PATCH 8/8] blk-throttle: enable throttling of task while dirtying pages Vivek Goyal
2011-06-30 14:52 ` Andrea Righi
2011-06-30 15:06 ` Andrea Righi
2011-06-30 17:14 ` Vivek Goyal
2011-06-30 21:22 ` Andrea Righi
2011-06-28 16:21 ` [PATCH 0/8][V2] blk-throttle: Throttle buffered WRITEs in balance_dirty_pages() Andrea Righi
2011-06-28 17:06 ` Vivek Goyal
2011-06-28 17:39 ` Andrea Righi
2011-06-29 16:05 ` Andrea Righi
2011-06-29 20:04 ` Vivek Goyal
2011-06-29 0:42 ` Dave Chinner
2011-06-29 1:53 ` Vivek Goyal
2011-06-30 20:04 ` fsync serialization on ext4 with blkio throttling (Was: Re: [PATCH 0/8][V2] blk-throttle: Throttle buffered WRITEs in balance_dirty_pages()) Vivek Goyal
2011-06-30 20:44 ` Vivek Goyal
2011-07-01 0:16 ` Dave Chinner [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20110701001625.GL561@dastard \
--to=david@fromorbit.com \
--cc=andrea@betterlinux.com \
--cc=jaxboe@fusionio.com \
--cc=linux-ext4@vger.kernel.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=vgoyal@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.