All of lore.kernel.org
 help / color / mirror / Atom feed
* [RESEND][PATCH v2] block: remove plugging at buffered write time
@ 2012-04-08  1:06 Wu Fengguang
  2012-04-09 14:34 ` Jeff Moyer
  0 siblings, 1 reply; 16+ messages in thread
From: Wu Fengguang @ 2012-04-08  1:06 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Jens Axboe, linux-fsdevel, LKML, Dave Chinner, Christoph Hellwig,
	Jan Kara, Chris Mason, Shaohua Li

Buffered write(2) is not directly tied to IO, so it's not suitable to
handle plug in generic_file_aio_write().

Also moves unplugging for direct I/O from around ->direct_IO() down to
do_blockdev_direct_IO().

Note that plugging for O_SYNC writes is also removed. The user may pass
arbitrary @size arguments, which may be much larger than the preferable
I/O size, or may cross extent/device boundaries. Let the lower layers
handle the plugging. Otherwise the plugging code here will turn the
low level plugging into no-ops.

CC: Jens Axboe <axboe@kernel.dk>
CC: Shaohua Li <shli@fusionio.com>
Signed-off-by: Wu Fengguang <fengguang.wu@intel.com>
---
 fs/direct-io.c |    5 +++++
 fs/sync.c      |   11 ++++++++++-
 mm/filemap.c   |    7 -------
 3 files changed, 15 insertions(+), 8 deletions(-)

--- linux-next.orig/mm/filemap.c	2012-02-08 19:33:29.000000000 +0800
+++ linux-next/mm/filemap.c	2012-02-09 15:59:47.000000000 +0800
@@ -1421,12 +1421,8 @@ generic_file_aio_read(struct kiocb *iocb
 			retval = filemap_write_and_wait_range(mapping, pos,
 					pos + iov_length(iov, nr_segs) - 1);
 			if (!retval) {
-				struct blk_plug plug;
-
-				blk_start_plug(&plug);
 				retval = mapping->a_ops->direct_IO(READ, iocb,
 							iov, pos, nr_segs);
-				blk_finish_plug(&plug);
 			}
 			if (retval > 0) {
 				*ppos = pos + retval;
@@ -2610,13 +2606,11 @@ ssize_t generic_file_aio_write(struct ki
 {
 	struct file *file = iocb->ki_filp;
 	struct inode *inode = file->f_mapping->host;
-	struct blk_plug plug;
 	ssize_t ret;
 
 	BUG_ON(iocb->ki_pos != pos);
 
 	mutex_lock(&inode->i_mutex);
-	blk_start_plug(&plug);
 	ret = __generic_file_aio_write(iocb, iov, nr_segs, &iocb->ki_pos);
 	mutex_unlock(&inode->i_mutex);
 
@@ -2627,7 +2621,6 @@ ssize_t generic_file_aio_write(struct ki
 		if (err < 0 && ret > 0)
 			ret = err;
 	}
-	blk_finish_plug(&plug);
 	return ret;
 }
 EXPORT_SYMBOL(generic_file_aio_write);
--- linux-next.orig/fs/direct-io.c	2012-02-08 19:33:29.000000000 +0800
+++ linux-next/fs/direct-io.c	2012-02-09 16:03:04.000000000 +0800
@@ -1106,6 +1106,7 @@ do_blockdev_direct_IO(int rw, struct kio
 	unsigned long user_addr;
 	size_t bytes;
 	struct buffer_head map_bh = { 0, };
+	struct blk_plug plug;
 
 	if (rw & WRITE)
 		rw = WRITE_ODIRECT;
@@ -1221,6 +1222,8 @@ do_blockdev_direct_IO(int rw, struct kio
 				PAGE_SIZE - user_addr / PAGE_SIZE);
 	}
 
+	blk_start_plug(&plug);
+
 	for (seg = 0; seg < nr_segs; seg++) {
 		user_addr = (unsigned long)iov[seg].iov_base;
 		sdio.size += bytes = iov[seg].iov_len;
@@ -1279,6 +1282,8 @@ do_blockdev_direct_IO(int rw, struct kio
 	if (sdio.bio)
 		dio_bio_submit(dio, &sdio);
 
+	blk_finish_plug(&plug);
+
 	/*
 	 * It is possible that, we return short IO due to end of file.
 	 * In that case, we need to release all the pages we got hold on.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [RESEND][PATCH v2] block: remove plugging at buffered write time
  2012-04-08  1:06 [RESEND][PATCH v2] block: remove plugging at buffered write time Wu Fengguang
@ 2012-04-09 14:34 ` Jeff Moyer
  2012-04-11 23:13   ` Andrew Morton
  0 siblings, 1 reply; 16+ messages in thread
From: Jeff Moyer @ 2012-04-09 14:34 UTC (permalink / raw)
  To: Wu Fengguang
  Cc: Andrew Morton, Jens Axboe, linux-fsdevel, LKML, Dave Chinner,
	Christoph Hellwig, Jan Kara, Chris Mason, Shaohua Li

Wu Fengguang <fengguang.wu@intel.com> writes:

> Buffered write(2) is not directly tied to IO, so it's not suitable to
> handle plug in generic_file_aio_write().
[snip, moved]
> Note that plugging for O_SYNC writes is also removed. The user may pass
> arbitrary @size arguments, which may be much larger than the preferable
> I/O size, or may cross extent/device boundaries. Let the lower layers
> handle the plugging. Otherwise the plugging code here will turn the
> low level plugging into no-ops.

I assume you have some numbers to back this up, right?  Care to share
those?

> Also moves unplugging for direct I/O from around ->direct_IO() down to
> do_blockdev_direct_IO().

This could be handled in a separate patch.  Acked-by: Jeff Moyer
<jmoyer@redhat.com> for this part.

Cheers,
Jeff

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [RESEND][PATCH v2] block: remove plugging at buffered write time
  2012-04-09 14:34 ` Jeff Moyer
@ 2012-04-11 23:13   ` Andrew Morton
  2012-04-12  1:32     ` Fengguang Wu
  0 siblings, 1 reply; 16+ messages in thread
From: Andrew Morton @ 2012-04-11 23:13 UTC (permalink / raw)
  To: Jeff Moyer
  Cc: Wu Fengguang, Jens Axboe, linux-fsdevel, LKML, Dave Chinner,
	Christoph Hellwig, Jan Kara, Chris Mason, Shaohua Li

On Mon, 09 Apr 2012 10:34:44 -0400
Jeff Moyer <jmoyer@redhat.com> wrote:

> Wu Fengguang <fengguang.wu@intel.com> writes:
> 
> > Buffered write(2) is not directly tied to IO, so it's not suitable to
> > handle plug in generic_file_aio_write().
> [snip, moved]
> > Note that plugging for O_SYNC writes is also removed. The user may pass
> > arbitrary @size arguments, which may be much larger than the preferable
> > I/O size, or may cross extent/device boundaries. Let the lower layers
> > handle the plugging. Otherwise the plugging code here will turn the
> > low level plugging into no-ops.
> 
> I assume you have some numbers to back this up, right?  Care to share
> those?

Yes please.

We've broken this stuff a few times recently - we should review and
test carefully.

> > Also moves unplugging for direct I/O from around ->direct_IO() down to
> > do_blockdev_direct_IO().
> 
> This could be handled in a separate patch.

I think that would be better.

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [RESEND][PATCH v2] block: remove plugging at buffered write time
  2012-04-11 23:13   ` Andrew Morton
@ 2012-04-12  1:32     ` Fengguang Wu
  2012-04-12  2:20       ` Fengguang Wu
  2012-05-06  9:58       ` Fengguang Wu
  0 siblings, 2 replies; 16+ messages in thread
From: Fengguang Wu @ 2012-04-12  1:32 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Jeff Moyer, Jens Axboe, linux-fsdevel, LKML, Dave Chinner,
	Christoph Hellwig, Jan Kara, Chris Mason, Shaohua Li

On Wed, Apr 11, 2012 at 04:13:44PM -0700, Andrew Morton wrote:
> On Mon, 09 Apr 2012 10:34:44 -0400
> Jeff Moyer <jmoyer@redhat.com> wrote:
> 
> > Wu Fengguang <fengguang.wu@intel.com> writes:
> > 
> > > Buffered write(2) is not directly tied to IO, so it's not suitable to
> > > handle plug in generic_file_aio_write().
> > [snip, moved]
> > > Note that plugging for O_SYNC writes is also removed. The user may pass
> > > arbitrary @size arguments, which may be much larger than the preferable
> > > I/O size, or may cross extent/device boundaries. Let the lower layers
> > > handle the plugging. Otherwise the plugging code here will turn the
> > > low level plugging into no-ops.
> > 
> > I assume you have some numbers to back this up, right?  Care to share
> > those?
> 
> Yes please.
> 
> We've broken this stuff a few times recently - we should review and
> test carefully.

Yes sure. Last time I posted the patch, I did some tests and found no
performance changes. Now for 3.3, the tests started days ago have not
finished now (partly because it is stalled for quite long time due to
unknown reason). The now-available numbers for bs=4k dd's look fine.
The pending tests are for bs=1M dd's and some random fio workloads.

$ ./compare bay/*/*-{3.3.0,3.3.0-plug+}
                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                  196.94        -0.0%       196.86  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  196.67        +0.2%       196.99  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  134.34        -0.1%       134.26  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                  141.75        +0.1%       141.92  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  166.74        +0.1%       166.89  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  165.90        -0.1%       165.69  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  186.83        +0.9%       188.60  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                  193.30        -0.1%       193.04  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  168.34        +0.1%       168.48  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  193.43        -0.3%       192.91  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  197.59        -0.2%       197.14  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  197.09        -0.1%       196.90  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                  197.20        +0.1%       197.48  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  139.07        -0.6%       138.24  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                  157.10        -0.1%       156.99  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  113.36        -0.6%       112.73  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  159.88        -0.5%       159.09  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  191.53        +0.3%       192.09  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   83.65        +0.4%        83.99  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  163.12        +0.3%       163.54  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  194.25        +0.3%       194.88  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  194.98        +0.2%       195.34  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  194.78        +0.1%       194.96  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  133.07        -0.1%       132.95  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  136.90        -0.8%       135.82  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  167.61        -1.2%       165.65  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  188.04        +1.1%       190.10  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                  187.74        -0.3%       187.27  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  169.01        +0.5%       169.80  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  192.03        +0.0%       192.10  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  196.06        +0.0%       196.14  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  189.04        -0.6%       187.92  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                  183.23        -0.6%       182.05  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  137.19        -0.1%       137.12  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                  140.61        -0.1%       140.54  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  114.32        -2.3%       111.71  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  162.42        -0.1%       162.31  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  183.84        -0.9%       182.19  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   89.10        -0.3%        88.84  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  162.48        -0.2%       162.20  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  188.15        +0.5%       189.12  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   97.79        -0.0%        97.76  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   97.69        +0.0%        97.71  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   69.84        -1.1%        69.05  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   76.62        -0.0%        76.59  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   83.97        -0.0%        83.95  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   94.87        -0.1%        94.79  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                   96.87        +0.6%        97.46  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                   84.57        -0.1%        84.50  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   95.51        +0.2%        95.67  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   97.44        -0.3%        97.12  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   97.69        -0.1%        97.56  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   97.75        +0.1%        97.87  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   66.68        -0.3%        66.45  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   76.23        -0.7%        75.73  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   59.36        +0.9%        59.93  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   82.63        -0.1%        82.56  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   95.77        +0.1%        95.86  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   43.77        -0.1%        43.71  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   84.24        +0.3%        84.48  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   97.81        -0.1%        97.76  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   99.27        -0.0%        99.25  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                   99.39        +0.0%        99.39  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                   69.75        +0.2%        69.86  bay/thresh=1000M/ext3-100dd-1-3.3.0
                   71.79        -0.0%        71.76  bay/thresh=1000M/ext3-10dd-1-3.3.0
                   80.00        -0.0%        79.99  bay/thresh=1000M/ext3-1dd-1-3.3.0
                   89.38        -0.1%        89.30  bay/thresh=1000M/ext4-100dd-1-3.3.0
                   96.43        +0.2%        96.63  bay/thresh=1000M/ext4-10dd-1-3.3.0
                   98.57        -0.3%        98.28  bay/thresh=1000M/ext4-1dd-1-3.3.0
                   93.97        -0.2%        93.76  bay/thresh=1000M/xfs-100dd-1-3.3.0
                   97.51        +0.1%        97.57  bay/thresh=1000M/xfs-10dd-1-3.3.0
                   99.68        +0.1%        99.74  bay/thresh=1000M/xfs-1dd-1-3.3.0
                   97.82        -0.2%        97.64  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                   98.93        +0.7%        99.58  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                   69.60        -0.0%        69.58  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                   72.97        -0.2%        72.83  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                   79.76        +0.3%        79.99  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                   90.58        -2.1%        88.70  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                   95.27        -0.2%        95.06  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                   96.91        +1.6%        98.48  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                   93.79        -0.4%        93.41  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                   97.07        -0.4%        96.70  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                   99.09        -0.0%        99.04  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                   96.28        -0.2%        96.06  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                   94.83        +1.4%        96.15  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                   69.70        +0.2%        69.81  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                   72.79        -0.3%        72.55  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                   79.75        -0.1%        79.69  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                   87.95        +1.6%        89.38  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                   95.18        +0.0%        95.20  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                   98.58        -2.3%        96.36  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                   92.54        -0.1%        92.46  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                   96.81        -0.4%        96.42  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                   99.29        -0.4%        98.91  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                   99.03        +0.1%        99.13  bay/thresh=100M/btrfs-10dd-1-3.3.0
                   99.55        -0.1%        99.40  bay/thresh=100M/btrfs-1dd-1-3.3.0
                   55.68        -0.4%        55.48  bay/thresh=100M/ext3-100dd-1-3.3.0
                   71.40        -0.6%        70.95  bay/thresh=100M/ext3-10dd-1-3.3.0
                   79.82        +0.1%        79.88  bay/thresh=100M/ext3-1dd-1-3.3.0
                   63.21        -0.4%        62.99  bay/thresh=100M/ext4-100dd-1-3.3.0
                   84.42        +0.0%        84.45  bay/thresh=100M/ext4-10dd-1-3.3.0
                   95.65        +1.9%        97.44  bay/thresh=100M/ext4-1dd-1-3.3.0
                   49.61        +0.5%        49.87  bay/thresh=100M/xfs-100dd-1-3.3.0
                   87.76        -0.0%        87.76  bay/thresh=100M/xfs-10dd-1-3.3.0
                   99.34        +0.0%        99.36  bay/thresh=100M/xfs-1dd-1-3.3.0
                   92.98        -0.7%        92.29  bay/thresh=10M/btrfs-10dd-1-3.3.0
                   89.07        +0.4%        89.44  bay/thresh=10M/btrfs-1dd-1-3.3.0
                   61.76        +0.1%        61.80  bay/thresh=10M/ext3-10dd-1-3.3.0
                   75.72        +0.3%        75.96  bay/thresh=10M/ext3-1dd-1-3.3.0
                   67.89        -0.2%        67.77  bay/thresh=10M/ext4-10dd-1-3.3.0
                   92.00        -0.3%        91.73  bay/thresh=10M/ext4-1dd-1-3.3.0
                   69.71        -0.4%        69.45  bay/thresh=10M/xfs-10dd-1-3.3.0
                   92.23        +1.8%        93.89  bay/thresh=10M/xfs-1dd-1-3.3.0
                    2.60        +7.1%         2.78  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    3.72       -12.5%         3.25  bay/thresh=1M/btrfs-1dd-1-3.3.0
                   62.41        +0.3%        62.60  bay/thresh=1M/ext3-10dd-1-3.3.0
                   58.61        +0.4%        58.82  bay/thresh=1M/ext3-1dd-1-3.3.0
                   70.70        -1.1%        69.92  bay/thresh=1M/ext4-10dd-1-3.3.0
                   84.81        +0.1%        84.90  bay/thresh=1M/ext4-1dd-1-3.3.0
                   66.05        -0.1%        65.97  bay/thresh=1M/xfs-10dd-1-3.3.0
                   74.81        +0.1%        74.85  bay/thresh=1M/xfs-1dd-1-3.3.0
                13467.55        -0.0%     13460.98  TOTAL write_bw

> > > Also moves unplugging for direct I/O from around ->direct_IO() down to
> > > do_blockdev_direct_IO().
> > 
> > This could be handled in a separate patch.
> 
> I think that would be better.

OK. I'll split it on next post, after the tests complete.

Thanks,
Fengguang

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [RESEND][PATCH v2] block: remove plugging at buffered write time
  2012-04-12  1:32     ` Fengguang Wu
@ 2012-04-12  2:20       ` Fengguang Wu
  2012-04-12 14:26         ` Jan Kara
  2012-05-06  9:58       ` Fengguang Wu
  1 sibling, 1 reply; 16+ messages in thread
From: Fengguang Wu @ 2012-04-12  2:20 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Jeff Moyer, Jens Axboe, linux-fsdevel, LKML, Dave Chinner,
	Christoph Hellwig, Jan Kara, Chris Mason, Shaohua Li

On Thu, Apr 12, 2012 at 09:32:24AM +0800, Fengguang Wu wrote:
> On Wed, Apr 11, 2012 at 04:13:44PM -0700, Andrew Morton wrote:
> > On Mon, 09 Apr 2012 10:34:44 -0400
> > Jeff Moyer <jmoyer@redhat.com> wrote:
> > 
> > > Wu Fengguang <fengguang.wu@intel.com> writes:
> > > 
> > > > Buffered write(2) is not directly tied to IO, so it's not suitable to
> > > > handle plug in generic_file_aio_write().
> > > [snip, moved]
> > > > Note that plugging for O_SYNC writes is also removed. The user may pass
> > > > arbitrary @size arguments, which may be much larger than the preferable
> > > > I/O size, or may cross extent/device boundaries. Let the lower layers
> > > > handle the plugging. Otherwise the plugging code here will turn the
> > > > low level plugging into no-ops.
> > > 
> > > I assume you have some numbers to back this up, right?  Care to share
> > > those?
> > 
> > Yes please.
> > 
> > We've broken this stuff a few times recently - we should review and
> > test carefully.
> 
> Yes sure. Last time I posted the patch, I did some tests and found no
> performance changes. Now for 3.3, the tests started days ago have not
> finished now (partly because it is stalled for quite long time due to
> unknown reason). The now-available numbers for bs=4k dd's look fine.
> The pending tests are for bs=1M dd's and some random fio workloads.
> 

The changes are basically small enough to be considered noises.
But anyway here are some interpretations:

- application visible data write performance (write_bw) is almost the same
- it slightly reduces the real IOs that hit disk (io_wkB_s, io_rkB_s)
- disk utilization slightly increased
- CPU time is slightly reduced

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                13693.44        -0.0%     13686.80  TOTAL write_bw                                                                                                      
             10323494.62        -0.6%  10265428.70  TOTAL io_wkB_s                                                                                                      
                29967.74        -0.8%     29737.85  TOTAL io_w_s                                                                                                        
               547020.75        -0.1%    546234.40  TOTAL io_wrqm_s                                                                                                     
                   78.82        -0.3%        78.58  TOTAL io_rkB_s                                                                                                      
                   19.84        -0.3%        19.78  TOTAL io_r_s                                                                                                        
                    0.00                      0.00  TOTAL io_rrqm_s                                                                                                     
               103601.20        +0.1%    103667.05  TOTAL io_avgrq_sz                                                                                                   
                10022.77        +0.8%     10106.45  TOTAL io_avgqu_sz                                                                                                   
                52986.24        +1.1%     53569.42  TOTAL io_await                                                                                                      
                  582.42        +0.6%       585.86  TOTAL io_svctm                                                                                                      
                11342.21        +0.3%     11372.47  TOTAL io_util                                                                                                       
                  320.82        -0.9%       317.94  TOTAL cpu_user                                                                                                      
                    0.00                      0.00  TOTAL cpu_nice                                                                                                      
                 2756.42        +0.1%      2760.34  TOTAL cpu_system                                                                                                    
                 6621.34        -0.2%      6609.32  TOTAL cpu_iowait                                                                                                    
                    0.00                      0.00  TOTAL cpu_steal                                                                                                     
                 2401.42        +0.5%      2412.41  TOTAL cpu_idle                                                                                                      

Thanks,
Fengguang
---
$ ./compare-io bay/*/*-{3.3.0,3.3.0-plug+}
                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                  196.94        -0.0%       196.86  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  196.67        +0.2%       196.99  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  134.34        -0.1%       134.26  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                  141.75        +0.1%       141.92  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  166.74        +0.1%       166.89  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  165.90        -0.1%       165.69  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  186.83        +0.9%       188.60  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                  193.30        -0.1%       193.04  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  168.34        +0.1%       168.48  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  193.43        -0.3%       192.91  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  197.59        -0.2%       197.14  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  197.09        -0.1%       196.90  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                  197.20        +0.1%       197.48  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   98.41        +0.0%        98.41  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                  139.07        -0.6%       138.24  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                  157.10        -0.1%       156.99  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  113.36        -0.6%       112.73  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  159.88        -0.5%       159.09  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  191.53        +0.3%       192.09  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   83.65        +0.4%        83.99  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  163.12        +0.3%       163.54  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  194.25        +0.3%       194.88  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  194.98        +0.2%       195.34  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  194.78        +0.1%       194.96  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  127.48        -0.1%       127.41  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                  133.07        -0.1%       132.95  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  136.90        -0.8%       135.82  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  167.61        -1.2%       165.65  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  188.04        +1.1%       190.10  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                  187.74        -0.3%       187.27  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  169.01        +0.5%       169.80  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  192.03        +0.0%       192.10  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  196.06        +0.0%       196.14  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  189.04        -0.6%       187.92  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                  183.23        -0.6%       182.05  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  137.19        -0.1%       137.12  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                  140.61        -0.1%       140.54  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  114.32        -2.3%       111.71  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  162.42        -0.1%       162.31  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  183.84        -0.9%       182.19  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   89.10        -0.3%        88.84  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  162.48        -0.2%       162.20  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  188.15        +0.5%       189.12  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   97.79        -0.0%        97.76  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   97.69        +0.0%        97.71  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   69.84        -1.1%        69.05  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   76.62        -0.0%        76.59  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   83.97        -0.0%        83.95  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   94.87        -0.1%        94.79  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                   96.87        +0.6%        97.46  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                   84.57        -0.1%        84.50  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   95.51        +0.2%        95.67  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   97.44        -0.3%        97.12  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   97.69        -0.1%        97.56  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   97.75        +0.1%        97.87  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   66.68        -0.3%        66.45  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   76.23        -0.7%        75.73  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   59.36        +0.9%        59.93  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   82.63        -0.1%        82.56  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   95.77        +0.1%        95.86  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   43.77        -0.1%        43.71  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   84.24        +0.3%        84.48  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   97.81        -0.1%        97.76  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   99.27        -0.0%        99.25  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                   99.39        +0.0%        99.39  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                   69.75        +0.2%        69.86  bay/thresh=1000M/ext3-100dd-1-3.3.0
                   71.79        -0.0%        71.76  bay/thresh=1000M/ext3-10dd-1-3.3.0
                   80.00        -0.0%        79.99  bay/thresh=1000M/ext3-1dd-1-3.3.0
                   89.38        -0.1%        89.30  bay/thresh=1000M/ext4-100dd-1-3.3.0
                   96.43        +0.2%        96.63  bay/thresh=1000M/ext4-10dd-1-3.3.0
                   98.57        -0.3%        98.28  bay/thresh=1000M/ext4-1dd-1-3.3.0
                   93.97        -0.2%        93.76  bay/thresh=1000M/xfs-100dd-1-3.3.0
                   97.51        +0.1%        97.57  bay/thresh=1000M/xfs-10dd-1-3.3.0
                   99.68        +0.1%        99.74  bay/thresh=1000M/xfs-1dd-1-3.3.0
                   97.82        -0.2%        97.64  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                   98.93        +0.7%        99.58  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                   69.60        -0.0%        69.58  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                   72.97        -0.2%        72.83  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                   79.76        +0.3%        79.99  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                   90.58        -2.1%        88.70  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                   95.27        -0.2%        95.06  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                   96.91        +1.6%        98.48  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                   93.79        -0.4%        93.41  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                   97.07        -0.4%        96.70  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                   99.09        -0.0%        99.04  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                   96.28        -0.2%        96.06  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                   94.83        +1.4%        96.15  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                   69.70        +0.2%        69.81  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                   72.79        -0.3%        72.55  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                   79.75        -0.1%        79.69  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                   87.95        +1.6%        89.38  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                   95.18        +0.0%        95.20  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                   98.58        -2.3%        96.36  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                   92.54        -0.1%        92.46  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                   96.81        -0.4%        96.42  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                   99.29        -0.4%        98.91  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                   99.03        +0.1%        99.13  bay/thresh=100M/btrfs-10dd-1-3.3.0
                   99.55        -0.1%        99.40  bay/thresh=100M/btrfs-1dd-1-3.3.0
                   55.68        -0.4%        55.48  bay/thresh=100M/ext3-100dd-1-3.3.0
                   71.40        -0.6%        70.95  bay/thresh=100M/ext3-10dd-1-3.3.0
                   79.82        +0.1%        79.88  bay/thresh=100M/ext3-1dd-1-3.3.0
                   63.21        -0.4%        62.99  bay/thresh=100M/ext4-100dd-1-3.3.0
                   84.42        +0.0%        84.45  bay/thresh=100M/ext4-10dd-1-3.3.0
                   95.65        +1.9%        97.44  bay/thresh=100M/ext4-1dd-1-3.3.0
                   49.61        +0.5%        49.87  bay/thresh=100M/xfs-100dd-1-3.3.0
                   87.76        -0.0%        87.76  bay/thresh=100M/xfs-10dd-1-3.3.0
                   99.34        +0.0%        99.36  bay/thresh=100M/xfs-1dd-1-3.3.0
                   92.98        -0.7%        92.29  bay/thresh=10M/btrfs-10dd-1-3.3.0
                   89.07        +0.4%        89.44  bay/thresh=10M/btrfs-1dd-1-3.3.0
                   61.76        +0.1%        61.80  bay/thresh=10M/ext3-10dd-1-3.3.0
                   75.72        +0.3%        75.96  bay/thresh=10M/ext3-1dd-1-3.3.0
                   67.89        -0.2%        67.77  bay/thresh=10M/ext4-10dd-1-3.3.0
                   92.00        -0.3%        91.73  bay/thresh=10M/ext4-1dd-1-3.3.0
                   69.71        -0.4%        69.45  bay/thresh=10M/xfs-10dd-1-3.3.0
                   92.23        +1.8%        93.89  bay/thresh=10M/xfs-1dd-1-3.3.0
                    2.60        +7.1%         2.78  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    3.72       -12.5%         3.25  bay/thresh=1M/btrfs-1dd-1-3.3.0
                   62.41        +0.3%        62.60  bay/thresh=1M/ext3-10dd-1-3.3.0
                   58.61        +0.4%        58.82  bay/thresh=1M/ext3-1dd-1-3.3.0
                   70.70        -1.1%        69.92  bay/thresh=1M/ext4-10dd-1-3.3.0
                   84.81        +0.1%        84.90  bay/thresh=1M/ext4-1dd-1-3.3.0
                   66.05        -0.1%        65.97  bay/thresh=1M/xfs-10dd-1-3.3.0
                   74.81        +0.1%        74.85  bay/thresh=1M/xfs-1dd-1-3.3.0
                13693.44        -0.0%     13686.80  TOTAL write_bw

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
               101640.86        -0.0%    101602.29  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
               101643.78        +0.3%    101912.28  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                69572.19        +0.7%     70058.07  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                74055.75        -0.5%     73713.74  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                86078.08        +0.3%     86357.25  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                86070.75        -0.1%     85958.97  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                97517.49        +0.2%     97725.90  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                99956.52        -0.1%     99810.09  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                87840.33        -0.1%     87736.60  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                99785.53        -0.1%     99720.41  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
               101985.83        -0.3%    101704.75  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
               101627.30        -0.1%    101507.43  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
               101823.19        +0.2%    102027.67  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                51901.25        +1.0%     52399.28  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                72728.58        -0.7%     72183.21  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                81197.21        -0.1%     81103.59  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                60224.51        -2.4%     58783.59  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                82948.21        -0.0%     82915.24  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                99523.29        -0.1%     99448.25  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                43359.48        +1.2%     43900.16  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                83334.32        +2.0%     84985.87  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                99362.29        +0.7%    100076.90  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
               100425.00        +0.2%    100616.22  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
               100311.31        +0.1%    100403.97  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                65373.38        -0.1%     65291.41  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                68084.17        -0.1%     68041.18  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                70125.99        -0.8%     69541.46  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                86026.54        -0.3%     85775.06  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                97278.22        +0.2%     97489.25  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                86555.74        +0.5%     87012.07  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                98437.78        +0.1%     98568.98  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
               100441.38        +0.0%    100491.46  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                96867.95        -0.6%     96278.70  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                93869.66        -0.7%     93237.95  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                53257.87                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                70234.34        -0.1%     70184.10  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                58539.99        -1.0%     57952.81  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                83153.07        -0.1%     83080.97  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                94438.74        -1.1%     93425.95  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                45496.05        -0.2%     45411.35  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                83113.73        -0.1%     82996.14  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                96256.29        +0.5%     96771.63  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
               101174.25        -0.1%    101120.42  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
               101051.64        +0.0%    101060.98  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                71552.32        -1.0%     70819.21  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                78470.68        -0.0%     78441.43  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                85865.09        +0.0%     85904.62  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                97057.20        +0.0%     97095.17  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                99616.64        +0.3%     99895.94  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                86609.12        +0.1%     86671.78  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                97831.45        +0.1%     97966.11  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                99737.82        -0.3%     99487.77  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
               100007.28        -0.1%     99857.60  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
               100078.91        +0.0%    100118.03  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                68307.00        -0.3%     68086.47  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                78100.75        -0.7%     77550.11  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                60829.99        +0.9%     61372.87  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                84694.23        -0.2%     84555.18  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                98230.92        +0.1%     98305.03  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                44663.17        -0.2%     44580.65  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                86262.06        +0.2%     86473.25  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
               100128.65        -0.0%    100094.55  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
               101683.30        -0.0%    101661.66  bay/thresh=1000M/btrfs-10dd-1-3.3.0
               101859.75        -0.0%    101825.18  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                71645.78        -0.2%     71510.35  bay/thresh=1000M/ext3-100dd-1-3.3.0
                73421.83        +0.1%     73470.29  bay/thresh=1000M/ext3-10dd-1-3.3.0
                81912.24        -0.1%     81869.11  bay/thresh=1000M/ext3-1dd-1-3.3.0
                91615.47        -0.1%     91565.19  bay/thresh=1000M/ext4-100dd-1-3.3.0
                98974.77        +0.1%     99084.46  bay/thresh=1000M/ext4-10dd-1-3.3.0
               100995.23        -0.2%    100810.81  bay/thresh=1000M/ext4-1dd-1-3.3.0
                96234.18        -0.2%     96001.29  bay/thresh=1000M/xfs-100dd-1-3.3.0
                99871.84        +0.1%     99939.46  bay/thresh=1000M/xfs-10dd-1-3.3.0
               102050.79        -0.2%    101861.32  bay/thresh=1000M/xfs-1dd-1-3.3.0
               100198.27        -0.1%    100062.74  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
               101629.25        +0.2%    101787.73  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                71434.39        -0.0%     71433.16  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                74674.93        -0.3%     74449.62  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                81449.47        +0.2%     81601.75  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                92865.95        -0.3%     92550.24  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                97600.82        -0.1%     97483.69  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
               100863.59        +0.1%    100954.38  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                95969.85        -0.4%     95623.51  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                99363.75        -0.4%     98990.09  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
               101578.69        -0.0%    101550.90  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                98650.77        -0.2%     98409.18  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                97495.77        +0.9%     98396.09  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                71637.30        -0.0%     71632.14  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                74504.62        -0.5%     74133.16  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                81391.31        -0.0%     81361.32  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                91814.88        +0.0%     91835.63  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                99013.30        -0.1%     98910.43  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
               101003.65        -0.7%    100289.76  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                94721.50        -0.1%     94646.42  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                99057.14        -0.4%     98618.02  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
               101934.82        -0.4%    101506.65  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
               101421.30        +0.1%    101538.94  bay/thresh=100M/btrfs-10dd-1-3.3.0
               101995.48        -0.2%    101812.77  bay/thresh=100M/btrfs-1dd-1-3.3.0
                56996.11        -0.2%     56897.74  bay/thresh=100M/ext3-100dd-1-3.3.0
                73091.19        -0.7%     72607.40  bay/thresh=100M/ext3-10dd-1-3.3.0
                81710.55        +0.1%     81769.28  bay/thresh=100M/ext3-1dd-1-3.3.0
                64727.84        -0.3%     64534.33  bay/thresh=100M/ext4-100dd-1-3.3.0
                86504.71        +0.0%     86528.82  bay/thresh=100M/ext4-10dd-1-3.3.0
                99861.85        +0.0%     99903.90  bay/thresh=100M/ext4-1dd-1-3.3.0
                50654.09        +0.5%     50894.71  bay/thresh=100M/xfs-100dd-1-3.3.0
                89777.53        +0.0%     89813.32  bay/thresh=100M/xfs-10dd-1-3.3.0
               101674.66        -0.0%    101670.73  bay/thresh=100M/xfs-1dd-1-3.3.0
                95313.57        -0.8%     94593.99  bay/thresh=10M/btrfs-10dd-1-3.3.0
                91350.14        +0.4%     91719.56  bay/thresh=10M/btrfs-1dd-1-3.3.0
                63147.34        +0.1%     63197.57  bay/thresh=10M/ext3-10dd-1-3.3.0
                77510.17        +0.2%     77699.07  bay/thresh=10M/ext3-1dd-1-3.3.0
                69766.12        -0.3%     69569.97  bay/thresh=10M/ext4-10dd-1-3.3.0
                94538.38        -0.0%     94533.81  bay/thresh=10M/ext4-1dd-1-3.3.0
                71228.45        -0.4%     70961.11  bay/thresh=10M/xfs-10dd-1-3.3.0
                94305.80        +1.8%     96036.20  bay/thresh=10M/xfs-1dd-1-3.3.0
                 2475.00        +8.3%      2680.86  bay/thresh=1M/btrfs-10dd-1-3.3.0
                 3693.17       -14.0%      3174.89  bay/thresh=1M/btrfs-1dd-1-3.3.0
                63667.22        +0.4%     63891.82  bay/thresh=1M/ext3-10dd-1-3.3.0
                59989.80        +0.3%     60193.30  bay/thresh=1M/ext3-1dd-1-3.3.0
                72651.56        -0.2%     72493.23  bay/thresh=1M/ext4-10dd-1-3.3.0
                89754.89        -0.3%     89476.44  bay/thresh=1M/ext4-1dd-1-3.3.0
                67329.57        -0.1%     67229.65  bay/thresh=1M/xfs-10dd-1-3.3.0
                76505.80        +0.0%     76520.19  bay/thresh=1M/xfs-1dd-1-3.3.0
             10323494.62        -0.6%  10265428.70  TOTAL io_wkB_s

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                  202.76        +0.0%       202.84  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  200.58        +0.3%       201.10  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  179.65        +0.2%       179.94  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                  181.10        -0.6%       179.96  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  185.39        +0.3%       185.89  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  179.76        -0.1%       179.62  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  192.51        +0.3%       193.04  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                  196.61        -0.2%       196.26  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  175.48        -0.1%       175.33  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  197.22        -0.1%       197.09  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  201.27        -0.3%       200.59  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  218.22        -0.0%       218.17  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                  201.62        +0.2%       202.09  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  206.70        -2.0%       202.50  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                  184.21        -0.5%       183.29  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                  180.62        +0.2%       181.07  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  196.13        +4.8%       205.52  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  184.02        -0.3%       183.54  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  198.86        -0.1%       198.75  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                  141.47        +0.3%       141.87  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  180.34        +2.0%       183.89  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  200.28        +0.7%       201.76  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  252.30        +1.5%       255.96  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  248.90        -0.5%       247.57  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  173.55        +0.0%       173.63  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                  178.16        -0.3%       177.68  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  157.75        -0.9%       156.36  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  174.76        -0.1%       174.59  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  191.55        +0.2%       191.93  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  175.51        +0.5%       176.38  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  193.88        +0.2%       194.27  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  197.24        -0.0%       197.24  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  387.58        +0.0%       387.59  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                  346.03       -10.6%       309.39  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  198.91                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                  185.40        -0.3%       184.79  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  191.10        -1.4%       188.37  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  177.19        -0.0%       177.13  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  191.14        +3.2%       197.18  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                  156.10        -0.2%       155.86  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  178.77        -0.3%       178.32  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  195.01        -0.0%       194.96  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  207.45        +0.0%       207.51  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  209.39        +0.4%       210.28  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  194.33        +0.0%       194.39  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  189.36        +0.3%       189.96  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  177.04        +0.1%       177.26  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  191.38        -0.0%       191.36  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                  195.46        +0.3%       196.13  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  173.35        +0.1%       173.49  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  193.08        +0.1%       193.36  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  196.93        -0.2%       196.46  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  230.67        +0.7%       232.18  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                  233.81        -0.0%       233.72  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  183.47        -0.1%       183.23  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                  182.54        -1.1%       180.62  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  167.55        -0.6%       166.54  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  178.96        -0.2%       178.57  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  194.71        +0.1%       194.88  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                  138.08        -0.3%       137.61  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  177.17        +0.4%       177.96  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  201.85        -0.0%       201.80  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  201.09        -0.0%       201.08  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                  201.20        -0.1%       201.10  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                  179.04        +0.1%       179.19  bay/thresh=1000M/ext3-100dd-1-3.3.0
                  180.46        -0.2%       180.05  bay/thresh=1000M/ext3-10dd-1-3.3.0
                  181.60        -0.1%       181.49  bay/thresh=1000M/ext3-1dd-1-3.3.0
                  185.06        -0.1%       184.94  bay/thresh=1000M/ext4-100dd-1-3.3.0
                  194.65        +0.1%       194.87  bay/thresh=1000M/ext4-10dd-1-3.3.0
                  197.54        -0.2%       197.18  bay/thresh=1000M/ext4-1dd-1-3.3.0
                  190.67        -0.4%       189.90  bay/thresh=1000M/xfs-100dd-1-3.3.0
                  197.11        +0.1%       197.22  bay/thresh=1000M/xfs-10dd-1-3.3.0
                  201.36        -0.2%       200.95  bay/thresh=1000M/xfs-1dd-1-3.3.0
                  198.37        -0.0%       198.28  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                  201.20        +0.2%       201.55  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                  180.22        -0.4%       179.42  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                  182.95        -0.1%       182.71  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                  180.58        +0.1%       180.73  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                  186.79        -0.4%       185.95  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                  192.85        -0.0%       192.76  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                  197.22        +0.1%       197.39  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                  190.55        -0.3%       189.92  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                  196.51        -0.4%       195.75  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                  200.38        -0.0%       200.33  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                  195.58        -0.4%       194.88  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                  193.11        +1.1%       195.30  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                  180.27        +0.0%       180.31  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                  183.17        -0.1%       182.99  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                  180.41        -0.2%       180.13  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                  184.72        +0.1%       184.93  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                  194.52        -0.1%       194.35  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                  197.50        -0.7%       196.09  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                  188.46        -0.1%       188.30  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                  195.67        -0.4%       194.87  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                  201.14        -0.4%       200.28  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                  210.03        -0.6%       208.69  bay/thresh=100M/btrfs-10dd-1-3.3.0
                  201.36        -0.2%       200.92  bay/thresh=100M/btrfs-1dd-1-3.3.0
                  186.85        -0.3%       186.36  bay/thresh=100M/ext3-100dd-1-3.3.0
                  181.69        -0.5%       180.70  bay/thresh=100M/ext3-10dd-1-3.3.0
                  180.88        +0.1%       181.05  bay/thresh=100M/ext3-1dd-1-3.3.0
                  171.29        -0.2%       170.99  bay/thresh=100M/ext4-100dd-1-3.3.0
                  180.67        +0.1%       180.82  bay/thresh=100M/ext4-10dd-1-3.3.0
                  197.88        +0.1%       198.12  bay/thresh=100M/ext4-1dd-1-3.3.0
                  133.34        -0.9%       132.08  bay/thresh=100M/xfs-100dd-1-3.3.0
                  184.25        +0.3%       184.80  bay/thresh=100M/xfs-10dd-1-3.3.0
                  204.99        -0.0%       204.98  bay/thresh=100M/xfs-1dd-1-3.3.0
                  508.86        -1.9%       499.05  bay/thresh=10M/btrfs-10dd-1-3.3.0
                  218.37        -0.6%       217.11  bay/thresh=10M/btrfs-1dd-1-3.3.0
                  212.74        +0.1%       213.05  bay/thresh=10M/ext3-10dd-1-3.3.0
                  187.51        +0.1%       187.63  bay/thresh=10M/ext3-1dd-1-3.3.0
                  223.76        -0.6%       222.37  bay/thresh=10M/ext4-10dd-1-3.3.0
                  205.08        -0.3%       204.38  bay/thresh=10M/ext4-1dd-1-3.3.0
                  202.05        +0.2%       202.53  bay/thresh=10M/xfs-10dd-1-3.3.0
                  207.27        +1.9%       211.11  bay/thresh=10M/xfs-1dd-1-3.3.0
                   71.17        +2.9%        73.21  bay/thresh=1M/btrfs-10dd-1-3.3.0
                   34.65       -13.0%        30.14  bay/thresh=1M/btrfs-1dd-1-3.3.0
                 1005.42        +1.0%      1015.53  bay/thresh=1M/ext3-10dd-1-3.3.0
                  226.99        -4.6%       216.45  bay/thresh=1M/ext3-1dd-1-3.3.0
                 1148.86        -5.3%      1087.80  bay/thresh=1M/ext4-10dd-1-3.3.0
                 3603.70        +2.3%      3687.69  bay/thresh=1M/ext4-1dd-1-3.3.0
                 1088.82        -0.1%      1087.97  bay/thresh=1M/xfs-10dd-1-3.3.0
                  325.12        -2.7%       316.49  bay/thresh=1M/xfs-1dd-1-3.3.0
                29967.74        -0.8%     29737.85  TOTAL io_w_s

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                    3.51        +1.0%         3.55  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    3.51        +2.2%         3.58  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                17211.59        +0.7%     17335.19  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                18332.25        -0.5%     18249.07  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                21336.37        +0.4%     21411.42  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   21.22        +1.3%        21.50  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    4.71        +0.9%         4.75  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    2.07        -1.0%         2.05  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    4.18        +3.6%         4.32  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.09       +47.1%         0.13  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.05       -32.1%         0.03  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    3.09        -2.0%         3.03  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    3.10        -2.1%         3.03  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                12767.94        +1.0%     12901.93  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                17989.84        -0.7%     17860.56  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                20108.73        -0.1%     20093.81  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   64.51        -5.1%        61.24  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   58.21       -11.7%        51.40  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   42.96       -42.2%        24.84  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   15.76       -15.6%        13.29  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    2.45        +4.1%         2.55  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    1.50       +21.3%         1.81  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                 1319.47        -0.1%      1318.45  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                 1318.46        +0.3%      1322.34  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                16151.73        -0.2%     16126.80  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                16825.76        -0.1%     16812.78  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                17374.02        -0.8%     17228.57  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   12.81        +1.7%        13.02  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    5.18       -15.3%         4.39  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    3.90        +1.1%         3.95  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.48       +23.9%         0.60  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.46       -10.9%         0.41  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                 1139.96        -0.8%      1130.65  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                 1121.96        +2.4%      1148.65  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                17365.25        -0.0%     17359.65  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   20.23        -4.3%        19.36  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   57.83        -0.8%        57.37  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  182.32       +18.4%       215.94  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    4.13       -18.7%         3.36  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    2.96        -4.3%         2.83  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    3.67       +70.9%         6.27  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                 1377.09        -0.1%      1376.23  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                 1371.19        -0.0%      1370.82  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                17667.59        -1.0%     17485.61  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                19400.03        -0.0%     19397.09  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   13.51        -1.3%        13.33  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    5.07        +6.0%         5.37  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    2.22       -21.6%         1.74  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.25       +41.5%         0.36  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.30       -49.4%         0.15  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.02       +69.2%         0.04  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                 1347.86        -0.3%      1343.43  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                 1330.60        +0.2%      1333.04  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                16879.18        -0.3%     16823.03  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                19332.90        -0.7%     19189.88  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   28.13        -1.3%        27.77  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   38.56       +15.1%        44.40  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   68.52       -21.2%        54.02  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.94      +121.8%         2.10  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    3.51        +7.9%         3.79  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    6.09       -50.6%         3.01  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    3.68        +0.7%         3.71  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                    3.65        +2.1%         3.73  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                17730.91        -0.2%     17695.91  bay/thresh=1000M/ext3-100dd-1-3.3.0
                18173.73        -0.0%     18173.48  bay/thresh=1000M/ext3-10dd-1-3.3.0
                20321.91        -0.1%     20298.21  bay/thresh=1000M/ext3-1dd-1-3.3.0
                   13.05        -3.3%        12.61  bay/thresh=1000M/ext4-100dd-1-3.3.0
                    4.24        -1.3%         4.18  bay/thresh=1000M/ext4-10dd-1-3.3.0
                    1.77        +0.9%         1.79  bay/thresh=1000M/ext4-1dd-1-3.3.0
                    3.55        +6.6%         3.78  bay/thresh=1000M/xfs-100dd-1-3.3.0
                    0.02       -23.1%         0.02  bay/thresh=1000M/xfs-10dd-1-3.3.0
                    0.05        -0.1%         0.05  bay/thresh=1000M/xfs-1dd-1-3.3.0
                    3.51        -2.5%         3.42  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                    3.38        -0.6%         3.36  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                17704.76        +0.0%     17705.81  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                18512.03        -0.3%     18458.41  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                20210.88        +0.2%     20248.93  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                   11.30        -4.4%        10.80  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                    7.90       -20.8%         6.25  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                    1.74        +0.6%         1.75  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                    2.07        +2.0%         2.11  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                    0.12       +19.2%         0.15  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                    0.05        -7.1%         0.04  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                    3.19       +14.5%         3.65  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                    3.32        -6.8%         3.10  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                17754.91        -0.1%     17742.03  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                18467.85        -0.5%     18375.65  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                20197.66        -0.0%     20190.27  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                   16.33        +8.9%        17.80  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                    3.75        +1.8%         3.81  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                    1.76        -4.0%         1.69  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                    0.63        -8.2%         0.58  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                    0.12        +2.9%         0.12  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                    0.05        +3.7%         0.05  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                    2.71        -1.4%         2.67  bay/thresh=100M/btrfs-10dd-1-3.3.0
                    2.95        +1.2%         2.98  bay/thresh=100M/btrfs-1dd-1-3.3.0
                14064.97        -0.1%     14044.49  bay/thresh=100M/ext3-100dd-1-3.3.0
                18088.22        -0.7%     17969.28  bay/thresh=100M/ext3-10dd-1-3.3.0
                20271.98        -0.0%     20263.52  bay/thresh=100M/ext3-1dd-1-3.3.0
                   31.45        +5.4%        33.15  bay/thresh=100M/ext4-100dd-1-3.3.0
                   50.06       +13.8%        56.98  bay/thresh=100M/ext4-10dd-1-3.3.0
                  158.56       -91.6%        13.26  bay/thresh=100M/ext4-1dd-1-3.3.0
                    2.60        +1.6%         2.64  bay/thresh=100M/xfs-100dd-1-3.3.0
                    1.94        -9.0%         1.77  bay/thresh=100M/xfs-10dd-1-3.3.0
                    1.56       -74.6%         0.40  bay/thresh=100M/xfs-1dd-1-3.3.0
                   55.16        +8.7%        59.99  bay/thresh=10M/btrfs-10dd-1-3.3.0
                    8.62        +0.4%         8.66  bay/thresh=10M/btrfs-1dd-1-3.3.0
                15573.83        +0.1%     15586.97  bay/thresh=10M/ext3-10dd-1-3.3.0
                19191.19        +0.2%     19237.79  bay/thresh=10M/ext3-1dd-1-3.3.0
                   31.50        -3.3%        30.47  bay/thresh=10M/ext4-10dd-1-3.3.0
                   69.49        -9.8%        62.68  bay/thresh=10M/ext4-1dd-1-3.3.0
                   10.31        +1.2%        10.43  bay/thresh=10M/xfs-10dd-1-3.3.0
                    2.31       -54.4%         1.05  bay/thresh=10M/xfs-1dd-1-3.3.0
                    2.89       +21.4%         3.51  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    0.12       -51.4%         0.06  bay/thresh=1M/btrfs-1dd-1-3.3.0
                14911.37        +0.3%     14957.49  bay/thresh=1M/ext3-10dd-1-3.3.0
                14770.14        +0.4%     14831.78  bay/thresh=1M/ext3-1dd-1-3.3.0
                   65.33        -7.7%        60.31  bay/thresh=1M/ext4-10dd-1-3.3.0
                  420.32        -1.7%       413.20  bay/thresh=1M/ext4-1dd-1-3.3.0
                   62.47        +2.3%        63.88  bay/thresh=1M/xfs-10dd-1-3.3.0
                  243.06        -3.7%       234.06  bay/thresh=1M/xfs-1dd-1-3.3.0
               547020.75        -0.1%    546234.40  TOTAL io_wrqm_s

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    2.40        +0.3%         2.41  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    2.40        -0.3%         2.39  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    2.79        -0.0%         2.79  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.25        +0.1%         0.25  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.27        +0.0%         0.27  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.28        -0.0%         0.28  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.05        +0.0%         0.05  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.03        +0.0%         0.03  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.03        +0.0%         0.03  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    1.81        +0.7%         1.82  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                    2.35        -0.8%         2.33  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    2.61        -0.2%         2.61  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.18        +0.0%         0.18  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.24        +0.0%         0.24  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.28        -0.0%         0.28  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.02        +0.0%         0.02  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.03        +0.0%         0.03  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    2.25        -0.0%         2.25  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    2.29        -0.3%         2.28  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    2.32        -0.6%         2.31  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.43        -1.6%         0.42  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.48        +1.4%         0.48  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    2.34        +0.0%         2.34  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.29        +2.2%         0.30  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.42        +0.0%         0.42  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.47        -1.4%         0.46  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    1.29        +1.6%         1.31  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    2.38        -2.0%         2.33  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.17       -20.0%         0.13  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.19        -3.4%         0.19  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.25        -5.4%         0.23  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.01                      0.00  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.02        +0.0%         0.02  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.02        +0.0%         0.02  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    1.83        +0.4%         1.84  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    2.26        -3.2%         2.19  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.06       +55.6%         0.09  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.13       -30.0%         0.09  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.21        -6.4%         0.19  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.01                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.02        +0.0%         0.02  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                    2.44        +0.3%         2.45  bay/thresh=1000M/ext3-100dd-1-3.3.0
                    2.39        -0.0%         2.39  bay/thresh=1000M/ext3-10dd-1-3.3.0
                    2.67        -0.5%         2.66  bay/thresh=1000M/ext3-1dd-1-3.3.0
                    0.27        +0.0%         0.27  bay/thresh=1000M/ext4-100dd-1-3.3.0
                    0.27        +2.5%         0.28  bay/thresh=1000M/ext4-10dd-1-3.3.0
                    0.28        +0.0%         0.28  bay/thresh=1000M/ext4-1dd-1-3.3.0
                    0.05        +0.0%         0.05  bay/thresh=1000M/xfs-100dd-1-3.3.0
                    0.03        +0.0%         0.03  bay/thresh=1000M/xfs-10dd-1-3.3.0
                    0.03       +50.0%         0.05  bay/thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                    2.43        +0.0%         2.43  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                    2.42        +0.0%         2.42  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                    2.65        +0.2%         2.66  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                    0.27        -2.5%         0.26  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                    0.27        +2.4%         0.28  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                    0.28        +0.0%         0.28  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                    0.05       -33.3%         0.03  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                    0.03       +49.5%         0.05  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                    0.05        +0.0%         0.05  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                    2.44        +0.0%         2.44  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                    2.42        -0.3%         2.41  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                    2.65        +0.0%         2.65  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                    0.29        +4.6%         0.30  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                    0.27        +0.0%         0.27  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                    0.28        +0.0%         0.28  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                    0.05       -33.3%         0.03  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                    0.05       -33.3%         0.03  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                    0.05        +0.0%         0.05  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/btrfs-1dd-1-3.3.0
                    1.98        -0.7%         1.96  bay/thresh=100M/ext3-100dd-1-3.3.0
                    2.37        -0.9%         2.34  bay/thresh=100M/ext3-10dd-1-3.3.0
                    2.63        +0.0%         2.63  bay/thresh=100M/ext3-1dd-1-3.3.0
                    0.19        +0.0%         0.19  bay/thresh=100M/ext4-100dd-1-3.3.0
                    0.24        +0.0%         0.24  bay/thresh=100M/ext4-10dd-1-3.3.0
                    0.27        +2.4%         0.28  bay/thresh=100M/ext4-1dd-1-3.3.0
                    0.01                      0.00  bay/thresh=100M/xfs-100dd-1-3.3.0
                    0.03        +0.0%         0.03  bay/thresh=100M/xfs-10dd-1-3.3.0
                    0.03        +0.0%         0.03  bay/thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/btrfs-1dd-1-3.3.0
                    2.05        +0.0%         2.05  bay/thresh=10M/ext3-10dd-1-3.3.0
                    2.49        +0.5%         2.51  bay/thresh=10M/ext3-1dd-1-3.3.0
                    0.21        +0.0%         0.21  bay/thresh=10M/ext4-10dd-1-3.3.0
                    0.27        +0.0%         0.27  bay/thresh=10M/ext4-1dd-1-3.3.0
                    0.02        +0.0%         0.02  bay/thresh=10M/xfs-10dd-1-3.3.0
                    0.02      +100.0%         0.03  bay/thresh=10M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/btrfs-1dd-1-3.3.0
                    2.06        +0.3%         2.07  bay/thresh=1M/ext3-10dd-1-3.3.0
                    1.94        +0.3%         1.94  bay/thresh=1M/ext3-1dd-1-3.3.0
                    0.22        -3.0%         0.21  bay/thresh=1M/ext4-10dd-1-3.3.0
                    0.25        +0.0%         0.25  bay/thresh=1M/ext4-1dd-1-3.3.0
                    0.02        +0.0%         0.02  bay/thresh=1M/xfs-10dd-1-3.3.0
                    0.02        +0.0%         0.02  bay/thresh=1M/xfs-1dd-1-3.3.0
                   78.82        -0.3%        78.58  TOTAL io_rkB_s

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.60        +0.3%         0.60  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    0.60        -0.3%         0.60  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.70        -0.0%         0.70  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.06        +0.1%         0.06  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.07        +0.0%         0.07  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.07        -0.0%         0.07  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.02        +0.0%         0.02  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.45        +0.7%         0.46  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                    0.59        -0.8%         0.58  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.65        -0.2%         0.65  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.05        +0.0%         0.05  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.06        +0.0%         0.06  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.07        -0.0%         0.07  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.56        -0.0%         0.56  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    0.57        -0.3%         0.57  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.58        -0.6%         0.58  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.11        -1.6%         0.11  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.12        +1.4%         0.12  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.59        +0.0%         0.59  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.07        +2.2%         0.08  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.10        +0.0%         0.10  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.12        -1.4%         0.12  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.32        +1.6%         0.33  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.59        -2.0%         0.58  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.04       -20.0%         0.03  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.05        -3.4%         0.05  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.06        -5.4%         0.06  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.01                      0.00  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.46        +0.4%         0.46  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.57        -3.2%         0.55  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.02       +55.6%         0.02  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.03       -30.0%         0.02  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.05        -6.4%         0.05  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.01                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                    0.61        +0.3%         0.61  bay/thresh=1000M/ext3-100dd-1-3.3.0
                    0.60        -0.0%         0.60  bay/thresh=1000M/ext3-10dd-1-3.3.0
                    0.67        -0.5%         0.66  bay/thresh=1000M/ext3-1dd-1-3.3.0
                    0.07        +0.0%         0.07  bay/thresh=1000M/ext4-100dd-1-3.3.0
                    0.07        +2.5%         0.07  bay/thresh=1000M/ext4-10dd-1-3.3.0
                    0.07        +0.0%         0.07  bay/thresh=1000M/ext4-1dd-1-3.3.0
                    0.02        +0.0%         0.02  bay/thresh=1000M/xfs-100dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/thresh=1000M/xfs-10dd-1-3.3.0
                    0.01       +50.0%         0.02  bay/thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                    0.61        +0.0%         0.61  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                    0.60        +0.0%         0.60  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                    0.66        +0.2%         0.66  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                    0.07        -2.5%         0.07  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                    0.07        +2.4%         0.07  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                    0.07        +0.0%         0.07  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                    0.02       -33.3%         0.01  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                    0.01       +49.5%         0.02  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                    0.02        +0.0%         0.02  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                    0.61        +0.0%         0.61  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                    0.60        -0.3%         0.60  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                    0.66        +0.0%         0.66  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                    0.07        +4.6%         0.08  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                    0.07        +0.0%         0.07  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                    0.07        +0.0%         0.07  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                    0.02       -33.3%         0.01  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                    0.02       -33.3%         0.01  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                    0.02        +0.0%         0.02  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/btrfs-1dd-1-3.3.0
                    0.49        -0.7%         0.49  bay/thresh=100M/ext3-100dd-1-3.3.0
                    0.59        -0.9%         0.59  bay/thresh=100M/ext3-10dd-1-3.3.0
                    0.66        +0.0%         0.66  bay/thresh=100M/ext3-1dd-1-3.3.0
                    0.05        +0.0%         0.05  bay/thresh=100M/ext4-100dd-1-3.3.0
                    0.06        +0.0%         0.06  bay/thresh=100M/ext4-10dd-1-3.3.0
                    0.07        +2.4%         0.07  bay/thresh=100M/ext4-1dd-1-3.3.0
                    0.01                      0.00  bay/thresh=100M/xfs-100dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/thresh=100M/xfs-10dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/btrfs-1dd-1-3.3.0
                    0.51        +0.0%         0.51  bay/thresh=10M/ext3-10dd-1-3.3.0
                    0.62        +0.5%         0.63  bay/thresh=10M/ext3-1dd-1-3.3.0
                    0.05        +0.0%         0.05  bay/thresh=10M/ext4-10dd-1-3.3.0
                    0.07        +0.0%         0.07  bay/thresh=10M/ext4-1dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/thresh=10M/xfs-10dd-1-3.3.0
                    0.01      +100.0%         0.01  bay/thresh=10M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/btrfs-1dd-1-3.3.0
                    0.52        +0.3%         0.52  bay/thresh=1M/ext3-10dd-1-3.3.0
                    0.48        +0.3%         0.49  bay/thresh=1M/ext3-1dd-1-3.3.0
                    0.06        -3.0%         0.05  bay/thresh=1M/ext4-10dd-1-3.3.0
                    0.06        +0.0%         0.06  bay/thresh=1M/ext4-1dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/thresh=1M/xfs-10dd-1-3.3.0
                    0.01        +0.0%         0.01  bay/thresh=1M/xfs-1dd-1-3.3.0
                   19.84        -0.3%        19.78  TOTAL io_r_s

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/xfs-1dd-1-3.3.0
                    0.00                      0.00  TOTAL io_rrqm_s

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                 1002.48        -0.1%      1001.70  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                 1013.23        +0.0%      1013.34  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  797.02        +0.9%       803.80  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                  833.69        +0.2%       835.53  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  932.91        +0.1%       933.59  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  957.27        -0.0%       956.81  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                 1012.54        -0.1%      1011.94  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                 1016.44        +0.0%      1016.73  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                 1001.16        -0.0%      1000.82  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                 1011.78        +0.0%      1011.81  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                 1013.29        +0.1%      1013.98  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  931.78        -0.1%       931.00  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                 1010.16        -0.0%      1009.79  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  505.73        +3.3%       522.54  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                  807.72        -0.6%       802.80  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                  922.23        -0.1%       921.53  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  617.24        -7.1%       573.51  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  901.88        +0.1%       903.21  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                 1000.46        -0.0%      1000.29  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                  614.40        +1.0%       620.45  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  924.51        +0.0%       924.60  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  992.19        -0.0%       992.03  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  854.68        -1.2%       844.84  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  857.31        +0.1%       858.09  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  767.71        -0.2%       765.85  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                  774.35        +0.2%       775.99  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  912.74        +0.1%       913.75  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  983.98        -0.2%       981.89  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                 1014.81        +0.0%      1015.13  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  986.34        +0.0%       986.66  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                 1015.44        -0.1%      1014.78  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                 1018.53        +0.1%      1019.11  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  532.27        +0.2%       533.27  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                  571.34        +9.3%       624.61  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  774.14        +0.2%       776.03  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  614.30        +0.4%       616.98  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  938.52        -0.1%       937.81  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  994.75        -1.7%       977.70  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                  584.04        -0.0%       583.97  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  930.05        +0.1%       931.05  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  988.33        +0.5%       993.71  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  980.97        -0.1%       979.53  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  971.95        -0.2%       969.74  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  747.84        -1.5%       736.54  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  839.11        -0.3%       836.69  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  968.14        -0.0%       967.71  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                 1013.58        +0.0%      1014.07  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                 1018.95        -0.1%      1018.35  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  999.44        -0.0%       999.36  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                 1013.36        -0.0%      1013.24  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                 1013.01        -0.0%      1012.95  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  872.96        -0.9%       865.45  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                  863.24        +0.1%       864.34  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  760.52        -0.2%       758.97  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                  878.42        -0.1%       877.78  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  731.68        +1.5%       742.40  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  946.18        -0.0%       946.08  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                 1008.63        -0.0%      1008.47  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                  650.88        +0.2%       652.42  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  973.65        -0.2%       971.79  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  992.17        -0.0%       992.05  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                 1011.16        -0.0%      1010.96  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                 1012.38        +0.0%      1012.57  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                  828.19        -0.1%       827.42  bay/thresh=1000M/ext3-100dd-1-3.3.0
                  826.71        +0.4%       830.01  bay/thresh=1000M/ext3-10dd-1-3.3.0
                  918.56        +0.2%       920.68  bay/thresh=1000M/ext3-1dd-1-3.3.0
                  989.47        +0.0%       989.67  bay/thresh=1000M/ext4-100dd-1-3.3.0
                 1016.34        +0.0%      1016.34  bay/thresh=1000M/ext4-10dd-1-3.3.0
                 1022.13        +0.0%      1022.14  bay/thresh=1000M/ext4-1dd-1-3.3.0
                 1009.33        +0.2%      1010.92  bay/thresh=1000M/xfs-100dd-1-3.3.0
                 1013.22        +0.0%      1013.35  bay/thresh=1000M/xfs-10dd-1-3.3.0
                 1013.58        +0.0%      1013.76  bay/thresh=1000M/xfs-1dd-1-3.3.0
                 1010.09        -0.1%      1009.34  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                 1008.47        -0.0%      1008.33  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                  826.13        +0.1%       827.29  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                  840.67        -0.1%       839.96  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                  932.23        +0.3%       934.61  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                  993.68        +0.1%       994.84  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                 1011.66        -0.1%      1010.89  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                 1022.46        +0.0%      1022.49  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                 1007.15        -0.0%      1006.87  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                 1011.03        +0.0%      1011.24  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                 1011.92        -0.1%      1011.01  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                 1008.97        +0.1%      1009.57  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                 1000.61        +0.7%      1007.96  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                  825.14        -0.0%       824.95  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                  839.74        -0.3%       837.55  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                  933.69        +0.1%       934.35  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                  993.32        -0.1%       992.47  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                 1017.46        -0.0%      1017.24  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                 1022.43        -0.0%      1022.25  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                 1005.10        +0.0%      1005.17  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                 1012.23        -0.0%      1011.88  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                 1011.73        -0.1%      1011.19  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                  966.31        +0.8%       973.91  bay/thresh=100M/btrfs-10dd-1-3.3.0
                 1013.37        +0.0%      1013.77  bay/thresh=100M/btrfs-1dd-1-3.3.0
                  619.62        +0.2%       620.77  bay/thresh=100M/ext3-100dd-1-3.3.0
                  817.55        -0.0%       817.35  bay/thresh=100M/ext3-10dd-1-3.3.0
                  926.93        +0.0%       927.03  bay/thresh=100M/ext3-1dd-1-3.3.0
                  755.42        -0.1%       754.83  bay/thresh=100M/ext4-100dd-1-3.3.0
                  957.15        -0.0%       956.69  bay/thresh=100M/ext4-10dd-1-3.3.0
                 1008.84        -0.1%      1008.10  bay/thresh=100M/ext4-1dd-1-3.3.0
                  760.80        +1.4%       771.56  bay/thresh=100M/xfs-100dd-1-3.3.0
                  974.57        -0.3%       972.01  bay/thresh=100M/xfs-10dd-1-3.3.0
                  991.96        -0.0%       991.93  bay/thresh=100M/xfs-1dd-1-3.3.0
                  472.16        +0.6%       474.97  bay/thresh=10M/btrfs-10dd-1-3.3.0
                  834.38        +0.6%       839.04  bay/thresh=10M/btrfs-1dd-1-3.3.0
                  597.77        +0.8%       602.60  bay/thresh=10M/ext3-10dd-1-3.3.0
                  840.61        +0.3%       842.83  bay/thresh=10M/ext3-1dd-1-3.3.0
                  626.47        +0.2%       627.91  bay/thresh=10M/ext4-10dd-1-3.3.0
                  928.26        +0.1%       929.19  bay/thresh=10M/ext4-1dd-1-3.3.0
                  705.53        -0.6%       701.19  bay/thresh=10M/xfs-10dd-1-3.3.0
                  911.86        -0.2%       910.08  bay/thresh=10M/xfs-1dd-1-3.3.0
                   35.27       +11.3%        39.27  bay/thresh=1M/btrfs-10dd-1-3.3.0
                  170.16        +2.2%       173.97  bay/thresh=1M/btrfs-1dd-1-3.3.0
                  128.90        -0.8%       127.85  bay/thresh=1M/ext3-10dd-1-3.3.0
                  567.74        +1.4%       575.48  bay/thresh=1M/ext3-1dd-1-3.3.0
                  132.42        +3.8%       137.47  bay/thresh=1M/ext4-10dd-1-3.3.0
                   57.51        -1.8%        56.51  bay/thresh=1M/ext4-1dd-1-3.3.0
                  129.70        -0.3%       129.33  bay/thresh=1M/xfs-10dd-1-3.3.0
                  484.52        +1.4%       491.16  bay/thresh=1M/xfs-1dd-1-3.3.0
               103601.20        +0.1%    103667.05  TOTAL io_avgrq_sz

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                  143.47        +0.0%       143.50  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  143.41        +0.0%       143.46  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  140.56        -0.2%       140.27  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                  140.17        -0.2%       139.85  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  138.42        -0.1%       138.29  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  138.29        +0.3%       138.74  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  135.02        +0.9%       136.30  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                  142.01        -0.5%       141.33  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  143.16        -0.0%       143.10  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  143.03        +0.1%       143.17  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  143.01        -0.0%       142.98  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   62.95       +14.7%        72.23  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   54.31        -1.7%        53.38  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  108.96        +4.6%       114.00  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                   85.35        -3.5%        82.37  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   53.63       -14.8%        45.70  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  102.28       -10.2%        91.80  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   71.87        +3.1%        74.10  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   61.91        -4.4%        59.18  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                  114.05        +0.7%       114.85  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   54.23       +28.9%        69.91  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   29.27       +10.0%        32.20  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   66.32        -2.1%        64.93  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   68.85        +0.1%        68.88  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   20.76        -0.4%        20.67  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                   14.36       +28.2%        18.41  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   23.11        -3.3%        22.33  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   35.01       +71.2%        59.92  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   91.18       -12.4%        79.89  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                   38.38        +3.0%        39.53  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   44.04        -0.1%        43.97  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   36.29        +3.3%        37.47  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   25.60        -1.8%        25.15  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   26.17        +0.0%        26.17  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   40.62       +10.3%        44.81  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   35.80        -1.2%        35.36  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   55.35        -1.9%        54.33  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   27.79        -4.0%        26.68  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   59.79       -13.7%        51.61  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   35.75        +1.3%        36.21  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   18.72        -1.7%        18.41  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   90.48       +29.5%       117.17  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   85.38        +2.8%        87.74  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   17.15       -33.8%        11.36  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   19.91        -3.7%        19.18  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   70.84        +2.2%        72.41  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  102.42        -2.4%        99.95  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                   45.77       -18.6%        37.26  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                   48.35        -7.1%        44.94  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   69.81        -2.1%        68.37  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   37.36       +11.4%        41.62  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   52.62        -4.7%        50.12  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   54.11        +0.9%        54.60  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   13.44        -7.1%        12.48  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   24.96       -10.7%        22.30  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   32.88        +4.2%        34.26  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   77.72        -4.7%        74.10  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   80.96        -0.4%        80.65  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   12.79      +534.1%        81.10  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   57.79        -0.4%        57.54  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   37.65        -0.8%        37.34  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  143.43        +0.1%       143.52  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                  143.48        +0.0%       143.55  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                  140.85        -0.0%       140.84  bay/thresh=1000M/ext3-100dd-1-3.3.0
                  140.59        -0.2%       140.29  bay/thresh=1000M/ext3-10dd-1-3.3.0
                  139.12        -0.2%       138.88  bay/thresh=1000M/ext3-1dd-1-3.3.0
                  140.84        +0.3%       141.22  bay/thresh=1000M/ext4-100dd-1-3.3.0
                  140.47        +0.1%       140.60  bay/thresh=1000M/ext4-10dd-1-3.3.0
                  141.69        -0.6%       140.88  bay/thresh=1000M/ext4-1dd-1-3.3.0
                  143.08        -0.0%       143.04  bay/thresh=1000M/xfs-100dd-1-3.3.0
                  143.03        -0.0%       143.02  bay/thresh=1000M/xfs-10dd-1-3.3.0
                  143.10        +0.0%       143.11  bay/thresh=1000M/xfs-1dd-1-3.3.0
                  125.49        +0.6%       126.19  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                  123.55        +1.1%       124.91  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                  135.09        -0.8%       133.99  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                  137.85        -1.1%       136.40  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                  137.62        -0.2%       137.32  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                  129.59        -0.5%       128.97  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                  132.99        -0.7%       132.01  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                  140.02        +0.5%       140.67  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                  135.24        +1.1%       136.67  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                  137.08        -0.2%       136.74  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                  142.13        -0.2%       141.91  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                  120.16        +0.6%       120.85  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                  118.54        -5.4%       112.13  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                  128.24        -0.5%       127.66  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                  139.52        +0.4%       140.07  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                  136.67        -0.1%       136.57  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                  123.81        +0.9%       124.98  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                  138.90        -0.3%       138.42  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                  140.89        -1.2%       139.23  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                  134.80        -0.2%       134.58  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                  136.91        -1.1%       135.46  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                  142.31        -0.2%       142.07  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                  118.35        +1.4%       120.01  bay/thresh=100M/btrfs-10dd-1-3.3.0
                  115.74        +0.1%       115.82  bay/thresh=100M/btrfs-1dd-1-3.3.0
                  131.25        -0.3%       130.88  bay/thresh=100M/ext3-100dd-1-3.3.0
                  123.90        -1.2%       122.35  bay/thresh=100M/ext3-10dd-1-3.3.0
                   98.00        +0.8%        98.79  bay/thresh=100M/ext3-1dd-1-3.3.0
                  115.46        +0.1%       115.60  bay/thresh=100M/ext4-100dd-1-3.3.0
                  108.86        -0.4%       108.37  bay/thresh=100M/ext4-10dd-1-3.3.0
                  113.65        +0.3%       114.00  bay/thresh=100M/ext4-1dd-1-3.3.0
                  131.63        +0.1%       131.70  bay/thresh=100M/xfs-100dd-1-3.3.0
                   99.71        +0.8%       100.51  bay/thresh=100M/xfs-10dd-1-3.3.0
                   75.69        +0.7%        76.20  bay/thresh=100M/xfs-1dd-1-3.3.0
                   15.01        -0.3%        14.97  bay/thresh=10M/btrfs-10dd-1-3.3.0
                    7.95        +1.7%         8.09  bay/thresh=10M/btrfs-1dd-1-3.3.0
                   19.94        -0.2%        19.90  bay/thresh=10M/ext3-10dd-1-3.3.0
                   13.22        -2.2%        12.93  bay/thresh=10M/ext3-1dd-1-3.3.0
                   19.99        -1.9%        19.60  bay/thresh=10M/ext4-10dd-1-3.3.0
                   11.81        +2.0%        12.05  bay/thresh=10M/ext4-1dd-1-3.3.0
                   15.51        +1.5%        15.75  bay/thresh=10M/xfs-10dd-1-3.3.0
                    7.99       -12.5%         6.98  bay/thresh=10M/xfs-1dd-1-3.3.0
                    0.04       +32.1%         0.05  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    0.04        -7.0%         0.03  bay/thresh=1M/btrfs-1dd-1-3.3.0
                    8.98        -2.3%         8.77  bay/thresh=1M/ext3-10dd-1-3.3.0
                    1.86        -0.3%         1.86  bay/thresh=1M/ext3-1dd-1-3.3.0
                    8.21        -3.2%         7.95  bay/thresh=1M/ext4-10dd-1-3.3.0
                    7.90        -6.8%         7.36  bay/thresh=1M/ext4-1dd-1-3.3.0
                    7.83        -0.3%         7.80  bay/thresh=1M/xfs-10dd-1-3.3.0
                    1.54        -6.4%         1.44  bay/thresh=1M/xfs-1dd-1-3.3.0
                10022.77        +0.8%     10106.45  TOTAL io_avgqu_sz

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                  710.04        -0.0%       709.94  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  717.51        -0.1%       716.61  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  789.74        -0.6%       785.30  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                  780.32        +0.2%       781.62  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  751.48        -0.5%       748.06  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  773.52        +0.4%       776.77  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  703.83        +0.8%       709.62  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                  726.98        -0.4%       724.39  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  816.73        +0.1%       817.15  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  728.16        +0.2%       729.75  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  712.91        +0.4%       715.74  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  290.25       +14.9%       333.60  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                  271.36        -2.1%       265.60  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  526.42        +6.4%       560.09  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                  466.68        -4.2%       447.24  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                  296.06       -15.9%       249.09  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  530.25       -13.6%       458.27  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  390.59        +3.7%       405.04  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  313.13        -4.4%       299.29  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                  808.79        +0.5%       812.80  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  300.07       +26.5%       379.65  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  146.03        +9.7%       160.15  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  311.83        -3.3%       301.45  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  319.18        -0.5%       317.68  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  122.70        -1.9%       120.36  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                   81.83       +27.6%       104.37  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  149.66        -3.3%       144.75  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  198.73       +71.4%       340.71  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  475.61       -12.7%       415.26  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  217.26        +2.5%       222.59  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  226.42        -0.2%       225.91  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  183.90        +2.9%       189.28  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   73.81        -0.5%        73.47  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   82.93        +9.9%        91.11  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                  222.98       +10.3%       245.99  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  186.80        -0.7%       185.58  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  314.42        -2.2%       307.63  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  145.86        -4.7%       139.07  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                  386.73       -13.6%       334.04  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  199.33        +1.5%       202.25  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   95.80        -1.8%        94.12  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  444.61       +29.9%       577.75  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  419.31        +2.5%       429.91  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   92.26       -34.1%        60.78  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  109.42        -3.6%       105.48  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  394.83        +3.5%       408.64  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                  532.32        -2.3%       520.10  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                  234.32       -19.0%       189.77  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  277.32        -7.5%       256.39  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  358.51        -2.2%       350.56  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  190.29       +11.0%       211.26  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                  233.47        -5.9%       219.79  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                  237.12        +1.1%       239.75  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   74.89        -6.7%        69.90  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                  144.44       -11.3%       128.13  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                  198.45        +5.2%       208.84  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                  436.30        -4.2%       418.06  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                  413.74        -0.5%       411.50  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   95.56      +512.3%       585.10  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                  325.33        -1.1%       321.68  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                  186.29        -0.9%       184.56  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  716.19        +0.0%       716.47  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                  716.50        -0.0%       716.31  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                  796.73        -0.2%       794.85  bay/thresh=1000M/ext3-100dd-1-3.3.0
                  784.58        -0.2%       782.86  bay/thresh=1000M/ext3-10dd-1-3.3.0
                  771.73        -0.0%       771.62  bay/thresh=1000M/ext3-1dd-1-3.3.0
                  765.60        +0.3%       767.86  bay/thresh=1000M/ext4-100dd-1-3.3.0
                  727.42        -0.2%       725.81  bay/thresh=1000M/ext4-10dd-1-3.3.0
                  720.30        -0.3%       718.32  bay/thresh=1000M/ext4-1dd-1-3.3.0
                  752.76        +0.5%       756.55  bay/thresh=1000M/xfs-100dd-1-3.3.0
                  728.62        -0.1%       728.04  bay/thresh=1000M/xfs-10dd-1-3.3.0
                  713.78        +0.2%       714.94  bay/thresh=1000M/xfs-1dd-1-3.3.0
                  635.41        +0.7%       639.61  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                  615.64        +1.0%       621.69  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                  759.36        -0.4%       755.98  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                  762.18        -1.0%       754.72  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                  767.34        -0.2%       765.64  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                  697.08        +0.0%       697.30  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                  692.33        -0.6%       688.18  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                  713.22        +0.4%       716.39  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                  712.29        +1.3%       721.26  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                  700.14        -0.0%       700.02  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                  709.89        -0.2%       708.76  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                  612.31        +1.0%       618.73  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                  596.29        -5.4%       563.82  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                  719.44        -0.7%       714.57  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                  770.52        +0.6%       775.08  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                  763.72        +0.1%       764.21  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                  671.14        +0.7%       675.87  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                  718.02        -0.0%       717.75  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                  716.33        -0.9%       709.97  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                  715.54        -0.1%       714.87  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                  700.61        -0.8%       695.32  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                  708.05        +0.2%       709.75  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                  565.71        +2.1%       577.53  bay/thresh=100M/btrfs-10dd-1-3.3.0
                  577.21        +0.3%       579.18  bay/thresh=100M/btrfs-1dd-1-3.3.0
                  699.80        -0.1%       699.16  bay/thresh=100M/ext3-100dd-1-3.3.0
                  685.16        -0.4%       682.18  bay/thresh=100M/ext3-10dd-1-3.3.0
                  545.13        +1.0%       550.66  bay/thresh=100M/ext3-1dd-1-3.3.0
                  679.24        +0.1%       680.03  bay/thresh=100M/ext4-100dd-1-3.3.0
                  606.42        -0.9%       600.66  bay/thresh=100M/ext4-10dd-1-3.3.0
                  576.89        -0.0%       576.69  bay/thresh=100M/ext4-1dd-1-3.3.0
                  987.64        +1.0%       997.71  bay/thresh=100M/xfs-100dd-1-3.3.0
                  542.48        +0.5%       545.17  bay/thresh=100M/xfs-10dd-1-3.3.0
                  371.49        +0.7%       374.27  bay/thresh=100M/xfs-1dd-1-3.3.0
                   35.96        +2.3%        36.79  bay/thresh=10M/btrfs-10dd-1-3.3.0
                   36.54        +1.8%        37.21  bay/thresh=10M/btrfs-1dd-1-3.3.0
                   98.48        -4.3%        94.23  bay/thresh=10M/ext3-10dd-1-3.3.0
                   71.41        -3.5%        68.90  bay/thresh=10M/ext3-1dd-1-3.3.0
                   92.89        -0.8%        92.12  bay/thresh=10M/ext4-10dd-1-3.3.0
                   58.65        +2.7%        60.22  bay/thresh=10M/ext4-1dd-1-3.3.0
                   76.82        +1.3%        77.80  bay/thresh=10M/xfs-10dd-1-3.3.0
                   39.00       -14.2%        33.45  bay/thresh=10M/xfs-1dd-1-3.3.0
                    0.51       +36.8%         0.69  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    1.13        +0.4%         1.14  bay/thresh=1M/btrfs-1dd-1-3.3.0
                   11.03        -1.4%        10.88  bay/thresh=1M/ext3-10dd-1-3.3.0
                    8.34        +2.7%         8.56  bay/thresh=1M/ext3-1dd-1-3.3.0
                    7.47        +2.1%         7.63  bay/thresh=1M/ext4-10dd-1-3.3.0
                    2.41        -8.9%         2.20  bay/thresh=1M/ext4-1dd-1-3.3.0
                    7.26        +0.0%         7.26  bay/thresh=1M/xfs-10dd-1-3.3.0
                    4.74        -1.8%         4.65  bay/thresh=1M/xfs-1dd-1-3.3.0
                52986.24        +1.1%     53569.42  TOTAL io_await

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                    4.97        -0.0%         4.97  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    5.03        -0.3%         5.01  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    5.67        +2.3%         5.80  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    5.64        +2.2%         5.76  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    5.51        -0.3%         5.49  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    5.64        -0.1%         5.64  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    5.27        -0.3%         5.25  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    5.14        +0.1%         5.15  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    5.72        +0.2%         5.73  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    5.11        +0.2%         5.12  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    5.00        +0.5%         5.03  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    4.60        +0.2%         4.61  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    5.00        -0.3%         4.98  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    5.02        +3.4%         5.19  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                    5.61        +0.9%         5.67  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    5.69        +0.0%         5.69  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    5.27        -3.9%         5.06  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    5.52        +0.3%         5.54  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    5.08        +0.1%         5.08  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    7.10        -0.2%         7.08  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    5.55        -1.6%         5.46  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    4.78        +1.2%         4.83  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    4.21        -1.3%         4.15  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    4.19        +0.3%         4.21  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    4.05        +0.1%         4.05  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    4.08        -0.3%         4.06  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    4.10        +1.5%         4.16  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    5.32        +4.8%         5.58  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    5.18        -0.8%         5.14  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    5.46        -0.1%         5.45  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    5.06        -0.4%         5.04  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    4.97        -0.2%         4.96  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    2.57        +0.1%         2.57  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    2.69        +9.0%         2.94  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    5.41        +0.2%         5.42  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    4.49        +0.1%         4.50  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    5.56        +0.2%         5.57  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    4.79        -2.1%         4.68  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    5.59        -7.7%         5.16  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    5.21        -0.4%         5.19  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    4.80        -0.0%         4.79  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    4.80        -0.3%         4.79  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    4.74        -0.1%         4.74  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    4.10       -12.6%         3.58  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    4.75        +2.3%         4.86  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    5.51        +3.9%         5.72  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    5.13        +0.0%         5.13  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    5.02        -0.1%         5.02  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    5.67        -0.7%         5.64  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    5.02        +0.3%         5.04  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    5.01        +0.3%         5.02  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    4.21        -1.0%         4.17  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    4.16        +0.2%         4.17  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    3.28        -4.4%         3.14  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    4.92        -4.2%         4.71  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    4.92        +3.1%         5.08  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    5.45        -0.2%         5.44  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    5.00        -0.4%         4.98  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    2.73      +120.6%         6.03  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    5.68        -1.1%         5.62  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    4.79        -0.3%         4.78  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    5.01        -0.1%         5.01  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                    5.01        -0.1%         5.01  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                    5.72        +1.5%         5.80  bay/thresh=1000M/ext3-100dd-1-3.3.0
                    5.68        +1.6%         5.77  bay/thresh=1000M/ext3-10dd-1-3.3.0
                    5.60        +0.3%         5.61  bay/thresh=1000M/ext3-1dd-1-3.3.0
                    5.50        -0.0%         5.50  bay/thresh=1000M/ext4-100dd-1-3.3.0
                    5.21        -0.3%         5.20  bay/thresh=1000M/ext4-10dd-1-3.3.0
                    5.11        +0.3%         5.12  bay/thresh=1000M/ext4-1dd-1-3.3.0
                    5.29        +0.5%         5.31  bay/thresh=1000M/xfs-100dd-1-3.3.0
                    5.12        -0.1%         5.11  bay/thresh=1000M/xfs-10dd-1-3.3.0
                    5.01        +0.1%         5.01  bay/thresh=1000M/xfs-1dd-1-3.3.0
                    5.06        +0.1%         5.06  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                    4.98        -0.2%         4.98  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                    5.75        +0.4%         5.78  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                    5.60        +0.3%         5.62  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                    5.70        -0.8%         5.65  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                    5.44        +0.7%         5.48  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                    5.26        +0.0%         5.26  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                    5.12        -0.1%         5.11  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                    5.26        +0.5%         5.29  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                    5.10        +0.3%         5.12  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                    5.00        +0.0%         5.00  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                    5.00        +0.6%         5.03  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                    4.86        +0.1%         4.86  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                    5.67        +0.4%         5.69  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                    5.64        -0.1%         5.63  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                    5.68        -0.3%         5.66  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                    5.45        -0.4%         5.43  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                    5.21        +0.3%         5.23  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                    5.10        +0.4%         5.12  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                    5.29        +0.0%         5.29  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                    5.09        +0.1%         5.09  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                    4.98        +0.4%         5.01  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                    4.79        +0.7%         4.82  bay/thresh=100M/btrfs-10dd-1-3.3.0
                    5.00        +0.4%         5.02  bay/thresh=100M/btrfs-1dd-1-3.3.0
                    5.72        -0.8%         5.67  bay/thresh=100M/ext3-100dd-1-3.3.0
                    5.78        -0.5%         5.75  bay/thresh=100M/ext3-10dd-1-3.3.0
                    5.68        +0.1%         5.69  bay/thresh=100M/ext3-1dd-1-3.3.0
                    5.99        +0.2%         6.00  bay/thresh=100M/ext4-100dd-1-3.3.0
                    5.62        -0.4%         5.60  bay/thresh=100M/ext4-10dd-1-3.3.0
                    5.11        -0.3%         5.10  bay/thresh=100M/ext4-1dd-1-3.3.0
                    7.52        +1.0%         7.60  bay/thresh=100M/xfs-100dd-1-3.3.0
                    5.48        -0.2%         5.47  bay/thresh=100M/xfs-10dd-1-3.3.0
                    4.91        +0.1%         4.92  bay/thresh=100M/xfs-1dd-1-3.3.0
                    2.29        +1.4%         2.33  bay/thresh=10M/btrfs-10dd-1-3.3.0
                    3.90        +0.8%         3.93  bay/thresh=10M/btrfs-1dd-1-3.3.0
                    4.99        -0.4%         4.97  bay/thresh=10M/ext3-10dd-1-3.3.0
                    5.47        +0.1%         5.47  bay/thresh=10M/ext3-1dd-1-3.3.0
                    4.77        +0.3%         4.79  bay/thresh=10M/ext4-10dd-1-3.3.0
                    4.99        +0.3%         5.00  bay/thresh=10M/ext4-1dd-1-3.3.0
                    4.96        -0.2%         4.95  bay/thresh=10M/xfs-10dd-1-3.3.0
                    4.53        -1.4%         4.46  bay/thresh=10M/xfs-1dd-1-3.3.0
                    0.15       +27.8%         0.20  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    0.76        +1.8%         0.77  bay/thresh=1M/btrfs-1dd-1-3.3.0
                    1.10        -0.5%         1.10  bay/thresh=1M/ext3-10dd-1-3.3.0
                    3.48        +1.6%         3.54  bay/thresh=1M/ext3-1dd-1-3.3.0
                    0.90        +4.2%         0.94  bay/thresh=1M/ext4-10dd-1-3.3.0
                    0.23        -3.2%         0.22  bay/thresh=1M/ext4-1dd-1-3.3.0
                    0.93        +0.2%         0.93  bay/thresh=1M/xfs-10dd-1-3.3.0
                    2.31        +2.2%         2.36  bay/thresh=1M/xfs-1dd-1-3.3.0
                  582.42        +0.6%       585.86  TOTAL io_svctm

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                  100.00        +0.0%       100.00  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                  100.00        +0.0%       100.00  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                  100.00        +0.0%       100.00  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   99.99        +0.0%       100.00  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   99.82        +0.2%        99.98  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   99.96        -0.0%        99.91  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   99.84        +0.2%        99.99  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                   99.54        +0.2%        99.77  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   99.80        -0.4%        99.44  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   99.96        -0.0%        99.96  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   99.90        -0.1%        99.76  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   99.93        -0.0%        99.92  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   99.75        +0.2%        99.95  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   95.24        +1.7%        96.88  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   97.88        +0.2%        98.08  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   97.20        +0.4%        97.55  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   68.06        +1.0%        68.76  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                   70.42        -0.2%        70.30  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   63.42        +0.0%        63.43  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   92.04        +4.6%        96.24  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   97.81        -0.3%        97.50  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                   95.58        +0.4%        95.94  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   97.33        -0.3%        97.08  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   97.17        -0.1%        97.08  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   92.63        -0.8%        91.86  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   87.80        -0.9%        87.04  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   94.67        +0.7%        95.36  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   85.37        -0.9%        84.58  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   97.20        +0.1%        97.29  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   90.30        -1.3%        89.13  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   86.88        -7.6%        80.24  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   93.17        -0.6%        92.64  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   92.85        -0.0%        92.81  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   98.34        -0.2%        98.19  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   97.82        +0.1%        97.96  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   77.28       -11.9%        68.09  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   87.47        +2.2%        89.43  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   95.83        +0.6%        96.41  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   97.05        -0.2%        96.87  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                   97.43        +0.1%        97.57  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                   97.68        -0.5%        97.23  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   96.36        +0.4%        96.75  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   97.57        +0.2%        97.81  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   96.03        -0.3%        95.74  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   96.00        +0.1%        96.08  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   58.53        -5.0%        55.57  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   86.28        -4.9%        82.08  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   81.02        +2.8%        83.30  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   96.08        -0.5%        95.57  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   96.62        -0.2%        96.45  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   37.04      +123.1%        82.66  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   99.23        -0.5%        98.70  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   96.24        -0.2%        96.01  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                  100.00        +0.0%       100.00  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                  100.00        +0.0%       100.00  bay/thresh=1000M/ext3-100dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/thresh=1000M/ext3-10dd-1-3.3.0
                   99.99        +0.0%       100.00  bay/thresh=1000M/ext3-1dd-1-3.3.0
                  100.00        +0.0%       100.00  bay/thresh=1000M/ext4-100dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/thresh=1000M/ext4-10dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/thresh=1000M/ext4-1dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/thresh=1000M/xfs-100dd-1-3.3.0
                  100.00        +0.0%       100.00  bay/thresh=1000M/xfs-10dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/thresh=1000M/xfs-1dd-1-3.3.0
                   99.46        +0.0%        99.50  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                   99.58        -0.0%        99.56  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                   99.95        +0.0%        99.97  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                   99.94        -0.0%        99.94  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                   99.29        +0.0%        99.29  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                  100.00        -0.0%        99.98  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                   99.98        -0.1%        99.92  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                   99.94        +0.1%       100.00  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                   99.43        +0.2%        99.65  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                   99.42        -0.2%        99.26  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                   99.71        -0.0%        99.67  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                   97.00        +0.3%        97.26  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                   94.11        +0.5%        94.58  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                   98.98        +0.1%        99.07  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                   99.97        +0.0%        99.99  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                   99.24        +0.1%        99.29  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                   98.88        -0.1%        98.77  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                   99.99        +0.0%        99.99  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                   99.88        -0.6%        99.32  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                   98.85        -0.1%        98.80  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                   98.84        -0.4%        98.47  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                   99.76        -0.1%        99.67  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                  100.00        +0.0%       100.00  bay/thresh=100M/btrfs-10dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/thresh=100M/btrfs-1dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/thresh=100M/ext3-100dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/thresh=100M/ext3-10dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/thresh=100M/ext3-1dd-1-3.3.0
                   99.99        +0.0%        99.99  bay/thresh=100M/ext4-100dd-1-3.3.0
                   99.99        +0.0%       100.00  bay/thresh=100M/ext4-10dd-1-3.3.0
                   99.88        +0.1%        99.99  bay/thresh=100M/ext4-1dd-1-3.3.0
                  100.00        +0.0%       100.00  bay/thresh=100M/xfs-100dd-1-3.3.0
                  100.00        +0.0%       100.00  bay/thresh=100M/xfs-10dd-1-3.3.0
                   99.99        +0.0%       100.00  bay/thresh=100M/xfs-1dd-1-3.3.0
                   90.35        -0.1%        90.25  bay/thresh=10M/btrfs-10dd-1-3.3.0
                   85.50        +0.6%        86.04  bay/thresh=10M/btrfs-1dd-1-3.3.0
                  100.00        +0.0%       100.00  bay/thresh=10M/ext3-10dd-1-3.3.0
                   99.84        +0.0%        99.86  bay/thresh=10M/ext3-1dd-1-3.3.0
                   99.99        -0.0%        99.99  bay/thresh=10M/ext4-10dd-1-3.3.0
                   99.92        +0.0%        99.93  bay/thresh=10M/ext4-1dd-1-3.3.0
                  100.00        -0.0%       100.00  bay/thresh=10M/xfs-10dd-1-3.3.0
                   92.93        +0.7%        93.54  bay/thresh=10M/xfs-1dd-1-3.3.0
                    1.42       +18.4%         1.69  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    1.99        -9.4%         1.80  bay/thresh=1M/btrfs-1dd-1-3.3.0
                   96.63        -0.2%        96.42  bay/thresh=1M/ext3-10dd-1-3.3.0
                   73.96        +0.5%        74.32  bay/thresh=1M/ext3-1dd-1-3.3.0
                   96.13        -0.0%        96.09  bay/thresh=1M/ext4-10dd-1-3.3.0
                   75.69        -1.3%        74.73  bay/thresh=1M/ext4-1dd-1-3.3.0
                   96.36        +0.1%        96.50  bay/thresh=1M/xfs-10dd-1-3.3.0
                   72.08        +0.6%        72.47  bay/thresh=1M/xfs-1dd-1-3.3.0
                11342.21        +0.3%     11372.47  TOTAL io_util

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                    2.90        -0.9%         2.88  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    1.85        -1.3%         1.82  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    3.86        -1.3%         3.81  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    1.70        -0.1%         1.70  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    1.47        +2.6%         1.51  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    4.89        +0.8%         4.94  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    2.27        +0.5%         2.28  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    1.67        -0.9%         1.65  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    4.68        +1.2%         4.74  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    2.36        -2.1%         2.31  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    1.69        -2.0%         1.66  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    3.67        -4.3%         3.51  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    2.07        -2.4%         2.02  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    9.96       -12.2%         8.74  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                    2.53        +5.5%         2.67  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    1.74        +0.2%         1.74  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    9.95        +2.7%        10.22  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    2.90        -0.0%         2.90  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    1.97        +1.6%         2.00  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    5.97        +0.6%         6.00  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    2.89        -1.9%         2.83  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    2.25        -0.3%         2.24  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    2.08        -2.1%         2.04  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    1.63        -1.9%         1.60  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    3.60        -0.0%         3.60  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    1.29        -1.4%         1.27  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.92        +1.1%         0.93  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    4.87        -3.7%         4.70  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    1.69        +0.8%         1.71  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    4.63        +0.3%         4.64  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    1.75        -2.7%         1.71  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    1.51        -3.3%         1.46  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    2.21        -2.1%         2.17  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    1.85        -2.9%         1.80  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    1.67        -2.3%         1.63  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    4.83        -5.0%         4.59  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    1.95        +0.2%         1.96  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    1.67        -2.2%         1.63  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    3.21        +1.2%         3.25  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    1.86        -0.4%         1.85  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    2.13        +1.5%         2.16  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    1.29        -0.4%         1.28  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.88        -1.8%         0.86  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.81        +1.3%         0.82  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.67        +0.7%         0.68  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    1.89        +2.5%         1.93  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.99        -3.2%         0.96  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.75        +1.6%         0.76  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    1.98        -0.2%         1.98  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    1.07        -3.1%         1.04  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.75        -1.3%         0.74  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    1.35        -3.0%         1.31  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.99        -2.6%         0.96  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    1.04        +1.1%         1.05  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.81        +1.2%         0.82  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    2.86        -1.6%         2.81  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    1.18        -0.6%         1.18  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.99        +1.1%         1.00  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    2.05        -1.1%         2.03  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    1.23        +0.3%         1.23  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    1.16        -3.8%         1.11  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    1.34        -1.0%         1.33  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                    0.88        -2.8%         0.86  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                    1.72        +0.0%         1.72  bay/thresh=1000M/ext3-100dd-1-3.3.0
                    0.81        +0.8%         0.82  bay/thresh=1000M/ext3-10dd-1-3.3.0
                    0.68        +0.8%         0.68  bay/thresh=1000M/ext3-1dd-1-3.3.0
                    2.20        +0.8%         2.22  bay/thresh=1000M/ext4-100dd-1-3.3.0
                    1.09        -0.3%         1.08  bay/thresh=1000M/ext4-10dd-1-3.3.0
                    0.78        +2.9%         0.80  bay/thresh=1000M/ext4-1dd-1-3.3.0
                    2.41        -1.4%         2.37  bay/thresh=1000M/xfs-100dd-1-3.3.0
                    1.11        -0.6%         1.10  bay/thresh=1000M/xfs-10dd-1-3.3.0
                    0.81        -4.1%         0.77  bay/thresh=1000M/xfs-1dd-1-3.3.0
                    1.51        +1.2%         1.53  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                    1.05        -2.3%         1.02  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                    2.73        +2.0%         2.78  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                    1.21        +0.3%         1.21  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                    0.85        +4.1%         0.89  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                    3.14        -3.4%         3.03  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                    1.34        +0.2%         1.34  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                    0.92        +5.0%         0.97  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                    3.65        -0.8%         3.62  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                    1.12        -1.2%         1.10  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                    1.01        -1.2%         1.00  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                    1.39        -0.6%         1.39  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                    1.25        -1.4%         1.23  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                    2.22        -1.8%         2.18  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                    1.18        -0.6%         1.18  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                    0.99        +1.7%         1.01  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                    2.17        +7.2%         2.32  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                    1.35        -0.7%         1.34  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                    1.30        -3.3%         1.25  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                    2.50        -0.8%         2.48  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                    1.16        -0.4%         1.16  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                    1.09        -2.6%         1.06  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                    1.37        -3.3%         1.33  bay/thresh=100M/btrfs-10dd-1-3.3.0
                    0.98        -1.0%         0.97  bay/thresh=100M/btrfs-1dd-1-3.3.0
                    3.02        -2.8%         2.93  bay/thresh=100M/ext3-100dd-1-3.3.0
                    1.02        -2.4%         0.99  bay/thresh=100M/ext3-10dd-1-3.3.0
                    0.83        +2.5%         0.85  bay/thresh=100M/ext3-1dd-1-3.3.0
                    2.94        -1.7%         2.89  bay/thresh=100M/ext4-100dd-1-3.3.0
                    1.13        +2.3%         1.16  bay/thresh=100M/ext4-10dd-1-3.3.0
                    0.90        +4.7%         0.94  bay/thresh=100M/ext4-1dd-1-3.3.0
                    1.68        -2.5%         1.64  bay/thresh=100M/xfs-100dd-1-3.3.0
                    1.22        -1.6%         1.20  bay/thresh=100M/xfs-10dd-1-3.3.0
                    1.12        -1.1%         1.10  bay/thresh=100M/xfs-1dd-1-3.3.0
                    8.58        -0.2%         8.56  bay/thresh=10M/btrfs-10dd-1-3.3.0
                    4.91        +0.3%         4.92  bay/thresh=10M/btrfs-1dd-1-3.3.0
                    3.35        +1.3%         3.39  bay/thresh=10M/ext3-10dd-1-3.3.0
                    2.76        +1.5%         2.80  bay/thresh=10M/ext3-1dd-1-3.3.0
                    3.69        +0.4%         3.71  bay/thresh=10M/ext4-10dd-1-3.3.0
                    3.09        +4.2%         3.22  bay/thresh=10M/ext4-1dd-1-3.3.0
                    3.95        +0.6%         3.97  bay/thresh=10M/xfs-10dd-1-3.3.0
                    3.06        +4.6%         3.20  bay/thresh=10M/xfs-1dd-1-3.3.0
                    2.30        +3.9%         2.39  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    0.77        -6.2%         0.72  bay/thresh=1M/btrfs-1dd-1-3.3.0
                   18.42        +0.3%        18.48  bay/thresh=1M/ext3-10dd-1-3.3.0
                    4.00        -3.2%         3.87  bay/thresh=1M/ext3-1dd-1-3.3.0
                   21.23        -4.2%        20.33  bay/thresh=1M/ext4-10dd-1-3.3.0
                    4.60        +3.4%         4.76  bay/thresh=1M/ext4-1dd-1-3.3.0
                   20.07        -0.2%        20.02  bay/thresh=1M/xfs-10dd-1-3.3.0
                    5.55        -4.7%         5.29  bay/thresh=1M/xfs-1dd-1-3.3.0
                  320.82        -0.9%       317.94  TOTAL cpu_user

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/xfs-1dd-1-3.3.0
                    0.00                      0.00  TOTAL cpu_nice

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                   67.68        +0.5%        68.04  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   68.53        +0.9%        69.12  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   22.77        +0.4%        22.87  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                   32.20        +0.7%        32.41  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   40.05        +0.9%        40.42  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   15.96        -0.7%        15.86  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   30.41        +2.1%        31.04  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                   34.98        +1.2%        35.41  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                   13.32        -0.6%        13.23  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   23.10        -0.3%        23.04  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   27.41        +0.5%        27.55  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   68.81        +0.5%        69.13  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   69.25        +0.9%        69.86  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   16.11        -3.7%        15.51  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                   31.89        -2.3%        31.16  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   37.91        +0.6%        38.14  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   11.61        -0.6%        11.54  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   23.94        +0.0%        23.95  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   33.71        +0.5%        33.88  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    6.05        -0.7%         6.01  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   20.63        -1.1%        20.39  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   28.29        +0.6%        28.46  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   70.31        +0.7%        70.82  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   70.65        +0.8%        71.22  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   23.36        -1.3%        23.04  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                   32.65        +0.9%        32.93  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   34.76        -0.3%        34.65  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   15.31       +11.8%        17.12  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   32.61        +1.0%        32.93  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                   13.21        +0.3%        13.25  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   21.59       +23.6%        26.69  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   28.41        +0.4%        28.53  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   74.84        +0.1%        74.90  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   71.97        -7.7%        66.41  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   34.62        +0.8%        34.90  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   11.62        +3.1%        11.98  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   26.03        +0.4%        26.14  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   33.52        +0.3%        33.63  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    5.75        -1.4%         5.67  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   21.24       -14.7%        18.12  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   28.34        +1.4%        28.73  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   34.17        +0.4%        34.30  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   32.94        +9.1%        35.94  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   20.55        +0.1%        20.56  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   23.30        -6.6%        21.75  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    5.92        -0.5%         5.89  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   15.79        +1.5%        16.03  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                   16.47        +0.6%        16.57  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    4.84        -0.9%         4.80  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   10.64        +0.0%        10.64  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   12.84        +0.1%        12.86  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   34.64        +0.2%        34.72  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   37.41        +0.9%        37.75  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   18.82        -0.3%        18.76  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   23.41        +1.0%        23.65  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    5.61        -1.6%         5.52  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   11.60        +0.5%        11.66  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   16.49        +1.1%        16.68  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    2.50        +0.3%         2.51  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   10.09        +0.3%        10.12  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   14.13        +0.6%        14.21  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   32.36        +0.8%        32.61  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                   33.93        -9.4%        30.73  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                    9.28        -1.6%         9.13  bay/thresh=1000M/ext3-100dd-1-3.3.0
                   15.41        +0.4%        15.48  bay/thresh=1000M/ext3-10dd-1-3.3.0
                   18.35        +0.6%        18.45  bay/thresh=1000M/ext3-1dd-1-3.3.0
                    5.57        -0.3%         5.56  bay/thresh=1000M/ext4-100dd-1-3.3.0
                   14.33        -1.0%        14.18  bay/thresh=1000M/ext4-10dd-1-3.3.0
                   16.88        +0.7%        17.00  bay/thresh=1000M/ext4-1dd-1-3.3.0
                    5.72        -0.4%         5.70  bay/thresh=1000M/xfs-100dd-1-3.3.0
                   10.50        +0.3%        10.53  bay/thresh=1000M/xfs-10dd-1-3.3.0
                   13.31        +0.5%        13.37  bay/thresh=1000M/xfs-1dd-1-3.3.0
                   36.77        +1.4%        37.30  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                   34.31        +0.1%        34.35  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                   13.78        +0.7%        13.88  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                   18.24        +0.9%        18.40  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                   18.47        +0.2%        18.51  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                   11.43        +1.7%        11.63  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                   18.59        +1.3%        18.83  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                   17.36        +2.1%        17.71  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                   13.37        -0.7%        13.27  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                   13.85        +0.2%        13.87  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                   14.30        +1.4%        14.51  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                   37.44        +0.1%        37.49  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                   33.21        -6.5%        31.05  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                   18.27        -0.1%        18.25  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                   18.91        -0.6%        18.81  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                   18.62        +0.5%        18.72  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                   16.50        +0.9%        16.65  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                   18.27        +0.8%        18.41  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                   18.21        -2.5%        17.75  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                   14.43        +0.0%        14.43  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                   14.72        -0.5%        14.65  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                   14.61        -0.3%        14.57  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                   32.34        +0.2%        32.40  bay/thresh=100M/btrfs-10dd-1-3.3.0
                   34.20        +0.6%        34.39  bay/thresh=100M/btrfs-1dd-1-3.3.0
                    9.51        -1.0%         9.42  bay/thresh=100M/ext3-100dd-1-3.3.0
                   14.54        +7.3%        15.61  bay/thresh=100M/ext3-10dd-1-3.3.0
                   18.29        +1.1%        18.50  bay/thresh=100M/ext3-1dd-1-3.3.0
                    5.36        +0.3%         5.38  bay/thresh=100M/ext4-100dd-1-3.3.0
                   11.21        -2.4%        10.94  bay/thresh=100M/ext4-10dd-1-3.3.0
                   15.86        +2.8%        16.29  bay/thresh=100M/ext4-1dd-1-3.3.0
                    2.52        -2.4%         2.46  bay/thresh=100M/xfs-100dd-1-3.3.0
                    9.99        +0.2%        10.01  bay/thresh=100M/xfs-10dd-1-3.3.0
                   14.13        +0.3%        14.18  bay/thresh=100M/xfs-1dd-1-3.3.0
                   41.85        -0.4%        41.66  bay/thresh=10M/btrfs-10dd-1-3.3.0
                   37.24        +0.6%        37.47  bay/thresh=10M/btrfs-1dd-1-3.3.0
                   12.42        -1.0%        12.29  bay/thresh=10M/ext3-10dd-1-3.3.0
                   18.25        +0.8%        18.40  bay/thresh=10M/ext3-1dd-1-3.3.0
                    6.71        +0.5%         6.75  bay/thresh=10M/ext4-10dd-1-3.3.0
                   16.21        +0.7%        16.32  bay/thresh=10M/ext4-1dd-1-3.3.0
                    5.40        +1.3%         5.47  bay/thresh=10M/xfs-10dd-1-3.3.0
                   13.69        -6.7%        12.77  bay/thresh=10M/xfs-1dd-1-3.3.0
                    1.33        +8.5%         1.45  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    1.62       -10.8%         1.45  bay/thresh=1M/btrfs-1dd-1-3.3.0
                   16.89        +1.3%        17.11  bay/thresh=1M/ext3-10dd-1-3.3.0
                   14.27        +0.2%        14.30  bay/thresh=1M/ext3-1dd-1-3.3.0
                   15.11        -3.0%        14.66  bay/thresh=1M/ext4-10dd-1-3.3.0
                   26.57        +1.9%        27.08  bay/thresh=1M/ext4-1dd-1-3.3.0
                   12.52        +0.6%        12.60  bay/thresh=1M/xfs-10dd-1-3.3.0
                   11.56        +0.5%        11.62  bay/thresh=1M/xfs-1dd-1-3.3.0
                 2756.42        +0.1%      2760.34  TOTAL cpu_system

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                   29.03        -1.2%        28.68  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   20.49        -0.4%        20.41  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   73.37        -0.1%        73.32  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                   66.00        -0.4%        65.77  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   38.07        -0.6%        37.83  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   79.14        -0.0%        79.11  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   66.89        -0.8%        66.34  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                   49.34        -2.4%        48.15  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                   82.00        +0.0%        82.03  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   74.40        +0.1%        74.50  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   47.04        -1.9%        46.16  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   27.13        -0.8%        26.92  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   13.28        -1.4%        13.10  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   73.93        +2.5%        75.75  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                   65.11        +0.7%        65.55  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   22.04        -0.8%        21.86  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   78.44        -0.3%        78.24  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   72.81        -0.1%        72.76  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   23.07        -0.1%        23.05  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   87.98        +0.0%        87.99  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   76.21        +0.4%        76.52  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   26.35        -0.2%        26.30  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   26.94        -1.7%        26.49  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   11.32        -2.1%        11.08  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   72.82        +0.4%        73.14  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                   63.71        -0.7%        63.25  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    3.72        -5.2%         3.53  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   79.04        -9.9%        71.23  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   59.08        +6.0%        62.61  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                   82.16        -0.1%        82.11  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   74.08        -6.0%        69.64  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   24.50        -1.7%        24.09  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   21.95        -0.1%        21.93  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    3.58       +61.1%         5.76  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   59.32        -0.3%        59.14  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   83.22        -5.8%        78.41  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   68.40        -0.2%        68.28  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    2.34        -4.1%         2.24  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   91.03        +0.1%        91.08  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   73.73        +3.8%        76.53  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    5.34        -5.5%         5.05  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   60.68        -0.0%        60.68  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   29.45        -2.8%        28.63  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                   73.51        -0.0%        73.49  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   30.55        +2.3%        31.25  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                   82.25        +0.3%        82.47  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                   74.03        -3.1%        71.71  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                   41.09        +0.4%        41.24  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                   93.17        +0.0%        93.22  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                   84.01        -0.3%        83.76  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   36.19        +0.8%        36.48  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                   59.25        +0.3%        59.44  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   17.85        -1.3%        17.62  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                   73.85        +0.1%        73.96  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   17.29        +1.0%        17.47  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                   91.05        +0.2%        91.25  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                   82.55        -0.1%        82.46  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   18.62        -1.2%        18.39  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                   95.45        +0.0%        95.46  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                   84.76        +0.1%        84.81  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   15.21        -0.7%        15.11  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                   61.74        -0.1%        61.69  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                   27.07        +6.0%        28.69  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                   88.82        +0.2%        88.97  bay/thresh=1000M/ext3-100dd-1-3.3.0
                   80.26        -0.2%        80.11  bay/thresh=1000M/ext3-10dd-1-3.3.0
                   32.92        -0.6%        32.74  bay/thresh=1000M/ext3-1dd-1-3.3.0
                   91.53        +0.1%        91.62  bay/thresh=1000M/ext4-100dd-1-3.3.0
                   79.35        +0.3%        79.59  bay/thresh=1000M/ext4-10dd-1-3.3.0
                   44.88        -2.0%        43.96  bay/thresh=1000M/ext4-1dd-1-3.3.0
                   91.87        +0.1%        91.93  bay/thresh=1000M/xfs-100dd-1-3.3.0
                   82.01        +1.2%        82.97  bay/thresh=1000M/xfs-10dd-1-3.3.0
                   34.90        -0.3%        34.80  bay/thresh=1000M/xfs-1dd-1-3.3.0
                   55.92        -0.8%        55.46  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                   24.53        +1.6%        24.93  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                   83.31        -0.2%        83.18  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                   76.08        -0.0%        76.08  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                   32.29        -0.4%        32.15  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                   85.08        -3.5%        82.07  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                   75.08        -1.2%        74.16  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                   43.62        -2.1%        42.72  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                   82.98        +0.1%        83.07  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                   82.41        -0.1%        82.31  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                   34.46        -2.5%        33.59  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                   51.23        -0.7%        50.85  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                   24.15        +2.9%        24.86  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                   77.15        +0.2%        77.32  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                   73.91        +0.7%        74.42  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                   31.63        +1.0%        31.96  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                   71.85        +3.2%        74.16  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                   68.08        -0.6%        67.67  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                   41.40        +5.8%        43.78  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                   77.17        -0.3%        76.94  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                   78.06        +0.1%        78.11  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                   33.36        +0.4%        33.50  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                   60.54        +0.1%        60.60  bay/thresh=100M/btrfs-10dd-1-3.3.0
                   18.48        -1.3%        18.24  bay/thresh=100M/btrfs-1dd-1-3.3.0
                   87.40        +0.2%        87.57  bay/thresh=100M/ext3-100dd-1-3.3.0
                   80.40        -1.5%        79.22  bay/thresh=100M/ext3-10dd-1-3.3.0
                   18.38        -5.3%        17.41  bay/thresh=100M/ext3-1dd-1-3.3.0
                   91.14        -0.3%        90.83  bay/thresh=100M/ext4-100dd-1-3.3.0
                   82.76        -0.0%        82.73  bay/thresh=100M/ext4-10dd-1-3.3.0
                   20.46        -0.8%        20.28  bay/thresh=100M/ext4-1dd-1-3.3.0
                   95.81        +0.1%        95.91  bay/thresh=100M/xfs-100dd-1-3.3.0
                   85.15        -0.1%        85.10  bay/thresh=100M/xfs-10dd-1-3.3.0
                   15.35        -0.2%        15.32  bay/thresh=100M/xfs-1dd-1-3.3.0
                   45.79        +0.7%        46.12  bay/thresh=10M/btrfs-10dd-1-3.3.0
                   13.64        -0.8%        13.53  bay/thresh=10M/btrfs-1dd-1-3.3.0
                   79.83        +0.3%        80.08  bay/thresh=10M/ext3-10dd-1-3.3.0
                   13.55        -1.6%        13.34  bay/thresh=10M/ext3-1dd-1-3.3.0
                   85.98        -0.0%        85.94  bay/thresh=10M/ext4-10dd-1-3.3.0
                   13.67        +1.1%        13.81  bay/thresh=10M/ext4-1dd-1-3.3.0
                   87.14        -0.5%        86.66  bay/thresh=10M/xfs-10dd-1-3.3.0
                   14.92        +6.2%        15.85  bay/thresh=10M/xfs-1dd-1-3.3.0
                   89.58        +2.1%        91.46  bay/thresh=1M/btrfs-10dd-1-3.3.0
                   24.35        +0.5%        24.48  bay/thresh=1M/btrfs-1dd-1-3.3.0
                   59.50        -0.6%        59.12  bay/thresh=1M/ext3-10dd-1-3.3.0
                   14.78        -0.5%        14.71  bay/thresh=1M/ext3-1dd-1-3.3.0
                   58.06        +3.0%        59.78  bay/thresh=1M/ext4-10dd-1-3.3.0
                    8.37        -2.5%         8.16  bay/thresh=1M/ext4-1dd-1-3.3.0
                   61.30        -0.1%        61.23  bay/thresh=1M/xfs-10dd-1-3.3.0
                   16.67        -0.3%        16.63  bay/thresh=1M/xfs-1dd-1-3.3.0
                 6621.34        -0.2%      6609.32  TOTAL cpu_iowait

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext3-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext4-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/xfs-100dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=100M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=10M/xfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/xfs-10dd-1-3.3.0
                    0.00                      0.00  bay/thresh=1M/xfs-1dd-1-3.3.0
                    0.00                      0.00  TOTAL cpu_steal

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                    0.39        +2.7%         0.40  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                    9.14        -5.4%         8.65  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    0.09       +23.5%         0.12  bay/JBOD-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   20.41        -0.8%        20.24  bay/JBOD-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.01     +1636.2%         0.09  bay/JBOD-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    0.43       -20.7%         0.34  bay/JBOD-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                   14.00        +5.6%        14.79  bay/JBOD-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    0.14        +9.1%         0.15  bay/JBOD-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   23.86        +3.2%        24.63  bay/JBOD-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    0.39       +14.2%         0.44  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   15.40        -2.4%        15.02  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    0.01                      0.00  bay/JBOD-2HDD-thresh=100M/ext3-100dd-1-3.3.0
                    0.46       +34.1%         0.62  bay/JBOD-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   38.31        -0.1%        38.26  bay/JBOD-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    0.35       +14.5%         0.40  bay/JBOD-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   41.26        -0.4%        41.08  bay/JBOD-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/JBOD-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    0.28        -9.3%         0.25  bay/JBOD-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   43.12        -0.3%        43.00  bay/JBOD-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    0.67        -3.8%         0.65  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   16.40        -1.9%        16.09  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    0.22        -4.5%         0.21  bay/RAID0-2HDD-thresh=1000M/ext3-100dd-1-3.3.0
                    2.35        +8.4%         2.55  bay/RAID0-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   60.60        +0.5%        60.89  bay/RAID0-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    0.78      +793.4%         6.95  bay/RAID0-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    6.61       -58.5%         2.75  bay/RAID0-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    2.58       -23.6%         1.97  bay/RAID0-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   45.58        +0.7%        45.91  bay/RAID0-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    1.00        +0.1%         1.00  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   22.60       +15.1%        26.03  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    4.38        -1.3%         4.33  bay/RAID0-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                    0.00                      0.00  bay/RAID0-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.33     +1435.4%         5.02  bay/RAID0-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    3.62        +0.2%         3.63  bay/RAID0-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   62.47        +0.0%        62.50  bay/RAID0-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.01       -43.3%         0.01  bay/RAID0-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    3.17       +10.4%         3.49  bay/RAID0-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   64.19        -0.2%        64.07  bay/RAID0-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    3.87        -3.2%         3.74  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.3.0
                   36.74        -5.9%        34.57  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.3.0
                    5.13        -0.0%         5.13  bay/RAID1-2HDD-thresh=1000M/ext3-10dd-1-3.3.0
                   45.48        +1.9%        46.33  bay/RAID1-2HDD-thresh=1000M/ext3-1dd-1-3.3.0
                    9.95        -2.5%         9.70  bay/RAID1-2HDD-thresh=1000M/ext4-100dd-1-3.3.0
                    9.19       +23.0%        11.30  bay/RAID1-2HDD-thresh=1000M/ext4-10dd-1-3.3.0
                   41.69        -0.6%        41.43  bay/RAID1-2HDD-thresh=1000M/ext4-1dd-1-3.3.0
                    0.00       -25.0%         0.00  bay/RAID1-2HDD-thresh=1000M/xfs-100dd-1-3.3.0
                    4.28        +6.4%         4.56  bay/RAID1-2HDD-thresh=1000M/xfs-10dd-1-3.3.0
                   50.22        -0.6%        49.93  bay/RAID1-2HDD-thresh=1000M/xfs-1dd-1-3.3.0
                    4.75        -4.6%         4.53  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.3.0
                   43.75        -0.2%        43.67  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.3.0
                    6.29        -0.9%         6.23  bay/RAID1-2HDD-thresh=100M/ext3-10dd-1-3.3.0
                   58.49        -0.7%        58.07  bay/RAID1-2HDD-thresh=100M/ext3-1dd-1-3.3.0
                    0.49       -12.5%         0.43  bay/RAID1-2HDD-thresh=100M/ext4-100dd-1-3.3.0
                    4.68        +0.6%         4.70  bay/RAID1-2HDD-thresh=100M/ext4-10dd-1-3.3.0
                   63.90        +0.0%        63.93  bay/RAID1-2HDD-thresh=100M/ext4-1dd-1-3.3.0
                    0.00      +252.8%         0.00  bay/RAID1-2HDD-thresh=100M/xfs-100dd-1-3.3.0
                    3.93        -2.1%         3.85  bay/RAID1-2HDD-thresh=100M/xfs-10dd-1-3.3.0
                   69.50        +0.1%        69.56  bay/RAID1-2HDD-thresh=100M/xfs-1dd-1-3.3.0
                    4.55        -4.2%         4.36  bay/thresh=1000M/btrfs-10dd-1-3.3.0
                   38.11        +4.2%        39.71  bay/thresh=1000M/btrfs-1dd-1-3.3.0
                    0.17        -0.3%         0.17  bay/thresh=1000M/ext3-100dd-1-3.3.0
                    3.52        +2.0%         3.59  bay/thresh=1000M/ext3-10dd-1-3.3.0
                   48.05        +0.2%        48.13  bay/thresh=1000M/ext3-1dd-1-3.3.0
                    0.69       -12.1%         0.61  bay/thresh=1000M/ext4-100dd-1-3.3.0
                    5.23        -1.7%         5.15  bay/thresh=1000M/ext4-10dd-1-3.3.0
                   37.46        +2.1%        38.23  bay/thresh=1000M/ext4-1dd-1-3.3.0
                    0.00       -60.3%         0.00  bay/thresh=1000M/xfs-100dd-1-3.3.0
                    6.38       -15.5%         5.39  bay/thresh=1000M/xfs-10dd-1-3.3.0
                   50.98        +0.1%        51.05  bay/thresh=1000M/xfs-1dd-1-3.3.0
                    5.80        -1.4%         5.72  bay/thresh=1000M:990M/btrfs-10dd-1-3.3.0
                   40.10        -1.0%        39.70  bay/thresh=1000M:990M/btrfs-1dd-1-3.3.0
                    0.18       -13.9%         0.16  bay/thresh=1000M:990M/ext3-100dd-1-3.3.0
                    4.48        -3.7%         4.31  bay/thresh=1000M:990M/ext3-10dd-1-3.3.0
                   48.38        +0.1%        48.45  bay/thresh=1000M:990M/ext3-1dd-1-3.3.0
                    0.34      +858.6%         3.26  bay/thresh=1000M:990M/ext4-100dd-1-3.3.0
                    4.99       +13.5%         5.67  bay/thresh=1000M:990M/ext4-10dd-1-3.3.0
                   38.10        +1.3%        38.60  bay/thresh=1000M:990M/ext4-1dd-1-3.3.0
                    0.00      +806.5%         0.04  bay/thresh=1000M:990M/xfs-100dd-1-3.3.0
                    2.62        +3.6%         2.71  bay/thresh=1000M:990M/xfs-10dd-1-3.3.0
                   50.22        +1.3%        50.90  bay/thresh=1000M:990M/xfs-1dd-1-3.3.0
                    9.93        +3.4%        10.27  bay/thresh=1000M:999M/btrfs-10dd-1-3.3.0
                   41.39        +3.6%        42.86  bay/thresh=1000M:999M/btrfs-1dd-1-3.3.0
                    2.35        -4.7%         2.24  bay/thresh=1000M:999M/ext3-100dd-1-3.3.0
                    5.99        -6.6%         5.60  bay/thresh=1000M:999M/ext3-10dd-1-3.3.0
                   48.76        -0.9%        48.31  bay/thresh=1000M:999M/ext3-1dd-1-3.3.0
                    9.48       -27.5%         6.87  bay/thresh=1000M:999M/ext4-100dd-1-3.3.0
                   12.30        +2.3%        12.58  bay/thresh=1000M:999M/ext4-10dd-1-3.3.0
                   39.10        -4.8%        37.22  bay/thresh=1000M:999M/ext4-1dd-1-3.3.0
                    5.90        +4.2%         6.15  bay/thresh=1000M:999M/xfs-100dd-1-3.3.0
                    6.06        +0.4%         6.09  bay/thresh=1000M:999M/xfs-10dd-1-3.3.0
                   50.93        -0.1%        50.87  bay/thresh=1000M:999M/xfs-1dd-1-3.3.0
                    5.75        -1.3%         5.68  bay/thresh=100M/btrfs-10dd-1-3.3.0
                   46.35        +0.1%        46.40  bay/thresh=100M/btrfs-1dd-1-3.3.0
                    0.07        +6.8%         0.07  bay/thresh=100M/ext3-100dd-1-3.3.0
                    4.04        +3.4%         4.18  bay/thresh=100M/ext3-10dd-1-3.3.0
                   62.49        +1.2%        63.23  bay/thresh=100M/ext3-1dd-1-3.3.0
                    0.56       +60.1%         0.90  bay/thresh=100M/ext4-100dd-1-3.3.0
                    4.90        +5.7%         5.17  bay/thresh=100M/ext4-10dd-1-3.3.0
                   62.79        -0.5%        62.48  bay/thresh=100M/ext4-1dd-1-3.3.0
                    0.00       +50.0%         0.00  bay/thresh=100M/xfs-100dd-1-3.3.0
                    3.64        +1.2%         3.69  bay/thresh=100M/xfs-10dd-1-3.3.0
                   69.40        -0.0%        69.40  bay/thresh=100M/xfs-1dd-1-3.3.0
                    3.79        -3.3%         3.66  bay/thresh=10M/btrfs-10dd-1-3.3.0
                   44.21        -0.3%        44.08  bay/thresh=10M/btrfs-1dd-1-3.3.0
                    4.40        -3.7%         4.24  bay/thresh=10M/ext3-10dd-1-3.3.0
                   65.44        +0.0%        65.46  bay/thresh=10M/ext3-1dd-1-3.3.0
                    3.61        -0.2%         3.60  bay/thresh=10M/ext4-10dd-1-3.3.0
                   67.03        -0.6%        66.64  bay/thresh=10M/ext4-1dd-1-3.3.0
                    3.52       +10.7%         3.90  bay/thresh=10M/xfs-10dd-1-3.3.0
                   68.33        -0.2%        68.17  bay/thresh=10M/xfs-1dd-1-3.3.0
                    6.79       -30.6%         4.71  bay/thresh=1M/btrfs-10dd-1-3.3.0
                   73.26        +0.1%        73.35  bay/thresh=1M/btrfs-1dd-1-3.3.0
                    5.19        +1.8%         5.28  bay/thresh=1M/ext3-10dd-1-3.3.0
                   66.95        +0.3%        67.12  bay/thresh=1M/ext3-1dd-1-3.3.0
                    5.60        -6.8%         5.22  bay/thresh=1M/ext4-10dd-1-3.3.0
                   60.46        -0.8%        60.00  bay/thresh=1M/ext4-1dd-1-3.3.0
                    6.12        +0.6%         6.15  bay/thresh=1M/xfs-10dd-1-3.3.0
                   66.22        +0.4%        66.46  bay/thresh=1M/xfs-1dd-1-3.3.0
                 2401.42        +0.5%      2412.41  TOTAL cpu_idle


^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [RESEND][PATCH v2] block: remove plugging at buffered write time
  2012-04-12  2:20       ` Fengguang Wu
@ 2012-04-12 14:26         ` Jan Kara
  2012-04-13  1:40           ` Fengguang Wu
  2012-05-06  6:01           ` [RESEND][PATCH v2] block: remove plugging at buffered write time Fengguang Wu
  0 siblings, 2 replies; 16+ messages in thread
From: Jan Kara @ 2012-04-12 14:26 UTC (permalink / raw)
  To: Fengguang Wu
  Cc: Andrew Morton, Jeff Moyer, Jens Axboe, linux-fsdevel, LKML,
	Dave Chinner, Christoph Hellwig, Jan Kara, Chris Mason,
	Shaohua Li

  Hi Fengguang,

On Thu 12-04-12 10:20:40, Wu Fengguang wrote:
> > > > > Note that plugging for O_SYNC writes is also removed. The user may pass
> > > > > arbitrary @size arguments, which may be much larger than the preferable
> > > > > I/O size, or may cross extent/device boundaries. Let the lower layers
> > > > > handle the plugging. Otherwise the plugging code here will turn the
> > > > > low level plugging into no-ops.
> > > > 
> > > > I assume you have some numbers to back this up, right?  Care to share
> > > > those?
> > > 
> > > Yes please.
> > > 
> > > We've broken this stuff a few times recently - we should review and
> > > test carefully.
> > 
> > Yes sure. Last time I posted the patch, I did some tests and found no
> > performance changes. Now for 3.3, the tests started days ago have not
> > finished now (partly because it is stalled for quite long time due to
> > unknown reason). The now-available numbers for bs=4k dd's look fine.
> > The pending tests are for bs=1M dd's and some random fio workloads.
> > 
> 
> The changes are basically small enough to be considered noises.
> But anyway here are some interpretations:
> 
> - application visible data write performance (write_bw) is almost the same
> - it slightly reduces the real IOs that hit disk (io_wkB_s, io_rkB_s)
> - disk utilization slightly increased
> - CPU time is slightly reduced
> 
  Well, two of the throughput numbers stand out (in both directions
actually) although they seem to be more extreme configurations so maybe it
is a noise. But maybe it would deserve further check:

> $ ./compare-io bay/*/*-{3.3.0,3.3.0-plug+}
>                    3.3.0               3.3.0-plug+
> ------------------------  ------------------------
...
>                     2.60        +7.1%         2.78  bay/thresh=1M/btrfs-10dd-1-3.3.0
>                     3.72       -12.5%         3.25  bay/thresh=1M/btrfs-1dd-1-3.3.0
...
  I looked at other iostat numbers as well, but seeing quite some changes
in both directions I'd say that those iostat numbers are too noisy to draw
serious conclusion from them.

								Honza

-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [RESEND][PATCH v2] block: remove plugging at buffered write time
  2012-04-12 14:26         ` Jan Kara
@ 2012-04-13  1:40           ` Fengguang Wu
  2012-05-03  3:43             ` [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold Fengguang Wu
  2012-05-06  6:01           ` [RESEND][PATCH v2] block: remove plugging at buffered write time Fengguang Wu
  1 sibling, 1 reply; 16+ messages in thread
From: Fengguang Wu @ 2012-04-13  1:40 UTC (permalink / raw)
  To: Jan Kara
  Cc: Andrew Morton, Jeff Moyer, Jens Axboe, linux-fsdevel, LKML,
	Dave Chinner, Christoph Hellwig, Chris Mason, Shaohua Li

On Thu, Apr 12, 2012 at 04:26:34PM +0200, Jan Kara wrote:
>   Hi Fengguang,
> 
> On Thu 12-04-12 10:20:40, Wu Fengguang wrote:
> > > > > > Note that plugging for O_SYNC writes is also removed. The user may pass
> > > > > > arbitrary @size arguments, which may be much larger than the preferable
> > > > > > I/O size, or may cross extent/device boundaries. Let the lower layers
> > > > > > handle the plugging. Otherwise the plugging code here will turn the
> > > > > > low level plugging into no-ops.
> > > > > 
> > > > > I assume you have some numbers to back this up, right?  Care to share
> > > > > those?
> > > > 
> > > > Yes please.
> > > > 
> > > > We've broken this stuff a few times recently - we should review and
> > > > test carefully.
> > > 
> > > Yes sure. Last time I posted the patch, I did some tests and found no
> > > performance changes. Now for 3.3, the tests started days ago have not
> > > finished now (partly because it is stalled for quite long time due to
> > > unknown reason). The now-available numbers for bs=4k dd's look fine.
> > > The pending tests are for bs=1M dd's and some random fio workloads.
> > > 
> > 
> > The changes are basically small enough to be considered noises.
> > But anyway here are some interpretations:
> > 
> > - application visible data write performance (write_bw) is almost the same
> > - it slightly reduces the real IOs that hit disk (io_wkB_s, io_rkB_s)
> > - disk utilization slightly increased
> > - CPU time is slightly reduced
> > 
>   Well, two of the throughput numbers stand out (in both directions
> actually) although they seem to be more extreme configurations so maybe it
> is a noise. But maybe it would deserve further check:
> 
> > $ ./compare-io bay/*/*-{3.3.0,3.3.0-plug+}
> >                    3.3.0               3.3.0-plug+
> > ------------------------  ------------------------
> ...
> >                     2.60        +7.1%         2.78  bay/thresh=1M/btrfs-10dd-1-3.3.0
> >                     3.72       -12.5%         3.25  bay/thresh=1M/btrfs-1dd-1-3.3.0
> ...

Yeah. In fact I noticed btrfs performing badly on the 1MB dirty
threshold cases comparing to other filesystems. thresh=10MB cases are
fine. It looks better to explore this first. My hunch is that fixing
this issue may also change its sensitiveness to the plugging.

                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                    2.60        +7.1%         2.78  bay/thresh=1M/btrfs-10dd-1-3.3.0
                    3.72       -12.5%         3.25  bay/thresh=1M/btrfs-1dd-1-3.3.0
                   62.41        +0.3%        62.60  bay/thresh=1M/ext3-10dd-1-3.3.0
                   58.61        +0.4%        58.82  bay/thresh=1M/ext3-1dd-1-3.3.0
                   70.70        -1.1%        69.92  bay/thresh=1M/ext4-10dd-1-3.3.0
                   84.81        +0.1%        84.90  bay/thresh=1M/ext4-1dd-1-3.3.0
                   66.05        -0.1%        65.97  bay/thresh=1M/xfs-10dd-1-3.3.0
                   74.81        +0.1%        74.85  bay/thresh=1M/xfs-1dd-1-3.3.0

>   I looked at other iostat numbers as well, but seeing quite some changes
> in both directions I'd say that those iostat numbers are too noisy to draw
> serious conclusion from them.

Agreed.

Thanks,
Fengguang

^ permalink raw reply	[flat|nested] 16+ messages in thread

* [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold
  2012-04-13  1:40           ` Fengguang Wu
@ 2012-05-03  3:43             ` Fengguang Wu
  2012-05-03  3:53               ` [PATCH] writeback: initialize global_dirty_limit Fengguang Wu
                                 ` (2 more replies)
  0 siblings, 3 replies; 16+ messages in thread
From: Fengguang Wu @ 2012-05-03  3:43 UTC (permalink / raw)
  To: Chris Mason
  Cc: Jan Kara, Andrew Morton, Jeff Moyer, Jens Axboe, linux-fsdevel,
	LKML, Dave Chinner, Christoph Hellwig, Shaohua Li

This helps write performance when setting the dirty threshold to tiny numbers.

     3.4.0-rc2         3.4.0-rc2-btrfs4+
  ------------  ------------------------
         96.92        -0.4%        96.54  bay/thresh=1000M/btrfs-100dd-1-3.4.0-rc2
         98.47        +0.0%        98.50  bay/thresh=1000M/btrfs-10dd-1-3.4.0-rc2
         99.38        -0.3%        99.06  bay/thresh=1000M/btrfs-1dd-1-3.4.0-rc2
         98.04        -0.0%        98.02  bay/thresh=100M/btrfs-100dd-1-3.4.0-rc2
         98.68        +0.3%        98.98  bay/thresh=100M/btrfs-10dd-1-3.4.0-rc2
         99.34        -0.0%        99.31  bay/thresh=100M/btrfs-1dd-1-3.4.0-rc2
  ==>    88.98        +9.6%        97.53  bay/thresh=10M/btrfs-10dd-1-3.4.0-rc2
  ==>    86.99       +13.1%        98.39  bay/thresh=10M/btrfs-1dd-1-3.4.0-rc2
  ==>     2.75     +2442.4%        69.88  bay/thresh=1M/btrfs-10dd-1-3.4.0-rc2
  ==>     3.31     +2634.1%        90.54  bay/thresh=1M/btrfs-1dd-1-3.4.0-rc2

Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
---
 fs/btrfs/disk-io.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

--- linux-next.orig/fs/btrfs/disk-io.c	2012-05-02 14:04:00.989262395 +0800
+++ linux-next/fs/btrfs/disk-io.c	2012-05-02 14:04:01.773262414 +0800
@@ -930,7 +930,8 @@ static int btree_writepages(struct addre
 
 		/* this is a bit racy, but that's ok */
 		num_dirty = root->fs_info->dirty_metadata_bytes;
-		if (num_dirty < thresh)
+		if (num_dirty < min(thresh,
+				    global_dirty_limit << (PAGE_CACHE_SHIFT-2)))
 			return 0;
 	}
 	return btree_write_cache_pages(mapping, wbc);

^ permalink raw reply	[flat|nested] 16+ messages in thread

* [PATCH] writeback: initialize global_dirty_limit
  2012-05-03  3:43             ` [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold Fengguang Wu
@ 2012-05-03  3:53               ` Fengguang Wu
  2012-05-03  9:25               ` [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold Jan Kara
  2012-05-03 14:08               ` Fengguang Wu
  2 siblings, 0 replies; 16+ messages in thread
From: Fengguang Wu @ 2012-05-03  3:53 UTC (permalink / raw)
  To: Jan Kara
  Cc: Chris Mason, Andrew Morton, Jeff Moyer, Jens Axboe,
	linux-fsdevel, LKML, Dave Chinner, Christoph Hellwig, Shaohua Li

This prevents global_dirty_limit from remaining 0 (the initial value)
for long time, since it's only updated in update_dirty_limit() when
above the dirty freerun area.

It will avoid unexpected consequences when some random code use it as a
convenient approximation of the global dirty threshold.

Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
---
 mm/page-writeback.c |    1 +
 1 file changed, 1 insertion(+)

--- linux-next.orig/mm/page-writeback.c	2012-04-25 20:16:12.766859391 +0800
+++ linux-next/mm/page-writeback.c	2012-05-03 11:44:32.746272930 +0800
@@ -1568,6 +1568,7 @@ void writeback_set_ratelimit(void)
 	unsigned long background_thresh;
 	unsigned long dirty_thresh;
 	global_dirty_limits(&background_thresh, &dirty_thresh);
+	global_dirty_limit = dirty_thresh;
 	ratelimit_pages = dirty_thresh / (num_online_cpus() * 32);
 	if (ratelimit_pages < 16)
 		ratelimit_pages = 16;

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold
  2012-05-03  3:43             ` [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold Fengguang Wu
  2012-05-03  3:53               ` [PATCH] writeback: initialize global_dirty_limit Fengguang Wu
@ 2012-05-03  9:25               ` Jan Kara
  2012-05-03 10:02                 ` Fengguang Wu
                                   ` (2 more replies)
  2012-05-03 14:08               ` Fengguang Wu
  2 siblings, 3 replies; 16+ messages in thread
From: Jan Kara @ 2012-05-03  9:25 UTC (permalink / raw)
  To: Fengguang Wu
  Cc: Chris Mason, Jan Kara, Andrew Morton, Jeff Moyer, Jens Axboe,
	linux-fsdevel, LKML, Dave Chinner, Christoph Hellwig, Shaohua Li

On Thu 03-05-12 11:43:11, Wu Fengguang wrote:
> This helps write performance when setting the dirty threshold to tiny numbers.
> 
>      3.4.0-rc2         3.4.0-rc2-btrfs4+
>   ------------  ------------------------
>          96.92        -0.4%        96.54  bay/thresh=1000M/btrfs-100dd-1-3.4.0-rc2
>          98.47        +0.0%        98.50  bay/thresh=1000M/btrfs-10dd-1-3.4.0-rc2
>          99.38        -0.3%        99.06  bay/thresh=1000M/btrfs-1dd-1-3.4.0-rc2
>          98.04        -0.0%        98.02  bay/thresh=100M/btrfs-100dd-1-3.4.0-rc2
>          98.68        +0.3%        98.98  bay/thresh=100M/btrfs-10dd-1-3.4.0-rc2
>          99.34        -0.0%        99.31  bay/thresh=100M/btrfs-1dd-1-3.4.0-rc2
>   ==>    88.98        +9.6%        97.53  bay/thresh=10M/btrfs-10dd-1-3.4.0-rc2
>   ==>    86.99       +13.1%        98.39  bay/thresh=10M/btrfs-1dd-1-3.4.0-rc2
>   ==>     2.75     +2442.4%        69.88  bay/thresh=1M/btrfs-10dd-1-3.4.0-rc2
>   ==>     3.31     +2634.1%        90.54  bay/thresh=1M/btrfs-1dd-1-3.4.0-rc2
> 
> Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
> ---
>  fs/btrfs/disk-io.c |    3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> --- linux-next.orig/fs/btrfs/disk-io.c	2012-05-02 14:04:00.989262395 +0800
> +++ linux-next/fs/btrfs/disk-io.c	2012-05-02 14:04:01.773262414 +0800
> @@ -930,7 +930,8 @@ static int btree_writepages(struct addre
>  
>  		/* this is a bit racy, but that's ok */
>  		num_dirty = root->fs_info->dirty_metadata_bytes;
> -		if (num_dirty < thresh)
> +		if (num_dirty < min(thresh,
> +				    global_dirty_limit << (PAGE_CACHE_SHIFT-2)))
>  			return 0;
>  	}
>  	return btree_write_cache_pages(mapping, wbc);
  Frankly, that whole condition on WB_SYNC_NONE in btree_writepages() looks
like a hack. I think we also had problems with this condition when we tried
to change b_more_io list handling. I found rather terse commit message
explaining the code:
Btrfs: Limit btree writeback to prevent seeks

  Which I kind of understand but is it that bad? Also I think last time we
stumbled over this code we were discussing that these dirty metadata would
be simply hidden from mm which would solve the problem of flusher thread
trying to outsmart the filesystem... But I guess noone had time to
implement this for btrfs.

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold
  2012-05-03  9:25               ` [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold Jan Kara
@ 2012-05-03 10:02                 ` Fengguang Wu
  2012-05-03 12:31                 ` Chris Mason
  2012-05-03 13:30                 ` Josef Bacik
  2 siblings, 0 replies; 16+ messages in thread
From: Fengguang Wu @ 2012-05-03 10:02 UTC (permalink / raw)
  To: Jan Kara
  Cc: Chris Mason, Andrew Morton, Jeff Moyer, Jens Axboe,
	linux-fsdevel, LKML, Dave Chinner, Christoph Hellwig, Shaohua Li

On Thu, May 03, 2012 at 11:25:28AM +0200, Jan Kara wrote:
> On Thu 03-05-12 11:43:11, Wu Fengguang wrote:
> > This helps write performance when setting the dirty threshold to tiny numbers.
> > 
> >      3.4.0-rc2         3.4.0-rc2-btrfs4+
> >   ------------  ------------------------
> >          96.92        -0.4%        96.54  bay/thresh=1000M/btrfs-100dd-1-3.4.0-rc2
> >          98.47        +0.0%        98.50  bay/thresh=1000M/btrfs-10dd-1-3.4.0-rc2
> >          99.38        -0.3%        99.06  bay/thresh=1000M/btrfs-1dd-1-3.4.0-rc2
> >          98.04        -0.0%        98.02  bay/thresh=100M/btrfs-100dd-1-3.4.0-rc2
> >          98.68        +0.3%        98.98  bay/thresh=100M/btrfs-10dd-1-3.4.0-rc2
> >          99.34        -0.0%        99.31  bay/thresh=100M/btrfs-1dd-1-3.4.0-rc2
> >   ==>    88.98        +9.6%        97.53  bay/thresh=10M/btrfs-10dd-1-3.4.0-rc2
> >   ==>    86.99       +13.1%        98.39  bay/thresh=10M/btrfs-1dd-1-3.4.0-rc2
> >   ==>     2.75     +2442.4%        69.88  bay/thresh=1M/btrfs-10dd-1-3.4.0-rc2
> >   ==>     3.31     +2634.1%        90.54  bay/thresh=1M/btrfs-1dd-1-3.4.0-rc2
> > 
> > Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
> > ---
> >  fs/btrfs/disk-io.c |    3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > --- linux-next.orig/fs/btrfs/disk-io.c	2012-05-02 14:04:00.989262395 +0800
> > +++ linux-next/fs/btrfs/disk-io.c	2012-05-02 14:04:01.773262414 +0800
> > @@ -930,7 +930,8 @@ static int btree_writepages(struct addre
> >  
> >  		/* this is a bit racy, but that's ok */
> >  		num_dirty = root->fs_info->dirty_metadata_bytes;
> > -		if (num_dirty < thresh)
> > +		if (num_dirty < min(thresh,
> > +				    global_dirty_limit << (PAGE_CACHE_SHIFT-2)))
> >  			return 0;
> >  	}
> >  	return btree_write_cache_pages(mapping, wbc);
>   Frankly, that whole condition on WB_SYNC_NONE in btree_writepages() looks
> like a hack. I think we also had problems with this condition when we tried
> to change b_more_io list handling. I found rather terse commit message
> explaining the code:
> Btrfs: Limit btree writeback to prevent seeks
> 
>   Which I kind of understand but is it that bad? Also I think last time we
> stumbled over this code we were discussing that these dirty metadata would
> be simply hidden from mm which would solve the problem of flusher thread
> trying to outsmart the filesystem... But I guess noone had time to
> implement this for btrfs.

Yeah I have the same uneasy feelings. Actually my first attempt was to
remove the heuristics in btree_writepages() altogether. The result is
more or less performance degradations in the normal cases:

wfg@bee /export/writeback% ./compare bay/*/*-{3.4.0-rc2,3.4.0-rc2-btrfs+} 
               3.4.0-rc2          3.4.0-rc2-btrfs+  
------------------------  ------------------------  
                  190.81        -6.8%       177.82  bay/JBOD-2HDD-thresh=1000M/btrfs-100dd-1-3.4.0-rc2
                  195.86        -3.3%       189.31  bay/JBOD-2HDD-thresh=1000M/btrfs-10dd-1-3.4.0-rc2
                  196.68        -1.7%       193.30  bay/JBOD-2HDD-thresh=1000M/btrfs-1dd-1-3.4.0-rc2
                  194.83       -24.4%       147.27  bay/JBOD-2HDD-thresh=100M/btrfs-100dd-1-3.4.0-rc2
                  196.60        -2.5%       191.61  bay/JBOD-2HDD-thresh=100M/btrfs-10dd-1-3.4.0-rc2
                  197.09        -0.7%       195.69  bay/JBOD-2HDD-thresh=100M/btrfs-1dd-1-3.4.0-rc2
                  181.64        -8.7%       165.80  bay/RAID0-2HDD-thresh=1000M/btrfs-100dd-1-3.4.0-rc2
                  186.14        -2.8%       180.85  bay/RAID0-2HDD-thresh=1000M/btrfs-10dd-1-3.4.0-rc2
                  191.10        -1.5%       188.23  bay/RAID0-2HDD-thresh=1000M/btrfs-1dd-1-3.4.0-rc2
                  191.30       -20.7%       151.63  bay/RAID0-2HDD-thresh=100M/btrfs-100dd-1-3.4.0-rc2
                  186.03        -2.4%       181.54  bay/RAID0-2HDD-thresh=100M/btrfs-10dd-1-3.4.0-rc2
                  170.18        -2.5%       165.97  bay/RAID0-2HDD-thresh=100M/btrfs-1dd-1-3.4.0-rc2
                   96.18        -1.9%        94.32  bay/RAID1-2HDD-thresh=1000M/btrfs-100dd-1-3.4.0-rc2
                   97.71        -1.4%        96.36  bay/RAID1-2HDD-thresh=1000M/btrfs-10dd-1-3.4.0-rc2
                   97.57        -0.4%        97.23  bay/RAID1-2HDD-thresh=1000M/btrfs-1dd-1-3.4.0-rc2
                   97.68        -6.0%        91.79  bay/RAID1-2HDD-thresh=100M/btrfs-100dd-1-3.4.0-rc2
                   97.76        -0.7%        97.07  bay/RAID1-2HDD-thresh=100M/btrfs-10dd-1-3.4.0-rc2
                   97.53        -0.3%        97.19  bay/RAID1-2HDD-thresh=100M/btrfs-1dd-1-3.4.0-rc2
                   96.92        -3.0%        94.03  bay/thresh=1000M/btrfs-100dd-1-3.4.0-rc2
                   98.47        -1.4%        97.08  bay/thresh=1000M/btrfs-10dd-1-3.4.0-rc2
                   99.38        -0.7%        98.66  bay/thresh=1000M/btrfs-1dd-1-3.4.0-rc2
                   98.04        -8.2%        89.99  bay/thresh=100M/btrfs-100dd-1-3.4.0-rc2
                   98.68        -0.6%        98.09  bay/thresh=100M/btrfs-10dd-1-3.4.0-rc2
                   99.34        -0.7%        98.62  bay/thresh=100M/btrfs-1dd-1-3.4.0-rc2
                   88.98        -0.5%        88.51  bay/thresh=10M/btrfs-10dd-1-3.4.0-rc2
                   86.99       +14.5%        99.60  bay/thresh=10M/btrfs-1dd-1-3.4.0-rc2
                    2.75     +1871.2%        54.18  bay/thresh=1M/btrfs-10dd-1-3.4.0-rc2
                    3.31     +2035.0%        70.70  bay/thresh=1M/btrfs-1dd-1-3.4.0-rc2
                 3635.55        -1.2%      3592.46  TOTAL write_bw

So I end up with the conservative fix in this patch.

FYI I also experimented with "global_dirty_limit << PAGE_CACHE_SHIFT"
w/o the further "/4" in this patch, however result is not good:

               3.4.0-rc2         3.4.0-rc2-btrfs3+
------------------------  ------------------------
                   96.92        -0.3%        96.62  bay/thresh=1000M/btrfs-100dd-1-3.4.0-rc2
                   98.47        +0.1%        98.56  bay/thresh=1000M/btrfs-10dd-1-3.4.0-rc2
                   99.38        -0.2%        99.23  bay/thresh=1000M/btrfs-1dd-1-3.4.0-rc2
                   98.04        +0.1%        98.15  bay/thresh=100M/btrfs-100dd-1-3.4.0-rc2
                   98.68        +0.3%        98.96  bay/thresh=100M/btrfs-10dd-1-3.4.0-rc2
                   99.34        -0.1%        99.20  bay/thresh=100M/btrfs-1dd-1-3.4.0-rc2
                   88.98        -0.3%        88.73  bay/thresh=10M/btrfs-10dd-1-3.4.0-rc2
                   86.99        +1.4%        88.23  bay/thresh=10M/btrfs-1dd-1-3.4.0-rc2
                    2.75      +232.0%         9.13  bay/thresh=1M/btrfs-10dd-1-3.4.0-rc2
                    3.31        +1.5%         3.36  bay/thresh=1M/btrfs-1dd-1-3.4.0-rc2

So this patch is kind of based on "experiment" rather than "reasoning".
And I took the easy way of using the global dirty threshold. Ideally
it should be based upon the per-bdi dirty threshold, but anyway...

Thanks,
Fengguang

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold
  2012-05-03  9:25               ` [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold Jan Kara
  2012-05-03 10:02                 ` Fengguang Wu
@ 2012-05-03 12:31                 ` Chris Mason
  2012-05-03 13:30                 ` Josef Bacik
  2 siblings, 0 replies; 16+ messages in thread
From: Chris Mason @ 2012-05-03 12:31 UTC (permalink / raw)
  To: Jan Kara
  Cc: Fengguang Wu, Andrew Morton, Jeff Moyer, Jens Axboe,
	linux-fsdevel, LKML, Dave Chinner, Christoph Hellwig, Shaohua Li

On Thu, May 03, 2012 at 11:25:28AM +0200, Jan Kara wrote:
> On Thu 03-05-12 11:43:11, Wu Fengguang wrote:
> > This helps write performance when setting the dirty threshold to tiny numbers.
> > 
> >      3.4.0-rc2         3.4.0-rc2-btrfs4+
> >   ------------  ------------------------
> >          96.92        -0.4%        96.54  bay/thresh=1000M/btrfs-100dd-1-3.4.0-rc2
> >          98.47        +0.0%        98.50  bay/thresh=1000M/btrfs-10dd-1-3.4.0-rc2
> >          99.38        -0.3%        99.06  bay/thresh=1000M/btrfs-1dd-1-3.4.0-rc2
> >          98.04        -0.0%        98.02  bay/thresh=100M/btrfs-100dd-1-3.4.0-rc2
> >          98.68        +0.3%        98.98  bay/thresh=100M/btrfs-10dd-1-3.4.0-rc2
> >          99.34        -0.0%        99.31  bay/thresh=100M/btrfs-1dd-1-3.4.0-rc2
> >   ==>    88.98        +9.6%        97.53  bay/thresh=10M/btrfs-10dd-1-3.4.0-rc2
> >   ==>    86.99       +13.1%        98.39  bay/thresh=10M/btrfs-1dd-1-3.4.0-rc2
> >   ==>     2.75     +2442.4%        69.88  bay/thresh=1M/btrfs-10dd-1-3.4.0-rc2
> >   ==>     3.31     +2634.1%        90.54  bay/thresh=1M/btrfs-1dd-1-3.4.0-rc2
> > 
> > Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
> > ---
> >  fs/btrfs/disk-io.c |    3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > --- linux-next.orig/fs/btrfs/disk-io.c	2012-05-02 14:04:00.989262395 +0800
> > +++ linux-next/fs/btrfs/disk-io.c	2012-05-02 14:04:01.773262414 +0800
> > @@ -930,7 +930,8 @@ static int btree_writepages(struct addre
> >  
> >  		/* this is a bit racy, but that's ok */
> >  		num_dirty = root->fs_info->dirty_metadata_bytes;
> > -		if (num_dirty < thresh)
> > +		if (num_dirty < min(thresh,
> > +				    global_dirty_limit << (PAGE_CACHE_SHIFT-2)))
> >  			return 0;
> >  	}
> >  	return btree_write_cache_pages(mapping, wbc);
>   Frankly, that whole condition on WB_SYNC_NONE in btree_writepages() looks
> like a hack. I think we also had problems with this condition when we tried
> to change b_more_io list handling. I found rather terse commit message
> explaining the code:
> Btrfs: Limit btree writeback to prevent seeks

It is definitely a hack ;)  The basic point is that once we write a
metadata block, we have to cow it for any future changes.  So writing
the metadata has a pretty big impact on performance, and I'd rather
write everything else that is dirty first.  When that code was added I
was finding the metadata going to disk very soon under memory pressure.

I'm open to any ideas on this one.

-chris

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold
  2012-05-03  9:25               ` [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold Jan Kara
  2012-05-03 10:02                 ` Fengguang Wu
  2012-05-03 12:31                 ` Chris Mason
@ 2012-05-03 13:30                 ` Josef Bacik
  2 siblings, 0 replies; 16+ messages in thread
From: Josef Bacik @ 2012-05-03 13:30 UTC (permalink / raw)
  To: Jan Kara
  Cc: Fengguang Wu, Chris Mason, Andrew Morton, Jeff Moyer, Jens Axboe,
	linux-fsdevel, LKML, Dave Chinner, Christoph Hellwig, Shaohua Li

On Thu, May 03, 2012 at 11:25:28AM +0200, Jan Kara wrote:
> On Thu 03-05-12 11:43:11, Wu Fengguang wrote:
> > This helps write performance when setting the dirty threshold to tiny numbers.
> > 
> >      3.4.0-rc2         3.4.0-rc2-btrfs4+
> >   ------------  ------------------------
> >          96.92        -0.4%        96.54  bay/thresh=1000M/btrfs-100dd-1-3.4.0-rc2
> >          98.47        +0.0%        98.50  bay/thresh=1000M/btrfs-10dd-1-3.4.0-rc2
> >          99.38        -0.3%        99.06  bay/thresh=1000M/btrfs-1dd-1-3.4.0-rc2
> >          98.04        -0.0%        98.02  bay/thresh=100M/btrfs-100dd-1-3.4.0-rc2
> >          98.68        +0.3%        98.98  bay/thresh=100M/btrfs-10dd-1-3.4.0-rc2
> >          99.34        -0.0%        99.31  bay/thresh=100M/btrfs-1dd-1-3.4.0-rc2
> >   ==>    88.98        +9.6%        97.53  bay/thresh=10M/btrfs-10dd-1-3.4.0-rc2
> >   ==>    86.99       +13.1%        98.39  bay/thresh=10M/btrfs-1dd-1-3.4.0-rc2
> >   ==>     2.75     +2442.4%        69.88  bay/thresh=1M/btrfs-10dd-1-3.4.0-rc2
> >   ==>     3.31     +2634.1%        90.54  bay/thresh=1M/btrfs-1dd-1-3.4.0-rc2
> > 
> > Signed-off-by: Fengguang Wu <fengguang.wu@intel.com>
> > ---
> >  fs/btrfs/disk-io.c |    3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> > 
> > --- linux-next.orig/fs/btrfs/disk-io.c	2012-05-02 14:04:00.989262395 +0800
> > +++ linux-next/fs/btrfs/disk-io.c	2012-05-02 14:04:01.773262414 +0800
> > @@ -930,7 +930,8 @@ static int btree_writepages(struct addre
> >  
> >  		/* this is a bit racy, but that's ok */
> >  		num_dirty = root->fs_info->dirty_metadata_bytes;
> > -		if (num_dirty < thresh)
> > +		if (num_dirty < min(thresh,
> > +				    global_dirty_limit << (PAGE_CACHE_SHIFT-2)))
> >  			return 0;
> >  	}
> >  	return btree_write_cache_pages(mapping, wbc);
>   Frankly, that whole condition on WB_SYNC_NONE in btree_writepages() looks
> like a hack. I think we also had problems with this condition when we tried
> to change b_more_io list handling. I found rather terse commit message
> explaining the code:
> Btrfs: Limit btree writeback to prevent seeks
> 
>   Which I kind of understand but is it that bad? Also I think last time we
> stumbled over this code we were discussing that these dirty metadata would
> be simply hidden from mm which would solve the problem of flusher thread
> trying to outsmart the filesystem... But I guess noone had time to
> implement this for btrfs.
> 

Actually I did but I ran into an OOM problem.  See we can have as much dirty
metadata as we have ram, and having no insight into what the global dirty and
writeback limits are for the system means btrfs was using wayyyyy more memory
for it's dirty and writeback metadata pages than would have normally been
allowed.  In order to avoid OOM I had to re-implement a sort of
balance_dirty_pages for btrfs, and again having no access to the global dirty
limits and such at the time (AFAIK, I could just be an idiot) it was very hacky
and prone to breaking.  The shrinker doesn't get called enough to handle this
sort of thing.  Dave mentioned at LSF that XFS will actually do the synchronous
writeout from the shrinker which will auto-throttle everything so I was going to
try that but I haven't gotten around to it.  Thanks,

Josef

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold
  2012-05-03  3:43             ` [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold Fengguang Wu
  2012-05-03  3:53               ` [PATCH] writeback: initialize global_dirty_limit Fengguang Wu
  2012-05-03  9:25               ` [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold Jan Kara
@ 2012-05-03 14:08               ` Fengguang Wu
  2 siblings, 0 replies; 16+ messages in thread
From: Fengguang Wu @ 2012-05-03 14:08 UTC (permalink / raw)
  To: Chris Mason
  Cc: Jan Kara, Andrew Morton, Jeff Moyer, Jens Axboe, linux-fsdevel,
	LKML, Dave Chinner, Christoph Hellwig, Shaohua Li

On Thu, May 03, 2012 at 11:43:11AM +0800, Fengguang Wu wrote:
> This helps write performance when setting the dirty threshold to tiny numbers.
> 
>      3.4.0-rc2         3.4.0-rc2-btrfs4+
>   ------------  ------------------------
>          96.92        -0.4%        96.54  bay/thresh=1000M/btrfs-100dd-1-3.4.0-rc2
>          98.47        +0.0%        98.50  bay/thresh=1000M/btrfs-10dd-1-3.4.0-rc2
>          99.38        -0.3%        99.06  bay/thresh=1000M/btrfs-1dd-1-3.4.0-rc2
>          98.04        -0.0%        98.02  bay/thresh=100M/btrfs-100dd-1-3.4.0-rc2
>          98.68        +0.3%        98.98  bay/thresh=100M/btrfs-10dd-1-3.4.0-rc2
>          99.34        -0.0%        99.31  bay/thresh=100M/btrfs-1dd-1-3.4.0-rc2
>   ==>    88.98        +9.6%        97.53  bay/thresh=10M/btrfs-10dd-1-3.4.0-rc2
>   ==>    86.99       +13.1%        98.39  bay/thresh=10M/btrfs-1dd-1-3.4.0-rc2
>   ==>     2.75     +2442.4%        69.88  bay/thresh=1M/btrfs-10dd-1-3.4.0-rc2
>   ==>     3.31     +2634.1%        90.54  bay/thresh=1M/btrfs-1dd-1-3.4.0-rc2

Well, further tests show that it behaves very unstable:

       3.4.0-rc2-btrfs4+         3.4.0-rc2-btrfs5+  
------------------------  ------------------------  
                   69.88       +16.4%        81.31  bay/thresh=1M/btrfs-10dd-1-3.4.0-rc2-btrfs4+
                   71.09        +1.4%        72.05  bay/thresh=1M/btrfs-10dd-2-3.4.0-rc2-btrfs4+
                   72.60        -1.7%        71.38  bay/thresh=1M/btrfs-10dd-3-3.4.0-rc2-btrfs4+
                   90.54        -0.9%        89.74  bay/thresh=1M/btrfs-1dd-1-3.4.0-rc2-btrfs4+
                   89.17       -90.2%     ==> 8.71  bay/thresh=1M/btrfs-1dd-2-3.4.0-rc2-btrfs4+
               ==> 14.96      +495.3%        89.06  bay/thresh=1M/btrfs-1dd-3-3.4.0-rc2-btrfs4+
                  408.23        +1.0%       412.26  TOTAL write_bw

Where the -btrfs5 kernel has one more patch to remove the write plug.

Thanks,
Fengguang

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [RESEND][PATCH v2] block: remove plugging at buffered write time
  2012-04-12 14:26         ` Jan Kara
  2012-04-13  1:40           ` Fengguang Wu
@ 2012-05-06  6:01           ` Fengguang Wu
  1 sibling, 0 replies; 16+ messages in thread
From: Fengguang Wu @ 2012-05-06  6:01 UTC (permalink / raw)
  To: Jan Kara
  Cc: Andrew Morton, Jeff Moyer, Jens Axboe, linux-fsdevel, LKML,
	Dave Chinner, Christoph Hellwig, Chris Mason, Shaohua Li, Yan,
	Zheng Z

On Thu, Apr 12, 2012 at 04:26:34PM +0200, Jan Kara wrote:
>   Hi Fengguang,
> 
> On Thu 12-04-12 10:20:40, Wu Fengguang wrote:
> > > > > > Note that plugging for O_SYNC writes is also removed. The user may pass
> > > > > > arbitrary @size arguments, which may be much larger than the preferable
> > > > > > I/O size, or may cross extent/device boundaries. Let the lower layers
> > > > > > handle the plugging. Otherwise the plugging code here will turn the
> > > > > > low level plugging into no-ops.
> > > > > 
> > > > > I assume you have some numbers to back this up, right?  Care to share
> > > > > those?
> > > > 
> > > > Yes please.
> > > > 
> > > > We've broken this stuff a few times recently - we should review and
> > > > test carefully.
> > > 
> > > Yes sure. Last time I posted the patch, I did some tests and found no
> > > performance changes. Now for 3.3, the tests started days ago have not
> > > finished now (partly because it is stalled for quite long time due to
> > > unknown reason). The now-available numbers for bs=4k dd's look fine.
> > > The pending tests are for bs=1M dd's and some random fio workloads.
> > > 
> > 
> > The changes are basically small enough to be considered noises.
> > But anyway here are some interpretations:
> > 
> > - application visible data write performance (write_bw) is almost the same
> > - it slightly reduces the real IOs that hit disk (io_wkB_s, io_rkB_s)
> > - disk utilization slightly increased
> > - CPU time is slightly reduced
> > 
>   Well, two of the throughput numbers stand out (in both directions
> actually) although they seem to be more extreme configurations so maybe it
> is a noise. But maybe it would deserve further check:
> 
> > $ ./compare-io bay/*/*-{3.3.0,3.3.0-plug+}
> >                    3.3.0               3.3.0-plug+
> > ------------------------  ------------------------
> ...
> >                     2.60        +7.1%         2.78  bay/thresh=1M/btrfs-10dd-1-3.3.0
> >                     3.72       -12.5%         3.25  bay/thresh=1M/btrfs-1dd-1-3.3.0

For that btrfs performance bug, Yan Zheng helped me track down the
root cause to be the hack in btree_writepages(), as we are discussing
solutions to this in the other emails.

>   I looked at other iostat numbers as well, but seeing quite some changes
> in both directions I'd say that those iostat numbers are too noisy to draw
> serious conclusion from them.

So let's proceed with the plugging change? I'll repost the split
plugging patches.

Thanks,
Fengguang

^ permalink raw reply	[flat|nested] 16+ messages in thread

* Re: [RESEND][PATCH v2] block: remove plugging at buffered write time
  2012-04-12  1:32     ` Fengguang Wu
  2012-04-12  2:20       ` Fengguang Wu
@ 2012-05-06  9:58       ` Fengguang Wu
  1 sibling, 0 replies; 16+ messages in thread
From: Fengguang Wu @ 2012-05-06  9:58 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Jeff Moyer, Jens Axboe, linux-fsdevel, LKML, Dave Chinner,
	Christoph Hellwig, Jan Kara, Chris Mason, Shaohua Li

On Thu, Apr 12, 2012 at 09:32:24AM +0800, Fengguang Wu wrote:
> On Wed, Apr 11, 2012 at 04:13:44PM -0700, Andrew Morton wrote:
> > On Mon, 09 Apr 2012 10:34:44 -0400
> > Jeff Moyer <jmoyer@redhat.com> wrote:
> > 
> > > Wu Fengguang <fengguang.wu@intel.com> writes:
> > > 
> > > > Buffered write(2) is not directly tied to IO, so it's not suitable to
> > > > handle plug in generic_file_aio_write().
> > > [snip, moved]
> > > > Note that plugging for O_SYNC writes is also removed. The user may pass
> > > > arbitrary @size arguments, which may be much larger than the preferable
> > > > I/O size, or may cross extent/device boundaries. Let the lower layers
> > > > handle the plugging. Otherwise the plugging code here will turn the
> > > > low level plugging into no-ops.
> > > 
> > > I assume you have some numbers to back this up, right?  Care to share
> > > those?
> > 
> > Yes please.
> > 
> > We've broken this stuff a few times recently - we should review and
> > test carefully.
> 
> Yes sure. Last time I posted the patch, I did some tests and found no
> performance changes. Now for 3.3, the tests started days ago have not
> finished now (partly because it is stalled for quite long time due to
> unknown reason). The now-available numbers for bs=4k dd's look fine.

> The pending tests are for bs=1M dd's and some random fio workloads.

The bs=1M and fio performance numbers are also unchanged (within the error range):

wfg@bee /export/writeback% ./compare -g bs=1M bay/*/*-{3.3.0,3.3.0-plug+}
                   3.3.0               3.3.0-plug+
------------------------  ------------------------
                  196.56        +0.4%       197.32  bay/JBOD-2HDD-thresh=1000M/xfs-1dd:bs=1M-1-3.3.0
                   99.47        -0.2%        99.31  bay/thresh=1000M/xfs-1dd:bs=1M-1-3.3.0
                   99.19        +0.0%        99.20  bay/thresh=1000M:990M/xfs-1dd:bs=1M-1-3.3.0
                   99.03        +0.6%        99.64  bay/thresh=1000M:999M/xfs-1dd:bs=1M-1-3.3.0
                   99.37        -0.1%        99.22  bay/thresh=100M/xfs-1dd:bs=1M-1-3.3.0
                   92.62        -0.6%        92.07  bay/thresh=10M/xfs-1dd:bs=1M-1-3.3.0
                    7.38        +4.2%         7.69  bay/thresh=1M/btrfs-10dd:bs=1M-1-3.3.0
                    7.52        -0.9%         7.45  bay/thresh=1M/btrfs-10dd:bs=1M-2-3.3.0
                    8.41        +0.3%         8.44  bay/thresh=1M/btrfs-1dd:bs=1M-1-3.3.0
                    8.63        -2.9%         8.38  bay/thresh=1M/btrfs-1dd:bs=1M-2-3.3.0
                   68.98        -1.0%        68.28  bay/thresh=1M/xfs-1dd:bs=1M-1-3.3.0
                  787.17        -0.0%       786.99  TOTAL write_bw

The random writes see more fluctuations:

wfg@bee /export/writeback% ./compare fat/*/*-{3.3.0,3.3.0-plug+}      
                   3.3.0               3.3.0-plug+  
------------------------  ------------------------  
                   15.38        -8.1%        14.13  fat/fio/btrfs-fio_fat_mmap_randwrite_4k-1-3.3.0
                   15.63       -15.9%        13.14  fat/fio/btrfs-fio_fat_mmap_randwrite_4k-2-3.3.0
                   53.96        -1.3%        53.26  fat/fio/btrfs-fio_fat_mmap_randwrite_64k-1-3.3.0
                   53.36        +0.4%        53.56  fat/fio/btrfs-fio_fat_mmap_randwrite_64k-2-3.3.0
                   56.40        +1.4%        57.19  fat/fio/btrfs-fio_fat_rates-1-3.3.0
                   56.15        -0.5%        55.88  fat/fio/btrfs-fio_fat_rates-2-3.3.0
                    9.55        -2.5%         9.31  fat/fio/ext3-fio_fat_mmap_randwrite_4k-1-3.3.0
                    9.28        +2.9%         9.55  fat/fio/ext3-fio_fat_mmap_randwrite_4k-2-3.3.0
                   10.55        +2.0%        10.77  fat/fio/ext3-fio_fat_mmap_randwrite_64k-1-3.3.0
                   10.23        +5.1%        10.76  fat/fio/ext3-fio_fat_mmap_randwrite_64k-2-3.3.0
                   40.24        -0.5%        40.05  fat/fio/ext3-fio_fat_rates-1-3.3.0
                   40.11        +0.2%        40.19  fat/fio/ext3-fio_fat_rates-2-3.3.0
                    2.49        -1.3%         2.46  fat/fio/ext4-fio_fat_mmap_randwrite_4k-1-3.3.0
                    2.33        -4.6%         2.23  fat/fio/ext4-fio_fat_mmap_randwrite_4k-2-3.3.0
                    8.61        -4.5%         8.22  fat/fio/ext4-fio_fat_mmap_randwrite_64k-1-3.3.0
                    8.32        -2.5%         8.11  fat/fio/ext4-fio_fat_mmap_randwrite_64k-2-3.3.0
                   49.43        +5.8%        52.30  fat/fio/ext4-fio_fat_rates-1-3.3.0
                   50.26        +5.9%        53.25  fat/fio/ext4-fio_fat_rates-2-3.3.0
                    2.29        -1.4%         2.25  fat/fio/ext4:wb-fio_fat_mmap_randwrite_4k-1-3.3.0
                    2.76        +0.0%         2.76  fat/fio/ext4:wb-fio_fat_mmap_randwrite_4k-2-3.3.0
                    8.36        +3.5%         8.65  fat/fio/ext4:wb-fio_fat_mmap_randwrite_64k-1-3.3.0
                    8.21        -0.7%         8.16  fat/fio/ext4:wb-fio_fat_mmap_randwrite_64k-2-3.3.0
                   51.32        +2.5%        52.62  fat/fio/ext4:wb-fio_fat_rates-1-3.3.0
                   53.53        -0.4%        53.31  fat/fio/ext4:wb-fio_fat_rates-2-3.3.0
                    9.17        +4.6%         9.59  fat/fio/xfs-fio_fat_mmap_randwrite_4k-1-3.3.0
                    9.36        -0.9%         9.28  fat/fio/xfs-fio_fat_mmap_randwrite_4k-2-3.3.0
                   44.24        -0.4%        44.06  fat/fio/xfs-fio_fat_mmap_randwrite_64k-1-3.3.0
                   43.03        -0.8%        42.70  fat/fio/xfs-fio_fat_mmap_randwrite_64k-2-3.3.0
                   55.74        -7.4%        51.61  fat/fio/xfs-fio_fat_rates-1-3.3.0
                   51.64        -1.1%        51.05  fat/fio/xfs-fio_fat_rates-2-3.3.0
                  831.94        -0.2%       830.38  TOTAL write_bw

Thanks,
Fengguang

^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2012-05-06  9:58 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-04-08  1:06 [RESEND][PATCH v2] block: remove plugging at buffered write time Wu Fengguang
2012-04-09 14:34 ` Jeff Moyer
2012-04-11 23:13   ` Andrew Morton
2012-04-12  1:32     ` Fengguang Wu
2012-04-12  2:20       ` Fengguang Wu
2012-04-12 14:26         ` Jan Kara
2012-04-13  1:40           ` Fengguang Wu
2012-05-03  3:43             ` [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold Fengguang Wu
2012-05-03  3:53               ` [PATCH] writeback: initialize global_dirty_limit Fengguang Wu
2012-05-03  9:25               ` [PATCH] btrfs: lower metadata writeback threshold on low dirty threshold Jan Kara
2012-05-03 10:02                 ` Fengguang Wu
2012-05-03 12:31                 ` Chris Mason
2012-05-03 13:30                 ` Josef Bacik
2012-05-03 14:08               ` Fengguang Wu
2012-05-06  6:01           ` [RESEND][PATCH v2] block: remove plugging at buffered write time Fengguang Wu
2012-05-06  9:58       ` Fengguang Wu

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.