linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* Re: [BENCHMARK] I/O regression after 2.6.0-test5
@ 2003-10-29  0:38 rwhron
  0 siblings, 0 replies; 25+ messages in thread
From: rwhron @ 2003-10-29  0:38 UTC (permalink / raw)
  To: piggin; +Cc: linux-kernel

On 28 Oct 2003 10:24:36 +1100,  Nick Piggin wrote:
> There is an #if 0 part in the patch. Could you try setting it to #if 1
> please.

Changing the #if 0 to #if 1 appears to affect AIM7 fserver the most.

Patch tested is very similar to:
http://marc.theaimsgroup.com/?l=linux-kernel&m=106729967301172&w=2

2.6.0-test9-as	as-fix patch
2.6.0-test9-as1	as-fix patch with #if 1

Either way, that version of as-fix patch looks good for both
AIM7 and tiobench.

AIM7 dbase workload
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test9-as           32	604.1		314.6	133.7
2.6.0-test9-as1          32	599.6		317.0	142.2
2.6.0-test5              32	590.2		322.0	131.9
2.6.0-test9              32	316.2		601.2	128.5

2.6.0-test9-as           64	783.3		485.3	256.3
2.6.0-test9-as1          64	776.8		489.4	257.6
2.6.0-test5              64	774.3		490.9	256.6
2.6.0-test9              64	460.9		824.8	246.2

2.6.0-test9-as           96	868.5		656.6	386.4
2.6.0-test9-as1          96	864.1		660.0	391.0
2.6.0-test5              96	859.0		663.9	389.5
2.6.0-test9              96	507.5		1123.5	368.1

2.6.0-test9-as           128	914.1		831.8	510.7
2.6.0-test9-as1          128	910.7		834.8	521.1
2.6.0-test5              128	910.3		835.2	528.6
2.6.0-test9              128	536.6		1416.9	487.0

2.6.0-test9-as           160	952.4		998.0	635.8
2.6.0-test9-as1          160	949.3		1001.2	652.2
2.6.0-test5              160	741.5		1281.7	644.8
2.6.0-test9              160	522.4		1819.2	619.9

2.6.0-test9-as1          192	981.4		1162.1	779.1
2.6.0-test9-as           192	976.6		1167.8	774.2
2.6.0-test5              192	840.0		1357.8	776.0
2.6.0-test9              192	513.5		2220.8	739.3

2.6.0-test9-as           224	1000.6		1329.7	917.0
2.6.0-test9-as1          224	994.9		1337.3	904.8
2.6.0-test5              224	986.9		1348.2	911.8
2.6.0-test9              224	503.9		2640.3	853.4

2.6.0-test9-as           256	1010.6		1504.7	1036.3
2.6.0-test9-as1          256	1009.0		1507.1	1030.0
2.6.0-test5              256	999.5		1521.3	1029.3
2.6.0-test9              256	530.8		2864.8	975.5

At low number of tasks, #if 0 does more work.   As the
number of tasks goes up, having #if 1 does more work.


AIM7 fserver workload
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test9-as           4	114.6		211.6	38.2
2.6.0-test5              4	79.7		304.0	38.3
2.6.0-test9-as1          4	79.3		305.8	38.4
2.6.0-test9              4	43.0		564.4	43.6

2.6.0-test9-as           8	175.0		277.0	65.9
2.6.0-test5              8	129.8		373.5	66.3
2.6.0-test9-as1          8	128.8		376.5	65.0
2.6.0-test9              8	61.4		789.1	68.5

2.6.0-test9-as           12	208.4		349.0	93.6
2.6.0-test5              12	164.1		443.0	93.1
2.6.0-test9-as1          12	158.4		459.1	91.6
2.6.0-test9              12	70.4		1033.1	97.6

2.6.0-test9-as           16	190.4		509.3	111.8
2.6.0-test9-as1          16	184.7		525.1	114.2
2.6.0-test5              16	173.4		559.0	116.5
2.6.0-test9              16	100.7		962.8	135.5

2.6.0-test9-as1          20	213.3		568.2	143.3
2.6.0-test9-as           20	190.3		636.8	142.8
2.6.0-test5              20	180.5		671.5	143.0
2.6.0-test9              20	113.0		1072.6	160.0

2.6.0-test9-as1          24	219.2		663.5	169.8
2.6.0-test9-as           24	193.6		751.2	167.3
2.6.0-test5              24	189.1		769.0	176.0
2.6.0-test9              24	116.8		1245.6	163.3

2.6.0-test9-as1          28	223.7		758.6	195.5
2.6.0-test9-as           28	205.7		825.0	200.6
2.6.0-test5              28	204.6		829.2	197.5
2.6.0-test9              28	132.7		1278.5	190.2

2.6.0-test9-as1          32	228.1		850.0	224.8
2.6.0-test5              32	213.7		907.4	228.9
2.6.0-test9-as           32	209.3		926.6	225.9
2.6.0-test9              32	130.4		1486.6	209.1

The regression seen after test5 is gone with the most
recent version of as-fix.

AIM7 shared workload
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test9-as1          64	1952.3		190.8	169.6
2.6.0-test9-as           64	1936.9		192.3	170.2
2.6.0-test5              64	1888.8		197.2	168.3
2.6.0-test9              64	1050.5		354.6	165.0

2.6.0-test9-as           128	2154.9		345.7	337.2
2.6.0-test9-as1          128	2146.3		347.1	338.1
2.6.0-test5              128	2094.9		355.6	338.1
2.6.0-test9              128	1316.6		565.8	330.6

2.6.0-test9-as           192	2267.5		492.8	509.8
2.6.0-test9-as1          192	2260.9		494.2	508.6
2.6.0-test5              192	2256.0		495.3	507.8
2.6.0-test9              192	1399.0		798.7	502.9

2.6.0-test9-as1          256	2417.6		616.3	678.9
2.6.0-test5              256	2389.0		623.7	683.8
2.6.0-test9-as           256	2389.0		623.7	679.8
2.6.0-test9              256	1472.1		1012.1	672.8

2.6.0-test5              320	2457.9		757.7	856.6
2.6.0-test9-as           320	2440.5		763.1	851.0
2.6.0-test9-as1          320	2420.7		769.4	855.9
2.6.0-test9              320	1590.3		1171.1	841.0

2.6.0-test9-as1          384	2498.2		894.6	1030.3
2.6.0-test9-as           384	2487.2		898.6	1038.3
2.6.0-test5              384	2480.4		901.0	1036.5
2.6.0-test9              384	1626.5		1374.1	1022.9

2.6.0-test5              448	2529.8		1030.7	1209.2
2.6.0-test9-as           448	2514.9		1036.8	1213.3
2.6.0-test9-as1          448	2499.6		1043.1	1206.1
2.6.0-test9              448	1636.7		1593.0	1199.5

2.6.0-test9-as           512	2598.7		1146.7	1390.1
2.6.0-test9-as1          512	2581.0		1154.5	1382.3
2.6.0-test5              512	2566.2		1161.2	1386.2
2.6.0-test9              512	1650.9		1805.0	1377.2


Tiobench below is not too exciting.  The main thing is the regression
seen in test[678] is gone.


tiobench-0.3.3
Unit information
================
File size = 8192 megabytes
Blk Size  = 4096 bytes
Rate      = megabytes per second
CPU%      = percentage of CPU used during the test
Latency   = milliseconds
Lat%      = percent of requests that took longer than X seconds
CPU Eff   = Rate divided by CPU% - throughput per cpu load

Sequential Reads ext2
                       Num                    Avg       Maximum     Lat%
Kernel                 Thr   Rate  (CPU%)   Latency     Latency      >2s
---------------------- ---  --------------------------------------------
2.6.0-test9-as           1   28.83 13.66%     0.404      753.63  0.00000
2.6.0-test5              1   28.55 13.89%     0.408      754.59  0.00000
2.6.0-test9-as1          1   28.47 13.76%     0.409      885.44  0.00000

2.6.0-test5              8   34.01 17.09%     2.620      834.85  0.00000
2.6.0-test9-as1          8   33.90 16.82%     2.650      943.85  0.00000
2.6.0-test9-as           8   30.59 15.02%     2.963      843.18  0.00000

2.6.0-test9-as1         16   31.26 15.30%     5.703     1078.85  0.00000
2.6.0-test5             16   30.13 15.04%     5.980     1337.69  0.00000
2.6.0-test9-as          16   26.53 13.00%     6.758     1036.38  0.00000

2.6.0-test9-as1         32   29.63 14.81%    12.057     1459.39  0.00000
2.6.0-test5             32   29.46 14.62%    11.954     1560.02  0.00000
2.6.0-test9-as          32   24.84 12.43%    14.389     1842.53  0.00000

2.6.0-test5             64   25.24 13.57%    26.901     1743.05  0.00000
2.6.0-test9-as1         64   22.77 12.34%    29.750     1783.29  0.00000
2.6.0-test9-as          64   21.53 11.64%    31.063     1901.78  0.00000

2.6.0-test5            128   10.41  9.76%   129.662     1820.78  0.00000
2.6.0-test9-as1        128   10.27  9.41%   130.097     2172.39  0.00000
2.6.0-test9-as         128   10.05  9.13%   133.312     1753.29  0.00000

2.6.0-test5            256    9.13 13.30%   305.097     1446.43  0.00000
2.6.0-test9-as         256    8.62 12.50%   322.473     1698.87  0.00000
2.6.0-test9-as1        256    8.57 12.63%   325.050     1653.00  0.00000

Random Reads ext2
                       Num                    Avg       Maximum     Lat%
Kernel                 Thr   Rate  (CPU%)   Latency     Latency      >2s
---------------------- ---  --------------------------------------------
2.6.0-test9-as1          1    0.98  0.99%    12.001      109.76  0.00000
2.6.0-test5              1    0.95  0.94%    12.373      114.89  0.00000
2.6.0-test9-as           1    0.81  0.80%    14.484      117.63  0.00000

2.6.0-test5              8    4.11  4.58%    19.433      137.74  0.00000
2.6.0-test9-as1          8    4.03  4.39%    20.775      177.47  0.00000
2.6.0-test9-as           8    3.61  3.79%    23.024      161.12  0.00000

2.6.0-test9-as1         16    4.58  3.69%    36.975      396.44  0.00000
2.6.0-test5             16    4.05  3.90%    42.424      236.49  0.00000
2.6.0-test9-as          16    3.33  3.02%    51.015      293.01  0.00000

2.6.0-test5             32    4.41  3.58%    70.910      516.28  0.00000
2.6.0-test9-as1         32    4.21  3.95%    80.533      451.20  0.00000
2.6.0-test9-as          32    3.26  3.13%   102.403      697.51  0.00000

2.6.0-test5             64    4.47  4.00%   137.277     1077.53  0.00000
2.6.0-test9-as          64    4.40  3.88%   134.358     1229.07  0.00000
2.6.0-test9-as1         64    4.40  3.74%   138.895     1233.53  0.00000

2.6.0-test5            128    4.86  4.83%   243.704     2287.09  0.00000
2.6.0-test9-as1        128    4.68  4.47%   243.701     2172.91  0.00000
2.6.0-test9-as         128    4.51  4.51%   249.319     2547.23  0.00000

2.6.0-test5            256    5.51  7.00%   427.503     2511.79  0.00000
2.6.0-test9-as         256    5.23  6.79%   450.963     3573.61  0.00000
2.6.0-test9-as1        256    5.01  6.61%   467.361     3305.41  0.00000

Sequential Writes ext2
                       Num                    Avg       Maximum     Lat%
Kernel                 Thr   Rate  (CPU%)   Latency     Latency      >2s
---------------------- ---  --------------------------------------------
2.6.0-test5              1   56.24 41.13%     0.170     3207.90  0.00000
2.6.0-test9-as1          1   55.71 39.50%     0.172     2225.67  0.00000
2.6.0-test9-as           1   55.49 39.25%     0.173     1806.75  0.00000

2.6.0-test5              8   31.86 31.70%     2.356    15355.31  0.05655
2.6.0-test9-as1          8   31.66 30.95%     2.364    14477.90  0.06008
2.6.0-test9-as           8   31.10 30.23%     2.453    14880.66  0.06271

2.6.0-test9-as1         16   32.63 31.80%     4.445    28724.28  0.08045
2.6.0-test9-as          16   32.10 31.67%     4.459    41263.17  0.07834
2.6.0-test5             16   32.07 32.22%     4.371    29214.57  0.07829

2.6.0-test9-as1         32   32.79 32.88%     8.351    70993.57  0.10638
2.6.0-test5             32   30.73 34.74%     8.843    59335.31  0.10657
2.6.0-test9-as          32   30.03 33.06%     8.852    61766.24  0.10576

2.6.0-test5             64   30.27 36.46%    16.798   119951.55  0.14372
2.6.0-test9-as1         64   28.70 34.79%    17.693   114731.30  0.15449
2.6.0-test9-as          64   27.10 32.73%    18.207   120362.26  0.15745

2.6.0-test5            128   29.98 38.86%    32.259   230569.79  0.20509
2.6.0-test9-as1        128   29.84 37.86%    31.890   224444.62  0.20604
2.6.0-test9-as         128   29.25 37.18%    32.428   210201.96  0.21028

2.6.0-test5            256   33.49 48.84%    53.571   391954.31  0.24166
2.6.0-test9-as         256   32.18 44.97%    56.356   453839.45  0.24067
2.6.0-test9-as1        256   31.23 43.69%    57.305   431916.55  0.23952

Random Writes ext2
                       Num                    Avg       Maximum     Lat%
Kernel                 Thr   Rate  (CPU%)   Latency     Latency      >2s
---------------------- ---  --------------------------------------------
2.6.0-test9-as           1    3.08  2.79%     0.582       25.12  0.00000
2.6.0-test5              1    2.86  2.88%     1.117       50.58  0.00000
2.6.0-test9-as1          1    2.63  2.52%     1.481      380.77  0.00000

2.6.0-test5              8    3.85  5.10%     4.885      861.30  0.00000
2.6.0-test9-as1          8    3.69  3.69%     7.495     1084.38  0.00000
2.6.0-test9-as           8    3.34  3.49%    11.409     1369.42  0.00000

2.6.0-test5             16    3.70  3.93%    17.617     1419.36  0.00000
2.6.0-test9-as1         16    3.59  3.90%    18.985     1561.92  0.00000
2.6.0-test9-as          16    3.43  3.33%    16.893     1458.50  0.00000

2.6.0-test5             32    3.72  5.11%    14.666     1191.07  0.00000
2.6.0-test9-as1         32    3.67  4.34%    13.096     1346.98  0.00000
2.6.0-test9-as          32    3.40  4.22%    16.468     1422.35  0.00000

2.6.0-test5             64    3.69  5.68%    14.445     1116.77  0.00000
2.6.0-test9-as1         64    3.53  5.15%    18.633     1450.11  0.00000
2.6.0-test9-as          64    3.39  4.76%    19.373     1659.40  0.00000

2.6.0-test5            128    3.72  6.48%    14.037     1354.59  0.00000
2.6.0-test9-as         128    3.44  5.97%    12.805     1776.43  0.00000
2.6.0-test9-as1        128    3.41  5.81%    15.659     1974.75  0.00000

2.6.0-test5            256    3.57  6.59%    13.418     1912.18  0.00000
2.6.0-test9-as1        256    3.48  6.76%     5.699     1468.80  0.00000
2.6.0-test9-as         256    3.43  6.53%     8.807     1795.06  0.00000

-- 
Randy Hron
http://home.earthlink.net/~rwhron/kernel/bigbox.html


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-26 10:38 rwhron
@ 2003-10-27 23:49 ` Dave Olien
  0 siblings, 0 replies; 25+ messages in thread
From: Dave Olien @ 2003-10-27 23:49 UTC (permalink / raw)
  To: rwhron, piggin; +Cc: linux-kernel, akpm

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]

On Sun, Oct 26, 2003 at 05:38:29AM -0500, rwhron@earthlink.net wrote:
> 2.6.0-test8-mm1 doesn't have the AIM7 database regression
> that 2.6.0-test8 has.  The AIM7 fileserver and shared 
> workloads regression between test5-mm2 and test8-mm1 is
> about 18%.  Mainline regression was about 38% between test5
> and test8 on AIM7 fileserver and shared workloads.
> 

Yes, I've noticed this as well.  This has been the pattern since
test5.  Every mainline release (test6, test7, test8, AND test9)
has a large regression in as-iosched vs deadline scheduler performance.
The regression is considerably smaller when the mm[1-4] patches are
applied.  See the attachment for a collection of data points, collected
running REAIM, an updated version of AIM7.

In the mm[1-4] broken-out patches lists, there is an O_SYNC-SPEEDUP-2.patch
that has been in all of the mm[1-4] patches since test5.  The comments at
the head of that patch says this patch reduces the blocking of writers on i_sem.

This patch probably allows more writer concurrency.  This benefits
only O_SYNC writes.   Reaim does O_SYNC writes.  My guess this patch
is what reduces the as-iosched regression for mm[1-4] patched kernels.

I imagine the O_SYNC-SPEEDUP is unlikely to get into any more TEST kernels.
But maybe it gives a hint is to what's going on with as-iosched.
Letting in more concurrent writers seems to reduce as-iosched's peformance
regression.


[-- Attachment #2: reaim_as_vs_deadline --]
[-- Type: text/plain, Size: 1737 bytes --]


Some data points (marked with XXX) haven't been collected yet.
This data has been collected on the Scalable Test Platform (STP)
8-way machines at OSDL.

REAIM is an updated version of AIM7.  Information on REAIM and
performance data collected on 2, 4, and 8 way machines can be found at

        http://developer.osdl.org/cliffw/

Hardware details of the STP 8-way machine are at the bottom of this page.
More details on each test run can be viewed by construcing a URL from the
STP ID #. For example, the first entry in the table can be seen at URL

	http://khack.osdl.org/279446

						Maximum Jobs Per Minute
Kernel Version  IO Scheduler   STP ID #	      Peak Load     Quick Convergence
2.6.0-test5	Deadline	279446		8950		8542	
		As-iosched	279444		8785		8489

	mm1	Deadline	279560		8303		7998
		As-iosched	279558		8401		7935

	mm2	Deadline	279929		8309		8063
		As-iosched	281948		8224		7741

	mm3	Deadline	281859		8438		8064
		As-iosched	281813		8207		7879

	mm4	Deadline	281814		8222		7991
		As-iosched	281815		8417		7963

2.6.0-test6	Deadline	281816		8302		7949   ****
		As-iosched	281817		6934		6945

	mm1	Deadline	281864		8375		7930
		As-iosched	281863		8163		7923

	mm2	Deadline	281948		8224		7741
		As-iosched	281865		8309		7616

	mm3	Deadline	281868		8280		8022
		As-iosched	281867		8348		8017

	mm4	Deadline	281870		8512		7868
		As-iosched	281869		8307		7749

2.6.0-test7	Deadline	281231		8413		8002   ****
		As-iosched	282095		6957		6790

	mm1	Deadline	281548		8567		8033
		As-iosched	282181		8346		7755

2.6.0-test8	Deadline	281671		8294		8020   ****
		As-iosched	281669		7018		6926

	mm1	Deadline	282183		8326		7997
		As-iosched	282182		8347		XXX

2.6.0-test9	Deadline			XXXX		XXXX
		As-iosched	282310		6868		6045   ****

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
@ 2003-10-27 23:33 John Hawkes
  0 siblings, 0 replies; 25+ messages in thread
From: John Hawkes @ 2003-10-27 23:33 UTC (permalink / raw)
  To: linux-kernel

I ran AIM7 on a 64p Altix NUMA (Itanium 2 Madison CPUs, 1.5 GHz), with an
XFS filesystem on each of 99 fiberchannel disks, with two different flavors
of AIM7 workloads, on both 2.6.0-test5 and 2.6.0-test8.  One workload shows
that -test8 is half the throughput of -test5.  The other workload shows
that -test8 is 22% faster.

The first workload is the default AIM7 "multiuser/shared system" workload.
2.6.0-test8 shows a peak throughput of 52% of -test5 -- it drops in half.
The -test8 peak at 64p is roughly the same as at 8p and 4p.  This
bottleneck cannot be attributed solely to I/O hardware, since our SGI
ProPack kernel (based upon 2.4.21) achieves a peak throughput at 64p that
is 30% higher than 2.6.0-test5's peak at 64p.

The second workload takes the same "multiuser/shared system" workload,
but turns off the O_SYNC flag for the three sync_* subtests.  This
generally produces higher peak throughput values than when using the
default O_SYNC, and the peak throughput values scale significantly better
as CPU counts increase.  Here the -test8 kernel is 22% *faster* than
-test5 at 64p (and 8x faster for no-O_SYNC than with-O_SYNC).

John Hawkes

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
@ 2003-10-26 10:38 rwhron
  2003-10-27 23:49 ` Dave Olien
  0 siblings, 1 reply; 25+ messages in thread
From: rwhron @ 2003-10-26 10:38 UTC (permalink / raw)
  To: linux-kernel; +Cc: akpm

2.6.0-test8-mm1 doesn't have the AIM7 database regression
that 2.6.0-test8 has.  The AIM7 fileserver and shared 
workloads regression between test5-mm2 and test8-mm1 is
about 18%.  Mainline regression was about 38% between test5
and test8 on AIM7 fileserver and shared workloads.

AIM7 dbase workload
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test5-mm2          32	609.3		311.9	139.5
2.6.0-test8-mm1          32	608.2		312.6	133.5
2.6.0-test5              32	590.2		322.0	131.9
2.6.0-test8              32	299.8		634.0	128.6

2.6.0-test5-mm2          64	786.2		483.5	253.2
2.6.0-test8-mm1          64	784.3		484.7	256.6
2.6.0-test5              64	774.3		490.9	256.6
2.6.0-test8              64	450.5		843.9	246.5

2.6.0-test8-mm1          96	876.2		650.8	384.0
2.6.0-test5-mm2          96	869.2		656.0	382.2
2.6.0-test5              96	859.0		663.9	389.5
2.6.0-test8              96	549.7		1037.4	366.0

2.6.0-test5-mm2          128	929.4		818.0	526.1
2.6.0-test8-mm1          128	924.7		822.2	523.3
2.6.0-test5              128	910.3		835.2	528.6
2.6.0-test8              128	558.7		1360.8	497.6

2.6.0-test5-mm2          160	962.5		987.4	674.6
2.6.0-test8-mm1          160	959.8		990.2	664.5
2.6.0-test5              160	741.5		1281.7	644.8
2.6.0-test8              160	522.1		1820.2	612.2

2.6.0-test5-mm2          192	993.9		1147.5	810.0
2.6.0-test8-mm1          192	983.7		1159.3	806.0
2.6.0-test5              192	840.0		1357.8	776.0
2.6.0-test8              192	515.5		2212.4	742.7

2.6.0-test5-mm2          224	1011.2		1315.8	943.5
2.6.0-test8-mm1          224	1011.1		1315.9	918.0
2.6.0-test5              224	986.9		1348.2	911.8
2.6.0-test8              224	529.9		2510.8	859.5

2.6.0-test5-mm2          256	1030.9		1475.1	1076.5
2.6.0-test8-mm1          256	1023.2		1486.2	1078.5
2.6.0-test5              256	999.5		1521.3	1029.3
2.6.0-test8              256	552.4		2753.0	980.4


-mm regressed about 20% going from test5 to test8 in AIM7
fileserver workload.  Linus tree regressed about 38%.

AIM7 fserver workload
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test5              4	79.7		304.0	38.3
2.6.0-test5-mm2          4	77.4		313.4	38.3
2.6.0-test8-mm1          4	66.5		364.5	34.1
2.6.0-test8              4	42.3		573.6	40.9

2.6.0-test5              8	129.8		373.5	66.3
2.6.0-test5-mm2          8	103.1		470.4	62.0
2.6.0-test8-mm1          8	85.1		569.6	63.9
2.6.0-test8              8	63.3		765.7	71.6

2.6.0-test5              12	164.1		443.0	93.1
2.6.0-test5-mm2          12	112.4		646.8	98.4
2.6.0-test8-mm1          12	92.3		788.0	101.8
2.6.0-test8              12	83.2		874.2	95.0

2.6.0-test5              16	173.4		559.0	116.5
2.6.0-test5-mm2          16	128.8		752.9	119.9
2.6.0-test8-mm1          16	104.4		928.5	124.1
2.6.0-test8              16	87.5		1108.4	102.5

2.6.0-test5              20	180.5		671.5	143.0
2.6.0-test5-mm2          20	141.7		855.3	144.1
2.6.0-test8-mm1          20	116.6		1039.9	143.9
2.6.0-test8              20	105.0		1154.6	136.7

2.6.0-test5              24	189.1		769.0	176.0
2.6.0-test5-mm2          24	156.2		930.9	180.2
2.6.0-test8-mm1          24	127.1		1144.1	174.4
2.6.0-test8              24	111.2		1308.1	162.2

2.6.0-test5              28	204.6		829.2	197.5
2.6.0-test5-mm2          28	158.7		1069.2	209.8
2.6.0-test8-mm1          28	127.8		1327.5	203.8
2.6.0-test8              28	130.7		1297.9	185.8

2.6.0-test5              32	213.7		907.4	228.9
2.6.0-test5-mm2          32	151.9		1276.7	239.7
2.6.0-test8-mm1          32	123.2		1573.5	243.7
2.6.0-test8              32	134.2		1444.5	212.9

On the shared workload, Linus tree regressed about 38% between
test5 and test8.  The -mm tree only regressed about 16%.

AIM7 shared workload
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test5              64	1888.8		197.2	168.3
2.6.0-test5-mm2          64	1517.6		245.4	177.0
2.6.0-test8-mm1          64	1265.2		294.4	174.8
2.6.0-test8              64	829.1		449.3	167.8

2.6.0-test5              128	2094.9		355.6	338.1
2.6.0-test5-mm2          128	1837.4		405.4	352.4
2.6.0-test8-mm1          128	1558.3		478.1	348.9
2.6.0-test8              128	1105.2		674.1	330.5

2.6.0-test5              192	2256.0		495.3	507.8
2.6.0-test5-mm2          192	2074.9		538.5	530.9
2.6.0-test8-mm1          192	1755.2		636.6	526.9
2.6.0-test8              192	1320.0		846.6	504.7

2.6.0-test5              256	2389.0		623.7	683.8
2.6.0-test5-mm2          256	2207.2		675.0	720.6
2.6.0-test8-mm1          256	1855.2		803.1	708.4
2.6.0-test8              256	1462.1		1019.0	672.7

2.6.0-test5              320	2457.9		757.7	856.6
2.6.0-test5-mm2          320	2327.9		800.0	919.4
2.6.0-test8-mm1          320	1958.4		951.0	894.6
2.6.0-test8              320	1573.0		1184.0	845.2

2.6.0-test5              384	2480.4		901.0	1036.5
2.6.0-test5-mm2          384	2440.2		915.9	1111.6
2.6.0-test8-mm1          384	1997.9		1118.6	1087.2
2.6.0-test8              384	1601.1		1395.9	1028.7

2.6.0-test5              448	2529.8		1030.7	1209.2
2.6.0-test5-mm2          448	2510.0		1038.8	1306.1
2.6.0-test8-mm1          448	2030.0		1284.4	1279.6
2.6.0-test8              448	1659.7		1571.0	1205.2

2.6.0-test5              512	2566.2		1161.2	1386.2
2.6.0-test5-mm2          512	2579.1		1155.4	1504.0
2.6.0-test8-mm1          512	2046.4		1456.1	1481.3
2.6.0-test8              512	1614.4		1845.8	1381.5


The big increase in max latency on tiobench sequential reads at
256 threads affects both test8 and test8-mm1.

tiobench-0.3.3
File size = 8192 megabytes
Blk Size  = 4096 bytes

Sequential Reads ext2
                    Num                    Avg       Maximum     Lat%
Kernel              Thr   Rate  (CPU%)   Latency     Latency      >2s
------------------  ---  --------------------------------------------
2.6.0-test5           1   28.55 13.89%     0.408      754.59  0.00000
2.6.0-test5-mm2       1   28.66 13.98%     0.406      742.76  0.00000
2.6.0-test8           1   28.66 13.57%     0.406     1010.39  0.00000
2.6.0-test8-mm1       1   28.53 13.79%     0.408      977.75  0.00000

2.6.0-test5           8   34.01 17.09%     2.620      834.85  0.00000
2.6.0-test5-mm2       8   32.44 16.44%     2.765      844.99  0.00000
2.6.0-test8           8   20.83  9.82%     4.437     1626.42  0.00000
2.6.0-test8-mm1       8   21.31 10.27%     4.334     1669.82  0.00000

2.6.0-test5          16   30.13 15.04%     5.980     1337.69  0.00000
2.6.0-test5-mm2      16   29.71 14.97%     6.024     1115.95  0.00000
2.6.0-test8          16   15.88  7.53%    11.604     2888.67  0.00000
2.6.0-test8-mm1      16   16.40  7.93%    11.203     2954.75  0.00000

2.6.0-test5          32   29.46 14.62%    11.954     1560.02  0.00000
2.6.0-test5-mm2      32   28.01 14.26%    12.808     1627.00  0.00000
2.6.0-test8          32   16.17  7.55%    22.170     5119.98  0.00000
2.6.0-test8-mm1      32   15.78  7.74%    23.080     4826.59  0.00000

2.6.0-test5          64   25.24 13.57%    26.901     1743.05  0.00000
2.6.0-test5-mm2      64   24.01 13.27%    27.922     1849.82  0.00000
2.6.0-test8          64    8.71  4.78%    82.514    11744.03  3.04523
2.6.0-test8-mm1      64    9.06  5.13%    79.839     9458.66  2.92964

2.6.0-test5         128   10.41  9.76%   129.662     1820.78  0.00000
2.6.0-test5-mm2     128   11.00 10.44%   120.118     1742.44  0.00000
2.6.0-test8         128    3.04  2.64%   466.928    26583.72  8.12159
2.6.0-test8-mm1     128    3.00  2.78%   476.711    22323.98  8.28352

2.6.0-test5         256    9.13 13.30%   305.097     1446.43  0.00000
2.6.0-test5-mm2     256    9.09 13.60%   304.019     1648.80  0.00000
2.6.0-test8         256    2.66  2.41%   960.563   679406.45  9.02886
2.6.0-test8-mm1     256    2.80  2.61%   916.388   678138.20  8.60138


test8 and test8-mm1 have an increase in requests taking more than 2 
seconds on tiobench random reads at 256 threads.  -mm1 regressed less
than mainline between test5 and test8.

Random Reads ext2
                    Num                    Avg       Maximum     Lat%
Kernel              Thr   Rate  (CPU%)   Latency     Latency      >2s
------------------  ---  --------------------------------------------
2.6.0-test5           1    0.95  0.94%    12.373      114.89  0.00000
2.6.0-test5-mm2       1    0.90  0.88%    13.061      113.19  0.00000
2.6.0-test8           1    0.95  0.85%    12.360      113.96  0.00000
2.6.0-test8-mm1       1    0.89  0.88%    13.222      107.35  0.00000

2.6.0-test5           8    4.11  4.58%    19.433      137.74  0.00000
2.6.0-test5-mm2       8    3.94  4.03%    20.931      144.16  0.00000
2.6.0-test8           8    3.52  3.59%    25.298      477.80  0.00000
2.6.0-test8-mm1       8    4.51  4.80%    19.777      299.43  0.00000

2.6.0-test5          16    4.05  3.90%    42.424      236.49  0.00000
2.6.0-test5-mm2      16    4.09  4.31%    41.651      255.17  0.00000
2.6.0-test8          16    3.92  3.56%    45.278      843.31  0.00000
2.6.0-test8-mm1      16    4.11  3.46%    42.043      608.63  0.00000

2.6.0-test5          32    4.41  3.58%    70.910      516.28  0.00000
2.6.0-test5-mm2      32    4.23  4.17%    78.492      372.63  0.00000
2.6.0-test8          32    4.25  3.45%    77.345     1324.85  0.00000
2.6.0-test8-mm1      32    3.70  3.69%    91.986     1270.38  0.00000

2.6.0-test5          64    4.47  4.00%   137.277     1077.53  0.00000
2.6.0-test5-mm2      64    4.79  4.53%   124.035     1144.38  0.00000
2.6.0-test8          64    3.57  3.32%   181.257     3029.47  0.00000
2.6.0-test8-mm1      64    3.66  3.24%   175.724     2493.85  0.00000

2.6.0-test5         128    4.86  4.83%   243.704     2287.09  0.00000
2.6.0-test5-mm2     128    5.08  4.85%   228.102     2391.21  0.00000
2.6.0-test8         128    3.28  3.08%   379.081     5007.00  0.00000
2.6.0-test8-mm1     128    3.40  3.52%   372.210     4671.71  0.00000

2.6.0-test5         256    5.51  7.00%   427.503     2511.79  0.00000
2.6.0-test5-mm2     256    5.23  7.39%   449.007     3169.28  0.00000
2.6.0-test8         256    2.07  2.40%  1162.555    13971.08 16.95313
2.6.0-test8-mm1     256    2.36  2.99%  1019.360    10935.31 10.65104

-- 
Randy Hron
http://home.earthlink.net/~rwhron/kernel/bigbox.html


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
@ 2003-10-24 23:43 rwhron
  0 siblings, 0 replies; 25+ messages in thread
From: rwhron @ 2003-10-24 23:43 UTC (permalink / raw)
  To: piggin; +Cc: linux-kernel, akpm

Quad P3 Xeon.  Only the as-iosched.c from test5
really fixes the regression.

2.6.0-test8 = vanilla
2.6.0-test8-as = vanilla + as-iosched.c from test5
2.6.0-test8-as2 = vanilla + first as-fix patch
2.6.0-test8-as = vanilla + second as-fix patch

AIM7 dbase workload
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test8-as           32	601.0		316.3	134.0
2.6.0-test8-as3          32	320.7		592.7	128.3
2.6.0-test8-as2          32	304.3		624.6	128.8
2.6.0-test8              32	299.8		634.0	128.6

2.6.0-test8-as           64	782.6		485.8	259.2
2.6.0-test8-as3          64	470.1		808.7	247.0
2.6.0-test8-as2          64	458.2		829.8	245.7
2.6.0-test8              64	450.5		843.9	246.5

2.6.0-test8-as           96	869.1		656.1	387.6
2.6.0-test8              96	549.7		1037.4	366.0
2.6.0-test8-as2          96	543.4		1049.4	370.0
2.6.0-test8-as3          96	521.4		1093.6	372.5

2.6.0-test8-as           128	919.3		827.0	522.8
2.6.0-test8-as2          128	564.1		1348.0	496.6
2.6.0-test8              128	558.7		1360.8	497.6
2.6.0-test8-as3          128	551.9		1377.6	499.9

2.6.0-test8-as           160	955.0		995.2	650.3
2.6.0-test8-as2          160	534.2		1779.2	618.2
2.6.0-test8              160	522.1		1820.2	612.2
2.6.0-test8-as3          160	517.0		1838.4	611.9

2.6.0-test8-as           192	981.9		1161.5	790.9
2.6.0-test8-as2          192	535.6		2129.3	746.5
2.6.0-test8-as3          192	530.1		2151.3	744.8
2.6.0-test8              192	515.5		2212.4	742.7

2.6.0-test8-as           224	1002.3		1327.5	910.2
2.6.0-test8-as3          224	531.7		2502.3	850.2
2.6.0-test8              224	529.9		2510.8	859.5
2.6.0-test8-as2          224	526.4		2527.5	852.9

2.6.0-test8-as           256	1009.9		1505.7	1027.7
2.6.0-test8-as3          256	555.3		2738.2	976.5
2.6.0-test8              256	552.4		2753.0	980.4
2.6.0-test8-as2          256	548.9		2770.4	975.3


AIM7 fserver workload
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test8-as           4	80.5		301.0	37.0
2.6.0-test8-as2          4	46.4		521.9	42.5
2.6.0-test8-as3          4	43.8		553.0	44.5
2.6.0-test8              4	42.3		573.6	40.9

2.6.0-test8-as           8	133.0		364.6	64.0
2.6.0-test8-as3          8	65.5		740.3	71.1
2.6.0-test8              8	63.3		765.7	71.6
2.6.0-test8-as2          8	59.9		809.1	74.4

2.6.0-test8-as           12	166.7		436.3	93.4
2.6.0-test8              12	83.2		874.2	95.0
2.6.0-test8-as3          12	67.2		1081.6	99.8
2.6.0-test8-as2          12	62.6		1161.5	99.5

2.6.0-test8-as           16	175.9		551.2	111.5
2.6.0-test8              16	87.5		1108.4	102.5
2.6.0-test8-as2          16	84.1		1152.9	109.6
2.6.0-test8-as3          16	82.0		1182.4	112.0

2.6.0-test8-as           20	186.3		650.7	133.8
2.6.0-test8              20	105.0		1154.6	136.7
2.6.0-test8-as2          20	101.3		1196.4	135.8
2.6.0-test8-as3          20	100.5		1206.3	128.8

2.6.0-test8-as           24	202.8		717.3	161.2
2.6.0-test8-as2          24	121.4		1198.0	164.1
2.6.0-test8              24	111.2		1308.1	162.2
2.6.0-test8-as3          24	104.8		1387.9	163.9

2.6.0-test8-as           28	207.1		819.3	200.0
2.6.0-test8              28	130.7		1297.9	185.8
2.6.0-test8-as2          28	128.0		1326.1	190.3
2.6.0-test8-as3          28	123.4		1374.5	186.9

2.6.0-test8-as           32	210.6		920.7	226.5
2.6.0-test8              32	134.2		1444.5	212.9
2.6.0-test8-as3          32	130.4		1487.1	217.7
2.6.0-test8-as2          32	122.4		1583.7	217.5


AIM7 shared workload
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test8-as           64	1949.6		191.1	170.8
2.6.0-test8-as2          64	880.0		423.3	166.6
2.6.0-test8-as3          64	878.8		423.9	166.3
2.6.0-test8              64	829.1		449.3	167.8

2.6.0-test8-as           128	2148.3		346.8	339.2
2.6.0-test8-as3          128	1176.2		633.4	331.1
2.6.0-test8-as2          128	1128.7		660.0	331.4
2.6.0-test8              128	1105.2		674.1	330.5

2.6.0-test8-as           192	2265.7		493.2	507.1
2.6.0-test8-as3          192	1330.7		839.7	505.0
2.6.0-test8              192	1320.0		846.6	504.7
2.6.0-test8-as2          192	1317.8		848.0	503.3

2.6.0-test8-as           256	2382.9		625.3	678.0
2.6.0-test8-as3          256	1490.3		999.7	673.6
2.6.0-test8              256	1462.1		1019.0	672.7
2.6.0-test8-as2          256	1453.3		1025.2	670.9

2.6.0-test8-as           320	2450.5		760.0	853.7
2.6.0-test8-as3          320	1579.7		1178.9	849.0
2.6.0-test8              320	1573.0		1184.0	845.2
2.6.0-test8-as2          320	1564.0		1190.8	848.2

2.6.0-test8-as           384	2517.4		887.8	1030.6
2.6.0-test8-as3          384	1633.0		1368.6	1020.7
2.6.0-test8-as2          384	1609.8		1388.3	1026.2
2.6.0-test8              384	1601.1		1395.9	1028.7

-- 
Randy Hron
http://home.earthlink.net/~rwhron/kernel/bigbox.html


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-20  0:37 rwhron
  2003-10-20  1:35 ` Nick Piggin
  2003-10-20  4:51 ` Dave Olien
@ 2003-10-24 13:52 ` Magnus Andersson
  2 siblings, 0 replies; 25+ messages in thread
From: Magnus Andersson @ 2003-10-24 13:52 UTC (permalink / raw)
  To: rwhron; +Cc: linux-kernel

On Sun, Oct 19, 2003 at 08:37:45PM -0400, rwhron@earthlink.net wrote:
> There was about a 50% regression in jobs/minute in AIM7
> database workload on quad P3 Xeon.  The CPU time has not
> gone up, so the extra run time is coming from something
> else.  (I/O or I/O scheduler?)

Hello all!

This sounds like the same problem I had with the 2.6 kernel.
I posted some stuff on the kernel-list about this.

Make a search for "2.6.0-test4 parallel seek & read problem"
on google to read my posts.

I think I ran into two different unrelated problems.
 
First problem was solved, but second problem is still there.
It is probably the same you are having.

My finding was that the problem is not the scheduler, but some
other part of the IO system.

1 seek & read issued by a program goes to the disk as 2 different
seeks & 2 reads, thus only giving 50 % performance.

I think the problem is that the elevators are flushed so often
so merges that should happend never happens.

/Magnus

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
@ 2003-10-24 12:46 rwhron
  0 siblings, 0 replies; 25+ messages in thread
From: rwhron @ 2003-10-24 12:46 UTC (permalink / raw)
  To: piggin; +Cc: linux-kernel

>> NOTPM was better with AS in test5? Does using as-iosched.c from test5
>> in a test8 kernel help?

> If that helps, can you try this patch.

That patch helps the AIM7 database load a little, but the
big regression is still there.

2.6.0-test8 = vanilla
2.6.0-test8-as = vanilla + as-iosched.c from test5
2.6.0-test8-asc2 = vanilla + patch.

AIM7 dbase workloa
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test8-as           32	601.0		316.3	134.0
2.6.0-test8-as2          32	304.3		624.6	128.8
2.6.0-test8              32	299.8		634.0	128.6

2.6.0-test8-as           64	782.6		485.8	259.2
2.6.0-test8-as2          64	458.2		829.8	245.7
2.6.0-test8              64	450.5		843.9	246.5

2.6.0-test8-as           96	869.1		656.1	387.6
2.6.0-test8              96	549.7		1037.4	366.0
2.6.0-test8-as2          96	543.4		1049.4	370.0

2.6.0-test8-as           128	919.3		827.0	522.8
2.6.0-test8-as2          128	564.1		1348.0	496.6
2.6.0-test8              128	558.7		1360.8	497.6

2.6.0-test8-as           160	955.0		995.2	650.3
2.6.0-test8-as2          160	534.2		1779.2	618.2
2.6.0-test8              160	522.1		1820.2	612.2

2.6.0-test8-as           192	981.9		1161.5	790.9
2.6.0-test8-as2          192	535.6		2129.3	746.5
2.6.0-test8              192	515.5		2212.4	742.7

2.6.0-test8-as           224	1002.3		1327.5	910.2
2.6.0-test8              224	529.9		2510.8	859.5
2.6.0-test8-as2          224	526.4		2527.5	852.9

2.6.0-test8-as           256	1009.9		1505.7	1027.7
2.6.0-test8              256	552.4		2753.0	980.4
2.6.0-test8-as2          256	548.9		2770.4	975.3


AIM7 fserver workload
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test8-as           4	80.5		301.0	37.0
2.6.0-test8-as2          4	46.4		521.9	42.5
2.6.0-test8              4	42.3		573.6	40.9

2.6.0-test8-as           8	133.0		364.6	64.0
2.6.0-test8              8	63.3		765.7	71.6
2.6.0-test8-as2          8	59.9		809.1	74.4

2.6.0-test8-as           12	166.7		436.3	93.4
2.6.0-test8              12	83.2		874.2	95.0
2.6.0-test8-as2          12	62.6		1161.5	99.5

2.6.0-test8-as           16	175.9		551.2	111.5
2.6.0-test8              16	87.5		1108.4	102.5
2.6.0-test8-as2          16	84.1		1152.9	109.6

2.6.0-test8-as           20	186.3		650.7	133.8
2.6.0-test8              20	105.0		1154.6	136.7
2.6.0-test8-as2          20	101.3		1196.4	135.8

2.6.0-test8-as           24	202.8		717.3	161.2
2.6.0-test8-as2          24	121.4		1198.0	164.1
2.6.0-test8              24	111.2		1308.1	162.2

2.6.0-test8-as           28	207.1		819.3	200.0
2.6.0-test8              28	130.7		1297.9	185.8
2.6.0-test8-as2          28	128.0		1326.1	190.3

2.6.0-test8-as           32	210.6		920.7	226.5
2.6.0-test8              32	134.2		1444.5	212.9
2.6.0-test8-as2          32	122.4		1583.7	217.5


AIM7 shared workload
kernel                   Tasks   Jobs/Min        Real    CPU
2.6.0-test8-as           64	1949.6		191.1	170.8
2.6.0-test8-as2          64	880.0		423.3	166.6
2.6.0-test8              64	829.1		449.3	167.8

2.6.0-test8-as           128	2148.3		346.8	339.2
2.6.0-test8-as2          128	1128.7		660.0	331.4
2.6.0-test8              128	1105.2		674.1	330.5

2.6.0-test8-as           192	2265.7		493.2	507.1
2.6.0-test8              192	1320.0		846.6	504.7
2.6.0-test8-as2          192	1317.8		848.0	503.3

2.6.0-test8-as           256	2382.9		625.3	678.0
2.6.0-test8              256	1462.1		1019.0	672.7
2.6.0-test8-as2          256	1453.3		1025.2	670.9

2.6.0-test8-as           320	2450.5		760.0	853.7
2.6.0-test8              320	1573.0		1184.0	845.2
2.6.0-test8-as2          320	1564.0		1190.8	848.2

2.6.0-test8-as           384	2517.4		887.8	1030.6
2.6.0-test8-as2          384	1609.8		1388.3	1026.2
2.6.0-test8              384	1601.1		1395.9	1028.7

2.6.0-test8-as           448	2547.5		1023.5	1214.4
2.6.0-test8              448	1659.7		1571.0	1205.2
2.6.0-test8-as2          448	1656.4		1574.2	1201.6

2.6.0-test8-as           512	2594.6		1148.5	1389.7
2.6.0-test8-as2          512	1620.4		1839.0	1378.0
2.6.0-test8              512	1614.4		1845.8	1381.5

--
Randy Hron
http://home.earthlink.net/~rwhron/kernel/bigbox.html


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
@ 2003-10-24  0:10 rwhron
  0 siblings, 0 replies; 25+ messages in thread
From: rwhron @ 2003-10-24  0:10 UTC (permalink / raw)
  To: piggin; +Cc: linux-kernel

> Randy can reproduce regressions, so I'll work on his first.

Below is test8 and test8 with the test5 as-iosched.c (2.6.0-test8-as).
Jobs/minute is way up, and total runtime is way down with the test5
as-iosched.c.   

The machine (quad P3 xeon) locked up during a bonnie++ run.  I can't
compare that to vanilla test8 because I stopped the test8 to run the
test5 as-iosched.   Both of these were compiled with gcc-3.3.1 because
of the RedHat gcc-2.96 bug in test8.  (In case the compiler was part
of the instability).

Earlier I ran 2.6.0-test3-mm2 compiled with gcc-3.3.1, and it
was stable on this box.

AIM7 dbase workload
kernel                   Tasks   Jobs/Min       Real    CPU
2.6.0-test8-as           32	601.0	       316.3	134.0
2.6.0-test8              32	299.8	       634.0	128.6

2.6.0-test8-as           64	782.6	       485.8	259.2
2.6.0-test8              64	450.5	       843.9	246.5

2.6.0-test8-as           96	869.1	       656.1	387.6
2.6.0-test8              96	549.7	       1037.4	366.0

2.6.0-test8-as           128	919.3	       827.0	522.8
2.6.0-test8              128	558.7	       1360.8	497.6

2.6.0-test8-as           160	955.0	       995.2	650.3
2.6.0-test8              160	522.1	       1820.2	612.2

2.6.0-test8-as           192	981.9	       1161.5	790.9
2.6.0-test8              192	515.5	       2212.4	742.7

2.6.0-test8-as           224	1002.3	       1327.5	910.2
2.6.0-test8              224	529.9	       2510.8	859.5

2.6.0-test8-as           256	1009.9	       1505.7	1027.7
2.6.0-test8              256	552.4	       2753.0	980.4


AIM7 fserver workload
kernel                   Tasks   Jobs/Min       Real    CPU
2.6.0-test8-as           4	80.5	       301.0	37.0
2.6.0-test8              4	42.3	       573.6	40.9

2.6.0-test8-as           8	133.0	       364.6	64.0
2.6.0-test8              8	63.3	       765.7	71.6

2.6.0-test8-as           12	166.7	       436.3	93.4
2.6.0-test8              12	83.2	       874.2	95.0

2.6.0-test8-as           16	175.9	       551.2	111.5
2.6.0-test8              16	87.5	       1108.4	102.5

2.6.0-test8-as           20	186.3	       650.7	133.8
2.6.0-test8              20	105.0	       1154.6	136.7

2.6.0-test8-as           24	202.8	       717.3	161.2
2.6.0-test8              24	111.2	       1308.1	162.2

2.6.0-test8-as           28	207.1	       819.3	200.0
2.6.0-test8              28	130.7	       1297.9	185.8

2.6.0-test8-as           32	210.6	       920.7	226.5
2.6.0-test8              32	134.2	       1444.5	212.9


AIM7 shared workload
kernel                   Tasks   Jobs/Min       Real    CPU
2.6.0-test8-as           64	1949.6	       191.1	170.8
2.6.0-test8              64	829.1	       449.3	167.8

2.6.0-test8-as           128	2148.3	       346.8	339.2
2.6.0-test8              128	1105.2	       674.1	330.5

2.6.0-test8-as           192	2265.7	       493.2	507.1
2.6.0-test8              192	1320.0	       846.6	504.7

2.6.0-test8-as           256	2382.9	       625.3	678.0
2.6.0-test8              256	1462.1	       1019.0	672.7

2.6.0-test8-as           320	2450.5	       760.0	853.7
2.6.0-test8              320	1573.0	       1184.0	845.2

2.6.0-test8-as           384	2517.4	       887.8	1030.6
2.6.0-test8              384	1601.1	       1395.9	1028.7

2.6.0-test8-as           448	2547.5	       1023.5	1214.4
2.6.0-test8              448	1659.7	       1571.0	1205.2

2.6.0-test8-as           512	2594.6	       1148.5	1389.7
2.6.0-test8              512	1614.4	       1845.8	1381.5

I'm running Nick's patch
http://marc.theaimsgroup.com/?l=linux-kernel&m=106688130229337&w=2
on test8 at the moment.  I should have some comparison numbers 
within 24 hours.

-- 
Randy Hron
http://home.earthlink.net/~rwhron/kernel/bigbox.html


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-23 20:35       ` Dave Olien
@ 2003-10-23 23:07         ` Nick Piggin
  0 siblings, 0 replies; 25+ messages in thread
From: Nick Piggin @ 2003-10-23 23:07 UTC (permalink / raw)
  To: Dave Olien; +Cc: rwhron, venom, linux-kernel, akpm, Mary Edie Meredith



Dave Olien wrote:

>test5 didn't work with dbt2.  The database run kept hanging.
>I've been running reaim, which doesn't have those problems with
>older kernels, but has shown similar performance problems.
>
>I need to also check again that reaim and dbt2 still show similar
>performance issues with newer kernel versions.
>
>I've lost my test machine for today.  But I'll try test5 as-iosched
>in a test8 kernel with reaim tomorrow.  If it works well, I'll try your
>following patch.  Probably get it in the mail tomorrow afternoon.
>

Hi Dave,
Yeah thats right - I forgot. It was hanging because of as-iosched.c
too, so don't bother trying to run it :(

Anyway, Randy can reproduce regressions, so I'll work on his first.



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-23  2:24     ` Nick Piggin
  2003-10-23  3:48       ` Nick Piggin
@ 2003-10-23 20:35       ` Dave Olien
  2003-10-23 23:07         ` Nick Piggin
  1 sibling, 1 reply; 25+ messages in thread
From: Dave Olien @ 2003-10-23 20:35 UTC (permalink / raw)
  To: Nick Piggin; +Cc: rwhron, venom, linux-kernel, akpm, Mary Edie Meredith


test5 didn't work with dbt2.  The database run kept hanging.
I've been running reaim, which doesn't have those problems with
older kernels, but has shown similar performance problems.

I need to also check again that reaim and dbt2 still show similar
performance issues with newer kernel versions.

I've lost my test machine for today.  But I'll try test5 as-iosched
in a test8 kernel with reaim tomorrow.  If it works well, I'll try your
following patch.  Probably get it in the mail tomorrow afternoon.

Dave

On Thu, Oct 23, 2003 at 12:24:09PM +1000, Nick Piggin wrote:
> 
> 
> Dave Olien wrote:
> 
> >Sorry, this patch didn't fix our performance problems.  Mary just
> >finished running dbt2 on test8 with your patch:
> >
> >NOTPM   kernel          scheduler
> >965     2.6.0-test8-np  AS
> >1632    2.6.-test6-mm4  deadline
> >
> 
> Thanks. hmm.
> And NOTPM was better with AS in test5? Does using as-iosched.c from test5
> in a test8 kernel help?
> 

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-23  2:24     ` Nick Piggin
@ 2003-10-23  3:48       ` Nick Piggin
  2003-10-23 20:35       ` Dave Olien
  1 sibling, 0 replies; 25+ messages in thread
From: Nick Piggin @ 2003-10-23  3:48 UTC (permalink / raw)
  To: Dave Olien; +Cc: rwhron, venom, linux-kernel, akpm, Mary Edie Meredith

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]



Nick Piggin wrote:

>
>
> Dave Olien wrote:
>
>> Sorry, this patch didn't fix our performance problems.  Mary just
>> finished running dbt2 on test8 with your patch:
>>
>> NOTPM   kernel          scheduler
>> 965     2.6.0-test8-np  AS
>> 1632    2.6.-test6-mm4  deadline
>>
>
> Thanks. hmm.
> And NOTPM was better with AS in test5? Does using as-iosched.c from test5
> in a test8 kernel help?
>

If that helps, can you try this patch.


[-- Attachment #2: as-fix.patch --]
[-- Type: text/plain, Size: 2416 bytes --]

 linux-2.6-npiggin/drivers/block/as-iosched.c |   23 +++--------------------
 1 files changed, 3 insertions(+), 20 deletions(-)

diff -puN drivers/block/as-iosched.c~as-fix drivers/block/as-iosched.c
--- linux-2.6/drivers/block/as-iosched.c~as-fix	2003-10-23 13:46:11.000000000 +1000
+++ linux-2.6-npiggin/drivers/block/as-iosched.c	2003-10-23 13:47:58.000000000 +1000
@@ -99,7 +99,6 @@ struct as_data {
 	sector_t last_sector[2];	/* last REQ_SYNC & REQ_ASYNC sectors */
 	struct list_head *dispatch;	/* driver dispatch queue */
 	struct list_head *hash;		/* request hash */
-	unsigned long new_success; /* anticipation success on new proc */
 	unsigned long current_batch_expires;
 	unsigned long last_check_fifo[2];
 	int changed_batch;		/* 1: waiting for old batch to end */
@@ -585,18 +584,11 @@ static void as_antic_stop(struct as_data
 	int status = ad->antic_status;
 
 	if (status == ANTIC_WAIT_REQ || status == ANTIC_WAIT_NEXT) {
-		struct as_io_context *aic;
-
 		if (status == ANTIC_WAIT_NEXT)
 			del_timer(&ad->antic_timer);
 		ad->antic_status = ANTIC_FINISHED;
 		/* see as_work_handler */
 		kblockd_schedule_work(&ad->antic_work);
-
-		aic = ad->io_context->aic;
-		if (aic->seek_samples == 0)
-			/* new process */
-			ad->new_success = (ad->new_success * 3) / 4 + 256;
 	}
 }
 
@@ -612,14 +604,8 @@ static void as_antic_timeout(unsigned lo
 	spin_lock_irqsave(q->queue_lock, flags);
 	if (ad->antic_status == ANTIC_WAIT_REQ
 			|| ad->antic_status == ANTIC_WAIT_NEXT) {
-		struct as_io_context *aic;
 		ad->antic_status = ANTIC_FINISHED;
 		kblockd_schedule_work(&ad->antic_work);
-
-		aic = ad->io_context->aic;
-		if (aic->seek_samples == 0)
-			/* new process */
-			ad->new_success = (ad->new_success * 3) / 4;
 	}
 	spin_unlock_irqrestore(q->queue_lock, flags);
 }
@@ -708,11 +694,10 @@ static int as_can_break_anticipation(str
 		return 1;
 	}
 
-	if (ad->new_success < 256 &&
-			(aic->seek_samples == 0 || aic->ttime_samples == 0)) {
+	if (aic->seek_samples == 0 || aic->ttime_samples == 0) {
 		/*
-		 * Process has just started IO and we have a bad history of
-		 * success anticipating on new processes!
+		 * Process has just started IO. Don't anticipate.
+		 * TODO! Must fix this up.
 		 */
 		return 1;
 	}
@@ -1822,8 +1807,6 @@ static int as_init(request_queue_t *q, e
 	if (ad->write_batch_count < 2)
 		ad->write_batch_count = 2;
 
-	ad->new_success = 512;
-
 	return 0;
 }
 

_

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-22 18:30   ` Dave Olien
  2003-10-22 19:08     ` Dave Olien
@ 2003-10-23  2:24     ` Nick Piggin
  2003-10-23  3:48       ` Nick Piggin
  2003-10-23 20:35       ` Dave Olien
  1 sibling, 2 replies; 25+ messages in thread
From: Nick Piggin @ 2003-10-23  2:24 UTC (permalink / raw)
  To: Dave Olien; +Cc: rwhron, venom, linux-kernel, akpm, Mary Edie Meredith



Dave Olien wrote:

>Sorry, this patch didn't fix our performance problems.  Mary just
>finished running dbt2 on test8 with your patch:
>
>NOTPM   kernel          scheduler
>965     2.6.0-test8-np  AS
>1632    2.6.-test6-mm4  deadline
>

Thanks. hmm.
And NOTPM was better with AS in test5? Does using as-iosched.c from test5
in a test8 kernel help?



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-22 18:30   ` Dave Olien
@ 2003-10-22 19:08     ` Dave Olien
  2003-10-23  2:24     ` Nick Piggin
  1 sibling, 0 replies; 25+ messages in thread
From: Dave Olien @ 2003-10-22 19:08 UTC (permalink / raw)
  To: Nick Piggin; +Cc: rwhron, venom, linux-kernel, akpm, Mary Edie Meredith



I don't know how useful this might but.  I've been trying to
track down which patches to as-iosched resulted in the performance
drop.  The results so far are a little confusing.  I have been
running reaim on an 8-way system, comparing as-iosched performance
with deadline, on each kernel version.  The numbers are "number
of jobs per minute".  So, larger is better.

			Deadline		As-iosched
2.6.0-test5   		8542			8589
2.6.0-test5-mm1		8303			8401
2.6.0-test5-mm2		8309			8224
2.6.0-test5-mm3		8222			8417
2.6.0-test6		8302			6934	****
2.6.0-test6-mm1		8375			8163
2.6.0-test6-mm2		???			8309

I'm still getting data on test6-mm*.  It's curious that
performance dropped for test6, but came back for test6-mm1.

On Wed, Oct 22, 2003 at 11:30:28AM -0700, Dave Olien wrote:
> 
> Sorry, this patch didn't fix our performance problems.  Mary just
> finished running dbt2 on test8 with your patch:
> 
> NOTPM   kernel          scheduler
> 965     2.6.0-test8-np  AS
> 1632    2.6.-test6-mm4  deadline
> 
> This is an 8-way system with DAC960 and 12 LUNs, using raw devices.
> That's still quite a sizeable drop.
> 

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-22  9:54 ` Nick Piggin
@ 2003-10-22 18:30   ` Dave Olien
  2003-10-22 19:08     ` Dave Olien
  2003-10-23  2:24     ` Nick Piggin
  0 siblings, 2 replies; 25+ messages in thread
From: Dave Olien @ 2003-10-22 18:30 UTC (permalink / raw)
  To: Nick Piggin; +Cc: rwhron, venom, linux-kernel, akpm, Mary Edie Meredith


Sorry, this patch didn't fix our performance problems.  Mary just
finished running dbt2 on test8 with your patch:

NOTPM   kernel          scheduler
965     2.6.0-test8-np  AS
1632    2.6.-test6-mm4  deadline

This is an 8-way system with DAC960 and 12 LUNs, using raw devices.
That's still quite a sizeable drop.

On Wed, Oct 22, 2003 at 07:54:46PM +1000, Nick Piggin wrote:
> 
> Could you please try the following patch against 2.6.0-test8.
> Thanks. I'll have to come up with something more comprehensive
> because this destroys find | xargs grep during other IO. It
> looks like this might be the cause of a lot of AS's database
> problems though.
> 
> I'm working on something...
> 

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-21 13:05 rwhron
  2003-10-21 15:57 ` Nick Piggin
  2003-10-21 21:07 ` venom
@ 2003-10-22  9:54 ` Nick Piggin
  2003-10-22 18:30   ` Dave Olien
  2 siblings, 1 reply; 25+ messages in thread
From: Nick Piggin @ 2003-10-22  9:54 UTC (permalink / raw)
  To: rwhron; +Cc: venom, linux-kernel, dmo, akpm

[-- Attachment #1: Type: text/plain, Size: 591 bytes --]



rwhron@earthlink.net wrote:

>>If you have time, would you please try testing as-iosched.c from
>>test5 in a later kernel (it won't go into test8-mm1 though).
>>
>
>copying drivers/block/as-iosched.c from 2.6.0-test5 to test8
>looks like it fixes the regression.  Here are the results so far.
>

Could you please try the following patch against 2.6.0-test8.
Thanks. I'll have to come up with something more comprehensive
because this destroys find | xargs grep during other IO. It
looks like this might be the cause of a lot of AS's database
problems though.

I'm working on something...


[-- Attachment #2: as-fix.patch --]
[-- Type: text/plain, Size: 2416 bytes --]

 linux-2.6-npiggin/drivers/block/as-iosched.c |   23 +++--------------------
 1 files changed, 3 insertions(+), 20 deletions(-)

diff -puN drivers/block/as-iosched.c~as-fix drivers/block/as-iosched.c
--- linux-2.6/drivers/block/as-iosched.c~as-fix	2003-10-22 19:47:48.000000000 +1000
+++ linux-2.6-npiggin/drivers/block/as-iosched.c	2003-10-22 19:49:41.000000000 +1000
@@ -99,7 +99,6 @@ struct as_data {
 	sector_t last_sector[2];	/* last REQ_SYNC & REQ_ASYNC sectors */
 	struct list_head *dispatch;	/* driver dispatch queue */
 	struct list_head *hash;		/* request hash */
-	unsigned long new_success; /* anticipation success on new proc */
 	unsigned long current_batch_expires;
 	unsigned long last_check_fifo[2];
 	int changed_batch;		/* 1: waiting for old batch to end */
@@ -585,18 +584,11 @@ static void as_antic_stop(struct as_data
 	int status = ad->antic_status;
 
 	if (status == ANTIC_WAIT_REQ || status == ANTIC_WAIT_NEXT) {
-		struct as_io_context *aic;
-
 		if (status == ANTIC_WAIT_NEXT)
 			del_timer(&ad->antic_timer);
 		ad->antic_status = ANTIC_FINISHED;
 		/* see as_work_handler */
 		kblockd_schedule_work(&ad->antic_work);
-
-		aic = ad->io_context->aic;
-		if (aic->seek_samples == 0)
-			/* new process */
-			ad->new_success = (ad->new_success * 3) / 4 + 256;
 	}
 }
 
@@ -612,14 +604,8 @@ static void as_antic_timeout(unsigned lo
 	spin_lock_irqsave(q->queue_lock, flags);
 	if (ad->antic_status == ANTIC_WAIT_REQ
 			|| ad->antic_status == ANTIC_WAIT_NEXT) {
-		struct as_io_context *aic;
 		ad->antic_status = ANTIC_FINISHED;
 		kblockd_schedule_work(&ad->antic_work);
-
-		aic = ad->io_context->aic;
-		if (aic->seek_samples == 0)
-			/* new process */
-			ad->new_success = (ad->new_success * 3) / 4;
 	}
 	spin_unlock_irqrestore(q->queue_lock, flags);
 }
@@ -708,11 +694,10 @@ static int as_can_break_anticipation(str
 		return 1;
 	}
 
-	if (ad->new_success < 256 &&
-			(aic->seek_samples == 0 || aic->ttime_samples == 0)) {
+	if (aic->seek_samples == 0 && aic->ttime_samples == 0) {
 		/*
-		 * Process has just started IO and we have a bad history of
-		 * success anticipating on new processes!
+		 * Process has just started IO. Don't anticipate.
+		 * TODO! Must fix this up.
 		 */
 		return 1;
 	}
@@ -1822,8 +1807,6 @@ static int as_init(request_queue_t *q, e
 	if (ad->write_batch_count < 2)
 		ad->write_batch_count = 2;
 
-	ad->new_success = 512;
-
 	return 0;
 }
 

_

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-21 13:05 rwhron
  2003-10-21 15:57 ` Nick Piggin
@ 2003-10-21 21:07 ` venom
  2003-10-22  9:54 ` Nick Piggin
  2 siblings, 0 replies; 25+ messages in thread
From: venom @ 2003-10-21 21:07 UTC (permalink / raw)
  To: rwhron; +Cc: piggin, linux-kernel, dmo, akpm


sql bench on mysql, una master, three slave, with old as-iosched.c
runs mutch better, thanx.


On Tue, 21 Oct 2003 rwhron@earthlink.net wrote:

> Date: Tue, 21 Oct 2003 09:05:01 -0400
> From: rwhron@earthlink.net
> To: piggin@cyberone.com.au, venom@sns.it
> Cc: linux-kernel@vger.kernel.org, dmo@osdl.org, akpm@osdl.org
> Subject: Re: [BENCHMARK] I/O regression after 2.6.0-test5
>
> > If you have time, would you please try testing as-iosched.c from
> > test5 in a later kernel (it won't go into test8-mm1 though).
>
> copying drivers/block/as-iosched.c from 2.6.0-test5 to test8
> looks like it fixes the regression.  Here are the results so far.
>
> In the AIM7 database benchmark, jobs/minute using test5 as-iosched.c
> doubled.  At 32 tasks, jobs/min went from 300 to 601.
> At 256 tasks, jobs/min went from 552 to 1010.
>
> test5 as-iosched.c:
> cat suite7-dbase-2.6.0-test8-as
> Benchmark       Version Machine Run Date
> AIM Multiuser Benchmark - Suite VII     "1.1"   dev4-003
>
> Tasks   Jobs/Min        Real    CPU
> 32      601.0           316.3   134.0
> 64      782.6           485.8   259.2
> 96      869.1           656.1   387.6
> 128     919.3           827.0   522.8
> 160     955.0           995.2   650.3
> 192     981.9           1161.5  790.9
> 224     1002.3          1327.5  910.2
> 256     1009.9          1505.7  1027.7
>
> Unmodififed test8.
> cat suite7-dbase-2.6.0-test8
> Benchmark       Version Machine Run Date
> AIM Multiuser Benchmark - Suite VII     "1.1"   dev4-003
>
> Tasks   Jobs/Min        Real    CPU
> 32      299.8           634.0   128.6
> 64      450.5           843.9   246.5
> 96      549.7           1037.4  366.0
> 128     558.7           1360.8  497.6
> 160     522.1           1820.2  612.2
> 192     515.5           2212.4  742.7
> 224     529.9           2510.8  859.5
> 256     552.4           2753.0  980.4
>
> The AIM7 fileserver benchmark is back up to previous
> levels when test8 uses the test5 as-iosched.c:
>
>
> test5 as-iosched.c with test8.
> cat suite7-fserver-2.6.0-test8-as
> Benchmark       Version Machine Run Date
> AIM Multiuser Benchmark - Suite VII     "1.1"   dev4-003
>
> Tasks   Jobs/Min        Real    CPU
> 4       80.5            301.0   37.0
> 8       133.0           364.6   64.0
> 12      166.7           436.3   93.4
> 16      175.9           551.2   111.5
> 20      186.3           650.7   133.8
> 24      202.8           717.3   161.2
> 28      207.1           819.3   200.0
> 32      210.6           920.7   226.5
>
>
> Unmodified test8.
> cat suite7-fserver-2.6.0-test8
> Benchmark       Version Machine Run Date
> AIM Multiuser Benchmark - Suite VII     "1.1"   dev4-003
>
> Tasks   Jobs/Min        Real    CPU
> 4       42.3            573.6   40.9
> 8       63.3            765.7   71.6
> 12      83.2            874.2   95.0
> 16      87.5            1108.4  102.5
> 20      105.0           1154.6  136.7
> 24      111.2           1308.1  162.2
> 28      130.7           1297.9  185.8
> 32      134.2           1444.5  212.9
>
>
> My tiobench test hasn't completed yet, but test8 with test5's
> as-iosched.c is making good progress.  I think Nick nailed the
> changeset.
>
> --
> Randy Hron
> http://home.earthlink.net/~rwhron/kernel/bigbox.html
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-21 13:05 rwhron
@ 2003-10-21 15:57 ` Nick Piggin
  2003-10-21 21:07 ` venom
  2003-10-22  9:54 ` Nick Piggin
  2 siblings, 0 replies; 25+ messages in thread
From: Nick Piggin @ 2003-10-21 15:57 UTC (permalink / raw)
  To: rwhron; +Cc: venom, linux-kernel, dmo, akpm



rwhron@earthlink.net wrote:

>>If you have time, would you please try testing as-iosched.c from
>>test5 in a later kernel (it won't go into test8-mm1 though).
>>
>
>copying drivers/block/as-iosched.c from 2.6.0-test5 to test8
>looks like it fixes the regression.  Here are the results so far.
>
>In the AIM7 database benchmark, jobs/minute using test5 as-iosched.c
>doubled.  At 32 tasks, jobs/min went from 300 to 601.
>At 256 tasks, jobs/min went from 552 to 1010.
>

Yep, looks right. Thanks for your thoroughness. There are a few
important direct IO bug fixes gone in there as well (hopefully
they aren't causing the regressions... unlikely), so for stability
you'll want to continue to use the test8 version.

I'll get a patch out soon.



^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
@ 2003-10-21 13:05 rwhron
  2003-10-21 15:57 ` Nick Piggin
                   ` (2 more replies)
  0 siblings, 3 replies; 25+ messages in thread
From: rwhron @ 2003-10-21 13:05 UTC (permalink / raw)
  To: piggin, venom; +Cc: linux-kernel, dmo, akpm

> If you have time, would you please try testing as-iosched.c from
> test5 in a later kernel (it won't go into test8-mm1 though).

copying drivers/block/as-iosched.c from 2.6.0-test5 to test8
looks like it fixes the regression.  Here are the results so far.

In the AIM7 database benchmark, jobs/minute using test5 as-iosched.c
doubled.  At 32 tasks, jobs/min went from 300 to 601.
At 256 tasks, jobs/min went from 552 to 1010.

test5 as-iosched.c:
cat suite7-dbase-2.6.0-test8-as
Benchmark       Version Machine Run Date
AIM Multiuser Benchmark - Suite VII     "1.1"   dev4-003

Tasks   Jobs/Min        Real    CPU     
32      601.0           316.3   134.0   
64      782.6           485.8   259.2   
96      869.1           656.1   387.6   
128     919.3           827.0   522.8   
160     955.0           995.2   650.3   
192     981.9           1161.5  790.9   
224     1002.3          1327.5  910.2   
256     1009.9          1505.7  1027.7  

Unmodififed test8.
cat suite7-dbase-2.6.0-test8
Benchmark       Version Machine Run Date
AIM Multiuser Benchmark - Suite VII     "1.1"   dev4-003

Tasks   Jobs/Min        Real    CPU     
32      299.8           634.0   128.6   
64      450.5           843.9   246.5   
96      549.7           1037.4  366.0   
128     558.7           1360.8  497.6   
160     522.1           1820.2  612.2   
192     515.5           2212.4  742.7   
224     529.9           2510.8  859.5   
256     552.4           2753.0  980.4   

The AIM7 fileserver benchmark is back up to previous
levels when test8 uses the test5 as-iosched.c:


test5 as-iosched.c with test8.
cat suite7-fserver-2.6.0-test8-as
Benchmark       Version Machine Run Date
AIM Multiuser Benchmark - Suite VII     "1.1"   dev4-003

Tasks   Jobs/Min        Real    CPU     
4       80.5            301.0   37.0    
8       133.0           364.6   64.0    
12      166.7           436.3   93.4    
16      175.9           551.2   111.5   
20      186.3           650.7   133.8   
24      202.8           717.3   161.2   
28      207.1           819.3   200.0   
32      210.6           920.7   226.5   


Unmodified test8.
cat suite7-fserver-2.6.0-test8
Benchmark       Version Machine Run Date
AIM Multiuser Benchmark - Suite VII     "1.1"   dev4-003

Tasks   Jobs/Min        Real    CPU     
4       42.3            573.6   40.9    
8       63.3            765.7   71.6    
12      83.2            874.2   95.0    
16      87.5            1108.4  102.5   
20      105.0           1154.6  136.7   
24      111.2           1308.1  162.2   
28      130.7           1297.9  185.8   
32      134.2           1444.5  212.9   


My tiobench test hasn't completed yet, but test8 with test5's
as-iosched.c is making good progress.  I think Nick nailed the
changeset.

-- 
Randy Hron
http://home.earthlink.net/~rwhron/kernel/bigbox.html


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-20  7:55   ` venom
@ 2003-10-21  4:49     ` Nick Piggin
  0 siblings, 0 replies; 25+ messages in thread
From: Nick Piggin @ 2003-10-21  4:49 UTC (permalink / raw)
  To: venom; +Cc: Dave Olien, rwhron, linux-kernel, akpm

Hi Guys,
If you have time, would you please try testing as-iosched.c from
test5 in a later kernel (it won't go into test8-mm1 though).

Thanks


venom@sns.it wrote:

>me too, on some self made db benchs on both mysql and postgresql,  testing, of
>course, I/O.
>
>On Sun, 19 Oct 2003, Dave Olien wrote:
>
>
>>Date: Sun, 19 Oct 2003 21:51:23 -0700
>>From: Dave Olien <dmo@osdl.org>
>>To: rwhron@earthlink.net
>>Cc: linux-kernel@vger.kernel.org, akpm@osdl.org
>>Subject: Re: [BENCHMARK] I/O regression after 2.6.0-test5
>>
>>
>>Yup, we've seen similar regression on tiobench and reaim workloads.
>>
>>On Sun, Oct 19, 2003 at 08:37:45PM -0400, rwhron@earthlink.net wrote:
>>
>>>There was about a 50% regression in jobs/minute in AIM7
>>>database workload on quad P3 Xeon.  The CPU time has not
>>>gone up, so the extra run time is coming from something
>>>else.  (I/O or I/O scheduler?)
>>>
>>>tiobench sequential reads has a significant regression too.
>>>
>>>Regression appears unrelated to filesystem type.
>>>
>>>dbench was not affected.
>>>
>>>The AIM7 was run on ext2.
>>>
>>>      
>>>


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-20  4:52   ` Andrew Morton
@ 2003-10-20  8:12     ` Nick Piggin
  0 siblings, 0 replies; 25+ messages in thread
From: Nick Piggin @ 2003-10-20  8:12 UTC (permalink / raw)
  To: Andrew Morton; +Cc: rwhron, linux-kernel



Andrew Morton wrote:

>Nick Piggin <piggin@cyberone.com.au> wrote:
>
>>
>>
>>rwhron@earthlink.net wrote:
>>
>>
>>>There was about a 50% regression in jobs/minute in AIM7
>>>database workload on quad P3 Xeon.  The CPU time has not
>>>gone up, so the extra run time is coming from something
>>>else.  (I/O or I/O scheduler?)
>>>
>>>tiobench sequential reads has a significant regression too.
>>>
>>>Regression appears unrelated to filesystem type.
>>>
>>>dbench was not affected.
>>>
>>>The AIM7 was run on ext2.
>>>
>>>
>>Yeah I'd say its all due to the IO scheduler. There is a problem
>>I'm thinking about how to fix - its the likely cause of this too.
>>
>>
>
>What change do you think it was due to?
>
>

I was thinking: [PATCH] fix AS crappy performance

(It still doesn't work properly)

>It's rather strange that test6 is slow but test6-mm is not: generally the
>IO scheduler regressions enter -mm first ;)
>

But if test6-mm isn't slow then maybe it is due to something else


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-20  4:51 ` Dave Olien
@ 2003-10-20  7:55   ` venom
  2003-10-21  4:49     ` Nick Piggin
  0 siblings, 1 reply; 25+ messages in thread
From: venom @ 2003-10-20  7:55 UTC (permalink / raw)
  To: Dave Olien; +Cc: rwhron, linux-kernel, akpm


me too, on some self made db benchs on both mysql and postgresql,  testing, of
course, I/O.

On Sun, 19 Oct 2003, Dave Olien wrote:

> Date: Sun, 19 Oct 2003 21:51:23 -0700
> From: Dave Olien <dmo@osdl.org>
> To: rwhron@earthlink.net
> Cc: linux-kernel@vger.kernel.org, akpm@osdl.org
> Subject: Re: [BENCHMARK] I/O regression after 2.6.0-test5
>
>
> Yup, we've seen similar regression on tiobench and reaim workloads.
>
> On Sun, Oct 19, 2003 at 08:37:45PM -0400, rwhron@earthlink.net wrote:
> > There was about a 50% regression in jobs/minute in AIM7
> > database workload on quad P3 Xeon.  The CPU time has not
> > gone up, so the extra run time is coming from something
> > else.  (I/O or I/O scheduler?)
> >
> > tiobench sequential reads has a significant regression too.
> >
> > Regression appears unrelated to filesystem type.
> >
> > dbench was not affected.
> >
> > The AIM7 was run on ext2.
> >
> > AIM7 dbase workload
> > kernel                   Tasks   Jobs/Min       Real    CPU
> > 2.6.0-test5-mm2          32	609.3	       311.9	139.5
> > 2.6.0-test5-mm4          32	595.2	       319.4	131.7
> > 2.6.0-test6-mm4          32	590.8	       321.8	130.7
> > 2.6.0-test5              32	590.2	       322.0	131.9
> > 2.6.0-test6              32	320.6	       592.9	129.1
> > 2.6.0-test7              32	290.3	       654.8	120.9
> >
> > 2.6.0-test5-mm2          64	786.2	       483.5	253.2
> > 2.6.0-test6-mm4          64	779.4	       487.8	253.8
> > 2.6.0-test5              64	774.3	       490.9	256.6
> > 2.6.0-test5-mm4          64	772.0	       492.4	253.7
> > 2.6.0-test6              64	430.5	       883.1	247.6
> > 2.6.0-test7              64	402.6	       944.2	231.5
> >
> > 2.6.0-test5-mm2          96	869.2	       656.0	382.2
> > 2.6.0-test6-mm4          96	867.6	       657.2	387.4
> > 2.6.0-test5-mm4          96	863.1	       660.7	385.2
> > 2.6.0-test5              96	859.0	       663.9	389.5
> > 2.6.0-test6              96	478.5	       1191.7	368.1
> > 2.6.0-test7              96	453.7	       1256.9	346.8
> >
> > 2.6.0-test5-mm2          128	929.4	       818.0	526.1
> > 2.6.0-test6-mm4          128	922.6	       824.1	523.3
> > 2.6.0-test5-mm4          128	916.6	       829.5	509.1
> > 2.6.0-test5              128	910.3	       835.2	528.6
> > 2.6.0-test7              128	500.5	       1519.1	461.3
> > 2.6.0-test6              128	500.2	       1520.1	497.4
> >
> > 2.6.0-test5-mm2          160	962.5	       987.4	674.6
> > 2.6.0-test5-mm4          160	952.7	       997.6	655.5
> > 2.6.0-test6-mm4          160	950.0	       1000.5	659.5
> > 2.6.0-test5              160	741.5	       1281.7	644.8
> > 2.6.0-test7              160	518.5	       1833.1	589.6
> > 2.6.0-test6              160	479.6	       1981.8	618.9
> >
> > 2.6.0-test5-mm2          192	993.9	       1147.5	810.0
> > 2.6.0-test5-mm4          192	982.7	       1160.5	788.2
> > 2.6.0-test6-mm4          192	978.8	       1165.2	808.4
> > 2.6.0-test5              192	840.0	       1357.8	776.0
> > 2.6.0-test7              192	537.9	       2120.2	709.3
> > 2.6.0-test6              192	494.3	       2307.1	748.2
> >
> > 2.6.0-test5-mm2          224	1011.2	       1315.8	943.5
> > 2.6.0-test5-mm4          224	1004.7	       1324.3	942.1
> > 2.6.0-test6-mm4          224	999.5	       1331.2	920.0
> > 2.6.0-test5              224	986.9	       1348.2	911.8
> > 2.6.0-test7              224	546.7	       2433.8	832.8
> > 2.6.0-test6              224	482.9	       2755.1	862.1
> >
> > 2.6.0-test5-mm2          256	1030.9	       1475.1	1076.5
> > 2.6.0-test5-mm4          256	1027.1	       1480.6	1071.4
> > 2.6.0-test6-mm4          256	1010.3	       1505.2	1074.8
> > 2.6.0-test5              256	999.5	       1521.3	1029.3
> > 2.6.0-test7              256	556.6	       2731.9	951.0
> > 2.6.0-test6              256	545.1	       2789.5	978.4
> >
> >
> > Similarly, AIM7 fileserver benchmark had a drop in throughput.
> > CPU time is down, but total runtime is way up.
> >
> > AIM7 fserver workload
> > kernel                   Tasks   Jobs/Min       Real    CPU
> > 2.6.0-test5              4	79.7	       304.0	38.3
> > 2.6.0-test5-mm2          4	77.4	       313.4	38.3
> > 2.6.0-test5-mm4          4	76.5	       317.1	36.7
> > 2.6.0-test6-mm4          4	65.5	       369.9	34.5
> > 2.6.0-test7              4	45.5	       532.5	32.2
> > 2.6.0-test6              4	45.0	       538.7	31.0
> >
> > 2.6.0-test5              8	129.8	       373.5	66.3
> > 2.6.0-test5-mm2          8	103.1	       470.4	62.0
> > 2.6.0-test5-mm4          8	102.2	       474.2	61.8
> > 2.6.0-test6              8	82.7	       585.9	84.5
> > 2.6.0-test6-mm4          8	82.4	       588.1	61.1
> > 2.6.0-test7              8	61.0	       795.0	59.9
> >
> > 2.6.0-test5              12	164.1	       443.0	93.1
> > 2.6.0-test5-mm2          12	112.4	       646.8	98.4
> > 2.6.0-test5-mm4          12	111.2	       653.8	96.7
> > 2.6.0-test6-mm4          12	95.2	       763.6	96.9
> > 2.6.0-test6              12	87.9	       827.6	118.9
> > 2.6.0-test7              12	54.5	       1335.0	91.8
> >
> > 2.6.0-test5              16	173.4	       559.0	116.5
> > 2.6.0-test5-mm2          16	128.8	       752.9	119.9
> > 2.6.0-test5-mm4          16	125.4	       773.1	124.3
> > 2.6.0-test6-mm4          16	104.2	       930.3	118.7
> > 2.6.0-test6              16	101.4	       956.0	107.4
> > 2.6.0-test7              16	70.6	       1372.6	112.2
> >
> > 2.6.0-test5              20	180.5	       671.5	143.0
> > 2.6.0-test5-mm4          20	144.9	       836.4	143.0
> > 2.6.0-test5-mm2          20	141.7	       855.3	144.1
> > 2.6.0-test6              20	117.7	       1030.0	131.1
> > 2.6.0-test6-mm4          20	114.4	       1059.5	143.0
> > 2.6.0-test7              20	78.7	       1539.9	134.0
> >
> > 2.6.0-test5              24	189.1	       769.0	176.0
> > 2.6.0-test5-mm2          24	156.2	       930.9	180.2
> > 2.6.0-test5-mm4          24	154.5	       941.6	176.9
> > 2.6.0-test6              24	134.5	       1081.1	156.5
> > 2.6.0-test6-mm4          24	124.5	       1167.8	170.2
> > 2.6.0-test7              24	76.9	       1892.1	169.0
> >
> > 2.6.0-test5              28	204.6	       829.2	197.5
> > 2.6.0-test5-mm2          28	158.7	       1069.2	209.8
> > 2.6.0-test5-mm4          28	156.0	       1087.7	211.5
> > 2.6.0-test6-mm4          28	134.2	       1264.0	196.9
> > 2.6.0-test6              28	133.7	       1268.7	174.7
> > 2.6.0-test7              28	84.1	       2016.4	188.5
> >
> > 2.6.0-test5              32	213.7	       907.4	228.9
> > 2.6.0-test5-mm2          32	151.9	       1276.7	239.7
> > 2.6.0-test5-mm4          32	151.5	       1279.6	241.4
> > 2.6.0-test6              32	147.0	       1318.9	197.0
> > 2.6.0-test6-mm4          32	138.8	       1397.6	225.4
> > 2.6.0-test7              32	88.5	       2190.3	214.9
> >
> > Same story for the shared workload.  Jobs/minute is down a
> > lot since 2.6.0-test5.
> >
> >
> > AIM7 shared workload
> > kernel                   Tasks   Jobs/Min       Real    CPU
> > 2.6.0-test5              64	1888.8	       197.2	168.3
> > 2.6.0-test5-mm2          64	1517.6	       245.4	177.0
> > 2.6.0-test5-mm4          64	1488.6	       250.2	180.5
> > 2.6.0-test6-mm4          64	1239.0	       300.6	175.0
> > 2.6.0-test7              64	1225.1	       304.0	166.6
> > 2.6.0-test6              64	880.0	       423.3	168.1
> >
> > 2.6.0-test5              128	2094.9	       355.6	338.1
> > 2.6.0-test5-mm2          128	1837.4	       405.4	352.4
> > 2.6.0-test5-mm4          128	1826.6	       407.8	358.4
> > 2.6.0-test6-mm4          128	1561.8	       477.0	346.6
> > 2.6.0-test7              128	1176.5	       633.2	332.6
> > 2.6.0-test6              128	1142.1	       652.3	332.6
> >
> > 2.6.0-test5              192	2256.0	       495.3	507.8
> > 2.6.0-test5-mm2          192	2074.9	       538.5	530.9
> > 2.6.0-test5-mm4          192	2044.8	       546.5	536.9
> > 2.6.0-test6-mm4          192	1711.8	       652.8	527.3
> > 2.6.0-test6              192	1328.3	       841.3	505.6
> > 2.6.0-test7              192	1317.9	       847.9	499.1
> >
> > 2.6.0-test5              256	2389.0	       623.7	683.8
> > 2.6.0-test5-mm2          256	2207.2	       675.0	720.6
> > 2.6.0-test5-mm4          256	2186.2	       681.5	729.9
> > 2.6.0-test6-mm4          256	1780.7	       836.7	719.0
> > 2.6.0-test6              256	1479.5	       1007.0	672.9
> > 2.6.0-test7              256	1282.0	       1162.2	672.8
> >
> > 2.6.0-test5              320	2457.9	       757.7	856.6
> > 2.6.0-test5-mm4          320	2330.2	       799.2	920.0
> > 2.6.0-test5-mm2          320	2327.9	       800.0	919.4
> > 2.6.0-test6-mm4          320	1864.7	       998.8	900.9
> > 2.6.0-test6              320	1575.9	       1181.8	853.5
> > 2.6.0-test7              320	1317.9	       1413.1	844.4
> >
> > 2.6.0-test5              384	2480.4	       901.0	1036.5
> > 2.6.0-test5-mm2          384	2440.2	       915.9	1111.6
> > 2.6.0-test5-mm4          384	2434.2	       918.1	1111.0
> > 2.6.0-test6-mm4          384	1883.9	       1186.3	1091.7
> > 2.6.0-test6              384	1629.8	       1371.3	1026.3
> > 2.6.0-test7              384	1326.3	       1685.0	1018.6
> >
> > 2.6.0-test5              448	2529.8	       1030.7	1209.2
> > 2.6.0-test5-mm4          448	2511.6	       1038.1	1306.8
> > 2.6.0-test5-mm2          448	2510.0	       1038.8	1306.1
> > 2.6.0-test6-mm4          448	1934.3	       1347.9	1283.6
> > 2.6.0-test6              448	1612.3	       1617.1	1203.3
> > 2.6.0-test7              448	1344.7	       1939.0	1200.9
> >
> > 2.6.0-test5-mm2          512	2579.1	       1155.4	1504.0
> > 2.6.0-test5              512	2566.2	       1161.2	1386.2
> > 2.6.0-test5-mm4          512	2552.9	       1167.2	1509.2
> > 2.6.0-test6-mm4          512	1921.5	       1550.8	1475.1
> > 2.6.0-test6              512	1538.6	       1936.7	1376.9
> > 2.6.0-test7              512	1370.5	       2174.3	1379.2
> >
> > tiobench read performance in test[67] has a big regression in throughput
> > as the number of threads goes up.  The max and long latency numbers get
> > bad too.  It appears to affect ext2, ext3, reiserfs, jfs and xfs.
> > Regression is in 2.6.0-test[67] and test6-mm4.
> >
> > tiobench-0.3.3
> > Unit information
> > ================
> > File size = 8192 megabytes
> > Blk Size  = 4096 bytes
> > Rate      = megabytes per second
> > CPU%      = percentage of CPU used during the test
> > Latency   = milliseconds
> > Lat%      = percent of requests that took longer than X seconds
> > CPU Eff   = Rate divided by CPU% - throughput per cpu load
> >
> > Sequential Reads ext2
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1   28.55 13.89%     0.408      754.59  0.00000
> > 2.6.0-test5-mm2              1   28.66 13.98%     0.406      742.76  0.00000
> > 2.6.0-test5-mm4              1   28.46 13.90%     0.409      887.02  0.00000
> > 2.6.0-test6                  1   28.47 13.81%     0.409     1069.74  0.00000
> > 2.6.0-test6-mm4              1   28.41 13.89%     0.410      723.14  0.00000
> > 2.6.0-test7                  1   28.71 13.92%     0.405      715.02  0.00000
> >
> > With one thread, performance is similar.
> >
> > 2.6.0-test5                  8   34.01 17.09%     2.620      834.85  0.00000
> > 2.6.0-test5-mm2              8   32.44 16.44%     2.765      844.99  0.00000
> > 2.6.0-test5-mm4              8   30.75 15.55%     2.967      952.18  0.00000
> > 2.6.0-test6                  8   20.67  9.90%     4.477     1569.30  0.00000
> > 2.6.0-test6-mm4              8   21.55 10.31%     4.260     1790.19  0.00000
> > 2.6.0-test7                  8   21.46 10.24%     4.312     1913.24  0.00000
> >
> > 2.6.0-test5                 16   30.13 15.04%     5.980     1337.69  0.00000
> > 2.6.0-test5-mm2             16   29.71 14.97%     6.024     1115.95  0.00000
> > 2.6.0-test5-mm4             16   27.10 13.65%     6.606     1008.67  0.00000
> > 2.6.0-test6                 16   16.44  7.90%    11.201     2820.85  0.00000
> > 2.6.0-test6-mm4             16   16.14  7.78%    11.382     2753.13  0.00000
> > 2.6.0-test7                 16   18.34  8.64%     9.877     2742.49  0.00000
> >
> > 2.6.0-test5                 32   29.46 14.62%    11.954     1560.02  0.00000
> > 2.6.0-test5-mm2             32   28.01 14.26%    12.808     1627.00  0.00000
> > 2.6.0-test5-mm4             32   28.78 14.63%    12.182     1859.56  0.00000
> > 2.6.0-test6                 32   15.60  7.55%    23.415     4877.75  0.00000
> > 2.6.0-test6-mm4             32   14.63  7.20%    24.713     4846.63  0.00000
> > 2.6.0-test7                 32   14.79  7.14%    24.234     4855.64  0.00000
> >
> > 2.6.0-test5                 64   25.24 13.57%    26.901     1743.05  0.00000
> > 2.6.0-test5-mm2             64   24.01 13.27%    27.922     1849.82  0.00000
> > 2.6.0-test5-mm4             64   23.67 13.15%    28.513     1956.58  0.00000
> > 2.6.0-test6                 64    9.45  5.24%    75.491     9578.85  2.75073
> > 2.6.0-test6-mm4             64    8.51  4.82%    84.118    13166.74  3.11647
> > 2.6.0-test7                 64    8.58  4.86%    83.430    13622.86  3.05581
> >
> > 2.6.0-test5                128   10.41  9.76%   129.662     1820.78  0.00000
> > 2.6.0-test5-mm2            128   11.00 10.44%   120.118     1742.44  0.00000
> > 2.6.0-test5-mm4            128   10.67 10.15%   122.871     1912.79  0.00000
> > 2.6.0-test6                128    2.97  2.66%   472.251    25694.03  8.39324
> > 2.6.0-test6-mm4            128    2.94  2.66%   473.943    25530.14  8.39028
> > 2.6.0-test7                128    2.94  2.68%   480.175    26306.38  8.39257
> >
> > At 256 threads, throughput is down and max latency skyrockets.
> >
> > 2.6.0-test5                256    9.13 13.30%   305.097     1446.43  0.00000
> > 2.6.0-test5-mm2            256    9.09 13.60%   304.019     1648.80  0.00000
> > 2.6.0-test5-mm4            256    8.63 13.66%   325.793     1542.62  0.00000
> > 2.6.0-test6                256    2.71  2.49%   937.603   529116.60  8.94007
> > 2.6.0-test6-mm4            256    2.62  2.46%   979.221   785760.77  9.12519
> > 2.6.0-test7                256    2.56  2.39%   988.719   582067.86  9.28287
> >
> > Random Reads ext2
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1    0.95  0.94%    12.373      114.89  0.00000
> > 2.6.0-test5-mm2              1    0.90  0.88%    13.061      113.19  0.00000
> > 2.6.0-test5-mm4              1    0.53  0.60%    21.903      151.35  0.00000
> > 2.6.0-test6                  1    0.73  0.82%    16.053      391.29  0.00000
> > 2.6.0-test6-mm4              1    0.76  0.81%    15.388      126.51  0.00000
> > 2.6.0-test7                  1    0.96  0.94%    12.192      101.93  0.00000
> >
> > 2.6.0-test5                  8    4.11  4.58%    19.433      137.74  0.00000
> > 2.6.0-test5-mm2              8    3.94  4.03%    20.931      144.16  0.00000
> > 2.6.0-test5-mm4              8    2.81  3.28%    30.086      222.27  0.00000
> > 2.6.0-test6                  8    3.25  3.21%    27.598      640.41  0.00000
> > 2.6.0-test6-mm4              8    3.98  3.82%    22.078      324.87  0.00000
> > 2.6.0-test7                  8    3.33  3.27%    27.069      417.17  0.00000
> >
> > 2.6.0-test5                 16    4.05  3.90%    42.424      236.49  0.00000
> > 2.6.0-test5-mm2             16    4.09  4.31%    41.651      255.17  0.00000
> > 2.6.0-test5-mm4             16    3.05  3.19%    55.847      332.10  0.00000
> > 2.6.0-test6                 16    3.43  3.28%    48.836      934.02  0.00000
> > 2.6.0-test6-mm4             16    3.21  2.69%    52.841      936.50  0.00000
> > 2.6.0-test7                 16    4.93  4.17%    34.720      630.63  0.00000
> >
> > 2.6.0-test5                 32    4.41  3.58%    70.910      516.28  0.00000
> > 2.6.0-test5-mm2             32    4.23  4.17%    78.492      372.63  0.00000
> > 2.6.0-test5-mm4             32    4.19  3.90%    76.001      472.90  0.00000
> > 2.6.0-test6                 32    3.74  3.20%    92.407     1578.33  0.00000
> > 2.6.0-test6-mm4             32    2.94  2.97%   116.663     1722.23  0.00000
> > 2.6.0-test7                 32    2.62  2.36%   129.093     2248.32  0.00000
> >
> > 2.6.0-test5                 64    4.47  4.00%   137.277     1077.53  0.00000
> > 2.6.0-test5-mm2             64    4.79  4.53%   124.035     1144.38  0.00000
> > 2.6.0-test5-mm4             64    4.13  3.81%   143.321     1265.44  0.00000
> > 2.6.0-test6                 64    3.89  3.62%   166.555     2679.86  0.00000
> > 2.6.0-test6-mm4             64    3.23  3.15%   200.698     3160.57  0.00000
> > 2.6.0-test7                 64    3.14  2.74%   206.849     3384.30  0.00000
> >
> > 2.6.0-test5                128    4.86  4.83%   243.704     2287.09  0.00000
> > 2.6.0-test5-mm2            128    5.08  4.85%   228.102     2391.21  0.00000
> > 2.6.0-test5-mm4            128    4.16  4.65%   270.840     2501.04  0.00000
> > 2.6.0-test6                128    3.19  3.09%   389.583     6092.04  0.12601
> > 2.6.0-test6-mm4            128    2.75  2.78%   439.379     6632.53  0.65524
> > 2.6.0-test7                128    3.16  3.16%   388.762     5810.95  0.07560
> >
> > 2.6.0-test5                256    5.51  7.00%   427.503     2511.79  0.00000
> > 2.6.0-test5-mm2            256    5.23  7.39%   449.007     3169.28  0.00000
> > 2.6.0-test5-mm4            256    4.48  6.76%   526.994     3618.17  0.00000
> > 2.6.0-test6                256    2.01  2.44%  1156.505    14117.02 16.82291
> > 2.6.0-test6-mm4            256    1.99  2.38%  1169.942    13903.51 16.56250
> > 2.6.0-test7                256    2.09  2.41%  1133.624    13195.27 14.97396
> >
> > Sequential and random writes on ext2 are not regressed.
> >
> > ext3 has a regression similar to ext2 for sequential and random reads.
> >
> > Sequential Reads ext3
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1   29.03 14.00%     0.401      745.26  0.00000
> > 2.6.0-test5-mm2              1   28.05 13.82%     0.415      763.49  0.00000
> > 2.6.0-test5-mm4              1   29.31 14.10%     0.397      867.89  0.00000
> > 2.6.0-test6                  1   28.92 13.83%     0.402      747.59  0.00000
> > 2.6.0-test6-mm4              1   28.01 13.67%     0.415     1162.40  0.00000
> > 2.6.0-test7                  1   27.95 13.54%     0.416      860.38  0.00000
> >
> > 2.6.0-test5                  8   14.53  8.42%     6.187     1280.16  0.00000
> > 2.6.0-test5-mm2              8   14.25  8.56%     6.428     1438.12  0.00000
> > 2.6.0-test5-mm4              8   14.42  8.60%     6.317     1335.28  0.00000
> > 2.6.0-test6                  8   10.37  6.00%     8.791     2065.27  0.00000
> > 2.6.0-test6-mm4              8    9.38  5.81%     9.765     1978.67  0.00000
> > 2.6.0-test7                  8   10.25  6.02%     8.931     1977.71  0.00000
> >
> > 2.6.0-test5                 16   11.38  7.09%    15.604     1387.68  0.00000
> > 2.6.0-test5-mm2             16   12.84  7.97%    14.206     1246.21  0.00000
> > 2.6.0-test5-mm4             16   12.89  8.00%    14.050     1790.63  0.00000
> > 2.6.0-test6                 16    5.13  3.45%    34.925     3982.64  0.00000
> > 2.6.0-test6-mm4             16    4.91  3.42%    37.057     4349.95  0.00000
> > 2.6.0-test7                 16    5.27  3.56%    34.774     3852.71  0.00000
> >
> > 2.6.0-test5                 32    8.77  6.17%    41.461     1856.08  0.00000
> > 2.6.0-test5-mm2             32    8.79  6.26%    40.822     1667.85  0.00000
> > 2.6.0-test5-mm4             32    8.74  6.41%    41.519     1797.28  0.00000
> > 2.6.0-test6                 32    3.32  2.60%   107.252     9267.00  0.00649
> > 2.6.0-test6-mm4             32    3.58  2.78%   100.364     8453.15  0.00496
> > 2.6.0-test7                 32    3.39  2.65%   104.283     9029.86  0.00367
> >
> > 2.6.0-test5                 64    6.29  5.30%   113.016     1947.14  0.00000
> > 2.6.0-test5-mm2             64    7.58  6.00%    81.717     1722.83  0.00000
> > 2.6.0-test5-mm4             64    7.63  6.07%    79.882     1709.56  0.00000
> > 2.6.0-test6                 64    2.63  2.25%   265.280    18736.77  8.76222
> > 2.6.0-test6-mm4             64    2.43  2.17%   277.430    17986.61  8.95066
> > 2.6.0-test7                 64    2.40  2.16%   286.947    17890.59  9.59925
> >
> > 2.6.0-test5                128    6.19  6.61%   216.851     1852.30  0.00000
> > 2.6.0-test5-mm2            128    6.73  7.19%   191.761     2049.82  0.00000
> > 2.6.0-test5-mm4            128    6.09  6.55%   209.661     1910.36  0.00000
> > 2.6.0-test6                128    2.06  1.95%   640.506    32660.27 12.41284
> > 2.6.0-test6-mm4            128    2.03  1.98%   645.423    30932.33 12.57601
> > 2.6.0-test7                128    2.05  2.02%   639.791    30885.16 12.72559
> >
> > 2.6.0-test5                256    6.15  9.88%   457.029     1880.47  0.00000
> > 2.6.0-test5-mm2            256    5.51  9.06%   518.943     2002.06  0.00000
> > 2.6.0-test5-mm4            256    5.30  8.81%   535.291     2103.11  0.00000
> > 2.6.0-test6                256    1.85  1.78%  1293.732   230380.55 13.73657
> > 2.6.0-test6-mm4            256    1.91  1.90%  1246.640   243088.10 13.26160
> > 2.6.0-test7                256    1.81  1.78%  1351.071   281394.59 13.80543
> >
> > Random Reads ext3
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1    0.90  0.85%    12.963      112.10  0.00000
> > 2.6.0-test5-mm2              1    0.93  0.97%    12.637      125.35  0.00000
> > 2.6.0-test5-mm4              1    0.85  0.95%    13.714      122.66  0.00000
> > 2.6.0-test6                  1    0.73  0.71%    16.054      516.87  0.00000
> > 2.6.0-test6-mm4              1    0.91  0.91%    12.907      125.50  0.00000
> > 2.6.0-test7                  1    0.94  0.82%    12.443      110.45  0.00000
> >
> > 2.6.0-test5                  8    3.78  4.24%    21.477      131.88  0.00000
> > 2.6.0-test5-mm2              8    4.01  4.49%    20.218      124.38  0.00000
> > 2.6.0-test5-mm4              8    4.43  4.63%    18.684      142.49  0.00000
> > 2.6.0-test6                  8    3.55  3.11%    23.596      362.51  0.00000
> > 2.6.0-test6-mm4              8    2.52  2.69%    32.927      534.18  0.00000
> > 2.6.0-test7                  8    4.21  3.80%    19.881      459.00  0.00000
> >
> > 2.6.0-test5                 16    4.10  3.74%    37.249      236.48  0.00000
> > 2.6.0-test5-mm2             16    4.42  4.48%    37.848      208.96  0.00000
> > 2.6.0-test5-mm4             16    4.25  4.51%    36.874      220.75  0.00000
> > 2.6.0-test6                 16    3.27  3.13%    49.687      803.39  0.00000
> > 2.6.0-test6-mm4             16    3.37  3.17%    49.112      951.18  0.00000
> > 2.6.0-test7                 16    4.49  3.87%    38.468      727.87  0.00000
> >
> > 2.6.0-test5                 32    4.28  4.06%    76.377      458.79  0.00000
> > 2.6.0-test5-mm2             32    4.53  4.40%    71.762      466.83  0.00000
> > 2.6.0-test5-mm4             32    4.48  4.23%    70.711      361.46  0.00000
> > 2.6.0-test6                 32    3.35  3.06%    98.621     1728.18  0.00000
> > 2.6.0-test6-mm4             32    4.13  3.54%    77.004     1321.89  0.00000
> > 2.6.0-test7                 32    3.95  3.30%    79.804     1402.47  0.00000
> >
> > 2.6.0-test5                 64    4.89  4.80%   134.964     1243.33  0.00000
> > 2.6.0-test5-mm2             64    4.63  4.17%   108.944     1006.23  0.00000
> > 2.6.0-test5-mm4             64    3.70  3.92%   132.534     1222.82  0.00000
> > 2.6.0-test6                 64    3.35  2.98%   189.642     2847.96  0.00000
> > 2.6.0-test6-mm4             64    3.12  3.03%   207.658     3362.27  0.00000
> > 2.6.0-test7                 64    3.02  2.89%   212.838     2652.28  0.00000
> >
> > 2.6.0-test5                128    5.26  5.11%   230.823     2088.49  0.00000
> > 2.6.0-test5-mm2            128    4.77  4.65%   252.354     2283.06  0.00000
> > 2.6.0-test5-mm4            128    4.07  4.26%   280.692     2923.29  0.00000
> > 2.6.0-test6                128    2.88  3.07%   431.937     5967.29  0.12601
> > 2.6.0-test6-mm4            128    3.09  3.26%   403.965     5559.31  0.02520
> > 2.6.0-test7                128    3.01  2.95%   407.900     6243.88  0.10081
> >
> > 2.6.0-test5                256    5.18  6.62%   463.528     3270.79  0.00000
> > 2.6.0-test5-mm2            256    5.20  7.12%   452.987     3055.03  0.00000
> > 2.6.0-test5-mm4            256    4.36  5.89%   539.848     3465.46  0.00000
> > 2.6.0-test6                256    2.26  2.92%  1013.076    11495.98 12.47396
> > 2.6.0-test6-mm4            256    2.49  2.98%   943.926    10902.18 10.72917
> > 2.6.0-test7                256    2.43  3.04%   996.420    11016.00 12.08334
> >
> > Sequential Writes ext3
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1   56.15 64.55%     0.170     5700.14  0.00010
> > 2.6.0-test5-mm2              1   56.14 67.38%     0.173     4383.06  0.00005
> > 2.6.0-test5-mm4              1   55.64 66.40%     0.173     4323.48  0.00000
> > 2.6.0-test6                  1   56.15 65.22%     0.171     4578.10  0.00005
> > 2.6.0-test6-mm4              1   55.81 66.36%     0.172     5623.30  0.00010
> > 2.6.0-test7                  1   55.94 65.10%     0.171     4544.88  0.00000
> >
> > 2.6.0-test5                  8    2.99  4.94%    23.859  1119399.58  0.08983
> > 2.6.0-test5-mm2              8    3.00  5.08%    24.803   613704.08  0.09155
> > 2.6.0-test5-mm4              8    2.98  5.06%    25.599   515385.61  0.09627
> > 2.6.0-test6                  8    4.65  7.53%    15.776   347393.72  0.07853
> > 2.6.0-test6-mm4              8    3.02  5.04%    25.421   399480.58  0.09512
> > 2.6.0-test7                  8    3.12  5.12%    23.991   197721.86  0.09255
> >
> > 2.6.0-test5                 16    2.58  4.55%    55.284   715963.30  0.14072
> > 2.6.0-test5-mm2             16    2.93  5.09%    50.697   648332.45  0.11883
> > 2.6.0-test5-mm4             16    3.21  5.62%    47.072   544422.32  0.12240
> > 2.6.0-test6                 16    3.45  6.12%    39.175   424983.10  0.11596
> > 2.6.0-test6-mm4             16    2.72  4.76%    54.117   429887.30  0.13976
> > 2.6.0-test7                 16    2.74  4.73%    52.594   886672.08  0.13404
> >
> > 2.6.0-test5                 32    2.28  4.43%   127.746   833586.10  0.20012
> > 2.6.0-test5-mm2             32    2.25  4.58%   124.219  1054925.33  0.20647
> > 2.6.0-test5-mm4             32    2.11  4.33%   139.662   787258.98  0.22011
> > 2.6.0-test6                 32    2.80  5.35%   107.223   835055.83  0.18424
> > 2.6.0-test6-mm4             32    2.28  4.26%   117.722   754407.32  0.18582
> > 2.6.0-test7                 32    2.21  4.38%   128.475   829790.72  0.21805
> >
> > 2.6.0-test5                 64    2.02  4.54%   265.546  1322591.44  0.40007
> > 2.6.0-test5-mm2             64    2.02  4.42%   278.978  1434418.75  0.42186
> > 2.6.0-test5-mm4             64    2.25  4.83%   242.598  1592337.06  0.35062
> > 2.6.0-test6                 64    2.44  5.25%   222.436  1233112.15  0.38619
> > 2.6.0-test6-mm4             64    1.98  4.45%   274.091  1442328.98  0.45623
> > 2.6.0-test7                 64    2.09  4.37%   249.893  1456365.96  0.30308
> >
> > 2.6.0-test5                128    2.21  4.96%   444.935  2758997.12  0.60168
> > 2.6.0-test5-mm2            128    1.93  4.62%   568.801  3209338.67  0.84395
> > 2.6.0-test5-mm4            128    1.99  4.67%   518.813  2778367.36  0.77595
> > 2.6.0-test6                128    2.00  4.47%   497.636  2933240.88  0.65131
> > 2.6.0-test6-mm4            128    2.03  4.71%   505.696  2839390.84  0.74044
> > 2.6.0-test7                128    2.00  4.69%   491.950  2994053.88  0.75612
> >
> > 2.6.0-test5                256    2.11  7.77%   941.664  4873713.39  1.83378
> > 2.6.0-test5-mm2            256    1.91  6.85%  1093.820  5897453.70  1.80015
> > 2.6.0-test5-mm4            256    1.97  6.74%  1017.367  5695863.45  1.77222
> > 2.6.0-test6                256    1.92  6.71%  1065.957  6104195.68  1.87802
> > 2.6.0-test6-mm4            256    1.93  7.02%  1038.493  5678570.32  1.92114
> > 2.6.0-test7                256    1.85  7.36%  1083.836  5720266.70  1.89524
> >
> > Random Writes ext3
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1    3.39  3.44%     0.399       37.37  0.00000
> > 2.6.0-test5-mm2              1    3.22  3.38%     0.516       40.33  0.00000
> > 2.6.0-test5-mm4              1    3.56  3.72%     0.081       10.61  0.00000
> > 2.6.0-test6                  1    2.34  2.39%     1.780       24.35  0.00000
> > 2.6.0-test6-mm4              1    3.00  3.19%     0.746       37.62  0.00000
> > 2.6.0-test7                  1    3.05  3.06%     0.653       54.78  0.00000
> >
> > 2.6.0-test5                  8    3.17  3.59%    13.525     1655.65  0.00000
> > 2.6.0-test5-mm2              8    3.05  3.44%    14.349     2391.37  0.00000
> > 2.6.0-test5-mm4              8    3.08  3.54%    14.093     1401.73  0.00000
> > 2.6.0-test6                  8    2.79  3.15%    14.906     3957.84  0.02500
> > 2.6.0-test6-mm4              8    2.88  3.06%    15.029     1283.97  0.00000
> > 2.6.0-test7                  8    3.18  3.54%    14.407     1671.73  0.00000
> >
> > 2.6.0-test5                 16    3.18  3.49%    26.252     1693.61  0.00000
> > 2.6.0-test5-mm2             16    3.28  3.73%    25.573     3583.93  0.02500
> > 2.6.0-test5-mm4             16    3.21  3.84%    25.857     6012.15  0.02500
> > 2.6.0-test6                 16    2.81  2.81%    30.448     2511.41  0.00000
> > 2.6.0-test6-mm4             16    3.17  4.08%    20.815     5062.41  0.02500
> > 2.6.0-test7                 16    3.14  3.43%    28.987     2120.35  0.00000
> >
> > 2.6.0-test5                 32    3.26  4.05%    46.822     3656.96  0.00000
> > 2.6.0-test5-mm2             32    3.25  4.28%    47.088     2834.74  0.00000
> > 2.6.0-test5-mm4             32    3.29  4.49%    44.765     2409.89  0.00000
> > 2.6.0-test6                 32    2.67  2.91%    62.319    11807.07  0.05000
> > 2.6.0-test6-mm4             32    2.98  3.57%    57.025     2250.13  0.00000
> > 2.6.0-test7                 32    3.06  3.75%    50.252     9420.48  0.05000
> >
> > 2.6.0-test5                 64    3.40  4.96%    46.634     2426.05  0.00000
> > 2.6.0-test5-mm2             64    3.24  4.72%    93.293     7875.05  1.86492
> > 2.6.0-test5-mm4             64    3.24  4.44%    79.998     5292.84  0.63004
> > 2.6.0-test6                 64    2.50  3.22%   137.484     6723.01  0.25202
> > 2.6.0-test6-mm4             64    2.83  3.92%    79.495     9204.46  0.68044
> > 2.6.0-test7                 64    2.85  4.00%    91.255     6050.08  0.63004
> >
> > 2.6.0-test5                128    3.46  5.89%   101.597     6086.33  0.80645
> > 2.6.0-test5-mm2            128    3.39  5.71%   129.468     6234.78  1.28528
> > 2.6.0-test5-mm4            128    3.27  5.82%   128.846     6601.87  0.65524
> > 2.6.0-test6                128    2.63  4.26%   123.641     6247.05  2.09173
> > 2.6.0-test6-mm4            128    2.69  4.58%   100.787     4669.66  0.00000
> > 2.6.0-test7                128    2.71  4.58%    91.826     4953.82  0.00000
> >
> > 2.6.0-test5                256    3.45  6.88%   210.999    10063.95  6.06770
> > 2.6.0-test5-mm2            256    3.45  6.80%   263.620     8774.35  8.25520
> > 2.6.0-test5-mm4            256    3.33  6.91%   209.373    10921.27  5.10416
> > 2.6.0-test6                256    2.58  5.45%   353.922     7824.45  5.31250
> > 2.6.0-test6-mm4            256    2.99  6.08%   152.068     6246.05  2.36979
> > 2.6.0-test7                256    2.96  6.15%    69.124     3923.92  0.00000
> >
> > Sequential Reads jfs
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> >
> > 2.6.0-test5                  1   28.93 13.16%     0.402      958.45  0.00000
> > 2.6.0-test5-mm2              1   28.84 13.33%     0.403      703.16  0.00000
> > 2.6.0-test5-mm4              1   29.03 13.50%     0.401      871.93  0.00000
> > 2.6.0-test6-mm4              1   28.90 13.54%     0.402      724.31  0.00000
> > 2.6.0-test7                  1   29.07 13.44%     0.400      677.09  0.00000
> >
> > 2.6.0-test5                  8   50.81 24.01%     1.815      942.91  0.00000
> > 2.6.0-test5-mm2              8   48.22 23.30%     1.918      959.38  0.00000
> > 2.6.0-test5-mm4              8   46.01 21.88%     1.988      840.79  0.00000
> > 2.6.0-test6-mm4              8   20.85  9.38%     4.417     2073.30  0.00000
> > 2.6.0-test7                  8   20.88  9.17%     4.402     1941.85  0.00000
> >
> > 2.6.0-test5                 16   37.26 17.05%     5.004      855.09  0.00000
> > 2.6.0-test5-mm2             16   40.27 18.83%     4.628     1024.08  0.00000
> > 2.6.0-test5-mm4             16   38.68 18.07%     4.831      820.64  0.00000
> > 2.6.0-test6-mm4             16   11.10  5.05%    16.392     3579.95  0.00000
> > 2.6.0-test7                 16   11.98  5.35%    15.258     3381.34  0.00000
> >
> > 2.6.0-test5                 32   35.48 16.56%    10.439     1742.70  0.00000
> > 2.6.0-test5-mm2             32   35.46 17.01%    10.292     1979.97  0.00000
> > 2.6.0-test5-mm4             32   33.58 15.89%    11.012     1885.26  0.00000
> > 2.6.0-test6-mm4             32   10.45  4.79%    34.665     6205.82  0.00743
> > 2.6.0-test7                 32   10.82  4.85%    33.369     6185.50  0.00668
> >
> > 2.6.0-test5                 64   36.58 17.71%    19.855     3297.14  0.00000
> > 2.6.0-test5-mm2             64   34.64 17.14%    21.114     3190.91  0.00000
> > 2.6.0-test5-mm4             64   34.98 17.66%    20.673     3254.24  0.00000
> > 2.6.0-test6-mm4             64   10.64  4.87%    67.501    11935.89  2.07191
> > 2.6.0-test7                 64   11.33  5.08%    61.240    11701.91  1.98450
> >
> > 2.6.0-test5                128   11.37  8.64%   109.981     2829.31  0.00000
> > 2.6.0-test5-mm2            128   17.63 11.71%    64.633     2799.09  0.00000
> > 2.6.0-test5-mm4            128   15.92 11.40%    73.376     3032.50  0.00000
> > 2.6.0-test6-mm4            128    2.71  2.52%   520.381    33565.98  8.79664
> > 2.6.0-test7                128    3.09  2.69%   409.208    33147.84  7.51347
> >
> > 2.6.0-test5                256    7.53 11.50%   348.180     2671.65  0.00000
> > 2.6.0-test5-mm2            256    7.77 11.45%   319.292     2285.73  0.00000
> > 2.6.0-test5-mm4            256    7.23 11.94%   364.323     2244.99  0.00000
> > 2.6.0-test6-mm4            256    2.00  1.95%  1023.629   216889.13 12.88433
> > 2.6.0-test7                256    1.85  1.88%  1090.929   455893.85 13.53616
> >
> > Random Reads jfs
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1    0.95  0.82%    12.368       86.79  0.00000
> > 2.6.0-test5-mm2              1    0.97  0.89%    12.013      108.08  0.00000
> > 2.6.0-test5-mm4              1    0.99  0.99%    11.796      110.46  0.00000
> > 2.6.0-test6-mm4              1    0.98  0.91%    11.987      112.94  0.00000
> > 2.6.0-test7                  1    0.96  0.83%    12.187      100.56  0.00000
> >
> > 2.6.0-test5                  8    4.68  4.23%    18.784      137.13  0.00000
> > 2.6.0-test5-mm2              8    4.06  3.84%    21.942      145.02  0.00000
> > 2.6.0-test5-mm4              8    4.01  3.80%    20.854      150.03  0.00000
> > 2.6.0-test6-mm4              8    0.81  0.61%   112.779     1277.22  0.00000
> > 2.6.0-test7                  8    0.82  0.66%   110.357     1250.05  0.00000
> >
> > 2.6.0-test5                 16    3.98  3.12%    44.566      188.99  0.00000
> > 2.6.0-test5-mm2             16    4.35  3.42%    40.240      189.97  0.00000
> > 2.6.0-test5-mm4             16    4.28  3.37%    41.846      199.23  0.00000
> > 2.6.0-test6-mm4             16    0.75  0.62%   231.931     2631.93  0.00000
> > 2.6.0-test7                 16    0.84  0.65%   212.690     2578.59  0.00000
> >
> > 2.6.0-test5                 32    4.09  2.87%    86.440      373.37  0.00000
> > 2.6.0-test5-mm2             32    3.86  3.07%    91.487      443.48  0.00000
> > 2.6.0-test5-mm4             32    3.88  3.07%    90.965      553.03  0.00000
> > 2.6.0-test6-mm4             32    0.76  0.64%   455.886     5230.06  0.00000
> > 2.6.0-test7                 32    0.71  0.57%   477.112     5188.75  0.00000
> >
> > 2.6.0-test5                 64    4.04  2.88%   169.495     1553.14  0.00000
> > 2.6.0-test5-mm2             64    3.84  3.25%   170.017     1413.19  0.00000
> > 2.6.0-test5-mm4             64    3.99  3.37%   167.645     1628.57  0.00000
> > 2.6.0-test6-mm4             64    0.70  0.61%   950.464    10451.50 26.76412
> > 2.6.0-test7                 64    0.71  0.63%   923.148    10352.12 26.38609
> >
> > 2.6.0-test5                128    4.32  3.83%   268.303     2322.59  0.00000
> > 2.6.0-test5-mm2            128    4.18  3.95%   253.072     2451.90  0.00000
> > 2.6.0-test5-mm4            128    4.05  3.95%   255.242     2232.10  0.00000
> > 2.6.0-test6-mm4            128    0.76  0.77%  1626.585    26333.82 35.78629
> > 2.6.0-test7                128    0.78  0.69%  1521.032    24112.21 33.84576
> >
> > 2.6.0-test5                256    4.03  5.43%   585.871     3402.15  0.00000
> > 2.6.0-test5-mm2            256    4.06  5.49%   556.512     3436.30  0.00000
> > 2.6.0-test5-mm4            256    4.11  5.47%   543.649     2806.43  0.00000
> > 2.6.0-test6-mm4            256    0.82  0.87%  2318.070    33523.86 44.84375
> > 2.6.0-test7                256    0.81  0.85%  2315.106    34969.85 44.94791
> >
> > Sequential Writes jfs
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1   56.47 47.37%     0.170     3391.01  0.00000
> > 2.6.0-test5-mm2              1   56.47 49.42%     0.172     2901.46  0.00000
> > 2.6.0-test5-mm4              1   56.40 48.44%     0.181     2578.21  0.00000
> > 2.6.0-test6-mm4              1   56.28 47.99%     0.175     2442.78  0.00000
> > 2.6.0-test7                  1   56.39 46.73%     0.171     3405.20  0.00000
> >
> > 2.6.0-test5                  8   56.19 65.59%     1.343     8360.10  0.00100
> > 2.6.0-test5-mm2              8   55.94 66.86%     1.336    17088.51  0.00162
> > 2.6.0-test5-mm4              8   56.03 67.52%     1.316    12716.34  0.00105
> > 2.6.0-test6-mm4              8   56.14 65.15%     1.320    13881.60  0.00100
> > 2.6.0-test7                  8   56.23 64.40%     1.335    11973.67  0.00109
> >
> > 2.6.0-test5                 16   55.15 67.48%     2.651    30109.56  0.03486
> > 2.6.0-test5-mm2             16   55.93 68.92%     2.576    39135.68  0.03081
> > 2.6.0-test5-mm4             16   54.88 69.40%     2.679    36823.40  0.03596
> > 2.6.0-test6-mm4             16   54.85 65.44%     2.641    21368.63  0.04229
> > 2.6.0-test7                 16   55.84 68.65%     2.635    63526.06  0.03019
> >
> > 2.6.0-test5                 32   54.38 69.36%     5.204    96795.80  0.10614
> > 2.6.0-test5-mm2             32   55.11 70.01%     4.989    55045.13  0.10557
> > 2.6.0-test5-mm4             32   53.92 70.42%     5.099    68154.16  0.10362
> > 2.6.0-test6-mm4             32   54.17 70.60%     5.074    81693.83  0.09809
> > 2.6.0-test7                 32   54.75 69.14%     4.998   108095.44  0.09742
> >
> > 2.6.0-test5                 64   54.58 75.03%     9.331   208382.29  0.10910
> > 2.6.0-test5-mm2             64   54.44 69.22%     9.195   156878.93  0.11329
> > 2.6.0-test5-mm4             64   53.35 72.64%     9.621   158954.21  0.10639
> > 2.6.0-test6-mm4             64   54.54 70.28%     9.386   148888.99  0.11191
> > 2.6.0-test7                 64   54.67 69.68%     8.969   189127.28  0.10605
> >
> > 2.6.0-test5                128   10.39 18.85%    93.764  2075500.73  0.09913
> > 2.6.0-test5-mm2            128   15.87 27.47%    61.244  1306699.59  0.08931
> > 2.6.0-test5-mm4            128   16.06 28.24%    56.744  1437340.48  0.08950
> > 2.6.0-test6-mm4            128   20.89 35.44%    42.971  1018802.82  0.10410
> > 2.6.0-test7                128   13.63 23.07%    69.588  1685679.31  0.08650
> >
> > 2.6.0-test5                256   10.65 20.80%   168.683  1900450.38  0.20290
> > 2.6.0-test5-mm2            256    9.72 18.90%   202.289  2024501.68  0.13385
> > 2.6.0-test5-mm4            256    8.52 17.57%   221.659  1967027.22  0.19260
> > 2.6.0-test6-mm4            256    9.20 17.32%   200.496  1843020.23  0.12737
> > 2.6.0-test7                256    8.85 17.47%   212.298  2080395.84  0.11768
> >
> > Random Writes jfs
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1    3.90  3.48%     0.060        0.56  0.00000
> > 2.6.0-test5-mm2              1    3.89  3.25%     0.061        0.28  0.00000
> > 2.6.0-test5-mm4              1    3.86  3.23%     0.061        0.35  0.00000
> > 2.6.0-test6-mm4              1    3.93  3.40%     0.060        0.37  0.00000
> > 2.6.0-test7                  1    3.69  3.03%     0.059        0.88  0.00000
> >
> > 2.6.0-test5                  8    3.86  6.54%     0.170       59.53  0.00000
> > 2.6.0-test5-mm2              8    3.68  6.19%     0.170       77.58  0.00000
> > 2.6.0-test5-mm4              8    3.69  6.03%     0.171       69.55  0.00000
> > 2.6.0-test6-mm4              8    3.55  5.93%     0.153       34.88  0.00000
> > 2.6.0-test7                  8    3.56  5.99%     0.162       52.74  0.00000
> >
> > 2.6.0-test5                 16    3.71  6.47%     0.162       43.67  0.00000
> > 2.6.0-test5-mm2             16    3.82  6.63%     0.166       54.86  0.00000
> > 2.6.0-test5-mm4             16    3.71  6.33%     0.168       43.72  0.00000
> > 2.6.0-test6-mm4             16    3.42  6.01%     0.151       14.21  0.00000
> > 2.6.0-test7                 16    3.56  5.94%     0.149        4.41  0.00000
> >
> > 2.6.0-test5                 32    3.71  6.48%     0.178      111.32  0.00000
> > 2.6.0-test5-mm2             32    3.50  5.99%     0.148        3.16  0.00000
> > 2.6.0-test5-mm4             32    3.56  6.47%     0.170       55.28  0.00000
> > 2.6.0-test6-mm4             32    3.44  6.09%     0.147        4.23  0.00000
> > 2.6.0-test7                 32    3.22  5.41%     0.158       59.51  0.00000
> >
> > 2.6.0-test5                 64    3.50  6.28%     0.148        3.96  0.00000
> > 2.6.0-test5-mm2             64    3.34  5.99%     0.160       51.30  0.00000
> > 2.6.0-test5-mm4             64    3.58  6.46%     0.157        7.48  0.00000
> > 2.6.0-test6-mm4             64    3.25  5.67%     0.146        3.61  0.00000
> > 2.6.0-test7                 64    3.28  5.74%     0.168       55.67  0.00000
> >
> > 2.6.0-test5                128    3.28  5.72%    20.122     1415.53  0.00000
> > 2.6.0-test5-mm2            128    3.29  5.92%     1.297     1229.30  0.00000
> > 2.6.0-test5-mm4            128    3.42  6.33%     1.499      541.00  0.00000
> > 2.6.0-test6-mm4            128    3.08  5.62%     1.341     1089.07  0.00000
> > 2.6.0-test7                128    3.13  5.51%     0.595      923.55  0.00000
> >
> > 2.6.0-test5                256    2.80  5.18%    89.100     3075.82  0.00000
> > 2.6.0-test5-mm2            256    2.98  5.76%    33.288     2094.84  0.00000
> > 2.6.0-test5-mm4            256    3.19  6.47%    13.738     1648.72  0.00000
> > 2.6.0-test6-mm4            256    2.65  5.49%    18.498     2632.76  0.00000
> > 2.6.0-test7                256    2.74  5.45%    10.049     2382.47  0.00000
> >
> > Sequential Reads reiserfs
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1   28.20 15.37%     0.413       97.43  0.00000
> > 2.6.0-test5-mm2              1   30.21 16.13%     0.385     1159.97  0.00000
> > 2.6.0-test5-mm4              1   28.15 15.29%     0.413      771.52  0.00000
> > 2.6.0-test6                  1   28.84 15.63%     0.404      111.79  0.00000
> > 2.6.0-test6-mm4              1   29.99 16.01%     0.388     1065.08  0.00000
> > 2.6.0-test7                  1   29.45 15.70%     0.395      113.20  0.00000
> >
> > 2.6.0-test5                  8   29.77 17.07%     2.974      854.50  0.00000
> > 2.6.0-test5-mm2              8   31.14 18.03%     2.811      653.61  0.00000
> > 2.6.0-test5-mm4              8   32.23 18.70%     2.727      542.20  0.00000
> > 2.6.0-test6                  8   21.41 11.92%     4.275     1487.18  0.00000
> > 2.6.0-test6-mm4              8   21.93 12.30%     4.185     1320.72  0.00000
> > 2.6.0-test7                  8   21.40 11.78%     4.263     1261.39  0.00000
> >
> > 2.6.0-test5                 16   29.99 17.01%     5.829      901.20  0.00000
> > 2.6.0-test5-mm2             16   29.66 17.05%     5.954     1288.12  0.00000
> > 2.6.0-test5-mm4             16   30.20 17.66%     5.872     1153.05  0.00000
> > 2.6.0-test6                 16   19.46 10.99%     9.435     2515.46  0.00000
> > 2.6.0-test6-mm4             16   18.92 10.48%     9.650     2572.79  0.00000
> > 2.6.0-test7                 16   18.34 10.11%    10.014     2648.17  0.00000
> >
> > 2.6.0-test5                 32   28.86 16.73%    12.003     1408.20  0.00000
> > 2.6.0-test5-mm2             32   28.21 16.52%    12.099     1385.36  0.00000
> > 2.6.0-test5-mm4             32   27.64 16.72%    12.706     1491.53  0.00000
> > 2.6.0-test6                 32   17.33  9.87%    20.872     5771.49  0.00005
> > 2.6.0-test6-mm4             32   18.21 10.25%    19.659     4621.02  0.00000
> > 2.6.0-test7                 32   17.89  9.88%    20.121     4681.65  0.00000
> >
> > 2.6.0-test5                 64   24.08 15.49%    27.936     1610.23  0.00000
> > 2.6.0-test5-mm2             64   23.23 15.01%    27.942     1478.59  0.00000
> > 2.6.0-test5-mm4             64   22.97 15.39%    29.020     1577.31  0.00000
> > 2.6.0-test6                 64    7.62  5.28%    95.479    16071.22  3.52058
> > 2.6.0-test6-mm4             64    7.89  5.41%    91.677    12720.86  3.22466
> > 2.6.0-test7                 64    7.51  5.16%    96.373    14738.26  3.39680
> >
> > 2.6.0-test5                128    9.77 11.49%   141.347     1501.12  0.00000
> > 2.6.0-test5-mm2            128   10.81 12.13%   121.146     1501.77  0.00000
> > 2.6.0-test5-mm4            128   10.08 12.01%   134.006     1616.24  0.00000
> > 2.6.0-test6                128    2.71  2.80%   534.963    33603.12 10.19945
> > 2.6.0-test6-mm4            128    2.76  2.91%   524.334    34758.05  9.84798
> > 2.6.0-test7                128    2.66  2.72%   543.918    33661.62 10.33516
> >
> > 2.6.0-test5                256    8.88 15.47%   327.283     2378.44  0.00000
> > 2.6.0-test5-mm2            256    8.59 15.37%   339.514     1905.29  0.00000
> > 2.6.0-test5-mm4            256    8.60 16.11%   338.642     1890.15  0.00000
> > 2.6.0-test6                256    2.29  2.42%  1177.401   291951.14 11.55243
> > 2.6.0-test6-mm4            256    2.32  2.50%  1158.512   343146.47 11.20739
> > 2.6.0-test7                256    2.28  2.39%  1172.902   277837.90 11.42778
> >
> > Random Reads reiserfs
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1    0.88  0.95%    13.375      121.48  0.00000
> > 2.6.0-test5-mm2              1    1.01  1.08%    11.543      382.20  0.00000
> > 2.6.0-test5-mm4              1    0.86  1.11%    13.564      123.23  0.00000
> > 2.6.0-test6                  1    0.96  1.10%    12.246      127.82  0.00000
> > 2.6.0-test6-mm4              1    1.02  1.16%    11.496      391.26  0.00000
> > 2.6.0-test7                  1    0.99  1.04%    11.863      111.96  0.00000
> >
> > 2.6.0-test5                  8    4.59  5.28%    17.392      126.60  0.00000
> > 2.6.0-test5-mm2              8    4.88  5.41%    15.978      105.27  0.00000
> > 2.6.0-test5-mm4              8    4.64  5.92%    17.234      119.30  0.00000
> > 2.6.0-test6                  8    3.12  3.28%    27.584      373.26  0.00000
> > 2.6.0-test6-mm4              8    3.80  4.12%    23.064      280.06  0.00000
> > 2.6.0-test7                  8    3.35  3.69%    24.895      337.23  0.00000
> >
> > 2.6.0-test5                 16    4.80  4.33%    34.457      189.80  0.00000
> > 2.6.0-test5-mm2             16    5.27  5.28%    30.882      160.61  0.00000
> > 2.6.0-test5-mm4             16    5.22  5.44%    32.357      494.99  0.00000
> > 2.6.0-test6                 16    3.93  4.12%    43.768      728.55  0.00000
> > 2.6.0-test6-mm4             16    3.96  3.79%    41.680      775.69  0.00000
> > 2.6.0-test7                 16    3.96  3.85%    42.797      705.06  0.00000
> >
> > 2.6.0-test5                 32    5.45  5.07%    59.014      323.30  0.00000
> > 2.6.0-test5-mm2             32    5.65  5.45%    57.673      313.96  0.00000
> > 2.6.0-test5-mm4             32    4.47  4.84%    75.004      455.38  0.00000
> > 2.6.0-test6                 32    4.39  4.58%    77.879     1232.67  0.00000
> > 2.6.0-test6-mm4             32    4.15  4.34%    80.355     1229.75  0.00000
> > 2.6.0-test7                 32    4.88  4.59%    67.103     1033.43  0.00000
> >
> > 2.6.0-test5                 64    5.49  5.42%   108.692      991.66  0.00000
> > 2.6.0-test5-mm2             64    5.10  5.60%   113.940      914.35  0.00000
> > 2.6.0-test5-mm4             64    4.72  5.35%   126.505     1182.30  0.00000
> > 2.6.0-test6                 64    3.90  4.03%   165.855     2554.36  0.00000
> > 2.6.0-test6-mm4             64    4.54  4.72%   140.927     2060.27  0.00000
> > 2.6.0-test7                 64    4.95  4.73%   129.947     2011.81  0.00000
> >
> > 2.6.0-test5                128    5.48  5.99%   221.089     2074.91  0.00000
> > 2.6.0-test5-mm2            128    5.71  6.49%   208.548     1781.53  0.00000
> > 2.6.0-test5-mm4            128    4.94  5.94%   240.781     2090.92  0.00000
> > 2.6.0-test6                128    3.55  4.55%   367.628     5086.89  0.00000
> > 2.6.0-test6-mm4            128    4.15  4.50%   314.516     3914.52  0.00000
> > 2.6.0-test7                128    3.98  4.56%   328.297     4300.67  0.00000
> >
> > 2.6.0-test5                256    5.39  7.63%   455.466     2448.36  0.00000
> > 2.6.0-test5-mm2            256    5.85  8.92%   415.957     2773.47  0.00000
> > 2.6.0-test5-mm4            256    4.88  8.30%   498.235     2859.83  0.00000
> > 2.6.0-test6                256    2.88  4.24%   861.877     8947.59  7.13542
> > 2.6.0-test6-mm4            256    3.28  4.52%   752.088     7585.14  3.09895
> > 2.6.0-test7                256    3.11  4.19%   794.027     8045.64  3.82812
> >
> > Sequential Writes reiserf
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1   55.30 90.83%     0.200     1317.87  0.00000
> > 2.6.0-test5-mm2              1   55.08 92.70%     0.201     1512.52  0.00000
> > 2.6.0-test5-mm4              1   54.62 92.30%     0.202     2146.88  0.00000
> > 2.6.0-test6                  1   55.09 91.14%     0.201     1243.21  0.00000
> > 2.6.0-test6-mm4              1   54.46 91.70%     0.202     1725.38  0.00000
> > 2.6.0-test7                  1   55.08 91.27%     0.200     2381.55  0.00000
> >
> > 2.6.0-test5                  8   27.22 99.99%     2.834    62892.07  0.02932
> > 2.6.0-test5-mm2              8   27.84 100.4%     2.721    20683.58  0.03290
> > 2.6.0-test5-mm4              8   28.18 96.26%     2.667    24253.31  0.03237
> > 2.6.0-test6                  8   28.88 102.9%     2.572    28097.64  0.03533
> > 2.6.0-test6-mm4              8   26.57 92.64%     2.855    24274.27  0.05517
> > 2.6.0-test7                  8   28.60 104.5%     2.571    19829.03  0.02890
> >
> > 2.6.0-test5                 16   30.58 102.6%     4.745    29788.03  0.07806
> > 2.6.0-test5-mm2             16   28.94 101.7%     5.051    37491.58  0.07902
> > 2.6.0-test5-mm4             16   28.45 97.86%     4.971    28952.01  0.08851
> > 2.6.0-test6                 16   27.13 98.05%     5.414    58605.69  0.08331
> > 2.6.0-test6-mm4             16   27.04 89.13%     5.347    24565.13  0.08516
> > 2.6.0-test7                 16   29.19 98.45%     5.031    29653.75  0.08398
> >
> > 2.6.0-test5                 32   27.88 104.6%    10.012    44186.44  0.11630
> > 2.6.0-test5-mm2             32   27.15 104.7%     9.911    47326.17  0.11076
> > 2.6.0-test5-mm4             32   27.57 102.6%     9.820    88173.93  0.10858
> > 2.6.0-test6                 32   27.80 100.8%     9.951    72577.05  0.11445
> > 2.6.0-test6-mm4             32   25.80 97.74%    10.563    65400.15  0.11706
> > 2.6.0-test7                 32   28.65 105.3%     9.613    44890.58  0.11087
> >
> > 2.6.0-test5                 64   27.12 109.5%    19.091    93664.16  0.15073
> > 2.6.0-test5-mm2             64   25.80 117.2%    20.100    73309.25  0.18029
> > 2.6.0-test5-mm4             64   25.64 112.9%    19.408    76139.55  0.16613
> > 2.6.0-test6                 64   26.30 112.0%    19.162    93011.66  0.16308
> > 2.6.0-test6-mm4             64   24.15 104.3%    21.613    86562.64  0.18483
> > 2.6.0-test7                 64   25.10 107.7%    20.311    80093.32  0.16470
> >
> > 2.6.0-test5                128   23.52 119.4%    42.286   116297.76  0.39005
> > 2.6.0-test5-mm2            128   24.88 127.8%    39.106   139584.40  0.35658
> > 2.6.0-test5-mm4            128   24.28 127.9%    39.111   159921.42  0.37918
> > 2.6.0-test6                128   24.79 122.4%    41.723   127648.00  0.39692
> > 2.6.0-test6-mm4            128   24.48 125.2%    39.716   128936.67  0.38275
> > 2.6.0-test7                128   24.29 122.0%    40.692   152191.29  0.35692
> >
> > 2.6.0-test5                256   19.14 163.4%   112.557   239944.32  1.00599
> > 2.6.0-test5-mm2            256   23.21 153.9%    84.422   200276.08  0.75154
> > 2.6.0-test5-mm4            256   23.45 161.4%    86.423   216520.39  0.71788
> > 2.6.0-test6                256   23.14 168.2%    83.087   247571.24  0.75640
> > 2.6.0-test6-mm4            256   20.87 136.1%    74.033   177291.47  0.66229
> > 2.6.0-test7                256   23.11 142.0%    79.651   233921.91  0.66109
> >
> > Random Writes reiserfs
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1    2.41  2.77%     1.925      282.65  0.00000
> > 2.6.0-test5-mm2              1    3.71  4.05%     0.222       18.72  0.00000
> > 2.6.0-test5-mm4              1    2.29  2.87%     2.191      299.62  0.00000
> > 2.6.0-test6                  1    2.89  3.32%     1.040       91.48  0.00000
> > 2.6.0-test6-mm4              1    3.60  3.66%     0.228       24.13  0.00000
> > 2.6.0-test7                  1    3.58  3.58%     0.347       30.23  0.00000
> >
> > 2.6.0-test5                  8    3.50  4.12%    11.316     1348.95  0.00000
> > 2.6.0-test5-mm2              8    3.74  5.07%    10.135     1133.62  0.00000
> > 2.6.0-test5-mm4              8    3.54  4.73%    10.168     1285.63  0.00000
> > 2.6.0-test6                  8    3.26  4.02%     9.287     1182.99  0.00000
> > 2.6.0-test6-mm4              8    3.39  4.53%     5.611     1106.14  0.00000
> > 2.6.0-test7                  8    3.29  3.77%    10.518     1191.14  0.00000
> >
> > 2.6.0-test5                 16    3.44  4.05%    22.370     4353.76  0.10000
> > 2.6.0-test5-mm2             16    3.54  3.72%    22.545     1478.64  0.00000
> > 2.6.0-test5-mm4             16    3.61  4.71%    20.853     1777.27  0.00000
> > 2.6.0-test6                 16    3.60  4.11%    17.724     1421.63  0.00000
> > 2.6.0-test6-mm4             16    3.37  3.72%    23.097     1422.10  0.00000
> > 2.6.0-test7                 16    3.46  4.40%    16.822     4822.00  0.02500
> >
> > 2.6.0-test5                 32    3.68  4.57%    22.011     1260.71  0.00000
> > 2.6.0-test5-mm2             32    3.60  4.50%    26.561     1836.41  0.00000
> > 2.6.0-test5-mm4             32    3.55  4.32%    33.055     1460.77  0.00000
> > 2.6.0-test6                 32    3.41  4.07%    23.498     2013.53  0.00000
> > 2.6.0-test6-mm4             32    3.42  4.36%    13.875     1214.49  0.00000
> > 2.6.0-test7                 32    3.42  4.04%    26.600     3437.82  0.02500
> >
> > 2.6.0-test5                 64    3.62  5.09%    37.320     8700.40  0.90725
> > 2.6.0-test5-mm2             64    3.81  6.11%    18.225     1814.21  0.00000
> > 2.6.0-test5-mm4             64    3.37  5.09%    43.372     3280.58  0.00000
> > 2.6.0-test6                 64    2.96  3.93%    73.537     4484.06  0.00000
> > 2.6.0-test6-mm4             64    3.29  4.98%    22.802     2206.78  0.00000
> > 2.6.0-test7                 64    3.42  5.00%    27.782     2453.44  0.00000
> >
> > 2.6.0-test5                128    3.73  5.92%    48.524     7848.01  1.10887
> > 2.6.0-test5-mm2            128    3.39  5.91%    36.168     5651.07  0.25202
> > 2.6.0-test5-mm4            128    3.55  6.74%    70.991     6856.35  1.94052
> > 2.6.0-test6                128    3.16  5.58%    43.277     5286.77  0.05040
> > 2.6.0-test6-mm4            128    3.17  5.25%    43.818     5534.13  0.30242
> > 2.6.0-test7                128    3.26  5.26%    39.051     3037.10  0.00000
> >
> > 2.6.0-test5                256    3.96  8.27%     8.806      963.34  0.00000
> > 2.6.0-test5-mm2            256    3.67  7.17%    20.000     1575.35  0.00000
> > 2.6.0-test5-mm4            256    3.36  6.99%   114.310     4266.84  1.61458
> > 2.6.0-test6                256    3.10  6.69%   132.312     8228.26  3.30729
> > 2.6.0-test6-mm4            256    3.21  6.79%    88.532     8631.04  1.79687
> > 2.6.0-test7                256    3.23  6.61%    48.643     6005.85  0.75521
> >
> > Sequential Reads xfs
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1   30.46 14.80%     0.382      718.97  0.00000
> > 2.6.0-test5-mm2              1   28.43 14.31%     0.409      729.27  0.00000
> > 2.6.0-test5-mm4              1   29.25 14.50%     0.398      751.10  0.00000
> > 2.6.0-test6                  1   29.25 14.52%     0.398      735.45  0.00000
> > 2.6.0-test6-mm4              1   28.68 14.60%     0.406      924.52  0.00000
> > 2.6.0-test7                  1   29.20 14.46%     0.398      727.21  0.00000
> >
> > 2.6.0-test5                  8   55.34 28.06%     1.652      831.66  0.00000
> > 2.6.0-test5-mm2              8   54.19 28.05%     1.707      852.34  0.00000
> > 2.6.0-test5-mm4              8   55.49 28.38%     1.591     1165.42  0.00000
> > 2.6.0-test6                  8   25.17 11.78%     3.562     1987.22  0.00000
> > 2.6.0-test6-mm4              8   23.25 11.32%     3.989     1872.56  0.00000
> > 2.6.0-test7                  8   25.91 12.19%     3.443     1894.36  0.00000
> >
> > 2.6.0-test5                 16   47.04 23.31%     3.903      846.86  0.00000
> > 2.6.0-test5-mm2             16   45.50 23.01%     4.086     1116.23  0.00000
> > 2.6.0-test5-mm4             16   47.00 23.78%     3.827      822.79  0.00000
> > 2.6.0-test6                 16   15.07  7.14%    12.090     3088.51  0.00000
> > 2.6.0-test6-mm4             16   12.58  6.23%    14.655     3469.24  0.00000
> > 2.6.0-test7                 16   14.79  7.03%    11.808     3178.50  0.00000
> >
> > 2.6.0-test5                 32   44.85 22.40%     7.699     1225.63  0.00000
> > 2.6.0-test5-mm2             32   45.89 23.39%     7.624     1173.54  0.00000
> > 2.6.0-test5-mm4             32   44.18 22.68%     7.940     1051.95  0.00000
> > 2.6.0-test6                 32   14.80  7.02%    24.021     5758.07  0.00005
> > 2.6.0-test6-mm4             32   14.18  6.98%    25.241     5709.74  0.00000
> > 2.6.0-test7                 32   12.46  6.03%    29.086     5836.49  0.00029
> >
> > 2.6.0-test5                 64   42.08 21.45%    16.269     1897.20  0.00000
> > 2.6.0-test5-mm2             64   42.36 21.75%    15.948     1925.10  0.00000
> > 2.6.0-test5-mm4             64   43.77 22.73%    15.396     1963.29  0.00000
> > 2.6.0-test6                 64   12.50  5.98%    53.864    10887.40  1.74727
> > 2.6.0-test6-mm4             64   12.37  6.19%    55.962    10690.33  1.82176
> > 2.6.0-test7                 64   12.25  5.95%    56.211    10899.32  1.78017
> >
> > 2.6.0-test5                128   15.41 12.23%    87.001     2911.16  0.00000
> > 2.6.0-test5-mm2            128   20.45 15.21%    61.549     2516.86  0.00000
> > 2.6.0-test5-mm4            128   18.55 14.38%    68.572     2527.97  0.00000
> > 2.6.0-test6                128    5.20  4.07%   251.782    18918.92  4.34985
> > 2.6.0-test6-mm4            128    4.47  3.56%   290.774    19729.60  5.13649
> > 2.6.0-test7                128    3.50  3.19%   385.879    20057.71  6.42790
> >
> > 2.6.0-test5                256   10.33 13.76%   225.449     6314.51  0.00167
> > 2.6.0-test5-mm2            256   10.58 14.19%   206.025     9145.94  0.00219
> > 2.6.0-test5-mm4            256   11.64 15.40%   175.294    11398.84  0.00677
> > 2.6.0-test6                256    3.84  3.42%   539.606   972621.75  5.75909
> > 2.6.0-test6-mm4            256    3.92  3.35%   527.548   867696.31  5.66021
> > 2.6.0-test7                256    3.99  3.52%   511.822   865743.62  5.56827
> >
> > Random Reads xfs
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1    1.03  0.94%    11.357      107.82  0.00000
> > 2.6.0-test5-mm2              1    1.03  1.02%    11.325      117.87  0.00000
> > 2.6.0-test5-mm4              1    1.04  1.03%    11.261      107.89  0.00000
> > 2.6.0-test6                  1    1.03  0.92%    11.386      109.39  0.00000
> > 2.6.0-test6-mm4              1    1.00  1.04%    11.659      119.70  0.00000
> > 2.6.0-test7                  1    1.05  0.92%    11.161       96.11  0.00000
> >
> > 2.6.0-test5                  8    5.42  5.19%    16.257      124.33  0.00000
> > 2.6.0-test5-mm2              8    5.36  5.79%    16.591      120.16  0.00000
> > 2.6.0-test5-mm4              8    4.67  4.80%    16.448      104.77  0.00000
> > 2.6.0-test6                  8    2.08  1.68%    38.061      775.61  0.00000
> > 2.6.0-test6-mm4              8    2.16  1.95%    37.981      686.42  0.00000
> > 2.6.0-test7                  8    1.88  1.64%    42.173      683.55  0.00000
> >
> > 2.6.0-test5                 16    5.20  4.62%    34.058      146.53  0.00000
> > 2.6.0-test5-mm2             16    5.21  4.88%    34.306      175.88  0.00000
> > 2.6.0-test5-mm4             16    5.18  4.84%    32.751      135.47  0.00000
> > 2.6.0-test6                 16    1.86  1.44%    85.898     1244.64  0.00000
> > 2.6.0-test6-mm4             16    1.95  1.64%    80.971     1175.91  0.00000
> > 2.6.0-test7                 16    2.35  1.98%    67.271     1001.72  0.00000
> >
> > 2.6.0-test5                 32    4.91  3.98%    68.586      265.91  0.00000
> > 2.6.0-test5-mm2             32    5.43  4.74%    63.283      279.95  0.00000
> > 2.6.0-test5-mm4             32    4.69  4.28%    68.452      638.24  0.00000
> > 2.6.0-test6                 32    2.92  2.41%   113.627     1626.83  0.00000
> > 2.6.0-test6-mm4             32    3.14  2.92%   105.714     1451.06  0.00000
> > 2.6.0-test7                 32    2.43  1.91%   134.936     1864.81  0.00000
> >
> > 2.6.0-test5                 64    4.76  4.02%   136.805      879.54  0.00000
> > 2.6.0-test5-mm2             64    4.66  4.68%   136.498      871.90  0.00000
> > 2.6.0-test5-mm4             64    4.81  4.75%   131.041      994.54  0.00000
> > 2.6.0-test6                 64    1.45  1.25%   425.661     5262.95  0.02520
> > 2.6.0-test6-mm4             64    1.66  1.40%   382.569     4759.16  0.00000
> > 2.6.0-test7                 64    1.41  1.21%   444.027     5820.44  0.30242
> >
> > 2.6.0-test5                128    4.70  4.50%   248.562     1396.77  0.00000
> > 2.6.0-test5-mm2            128    5.29  5.72%   206.313     1395.33  0.00000
> > 2.6.0-test5-mm4            128    5.16  4.95%   209.001     1485.37  0.00000
> > 2.6.0-test6                128    1.65  1.57%   753.959     9498.63  9.02218
> > 2.6.0-test6-mm4            128    1.47  1.43%   833.261     9951.50 11.54233
> > 2.6.0-test7                128    1.23  1.06%   969.453    10793.66 18.04436
> >
> > 2.6.0-test5                256    4.82  6.71%   469.459     2140.97  0.00000
> > 2.6.0-test5-mm2            256    4.60  6.06%   477.682     1934.27  0.00000
> > 2.6.0-test5-mm4            256    5.01  6.60%   402.622     1881.24  0.00000
> > 2.6.0-test6                256    0.90  0.99%  2204.619    32794.62 42.68229
> > 2.6.0-test6-mm4            256    0.88  0.98%  2165.705    33928.64 41.09376
> > 2.6.0-test7                256    0.92  1.05%  2113.981    33637.27 39.11458
> >
> > Sequential Writes xfs
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1   56.44 46.30%     0.182    68379.68  0.00030
> > 2.6.0-test5-mm2              1   56.33 47.39%     0.186    68474.74  0.00030
> > 2.6.0-test5-mm4              1   56.30 46.74%     0.182    68268.99  0.00030
> > 2.6.0-test6                  1   56.24 46.08%     0.177    68501.77  0.00030
> > 2.6.0-test6-mm4              1   56.17 47.57%     0.175    68404.17  0.00030
> > 2.6.0-test7                  1   56.26 46.03%     0.171    68451.18  0.00030
> >
> > 2.6.0-test5                  8   57.54 65.47%     0.991   166706.97  0.00424
> > 2.6.0-test5-mm2              8   57.70 70.44%     1.099   211950.97  0.00420
> > 2.6.0-test5-mm4              8   57.23 66.09%     1.042   205671.61  0.00415
> > 2.6.0-test6                  8   57.19 69.31%     1.058   185132.73  0.00410
> > 2.6.0-test6-mm4              8   57.46 66.82%     1.100   178544.18  0.00486
> > 2.6.0-test7                  8   57.24 64.93%     1.096   207662.87  0.00481
> >
> > 2.6.0-test5                 16   57.26 71.20%     2.069   186551.11  0.01030
> > 2.6.0-test5-mm2             16   57.26 72.02%     2.079   187786.90  0.01059
> > 2.6.0-test5-mm4             16   57.38 68.91%     1.999   193908.75  0.00958
> > 2.6.0-test6                 16   57.24 70.72%     2.072   185124.29  0.01092
> > 2.6.0-test6-mm4             16   57.37 68.99%     2.080   208981.25  0.01045
> > 2.6.0-test7                 16   57.17 68.79%     2.051   181396.29  0.01135
> >
> > 2.6.0-test5                 32   57.29 75.00%     4.076   211670.36  0.01840
> > 2.6.0-test5-mm2             32   57.34 73.21%     3.768   213556.33  0.01560
> > 2.6.0-test5-mm4             32   57.27 75.59%     4.007   199887.88  0.01708
> > 2.6.0-test6                 32   57.12 73.89%     4.007   198273.60  0.01750
> > 2.6.0-test6-mm4             32   57.04 73.27%     4.034   181071.17  0.01688
> > 2.6.0-test7                 32   57.04 72.85%     4.066   193571.15  0.02007
> >
> > 2.6.0-test5                 64   57.13 83.01%     8.186   336901.61  0.03567
> > 2.6.0-test5-mm2             64   56.93 83.63%     8.062   340355.98  0.03066
> > 2.6.0-test5-mm4             64   56.33 82.10%     7.834   325305.64  0.02895
> > 2.6.0-test6                 64   56.48 80.10%     7.829   306781.15  0.03157
> > 2.6.0-test6-mm4             64   56.87 81.05%     7.808   257430.27  0.03662
> > 2.6.0-test7                 64   56.54 77.01%     7.746   347269.10  0.02880
> >
> > 2.6.0-test5                128   56.58 86.45%    15.451   330514.59  0.06585
> > 2.6.0-test5-mm2            128   56.78 82.31%    14.164   364774.73  0.05298
> > 2.6.0-test5-mm4            128   55.57 82.39%    14.088   350137.61  0.05364
> > 2.6.0-test6                128   56.08 81.22%    13.917   377412.59  0.04669
> > 2.6.0-test6-mm4            128   56.43 79.05%    14.332   357252.17  0.05203
> > 2.6.0-test7                128   56.28 77.14%    14.465   351042.94  0.05546
> >
> > 2.6.0-test5                256   56.13 83.69%    19.836   383235.17  0.07386
> > 2.6.0-test5-mm2            256   56.19 80.90%    16.880   294598.89  0.07434
> > 2.6.0-test5-mm4            256   56.09 87.58%    13.343   308894.90  0.05746
> > 2.6.0-test6                256   55.39 80.57%    18.269   350145.38  0.07200
> > 2.6.0-test6-mm4            256   55.67 81.98%    17.338   337290.04  0.06890
> > 2.6.0-test7                256   55.49 82.29%    15.776   351541.33  0.06071
> >
> > Random Writes xfs
> >                            Num                    Avg       Maximum     Lat%
> > Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> > -------------------------  ---  --------------------------------------------
> > 2.6.0-test5                  1    3.80  3.61%     0.074        0.46  0.00000
> > 2.6.0-test5-mm2              1    3.86  3.77%     0.077        0.42  0.00000
> > 2.6.0-test5-mm4              1    3.77  3.65%     0.078        1.42  0.00000
> > 2.6.0-test6                  1    3.76  3.53%     0.075        1.43  0.00000
> > 2.6.0-test6-mm4              1    3.71  3.64%     0.078        0.68  0.00000
> > 2.6.0-test7                  1    3.76  3.53%     0.075        0.87  0.00000
> >
> > 2.6.0-test5                  8    3.97  6.78%     0.202       95.59  0.00000
> > 2.6.0-test5-mm2              8    3.91  6.75%     0.181       75.97  0.00000
> > 2.6.0-test5-mm4              8    3.87  6.57%     0.191       81.43  0.00000
> > 2.6.0-test6                  8    3.86  6.63%     0.165       25.62  0.00000
> > 2.6.0-test6-mm4              8    3.82  6.33%     0.166       39.50  0.00000
> > 2.6.0-test7                  8    3.92  6.44%     0.181       74.95  0.00000
> >
> > 2.6.0-test5                 16    3.95  6.81%     0.168       32.20  0.00000
> > 2.6.0-test5-mm2             16    3.94  6.99%     0.175       48.13  0.00000
> > 2.6.0-test5-mm4             16    3.82  6.53%     0.186       87.83  0.00000
> > 2.6.0-test6                 16    3.76  6.58%     0.168       25.32  0.00000
> > 2.6.0-test6-mm4             16    3.81  6.62%     0.193       83.97  0.00000
> > 2.6.0-test7                 16    3.89  6.57%     0.181       92.60  0.00000
> >
> > 2.6.0-test5                 32    3.85  6.64%     0.159        3.38  0.00000
> > 2.6.0-test5-mm2             32    3.80  6.75%     0.175       59.86  0.00000
> > 2.6.0-test5-mm4             32    3.72  6.60%     0.180       55.04  0.00000
> > 2.6.0-test6                 32    3.80  6.73%     0.175       56.73  0.00000
> > 2.6.0-test6-mm4             32    3.81  6.61%     0.171       55.65  0.00000
> > 2.6.0-test7                 32    3.60  5.97%     0.155        3.09  0.00000
> >
> > 2.6.0-test5                 64    3.68  6.56%     0.161        3.90  0.00000
> > 2.6.0-test5-mm2             64    3.60  6.51%     0.163        6.67  0.00000
> > 2.6.0-test5-mm4             64    3.71  6.58%     0.166        5.44  0.00000
> > 2.6.0-test6                 64    3.53  6.32%     0.160        4.14  0.00000
> > 2.6.0-test6-mm4             64    3.54  6.21%     0.185      106.22  0.00000
> > 2.6.0-test7                 64    3.61  6.15%     0.156        4.34  0.00000
> >
> > 2.6.0-test5                128    3.32  6.23%     0.166        4.42  0.00000
> > 2.6.0-test5-mm2            128    3.64  6.70%     0.163        2.98  0.00000
> > 2.6.0-test5-mm4            128    3.71  6.85%     0.169        5.37  0.00000
> > 2.6.0-test6                128    3.44  6.25%     0.174       63.35  0.00000
> > 2.6.0-test6-mm4            128    3.49  6.36%     0.174       64.77  0.00000
> > 2.6.0-test7                128    3.43  6.15%     0.159        4.05  0.00000
> >
> > 2.6.0-test5                256    3.35  6.56%     0.161        3.65  0.00000
> > 2.6.0-test5-mm2            256    3.29  6.41%     0.164        3.68  0.00000
> > 2.6.0-test5-mm4            256    3.27  6.35%     0.166        4.84  0.00000
> > 2.6.0-test6                256    2.97  5.80%     0.171       47.92  0.00000
> > 2.6.0-test6-mm4            256    3.06  5.93%     0.159        3.02  0.00000
> > 2.6.0-test7                256    3.11  5.88%     0.159        3.91  0.00000
> >
> >
> > dbench did not regress.
> >
> > dbench-1.3 reiserfs 192 processes     Average		+/- MB/sec
> > 2.6.0-test7                             77.3		   7.1
> > 2.6.0-test6-mm4                         76.6		   7.2
> > 2.6.0-test5-mm2                         76.1		   6.8
> > 2.6.0-test6                             73.0		   5.1
> > 2.6.0-test5-mm4                         72.5		   6.6
> > 2.6.0-test5                             65.0		   1.2
> >
> > dbench-1.3 reiserfs 64 processes      Average		+/- MB/sec
> > 2.6.0-test5-mm4                        111.4		   9.1
> > 2.6.0-test6-mm4                        109.6		   7.5
> > 2.6.0-test6                            108.8		   4.2
> > 2.6.0-test5-mm2                        104.4		   8.1
> > 2.6.0-test7                            102.3		   8.1
> > 2.6.0-test5                            100.6		   4.7
> >
> > dbench-1.3 xfs 192 processes          Average		+/- MB/sec
> > 2.6.0-test5                            112.5		   4.0
> > 2.6.0-test5-mm2                        110.8		   1.6
> > 2.6.0-test6-mm4                        110.7		   3.4
> > 2.6.0-test5-mm4                        109.7		   2.0
> > 2.6.0-test7                            109.0		   5.1
> > 2.6.0-test6                            108.5		   6.1
> >
> > dbench-1.3 xfs 64 processes           Average		+/- MB/sec
> > 2.6.0-test5                            143.7		   0.9
> > 2.6.0-test7                            143.6		   2.1
> > 2.6.0-test5-mm2                        141.5		   1.6
> > 2.6.0-test6                            141.4		   0.5
> > 2.6.0-test6-mm4                        141.2		   1.5
> > 2.6.0-test5-mm4                        140.0		   1.4
> >
> > dbench-1.3 ext2 192 processes         Average		+/- MB/sec
> > 2.6.0-test6                            209.5		  11.8
> > 2.6.0-test7                            205.2		  20.8
> > 2.6.0-test5-mm4                        200.8		   6.2
> > 2.6.0-test5-mm2                        200.0		  10.3
> > 2.6.0-test5                            198.1		  19.5
> > 2.6.0-test6-mm4                        193.4		  12.0
> >
> > dbench-1.3 ext2 64 processes          Average		+/- MB/sec
> > 2.6.0-test7                            256.8		   0.6
> > 2.6.0-test6                            256.7		   1.6
> > 2.6.0-test5                            256.2		   1.6
> > 2.6.0-test5-mm4                        255.5		   1.4
> > 2.6.0-test5-mm2                        255.3		   0.9
> > 2.6.0-test6-mm4                        255.3		   1.7
> >
> > dbench-1.3 jfs 192 processes          Average		+/- MB/sec
> > 2.6.0-test5-mm4                         11.3		   0.5
> > 2.6.0-test7                             11.1		   0.2
> > 2.6.0-test6-mm4                         11.0		   0.3
> >
> > dbench-1.3 jfs 64 processes           Average		+/- MB/sec
> > 2.6.0-test5-mm2                         20.0		   2.9
> > 2.6.0-test5                             19.5		   2.4
> > 2.6.0-test5-mm4                         18.6		   2.4
> > 2.6.0-test7                             18.5		   1.1
> > 2.6.0-test6-mm4                         18.1		   1.6
> >
> > dbench-1.3 ext3 192 processes         Average		+/- MB/sec
> > 2.6.0-test7                            110.8		  23.1
> > 2.6.0-test5-mm2                        107.5		  39.7
> > 2.6.0-test6                            103.9		  39.0
> > 2.6.0-test6-mm4                         99.7		  38.7
> > 2.6.0-test5-mm4                         82.6		  35.5
> > 2.6.0-test5                             73.6		  15.1
> >
> > dbench-1.3 ext3 64 processes          Average		+/- MB/sec
> > 2.6.0-test7                            177.9		   3.4
> > 2.6.0-test6                            175.1		   3.6
> > 2.6.0-test5                            174.5		   4.2
> > 2.6.0-test6-mm4                        174.2		   2.5
> > 2.6.0-test5-mm2                        172.8		   4.0
> > 2.6.0-test5-mm4                        172.6		   2.3
> >
> >
> > --
> > Randy Hron
> > http://home.earthlink.net/~rwhron/kernel/bigbox.html
> >
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-20  1:35 ` Nick Piggin
@ 2003-10-20  4:52   ` Andrew Morton
  2003-10-20  8:12     ` Nick Piggin
  0 siblings, 1 reply; 25+ messages in thread
From: Andrew Morton @ 2003-10-20  4:52 UTC (permalink / raw)
  To: Nick Piggin; +Cc: rwhron, linux-kernel

Nick Piggin <piggin@cyberone.com.au> wrote:
>
> 
> 
> rwhron@earthlink.net wrote:
> 
> >There was about a 50% regression in jobs/minute in AIM7
> >database workload on quad P3 Xeon.  The CPU time has not
> >gone up, so the extra run time is coming from something
> >else.  (I/O or I/O scheduler?)
> >
> >tiobench sequential reads has a significant regression too.
> >
> >Regression appears unrelated to filesystem type.
> >
> >dbench was not affected.
> >
> >The AIM7 was run on ext2.
> >
> 
> Yeah I'd say its all due to the IO scheduler. There is a problem
> I'm thinking about how to fix - its the likely cause of this too.
> 

What change do you think it was due to?

It's rather strange that test6 is slow but test6-mm is not: generally the
IO scheduler regressions enter -mm first ;)

Testing versus deadline would be interesting.

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-20  0:37 rwhron
  2003-10-20  1:35 ` Nick Piggin
@ 2003-10-20  4:51 ` Dave Olien
  2003-10-20  7:55   ` venom
  2003-10-24 13:52 ` Magnus Andersson
  2 siblings, 1 reply; 25+ messages in thread
From: Dave Olien @ 2003-10-20  4:51 UTC (permalink / raw)
  To: rwhron; +Cc: linux-kernel, akpm


Yup, we've seen similar regression on tiobench and reaim workloads.

On Sun, Oct 19, 2003 at 08:37:45PM -0400, rwhron@earthlink.net wrote:
> There was about a 50% regression in jobs/minute in AIM7
> database workload on quad P3 Xeon.  The CPU time has not
> gone up, so the extra run time is coming from something
> else.  (I/O or I/O scheduler?)
> 
> tiobench sequential reads has a significant regression too.
> 
> Regression appears unrelated to filesystem type.
> 
> dbench was not affected.
> 
> The AIM7 was run on ext2.
> 
> AIM7 dbase workload
> kernel                   Tasks   Jobs/Min       Real    CPU
> 2.6.0-test5-mm2          32	609.3	       311.9	139.5
> 2.6.0-test5-mm4          32	595.2	       319.4	131.7
> 2.6.0-test6-mm4          32	590.8	       321.8	130.7
> 2.6.0-test5              32	590.2	       322.0	131.9
> 2.6.0-test6              32	320.6	       592.9	129.1
> 2.6.0-test7              32	290.3	       654.8	120.9
> 
> 2.6.0-test5-mm2          64	786.2	       483.5	253.2
> 2.6.0-test6-mm4          64	779.4	       487.8	253.8
> 2.6.0-test5              64	774.3	       490.9	256.6
> 2.6.0-test5-mm4          64	772.0	       492.4	253.7
> 2.6.0-test6              64	430.5	       883.1	247.6
> 2.6.0-test7              64	402.6	       944.2	231.5
> 
> 2.6.0-test5-mm2          96	869.2	       656.0	382.2
> 2.6.0-test6-mm4          96	867.6	       657.2	387.4
> 2.6.0-test5-mm4          96	863.1	       660.7	385.2
> 2.6.0-test5              96	859.0	       663.9	389.5
> 2.6.0-test6              96	478.5	       1191.7	368.1
> 2.6.0-test7              96	453.7	       1256.9	346.8
> 
> 2.6.0-test5-mm2          128	929.4	       818.0	526.1
> 2.6.0-test6-mm4          128	922.6	       824.1	523.3
> 2.6.0-test5-mm4          128	916.6	       829.5	509.1
> 2.6.0-test5              128	910.3	       835.2	528.6
> 2.6.0-test7              128	500.5	       1519.1	461.3
> 2.6.0-test6              128	500.2	       1520.1	497.4
> 
> 2.6.0-test5-mm2          160	962.5	       987.4	674.6
> 2.6.0-test5-mm4          160	952.7	       997.6	655.5
> 2.6.0-test6-mm4          160	950.0	       1000.5	659.5
> 2.6.0-test5              160	741.5	       1281.7	644.8
> 2.6.0-test7              160	518.5	       1833.1	589.6
> 2.6.0-test6              160	479.6	       1981.8	618.9
> 
> 2.6.0-test5-mm2          192	993.9	       1147.5	810.0
> 2.6.0-test5-mm4          192	982.7	       1160.5	788.2
> 2.6.0-test6-mm4          192	978.8	       1165.2	808.4
> 2.6.0-test5              192	840.0	       1357.8	776.0
> 2.6.0-test7              192	537.9	       2120.2	709.3
> 2.6.0-test6              192	494.3	       2307.1	748.2
> 
> 2.6.0-test5-mm2          224	1011.2	       1315.8	943.5
> 2.6.0-test5-mm4          224	1004.7	       1324.3	942.1
> 2.6.0-test6-mm4          224	999.5	       1331.2	920.0
> 2.6.0-test5              224	986.9	       1348.2	911.8
> 2.6.0-test7              224	546.7	       2433.8	832.8
> 2.6.0-test6              224	482.9	       2755.1	862.1
> 
> 2.6.0-test5-mm2          256	1030.9	       1475.1	1076.5
> 2.6.0-test5-mm4          256	1027.1	       1480.6	1071.4
> 2.6.0-test6-mm4          256	1010.3	       1505.2	1074.8
> 2.6.0-test5              256	999.5	       1521.3	1029.3
> 2.6.0-test7              256	556.6	       2731.9	951.0
> 2.6.0-test6              256	545.1	       2789.5	978.4
> 
> 
> Similarly, AIM7 fileserver benchmark had a drop in throughput.
> CPU time is down, but total runtime is way up.
> 
> AIM7 fserver workload
> kernel                   Tasks   Jobs/Min       Real    CPU
> 2.6.0-test5              4	79.7	       304.0	38.3
> 2.6.0-test5-mm2          4	77.4	       313.4	38.3
> 2.6.0-test5-mm4          4	76.5	       317.1	36.7
> 2.6.0-test6-mm4          4	65.5	       369.9	34.5
> 2.6.0-test7              4	45.5	       532.5	32.2
> 2.6.0-test6              4	45.0	       538.7	31.0
> 
> 2.6.0-test5              8	129.8	       373.5	66.3
> 2.6.0-test5-mm2          8	103.1	       470.4	62.0
> 2.6.0-test5-mm4          8	102.2	       474.2	61.8
> 2.6.0-test6              8	82.7	       585.9	84.5
> 2.6.0-test6-mm4          8	82.4	       588.1	61.1
> 2.6.0-test7              8	61.0	       795.0	59.9
> 
> 2.6.0-test5              12	164.1	       443.0	93.1
> 2.6.0-test5-mm2          12	112.4	       646.8	98.4
> 2.6.0-test5-mm4          12	111.2	       653.8	96.7
> 2.6.0-test6-mm4          12	95.2	       763.6	96.9
> 2.6.0-test6              12	87.9	       827.6	118.9
> 2.6.0-test7              12	54.5	       1335.0	91.8
> 
> 2.6.0-test5              16	173.4	       559.0	116.5
> 2.6.0-test5-mm2          16	128.8	       752.9	119.9
> 2.6.0-test5-mm4          16	125.4	       773.1	124.3
> 2.6.0-test6-mm4          16	104.2	       930.3	118.7
> 2.6.0-test6              16	101.4	       956.0	107.4
> 2.6.0-test7              16	70.6	       1372.6	112.2
> 
> 2.6.0-test5              20	180.5	       671.5	143.0
> 2.6.0-test5-mm4          20	144.9	       836.4	143.0
> 2.6.0-test5-mm2          20	141.7	       855.3	144.1
> 2.6.0-test6              20	117.7	       1030.0	131.1
> 2.6.0-test6-mm4          20	114.4	       1059.5	143.0
> 2.6.0-test7              20	78.7	       1539.9	134.0
> 
> 2.6.0-test5              24	189.1	       769.0	176.0
> 2.6.0-test5-mm2          24	156.2	       930.9	180.2
> 2.6.0-test5-mm4          24	154.5	       941.6	176.9
> 2.6.0-test6              24	134.5	       1081.1	156.5
> 2.6.0-test6-mm4          24	124.5	       1167.8	170.2
> 2.6.0-test7              24	76.9	       1892.1	169.0
> 
> 2.6.0-test5              28	204.6	       829.2	197.5
> 2.6.0-test5-mm2          28	158.7	       1069.2	209.8
> 2.6.0-test5-mm4          28	156.0	       1087.7	211.5
> 2.6.0-test6-mm4          28	134.2	       1264.0	196.9
> 2.6.0-test6              28	133.7	       1268.7	174.7
> 2.6.0-test7              28	84.1	       2016.4	188.5
> 
> 2.6.0-test5              32	213.7	       907.4	228.9
> 2.6.0-test5-mm2          32	151.9	       1276.7	239.7
> 2.6.0-test5-mm4          32	151.5	       1279.6	241.4
> 2.6.0-test6              32	147.0	       1318.9	197.0
> 2.6.0-test6-mm4          32	138.8	       1397.6	225.4
> 2.6.0-test7              32	88.5	       2190.3	214.9
> 
> Same story for the shared workload.  Jobs/minute is down a
> lot since 2.6.0-test5.
> 
> 
> AIM7 shared workload
> kernel                   Tasks   Jobs/Min       Real    CPU
> 2.6.0-test5              64	1888.8	       197.2	168.3
> 2.6.0-test5-mm2          64	1517.6	       245.4	177.0
> 2.6.0-test5-mm4          64	1488.6	       250.2	180.5
> 2.6.0-test6-mm4          64	1239.0	       300.6	175.0
> 2.6.0-test7              64	1225.1	       304.0	166.6
> 2.6.0-test6              64	880.0	       423.3	168.1
> 
> 2.6.0-test5              128	2094.9	       355.6	338.1
> 2.6.0-test5-mm2          128	1837.4	       405.4	352.4
> 2.6.0-test5-mm4          128	1826.6	       407.8	358.4
> 2.6.0-test6-mm4          128	1561.8	       477.0	346.6
> 2.6.0-test7              128	1176.5	       633.2	332.6
> 2.6.0-test6              128	1142.1	       652.3	332.6
> 
> 2.6.0-test5              192	2256.0	       495.3	507.8
> 2.6.0-test5-mm2          192	2074.9	       538.5	530.9
> 2.6.0-test5-mm4          192	2044.8	       546.5	536.9
> 2.6.0-test6-mm4          192	1711.8	       652.8	527.3
> 2.6.0-test6              192	1328.3	       841.3	505.6
> 2.6.0-test7              192	1317.9	       847.9	499.1
> 
> 2.6.0-test5              256	2389.0	       623.7	683.8
> 2.6.0-test5-mm2          256	2207.2	       675.0	720.6
> 2.6.0-test5-mm4          256	2186.2	       681.5	729.9
> 2.6.0-test6-mm4          256	1780.7	       836.7	719.0
> 2.6.0-test6              256	1479.5	       1007.0	672.9
> 2.6.0-test7              256	1282.0	       1162.2	672.8
> 
> 2.6.0-test5              320	2457.9	       757.7	856.6
> 2.6.0-test5-mm4          320	2330.2	       799.2	920.0
> 2.6.0-test5-mm2          320	2327.9	       800.0	919.4
> 2.6.0-test6-mm4          320	1864.7	       998.8	900.9
> 2.6.0-test6              320	1575.9	       1181.8	853.5
> 2.6.0-test7              320	1317.9	       1413.1	844.4
> 
> 2.6.0-test5              384	2480.4	       901.0	1036.5
> 2.6.0-test5-mm2          384	2440.2	       915.9	1111.6
> 2.6.0-test5-mm4          384	2434.2	       918.1	1111.0
> 2.6.0-test6-mm4          384	1883.9	       1186.3	1091.7
> 2.6.0-test6              384	1629.8	       1371.3	1026.3
> 2.6.0-test7              384	1326.3	       1685.0	1018.6
> 
> 2.6.0-test5              448	2529.8	       1030.7	1209.2
> 2.6.0-test5-mm4          448	2511.6	       1038.1	1306.8
> 2.6.0-test5-mm2          448	2510.0	       1038.8	1306.1
> 2.6.0-test6-mm4          448	1934.3	       1347.9	1283.6
> 2.6.0-test6              448	1612.3	       1617.1	1203.3
> 2.6.0-test7              448	1344.7	       1939.0	1200.9
> 
> 2.6.0-test5-mm2          512	2579.1	       1155.4	1504.0
> 2.6.0-test5              512	2566.2	       1161.2	1386.2
> 2.6.0-test5-mm4          512	2552.9	       1167.2	1509.2
> 2.6.0-test6-mm4          512	1921.5	       1550.8	1475.1
> 2.6.0-test6              512	1538.6	       1936.7	1376.9
> 2.6.0-test7              512	1370.5	       2174.3	1379.2
> 
> tiobench read performance in test[67] has a big regression in throughput
> as the number of threads goes up.  The max and long latency numbers get
> bad too.  It appears to affect ext2, ext3, reiserfs, jfs and xfs.
> Regression is in 2.6.0-test[67] and test6-mm4.
> 
> tiobench-0.3.3
> Unit information
> ================
> File size = 8192 megabytes
> Blk Size  = 4096 bytes
> Rate      = megabytes per second
> CPU%      = percentage of CPU used during the test
> Latency   = milliseconds
> Lat%      = percent of requests that took longer than X seconds
> CPU Eff   = Rate divided by CPU% - throughput per cpu load
> 
> Sequential Reads ext2
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1   28.55 13.89%     0.408      754.59  0.00000
> 2.6.0-test5-mm2              1   28.66 13.98%     0.406      742.76  0.00000
> 2.6.0-test5-mm4              1   28.46 13.90%     0.409      887.02  0.00000
> 2.6.0-test6                  1   28.47 13.81%     0.409     1069.74  0.00000
> 2.6.0-test6-mm4              1   28.41 13.89%     0.410      723.14  0.00000
> 2.6.0-test7                  1   28.71 13.92%     0.405      715.02  0.00000
> 
> With one thread, performance is similar.
> 
> 2.6.0-test5                  8   34.01 17.09%     2.620      834.85  0.00000
> 2.6.0-test5-mm2              8   32.44 16.44%     2.765      844.99  0.00000
> 2.6.0-test5-mm4              8   30.75 15.55%     2.967      952.18  0.00000
> 2.6.0-test6                  8   20.67  9.90%     4.477     1569.30  0.00000
> 2.6.0-test6-mm4              8   21.55 10.31%     4.260     1790.19  0.00000
> 2.6.0-test7                  8   21.46 10.24%     4.312     1913.24  0.00000
> 
> 2.6.0-test5                 16   30.13 15.04%     5.980     1337.69  0.00000
> 2.6.0-test5-mm2             16   29.71 14.97%     6.024     1115.95  0.00000
> 2.6.0-test5-mm4             16   27.10 13.65%     6.606     1008.67  0.00000
> 2.6.0-test6                 16   16.44  7.90%    11.201     2820.85  0.00000
> 2.6.0-test6-mm4             16   16.14  7.78%    11.382     2753.13  0.00000
> 2.6.0-test7                 16   18.34  8.64%     9.877     2742.49  0.00000
> 
> 2.6.0-test5                 32   29.46 14.62%    11.954     1560.02  0.00000
> 2.6.0-test5-mm2             32   28.01 14.26%    12.808     1627.00  0.00000
> 2.6.0-test5-mm4             32   28.78 14.63%    12.182     1859.56  0.00000
> 2.6.0-test6                 32   15.60  7.55%    23.415     4877.75  0.00000
> 2.6.0-test6-mm4             32   14.63  7.20%    24.713     4846.63  0.00000
> 2.6.0-test7                 32   14.79  7.14%    24.234     4855.64  0.00000
> 
> 2.6.0-test5                 64   25.24 13.57%    26.901     1743.05  0.00000
> 2.6.0-test5-mm2             64   24.01 13.27%    27.922     1849.82  0.00000
> 2.6.0-test5-mm4             64   23.67 13.15%    28.513     1956.58  0.00000
> 2.6.0-test6                 64    9.45  5.24%    75.491     9578.85  2.75073
> 2.6.0-test6-mm4             64    8.51  4.82%    84.118    13166.74  3.11647
> 2.6.0-test7                 64    8.58  4.86%    83.430    13622.86  3.05581
> 
> 2.6.0-test5                128   10.41  9.76%   129.662     1820.78  0.00000
> 2.6.0-test5-mm2            128   11.00 10.44%   120.118     1742.44  0.00000
> 2.6.0-test5-mm4            128   10.67 10.15%   122.871     1912.79  0.00000
> 2.6.0-test6                128    2.97  2.66%   472.251    25694.03  8.39324
> 2.6.0-test6-mm4            128    2.94  2.66%   473.943    25530.14  8.39028
> 2.6.0-test7                128    2.94  2.68%   480.175    26306.38  8.39257
> 
> At 256 threads, throughput is down and max latency skyrockets.
> 
> 2.6.0-test5                256    9.13 13.30%   305.097     1446.43  0.00000
> 2.6.0-test5-mm2            256    9.09 13.60%   304.019     1648.80  0.00000
> 2.6.0-test5-mm4            256    8.63 13.66%   325.793     1542.62  0.00000
> 2.6.0-test6                256    2.71  2.49%   937.603   529116.60  8.94007
> 2.6.0-test6-mm4            256    2.62  2.46%   979.221   785760.77  9.12519
> 2.6.0-test7                256    2.56  2.39%   988.719   582067.86  9.28287
> 
> Random Reads ext2
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1    0.95  0.94%    12.373      114.89  0.00000
> 2.6.0-test5-mm2              1    0.90  0.88%    13.061      113.19  0.00000
> 2.6.0-test5-mm4              1    0.53  0.60%    21.903      151.35  0.00000
> 2.6.0-test6                  1    0.73  0.82%    16.053      391.29  0.00000
> 2.6.0-test6-mm4              1    0.76  0.81%    15.388      126.51  0.00000
> 2.6.0-test7                  1    0.96  0.94%    12.192      101.93  0.00000
> 
> 2.6.0-test5                  8    4.11  4.58%    19.433      137.74  0.00000
> 2.6.0-test5-mm2              8    3.94  4.03%    20.931      144.16  0.00000
> 2.6.0-test5-mm4              8    2.81  3.28%    30.086      222.27  0.00000
> 2.6.0-test6                  8    3.25  3.21%    27.598      640.41  0.00000
> 2.6.0-test6-mm4              8    3.98  3.82%    22.078      324.87  0.00000
> 2.6.0-test7                  8    3.33  3.27%    27.069      417.17  0.00000
> 
> 2.6.0-test5                 16    4.05  3.90%    42.424      236.49  0.00000
> 2.6.0-test5-mm2             16    4.09  4.31%    41.651      255.17  0.00000
> 2.6.0-test5-mm4             16    3.05  3.19%    55.847      332.10  0.00000
> 2.6.0-test6                 16    3.43  3.28%    48.836      934.02  0.00000
> 2.6.0-test6-mm4             16    3.21  2.69%    52.841      936.50  0.00000
> 2.6.0-test7                 16    4.93  4.17%    34.720      630.63  0.00000
> 
> 2.6.0-test5                 32    4.41  3.58%    70.910      516.28  0.00000
> 2.6.0-test5-mm2             32    4.23  4.17%    78.492      372.63  0.00000
> 2.6.0-test5-mm4             32    4.19  3.90%    76.001      472.90  0.00000
> 2.6.0-test6                 32    3.74  3.20%    92.407     1578.33  0.00000
> 2.6.0-test6-mm4             32    2.94  2.97%   116.663     1722.23  0.00000
> 2.6.0-test7                 32    2.62  2.36%   129.093     2248.32  0.00000
> 
> 2.6.0-test5                 64    4.47  4.00%   137.277     1077.53  0.00000
> 2.6.0-test5-mm2             64    4.79  4.53%   124.035     1144.38  0.00000
> 2.6.0-test5-mm4             64    4.13  3.81%   143.321     1265.44  0.00000
> 2.6.0-test6                 64    3.89  3.62%   166.555     2679.86  0.00000
> 2.6.0-test6-mm4             64    3.23  3.15%   200.698     3160.57  0.00000
> 2.6.0-test7                 64    3.14  2.74%   206.849     3384.30  0.00000
> 
> 2.6.0-test5                128    4.86  4.83%   243.704     2287.09  0.00000
> 2.6.0-test5-mm2            128    5.08  4.85%   228.102     2391.21  0.00000
> 2.6.0-test5-mm4            128    4.16  4.65%   270.840     2501.04  0.00000
> 2.6.0-test6                128    3.19  3.09%   389.583     6092.04  0.12601
> 2.6.0-test6-mm4            128    2.75  2.78%   439.379     6632.53  0.65524
> 2.6.0-test7                128    3.16  3.16%   388.762     5810.95  0.07560
> 
> 2.6.0-test5                256    5.51  7.00%   427.503     2511.79  0.00000
> 2.6.0-test5-mm2            256    5.23  7.39%   449.007     3169.28  0.00000
> 2.6.0-test5-mm4            256    4.48  6.76%   526.994     3618.17  0.00000
> 2.6.0-test6                256    2.01  2.44%  1156.505    14117.02 16.82291
> 2.6.0-test6-mm4            256    1.99  2.38%  1169.942    13903.51 16.56250
> 2.6.0-test7                256    2.09  2.41%  1133.624    13195.27 14.97396
> 
> Sequential and random writes on ext2 are not regressed.
> 
> ext3 has a regression similar to ext2 for sequential and random reads.
> 
> Sequential Reads ext3
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1   29.03 14.00%     0.401      745.26  0.00000
> 2.6.0-test5-mm2              1   28.05 13.82%     0.415      763.49  0.00000
> 2.6.0-test5-mm4              1   29.31 14.10%     0.397      867.89  0.00000
> 2.6.0-test6                  1   28.92 13.83%     0.402      747.59  0.00000
> 2.6.0-test6-mm4              1   28.01 13.67%     0.415     1162.40  0.00000
> 2.6.0-test7                  1   27.95 13.54%     0.416      860.38  0.00000
> 
> 2.6.0-test5                  8   14.53  8.42%     6.187     1280.16  0.00000
> 2.6.0-test5-mm2              8   14.25  8.56%     6.428     1438.12  0.00000
> 2.6.0-test5-mm4              8   14.42  8.60%     6.317     1335.28  0.00000
> 2.6.0-test6                  8   10.37  6.00%     8.791     2065.27  0.00000
> 2.6.0-test6-mm4              8    9.38  5.81%     9.765     1978.67  0.00000
> 2.6.0-test7                  8   10.25  6.02%     8.931     1977.71  0.00000
> 
> 2.6.0-test5                 16   11.38  7.09%    15.604     1387.68  0.00000
> 2.6.0-test5-mm2             16   12.84  7.97%    14.206     1246.21  0.00000
> 2.6.0-test5-mm4             16   12.89  8.00%    14.050     1790.63  0.00000
> 2.6.0-test6                 16    5.13  3.45%    34.925     3982.64  0.00000
> 2.6.0-test6-mm4             16    4.91  3.42%    37.057     4349.95  0.00000
> 2.6.0-test7                 16    5.27  3.56%    34.774     3852.71  0.00000
> 
> 2.6.0-test5                 32    8.77  6.17%    41.461     1856.08  0.00000
> 2.6.0-test5-mm2             32    8.79  6.26%    40.822     1667.85  0.00000
> 2.6.0-test5-mm4             32    8.74  6.41%    41.519     1797.28  0.00000
> 2.6.0-test6                 32    3.32  2.60%   107.252     9267.00  0.00649
> 2.6.0-test6-mm4             32    3.58  2.78%   100.364     8453.15  0.00496
> 2.6.0-test7                 32    3.39  2.65%   104.283     9029.86  0.00367
> 
> 2.6.0-test5                 64    6.29  5.30%   113.016     1947.14  0.00000
> 2.6.0-test5-mm2             64    7.58  6.00%    81.717     1722.83  0.00000
> 2.6.0-test5-mm4             64    7.63  6.07%    79.882     1709.56  0.00000
> 2.6.0-test6                 64    2.63  2.25%   265.280    18736.77  8.76222
> 2.6.0-test6-mm4             64    2.43  2.17%   277.430    17986.61  8.95066
> 2.6.0-test7                 64    2.40  2.16%   286.947    17890.59  9.59925
> 
> 2.6.0-test5                128    6.19  6.61%   216.851     1852.30  0.00000
> 2.6.0-test5-mm2            128    6.73  7.19%   191.761     2049.82  0.00000
> 2.6.0-test5-mm4            128    6.09  6.55%   209.661     1910.36  0.00000
> 2.6.0-test6                128    2.06  1.95%   640.506    32660.27 12.41284
> 2.6.0-test6-mm4            128    2.03  1.98%   645.423    30932.33 12.57601
> 2.6.0-test7                128    2.05  2.02%   639.791    30885.16 12.72559
> 
> 2.6.0-test5                256    6.15  9.88%   457.029     1880.47  0.00000
> 2.6.0-test5-mm2            256    5.51  9.06%   518.943     2002.06  0.00000
> 2.6.0-test5-mm4            256    5.30  8.81%   535.291     2103.11  0.00000
> 2.6.0-test6                256    1.85  1.78%  1293.732   230380.55 13.73657
> 2.6.0-test6-mm4            256    1.91  1.90%  1246.640   243088.10 13.26160
> 2.6.0-test7                256    1.81  1.78%  1351.071   281394.59 13.80543
> 
> Random Reads ext3
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1    0.90  0.85%    12.963      112.10  0.00000
> 2.6.0-test5-mm2              1    0.93  0.97%    12.637      125.35  0.00000
> 2.6.0-test5-mm4              1    0.85  0.95%    13.714      122.66  0.00000
> 2.6.0-test6                  1    0.73  0.71%    16.054      516.87  0.00000
> 2.6.0-test6-mm4              1    0.91  0.91%    12.907      125.50  0.00000
> 2.6.0-test7                  1    0.94  0.82%    12.443      110.45  0.00000
> 
> 2.6.0-test5                  8    3.78  4.24%    21.477      131.88  0.00000
> 2.6.0-test5-mm2              8    4.01  4.49%    20.218      124.38  0.00000
> 2.6.0-test5-mm4              8    4.43  4.63%    18.684      142.49  0.00000
> 2.6.0-test6                  8    3.55  3.11%    23.596      362.51  0.00000
> 2.6.0-test6-mm4              8    2.52  2.69%    32.927      534.18  0.00000
> 2.6.0-test7                  8    4.21  3.80%    19.881      459.00  0.00000
> 
> 2.6.0-test5                 16    4.10  3.74%    37.249      236.48  0.00000
> 2.6.0-test5-mm2             16    4.42  4.48%    37.848      208.96  0.00000
> 2.6.0-test5-mm4             16    4.25  4.51%    36.874      220.75  0.00000
> 2.6.0-test6                 16    3.27  3.13%    49.687      803.39  0.00000
> 2.6.0-test6-mm4             16    3.37  3.17%    49.112      951.18  0.00000
> 2.6.0-test7                 16    4.49  3.87%    38.468      727.87  0.00000
> 
> 2.6.0-test5                 32    4.28  4.06%    76.377      458.79  0.00000
> 2.6.0-test5-mm2             32    4.53  4.40%    71.762      466.83  0.00000
> 2.6.0-test5-mm4             32    4.48  4.23%    70.711      361.46  0.00000
> 2.6.0-test6                 32    3.35  3.06%    98.621     1728.18  0.00000
> 2.6.0-test6-mm4             32    4.13  3.54%    77.004     1321.89  0.00000
> 2.6.0-test7                 32    3.95  3.30%    79.804     1402.47  0.00000
> 
> 2.6.0-test5                 64    4.89  4.80%   134.964     1243.33  0.00000
> 2.6.0-test5-mm2             64    4.63  4.17%   108.944     1006.23  0.00000
> 2.6.0-test5-mm4             64    3.70  3.92%   132.534     1222.82  0.00000
> 2.6.0-test6                 64    3.35  2.98%   189.642     2847.96  0.00000
> 2.6.0-test6-mm4             64    3.12  3.03%   207.658     3362.27  0.00000
> 2.6.0-test7                 64    3.02  2.89%   212.838     2652.28  0.00000
> 
> 2.6.0-test5                128    5.26  5.11%   230.823     2088.49  0.00000
> 2.6.0-test5-mm2            128    4.77  4.65%   252.354     2283.06  0.00000
> 2.6.0-test5-mm4            128    4.07  4.26%   280.692     2923.29  0.00000
> 2.6.0-test6                128    2.88  3.07%   431.937     5967.29  0.12601
> 2.6.0-test6-mm4            128    3.09  3.26%   403.965     5559.31  0.02520
> 2.6.0-test7                128    3.01  2.95%   407.900     6243.88  0.10081
> 
> 2.6.0-test5                256    5.18  6.62%   463.528     3270.79  0.00000
> 2.6.0-test5-mm2            256    5.20  7.12%   452.987     3055.03  0.00000
> 2.6.0-test5-mm4            256    4.36  5.89%   539.848     3465.46  0.00000
> 2.6.0-test6                256    2.26  2.92%  1013.076    11495.98 12.47396
> 2.6.0-test6-mm4            256    2.49  2.98%   943.926    10902.18 10.72917
> 2.6.0-test7                256    2.43  3.04%   996.420    11016.00 12.08334
> 
> Sequential Writes ext3
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1   56.15 64.55%     0.170     5700.14  0.00010
> 2.6.0-test5-mm2              1   56.14 67.38%     0.173     4383.06  0.00005
> 2.6.0-test5-mm4              1   55.64 66.40%     0.173     4323.48  0.00000
> 2.6.0-test6                  1   56.15 65.22%     0.171     4578.10  0.00005
> 2.6.0-test6-mm4              1   55.81 66.36%     0.172     5623.30  0.00010
> 2.6.0-test7                  1   55.94 65.10%     0.171     4544.88  0.00000
> 
> 2.6.0-test5                  8    2.99  4.94%    23.859  1119399.58  0.08983
> 2.6.0-test5-mm2              8    3.00  5.08%    24.803   613704.08  0.09155
> 2.6.0-test5-mm4              8    2.98  5.06%    25.599   515385.61  0.09627
> 2.6.0-test6                  8    4.65  7.53%    15.776   347393.72  0.07853
> 2.6.0-test6-mm4              8    3.02  5.04%    25.421   399480.58  0.09512
> 2.6.0-test7                  8    3.12  5.12%    23.991   197721.86  0.09255
> 
> 2.6.0-test5                 16    2.58  4.55%    55.284   715963.30  0.14072
> 2.6.0-test5-mm2             16    2.93  5.09%    50.697   648332.45  0.11883
> 2.6.0-test5-mm4             16    3.21  5.62%    47.072   544422.32  0.12240
> 2.6.0-test6                 16    3.45  6.12%    39.175   424983.10  0.11596
> 2.6.0-test6-mm4             16    2.72  4.76%    54.117   429887.30  0.13976
> 2.6.0-test7                 16    2.74  4.73%    52.594   886672.08  0.13404
> 
> 2.6.0-test5                 32    2.28  4.43%   127.746   833586.10  0.20012
> 2.6.0-test5-mm2             32    2.25  4.58%   124.219  1054925.33  0.20647
> 2.6.0-test5-mm4             32    2.11  4.33%   139.662   787258.98  0.22011
> 2.6.0-test6                 32    2.80  5.35%   107.223   835055.83  0.18424
> 2.6.0-test6-mm4             32    2.28  4.26%   117.722   754407.32  0.18582
> 2.6.0-test7                 32    2.21  4.38%   128.475   829790.72  0.21805
> 
> 2.6.0-test5                 64    2.02  4.54%   265.546  1322591.44  0.40007
> 2.6.0-test5-mm2             64    2.02  4.42%   278.978  1434418.75  0.42186
> 2.6.0-test5-mm4             64    2.25  4.83%   242.598  1592337.06  0.35062
> 2.6.0-test6                 64    2.44  5.25%   222.436  1233112.15  0.38619
> 2.6.0-test6-mm4             64    1.98  4.45%   274.091  1442328.98  0.45623
> 2.6.0-test7                 64    2.09  4.37%   249.893  1456365.96  0.30308
> 
> 2.6.0-test5                128    2.21  4.96%   444.935  2758997.12  0.60168
> 2.6.0-test5-mm2            128    1.93  4.62%   568.801  3209338.67  0.84395
> 2.6.0-test5-mm4            128    1.99  4.67%   518.813  2778367.36  0.77595
> 2.6.0-test6                128    2.00  4.47%   497.636  2933240.88  0.65131
> 2.6.0-test6-mm4            128    2.03  4.71%   505.696  2839390.84  0.74044
> 2.6.0-test7                128    2.00  4.69%   491.950  2994053.88  0.75612
> 
> 2.6.0-test5                256    2.11  7.77%   941.664  4873713.39  1.83378
> 2.6.0-test5-mm2            256    1.91  6.85%  1093.820  5897453.70  1.80015
> 2.6.0-test5-mm4            256    1.97  6.74%  1017.367  5695863.45  1.77222
> 2.6.0-test6                256    1.92  6.71%  1065.957  6104195.68  1.87802
> 2.6.0-test6-mm4            256    1.93  7.02%  1038.493  5678570.32  1.92114
> 2.6.0-test7                256    1.85  7.36%  1083.836  5720266.70  1.89524
> 
> Random Writes ext3
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1    3.39  3.44%     0.399       37.37  0.00000
> 2.6.0-test5-mm2              1    3.22  3.38%     0.516       40.33  0.00000
> 2.6.0-test5-mm4              1    3.56  3.72%     0.081       10.61  0.00000
> 2.6.0-test6                  1    2.34  2.39%     1.780       24.35  0.00000
> 2.6.0-test6-mm4              1    3.00  3.19%     0.746       37.62  0.00000
> 2.6.0-test7                  1    3.05  3.06%     0.653       54.78  0.00000
> 
> 2.6.0-test5                  8    3.17  3.59%    13.525     1655.65  0.00000
> 2.6.0-test5-mm2              8    3.05  3.44%    14.349     2391.37  0.00000
> 2.6.0-test5-mm4              8    3.08  3.54%    14.093     1401.73  0.00000
> 2.6.0-test6                  8    2.79  3.15%    14.906     3957.84  0.02500
> 2.6.0-test6-mm4              8    2.88  3.06%    15.029     1283.97  0.00000
> 2.6.0-test7                  8    3.18  3.54%    14.407     1671.73  0.00000
> 
> 2.6.0-test5                 16    3.18  3.49%    26.252     1693.61  0.00000
> 2.6.0-test5-mm2             16    3.28  3.73%    25.573     3583.93  0.02500
> 2.6.0-test5-mm4             16    3.21  3.84%    25.857     6012.15  0.02500
> 2.6.0-test6                 16    2.81  2.81%    30.448     2511.41  0.00000
> 2.6.0-test6-mm4             16    3.17  4.08%    20.815     5062.41  0.02500
> 2.6.0-test7                 16    3.14  3.43%    28.987     2120.35  0.00000
> 
> 2.6.0-test5                 32    3.26  4.05%    46.822     3656.96  0.00000
> 2.6.0-test5-mm2             32    3.25  4.28%    47.088     2834.74  0.00000
> 2.6.0-test5-mm4             32    3.29  4.49%    44.765     2409.89  0.00000
> 2.6.0-test6                 32    2.67  2.91%    62.319    11807.07  0.05000
> 2.6.0-test6-mm4             32    2.98  3.57%    57.025     2250.13  0.00000
> 2.6.0-test7                 32    3.06  3.75%    50.252     9420.48  0.05000
> 
> 2.6.0-test5                 64    3.40  4.96%    46.634     2426.05  0.00000
> 2.6.0-test5-mm2             64    3.24  4.72%    93.293     7875.05  1.86492
> 2.6.0-test5-mm4             64    3.24  4.44%    79.998     5292.84  0.63004
> 2.6.0-test6                 64    2.50  3.22%   137.484     6723.01  0.25202
> 2.6.0-test6-mm4             64    2.83  3.92%    79.495     9204.46  0.68044
> 2.6.0-test7                 64    2.85  4.00%    91.255     6050.08  0.63004
> 
> 2.6.0-test5                128    3.46  5.89%   101.597     6086.33  0.80645
> 2.6.0-test5-mm2            128    3.39  5.71%   129.468     6234.78  1.28528
> 2.6.0-test5-mm4            128    3.27  5.82%   128.846     6601.87  0.65524
> 2.6.0-test6                128    2.63  4.26%   123.641     6247.05  2.09173
> 2.6.0-test6-mm4            128    2.69  4.58%   100.787     4669.66  0.00000
> 2.6.0-test7                128    2.71  4.58%    91.826     4953.82  0.00000
> 
> 2.6.0-test5                256    3.45  6.88%   210.999    10063.95  6.06770
> 2.6.0-test5-mm2            256    3.45  6.80%   263.620     8774.35  8.25520
> 2.6.0-test5-mm4            256    3.33  6.91%   209.373    10921.27  5.10416
> 2.6.0-test6                256    2.58  5.45%   353.922     7824.45  5.31250
> 2.6.0-test6-mm4            256    2.99  6.08%   152.068     6246.05  2.36979
> 2.6.0-test7                256    2.96  6.15%    69.124     3923.92  0.00000
> 
> Sequential Reads jfs
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 
> 2.6.0-test5                  1   28.93 13.16%     0.402      958.45  0.00000
> 2.6.0-test5-mm2              1   28.84 13.33%     0.403      703.16  0.00000
> 2.6.0-test5-mm4              1   29.03 13.50%     0.401      871.93  0.00000
> 2.6.0-test6-mm4              1   28.90 13.54%     0.402      724.31  0.00000
> 2.6.0-test7                  1   29.07 13.44%     0.400      677.09  0.00000
> 
> 2.6.0-test5                  8   50.81 24.01%     1.815      942.91  0.00000
> 2.6.0-test5-mm2              8   48.22 23.30%     1.918      959.38  0.00000
> 2.6.0-test5-mm4              8   46.01 21.88%     1.988      840.79  0.00000
> 2.6.0-test6-mm4              8   20.85  9.38%     4.417     2073.30  0.00000
> 2.6.0-test7                  8   20.88  9.17%     4.402     1941.85  0.00000
> 
> 2.6.0-test5                 16   37.26 17.05%     5.004      855.09  0.00000
> 2.6.0-test5-mm2             16   40.27 18.83%     4.628     1024.08  0.00000
> 2.6.0-test5-mm4             16   38.68 18.07%     4.831      820.64  0.00000
> 2.6.0-test6-mm4             16   11.10  5.05%    16.392     3579.95  0.00000
> 2.6.0-test7                 16   11.98  5.35%    15.258     3381.34  0.00000
> 
> 2.6.0-test5                 32   35.48 16.56%    10.439     1742.70  0.00000
> 2.6.0-test5-mm2             32   35.46 17.01%    10.292     1979.97  0.00000
> 2.6.0-test5-mm4             32   33.58 15.89%    11.012     1885.26  0.00000
> 2.6.0-test6-mm4             32   10.45  4.79%    34.665     6205.82  0.00743
> 2.6.0-test7                 32   10.82  4.85%    33.369     6185.50  0.00668
> 
> 2.6.0-test5                 64   36.58 17.71%    19.855     3297.14  0.00000
> 2.6.0-test5-mm2             64   34.64 17.14%    21.114     3190.91  0.00000
> 2.6.0-test5-mm4             64   34.98 17.66%    20.673     3254.24  0.00000
> 2.6.0-test6-mm4             64   10.64  4.87%    67.501    11935.89  2.07191
> 2.6.0-test7                 64   11.33  5.08%    61.240    11701.91  1.98450
> 
> 2.6.0-test5                128   11.37  8.64%   109.981     2829.31  0.00000
> 2.6.0-test5-mm2            128   17.63 11.71%    64.633     2799.09  0.00000
> 2.6.0-test5-mm4            128   15.92 11.40%    73.376     3032.50  0.00000
> 2.6.0-test6-mm4            128    2.71  2.52%   520.381    33565.98  8.79664
> 2.6.0-test7                128    3.09  2.69%   409.208    33147.84  7.51347
> 
> 2.6.0-test5                256    7.53 11.50%   348.180     2671.65  0.00000
> 2.6.0-test5-mm2            256    7.77 11.45%   319.292     2285.73  0.00000
> 2.6.0-test5-mm4            256    7.23 11.94%   364.323     2244.99  0.00000
> 2.6.0-test6-mm4            256    2.00  1.95%  1023.629   216889.13 12.88433
> 2.6.0-test7                256    1.85  1.88%  1090.929   455893.85 13.53616
> 
> Random Reads jfs
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1    0.95  0.82%    12.368       86.79  0.00000
> 2.6.0-test5-mm2              1    0.97  0.89%    12.013      108.08  0.00000
> 2.6.0-test5-mm4              1    0.99  0.99%    11.796      110.46  0.00000
> 2.6.0-test6-mm4              1    0.98  0.91%    11.987      112.94  0.00000
> 2.6.0-test7                  1    0.96  0.83%    12.187      100.56  0.00000
> 
> 2.6.0-test5                  8    4.68  4.23%    18.784      137.13  0.00000
> 2.6.0-test5-mm2              8    4.06  3.84%    21.942      145.02  0.00000
> 2.6.0-test5-mm4              8    4.01  3.80%    20.854      150.03  0.00000
> 2.6.0-test6-mm4              8    0.81  0.61%   112.779     1277.22  0.00000
> 2.6.0-test7                  8    0.82  0.66%   110.357     1250.05  0.00000
> 
> 2.6.0-test5                 16    3.98  3.12%    44.566      188.99  0.00000
> 2.6.0-test5-mm2             16    4.35  3.42%    40.240      189.97  0.00000
> 2.6.0-test5-mm4             16    4.28  3.37%    41.846      199.23  0.00000
> 2.6.0-test6-mm4             16    0.75  0.62%   231.931     2631.93  0.00000
> 2.6.0-test7                 16    0.84  0.65%   212.690     2578.59  0.00000
> 
> 2.6.0-test5                 32    4.09  2.87%    86.440      373.37  0.00000
> 2.6.0-test5-mm2             32    3.86  3.07%    91.487      443.48  0.00000
> 2.6.0-test5-mm4             32    3.88  3.07%    90.965      553.03  0.00000
> 2.6.0-test6-mm4             32    0.76  0.64%   455.886     5230.06  0.00000
> 2.6.0-test7                 32    0.71  0.57%   477.112     5188.75  0.00000
> 
> 2.6.0-test5                 64    4.04  2.88%   169.495     1553.14  0.00000
> 2.6.0-test5-mm2             64    3.84  3.25%   170.017     1413.19  0.00000
> 2.6.0-test5-mm4             64    3.99  3.37%   167.645     1628.57  0.00000
> 2.6.0-test6-mm4             64    0.70  0.61%   950.464    10451.50 26.76412
> 2.6.0-test7                 64    0.71  0.63%   923.148    10352.12 26.38609
> 
> 2.6.0-test5                128    4.32  3.83%   268.303     2322.59  0.00000
> 2.6.0-test5-mm2            128    4.18  3.95%   253.072     2451.90  0.00000
> 2.6.0-test5-mm4            128    4.05  3.95%   255.242     2232.10  0.00000
> 2.6.0-test6-mm4            128    0.76  0.77%  1626.585    26333.82 35.78629
> 2.6.0-test7                128    0.78  0.69%  1521.032    24112.21 33.84576
> 
> 2.6.0-test5                256    4.03  5.43%   585.871     3402.15  0.00000
> 2.6.0-test5-mm2            256    4.06  5.49%   556.512     3436.30  0.00000
> 2.6.0-test5-mm4            256    4.11  5.47%   543.649     2806.43  0.00000
> 2.6.0-test6-mm4            256    0.82  0.87%  2318.070    33523.86 44.84375
> 2.6.0-test7                256    0.81  0.85%  2315.106    34969.85 44.94791
> 
> Sequential Writes jfs
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1   56.47 47.37%     0.170     3391.01  0.00000
> 2.6.0-test5-mm2              1   56.47 49.42%     0.172     2901.46  0.00000
> 2.6.0-test5-mm4              1   56.40 48.44%     0.181     2578.21  0.00000
> 2.6.0-test6-mm4              1   56.28 47.99%     0.175     2442.78  0.00000
> 2.6.0-test7                  1   56.39 46.73%     0.171     3405.20  0.00000
> 
> 2.6.0-test5                  8   56.19 65.59%     1.343     8360.10  0.00100
> 2.6.0-test5-mm2              8   55.94 66.86%     1.336    17088.51  0.00162
> 2.6.0-test5-mm4              8   56.03 67.52%     1.316    12716.34  0.00105
> 2.6.0-test6-mm4              8   56.14 65.15%     1.320    13881.60  0.00100
> 2.6.0-test7                  8   56.23 64.40%     1.335    11973.67  0.00109
> 
> 2.6.0-test5                 16   55.15 67.48%     2.651    30109.56  0.03486
> 2.6.0-test5-mm2             16   55.93 68.92%     2.576    39135.68  0.03081
> 2.6.0-test5-mm4             16   54.88 69.40%     2.679    36823.40  0.03596
> 2.6.0-test6-mm4             16   54.85 65.44%     2.641    21368.63  0.04229
> 2.6.0-test7                 16   55.84 68.65%     2.635    63526.06  0.03019
> 
> 2.6.0-test5                 32   54.38 69.36%     5.204    96795.80  0.10614
> 2.6.0-test5-mm2             32   55.11 70.01%     4.989    55045.13  0.10557
> 2.6.0-test5-mm4             32   53.92 70.42%     5.099    68154.16  0.10362
> 2.6.0-test6-mm4             32   54.17 70.60%     5.074    81693.83  0.09809
> 2.6.0-test7                 32   54.75 69.14%     4.998   108095.44  0.09742
> 
> 2.6.0-test5                 64   54.58 75.03%     9.331   208382.29  0.10910
> 2.6.0-test5-mm2             64   54.44 69.22%     9.195   156878.93  0.11329
> 2.6.0-test5-mm4             64   53.35 72.64%     9.621   158954.21  0.10639
> 2.6.0-test6-mm4             64   54.54 70.28%     9.386   148888.99  0.11191
> 2.6.0-test7                 64   54.67 69.68%     8.969   189127.28  0.10605
> 
> 2.6.0-test5                128   10.39 18.85%    93.764  2075500.73  0.09913
> 2.6.0-test5-mm2            128   15.87 27.47%    61.244  1306699.59  0.08931
> 2.6.0-test5-mm4            128   16.06 28.24%    56.744  1437340.48  0.08950
> 2.6.0-test6-mm4            128   20.89 35.44%    42.971  1018802.82  0.10410
> 2.6.0-test7                128   13.63 23.07%    69.588  1685679.31  0.08650
> 
> 2.6.0-test5                256   10.65 20.80%   168.683  1900450.38  0.20290
> 2.6.0-test5-mm2            256    9.72 18.90%   202.289  2024501.68  0.13385
> 2.6.0-test5-mm4            256    8.52 17.57%   221.659  1967027.22  0.19260
> 2.6.0-test6-mm4            256    9.20 17.32%   200.496  1843020.23  0.12737
> 2.6.0-test7                256    8.85 17.47%   212.298  2080395.84  0.11768
> 
> Random Writes jfs
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1    3.90  3.48%     0.060        0.56  0.00000
> 2.6.0-test5-mm2              1    3.89  3.25%     0.061        0.28  0.00000
> 2.6.0-test5-mm4              1    3.86  3.23%     0.061        0.35  0.00000
> 2.6.0-test6-mm4              1    3.93  3.40%     0.060        0.37  0.00000
> 2.6.0-test7                  1    3.69  3.03%     0.059        0.88  0.00000
> 
> 2.6.0-test5                  8    3.86  6.54%     0.170       59.53  0.00000
> 2.6.0-test5-mm2              8    3.68  6.19%     0.170       77.58  0.00000
> 2.6.0-test5-mm4              8    3.69  6.03%     0.171       69.55  0.00000
> 2.6.0-test6-mm4              8    3.55  5.93%     0.153       34.88  0.00000
> 2.6.0-test7                  8    3.56  5.99%     0.162       52.74  0.00000
> 
> 2.6.0-test5                 16    3.71  6.47%     0.162       43.67  0.00000
> 2.6.0-test5-mm2             16    3.82  6.63%     0.166       54.86  0.00000
> 2.6.0-test5-mm4             16    3.71  6.33%     0.168       43.72  0.00000
> 2.6.0-test6-mm4             16    3.42  6.01%     0.151       14.21  0.00000
> 2.6.0-test7                 16    3.56  5.94%     0.149        4.41  0.00000
> 
> 2.6.0-test5                 32    3.71  6.48%     0.178      111.32  0.00000
> 2.6.0-test5-mm2             32    3.50  5.99%     0.148        3.16  0.00000
> 2.6.0-test5-mm4             32    3.56  6.47%     0.170       55.28  0.00000
> 2.6.0-test6-mm4             32    3.44  6.09%     0.147        4.23  0.00000
> 2.6.0-test7                 32    3.22  5.41%     0.158       59.51  0.00000
> 
> 2.6.0-test5                 64    3.50  6.28%     0.148        3.96  0.00000
> 2.6.0-test5-mm2             64    3.34  5.99%     0.160       51.30  0.00000
> 2.6.0-test5-mm4             64    3.58  6.46%     0.157        7.48  0.00000
> 2.6.0-test6-mm4             64    3.25  5.67%     0.146        3.61  0.00000
> 2.6.0-test7                 64    3.28  5.74%     0.168       55.67  0.00000
> 
> 2.6.0-test5                128    3.28  5.72%    20.122     1415.53  0.00000
> 2.6.0-test5-mm2            128    3.29  5.92%     1.297     1229.30  0.00000
> 2.6.0-test5-mm4            128    3.42  6.33%     1.499      541.00  0.00000
> 2.6.0-test6-mm4            128    3.08  5.62%     1.341     1089.07  0.00000
> 2.6.0-test7                128    3.13  5.51%     0.595      923.55  0.00000
> 
> 2.6.0-test5                256    2.80  5.18%    89.100     3075.82  0.00000
> 2.6.0-test5-mm2            256    2.98  5.76%    33.288     2094.84  0.00000
> 2.6.0-test5-mm4            256    3.19  6.47%    13.738     1648.72  0.00000
> 2.6.0-test6-mm4            256    2.65  5.49%    18.498     2632.76  0.00000
> 2.6.0-test7                256    2.74  5.45%    10.049     2382.47  0.00000
> 
> Sequential Reads reiserfs
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1   28.20 15.37%     0.413       97.43  0.00000
> 2.6.0-test5-mm2              1   30.21 16.13%     0.385     1159.97  0.00000
> 2.6.0-test5-mm4              1   28.15 15.29%     0.413      771.52  0.00000
> 2.6.0-test6                  1   28.84 15.63%     0.404      111.79  0.00000
> 2.6.0-test6-mm4              1   29.99 16.01%     0.388     1065.08  0.00000
> 2.6.0-test7                  1   29.45 15.70%     0.395      113.20  0.00000
> 
> 2.6.0-test5                  8   29.77 17.07%     2.974      854.50  0.00000
> 2.6.0-test5-mm2              8   31.14 18.03%     2.811      653.61  0.00000
> 2.6.0-test5-mm4              8   32.23 18.70%     2.727      542.20  0.00000
> 2.6.0-test6                  8   21.41 11.92%     4.275     1487.18  0.00000
> 2.6.0-test6-mm4              8   21.93 12.30%     4.185     1320.72  0.00000
> 2.6.0-test7                  8   21.40 11.78%     4.263     1261.39  0.00000
> 
> 2.6.0-test5                 16   29.99 17.01%     5.829      901.20  0.00000
> 2.6.0-test5-mm2             16   29.66 17.05%     5.954     1288.12  0.00000
> 2.6.0-test5-mm4             16   30.20 17.66%     5.872     1153.05  0.00000
> 2.6.0-test6                 16   19.46 10.99%     9.435     2515.46  0.00000
> 2.6.0-test6-mm4             16   18.92 10.48%     9.650     2572.79  0.00000
> 2.6.0-test7                 16   18.34 10.11%    10.014     2648.17  0.00000
> 
> 2.6.0-test5                 32   28.86 16.73%    12.003     1408.20  0.00000
> 2.6.0-test5-mm2             32   28.21 16.52%    12.099     1385.36  0.00000
> 2.6.0-test5-mm4             32   27.64 16.72%    12.706     1491.53  0.00000
> 2.6.0-test6                 32   17.33  9.87%    20.872     5771.49  0.00005
> 2.6.0-test6-mm4             32   18.21 10.25%    19.659     4621.02  0.00000
> 2.6.0-test7                 32   17.89  9.88%    20.121     4681.65  0.00000
> 
> 2.6.0-test5                 64   24.08 15.49%    27.936     1610.23  0.00000
> 2.6.0-test5-mm2             64   23.23 15.01%    27.942     1478.59  0.00000
> 2.6.0-test5-mm4             64   22.97 15.39%    29.020     1577.31  0.00000
> 2.6.0-test6                 64    7.62  5.28%    95.479    16071.22  3.52058
> 2.6.0-test6-mm4             64    7.89  5.41%    91.677    12720.86  3.22466
> 2.6.0-test7                 64    7.51  5.16%    96.373    14738.26  3.39680
> 
> 2.6.0-test5                128    9.77 11.49%   141.347     1501.12  0.00000
> 2.6.0-test5-mm2            128   10.81 12.13%   121.146     1501.77  0.00000
> 2.6.0-test5-mm4            128   10.08 12.01%   134.006     1616.24  0.00000
> 2.6.0-test6                128    2.71  2.80%   534.963    33603.12 10.19945
> 2.6.0-test6-mm4            128    2.76  2.91%   524.334    34758.05  9.84798
> 2.6.0-test7                128    2.66  2.72%   543.918    33661.62 10.33516
> 
> 2.6.0-test5                256    8.88 15.47%   327.283     2378.44  0.00000
> 2.6.0-test5-mm2            256    8.59 15.37%   339.514     1905.29  0.00000
> 2.6.0-test5-mm4            256    8.60 16.11%   338.642     1890.15  0.00000
> 2.6.0-test6                256    2.29  2.42%  1177.401   291951.14 11.55243
> 2.6.0-test6-mm4            256    2.32  2.50%  1158.512   343146.47 11.20739
> 2.6.0-test7                256    2.28  2.39%  1172.902   277837.90 11.42778
> 
> Random Reads reiserfs
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1    0.88  0.95%    13.375      121.48  0.00000
> 2.6.0-test5-mm2              1    1.01  1.08%    11.543      382.20  0.00000
> 2.6.0-test5-mm4              1    0.86  1.11%    13.564      123.23  0.00000
> 2.6.0-test6                  1    0.96  1.10%    12.246      127.82  0.00000
> 2.6.0-test6-mm4              1    1.02  1.16%    11.496      391.26  0.00000
> 2.6.0-test7                  1    0.99  1.04%    11.863      111.96  0.00000
> 
> 2.6.0-test5                  8    4.59  5.28%    17.392      126.60  0.00000
> 2.6.0-test5-mm2              8    4.88  5.41%    15.978      105.27  0.00000
> 2.6.0-test5-mm4              8    4.64  5.92%    17.234      119.30  0.00000
> 2.6.0-test6                  8    3.12  3.28%    27.584      373.26  0.00000
> 2.6.0-test6-mm4              8    3.80  4.12%    23.064      280.06  0.00000
> 2.6.0-test7                  8    3.35  3.69%    24.895      337.23  0.00000
> 
> 2.6.0-test5                 16    4.80  4.33%    34.457      189.80  0.00000
> 2.6.0-test5-mm2             16    5.27  5.28%    30.882      160.61  0.00000
> 2.6.0-test5-mm4             16    5.22  5.44%    32.357      494.99  0.00000
> 2.6.0-test6                 16    3.93  4.12%    43.768      728.55  0.00000
> 2.6.0-test6-mm4             16    3.96  3.79%    41.680      775.69  0.00000
> 2.6.0-test7                 16    3.96  3.85%    42.797      705.06  0.00000
> 
> 2.6.0-test5                 32    5.45  5.07%    59.014      323.30  0.00000
> 2.6.0-test5-mm2             32    5.65  5.45%    57.673      313.96  0.00000
> 2.6.0-test5-mm4             32    4.47  4.84%    75.004      455.38  0.00000
> 2.6.0-test6                 32    4.39  4.58%    77.879     1232.67  0.00000
> 2.6.0-test6-mm4             32    4.15  4.34%    80.355     1229.75  0.00000
> 2.6.0-test7                 32    4.88  4.59%    67.103     1033.43  0.00000
> 
> 2.6.0-test5                 64    5.49  5.42%   108.692      991.66  0.00000
> 2.6.0-test5-mm2             64    5.10  5.60%   113.940      914.35  0.00000
> 2.6.0-test5-mm4             64    4.72  5.35%   126.505     1182.30  0.00000
> 2.6.0-test6                 64    3.90  4.03%   165.855     2554.36  0.00000
> 2.6.0-test6-mm4             64    4.54  4.72%   140.927     2060.27  0.00000
> 2.6.0-test7                 64    4.95  4.73%   129.947     2011.81  0.00000
> 
> 2.6.0-test5                128    5.48  5.99%   221.089     2074.91  0.00000
> 2.6.0-test5-mm2            128    5.71  6.49%   208.548     1781.53  0.00000
> 2.6.0-test5-mm4            128    4.94  5.94%   240.781     2090.92  0.00000
> 2.6.0-test6                128    3.55  4.55%   367.628     5086.89  0.00000
> 2.6.0-test6-mm4            128    4.15  4.50%   314.516     3914.52  0.00000
> 2.6.0-test7                128    3.98  4.56%   328.297     4300.67  0.00000
> 
> 2.6.0-test5                256    5.39  7.63%   455.466     2448.36  0.00000
> 2.6.0-test5-mm2            256    5.85  8.92%   415.957     2773.47  0.00000
> 2.6.0-test5-mm4            256    4.88  8.30%   498.235     2859.83  0.00000
> 2.6.0-test6                256    2.88  4.24%   861.877     8947.59  7.13542
> 2.6.0-test6-mm4            256    3.28  4.52%   752.088     7585.14  3.09895
> 2.6.0-test7                256    3.11  4.19%   794.027     8045.64  3.82812
> 
> Sequential Writes reiserf
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1   55.30 90.83%     0.200     1317.87  0.00000
> 2.6.0-test5-mm2              1   55.08 92.70%     0.201     1512.52  0.00000
> 2.6.0-test5-mm4              1   54.62 92.30%     0.202     2146.88  0.00000
> 2.6.0-test6                  1   55.09 91.14%     0.201     1243.21  0.00000
> 2.6.0-test6-mm4              1   54.46 91.70%     0.202     1725.38  0.00000
> 2.6.0-test7                  1   55.08 91.27%     0.200     2381.55  0.00000
> 
> 2.6.0-test5                  8   27.22 99.99%     2.834    62892.07  0.02932
> 2.6.0-test5-mm2              8   27.84 100.4%     2.721    20683.58  0.03290
> 2.6.0-test5-mm4              8   28.18 96.26%     2.667    24253.31  0.03237
> 2.6.0-test6                  8   28.88 102.9%     2.572    28097.64  0.03533
> 2.6.0-test6-mm4              8   26.57 92.64%     2.855    24274.27  0.05517
> 2.6.0-test7                  8   28.60 104.5%     2.571    19829.03  0.02890
> 
> 2.6.0-test5                 16   30.58 102.6%     4.745    29788.03  0.07806
> 2.6.0-test5-mm2             16   28.94 101.7%     5.051    37491.58  0.07902
> 2.6.0-test5-mm4             16   28.45 97.86%     4.971    28952.01  0.08851
> 2.6.0-test6                 16   27.13 98.05%     5.414    58605.69  0.08331
> 2.6.0-test6-mm4             16   27.04 89.13%     5.347    24565.13  0.08516
> 2.6.0-test7                 16   29.19 98.45%     5.031    29653.75  0.08398
> 
> 2.6.0-test5                 32   27.88 104.6%    10.012    44186.44  0.11630
> 2.6.0-test5-mm2             32   27.15 104.7%     9.911    47326.17  0.11076
> 2.6.0-test5-mm4             32   27.57 102.6%     9.820    88173.93  0.10858
> 2.6.0-test6                 32   27.80 100.8%     9.951    72577.05  0.11445
> 2.6.0-test6-mm4             32   25.80 97.74%    10.563    65400.15  0.11706
> 2.6.0-test7                 32   28.65 105.3%     9.613    44890.58  0.11087
> 
> 2.6.0-test5                 64   27.12 109.5%    19.091    93664.16  0.15073
> 2.6.0-test5-mm2             64   25.80 117.2%    20.100    73309.25  0.18029
> 2.6.0-test5-mm4             64   25.64 112.9%    19.408    76139.55  0.16613
> 2.6.0-test6                 64   26.30 112.0%    19.162    93011.66  0.16308
> 2.6.0-test6-mm4             64   24.15 104.3%    21.613    86562.64  0.18483
> 2.6.0-test7                 64   25.10 107.7%    20.311    80093.32  0.16470
> 
> 2.6.0-test5                128   23.52 119.4%    42.286   116297.76  0.39005
> 2.6.0-test5-mm2            128   24.88 127.8%    39.106   139584.40  0.35658
> 2.6.0-test5-mm4            128   24.28 127.9%    39.111   159921.42  0.37918
> 2.6.0-test6                128   24.79 122.4%    41.723   127648.00  0.39692
> 2.6.0-test6-mm4            128   24.48 125.2%    39.716   128936.67  0.38275
> 2.6.0-test7                128   24.29 122.0%    40.692   152191.29  0.35692
> 
> 2.6.0-test5                256   19.14 163.4%   112.557   239944.32  1.00599
> 2.6.0-test5-mm2            256   23.21 153.9%    84.422   200276.08  0.75154
> 2.6.0-test5-mm4            256   23.45 161.4%    86.423   216520.39  0.71788
> 2.6.0-test6                256   23.14 168.2%    83.087   247571.24  0.75640
> 2.6.0-test6-mm4            256   20.87 136.1%    74.033   177291.47  0.66229
> 2.6.0-test7                256   23.11 142.0%    79.651   233921.91  0.66109
> 
> Random Writes reiserfs
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1    2.41  2.77%     1.925      282.65  0.00000
> 2.6.0-test5-mm2              1    3.71  4.05%     0.222       18.72  0.00000
> 2.6.0-test5-mm4              1    2.29  2.87%     2.191      299.62  0.00000
> 2.6.0-test6                  1    2.89  3.32%     1.040       91.48  0.00000
> 2.6.0-test6-mm4              1    3.60  3.66%     0.228       24.13  0.00000
> 2.6.0-test7                  1    3.58  3.58%     0.347       30.23  0.00000
> 
> 2.6.0-test5                  8    3.50  4.12%    11.316     1348.95  0.00000
> 2.6.0-test5-mm2              8    3.74  5.07%    10.135     1133.62  0.00000
> 2.6.0-test5-mm4              8    3.54  4.73%    10.168     1285.63  0.00000
> 2.6.0-test6                  8    3.26  4.02%     9.287     1182.99  0.00000
> 2.6.0-test6-mm4              8    3.39  4.53%     5.611     1106.14  0.00000
> 2.6.0-test7                  8    3.29  3.77%    10.518     1191.14  0.00000
> 
> 2.6.0-test5                 16    3.44  4.05%    22.370     4353.76  0.10000
> 2.6.0-test5-mm2             16    3.54  3.72%    22.545     1478.64  0.00000
> 2.6.0-test5-mm4             16    3.61  4.71%    20.853     1777.27  0.00000
> 2.6.0-test6                 16    3.60  4.11%    17.724     1421.63  0.00000
> 2.6.0-test6-mm4             16    3.37  3.72%    23.097     1422.10  0.00000
> 2.6.0-test7                 16    3.46  4.40%    16.822     4822.00  0.02500
> 
> 2.6.0-test5                 32    3.68  4.57%    22.011     1260.71  0.00000
> 2.6.0-test5-mm2             32    3.60  4.50%    26.561     1836.41  0.00000
> 2.6.0-test5-mm4             32    3.55  4.32%    33.055     1460.77  0.00000
> 2.6.0-test6                 32    3.41  4.07%    23.498     2013.53  0.00000
> 2.6.0-test6-mm4             32    3.42  4.36%    13.875     1214.49  0.00000
> 2.6.0-test7                 32    3.42  4.04%    26.600     3437.82  0.02500
> 
> 2.6.0-test5                 64    3.62  5.09%    37.320     8700.40  0.90725
> 2.6.0-test5-mm2             64    3.81  6.11%    18.225     1814.21  0.00000
> 2.6.0-test5-mm4             64    3.37  5.09%    43.372     3280.58  0.00000
> 2.6.0-test6                 64    2.96  3.93%    73.537     4484.06  0.00000
> 2.6.0-test6-mm4             64    3.29  4.98%    22.802     2206.78  0.00000
> 2.6.0-test7                 64    3.42  5.00%    27.782     2453.44  0.00000
> 
> 2.6.0-test5                128    3.73  5.92%    48.524     7848.01  1.10887
> 2.6.0-test5-mm2            128    3.39  5.91%    36.168     5651.07  0.25202
> 2.6.0-test5-mm4            128    3.55  6.74%    70.991     6856.35  1.94052
> 2.6.0-test6                128    3.16  5.58%    43.277     5286.77  0.05040
> 2.6.0-test6-mm4            128    3.17  5.25%    43.818     5534.13  0.30242
> 2.6.0-test7                128    3.26  5.26%    39.051     3037.10  0.00000
> 
> 2.6.0-test5                256    3.96  8.27%     8.806      963.34  0.00000
> 2.6.0-test5-mm2            256    3.67  7.17%    20.000     1575.35  0.00000
> 2.6.0-test5-mm4            256    3.36  6.99%   114.310     4266.84  1.61458
> 2.6.0-test6                256    3.10  6.69%   132.312     8228.26  3.30729
> 2.6.0-test6-mm4            256    3.21  6.79%    88.532     8631.04  1.79687
> 2.6.0-test7                256    3.23  6.61%    48.643     6005.85  0.75521
> 
> Sequential Reads xfs
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1   30.46 14.80%     0.382      718.97  0.00000
> 2.6.0-test5-mm2              1   28.43 14.31%     0.409      729.27  0.00000
> 2.6.0-test5-mm4              1   29.25 14.50%     0.398      751.10  0.00000
> 2.6.0-test6                  1   29.25 14.52%     0.398      735.45  0.00000
> 2.6.0-test6-mm4              1   28.68 14.60%     0.406      924.52  0.00000
> 2.6.0-test7                  1   29.20 14.46%     0.398      727.21  0.00000
> 
> 2.6.0-test5                  8   55.34 28.06%     1.652      831.66  0.00000
> 2.6.0-test5-mm2              8   54.19 28.05%     1.707      852.34  0.00000
> 2.6.0-test5-mm4              8   55.49 28.38%     1.591     1165.42  0.00000
> 2.6.0-test6                  8   25.17 11.78%     3.562     1987.22  0.00000
> 2.6.0-test6-mm4              8   23.25 11.32%     3.989     1872.56  0.00000
> 2.6.0-test7                  8   25.91 12.19%     3.443     1894.36  0.00000
> 
> 2.6.0-test5                 16   47.04 23.31%     3.903      846.86  0.00000
> 2.6.0-test5-mm2             16   45.50 23.01%     4.086     1116.23  0.00000
> 2.6.0-test5-mm4             16   47.00 23.78%     3.827      822.79  0.00000
> 2.6.0-test6                 16   15.07  7.14%    12.090     3088.51  0.00000
> 2.6.0-test6-mm4             16   12.58  6.23%    14.655     3469.24  0.00000
> 2.6.0-test7                 16   14.79  7.03%    11.808     3178.50  0.00000
> 
> 2.6.0-test5                 32   44.85 22.40%     7.699     1225.63  0.00000
> 2.6.0-test5-mm2             32   45.89 23.39%     7.624     1173.54  0.00000
> 2.6.0-test5-mm4             32   44.18 22.68%     7.940     1051.95  0.00000
> 2.6.0-test6                 32   14.80  7.02%    24.021     5758.07  0.00005
> 2.6.0-test6-mm4             32   14.18  6.98%    25.241     5709.74  0.00000
> 2.6.0-test7                 32   12.46  6.03%    29.086     5836.49  0.00029
> 
> 2.6.0-test5                 64   42.08 21.45%    16.269     1897.20  0.00000
> 2.6.0-test5-mm2             64   42.36 21.75%    15.948     1925.10  0.00000
> 2.6.0-test5-mm4             64   43.77 22.73%    15.396     1963.29  0.00000
> 2.6.0-test6                 64   12.50  5.98%    53.864    10887.40  1.74727
> 2.6.0-test6-mm4             64   12.37  6.19%    55.962    10690.33  1.82176
> 2.6.0-test7                 64   12.25  5.95%    56.211    10899.32  1.78017
> 
> 2.6.0-test5                128   15.41 12.23%    87.001     2911.16  0.00000
> 2.6.0-test5-mm2            128   20.45 15.21%    61.549     2516.86  0.00000
> 2.6.0-test5-mm4            128   18.55 14.38%    68.572     2527.97  0.00000
> 2.6.0-test6                128    5.20  4.07%   251.782    18918.92  4.34985
> 2.6.0-test6-mm4            128    4.47  3.56%   290.774    19729.60  5.13649
> 2.6.0-test7                128    3.50  3.19%   385.879    20057.71  6.42790
> 
> 2.6.0-test5                256   10.33 13.76%   225.449     6314.51  0.00167
> 2.6.0-test5-mm2            256   10.58 14.19%   206.025     9145.94  0.00219
> 2.6.0-test5-mm4            256   11.64 15.40%   175.294    11398.84  0.00677
> 2.6.0-test6                256    3.84  3.42%   539.606   972621.75  5.75909
> 2.6.0-test6-mm4            256    3.92  3.35%   527.548   867696.31  5.66021
> 2.6.0-test7                256    3.99  3.52%   511.822   865743.62  5.56827
> 
> Random Reads xfs
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1    1.03  0.94%    11.357      107.82  0.00000
> 2.6.0-test5-mm2              1    1.03  1.02%    11.325      117.87  0.00000
> 2.6.0-test5-mm4              1    1.04  1.03%    11.261      107.89  0.00000
> 2.6.0-test6                  1    1.03  0.92%    11.386      109.39  0.00000
> 2.6.0-test6-mm4              1    1.00  1.04%    11.659      119.70  0.00000
> 2.6.0-test7                  1    1.05  0.92%    11.161       96.11  0.00000
> 
> 2.6.0-test5                  8    5.42  5.19%    16.257      124.33  0.00000
> 2.6.0-test5-mm2              8    5.36  5.79%    16.591      120.16  0.00000
> 2.6.0-test5-mm4              8    4.67  4.80%    16.448      104.77  0.00000
> 2.6.0-test6                  8    2.08  1.68%    38.061      775.61  0.00000
> 2.6.0-test6-mm4              8    2.16  1.95%    37.981      686.42  0.00000
> 2.6.0-test7                  8    1.88  1.64%    42.173      683.55  0.00000
> 
> 2.6.0-test5                 16    5.20  4.62%    34.058      146.53  0.00000
> 2.6.0-test5-mm2             16    5.21  4.88%    34.306      175.88  0.00000
> 2.6.0-test5-mm4             16    5.18  4.84%    32.751      135.47  0.00000
> 2.6.0-test6                 16    1.86  1.44%    85.898     1244.64  0.00000
> 2.6.0-test6-mm4             16    1.95  1.64%    80.971     1175.91  0.00000
> 2.6.0-test7                 16    2.35  1.98%    67.271     1001.72  0.00000
> 
> 2.6.0-test5                 32    4.91  3.98%    68.586      265.91  0.00000
> 2.6.0-test5-mm2             32    5.43  4.74%    63.283      279.95  0.00000
> 2.6.0-test5-mm4             32    4.69  4.28%    68.452      638.24  0.00000
> 2.6.0-test6                 32    2.92  2.41%   113.627     1626.83  0.00000
> 2.6.0-test6-mm4             32    3.14  2.92%   105.714     1451.06  0.00000
> 2.6.0-test7                 32    2.43  1.91%   134.936     1864.81  0.00000
> 
> 2.6.0-test5                 64    4.76  4.02%   136.805      879.54  0.00000
> 2.6.0-test5-mm2             64    4.66  4.68%   136.498      871.90  0.00000
> 2.6.0-test5-mm4             64    4.81  4.75%   131.041      994.54  0.00000
> 2.6.0-test6                 64    1.45  1.25%   425.661     5262.95  0.02520
> 2.6.0-test6-mm4             64    1.66  1.40%   382.569     4759.16  0.00000
> 2.6.0-test7                 64    1.41  1.21%   444.027     5820.44  0.30242
> 
> 2.6.0-test5                128    4.70  4.50%   248.562     1396.77  0.00000
> 2.6.0-test5-mm2            128    5.29  5.72%   206.313     1395.33  0.00000
> 2.6.0-test5-mm4            128    5.16  4.95%   209.001     1485.37  0.00000
> 2.6.0-test6                128    1.65  1.57%   753.959     9498.63  9.02218
> 2.6.0-test6-mm4            128    1.47  1.43%   833.261     9951.50 11.54233
> 2.6.0-test7                128    1.23  1.06%   969.453    10793.66 18.04436
> 
> 2.6.0-test5                256    4.82  6.71%   469.459     2140.97  0.00000
> 2.6.0-test5-mm2            256    4.60  6.06%   477.682     1934.27  0.00000
> 2.6.0-test5-mm4            256    5.01  6.60%   402.622     1881.24  0.00000
> 2.6.0-test6                256    0.90  0.99%  2204.619    32794.62 42.68229
> 2.6.0-test6-mm4            256    0.88  0.98%  2165.705    33928.64 41.09376
> 2.6.0-test7                256    0.92  1.05%  2113.981    33637.27 39.11458
> 
> Sequential Writes xfs
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1   56.44 46.30%     0.182    68379.68  0.00030
> 2.6.0-test5-mm2              1   56.33 47.39%     0.186    68474.74  0.00030
> 2.6.0-test5-mm4              1   56.30 46.74%     0.182    68268.99  0.00030
> 2.6.0-test6                  1   56.24 46.08%     0.177    68501.77  0.00030
> 2.6.0-test6-mm4              1   56.17 47.57%     0.175    68404.17  0.00030
> 2.6.0-test7                  1   56.26 46.03%     0.171    68451.18  0.00030
> 
> 2.6.0-test5                  8   57.54 65.47%     0.991   166706.97  0.00424
> 2.6.0-test5-mm2              8   57.70 70.44%     1.099   211950.97  0.00420
> 2.6.0-test5-mm4              8   57.23 66.09%     1.042   205671.61  0.00415
> 2.6.0-test6                  8   57.19 69.31%     1.058   185132.73  0.00410
> 2.6.0-test6-mm4              8   57.46 66.82%     1.100   178544.18  0.00486
> 2.6.0-test7                  8   57.24 64.93%     1.096   207662.87  0.00481
> 
> 2.6.0-test5                 16   57.26 71.20%     2.069   186551.11  0.01030
> 2.6.0-test5-mm2             16   57.26 72.02%     2.079   187786.90  0.01059
> 2.6.0-test5-mm4             16   57.38 68.91%     1.999   193908.75  0.00958
> 2.6.0-test6                 16   57.24 70.72%     2.072   185124.29  0.01092
> 2.6.0-test6-mm4             16   57.37 68.99%     2.080   208981.25  0.01045
> 2.6.0-test7                 16   57.17 68.79%     2.051   181396.29  0.01135
> 
> 2.6.0-test5                 32   57.29 75.00%     4.076   211670.36  0.01840
> 2.6.0-test5-mm2             32   57.34 73.21%     3.768   213556.33  0.01560
> 2.6.0-test5-mm4             32   57.27 75.59%     4.007   199887.88  0.01708
> 2.6.0-test6                 32   57.12 73.89%     4.007   198273.60  0.01750
> 2.6.0-test6-mm4             32   57.04 73.27%     4.034   181071.17  0.01688
> 2.6.0-test7                 32   57.04 72.85%     4.066   193571.15  0.02007
> 
> 2.6.0-test5                 64   57.13 83.01%     8.186   336901.61  0.03567
> 2.6.0-test5-mm2             64   56.93 83.63%     8.062   340355.98  0.03066
> 2.6.0-test5-mm4             64   56.33 82.10%     7.834   325305.64  0.02895
> 2.6.0-test6                 64   56.48 80.10%     7.829   306781.15  0.03157
> 2.6.0-test6-mm4             64   56.87 81.05%     7.808   257430.27  0.03662
> 2.6.0-test7                 64   56.54 77.01%     7.746   347269.10  0.02880
> 
> 2.6.0-test5                128   56.58 86.45%    15.451   330514.59  0.06585
> 2.6.0-test5-mm2            128   56.78 82.31%    14.164   364774.73  0.05298
> 2.6.0-test5-mm4            128   55.57 82.39%    14.088   350137.61  0.05364
> 2.6.0-test6                128   56.08 81.22%    13.917   377412.59  0.04669
> 2.6.0-test6-mm4            128   56.43 79.05%    14.332   357252.17  0.05203
> 2.6.0-test7                128   56.28 77.14%    14.465   351042.94  0.05546
> 
> 2.6.0-test5                256   56.13 83.69%    19.836   383235.17  0.07386
> 2.6.0-test5-mm2            256   56.19 80.90%    16.880   294598.89  0.07434
> 2.6.0-test5-mm4            256   56.09 87.58%    13.343   308894.90  0.05746
> 2.6.0-test6                256   55.39 80.57%    18.269   350145.38  0.07200
> 2.6.0-test6-mm4            256   55.67 81.98%    17.338   337290.04  0.06890
> 2.6.0-test7                256   55.49 82.29%    15.776   351541.33  0.06071
> 
> Random Writes xfs
>                            Num                    Avg       Maximum     Lat%
> Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
> -------------------------  ---  --------------------------------------------
> 2.6.0-test5                  1    3.80  3.61%     0.074        0.46  0.00000
> 2.6.0-test5-mm2              1    3.86  3.77%     0.077        0.42  0.00000
> 2.6.0-test5-mm4              1    3.77  3.65%     0.078        1.42  0.00000
> 2.6.0-test6                  1    3.76  3.53%     0.075        1.43  0.00000
> 2.6.0-test6-mm4              1    3.71  3.64%     0.078        0.68  0.00000
> 2.6.0-test7                  1    3.76  3.53%     0.075        0.87  0.00000
> 
> 2.6.0-test5                  8    3.97  6.78%     0.202       95.59  0.00000
> 2.6.0-test5-mm2              8    3.91  6.75%     0.181       75.97  0.00000
> 2.6.0-test5-mm4              8    3.87  6.57%     0.191       81.43  0.00000
> 2.6.0-test6                  8    3.86  6.63%     0.165       25.62  0.00000
> 2.6.0-test6-mm4              8    3.82  6.33%     0.166       39.50  0.00000
> 2.6.0-test7                  8    3.92  6.44%     0.181       74.95  0.00000
> 
> 2.6.0-test5                 16    3.95  6.81%     0.168       32.20  0.00000
> 2.6.0-test5-mm2             16    3.94  6.99%     0.175       48.13  0.00000
> 2.6.0-test5-mm4             16    3.82  6.53%     0.186       87.83  0.00000
> 2.6.0-test6                 16    3.76  6.58%     0.168       25.32  0.00000
> 2.6.0-test6-mm4             16    3.81  6.62%     0.193       83.97  0.00000
> 2.6.0-test7                 16    3.89  6.57%     0.181       92.60  0.00000
> 
> 2.6.0-test5                 32    3.85  6.64%     0.159        3.38  0.00000
> 2.6.0-test5-mm2             32    3.80  6.75%     0.175       59.86  0.00000
> 2.6.0-test5-mm4             32    3.72  6.60%     0.180       55.04  0.00000
> 2.6.0-test6                 32    3.80  6.73%     0.175       56.73  0.00000
> 2.6.0-test6-mm4             32    3.81  6.61%     0.171       55.65  0.00000
> 2.6.0-test7                 32    3.60  5.97%     0.155        3.09  0.00000
> 
> 2.6.0-test5                 64    3.68  6.56%     0.161        3.90  0.00000
> 2.6.0-test5-mm2             64    3.60  6.51%     0.163        6.67  0.00000
> 2.6.0-test5-mm4             64    3.71  6.58%     0.166        5.44  0.00000
> 2.6.0-test6                 64    3.53  6.32%     0.160        4.14  0.00000
> 2.6.0-test6-mm4             64    3.54  6.21%     0.185      106.22  0.00000
> 2.6.0-test7                 64    3.61  6.15%     0.156        4.34  0.00000
> 
> 2.6.0-test5                128    3.32  6.23%     0.166        4.42  0.00000
> 2.6.0-test5-mm2            128    3.64  6.70%     0.163        2.98  0.00000
> 2.6.0-test5-mm4            128    3.71  6.85%     0.169        5.37  0.00000
> 2.6.0-test6                128    3.44  6.25%     0.174       63.35  0.00000
> 2.6.0-test6-mm4            128    3.49  6.36%     0.174       64.77  0.00000
> 2.6.0-test7                128    3.43  6.15%     0.159        4.05  0.00000
> 
> 2.6.0-test5                256    3.35  6.56%     0.161        3.65  0.00000
> 2.6.0-test5-mm2            256    3.29  6.41%     0.164        3.68  0.00000
> 2.6.0-test5-mm4            256    3.27  6.35%     0.166        4.84  0.00000
> 2.6.0-test6                256    2.97  5.80%     0.171       47.92  0.00000
> 2.6.0-test6-mm4            256    3.06  5.93%     0.159        3.02  0.00000
> 2.6.0-test7                256    3.11  5.88%     0.159        3.91  0.00000
> 
> 
> dbench did not regress.  
> 
> dbench-1.3 reiserfs 192 processes     Average		+/- MB/sec
> 2.6.0-test7                             77.3		   7.1
> 2.6.0-test6-mm4                         76.6		   7.2
> 2.6.0-test5-mm2                         76.1		   6.8
> 2.6.0-test6                             73.0		   5.1
> 2.6.0-test5-mm4                         72.5		   6.6
> 2.6.0-test5                             65.0		   1.2
> 
> dbench-1.3 reiserfs 64 processes      Average		+/- MB/sec
> 2.6.0-test5-mm4                        111.4		   9.1
> 2.6.0-test6-mm4                        109.6		   7.5
> 2.6.0-test6                            108.8		   4.2
> 2.6.0-test5-mm2                        104.4		   8.1
> 2.6.0-test7                            102.3		   8.1
> 2.6.0-test5                            100.6		   4.7
> 
> dbench-1.3 xfs 192 processes          Average		+/- MB/sec
> 2.6.0-test5                            112.5		   4.0
> 2.6.0-test5-mm2                        110.8		   1.6
> 2.6.0-test6-mm4                        110.7		   3.4
> 2.6.0-test5-mm4                        109.7		   2.0
> 2.6.0-test7                            109.0		   5.1
> 2.6.0-test6                            108.5		   6.1
> 
> dbench-1.3 xfs 64 processes           Average		+/- MB/sec
> 2.6.0-test5                            143.7		   0.9
> 2.6.0-test7                            143.6		   2.1
> 2.6.0-test5-mm2                        141.5		   1.6
> 2.6.0-test6                            141.4		   0.5
> 2.6.0-test6-mm4                        141.2		   1.5
> 2.6.0-test5-mm4                        140.0		   1.4
> 
> dbench-1.3 ext2 192 processes         Average		+/- MB/sec
> 2.6.0-test6                            209.5		  11.8
> 2.6.0-test7                            205.2		  20.8
> 2.6.0-test5-mm4                        200.8		   6.2
> 2.6.0-test5-mm2                        200.0		  10.3
> 2.6.0-test5                            198.1		  19.5
> 2.6.0-test6-mm4                        193.4		  12.0
> 
> dbench-1.3 ext2 64 processes          Average		+/- MB/sec
> 2.6.0-test7                            256.8		   0.6
> 2.6.0-test6                            256.7		   1.6
> 2.6.0-test5                            256.2		   1.6
> 2.6.0-test5-mm4                        255.5		   1.4
> 2.6.0-test5-mm2                        255.3		   0.9
> 2.6.0-test6-mm4                        255.3		   1.7
> 
> dbench-1.3 jfs 192 processes          Average		+/- MB/sec
> 2.6.0-test5-mm4                         11.3		   0.5
> 2.6.0-test7                             11.1		   0.2
> 2.6.0-test6-mm4                         11.0		   0.3
> 
> dbench-1.3 jfs 64 processes           Average		+/- MB/sec
> 2.6.0-test5-mm2                         20.0		   2.9
> 2.6.0-test5                             19.5		   2.4
> 2.6.0-test5-mm4                         18.6		   2.4
> 2.6.0-test7                             18.5		   1.1
> 2.6.0-test6-mm4                         18.1		   1.6
> 
> dbench-1.3 ext3 192 processes         Average		+/- MB/sec
> 2.6.0-test7                            110.8		  23.1
> 2.6.0-test5-mm2                        107.5		  39.7
> 2.6.0-test6                            103.9		  39.0
> 2.6.0-test6-mm4                         99.7		  38.7
> 2.6.0-test5-mm4                         82.6		  35.5
> 2.6.0-test5                             73.6		  15.1
> 
> dbench-1.3 ext3 64 processes          Average		+/- MB/sec
> 2.6.0-test7                            177.9		   3.4
> 2.6.0-test6                            175.1		   3.6
> 2.6.0-test5                            174.5		   4.2
> 2.6.0-test6-mm4                        174.2		   2.5
> 2.6.0-test5-mm2                        172.8		   4.0
> 2.6.0-test5-mm4                        172.6		   2.3
> 
> 
> -- 
> Randy Hron
> http://home.earthlink.net/~rwhron/kernel/bigbox.html
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [BENCHMARK] I/O regression after 2.6.0-test5
  2003-10-20  0:37 rwhron
@ 2003-10-20  1:35 ` Nick Piggin
  2003-10-20  4:52   ` Andrew Morton
  2003-10-20  4:51 ` Dave Olien
  2003-10-24 13:52 ` Magnus Andersson
  2 siblings, 1 reply; 25+ messages in thread
From: Nick Piggin @ 2003-10-20  1:35 UTC (permalink / raw)
  To: rwhron; +Cc: linux-kernel, akpm



rwhron@earthlink.net wrote:

>There was about a 50% regression in jobs/minute in AIM7
>database workload on quad P3 Xeon.  The CPU time has not
>gone up, so the extra run time is coming from something
>else.  (I/O or I/O scheduler?)
>
>tiobench sequential reads has a significant regression too.
>
>Regression appears unrelated to filesystem type.
>
>dbench was not affected.
>
>The AIM7 was run on ext2.
>

Yeah I'd say its all due to the IO scheduler. There is a problem
I'm thinking about how to fix - its the likely cause of this too.



^ permalink raw reply	[flat|nested] 25+ messages in thread

* [BENCHMARK] I/O regression after 2.6.0-test5
@ 2003-10-20  0:37 rwhron
  2003-10-20  1:35 ` Nick Piggin
                   ` (2 more replies)
  0 siblings, 3 replies; 25+ messages in thread
From: rwhron @ 2003-10-20  0:37 UTC (permalink / raw)
  To: linux-kernel; +Cc: akpm

There was about a 50% regression in jobs/minute in AIM7
database workload on quad P3 Xeon.  The CPU time has not
gone up, so the extra run time is coming from something
else.  (I/O or I/O scheduler?)

tiobench sequential reads has a significant regression too.

Regression appears unrelated to filesystem type.

dbench was not affected.

The AIM7 was run on ext2.

AIM7 dbase workload
kernel                   Tasks   Jobs/Min       Real    CPU
2.6.0-test5-mm2          32	609.3	       311.9	139.5
2.6.0-test5-mm4          32	595.2	       319.4	131.7
2.6.0-test6-mm4          32	590.8	       321.8	130.7
2.6.0-test5              32	590.2	       322.0	131.9
2.6.0-test6              32	320.6	       592.9	129.1
2.6.0-test7              32	290.3	       654.8	120.9

2.6.0-test5-mm2          64	786.2	       483.5	253.2
2.6.0-test6-mm4          64	779.4	       487.8	253.8
2.6.0-test5              64	774.3	       490.9	256.6
2.6.0-test5-mm4          64	772.0	       492.4	253.7
2.6.0-test6              64	430.5	       883.1	247.6
2.6.0-test7              64	402.6	       944.2	231.5

2.6.0-test5-mm2          96	869.2	       656.0	382.2
2.6.0-test6-mm4          96	867.6	       657.2	387.4
2.6.0-test5-mm4          96	863.1	       660.7	385.2
2.6.0-test5              96	859.0	       663.9	389.5
2.6.0-test6              96	478.5	       1191.7	368.1
2.6.0-test7              96	453.7	       1256.9	346.8

2.6.0-test5-mm2          128	929.4	       818.0	526.1
2.6.0-test6-mm4          128	922.6	       824.1	523.3
2.6.0-test5-mm4          128	916.6	       829.5	509.1
2.6.0-test5              128	910.3	       835.2	528.6
2.6.0-test7              128	500.5	       1519.1	461.3
2.6.0-test6              128	500.2	       1520.1	497.4

2.6.0-test5-mm2          160	962.5	       987.4	674.6
2.6.0-test5-mm4          160	952.7	       997.6	655.5
2.6.0-test6-mm4          160	950.0	       1000.5	659.5
2.6.0-test5              160	741.5	       1281.7	644.8
2.6.0-test7              160	518.5	       1833.1	589.6
2.6.0-test6              160	479.6	       1981.8	618.9

2.6.0-test5-mm2          192	993.9	       1147.5	810.0
2.6.0-test5-mm4          192	982.7	       1160.5	788.2
2.6.0-test6-mm4          192	978.8	       1165.2	808.4
2.6.0-test5              192	840.0	       1357.8	776.0
2.6.0-test7              192	537.9	       2120.2	709.3
2.6.0-test6              192	494.3	       2307.1	748.2

2.6.0-test5-mm2          224	1011.2	       1315.8	943.5
2.6.0-test5-mm4          224	1004.7	       1324.3	942.1
2.6.0-test6-mm4          224	999.5	       1331.2	920.0
2.6.0-test5              224	986.9	       1348.2	911.8
2.6.0-test7              224	546.7	       2433.8	832.8
2.6.0-test6              224	482.9	       2755.1	862.1

2.6.0-test5-mm2          256	1030.9	       1475.1	1076.5
2.6.0-test5-mm4          256	1027.1	       1480.6	1071.4
2.6.0-test6-mm4          256	1010.3	       1505.2	1074.8
2.6.0-test5              256	999.5	       1521.3	1029.3
2.6.0-test7              256	556.6	       2731.9	951.0
2.6.0-test6              256	545.1	       2789.5	978.4


Similarly, AIM7 fileserver benchmark had a drop in throughput.
CPU time is down, but total runtime is way up.

AIM7 fserver workload
kernel                   Tasks   Jobs/Min       Real    CPU
2.6.0-test5              4	79.7	       304.0	38.3
2.6.0-test5-mm2          4	77.4	       313.4	38.3
2.6.0-test5-mm4          4	76.5	       317.1	36.7
2.6.0-test6-mm4          4	65.5	       369.9	34.5
2.6.0-test7              4	45.5	       532.5	32.2
2.6.0-test6              4	45.0	       538.7	31.0

2.6.0-test5              8	129.8	       373.5	66.3
2.6.0-test5-mm2          8	103.1	       470.4	62.0
2.6.0-test5-mm4          8	102.2	       474.2	61.8
2.6.0-test6              8	82.7	       585.9	84.5
2.6.0-test6-mm4          8	82.4	       588.1	61.1
2.6.0-test7              8	61.0	       795.0	59.9

2.6.0-test5              12	164.1	       443.0	93.1
2.6.0-test5-mm2          12	112.4	       646.8	98.4
2.6.0-test5-mm4          12	111.2	       653.8	96.7
2.6.0-test6-mm4          12	95.2	       763.6	96.9
2.6.0-test6              12	87.9	       827.6	118.9
2.6.0-test7              12	54.5	       1335.0	91.8

2.6.0-test5              16	173.4	       559.0	116.5
2.6.0-test5-mm2          16	128.8	       752.9	119.9
2.6.0-test5-mm4          16	125.4	       773.1	124.3
2.6.0-test6-mm4          16	104.2	       930.3	118.7
2.6.0-test6              16	101.4	       956.0	107.4
2.6.0-test7              16	70.6	       1372.6	112.2

2.6.0-test5              20	180.5	       671.5	143.0
2.6.0-test5-mm4          20	144.9	       836.4	143.0
2.6.0-test5-mm2          20	141.7	       855.3	144.1
2.6.0-test6              20	117.7	       1030.0	131.1
2.6.0-test6-mm4          20	114.4	       1059.5	143.0
2.6.0-test7              20	78.7	       1539.9	134.0

2.6.0-test5              24	189.1	       769.0	176.0
2.6.0-test5-mm2          24	156.2	       930.9	180.2
2.6.0-test5-mm4          24	154.5	       941.6	176.9
2.6.0-test6              24	134.5	       1081.1	156.5
2.6.0-test6-mm4          24	124.5	       1167.8	170.2
2.6.0-test7              24	76.9	       1892.1	169.0

2.6.0-test5              28	204.6	       829.2	197.5
2.6.0-test5-mm2          28	158.7	       1069.2	209.8
2.6.0-test5-mm4          28	156.0	       1087.7	211.5
2.6.0-test6-mm4          28	134.2	       1264.0	196.9
2.6.0-test6              28	133.7	       1268.7	174.7
2.6.0-test7              28	84.1	       2016.4	188.5

2.6.0-test5              32	213.7	       907.4	228.9
2.6.0-test5-mm2          32	151.9	       1276.7	239.7
2.6.0-test5-mm4          32	151.5	       1279.6	241.4
2.6.0-test6              32	147.0	       1318.9	197.0
2.6.0-test6-mm4          32	138.8	       1397.6	225.4
2.6.0-test7              32	88.5	       2190.3	214.9

Same story for the shared workload.  Jobs/minute is down a
lot since 2.6.0-test5.


AIM7 shared workload
kernel                   Tasks   Jobs/Min       Real    CPU
2.6.0-test5              64	1888.8	       197.2	168.3
2.6.0-test5-mm2          64	1517.6	       245.4	177.0
2.6.0-test5-mm4          64	1488.6	       250.2	180.5
2.6.0-test6-mm4          64	1239.0	       300.6	175.0
2.6.0-test7              64	1225.1	       304.0	166.6
2.6.0-test6              64	880.0	       423.3	168.1

2.6.0-test5              128	2094.9	       355.6	338.1
2.6.0-test5-mm2          128	1837.4	       405.4	352.4
2.6.0-test5-mm4          128	1826.6	       407.8	358.4
2.6.0-test6-mm4          128	1561.8	       477.0	346.6
2.6.0-test7              128	1176.5	       633.2	332.6
2.6.0-test6              128	1142.1	       652.3	332.6

2.6.0-test5              192	2256.0	       495.3	507.8
2.6.0-test5-mm2          192	2074.9	       538.5	530.9
2.6.0-test5-mm4          192	2044.8	       546.5	536.9
2.6.0-test6-mm4          192	1711.8	       652.8	527.3
2.6.0-test6              192	1328.3	       841.3	505.6
2.6.0-test7              192	1317.9	       847.9	499.1

2.6.0-test5              256	2389.0	       623.7	683.8
2.6.0-test5-mm2          256	2207.2	       675.0	720.6
2.6.0-test5-mm4          256	2186.2	       681.5	729.9
2.6.0-test6-mm4          256	1780.7	       836.7	719.0
2.6.0-test6              256	1479.5	       1007.0	672.9
2.6.0-test7              256	1282.0	       1162.2	672.8

2.6.0-test5              320	2457.9	       757.7	856.6
2.6.0-test5-mm4          320	2330.2	       799.2	920.0
2.6.0-test5-mm2          320	2327.9	       800.0	919.4
2.6.0-test6-mm4          320	1864.7	       998.8	900.9
2.6.0-test6              320	1575.9	       1181.8	853.5
2.6.0-test7              320	1317.9	       1413.1	844.4

2.6.0-test5              384	2480.4	       901.0	1036.5
2.6.0-test5-mm2          384	2440.2	       915.9	1111.6
2.6.0-test5-mm4          384	2434.2	       918.1	1111.0
2.6.0-test6-mm4          384	1883.9	       1186.3	1091.7
2.6.0-test6              384	1629.8	       1371.3	1026.3
2.6.0-test7              384	1326.3	       1685.0	1018.6

2.6.0-test5              448	2529.8	       1030.7	1209.2
2.6.0-test5-mm4          448	2511.6	       1038.1	1306.8
2.6.0-test5-mm2          448	2510.0	       1038.8	1306.1
2.6.0-test6-mm4          448	1934.3	       1347.9	1283.6
2.6.0-test6              448	1612.3	       1617.1	1203.3
2.6.0-test7              448	1344.7	       1939.0	1200.9

2.6.0-test5-mm2          512	2579.1	       1155.4	1504.0
2.6.0-test5              512	2566.2	       1161.2	1386.2
2.6.0-test5-mm4          512	2552.9	       1167.2	1509.2
2.6.0-test6-mm4          512	1921.5	       1550.8	1475.1
2.6.0-test6              512	1538.6	       1936.7	1376.9
2.6.0-test7              512	1370.5	       2174.3	1379.2

tiobench read performance in test[67] has a big regression in throughput
as the number of threads goes up.  The max and long latency numbers get
bad too.  It appears to affect ext2, ext3, reiserfs, jfs and xfs.
Regression is in 2.6.0-test[67] and test6-mm4.

tiobench-0.3.3
Unit information
================
File size = 8192 megabytes
Blk Size  = 4096 bytes
Rate      = megabytes per second
CPU%      = percentage of CPU used during the test
Latency   = milliseconds
Lat%      = percent of requests that took longer than X seconds
CPU Eff   = Rate divided by CPU% - throughput per cpu load

Sequential Reads ext2
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1   28.55 13.89%     0.408      754.59  0.00000
2.6.0-test5-mm2              1   28.66 13.98%     0.406      742.76  0.00000
2.6.0-test5-mm4              1   28.46 13.90%     0.409      887.02  0.00000
2.6.0-test6                  1   28.47 13.81%     0.409     1069.74  0.00000
2.6.0-test6-mm4              1   28.41 13.89%     0.410      723.14  0.00000
2.6.0-test7                  1   28.71 13.92%     0.405      715.02  0.00000

With one thread, performance is similar.

2.6.0-test5                  8   34.01 17.09%     2.620      834.85  0.00000
2.6.0-test5-mm2              8   32.44 16.44%     2.765      844.99  0.00000
2.6.0-test5-mm4              8   30.75 15.55%     2.967      952.18  0.00000
2.6.0-test6                  8   20.67  9.90%     4.477     1569.30  0.00000
2.6.0-test6-mm4              8   21.55 10.31%     4.260     1790.19  0.00000
2.6.0-test7                  8   21.46 10.24%     4.312     1913.24  0.00000

2.6.0-test5                 16   30.13 15.04%     5.980     1337.69  0.00000
2.6.0-test5-mm2             16   29.71 14.97%     6.024     1115.95  0.00000
2.6.0-test5-mm4             16   27.10 13.65%     6.606     1008.67  0.00000
2.6.0-test6                 16   16.44  7.90%    11.201     2820.85  0.00000
2.6.0-test6-mm4             16   16.14  7.78%    11.382     2753.13  0.00000
2.6.0-test7                 16   18.34  8.64%     9.877     2742.49  0.00000

2.6.0-test5                 32   29.46 14.62%    11.954     1560.02  0.00000
2.6.0-test5-mm2             32   28.01 14.26%    12.808     1627.00  0.00000
2.6.0-test5-mm4             32   28.78 14.63%    12.182     1859.56  0.00000
2.6.0-test6                 32   15.60  7.55%    23.415     4877.75  0.00000
2.6.0-test6-mm4             32   14.63  7.20%    24.713     4846.63  0.00000
2.6.0-test7                 32   14.79  7.14%    24.234     4855.64  0.00000

2.6.0-test5                 64   25.24 13.57%    26.901     1743.05  0.00000
2.6.0-test5-mm2             64   24.01 13.27%    27.922     1849.82  0.00000
2.6.0-test5-mm4             64   23.67 13.15%    28.513     1956.58  0.00000
2.6.0-test6                 64    9.45  5.24%    75.491     9578.85  2.75073
2.6.0-test6-mm4             64    8.51  4.82%    84.118    13166.74  3.11647
2.6.0-test7                 64    8.58  4.86%    83.430    13622.86  3.05581

2.6.0-test5                128   10.41  9.76%   129.662     1820.78  0.00000
2.6.0-test5-mm2            128   11.00 10.44%   120.118     1742.44  0.00000
2.6.0-test5-mm4            128   10.67 10.15%   122.871     1912.79  0.00000
2.6.0-test6                128    2.97  2.66%   472.251    25694.03  8.39324
2.6.0-test6-mm4            128    2.94  2.66%   473.943    25530.14  8.39028
2.6.0-test7                128    2.94  2.68%   480.175    26306.38  8.39257

At 256 threads, throughput is down and max latency skyrockets.

2.6.0-test5                256    9.13 13.30%   305.097     1446.43  0.00000
2.6.0-test5-mm2            256    9.09 13.60%   304.019     1648.80  0.00000
2.6.0-test5-mm4            256    8.63 13.66%   325.793     1542.62  0.00000
2.6.0-test6                256    2.71  2.49%   937.603   529116.60  8.94007
2.6.0-test6-mm4            256    2.62  2.46%   979.221   785760.77  9.12519
2.6.0-test7                256    2.56  2.39%   988.719   582067.86  9.28287

Random Reads ext2
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1    0.95  0.94%    12.373      114.89  0.00000
2.6.0-test5-mm2              1    0.90  0.88%    13.061      113.19  0.00000
2.6.0-test5-mm4              1    0.53  0.60%    21.903      151.35  0.00000
2.6.0-test6                  1    0.73  0.82%    16.053      391.29  0.00000
2.6.0-test6-mm4              1    0.76  0.81%    15.388      126.51  0.00000
2.6.0-test7                  1    0.96  0.94%    12.192      101.93  0.00000

2.6.0-test5                  8    4.11  4.58%    19.433      137.74  0.00000
2.6.0-test5-mm2              8    3.94  4.03%    20.931      144.16  0.00000
2.6.0-test5-mm4              8    2.81  3.28%    30.086      222.27  0.00000
2.6.0-test6                  8    3.25  3.21%    27.598      640.41  0.00000
2.6.0-test6-mm4              8    3.98  3.82%    22.078      324.87  0.00000
2.6.0-test7                  8    3.33  3.27%    27.069      417.17  0.00000

2.6.0-test5                 16    4.05  3.90%    42.424      236.49  0.00000
2.6.0-test5-mm2             16    4.09  4.31%    41.651      255.17  0.00000
2.6.0-test5-mm4             16    3.05  3.19%    55.847      332.10  0.00000
2.6.0-test6                 16    3.43  3.28%    48.836      934.02  0.00000
2.6.0-test6-mm4             16    3.21  2.69%    52.841      936.50  0.00000
2.6.0-test7                 16    4.93  4.17%    34.720      630.63  0.00000

2.6.0-test5                 32    4.41  3.58%    70.910      516.28  0.00000
2.6.0-test5-mm2             32    4.23  4.17%    78.492      372.63  0.00000
2.6.0-test5-mm4             32    4.19  3.90%    76.001      472.90  0.00000
2.6.0-test6                 32    3.74  3.20%    92.407     1578.33  0.00000
2.6.0-test6-mm4             32    2.94  2.97%   116.663     1722.23  0.00000
2.6.0-test7                 32    2.62  2.36%   129.093     2248.32  0.00000

2.6.0-test5                 64    4.47  4.00%   137.277     1077.53  0.00000
2.6.0-test5-mm2             64    4.79  4.53%   124.035     1144.38  0.00000
2.6.0-test5-mm4             64    4.13  3.81%   143.321     1265.44  0.00000
2.6.0-test6                 64    3.89  3.62%   166.555     2679.86  0.00000
2.6.0-test6-mm4             64    3.23  3.15%   200.698     3160.57  0.00000
2.6.0-test7                 64    3.14  2.74%   206.849     3384.30  0.00000

2.6.0-test5                128    4.86  4.83%   243.704     2287.09  0.00000
2.6.0-test5-mm2            128    5.08  4.85%   228.102     2391.21  0.00000
2.6.0-test5-mm4            128    4.16  4.65%   270.840     2501.04  0.00000
2.6.0-test6                128    3.19  3.09%   389.583     6092.04  0.12601
2.6.0-test6-mm4            128    2.75  2.78%   439.379     6632.53  0.65524
2.6.0-test7                128    3.16  3.16%   388.762     5810.95  0.07560

2.6.0-test5                256    5.51  7.00%   427.503     2511.79  0.00000
2.6.0-test5-mm2            256    5.23  7.39%   449.007     3169.28  0.00000
2.6.0-test5-mm4            256    4.48  6.76%   526.994     3618.17  0.00000
2.6.0-test6                256    2.01  2.44%  1156.505    14117.02 16.82291
2.6.0-test6-mm4            256    1.99  2.38%  1169.942    13903.51 16.56250
2.6.0-test7                256    2.09  2.41%  1133.624    13195.27 14.97396

Sequential and random writes on ext2 are not regressed.

ext3 has a regression similar to ext2 for sequential and random reads.

Sequential Reads ext3
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1   29.03 14.00%     0.401      745.26  0.00000
2.6.0-test5-mm2              1   28.05 13.82%     0.415      763.49  0.00000
2.6.0-test5-mm4              1   29.31 14.10%     0.397      867.89  0.00000
2.6.0-test6                  1   28.92 13.83%     0.402      747.59  0.00000
2.6.0-test6-mm4              1   28.01 13.67%     0.415     1162.40  0.00000
2.6.0-test7                  1   27.95 13.54%     0.416      860.38  0.00000

2.6.0-test5                  8   14.53  8.42%     6.187     1280.16  0.00000
2.6.0-test5-mm2              8   14.25  8.56%     6.428     1438.12  0.00000
2.6.0-test5-mm4              8   14.42  8.60%     6.317     1335.28  0.00000
2.6.0-test6                  8   10.37  6.00%     8.791     2065.27  0.00000
2.6.0-test6-mm4              8    9.38  5.81%     9.765     1978.67  0.00000
2.6.0-test7                  8   10.25  6.02%     8.931     1977.71  0.00000

2.6.0-test5                 16   11.38  7.09%    15.604     1387.68  0.00000
2.6.0-test5-mm2             16   12.84  7.97%    14.206     1246.21  0.00000
2.6.0-test5-mm4             16   12.89  8.00%    14.050     1790.63  0.00000
2.6.0-test6                 16    5.13  3.45%    34.925     3982.64  0.00000
2.6.0-test6-mm4             16    4.91  3.42%    37.057     4349.95  0.00000
2.6.0-test7                 16    5.27  3.56%    34.774     3852.71  0.00000

2.6.0-test5                 32    8.77  6.17%    41.461     1856.08  0.00000
2.6.0-test5-mm2             32    8.79  6.26%    40.822     1667.85  0.00000
2.6.0-test5-mm4             32    8.74  6.41%    41.519     1797.28  0.00000
2.6.0-test6                 32    3.32  2.60%   107.252     9267.00  0.00649
2.6.0-test6-mm4             32    3.58  2.78%   100.364     8453.15  0.00496
2.6.0-test7                 32    3.39  2.65%   104.283     9029.86  0.00367

2.6.0-test5                 64    6.29  5.30%   113.016     1947.14  0.00000
2.6.0-test5-mm2             64    7.58  6.00%    81.717     1722.83  0.00000
2.6.0-test5-mm4             64    7.63  6.07%    79.882     1709.56  0.00000
2.6.0-test6                 64    2.63  2.25%   265.280    18736.77  8.76222
2.6.0-test6-mm4             64    2.43  2.17%   277.430    17986.61  8.95066
2.6.0-test7                 64    2.40  2.16%   286.947    17890.59  9.59925

2.6.0-test5                128    6.19  6.61%   216.851     1852.30  0.00000
2.6.0-test5-mm2            128    6.73  7.19%   191.761     2049.82  0.00000
2.6.0-test5-mm4            128    6.09  6.55%   209.661     1910.36  0.00000
2.6.0-test6                128    2.06  1.95%   640.506    32660.27 12.41284
2.6.0-test6-mm4            128    2.03  1.98%   645.423    30932.33 12.57601
2.6.0-test7                128    2.05  2.02%   639.791    30885.16 12.72559

2.6.0-test5                256    6.15  9.88%   457.029     1880.47  0.00000
2.6.0-test5-mm2            256    5.51  9.06%   518.943     2002.06  0.00000
2.6.0-test5-mm4            256    5.30  8.81%   535.291     2103.11  0.00000
2.6.0-test6                256    1.85  1.78%  1293.732   230380.55 13.73657
2.6.0-test6-mm4            256    1.91  1.90%  1246.640   243088.10 13.26160
2.6.0-test7                256    1.81  1.78%  1351.071   281394.59 13.80543

Random Reads ext3
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1    0.90  0.85%    12.963      112.10  0.00000
2.6.0-test5-mm2              1    0.93  0.97%    12.637      125.35  0.00000
2.6.0-test5-mm4              1    0.85  0.95%    13.714      122.66  0.00000
2.6.0-test6                  1    0.73  0.71%    16.054      516.87  0.00000
2.6.0-test6-mm4              1    0.91  0.91%    12.907      125.50  0.00000
2.6.0-test7                  1    0.94  0.82%    12.443      110.45  0.00000

2.6.0-test5                  8    3.78  4.24%    21.477      131.88  0.00000
2.6.0-test5-mm2              8    4.01  4.49%    20.218      124.38  0.00000
2.6.0-test5-mm4              8    4.43  4.63%    18.684      142.49  0.00000
2.6.0-test6                  8    3.55  3.11%    23.596      362.51  0.00000
2.6.0-test6-mm4              8    2.52  2.69%    32.927      534.18  0.00000
2.6.0-test7                  8    4.21  3.80%    19.881      459.00  0.00000

2.6.0-test5                 16    4.10  3.74%    37.249      236.48  0.00000
2.6.0-test5-mm2             16    4.42  4.48%    37.848      208.96  0.00000
2.6.0-test5-mm4             16    4.25  4.51%    36.874      220.75  0.00000
2.6.0-test6                 16    3.27  3.13%    49.687      803.39  0.00000
2.6.0-test6-mm4             16    3.37  3.17%    49.112      951.18  0.00000
2.6.0-test7                 16    4.49  3.87%    38.468      727.87  0.00000

2.6.0-test5                 32    4.28  4.06%    76.377      458.79  0.00000
2.6.0-test5-mm2             32    4.53  4.40%    71.762      466.83  0.00000
2.6.0-test5-mm4             32    4.48  4.23%    70.711      361.46  0.00000
2.6.0-test6                 32    3.35  3.06%    98.621     1728.18  0.00000
2.6.0-test6-mm4             32    4.13  3.54%    77.004     1321.89  0.00000
2.6.0-test7                 32    3.95  3.30%    79.804     1402.47  0.00000

2.6.0-test5                 64    4.89  4.80%   134.964     1243.33  0.00000
2.6.0-test5-mm2             64    4.63  4.17%   108.944     1006.23  0.00000
2.6.0-test5-mm4             64    3.70  3.92%   132.534     1222.82  0.00000
2.6.0-test6                 64    3.35  2.98%   189.642     2847.96  0.00000
2.6.0-test6-mm4             64    3.12  3.03%   207.658     3362.27  0.00000
2.6.0-test7                 64    3.02  2.89%   212.838     2652.28  0.00000

2.6.0-test5                128    5.26  5.11%   230.823     2088.49  0.00000
2.6.0-test5-mm2            128    4.77  4.65%   252.354     2283.06  0.00000
2.6.0-test5-mm4            128    4.07  4.26%   280.692     2923.29  0.00000
2.6.0-test6                128    2.88  3.07%   431.937     5967.29  0.12601
2.6.0-test6-mm4            128    3.09  3.26%   403.965     5559.31  0.02520
2.6.0-test7                128    3.01  2.95%   407.900     6243.88  0.10081

2.6.0-test5                256    5.18  6.62%   463.528     3270.79  0.00000
2.6.0-test5-mm2            256    5.20  7.12%   452.987     3055.03  0.00000
2.6.0-test5-mm4            256    4.36  5.89%   539.848     3465.46  0.00000
2.6.0-test6                256    2.26  2.92%  1013.076    11495.98 12.47396
2.6.0-test6-mm4            256    2.49  2.98%   943.926    10902.18 10.72917
2.6.0-test7                256    2.43  3.04%   996.420    11016.00 12.08334

Sequential Writes ext3
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1   56.15 64.55%     0.170     5700.14  0.00010
2.6.0-test5-mm2              1   56.14 67.38%     0.173     4383.06  0.00005
2.6.0-test5-mm4              1   55.64 66.40%     0.173     4323.48  0.00000
2.6.0-test6                  1   56.15 65.22%     0.171     4578.10  0.00005
2.6.0-test6-mm4              1   55.81 66.36%     0.172     5623.30  0.00010
2.6.0-test7                  1   55.94 65.10%     0.171     4544.88  0.00000

2.6.0-test5                  8    2.99  4.94%    23.859  1119399.58  0.08983
2.6.0-test5-mm2              8    3.00  5.08%    24.803   613704.08  0.09155
2.6.0-test5-mm4              8    2.98  5.06%    25.599   515385.61  0.09627
2.6.0-test6                  8    4.65  7.53%    15.776   347393.72  0.07853
2.6.0-test6-mm4              8    3.02  5.04%    25.421   399480.58  0.09512
2.6.0-test7                  8    3.12  5.12%    23.991   197721.86  0.09255

2.6.0-test5                 16    2.58  4.55%    55.284   715963.30  0.14072
2.6.0-test5-mm2             16    2.93  5.09%    50.697   648332.45  0.11883
2.6.0-test5-mm4             16    3.21  5.62%    47.072   544422.32  0.12240
2.6.0-test6                 16    3.45  6.12%    39.175   424983.10  0.11596
2.6.0-test6-mm4             16    2.72  4.76%    54.117   429887.30  0.13976
2.6.0-test7                 16    2.74  4.73%    52.594   886672.08  0.13404

2.6.0-test5                 32    2.28  4.43%   127.746   833586.10  0.20012
2.6.0-test5-mm2             32    2.25  4.58%   124.219  1054925.33  0.20647
2.6.0-test5-mm4             32    2.11  4.33%   139.662   787258.98  0.22011
2.6.0-test6                 32    2.80  5.35%   107.223   835055.83  0.18424
2.6.0-test6-mm4             32    2.28  4.26%   117.722   754407.32  0.18582
2.6.0-test7                 32    2.21  4.38%   128.475   829790.72  0.21805

2.6.0-test5                 64    2.02  4.54%   265.546  1322591.44  0.40007
2.6.0-test5-mm2             64    2.02  4.42%   278.978  1434418.75  0.42186
2.6.0-test5-mm4             64    2.25  4.83%   242.598  1592337.06  0.35062
2.6.0-test6                 64    2.44  5.25%   222.436  1233112.15  0.38619
2.6.0-test6-mm4             64    1.98  4.45%   274.091  1442328.98  0.45623
2.6.0-test7                 64    2.09  4.37%   249.893  1456365.96  0.30308

2.6.0-test5                128    2.21  4.96%   444.935  2758997.12  0.60168
2.6.0-test5-mm2            128    1.93  4.62%   568.801  3209338.67  0.84395
2.6.0-test5-mm4            128    1.99  4.67%   518.813  2778367.36  0.77595
2.6.0-test6                128    2.00  4.47%   497.636  2933240.88  0.65131
2.6.0-test6-mm4            128    2.03  4.71%   505.696  2839390.84  0.74044
2.6.0-test7                128    2.00  4.69%   491.950  2994053.88  0.75612

2.6.0-test5                256    2.11  7.77%   941.664  4873713.39  1.83378
2.6.0-test5-mm2            256    1.91  6.85%  1093.820  5897453.70  1.80015
2.6.0-test5-mm4            256    1.97  6.74%  1017.367  5695863.45  1.77222
2.6.0-test6                256    1.92  6.71%  1065.957  6104195.68  1.87802
2.6.0-test6-mm4            256    1.93  7.02%  1038.493  5678570.32  1.92114
2.6.0-test7                256    1.85  7.36%  1083.836  5720266.70  1.89524

Random Writes ext3
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1    3.39  3.44%     0.399       37.37  0.00000
2.6.0-test5-mm2              1    3.22  3.38%     0.516       40.33  0.00000
2.6.0-test5-mm4              1    3.56  3.72%     0.081       10.61  0.00000
2.6.0-test6                  1    2.34  2.39%     1.780       24.35  0.00000
2.6.0-test6-mm4              1    3.00  3.19%     0.746       37.62  0.00000
2.6.0-test7                  1    3.05  3.06%     0.653       54.78  0.00000

2.6.0-test5                  8    3.17  3.59%    13.525     1655.65  0.00000
2.6.0-test5-mm2              8    3.05  3.44%    14.349     2391.37  0.00000
2.6.0-test5-mm4              8    3.08  3.54%    14.093     1401.73  0.00000
2.6.0-test6                  8    2.79  3.15%    14.906     3957.84  0.02500
2.6.0-test6-mm4              8    2.88  3.06%    15.029     1283.97  0.00000
2.6.0-test7                  8    3.18  3.54%    14.407     1671.73  0.00000

2.6.0-test5                 16    3.18  3.49%    26.252     1693.61  0.00000
2.6.0-test5-mm2             16    3.28  3.73%    25.573     3583.93  0.02500
2.6.0-test5-mm4             16    3.21  3.84%    25.857     6012.15  0.02500
2.6.0-test6                 16    2.81  2.81%    30.448     2511.41  0.00000
2.6.0-test6-mm4             16    3.17  4.08%    20.815     5062.41  0.02500
2.6.0-test7                 16    3.14  3.43%    28.987     2120.35  0.00000

2.6.0-test5                 32    3.26  4.05%    46.822     3656.96  0.00000
2.6.0-test5-mm2             32    3.25  4.28%    47.088     2834.74  0.00000
2.6.0-test5-mm4             32    3.29  4.49%    44.765     2409.89  0.00000
2.6.0-test6                 32    2.67  2.91%    62.319    11807.07  0.05000
2.6.0-test6-mm4             32    2.98  3.57%    57.025     2250.13  0.00000
2.6.0-test7                 32    3.06  3.75%    50.252     9420.48  0.05000

2.6.0-test5                 64    3.40  4.96%    46.634     2426.05  0.00000
2.6.0-test5-mm2             64    3.24  4.72%    93.293     7875.05  1.86492
2.6.0-test5-mm4             64    3.24  4.44%    79.998     5292.84  0.63004
2.6.0-test6                 64    2.50  3.22%   137.484     6723.01  0.25202
2.6.0-test6-mm4             64    2.83  3.92%    79.495     9204.46  0.68044
2.6.0-test7                 64    2.85  4.00%    91.255     6050.08  0.63004

2.6.0-test5                128    3.46  5.89%   101.597     6086.33  0.80645
2.6.0-test5-mm2            128    3.39  5.71%   129.468     6234.78  1.28528
2.6.0-test5-mm4            128    3.27  5.82%   128.846     6601.87  0.65524
2.6.0-test6                128    2.63  4.26%   123.641     6247.05  2.09173
2.6.0-test6-mm4            128    2.69  4.58%   100.787     4669.66  0.00000
2.6.0-test7                128    2.71  4.58%    91.826     4953.82  0.00000

2.6.0-test5                256    3.45  6.88%   210.999    10063.95  6.06770
2.6.0-test5-mm2            256    3.45  6.80%   263.620     8774.35  8.25520
2.6.0-test5-mm4            256    3.33  6.91%   209.373    10921.27  5.10416
2.6.0-test6                256    2.58  5.45%   353.922     7824.45  5.31250
2.6.0-test6-mm4            256    2.99  6.08%   152.068     6246.05  2.36979
2.6.0-test7                256    2.96  6.15%    69.124     3923.92  0.00000

Sequential Reads jfs
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------

2.6.0-test5                  1   28.93 13.16%     0.402      958.45  0.00000
2.6.0-test5-mm2              1   28.84 13.33%     0.403      703.16  0.00000
2.6.0-test5-mm4              1   29.03 13.50%     0.401      871.93  0.00000
2.6.0-test6-mm4              1   28.90 13.54%     0.402      724.31  0.00000
2.6.0-test7                  1   29.07 13.44%     0.400      677.09  0.00000

2.6.0-test5                  8   50.81 24.01%     1.815      942.91  0.00000
2.6.0-test5-mm2              8   48.22 23.30%     1.918      959.38  0.00000
2.6.0-test5-mm4              8   46.01 21.88%     1.988      840.79  0.00000
2.6.0-test6-mm4              8   20.85  9.38%     4.417     2073.30  0.00000
2.6.0-test7                  8   20.88  9.17%     4.402     1941.85  0.00000

2.6.0-test5                 16   37.26 17.05%     5.004      855.09  0.00000
2.6.0-test5-mm2             16   40.27 18.83%     4.628     1024.08  0.00000
2.6.0-test5-mm4             16   38.68 18.07%     4.831      820.64  0.00000
2.6.0-test6-mm4             16   11.10  5.05%    16.392     3579.95  0.00000
2.6.0-test7                 16   11.98  5.35%    15.258     3381.34  0.00000

2.6.0-test5                 32   35.48 16.56%    10.439     1742.70  0.00000
2.6.0-test5-mm2             32   35.46 17.01%    10.292     1979.97  0.00000
2.6.0-test5-mm4             32   33.58 15.89%    11.012     1885.26  0.00000
2.6.0-test6-mm4             32   10.45  4.79%    34.665     6205.82  0.00743
2.6.0-test7                 32   10.82  4.85%    33.369     6185.50  0.00668

2.6.0-test5                 64   36.58 17.71%    19.855     3297.14  0.00000
2.6.0-test5-mm2             64   34.64 17.14%    21.114     3190.91  0.00000
2.6.0-test5-mm4             64   34.98 17.66%    20.673     3254.24  0.00000
2.6.0-test6-mm4             64   10.64  4.87%    67.501    11935.89  2.07191
2.6.0-test7                 64   11.33  5.08%    61.240    11701.91  1.98450

2.6.0-test5                128   11.37  8.64%   109.981     2829.31  0.00000
2.6.0-test5-mm2            128   17.63 11.71%    64.633     2799.09  0.00000
2.6.0-test5-mm4            128   15.92 11.40%    73.376     3032.50  0.00000
2.6.0-test6-mm4            128    2.71  2.52%   520.381    33565.98  8.79664
2.6.0-test7                128    3.09  2.69%   409.208    33147.84  7.51347

2.6.0-test5                256    7.53 11.50%   348.180     2671.65  0.00000
2.6.0-test5-mm2            256    7.77 11.45%   319.292     2285.73  0.00000
2.6.0-test5-mm4            256    7.23 11.94%   364.323     2244.99  0.00000
2.6.0-test6-mm4            256    2.00  1.95%  1023.629   216889.13 12.88433
2.6.0-test7                256    1.85  1.88%  1090.929   455893.85 13.53616

Random Reads jfs
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1    0.95  0.82%    12.368       86.79  0.00000
2.6.0-test5-mm2              1    0.97  0.89%    12.013      108.08  0.00000
2.6.0-test5-mm4              1    0.99  0.99%    11.796      110.46  0.00000
2.6.0-test6-mm4              1    0.98  0.91%    11.987      112.94  0.00000
2.6.0-test7                  1    0.96  0.83%    12.187      100.56  0.00000

2.6.0-test5                  8    4.68  4.23%    18.784      137.13  0.00000
2.6.0-test5-mm2              8    4.06  3.84%    21.942      145.02  0.00000
2.6.0-test5-mm4              8    4.01  3.80%    20.854      150.03  0.00000
2.6.0-test6-mm4              8    0.81  0.61%   112.779     1277.22  0.00000
2.6.0-test7                  8    0.82  0.66%   110.357     1250.05  0.00000

2.6.0-test5                 16    3.98  3.12%    44.566      188.99  0.00000
2.6.0-test5-mm2             16    4.35  3.42%    40.240      189.97  0.00000
2.6.0-test5-mm4             16    4.28  3.37%    41.846      199.23  0.00000
2.6.0-test6-mm4             16    0.75  0.62%   231.931     2631.93  0.00000
2.6.0-test7                 16    0.84  0.65%   212.690     2578.59  0.00000

2.6.0-test5                 32    4.09  2.87%    86.440      373.37  0.00000
2.6.0-test5-mm2             32    3.86  3.07%    91.487      443.48  0.00000
2.6.0-test5-mm4             32    3.88  3.07%    90.965      553.03  0.00000
2.6.0-test6-mm4             32    0.76  0.64%   455.886     5230.06  0.00000
2.6.0-test7                 32    0.71  0.57%   477.112     5188.75  0.00000

2.6.0-test5                 64    4.04  2.88%   169.495     1553.14  0.00000
2.6.0-test5-mm2             64    3.84  3.25%   170.017     1413.19  0.00000
2.6.0-test5-mm4             64    3.99  3.37%   167.645     1628.57  0.00000
2.6.0-test6-mm4             64    0.70  0.61%   950.464    10451.50 26.76412
2.6.0-test7                 64    0.71  0.63%   923.148    10352.12 26.38609

2.6.0-test5                128    4.32  3.83%   268.303     2322.59  0.00000
2.6.0-test5-mm2            128    4.18  3.95%   253.072     2451.90  0.00000
2.6.0-test5-mm4            128    4.05  3.95%   255.242     2232.10  0.00000
2.6.0-test6-mm4            128    0.76  0.77%  1626.585    26333.82 35.78629
2.6.0-test7                128    0.78  0.69%  1521.032    24112.21 33.84576

2.6.0-test5                256    4.03  5.43%   585.871     3402.15  0.00000
2.6.0-test5-mm2            256    4.06  5.49%   556.512     3436.30  0.00000
2.6.0-test5-mm4            256    4.11  5.47%   543.649     2806.43  0.00000
2.6.0-test6-mm4            256    0.82  0.87%  2318.070    33523.86 44.84375
2.6.0-test7                256    0.81  0.85%  2315.106    34969.85 44.94791

Sequential Writes jfs
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1   56.47 47.37%     0.170     3391.01  0.00000
2.6.0-test5-mm2              1   56.47 49.42%     0.172     2901.46  0.00000
2.6.0-test5-mm4              1   56.40 48.44%     0.181     2578.21  0.00000
2.6.0-test6-mm4              1   56.28 47.99%     0.175     2442.78  0.00000
2.6.0-test7                  1   56.39 46.73%     0.171     3405.20  0.00000

2.6.0-test5                  8   56.19 65.59%     1.343     8360.10  0.00100
2.6.0-test5-mm2              8   55.94 66.86%     1.336    17088.51  0.00162
2.6.0-test5-mm4              8   56.03 67.52%     1.316    12716.34  0.00105
2.6.0-test6-mm4              8   56.14 65.15%     1.320    13881.60  0.00100
2.6.0-test7                  8   56.23 64.40%     1.335    11973.67  0.00109

2.6.0-test5                 16   55.15 67.48%     2.651    30109.56  0.03486
2.6.0-test5-mm2             16   55.93 68.92%     2.576    39135.68  0.03081
2.6.0-test5-mm4             16   54.88 69.40%     2.679    36823.40  0.03596
2.6.0-test6-mm4             16   54.85 65.44%     2.641    21368.63  0.04229
2.6.0-test7                 16   55.84 68.65%     2.635    63526.06  0.03019

2.6.0-test5                 32   54.38 69.36%     5.204    96795.80  0.10614
2.6.0-test5-mm2             32   55.11 70.01%     4.989    55045.13  0.10557
2.6.0-test5-mm4             32   53.92 70.42%     5.099    68154.16  0.10362
2.6.0-test6-mm4             32   54.17 70.60%     5.074    81693.83  0.09809
2.6.0-test7                 32   54.75 69.14%     4.998   108095.44  0.09742

2.6.0-test5                 64   54.58 75.03%     9.331   208382.29  0.10910
2.6.0-test5-mm2             64   54.44 69.22%     9.195   156878.93  0.11329
2.6.0-test5-mm4             64   53.35 72.64%     9.621   158954.21  0.10639
2.6.0-test6-mm4             64   54.54 70.28%     9.386   148888.99  0.11191
2.6.0-test7                 64   54.67 69.68%     8.969   189127.28  0.10605

2.6.0-test5                128   10.39 18.85%    93.764  2075500.73  0.09913
2.6.0-test5-mm2            128   15.87 27.47%    61.244  1306699.59  0.08931
2.6.0-test5-mm4            128   16.06 28.24%    56.744  1437340.48  0.08950
2.6.0-test6-mm4            128   20.89 35.44%    42.971  1018802.82  0.10410
2.6.0-test7                128   13.63 23.07%    69.588  1685679.31  0.08650

2.6.0-test5                256   10.65 20.80%   168.683  1900450.38  0.20290
2.6.0-test5-mm2            256    9.72 18.90%   202.289  2024501.68  0.13385
2.6.0-test5-mm4            256    8.52 17.57%   221.659  1967027.22  0.19260
2.6.0-test6-mm4            256    9.20 17.32%   200.496  1843020.23  0.12737
2.6.0-test7                256    8.85 17.47%   212.298  2080395.84  0.11768

Random Writes jfs
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1    3.90  3.48%     0.060        0.56  0.00000
2.6.0-test5-mm2              1    3.89  3.25%     0.061        0.28  0.00000
2.6.0-test5-mm4              1    3.86  3.23%     0.061        0.35  0.00000
2.6.0-test6-mm4              1    3.93  3.40%     0.060        0.37  0.00000
2.6.0-test7                  1    3.69  3.03%     0.059        0.88  0.00000

2.6.0-test5                  8    3.86  6.54%     0.170       59.53  0.00000
2.6.0-test5-mm2              8    3.68  6.19%     0.170       77.58  0.00000
2.6.0-test5-mm4              8    3.69  6.03%     0.171       69.55  0.00000
2.6.0-test6-mm4              8    3.55  5.93%     0.153       34.88  0.00000
2.6.0-test7                  8    3.56  5.99%     0.162       52.74  0.00000

2.6.0-test5                 16    3.71  6.47%     0.162       43.67  0.00000
2.6.0-test5-mm2             16    3.82  6.63%     0.166       54.86  0.00000
2.6.0-test5-mm4             16    3.71  6.33%     0.168       43.72  0.00000
2.6.0-test6-mm4             16    3.42  6.01%     0.151       14.21  0.00000
2.6.0-test7                 16    3.56  5.94%     0.149        4.41  0.00000

2.6.0-test5                 32    3.71  6.48%     0.178      111.32  0.00000
2.6.0-test5-mm2             32    3.50  5.99%     0.148        3.16  0.00000
2.6.0-test5-mm4             32    3.56  6.47%     0.170       55.28  0.00000
2.6.0-test6-mm4             32    3.44  6.09%     0.147        4.23  0.00000
2.6.0-test7                 32    3.22  5.41%     0.158       59.51  0.00000

2.6.0-test5                 64    3.50  6.28%     0.148        3.96  0.00000
2.6.0-test5-mm2             64    3.34  5.99%     0.160       51.30  0.00000
2.6.0-test5-mm4             64    3.58  6.46%     0.157        7.48  0.00000
2.6.0-test6-mm4             64    3.25  5.67%     0.146        3.61  0.00000
2.6.0-test7                 64    3.28  5.74%     0.168       55.67  0.00000

2.6.0-test5                128    3.28  5.72%    20.122     1415.53  0.00000
2.6.0-test5-mm2            128    3.29  5.92%     1.297     1229.30  0.00000
2.6.0-test5-mm4            128    3.42  6.33%     1.499      541.00  0.00000
2.6.0-test6-mm4            128    3.08  5.62%     1.341     1089.07  0.00000
2.6.0-test7                128    3.13  5.51%     0.595      923.55  0.00000

2.6.0-test5                256    2.80  5.18%    89.100     3075.82  0.00000
2.6.0-test5-mm2            256    2.98  5.76%    33.288     2094.84  0.00000
2.6.0-test5-mm4            256    3.19  6.47%    13.738     1648.72  0.00000
2.6.0-test6-mm4            256    2.65  5.49%    18.498     2632.76  0.00000
2.6.0-test7                256    2.74  5.45%    10.049     2382.47  0.00000

Sequential Reads reiserfs
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1   28.20 15.37%     0.413       97.43  0.00000
2.6.0-test5-mm2              1   30.21 16.13%     0.385     1159.97  0.00000
2.6.0-test5-mm4              1   28.15 15.29%     0.413      771.52  0.00000
2.6.0-test6                  1   28.84 15.63%     0.404      111.79  0.00000
2.6.0-test6-mm4              1   29.99 16.01%     0.388     1065.08  0.00000
2.6.0-test7                  1   29.45 15.70%     0.395      113.20  0.00000

2.6.0-test5                  8   29.77 17.07%     2.974      854.50  0.00000
2.6.0-test5-mm2              8   31.14 18.03%     2.811      653.61  0.00000
2.6.0-test5-mm4              8   32.23 18.70%     2.727      542.20  0.00000
2.6.0-test6                  8   21.41 11.92%     4.275     1487.18  0.00000
2.6.0-test6-mm4              8   21.93 12.30%     4.185     1320.72  0.00000
2.6.0-test7                  8   21.40 11.78%     4.263     1261.39  0.00000

2.6.0-test5                 16   29.99 17.01%     5.829      901.20  0.00000
2.6.0-test5-mm2             16   29.66 17.05%     5.954     1288.12  0.00000
2.6.0-test5-mm4             16   30.20 17.66%     5.872     1153.05  0.00000
2.6.0-test6                 16   19.46 10.99%     9.435     2515.46  0.00000
2.6.0-test6-mm4             16   18.92 10.48%     9.650     2572.79  0.00000
2.6.0-test7                 16   18.34 10.11%    10.014     2648.17  0.00000

2.6.0-test5                 32   28.86 16.73%    12.003     1408.20  0.00000
2.6.0-test5-mm2             32   28.21 16.52%    12.099     1385.36  0.00000
2.6.0-test5-mm4             32   27.64 16.72%    12.706     1491.53  0.00000
2.6.0-test6                 32   17.33  9.87%    20.872     5771.49  0.00005
2.6.0-test6-mm4             32   18.21 10.25%    19.659     4621.02  0.00000
2.6.0-test7                 32   17.89  9.88%    20.121     4681.65  0.00000

2.6.0-test5                 64   24.08 15.49%    27.936     1610.23  0.00000
2.6.0-test5-mm2             64   23.23 15.01%    27.942     1478.59  0.00000
2.6.0-test5-mm4             64   22.97 15.39%    29.020     1577.31  0.00000
2.6.0-test6                 64    7.62  5.28%    95.479    16071.22  3.52058
2.6.0-test6-mm4             64    7.89  5.41%    91.677    12720.86  3.22466
2.6.0-test7                 64    7.51  5.16%    96.373    14738.26  3.39680

2.6.0-test5                128    9.77 11.49%   141.347     1501.12  0.00000
2.6.0-test5-mm2            128   10.81 12.13%   121.146     1501.77  0.00000
2.6.0-test5-mm4            128   10.08 12.01%   134.006     1616.24  0.00000
2.6.0-test6                128    2.71  2.80%   534.963    33603.12 10.19945
2.6.0-test6-mm4            128    2.76  2.91%   524.334    34758.05  9.84798
2.6.0-test7                128    2.66  2.72%   543.918    33661.62 10.33516

2.6.0-test5                256    8.88 15.47%   327.283     2378.44  0.00000
2.6.0-test5-mm2            256    8.59 15.37%   339.514     1905.29  0.00000
2.6.0-test5-mm4            256    8.60 16.11%   338.642     1890.15  0.00000
2.6.0-test6                256    2.29  2.42%  1177.401   291951.14 11.55243
2.6.0-test6-mm4            256    2.32  2.50%  1158.512   343146.47 11.20739
2.6.0-test7                256    2.28  2.39%  1172.902   277837.90 11.42778

Random Reads reiserfs
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1    0.88  0.95%    13.375      121.48  0.00000
2.6.0-test5-mm2              1    1.01  1.08%    11.543      382.20  0.00000
2.6.0-test5-mm4              1    0.86  1.11%    13.564      123.23  0.00000
2.6.0-test6                  1    0.96  1.10%    12.246      127.82  0.00000
2.6.0-test6-mm4              1    1.02  1.16%    11.496      391.26  0.00000
2.6.0-test7                  1    0.99  1.04%    11.863      111.96  0.00000

2.6.0-test5                  8    4.59  5.28%    17.392      126.60  0.00000
2.6.0-test5-mm2              8    4.88  5.41%    15.978      105.27  0.00000
2.6.0-test5-mm4              8    4.64  5.92%    17.234      119.30  0.00000
2.6.0-test6                  8    3.12  3.28%    27.584      373.26  0.00000
2.6.0-test6-mm4              8    3.80  4.12%    23.064      280.06  0.00000
2.6.0-test7                  8    3.35  3.69%    24.895      337.23  0.00000

2.6.0-test5                 16    4.80  4.33%    34.457      189.80  0.00000
2.6.0-test5-mm2             16    5.27  5.28%    30.882      160.61  0.00000
2.6.0-test5-mm4             16    5.22  5.44%    32.357      494.99  0.00000
2.6.0-test6                 16    3.93  4.12%    43.768      728.55  0.00000
2.6.0-test6-mm4             16    3.96  3.79%    41.680      775.69  0.00000
2.6.0-test7                 16    3.96  3.85%    42.797      705.06  0.00000

2.6.0-test5                 32    5.45  5.07%    59.014      323.30  0.00000
2.6.0-test5-mm2             32    5.65  5.45%    57.673      313.96  0.00000
2.6.0-test5-mm4             32    4.47  4.84%    75.004      455.38  0.00000
2.6.0-test6                 32    4.39  4.58%    77.879     1232.67  0.00000
2.6.0-test6-mm4             32    4.15  4.34%    80.355     1229.75  0.00000
2.6.0-test7                 32    4.88  4.59%    67.103     1033.43  0.00000

2.6.0-test5                 64    5.49  5.42%   108.692      991.66  0.00000
2.6.0-test5-mm2             64    5.10  5.60%   113.940      914.35  0.00000
2.6.0-test5-mm4             64    4.72  5.35%   126.505     1182.30  0.00000
2.6.0-test6                 64    3.90  4.03%   165.855     2554.36  0.00000
2.6.0-test6-mm4             64    4.54  4.72%   140.927     2060.27  0.00000
2.6.0-test7                 64    4.95  4.73%   129.947     2011.81  0.00000

2.6.0-test5                128    5.48  5.99%   221.089     2074.91  0.00000
2.6.0-test5-mm2            128    5.71  6.49%   208.548     1781.53  0.00000
2.6.0-test5-mm4            128    4.94  5.94%   240.781     2090.92  0.00000
2.6.0-test6                128    3.55  4.55%   367.628     5086.89  0.00000
2.6.0-test6-mm4            128    4.15  4.50%   314.516     3914.52  0.00000
2.6.0-test7                128    3.98  4.56%   328.297     4300.67  0.00000

2.6.0-test5                256    5.39  7.63%   455.466     2448.36  0.00000
2.6.0-test5-mm2            256    5.85  8.92%   415.957     2773.47  0.00000
2.6.0-test5-mm4            256    4.88  8.30%   498.235     2859.83  0.00000
2.6.0-test6                256    2.88  4.24%   861.877     8947.59  7.13542
2.6.0-test6-mm4            256    3.28  4.52%   752.088     7585.14  3.09895
2.6.0-test7                256    3.11  4.19%   794.027     8045.64  3.82812

Sequential Writes reiserf
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1   55.30 90.83%     0.200     1317.87  0.00000
2.6.0-test5-mm2              1   55.08 92.70%     0.201     1512.52  0.00000
2.6.0-test5-mm4              1   54.62 92.30%     0.202     2146.88  0.00000
2.6.0-test6                  1   55.09 91.14%     0.201     1243.21  0.00000
2.6.0-test6-mm4              1   54.46 91.70%     0.202     1725.38  0.00000
2.6.0-test7                  1   55.08 91.27%     0.200     2381.55  0.00000

2.6.0-test5                  8   27.22 99.99%     2.834    62892.07  0.02932
2.6.0-test5-mm2              8   27.84 100.4%     2.721    20683.58  0.03290
2.6.0-test5-mm4              8   28.18 96.26%     2.667    24253.31  0.03237
2.6.0-test6                  8   28.88 102.9%     2.572    28097.64  0.03533
2.6.0-test6-mm4              8   26.57 92.64%     2.855    24274.27  0.05517
2.6.0-test7                  8   28.60 104.5%     2.571    19829.03  0.02890

2.6.0-test5                 16   30.58 102.6%     4.745    29788.03  0.07806
2.6.0-test5-mm2             16   28.94 101.7%     5.051    37491.58  0.07902
2.6.0-test5-mm4             16   28.45 97.86%     4.971    28952.01  0.08851
2.6.0-test6                 16   27.13 98.05%     5.414    58605.69  0.08331
2.6.0-test6-mm4             16   27.04 89.13%     5.347    24565.13  0.08516
2.6.0-test7                 16   29.19 98.45%     5.031    29653.75  0.08398

2.6.0-test5                 32   27.88 104.6%    10.012    44186.44  0.11630
2.6.0-test5-mm2             32   27.15 104.7%     9.911    47326.17  0.11076
2.6.0-test5-mm4             32   27.57 102.6%     9.820    88173.93  0.10858
2.6.0-test6                 32   27.80 100.8%     9.951    72577.05  0.11445
2.6.0-test6-mm4             32   25.80 97.74%    10.563    65400.15  0.11706
2.6.0-test7                 32   28.65 105.3%     9.613    44890.58  0.11087

2.6.0-test5                 64   27.12 109.5%    19.091    93664.16  0.15073
2.6.0-test5-mm2             64   25.80 117.2%    20.100    73309.25  0.18029
2.6.0-test5-mm4             64   25.64 112.9%    19.408    76139.55  0.16613
2.6.0-test6                 64   26.30 112.0%    19.162    93011.66  0.16308
2.6.0-test6-mm4             64   24.15 104.3%    21.613    86562.64  0.18483
2.6.0-test7                 64   25.10 107.7%    20.311    80093.32  0.16470

2.6.0-test5                128   23.52 119.4%    42.286   116297.76  0.39005
2.6.0-test5-mm2            128   24.88 127.8%    39.106   139584.40  0.35658
2.6.0-test5-mm4            128   24.28 127.9%    39.111   159921.42  0.37918
2.6.0-test6                128   24.79 122.4%    41.723   127648.00  0.39692
2.6.0-test6-mm4            128   24.48 125.2%    39.716   128936.67  0.38275
2.6.0-test7                128   24.29 122.0%    40.692   152191.29  0.35692

2.6.0-test5                256   19.14 163.4%   112.557   239944.32  1.00599
2.6.0-test5-mm2            256   23.21 153.9%    84.422   200276.08  0.75154
2.6.0-test5-mm4            256   23.45 161.4%    86.423   216520.39  0.71788
2.6.0-test6                256   23.14 168.2%    83.087   247571.24  0.75640
2.6.0-test6-mm4            256   20.87 136.1%    74.033   177291.47  0.66229
2.6.0-test7                256   23.11 142.0%    79.651   233921.91  0.66109

Random Writes reiserfs
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1    2.41  2.77%     1.925      282.65  0.00000
2.6.0-test5-mm2              1    3.71  4.05%     0.222       18.72  0.00000
2.6.0-test5-mm4              1    2.29  2.87%     2.191      299.62  0.00000
2.6.0-test6                  1    2.89  3.32%     1.040       91.48  0.00000
2.6.0-test6-mm4              1    3.60  3.66%     0.228       24.13  0.00000
2.6.0-test7                  1    3.58  3.58%     0.347       30.23  0.00000

2.6.0-test5                  8    3.50  4.12%    11.316     1348.95  0.00000
2.6.0-test5-mm2              8    3.74  5.07%    10.135     1133.62  0.00000
2.6.0-test5-mm4              8    3.54  4.73%    10.168     1285.63  0.00000
2.6.0-test6                  8    3.26  4.02%     9.287     1182.99  0.00000
2.6.0-test6-mm4              8    3.39  4.53%     5.611     1106.14  0.00000
2.6.0-test7                  8    3.29  3.77%    10.518     1191.14  0.00000

2.6.0-test5                 16    3.44  4.05%    22.370     4353.76  0.10000
2.6.0-test5-mm2             16    3.54  3.72%    22.545     1478.64  0.00000
2.6.0-test5-mm4             16    3.61  4.71%    20.853     1777.27  0.00000
2.6.0-test6                 16    3.60  4.11%    17.724     1421.63  0.00000
2.6.0-test6-mm4             16    3.37  3.72%    23.097     1422.10  0.00000
2.6.0-test7                 16    3.46  4.40%    16.822     4822.00  0.02500

2.6.0-test5                 32    3.68  4.57%    22.011     1260.71  0.00000
2.6.0-test5-mm2             32    3.60  4.50%    26.561     1836.41  0.00000
2.6.0-test5-mm4             32    3.55  4.32%    33.055     1460.77  0.00000
2.6.0-test6                 32    3.41  4.07%    23.498     2013.53  0.00000
2.6.0-test6-mm4             32    3.42  4.36%    13.875     1214.49  0.00000
2.6.0-test7                 32    3.42  4.04%    26.600     3437.82  0.02500

2.6.0-test5                 64    3.62  5.09%    37.320     8700.40  0.90725
2.6.0-test5-mm2             64    3.81  6.11%    18.225     1814.21  0.00000
2.6.0-test5-mm4             64    3.37  5.09%    43.372     3280.58  0.00000
2.6.0-test6                 64    2.96  3.93%    73.537     4484.06  0.00000
2.6.0-test6-mm4             64    3.29  4.98%    22.802     2206.78  0.00000
2.6.0-test7                 64    3.42  5.00%    27.782     2453.44  0.00000

2.6.0-test5                128    3.73  5.92%    48.524     7848.01  1.10887
2.6.0-test5-mm2            128    3.39  5.91%    36.168     5651.07  0.25202
2.6.0-test5-mm4            128    3.55  6.74%    70.991     6856.35  1.94052
2.6.0-test6                128    3.16  5.58%    43.277     5286.77  0.05040
2.6.0-test6-mm4            128    3.17  5.25%    43.818     5534.13  0.30242
2.6.0-test7                128    3.26  5.26%    39.051     3037.10  0.00000

2.6.0-test5                256    3.96  8.27%     8.806      963.34  0.00000
2.6.0-test5-mm2            256    3.67  7.17%    20.000     1575.35  0.00000
2.6.0-test5-mm4            256    3.36  6.99%   114.310     4266.84  1.61458
2.6.0-test6                256    3.10  6.69%   132.312     8228.26  3.30729
2.6.0-test6-mm4            256    3.21  6.79%    88.532     8631.04  1.79687
2.6.0-test7                256    3.23  6.61%    48.643     6005.85  0.75521

Sequential Reads xfs
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1   30.46 14.80%     0.382      718.97  0.00000
2.6.0-test5-mm2              1   28.43 14.31%     0.409      729.27  0.00000
2.6.0-test5-mm4              1   29.25 14.50%     0.398      751.10  0.00000
2.6.0-test6                  1   29.25 14.52%     0.398      735.45  0.00000
2.6.0-test6-mm4              1   28.68 14.60%     0.406      924.52  0.00000
2.6.0-test7                  1   29.20 14.46%     0.398      727.21  0.00000

2.6.0-test5                  8   55.34 28.06%     1.652      831.66  0.00000
2.6.0-test5-mm2              8   54.19 28.05%     1.707      852.34  0.00000
2.6.0-test5-mm4              8   55.49 28.38%     1.591     1165.42  0.00000
2.6.0-test6                  8   25.17 11.78%     3.562     1987.22  0.00000
2.6.0-test6-mm4              8   23.25 11.32%     3.989     1872.56  0.00000
2.6.0-test7                  8   25.91 12.19%     3.443     1894.36  0.00000

2.6.0-test5                 16   47.04 23.31%     3.903      846.86  0.00000
2.6.0-test5-mm2             16   45.50 23.01%     4.086     1116.23  0.00000
2.6.0-test5-mm4             16   47.00 23.78%     3.827      822.79  0.00000
2.6.0-test6                 16   15.07  7.14%    12.090     3088.51  0.00000
2.6.0-test6-mm4             16   12.58  6.23%    14.655     3469.24  0.00000
2.6.0-test7                 16   14.79  7.03%    11.808     3178.50  0.00000

2.6.0-test5                 32   44.85 22.40%     7.699     1225.63  0.00000
2.6.0-test5-mm2             32   45.89 23.39%     7.624     1173.54  0.00000
2.6.0-test5-mm4             32   44.18 22.68%     7.940     1051.95  0.00000
2.6.0-test6                 32   14.80  7.02%    24.021     5758.07  0.00005
2.6.0-test6-mm4             32   14.18  6.98%    25.241     5709.74  0.00000
2.6.0-test7                 32   12.46  6.03%    29.086     5836.49  0.00029

2.6.0-test5                 64   42.08 21.45%    16.269     1897.20  0.00000
2.6.0-test5-mm2             64   42.36 21.75%    15.948     1925.10  0.00000
2.6.0-test5-mm4             64   43.77 22.73%    15.396     1963.29  0.00000
2.6.0-test6                 64   12.50  5.98%    53.864    10887.40  1.74727
2.6.0-test6-mm4             64   12.37  6.19%    55.962    10690.33  1.82176
2.6.0-test7                 64   12.25  5.95%    56.211    10899.32  1.78017

2.6.0-test5                128   15.41 12.23%    87.001     2911.16  0.00000
2.6.0-test5-mm2            128   20.45 15.21%    61.549     2516.86  0.00000
2.6.0-test5-mm4            128   18.55 14.38%    68.572     2527.97  0.00000
2.6.0-test6                128    5.20  4.07%   251.782    18918.92  4.34985
2.6.0-test6-mm4            128    4.47  3.56%   290.774    19729.60  5.13649
2.6.0-test7                128    3.50  3.19%   385.879    20057.71  6.42790

2.6.0-test5                256   10.33 13.76%   225.449     6314.51  0.00167
2.6.0-test5-mm2            256   10.58 14.19%   206.025     9145.94  0.00219
2.6.0-test5-mm4            256   11.64 15.40%   175.294    11398.84  0.00677
2.6.0-test6                256    3.84  3.42%   539.606   972621.75  5.75909
2.6.0-test6-mm4            256    3.92  3.35%   527.548   867696.31  5.66021
2.6.0-test7                256    3.99  3.52%   511.822   865743.62  5.56827

Random Reads xfs
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1    1.03  0.94%    11.357      107.82  0.00000
2.6.0-test5-mm2              1    1.03  1.02%    11.325      117.87  0.00000
2.6.0-test5-mm4              1    1.04  1.03%    11.261      107.89  0.00000
2.6.0-test6                  1    1.03  0.92%    11.386      109.39  0.00000
2.6.0-test6-mm4              1    1.00  1.04%    11.659      119.70  0.00000
2.6.0-test7                  1    1.05  0.92%    11.161       96.11  0.00000

2.6.0-test5                  8    5.42  5.19%    16.257      124.33  0.00000
2.6.0-test5-mm2              8    5.36  5.79%    16.591      120.16  0.00000
2.6.0-test5-mm4              8    4.67  4.80%    16.448      104.77  0.00000
2.6.0-test6                  8    2.08  1.68%    38.061      775.61  0.00000
2.6.0-test6-mm4              8    2.16  1.95%    37.981      686.42  0.00000
2.6.0-test7                  8    1.88  1.64%    42.173      683.55  0.00000

2.6.0-test5                 16    5.20  4.62%    34.058      146.53  0.00000
2.6.0-test5-mm2             16    5.21  4.88%    34.306      175.88  0.00000
2.6.0-test5-mm4             16    5.18  4.84%    32.751      135.47  0.00000
2.6.0-test6                 16    1.86  1.44%    85.898     1244.64  0.00000
2.6.0-test6-mm4             16    1.95  1.64%    80.971     1175.91  0.00000
2.6.0-test7                 16    2.35  1.98%    67.271     1001.72  0.00000

2.6.0-test5                 32    4.91  3.98%    68.586      265.91  0.00000
2.6.0-test5-mm2             32    5.43  4.74%    63.283      279.95  0.00000
2.6.0-test5-mm4             32    4.69  4.28%    68.452      638.24  0.00000
2.6.0-test6                 32    2.92  2.41%   113.627     1626.83  0.00000
2.6.0-test6-mm4             32    3.14  2.92%   105.714     1451.06  0.00000
2.6.0-test7                 32    2.43  1.91%   134.936     1864.81  0.00000

2.6.0-test5                 64    4.76  4.02%   136.805      879.54  0.00000
2.6.0-test5-mm2             64    4.66  4.68%   136.498      871.90  0.00000
2.6.0-test5-mm4             64    4.81  4.75%   131.041      994.54  0.00000
2.6.0-test6                 64    1.45  1.25%   425.661     5262.95  0.02520
2.6.0-test6-mm4             64    1.66  1.40%   382.569     4759.16  0.00000
2.6.0-test7                 64    1.41  1.21%   444.027     5820.44  0.30242

2.6.0-test5                128    4.70  4.50%   248.562     1396.77  0.00000
2.6.0-test5-mm2            128    5.29  5.72%   206.313     1395.33  0.00000
2.6.0-test5-mm4            128    5.16  4.95%   209.001     1485.37  0.00000
2.6.0-test6                128    1.65  1.57%   753.959     9498.63  9.02218
2.6.0-test6-mm4            128    1.47  1.43%   833.261     9951.50 11.54233
2.6.0-test7                128    1.23  1.06%   969.453    10793.66 18.04436

2.6.0-test5                256    4.82  6.71%   469.459     2140.97  0.00000
2.6.0-test5-mm2            256    4.60  6.06%   477.682     1934.27  0.00000
2.6.0-test5-mm4            256    5.01  6.60%   402.622     1881.24  0.00000
2.6.0-test6                256    0.90  0.99%  2204.619    32794.62 42.68229
2.6.0-test6-mm4            256    0.88  0.98%  2165.705    33928.64 41.09376
2.6.0-test7                256    0.92  1.05%  2113.981    33637.27 39.11458

Sequential Writes xfs
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1   56.44 46.30%     0.182    68379.68  0.00030
2.6.0-test5-mm2              1   56.33 47.39%     0.186    68474.74  0.00030
2.6.0-test5-mm4              1   56.30 46.74%     0.182    68268.99  0.00030
2.6.0-test6                  1   56.24 46.08%     0.177    68501.77  0.00030
2.6.0-test6-mm4              1   56.17 47.57%     0.175    68404.17  0.00030
2.6.0-test7                  1   56.26 46.03%     0.171    68451.18  0.00030

2.6.0-test5                  8   57.54 65.47%     0.991   166706.97  0.00424
2.6.0-test5-mm2              8   57.70 70.44%     1.099   211950.97  0.00420
2.6.0-test5-mm4              8   57.23 66.09%     1.042   205671.61  0.00415
2.6.0-test6                  8   57.19 69.31%     1.058   185132.73  0.00410
2.6.0-test6-mm4              8   57.46 66.82%     1.100   178544.18  0.00486
2.6.0-test7                  8   57.24 64.93%     1.096   207662.87  0.00481

2.6.0-test5                 16   57.26 71.20%     2.069   186551.11  0.01030
2.6.0-test5-mm2             16   57.26 72.02%     2.079   187786.90  0.01059
2.6.0-test5-mm4             16   57.38 68.91%     1.999   193908.75  0.00958
2.6.0-test6                 16   57.24 70.72%     2.072   185124.29  0.01092
2.6.0-test6-mm4             16   57.37 68.99%     2.080   208981.25  0.01045
2.6.0-test7                 16   57.17 68.79%     2.051   181396.29  0.01135

2.6.0-test5                 32   57.29 75.00%     4.076   211670.36  0.01840
2.6.0-test5-mm2             32   57.34 73.21%     3.768   213556.33  0.01560
2.6.0-test5-mm4             32   57.27 75.59%     4.007   199887.88  0.01708
2.6.0-test6                 32   57.12 73.89%     4.007   198273.60  0.01750
2.6.0-test6-mm4             32   57.04 73.27%     4.034   181071.17  0.01688
2.6.0-test7                 32   57.04 72.85%     4.066   193571.15  0.02007

2.6.0-test5                 64   57.13 83.01%     8.186   336901.61  0.03567
2.6.0-test5-mm2             64   56.93 83.63%     8.062   340355.98  0.03066
2.6.0-test5-mm4             64   56.33 82.10%     7.834   325305.64  0.02895
2.6.0-test6                 64   56.48 80.10%     7.829   306781.15  0.03157
2.6.0-test6-mm4             64   56.87 81.05%     7.808   257430.27  0.03662
2.6.0-test7                 64   56.54 77.01%     7.746   347269.10  0.02880

2.6.0-test5                128   56.58 86.45%    15.451   330514.59  0.06585
2.6.0-test5-mm2            128   56.78 82.31%    14.164   364774.73  0.05298
2.6.0-test5-mm4            128   55.57 82.39%    14.088   350137.61  0.05364
2.6.0-test6                128   56.08 81.22%    13.917   377412.59  0.04669
2.6.0-test6-mm4            128   56.43 79.05%    14.332   357252.17  0.05203
2.6.0-test7                128   56.28 77.14%    14.465   351042.94  0.05546

2.6.0-test5                256   56.13 83.69%    19.836   383235.17  0.07386
2.6.0-test5-mm2            256   56.19 80.90%    16.880   294598.89  0.07434
2.6.0-test5-mm4            256   56.09 87.58%    13.343   308894.90  0.05746
2.6.0-test6                256   55.39 80.57%    18.269   350145.38  0.07200
2.6.0-test6-mm4            256   55.67 81.98%    17.338   337290.04  0.06890
2.6.0-test7                256   55.49 82.29%    15.776   351541.33  0.06071

Random Writes xfs
                           Num                    Avg       Maximum     Lat%
Kernel                     Thr   Rate  (CPU%)   Latency     Latency      >2s
-------------------------  ---  --------------------------------------------
2.6.0-test5                  1    3.80  3.61%     0.074        0.46  0.00000
2.6.0-test5-mm2              1    3.86  3.77%     0.077        0.42  0.00000
2.6.0-test5-mm4              1    3.77  3.65%     0.078        1.42  0.00000
2.6.0-test6                  1    3.76  3.53%     0.075        1.43  0.00000
2.6.0-test6-mm4              1    3.71  3.64%     0.078        0.68  0.00000
2.6.0-test7                  1    3.76  3.53%     0.075        0.87  0.00000

2.6.0-test5                  8    3.97  6.78%     0.202       95.59  0.00000
2.6.0-test5-mm2              8    3.91  6.75%     0.181       75.97  0.00000
2.6.0-test5-mm4              8    3.87  6.57%     0.191       81.43  0.00000
2.6.0-test6                  8    3.86  6.63%     0.165       25.62  0.00000
2.6.0-test6-mm4              8    3.82  6.33%     0.166       39.50  0.00000
2.6.0-test7                  8    3.92  6.44%     0.181       74.95  0.00000

2.6.0-test5                 16    3.95  6.81%     0.168       32.20  0.00000
2.6.0-test5-mm2             16    3.94  6.99%     0.175       48.13  0.00000
2.6.0-test5-mm4             16    3.82  6.53%     0.186       87.83  0.00000
2.6.0-test6                 16    3.76  6.58%     0.168       25.32  0.00000
2.6.0-test6-mm4             16    3.81  6.62%     0.193       83.97  0.00000
2.6.0-test7                 16    3.89  6.57%     0.181       92.60  0.00000

2.6.0-test5                 32    3.85  6.64%     0.159        3.38  0.00000
2.6.0-test5-mm2             32    3.80  6.75%     0.175       59.86  0.00000
2.6.0-test5-mm4             32    3.72  6.60%     0.180       55.04  0.00000
2.6.0-test6                 32    3.80  6.73%     0.175       56.73  0.00000
2.6.0-test6-mm4             32    3.81  6.61%     0.171       55.65  0.00000
2.6.0-test7                 32    3.60  5.97%     0.155        3.09  0.00000

2.6.0-test5                 64    3.68  6.56%     0.161        3.90  0.00000
2.6.0-test5-mm2             64    3.60  6.51%     0.163        6.67  0.00000
2.6.0-test5-mm4             64    3.71  6.58%     0.166        5.44  0.00000
2.6.0-test6                 64    3.53  6.32%     0.160        4.14  0.00000
2.6.0-test6-mm4             64    3.54  6.21%     0.185      106.22  0.00000
2.6.0-test7                 64    3.61  6.15%     0.156        4.34  0.00000

2.6.0-test5                128    3.32  6.23%     0.166        4.42  0.00000
2.6.0-test5-mm2            128    3.64  6.70%     0.163        2.98  0.00000
2.6.0-test5-mm4            128    3.71  6.85%     0.169        5.37  0.00000
2.6.0-test6                128    3.44  6.25%     0.174       63.35  0.00000
2.6.0-test6-mm4            128    3.49  6.36%     0.174       64.77  0.00000
2.6.0-test7                128    3.43  6.15%     0.159        4.05  0.00000

2.6.0-test5                256    3.35  6.56%     0.161        3.65  0.00000
2.6.0-test5-mm2            256    3.29  6.41%     0.164        3.68  0.00000
2.6.0-test5-mm4            256    3.27  6.35%     0.166        4.84  0.00000
2.6.0-test6                256    2.97  5.80%     0.171       47.92  0.00000
2.6.0-test6-mm4            256    3.06  5.93%     0.159        3.02  0.00000
2.6.0-test7                256    3.11  5.88%     0.159        3.91  0.00000


dbench did not regress.  

dbench-1.3 reiserfs 192 processes     Average		+/- MB/sec
2.6.0-test7                             77.3		   7.1
2.6.0-test6-mm4                         76.6		   7.2
2.6.0-test5-mm2                         76.1		   6.8
2.6.0-test6                             73.0		   5.1
2.6.0-test5-mm4                         72.5		   6.6
2.6.0-test5                             65.0		   1.2

dbench-1.3 reiserfs 64 processes      Average		+/- MB/sec
2.6.0-test5-mm4                        111.4		   9.1
2.6.0-test6-mm4                        109.6		   7.5
2.6.0-test6                            108.8		   4.2
2.6.0-test5-mm2                        104.4		   8.1
2.6.0-test7                            102.3		   8.1
2.6.0-test5                            100.6		   4.7

dbench-1.3 xfs 192 processes          Average		+/- MB/sec
2.6.0-test5                            112.5		   4.0
2.6.0-test5-mm2                        110.8		   1.6
2.6.0-test6-mm4                        110.7		   3.4
2.6.0-test5-mm4                        109.7		   2.0
2.6.0-test7                            109.0		   5.1
2.6.0-test6                            108.5		   6.1

dbench-1.3 xfs 64 processes           Average		+/- MB/sec
2.6.0-test5                            143.7		   0.9
2.6.0-test7                            143.6		   2.1
2.6.0-test5-mm2                        141.5		   1.6
2.6.0-test6                            141.4		   0.5
2.6.0-test6-mm4                        141.2		   1.5
2.6.0-test5-mm4                        140.0		   1.4

dbench-1.3 ext2 192 processes         Average		+/- MB/sec
2.6.0-test6                            209.5		  11.8
2.6.0-test7                            205.2		  20.8
2.6.0-test5-mm4                        200.8		   6.2
2.6.0-test5-mm2                        200.0		  10.3
2.6.0-test5                            198.1		  19.5
2.6.0-test6-mm4                        193.4		  12.0

dbench-1.3 ext2 64 processes          Average		+/- MB/sec
2.6.0-test7                            256.8		   0.6
2.6.0-test6                            256.7		   1.6
2.6.0-test5                            256.2		   1.6
2.6.0-test5-mm4                        255.5		   1.4
2.6.0-test5-mm2                        255.3		   0.9
2.6.0-test6-mm4                        255.3		   1.7

dbench-1.3 jfs 192 processes          Average		+/- MB/sec
2.6.0-test5-mm4                         11.3		   0.5
2.6.0-test7                             11.1		   0.2
2.6.0-test6-mm4                         11.0		   0.3

dbench-1.3 jfs 64 processes           Average		+/- MB/sec
2.6.0-test5-mm2                         20.0		   2.9
2.6.0-test5                             19.5		   2.4
2.6.0-test5-mm4                         18.6		   2.4
2.6.0-test7                             18.5		   1.1
2.6.0-test6-mm4                         18.1		   1.6

dbench-1.3 ext3 192 processes         Average		+/- MB/sec
2.6.0-test7                            110.8		  23.1
2.6.0-test5-mm2                        107.5		  39.7
2.6.0-test6                            103.9		  39.0
2.6.0-test6-mm4                         99.7		  38.7
2.6.0-test5-mm4                         82.6		  35.5
2.6.0-test5                             73.6		  15.1

dbench-1.3 ext3 64 processes          Average		+/- MB/sec
2.6.0-test7                            177.9		   3.4
2.6.0-test6                            175.1		   3.6
2.6.0-test5                            174.5		   4.2
2.6.0-test6-mm4                        174.2		   2.5
2.6.0-test5-mm2                        172.8		   4.0
2.6.0-test5-mm4                        172.6		   2.3


-- 
Randy Hron
http://home.earthlink.net/~rwhron/kernel/bigbox.html


^ permalink raw reply	[flat|nested] 25+ messages in thread

end of thread, other threads:[~2003-10-29  0:35 UTC | newest]

Thread overview: 25+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-10-29  0:38 [BENCHMARK] I/O regression after 2.6.0-test5 rwhron
  -- strict thread matches above, loose matches on Subject: below --
2003-10-27 23:33 John Hawkes
2003-10-26 10:38 rwhron
2003-10-27 23:49 ` Dave Olien
2003-10-24 23:43 rwhron
2003-10-24 12:46 rwhron
2003-10-24  0:10 rwhron
2003-10-21 13:05 rwhron
2003-10-21 15:57 ` Nick Piggin
2003-10-21 21:07 ` venom
2003-10-22  9:54 ` Nick Piggin
2003-10-22 18:30   ` Dave Olien
2003-10-22 19:08     ` Dave Olien
2003-10-23  2:24     ` Nick Piggin
2003-10-23  3:48       ` Nick Piggin
2003-10-23 20:35       ` Dave Olien
2003-10-23 23:07         ` Nick Piggin
2003-10-20  0:37 rwhron
2003-10-20  1:35 ` Nick Piggin
2003-10-20  4:52   ` Andrew Morton
2003-10-20  8:12     ` Nick Piggin
2003-10-20  4:51 ` Dave Olien
2003-10-20  7:55   ` venom
2003-10-21  4:49     ` Nick Piggin
2003-10-24 13:52 ` Magnus Andersson

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).