Ok, I grabbed Jens' patch from yesterday and made one small change to limit the total amount of io past blk_oversized_batch. The new patch is attached as axboe-2l.diff. We should probably drop the BH_Sync stuff, I need more time to make sure it is worth while. Also, the patch needs to be changed to calculate rlim once instead of in every get_request call. But I wanted to get some numbers out. Starting with my simple script yesterday to combine rio write loads and tiobench (8 tiotest threads): 2.4.22-axboe2 was Jens' patch (adds q->pending[rw]), 2.4.22-axboe-2l is from the patch attached. Sequential Reads File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.4.21 1792 4096 8 8.15 3.344% 3.564 9557.91 0.00022 0.00000 244 2.4.22-pre5 1792 4096 8 5.15 2.168% 5.964 2781.71 0.00022 0.00000 237 2.4.22-axboe2 1792 4096 8 5.07 2.119% 6.105 1095.73 0.00000 0.00000 239 2.4.22-axboe-2l 1792 4096 8 5.14 2.211% 5.998 970.74 0.00000 0.00000 232 Random Reads File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.4.21 1792 4096 8 0.83 0.318% 28.295 529.56 0.00000 0.00000 260 2.4.22-pre5 1792 4096 8 0.53 0.203% 45.613 770.82 0.00000 0.00000 260 2.4.22-axboe2 1792 4096 8 0.66 0.503% 41.791 637.23 0.00000 0.00000 130 2.4.22-axboe-2l 1792 4096 8 0.64 0.163% 46.923 742.38 0.00000 0.00000 391 Sequential Writes File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.4.21 1792 4096 8 15.51 11.40% 1.243 11776.20 0.01460 0.00109 136 2.4.22-pre5 1792 4096 8 11.99 9.960% 1.695 4628.58 0.00371 0.00000 120 2.4.22-axboe2 1792 4096 8 11.75 9.324% 1.755 2406.28 0.00065 0.00000 126 2.4.22-axboe-2l 1792 4096 8 11.73 9.320% 1.752 2924.61 0.00109 0.00000 126 Random Writes File Blk Num Avg Maximum Lat% Lat% CPU Identifier Size Size Thr Rate (CPU%) Latency Latency >2s >10s Eff ---------------- ------ ----- --- ------ ------ --------- ----------- -------- -------- ----- 2.4.21 1792 4096 8 0.41 0.318% 0.033 18.81 0.00000 0.00000 130 2.4.22-pre5 1792 4096 8 0.57 1.234% 0.041 37.63 0.00000 0.00000 46 2.4.22-axboe2 1792 4096 8 0.64 1.435% 0.040 29.32 0.00000 0.00000 45 2.4.22-axboe-2l 1792 4096 8 0.64 1.355% 0.033 20.79 0.00000 0.00000 47 So the max sequential read and write latencies were helped by elevator-low-latency while the averages were not. A good std deviation setup would help here, but tiobench's percentage setup shows some numbers. In practice, those max latencies are visible for interactive loads. Why did max sequential write latencies go down with -axboe? my guess is that we read the ext2 indirect blocks faster, but that's only a guess. For a little variety, I measured some read latencies in the face of a dbench 50 workload (after adding some latency stats to rio). These were done on scsi because my IDE drive wasn't big enough. You're looking at the last line from dbench (worthless) and the read throughput/latency stats from rio (rio was killed right after dbench finished). 2.4.21 Throughput 100.617 MB/sec (NB=125.771 MB/sec 1006.17 MBit/sec) 2.4.22-pre5 Throughput 111.563 MB/sec (NB=139.454 MB/sec 1115.63 MBit/sec) 2.4.21-axboe-2l Throughput 126.298 MB/sec (NB=157.873 MB/sec 1262.98 MBit/sec) 2.4.21 reads: 34.160 MB, 0.512 MB/s 2.4.22-pre5 reads: 34.254 MB, 0.569 MB/s 2.4.21-axboe-2l reads: 47.733 MB, 0.896 MB/s 2.4.21 read latency min 0.00 avg 7.54 max 5742.93 > 2s 0.00% > 10s 0.00% 2.4.22-pre5 read latency min 0.00 avg 6.74 max 7225.02 > 2s 0.00% > 10s 0.00% 2.4.21-axboe-2l read latency min 0.00 avg 4.32 max 2128.92 > 2s 0.00% > 10s 0.00% Both the dbench throughput numbers and the rio stats were fairly stable across runs. As the number of dbench procs increases vanilla 2.4.21 does worse on latency. -chris