file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.0 Starting 8 processes Jobs: 8 (f=8): [W(8)][100.0%][r=0KiB/s,w=82.1MiB/s][r=0,w=21.0k IOPS][eta 00m:00s] file1: (groupid=0, jobs=1): err= 0: pid=1178: Wed Dec 13 14:21:23 2017 write: IOPS=2809, BW=10.0MiB/s (11.5MB/s)(6584MiB/600001msec) slat (usec): min=5, max=1731.0k, avg=321.00, stdev=4481.26 clat (usec): min=76, max=1736.5k, avg=5171.63, stdev=17687.91 lat (usec): min=141, max=1736.8k, avg=5495.42, stdev=18290.97 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2409], 10.00th=[ 2802], 20.00th=[ 3195], | 30.00th=[ 3556], 40.00th=[ 3818], 50.00th=[ 4015], 60.00th=[ 4228], | 70.00th=[ 4490], 80.00th=[ 4883], 90.00th=[ 5407], 95.00th=[ 6063], | 99.00th=[ 14484], 99.50th=[ 69731], 99.90th=[250610], 99.95th=[337642], | 99.99th=[574620] bw ( KiB/s): min= 8, max=24673, per=13.42%, avg=11763.36, stdev=3943.17, samples=1148 iops : min= 2, max= 6168, avg=2940.54, stdev=985.79, samples=1148 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.03%, 1000=0.12% lat (msec) : 2=1.21%, 4=47.10%, 10=50.36%, 20=0.29%, 50=0.26% lat (msec) : 100=0.21%, 250=0.29%, 500=0.09%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.68%, sys=79.81%, ctx=652388, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1685538,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1179: Wed Dec 13 14:21:23 2017 write: IOPS=2557, BW=9.99MiB/s (10.5MB/s)(5993MiB/600001msec) slat (usec): min=5, max=1827.2k, avg=355.53, stdev=5641.37 clat (usec): min=54, max=1833.8k, avg=5680.10, stdev=22700.65 lat (usec): min=139, max=1834.2k, avg=6038.34, stdev=23494.13 clat percentiles (usec): | 1.00th=[ 1778], 5.00th=[ 2343], 10.00th=[ 2704], 20.00th=[ 3097], | 30.00th=[ 3490], 40.00th=[ 3752], 50.00th=[ 3982], 60.00th=[ 4178], | 70.00th=[ 4424], 80.00th=[ 4817], 90.00th=[ 5407], 95.00th=[ 6128], | 99.00th=[ 35390], 99.50th=[122160], 99.90th=[320865], 99.95th=[387974], | 99.99th=[918553] bw ( KiB/s): min= 24, max=26497, per=12.28%, avg=10761.46, stdev=4879.74, samples=1143 iops : min= 6, max= 6624, avg=2690.01, stdev=1219.92, samples=1143 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.03%, 1000=0.14% lat (msec) : 2=1.48%, 4=49.67%, 10=47.13%, 20=0.31%, 50=0.34% lat (msec) : 100=0.29%, 250=0.42%, 500=0.15%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.63%, sys=72.85%, ctx=984304, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1534249,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1180: Wed Dec 13 14:21:23 2017 write: IOPS=2817, BW=11.0MiB/s (11.5MB/s)(6604MiB/600001msec) slat (usec): min=5, max=1751.3k, avg=320.34, stdev=4558.52 clat (usec): min=69, max=1755.7k, avg=5156.99, stdev=18063.06 lat (usec): min=138, max=1755.0k, avg=5480.10, stdev=18680.79 clat percentiles (usec): | 1.00th=[ 1876], 5.00th=[ 2409], 10.00th=[ 2802], 20.00th=[ 3195], | 30.00th=[ 3556], 40.00th=[ 3785], 50.00th=[ 4015], 60.00th=[ 4228], | 70.00th=[ 4490], 80.00th=[ 4883], 90.00th=[ 5407], 95.00th=[ 6063], | 99.00th=[ 14353], 99.50th=[ 67634], 99.90th=[263193], 99.95th=[337642], | 99.99th=[683672] bw ( KiB/s): min= 80, max=20811, per=13.49%, avg=11827.88, stdev=3987.58, samples=1146 iops : min= 20, max= 5202, avg=2956.61, stdev=996.89, samples=1146 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.04%, 1000=0.12% lat (msec) : 2=1.20%, 4=48.00%, 10=49.47%, 20=0.30%, 50=0.27% lat (msec) : 100=0.20%, 250=0.29%, 500=0.09%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.66%, sys=79.82%, ctx=604829, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1690561,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1181: Wed Dec 13 14:21:23 2017 write: IOPS=2837, BW=11.1MiB/s (11.6MB/s)(6651MiB/600001msec) slat (usec): min=5, max=1700.6k, avg=317.64, stdev=4338.76 clat (usec): min=36, max=1704.3k, avg=5119.44, stdev=16972.87 lat (usec): min=148, max=1704.8k, avg=5439.85, stdev=17539.15 clat percentiles (usec): | 1.00th=[ 1975], 5.00th=[ 2507], 10.00th=[ 2835], 20.00th=[ 3261], | 30.00th=[ 3589], 40.00th=[ 3851], 50.00th=[ 4047], 60.00th=[ 4293], | 70.00th=[ 4555], 80.00th=[ 4883], 90.00th=[ 5407], 95.00th=[ 6063], | 99.00th=[ 13829], 99.50th=[ 62129], 99.90th=[240124], 99.95th=[320865], | 99.99th=[608175] bw ( KiB/s): min= 48, max=23214, per=13.56%, avg=11881.25, stdev=3784.78, samples=1148 iops : min= 12, max= 5803, avg=2970.05, stdev=946.19, samples=1148 lat (usec) : 50=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.10% lat (msec) : 2=0.96%, 4=46.54%, 10=51.25%, 20=0.27%, 50=0.30% lat (msec) : 100=0.21%, 250=0.26%, 500=0.08%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.66%, sys=80.94%, ctx=559902, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1702735,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1182: Wed Dec 13 14:21:23 2017 write: IOPS=2498, BW=9994KiB/s (10.2MB/s)(5856MiB/600001msec) slat (usec): min=5, max=1758.3k, avg=364.78, stdev=5607.81 clat (usec): min=51, max=1763.5k, avg=5813.05, stdev=22946.68 lat (usec): min=137, max=1763.8k, avg=6180.47, stdev=23776.43 clat percentiles (usec): | 1.00th=[ 1745], 5.00th=[ 2376], 10.00th=[ 2769], 20.00th=[ 3228], | 30.00th=[ 3589], 40.00th=[ 3818], 50.00th=[ 4015], 60.00th=[ 4228], | 70.00th=[ 4490], 80.00th=[ 4883], 90.00th=[ 5407], 95.00th=[ 6128], | 99.00th=[ 39060], 99.50th=[128451], 99.90th=[341836], 99.95th=[434111], | 99.99th=[826278] bw ( KiB/s): min= 8, max=23134, per=11.96%, avg=10486.76, stdev=5172.17, samples=1146 iops : min= 2, max= 5783, avg=2621.32, stdev=1293.02, samples=1146 lat (usec) : 100=0.01%, 250=0.03%, 500=0.01%, 750=0.03%, 1000=0.13% lat (msec) : 2=1.44%, 4=47.17%, 10=49.65%, 20=0.28%, 50=0.33% lat (msec) : 100=0.29%, 250=0.42%, 500=0.17%, 750=0.02%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.63%, sys=72.48%, ctx=1079470, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1499031,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1183: Wed Dec 13 14:21:23 2017 write: IOPS=2743, BW=10.7MiB/s (11.2MB/s)(6429MiB/600001msec) slat (usec): min=5, max=1738.5k, avg=330.39, stdev=4825.27 clat (usec): min=46, max=1743.0k, avg=5295.46, stdev=19183.60 lat (usec): min=138, max=1744.3k, avg=5628.41, stdev=19855.76 clat percentiles (usec): | 1.00th=[ 1860], 5.00th=[ 2343], 10.00th=[ 2737], 20.00th=[ 3130], | 30.00th=[ 3523], 40.00th=[ 3785], 50.00th=[ 3982], 60.00th=[ 4178], | 70.00th=[ 4424], 80.00th=[ 4817], 90.00th=[ 5342], 95.00th=[ 6063], | 99.00th=[ 18744], 99.50th=[ 83362], 99.90th=[267387], 99.95th=[379585], | 99.99th=[675283] bw ( KiB/s): min= 64, max=23616, per=13.09%, avg=11474.50, stdev=4531.36, samples=1149 iops : min= 16, max= 5904, avg=2868.36, stdev=1132.85, samples=1149 lat (usec) : 50=0.01%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.20% lat (msec) : 2=1.28%, 4=48.96%, 10=48.23%, 20=0.32%, 50=0.30% lat (msec) : 100=0.25%, 250=0.31%, 500=0.10%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.63%, sys=77.48%, ctx=758312, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1645817,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1184: Wed Dec 13 14:21:23 2017 write: IOPS=2850, BW=11.1MiB/s (11.7MB/s)(6680MiB/600001msec) slat (usec): min=5, max=1758.4k, avg=316.41, stdev=4456.60 clat (usec): min=61, max=1763.5k, avg=5098.26, stdev=17493.16 lat (usec): min=140, max=1763.7k, avg=5417.38, stdev=18076.15 clat percentiles (usec): | 1.00th=[ 1942], 5.00th=[ 2442], 10.00th=[ 2802], 20.00th=[ 3228], | 30.00th=[ 3589], 40.00th=[ 3818], 50.00th=[ 4047], 60.00th=[ 4228], | 70.00th=[ 4490], 80.00th=[ 4883], 90.00th=[ 5407], 95.00th=[ 6063], | 99.00th=[ 13042], 99.50th=[ 58983], 99.90th=[248513], 99.95th=[333448], | 99.99th=[599786] bw ( KiB/s): min= 32, max=21843, per=13.64%, avg=11953.28, stdev=3732.91, samples=1147 iops : min= 8, max= 5460, avg=2987.95, stdev=933.22, samples=1147 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.17% lat (msec) : 2=1.03%, 4=47.02%, 10=50.64%, 20=0.30%, 50=0.29% lat (msec) : 100=0.19%, 250=0.25%, 500=0.08%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.70%, sys=80.63%, ctx=521547, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1710059,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1185: Wed Dec 13 14:21:23 2017 write: IOPS=2798, BW=10.9MiB/s (11.5MB/s)(6559MiB/600001msec) slat (usec): min=5, max=1848.6k, avg=322.54, stdev=4583.84 clat (usec): min=38, max=1854.1k, avg=5191.27, stdev=18112.72 lat (usec): min=141, max=1854.5k, avg=5516.58, stdev=18734.21 clat percentiles (usec): | 1.00th=[ 1909], 5.00th=[ 2409], 10.00th=[ 2769], 20.00th=[ 3163], | 30.00th=[ 3556], 40.00th=[ 3785], 50.00th=[ 4015], 60.00th=[ 4228], | 70.00th=[ 4490], 80.00th=[ 4883], 90.00th=[ 5407], 95.00th=[ 6063], | 99.00th=[ 16057], 99.50th=[ 70779], 99.90th=[270533], 99.95th=[358613], | 99.99th=[616563] bw ( KiB/s): min= 80, max=23390, per=13.36%, avg=11711.75, stdev=4136.20, samples=1148 iops : min= 20, max= 5847, avg=2927.75, stdev=1034.06, samples=1148 lat (usec) : 50=0.01%, 250=0.01%, 500=0.01%, 750=0.02%, 1000=0.14% lat (msec) : 2=1.13%, 4=48.61%, 10=48.88%, 20=0.29%, 50=0.29% lat (msec) : 100=0.22%, 250=0.28%, 500=0.10%, 750=0.01%, 1000=0.01% lat (msec) : 2000=0.01% cpu : usr=0.69%, sys=79.14%, ctx=652094, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1679114,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=85.6MiB/s (89.8MB/s), 9994KiB/s-11.1MiB/s (10.2MB/s-11.7MB/s), io=50.2GiB (53.8GB), run=600001-600001msec Disk stats (read/write): vdb: ios=0/32561, merge=0/6598, ticks=0/5542527, in_queue=3752820, util=15.70%