file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.0 Starting 8 processes file1: Laying out IO file (1 file / 10240MiB) Jobs: 8 (f=8): [W(8)][100.0%][r=0KiB/s,w=23.9MiB/s][r=0,w=6130 IOPS][eta 00m:00s] file1: (groupid=0, jobs=1): err= 0: pid=1219: Wed Dec 13 16:25:59 2017 write: IOPS=1737, BW=6949KiB/s (7116kB/s)(4072MiB/600003msec) slat (usec): min=12, max=4505.4k, avg=556.60, stdev=17945.08 clat (usec): min=37, max=4509.3k, avg=8637.16, stdev=71426.10 lat (usec): min=261, max=4509.5k, avg=9196.04, stdev=73807.97 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 1336], 10.00th=[ 1762], | 20.00th=[ 2343], 30.00th=[ 2769], 40.00th=[ 2999], | 50.00th=[ 3326], 60.00th=[ 3523], 70.00th=[ 4015], | 80.00th=[ 4752], 90.00th=[ 5997], 95.00th=[ 6980], | 99.00th=[ 74974], 99.50th=[ 308282], 99.90th=[1098908], | 99.95th=[1518339], 99.99th=[2600469] bw ( KiB/s): min= 8, max=38012, per=17.25%, avg=8670.43, stdev=6123.73, samples=962 iops : min= 2, max= 9503, avg=2167.52, stdev=1530.93, samples=962 lat (usec) : 50=0.01%, 250=0.01%, 500=0.03%, 750=0.16%, 1000=2.57% lat (msec) : 2=8.62%, 4=58.13%, 10=28.77%, 20=0.27%, 50=0.27% lat (msec) : 100=0.24%, 250=0.32%, 500=0.29%, 750=0.12%, 1000=0.07% lat (msec) : 2000=0.10%, >=2000=0.02% cpu : usr=0.40%, sys=43.14%, ctx=761137, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1042323,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1220: Wed Dec 13 16:25:59 2017 write: IOPS=1523, BW=6094KiB/s (6240kB/s)(3571MiB/600001msec) slat (usec): min=11, max=3106.5k, avg=636.61, stdev=19297.35 clat (usec): min=8, max=3326.4k, avg=9841.72, stdev=76303.68 lat (usec): min=153, max=3326.7k, avg=10480.72, stdev=78931.46 clat percentiles (usec): | 1.00th=[ 881], 5.00th=[ 1336], 10.00th=[ 2024], | 20.00th=[ 2343], 30.00th=[ 2802], 40.00th=[ 3064], | 50.00th=[ 3359], 60.00th=[ 3589], 70.00th=[ 4146], | 80.00th=[ 4752], 90.00th=[ 5866], 95.00th=[ 6915], | 99.00th=[ 135267], 99.50th=[ 425722], 99.90th=[1182794], | 99.95th=[1551893], 99.99th=[2600469] bw ( KiB/s): min= 8, max=37835, per=15.13%, avg=7604.81, stdev=6137.33, samples=963 iops : min= 2, max= 9458, avg=1900.95, stdev=1534.30, samples=963 lat (usec) : 10=0.01%, 250=0.03%, 500=0.08%, 750=0.18%, 1000=2.16% lat (msec) : 2=7.24%, 4=57.60%, 10=30.73%, 20=0.28%, 50=0.36% lat (msec) : 100=0.23%, 250=0.35%, 500=0.31%, 750=0.19%, 1000=0.09% lat (msec) : 2000=0.14%, >=2000=0.02% cpu : usr=0.34%, sys=39.27%, ctx=1024014, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,914111,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1221: Wed Dec 13 16:25:59 2017 write: IOPS=1579, BW=6317KiB/s (6468kB/s)(3701MiB/600001msec) slat (usec): min=11, max=3378.7k, avg=613.95, stdev=19369.95 clat (usec): min=30, max=3382.7k, avg=9505.30, stdev=77110.08 lat (usec): min=236, max=3382.0k, avg=10121.57, stdev=79761.57 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 1385], 10.00th=[ 2040], | 20.00th=[ 2376], 30.00th=[ 2769], 40.00th=[ 3032], | 50.00th=[ 3326], 60.00th=[ 3589], 70.00th=[ 4178], | 80.00th=[ 4817], 90.00th=[ 6063], 95.00th=[ 6980], | 99.00th=[ 111674], 99.50th=[ 383779], 99.90th=[1233126], | 99.95th=[1551893], 99.99th=[2835350] bw ( KiB/s): min= 8, max=32344, per=15.92%, avg=7999.22, stdev=6196.12, samples=950 iops : min= 2, max= 8086, avg=1999.49, stdev=1549.00, samples=950 lat (usec) : 50=0.01%, 250=0.01%, 500=0.06%, 750=0.14%, 1000=1.96% lat (msec) : 2=7.38%, 4=57.50%, 10=31.13%, 20=0.25%, 50=0.32% lat (msec) : 100=0.23%, 250=0.35%, 500=0.30%, 750=0.14%, 1000=0.08% lat (msec) : 2000=0.13%, >=2000=0.03% cpu : usr=0.36%, sys=40.24%, ctx=845216, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,947496,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1222: Wed Dec 13 16:25:59 2017 write: IOPS=1504, BW=6019KiB/s (6163kB/s)(3527MiB/600003msec) slat (usec): min=11, max=3352.6k, avg=645.71, stdev=19845.49 clat (usec): min=28, max=3701.9k, avg=9978.31, stdev=79408.41 lat (usec): min=237, max=3702.1k, avg=10626.31, stdev=82130.21 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 1336], 10.00th=[ 1975], | 20.00th=[ 2343], 30.00th=[ 2802], 40.00th=[ 3032], | 50.00th=[ 3326], 60.00th=[ 3556], 70.00th=[ 4113], | 80.00th=[ 4752], 90.00th=[ 5997], 95.00th=[ 7046], | 99.00th=[ 132645], 99.50th=[ 434111], 99.90th=[1283458], | 99.95th=[1619002], 99.99th=[2600469] bw ( KiB/s): min= 7, max=35528, per=15.28%, avg=7681.10, stdev=5994.99, samples=940 iops : min= 1, max= 8882, avg=1920.22, stdev=1498.76, samples=940 lat (usec) : 50=0.01%, 250=0.01%, 500=0.03%, 750=0.14%, 1000=2.58% lat (msec) : 2=7.65%, 4=57.94%, 10=29.71%, 20=0.30%, 50=0.35% lat (msec) : 100=0.22%, 250=0.34%, 500=0.29%, 750=0.18%, 1000=0.10% lat (msec) : 2000=0.15%, >=2000=0.03% cpu : usr=0.32%, sys=38.58%, ctx=962514, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,902859,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1223: Wed Dec 13 16:25:59 2017 write: IOPS=1752, BW=7011KiB/s (7179kB/s)(4108MiB/600002msec) slat (usec): min=12, max=4505.4k, avg=550.95, stdev=16994.58 clat (usec): min=29, max=5045.7k, avg=8568.87, stdev=68091.35 lat (usec): min=241, max=5045.9k, avg=9122.18, stdev=70418.08 clat percentiles (usec): | 1.00th=[ 889], 5.00th=[ 1483], 10.00th=[ 2180], | 20.00th=[ 2442], 30.00th=[ 2900], 40.00th=[ 3195], | 50.00th=[ 3392], 60.00th=[ 3621], 70.00th=[ 4146], | 80.00th=[ 4817], 90.00th=[ 5997], 95.00th=[ 6915], | 99.00th=[ 78119], 99.50th=[ 320865], 99.90th=[1027605], | 99.95th=[1417675], 99.99th=[2533360] bw ( KiB/s): min= 8, max=35574, per=16.97%, avg=8529.81, stdev=6213.26, samples=989 iops : min= 2, max= 8893, avg=2132.10, stdev=1553.27, samples=989 lat (usec) : 50=0.01%, 250=0.02%, 500=0.07%, 750=0.08%, 1000=1.69% lat (msec) : 2=5.78%, 4=59.87%, 10=30.81%, 20=0.25%, 50=0.29% lat (msec) : 100=0.21%, 250=0.33%, 500=0.30%, 750=0.11%, 1000=0.08% lat (msec) : 2000=0.09%, >=2000=0.02% cpu : usr=0.42%, sys=44.71%, ctx=804193, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,1051592,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1224: Wed Dec 13 16:25:59 2017 write: IOPS=1457, BW=5831KiB/s (5971kB/s)(3417MiB/600001msec) slat (usec): min=11, max=3408.5k, avg=667.13, stdev=20675.53 clat (usec): min=31, max=3731.3k, avg=10298.98, stdev=83065.95 lat (usec): min=220, max=3731.4k, avg=10968.38, stdev=85913.83 clat percentiles (usec): | 1.00th=[ 824], 5.00th=[ 1254], 10.00th=[ 1500], | 20.00th=[ 2278], 30.00th=[ 2573], 40.00th=[ 2933], | 50.00th=[ 3195], 60.00th=[ 3425], 70.00th=[ 3851], | 80.00th=[ 4621], 90.00th=[ 5932], 95.00th=[ 6980], | 99.00th=[ 160433], 99.50th=[ 446694], 99.90th=[1300235], | 99.95th=[1736442], 99.99th=[2902459] bw ( KiB/s): min= 8, max=35558, per=15.21%, avg=7644.58, stdev=6275.71, samples=918 iops : min= 2, max= 8889, avg=1910.82, stdev=1568.89, samples=918 lat (usec) : 50=0.01%, 250=0.01%, 500=0.09%, 750=0.16%, 1000=3.44% lat (msec) : 2=8.67%, 4=59.58%, 10=26.04%, 20=0.23%, 50=0.35% lat (msec) : 100=0.26%, 250=0.38%, 500=0.35%, 750=0.16%, 1000=0.12% lat (msec) : 2000=0.14%, >=2000=0.03% cpu : usr=0.30%, sys=36.04%, ctx=992046, majf=0, minf=8 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,874718,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1225: Wed Dec 13 16:25:59 2017 write: IOPS=1595, BW=6382KiB/s (6535kB/s)(3739MiB/600001msec) slat (usec): min=11, max=3222.8k, avg=607.54, stdev=18877.57 clat (usec): min=27, max=3226.6k, avg=9412.18, stdev=74707.16 lat (usec): min=167, max=3226.8k, avg=10022.04, stdev=77241.60 clat percentiles (usec): | 1.00th=[ 873], 5.00th=[ 1336], 10.00th=[ 1975], | 20.00th=[ 2311], 30.00th=[ 2737], 40.00th=[ 2999], | 50.00th=[ 3326], 60.00th=[ 3523], 70.00th=[ 4080], | 80.00th=[ 4817], 90.00th=[ 5997], 95.00th=[ 6915], | 99.00th=[ 113771], 99.50th=[ 362808], 99.90th=[1199571], | 99.95th=[1535116], 99.99th=[2600469] bw ( KiB/s): min= 8, max=37675, per=16.03%, avg=8055.05, stdev=6107.68, samples=953 iops : min= 2, max= 9418, avg=2013.42, stdev=1526.91, samples=953 lat (usec) : 50=0.01%, 250=0.06%, 500=0.06%, 750=0.14%, 1000=2.53% lat (msec) : 2=7.54%, 4=57.91%, 10=29.88%, 20=0.23%, 50=0.37% lat (msec) : 100=0.25%, 250=0.33%, 500=0.35%, 750=0.12%, 1000=0.09% lat (msec) : 2000=0.13%, >=2000=0.02% cpu : usr=0.36%, sys=40.54%, ctx=986375, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,957247,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1226: Wed Dec 13 16:25:59 2017 write: IOPS=1413, BW=5656KiB/s (5792kB/s)(3314MiB/600001msec) slat (usec): min=11, max=3209.1k, avg=688.54, stdev=20245.71 clat (usec): min=43, max=3396.6k, avg=10618.11, stdev=82474.16 lat (usec): min=234, max=3396.8k, avg=11308.90, stdev=85346.97 clat percentiles (usec): | 1.00th=[ 848], 5.00th=[ 1221], 10.00th=[ 1565], | 20.00th=[ 2311], 30.00th=[ 2737], 40.00th=[ 3032], | 50.00th=[ 3326], 60.00th=[ 3523], 70.00th=[ 4113], | 80.00th=[ 4817], 90.00th=[ 6128], 95.00th=[ 7111], | 99.00th=[ 185598], 99.50th=[ 488637], 99.90th=[1317012], | 99.95th=[1702888], 99.99th=[2768241] bw ( KiB/s): min= 8, max=40056, per=14.11%, avg=7089.66, stdev=6368.59, samples=960 iops : min= 2, max=10014, avg=1772.08, stdev=1592.12, samples=960 lat (usec) : 50=0.01%, 250=0.01%, 500=0.19%, 750=0.18%, 1000=3.21% lat (msec) : 2=9.05%, 4=55.64%, 10=29.63%, 20=0.27%, 50=0.35% lat (msec) : 100=0.24%, 250=0.37%, 500=0.38%, 750=0.22%, 1000=0.10% lat (msec) : 2000=0.15%, >=2000=0.03% cpu : usr=0.31%, sys=36.04%, ctx=914036, majf=0, minf=9 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,848399,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=49.1MiB/s (51.5MB/s), 5656KiB/s-7011KiB/s (5792kB/s-7179kB/s), io=28.8GiB (30.9GB), run=600001-600003msec