file1: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16 ... fio-3.0 Starting 8 processes Jobs: 8 (f=8): [W(8)][100.0%][r=0KiB/s,w=104MiB/s][r=0,w=26.7k IOPS][eta 00m:00s] file1: (groupid=0, jobs=1): err= 0: pid=1191: Wed Dec 13 14:48:18 2017 write: IOPS=3478, BW=13.6MiB/s (14.2MB/s)(8152MiB/600001msec) slat (usec): min=4, max=1085.0k, avg=240.14, stdev=1750.72 clat (usec): min=58, max=1088.8k, avg=4143.72, stdev=7043.30 lat (usec): min=110, max=1088.9k, avg=4387.89, stdev=7296.91 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 12], 99.50th=[ 35], 99.90th=[ 102], 99.95th=[ 136], | 99.99th=[ 218] bw ( KiB/s): min= 993, max=19783, per=12.98%, avg=14575.57, stdev=3800.59, samples=1147 iops : min= 248, max= 4945, avg=3643.57, stdev=950.17, samples=1147 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.16%, 4=70.13%, 10=28.59%, 20=0.36%, 50=0.39% lat (msec) : 100=0.25%, 250=0.10%, 500=0.01%, 2000=0.01% cpu : usr=0.95%, sys=89.11%, ctx=452539, majf=0, minf=13 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2086817,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1192: Wed Dec 13 14:48:18 2017 write: IOPS=3626, BW=14.2MiB/s (14.9MB/s)(8500MiB/600001msec) slat (usec): min=6, max=1084.3k, avg=234.36, stdev=1416.96 clat (usec): min=44, max=1114.2k, avg=3971.34, stdev=5623.69 lat (usec): min=145, max=1114.5k, avg=4208.76, stdev=5823.63 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 7], 99.50th=[ 20], 99.90th=[ 78], 99.95th=[ 106], | 99.99th=[ 174] bw ( KiB/s): min= 1595, max=21112, per=13.53%, avg=15201.53, stdev=2969.29, samples=1147 iops : min= 398, max= 5278, avg=3800.04, stdev=742.34, samples=1147 lat (usec) : 50=0.01%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.19%, 4=69.92%, 10=29.06%, 20=0.28%, 50=0.27% lat (msec) : 100=0.15%, 250=0.05%, 500=0.01%, 2000=0.01% cpu : usr=0.97%, sys=92.24%, ctx=261214, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2175888,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1193: Wed Dec 13 14:48:18 2017 write: IOPS=3452, BW=13.5MiB/s (14.1MB/s)(8092MiB/600001msec) slat (usec): min=4, max=1154.9k, avg=242.30, stdev=1896.64 clat (usec): min=64, max=1164.4k, avg=4174.23, stdev=7654.32 lat (usec): min=109, max=1164.5k, avg=4420.55, stdev=7936.35 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 13], 99.50th=[ 36], 99.90th=[ 106], 99.95th=[ 144], | 99.99th=[ 275] bw ( KiB/s): min= 529, max=21258, per=12.90%, avg=14493.62, stdev=3880.36, samples=1147 iops : min= 132, max= 5314, avg=3623.03, stdev=970.10, samples=1147 lat (usec) : 100=0.01%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.15%, 4=70.16%, 10=28.51%, 20=0.37%, 50=0.40% lat (msec) : 100=0.24%, 250=0.10%, 500=0.01%, 750=0.01%, 2000=0.01% cpu : usr=0.94%, sys=88.67%, ctx=522998, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2071479,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1194: Wed Dec 13 14:48:18 2017 write: IOPS=3511, BW=13.7MiB/s (14.4MB/s)(8230MiB/600001msec) slat (usec): min=4, max=1085.1k, avg=237.86, stdev=1687.05 clat (usec): min=39, max=1089.1k, avg=4104.43, stdev=6668.41 lat (usec): min=120, max=1089.4k, avg=4346.31, stdev=6898.94 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 12], 99.50th=[ 30], 99.90th=[ 96], 99.95th=[ 132], | 99.99th=[ 211] bw ( KiB/s): min= 1112, max=19776, per=13.09%, avg=14707.65, stdev=3393.04, samples=1147 iops : min= 278, max= 4944, avg=3676.69, stdev=848.28, samples=1147 lat (usec) : 50=0.01%, 250=0.02%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.11%, 4=69.99%, 10=28.82%, 20=0.37%, 50=0.39% lat (msec) : 100=0.19%, 250=0.09%, 500=0.01%, 2000=0.01% cpu : usr=1.00%, sys=89.85%, ctx=413785, majf=0, minf=12 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2106782,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1195: Wed Dec 13 14:48:18 2017 write: IOPS=3463, BW=13.5MiB/s (14.2MB/s)(8117MiB/600001msec) slat (usec): min=4, max=1082.7k, avg=241.60, stdev=1838.26 clat (usec): min=63, max=1088.4k, avg=4161.37, stdev=7332.29 lat (usec): min=110, max=1088.7k, avg=4406.97, stdev=7592.92 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 13], 99.50th=[ 35], 99.90th=[ 107], 99.95th=[ 144], | 99.99th=[ 243] bw ( KiB/s): min= 496, max=21619, per=12.93%, avg=14528.62, stdev=3710.11, samples=1147 iops : min= 124, max= 5404, avg=3631.78, stdev=927.53, samples=1147 lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.15%, 4=70.22%, 10=28.49%, 20=0.38%, 50=0.40% lat (msec) : 100=0.24%, 250=0.10%, 500=0.01%, 750=0.01%, 2000=0.01% cpu : usr=0.99%, sys=88.74%, ctx=465671, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2077865,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1196: Wed Dec 13 14:48:18 2017 write: IOPS=3550, BW=13.9MiB/s (14.5MB/s)(8321MiB/600001msec) slat (usec): min=4, max=1084.5k, avg=234.71, stdev=1655.30 clat (usec): min=48, max=1089.2k, avg=4059.46, stdev=6479.65 lat (usec): min=110, max=1089.4k, avg=4298.19, stdev=6697.09 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 5], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 9], 99.50th=[ 25], 99.90th=[ 88], 99.95th=[ 124], | 99.99th=[ 222] bw ( KiB/s): min= 1386, max=19895, per=13.27%, avg=14907.27, stdev=2940.34, samples=1147 iops : min= 346, max= 4973, avg=3726.44, stdev=735.09, samples=1147 lat (usec) : 50=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.02% lat (msec) : 2=0.12%, 4=69.78%, 10=29.15%, 20=0.34%, 50=0.33% lat (msec) : 100=0.18%, 250=0.07%, 500=0.01%, 750=0.01%, 2000=0.01% cpu : usr=0.98%, sys=90.70%, ctx=321526, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2130283,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1197: Wed Dec 13 14:48:18 2017 write: IOPS=3520, BW=13.8MiB/s (14.4MB/s)(8252MiB/600001msec) slat (usec): min=4, max=1083.3k, avg=237.04, stdev=1668.77 clat (usec): min=45, max=1092.4k, avg=4093.45, stdev=6715.18 lat (usec): min=110, max=1092.6k, avg=4334.50, stdev=6950.68 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 11], 99.50th=[ 31], 99.90th=[ 95], 99.95th=[ 122], | 99.99th=[ 215] bw ( KiB/s): min= 128, max=22949, per=13.16%, avg=14785.06, stdev=3438.70, samples=1147 iops : min= 32, max= 5737, avg=3695.90, stdev=859.68, samples=1147 lat (usec) : 50=0.01%, 250=0.03%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.15%, 4=70.79%, 10=27.97%, 20=0.35%, 50=0.38% lat (msec) : 100=0.21%, 250=0.08%, 500=0.01%, 750=0.01%, 2000=0.01% cpu : usr=1.00%, sys=89.70%, ctx=410771, majf=0, minf=10 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2112511,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 file1: (groupid=0, jobs=1): err= 0: pid=1198: Wed Dec 13 14:48:18 2017 write: IOPS=3477, BW=13.6MiB/s (14.2MB/s)(8151MiB/600001msec) slat (usec): min=4, max=1082.9k, avg=240.13, stdev=1795.84 clat (usec): min=45, max=1115.8k, avg=4144.14, stdev=7175.43 lat (usec): min=115, max=1116.1k, avg=4388.33, stdev=7427.75 clat percentiles (msec): | 1.00th=[ 3], 5.00th=[ 3], 10.00th=[ 3], 20.00th=[ 4], | 30.00th=[ 4], 40.00th=[ 4], 50.00th=[ 4], 60.00th=[ 4], | 70.00th=[ 4], 80.00th=[ 5], 90.00th=[ 5], 95.00th=[ 5], | 99.00th=[ 12], 99.50th=[ 34], 99.90th=[ 103], 99.95th=[ 146], | 99.99th=[ 222] bw ( KiB/s): min= 963, max=19951, per=12.99%, avg=14591.74, stdev=3637.39, samples=1147 iops : min= 240, max= 4987, avg=3647.55, stdev=909.34, samples=1147 lat (usec) : 50=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.15%, 4=70.37%, 10=28.36%, 20=0.38%, 50=0.37% lat (msec) : 100=0.24%, 250=0.10%, 500=0.01%, 2000=0.01% cpu : usr=0.97%, sys=88.97%, ctx=452510, majf=0, minf=11 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwt: total=0,2086619,0, short=0,0,0, dropped=0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): WRITE: bw=110MiB/s (115MB/s), 13.5MiB/s-14.2MiB/s (14.1MB/s-14.9MB/s), io=64.3GiB (69.0GB), run=600001-600001msec Disk stats (read/write): vdb: ios=0/46063, merge=0/6986, ticks=0/2294756, in_queue=1430067, util=8.76%