Testing with SSD 1) Without dm-integrity 4ktest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=16 ... fio-3.14 Starting 8 processes 4ktest: Laying out IO file (1 file / 4096MiB) Jobs: 8 (f=8): [m(8)][100.0%][r=3347KiB/s,w=1205KiB/s][r=836,w=301 IOPS][eta 00m:00s] 4ktest: (groupid=0, jobs=8): err= 0: pid=429966: Sat Jan 4 02:34:22 2020 read: IOPS=806, BW=3225KiB/s (3302kB/s)(189MiB/60007msec) clat (usec): min=91, max=1333.0k, avg=1449.50, stdev=9061.33 lat (usec): min=91, max=1333.0k, avg=1449.86, stdev=9061.33 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 116], 10.00th=[ 125], 20.00th=[ 145], | 30.00th=[ 174], 40.00th=[ 202], 50.00th=[ 253], 60.00th=[ 363], | 70.00th=[ 1418], 80.00th=[ 3032], 90.00th=[ 4359], 95.00th=[ 6128], | 99.00th=[ 9241], 99.50th=[10290], 99.90th=[13042], 99.95th=[14484], | 99.99th=[18744] bw ( KiB/s): min= 1846, max= 4150, per=100.00%, avg=3279.09, stdev=44.56, samples=944 iops : min= 460, max= 1037, avg=819.37, stdev=11.15, samples=944 write: IOPS=270, BW=1082KiB/s (1108kB/s)(63.4MiB/60007msec); 0 zone resets clat (usec): min=39, max=1334.8k, avg=2561.63, stdev=15018.30 lat (usec): min=40, max=1334.8k, avg=2562.07, stdev=15018.30 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 83], 10.00th=[ 97], | 20.00th=[ 153], 30.00th=[ 249], 40.00th=[ 461], | 50.00th=[ 1500], 60.00th=[ 2835], 70.00th=[ 3130], | 80.00th=[ 4146], 90.00th=[ 6128], 95.00th=[ 7570], | 99.00th=[ 11076], 99.50th=[ 11994], 99.90th=[ 16581], | 99.95th=[ 17957], 99.99th=[1333789] bw ( KiB/s): min= 455, max= 1728, per=100.00%, avg=1100.00, stdev=30.18, samples=944 iops : min= 113, max= 432, avg=274.59, stdev= 7.55, samples=944 lat (usec) : 50=0.11%, 100=2.76%, 250=41.86%, 500=13.99%, 750=3.52% lat (usec) : 1000=0.20% lat (msec) : 2=7.62%, 4=16.54%, 10=12.44%, 20=0.95%, 50=0.01% lat (msec) : 500=0.01%, 2000=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=194, max=1339.7k, avg=5697.28, stdev=10037.09 sync percentiles (usec): | 1.00th=[ 461], 5.00th=[ 1483], 10.00th=[ 2835], 20.00th=[ 3195], | 30.00th=[ 4228], 40.00th=[ 5145], 50.00th=[ 5735], 60.00th=[ 5997], | 70.00th=[ 6783], 80.00th=[ 7439], 90.00th=[ 8717], 95.00th=[ 9765], | 99.00th=[12387], 99.50th=[13829], 99.90th=[16581], 99.95th=[17695], | 99.99th=[20579] cpu : usr=0.22%, sys=1.24%, ctx=308172, majf=0, minf=108 IO depths : 1=199.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=48377,16234,0,64566 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3225KiB/s (3302kB/s), 3225KiB/s-3225KiB/s (3302kB/s-3302kB/s), io=189MiB (198MB), run=60007-60007msec WRITE: bw=1082KiB/s (1108kB/s), 1082KiB/s-1082KiB/s (1108kB/s-1108kB/s), io=63.4MiB (66.5MB), run=60007-60007msec Disk stats (read/write): dm-15: ios=48215/96083, merge=0/0, ticks=13910/204053, in_queue=217963, util=52.52%, aggrios=48384/100562, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% md63: ios=48384/100562, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=24194/46905, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% md62: ios=24360/46880, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12186/47157, aggrmerge=0/86, aggrticks=3328/25276, aggrin_queue=16938, aggrutil=52.70% sdd: ios=7665/47426, merge=0/162, ticks=4452/40035, in_queue=31377, util=50.48% sdg: ios=16707/46888, merge=0/10, ticks=2205/10517, in_queue=2500, util=52.70% md61: ios=24029/46931, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12024/47174, aggrmerge=0/122, aggrticks=3502/25432, aggrin_queue=17289, aggrutil=52.36% sdf: ios=16251/46906, merge=0/46, ticks=2450/10909, in_queue=3265, util=52.36% sdb: ios=7798/47443, merge=0/199, ticks=4555/39956, in_queue=31313, util=50.55% 2) With dm-integrity 4ktest: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=16 ... fio-3.14 Starting 8 processes 4ktest: Laying out IO file (1 file / 4096MiB) Jobs: 8 (f=8): [m(8)][100.0%][r=3347KiB/s,w=1205KiB/s][r=836,w=301 IOPS][eta 00m:00s] 4ktest: (groupid=0, jobs=8): err= 0: pid=429966: Sat Jan 4 02:34:22 2020 read: IOPS=806, BW=3225KiB/s (3302kB/s)(189MiB/60007msec) clat (usec): min=91, max=1333.0k, avg=1449.50, stdev=9061.33 lat (usec): min=91, max=1333.0k, avg=1449.86, stdev=9061.33 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 116], 10.00th=[ 125], 20.00th=[ 145], | 30.00th=[ 174], 40.00th=[ 202], 50.00th=[ 253], 60.00th=[ 363], | 70.00th=[ 1418], 80.00th=[ 3032], 90.00th=[ 4359], 95.00th=[ 6128], | 99.00th=[ 9241], 99.50th=[10290], 99.90th=[13042], 99.95th=[14484], | 99.99th=[18744] bw ( KiB/s): min= 1846, max= 4150, per=100.00%, avg=3279.09, stdev=44.56, samples=944 iops : min= 460, max= 1037, avg=819.37, stdev=11.15, samples=944 write: IOPS=270, BW=1082KiB/s (1108kB/s)(63.4MiB/60007msec); 0 zone resets clat (usec): min=39, max=1334.8k, avg=2561.63, stdev=15018.30 lat (usec): min=40, max=1334.8k, avg=2562.07, stdev=15018.30 clat percentiles (usec): | 1.00th=[ 58], 5.00th=[ 83], 10.00th=[ 97], | 20.00th=[ 153], 30.00th=[ 249], 40.00th=[ 461], | 50.00th=[ 1500], 60.00th=[ 2835], 70.00th=[ 3130], | 80.00th=[ 4146], 90.00th=[ 6128], 95.00th=[ 7570], | 99.00th=[ 11076], 99.50th=[ 11994], 99.90th=[ 16581], | 99.95th=[ 17957], 99.99th=[1333789] bw ( KiB/s): min= 455, max= 1728, per=100.00%, avg=1100.00, stdev=30.18, samples=944 iops : min= 113, max= 432, avg=274.59, stdev= 7.55, samples=944 lat (usec) : 50=0.11%, 100=2.76%, 250=41.86%, 500=13.99%, 750=3.52% lat (usec) : 1000=0.20% lat (msec) : 2=7.62%, 4=16.54%, 10=12.44%, 20=0.95%, 50=0.01% lat (msec) : 500=0.01%, 2000=0.01% fsync/fdatasync/sync_file_range: sync (usec): min=194, max=1339.7k, avg=5697.28, stdev=10037.09 sync percentiles (usec): | 1.00th=[ 461], 5.00th=[ 1483], 10.00th=[ 2835], 20.00th=[ 3195], | 30.00th=[ 4228], 40.00th=[ 5145], 50.00th=[ 5735], 60.00th=[ 5997], | 70.00th=[ 6783], 80.00th=[ 7439], 90.00th=[ 8717], 95.00th=[ 9765], | 99.00th=[12387], 99.50th=[13829], 99.90th=[16581], 99.95th=[17695], | 99.99th=[20579] cpu : usr=0.22%, sys=1.24%, ctx=308172, majf=0, minf=108 IO depths : 1=199.9%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=48377,16234,0,64566 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=16 Run status group 0 (all jobs): READ: bw=3225KiB/s (3302kB/s), 3225KiB/s-3225KiB/s (3302kB/s-3302kB/s), io=189MiB (198MB), run=60007-60007msec WRITE: bw=1082KiB/s (1108kB/s), 1082KiB/s-1082KiB/s (1108kB/s-1108kB/s), io=63.4MiB (66.5MB), run=60007-60007msec Disk stats (read/write): dm-15: ios=48215/96083, merge=0/0, ticks=13910/204053, in_queue=217963, util=52.52%, aggrios=48384/100562, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% md63: ios=48384/100562, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=24194/46905, aggrmerge=0/0, aggrticks=0/0, aggrin_queue=0, aggrutil=0.00% md62: ios=24360/46880, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12186/47157, aggrmerge=0/86, aggrticks=3328/25276, aggrin_queue=16938, aggrutil=52.70% sdd: ios=7665/47426, merge=0/162, ticks=4452/40035, in_queue=31377, util=50.48% sdg: ios=16707/46888, merge=0/10, ticks=2205/10517, in_queue=2500, util=52.70% md61: ios=24029/46931, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=12024/47174, aggrmerge=0/122, aggrticks=3502/25432, aggrin_queue=17289, aggrutil=52.36% sdf: ios=16251/46906, merge=0/46, ticks=2450/10909, in_queue=3265, util=52.36% sdb: ios=7798/47443, merge=0/199, ticks=4555/39956, in_queue=31313, util=50.55%