Re: [PATCH] md: improve io stats accounting

[Artur Paszkiewicz <artur.paszkiewicz@intel.com>]

On 6/2/20 8:48 AM, Song Liu wrote:
>> +               clone = bio_clone_fast(bio, GFP_NOIO, &mddev->md_io_bs);
> 
> Handle clone == NULL?

I think this should never fail - bio_alloc_bioset() guarantees that. It
is used in a similar manner in raid1 and raid10. How about
BUG_ON(clone == NULL)?

> Also, have you done benchmarks with this change?

I tested 4k random reads on a raid0 (4x P4510 2TB) and it was 2550k vs
2567k IOPS, that's slower only by about 0.66%:

without patch:

# fio --direct=1 --thread --rw=randread --ioengine=libaio --iodepth=64 --bs=4k --name=fio --filename=/dev/md0 --time_based --runtime=300 --numjobs=16 --group_reporting --norandommap --randrepeat=0
fio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.20
Starting 16 threads
Jobs: 16 (f=16): [r(16)][100.0%][r=9.81GiB/s][r=2571k IOPS][eta 00m:00s]
fio: (groupid=0, jobs=16): err= 0: pid=8678: Tue Jun  2 13:19:38 2020
  read: IOPS=2567k, BW=9.79GiB/s (10.5GB/s)(2938GiB/300002msec)
    slat (nsec): min=1384, max=798130, avg=2852.52, stdev=1108.15
    clat (usec): min=48, max=12387, avg=395.81, stdev=260.52
     lat (usec): min=51, max=12389, avg=398.70, stdev=260.51
    clat percentiles (usec):
     |  1.00th=[  101],  5.00th=[  135], 10.00th=[  157], 20.00th=[  196],
     | 30.00th=[  233], 40.00th=[  273], 50.00th=[  322], 60.00th=[  379],
     | 70.00th=[  457], 80.00th=[  562], 90.00th=[  734], 95.00th=[  889],
     | 99.00th=[ 1287], 99.50th=[ 1500], 99.90th=[ 2147], 99.95th=[ 2442],
     | 99.99th=[ 3130]
   bw (  MiB/s): min= 9664, max=10215, per=100.00%, avg=10033.54, stdev= 4.59, samples=9584
   iops        : min=2474063, max=2615178, avg=2568585.96, stdev=1176.06, samples=9584
  lat (usec)   : 50=0.01%, 100=0.94%, 250=33.51%, 500=39.98%, 750=16.36%
  lat (usec)   : 1000=6.16%
  lat (msec)   : 2=2.92%, 4=0.14%, 10=0.01%, 20=0.01%
  cpu          : usr=16.07%, sys=41.29%, ctx=193003298, majf=0, minf=90976
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=770061876,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=9.79GiB/s (10.5GB/s), 9.79GiB/s-9.79GiB/s (10.5GB/s-10.5GB/s), io=2938GiB (3154GB), run=300002-300002msec

Disk stats (read/write):
    md0: ios=769770571/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=192515469/0, aggrmerge=0/0, aggrticks=74947558/0, aggrin_queue=6006362, aggrutil=100.00%
  nvme3n1: ios=192511957/0, merge=0/0, ticks=76700358/0, in_queue=6215887, util=100.00%
  nvme6n1: ios=192503722/0, merge=0/0, ticks=72629807/0, in_queue=5520156, util=100.00%
  nvme2n1: ios=192518930/0, merge=0/0, ticks=74719743/0, in_queue=5979779, util=100.00%
  nvme1n1: ios=192527267/0, merge=0/0, ticks=75740325/0, in_queue=6309628, util=100.00%

with patch:

# fio --direct=1 --thread --rw=randread --ioengine=libaio --iodepth=64 --bs=4k --name=fio --filename=/dev/md0 --time_based --runtime=300 --numjobs=16 --group_reporting --norandommap --randrepeat=0
fio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.20
Starting 16 threads
Jobs: 16 (f=16): [r(16)][100.0%][r=9934MiB/s][r=2543k IOPS][eta 00m:00s]
fio: (groupid=0, jobs=16): err= 0: pid=8463: Tue Jun  2 13:32:12 2020
  read: IOPS=2550k, BW=9961MiB/s (10.4GB/s)(2918GiB/300002msec)
    slat (nsec): min=1512, max=3578.1k, avg=5145.36, stdev=2145.71
    clat (usec): min=50, max=12421, avg=396.13, stdev=210.38
     lat (usec): min=52, max=12428, avg=401.33, stdev=210.45
    clat percentiles (usec):
     |  1.00th=[  133],  5.00th=[  178], 10.00th=[  208], 20.00th=[  247],
     | 30.00th=[  281], 40.00th=[  314], 50.00th=[  347], 60.00th=[  383],
     | 70.00th=[  437], 80.00th=[  510], 90.00th=[  644], 95.00th=[  783],
     | 99.00th=[ 1156], 99.50th=[ 1369], 99.90th=[ 1991], 99.95th=[ 2311],
     | 99.99th=[ 2999]
   bw (  MiB/s): min= 9266, max=10648, per=100.00%, avg=9967.23, stdev=13.31, samples=9584
   iops        : min=2372118, max=2725915, avg=2551610.09, stdev=3407.18, samples=9584
  lat (usec)   : 100=0.13%, 250=20.62%, 500=58.25%, 750=15.25%, 1000=3.92%
  lat (msec)   : 2=1.72%, 4=0.10%, 10=0.01%, 20=0.01%
  cpu          : usr=15.97%, sys=66.59%, ctx=11235674, majf=0, minf=41238
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=764997277,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=9961MiB/s (10.4GB/s), 9961MiB/s-9961MiB/s (10.4GB/s-10.4GB/s), io=2918GiB (3133GB), run=300002-300002msec

Disk stats (read/write):
    md0: ios=764702549/0, merge=0/0, ticks=242091778/0, in_queue=242091754, util=100.00%, aggrios=191249319/0, aggrmerge=0/0, aggrticks=59760064/0, aggrin_queue=2798855, aggrutil=100.00%
  nvme3n1: ios=191250967/0, merge=0/0, ticks=61633420/0, in_queue=3032943, util=100.00%
  nvme6n1: ios=191257919/0, merge=0/0, ticks=59065688/0, in_queue=2784603, util=100.00%
  nvme2n1: ios=191255129/0, merge=0/0, ticks=58520284/0, in_queue=2461116, util=100.00%
  nvme1n1: ios=191233262/0, merge=0/0, ticks=59820864/0, in_queue=2916760, util=100.00%