hitting 2048 threads/jobs limit

hitting 2048 threads/jobs limit

[Tobias Oberstein]

Hi,

is there a way to increase the 2048 threads/jobs limit in FIO?

I am hitting a wall trying to saturate a storage on random 8kB reads .. 
can't get over 5 mio IOPS.

Please see the top most / right most IOPS value 4.597.700 in here:

https://github.com/oberstet/scratchbox/raw/master/cruncher/sql19/Performance%20Results%20-%20NVMe%20Scaling%20with%20IO%20Concurrency.pdf

The box has still more CPU cycles avail .. and the IOPS over IO 
concurrency seems to indicate there is more room ..

Cheers,
/Tobias

PS: thanks for an _awesome_ tool!

Re: hitting 2048 threads/jobs limit

[Tobias Oberstein]

Hi Andrey,

thanks, got it .. FIO_MAX_JOB was bumped to 4096 like 4 months ago in 
this commit

https://github.com/axboe/fio/commit/cab2472e2e9e84877e65c6aa68b86899956c8a28

So I just needed to rebuild from source.

Now I can max out this box using 2800 threads at 4.7 mio 8kB random read 
IOPS =)

Awesome.

Cheers,
/Tobias


oberstet@svr-psql19:~/scm/parcit/RA/adr/system/docs$ sudo 
/opt/fio/bin/fio postgresql_storage_workload.fio
randread: (g=0): rw=randread, bs=8192B-8192B,8192B-8192B,8192B-8192B, 
ioengine=sync, iodepth=1
...
fio-2.17-17-g9cf1
Starting 2800 threads
Jobs: 2736 (f=37103): 
[f(7),_(1),f(3),_(1),f(3),_(2),f(1),_(1),f(1),_(14),f(1),_(2),f(3),_(4),f(1),_(1),f(1),_(3),f(2),_(11),f(1),_(1),f(5),_(1),f(17),_(1),f(3),_(1),f(56),_(1),f(10),_(1),f(4),_(1),f(5),_(1),f(6),_(1),f(489),_(1),f(9),_(1),f(191),_(1),f(139),_(1),f(22),_(1),f(229),_(1),f(19),_(1),f(28),_(1),f(138),_(1),f(480),_(1),f(6),_(1),f(9),_(1),f(2),_(1),f(284),_(1),f(338),_(1),f(223)][100.0%][r=32.8GiB/s,w=0KiB/s][r=4297k,w=0 
IOPS][eta 00m:00s]
randread: (groupid=0, jobs=2800): err= 0: pid=99065: Mon Jan 23 15:16:24 
2017
    read: IOPS=4732k, BW=36.2GiB/s (38.8GB/s)(4332GiB/120007msec)
     clat (usec): min=43, max=26678, avg=583.62, stdev=572.89
      lat (usec): min=43, max=26678, avg=583.70, stdev=572.89
     clat percentiles (usec):
      |  1.00th=[  153],  5.00th=[  185], 10.00th=[  209], 20.00th=[  253],
      | 30.00th=[  302], 40.00th=[  354], 50.00th=[  418], 60.00th=[  502],
      | 70.00th=[  612], 80.00th=[  772], 90.00th=[ 1064], 95.00th=[ 1448],
      | 99.00th=[ 3152], 99.50th=[ 3984], 99.90th=[ 5856], 99.95th=[ 6688],
      | 99.99th=[ 8768]
     lat (usec) : 50=0.01%, 100=0.01%, 250=19.14%, 500=40.68%, 750=19.20%
     lat (usec) : 1000=9.54%
     lat (msec) : 2=8.76%, 4=2.19%, 10=0.48%, 20=0.01%, 50=0.01%
   cpu          : usr=0.45%, sys=3.68%, ctx=568493883, majf=0, minf=5598
   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
 >=64=0.0%
      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 >=64=0.0%
      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
 >=64=0.0%
      issued rwt: total=567819066,0,0, short=0,0,0, dropped=0,0,0
      latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
    READ: bw=36.2GiB/s (38.8GB/s), 36.2GiB/s-36.2GiB/s 
(38.8GB/s-38.8GB/s), io=4332GiB (4652GB), run=120007-120007msec

Disk stats (read/write):
   nvme0n1: ios=35440924/0, merge=0/0, ticks=18474800/0, 
in_queue=18776444, util=99.79%
   nvme1n1: ios=35489875/0, merge=0/0, ticks=16143360/0, 
in_queue=16950032, util=100.00%
   nvme2n1: ios=35489702/0, merge=0/0, ticks=27449436/0, 
in_queue=28772208, util=100.00%
   nvme3n1: ios=35489494/0, merge=0/0, ticks=19419180/0, 
in_queue=19731140, util=99.71%
   nvme4n1: ios=35489290/0, merge=0/0, ticks=18055916/0, 
in_queue=19064904, util=100.00%
   nvme5n1: ios=35489126/0, merge=0/0, ticks=23692336/0, 
in_queue=24074852, util=99.79%
   nvme6n1: ios=35488938/0, merge=0/0, ticks=19653808/0, 
in_queue=19951296, util=99.67%
   nvme7n1: ios=35488789/0, merge=0/0, ticks=17548548/0, 
in_queue=17833728, util=99.70%
   nvme8n1: ios=35488597/0, merge=0/0, ticks=25328664/0, 
in_queue=25684916, util=99.69%
   nvme9n1: ios=35488409/0, merge=0/0, ticks=15241580/0, 
in_queue=15956076, util=100.00%
   nvme10n1: ios=35488240/0, merge=0/0, ticks=16489904/0, 
in_queue=16750844, util=99.82%
   nvme11n1: ios=35488059/0, merge=0/0, ticks=25966172/0, 
in_queue=27363044, util=100.00%
   nvme12n1: ios=35487878/0, merge=0/0, ticks=16283216/0, 
in_queue=16521860, util=99.78%
   nvme13n1: ios=35487713/0, merge=0/0, ticks=17994988/0, 
in_queue=18756864, util=100.00%
   nvme14n1: ios=35487556/0, merge=0/0, ticks=16564664/0, 
in_queue=16794600, util=99.90%
   nvme15n1: ios=35487377/0, merge=0/0, ticks=14727728/0, 
in_queue=14983204, util=99.85%
oberstet@svr-psql19:~/scm/parcit/RA/adr/system/docs$