RE: CPUs, threads, and speed

["Elliott, Robert (Servers)" <elliott@hpe.com>]

> -----Original Message-----
> From: fio-owner@vger.kernel.org <fio-owner@vger.kernel.org> On Behalf Of
> Mauricio Tavares
> Sent: Wednesday, January 15, 2020 9:51 AM
> Subject: CPUs, threads, and speed
> 
...
> [global]
> name=4k random write 4 ios in the queue in 32 queues
> filename=/dev/nvme0n1
> ioengine=libaio
> direct=1
> bs=4k
> rw=randwrite
> iodepth=4
> numjobs=32
> buffered=0
> size=100%
> loops=2
> randrepeat=0
> norandommap
> refill_buffers
> 
> [job1]
> 
> That is taking a ton of time, like days to go. Is there anything I can
> do to speed it up? For instance, what is the default value for
> cpus_allowed (or cpumask)[2]? Is it all CPUs? If not what would I gain
> by throwing more cpus at the problem?
> 
> I also read[2] by default fio uses fork. What would I get by going to
> threads?

> Jobs: 32 (f=32): [w(32)][10.8%][w=301MiB/s][w=77.0k IOPS][eta 06d:13h:56m:51s]]

77 kIOPs for random writes isn't bad - check your drive data sheet.
If the drive is 1 TB, it should take 
    1 TB / (77k * 4 KiB) = 3170 s = 52.8 minutes 
to write the whole drive.

Best practice is to use all CPU cores, lock threads to cores, and
be NUMA aware. If the device is attached to physical CPU 0 and that CPU
has 12 cores known to linux as 0-11 (per "lscpu" or "numactl --hardware"),
try:
  iodepth=16
  numjobs=12
  cpus_allowed=0-11
  cpus_allowed_policy=split

Based on these:
  numjobs=32, size=100%, loops=2
fio will run each job for that many bytes, so a 1 TB drive will result 
in IOs for 64 TB rather than 1 TB. That could easily result in the
multi-day estimate.

Other nits:
* thread - threading might be slightly more efficient than 
  spawning full processes
* gtod_reduce=1 - precision latency measurements don't matter for this
* refill_buffers - presuming you don't care about the data contents,
  don't include this. zero_buffers is the simplest/fastest, unless you're
  concerned that the device might do compression or zero detection
* norandommap - if you want it to hit each LBA a precise number
  of times, you can't include this; fio won't remember what it's 
  done. There is a lot of overhead in keeping track, though.