* IO Depth lower than requested
@ 2022-04-22 21:57 Elliott Balsley
2022-04-23 0:24 ` Damien Le Moal
0 siblings, 1 reply; 2+ messages in thread
From: Elliott Balsley @ 2022-04-22 21:57 UTC (permalink / raw)
To: fio
I'm testing some NVMe drives and I notice the IO Depth reported by fio
output is lower than what I specify in the job. Is the command-line
output showing the attempted depth or the actual achieved depth?
Here's an example output where I tried iodepth=32. The first line (IO
depths) shows a mix of 8 and 16. The next 2 lines (submit and
complete) show 4. What's the difference between the first and second
lines? I was expecting "submit" to be 32 and "complete" to maybe be
lower based on hardware and OS limitations. Oddly enough, if I
specify iodepth=4 then performance is actually a bit better (32K
instead of 35K) even though the submit/complete depths shown remain
the same.
Also — is there a way to find out what is limiting this? The Micron
9300 Pro should support QD=512 according to its spec sheet.
% fio --randrepeat=1 --name=test --bs=4k --iodepth=32 --size=1G
--readwrite=randread --directory /Volumes/Micron9300/elliott
--ioengine posixaio
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
4096B-4096B, ioengine=posixaio, iodepth=32
fio-3.30
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=137MiB/s][r=35.1k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=6903: Fri Apr 22 12:25:34 2022
read: IOPS=35.0k, BW=137MiB/s (143MB/s)(1024MiB/7498msec)
slat (nsec): min=0, max=46000, avg=823.84, stdev=784.94
clat (usec): min=179, max=874, avg=455.01, stdev=50.98
lat (usec): min=180, max=875, avg=455.84, stdev=50.88
clat percentiles (usec):
| 1.00th=[ 318], 5.00th=[ 367], 10.00th=[ 392], 20.00th=[ 416],
| 30.00th=[ 433], 40.00th=[ 445], 50.00th=[ 457], 60.00th=[ 469],
| 70.00th=[ 482], 80.00th=[ 494], 90.00th=[ 515], 95.00th=[ 537],
| 99.00th=[ 578], 99.50th=[ 594], 99.90th=[ 652], 99.95th=[ 668],
| 99.99th=[ 717]
bw ( KiB/s): min=139419, max=140595, per=100.00%, avg=140110.71,
stdev=340.75, samples=14
iops : min=34854, max=35148, avg=35027.43, stdev=85.14, samples=14
lat (usec) : 250=0.06%, 500=82.67%, 750=17.27%, 1000=0.01%
cpu : usr=19.97%, sys=13.74%, ctx=195959, majf=0, minf=28
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=43.4%, 16=56.6%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.1%, 8=0.9%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32
Run status group 0 (all jobs):
READ: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s),
io=1024MiB (1074MB), run=7498-7498msec
Thanks,
Elliott
^ permalink raw reply [flat|nested] 2+ messages in thread
* Re: IO Depth lower than requested
2022-04-22 21:57 IO Depth lower than requested Elliott Balsley
@ 2022-04-23 0:24 ` Damien Le Moal
0 siblings, 0 replies; 2+ messages in thread
From: Damien Le Moal @ 2022-04-23 0:24 UTC (permalink / raw)
To: Elliott Balsley, fio
On 4/23/22 06:57, Elliott Balsley wrote:
> I'm testing some NVMe drives and I notice the IO Depth reported by fio
> output is lower than what I specify in the job. Is the command-line
> output showing the attempted depth or the actual achieved depth?
> Here's an example output where I tried iodepth=32. The first line (IO
> depths) shows a mix of 8 and 16. The next 2 lines (submit and
> complete) show 4. What's the difference between the first and second
> lines? I was expecting "submit" to be 32 and "complete" to maybe be
> lower based on hardware and OS limitations. Oddly enough, if I
> specify iodepth=4 then performance is actually a bit better (32K
> instead of 35K) even though the submit/complete depths shown remain
> the same.
> Also — is there a way to find out what is limiting this? The Micron
> 9300 Pro should support QD=512 according to its spec sheet.
>
> % fio --randrepeat=1 --name=test --bs=4k --iodepth=32 --size=1G
> --readwrite=randread --directory /Volumes/Micron9300/elliott
> --ioengine posixaio
If it is on Linux, try libaio.
The file system may be limiting (blocking) IO submission due to some
internal IO processing/inode locking.
Try testing a similar workload on the raw drive first
(--filename=/dev/nvmeXnY) and see what the queue depth is.
> test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T)
> 4096B-4096B, ioengine=posixaio, iodepth=32
> fio-3.30
> Starting 1 process
> Jobs: 1 (f=1): [r(1)][100.0%][r=137MiB/s][r=35.1k IOPS][eta 00m:00s]
> test: (groupid=0, jobs=1): err= 0: pid=6903: Fri Apr 22 12:25:34 2022
> read: IOPS=35.0k, BW=137MiB/s (143MB/s)(1024MiB/7498msec)
> slat (nsec): min=0, max=46000, avg=823.84, stdev=784.94
> clat (usec): min=179, max=874, avg=455.01, stdev=50.98
> lat (usec): min=180, max=875, avg=455.84, stdev=50.88
> clat percentiles (usec):
> | 1.00th=[ 318], 5.00th=[ 367], 10.00th=[ 392], 20.00th=[ 416],
> | 30.00th=[ 433], 40.00th=[ 445], 50.00th=[ 457], 60.00th=[ 469],
> | 70.00th=[ 482], 80.00th=[ 494], 90.00th=[ 515], 95.00th=[ 537],
> | 99.00th=[ 578], 99.50th=[ 594], 99.90th=[ 652], 99.95th=[ 668],
> | 99.99th=[ 717]
> bw ( KiB/s): min=139419, max=140595, per=100.00%, avg=140110.71,
> stdev=340.75, samples=14
> iops : min=34854, max=35148, avg=35027.43, stdev=85.14, samples=14
> lat (usec) : 250=0.06%, 500=82.67%, 750=17.27%, 1000=0.01%
> cpu : usr=19.97%, sys=13.74%, ctx=195959, majf=0, minf=28
> IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=43.4%, 16=56.6%, 32=0.0%, >=64=0.0%
> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
> complete : 0=0.0%, 4=99.1%, 8=0.9%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
> issued rwts: total=262144,0,0,0 short=0,0,0,0 dropped=0,0,0,0
> latency : target=0, window=0, percentile=100.00%, depth=32
>
> Run status group 0 (all jobs):
> READ: bw=137MiB/s (143MB/s), 137MiB/s-137MiB/s (143MB/s-143MB/s),
> io=1024MiB (1074MB), run=7498-7498msec
>
> Thanks,
> Elliott
--
Damien Le Moal
Western Digital Research
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2022-04-23 0:25 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-04-22 21:57 IO Depth lower than requested Elliott Balsley
2022-04-23 0:24 ` Damien Le Moal
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.