* [PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns
@ 2022-02-02 9:04 Chaitanya Kulkarni
2022-02-02 9:04 ` [PATCH V3 1/2] nvmet: use i_size_read() to set size for file-ns Chaitanya Kulkarni
` (2 more replies)
0 siblings, 3 replies; 4+ messages in thread
From: Chaitanya Kulkarni @ 2022-02-02 9:04 UTC (permalink / raw)
To: linux-nvme; +Cc: hch, kbusch, sagi, Chaitanya Kulkarni
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset="y", Size: 98751 bytes --]
From: Chaitanya Kulkarni <kch@nvidia.com>
Hi,
In current implementation like NVMeOF target file backed namespace we
don't support buffered_io for block device backed namesapce. This allows
user to configure block device backend in buffered_io mode so user can
get the benefits of caching.
Please note that using this combination host needs to periodically sync
between cache and storage or set the system cache parameters
accordingly. NVMeOF target is not responsible for data consistency
between cache and storage.
fio data verification, rand read/write performance numbers :-
1. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=0 :-
fio Verify job :-
----------------------
read: IOPS=24.3k, BW=94.9MiB/s (99.5MB/s)(506MiB/5335msec)
write: IOPS=27.9k, BW=109MiB/s (114MB/s)(800MiB/7353msec);
fio Random Write :-
----------------------
write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60002msec);
write: IOPS=68.6k, BW=268MiB/s (281MB/s)(15.7GiB/60003msec);
write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60003msec);
write: IOPS=68.3k, BW=267MiB/s (280MB/s)(15.6GiB/60003msec);
write: IOPS=69.0k, BW=269MiB/s (283MB/s)(15.8GiB/60002msec);
write: IOPS=69.6k, BW=272MiB/s (285MB/s)(15.9GiB/60003msec);
write: IOPS=69.8k, BW=272MiB/s (286MB/s)(16.0GiB/60003msec);
write: IOPS=69.8k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec);
write: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60003msec);
write: IOPS=71.0k, BW=277MiB/s (291MB/s)(16.2GiB/60003msec);
fio Random Read :-
----------------------
read: IOPS=72.8k, BW=284MiB/s (298MB/s)(16.7GiB/60037msec)
read: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60037msec)
read: IOPS=67.5k, BW=264MiB/s (277MB/s)(15.5GiB/60033msec)
read: IOPS=74.5k, BW=291MiB/s (305MB/s)(17.1GiB/60035msec)
read: IOPS=72.4k, BW=283MiB/s (296MB/s)(16.6GiB/60036msec)
read: IOPS=71.8k, BW=281MiB/s (294MB/s)(16.5GiB/60036msec)
read: IOPS=73.7k, BW=288MiB/s (302MB/s)(16.9GiB/60035msec)
read: IOPS=72.3k, BW=283MiB/s (296MB/s)(16.6GiB/60039msec)
read: IOPS=72.8k, BW=285MiB/s (298MB/s)(16.7GiB/60037msec)
read: IOPS=74.6k, BW=291MiB/s (306MB/s)(17.1GiB/60036msec)
2. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=1 :-
fio Verify job :-
----------------------
read: IOPS=195k, BW=763MiB/s (800MB/s)(506MiB/663msec)
write: IOPS=123k, BW=481MiB/s (505MB/s)(800MiB/1662msec);
fio Random Write :-
----------------------
write: IOPS=4248k, BW=16.2GiB/s (17.4GB/s)(972GiB/60002msec);
write: IOPS=4157k, BW=15.9GiB/s (17.0GB/s)(951GiB/60002msec);
write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
write: IOPS=4130k, BW=15.8GiB/s (16.9GB/s)(945GiB/60002msec);
write: IOPS=4170k, BW=15.9GiB/s (17.1GB/s)(955GiB/60002msec);
write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
write: IOPS=4166k, BW=15.9GiB/s (17.1GB/s)(954GiB/60002msec);
write: IOPS=4106k, BW=15.7GiB/s (16.8GB/s)(940GiB/60002msec);
write: IOPS=4181k, BW=15.9GiB/s (17.1GB/s)(957GiB/60001msec);
write: IOPS=4153k, BW=15.8GiB/s (17.0GB/s)(951GiB/60002msec);
fio Random Read :-
----------------------
read: IOPS=4942k, BW=18.9GiB/s (20.2GB/s)(1131GiB/60002msec)
read: IOPS=4940k, BW=18.8GiB/s (20.2GB/s)(1131GiB/60002msec)
read: IOPS=5041k, BW=19.2GiB/s (20.6GB/s)(1154GiB/60002msec)
read: IOPS=4989k, BW=19.0GiB/s (20.4GB/s)(1142GiB/60001msec)
read: IOPS=4982k, BW=19.0GiB/s (20.4GB/s)(1140GiB/60001msec)
read: IOPS=5031k, BW=19.2GiB/s (20.6GB/s)(1152GiB/60001msec)
read: IOPS=4948k, BW=18.9GiB/s (20.3GB/s)(1133GiB/60002msec)
read: IOPS=4720k, BW=18.0GiB/s (19.3GB/s)(1080GiB/60002msec)
read: IOPS=4830k, BW=18.4GiB/s (19.8GB/s)(1106GiB/60002msec)
read: IOPS=4915k, BW=18.7GiB/s (20.1GB/s)(1125GiB/60002msec)
-ck
changes from V2:
1. Add a comment in the nvmet_bdev_ns_enable() when returning early
for buffered_io case. Feel free to edit the comment at the time
of applying patch.
changes from v1:
1. Remove first two patches and don't check for stat mode for
block device.
2. use i_size_read() to get the size of the file.
Chaitanya Kulkarni (2):
nvmet: use i_size_read() to set size for file-ns
nvmet: allow bdev in buffered_io mode
drivers/nvme/target/io-cmd-bdev.c | 8 ++++++++
drivers/nvme/target/io-cmd-file.c | 17 ++++-------------
drivers/nvme/target/nvmet.h | 2 +-
3 files changed, 13 insertions(+), 14 deletions(-)
root@dev nvme (nvme-5.17) # ./bdev-buffered-io
bdev-buffered-io/ bdev-buffered-io-test.sh
root@dev nvme (nvme-5.17) # ./bdev-buffered-io-test.sh
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)
real 0m0.350s
user 0m0.003s
sys 0m0.002s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
+ umount /mnt/nvme0n1
+ clear_dmesg
./compile_nvme.sh: line 3: clear_dmesg: command not found
umount: /mnt/nvme0n1: no mount point specified.
+ ./delete.sh 1
+ NQN=testnqn
+ nvme disconnect -n testnqn
Failed to disconnect by NQN: testnqn
real 0m0.001s
user 0m0.000s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ sleep 1
+ git diff
+ sleep 1
++ nproc
+ make -j 48 M=drivers/nvme/ modules
+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/
++ uname -r
+ TARGET_DEST=/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/ /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//
/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/:
total 6.7M
-rw-r--r--. 1 root root 2.8M Feb 1 23:42 nvme-core.ko
-rw-r--r--. 1 root root 448K Feb 1 23:42 nvme-fabrics.ko
-rw-r--r--. 1 root root 1015K Feb 1 23:42 nvme-fc.ko
-rw-r--r--. 1 root root 745K Feb 1 23:42 nvme.ko
-rw-r--r--. 1 root root 958K Feb 1 23:42 nvme-rdma.ko
-rw-r--r--. 1 root root 902K Feb 1 23:42 nvme-tcp.ko
/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//:
total 6.7M
-rw-r--r--. 1 root root 498K Feb 1 23:42 nvme-fcloop.ko
-rw-r--r--. 1 root root 439K Feb 1 23:42 nvme-loop.ko
-rw-r--r--. 1 root root 768K Feb 1 23:42 nvmet-fc.ko
-rw-r--r--. 1 root root 3.3M Feb 1 23:42 nvmet.ko
-rw-r--r--. 1 root root 929K Feb 1 23:42 nvmet-rdma.ko
-rw-r--r--. 1 root root 775K Feb 1 23:42 nvmet-tcp.ko
+ modprobe nvme
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 0 controller(s)
real 0m0.001s
user 0m0.001s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
+++ NN=1
+++ NQN=testnqn
+++ let NR_DEVICES=NN+1
+++ modprobe -r null_blk
+++ modprobe -r nvme
+++ modprobe null_blk nr_devices=0
+++ modprobe nvme
+++ modprobe nvme-fabrics
+++ modprobe nvmet
+++ modprobe nvme-loop
+++ dmesg -c
+++ sleep 2
+++ tree /sys/kernel/config
/sys/kernel/config
├── nullb
│ └── features
└── nvmet
├── hosts
├── ports
└── subsystems
5 directories, 1 file
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+++ echo -n /dev/nvme0n1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path
/dev/nvme0n1
+++ echo 1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
1
+++ echo 1
+++ mkdir /sys/kernel/config/nvmet/ports/1/
+++ echo -n loop
+++ echo -n 1
+++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/
+++ sleep 1
+++ echo transport=loop,nqn=testnqn
+++ sleep 1
+++ mount
+++ column -t
+++ grep nvme
+++ dmesg -c
[ 2348.622209] nvme nvme0: 48/0/0 default/read/poll queues
[ 2350.605991] nvmet: adding nsid 1 to subsystem testnqn
[ 2351.616737] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:bb4f2c46-9ac7-4283-902a-420883b62f3c.
[ 2351.616857] nvme nvme1: creating 48 I/O queues.
[ 2351.621168] nvme nvme1: new ctrl: "testnqn"
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.27
Starting 1 process
Jobs: 1 (f=1)
write-and-verify: (groupid=0, jobs=1): err= 0: pid=20757: Tue Feb 1 23:42:57 2022
read: IOPS=195k, BW=763MiB/s (800MB/s)(506MiB/663msec)
slat (usec): min=3, max=422, avg= 3.96, stdev= 1.90
clat (nsec): min=1072, max=483948, avg=15610.29, stdev=5064.35
lat (usec): min=4, max=494, avg=19.64, stdev= 6.20
clat percentiles (nsec):
| 1.00th=[13248], 5.00th=[13376], 10.00th=[13504], 20.00th=[13632],
| 30.00th=[13760], 40.00th=[13888], 50.00th=[14016], 60.00th=[14144],
| 70.00th=[14272], 80.00th=[14528], 90.00th=[23424], 95.00th=[26752],
| 99.00th=[28544], 99.50th=[32384], 99.90th=[42752], 99.95th=[47360],
| 99.99th=[82432]
write: IOPS=123k, BW=481MiB/s (505MB/s)(800MiB/1662msec);
slat (usec): min=4, max=221, avg= 7.36, stdev= 2.87
clat (nsec): min=841, max=222422, avg=24669.34, stdev=8735.58
lat (usec): min=6, max=263, avg=32.11, stdev=11.14
clat percentiles (usec):
| 1.00th=[ 18], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19],
| 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 20], 60.00th=[ 21],
| 70.00th=[ 28], 80.00th=[ 36], 90.00th=[ 38], 95.00th=[ 40],
| 99.00th=[ 51], 99.50th=[ 58], 99.90th=[ 70], 99.95th=[ 81],
| 99.99th=[ 155]
bw ( KiB/s): min=150160, max=525424, per=83.10%, avg=409600.00, stdev=174551.65, samples=4
iops : min=37540, max=131356, avg=102400.00, stdev=43637.91, samples=4
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=64.99%, 50=34.34%, 100=0.65%
lat (usec) : 250=0.02%, 500=0.01%
cpu : usr=27.97%, sys=32.66%, ctx=333157, majf=0, minf=3053
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=763MiB/s (800MB/s), 763MiB/s-763MiB/s (800MB/s-800MB/s), io=506MiB (531MB), run=663-663msec
WRITE: bw=481MiB/s (505MB/s), 481MiB/s-481MiB/s (505MB/s-505MB/s), io=800MiB (839MB), run=1662-1662msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.2GiB/s][w=4236k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=20761: Tue Feb 1 23:43:57 2022
write: IOPS=4248k, BW=16.2GiB/s (17.4GB/s)(972GiB/60002msec);
slat (nsec): min=1112, max=24135k, avg=6779.81, stdev=5320.08
clat (nsec): min=721, max=24222k, avg=53033.78, stdev=21465.11
lat (usec): min=6, max=24230, avg=59.89, stdev=23.52
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 57],
| 70.00th=[ 58], 80.00th=[ 60], 90.00th=[ 63], 95.00th=[ 88],
| 99.00th=[ 126], 99.50th=[ 133], 99.90th=[ 186], 99.95th=[ 223],
| 99.99th=[ 277]
bw ( MiB/s): min=11452, max=20170, per=100.00%, avg=16608.41, stdev=84.92, samples=3808
iops : min=2931936, max=5163618, avg=4251753.76, stdev=21739.07, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=54.82%, 100=42.95%
lat (usec) : 250=2.21%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=11.32%, sys=36.88%, ctx=254865246, majf=0, minf=439
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,254859948,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=16.2GiB/s (17.4GB/s), 16.2GiB/s-16.2GiB/s (17.4GB/s-17.4GB/s), io=972GiB (1044GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=15.3GiB/s][w=4015k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=20878: Tue Feb 1 23:44:57 2022
write: IOPS=4157k, BW=15.9GiB/s (17.0GB/s)(951GiB/60002msec);
slat (nsec): min=1773, max=23957k, avg=6930.59, stdev=5690.00
clat (nsec): min=672, max=24058k, avg=54190.75, stdev=22713.63
lat (usec): min=6, max=24086, avg=61.20, stdev=24.89
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 94],
| 99.00th=[ 129], 99.50th=[ 137], 99.90th=[ 192], 99.95th=[ 225],
| 99.99th=[ 269]
bw ( MiB/s): min=11359, max=20085, per=100.00%, avg=16257.07, stdev=84.39, samples=3808
iops : min=2907927, max=5141948, avg=4161809.03, stdev=21605.08, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=54.02%, 100=42.99%
lat (usec) : 250=2.97%, 500=0.02%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=11.06%, sys=36.43%, ctx=249428122, majf=0, minf=464
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,249423456,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.0GB/s), 15.9GiB/s-15.9GiB/s (17.0GB/s-17.0GB/s), io=951GiB (1022GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.3GiB/s][w=4283k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=20954: Tue Feb 1 23:45:57 2022
write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
slat (nsec): min=1783, max=23922k, avg=6853.24, stdev=4821.07
clat (nsec): min=711, max=24147k, avg=53603.75, stdev=19747.55
lat (usec): min=5, max=24155, avg=60.53, stdev=21.64
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 60], 90.00th=[ 65], 95.00th=[ 85],
| 99.00th=[ 118], 99.50th=[ 133], 99.90th=[ 167], 99.95th=[ 190],
| 99.99th=[ 251]
bw ( MiB/s): min=11876, max=20042, per=100.00%, avg=16429.89, stdev=81.89, samples=3808
iops : min=3040376, max=5130956, avg=4206051.45, stdev=20963.19, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.72%, 100=45.12%
lat (usec) : 250=2.14%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=11.02%, sys=37.05%, ctx=252169523, majf=0, minf=473
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,252164251,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=962GiB (1033GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=14.1GiB/s][w=3695k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=21089: Tue Feb 1 23:46:58 2022
write: IOPS=4130k, BW=15.8GiB/s (16.9GB/s)(945GiB/60002msec);
slat (nsec): min=1222, max=12690k, avg=6987.00, stdev=5499.44
clat (nsec): min=711, max=16691k, avg=54537.79, stdev=23234.80
lat (usec): min=5, max=16701, avg=61.61, stdev=25.56
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 69], 95.00th=[ 93],
| 99.00th=[ 131], 99.50th=[ 141], 99.90th=[ 184], 99.95th=[ 204],
| 99.99th=[ 260]
bw ( MiB/s): min=11551, max=20105, per=100.00%, avg=16156.07, stdev=74.80, samples=3808
iops : min=2957123, max=5146990, avg=4135953.66, stdev=19148.37, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.25%, 100=44.71%
lat (usec) : 250=3.03%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=10.94%, sys=36.18%, ctx=247812660, majf=0, minf=472
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,247805833,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.8GiB/s (16.9GB/s), 15.8GiB/s-15.8GiB/s (16.9GB/s-16.9GB/s), io=945GiB (1015GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.3GiB/s][w=4268k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22043: Tue Feb 1 23:47:58 2022
write: IOPS=4170k, BW=15.9GiB/s (17.1GB/s)(955GiB/60002msec);
slat (nsec): min=1763, max=17410k, avg=6908.89, stdev=5766.44
clat (nsec): min=712, max=17505k, avg=54011.25, stdev=23339.67
lat (usec): min=5, max=17513, avg=61.00, stdev=25.50
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 68], 95.00th=[ 87],
| 99.00th=[ 121], 99.50th=[ 135], 99.90th=[ 178], 99.95th=[ 204],
| 99.99th=[ 310]
bw ( MiB/s): min=11707, max=19992, per=100.00%, avg=16303.82, stdev=73.70, samples=3808
iops : min=2997012, max=5118104, avg=4173776.76, stdev=18867.69, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.47%, 100=45.42%
lat (usec) : 250=2.10%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=11.04%, sys=36.68%, ctx=250242780, majf=0, minf=461
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,250236670,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=955GiB (1025GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.0GiB/s][w=4186k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22119: Tue Feb 1 23:48:58 2022
write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec);
slat (nsec): min=1162, max=13647k, avg=6841.84, stdev=4990.36
clat (nsec): min=832, max=17776k, avg=53604.58, stdev=21052.59
lat (usec): min=6, max=17783, avg=60.53, stdev=22.89
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 50], 60.00th=[ 58],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 65], 95.00th=[ 79],
| 99.00th=[ 103], 99.50th=[ 121], 99.90th=[ 153], 99.95th=[ 176],
| 99.99th=[ 306]
bw ( MiB/s): min=11133, max=20082, per=100.00%, avg=16434.73, stdev=79.43, samples=3808
iops : min=2850202, max=5141134, avg=4207290.84, stdev=20334.84, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=50.62%
lat (usec) : 100=48.12%, 250=1.25%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=11.77%, sys=36.68%, ctx=252168793, majf=0, minf=462
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,252163391,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=962GiB (1033GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.5GiB/s][w=4326k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22221: Tue Feb 1 23:49:58 2022
write: IOPS=4166k, BW=15.9GiB/s (17.1GB/s)(954GiB/60002msec);
slat (nsec): min=1804, max=16555k, avg=6911.03, stdev=4216.56
clat (nsec): min=731, max=16639k, avg=54075.05, stdev=17110.09
lat (usec): min=5, max=16647, avg=61.07, stdev=18.98
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 49], 60.00th=[ 58],
| 70.00th=[ 60], 80.00th=[ 62], 90.00th=[ 67], 95.00th=[ 81],
| 99.00th=[ 123], 99.50th=[ 135], 99.90th=[ 165], 99.95th=[ 184],
| 99.99th=[ 210]
bw ( MiB/s): min=11319, max=20001, per=100.00%, avg=16285.86, stdev=86.40, samples=3808
iops : min=2897836, max=5120368, avg=4169180.79, stdev=22118.90, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=51.04%
lat (usec) : 100=47.21%, 250=1.75%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=11.43%, sys=36.52%, ctx=249967922, majf=0, minf=464
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,249962350,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=954GiB (1024GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.1GiB/s][w=4224k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22298: Tue Feb 1 23:50:59 2022
write: IOPS=4106k, BW=15.7GiB/s (16.8GB/s)(940GiB/60002msec);
slat (nsec): min=1372, max=18974k, avg=7033.73, stdev=4335.83
clat (nsec): min=721, max=19073k, avg=54854.61, stdev=19806.39
lat (usec): min=6, max=19082, avg=61.97, stdev=22.12
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 74], 95.00th=[ 95],
| 99.00th=[ 133], 99.50th=[ 147], 99.90th=[ 184], 99.95th=[ 196],
| 99.99th=[ 219]
bw ( MiB/s): min=11060, max=20108, per=100.00%, avg=16051.21, stdev=92.86, samples=3808
iops : min=2831464, max=5147868, avg=4109110.77, stdev=23772.60, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=53.51%
lat (usec) : 100=42.99%, 250=3.49%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=10.98%, sys=35.65%, ctx=246371950, majf=0, minf=458
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,246367496,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.7GiB/s (16.8GB/s), 15.7GiB/s-15.7GiB/s (16.8GB/s-16.8GB/s), io=940GiB (1009GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=14.9GiB/s][w=3895k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=22372: Tue Feb 1 23:51:59 2022
write: IOPS=4181k, BW=15.9GiB/s (17.1GB/s)(957GiB/60001msec);
slat (nsec): min=1623, max=16114k, avg=6882.50, stdev=4431.39
clat (nsec): min=712, max=16232k, avg=53879.81, stdev=16614.37
lat (usec): min=6, max=16241, avg=60.84, stdev=18.29
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 51], 60.00th=[ 58],
| 70.00th=[ 60], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 80],
| 99.00th=[ 104], 99.50th=[ 129], 99.90th=[ 161], 99.95th=[ 182],
| 99.99th=[ 219]
bw ( MiB/s): min=12006, max=19952, per=100.00%, avg=16360.11, stdev=79.02, samples=3808
iops : min=3073692, max=5107912, avg=4188189.50, stdev=20228.57, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=49.59%, 100=49.27%
lat (usec) : 250=1.14%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=11.16%, sys=37.05%, ctx=250877181, majf=0, minf=498
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,250866850,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=957GiB (1028GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.1GiB/s][w=4230k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=23453: Tue Feb 1 23:52:59 2022
write: IOPS=4153k, BW=15.8GiB/s (17.0GB/s)(951GiB/60002msec);
slat (nsec): min=1824, max=12568k, avg=6939.60, stdev=3999.18
clat (nsec): min=701, max=16670k, avg=54238.72, stdev=17869.32
lat (usec): min=6, max=16677, avg=61.26, stdev=19.85
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57],
| 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 69], 95.00th=[ 84],
| 99.00th=[ 129], 99.50th=[ 141], 99.90th=[ 176], 99.95th=[ 194],
| 99.99th=[ 217]
bw ( MiB/s): min=11198, max=20030, per=100.00%, avg=16238.98, stdev=82.50, samples=3808
iops : min=2866772, max=5127914, avg=4157179.99, stdev=21119.50, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.21%, 100=45.55%
lat (usec) : 250=2.23%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=10.99%, sys=36.59%, ctx=249190314, majf=0, minf=490
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,249183943,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.8GiB/s (17.0GB/s), 15.8GiB/s-15.8GiB/s (17.0GB/s-17.0GB/s), io=951GiB (1021GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.9GiB/s][r=4960k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=23530: Tue Feb 1 23:53:59 2022
read: IOPS=4942k, BW=18.9GiB/s (20.2GB/s)(1131GiB/60002msec)
slat (nsec): min=1142, max=49607k, avg=8828.54, stdev=15569.98
clat (nsec): min=441, max=243040k, avg=612190.41, stdev=339908.18
lat (usec): min=6, max=243043, avg=621.12, stdev=343.94
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1139], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1926], 99.95th=[ 4817],
| 99.99th=[11469]
bw ( MiB/s): min= 1691, max=27419, per=100.00%, avg=19320.99, stdev=89.34, samples=5712
iops : min=433098, max=7019516, avg=4946173.24, stdev=22869.85, samples=5712
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=15.40%, 750=69.25%, 1000=4.03%
lat (msec) : 2=11.24%, 4=0.03%, 10=0.04%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%, 250=0.01%
cpu : usr=9.29%, sys=35.84%, ctx=296529230, majf=0, minf=3836
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=296546544,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.9GiB/s (20.2GB/s), 18.9GiB/s-18.9GiB/s (20.2GB/s-20.2GB/s), io=1131GiB (1215GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.4GiB/s][r=4834k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=23882: Tue Feb 1 23:55:00 2022
read: IOPS=4940k, BW=18.8GiB/s (20.2GB/s)(1131GiB/60002msec)
slat (usec): min=2, max=16996, avg= 8.84, stdev= 6.11
clat (nsec): min=911, max=21566k, avg=612493.08, stdev=234517.40
lat (usec): min=4, max=21574, avg=621.43, stdev=238.08
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1123], 95.00th=[ 1172],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1549], 99.95th=[ 2024],
| 99.99th=[ 3261]
bw ( MiB/s): min=12590, max=26589, per=100.00%, avg=19314.78, stdev=65.21, samples=5712
iops : min=3223108, max=6806844, avg=4944582.49, stdev=16693.22, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=14.27%, 750=69.32%, 1000=5.38%
lat (msec) : 2=10.98%, 4=0.05%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.29%, sys=36.21%, ctx=296412664, majf=0, minf=3763
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=296403321,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.8GiB/s (20.2GB/s), 18.8GiB/s-18.8GiB/s (20.2GB/s-20.2GB/s), io=1131GiB (1214GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.6GiB/s][r=4885k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=24112: Tue Feb 1 23:56:00 2022
read: IOPS=5041k, BW=19.2GiB/s (20.6GB/s)(1154GiB/60002msec)
slat (usec): min=2, max=22808, avg= 8.65, stdev= 6.86
clat (nsec): min=912, max=26105k, avg=600221.79, stdev=221404.88
lat (usec): min=5, max=26112, avg=608.96, stdev=224.71
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 424], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1106], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1287], 99.95th=[ 1319],
| 99.99th=[ 2057]
bw ( MiB/s): min=16300, max=26946, per=100.00%, avg=19713.39, stdev=67.57, samples=5712
iops : min=4173052, max=6898318, avg=5046626.47, stdev=17299.17, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=12.91%, 750=73.53%, 1000=3.26%
lat (msec) : 2=10.28%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.37%, sys=36.74%, ctx=302479289, majf=0, minf=3758
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=302468143,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=19.2GiB/s (20.6GB/s), 19.2GiB/s-19.2GiB/s (20.6GB/s-20.6GB/s), io=1154GiB (1239GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.9GiB/s][r=4683k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=24212: Tue Feb 1 23:57:00 2022
read: IOPS=4989k, BW=19.0GiB/s (20.4GB/s)(1142GiB/60001msec)
slat (nsec): min=1953, max=52925k, avg=8751.89, stdev=11303.93
clat (nsec): min=912, max=56316k, avg=606515.90, stdev=246434.14
lat (usec): min=5, max=56326, avg=615.36, stdev=249.97
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 519],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1123], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1270], 99.90th=[ 1745], 99.95th=[ 1827],
| 99.99th=[ 4752]
bw ( MiB/s): min=13300, max=27673, per=100.00%, avg=19511.23, stdev=82.41, samples=5712
iops : min=3405011, max=7084445, avg=4994872.61, stdev=21096.46, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=14.24%, 750=70.95%, 1000=3.71%
lat (msec) : 2=11.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=9.31%, sys=36.30%, ctx=299345988, majf=0, minf=3730
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=299326708,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=19.0GiB/s (20.4GB/s), 19.0GiB/s-19.0GiB/s (20.4GB/s-20.4GB/s), io=1142GiB (1226GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.8GiB/s][r=4674k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25485: Tue Feb 1 23:58:00 2022
read: IOPS=4982k, BW=19.0GiB/s (20.4GB/s)(1140GiB/60001msec)
slat (nsec): min=1473, max=41255k, avg=8761.08, stdev=7916.28
clat (nsec): min=871, max=45519k, avg=607322.90, stdev=229101.48
lat (usec): min=5, max=45526, avg=616.18, stdev=232.51
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 519],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1106], 95.00th=[ 1172],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1287], 99.95th=[ 1303],
| 99.99th=[ 1745]
bw ( MiB/s): min=15677, max=27351, per=100.00%, avg=19485.07, stdev=73.52, samples=5712
iops : min=4013328, max=7002048, avg=4988177.24, stdev=18821.75, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=14.54%, 750=69.53%, 1000=5.29%
lat (msec) : 2=10.62%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.33%, sys=36.13%, ctx=298932803, majf=0, minf=3800
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=298924191,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=19.0GiB/s (20.4GB/s), 19.0GiB/s-19.0GiB/s (20.4GB/s-20.4GB/s), io=1140GiB (1224GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.2GiB/s][r=4763k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25572: Tue Feb 1 23:59:01 2022
read: IOPS=5031k, BW=19.2GiB/s (20.6GB/s)(1152GiB/60001msec)
slat (nsec): min=1985, max=21488k, avg=8673.54, stdev=5859.90
clat (nsec): min=902, max=22056k, avg=601397.58, stdev=214885.23
lat (usec): min=5, max=22064, avg=610.17, stdev=218.14
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 424], 20.00th=[ 523],
| 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 553], 80.00th=[ 553], 90.00th=[ 971], 95.00th=[ 1172],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1270], 99.95th=[ 1287],
| 99.99th=[ 1631]
bw ( MiB/s): min=14781, max=27237, per=100.00%, avg=19676.38, stdev=75.29, samples=5712
iops : min=3784135, max=6972871, avg=5037153.12, stdev=19275.07, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=13.17%, 750=72.94%, 1000=4.22%
lat (msec) : 2=9.67%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.32%, sys=36.93%, ctx=301882480, majf=0, minf=3751
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=301872061,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=19.2GiB/s (20.6GB/s), 19.2GiB/s-19.2GiB/s (20.6GB/s-20.6GB/s), io=1152GiB (1236GB), run=60001-60001msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.6GiB/s][r=4607k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25664: Wed Feb 2 00:00:01 2022
read: IOPS=4948k, BW=18.9GiB/s (20.3GB/s)(1133GiB/60002msec)
slat (usec): min=2, max=21123, avg= 8.83, stdev= 5.96
clat (nsec): min=942, max=22372k, avg=611470.57, stdev=236265.95
lat (usec): min=6, max=22391, avg=620.40, stdev=239.84
clat percentiles (usec):
| 1.00th=[ 396], 5.00th=[ 408], 10.00th=[ 416], 20.00th=[ 515],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 545], 80.00th=[ 562], 90.00th=[ 1139], 95.00th=[ 1188],
| 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1287], 99.95th=[ 1303],
| 99.99th=[ 1795]
bw ( MiB/s): min=15222, max=27543, per=100.00%, avg=19354.61, stdev=79.33, samples=5712
iops : min=3896862, max=7051174, avg=4954779.41, stdev=20309.68, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=16.87%, 750=66.30%, 1000=4.54%
lat (msec) : 2=12.28%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.19%, sys=35.87%, ctx=296904832, majf=0, minf=3743
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=296894931,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.9GiB/s (20.3GB/s), 18.9GiB/s-18.9GiB/s (20.3GB/s-20.3GB/s), io=1133GiB (1216GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.4GiB/s][r=4554k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25785: Wed Feb 2 00:01:01 2022
read: IOPS=4720k, BW=18.0GiB/s (19.3GB/s)(1080GiB/60002msec)
slat (usec): min=2, max=28533, avg= 9.30, stdev= 7.03
clat (nsec): min=921, max=29089k, avg=640985.06, stdev=258045.84
lat (usec): min=5, max=29096, avg=650.39, stdev=261.95
clat percentiles (usec):
| 1.00th=[ 392], 5.00th=[ 404], 10.00th=[ 412], 20.00th=[ 510],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 553], 80.00th=[ 906], 90.00th=[ 1156], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1303], 99.95th=[ 1401],
| 99.99th=[ 1876]
bw ( MiB/s): min=13222, max=27706, per=100.00%, avg=18458.01, stdev=98.55, samples=5712
iops : min=3384868, max=7092862, avg=4725251.34, stdev=25229.04, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=18.90%, 750=56.44%, 1000=10.71%
lat (msec) : 2=13.94%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=8.95%, sys=33.65%, ctx=283232675, majf=0, minf=3746
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=283217843,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.0GiB/s (19.3GB/s), 18.0GiB/s-18.0GiB/s (19.3GB/s-19.3GB/s), io=1080GiB (1160GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.2GiB/s][r=4774k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=25897: Wed Feb 2 00:02:01 2022
read: IOPS=4830k, BW=18.4GiB/s (19.8GB/s)(1106GiB/60002msec)
slat (usec): min=2, max=37297, avg= 9.06, stdev=11.53
clat (nsec): min=892, max=37911k, avg=626423.87, stdev=281337.79
lat (usec): min=5, max=37919, avg=635.59, stdev=285.46
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 429], 20.00th=[ 515],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537],
| 70.00th=[ 545], 80.00th=[ 562], 90.00th=[ 1156], 95.00th=[ 1205],
| 99.00th=[ 1549], 99.50th=[ 1680], 99.90th=[ 2835], 99.95th=[ 3392],
| 99.99th=[ 5407]
bw ( MiB/s): min= 6580, max=27466, per=100.00%, avg=18885.17, stdev=100.72, samples=5712
iops : min=1684599, max=7031464, avg=4834602.87, stdev=25783.81, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=12.90%, 750=70.01%, 1000=4.60%
lat (msec) : 2=12.35%, 4=0.13%, 10=0.02%, 20=0.01%, 50=0.01%
cpu : usr=9.17%, sys=37.36%, ctx=289835159, majf=0, minf=3741
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=289804479,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.4GiB/s (19.8GB/s), 18.4GiB/s-18.4GiB/s (19.8GB/s-19.8GB/s), io=1106GiB (1187GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=18.3GiB/s][r=4805k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=26489: Wed Feb 2 00:03:02 2022
read: IOPS=4915k, BW=18.7GiB/s (20.1GB/s)(1125GiB/60002msec)
slat (nsec): min=1714, max=56074k, avg=8893.16, stdev=12899.01
clat (nsec): min=922, max=57329k, avg=615610.19, stdev=258832.65
lat (usec): min=5, max=57337, avg=624.60, stdev=262.51
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 408], 10.00th=[ 420], 20.00th=[ 515],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 545], 80.00th=[ 562], 90.00th=[ 1139], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1418], 99.95th=[ 1811],
| 99.99th=[ 4948]
bw ( MiB/s): min=14198, max=26839, per=100.00%, avg=19217.89, stdev=63.10, samples=5712
iops : min=3634881, max=6870838, avg=4919778.50, stdev=16154.49, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=15.84%, 750=67.55%, 1000=3.63%
lat (msec) : 2=12.94%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=9.32%, sys=35.56%, ctx=294925914, majf=0, minf=3702
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=294900613,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.7GiB/s (20.1GB/s), 18.7GiB/s-18.7GiB/s (20.1GB/s-20.1GB/s), io=1125GiB (1208GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
+ NQN=testnqn
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)
real 0m0.349s
user 0m0.000s
sys 0m0.003s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
+++ NN=1
+++ NQN=testnqn
+++ let NR_DEVICES=NN+1
+++ modprobe -r null_blk
+++ modprobe -r nvme
+++ modprobe null_blk nr_devices=0
+++ modprobe nvme
+++ modprobe nvme-fabrics
+++ modprobe nvmet
+++ modprobe nvme-loop
+++ dmesg -c
+++ sleep 2
+++ tree /sys/kernel/config
/sys/kernel/config
├── nullb
│ └── features
└── nvmet
├── hosts
├── ports
└── subsystems
5 directories, 1 file
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
+++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+++ echo -n /dev/nvme0n1
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path
/dev/nvme0n1
+++ echo 0
+++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
0
+++ echo 1
+++ mkdir /sys/kernel/config/nvmet/ports/1/
+++ echo -n loop
+++ echo -n 1
+++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/
+++ sleep 1
+++ echo transport=loop,nqn=testnqn
+++ sleep 1
+++ mount
+++ column -t
+++ grep nvme
+++ dmesg -c
[ 3560.886773] nvme nvme0: 48/0/0 default/read/poll queues
[ 3562.872451] nvmet: adding nsid 1 to subsystem testnqn
[ 3563.883217] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:bc654437-83be-4628-91bd-a439b8e830d5.
[ 3563.883353] nvme nvme1: creating 48 I/O queues.
[ 3563.887931] nvme nvme1: new ctrl: "testnqn"
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.27
Starting 1 process
Jobs: 1 (f=1): [V(1)][80.0%][r=99.2MiB/s][r=25.4k IOPS][eta 00m:03s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=27715: Wed Feb 2 00:03:19 2022
read: IOPS=24.3k, BW=94.9MiB/s (99.5MB/s)(506MiB/5335msec)
slat (nsec): min=1032, max=1447.4k, avg=8870.69, stdev=27575.11
clat (usec): min=11, max=4669, avg=153.54, stdev=131.98
lat (usec): min=47, max=4672, avg=162.54, stdev=137.66
clat percentiles (usec):
| 1.00th=[ 62], 5.00th=[ 86], 10.00th=[ 96], 20.00th=[ 106],
| 30.00th=[ 113], 40.00th=[ 116], 50.00th=[ 120], 60.00th=[ 125],
| 70.00th=[ 135], 80.00th=[ 198], 90.00th=[ 221], 95.00th=[ 251],
| 99.00th=[ 709], 99.50th=[ 947], 99.90th=[ 1663], 99.95th=[ 2114],
| 99.99th=[ 2933]
write: IOPS=27.9k, BW=109MiB/s (114MB/s)(800MiB/7353msec);
slat (usec): min=2, max=3897, avg=13.26, stdev=21.42
clat (usec): min=15, max=5720, avg=129.39, stdev=79.93
lat (usec): min=37, max=6857, avg=142.77, stdev=84.05
clat percentiles (usec):
| 1.00th=[ 63], 5.00th=[ 77], 10.00th=[ 84], 20.00th=[ 96],
| 30.00th=[ 105], 40.00th=[ 115], 50.00th=[ 123], 60.00th=[ 129],
| 70.00th=[ 137], 80.00th=[ 151], 90.00th=[ 169], 95.00th=[ 215],
| 99.00th=[ 273], 99.50th=[ 310], 99.90th=[ 1057], 99.95th=[ 1696],
| 99.99th=[ 2868]
bw ( KiB/s): min=82496, max=143312, per=98.04%, avg=109226.13, stdev=16354.90, samples=15
iops : min=20624, max=35828, avg=27306.67, stdev=4088.95, samples=15
lat (usec) : 20=0.01%, 50=0.28%, 100=19.29%, 250=77.38%, 500=2.09%
lat (usec) : 750=0.52%, 1000=0.20%
lat (msec) : 2=0.19%, 4=0.05%, 10=0.01%
cpu : usr=18.65%, sys=16.14%, ctx=141033, majf=0, minf=3053
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=94.9MiB/s (99.5MB/s), 94.9MiB/s-94.9MiB/s (99.5MB/s-99.5MB/s), io=506MiB (531MB), run=5335-5335msec
WRITE: bw=109MiB/s (114MB/s), 109MiB/s-109MiB/s (114MB/s-114MB/s), io=800MiB (839MB), run=7353-7353msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=253MiB/s][w=64.8k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=27744: Wed Feb 2 00:04:20 2022
write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60002msec);
slat (nsec): min=1072, max=42904k, avg=162051.61, stdev=467637.32
clat (nsec): min=752, max=72679k, avg=3588602.81, stdev=2243011.38
lat (usec): min=52, max=72681, avg=3750.87, stdev=2268.70
clat percentiles (usec):
| 1.00th=[ 562], 5.00th=[ 1029], 10.00th=[ 1369], 20.00th=[ 1860],
| 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3589],
| 70.00th=[ 4178], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7701],
| 99.00th=[11076], 99.50th=[12780], 99.90th=[18220], 99.95th=[21890],
| 99.99th=[30278]
bw ( KiB/s): min=193029, max=335346, per=100.00%, avg=273269.63, stdev=877.09, samples=3808
iops : min=48248, max=83826, avg=68310.92, stdev=219.29, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.16%
lat (usec) : 100=0.04%, 250=0.09%, 500=0.46%, 750=1.38%, 1000=2.46%
lat (msec) : 2=18.84%, 4=43.56%, 10=31.38%, 20=1.55%, 50=0.07%
lat (msec) : 100=0.01%
cpu : usr=0.63%, sys=1.57%, ctx=1935887, majf=0, minf=485
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4093399,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=266MiB/s (279MB/s), 266MiB/s-266MiB/s (279MB/s-279MB/s), io=15.6GiB (16.8GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=251MiB/s][w=64.3k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=27840: Wed Feb 2 00:05:20 2022
write: IOPS=68.6k, BW=268MiB/s (281MB/s)(15.7GiB/60003msec);
slat (nsec): min=1052, max=52671k, avg=146009.86, stdev=449344.58
clat (nsec): min=641, max=64251k, avg=3581258.82, stdev=2231979.80
lat (usec): min=55, max=64976, avg=3727.49, stdev=2255.52
clat percentiles (usec):
| 1.00th=[ 570], 5.00th=[ 1037], 10.00th=[ 1369], 20.00th=[ 1844],
| 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3589],
| 70.00th=[ 4178], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7701],
| 99.00th=[10945], 99.50th=[12649], 99.90th=[18220], 99.95th=[21627],
| 99.99th=[30016]
bw ( KiB/s): min=203172, max=338441, per=100.00%, avg=275054.71, stdev=839.49, samples=3808
iops : min=50788, max=84602, avg=68758.55, stdev=209.87, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.15%
lat (usec) : 100=0.03%, 250=0.08%, 500=0.45%, 750=1.40%, 1000=2.44%
lat (msec) : 2=18.96%, 4=43.62%, 10=31.29%, 20=1.51%, 50=0.07%
lat (msec) : 100=0.01%
cpu : usr=0.63%, sys=1.56%, ctx=1832905, majf=0, minf=488
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4119099,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=268MiB/s (281MB/s), 268MiB/s-268MiB/s (281MB/s-281MB/s), io=15.7GiB (16.9GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=270MiB/s][w=69.1k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=27949: Wed Feb 2 00:06:20 2022
write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60003msec);
slat (nsec): min=1082, max=41371k, avg=222396.07, stdev=539362.31
clat (nsec): min=631, max=67998k, avg=3529440.50, stdev=2240119.90
lat (usec): min=54, max=69361, avg=3752.10, stdev=2276.78
clat percentiles (usec):
| 1.00th=[ 529], 5.00th=[ 1012], 10.00th=[ 1352], 20.00th=[ 1827],
| 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3523],
| 70.00th=[ 4113], 80.00th=[ 4883], 90.00th=[ 6194], 95.00th=[ 7570],
| 99.00th=[11076], 99.50th=[12780], 99.90th=[20055], 99.95th=[23987],
| 99.99th=[31327]
bw ( KiB/s): min=195270, max=336343, per=100.00%, avg=273029.60, stdev=880.99, samples=3808
iops : min=48813, max=84075, avg=68253.36, stdev=220.24, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.20%
lat (usec) : 100=0.05%, 250=0.13%, 500=0.52%, 750=1.45%, 1000=2.48%
lat (msec) : 2=19.07%, 4=44.45%, 10=30.09%, 20=1.45%, 50=0.10%
lat (msec) : 100=0.01%
cpu : usr=0.64%, sys=1.69%, ctx=2408175, majf=0, minf=508
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4091921,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=266MiB/s (279MB/s), 266MiB/s-266MiB/s (279MB/s-279MB/s), io=15.6GiB (16.8GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=268MiB/s][w=68.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=28031: Wed Feb 2 00:07:20 2022
write: IOPS=68.3k, BW=267MiB/s (280MB/s)(15.6GiB/60003msec);
slat (nsec): min=1062, max=50611k, avg=206709.24, stdev=515844.58
clat (nsec): min=491, max=58217k, avg=3538847.51, stdev=2211954.38
lat (usec): min=46, max=58793, avg=3745.81, stdev=2241.07
clat percentiles (usec):
| 1.00th=[ 529], 5.00th=[ 1020], 10.00th=[ 1352], 20.00th=[ 1844],
| 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3556],
| 70.00th=[ 4146], 80.00th=[ 4883], 90.00th=[ 6259], 95.00th=[ 7570],
| 99.00th=[10945], 99.50th=[12649], 99.90th=[18744], 99.95th=[22414],
| 99.99th=[30540]
bw ( KiB/s): min=205666, max=333724, per=100.00%, avg=273520.75, stdev=810.61, samples=3808
iops : min=51408, max=83423, avg=68374.75, stdev=202.66, samples=3808
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.21%
lat (usec) : 100=0.05%, 250=0.12%, 500=0.50%, 750=1.42%, 1000=2.45%
lat (msec) : 2=18.99%, 4=44.21%, 10=30.54%, 20=1.43%, 50=0.08%
lat (msec) : 100=0.01%
cpu : usr=0.66%, sys=1.67%, ctx=2291683, majf=0, minf=481
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4098881,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=15.6GiB (16.8GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=268MiB/s][w=68.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=28274: Wed Feb 2 00:08:21 2022
write: IOPS=69.0k, BW=269MiB/s (283MB/s)(15.8GiB/60002msec);
slat (nsec): min=1062, max=68056k, avg=176848.85, stdev=488062.07
clat (nsec): min=692, max=77359k, avg=3532992.98, stdev=2215070.84
lat (usec): min=42, max=78568, avg=3710.06, stdev=2245.78
clat percentiles (usec):
| 1.00th=[ 545], 5.00th=[ 1012], 10.00th=[ 1352], 20.00th=[ 1827],
| 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3556],
| 70.00th=[ 4146], 80.00th=[ 4948], 90.00th=[ 6259], 95.00th=[ 7570],
| 99.00th=[10814], 99.50th=[12518], 99.90th=[18482], 99.95th=[22414],
| 99.99th=[30540]
bw ( KiB/s): min=201233, max=338661, per=100.00%, avg=276170.71, stdev=819.64, samples=3808
iops : min=50302, max=84658, avg=69037.91, stdev=204.91, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.16%
lat (usec) : 100=0.04%, 250=0.11%, 500=0.50%, 750=1.46%, 1000=2.53%
lat (msec) : 2=19.26%, 4=43.92%, 10=30.52%, 20=1.41%, 50=0.08%
lat (msec) : 100=0.01%
cpu : usr=0.64%, sys=1.61%, ctx=2102892, majf=0, minf=494
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4138405,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=269MiB/s (283MB/s), 269MiB/s-269MiB/s (283MB/s-283MB/s), io=15.8GiB (16.9GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=0): [f(32)][100.0%][w=277MiB/s][w=71.0k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29446: Wed Feb 2 00:09:21 2022
write: IOPS=69.6k, BW=272MiB/s (285MB/s)(15.9GiB/60003msec);
slat (nsec): min=1052, max=50297k, avg=138125.49, stdev=435715.13
clat (nsec): min=521, max=59414k, avg=3537010.84, stdev=2183018.58
lat (usec): min=51, max=61090, avg=3675.34, stdev=2211.08
clat percentiles (usec):
| 1.00th=[ 570], 5.00th=[ 1029], 10.00th=[ 1352], 20.00th=[ 1827],
| 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3556],
| 70.00th=[ 4146], 80.00th=[ 4948], 90.00th=[ 6259], 95.00th=[ 7570],
| 99.00th=[10683], 99.50th=[12256], 99.90th=[17433], 99.95th=[21365],
| 99.99th=[30540]
bw ( KiB/s): min=206519, max=341031, per=100.00%, avg=278664.74, stdev=817.73, samples=3808
iops : min=51622, max=85253, avg=69660.15, stdev=204.42, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.12%
lat (usec) : 100=0.03%, 250=0.08%, 500=0.47%, 750=1.43%, 1000=2.51%
lat (msec) : 2=19.26%, 4=43.83%, 10=30.86%, 20=1.34%, 50=0.06%
lat (msec) : 100=0.01%
cpu : usr=0.64%, sys=1.55%, ctx=1803573, majf=0, minf=455
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4177602,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=15.9GiB (17.1GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=0): [f(32)][100.0%][w=272MiB/s][w=69.7k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29556: Wed Feb 2 00:10:21 2022
write: IOPS=69.8k, BW=272MiB/s (286MB/s)(16.0GiB/60003msec);
slat (nsec): min=1052, max=35550k, avg=79344.28, stdev=323613.65
clat (nsec): min=771, max=53581k, avg=3588937.03, stdev=2176963.25
lat (usec): min=58, max=53842, avg=3668.46, stdev=2181.80
clat percentiles (usec):
| 1.00th=[ 586], 5.00th=[ 1037], 10.00th=[ 1369], 20.00th=[ 1844],
| 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3621],
| 70.00th=[ 4228], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701],
| 99.00th=[10683], 99.50th=[12125], 99.90th=[16057], 99.95th=[18744],
| 99.99th=[27395]
bw ( KiB/s): min=204774, max=339176, per=100.00%, avg=279266.54, stdev=802.82, samples=3808
iops : min=51191, max=84786, avg=69810.42, stdev=200.71, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.13%
lat (usec) : 100=0.03%, 250=0.06%, 500=0.44%, 750=1.37%, 1000=2.49%
lat (msec) : 2=18.96%, 4=43.01%, 10=32.04%, 20=1.42%, 50=0.04%
lat (msec) : 100=0.01%
cpu : usr=0.62%, sys=1.45%, ctx=1356660, majf=0, minf=493
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4185468,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=272MiB/s (286MB/s), 272MiB/s-272MiB/s (286MB/s-286MB/s), io=16.0GiB (17.1GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][98.4%][w=280MiB/s][w=71.7k IOPS][eta 00m:01s]
Jobs: 32 (f=32): [w(32)][100.0%][w=277MiB/s][w=70.9k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29643: Wed Feb 2 00:11:21 2022
write: IOPS=69.8k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec);
slat (nsec): min=1052, max=34378k, avg=59061.75, stdev=277238.20
clat (nsec): min=742, max=47279k, avg=3608224.03, stdev=2171757.46
lat (usec): min=48, max=47679, avg=3667.45, stdev=2172.85
clat percentiles (usec):
| 1.00th=[ 594], 5.00th=[ 1045], 10.00th=[ 1369], 20.00th=[ 1860],
| 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3654],
| 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701],
| 99.00th=[10683], 99.50th=[12125], 99.90th=[15664], 99.95th=[17433],
| 99.99th=[23725]
bw ( KiB/s): min=201486, max=341296, per=100.00%, avg=279357.72, stdev=825.73, samples=3808
iops : min=50369, max=85319, avg=69833.71, stdev=206.44, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.11%
lat (usec) : 100=0.03%, 250=0.05%, 500=0.42%, 750=1.37%, 1000=2.47%
lat (msec) : 2=18.87%, 4=42.63%, 10=32.55%, 20=1.47%, 50=0.02%
cpu : usr=0.60%, sys=1.42%, ctx=1195286, majf=0, minf=481
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4186642,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=16.0GiB (17.1GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=0): [f(32)][100.0%][w=272MiB/s][w=69.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29748: Wed Feb 2 00:12:22 2022
write: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60003msec);
slat (nsec): min=1072, max=25141k, avg=14918.87, stdev=115673.29
clat (nsec): min=752, max=37983k, avg=3564960.33, stdev=2091487.49
lat (usec): min=57, max=37985, avg=3580.03, stdev=2089.00
clat percentiles (usec):
| 1.00th=[ 635], 5.00th=[ 1057], 10.00th=[ 1369], 20.00th=[ 1860],
| 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3130], 60.00th=[ 3621],
| 70.00th=[ 4228], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7570],
| 99.00th=[10421], 99.50th=[11600], 99.90th=[14615], 99.95th=[15926],
| 99.99th=[18744]
bw ( KiB/s): min=223459, max=340302, per=100.00%, avg=286170.92, stdev=703.56, samples=3808
iops : min=55856, max=85068, avg=71536.67, stdev=175.90, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.04%
lat (usec) : 100=0.02%, 250=0.02%, 500=0.37%, 750=1.33%, 1000=2.48%
lat (msec) : 2=19.22%, 4=43.12%, 10=32.14%, 20=1.26%, 50=0.01%
cpu : usr=0.61%, sys=1.38%, ctx=872668, majf=0, minf=472
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4288951,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=279MiB/s (293MB/s), 279MiB/s-279MiB/s (293MB/s-293MB/s), io=16.4GiB (17.6GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 2 (f=0): [_(12),f(2),_(18)][100.0%][w=266MiB/s][w=68.1k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=29827: Wed Feb 2 00:13:22 2022
write: IOPS=71.0k, BW=277MiB/s (291MB/s)(16.2GiB/60003msec);
slat (nsec): min=1062, max=32432k, avg=19937.63, stdev=148066.81
clat (nsec): min=811, max=59246k, avg=3585601.00, stdev=2115468.91
lat (usec): min=52, max=59254, avg=3605.70, stdev=2113.21
clat percentiles (usec):
| 1.00th=[ 635], 5.00th=[ 1074], 10.00th=[ 1385], 20.00th=[ 1860],
| 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3621],
| 70.00th=[ 4228], 80.00th=[ 5014], 90.00th=[ 6390], 95.00th=[ 7635],
| 99.00th=[10552], 99.50th=[11731], 99.90th=[14877], 99.95th=[16188],
| 99.99th=[19792]
bw ( KiB/s): min=222503, max=339437, per=100.00%, avg=284148.08, stdev=711.74, samples=3808
iops : min=55619, max=84852, avg=71031.03, stdev=177.94, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.05%
lat (usec) : 100=0.02%, 250=0.02%, 500=0.36%, 750=1.29%, 1000=2.44%
lat (msec) : 2=18.96%, 4=43.25%, 10=32.27%, 20=1.33%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=0.61%, sys=1.42%, ctx=918578, majf=0, minf=478
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4258348,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=277MiB/s (291MB/s), 277MiB/s-277MiB/s (291MB/s-291MB/s), io=16.2GiB (17.4GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=295MiB/s][r=75.5k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=30964: Wed Feb 2 00:14:22 2022
read: IOPS=72.8k, BW=284MiB/s (298MB/s)(16.7GiB/60037msec)
slat (nsec): min=971, max=173108k, avg=39325.53, stdev=826441.69
clat (nsec): min=1894, max=316249k, avg=42166512.42, stdev=14952004.11
lat (usec): min=417, max=316251, avg=42205.96, stdev=15032.61
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 24], 10.00th=[ 28], 20.00th=[ 31],
| 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 59], 95.00th=[ 68],
| 99.00th=[ 91], 99.50th=[ 99], 99.90th=[ 121], 99.95th=[ 140],
| 99.99th=[ 201]
bw ( KiB/s): min=142106, max=403311, per=100.00%, avg=291270.21, stdev=902.19, samples=5664
iops : min=35512, max=100811, avg=72801.07, stdev=225.55, samples=5664
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.05%, 4=0.22%, 10=1.02%, 20=3.07%, 50=76.23%
lat (msec) : 100=18.97%, 250=0.42%, 500=0.01%
cpu : usr=0.24%, sys=0.55%, ctx=261137, majf=0, minf=3757
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4368373,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=284MiB/s (298MB/s), 284MiB/s-284MiB/s (298MB/s-298MB/s), io=16.7GiB (17.9GB), run=60037-60037msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=300MiB/s][r=76.9k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31076: Wed Feb 2 00:15:23 2022
read: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60037msec)
slat (nsec): min=961, max=88836k, avg=17391.79, stdev=551047.29
clat (usec): min=22, max=221605, avg=42951.57, stdev=12630.75
lat (usec): min=383, max=222644, avg=42969.08, stdev=12673.43
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 32],
| 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 51], 90.00th=[ 58], 95.00th=[ 63],
| 99.00th=[ 84], 99.50th=[ 93], 99.90th=[ 112], 99.95th=[ 123],
| 99.99th=[ 150]
bw ( KiB/s): min=170354, max=369108, per=100.00%, avg=286058.08, stdev=686.26, samples=5664
iops : min=42573, max=92262, avg=71497.39, stdev=171.56, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.06%, 10=0.36%, 20=1.78%, 50=77.37%
lat (msec) : 100=20.13%, 250=0.27%
cpu : usr=0.22%, sys=0.53%, ctx=218767, majf=0, minf=3743
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4290648,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=279MiB/s (293MB/s), 279MiB/s-279MiB/s (293MB/s-293MB/s), io=16.4GiB (17.6GB), run=60037-60037msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=299MiB/s][r=76.4k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31174: Wed Feb 2 00:16:23 2022
read: IOPS=67.5k, BW=264MiB/s (277MB/s)(15.5GiB/60033msec)
slat (nsec): min=941, max=113553k, avg=20605.09, stdev=608598.15
clat (usec): min=20, max=244312, avg=45449.46, stdev=13495.74
lat (usec): min=78, max=244315, avg=45470.18, stdev=13549.72
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 28], 10.00th=[ 30], 20.00th=[ 34],
| 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 45], 60.00th=[ 47],
| 70.00th=[ 50], 80.00th=[ 55], 90.00th=[ 61], 95.00th=[ 66],
| 99.00th=[ 92], 99.50th=[ 101], 99.90th=[ 120], 99.95th=[ 129],
| 99.99th=[ 163]
bw ( KiB/s): min=147119, max=347537, per=100.00%, avg=270244.42, stdev=628.01, samples=5664
iops : min=36762, max=86866, avg=67544.04, stdev=157.00, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.08%, 10=0.40%, 20=1.57%, 50=70.55%
lat (msec) : 100=26.85%, 250=0.52%
cpu : usr=0.21%, sys=0.51%, ctx=217032, majf=0, minf=3833
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4054701,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=264MiB/s (277MB/s), 264MiB/s-264MiB/s (277MB/s-277MB/s), io=15.5GiB (16.6GB), run=60033-60033msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=296MiB/s][r=75.8k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31271: Wed Feb 2 00:17:23 2022
read: IOPS=74.5k, BW=291MiB/s (305MB/s)(17.1GiB/60035msec)
slat (nsec): min=952, max=94814k, avg=15742.78, stdev=457046.68
clat (usec): min=38, max=202717, avg=41217.82, stdev=11676.36
lat (usec): min=65, max=202721, avg=41233.69, stdev=11708.54
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 44],
| 70.00th=[ 46], 80.00th=[ 48], 90.00th=[ 56], 95.00th=[ 61],
| 99.00th=[ 79], 99.50th=[ 88], 99.90th=[ 105], 99.95th=[ 112],
| 99.99th=[ 140]
bw ( KiB/s): min=182011, max=362829, per=100.00%, avg=298134.45, stdev=603.05, samples=5664
iops : min=45487, max=90694, avg=74515.97, stdev=150.76, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.06%, 10=0.30%, 20=1.80%, 50=82.63%
lat (msec) : 100=15.05%, 250=0.14%
cpu : usr=0.23%, sys=0.55%, ctx=232008, majf=0, minf=3740
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4471696,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=291MiB/s (305MB/s), 291MiB/s-291MiB/s (305MB/s-305MB/s), io=17.1GiB (18.3GB), run=60035-60035msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=294MiB/s][r=75.2k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=31366: Wed Feb 2 00:18:23 2022
read: IOPS=72.4k, BW=283MiB/s (296MB/s)(16.6GiB/60036msec)
slat (nsec): min=952, max=131203k, avg=21974.73, stdev=596139.49
clat (usec): min=3, max=270950, avg=42409.20, stdev=13195.67
lat (usec): min=80, max=270961, avg=42431.29, stdev=13247.16
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 27], 10.00th=[ 28], 20.00th=[ 31],
| 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 43], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 58], 95.00th=[ 64],
| 99.00th=[ 86], 99.50th=[ 93], 99.90th=[ 109], 99.95th=[ 120],
| 99.99th=[ 161]
bw ( KiB/s): min=163161, max=388414, per=100.00%, avg=289753.58, stdev=752.52, samples=5664
iops : min=40771, max=97088, avg=72421.87, stdev=188.12, samples=5664
lat (usec) : 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.03%, 4=0.14%, 10=0.67%, 20=2.25%, 50=77.40%
lat (msec) : 100=19.26%, 250=0.25%, 500=0.01%
cpu : usr=0.23%, sys=0.53%, ctx=232559, majf=0, minf=3758
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4345429,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=283MiB/s (296MB/s), 283MiB/s-283MiB/s (296MB/s-296MB/s), io=16.6GiB (17.8GB), run=60036-60036msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=288MiB/s][r=73.8k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32520: Wed Feb 2 00:19:24 2022
read: IOPS=71.8k, BW=281MiB/s (294MB/s)(16.5GiB/60036msec)
slat (nsec): min=961, max=155112k, avg=23329.58, stdev=600906.79
clat (usec): min=18, max=269118, avg=42721.27, stdev=13343.53
lat (usec): min=109, max=269121, avg=42744.72, stdev=13395.21
clat percentiles (msec):
| 1.00th=[ 13], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 51], 90.00th=[ 58], 95.00th=[ 64],
| 99.00th=[ 88], 99.50th=[ 96], 99.90th=[ 113], 99.95th=[ 125],
| 99.99th=[ 176]
bw ( KiB/s): min=155002, max=377376, per=100.00%, avg=287593.81, stdev=736.00, samples=5664
iops : min=38739, max=94329, avg=71881.32, stdev=184.00, samples=5664
lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.02%, 4=0.10%, 10=0.54%, 20=2.09%, 50=77.36%
lat (msec) : 100=19.56%, 250=0.33%, 500=0.01%
cpu : usr=0.22%, sys=0.54%, ctx=235545, majf=0, minf=3694
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4313415,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=281MiB/s (294MB/s), 281MiB/s-281MiB/s (294MB/s-294MB/s), io=16.5GiB (17.7GB), run=60036-60036msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=291MiB/s][r=74.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32630: Wed Feb 2 00:20:24 2022
read: IOPS=73.7k, BW=288MiB/s (302MB/s)(16.9GiB/60035msec)
slat (nsec): min=962, max=177671k, avg=38019.32, stdev=644096.49
clat (usec): min=29, max=287028, avg=41615.50, stdev=13238.93
lat (usec): min=94, max=319545, avg=41653.64, stdev=13291.04
clat percentiles (msec):
| 1.00th=[ 12], 5.00th=[ 26], 10.00th=[ 28], 20.00th=[ 31],
| 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 45],
| 70.00th=[ 46], 80.00th=[ 49], 90.00th=[ 57], 95.00th=[ 63],
| 99.00th=[ 88], 99.50th=[ 95], 99.90th=[ 112], 99.95th=[ 123],
| 99.99th=[ 176]
bw ( KiB/s): min=156946, max=373773, per=100.00%, avg=295110.41, stdev=718.25, samples=5664
iops : min=39217, max=93429, avg=73760.01, stdev=179.56, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.11%, 10=0.60%, 20=2.37%, 50=80.04%
lat (msec) : 100=16.54%, 250=0.31%, 500=0.01%
cpu : usr=0.23%, sys=0.56%, ctx=327850, majf=0, minf=3771
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4426283,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=288MiB/s (302MB/s), 288MiB/s-288MiB/s (302MB/s-302MB/s), io=16.9GiB (18.1GB), run=60035-60035msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=292MiB/s][r=74.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32752: Wed Feb 2 00:21:24 2022
read: IOPS=72.3k, BW=283MiB/s (296MB/s)(16.6GiB/60039msec)
slat (nsec): min=972, max=125387k, avg=20090.72, stdev=496977.06
clat (nsec): min=1563, max=210556k, avg=42435419.14, stdev=12289751.96
lat (usec): min=123, max=210559, avg=42455.63, stdev=12326.24
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 45],
| 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 58], 95.00th=[ 63],
| 99.00th=[ 82], 99.50th=[ 91], 99.90th=[ 108], 99.95th=[ 116],
| 99.99th=[ 146]
bw ( KiB/s): min=169789, max=367924, per=100.00%, avg=289626.85, stdev=669.34, samples=5664
iops : min=42433, max=91970, avg=72388.99, stdev=167.32, samples=5664
lat (usec) : 2=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.02%, 4=0.07%, 10=0.34%, 20=1.74%, 50=79.17%
lat (msec) : 100=18.44%, 250=0.22%
cpu : usr=0.23%, sys=0.52%, ctx=248980, majf=0, minf=3751
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4343055,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=283MiB/s (296MB/s), 283MiB/s-283MiB/s (296MB/s-296MB/s), io=16.6GiB (17.8GB), run=60039-60039msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=294MiB/s][r=75.3k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32851: Wed Feb 2 00:22:25 2022
read: IOPS=72.8k, BW=285MiB/s (298MB/s)(16.7GiB/60037msec)
slat (nsec): min=961, max=88402k, avg=16580.98, stdev=499828.64
clat (usec): min=16, max=190979, avg=42147.65, stdev=12291.43
lat (usec): min=76, max=200180, avg=42164.34, stdev=12331.85
clat percentiles (msec):
| 1.00th=[ 14], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 43], 60.00th=[ 45],
| 70.00th=[ 46], 80.00th=[ 50], 90.00th=[ 57], 95.00th=[ 62],
| 99.00th=[ 83], 99.50th=[ 92], 99.90th=[ 108], 99.95th=[ 116],
| 99.99th=[ 142]
bw ( KiB/s): min=170567, max=369399, per=100.00%, avg=291617.48, stdev=663.74, samples=5664
iops : min=42628, max=92333, avg=72887.66, stdev=165.93, samples=5664
lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
lat (usec) : 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.07%, 10=0.39%, 20=1.84%, 50=79.83%
lat (msec) : 100=17.63%, 250=0.22%
cpu : usr=0.23%, sys=0.53%, ctx=222791, majf=0, minf=3753
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4372949,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=285MiB/s (298MB/s), 285MiB/s-285MiB/s (298MB/s-298MB/s), io=16.7GiB (17.9GB), run=60037-60037msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=296MiB/s][r=75.7k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=32948: Wed Feb 2 00:23:25 2022
read: IOPS=74.6k, BW=291MiB/s (306MB/s)(17.1GiB/60036msec)
slat (nsec): min=982, max=143449k, avg=14518.04, stdev=453275.74
clat (usec): min=39, max=211504, avg=41151.34, stdev=11534.04
lat (usec): min=227, max=230931, avg=41165.97, stdev=11567.45
clat percentiles (msec):
| 1.00th=[ 15], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 44],
| 70.00th=[ 46], 80.00th=[ 48], 90.00th=[ 55], 95.00th=[ 60],
| 99.00th=[ 79], 99.50th=[ 87], 99.90th=[ 103], 99.95th=[ 112],
| 99.99th=[ 148]
bw ( KiB/s): min=182147, max=368337, per=100.00%, avg=298660.44, stdev=576.72, samples=5664
iops : min=45520, max=92066, avg=74647.02, stdev=144.17, samples=5664
lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.07%, 10=0.33%, 20=1.75%, 50=83.09%
lat (msec) : 100=14.62%, 250=0.13%
cpu : usr=0.23%, sys=0.54%, ctx=233134, majf=0, minf=3732
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4479122,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=291MiB/s (306MB/s), 291MiB/s-291MiB/s (306MB/s-306MB/s), io=17.1GiB (18.3GB), run=60036-60036msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
--
2.29.0
^ permalink raw reply [flat|nested] 4+ messages in thread
* [PATCH V3 1/2] nvmet: use i_size_read() to set size for file-ns
2022-02-02 9:04 [PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns Chaitanya Kulkarni
@ 2022-02-02 9:04 ` Chaitanya Kulkarni
2022-02-02 9:04 ` [PATCH V3 2/2] nvmet: allow bdev in buffered_io mode Chaitanya Kulkarni
2022-02-02 16:07 ` [PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns Christoph Hellwig
2 siblings, 0 replies; 4+ messages in thread
From: Chaitanya Kulkarni @ 2022-02-02 9:04 UTC (permalink / raw)
To: linux-nvme; +Cc: hch, kbusch, sagi, Chaitanya Kulkarni
From: Chaitanya Kulkarni <kch@nvidia.com>
Instead of calling vfs_getattr() use i_size_read() to read the size of
file so we can read the size of not only file type but also block type
with one call. This is needed to implement buffered_io support for the
NVMeOF block device backend.
We also change return type of function nvmet_file_ns_revalidate() from
int to void, since this function does not return any meaning value.
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/nvme/target/io-cmd-file.c | 17 ++++-------------
drivers/nvme/target/nvmet.h | 2 +-
2 files changed, 5 insertions(+), 14 deletions(-)
diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c
index 6be6e59d273b..6485dc8eb974 100644
--- a/drivers/nvme/target/io-cmd-file.c
+++ b/drivers/nvme/target/io-cmd-file.c
@@ -14,16 +14,9 @@
#define NVMET_MAX_MPOOL_BVEC 16
#define NVMET_MIN_MPOOL_OBJ 16
-int nvmet_file_ns_revalidate(struct nvmet_ns *ns)
+void nvmet_file_ns_revalidate(struct nvmet_ns *ns)
{
- struct kstat stat;
- int ret;
-
- ret = vfs_getattr(&ns->file->f_path, &stat, STATX_SIZE,
- AT_STATX_FORCE_SYNC);
- if (!ret)
- ns->size = stat.size;
- return ret;
+ ns->size = i_size_read(ns->file->f_mapping->host);
}
void nvmet_file_ns_disable(struct nvmet_ns *ns)
@@ -43,7 +36,7 @@ void nvmet_file_ns_disable(struct nvmet_ns *ns)
int nvmet_file_ns_enable(struct nvmet_ns *ns)
{
int flags = O_RDWR | O_LARGEFILE;
- int ret;
+ int ret = 0;
if (!ns->buffered_io)
flags |= O_DIRECT;
@@ -57,9 +50,7 @@ int nvmet_file_ns_enable(struct nvmet_ns *ns)
return ret;
}
- ret = nvmet_file_ns_revalidate(ns);
- if (ret)
- goto err;
+ nvmet_file_ns_revalidate(ns);
/*
* i_blkbits can be greater than the universally accepted upper bound,
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index af193423c10b..69637bf8f8e1 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -541,7 +541,7 @@ u16 nvmet_bdev_flush(struct nvmet_req *req);
u16 nvmet_file_flush(struct nvmet_req *req);
void nvmet_ns_changed(struct nvmet_subsys *subsys, u32 nsid);
void nvmet_bdev_ns_revalidate(struct nvmet_ns *ns);
-int nvmet_file_ns_revalidate(struct nvmet_ns *ns);
+void nvmet_file_ns_revalidate(struct nvmet_ns *ns);
void nvmet_ns_revalidate(struct nvmet_ns *ns);
u16 blk_to_nvme_status(struct nvmet_req *req, blk_status_t blk_sts);
--
2.29.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* [PATCH V3 2/2] nvmet: allow bdev in buffered_io mode
2022-02-02 9:04 [PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns Chaitanya Kulkarni
2022-02-02 9:04 ` [PATCH V3 1/2] nvmet: use i_size_read() to set size for file-ns Chaitanya Kulkarni
@ 2022-02-02 9:04 ` Chaitanya Kulkarni
2022-02-02 16:07 ` [PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns Christoph Hellwig
2 siblings, 0 replies; 4+ messages in thread
From: Chaitanya Kulkarni @ 2022-02-02 9:04 UTC (permalink / raw)
To: linux-nvme; +Cc: hch, kbusch, sagi, Chaitanya Kulkarni
From: Chaitanya Kulkarni <kch@nvidia.com>
Allow block device to be configured in the buffered I/O mode by using
the file backend. In this way now we can use cache for the block
device namespace which shows significant performance improvement.
We update the block device ns enable function and return early when
buffered_io flag is set.
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/nvme/target/io-cmd-bdev.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index 70ca9dfc1771..8b2cfde7ad63 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -76,6 +76,14 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
{
int ret;
+ /*
+ * When buffered_io namespace attribute is enabled that means user want
+ * this block device to be used as a file, so block device can take
+ * an advantage of cache.
+ */
+ if (ns->buffered_io)
+ return -ENOTBLK;
+
ns->bdev = blkdev_get_by_path(ns->device_path,
FMODE_READ | FMODE_WRITE, NULL);
if (IS_ERR(ns->bdev)) {
--
2.29.0
^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns
2022-02-02 9:04 [PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns Chaitanya Kulkarni
2022-02-02 9:04 ` [PATCH V3 1/2] nvmet: use i_size_read() to set size for file-ns Chaitanya Kulkarni
2022-02-02 9:04 ` [PATCH V3 2/2] nvmet: allow bdev in buffered_io mode Chaitanya Kulkarni
@ 2022-02-02 16:07 ` Christoph Hellwig
2 siblings, 0 replies; 4+ messages in thread
From: Christoph Hellwig @ 2022-02-02 16:07 UTC (permalink / raw)
To: Chaitanya Kulkarni; +Cc: linux-nvme, hch, kbusch, sagi, Chaitanya Kulkarni
Thanks,
applied to nvme-5.18.
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-02-02 16:18 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-02-02 9:04 [PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns Chaitanya Kulkarni
2022-02-02 9:04 ` [PATCH V3 1/2] nvmet: use i_size_read() to set size for file-ns Chaitanya Kulkarni
2022-02-02 9:04 ` [PATCH V3 2/2] nvmet: allow bdev in buffered_io mode Chaitanya Kulkarni
2022-02-02 16:07 ` [PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns Christoph Hellwig
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.