* [PATCH 0/3] nvmet: add buffered_io support for bdev-ns
@ 2022-01-19 7:20 Chaitanya Kulkarni
2022-01-19 7:20 ` [PATCH 1/3] nvmet: add helper to set ns->blkshift value Chaitanya Kulkarni
` (3 more replies)
0 siblings, 4 replies; 11+ messages in thread
From: Chaitanya Kulkarni @ 2022-01-19 7:20 UTC (permalink / raw)
To: linux-nvme; +Cc: hch, kbusch, sagi, Chaitanya Kulkarni
From: Chaitanya Kulkarni <kch@nvidia.com>
Hi,
In current implementation like NVMeOF target file backed namespace we
don't support buffered_io for block device backed namesapce. This allows
user to configure block device backend in buffered_io mode so user can
get the benefits of caching.
Please note that using this combination host needs to periodically sync
between cache and storage or set the system cache parameters
accordingly. NVMeOF target is not responsible for data consistency
between cache and storage.
First two patches create helper functions which are used in the third
patch to avoid code duplication.
Following is the summary of fio performance numbers, see complete log at
the end of this cover-letter.
Data verification and fio performance numbers :-
1. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=0 :-
fio Verify job :-
----------------------
read: IOPS=32.1k, BW=125MiB/s (131MB/s)(506MiB/4042msec)
write: IOPS=27.3k, BW=107MiB/s (112MB/s)(800MiB/7489msec)
fio Random Write :-
----------------------
write: IOPS=69.9k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec)
write: IOPS=69.6k, BW=272MiB/s (285MB/s)(15.9GiB/60003msec)
write: IOPS=70.1k, BW=274MiB/s (287MB/s)(16.1GiB/60002msec)
write: IOPS=69.9k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec)
write: IOPS=70.1k, BW=274MiB/s (287MB/s)(16.1GiB/60004msec)
write: IOPS=69.9k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec)
write: IOPS=70.0k, BW=273MiB/s (287MB/s)(16.0GiB/60003msec)
write: IOPS=69.9k, BW=273MiB/s (287MB/s)(16.0GiB/60003msec)
write: IOPS=70.3k, BW=274MiB/s (288MB/s)(16.1GiB/60003msec)
write: IOPS=70.1k, BW=274MiB/s (287MB/s)(16.1GiB/60003msec)
fio Random Read :-
----------------------
read: IOPS=74.6k, BW=292MiB/s (306MB/s)(17.1GiB/60034msec)
read: IOPS=67.4k, BW=263MiB/s (276MB/s)(15.4GiB/60035msec)
read: IOPS=74.2k, BW=290MiB/s (304MB/s)(17.0GiB/60033msec)
read: IOPS=73.4k, BW=287MiB/s (301MB/s)(16.8GiB/60034msec)
read: IOPS=75.0k, BW=293MiB/s (307MB/s)(17.2GiB/60036msec)
read: IOPS=75.1k, BW=293MiB/s (308MB/s)(17.2GiB/60034msec)
read: IOPS=73.7k, BW=288MiB/s (302MB/s)(16.9GiB/60036msec)
read: IOPS=74.2k, BW=290MiB/s (304MB/s)(17.0GiB/60033msec)
read: IOPS=75.9k, BW=296MiB/s (311MB/s)(17.4GiB/60035msec)
read: IOPS=72.3k, BW=282MiB/s (296MB/s)(16.6GiB/60037msec)
2. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=1 :-
fio Verify job :-
----------------------
read: IOPS=163k, BW=638MiB/s (669MB/s)(506MiB/793msec)
write: IOPS=138k, BW=540MiB/s (566MB/s)(800MiB/1481msec);
fio Random Write :-
----------------------
write: IOPS=4129k, BW=15.7GiB/s (16.9GB/s)(945GiB/60001msec)
write: IOPS=4010k, BW=15.3GiB/s (16.4GB/s)(918GiB/60002msec)
write: IOPS=4170k, BW=15.9GiB/s (17.1GB/s)(955GiB/60001msec)
write: IOPS=4064k, BW=15.5GiB/s (16.6GB/s)(930GiB/60002msec)
write: IOPS=4078k, BW=15.6GiB/s (16.7GB/s)(933GiB/60002msec)
write: IOPS=4078k, BW=15.6GiB/s (16.7GB/s)(933GiB/60002msec)
write: IOPS=4136k, BW=15.8GiB/s (16.9GB/s)(947GiB/60002msec)
write: IOPS=3885k, BW=14.8GiB/s (15.9GB/s)(889GiB/60002msec)
write: IOPS=4020k, BW=15.3GiB/s (16.5GB/s)(920GiB/60002msec)
write: IOPS=4136k, BW=15.8GiB/s (16.9GB/s)(947GiB/60002msec)
fio Random Read :-
----------------------
read: IOPS=4859k, BW=18.5GiB/s (19.9GB/s)(1112GiB/60001msec)
read: IOPS=4862k, BW=18.5GiB/s (19.9GB/s)(1113GiB/60001msec)
read: IOPS=5134k, BW=19.6GiB/s (21.0GB/s)(1175GiB/60002msec)
read: IOPS=4860k, BW=18.5GiB/s (19.9GB/s)(1112GiB/60002msec)
read: IOPS=4756k, BW=18.1GiB/s (19.5GB/s)(1089GiB/60002msec)
read: IOPS=4706k, BW=18.0GiB/s (19.3GB/s)(1077GiB/60001msec)
read: IOPS=5039k, BW=19.2GiB/s (20.6GB/s)(1153GiB/60002msec)
read: IOPS=4670k, BW=17.8GiB/s (19.1GB/s)(1069GiB/60002msec)
read: IOPS=4882k, BW=18.6GiB/s (20.0GB/s)(1117GiB/60002msec)
read: IOPS=4861k, BW=18.5GiB/s (19.9GB/s)(1113GiB/60002msec)
-ck
Chaitanya Kulkarni (3):
nvmet: add helper to set ns->blkshift value
nvmet: add helper to open block device
nvmet: allow bdev in buffered_io mode
drivers/nvme/target/io-cmd-bdev.c | 22 ++++++++++++++++--
drivers/nvme/target/io-cmd-file.c | 38 ++++++++++++++++++++++++-------
drivers/nvme/target/nvmet.h | 1 +
3 files changed, 51 insertions(+), 10 deletions(-)
* FIO Verify, Random Read. Random write log :-
----------------------------------------------
1. Verbose log Block Device with buffered_io = 0 :-
---------------------------------------------------
root@dev nvme (nvme-5.17) # sh bdev-buffered-io-test.sh
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 0 controller(s)
real 0m0.001s
user 0m0.000s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
+ umount /mnt/nvme0n1
+ clear_dmesg
./compile_nvme.sh: line 3: clear_dmesg: command not found
umount: /mnt/nvme0n1: no mount point specified.
+ modprobe -r nvme-fabrics
+ modprobe -r nvme_loop
+ modprobe -r nvmet
+ modprobe -r nvme
+ sleep 1
+ modprobe -r nvme-core
+ lsmod
+ grep nvme
+ sleep 1
+ git diff
+ sleep 1
++ nproc
+ make -j 48 M=drivers/nvme/ modules
+ HOST=drivers/nvme/host
+ TARGET=drivers/nvme/target
++ uname -r
+ HOST_DEST=/lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/host/
++ uname -r
+ TARGET_DEST=/lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/target/
+ cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/host//
+ cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/target//
+ ls -lrth /lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/host/ /lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/target//
/lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/host/:
total 6.6M
-rw-r--r--. 1 root root 2.7M Jan 18 13:22 nvme-core.ko
-rw-r--r--. 1 root root 447K Jan 18 13:22 nvme-fabrics.ko
-rw-r--r--. 1 root root 965K Jan 18 13:22 nvme-fc.ko
-rw-r--r--. 1 root root 732K Jan 18 13:22 nvme.ko
-rw-r--r--. 1 root root 1021K Jan 18 13:22 nvme-rdma.ko
-rw-r--r--. 1 root root 897K Jan 18 13:22 nvme-tcp.ko
/lib/modules/5.16.0-rc3nvme+/kernel/drivers/nvme/target//:
total 6.7M
-rw-r--r--. 1 root root 498K Jan 18 13:22 nvme-fcloop.ko
-rw-r--r--. 1 root root 438K Jan 18 13:22 nvme-loop.ko
-rw-r--r--. 1 root root 767K Jan 18 13:22 nvmet-fc.ko
-rw-r--r--. 1 root root 3.3M Jan 18 13:22 nvmet.ko
-rw-r--r--. 1 root root 992K Jan 18 13:22 nvmet-rdma.ko
-rw-r--r--. 1 root root 768K Jan 18 13:22 nvmet-tcp.ko
+ modprobe nvme
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 0 controller(s)
real 0m0.001s
user 0m0.000s
sys 0m0.001s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory
+ rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*'
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
0 directories, 0 files
++ NN=1
++ NQN=testnqn
++ let NR_DEVICES=NN+1
++ modprobe -r null_blk
++ modprobe -r nvme
++ modprobe null_blk nr_devices=0
++ modprobe nvme
++ modprobe nvme-fabrics
++ modprobe nvmet
++ modprobe nvme-loop
++ dmesg -c
++ sleep 2
++ tree /sys/kernel/config
/sys/kernel/config
├── nullb
│ └── features
└── nvmet
├── hosts
├── ports
└── subsystems
5 directories, 1 file
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
++ echo -n /dev/nvme0n1
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path
/dev/nvme0n1
++ echo 0
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
0
++ echo 1
++ mkdir /sys/kernel/config/nvmet/ports/1/
++ echo -n loop
++ echo -n 1
++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/
++ sleep 1
++ echo transport=loop,nqn=testnqn
++ sleep 1
++ mount
++ column -t
++ grep nvme
++ '[' 1 ']'
+++ wc -l
+++ ls -l /dev/nvme1 /dev/nvme1n1
++ cnt=2
++ echo 2
2
++ '[' 2 -gt 1 ']'
++ break
++ dmesg -c
[55445.819019] nvme nvme0: 48/0/0 default/read/poll queues
[55447.805257] nvmet: adding nsid 1 to subsystem testnqn
[55448.815307] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:3aed9b58-95ea-477b-ac51-d87bc7d10326.
[55448.815470] nvme nvme1: creating 48 I/O queues.
[55448.819432] nvme nvme1: new ctrl: "testnqn"
++ sleep 5
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.27
Starting 1 process
Jobs: 1 (f=1): [V(1)][86.7%][r=102MiB/s][r=26.2k IOPS][eta 00m:02s]
write-and-verify: (groupid=0, jobs=1): err= 0: pid=11490: Tue Jan 18 13:22:52 2022
read: IOPS=27.7k, BW=108MiB/s (113MB/s)(506MiB/4680msec)
slat (nsec): min=1052, max=1431.4k, avg=4089.33, stdev=9877.17
clat (usec): min=12, max=3429, avg=138.18, stdev=75.13
lat (usec): min=44, max=3433, avg=142.39, stdev=75.17
clat percentiles (usec):
| 1.00th=[ 60], 5.00th=[ 78], 10.00th=[ 88], 20.00th=[ 101],
| 30.00th=[ 110], 40.00th=[ 114], 50.00th=[ 117], 60.00th=[ 122],
| 70.00th=[ 139], 80.00th=[ 196], 90.00th=[ 212], 95.00th=[ 223],
| 99.00th=[ 343], 99.50th=[ 457], 99.90th=[ 955], 99.95th=[ 1614],
| 99.99th=[ 1991]
write: IOPS=24.8k, BW=96.7MiB/s (101MB/s)(800MiB/8273msec); 0 zone resets
slat (usec): min=2, max=2817, avg=10.17, stdev=12.87
clat (usec): min=6, max=3742, avg=150.46, stdev=85.68
lat (usec): min=49, max=3748, avg=160.74, stdev=85.53
clat percentiles (usec):
| 1.00th=[ 59], 5.00th=[ 79], 10.00th=[ 90], 20.00th=[ 108],
| 30.00th=[ 119], 40.00th=[ 127], 50.00th=[ 133], 60.00th=[ 143],
| 70.00th=[ 159], 80.00th=[ 186], 90.00th=[ 229], 95.00th=[ 253],
| 99.00th=[ 408], 99.50th=[ 529], 99.90th=[ 766], 99.95th=[ 1598],
| 99.99th=[ 2999]
bw ( KiB/s): min=54560, max=141576, per=97.33%, avg=96376.47, stdev=19561.14, samples=17
iops : min=13640, max=35394, avg=24094.12, stdev=4890.28, samples=17
lat (usec) : 10=0.01%, 20=0.02%, 50=0.29%, 100=16.34%, 250=79.47%
lat (usec) : 500=3.41%, 750=0.36%, 1000=0.02%
lat (msec) : 2=0.07%, 4=0.02%
cpu : usr=19.11%, sys=15.27%, ctx=122606, majf=0, minf=3054
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=108MiB/s (113MB/s), 108MiB/s-108MiB/s (113MB/s-113MB/s), io=506MiB (531MB), run=4680-4680msec
WRITE: bw=96.7MiB/s (101MB/s), 96.7MiB/s-96.7MiB/s (101MB/s-101MB/s), io=800MiB (839MB), run=8273-8273msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 25 (f=0): [E(2),f(5),E(1),f(3),E(1),f(3),E(1),f(7),E(1),f(2),E(1),f(5)][100.0%][w=259MiB/s][w=66.4k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=12227: Tue Jan 18 13:23:52 2022
write: IOPS=68.8k, BW=269MiB/s (282MB/s)(15.7GiB/60003msec); 0 zone resets
slat (nsec): min=1042, max=29831k, avg=12162.75, stdev=103918.07
clat (nsec): min=511, max=38271k, avg=3708415.18, stdev=2189164.56
lat (usec): min=47, max=38800, avg=3720.72, stdev=2186.96
clat percentiles (usec):
| 1.00th=[ 652], 5.00th=[ 1090], 10.00th=[ 1418], 20.00th=[ 1926],
| 30.00th=[ 2343], 40.00th=[ 2802], 50.00th=[ 3228], 60.00th=[ 3752],
| 70.00th=[ 4359], 80.00th=[ 5211], 90.00th=[ 6587], 95.00th=[ 7898],
| 99.00th=[10814], 99.50th=[12256], 99.90th=[15401], 99.95th=[16909],
| 99.99th=[19530]
bw ( KiB/s): min=217922, max=328001, per=100.00%, avg=275365.13, stdev=666.91, samples=3808
iops : min=54470, max=81992, avg=68834.13, stdev=166.73, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.03%
lat (usec) : 100=0.01%, 250=0.02%, 500=0.33%, 750=1.22%, 1000=2.24%
lat (msec) : 2=17.95%, 4=42.33%, 10=34.25%, 20=1.60%, 50=0.01%
cpu : usr=0.59%, sys=1.35%, ctx=825570, majf=0, minf=480
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4126796,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=269MiB/s (282MB/s), 269MiB/s-269MiB/s (282MB/s-282MB/s), io=15.7GiB (16.9GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=0): [f(32)][100.0%][w=271MiB/s][w=69.4k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=12784: Tue Jan 18 13:24:53 2022
write: IOPS=69.3k, BW=271MiB/s (284MB/s)(15.9GiB/60003msec); 0 zone resets
slat (nsec): min=1052, max=18635k, avg=8508.06, stdev=73079.65
clat (nsec): min=761, max=33431k, avg=3684368.79, stdev=2159290.61
lat (usec): min=45, max=33436, avg=3693.02, stdev=2156.87
clat percentiles (usec):
| 1.00th=[ 652], 5.00th=[ 1090], 10.00th=[ 1418], 20.00th=[ 1909],
| 30.00th=[ 2343], 40.00th=[ 2769], 50.00th=[ 3228], 60.00th=[ 3752],
| 70.00th=[ 4359], 80.00th=[ 5211], 90.00th=[ 6521], 95.00th=[ 7832],
| 99.00th=[10814], 99.50th=[11994], 99.90th=[15008], 99.95th=[16319],
| 99.99th=[19006]
bw ( KiB/s): min=218885, max=330648, per=100.00%, avg=277393.20, stdev=661.31, samples=3808
iops : min=54714, max=82657, avg=69342.65, stdev=165.34, samples=3808
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.02%
lat (usec) : 100=0.01%, 250=0.01%, 500=0.33%, 750=1.20%, 1000=2.28%
lat (msec) : 2=18.09%, 4=42.50%, 10=34.04%, 20=1.50%, 50=0.01%
cpu : usr=0.59%, sys=1.34%, ctx=799313, majf=0, minf=473
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,4157753,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=271MiB/s (284MB/s), 271MiB/s-271MiB/s (284MB/s-284MB/s), io=15.9GiB (17.0GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=268MiB/s][w=68.6k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=12998: Tue Jan 18 13:25:53 2022
write: IOPS=65.0k, BW=254MiB/s (266MB/s)(14.9GiB/60003msec); 0 zone resets
slat (nsec): min=1042, max=43641k, avg=109484.94, stdev=394081.46
clat (nsec): min=632, max=55539k, avg=3829004.51, stdev=2354852.14
lat (usec): min=51, max=55541, avg=3938.69, stdev=2356.96
clat percentiles (usec):
| 1.00th=[ 594], 5.00th=[ 1090], 10.00th=[ 1434], 20.00th=[ 1958],
| 30.00th=[ 2409], 40.00th=[ 2868], 50.00th=[ 3326], 60.00th=[ 3851],
| 70.00th=[ 4490], 80.00th=[ 5407], 90.00th=[ 6849], 95.00th=[ 8291],
| 99.00th=[11600], 99.50th=[13173], 99.90th=[17695], 99.95th=[20579],
| 99.99th=[26870]
bw ( KiB/s): min=193792, max=318358, per=100.00%, avg=260028.15, stdev=771.99, samples=3808
iops : min=48444, max=79586, avg=65002.81, stdev=192.98, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.17%
lat (usec) : 100=0.04%, 250=0.06%, 500=0.39%, 750=1.20%, 1000=2.13%
lat (msec) : 2=16.86%, 4=41.57%, 10=35.41%, 20=2.09%, 50=0.06%
lat (msec) : 100=0.01%
cpu : usr=0.59%, sys=1.42%, ctx=1428856, majf=0, minf=477
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,3898434,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=254MiB/s (266MB/s), 254MiB/s-254MiB/s (266MB/s-266MB/s), io=14.9GiB (16.0GB), run=60003-60003msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=291MiB/s][r=74.5k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=13629: Tue Jan 18 13:26:53 2022
read: IOPS=73.8k, BW=288MiB/s (302MB/s)(16.9GiB/60034msec)
slat (nsec): min=971, max=92291k, avg=7601.08, stdev=294169.16
clat (nsec): min=881, max=172495k, avg=41609264.60, stdev=10573216.33
lat (usec): min=73, max=186206, avg=41616.97, stdev=10588.91
clat percentiles (msec):
| 1.00th=[ 17], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 43], 60.00th=[ 44],
| 70.00th=[ 46], 80.00th=[ 48], 90.00th=[ 56], 95.00th=[ 60],
| 99.00th=[ 69], 99.50th=[ 79], 99.90th=[ 96], 99.95th=[ 104],
| 99.99th=[ 127]
bw ( KiB/s): min=197554, max=351693, per=100.00%, avg=295385.18, stdev=487.78, samples=5664
iops : min=49373, max=87909, avg=73828.39, stdev=121.94, samples=5664
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%
lat (usec) : 1000=0.01%
lat (msec) : 2=0.01%, 4=0.02%, 10=0.12%, 20=1.19%, 50=83.06%
lat (msec) : 100=15.54%, 250=0.06%
cpu : usr=0.22%, sys=0.54%, ctx=211466, majf=0, minf=3783
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4430082,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=288MiB/s (302MB/s), 288MiB/s-288MiB/s (302MB/s-302MB/s), io=16.9GiB (18.1GB), run=60034-60034msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=290MiB/s][r=74.3k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=13758: Tue Jan 18 13:27:54 2022
read: IOPS=74.7k, BW=292MiB/s (306MB/s)(17.1GiB/60033msec)
slat (nsec): min=961, max=67006k, avg=5644.52, stdev=214199.20
clat (usec): min=54, max=161646, avg=41099.12, stdev=9934.03
lat (usec): min=748, max=161649, avg=41104.87, stdev=9941.10
clat percentiles (msec):
| 1.00th=[ 18], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 35], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 44],
| 70.00th=[ 46], 80.00th=[ 48], 90.00th=[ 54], 95.00th=[ 59],
| 99.00th=[ 65], 99.50th=[ 70], 99.90th=[ 86], 99.95th=[ 92],
| 99.99th=[ 109]
bw ( KiB/s): min=223837, max=345279, per=100.00%, avg=299116.64, stdev=387.45, samples=5664
iops : min=55941, max=86304, avg=74761.37, stdev=96.85, samples=5664
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.07%, 20=1.08%, 50=84.88%
lat (msec) : 100=13.93%, 250=0.02%
cpu : usr=0.22%, sys=0.55%, ctx=212760, majf=0, minf=3830
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4485508,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=17.1GiB (18.4GB), run=60033-60033msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=295MiB/s][r=75.5k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=13832: Tue Jan 18 13:28:54 2022
read: IOPS=74.7k, BW=292MiB/s (306MB/s)(17.1GiB/60033msec)
slat (nsec): min=972, max=88341k, avg=5140.64, stdev=192310.77
clat (usec): min=62, max=175937, avg=41091.41, stdev=9929.06
lat (usec): min=566, max=228627, avg=41096.66, stdev=9935.15
clat percentiles (msec):
| 1.00th=[ 18], 5.00th=[ 28], 10.00th=[ 29], 20.00th=[ 31],
| 30.00th=[ 35], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 44],
| 70.00th=[ 46], 80.00th=[ 48], 90.00th=[ 54], 95.00th=[ 59],
| 99.00th=[ 65], 99.50th=[ 70], 99.90th=[ 87], 99.95th=[ 94],
| 99.99th=[ 110]
bw ( KiB/s): min=223518, max=345146, per=100.00%, avg=299138.52, stdev=394.76, samples=5664
iops : min=55860, max=86264, avg=74766.90, stdev=98.69, samples=5664
lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.06%, 20=1.06%, 50=85.06%
lat (msec) : 100=13.77%, 250=0.02%
cpu : usr=0.23%, sys=0.54%, ctx=211810, majf=0, minf=3779
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=4486204,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=17.1GiB (18.4GB), run=60033-60033msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
+ nvme disconnect -n testnqn
NQN:testnqn disconnected 1 controller(s)
real 0m0.358s
user 0m0.000s
sys 0m0.004s
+ for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/*
+ echo 0
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
+ rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn
+ rmdir /sys/kernel/config/nvmet/ports/1
+ rmdir /sys/kernel/config/nvmet/subsystems/testnqn
+ rmdir 'config/nullb/nullb*'
rmdir: failed to remove 'config/nullb/nullb*': No such file or directory
+ umount /mnt/nvme0n1
umount: /mnt/nvme0n1: no mount point specified.
+ umount /mnt/backend
umount: /mnt/backend: not mounted.
+ modprobe -r nvme_loop
+ modprobe -r nvme_fabrics
+ modprobe -r nvmet
+ modprobe -r nvme
+ modprobe -r null_blk
+ tree /sys/kernel/config
/sys/kernel/config
2. Verbose log Block Device with buffered_io = 1 :-
---------------------------------------------------
0 directories, 0 files
++ NN=1
++ NQN=testnqn
++ let NR_DEVICES=NN+1
++ modprobe -r null_blk
++ modprobe -r nvme
++ modprobe null_blk nr_devices=0
++ modprobe nvme
++ modprobe nvme-fabrics
++ modprobe nvmet
++ modprobe nvme-loop
++ dmesg -c
++ sleep 2
++ tree /sys/kernel/config
/sys/kernel/config
├── nullb
│ └── features
└── nvmet
├── hosts
├── ports
└── subsystems
5 directories, 1 file
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn
++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1
++ echo -n /dev/nvme0n1
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path
/dev/nvme0n1
++ echo 1
++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io
1
++ echo 1
++ mkdir /sys/kernel/config/nvmet/ports/1/
++ echo -n loop
++ echo -n 1
++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/
++ sleep 1
++ echo transport=loop,nqn=testnqn
++ sleep 1
++ mount
++ column -t
++ grep nvme
++ '[' 1 ']'
+++ wc -l
+++ ls -l /dev/nvme1 /dev/nvme1n1
++ cnt=2
++ echo 2
2
++ '[' 2 -gt 1 ']'
++ break
++ dmesg -c
[55830.215441] nvme nvme0: 48/0/0 default/read/poll queues
[55832.213819] nvmet: adding nsid 1 to subsystem testnqn
[55833.223744] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:2ba90169-7cb8-47b3-b20f-9c2bbcb8b092.
[55833.223872] nvme nvme1: creating 48 I/O queues.
[55833.228124] nvme nvme1: new ctrl: "testnqn"
++ sleep 5
write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.27
Starting 1 process
Jobs: 1 (f=1)
write-and-verify: (groupid=0, jobs=1): err= 0: pid=13967: Tue Jan 18 13:29:06 2022
read: IOPS=180k, BW=703MiB/s (737MB/s)(506MiB/720msec)
slat (usec): min=2, max=118, avg= 4.20, stdev= 1.61
clat (nsec): min=1113, max=224236, avg=17031.32, stdev=5565.81
lat (usec): min=4, max=342, avg=21.31, stdev= 6.80
clat percentiles (usec):
| 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 14],
| 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15],
| 70.00th=[ 15], 80.00th=[ 22], 90.00th=[ 27], 95.00th=[ 28],
| 99.00th=[ 35], 99.50th=[ 38], 99.90th=[ 45], 99.95th=[ 50],
| 99.99th=[ 113]
write: IOPS=125k, BW=489MiB/s (513MB/s)(800MiB/1635msec); 0 zone resets
slat (usec): min=2, max=248, avg= 6.78, stdev= 2.59
clat (nsec): min=661, max=202274, avg=24724.98, stdev=8660.56
lat (usec): min=6, max=293, avg=31.59, stdev=10.27
clat percentiles (usec):
| 1.00th=[ 18], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 20],
| 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 21], 60.00th=[ 21],
| 70.00th=[ 25], 80.00th=[ 35], 90.00th=[ 38], 95.00th=[ 42],
| 99.00th=[ 51], 99.50th=[ 55], 99.90th=[ 65], 99.95th=[ 71],
| 99.99th=[ 121]
bw ( KiB/s): min=94024, max=525016, per=81.75%, avg=409600.00, stdev=210508.70, samples=4
iops : min=23506, max=131254, avg=102400.00, stdev=52627.18, samples=4
lat (nsec) : 750=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=57.13%, 50=42.10%, 100=0.76%
lat (usec) : 250=0.02%
cpu : usr=28.33%, sys=32.75%, ctx=319653, majf=0, minf=3055
IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=4
Run status group 0 (all jobs):
READ: bw=703MiB/s (737MB/s), 703MiB/s-703MiB/s (737MB/s-737MB/s), io=506MiB (531MB), run=720-720msec
WRITE: bw=489MiB/s (513MB/s), 489MiB/s-489MiB/s (513MB/s-513MB/s), io=800MiB (839MB), run=1635-1635msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=15.7GiB/s][w=4108k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=13970: Tue Jan 18 13:30:06 2022
write: IOPS=4140k, BW=15.8GiB/s (17.0GB/s)(948GiB/60002msec); 0 zone resets
slat (nsec): min=1143, max=23104k, avg=6968.02, stdev=4184.92
clat (nsec): min=731, max=23218k, avg=54413.34, stdev=20272.07
lat (usec): min=5, max=23227, avg=61.46, stdev=22.72
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 41], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 56],
| 70.00th=[ 58], 80.00th=[ 60], 90.00th=[ 75], 95.00th=[ 97],
| 99.00th=[ 135], 99.50th=[ 149], 99.90th=[ 188], 99.95th=[ 198],
| 99.99th=[ 231]
bw ( MiB/s): min=11988, max=19957, per=100.00%, avg=16186.94, stdev=76.54, samples=3808
iops : min=3069065, max=5108996, avg=4143856.81, stdev=19593.23, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=54.78%
lat (usec) : 100=41.05%, 250=4.16%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%
cpu : usr=10.96%, sys=35.89%, ctx=248415838, majf=0, minf=446
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,248414657,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.8GiB/s (17.0GB/s), 15.8GiB/s-15.8GiB/s (17.0GB/s-17.0GB/s), io=948GiB (1018GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=16.1GiB/s][w=4215k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=14054: Tue Jan 18 13:31:06 2022
write: IOPS=4232k, BW=16.1GiB/s (17.3GB/s)(969GiB/60002msec); 0 zone resets
slat (nsec): min=1312, max=5006.7k, avg=6792.48, stdev=3245.46
clat (nsec): min=721, max=23279k, avg=53238.95, stdev=14126.26
lat (usec): min=5, max=23285, avg=60.11, stdev=15.65
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 51], 60.00th=[ 57],
| 70.00th=[ 58], 80.00th=[ 60], 90.00th=[ 66], 95.00th=[ 79],
| 99.00th=[ 102], 99.50th=[ 126], 99.90th=[ 155], 99.95th=[ 172],
| 99.99th=[ 206]
bw ( MiB/s): min=12424, max=19988, per=100.00%, avg=16547.88, stdev=76.36, samples=3808
iops : min=3180626, max=5117092, avg=4236256.72, stdev=19547.40, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=49.68%, 100=49.24%
lat (usec) : 250=1.07%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 50=0.01%
cpu : usr=11.28%, sys=37.25%, ctx=253938868, majf=0, minf=462
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,253934166,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=16.1GiB/s (17.3GB/s), 16.1GiB/s-16.1GiB/s (17.3GB/s-17.3GB/s), io=969GiB (1040GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8
...
fio-3.27
Starting 32 processes
Jobs: 32 (f=32): [w(32)][100.0%][w=15.7GiB/s][w=4119k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=32): err= 0: pid=14113: Tue Jan 18 13:32:07 2022
write: IOPS=4125k, BW=15.7GiB/s (16.9GB/s)(944GiB/60002msec); 0 zone resets
slat (nsec): min=1282, max=5173.6k, avg=6989.07, stdev=3699.41
clat (nsec): min=741, max=5243.0k, avg=54604.96, stdev=18594.18
lat (usec): min=5, max=5251, avg=61.67, stdev=20.85
clat percentiles (usec):
| 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43],
| 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57],
| 70.00th=[ 58], 80.00th=[ 61], 90.00th=[ 72], 95.00th=[ 93],
| 99.00th=[ 133], 99.50th=[ 145], 99.90th=[ 186], 99.95th=[ 198],
| 99.99th=[ 258]
bw ( MiB/s): min=10106, max=20077, per=100.00%, avg=16129.59, stdev=91.59, samples=3808
iops : min=2587208, max=5139896, avg=4129175.92, stdev=23445.99, samples=3808
lat (nsec) : 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.41%, 100=44.26%
lat (usec) : 250=3.32%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%
cpu : usr=11.27%, sys=35.70%, ctx=247529856, majf=0, minf=488
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=0,247525302,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8
Run status group 0 (all jobs):
WRITE: bw=15.7GiB/s (16.9GB/s), 15.7GiB/s-15.7GiB/s (16.9GB/s-16.9GB/s), io=944GiB (1014GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.5GiB/s][r=4580k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=14172: Tue Jan 18 13:33:07 2022
read: IOPS=4731k, BW=18.0GiB/s (19.4GB/s)(1083GiB/60002msec)
slat (nsec): min=1001, max=69146k, avg=9264.71, stdev=16039.18
clat (nsec): min=481, max=83414k, avg=639474.44, stdev=334536.92
lat (usec): min=5, max=83416, avg=648.84, stdev=338.99
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 424], 20.00th=[ 510],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 553], 80.00th=[ 914], 90.00th=[ 1139], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1270], 99.90th=[ 1483], 99.95th=[ 4883],
| 99.99th=[11076]
bw ( MiB/s): min= 1806, max=26193, per=100.00%, avg=18499.19, stdev=75.70, samples=5712
iops : min=462371, max=6705466, avg=4735791.68, stdev=19380.33, samples=5712
lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=17.63%, 750=60.60%, 1000=8.11%
lat (msec) : 2=13.58%, 4=0.03%, 10=0.05%, 20=0.01%, 50=0.01%
lat (msec) : 100=0.01%
cpu : usr=8.84%, sys=34.58%, ctx=283862866, majf=0, minf=3785
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=283879844,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.0GiB/s (19.4GB/s), 18.0GiB/s-18.0GiB/s (19.4GB/s-19.4GB/s), io=1083GiB (1163GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.2GiB/s][r=4506k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=14451: Tue Jan 18 13:34:07 2022
read: IOPS=4790k, BW=18.3GiB/s (19.6GB/s)(1096GiB/60002msec)
slat (usec): min=2, max=9735, avg= 9.15, stdev= 4.82
clat (nsec): min=842, max=10334k, avg=631631.33, stdev=249341.17
lat (usec): min=5, max=10342, avg=640.88, stdev=253.17
clat percentiles (usec):
| 1.00th=[ 400], 5.00th=[ 408], 10.00th=[ 420], 20.00th=[ 510],
| 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 553], 80.00th=[ 906], 90.00th=[ 1156], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1287], 99.95th=[ 1319],
| 99.99th=[ 1434]
bw ( MiB/s): min=15109, max=25642, per=100.00%, avg=18732.45, stdev=71.99, samples=5712
iops : min=3868088, max=6564526, avg=4795506.90, stdev=18429.43, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01%
lat (usec) : 250=0.01%, 500=18.55%, 750=60.35%, 1000=7.37%
lat (msec) : 2=13.73%, 4=0.01%, 10=0.01%, 20=0.01%
cpu : usr=9.18%, sys=34.74%, ctx=287421571, majf=0, minf=3694
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=287412242,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.3GiB/s (19.6GB/s), 18.3GiB/s-18.3GiB/s (19.6GB/s-19.6GB/s), io=1096GiB (1177GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
...
fio-3.27
Starting 48 processes
Jobs: 48 (f=48): [r(48)][100.0%][r=17.4GiB/s][r=4562k IOPS][eta 00m:00s]
RANDREAD: (groupid=0, jobs=48): err= 0: pid=14522: Tue Jan 18 13:35:08 2022
read: IOPS=4925k, BW=18.8GiB/s (20.2GB/s)(1127GiB/60002msec)
slat (usec): min=3, max=22231, avg= 8.88, stdev= 4.65
clat (nsec): min=892, max=24497k, avg=614341.36, stdev=226405.16
lat (usec): min=6, max=24528, avg=623.32, stdev=229.88
clat percentiles (usec):
| 1.00th=[ 404], 5.00th=[ 416], 10.00th=[ 429], 20.00th=[ 519],
| 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545],
| 70.00th=[ 553], 80.00th=[ 562], 90.00th=[ 1123], 95.00th=[ 1188],
| 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1303], 99.95th=[ 1319],
| 99.99th=[ 1418]
bw ( MiB/s): min=15698, max=26142, per=100.00%, avg=19262.82, stdev=68.27, samples=5712
iops : min=4018694, max=6692394, avg=4931281.76, stdev=17477.36, samples=5712
lat (nsec) : 1000=0.01%
lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
lat (usec) : 100=0.01%, 250=0.01%, 500=14.73%, 750=68.64%, 1000=5.40%
lat (msec) : 2=11.22%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
cpu : usr=9.25%, sys=36.21%, ctx=295513468, majf=0, minf=3792
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=295503180,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=18.8GiB/s (20.2GB/s), 18.8GiB/s-18.8GiB/s (20.2GB/s-20.2GB/s), io=1127GiB (1210GB), run=60002-60002msec
Disk stats (read/write):
nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
root@dev nvme (nvme-5.17) #
--
2.29.0
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 1/3] nvmet: add helper to set ns->blkshift value
2022-01-19 7:20 [PATCH 0/3] nvmet: add buffered_io support for bdev-ns Chaitanya Kulkarni
@ 2022-01-19 7:20 ` Chaitanya Kulkarni
2022-01-23 9:18 ` Sagi Grimberg
2022-01-19 7:20 ` [PATCH 2/3] nvmet: add helper to open block device Chaitanya Kulkarni
` (2 subsequent siblings)
3 siblings, 1 reply; 11+ messages in thread
From: Chaitanya Kulkarni @ 2022-01-19 7:20 UTC (permalink / raw)
To: linux-nvme; +Cc: hch, kbusch, sagi, Chaitanya Kulkarni
From: Chaitanya Kulkarni <kch@nvidia.com>
In order to support the block device in the buffered I/O mode, we use
file backend. Calculating the blksize_shift value for the block device
backend is different from the file backend.
Move existing code for calculating the namespace's blksize_shift value
into the helper for file backend. The last patch in this series will add
the code for block device blksize_shift calculation in the same helper.
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/nvme/target/io-cmd-file.c | 21 ++++++++++++++-------
1 file changed, 14 insertions(+), 7 deletions(-)
diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c
index 6be6e59d273b..7aa4cdcb80ce 100644
--- a/drivers/nvme/target/io-cmd-file.c
+++ b/drivers/nvme/target/io-cmd-file.c
@@ -26,6 +26,16 @@ int nvmet_file_ns_revalidate(struct nvmet_ns *ns)
return ret;
}
+static int nvmet_file_ns_set_blkshift(struct nvmet_ns *ns)
+{
+ /*
+ * i_blkbits can be greater than the universally accepted upper bound,
+ * so make sure we export a sane namespace lba_shift.
+ */
+ ns->blksize_shift = min_t(u8, file_inode(ns->file)->i_blkbits, 12);
+ return 0;
+}
+
void nvmet_file_ns_disable(struct nvmet_ns *ns)
{
if (ns->file) {
@@ -57,17 +67,14 @@ int nvmet_file_ns_enable(struct nvmet_ns *ns)
return ret;
}
+ ret = nvmet_file_ns_set_blkshift(ns);
+ if (ret)
+ return ret;
+
ret = nvmet_file_ns_revalidate(ns);
if (ret)
goto err;
- /*
- * i_blkbits can be greater than the universally accepted upper bound,
- * so make sure we export a sane namespace lba_shift.
- */
- ns->blksize_shift = min_t(u8,
- file_inode(ns->file)->i_blkbits, 12);
-
ns->bvec_cache = kmem_cache_create("nvmet-bvec",
NVMET_MAX_MPOOL_BVEC * sizeof(struct bio_vec),
0, SLAB_HWCACHE_ALIGN, NULL);
--
2.29.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 2/3] nvmet: add helper to open block device
2022-01-19 7:20 [PATCH 0/3] nvmet: add buffered_io support for bdev-ns Chaitanya Kulkarni
2022-01-19 7:20 ` [PATCH 1/3] nvmet: add helper to set ns->blkshift value Chaitanya Kulkarni
@ 2022-01-19 7:20 ` Chaitanya Kulkarni
2022-01-19 7:20 ` [PATCH 3/3] nvmet: allow bdev in buffered_io mode Chaitanya Kulkarni
2022-01-26 16:27 ` [PATCH 0/3] nvmet: add buffered_io support for bdev-ns Christoph Hellwig
3 siblings, 0 replies; 11+ messages in thread
From: Chaitanya Kulkarni @ 2022-01-19 7:20 UTC (permalink / raw)
To: linux-nvme; +Cc: hch, kbusch, sagi, Chaitanya Kulkarni
From: Chaitanya Kulkarni <kch@nvidia.com>
In order to support block device backend in the buffered I/O mode
we keep the ns->bdev handle open so we can reavalidate the ns->size
in nvmet_file_ns_revalidate() using bdev_nr_bytes(). Also, instead of
duplicating the code for opening block device we move the existing code
into the nvme_bdev_open() helper and call from nvmet_bdev_ns_enable().
This newly added helper will be called in the next patch to open and set
the ns->size and ns->blksize_shift values when block device is
configured with buffere I/O mode from file backend code.
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/nvme/target/io-cmd-bdev.c | 14 ++++++++++++--
drivers/nvme/target/nvmet.h | 1 +
2 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index 70ca9dfc1771..58907304587e 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -72,9 +72,9 @@ static void nvmet_bdev_ns_enable_integrity(struct nvmet_ns *ns)
}
}
-int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
+int nvmet_bdev_ns_open(struct nvmet_ns *ns)
{
- int ret;
+ int ret = 0;
ns->bdev = blkdev_get_by_path(ns->device_path,
FMODE_READ | FMODE_WRITE, NULL);
@@ -89,6 +89,16 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
}
ns->size = bdev_nr_bytes(ns->bdev);
ns->blksize_shift = blksize_bits(bdev_logical_block_size(ns->bdev));
+ return ret;
+}
+
+int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
+{
+ int ret;
+
+ ret = nvmet_bdev_ns_open(ns);
+ if (ret)
+ return ret;
ns->pi_type = 0;
ns->metadata_size = 0;
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index af193423c10b..534fe2b32e99 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -533,6 +533,7 @@ extern struct rw_semaphore nvmet_ana_sem;
bool nvmet_host_allowed(struct nvmet_subsys *subsys, const char *hostnqn);
+int nvmet_bdev_ns_open(struct nvmet_ns *ns);
int nvmet_bdev_ns_enable(struct nvmet_ns *ns);
int nvmet_file_ns_enable(struct nvmet_ns *ns);
void nvmet_bdev_ns_disable(struct nvmet_ns *ns);
--
2.29.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* [PATCH 3/3] nvmet: allow bdev in buffered_io mode
2022-01-19 7:20 [PATCH 0/3] nvmet: add buffered_io support for bdev-ns Chaitanya Kulkarni
2022-01-19 7:20 ` [PATCH 1/3] nvmet: add helper to set ns->blkshift value Chaitanya Kulkarni
2022-01-19 7:20 ` [PATCH 2/3] nvmet: add helper to open block device Chaitanya Kulkarni
@ 2022-01-19 7:20 ` Chaitanya Kulkarni
2022-01-26 16:27 ` [PATCH 0/3] nvmet: add buffered_io support for bdev-ns Christoph Hellwig
3 siblings, 0 replies; 11+ messages in thread
From: Chaitanya Kulkarni @ 2022-01-19 7:20 UTC (permalink / raw)
To: linux-nvme; +Cc: hch, kbusch, sagi, Chaitanya Kulkarni
From: Chaitanya Kulkarni <kch@nvidia.com>
Allow block device to be configured in the buffered I/O mode by using
the file backend. In this way now we can use cache for the block
device namespace which shows significant performance improvement.
We update the block device ns enable function and return early when
buffered_io flag is set, when revalidating the file backed namespace
open block device handle and consider block device size, use
previously exported block device API to set the namespace size and
blksize_shift value, and call nvmet_bdev_ns_disable() when disabling
file backed namespace.
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/nvme/target/io-cmd-bdev.c | 8 ++++++++
drivers/nvme/target/io-cmd-file.c | 17 ++++++++++++++++-
2 files changed, 24 insertions(+), 1 deletion(-)
diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index 58907304587e..5e5eb49a0255 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -96,6 +96,14 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
{
int ret;
+ /*
+ * When buffered_io namespace attribute is enabled that means user want
+ * this block device to be used as a file, so block device can take
+ * an advantage of cache.
+ */
+ if (ns->buffered_io)
+ return -ENOTBLK;
+
ret = nvmet_bdev_ns_open(ns);
if (ret)
return ret;
diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c
index 7aa4cdcb80ce..067c45b23bba 100644
--- a/drivers/nvme/target/io-cmd-file.c
+++ b/drivers/nvme/target/io-cmd-file.c
@@ -22,12 +22,25 @@ int nvmet_file_ns_revalidate(struct nvmet_ns *ns)
ret = vfs_getattr(&ns->file->f_path, &stat, STATX_SIZE,
AT_STATX_FORCE_SYNC);
if (!ret)
- ns->size = stat.size;
+ ns->size = ns->bdev ? bdev_nr_bytes(ns->bdev) : stat.size;
+
return ret;
}
static int nvmet_file_ns_set_blkshift(struct nvmet_ns *ns)
{
+ struct kstat stat;
+ int ret;
+
+ ret = vfs_getattr(&ns->file->f_path, &stat, STATX_MODE,
+ AT_STATX_FORCE_SYNC);
+ if (ret)
+ return ret;
+
+ /* nvmet_bdev_ns_open() sets size & blksize_shift values for ns->bdev */
+ if (S_ISBLK(stat.mode) && ns->buffered_io)
+ return nvmet_bdev_ns_open(ns);
+
/*
* i_blkbits can be greater than the universally accepted upper bound,
* so make sure we export a sane namespace lba_shift.
@@ -45,6 +58,8 @@ void nvmet_file_ns_disable(struct nvmet_ns *ns)
ns->bvec_pool = NULL;
kmem_cache_destroy(ns->bvec_cache);
ns->bvec_cache = NULL;
+ /* if block device is configured with buffered_io */
+ nvmet_bdev_ns_disable(ns);
fput(ns->file);
ns->file = NULL;
}
--
2.29.0
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 1/3] nvmet: add helper to set ns->blkshift value
2022-01-19 7:20 ` [PATCH 1/3] nvmet: add helper to set ns->blkshift value Chaitanya Kulkarni
@ 2022-01-23 9:18 ` Sagi Grimberg
2022-01-23 23:37 ` Chaitanya Kulkarni
0 siblings, 1 reply; 11+ messages in thread
From: Sagi Grimberg @ 2022-01-23 9:18 UTC (permalink / raw)
To: Chaitanya Kulkarni, linux-nvme; +Cc: hch, kbusch, Chaitanya Kulkarni
> void nvmet_file_ns_disable(struct nvmet_ns *ns)
> {
> if (ns->file) {
> @@ -57,17 +67,14 @@ int nvmet_file_ns_enable(struct nvmet_ns *ns)
> return ret;
> }
>
> + ret = nvmet_file_ns_set_blkshift(ns);
> + if (ret)
> + return ret;
> +
> ret = nvmet_file_ns_revalidate(ns);
> if (ret)
> goto err;
>
> - /*
> - * i_blkbits can be greater than the universally accepted upper bound,
> - * so make sure we export a sane namespace lba_shift.
> - */
> - ns->blksize_shift = min_t(u8,
> - file_inode(ns->file)->i_blkbits, 12);
> -
Any particular reason for the change in location?
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/3] nvmet: add helper to set ns->blkshift value
2022-01-23 9:18 ` Sagi Grimberg
@ 2022-01-23 23:37 ` Chaitanya Kulkarni
0 siblings, 0 replies; 11+ messages in thread
From: Chaitanya Kulkarni @ 2022-01-23 23:37 UTC (permalink / raw)
To: Sagi Grimberg; +Cc: hch, kbusch, Chaitanya Kulkarni, linux-nvme
Sagi,
On 1/23/22 1:18 AM, Sagi Grimberg wrote:
> External email: Use caution opening links or attachments
>
>
>> void nvmet_file_ns_disable(struct nvmet_ns *ns)
>> {
>> if (ns->file) {
>> @@ -57,17 +67,14 @@ int nvmet_file_ns_enable(struct nvmet_ns *ns)
>> return ret;
>> }
>>
>> + ret = nvmet_file_ns_set_blkshift(ns);
>> + if (ret)
>> + return ret;
>> +
>> ret = nvmet_file_ns_revalidate(ns);
>> if (ret)
>> goto err;
>>
>> - /*
>> - * i_blkbits can be greater than the universally accepted upper
>> bound,
>> - * so make sure we export a sane namespace lba_shift.
>> - */
>> - ns->blksize_shift = min_t(u8,
>> - file_inode(ns->file)->i_blkbits, 12);
>> -
>
> Any particular reason for the change in location?
So third patch in this series sets ns->bdev if file type is block
device in nvmet_file_ns_set_blkshift(). Based on ns->bdev is NULL
or not NULL that is used to determine how we calculate ns->size
in the nvmet_file_ns_revalidate(), see last patch.
Hence move blksize_shift setting before nvmet_file_ns_revalidate().
-ck
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 0/3] nvmet: add buffered_io support for bdev-ns
2022-01-19 7:20 [PATCH 0/3] nvmet: add buffered_io support for bdev-ns Chaitanya Kulkarni
` (2 preceding siblings ...)
2022-01-19 7:20 ` [PATCH 3/3] nvmet: allow bdev in buffered_io mode Chaitanya Kulkarni
@ 2022-01-26 16:27 ` Christoph Hellwig
2022-01-27 7:46 ` Chaitanya Kulkarni
3 siblings, 1 reply; 11+ messages in thread
From: Christoph Hellwig @ 2022-01-26 16:27 UTC (permalink / raw)
To: Chaitanya Kulkarni; +Cc: linux-nvme, hch, kbusch, sagi, Chaitanya Kulkarni
Shouldn't something like the patch below be all we need to use
buffered I/O on a block device?
diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
index 70ca9dfc1771a..7d9da2f77b83a 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -76,6 +76,10 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
{
int ret;
+ /* force fallback to the file backend for buffered I/O mode */
+ if (ns->buffered_io)
+ return -ENOTBLK;
+
ns->bdev = blkdev_get_by_path(ns->device_path,
FMODE_READ | FMODE_WRITE, NULL);
if (IS_ERR(ns->bdev)) {
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 0/3] nvmet: add buffered_io support for bdev-ns
2022-01-26 16:27 ` [PATCH 0/3] nvmet: add buffered_io support for bdev-ns Christoph Hellwig
@ 2022-01-27 7:46 ` Chaitanya Kulkarni
2022-01-27 8:04 ` Chaitanya Kulkarni
2022-01-27 10:07 ` Christoph Hellwig
0 siblings, 2 replies; 11+ messages in thread
From: Chaitanya Kulkarni @ 2022-01-27 7:46 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: linux-nvme, kbusch, sagi, Chaitanya Kulkarni
Christoph,
On 1/26/22 8:27 AM, Christoph Hellwig wrote:
> External email: Use caution opening links or attachments
>
>
> Shouldn't something like the patch below be all we need to use
> buffered I/O on a block device?
>
> diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
> index 70ca9dfc1771a..7d9da2f77b83a 100644
> --- a/drivers/nvme/target/io-cmd-bdev.c
> +++ b/drivers/nvme/target/io-cmd-bdev.c
> @@ -76,6 +76,10 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
> {
> int ret;
>
> + /* force fallback to the file backend for buffered I/O mode */
> + if (ns->buffered_io)
> + return -ENOTBLK;
> +
> ns->bdev = blkdev_get_by_path(ns->device_path,
> FMODE_READ | FMODE_WRITE, NULL);
> if (IS_ERR(ns->bdev)) {
>
This is exactly how I stared this patch-series, but above doesn't
work since it errors out in the nvme_identify_ns() due to ncap value
set to 0 (set from ns->size in id-ns admin on target side) see [1],
since ns->size value we get is 0, that is set using call to
vfs_getattr(STATX_SIZE) in the nvmet_file_ns_revalidate()
for file backend.
[1]
[ 579.444478] nvme nvme0: 48/0/0 default/read/poll queues
[ 581.430805] nvmet: adding nsid 1 to subsystem testnqn
[ 582.439296] nvmet: creating nvm controller 1 for subsystem testnqn
for NQN
nqn.2014-08.org.nvmexpress:uuid:23f0992c-7ef7-4fbd-80f7-209cda6d872a.
[ 582.439429] nvme nvme1: creating 48 I/O queues.
[ 582.444036] nvme nvme1: new ctrl: "testnqn"
**[ 582.444095] nvmet: nvmet_execute_identify_ns 520 req->ns->size 0 >>
req->ns->blksize_shift 12 = 0**
**[ 582.444103] ERROR nvme_identify_ns 1473**
++ sleep 5
Debug diff :-
root@dev nvme (nvme-5.17) # git diff
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index dd18861f77c0..5a33b3f73b3b 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1469,8 +1469,10 @@ static int nvme_identify_ns(struct nvme_ctrl
*ctrl, unsigned nsid,
}
error = NVME_SC_INVALID_NS | NVME_SC_DNR;
- if ((*id)->ncap == 0) /* namespace not allocated or attached */
+ if ((*id)->ncap == 0) { /* namespace not allocated or attached */
+ pr_info(" ERROR %s %d\n", __func__, __LINE__);
goto out_free_id;
+ }
if (ctrl->vs >= NVME_VS(1, 1, 0) &&
!memchr_inv(ids->eui64, 0, sizeof(ids->eui64)))
diff --git a/drivers/nvme/target/admin-cmd.c
b/drivers/nvme/target/admin-cmd.c
index 6fb24746de06..94f158aa7bfa 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -517,6 +517,9 @@ static void nvmet_execute_identify_ns(struct
nvmet_req *req)
* nuse = ncap = nsze isn't always true, but we have no way to find
* that out from the underlying device.
*/
+ pr_info("%s %d req->ns->size %u >> req->ns->blksize_shift %u =
%u\n",
+ __func__, __LINE__, req->ns->size,
req->ns->blksize_shift,
+ req->ns->size >> req->ns->blksize_shift);
id->ncap = id->nsze =
cpu_to_le64(req->ns->size >> req->ns->blksize_shift);
switch (req->port->ana_state[req->ns->anagrpid]) {
diff --git a/drivers/nvme/target/io-cmd-bdev.c
b/drivers/nvme/target/io-cmd-bdev.c
index 70ca9dfc1771..7d9da2f77b83 100644
--- a/drivers/nvme/target/io-cmd-bdev.c
+++ b/drivers/nvme/target/io-cmd-bdev.c
@@ -76,6 +76,10 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
{
int ret;
+ /* force fallback to the file backend for buffered I/O mode */
+ if (ns->buffered_io)
+ return -ENOTBLK;
+
ns->bdev = blkdev_get_by_path(ns->device_path,
FMODE_READ | FMODE_WRITE, NULL);
if (IS_ERR(ns->bdev)) {
root@dev nvme (nvme-5.17) #
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 0/3] nvmet: add buffered_io support for bdev-ns
2022-01-27 7:46 ` Chaitanya Kulkarni
@ 2022-01-27 8:04 ` Chaitanya Kulkarni
2022-01-27 10:07 ` Christoph Hellwig
2022-01-27 10:07 ` Christoph Hellwig
1 sibling, 1 reply; 11+ messages in thread
From: Chaitanya Kulkarni @ 2022-01-27 8:04 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: linux-nvme, kbusch, sagi
On 1/26/22 11:46 PM, Chaitanya Kulkarni wrote:
> Christoph,
>
> On 1/26/22 8:27 AM, Christoph Hellwig wrote:
>> External email: Use caution opening links or attachments
>>
>>
>> Shouldn't something like the patch below be all we need to use
>> buffered I/O on a block device?
>>
>> diff --git a/drivers/nvme/target/io-cmd-bdev.c b/drivers/nvme/target/io-cmd-bdev.c
>> index 70ca9dfc1771a..7d9da2f77b83a 100644
>> --- a/drivers/nvme/target/io-cmd-bdev.c
>> +++ b/drivers/nvme/target/io-cmd-bdev.c
>> @@ -76,6 +76,10 @@ int nvmet_bdev_ns_enable(struct nvmet_ns *ns)
>> {
>> int ret;
>>
>> + /* force fallback to the file backend for buffered I/O mode */
>> + if (ns->buffered_io)
>> + return -ENOTBLK;
>> +
>> ns->bdev = blkdev_get_by_path(ns->device_path,
>> FMODE_READ | FMODE_WRITE, NULL);
>> if (IS_ERR(ns->bdev)) {
>>
>
> This is exactly how I stared this patch-series, but above doesn't
> work since it errors out in the nvme_identify_ns() due to ncap value
> set to 0 (set from ns->size in id-ns admin on target side) see [1],
> since ns->size value we get is 0, that is set using call to
> vfs_getattr(STATX_SIZE) in the nvmet_file_ns_revalidate()
> for file backend.
>
>
Also, I believe we need to make sure ns->device_path is block device
type with vfs_getattr(STATX_MODE).
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 0/3] nvmet: add buffered_io support for bdev-ns
2022-01-27 7:46 ` Chaitanya Kulkarni
2022-01-27 8:04 ` Chaitanya Kulkarni
@ 2022-01-27 10:07 ` Christoph Hellwig
1 sibling, 0 replies; 11+ messages in thread
From: Christoph Hellwig @ 2022-01-27 10:07 UTC (permalink / raw)
To: Chaitanya Kulkarni; +Cc: Christoph Hellwig, linux-nvme, kbusch, sagi
On Thu, Jan 27, 2022 at 07:46:22AM +0000, Chaitanya Kulkarni wrote:
> This is exactly how I stared this patch-series, but above doesn't
> work since it errors out in the nvme_identify_ns() due to ncap value
> set to 0 (set from ns->size in id-ns admin on target side) see [1],
> since ns->size value we get is 0, that is set using call to
> vfs_getattr(STATX_SIZE) in the nvmet_file_ns_revalidate()
> for file backend.
Something like this should solve that issue:
diff --git a/drivers/nvme/target/io-cmd-file.c b/drivers/nvme/target/io-cmd-file.c
index 6be6e59d273bb..fb7f4dd72daf6 100644
--- a/drivers/nvme/target/io-cmd-file.c
+++ b/drivers/nvme/target/io-cmd-file.c
@@ -16,14 +16,8 @@
int nvmet_file_ns_revalidate(struct nvmet_ns *ns)
{
- struct kstat stat;
- int ret;
-
- ret = vfs_getattr(&ns->file->f_path, &stat, STATX_SIZE,
- AT_STATX_FORCE_SYNC);
- if (!ret)
- ns->size = stat.size;
- return ret;
+ ns->size = i_size_read(ns->file->f_mapping->host);
+ return 0;
}
void nvmet_file_ns_disable(struct nvmet_ns *ns)
^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 0/3] nvmet: add buffered_io support for bdev-ns
2022-01-27 8:04 ` Chaitanya Kulkarni
@ 2022-01-27 10:07 ` Christoph Hellwig
0 siblings, 0 replies; 11+ messages in thread
From: Christoph Hellwig @ 2022-01-27 10:07 UTC (permalink / raw)
To: Chaitanya Kulkarni; +Cc: Christoph Hellwig, linux-nvme, kbusch, sagi
On Thu, Jan 27, 2022 at 08:04:45AM +0000, Chaitanya Kulkarni wrote:
> Also, I believe we need to make sure ns->device_path is block device
> type with vfs_getattr(STATX_MODE).
I don't think we care about the type of filp_open works.
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2022-01-27 10:09 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-01-19 7:20 [PATCH 0/3] nvmet: add buffered_io support for bdev-ns Chaitanya Kulkarni
2022-01-19 7:20 ` [PATCH 1/3] nvmet: add helper to set ns->blkshift value Chaitanya Kulkarni
2022-01-23 9:18 ` Sagi Grimberg
2022-01-23 23:37 ` Chaitanya Kulkarni
2022-01-19 7:20 ` [PATCH 2/3] nvmet: add helper to open block device Chaitanya Kulkarni
2022-01-19 7:20 ` [PATCH 3/3] nvmet: allow bdev in buffered_io mode Chaitanya Kulkarni
2022-01-26 16:27 ` [PATCH 0/3] nvmet: add buffered_io support for bdev-ns Christoph Hellwig
2022-01-27 7:46 ` Chaitanya Kulkarni
2022-01-27 8:04 ` Chaitanya Kulkarni
2022-01-27 10:07 ` Christoph Hellwig
2022-01-27 10:07 ` Christoph Hellwig
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.