Re: [SPDK] Request for clarification of NVMeoF test configuration and setup in initiator and target.

Re: [SPDK] Request for clarification of NVMeoF test configuration and setup in initiator and target.

[Sreeni Busam]

[-- Attachment #1: Type: text/plain, Size: 3196 bytes --]

Here is some more information regarding the problem I am facing.
I have configured the target for 2 subsystems with NVMe devices. Here are the errors I am getting from starting the target application.
DPDK EAL parameters: nvmf -c 0x1 --file-prefix=spdk_pid6390 ]
EAL: Detected 32 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
Total cores available: 1
Occupied cpu socket mask is 0x1
reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled
bdev.c: 522:spdk_bdev_initialize: *ERROR*: create rbuf small pool failed
subsystem.c: 122:spdk_subsystem_init_next: *ERROR*: Init subsystem bdev failed
My configuration file in the target
[Subsystem1]
  NQN nqn.2017-06.io.spdk-MPcnode1
  Core 0
  Mode Direct
  #SN  SPDK000000000000015
  Listen RDMA 10.0.0.1:11345
  AllowAnyHost Yes
NVMe 0000:84:00.0
Please let me know if you have any suggestions how to fix it.

Thanks for your ideas,
Sreeni

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Monday, December 11, 2017 10:13 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] Request for clarification of NVMeoF test configuration and setup in initiator and target.

Hi Ben, Lee, John and Teng,

I am looking at the set of instructions from one of the email discussion regarding setting up a NVMeoF setup for running SPDK test. I understood some of the steps, have are few more questions, for which I need some information. Please take look and reply when you get a chance. I have attached the steps from the email for your reference.
1. NQN nqn.2016-06.io.spdk:cnode1
How was the nqn configured? Is it just node name of the target or what command was used to configure?
What does cnode1 mean? How is it used? I am trying to understand how it is relevant to initiator.
2. Host nqn.2016-06.io.spdk:init
It looks like nqn and host name the same. Please confirm.
3. "3)      The nvme-cli tool is capable of configuring the Linux kernel's built-in NVMe-oF initiator to connect to a target. You simply type something like "nvme connect -t rdma -a <ip> -s <port> -n <nqn>", where the values match the ones you put into the SPDK NVMe-oF target's configuration file. The initiator will then connect and a block device will appear on the initiator system at /dev/nvmeX."
4. What application is used to test the device /dev/nvmeX which appears on the initiator when the configuration is successful?
Can I use the fio to test the NVMeoF setup as per this configuration? Please tell any specific changes I need to make to fio configuration file.
5. In the similar way to local SSD NVME API's spdk_nvme_ns_cmd_read() and spdk_nvme_ns_write() used in hello_world program, can you please explain what are the API's used in the target application to send the commands to the underlying device?
Please give some pointers how the NVME commands are sent from initiator to target using the RDMA protocol.
Any help would be greatly appreciated?

Thanks a lot for your help,
Sreeni

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7964 bytes --]

[SPDK] Request for clarification of NVMeoF test configuration and setup in initiator and target.

[Sreeni Busam]

[-- Attachment #1: Type: text/plain, Size: 1810 bytes --]

Hi Ben, Lee, John and Teng,

I am looking at the set of instructions from one of the email discussion regarding setting up a NVMeoF setup for running SPDK test. I understood some of the steps, have are few more questions, for which I need some information. Please take look and reply when you get a chance. I have attached the steps from the email for your reference.
1. NQN nqn.2016-06.io.spdk:cnode1
How was the nqn configured? Is it just node name of the target or what command was used to configure?
What does cnode1 mean? How is it used? I am trying to understand how it is relevant to initiator.
2. Host nqn.2016-06.io.spdk:init
It looks like nqn and host name the same. Please confirm.
3. "3)      The nvme-cli tool is capable of configuring the Linux kernel's built-in NVMe-oF initiator to connect to a target. You simply type something like "nvme connect -t rdma -a <ip> -s <port> -n <nqn>", where the values match the ones you put into the SPDK NVMe-oF target's configuration file. The initiator will then connect and a block device will appear on the initiator system at /dev/nvmeX."
4. What application is used to test the device /dev/nvmeX which appears on the initiator when the configuration is successful?
Can I use the fio to test the NVMeoF setup as per this configuration? Please tell any specific changes I need to make to fio configuration file.
5. In the similar way to local SSD NVME API's spdk_nvme_ns_cmd_read() and spdk_nvme_ns_write() used in hello_world program, can you please explain what are the API's used in the target application to send the commands to the underlying device?
Please give some pointers how the NVME commands are sent from initiator to target using the RDMA protocol.
Any help would be greatly appreciated?

Thanks a lot for your help,
Sreeni

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 4245 bytes --]

[-- Attachment #3: Steps_spdk_nvmeof_test_setup_v1.txt --]
[-- Type: text/plain, Size: 4265 bytes --]

Hi Lei, Ben and John,

Thanks so much for your instructions! I heard that
intel is going to release its own SPDK initiator, may
I know the approximate time for this release?

Thanks,
Teng

From: SPDK [mailto:spdk-bounces at lists.01.org] On Behalf Of Kariuki, John K
Sent: Wednesday, November 30, 2016 2:21 PM
To: Storage Performance Development Kit <spdk at lists.01.org>
Subject: Re: [SPDK] how to use spdk's nvme over fabrics

Teng
Here are some notes that I took on my system when starting the nvmf target. I put steps 1 – 3 and 5 in a script that I run whenever my system reboots.
1.      Load ib_rdma_modules
sudo modprobe ib_cm
sudo modprobe ib_core
sudo modprobe ib_ucm
sudo modprobe ib_umad
sudo modprobe ib_uverbs
sudo modprobe iw_cm
sudo modprobe rdma_cm
sudo modprobe rdma_ucm
2.      Detect Mellanox NICs
sudo modprobe mlx5_core
sudo modprobe mlx5_ib
3.      Assign NICs ip addresses
a.      sudo ifconfig eth1 192.168.100.8 netmask 255.255.255.0 up
b.      sudo ifconfig eth2 192.168.100.9 netmask 255.255.255.0 up
4.      Start the nvmf target
cd ~/spdk/app/nvmf_tgt
sudo ./nvmf_tgt -c nvmf.conf.in &

My configuration file has the following Subsystems

[Subsystem1]
  NQN nqn.2016-06.io.spdk:cnode1
  Core 0
  Mode Direct
  Listen RDMA 192.168.100.8:4420
  Host nqn.2016-06.io.spdk:init
  NVMe 0000:81:00.0

# Multiple subsystems are allowed.
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Core 0
  Mode Direct
  Listen RDMA 192.168.100.9:4420
  Host nqn.2016-06.io.spdk:init
 NVMe 0000:86:00.0


5.      On the client load nvme-rdma kernel module
sudo modprobe -v nvme-rdma
6.      Use the cliet to discover the nvme target
sudo nvme discover -t rdma -a 192.168.100.8 -s 4420


From: SPDK [mailto:spdk-bounces at lists.01.org] On Behalf Of Walker, Benjamin
Sent: Wednesday, November 30, 2016 9:57 AM
To: Storage Performance Development Kit <spdk at lists.01.org<mailto:spdk at lists.01.org>>
Subject: Re: [SPDK] how to use spdk's nvme over fabrics

The guide linked below is our best attempt at documentation so far. I do really need to write more. The general outline is the following, assuming you have two systems (initiator and target) connected via RDMA NICs:


1)     On the target system, compile SPDK. Follow the guide linked below to start the SPDK NVMe-oF target application. A fully documented example configuration file is in <spdk repo>/etc/nvmf/nvmf.conf.in – use that as the starting point.

2)     On the initiator system, make sure you are running Linux kernel 4.8 and have nvme-cli installed. This is outside of SPDK, so we don’t provide instructions on how to do this. By far, the easiest way is to just install Fedora 24/25 on the initiator. It has a suitable kernel by default and the nvme-cli tool is packaged, so you have zero additional configuration to do.

3)     The nvme-cli tool is capable of configuring the Linux kernel’s built-in NVMe-oF initiator to connect to a target. You simply type something like “nvme connect –t rdma –a <ip> -s <port> -n <nqn>”, where the values match the ones you put into the SPDK NVMe-oF target’s configuration file. The initiator will then connect and a block device will appear on the initiator system at /dev/nvmeX.

Only step #1 is SPDK-specific. Step #2 and #3 are the same if you are using the Linux kernel’s NVMe-oF target – that’s why we don’t really document those today.

From: SPDK [mailto:spdk-bounces at lists.01.org] On Behalf Of Lei Zhu
Sent: Tuesday, November 29, 2016 7:54 PM
To: Storage Performance Development Kit <spdk at lists.01.org<mailto:spdk at lists.01.org>>
Subject: [SPDK] ??: how to use spdk's nvme over fabrics

Hi,

You can follow this guide:
NVMe over Fabrics Target Getting Started Guide
http://www.spdk.io/spdk/doc/nvmf_getting_started.html

???: SPDK [mailto:spdk-bounces at lists.01.org] ?? Wang, Teng
????: 2016?11?30? 6:45
???: spdk at lists.01.org<mailto:spdk at lists.01.org>
??: [SPDK] how to use spdk's nvme over fabrics

Hi all,

Is there any detailed instruction or sample code for using SPDK's NVMe over fabrics?
How to set up the experiment? I would really appreciate if anyone give me some
Instructions or materials on this.

Thanks,
Teng