Here is some more information regarding the problem I am facing.

I have configured the target for 2 subsystems with NVMe devices. Here are the errors I am getting from starting the target application.

DPDK EAL parameters: nvmf -c 0x1 --file-prefix=spdk_pid6390 ]

EAL: Detected 32 lcore(s)

EAL: No free hugepages reported in hugepages-1048576kB

EAL: Probing VFIO support...

Total cores available: 1

Occupied cpu socket mask is 0x1

reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0

copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled

bdev.c: 522:spdk_bdev_initialize: *ERROR*: create rbuf small pool failed

subsystem.c: 122:spdk_subsystem_init_next: *ERROR*: Init subsystem bdev failed

My configuration file in the target

[Subsystem1]

  NQN nqn.2017-06.io.spdk-MPcnode1

  Core 0

  Mode Direct

  #SN  SPDK000000000000015

  Listen RDMA 10.0.0.1:11345

  AllowAnyHost Yes

NVMe 0000:84:00.0

Please let me know if you have any suggestions how to fix it.

Thanks for your ideas,

Sreeni

From: SPDK [mailto:spdk-bounces@lists.01.org] On Behalf Of Sreeni (Sreenivasa) Busam (Stellus)
Sent: Monday, December 11, 2017 10:13 AM
To: Storage Performance Development Kit <spdk@lists.01.org>
Subject: [SPDK] Request for clarification of NVMeoF test configuration and setup in initiator and target.

Hi Ben, Lee, John and Teng,

I am looking at the set of instructions from one of the email discussion regarding setting up a NVMeoF setup for running SPDK test. I understood some of the steps, have are few more questions, for which I need some information. Please take look and reply when you get a chance. I have attached the steps from the email for your reference.

1. NQN nqn.2016-06.io.spdk:cnode1

How was the nqn configured? Is it just node name of the target or what command was used to configure?

What does cnode1 mean? How is it used? I am trying to understand how it is relevant to initiator.

2. Host nqn.2016-06.io.spdk:init

It looks like nqn and host name the same. Please confirm.

3. “3)      The nvme-cli tool is capable of configuring the Linux kernel’s built-in NVMe-oF initiator to connect to a target. You simply type something like “nvme connect –t rdma –a <ip> -s <port> -n <nqn>”, where the values match the ones you put into the SPDK NVMe-oF target’s configuration file. The initiator will then connect and a block device will appear on the initiator system at /dev/nvmeX.”

4. What application is used to test the device /dev/nvmeX which appears on the initiator when the configuration is successful?

Can I use the fio to test the NVMeoF setup as per this configuration? Please tell any specific changes I need to make to fio configuration file.

5. In the similar way to local SSD NVME API’s spdk_nvme_ns_cmd_read() and spdk_nvme_ns_write() used in hello_world program, can you please explain what are the API’s used in the target application to send the commands to the underlying device?

Please give some pointers how the NVME commands are sent from initiator to target using the RDMA protocol.

Any help would be greatly appreciated?

Thanks a lot for your help,

Sreeni