Re: [SPDK] spdk user mode initiator

* Re: [SPDK] spdk user mode initiator
@ 2017-09-25 16:08 Cao, Gang
  0 siblings, 0 replies; 8+ messages in thread
From: Cao, Gang @ 2017-09-25 16:08 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 5575 bytes --]

Here nvme-cli (with SPDK support) and SPDK NVMe-oF initiator will both run at the user mode state. We have previously verified that SPDK NVMe-oF initiator can run on kernel 3.10 with some required steps. And use the SPDK perf tool to connect the remote SPDK NVMe-oF target and do the IOs.

Below are the major steps at that time we verified on the OS 3.10 (and for 3.18, please check for the similar steps):

Please keep us updated whether this works for you. Thanks. Gang

Steps to run SPDK nvmf target/host using CentOS7.2 and 3.10 kernel

The 3.10 kernel can’t run SPDK nvmf target/host since the lack of dependent driver and library.
So we need to install Mellanox official driver to get the latest driver and library.
Steps to install latest Mellanox driver
The web site to download the Mellanox official driver:
http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
Choose the version which support RHEL/CentOS 7.2. The latest version may not support RHEL/CentOS 7.2.
Download the ISO file since it contains a script which can do installation job.

Mount the ISO file to a directory.
Get into the directory. Run mlnxofedinstall scripts. Follow the steps to install it.
To make the driver work, it is better to reboot the machine.

Run nvmf target in target machine
Clone latest spdk codes from github into the target machines.
Run “git submodule update --init” command to get the DPDK codes.
Run following command in SPDK directory.
[localhost]./configure --with-rdma
[localshot]make

Run SPDK nvmf host in host machine  <<<<<< Here you can use the SPDK supported nvme-cli to connect

Clone latest spdk codes from github into the target machines.
Run “git submodule update --init” command to get the DPDK codes.
Run following command in SPDK directory.
[localhost]./configure --with-rdma
[localshot]make
Assign IP to the Mellanox NIC port. The IP should can ping the SPDK nvmf target’s IP.
Run the following command:
[localhost]NRHUGE=8192 /path/to/spdk/scripts/setup.sh

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Kirubakaran Kaliannan
Sent: Sunday, September 24, 2017 10:29 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] spdk user mode initiator

Hi Gang,

I am on 3.18 kernel on initiator. I assume, I cannot use the nvme-cli command on the initiator with 3.18 kernel. Right  (as this needs nvme_fabrics driver !! )?
or am I missing some option in the command to expose the nvme devices through spdk user-mode with out any support from the nvme kernel divers ?

Thanks
-kiru

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Cao, Gang
Sent: Friday, September 22, 2017 9:16 PM
To: Storage Performance Development Kit
Subject: Re: [SPDK] spdk user mode initiator

One thing is that you can try the nvme cli with SPDK support. Reference here http://www.spdk.io/doc/nvme-cli.html

You can also use nvme connect command which through spdk user mode initiator to connect the spdk nvme-of target and expose the /dev/nvmeX and /dev/nvmeXn1 devices locally. Once these devices are listed, can run the application on it.

If you have some issue when using SPDK supported nvme cli, please let us know.

Thanks,
Gang

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Kirubakaran Kaliannan
Sent: Friday, September 22, 2017 2:19 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] spdk user mode initiator

Just to cross check,  we need to build a plugin (to use the spdk SDK) for the application we have and to compile it with the spdk to use it as the initiator. Right ?

As I understand there is no traditional way of exposing the device for the application to consume. Please confirm.

Thanks,
-kiru

From: Kirubakaran Kaliannan [mailto:kirubak(a)zadarastorage.com<mailto:kirubak(a)zadarastorage.com>]
Sent: Friday, September 22, 2017 2:43 PM
To: 'Storage Performance Development Kit'
Subject: RE: [SPDK] spdk user mode initiator

Thanks Liu.

Now i can run perf test using
./perf -q 1 -s 1048576 -w read -M 100 -t 10 -r 'trtype:rdma adrfam:IPv4 traddr:10.3.7.2 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'

How can I import this as a block deivce (/dev/sd? Or /dev/nvme0?? Or some other device format), so I can use this for the application locally ?

Thanks,
-kiru

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Liu, Changpeng
Sent: Friday, September 22, 2017 10:42 AM
To: Storage Performance Development Kit
Subject: Re: [SPDK] spdk user mode initiator

Hi Kiru,

Yes, actually SPDK nvme library already support NVM-oF initiator, please see examples/nvme/perf as an example.
It can connect local PCI NVMe or remote NVM-oF target.

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Kirubakaran Kaliannan
Sent: Friday, September 22, 2017 12:57 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: [SPDK] spdk user mode initiator

Hi,

We are evaluating the spdk-target with NVMf initiator (installing linux 4.9 only on the initator but 3.18 on the target).
This is good so for, and we do have customers don’t want to upgrade 3.18 kernel even on the initiator side.

So, Similarly do we have SPDK initiator (at user land) so I can have this configured on 3.18 kernel ?
If we have one, any pointer to the documentation will help ?

Thanks
-kiru

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 21975 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread