Re: [SPDK] Need help for fixing NVMe probe problem in NVMeoF initiator running fio.

Re: [SPDK] Need help for fixing NVMe probe problem in NVMeoF initiator running fio.

[Walker, Benjamin]

[-- Attachment #1: Type: text/plain, Size: 2031 bytes --]

On Wed, 2017-12-20 at 21:31 +0000, Sreeni (Sreenivasa) Busam (Stellus) wrote:
>   Initiator log:
>     ./fio --name=nvme --numjobs=1 --filename="trtype=RDMA adrfam=IPV4
> traddr=172.17.2.175 trsvcid=11345 subnqn=nqn.2017-06.io.spdk-MPcnode1 ns=1" --
> bs=4K --iodepth=1 --
> ioengine=/home.local/sfast/spdk20/spdk/examples/nvme/fio_plugin/fio_plugin --
> sync=0 --norandommap --group_reporting --size=12K --runtime=3 -rwmixwrite=30
> --thread=1 --rw=rw
>     nvme: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B,
> ioengine=spdk, iodepth=1
>     fio-3.3
>     Starting 1 thread
>     Starting DPDK 17.11.0 initialization...
>     [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid28214 ]
>     EAL: Detected 32 lcore(s)
>     EAL: No free hugepages reported in hugepages-1048576kB
>     EAL: Probing VFIO support...
>     nvme_ctrlr.c:1031:nvme_ctrlr_construct_namespaces: *ERROR*: controller has
> 0 namespaces
>     fio_plugin.c: 298:spdk_fio_setup: *ERROR*: spdk_nvme_probe()

This error means your NVMe-oF subsystem does not have any namespaces. That
usually means there is a problem in the configuration file.

>     NVMF configuration file:
>     [Nvmf]
>       MaxQueuesPerSession 4
>       AcceptorPollRate 10000
>  
>     [Subsystem1]
>       NQN nqn.2017-06.io.spdk-MPcnode1
>       Core 1
>       SN  SPDK0000000000000001
>       Listen RDMA 172.17.2.175:11345
>       AllowAnyHost Yes
>       NVMe 0000:84:00.0

What version of SPDK are you using? The NVMe directive inside of the [Subsystem]
section was replaced with syntax that looks like this:

Namespace Nvme0n1 1

And Nvme0n1 is defined by the [Nvme] section with syntax like this:

[Nvme]
TransportId "trtype:PCIe traddr:0000:00:00.0" Nvme0

See etc/spdk/nvmf.conf.in for a full example. If you upgraded SPDK to a newer
version but didn't update your configuration file, you'll end up with a
subsystem with 0 namespaces because it doesn't understand the NVMe directive
anymore.

[-- Attachment #2: smime.p7s --]
[-- Type: application/x-pkcs7-signature, Size: 3274 bytes --]

Re: [SPDK] Need help for fixing NVMe probe problem in NVMeoF initiator running fio.

[Sreeni Busam]

[-- Attachment #1: Type: text/plain, Size: 2738 bytes --]

Hi Ben,

Thank you for your reply. I fully forgot about the email thread, but I got the fio working and able to get my project related to developing a tool for IOPS and latency completed. The problem was device was removed by the administrator and I got it working. My apologies for not getting back.
I have few questions about how to handle I/O errors in NVMF initiator, am going to start a different thread.

Sreeni

-----Original Message-----
From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Walker, Benjamin
Sent: Thursday, December 21, 2017 8:51 AM
To: spdk(a)lists.01.org
Subject: Re: [SPDK] Need help for fixing NVMe probe problem in NVMeoF initiator running fio.

On Wed, 2017-12-20 at 21:31 +0000, Sreeni (Sreenivasa) Busam (Stellus) wrote:
>   Initiator log:
>     ./fio --name=nvme --numjobs=1 --filename="trtype=RDMA adrfam=IPV4
> traddr=172.17.2.175 trsvcid=11345 subnqn=nqn.2017-06.io.spdk-MPcnode1 
> ns=1" -- bs=4K --iodepth=1 -- 
> ioengine=/home.local/sfast/spdk20/spdk/examples/nvme/fio_plugin/fio_pl
> ugin --
> sync=0 --norandommap --group_reporting --size=12K --runtime=3 
> -rwmixwrite=30
> --thread=1 --rw=rw
>     nvme: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 
> 4096B-4096B, ioengine=spdk, iodepth=1
>     fio-3.3
>     Starting 1 thread
>     Starting DPDK 17.11.0 initialization...
>     [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid28214 ]
>     EAL: Detected 32 lcore(s)
>     EAL: No free hugepages reported in hugepages-1048576kB
>     EAL: Probing VFIO support...
>     nvme_ctrlr.c:1031:nvme_ctrlr_construct_namespaces: *ERROR*: 
> controller has
> 0 namespaces
>     fio_plugin.c: 298:spdk_fio_setup: *ERROR*: spdk_nvme_probe()

This error means your NVMe-oF subsystem does not have any namespaces. That usually means there is a problem in the configuration file.

>     NVMF configuration file:
>     [Nvmf]
>       MaxQueuesPerSession 4
>       AcceptorPollRate 10000
>  
>     [Subsystem1]
>       NQN nqn.2017-06.io.spdk-MPcnode1
>       Core 1
>       SN  SPDK0000000000000001
>       Listen RDMA 172.17.2.175:11345
>       AllowAnyHost Yes
>       NVMe 0000:84:00.0

What version of SPDK are you using? The NVMe directive inside of the [Subsystem] section was replaced with syntax that looks like this:

Namespace Nvme0n1 1

And Nvme0n1 is defined by the [Nvme] section with syntax like this:

[Nvme]
TransportId "trtype:PCIe traddr:0000:00:00.0" Nvme0

See etc/spdk/nvmf.conf.in for a full example. If you upgraded SPDK to a newer version but didn't update your configuration file, you'll end up with a subsystem with 0 namespaces because it doesn't understand the NVMe directive anymore.

[SPDK] Need help for fixing NVMe probe problem in NVMeoF initiator running fio.

[Sreeni Busam]

[-- Attachment #1: Type: text/plain, Size: 2864 bytes --]

Hello,

I have configured the target and initiator for a subsystem with 1 NVMe device in target.
Here are the errors I am getting on the initiator. I have a good NVMe device on the target side, but I am getting the error below.
If you know why the initiator does not initialize the controller and reason for the error, please let me know.
    Target log:
    Starting DPDK 17.08.0 initialization...
    [ DPDK EAL parameters: nvmf -c 0x1 --file-prefix=spdk_pid27838 ]
    EAL: Detected 32 lcore(s)
    EAL: No free hugepages reported in hugepages-1048576kB
    EAL: Probing VFIO support...
    Total cores available: 1
    Occupied cpu socket mask is 0x1
    reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
    copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled
    nvmf_tgt.c: 178:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 0 on socket 0
    nvmf_tgt.c: 178:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2017-06.io.spdk-MPcnode1 on lcore 0 on socket 0
    rdma.c:1146:spdk_nvmf_rdma_create: *NOTICE*: *** RDMA Transport Init ***
    rdma.c:1353:spdk_nvmf_rdma_listen: *NOTICE*: *** NVMf Target Listening on 172.17.2.175 port 11345 ***
    nvmf_tgt.c: 255:spdk_nvmf_startup: *NOTICE*: Acceptor running on core 0 on socket 0
    rdma.c:1515:spdk_nvmf_rdma_poll_group_create: *NOTICE*: Skipping unused RDMA device when creating poll group.
    Everything seems to be fine on the target application until the initiator connects to it and create a namespace.
    NVMF configuration file:
    [Nvmf]
      MaxQueuesPerSession 4
      AcceptorPollRate 10000

    [Subsystem1]
      NQN nqn.2017-06.io.spdk-MPcnode1
      Core 1
      SN  SPDK0000000000000001
      Listen RDMA 172.17.2.175:11345
      AllowAnyHost Yes
    NVMe 0000:84:00.0

    Initiator log:
    ./fio --name=nvme --numjobs=1 --filename="trtype=RDMA adrfam=IPV4 traddr=172.17.2.175 trsvcid=11345 subnqn=nqn.2017-06.io.spdk-MPcnode1 ns=1" --bs=4K --iodepth=1 --ioengine=/home.local/sfast/spdk20/spdk/examples/nvme/fio_plugin/fio_plugin --sync=0 --norandommap --group_reporting --size=12K --runtime=3 -rwmixwrite=30 --thread=1 --rw=rw
    nvme: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=spdk, iodepth=1
    fio-3.3
    Starting 1 thread
    Starting DPDK 17.11.0 initialization...
    [ DPDK EAL parameters: fio -c 0x1 -m 512 --file-prefix=spdk_pid28214 ]
    EAL: Detected 32 lcore(s)
    EAL: No free hugepages reported in hugepages-1048576kB
    EAL: Probing VFIO support...
    nvme_ctrlr.c:1031:nvme_ctrlr_construct_namespaces: *ERROR*: controller has 0 namespaces
    fio_plugin.c: 298:spdk_fio_setup: *ERROR*: spdk_nvme_probe()

Thanks for your suggestion
Sreeni

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 7889 bytes --]