[SPDK] FreeBSD - SPDK initiator - memory map failure

[SPDK] FreeBSD - SPDK initiator - memory map failure

[Sai Rajesh]

[-- Attachment #1: Type: text/plain, Size: 2808 bytes --]

Hi,

A month ago, I ran SPDK v18.01x on BSD12 (head) and didnot see any issues.
However, my lab assistant downgraded the setup to BSD11. Either downgrade
process went bad or something is not right with respect to SPDK/DPDK. Now
on BSD11, I see failure at spdk_mem_map_init() -> spdk_mem_register().

It says,

[root(a)bsd02 ~/spdk]# ./examples/nvme/perf/perf -q 64 -s 4096 -w write -t 3
-r 'trtype:RDMA adrfam:IPv4 traddr:192.168.66.55 trsvcid:4420'
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --no-pci ]
EAL: Sysctl reports 20 cpus
EAL: Detected 20 lcore(s)
EAL: Contigmem driver has 8 buffers, each of size 256MB
EAL: Mapped memory segment 1 @ 0x8027ff000: physaddr:0x2b50000000, len
268435456
EAL: Mapped memory segment 2 @ 0x8127ff000: physaddr:0x2b60000000, len
268435456
EAL: Mapped memory segment 3 @ 0x8227ff000: physaddr:0x2b70000000, len
268435456
EAL: Mapped memory segment 4 @ 0x8327ff000: physaddr:0x2b80000000, len
268435456
EAL: Mapped memory segment 5 @ 0x8427ff000: physaddr:0x2b90000000, len
268435456
EAL: Mapped memory segment 6 @ 0x8527ff000: physaddr:0x2ba0000000, len
268435456
EAL: Mapped memory segment 7 @ 0x8627ff000: physaddr:0x2bb0000000, len
268435456
EAL: Mapped memory segment 8 @ 0x8727ff000: physaddr:0x2bc0000000, len
268435456
Unable to unlink shared memory file: /var/run/.spdk_pid96250_config. Error
code: 2
Unable to unlink shared memory file: /var/run/.spdk_pid96250_hugepage_info.
Error code: 2
*invalid spdk_mem_register parameters, vaddr=0x8027ff000 len=268435456*
*invalid spdk_mem_register parameters, vaddr=0x8127ff000 len=268435456*
*invalid spdk_mem_register parameters, vaddr=0x8227ff000 len=268435456*
*invalid spdk_mem_register parameters, vaddr=0x8327ff000 len=268435456*
*invalid spdk_mem_register parameters, vaddr=0x8427ff000 len=268435456*
*invalid spdk_mem_register parameters, vaddr=0x8527ff000 len=268435456*
*invalid spdk_mem_register parameters, vaddr=0x8627ff000 len=268435456*
*invalid spdk_mem_register parameters, vaddr=0x8727ff000 len=268435456*

nvme_rdma.c:1432:nvme_rdma_qpair_submit_request: *ERROR*:
nvme_rdma_req_init() failed
nvme_rdma.c: 639:nvme_rdma_qpair_fabric_connect: *ERROR*:
spdk_nvme_rdma_req_fabric_connect failed
nvme_rdma.c: 829:nvme_rdma_qpair_connect: *ERROR*: Failed to send an
NVMe-oF Fabric CONNECT command
nvme_rdma.c:1346:nvme_rdma_ctrlr_construct: *ERROR*: failed to create admin
qpair
No valid NVMe controllers or AIO devices found
[root(a)bsd02 ~/spdk]#

"rping" works fine on same setup, so atleast wrt RDMA we are good with
downgrade process. From above errors it appears something bad happened with
DPDK. If that's fixed, I believe SPDK-RDMA should work fine.

Any clue or workaround to fix this?

Thanks,
Rajesh.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 3410 bytes --]

Re: [SPDK] FreeBSD - SPDK initiator - memory map failure

[Sai Rajesh]

[-- Attachment #1: Type: text/plain, Size: 7382 bytes --]

Hi Jim,

Thanks for your reply. And your suggestion worked for me.

I reverted your said commit 190ce864 http://dpdk.org/commit/190ce864 and
re-build SPDK/DPDK.

cd spdk
git revert 190ce864
kldunload contigmem.ko
./configure --with-rdma
gmake
cp dpdk/build/kmod/contigmem.ko /boot/kernel/
./scripts/setup.sh

Thanks,
Rajesh.

On Tue, Apr 24, 2018 at 3:01 PM Harris, James R <james.r.harris(a)intel.com>
wrote:

> Hi Rajesh,
>
>
>
> We’ve been hitting this issue internally as part of bringing up a new
> FreeBSD VM for the SPDK test pool.  But I don’t think this is a FreeBSD 11
> v 12 issue.  Let me explain.
>
>
>
> I have FreeBSD 10.2 VM on my laptop and updated to latest SPDK from the
> master branch.  I hit this exact same issue.  We also have a bare metal
> FreeBSD system in our test pool running FreeBSD 11.1 which does not hit
> this issue.  Incidentally though – this 11.1 system is using an older
> FreeBSD contigmem driver – it is not (currently) rebuilt as part of the
> automated test process, so it hasn’t been testing the latest DPDK changes
> to this driver.
>
>
>
> If I revert DPDK commit 190ce864, I get 2MB aligned addresses again.
> Could you try this as well?  Note this is a DPDK commit – not SPDK.  You’ll
> need to rebuild and reinstall the contigmem.ko driver.  I’d also suggest
> explicitly kldunload’ing contigmem and then running the setup.sh script
> again.
>
>
>
> The 2MB alignment allows the SPDK userspace page tables to use very simple
> 2MB page translation.  4KB alignment is possible, but it would be higher
> overhead.  There are other use cases where 4KB alignment is desired (for
> example, Stephen Bates’ recently added NVMe CMB work) so it’s likely this
> will be added at some point.  But it still would be more efficient to have
> 2MB aligned mappings.
>
>
>
> If you can confirm that reverting this patch fixes the alignment issue, we
> plan to revert this commit SPDK’s DPDK submodule and then look at how to
> fix this in upstream DPDK.  This commit can be safely reverted for SPDK,
> since we don’t support FreeBSD multi-process yet.
>
>
>
> Thanks,
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Sai Rajesh <
> rajeshsai810(a)gmail.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Tuesday, April 24, 2018 at 10:46 AM
> *To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Subject: *Re: [SPDK] FreeBSD - SPDK initiator - memory map failure
>
>
>
> On BSD 12 (head), for same configuration
>
>
>
> *EAL: Contigmem driver has 8 buffers, each of size 256MB*
>
> *EAL: Mapped memory segment 1 @ 0x801400000: physaddr:0x460000000, len
> 268435456*
>
> *EAL: Mapped memory segment 2 @ 0x811400000: physaddr:0x470000000, len
> 268435456*
>
> *EAL: Mapped memory segment 3 @ 0x821400000: physaddr:0x500000000, len
> 268435456*
>
> *EAL: Mapped memory segment 4 @ 0x831400000: physaddr:0x510000000, len
> 268435456*
>
> *EAL: Mapped memory segment 5 @ 0x841400000: physaddr:0x520000000, len
> 268435456*
>
> *EAL: Mapped memory segment 6 @ 0x851400000: physaddr:0x530000000, len
> 268435456*
>
> *EAL: Mapped memory segment 7 @ 0x861400000: physaddr:0x540000000, len
> 268435456*
>
> *EAL: Mapped memory segment 8 @ 0x871400000: physaddr:0x550000000, len
> 268435456*
>
>
>
> And everything works fine.
>
>
>
> One thing is still unclear for me. These addresses *0x801400000 *on BSD12
> or* 0x8027ff000* on BSD11 is obtained as a result of mmap() during
> rte_eal_hugepage_init(). How can I enforce mmap() to return address at
> 2MB boundary or multiple? Why do we even have this restriction in DPDK?
>
>
>
> if (((uintptr_t)vaddr & MASK_2MB) || (len & MASK_2MB)) {
>
>                 DEBUG_PRINT("invalid %s parameters, vaddr=%p len=%ju\n",
>
>                             __func__, vaddr, len);
>
>                 return -EINVAL;
>
>         }
>
>
>
>
>
> Thanks,
>
> Rajesh.
>
>
>
>
>
> On Mon, Apr 23, 2018 at 5:42 PM Sai Rajesh <rajeshsai810(a)gmail.com> wrote:
>
> Hi,
>
>
>
> A month ago, I ran SPDK v18.01x on BSD12 (head) and didnot see any issues.
> However, my lab assistant downgraded the setup to BSD11. Either downgrade
> process went bad or something is not right with respect to SPDK/DPDK. Now
> on BSD11, I see failure at spdk_mem_map_init() -> spdk_mem_register().
>
>
>
> It says,
>
>
>
> [root(a)bsd02 ~/spdk]# ./examples/nvme/perf/perf -q 64 -s 4096 -w write -t
> 3 -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.66.55 trsvcid:4420'
>
> Starting DPDK 17.11.0 initialization...
>
> [ DPDK EAL parameters: perf -c 0x1 --no-pci ]
>
> EAL: Sysctl reports 20 cpus
>
> EAL: Detected 20 lcore(s)
>
> EAL: Contigmem driver has 8 buffers, each of size 256MB
>
> EAL: Mapped memory segment 1 @ 0x8027ff000: physaddr:0x2b50000000, len
> 268435456
>
> EAL: Mapped memory segment 2 @ 0x8127ff000: physaddr:0x2b60000000, len
> 268435456
>
> EAL: Mapped memory segment 3 @ 0x8227ff000: physaddr:0x2b70000000, len
> 268435456
>
> EAL: Mapped memory segment 4 @ 0x8327ff000: physaddr:0x2b80000000, len
> 268435456
>
> EAL: Mapped memory segment 5 @ 0x8427ff000: physaddr:0x2b90000000, len
> 268435456
>
> EAL: Mapped memory segment 6 @ 0x8527ff000: physaddr:0x2ba0000000, len
> 268435456
>
> EAL: Mapped memory segment 7 @ 0x8627ff000: physaddr:0x2bb0000000, len
> 268435456
>
> EAL: Mapped memory segment 8 @ 0x8727ff000: physaddr:0x2bc0000000, len
> 268435456
>
> Unable to unlink shared memory file: /var/run/.spdk_pid96250_config. Error
> code: 2
>
> Unable to unlink shared memory file:
> /var/run/.spdk_pid96250_hugepage_info. Error code: 2
>
> *invalid spdk_mem_register parameters, vaddr=0x8027ff000 len=268435456*
>
> *invalid spdk_mem_register parameters, vaddr=0x8127ff000 len=268435456*
>
> *invalid spdk_mem_register parameters, vaddr=0x8227ff000 len=268435456*
>
> *invalid spdk_mem_register parameters, vaddr=0x8327ff000 len=268435456*
>
> *invalid spdk_mem_register parameters, vaddr=0x8427ff000 len=268435456*
>
> *invalid spdk_mem_register parameters, vaddr=0x8527ff000 len=268435456*
>
> *invalid spdk_mem_register parameters, vaddr=0x8627ff000 len=268435456*
>
> *invalid spdk_mem_register parameters, vaddr=0x8727ff000 len=268435456*
>
>
>
> nvme_rdma.c:1432:nvme_rdma_qpair_submit_request: *ERROR*:
> nvme_rdma_req_init() failed
>
> nvme_rdma.c: 639:nvme_rdma_qpair_fabric_connect: *ERROR*:
> spdk_nvme_rdma_req_fabric_connect failed
>
> nvme_rdma.c: 829:nvme_rdma_qpair_connect: *ERROR*: Failed to send an
> NVMe-oF Fabric CONNECT command
>
> nvme_rdma.c:1346:nvme_rdma_ctrlr_construct: *ERROR*: failed to create
> admin qpair
>
> No valid NVMe controllers or AIO devices found
>
> [root(a)bsd02 ~/spdk]#
>
>
>
> "rping" works fine on same setup, so atleast wrt RDMA we are good with
> downgrade process. From above errors it appears something bad happened with
> DPDK. If that's fixed, I believe SPDK-RDMA should work fine.
>
>
>
> Any clue or workaround to fix this?
>
>
>
> Thanks,
>
> Rajesh.
>
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 17848 bytes --]

Re: [SPDK] FreeBSD - SPDK initiator - memory map failure

[Harris, James R]

[-- Attachment #1: Type: text/plain, Size: 6209 bytes --]

Hi Rajesh,

We’ve been hitting this issue internally as part of bringing up a new FreeBSD VM for the SPDK test pool.  But I don’t think this is a FreeBSD 11 v 12 issue.  Let me explain.

I have FreeBSD 10.2 VM on my laptop and updated to latest SPDK from the master branch.  I hit this exact same issue.  We also have a bare metal FreeBSD system in our test pool running FreeBSD 11.1 which does not hit this issue.  Incidentally though – this 11.1 system is using an older FreeBSD contigmem driver – it is not (currently) rebuilt as part of the automated test process, so it hasn’t been testing the latest DPDK changes to this driver.

If I revert DPDK commit 190ce864, I get 2MB aligned addresses again.  Could you try this as well?  Note this is a DPDK commit – not SPDK.  You’ll need to rebuild and reinstall the contigmem.ko driver.  I’d also suggest explicitly kldunload’ing contigmem and then running the setup.sh script again.

The 2MB alignment allows the SPDK userspace page tables to use very simple 2MB page translation.  4KB alignment is possible, but it would be higher overhead.  There are other use cases where 4KB alignment is desired (for example, Stephen Bates’ recently added NVMe CMB work) so it’s likely this will be added at some point.  But it still would be more efficient to have 2MB aligned mappings.

If you can confirm that reverting this patch fixes the alignment issue, we plan to revert this commit SPDK’s DPDK submodule and then look at how to fix this in upstream DPDK.  This commit can be safely reverted for SPDK, since we don’t support FreeBSD multi-process yet.

Thanks,

-Jim


From: SPDK <spdk-bounces(a)lists.01.org> on behalf of Sai Rajesh <rajeshsai810(a)gmail.com>
Reply-To: Storage Performance Development Kit <spdk(a)lists.01.org>
Date: Tuesday, April 24, 2018 at 10:46 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] FreeBSD - SPDK initiator - memory map failure

On BSD 12 (head), for same configuration

EAL: Contigmem driver has 8 buffers, each of size 256MB
EAL: Mapped memory segment 1 @ 0x801400000: physaddr:0x460000000, len 268435456
EAL: Mapped memory segment 2 @ 0x811400000: physaddr:0x470000000, len 268435456
EAL: Mapped memory segment 3 @ 0x821400000: physaddr:0x500000000, len 268435456
EAL: Mapped memory segment 4 @ 0x831400000: physaddr:0x510000000, len 268435456
EAL: Mapped memory segment 5 @ 0x841400000: physaddr:0x520000000, len 268435456
EAL: Mapped memory segment 6 @ 0x851400000: physaddr:0x530000000, len 268435456
EAL: Mapped memory segment 7 @ 0x861400000: physaddr:0x540000000, len 268435456
EAL: Mapped memory segment 8 @ 0x871400000: physaddr:0x550000000, len 268435456

And everything works fine.

One thing is still unclear for me. These addresses 0x801400000 on BSD12 or 0x8027ff000 on BSD11 is obtained as a result of mmap() during rte_eal_hugepage_init(). How can I enforce mmap() to return address at 2MB boundary or multiple? Why do we even have this restriction in DPDK?


if (((uintptr_t)vaddr & MASK_2MB) || (len & MASK_2MB)) {

                DEBUG_PRINT("invalid %s parameters, vaddr=%p len=%ju\n",

                            __func__, vaddr, len);

                return -EINVAL;

        }



Thanks,
Rajesh.


On Mon, Apr 23, 2018 at 5:42 PM Sai Rajesh <rajeshsai810(a)gmail.com<mailto:rajeshsai810(a)gmail.com>> wrote:
Hi,

A month ago, I ran SPDK v18.01x on BSD12 (head) and didnot see any issues. However, my lab assistant downgraded the setup to BSD11. Either downgrade process went bad or something is not right with respect to SPDK/DPDK. Now on BSD11, I see failure at spdk_mem_map_init() -> spdk_mem_register().

It says,

[root(a)bsd02 ~/spdk]# ./examples/nvme/perf/perf -q 64 -s 4096 -w write -t 3 -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.66.55 trsvcid:4420'
Starting DPDK 17.11.0 initialization...
[ DPDK EAL parameters: perf -c 0x1 --no-pci ]
EAL: Sysctl reports 20 cpus
EAL: Detected 20 lcore(s)
EAL: Contigmem driver has 8 buffers, each of size 256MB
EAL: Mapped memory segment 1 @ 0x8027ff000: physaddr:0x2b50000000, len 268435456
EAL: Mapped memory segment 2 @ 0x8127ff000: physaddr:0x2b60000000, len 268435456
EAL: Mapped memory segment 3 @ 0x8227ff000: physaddr:0x2b70000000, len 268435456
EAL: Mapped memory segment 4 @ 0x8327ff000: physaddr:0x2b80000000, len 268435456
EAL: Mapped memory segment 5 @ 0x8427ff000: physaddr:0x2b90000000, len 268435456
EAL: Mapped memory segment 6 @ 0x8527ff000: physaddr:0x2ba0000000, len 268435456
EAL: Mapped memory segment 7 @ 0x8627ff000: physaddr:0x2bb0000000, len 268435456
EAL: Mapped memory segment 8 @ 0x8727ff000: physaddr:0x2bc0000000, len 268435456
Unable to unlink shared memory file: /var/run/.spdk_pid96250_config. Error code: 2
Unable to unlink shared memory file: /var/run/.spdk_pid96250_hugepage_info. Error code: 2
invalid spdk_mem_register parameters, vaddr=0x8027ff000 len=268435456
invalid spdk_mem_register parameters, vaddr=0x8127ff000 len=268435456
invalid spdk_mem_register parameters, vaddr=0x8227ff000 len=268435456
invalid spdk_mem_register parameters, vaddr=0x8327ff000 len=268435456
invalid spdk_mem_register parameters, vaddr=0x8427ff000 len=268435456
invalid spdk_mem_register parameters, vaddr=0x8527ff000 len=268435456
invalid spdk_mem_register parameters, vaddr=0x8627ff000 len=268435456
invalid spdk_mem_register parameters, vaddr=0x8727ff000 len=268435456

nvme_rdma.c:1432:nvme_rdma_qpair_submit_request: *ERROR*: nvme_rdma_req_init() failed
nvme_rdma.c: 639:nvme_rdma_qpair_fabric_connect: *ERROR*: spdk_nvme_rdma_req_fabric_connect failed
nvme_rdma.c: 829:nvme_rdma_qpair_connect: *ERROR*: Failed to send an NVMe-oF Fabric CONNECT command
nvme_rdma.c:1346:nvme_rdma_ctrlr_construct: *ERROR*: failed to create admin qpair
No valid NVMe controllers or AIO devices found
[root(a)bsd02 ~/spdk]#

"rping" works fine on same setup, so atleast wrt RDMA we are good with downgrade process. From above errors it appears something bad happened with DPDK. If that's fixed, I believe SPDK-RDMA should work fine.

Any clue or workaround to fix this?

Thanks,
Rajesh.

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 16888 bytes --]

Re: [SPDK] FreeBSD - SPDK initiator - memory map failure

[Sai Rajesh]

[-- Attachment #1: Type: text/plain, Size: 4374 bytes --]

On BSD 12 (head), for same configuration

*EAL: Contigmem driver has 8 buffers, each of size 256MB*
*EAL: Mapped memory segment 1 @ 0x801400000: physaddr:0x460000000, len
268435456*
*EAL: Mapped memory segment 2 @ 0x811400000: physaddr:0x470000000, len
268435456*
*EAL: Mapped memory segment 3 @ 0x821400000: physaddr:0x500000000, len
268435456*
*EAL: Mapped memory segment 4 @ 0x831400000: physaddr:0x510000000, len
268435456*
*EAL: Mapped memory segment 5 @ 0x841400000: physaddr:0x520000000, len
268435456*
*EAL: Mapped memory segment 6 @ 0x851400000: physaddr:0x530000000, len
268435456*
*EAL: Mapped memory segment 7 @ 0x861400000: physaddr:0x540000000, len
268435456*
*EAL: Mapped memory segment 8 @ 0x871400000: physaddr:0x550000000, len
268435456*

And everything works fine.

One thing is still unclear for me. These addresses *0x801400000 *on BSD12 or
 *0x8027ff000* on BSD11 is obtained as a result of mmap() during
rte_eal_hugepage_init(). How can I enforce mmap() to return address at 2MB
boundary or multiple? Why do we even have this restriction in DPDK?

if (((uintptr_t)vaddr & MASK_2MB) || (len & MASK_2MB)) {

                DEBUG_PRINT("invalid %s parameters, vaddr=%p len=%ju\n",

                            __func__, vaddr, len);

                return -EINVAL;

        }



Thanks,
Rajesh.


On Mon, Apr 23, 2018 at 5:42 PM Sai Rajesh <rajeshsai810(a)gmail.com> wrote:

> Hi,
>
> A month ago, I ran SPDK v18.01x on BSD12 (head) and didnot see any issues.
> However, my lab assistant downgraded the setup to BSD11. Either downgrade
> process went bad or something is not right with respect to SPDK/DPDK. Now
> on BSD11, I see failure at spdk_mem_map_init() -> spdk_mem_register().
>
> It says,
>
> [root(a)bsd02 ~/spdk]# ./examples/nvme/perf/perf -q 64 -s 4096 -w write -t
> 3 -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.66.55 trsvcid:4420'
> Starting DPDK 17.11.0 initialization...
> [ DPDK EAL parameters: perf -c 0x1 --no-pci ]
> EAL: Sysctl reports 20 cpus
> EAL: Detected 20 lcore(s)
> EAL: Contigmem driver has 8 buffers, each of size 256MB
> EAL: Mapped memory segment 1 @ 0x8027ff000: physaddr:0x2b50000000, len
> 268435456
> EAL: Mapped memory segment 2 @ 0x8127ff000: physaddr:0x2b60000000, len
> 268435456
> EAL: Mapped memory segment 3 @ 0x8227ff000: physaddr:0x2b70000000, len
> 268435456
> EAL: Mapped memory segment 4 @ 0x8327ff000: physaddr:0x2b80000000, len
> 268435456
> EAL: Mapped memory segment 5 @ 0x8427ff000: physaddr:0x2b90000000, len
> 268435456
> EAL: Mapped memory segment 6 @ 0x8527ff000: physaddr:0x2ba0000000, len
> 268435456
> EAL: Mapped memory segment 7 @ 0x8627ff000: physaddr:0x2bb0000000, len
> 268435456
> EAL: Mapped memory segment 8 @ 0x8727ff000: physaddr:0x2bc0000000, len
> 268435456
> Unable to unlink shared memory file: /var/run/.spdk_pid96250_config. Error
> code: 2
> Unable to unlink shared memory file:
> /var/run/.spdk_pid96250_hugepage_info. Error code: 2
> *invalid spdk_mem_register parameters, vaddr=0x8027ff000 len=268435456*
> *invalid spdk_mem_register parameters, vaddr=0x8127ff000 len=268435456*
> *invalid spdk_mem_register parameters, vaddr=0x8227ff000 len=268435456*
> *invalid spdk_mem_register parameters, vaddr=0x8327ff000 len=268435456*
> *invalid spdk_mem_register parameters, vaddr=0x8427ff000 len=268435456*
> *invalid spdk_mem_register parameters, vaddr=0x8527ff000 len=268435456*
> *invalid spdk_mem_register parameters, vaddr=0x8627ff000 len=268435456*
> *invalid spdk_mem_register parameters, vaddr=0x8727ff000 len=268435456*
>
> nvme_rdma.c:1432:nvme_rdma_qpair_submit_request: *ERROR*:
> nvme_rdma_req_init() failed
> nvme_rdma.c: 639:nvme_rdma_qpair_fabric_connect: *ERROR*:
> spdk_nvme_rdma_req_fabric_connect failed
> nvme_rdma.c: 829:nvme_rdma_qpair_connect: *ERROR*: Failed to send an
> NVMe-oF Fabric CONNECT command
> nvme_rdma.c:1346:nvme_rdma_ctrlr_construct: *ERROR*: failed to create
> admin qpair
> No valid NVMe controllers or AIO devices found
> [root(a)bsd02 ~/spdk]#
>
> "rping" works fine on same setup, so atleast wrt RDMA we are good with
> downgrade process. From above errors it appears something bad happened with
> DPDK. If that's fixed, I believe SPDK-RDMA should work fine.
>
> Any clue or workaround to fix this?
>
> Thanks,
> Rajesh.
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 6125 bytes --]