Hi Rajesh,

 

For the first failure – do you have a kernel w/o INVARIANTS that you could try?  The FreeBSD NVMe driver is setting the ETERNAL flag when registering the device meaning the expectation is that it won’t be removed – this needs to be removed.  I’m guessing the systems in the SPDK test pool are running a STABLE branch which is why this hasn’t come up before.

 

So you could either try with most recent FreeBSD release, rebuild your current kernel and disable INVARIANTS, or rebuild your kernel and make nvme a loadable module and then don’t load it.  A FreeBSD kernel patch will also be needed but that won’t help you in the short term.

 

Regarding NVMe-oF, I don’t believe this has been tested on FreeBSD.  This is a bug in the SPDK build environment – it should not be building the RDMA transport on FreeBSD.  If you have any experience with using userspace RDMA on FreeBSD, and could provide any pointers, it would be appreciated.  I’m guessing the error message about not finding the config directory has something to do with it.

 

Thanks,

 

-Jim

 

 

From: SPDK <spdk-bounces@lists.01.org> on behalf of Sai Rajesh <rajeshsai810@gmail.com>
Reply-To: Storage Performance Development Kit <spdk@lists.01.org>
Date: Wednesday, February 28, 2018 at 6:55 PM
To: "spdk@lists.01.org" <spdk@lists.01.org>
Subject: [SPDK] SPDK on FreeBSD

 

Hi,

 

I'm running SPDK on FreeBSD master branch. When I try to run setup.sh it crashes my kernel. For now, I hardcoded kenv variable hw.nic_uio.bdfs to my nvme disk PCI address. I grabbed the address from command "pciconf -l | grep nvme".

 

Looks like nic_uio driver is trying to unload/destroy something its not supposed to?

 

I plan to run this node as only initiator node, so I donot need any fancy setup. All I need is able to setup SPDK on it and RDMA connect to target node which is a Linux node. 

 

If I skip setup.sh, initiator complains SPDK/DPDK environment is not set.

Is there a way to setup SPDK/DPDK without any overhead?

 

[root@rajesh ~/spdk]#./scripts/setup.sh

kldunload: can't find file nic_uio.ko

hw.nnic_uio: hw.nic_uio.bdfs = '5:0:0'

nic_uio_load: detaching and storing dev=0xfffff80108c1e100

ic_uio.bdfs="5:0:0"

panic: WARNING: Driver mistake: destroy_dev on eternal 0

 

cpuid = 19

time = 1519867718

KDB: stack backtrace:

db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe01a1161a20

vpanic() at vpanic+0x18d/frame 0xfffffe01a1161a80

vpanic() at vpanic/frame 0xfffffe01a1161b00

destroy_devl() at destroy_devl+0x541/frame 0xfffffe01a1161b40

destroy_dev() at destroy_dev+0x40/frame 0xfffffe01a1161b60

nvme_ctrlr_destruct() at nvme_ctrlr_destruct+0x38/frame 0xfffffe01a1161b90

nvme_detach() at nvme_detach+0x1a/frame 0xfffffe01a1161bb0

device_detach() at device_detach+0x167/frame 0xfffffe01a1161bf0

nic_uio_load() at nic_uio_load+0x2e4/frame 0xfffffe01a1161db0

nic_uio_modevent() at nic_uio_modevent+0x53/frame 0xfffffe01a1161df0

driver_module_handler() at driver_module_handler+0x5b/frame 0xfffffe01a1161e40

module_register_init() at module_register_init+0xc0/frame 0xfffffe01a1161e70

linker_load_module() at linker_load_module+0xb78/frame 0xfffffe01a1162180

kern_kldload() at kern_kldload+0xf1/frame 0xfffffe01a11621d0

sys_kldload() at sys_kldload+0x5b/frame 0xfffffe01a1162200

amd64_syscall() at amd64_syscall+0x79b/frame 0xfffffe01a1162330

fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffffffeb60

KDB: enter: panic

[ thread pid 1609 tid 100605 ]

Stopped at      kdb_enter+0x3b: movq    $0,kdb_why

db>

 

And if I comment out kldload nic_uio.ko from setup.sh, it fails to create rdma event channel.

 

root@rajesh:~/spdk # ./examples/nvme/perf/perf -q 1 -s 4096 -w randread -t 10 -r "trtype:RDMA adrfam:IPv4 traddr:192.168.22.55 trsvcid:9797"

Starting DPDK 17.11.0 initialization...

[ DPDK EAL parameters: perf -c 0x1 --no-pci ]

EAL: Sysctl reports 20 cpus

EAL: Detected 20 lcore(s)

EAL: Contigmem driver has 8 buffers, each of size 256MB

EAL: Mapped memory segment 1 @ 0x801400000: physaddr:0x390000000, len 268435456

EAL: Mapped memory segment 2 @ 0x811400000: physaddr:0x3a0000000, len 268435456

EAL: Mapped memory segment 3 @ 0x821400000: physaddr:0x3b0000000, len 268435456

EAL: Mapped memory segment 4 @ 0x831400000: physaddr:0x3c0000000, len 268435456

EAL: Mapped memory segment 5 @ 0x841400000: physaddr:0x3d0000000, len 268435456

EAL: Mapped memory segment 6 @ 0x851400000: physaddr:0x3e0000000, len 268435456

EAL: Mapped memory segment 7 @ 0x861400000: physaddr:0x3f0000000, len 268435456

EAL: Mapped memory segment 8 @ 0x871400000: physaddr:0x400000000, len 268435456

Unable to unlink shared memory file: /var/run/.spdk_pid1425_config. Error code: 2

Unable to unlink shared memory file: /var/run/.spdk_pid1425_hugepage_info. Error code: 2

Initializing NVMe Controllers

libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.

nvme_rdma.c: 736:nvme_rdma_qpair_connect: *ERROR*: rdma_create_event_channel() failed

nvme_rdma.c:1339:nvme_rdma_ctrlr_construct: *ERROR*: failed to create admin qpair

No valid NVMe controllers or AIO devices found

Feb 28 20:47:24 saber1b perf: nvme_rdma.c: 736:nvme_rdma_qpair_connect: *ERROR*: rdma_create_event_channel() failed

Feb 28 20:47:24 saber1b perf: nvme_rdma.c:1339:nvme_rdma_ctrlr_construct: *ERROR*: failed to create admin qpair

 

Any help is appreciated to get me started as FreeBSD SPDK initiator.

 

Thanks,

Rajesh.

_______________________________________________
SPDK mailing list
SPDK@lists.01.org
https://lists.01.org/mailman/listinfo/spdk