Re: [SPDK] SPDK on FreeBSD

* Re: [SPDK] SPDK on FreeBSD
@ 2018-03-04  2:35 Sai Rajesh
  0 siblings, 0 replies; 3+ messages in thread
From: Sai Rajesh @ 2018-03-04  2:35 UTC (permalink / raw)
  To: spdk

[-- Attachment #1: Type: text/plain, Size: 6903 bytes --]

Hi Jim,

I will work on your suggestion for issue 1.

For now, in my experiment I would need FreeBSD as a initiator. So, I do not
require any devices that need nic_uio.ko. I have commented out
"configure_freebsd_pci" in scripts/setup.sh to proceed forward.

For issue 2, I resolved the problem my linking RDMA userspace libraries
with SPDK. I changed spdk/mk/spdk.common.mk, and added following lines.

ifeq ($(CONFIG_RDMA), y)
ifeq ($(OS),FreeBSD)
LDFLAGS+= -libverbs -lrdmacm -lmlx4 -lpthread
endif
endif

I have created issue # 253 for same. Hope this fix would help other FreeBSD
users.

Thanks,
Rajesh.

On Thu, Mar 1, 2018 at 7:04 PM Harris, James R <james.r.harris(a)intel.com>
wrote:

> Hi Rajesh,
>
>
>
> For the first failure – do you have a kernel w/o INVARIANTS that you could
> try?  The FreeBSD NVMe driver is setting the ETERNAL flag when registering
> the device meaning the expectation is that it won’t be removed – this needs
> to be removed.  I’m guessing the systems in the SPDK test pool are running
> a STABLE branch which is why this hasn’t come up before.
>
>
>
> So you could either try with most recent FreeBSD release, rebuild your
> current kernel and disable INVARIANTS, or rebuild your kernel and make nvme
> a loadable module and then don’t load it.  A FreeBSD kernel patch will also
> be needed but that won’t help you in the short term.
>
>
>
> Regarding NVMe-oF, I don’t believe this has been tested on FreeBSD.  This
> is a bug in the SPDK build environment – it should not be building the RDMA
> transport on FreeBSD.  If you have any experience with using userspace RDMA
> on FreeBSD, and could provide any pointers, it would be appreciated.  I’m
> guessing the error message about not finding the config directory has
> something to do with it.
>
>
>
> Thanks,
>
>
>
> -Jim
>
>
>
>
>
> *From: *SPDK <spdk-bounces(a)lists.01.org> on behalf of Sai Rajesh <
> rajeshsai810(a)gmail.com>
> *Reply-To: *Storage Performance Development Kit <spdk(a)lists.01.org>
> *Date: *Wednesday, February 28, 2018 at 6:55 PM
> *To: *"spdk(a)lists.01.org" <spdk(a)lists.01.org>
> *Subject: *[SPDK] SPDK on FreeBSD
>
>
>
> Hi,
>
>
>
> I'm running SPDK on FreeBSD master branch. When I try to run setup.sh it
> crashes my kernel. For now, I hardcoded kenv variable *hw.nic_uio.bdfs*
> to my nvme disk PCI address. I grabbed the address from command "pciconf -l
> | grep nvme".
>
>
>
> Looks like nic_uio driver is trying to unload/destroy something its not
> supposed to?
>
>
>
> I plan to run this node as only initiator node, so I donot need any fancy
> setup. All I need is able to setup SPDK on it and RDMA connect to target
> node which is a Linux node.
>
>
>
> If I skip setup.sh, initiator complains SPDK/DPDK environment is not set.
>
> Is there a way to setup SPDK/DPDK without any overhead?
>
>
>
> [root(a)rajesh ~/spdk]#./scripts/setup.sh
>
> kldunload: can't find file nic_uio.ko
>
> hw.nnic_uio: hw.nic_uio.bdfs = '5:0:0'
>
> nic_uio_load: detaching and storing dev=0xfffff80108c1e100
>
> ic_uio.bdfs="5:0:0"
>
> panic: WARNING: Driver mistake: destroy_dev on eternal 0
>
>
>
> cpuid = 19
>
> time = 1519867718
>
> KDB: stack backtrace:
>
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> 0xfffffe01a1161a20
>
> vpanic() at vpanic+0x18d/frame 0xfffffe01a1161a80
>
> vpanic() at vpanic/frame 0xfffffe01a1161b00
>
> destroy_devl() at destroy_devl+0x541/frame 0xfffffe01a1161b40
>
> destroy_dev() at destroy_dev+0x40/frame 0xfffffe01a1161b60
>
> nvme_ctrlr_destruct() at nvme_ctrlr_destruct+0x38/frame 0xfffffe01a1161b90
>
> nvme_detach() at nvme_detach+0x1a/frame 0xfffffe01a1161bb0
>
> device_detach() at device_detach+0x167/frame 0xfffffe01a1161bf0
>
> nic_uio_load() at nic_uio_load+0x2e4/frame 0xfffffe01a1161db0
>
> nic_uio_modevent() at nic_uio_modevent+0x53/frame 0xfffffe01a1161df0
>
> driver_module_handler() at driver_module_handler+0x5b/frame
> 0xfffffe01a1161e40
>
> module_register_init() at module_register_init+0xc0/frame
> 0xfffffe01a1161e70
>
> linker_load_module() at linker_load_module+0xb78/frame 0xfffffe01a1162180
>
> kern_kldload() at kern_kldload+0xf1/frame 0xfffffe01a11621d0
>
> sys_kldload() at sys_kldload+0x5b/frame 0xfffffe01a1162200
>
> amd64_syscall() at amd64_syscall+0x79b/frame 0xfffffe01a1162330
>
> fast_syscall_common() at fast_syscall_common+0x101/frame 0x7fffffffeb60
>
> KDB: enter: panic
>
> [ thread pid 1609 tid 100605 ]
>
> Stopped at      kdb_enter+0x3b: movq    $0,kdb_why
>
> db>
>
>
>
> And if I comment out kldload nic_uio.ko from setup.sh, it fails to create
> rdma event channel.
>
>
>
> root(a)rajesh:~/spdk # ./examples/nvme/perf/perf -q 1 -s 4096 -w randread
> -t 10 -r "trtype:RDMA adrfam:IPv4 traddr:192.168.22.55 trsvcid:9797"
>
> Starting DPDK 17.11.0 initialization...
>
> [ DPDK EAL parameters: perf -c 0x1 --no-pci ]
>
> EAL: Sysctl reports 20 cpus
>
> EAL: Detected 20 lcore(s)
>
> EAL: Contigmem driver has 8 buffers, each of size 256MB
>
> EAL: Mapped memory segment 1 @ 0x801400000: physaddr:0x390000000, len
> 268435456
>
> EAL: Mapped memory segment 2 @ 0x811400000: physaddr:0x3a0000000, len
> 268435456
>
> EAL: Mapped memory segment 3 @ 0x821400000: physaddr:0x3b0000000, len
> 268435456
>
> EAL: Mapped memory segment 4 @ 0x831400000: physaddr:0x3c0000000, len
> 268435456
>
> EAL: Mapped memory segment 5 @ 0x841400000: physaddr:0x3d0000000, len
> 268435456
>
> EAL: Mapped memory segment 6 @ 0x851400000: physaddr:0x3e0000000, len
> 268435456
>
> EAL: Mapped memory segment 7 @ 0x861400000: physaddr:0x3f0000000, len
> 268435456
>
> EAL: Mapped memory segment 8 @ 0x871400000: physaddr:0x400000000, len
> 268435456
>
> Unable to unlink shared memory file: /var/run/.spdk_pid1425_config. Error
> code: 2
>
> Unable to unlink shared memory file: /var/run/.spdk_pid1425_hugepage_info.
> Error code: 2
>
> Initializing NVMe Controllers
>
> libibverbs: Warning: couldn't open config directory '/etc/ibverbs/'.
>
> nvme_rdma.c: 736:nvme_rdma_qpair_connect: *ERROR*:
> rdma_create_event_channel() failed
>
> nvme_rdma.c:1339:nvme_rdma_ctrlr_construct: *ERROR*: failed to create
> admin qpair
>
> No valid NVMe controllers or AIO devices found
>
> Feb 28 20:47:24 saber1b perf: nvme_rdma.c: 736:nvme_rdma_qpair_connect:
> *ERROR*: rdma_create_event_channel() failed
>
> Feb 28 20:47:24 saber1b perf: nvme_rdma.c:1339:nvme_rdma_ctrlr_construct:
> *ERROR*: failed to create admin qpair
>
>
>
> Any help is appreciated to get me started as FreeBSD SPDK initiator.
>
>
>
> Thanks,
>
> Rajesh.
> _______________________________________________
> SPDK mailing list
> SPDK(a)lists.01.org
> https://lists.01.org/mailman/listinfo/spdk
>

[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 14974 bytes --]

^ permalink raw reply	[flat|nested] 3+ messages in thread