[SPDK] SPDK and RocksDB integration -- setup problem

[SPDK] SPDK and RocksDB integration -- setup problem

[Meneghini, John]

[-- Attachment #1: Type: text/plain, Size: 5119 bytes --]

I have the followed the instructions here -- https://spdk.io/doc/blobfs.html to setup RockDB integration with SPDK environment. And I’m using top of the tree SPDK from github.

Here is the output from running scripts/gen_nvme.sh:


[lalsangi(a)spdk]$ scripts/gen_nvme.sh

0000:84:00.0 ....

{

"subsystem": "bdev",

"config": [

{

"method": "bdev_nvme_attach_controller",

"params": {

"trtype": "PCIe",

"name":"Nvme0",

"traddr":"0000:84:00.0"

}

}

]

}

And the output from scripts/setup.sh: (with some tracing)


[root(a)spdk]# HUGEMEM=5120 scripts/setup.sh

START ...>

----------

TARGET_USER .. defined.

lalsangi ..

OS is Linux! .. config

configure_linux_pci ..

vfio-pci ..

bdf related stuff ..

0000:84:00.0 (8086 0953): no driver -> vfio-pci

8086 0953

0000:84:00.0

1

end of config_linux_pci ..

config_linux_pci .. DONE.

-------------------------

hugetblfs_mounts ..

2560

Setting 2560 in /proc/sys/vm/nr_hugepages ..

/dev/hugepages

...............

Target user is: lalsangi

Password:

MEMLOCK_AMNT = 65536 ..

"lalsangi" user memlock limit: 64 MB



This is the maximum amount of memory you will be

able to use with DPDK and VFIO if run as user "lalsangi".

To change this, please adjust limits.conf memlock limit for user "lalsangi".

[root(a)spdk]#

And finally, the problem when I run ‘test/blobfs/mkfs/mkfs /usr/local/etc/spdk/rocksdb.json Nvme0n1: (with some tracing)
(I tried with different bdev names -- e.g., Nvme0 -- still same problem)


[root(a)ssan-rx2560-03 spdk]# test/blobfs/mkfs/mkfs /usr/local/etc/spdk/rocksdb.json Nvme0n1

[2020-12-14 21:30:05.201555] Launching mkfs, config_file: /usr/local/etc/spdk/rocksdb.json

[2020-12-14 21:30:05.201832] Starting SPDK v21.01-pre git sha1 602b134fa / DPDK 20.08.0 initialization...

[2020-12-14 21:30:05.201879] [ DPDK EAL parameters: [2020-12-14 21:30:05.201906] spdk_mkfs [2020-12-14 21:30:05.201928] --no-shconf [2020-12-14 21:30:05.201948] -c 0x3 [2020-12-14 21:30:05.201970] --log-level=lib.eal:6 [2020-12-14 21:30:05.201992] --log-level=lib.cryptodev:5 [2020-12-14 21:30:05.202012] --log-level=user1:6 [2020-12-14 21:30:05.202031] --base-virtaddr=0x200000000000 [2020-12-14 21:30:05.202051] --match-allocations [2020-12-14 21:30:05.202071] --file-prefix=spdk_pid103642 [2020-12-14 21:30:05.202092] ]

EAL: No available hugepages reported in hugepages-1048576kB

EAL: No legacy callbacks, legacy socket not created

[2020-12-14 21:30:05.250379] app.c: 465:spdk_app_start: *NOTICE*: Total cores available: 2

[2020-12-14 21:30:05.250411] Total cores available: 2

[2020-12-14 21:30:05.250538] spdk_reactor_init ..

[2020-12-14 21:30:05.347656] Reactor init done ..

[2020-12-14 21:30:05.347710] cpuset_set_cpu done ..

[2020-12-14 21:30:05.347719] spdk_thread_create init done ..

[2020-12-14 21:30:05.348539] spdk_mempool_get_bulk done ..

[2020-12-14 21:30:05.348556] Allowing new thread app_thread ..

[2020-12-14 21:30:05.348568] Ran the thread function app_thread ..

[2020-12-14 21:30:05.461055] send msg to run bootstrap_fn ..

[2020-12-14 21:30:05.461181] reactor.c: 701:reactor_run: *NOTICE*: Reactor started on core 1

[2020-12-14 21:30:05.461184] spdk_thread_create init done ..

[2020-12-14 21:30:05.462004] spdk_mempool_get_bulk done ..

[2020-12-14 21:30:05.462035] Allowing new thread reactor_1 ..

[2020-12-14 21:30:05.462045] Ran the thread function reactor_1 ..

[2020-12-14 21:30:05.462052] reactor.c: 701:reactor_run: *NOTICE*: Reactor started on core 0

[2020-12-14 21:30:05.462080] Load json config /usr/local/etc/spdk/rocksdb.json ..Inside spdk_json_parse ..

Inside spdk_json_parse ..

[2020-12-14 21:30:05.462126] Read the json config file /usr/local/etc/spdk/rocksdb.json ..

[2020-12-14 21:30:05.462155] Find the key_name: subsystem ..

[2020-12-14 21:30:05.462164] parsing token  #??..

[2020-12-14 21:30:05.462172] key  #??, value 0??

[2020-12-14 21:30:05.462180] json_config.c: 597:spdk_app_json_config_load: *WARNING*: No 'subsystems' key JSON configuration file.

[2020-12-14 21:30:05.462524] accel_engine.c: 692:spdk_accel_engine_initialize: *NOTICE*: Accel engine initialized to use software engine.

Initializing filesystem on bdev Nvme0n1...[2020-12-14 21:30:05.527735] bdev.c:5562:spdk_bdev_open_ext: *NOTICE*: Currently unable to find bdev with name: Nvme0n1



Failed to initialize filesystem on bdev Nvme0n1...done.

First, I noticed that the code is expecting ‘subsystems’ key_name, but the gen_nvme.sh() created the .json with the key ‘subsystem’.  I changed the code to match the key_name. But still the same issue exists.

Based on the above, it seems that either I’m making a silly mistake, or something basic is not working.

Any hints / suggestions on debugging this problem is highly appreciated!

Or should I stick with code bits with particular tags for SPDK and RocksDB integration?

Thanks,

John Meneghini
ONTAP SAN Target Architect
978-930-3519 (cell)
johnm(a)netapp.com