[SPDK] Initiator IO errors using bdev libaio

[SPDK] Initiator IO errors using bdev libaio

[Shahar Salzman]

[-- Attachment #1: Type: text/plain, Size: 2454 bytes --]

Hi,

We are building a setup which exposes a local non nvme SSD connected via SAS Megaraid controller as a NVMeF device, using libaio bdev, I added the entire setup detail bellow. When doing single threaded IO, everything is OK, but when running multithreaded IO (vdbench) with data verification (-vr: verify via read after each write) we get IO errors.

Looking at the IO errors, the problem seems to be that the data buffer given by the read is missing.
This is identified by vdbench which fills the buffer with a predefined pattern which should not be in the buffer post read.
As written above, this does not happen when I do single threaded IO. In addition, if we read the contents from the error LBA using dd, the contents are correct.

We ran the same vdbench workload directly on the disk without any issue, in addition, we tried replacing the physical device with a RAM disk, and with a SATA device directly connected (not via megaraid controller), and the problem does not recreate (i.e. no IO error).

Here is the vdbench error:
16:34:59.377 localhost-0: 16:34:59.376 op: read   lun: /dev/nvme0n1 lba: 789422931968 0xB7CD460000 xfer:   131072 errno: BAD_READ_RETURN: 'Read was successful, but data buffer contents not changed'

Shahar

P.S. system information and configuration details:

The setup is: Supermicro server.
ConnectX-4 Lx
8xSSD’s aggregated to single RAID0 volume configured 64K stripe size with WT and DirectIO. Presented to OS by megaraid module.
CentOS 7.6 + OFED-4.6-1.0.1

SPDK configuration script we are using:
/root/spdk/scripts/setup.sh
sleep 3
/root/spdk/app/nvmf_tgt/nvmf_tgt -m 0xF000000 &
sleep 3
spdk/scripts/rpc.py  nvmf_subsystem_create nqn.2016-06.io.spdk:jbof01 -a -s 000000000 -d jbof01
spdk/scripts/rpc.py  construct_aio_bdev /dev/sdc megaraid
spdk/scripts/rpc.py  nvmf_subsystem_add_ns nqn.2016-06.io.spdk:jbof01 megaraid

spdk/scripts/rpc.py nvmf_create_transport -t RDMA -u 8192
spdk/scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:jbof01 -t rdma -a 172.20.95.222 -s 1023

nvme connect -t rdma -s 1023 -a 172.20.95.222 -n nqn.2016-06.io.spdk:jbof01

Vdbench input file:
debug=88
data_errors=1
sd=sd1,lun=/dev/nvme0n1,openflags=o_direct
wd=wd1,sd=sd*,rdpct=0,rhpct=0,whpct=0,xfersize=(128k,50,32k,10,4k,20,512k,20),seekpct=100
rd=rd1,wd=wd*,interval=1,iorate=MAX,elapsed=2600000,threads=(20)


** If we change the threads to 1, there is no issue.