Re: [SPDK] SPDK errors

[Luse, Paul E <paul.e.luse at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 20240 bytes --]

Not me, I’ve been out for 2 days and then M-W we have the SPDK developer meetup in Chandler so things will be a little quiet for the first half of the week at least..

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org] On Behalf Of Santhebachalli Ganesh
Sent: Friday, November 3, 2017 3:13 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: Re: [SPDK] SPDK errors

Thanks in advance.
Did anyone have a chance to look?

I can also try a conf file which someone is using on a working setup...
which has 2 sub-systems with unique NQN's on the target listening on IP's (say, 1.1.1.80, 1.1.2.80) on separate subnet,
2 (say, 1.1.1.81, 1.1.2.81) initiators connected with perf used (at the same time) to run metrics from initiators to target.


On Wed, Nov 1, 2017 at 4:57 PM, Santhebachalli Ganesh <svg.eid(a)gmail.com<mailto:svg.eid(a)gmail.com>> wrote:
Some typo; should read.
1.1.1.80, 1.1.2.80 - Target
1.1.1.81 - Ini 1
1.1.2.83 - Ini 2

On Wed, Nov 1, 2017 at 4:00 PM, Santhebachalli Ganesh <svg.eid(a)gmail.com<mailto:svg.eid(a)gmail.com>> wrote:
>What did you do to get past your previous issues?
Same CX-3 and HW.
Just latest SW/Distro - CentOS, latest (as of my install) kernel-ml, latest SPDK/DPDK.

Have got some CX-5's now. Not yet installed.
Will try to get some CX-4's.

Thanks.
Ganesh


On Wed, Nov 1, 2017 at 3:52 PM, Luse, Paul E <paul.e.luse(a)intel.com<mailto:paul.e.luse(a)intel.com>> wrote:
Hi Ganesh,

That’s progress at least.  What did you do to get past your previous issues?  FYI I finally broke down and bought some CX-4’s, just got them today actually so will work on trying to get them up and running end of next week.

In the meantime, I’m sure John or someone else can assist with the questions below.

Thx
Paul

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Santhebachalli Ganesh
Sent: Wednesday, November 1, 2017 3:10 PM

To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK errors

Paul and John,
Back to some perf work.
As always, thanks for your time and assistance in advance.

Update:
Able to run perf (ini to target) consistently with updated CentOS, kernel-ml and latest SPDK/DPDK.
Same HW: CX-3 and PMC NVME, PCI card.
Able to get 1M IOP's (4KB) on generic HW.
[adminuser(a)dell730-80 ~]$ uname -r
4.13.9-1.el7.elrepo.x86_64
[adminuser(a)dell730-80 ~]$ lsb_release -d
Description:    CentOS Linux release 7.4.1708 (Core)

Issue being faced now:
Can run perf at one initiator at a time.
Unable to run perf from multiple initiators to target at the same time. Able to discover and connect.
Just "hangs" on initiator side when running perf. Don't seem to see any messages anywhere.
Maybe, I am missing some config to be done.

Added info below. Let me know any more info you need.

---
Initiators connected to target NIC ports on separate subnet
Target has a dual port CX-3 NIC with IP addresses on separate subnet.
1.1.1.80, 1.2.2.80 - Target
1.1.1.81 - Ini 1
1.2.2.83 - Ini 2

-- target
[adminuser(a)dell730-80 ~]$ sudo ~/gits/spdk/app/nvmf_tgt/nvmf_tgt -c ~/gits/spdk/etc/spdk/nvmf.conf
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: nvmf -c 0x5555 --file-prefix=spdk_pid3534 ]
EAL: Detected 48 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
Total cores available: 8
Occupied cpu socket mask is 0x1
reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 2 on socket 0
reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 4 on socket 0
reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 6 on socket 0
reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 8 on socket 0
reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 10 on socket 0
reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 12 on socket 0
reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 14 on socket 0
reactor.c: 364:_spdk_reactor_run: *NOTICE*: Reactor started on core 0 on socket 0
copy_engine_ioat.c: 306:copy_engine_ioat_init: *NOTICE*: Ioat Copy Engine Offload Enabled
EAL: PCI device 0000:04:00.0 on NUMA socket 0
EAL:   probe driver: 11f8:f117 spdk_nvme
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 11f8:f117 spdk_nvme
nvmf_tgt.c: 178:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2014-08.org.nvmexpress.discovery on lcore 2 on socket 0
nvmf_tgt.c: 178:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2016-06.io.spdk:cnode1 on lcore 2 on socket 0
rdma.c:1145:spdk_nvmf_rdma_create: *NOTICE*: *** RDMA Transport Init ***
rdma.c:1352:spdk_nvmf_rdma_listen: *NOTICE*: *** NVMf Target Listening on 1.1.1.80 port 4420 ***
conf.c: 500:spdk_nvmf_construct_subsystem: *NOTICE*: Attaching block device Nvme0n1 to subsystem nqn.2016-06.io.spdk:cnode1
nvmf_tgt.c: 178:nvmf_tgt_create_subsystem: *NOTICE*: allocated subsystem nqn.2016-06.io.spdk:cnode2 on lcore 4 on socket 0
rdma.c:1352:spdk_nvmf_rdma_listen: *NOTICE*: *** NVMf Target Listening on 1.1.2.80 port 4420 ***
conf.c: 500:spdk_nvmf_construct_subsystem: *NOTICE*: Attaching block device Nvme1n1 to subsystem nqn.2016-06.io.spdk:cnode2
nvmf_tgt.c: 255:spdk_nvmf_startup: *NOTICE*: Acceptor running on core 2 on socket 0

--- ini
Can see both NQN's on discovery
Can connect to both NQN's
[adminuser(a)dell730-81 ~]$ sudo nvme discover -t rdma -a 1.1.1.80 -s 4420

Discovery Log Number of Records 2, Generation counter 5
=====Discovery Log Entry 0======
trtype:  rdma
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified
portid:  0
trsvcid: 4420
subnqn:  nqn.2016-06.io.spdk:cnode1
traddr:  1.1.1.80
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms:    rdma-cm
rdma_pkey: 0x0000
=====Discovery Log Entry 1======
trtype:  rdma
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified
portid:  1
trsvcid: 4420
subnqn:  nqn.2016-06.io.spdk:cnode2
traddr:  1.1.2.80
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms:    rdma-cm
rdma_pkey: 0x0000

sudo nvme connect -t rdma -n "nqn.2016-06.io.spdk:cnode1" -a 1.1.1.80 -w 1.1.1.81
sudo nvme connect -t rdma -n "nqn.2016-06.io.spdk:cnode2" -a 1.1.2.80 -w 1.1.2.83

[adminuser(a)dell730-81 ~]$ sudo ~/gits/spdk/examples/nvme/perf/perf -q 1 -s 4096 -w read -t 30 -r 'trtype:RDMA adrfam:IPv4 traddr:1.1.1.80 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1' -c 0x2
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: perf -c 0x2 --no-pci --file-prefix=spdk_pid3383 ]
EAL: Detected 48 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
Initializing NVMe Controllers
Attaching to NVMe over Fabrics controller at 1.1.1.80:4420<http://1.1.1.80:4420>: nqn.2016-06.io.spdk:cnode1
Attached to NVMe over Fabrics controller at 1.1.1.80:4420<http://1.1.1.80:4420>: nqn.2016-06.io.spdk:cnode1
Associating SPDK bdev Controller (SPDK00000000000001  ) with lcore 1
Initialization complete. Launching workers.
Starting thread on core 1

[adminuser(a)dell730-83 ~]$ sudo ~/gits/spdk/spdk/examples/nvme/perf/perf -q 1 -s 4096 -w randread -t 30 -r 'trtype:RDMA adrfam:IPv4 traddr:1.1.2.80 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode2' -c 0x2
Starting DPDK 17.08.0 initialization...
[ DPDK EAL parameters: perf -c 0x2 --no-pci --file-prefix=spdk_pid3633 ]
EAL: Detected 48 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
Initializing NVMe Controllers
Attaching to NVMe over Fabrics controller at 1.1.2.80:4420<http://1.1.2.80:4420>: nqn.2016-06.io.spdk:cnode2
Attached to NVMe over Fabrics controller at 1.1.2.80:4420<http://1.1.2.80:4420>: nqn.2016-06.io.spdk:cnode2
Associating SPDK bdev Controller (SPDK00000000000002  ) with lcore 1
Initialization complete. Launching workers.
Starting thread on core 1

--- snippet from my nvmf.conf on target
  TransportId "trtype:PCIe traddr:0000:04:00.0" Nvme0
  TransportId "trtype:PCIe traddr:0000:06:00.0" Nvme1

# Namespaces backed by physical NVMe devices
[Subsystem1]
  NQN nqn.2016-06.io.spdk:cnode1
  Core 2
  Listen RDMA 1.1.1.80:4420<http://1.1.1.80:4420>
  AllowAnyHost Yes
  SN SPDK00000000000001
  Namespace Nvme0n1 1

# Namespaces backed by physical NVMe devices
[Subsystem2]
  NQN nqn.2016-06.io.spdk:cnode2
  Core 4
  Listen RDMA 1.1.2.80:4420<http://1.1.2.80:4420>
  AllowAnyHost Yes
  SN SPDK00000000000002
  Namespace Nvme1n1 1

On Tue, Sep 12, 2017 at 4:00 PM, Luse, Paul E <paul.e.luse(a)intel.com<mailto:paul.e.luse(a)intel.com>> wrote:
Yeah, thanks John!  FYI Ganesh I didn’t forget about you I’m at SNIA SDC this week.  Cool to see SPDK mentioned in lots of talks including a nice shout out from Sage (Ceph) on SPDK in Bluestore…

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Santhebachalli Ganesh
Sent: Tuesday, September 12, 2017 4:29 AM

To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK errors

John,
Thanks for looking into this.
Yes, this is similar.
One of the significant difference is ConnectX-4 NIC. My setup uses ConnectX-3 NIC.

I don't have access to my lab as it is being physically moved. Hope to get access this or next week.


On Tue, Sep 12, 2017 at 5:36 AM, Kariuki, John K <john.k.kariuki(a)intel.com<mailto:john.k.kariuki(a)intel.com>> wrote:
Ganesh
I am trying to reproduce the issue in my environment without any luck. My environment has 2 CPUs with 22 cores/socket, Mellanox ConnectX-4 Lx NICs, Ubuntu 17.04 with kernel 4.10 and OFED 4.1. I have tried several different QD above 16 (all the way to 256) and perf ran successfully. Here is the workload parameters I am using. Is this similar to what you’re doing to reproduce the issue?

./perf -q 32 -s 512 -w randread -t 30 -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.51.2 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'

./perf -q 64 -s 512 -w randread -t 30 -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.51.2 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'

./perf -q 64 -s 512 -w randread -t 30 -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.51.2 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'

./perf -q 128 -s 512 -w randread -t 30 -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.51.2 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'

./perf -q 256 -s 512 -w randread -t 30 -r 'trtype:RDMA adrfam:IPv4 traddr:192.168.51.2 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1'



From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Santhebachalli Ganesh
Sent: Tuesday, September 05, 2017 4:52 AM

To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK errors

Thanks.
Traveling. My lab machines are shutdown as lab is being moved this week.
When I land, will try to reach out.
On Tue, Sep 5, 2017 at 3:40 AM Luse, Paul E <paul.e.luse(a)intel.com<mailto:paul.e.luse(a)intel.com>> wrote:
Hi Ganesh,

So I have my hardware installed , the Chelsio cards below, and I believe I’ve got the drivers built and installed correctly and ethtool is showing Link Detected on both ends but I’m not having much luck testing the connection in any other way and its not connecting w/SPDK. I’m far from an expexrt in this area, if you have some time and want to see if you can help me get this working we can use this setup to troubleshoot yours.  Let me know… I’m probably done for today though

Thx
Paul

Chelsio Communications Inc T580-LP-CR Unified Wire Ethernet Controller

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Santhebachalli Ganesh
Sent: Friday, September 1, 2017 11:59 AM

To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK errors

Update:
Just in case...
Tried with CentOS 7 (Release: 7.3.1611) with both built in latest kernel (3.10.0-514.26.2.el7.x86_64),
and updated latest stable (4.12.10-1.el7.elrepo.x86_64) installed.

Similar behavior. ;(

Maybe, next is to try with MOFED drivers.

On Thu, Aug 31, 2017 at 3:49 PM, Santhebachalli Ganesh <svg.eid(a)gmail.com<mailto:svg.eid(a)gmail.com>> wrote:
Thanks!

Update:
Tried using IB mode instead of Eth on the Connectx-3 NIC.
No change in behavior. Similar behavior and errors observed.



On Thu, Aug 31, 2017 at 3:36 PM, Luse, Paul E <paul.e.luse(a)intel.com<mailto:paul.e.luse(a)intel.com>> wrote:
Well those are good steps… hopefully someone else will jump in as well. I will see if I can get my HW setup to repro over the long weekend and let ya know how it goes…

From: SPDK [mailto:spdk-bounces(a)lists.01.org<mailto:spdk-bounces(a)lists.01.org>] On Behalf Of Santhebachalli Ganesh
Sent: Wednesday, August 30, 2017 12:44 PM
To: Storage Performance Development Kit <spdk(a)lists.01.org<mailto:spdk(a)lists.01.org>>
Subject: Re: [SPDK] SPDK errors

Thanks.

I did change a few cables, and even targets. Assuming that this took care of insertion force statistically.
Could not do IB instead of Eth as my adapter does not support IB.

Also, the error messages are not consistent. That was a snapshot of one of the runs.
Then, there is also the older Connectx3 adapters (latest FW flashed) with newer kernels and latest SPDK/DPDK.

Seen the following:
>nvme_pcie.c:1910:nvme_pcie_qpair_process_completions: *ERROR*: cpl does not map to outstanding cmd
>bdev_nvme.c:1248:bdev_nvme_queue_cmd: *ERROR*: readv failed: rc = -12
>bdev.c: 511:spdk_bdev_finish: *ERROR*: bdev IO pool count is 65533 but should be 65536

On Tue, Aug 29, 2017 at 7:33 PM, Vladislav Bolkhovitin <vst(a)vlnb.net<mailto:vst(a)vlnb.net>> wrote:

Santhebachalli Ganesh wrote on 08/29/2017 10:14 AM:
> Folks,
> My name is Ganesh, and I am working on NVEMoF performance metrics using SPDK (and kernel).
> I would appreciate your expert insights.
>
> I am observing errors when QD on perf is increased above >=64 most of the
> times. Sometimes, even for <=16
> Errors are not consistent.
>
> Attached are some details.
>
> Please let me know if have any additional questions.
>
> Thanks.
> -Ganesh
>
> SPDK errors 1.txt
>
>
> Setup details:
> -- Some info on setup
> Same HW/SW on target and initiator.
>
> adminuser(a)dell730-80:~> hostnamectl
>    Static hostname: dell730-80
>          Icon name: computer-server
>            Chassis: server
>         Machine ID: b5abb0fe67afd04c59521c40599b3115
>            Boot ID: f825aa6338194338a6f80125caa836c7
>   Operating System: openSUSE Leap 42.3
>        CPE OS Name: cpe:/o:opensuse:leap:42.3
>             Kernel: Linux 4.12.8-1.g4d7933a-default
>       Architecture: x86-64
>
> adminuser(a)dell730-80:~> lscpu | grep -i socket
> Core(s) per socket:    12
> Socket(s):             2
>
> 2MB and/or 1GB huge pages set,
>
> Latest spdk/dpdk from respective GIT,
>
> compiled with RDMA flag,
>
> nvmf.conf file: (have played around with the values)
> reactor mask 0x5555
> AcceptorCore 2
> 1 - 3 Subsystems on cores 4,8,10
>
> adminuser(a)dell730-80:~> sudo gits/spdk/app/nvmf_tgt/nvmf_tgt -c gits/spdk/etc/spdk/nvmf.conf -p 6
>
> PCI, NVME cards (16GB)
> adminuser(a)dell730-80:~> sudo lspci | grep -i pmc
> 04:00.0 Non-Volatile memory controller: PMC-Sierra Inc. Device f117 (rev 06)
> 06:00.0 Non-Volatile memory controller: PMC-Sierra Inc. Device f117 (rev 06)
> 85:00.0 Non-Volatile memory controller: PMC-Sierra Inc. Device f117 (rev 06)
>
> Network cards: (latest associated FW from vendor)
> adminuser(a)dell730-80:~> sudo lspci | grep -i connect
> 05:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
>
> --- initiator cmd line
> sudo ./perf -q 32 -s 512 -w randread -t 30 -r 'trtype:RDMA adrfam:IPv4 traddr:1.1.1.80 trsvcid:4420 subnqn:nqn.2016-06.io.spdk:cnode1' -c 0x2
>
> --errors on stdout on target
> Aug 24 17:14:09 dell730-80 nvmf[38006]: nvme_pcie.c:1910:nvme_pcie_qpair_process_completions: *ERROR*: cpl does not map to outstanding cmd
> Aug 24 17:14:09 dell730-80 nvmf[38006]: nvme_qpair.c: 284:nvme_qpair_print_completion: *NOTICE*: SUCCESS (00/00) sqid:1 cid:201 cdw0:0 sqhd:0094 p:0 m:0 dnr:0
> Aug 24 17:14:09 dell730-80 nvmf[38006]: nvme_pcie.c:1910:nvme_pcie_qpair_process_completions: *ERROR*: cpl does not map to outstanding cmd
> Aug 24 17:14:09 dell730-80 nvmf[38006]: nvme_qpair.c: 284:nvme_qpair_print_completion: *NOTICE*: SUCCESS (00/00) sqid:1 cid:201 cdw0:0 sqhd:0094 p:0 m:0 dnr:0
> Aug 24 17:14:09 dell730-80 nvmf[38006]: nvme_pcie.c:1910:nvme_pcie_qpair_process_completions: *ERROR*: cpl does not map to outstanding cmd
> Aug 24 17:14:09 dell730-80 nvmf[38006]: nvme_qpair.c: 284:nvme_qpair_print_completion: *NOTICE*: SUCCESS (00/00) sqid:1 cid:198 cdw0:0 sqhd:0094 p:0 m:0 dnr:0
> Aug 24 17:14:09 dell730-80 nvmf[38006]: nvme_pcie.c:1910:nvme_pcie_qpair_process_completions: *ERROR*: cpl does not map to outstanding cmd
> Aug 24 17:14:09 dell730-80 nvmf[38006]: nvme_qpair.c: 284:nvme_qpair_print_completion: *NOTICE*: SUCCESS (00/00) sqid:1 cid:222 cdw0:0 sqhd:0094 p:0 m:0 dnr:0
> Aug 24 17:14:09 dell730-80 nvmf[38006]: nvme_pcie.c:1910:nvme_pcie_qpair_process_completions: *ERROR*: cpl does not map to outstanding cmd
> Aug 24 17:14:09 dell730-80 nvmf[38006]: nvme_qpair.c: 284:nvme_qpair_print_completion: *NOTICE*: SUCCESS (00/00) sqid:1 cid:222 cdw0:0 sqhd:0094 p:0 m:0 dnr:0
> Aug 24 17:14:09 dell730-80 nvmf[38006]: bdev_nvme.c:1248:bdev_nvme_queue_cmd: *ERROR*: readv failed: rc = -12
> Aug 24 17:14:09 dell730-80 nvmf[38006]: bdev_nvme.c:1248:bdev_nvme_queue_cmd: *ERROR*: readv failed: rc = -12
> Aug 24 17:14:09 dell730-80 nvmf[38006]: bdev_nvme.c:1248:bdev_nvme_queue_cmd: *ERROR*: readv failed: rc = -12
> Aug 24 17:14:09 dell730-80 nvmf[38006]: bdev_nvme.c:1248:bdev_nvme_queue_cmd: *ERROR*: readv failed: rc = -12
> Aug 24 17:14:09 dell730-80 nvmf[38006]: bdev_nvme.c:1248:bdev_nvme_queue_cmd: *ERROR*: readv failed: rc = -12
> Aug 24 17:14:13 dell730-80 nvmf[38006]: rdma.c:1622:spdk_nvmf_rdma_poll: *ERROR*: CQ error on CQ 0x7f8a3803cae0, Request 0x140231622050400 (12): transport retry counter exceeded
>
> --- errros seen on client
> nvme_rdma.c:1470:nvme_rdma_qpair_process_completions: *ERROR*: CQ error on Queue Pair 0x1fdb580, Response Index 33408520 (13): RNR retry counter exceeded
> nvme_rdma.c:1470:nvme_rdma_qpair_process_completions: *ERROR*: CQ error on Queue Pair 0x1fdb580, Response Index 33408016 (5): Work Request Flushed Error
> nvme_rdma.c:1470:nvme_rdma_qpair_process_completions: *ERROR*: CQ error on Queue Pair 0x1fdb580, Response Index 14 (5): Work Request Flushed Error
> nvme_rdma.c:1470:nvme_rdma_qpair_process_completions: *ERROR*: CQ error on Queue Pair 0x1fdb580, Response Index 15 (5): Work Request Flushed Error

It's, actually, might be HW errors, because retries supposed to be engaged only on
packets loss/corruptions. Might be bad or not too well inserted cables.

Vlad

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk

_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk


_______________________________________________
SPDK mailing list
SPDK(a)lists.01.org<mailto:SPDK(a)lists.01.org>
https://lists.01.org/mailman/listinfo/spdk




[-- Attachment #2: attachment.html --]
[-- Type: text/html, Size: 62164 bytes --]