[SPDK] Re: NVMe hotplug for RDMA and TCP transports

[Howell, Seth <seth.howell at intel.com>]

[-- Attachment #1: Type: text/plain, Size: 2888 bytes --]

Hi Andrey,

Typically when we refer to hotplug (removal) in fabrics transports, we are talking about the target side of the connection suddenly disconnecting the admin and I/O qpairs by the target side of the connection. This definition of hotplug is already supported in the NVMe initiator. If your definition of hotplug is something different, please correct me so that I can better answer your question.

In RDMA for example, when we receive a disconnect event on the admin qpair for a given controller, we mark that controller as failed and fail up all I/O corresponding to I/O qpairs on that controller. Then subsequent calls to either submit I/O or process completions on any qpair associated with that controller return -ENXIO indicating to the initiator application that the drive has been failed by the target side.
There are a couple of reasons that could happen:
1. The actual drive itself has been hotplugged from the target application (i.e. nvme pcie hotplug on the target side)
2. There wsa some network event that caused the target application to disconnect (NIC failure, RDMA error, etc)

Because there are multiple reasons we could receive a "hotplug" event from the target application we leave it up to the initator application to decide what they want to do with this. Either destroy the controller from the initiator side, try reconnecting to the controller from the same TRID or attempting to connect to the controller from a different TRID (something like target side port failover).

In terms of hotplug insertion, I assume that would mean you want the initiator to automatically connect to a target subsystem that can be presented at any point in time during the running of the application. There isn't a specific driver level implementation of this feature for fabrics controllers, I think mostly because it would be very easy to implement and customize this functionality at the application layer. For example, one could periodically call discover on the targets they want to connect to and when new controllers/subsystems appear, connect to them at that time.

I hope that this answers your question. Please let me know if I am talking about a different definition of hotplug than the one you are using.

Thanks,

Seth



-----Original Message-----
From: Andrey Kuzmin <andrey.v.kuzmin(a)gmail.com> 
Sent: Friday, May 22, 2020 1:47 AM
To: Storage Performance Development Kit <spdk(a)lists.01.org>
Subject: [SPDK] NVMe hotplug for RDMA and TCP transports

Hi team,

is NVMe hotplug functionality as implemented limited to PCIe transport or does it also work for other transports? If it's currently PCIe only, are there any plans to extend the support to RDMA/TCP?

Thanks,
Andrey
_______________________________________________
SPDK mailing list -- spdk(a)lists.01.org
To unsubscribe send an email to spdk-leave(a)lists.01.org