NVME Subsystem Reset Question

NVME Subsystem Reset Question

[Jeffrey Lien]

Keith, Jens,
I have a question on what the state of the nvme device should be after a subsystem reset (ie. Writing "NVMe" to NSSR).   The spec says it should be in LTSSM detect state which seems to mean it no longer showing up on the PCI bus which requires a pci bus rescan to get it back.   Does that match your interpretation?   

If so, who's responsible for initiating the bus rescan to get the device back?  I didn't see anything in the driver code that would do this so I'm assuming it would be the App's (that issued the subsystem reset ioctl) responsibility.  Agree?  Or is there something else that should initiate the rescan?

Thanks

Jeff Lien
Linux Device Driver Development
Device Host Apps and Drivers
jeff.lien at hgst.com
o: 507-322-2416 (ext. 23-2416)
m: 507-273-9124

NVME Subsystem Reset Question

[Keith Busch]

On Wed, Oct 12, 2016@05:04:40PM +0000, Jeffrey Lien wrote:
> Keith, Jens,
> I have a question on what the state of the nvme device should be after a subsystem reset (ie. Writing "NVMe" to NSSR).   The spec says it should be in LTSSM detect state which seems to mean it no longer showing up on the PCI bus which requires a pci bus rescan to get it back.   Does that match your interpretation?
>
> If so, who's responsible for initiating the bus rescan to get the device back?  I didn't see anything in the driver code that would do this so I'm assuming it would be the App's (that issued the subsystem reset ioctl) responsibility.  Agree?  Or is there something else that should initiate the rescan?

A bus rescan shouldn't be necessary. The device is supposed to go
from detect to polling automatically if a receiver is detected,
then configuration, recovery and finally L0 without the kernel doing
anything. At least that's my understanding.

Since power is never lost, the controllers in the subsystem should have
CSTS.NSSRO set to 1 so the driver can observe this after the controller
completes link training.

It wouldn't make much sense to require the host do anything to initiate
a rescan to detect the pci device after a subsystem reset. Different
controllers in the subsystem could be connected to different hosts,
and you can't ensure that's coordinated from the host that initiated the
subsystem reset.

NVME Subsystem Reset Question

[Jeffrey Lien]

Keith,
We agree with your thoughts on the fact that nvme device should recover automatically without requiring a pci bus rescan, but are seeing behavior to the contrary  with our NVME device as well as an Intel 3600 drive we also have.   Both devices disappear of the pci and require a pci bus rescan to show back up after the subsystem reset is done.  

After reading thru the PCI Express spec, it seems like the LTSSM detect state is a transitional state that the device should eventually and automatically leave on its way back to L0 state.   Could you confirm that from your side?  We're just looking for evidence and confirmation from you and the nvme community that this is not correct or expected behavior so  we can convince our firmware team that they need to make a change.  

Thanks in advance for your input.

Jeff Lien

-----Original Message-----
From: Keith Busch [mailto:keith.busch@intel.com] 
Sent: Wednesday, October 12, 2016 12:34 PM
To: Jeffrey Lien
Cc: axboe at fb.com; linux-nvme at lists.infradead.org; David Darrington; Nathan Rabe
Subject: Re: NVME Subsystem Reset Question

On Wed, Oct 12, 2016@05:04:40PM +0000, Jeffrey Lien wrote:
> Keith, Jens,
> I have a question on what the state of the nvme device should be after a subsystem reset (ie. Writing "NVMe" to NSSR).   The spec says it should be in LTSSM detect state which seems to mean it no longer showing up on the PCI bus which requires a pci bus rescan to get it back.   Does that match your interpretation?
>
> If so, who's responsible for initiating the bus rescan to get the device back?  I didn't see anything in the driver code that would do this so I'm assuming it would be the App's (that issued the subsystem reset ioctl) responsibility.  Agree?  Or is there something else that should initiate the rescan?

A bus rescan shouldn't be necessary. The device is supposed to go from detect to polling automatically if a receiver is detected, then configuration, recovery and finally L0 without the kernel doing anything. At least that's my understanding.

Since power is never lost, the controllers in the subsystem should have CSTS.NSSRO set to 1 so the driver can observe this after the controller completes link training.

It wouldn't make much sense to require the host do anything to initiate a rescan to detect the pci device after a subsystem reset. Different controllers in the subsystem could be connected to different hosts, and you can't ensure that's coordinated from the host that initiated the subsystem reset.

NVME Subsystem Reset Question

[Keith Busch]

On Fri, Oct 14, 2016@05:05:00PM +0000, Jeffrey Lien wrote:
> We agree with your thoughts on the fact that nvme device should recover automatically without requiring a pci bus rescan, but are seeing behavior to the contrary  with our NVME device as well as an Intel 3600 drive we also have.   Both devices disappear of the pci and require a pci bus rescan to show back up after the subsystem reset is done.  

What kernel version are you using? Can you send me a dmesg of the event?

NVME Subsystem Reset Question

[Jeffrey Lien]

Keith,
We're using the 3.10.0-493 kernel from a Red hat 7.3 Beta install.  I believe the nvme inbox driver here was back ported from the 4.5 kernel release.   I've attached the dmesg output from both the intel and western digital device.  Both are very similar if not identical.  


Jeff Lien

-----Original Message-----
From: Keith Busch [mailto:keith.busch@intel.com] 
Sent: Friday, October 14, 2016 4:28 PM
To: Jeffrey Lien
Cc: axboe at fb.com; linux-nvme at lists.infradead.org; David Darrington; Nathan Rabe
Subject: Re: NVME Subsystem Reset Question

On Fri, Oct 14, 2016@05:05:00PM +0000, Jeffrey Lien wrote:
> We agree with your thoughts on the fact that nvme device should recover automatically without requiring a pci bus rescan, but are seeing behavior to the contrary  with our NVME device as well as an Intel 3600 drive we also have.   Both devices disappear of the pci and require a pci bus rescan to show back up after the subsystem reset is done.  

What kernel version are you using? Can you send me a dmesg of the event?
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: subsys-reset-dmesg.txt
URL: <http://lists.infradead.org/pipermail/linux-nvme/attachments/20161017/ee5043ed/attachment.txt>

NVME Subsystem Reset Question

[Keith Busch]

On Mon, Oct 17, 2016@02:42:56PM +0000, Jeffrey Lien wrote:
> We're using the 3.10.0-493 kernel from a Red hat 7.3 Beta install.  I believe the nvme inbox driver here was back ported from the 4.5 kernel release.   I've attached the dmesg output from both the intel and western digital device.  Both are very similar if not identical.  

It does appear the link is disabled when the driver's health check
runs. It may be possible it observes all 1's on a transient link state
change in the health check. I'm not sure right now of a safe way to
distinguish a permanently disabled device vs a transient condition with
the information available.

How are you sending a subsystem reset? On the Intel device, I assumed
you were using a D3600, but this looks like a P3600 based on the model
and firmware revision. That controller doesn't support subsystem reset,
so the ioctl method should have returned -ENOTTY.

NVME Subsystem Reset Question

[Jeffrey Lien]

I'm using the nvme cli to send the subsystem reset - "nvme subsystem-reset /dev/nvme0".


Jeff Lien

-----Original Message-----
From: Linux-nvme [mailto:linux-nvme-bounces@lists.infradead.org] On Behalf Of Keith Busch
Sent: Monday, October 17, 2016 11:21 AM
To: Jeffrey Lien
Cc: axboe at fb.com; Nathan Rabe; David Darrington; linux-nvme at lists.infradead.org
Subject: Re: NVME Subsystem Reset Question

On Mon, Oct 17, 2016@02:42:56PM +0000, Jeffrey Lien wrote:
> We're using the 3.10.0-493 kernel from a Red hat 7.3 Beta install.  I believe the nvme inbox driver here was back ported from the 4.5 kernel release.   I've attached the dmesg output from both the intel and western digital device.  Both are very similar if not identical.  

It does appear the link is disabled when the driver's health check runs. It may be possible it observes all 1's on a transient link state change in the health check. I'm not sure right now of a safe way to distinguish a permanently disabled device vs a transient condition with the information available.

How are you sending a subsystem reset? On the Intel device, I assumed you were using a D3600, but this looks like a P3600 based on the model and firmware revision. That controller doesn't support subsystem reset, so the ioctl method should have returned -ENOTTY.

_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme

NVME Subsystem Reset Question

[Jeffrey Lien]

Keith,
After reading over the LTSSM detect state description in the PCI spec, it certainly appears like the link recovery should be automatic.  

Are you thinking the surprise removal checking by the driver is causing the link to be disabled - until a bus scan is done?  If so, maybe it would be worth putting in a retry of the call to nvme_dev_resume (after a short delay) in nvme_remove_dead_ctrl before making the call to pci_stop_and_remove_bus_device_locked.  I'll do some experimenting with this to see if it works and let you know what I find.


Jeff Lien

-----Original Message-----
From: Linux-nvme [mailto:linux-nvme-bounces@lists.infradead.org] On Behalf Of Keith Busch
Sent: Monday, October 17, 2016 11:21 AM
To: Jeffrey Lien
Cc: axboe at fb.com; Nathan Rabe; David Darrington; linux-nvme at lists.infradead.org
Subject: Re: NVME Subsystem Reset Question

On Mon, Oct 17, 2016@02:42:56PM +0000, Jeffrey Lien wrote:
> We're using the 3.10.0-493 kernel from a Red hat 7.3 Beta install.  I believe the nvme inbox driver here was back ported from the 4.5 kernel release.   I've attached the dmesg output from both the intel and western digital device.  Both are very similar if not identical.  

It does appear the link is disabled when the driver's health check runs. It may be possible it observes all 1's on a transient link state change in the health check. I'm not sure right now of a safe way to distinguish a permanently disabled device vs a transient condition with the information available.

How are you sending a subsystem reset? On the Intel device, I assumed you were using a D3600, but this looks like a P3600 based on the model and firmware revision. That controller doesn't support subsystem reset, so the ioctl method should have returned -ENOTTY.

_______________________________________________
Linux-nvme mailing list
Linux-nvme at lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-nvme