[PATCH] nvme: Add abrupt shutdown support

[PATCH] nvme: Add abrupt shutdown support

[shiva.linuxworks]

From: Shivamurthy Shastri <sshivamurthy@micron.com>

Enabling the abrupt shutdown support. In this shutdown type host does
not need to send Delete I/O Submission Queue and Delete I/O Completion
queue commands to the device.

Signed-off-by: Shivamurthy Shastri <sshivamurthy@micron.com>
---
 drivers/nvme/host/core.c | 11 ++++++++++-
 drivers/nvme/host/nvme.h |  2 ++
 drivers/nvme/host/pci.c  |  5 +++--
 3 files changed, 15 insertions(+), 3 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 80c656dcbbac..c1d1bd6a03af 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -40,6 +40,11 @@ module_param_named(io_timeout, nvme_io_timeout, uint, 0644);
 MODULE_PARM_DESC(io_timeout, "timeout in seconds for I/O");
 EXPORT_SYMBOL_GPL(nvme_io_timeout);
 
+bool shutdown_type;
+EXPORT_SYMBOL_GPL(shutdown_type);
+module_param(shutdown_type, bool, 0644);
+MODULE_PARM_DESC(shutdown_type, "Type of controller shutdown");
+
 static unsigned char shutdown_timeout = 5;
 module_param(shutdown_timeout, byte, 0644);
 MODULE_PARM_DESC(shutdown_timeout, "timeout in seconds for controller shutdown");
@@ -2166,7 +2171,11 @@ int nvme_shutdown_ctrl(struct nvme_ctrl *ctrl)
 	int ret;
 
 	ctrl->ctrl_config &= ~NVME_CC_SHN_MASK;
-	ctrl->ctrl_config |= NVME_CC_SHN_NORMAL;
+
+	if (shutdown_type)
+		ctrl->ctrl_config |= NVME_CC_SHN_ABRUPT;
+	else
+		ctrl->ctrl_config |= NVME_CC_SHN_NORMAL;
 
 	ret = ctrl->ops->reg_write32(ctrl, NVME_REG_CC, ctrl->ctrl_config);
 	if (ret)
diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index 75420ceacc10..af14ee1cd7d8 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -23,6 +23,8 @@
 extern unsigned int nvme_io_timeout;
 #define NVME_IO_TIMEOUT	(nvme_io_timeout * HZ)
 
+extern bool shutdown_type;
+
 extern unsigned int admin_timeout;
 #define NVME_ADMIN_TIMEOUT	(admin_timeout * HZ)
 
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index d3c5086673bc..6a78233d9645 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2474,13 +2474,14 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown)
 	 * Give the controller a chance to complete all entered requests if
 	 * doing a safe shutdown.
 	 */
-	if (!dead && shutdown && freeze)
+	if (!dead && shutdown && !shutdown_type && freeze)
 		nvme_wait_freeze_timeout(&dev->ctrl, NVME_IO_TIMEOUT);
 
 	nvme_stop_queues(&dev->ctrl);
 
 	if (!dead && dev->ctrl.queue_count > 0) {
-		nvme_disable_io_queues(dev);
+		if (!shutdown_type)
+			nvme_disable_io_queues(dev);
 		nvme_disable_admin_queue(dev, shutdown);
 	}
 	nvme_suspend_io_queues(dev);
-- 
2.25.1

Re: [PATCH] nvme: Add abrupt shutdown support

[Christoph Hellwig]

On Mon, Jul 05, 2021 at 10:11:57AM +0000, shiva.linuxworks@gmail.com wrote:
> From: Shivamurthy Shastri <sshivamurthy@micron.com>
> 
> Enabling the abrupt shutdown support. In this shutdown type host does
> not need to send Delete I/O Submission Queue and Delete I/O Completion
> queue commands to the device.

Why?

RE: [EXT] Re: [PATCH] nvme: Add abrupt shutdown support

[Shivamurthy Shastri (sshivamurthy)]

Micron Confidential

Hi Christoph,

> 
> 
> On Mon, Jul 05, 2021 at 10:11:57AM +0000, shiva.linuxworks@gmail.com wrote:
> > From: Shivamurthy Shastri <sshivamurthy@micron.com>
> >
> > Enabling the abrupt shutdown support. In this shutdown type host does
> > not need to send Delete I/O Submission Queue and Delete I/O Completion
> > queue commands to the device.
> 
> Why?

This is to achieve faster shutdown at the cost of data in-flight.

As per the spec Deleting Submission queues and Completion queues are not explicitly 
mentioned for performing the abrupt shutdown (CC.SHN=10b).

Thanks,
Shiva


Micron Confidential

Re: [EXT] Re: [PATCH] nvme: Add abrupt shutdown support

[Christoph Hellwig]

On Mon, Jul 05, 2021 at 10:40:21AM +0000, Shivamurthy Shastri (sshivamurthy) wrote:
> Micron Confidential
> 
> Hi Christoph,
> 
> > 
> > 
> > On Mon, Jul 05, 2021 at 10:11:57AM +0000, shiva.linuxworks@gmail.com wrote:
> > > From: Shivamurthy Shastri <sshivamurthy@micron.com>
> > >
> > > Enabling the abrupt shutdown support. In this shutdown type host does
> > > not need to send Delete I/O Submission Queue and Delete I/O Completion
> > > queue commands to the device.
> > 
> > Why?
> 
> This is to achieve faster shutdown at the cost of data in-flight.

What is the actual use case, and why would a module paramter be a good
interface?

RE: [EXT] Re: [PATCH] nvme: Add abrupt shutdown support

[Shivamurthy Shastri (sshivamurthy)]

Micron Confidential

> >
> > Hi Christoph,
> >
> > >
> > >
> > > On Mon, Jul 05, 2021 at 10:11:57AM +0000, shiva.linuxworks@gmail.com
> wrote:
> > > > From: Shivamurthy Shastri <sshivamurthy@micron.com>
> > > >
> > > > Enabling the abrupt shutdown support. In this shutdown type host does
> > > > not need to send Delete I/O Submission Queue and Delete I/O Completion
> > > > queue commands to the device.
> > >
> > > Why?
> >
> > This is to achieve faster shutdown at the cost of data in-flight.
> 
> What is the actual use case, and why would a module paramter be a good
> interface?

For systems with passive electronic components to handle power-loss/battery-removal,
CC.SHN=10b will help to achieve safer shutdown with relatively lower power back-up 
requirements (at lower cost).

The module parameter gives flexibility to the host to decide shutdown type even during
boot failure. 

Thanks,
Shiva

Micron Confidential

Re: [EXT] Re: [PATCH] nvme: Add abrupt shutdown support

[Christoph Hellwig]

On Mon, Jul 05, 2021 at 11:15:51AM +0000, Shivamurthy Shastri (sshivamurthy) wrote:
> For systems with passive electronic components to handle power-loss/battery-removal,
> CC.SHN=10b will help to achieve safer shutdown with relatively lower power back-up 
> requirements (at lower cost).
> 
> The module parameter gives flexibility to the host to decide shutdown type even during
> boot failure. 

If you have a device that always prefers an abrupt please introduce a TP
in the NVMe technical working group so that the hosts automatically
do the right thing.  Requiring sysadmin intervention using a global
kernel parameter is not the way to do this.

Re: [EXT] Re: [PATCH] nvme: Add abrupt shutdown support

[Keith Busch]

On Mon, Jul 05, 2021 at 01:53:49PM +0200, Christoph Hellwig wrote:
> On Mon, Jul 05, 2021 at 11:15:51AM +0000, Shivamurthy Shastri (sshivamurthy) wrote:
> > For systems with passive electronic components to handle power-loss/battery-removal,
> > CC.SHN=10b will help to achieve safer shutdown with relatively lower power back-up 
> > requirements (at lower cost).
> > 
> > The module parameter gives flexibility to the host to decide shutdown type even during
> > boot failure. 
> 
> If you have a device that always prefers an abrupt please introduce a TP
> in the NVMe technical working group so that the hosts automatically
> do the right thing.  Requiring sysadmin intervention using a global
> kernel parameter is not the way to do this.

Is it the device that wants an abrupt shutdown or the platform? If the
platform's power is running on limited back-up supply, and could inform
the kernel's power management subsystem of this, then a driver could use
that to determine if the quick shutdown is appropriate.

RE: [EXT] Re: [PATCH] nvme: Add abrupt shutdown support

[Shivamurthy Shastri (sshivamurthy)]

Micron Confidential

Hi,

> 
> 
> On Mon, Jul 05, 2021 at 01:53:49PM +0200, Christoph Hellwig wrote:
> > On Mon, Jul 05, 2021 at 11:15:51AM +0000, Shivamurthy Shastri
> (sshivamurthy) wrote:
> > > For systems with passive electronic components to handle power-
> loss/battery-removal,
> > > CC.SHN=10b will help to achieve safer shutdown with relatively lower power
> back-up
> > > requirements (at lower cost).
> > >
> > > The module parameter gives flexibility to the host to decide shutdown type
> even during
> > > boot failure.
> >
> > If you have a device that always prefers an abrupt please introduce a TP
> > in the NVMe technical working group so that the hosts automatically
> > do the right thing.  Requiring sysadmin intervention using a global
> > kernel parameter is not the way to do this.

The abrupt shutdown notification is already part of NVMe Specification.

> 
> Is it the device that wants an abrupt shutdown or the platform? If the
> platform's power is running on limited back-up supply, and could inform
> the kernel's power management subsystem of this, then a driver could use
> that to determine if the quick shutdown is appropriate.

Yes, the platform is running on limited back-up supply. In this case, abrupt shutdown
option will help to achieve shutdown with relatively low back-up requirements.

Please let me know if you want to include the abrupt shutdown as part of 

static const struct dev_pm_ops nvme_dev_pm_ops = {
	.suspend	= nvme_suspend,
               .....


Thanks,
Shiva



Micron Confidential

Re: [EXT] Re: [PATCH] nvme: Add abrupt shutdown support

[Keith Busch]

On Tue, Jul 06, 2021 at 08:14:45AM +0000, Shivamurthy Shastri (sshivamurthy) wrote:
> > Is it the device that wants an abrupt shutdown or the platform? If the
> > platform's power is running on limited back-up supply, and could inform
> > the kernel's power management subsystem of this, then a driver could use
> > that to determine if the quick shutdown is appropriate.
> 
> Yes, the platform is running on limited back-up supply. In this case, abrupt shutdown
> option will help to achieve shutdown with relatively low back-up requirements.
> 
> Please let me know if you want to include the abrupt shutdown as part of 
> 
> static const struct dev_pm_ops nvme_dev_pm_ops = {
> 	.suspend	= nvme_suspend,
>                .....

Are you suggesting creating a new pm callback just for this? I don't
think that's necessary.

I was considering just adding a bit to the pm_suspend_global_flags,
something like the patch below. You just have to fill in the platform
specific parts to set the flag.

---
diff --git a/include/linux/suspend.h b/include/linux/suspend.h
index 8af13ba60c7e..f43c0b2313d0 100644
--- a/include/linux/suspend.h
+++ b/include/linux/suspend.h
@@ -210,9 +210,10 @@ extern int suspend_valid_only_mem(suspend_state_t state);
 
 extern unsigned int pm_suspend_global_flags;
 
-#define PM_SUSPEND_FLAG_FW_SUSPEND	BIT(0)
-#define PM_SUSPEND_FLAG_FW_RESUME	BIT(1)
-#define PM_SUSPEND_FLAG_NO_PLATFORM	BIT(2)
+#define PM_SUSPEND_FLAG_FW_SUSPEND		BIT(0)
+#define PM_SUSPEND_FLAG_FW_RESUME		BIT(1)
+#define PM_SUSPEND_FLAG_NO_PLATFORM		BIT(2)
+#define PM_SUSPEND_FLAG_POWER_LOSS_IMMINENT	BIT(3)
 
 static inline void pm_suspend_clear_flags(void)
 {
@@ -234,6 +235,11 @@ static inline void pm_set_suspend_no_platform(void)
 	pm_suspend_global_flags |= PM_SUSPEND_FLAG_NO_PLATFORM;
 }
 
+static inline void pm_set_power_loss_imminent(void)
+{
+	pm_suspend_global_flags |= PM_SUSPEND_FLAG_POWER_LOSS_IMMINENT;
+}
+
 /**
  * pm_suspend_via_firmware - Check if platform firmware will suspend the system.
  *
@@ -291,6 +297,21 @@ static inline bool pm_suspend_no_platform(void)
 	return !!(pm_suspend_global_flags & PM_SUSPEND_FLAG_NO_PLATFORM);
 }
 
+/**
+ * pm_power_loss_imminent - Check if platform is running on limited backup
+ * 			    power source
+ *
+ * To be called during system-wide power management transitions to sleep states.
+ *
+ * Return 'true' if power loss may be imminent due to platform running on
+ * limited backup supply. If set during a shutdown, drivers should use any
+ * available shortcuts to prepare their device for abrupt power loss.
+ */
+static inline bool pm_power_loss_imminent(void)
+{
+	return !!(pm_suspend_global_flags & PM_SUSPEND_FLAG_POWER_LOSS_IMMINENT);
+}
+
 /* Suspend-to-idle state machnine. */
 enum s2idle_states {
 	S2IDLE_STATE_NONE,      /* Not suspended/suspending. */
--