[PATCH 0/2] nvme APST fixes

[PATCH 0/2] nvme APST fixes

[Andy Lutomirski]

This should fix all known regressions.

Andy Lutomirski (2):
  nvme: Wait at least 6000ms before entering the deepest idle state
  nvme: Quirk APST on Intel 600P/P3100 devices

 drivers/nvme/host/core.c | 38 +++++++++++++++++++++++++++++++-------
 drivers/nvme/host/pci.c  |  2 ++
 2 files changed, 33 insertions(+), 7 deletions(-)

-- 
2.9.4

[PATCH 0/2] nvme APST fixes

[Andy Lutomirski]

This should fix all known regressions.

Andy Lutomirski (2):
  nvme: Wait at least 6000ms before entering the deepest idle state
  nvme: Quirk APST on Intel 600P/P3100 devices

 drivers/nvme/host/core.c | 38 +++++++++++++++++++++++++++++++-------
 drivers/nvme/host/pci.c  |  2 ++
 2 files changed, 33 insertions(+), 7 deletions(-)

-- 
2.9.4

[PATCH 1/2] nvme: Wait at least 6000ms before entering the deepest idle state

[Andy Lutomirski]

This should at least make vendors less nervous about Linux's APST
policy.  I'm not aware of any concrete bugs it would fix (although I
was hoping it would fix the Samsung/Dell quirk).

Cc: stable@vger.kernel.org # v4.11
Cc: Kai-Heng Feng <kai.heng.feng@canonical.com>
Cc: Mario Limonciello <mario_limonciello@dell.com>
Signed-off-by: Andy Lutomirski <luto@kernel.org>
---
 drivers/nvme/host/core.c | 38 +++++++++++++++++++++++++++++++-------
 1 file changed, 31 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index d5e0906262ea..381e9f813385 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1325,13 +1325,7 @@ static void nvme_configure_apst(struct nvme_ctrl *ctrl)
 	/*
 	 * APST (Autonomous Power State Transition) lets us program a
 	 * table of power state transitions that the controller will
-	 * perform automatically.  We configure it with a simple
-	 * heuristic: we are willing to spend at most 2% of the time
-	 * transitioning between power states.  Therefore, when running
-	 * in any given state, we will enter the next lower-power
-	 * non-operational state after waiting 50 * (enlat + exlat)
-	 * microseconds, as long as that state's total latency is under
-	 * the requested maximum latency.
+	 * perform automatically.
 	 *
 	 * We will not autonomously enter any non-operational state for
 	 * which the total latency exceeds ps_max_latency_us.  Users
@@ -1405,9 +1399,39 @@ static void nvme_configure_apst(struct nvme_ctrl *ctrl)
 			/*
 			 * This state is good.  Use it as the APST idle
 			 * target for higher power states.
+			 *
+			 * Intel RSTe supposedly uses the following algorithm:
+			 * 60ms delay to transition to the first
+			 * non-operational state and 1000*exlat to each
+			 * additional state.  This is problematic.  60ms is
+			 * too short if the first non-operational state has
+			 * high latency, and 1000*exlat into a state is
+			 * absurdly slow.  (exlat=22ms seems typical for the
+			 * deepest state.  A delay of 22 seconds to enter that
+			 * state means that it will almost never be entered at
+			 * all, wasting power and, worse, turning otherwise
+			 * easy-to-detect hardware/firmware bugs into sporadic
+			 * problems.
+			 *
+			 * Linux is willing to spend at most 2% of the time
+			 * transitioning between power states.  Therefore,
+			 * when running in any given state, we will enter the
+			 * next lower-power non-operational state after
+			 * waiting 50 * (enlat + exlat) microseconds, as long
+			 * as that state's total latency is under the
+			 * requested maximum latency.
 			 */
 			transition_ms = total_latency_us + 19;
 			do_div(transition_ms, 20);
+
+			/*
+			 * Some vendors have expressed nervousness about
+			 * entering the deepest state after less than six
+			 * seconds.
+			 */
+			if (state == ctrl->npss && transition_ms < 6000)
+				transition_ms = 6000;
+
 			if (transition_ms > (1 << 24) - 1)
 				transition_ms = (1 << 24) - 1;
 
-- 
2.9.4

[PATCH 1/2] nvme: Wait at least 6000ms before entering the deepest idle state

[Andy Lutomirski]

This should at least make vendors less nervous about Linux's APST
policy.  I'm not aware of any concrete bugs it would fix (although I
was hoping it would fix the Samsung/Dell quirk).

Cc: stable at vger.kernel.org # v4.11
Cc: Kai-Heng Feng <kai.heng.feng at canonical.com>
Cc: Mario Limonciello <mario_limonciello at dell.com>
Signed-off-by: Andy Lutomirski <luto at kernel.org>
---
 drivers/nvme/host/core.c | 38 +++++++++++++++++++++++++++++++-------
 1 file changed, 31 insertions(+), 7 deletions(-)

diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index d5e0906262ea..381e9f813385 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -1325,13 +1325,7 @@ static void nvme_configure_apst(struct nvme_ctrl *ctrl)
 	/*
 	 * APST (Autonomous Power State Transition) lets us program a
 	 * table of power state transitions that the controller will
-	 * perform automatically.  We configure it with a simple
-	 * heuristic: we are willing to spend at most 2% of the time
-	 * transitioning between power states.  Therefore, when running
-	 * in any given state, we will enter the next lower-power
-	 * non-operational state after waiting 50 * (enlat + exlat)
-	 * microseconds, as long as that state's total latency is under
-	 * the requested maximum latency.
+	 * perform automatically.
 	 *
 	 * We will not autonomously enter any non-operational state for
 	 * which the total latency exceeds ps_max_latency_us.  Users
@@ -1405,9 +1399,39 @@ static void nvme_configure_apst(struct nvme_ctrl *ctrl)
 			/*
 			 * This state is good.  Use it as the APST idle
 			 * target for higher power states.
+			 *
+			 * Intel RSTe supposedly uses the following algorithm:
+			 * 60ms delay to transition to the first
+			 * non-operational state and 1000*exlat to each
+			 * additional state.  This is problematic.  60ms is
+			 * too short if the first non-operational state has
+			 * high latency, and 1000*exlat into a state is
+			 * absurdly slow.  (exlat=22ms seems typical for the
+			 * deepest state.  A delay of 22 seconds to enter that
+			 * state means that it will almost never be entered at
+			 * all, wasting power and, worse, turning otherwise
+			 * easy-to-detect hardware/firmware bugs into sporadic
+			 * problems.
+			 *
+			 * Linux is willing to spend at most 2% of the time
+			 * transitioning between power states.  Therefore,
+			 * when running in any given state, we will enter the
+			 * next lower-power non-operational state after
+			 * waiting 50 * (enlat + exlat) microseconds, as long
+			 * as that state's total latency is under the
+			 * requested maximum latency.
 			 */
 			transition_ms = total_latency_us + 19;
 			do_div(transition_ms, 20);
+
+			/*
+			 * Some vendors have expressed nervousness about
+			 * entering the deepest state after less than six
+			 * seconds.
+			 */
+			if (state == ctrl->npss && transition_ms < 6000)
+				transition_ms = 6000;
+
 			if (transition_ms > (1 << 24) - 1)
 				transition_ms = (1 << 24) - 1;
 
-- 
2.9.4

Re: [PATCH 1/2] nvme: Wait at least 6000ms before entering the deepest idle state

[Christoph Hellwig]

On Wed, May 24, 2017 at 03:06:30PM -0700, Andy Lutomirski wrote:
> This should at least make vendors less nervous about Linux's APST
> policy.  I'm not aware of any concrete bugs it would fix (although I
> was hoping it would fix the Samsung/Dell quirk).
> 
> Cc: stable@vger.kernel.org # v4.11

Just following a somewhat odd and arbitray policy from another driver
that doesn't fix anything by itself certainly isn't stable material.

And to be honest I'm not sure what applying it will gain us at all.

[PATCH 1/2] nvme: Wait at least 6000ms before entering the deepest idle state

[Christoph Hellwig]

On Wed, May 24, 2017@03:06:30PM -0700, Andy Lutomirski wrote:
> This should at least make vendors less nervous about Linux's APST
> policy.  I'm not aware of any concrete bugs it would fix (although I
> was hoping it would fix the Samsung/Dell quirk).
> 
> Cc: stable at vger.kernel.org # v4.11

Just following a somewhat odd and arbitray policy from another driver
that doesn't fix anything by itself certainly isn't stable material.

And to be honest I'm not sure what applying it will gain us at all.

Re: [PATCH 1/2] nvme: Wait at least 6000ms before entering the deepest idle state

[Andy Lutomirski]

On Fri, May 26, 2017 at 1:52 AM, Christoph Hellwig <hch@lst.de> wrote:
> On Wed, May 24, 2017 at 03:06:30PM -0700, Andy Lutomirski wrote:
>> This should at least make vendors less nervous about Linux's APST
>> policy.  I'm not aware of any concrete bugs it would fix (although I
>> was hoping it would fix the Samsung/Dell quirk).
>>
>> Cc: stable@vger.kernel.org # v4.11
>
> Just following a somewhat odd and arbitray policy from another driver
> that doesn't fix anything by itself certainly isn't stable material.

Fair enough.  The Cc: stable should certainly be dropped.

>
> And to be honest I'm not sure what applying it will gain us at all.

I'd be fine with skipping this patch entirely at least until we find
some evidence that it solves a problem instead.

[PATCH 1/2] nvme: Wait at least 6000ms before entering the deepest idle state

[Andy Lutomirski]

On Fri, May 26, 2017@1:52 AM, Christoph Hellwig <hch@lst.de> wrote:
> On Wed, May 24, 2017@03:06:30PM -0700, Andy Lutomirski wrote:
>> This should at least make vendors less nervous about Linux's APST
>> policy.  I'm not aware of any concrete bugs it would fix (although I
>> was hoping it would fix the Samsung/Dell quirk).
>>
>> Cc: stable at vger.kernel.org # v4.11
>
> Just following a somewhat odd and arbitray policy from another driver
> that doesn't fix anything by itself certainly isn't stable material.

Fair enough.  The Cc: stable should certainly be dropped.

>
> And to be honest I'm not sure what applying it will gain us at all.

I'd be fine with skipping this patch entirely at least until we find
some evidence that it solves a problem instead.

Re: [PATCH 1/2] nvme: Wait at least 6000ms before entering the deepest idle state

[Linus Torvalds]

On Sat, May 27, 2017 at 9:08 AM, Andy Lutomirski <luto@kernel.org> wrote:
> On Fri, May 26, 2017 at 1:52 AM, Christoph Hellwig <hch@lst.de> wrote:
>>
>> Just following a somewhat odd and arbitray policy from another driver
>> that doesn't fix anything by itself certainly isn't stable material.
>
> I'd be fine with skipping this patch entirely at least until we find
> some evidence that it solves a problem instead.

It's certainly worth waiting for confirmation that it actually changes
anything, but I did want to pipe up to say that "following a somewhat
odd and arbitrary policy from another driver" is not wrong per se.

In fact, on the power management we pretty much had to do that, simply
because that "odd and arbitrary policy" (Windows behavior) was the
only thing that had ever been tested by anybody. Same goes for a lot
of PCI subsystem behavior etc.

So it's not wrong per se to just emulate others' behaviors. But yes,
we should have some reason for doing so, not just doing so blindly.

           Linus

[PATCH 1/2] nvme: Wait at least 6000ms before entering the deepest idle state

[Linus Torvalds]

On Sat, May 27, 2017@9:08 AM, Andy Lutomirski <luto@kernel.org> wrote:
> On Fri, May 26, 2017@1:52 AM, Christoph Hellwig <hch@lst.de> wrote:
>>
>> Just following a somewhat odd and arbitray policy from another driver
>> that doesn't fix anything by itself certainly isn't stable material.
>
> I'd be fine with skipping this patch entirely at least until we find
> some evidence that it solves a problem instead.

It's certainly worth waiting for confirmation that it actually changes
anything, but I did want to pipe up to say that "following a somewhat
odd and arbitrary policy from another driver" is not wrong per se.

In fact, on the power management we pretty much had to do that, simply
because that "odd and arbitrary policy" (Windows behavior) was the
only thing that had ever been tested by anybody. Same goes for a lot
of PCI subsystem behavior etc.

So it's not wrong per se to just emulate others' behaviors. But yes,
we should have some reason for doing so, not just doing so blindly.

           Linus

[PATCH 2/2] nvme: Quirk APST on Intel 600P/P3100 devices

[Andy Lutomirski]

They have known firmware bugs.  A fix is apparently in the works --
once fixed firmware is available, someone from Intel (Hi, Keith!)
can adjust the quirk accordingly.

Cc: stable@vger.kernel.org # v4.11
Cc: Kai-Heng Feng <kai.heng.feng@canonical.com>
Cc: Mario Limonciello <mario_limonciello@dell.com>
Signed-off-by: Andy Lutomirski <luto@kernel.org>
---
 drivers/nvme/host/pci.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index fed803232edc..24cf32afaa34 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2288,6 +2288,8 @@ static const struct pci_device_id nvme_id_table[] = {
 	{ PCI_VDEVICE(INTEL, 0x0a54),
 		.driver_data = NVME_QUIRK_STRIPE_SIZE |
 				NVME_QUIRK_DEALLOCATE_ZEROES, },
+	{ PCI_VDEVICE(INTEL, 0xf1a5),	/* Intel 600P/P3100 */
+		.driver_data = NVME_QUIRK_NO_DEEPEST_PS },
 	{ PCI_VDEVICE(INTEL, 0x5845),	/* Qemu emulated controller */
 		.driver_data = NVME_QUIRK_IDENTIFY_CNS, },
 	{ PCI_DEVICE(0x1c58, 0x0003),	/* HGST adapter */
-- 
2.9.4

[PATCH 2/2] nvme: Quirk APST on Intel 600P/P3100 devices

[Andy Lutomirski]

They have known firmware bugs.  A fix is apparently in the works --
once fixed firmware is available, someone from Intel (Hi, Keith!)
can adjust the quirk accordingly.

Cc: stable at vger.kernel.org # v4.11
Cc: Kai-Heng Feng <kai.heng.feng at canonical.com>
Cc: Mario Limonciello <mario_limonciello at dell.com>
Signed-off-by: Andy Lutomirski <luto at kernel.org>
---
 drivers/nvme/host/pci.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index fed803232edc..24cf32afaa34 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2288,6 +2288,8 @@ static const struct pci_device_id nvme_id_table[] = {
 	{ PCI_VDEVICE(INTEL, 0x0a54),
 		.driver_data = NVME_QUIRK_STRIPE_SIZE |
 				NVME_QUIRK_DEALLOCATE_ZEROES, },
+	{ PCI_VDEVICE(INTEL, 0xf1a5),	/* Intel 600P/P3100 */
+		.driver_data = NVME_QUIRK_NO_DEEPEST_PS },
 	{ PCI_VDEVICE(INTEL, 0x5845),	/* Qemu emulated controller */
 		.driver_data = NVME_QUIRK_IDENTIFY_CNS, },
 	{ PCI_DEVICE(0x1c58, 0x0003),	/* HGST adapter */
-- 
2.9.4

Re: [PATCH 2/2] nvme: Quirk APST on Intel 600P/P3100 devices

[Christoph Hellwig]

Thanks,

applied to nvme-4.12

[PATCH 2/2] nvme: Quirk APST on Intel 600P/P3100 devices

[Christoph Hellwig]

Thanks,

applied to nvme-4.12

Re: [PATCH 2/2] nvme: Quirk APST on Intel 600P/P3100 devices

[Keith Busch]

On Wed, May 24, 2017 at 03:06:31PM -0700, Andy Lutomirski wrote:
> They have known firmware bugs.  A fix is apparently in the works --
> once fixed firmware is available, someone from Intel (Hi, Keith!)
> can adjust the quirk accordingly.

Here's the latest firmware with all the known fixes:

  https://downloadcenter.intel.com/download/26491?v=t

I am not 100% sure if this fixes the issue you're quirking, but I think
it falls under the issues the link initialization and ASPM comments in
the release notes.

[PATCH 2/2] nvme: Quirk APST on Intel 600P/P3100 devices

[Keith Busch]

On Wed, May 24, 2017@03:06:31PM -0700, Andy Lutomirski wrote:
> They have known firmware bugs.  A fix is apparently in the works --
> once fixed firmware is available, someone from Intel (Hi, Keith!)
> can adjust the quirk accordingly.

Here's the latest firmware with all the known fixes:

  https://downloadcenter.intel.com/download/26491?v=t

I am not 100% sure if this fixes the issue you're quirking, but I think
it falls under the issues the link initialization and ASPM comments in
the release notes.

Re: [PATCH 2/2] nvme: Quirk APST on Intel 600P/P3100 devices

[Andy Lutomirski]

On Tue, May 30, 2017 at 8:35 AM, Keith Busch <keith.busch@intel.com> wrote:
> On Wed, May 24, 2017 at 03:06:31PM -0700, Andy Lutomirski wrote:
>> They have known firmware bugs.  A fix is apparently in the works --
>> once fixed firmware is available, someone from Intel (Hi, Keith!)
>> can adjust the quirk accordingly.
>
> Here's the latest firmware with all the known fixes:
>
>   https://downloadcenter.intel.com/download/26491?v=t
>
> I am not 100% sure if this fixes the issue you're quirking, but I think
> it falls under the issues the link initialization and ASPM comments in
> the release notes.

Can you or someone at Intel supply a reliable way to distinguish a
fixed firmware from an unfixed firmware?  We have access to the "fr"
string, but I don't know what to do with it.

[PATCH 2/2] nvme: Quirk APST on Intel 600P/P3100 devices

[Andy Lutomirski]

On Tue, May 30, 2017@8:35 AM, Keith Busch <keith.busch@intel.com> wrote:
> On Wed, May 24, 2017@03:06:31PM -0700, Andy Lutomirski wrote:
>> They have known firmware bugs.  A fix is apparently in the works --
>> once fixed firmware is available, someone from Intel (Hi, Keith!)
>> can adjust the quirk accordingly.
>
> Here's the latest firmware with all the known fixes:
>
>   https://downloadcenter.intel.com/download/26491?v=t
>
> I am not 100% sure if this fixes the issue you're quirking, but I think
> it falls under the issues the link initialization and ASPM comments in
> the release notes.

Can you or someone at Intel supply a reliable way to distinguish a
fixed firmware from an unfixed firmware?  We have access to the "fr"
string, but I don't know what to do with it.

[PATCH 2/2] nvme: Quirk APST on Intel 600P/P3100 devices

[Nicholas Sielicki]

On Wed, May 31 06:54:51 PDT 2017, Andy Lutomirski wrote:
>On Tue, May 30, 2017@8:35 AM, Keith Busch <keith.busch@intel.com> wrote:
>> On Wed, May 24, 2017@03:06:31PM -0700, Andy Lutomirski wrote:
>>> They have known firmware bugs.  A fix is apparently in the works --
>>> once fixed firmware is available, someone from Intel (Hi, Keith!)
>>> can adjust the quirk accordingly.
>>
>> Here's the latest firmware with all the known fixes:
>>
>>   https://downloadcenter.intel.com/download/26491?v=t
>>
>> I am not 100% sure if this fixes the issue you're quirking, but I
>> think it falls under the issues the link initialization and ASPM
>> comments in the release notes.
>
>Can you or someone at Intel supply a reliable way to distinguish a
>fixed firmware from an unfixed firmware?  We have access to the "fr"
>string, but I don't know what to do with it.

FWIW, on a system with a 600P on the old firmware (109C) running 4.11.3,
I went out to lunch and returned to find my system came down with a
message something like

> controller is down; will reset: CSTS=<???>, PCI_STATUS=0xffff

I updated to the new firmware (121C) and saw it again a few hours later
on the same 4.11.3 kernel. So I think it's prudent to quirk regardless
of firmware revision.