[PATCH 1/2] pci: prevent sk hynix nvme from entering D3

[PATCH 1/2] pci: prevent sk hynix nvme from entering D3

[AceLan Kao]

It leads to the power consumption raises to 2.2W during s2idle, while
it consumes less than 1W during long idle if put SK hynix nvme to D3
and then enter s2idle.
From SK hynix FE, MS Windows doesn't put nvme to D3, and uses its own
APST feature to do the power management.
To leverage its APST feature during s2idle, we can't disable nvme
device while suspending, too.

BTW, prevent it from entering D3 will increase the power consumtion around
0.13W ~ 0.15W during short/long idle, and the power consumption during
s2idle becomes 0.77W.

Signed-off-by: AceLan Kao <acelan.kao@canonical.com>
---
 drivers/pci/quirks.c    | 1 +
 include/linux/pci_ids.h | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 4700d24e5d55..b7e6492e8311 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -1332,6 +1332,7 @@ DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_VENDOR_ID_AL, PCI_ANY_ID,
    occur when mode detecting */
 DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_VENDOR_ID_VIA, PCI_ANY_ID,
 				PCI_CLASS_STORAGE_IDE, 8, quirk_no_ata_d3);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_SK_HYNIX, 0x1527, quirk_no_ata_d3);
 
 /*
  * This was originally an Alpha-specific thing, but it really fits here.
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index 69f0abe1ba1a..5f5adda07de0 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -3090,4 +3090,6 @@
 
 #define PCI_VENDOR_ID_NCUBE		0x10ff
 
+#define PCI_VENDOR_ID_SK_HYNIX		0x1c5c
+
 #endif /* _LINUX_PCI_IDS_H */
-- 
2.17.1

[PATCH 1/2] pci: prevent sk hynix nvme from entering D3

[AceLan Kao]

It leads to the power consumption raises to 2.2W during s2idle, while
it consumes less than 1W during long idle if put SK hynix nvme to D3
and then enter s2idle.
>From SK hynix FE, MS Windows doesn't put nvme to D3, and uses its own
APST feature to do the power management.
To leverage its APST feature during s2idle, we can't disable nvme
device while suspending, too.

BTW, prevent it from entering D3 will increase the power consumtion around
0.13W ~ 0.15W during short/long idle, and the power consumption during
s2idle becomes 0.77W.

Signed-off-by: AceLan Kao <acelan.kao at canonical.com>
---
 drivers/pci/quirks.c    | 1 +
 include/linux/pci_ids.h | 2 ++
 2 files changed, 3 insertions(+)

diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 4700d24e5d55..b7e6492e8311 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -1332,6 +1332,7 @@ DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_VENDOR_ID_AL, PCI_ANY_ID,
    occur when mode detecting */
 DECLARE_PCI_FIXUP_CLASS_EARLY(PCI_VENDOR_ID_VIA, PCI_ANY_ID,
 				PCI_CLASS_STORAGE_IDE, 8, quirk_no_ata_d3);
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_SK_HYNIX, 0x1527, quirk_no_ata_d3);
 
 /*
  * This was originally an Alpha-specific thing, but it really fits here.
diff --git a/include/linux/pci_ids.h b/include/linux/pci_ids.h
index 69f0abe1ba1a..5f5adda07de0 100644
--- a/include/linux/pci_ids.h
+++ b/include/linux/pci_ids.h
@@ -3090,4 +3090,6 @@
 
 #define PCI_VENDOR_ID_NCUBE		0x10ff
 
+#define PCI_VENDOR_ID_SK_HYNIX		0x1c5c
+
 #endif /* _LINUX_PCI_IDS_H */
-- 
2.17.1

[PATCH 2/2] nvme: add quirk to not call disable function when suspending

[AceLan Kao]

Call nvme_dev_disable() function leads to the power consumption goes
up to 2.2 Watt during suspend-to-idle, and from SK hynix FE, they
suggest us to use its own APST feature to do the power management during
s2idle.
After D3 is diabled and nvme_dev_disable() is not called while
suspending, the power consumption drops to 0.77 Watt during s2idle.

Signed-off-by: AceLan Kao <acelan.kao@canonical.com>
---
 drivers/nvme/host/nvme.h | 5 +++++
 drivers/nvme/host/pci.c  | 8 +++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index cee79cb388af..35d260a4cf46 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -90,6 +90,11 @@ enum nvme_quirks {
 	 * Set MEDIUM priority on SQ creation
 	 */
 	NVME_QUIRK_MEDIUM_PRIO_SQ		= (1 << 7),
+
+	/*
+	 * Do not disable nvme when suspending(s2idle)
+	 */
+	NVME_QUIRK_NO_DISABLE			= (1 << 8),
 };
 
 /*
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index c33bb201b884..586f3516e36b 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -31,6 +31,7 @@
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/sed-opal.h>
 #include <linux/pci-p2pdma.h>
+#include <linux/suspend.h>
 
 #include "nvme.h"
 
@@ -2612,8 +2613,11 @@ static int nvme_suspend(struct device *dev)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
 	struct nvme_dev *ndev = pci_get_drvdata(pdev);
+	struct nvme_ctrl *ctrl = &ndev->ctrl;
+
+	if (!(pm_suspend_via_s2idle() && (ctrl->quirks & NVME_QUIRK_NO_DISABLE)))
+		nvme_dev_disable(ndev, true);
 
-	nvme_dev_disable(ndev, true);
 	return 0;
 }
 
@@ -2716,6 +2720,8 @@ static const struct pci_device_id nvme_id_table[] = {
 		.driver_data = NVME_QUIRK_LIGHTNVM, },
 	{ PCI_DEVICE(0x1d1d, 0x2601),	/* CNEX Granby */
 		.driver_data = NVME_QUIRK_LIGHTNVM, },
+	{ PCI_DEVICE(0x1c5c, 0x1527),   /* Sk Hynix */
+		.driver_data = NVME_QUIRK_NO_DISABLE, },
 	{ PCI_DEVICE_CLASS(PCI_CLASS_STORAGE_EXPRESS, 0xffffff) },
 	{ PCI_DEVICE(PCI_VENDOR_ID_APPLE, 0x2001) },
 	{ PCI_DEVICE(PCI_VENDOR_ID_APPLE, 0x2003) },
-- 
2.17.1

[PATCH 2/2] nvme: add quirk to not call disable function when suspending

[AceLan Kao]

Call nvme_dev_disable() function leads to the power consumption goes
up to 2.2 Watt during suspend-to-idle, and from SK hynix FE, they
suggest us to use its own APST feature to do the power management during
s2idle.
After D3 is diabled and nvme_dev_disable() is not called while
suspending, the power consumption drops to 0.77 Watt during s2idle.

Signed-off-by: AceLan Kao <acelan.kao at canonical.com>
---
 drivers/nvme/host/nvme.h | 5 +++++
 drivers/nvme/host/pci.c  | 8 +++++++-
 2 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h
index cee79cb388af..35d260a4cf46 100644
--- a/drivers/nvme/host/nvme.h
+++ b/drivers/nvme/host/nvme.h
@@ -90,6 +90,11 @@ enum nvme_quirks {
 	 * Set MEDIUM priority on SQ creation
 	 */
 	NVME_QUIRK_MEDIUM_PRIO_SQ		= (1 << 7),
+
+	/*
+	 * Do not disable nvme when suspending(s2idle)
+	 */
+	NVME_QUIRK_NO_DISABLE			= (1 << 8),
 };
 
 /*
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index c33bb201b884..586f3516e36b 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -31,6 +31,7 @@
 #include <linux/io-64-nonatomic-lo-hi.h>
 #include <linux/sed-opal.h>
 #include <linux/pci-p2pdma.h>
+#include <linux/suspend.h>
 
 #include "nvme.h"
 
@@ -2612,8 +2613,11 @@ static int nvme_suspend(struct device *dev)
 {
 	struct pci_dev *pdev = to_pci_dev(dev);
 	struct nvme_dev *ndev = pci_get_drvdata(pdev);
+	struct nvme_ctrl *ctrl = &ndev->ctrl;
+
+	if (!(pm_suspend_via_s2idle() && (ctrl->quirks & NVME_QUIRK_NO_DISABLE)))
+		nvme_dev_disable(ndev, true);
 
-	nvme_dev_disable(ndev, true);
 	return 0;
 }
 
@@ -2716,6 +2720,8 @@ static const struct pci_device_id nvme_id_table[] = {
 		.driver_data = NVME_QUIRK_LIGHTNVM, },
 	{ PCI_DEVICE(0x1d1d, 0x2601),	/* CNEX Granby */
 		.driver_data = NVME_QUIRK_LIGHTNVM, },
+	{ PCI_DEVICE(0x1c5c, 0x1527),   /* Sk Hynix */
+		.driver_data = NVME_QUIRK_NO_DISABLE, },
 	{ PCI_DEVICE_CLASS(PCI_CLASS_STORAGE_EXPRESS, 0xffffff) },
 	{ PCI_DEVICE(PCI_VENDOR_ID_APPLE, 0x2001) },
 	{ PCI_DEVICE(PCI_VENDOR_ID_APPLE, 0x2003) },
-- 
2.17.1

Re: [PATCH 2/2] nvme: add quirk to not call disable function when suspending

[Sinan Kaya]

On 11/5/2018 6:04 PM, AceLan Kao wrote:
> +	{ PCI_DEVICE(0x1c5c, 0x1527),   /* Sk Hynix */
> +		.driver_data = NVME_QUIRK_NO_DISABLE, },

Now that you added PCI_VENDOR_ID_SK_HYNIX to pci_ids.h, you
could use PCI_VENDOR_ID_SK_HYNIX above instead of 0x1c5c.

[PATCH 2/2] nvme: add quirk to not call disable function when suspending

[Sinan Kaya]

On 11/5/2018 6:04 PM, AceLan Kao wrote:
> +	{ PCI_DEVICE(0x1c5c, 0x1527),   /* Sk Hynix */
> +		.driver_data = NVME_QUIRK_NO_DISABLE, },

Now that you added PCI_VENDOR_ID_SK_HYNIX to pci_ids.h, you
could use PCI_VENDOR_ID_SK_HYNIX above instead of 0x1c5c.

Re: [PATCH 2/2] nvme: add quirk to not call disable function when suspending

[AceLan Kao]

Right, should replace it with SK_HYNIX.
I'll prepare v2 patch for that.
Sinan Kaya <okaya@kernel.org> 於 2018年11月6日 週二 上午10:24寫道：
>
> On 11/5/2018 6:04 PM, AceLan Kao wrote:
> > +     { PCI_DEVICE(0x1c5c, 0x1527),   /* Sk Hynix */
> > +             .driver_data = NVME_QUIRK_NO_DISABLE, },
>
> Now that you added PCI_VENDOR_ID_SK_HYNIX to pci_ids.h, you
> could use PCI_VENDOR_ID_SK_HYNIX above instead of 0x1c5c.

[PATCH 2/2] nvme: add quirk to not call disable function when suspending

[AceLan Kao]

Right, should replace it with SK_HYNIX.
I'll prepare v2 patch for that.
Sinan Kaya <okaya at kernel.org> ? 2018?11?6? ?? ??10:24???
>
> On 11/5/2018 6:04 PM, AceLan Kao wrote:
> > +     { PCI_DEVICE(0x1c5c, 0x1527),   /* Sk Hynix */
> > +             .driver_data = NVME_QUIRK_NO_DISABLE, },
>
> Now that you added PCI_VENDOR_ID_SK_HYNIX to pci_ids.h, you
> could use PCI_VENDOR_ID_SK_HYNIX above instead of 0x1c5c.

Re: [PATCH 2/2] nvme: add quirk to not call disable function when suspending

[Christoph Hellwig]

On Tue, Nov 06, 2018 at 10:04:02AM +0800, AceLan Kao wrote:
> Call nvme_dev_disable() function leads to the power consumption goes
> up to 2.2 Watt during suspend-to-idle, and from SK hynix FE, they
> suggest us to use its own APST feature to do the power management during
> s2idle.
> After D3 is diabled and nvme_dev_disable() is not called while
> suspending, the power consumption drops to 0.77 Watt during s2idle.

Same comment as previous one.  Get the firmware fixed please - this
doesn't prevent the drive from working, but it is a grave implementation
bug in it that needs to get fixed.

[PATCH 2/2] nvme: add quirk to not call disable function when suspending

[Christoph Hellwig]

On Tue, Nov 06, 2018@10:04:02AM +0800, AceLan Kao wrote:
> Call nvme_dev_disable() function leads to the power consumption goes
> up to 2.2 Watt during suspend-to-idle, and from SK hynix FE, they
> suggest us to use its own APST feature to do the power management during
> s2idle.
> After D3 is diabled and nvme_dev_disable() is not called while
> suspending, the power consumption drops to 0.77 Watt during s2idle.

Same comment as previous one.  Get the firmware fixed please - this
doesn't prevent the drive from working, but it is a grave implementation
bug in it that needs to get fixed.

Re: [PATCH 1/2] pci: prevent sk hynix nvme from entering D3

[Christoph Hellwig]

On Tue, Nov 06, 2018 at 10:04:01AM +0800, AceLan Kao wrote:
> It leads to the power consumption raises to 2.2W during s2idle, while
> it consumes less than 1W during long idle if put SK hynix nvme to D3
> and then enter s2idle.
> >From SK hynix FE, MS Windows doesn't put nvme to D3, and uses its own
> APST feature to do the power management.
> To leverage its APST feature during s2idle, we can't disable nvme
> device while suspending, too.
> 
> BTW, prevent it from entering D3 will increase the power consumtion around
> 0.13W ~ 0.15W during short/long idle, and the power consumption during
> s2idle becomes 0.77W.

Please get this fixed in firmware.  And yes, depending on which Windows
driver you use Windows might be doing very agressive runtime D3, which
is parted of Windows modern standby, so it'll trip up there eventually 
as well.

[PATCH 1/2] pci: prevent sk hynix nvme from entering D3

[Christoph Hellwig]

On Tue, Nov 06, 2018@10:04:01AM +0800, AceLan Kao wrote:
> It leads to the power consumption raises to 2.2W during s2idle, while
> it consumes less than 1W during long idle if put SK hynix nvme to D3
> and then enter s2idle.
> >From SK hynix FE, MS Windows doesn't put nvme to D3, and uses its own
> APST feature to do the power management.
> To leverage its APST feature during s2idle, we can't disable nvme
> device while suspending, too.
> 
> BTW, prevent it from entering D3 will increase the power consumtion around
> 0.13W ~ 0.15W during short/long idle, and the power consumption during
> s2idle becomes 0.77W.

Please get this fixed in firmware.  And yes, depending on which Windows
driver you use Windows might be doing very agressive runtime D3, which
is parted of Windows modern standby, so it'll trip up there eventually 
as well.

Re: [PATCH 1/2] pci: prevent sk hynix nvme from entering D3

[AceLan Kao]

Christoph Hellwig <hch@infradead.org> 於 2018年11月6日 週二 下午3:10寫道：
>
> On Tue, Nov 06, 2018 at 10:04:01AM +0800, AceLan Kao wrote:
> > It leads to the power consumption raises to 2.2W during s2idle, while
> > it consumes less than 1W during long idle if put SK hynix nvme to D3
> > and then enter s2idle.
> > >From SK hynix FE, MS Windows doesn't put nvme to D3, and uses its own
> > APST feature to do the power management.
> > To leverage its APST feature during s2idle, we can't disable nvme
> > device while suspending, too.
> >
> > BTW, prevent it from entering D3 will increase the power consumtion around
> > 0.13W ~ 0.15W during short/long idle, and the power consumption during
> > s2idle becomes 0.77W.
>
> Please get this fixed in firmware.  And yes, depending on which Windows
> driver you use Windows might be doing very agressive runtime D3, which
> is parted of Windows modern standby, so it'll trip up there eventually
> as well.
Agree, this is not a good fix for Sk hynix nvme, so Dell is still pushing
Sk hynix to fix it from firmware.
But before the firmware is ready, this is still a issue that need to be fixed in
kernel side, and the new firmware may not be applied on the old nvme
modules.
This list won't keep growing, and we'll keep an eye on the new firmware and
co-work with engineers from Sky hynix to make sure this issue won't happen
again.

[PATCH 1/2] pci: prevent sk hynix nvme from entering D3

[AceLan Kao]

Christoph Hellwig <hch at infradead.org> ? 2018?11?6? ?? ??3:10???
>
> On Tue, Nov 06, 2018@10:04:01AM +0800, AceLan Kao wrote:
> > It leads to the power consumption raises to 2.2W during s2idle, while
> > it consumes less than 1W during long idle if put SK hynix nvme to D3
> > and then enter s2idle.
> > >From SK hynix FE, MS Windows doesn't put nvme to D3, and uses its own
> > APST feature to do the power management.
> > To leverage its APST feature during s2idle, we can't disable nvme
> > device while suspending, too.
> >
> > BTW, prevent it from entering D3 will increase the power consumtion around
> > 0.13W ~ 0.15W during short/long idle, and the power consumption during
> > s2idle becomes 0.77W.
>
> Please get this fixed in firmware.  And yes, depending on which Windows
> driver you use Windows might be doing very agressive runtime D3, which
> is parted of Windows modern standby, so it'll trip up there eventually
> as well.
Agree, this is not a good fix for Sk hynix nvme, so Dell is still pushing
Sk hynix to fix it from firmware.
But before the firmware is ready, this is still a issue that need to be fixed in
kernel side, and the new firmware may not be applied on the old nvme
modules.
This list won't keep growing, and we'll keep an eye on the new firmware and
co-work with engineers from Sky hynix to make sure this issue won't happen
again.

Re: [PATCH 1/2] pci: prevent sk hynix nvme from entering D3

[Christoph Hellwig]

On Tue, Nov 06, 2018 at 06:11:31PM +0800, AceLan Kao wrote:
> Agree, this is not a good fix for Sk hynix nvme, so Dell is still pushing
> Sk hynix to fix it from firmware.
> But before the firmware is ready, this is still a issue that need to be fixed in
> kernel side, and the new firmware may not be applied on the old nvme
> modules.
> This list won't keep growing, and we'll keep an eye on the new firmware and
> co-work with engineers from Sky hynix to make sure this issue won't happen
> again.

We generally quirk for grave issues that make devices unusable.

To me working around a D3 mode that consumers more power than D0
does not quite fit that bill.

[PATCH 1/2] pci: prevent sk hynix nvme from entering D3

[Christoph Hellwig]

On Tue, Nov 06, 2018@06:11:31PM +0800, AceLan Kao wrote:
> Agree, this is not a good fix for Sk hynix nvme, so Dell is still pushing
> Sk hynix to fix it from firmware.
> But before the firmware is ready, this is still a issue that need to be fixed in
> kernel side, and the new firmware may not be applied on the old nvme
> modules.
> This list won't keep growing, and we'll keep an eye on the new firmware and
> co-work with engineers from Sky hynix to make sure this issue won't happen
> again.

We generally quirk for grave issues that make devices unusable.

To me working around a D3 mode that consumers more power than D0
does not quite fit that bill.