linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges
@ 2019-10-17 12:19 Karol Herbst
  2019-11-14 19:17 ` Karol Herbst
                   ` (2 more replies)
  0 siblings, 3 replies; 67+ messages in thread
From: Karol Herbst @ 2019-10-17 12:19 UTC (permalink / raw)
  To: linux-kernel
  Cc: Karol Herbst, Bjorn Helgaas, Lyude Paul, Rafael J . Wysocki,
	Mika Westerberg, linux-pci, linux-pm, dri-devel, nouveau

Fixes state transitions of Nvidia Pascal GPUs from D3cold into higher device
states.

v2: convert to pci_dev quirk
    put a proper technical explanation of the issue as a in-code comment
v3: disable it only for certain combinations of intel and nvidia hardware
v4: simplify quirk by setting flag on the GPU itself

Signed-off-by: Karol Herbst <kherbst@redhat.com>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Lyude Paul <lyude@redhat.com>
Cc: Rafael J. Wysocki <rjw@rjwysocki.net>
Cc: Mika Westerberg <mika.westerberg@intel.com>
Cc: linux-pci@vger.kernel.org
Cc: linux-pm@vger.kernel.org
Cc: dri-devel@lists.freedesktop.org
Cc: nouveau@lists.freedesktop.org
---
 drivers/pci/pci.c    |  7 ++++++
 drivers/pci/quirks.c | 53 ++++++++++++++++++++++++++++++++++++++++++++
 include/linux/pci.h  |  1 +
 3 files changed, 61 insertions(+)

diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index b97d9e10c9cc..02e71e0bcdd7 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -850,6 +850,13 @@ static int pci_raw_set_power_state(struct pci_dev *dev, pci_power_t state)
 	   || (state == PCI_D2 && !dev->d2_support))
 		return -EIO;
 
+	/*
+	 * check if we have a bad combination of bridge controller and nvidia
+         * GPU, see quirk_broken_nv_runpm for more info
+	 */
+	if (state != PCI_D0 && dev->broken_nv_runpm)
+		return 0;
+
 	pci_read_config_word(dev, dev->pm_cap + PCI_PM_CTRL, &pmcsr);
 
 	/*
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 44c4ae1abd00..0006c9e37b6f 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -5268,3 +5268,56 @@ static void quirk_reset_lenovo_thinkpad_p50_nvgpu(struct pci_dev *pdev)
 DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_NVIDIA, 0x13b1,
 			      PCI_CLASS_DISPLAY_VGA, 8,
 			      quirk_reset_lenovo_thinkpad_p50_nvgpu);
+
+/*
+ * Some Intel PCIe bridges cause devices to disappear from the PCIe bus after
+ * those were put into D3cold state if they were put into a non D0 PCI PM
+ * device state before doing so.
+ *
+ * This leads to various issue different issues which all manifest differently,
+ * but have the same root cause:
+ *  - AIML code execution hits an infinite loop (as the coe waits on device
+ *    memory to change).
+ *  - kernel crashes, as all pci reads return -1, which most code isn't able
+ *    to handle well enough.
+ *  - sudden shutdowns, as the kernel identified an unrecoverable error after
+ *    userspace tries to access the GPU.
+ *
+ * In all cases dmesg will contain at least one line like this:
+ * 'nouveau 0000:01:00.0: Refused to change power state, currently in D3'
+ * followed by a lot of nouveau timeouts.
+ *
+ * ACPI code writes bit 0x80 to the not documented PCI register 0x248 of the
+ * PCIe bridge controller in order to power down the GPU.
+ * Nonetheless, there are other code paths inside the ACPI firmware which use
+ * other registers, which seem to work fine:
+ *  - 0xbc bit 0x20 (publicly available documentation claims 'reserved')
+ *  - 0xb0 bit 0x10 (link disable)
+ * Changing the conditions inside the firmware by poking into the relevant
+ * addresses does resolve the issue, but it seemed to be ACPI private memory
+ * and not any device accessible memory at all, so there is no portable way of
+ * changing the conditions.
+ *
+ * The only systems where this behavior can be seen are hybrid graphics laptops
+ * with a secondary Nvidia Pascal GPU. It cannot be ruled out that this issue
+ * only occurs in combination with listed Intel PCIe bridge controllers and
+ * the mentioned GPUs or if it's only a hw bug in the bridge controller.
+ *
+ * But because this issue was NOT seen on laptops with an Nvidia Pascal GPU
+ * and an Intel Coffee Lake SoC, there is a higher chance of there being a bug
+ * in the bridge controller rather than in the GPU.
+ *
+ * This issue was not able to be reproduced on non laptop systems.
+ */
+
+static void quirk_broken_nv_runpm(struct pci_dev *dev)
+{
+	struct pci_dev *bridge = pci_upstream_bridge(dev);
+
+	if (bridge->vendor == PCI_VENDOR_ID_INTEL &&
+	    bridge->device == 0x1901)
+		dev->broken_nv_runpm = 1;
+}
+DECLARE_PCI_FIXUP_CLASS_FINAL(PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
+			      PCI_BASE_CLASS_DISPLAY, 16,
+			      quirk_broken_nv_runpm);
diff --git a/include/linux/pci.h b/include/linux/pci.h
index ac8a6c4e1792..903a0b3a39ec 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -416,6 +416,7 @@ struct pci_dev {
 	unsigned int	__aer_firmware_first_valid:1;
 	unsigned int	__aer_firmware_first:1;
 	unsigned int	broken_intx_masking:1;	/* INTx masking can't be used */
+	unsigned int	broken_nv_runpm:1;	/* some combinations of intel bridge controller and nvidia GPUs break rtd3 */
 	unsigned int	io_window_1k:1;		/* Intel bridge 1K I/O windows */
 	unsigned int	irq_managed:1;
 	unsigned int	has_secondary_link:1;
-- 
2.21.0


^ permalink raw reply related	[flat|nested] 67+ messages in thread

end of thread, other threads:[~2020-01-13 15:32 UTC | newest]

Thread overview: 67+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-17 12:19 [PATCH v4] pci: prevent putting nvidia GPUs into lower device states on certain intel bridges Karol Herbst
2019-11-14 19:17 ` Karol Herbst
2019-11-19 20:06 ` Dave Airlie
2019-11-19 21:49 ` Bjorn Helgaas
2019-11-19 22:26   ` Karol Herbst
2019-11-19 22:57     ` Bjorn Helgaas
2019-11-20 10:18     ` Mika Westerberg
2019-11-20 10:52       ` Rafael J. Wysocki
2019-11-20 11:22         ` Mika Westerberg
2019-11-20 11:48           ` Rafael J. Wysocki
2019-11-20 11:51             ` Karol Herbst
2019-11-20 12:06               ` Rafael J. Wysocki
2019-11-20 12:09                 ` Karol Herbst
2019-11-20 12:14                   ` Rafael J. Wysocki
2019-11-20 12:19                     ` Karol Herbst
2019-11-20 12:11                 ` Rafael J. Wysocki
2019-11-20 11:51           ` Mika Westerberg
2019-11-20 11:54             ` Karol Herbst
2019-11-20 11:58               ` Karol Herbst
2019-11-20 12:09                 ` Mika Westerberg
2019-11-20 12:11                   ` Karol Herbst
2019-11-20 15:15                     ` Mika Westerberg
2019-11-20 15:37                       ` Karol Herbst
2019-11-20 15:53                         ` Mika Westerberg
2019-11-20 16:23                           ` Mika Westerberg
2019-11-20 21:36                             ` Karol Herbst
2019-11-21 10:14                               ` Mika Westerberg
2019-11-21 11:03                                 ` Rafael J. Wysocki
2019-11-21 11:08                                   ` Rafael J. Wysocki
2019-11-21 11:15                                     ` Rafael J. Wysocki
2019-11-21 11:17                                   ` Mika Westerberg
2019-11-21 11:31                                     ` Rafael J. Wysocki
2019-11-20 21:37                           ` Rafael J. Wysocki
2019-11-20 21:40                             ` Karol Herbst
2019-11-20 22:29                               ` Rafael J. Wysocki
2019-11-21 11:28                                 ` Mika Westerberg
2019-11-21 11:34                                   ` Rafael J. Wysocki
2019-11-21 11:46                                     ` Mika Westerberg
2019-11-21 12:52                                       ` Mika Westerberg
2019-11-21 12:56                                         ` Karol Herbst
2019-11-21 15:43                                         ` Rafael J. Wysocki
2019-11-21 19:49                                           ` Mika Westerberg
2019-11-21 22:39                                             ` Rafael J. Wysocki
2019-11-21 22:50                                               ` Karol Herbst
2019-11-22  0:13                                                 ` Karol Herbst
2019-11-22  9:07                                                   ` Rafael J. Wysocki
2019-11-22 11:30                                                     ` Karol Herbst
2019-11-22 10:36                                               ` Mika Westerberg
2019-11-22 11:30                                                 ` Rafael J. Wysocki
2019-11-22 11:34                                                   ` Karol Herbst
2019-11-22 11:54                                                     ` Rafael J. Wysocki
2019-11-22 11:52                                                   ` Mika Westerberg
2019-11-22 12:15                                                     ` Rafael J. Wysocki
2019-11-21 12:52                                       ` Karol Herbst
2019-11-21 15:47                                         ` Rafael J. Wysocki
2019-11-21 16:06                                           ` Karol Herbst
2019-11-21 16:39                                             ` Rafael J. Wysocki
2019-11-26 23:10                                               ` Lyude Paul
2019-11-27 11:48                                                 ` Mika Westerberg
2019-11-27 11:51                                                   ` Karol Herbst
2019-11-27 19:51                                                     ` Lyude Paul
2019-12-09 11:17                                                       ` Karol Herbst
2019-12-09 11:38                                                         ` Rafael J. Wysocki
2019-12-09 12:24                                                           ` Karol Herbst
2019-12-10 19:58                                                           ` Dave Airlie
2019-12-10 20:49                                                             ` Karol Herbst
2020-01-13 15:31                                                               ` Karol Herbst

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).