* [PATCH 1/2] pci: Add boot interrupt quirk mechanism for Xeon chipsets
2020-02-14 21:33 [PATCH 0/2] Add boot interrupt quirk mechanism for Xeon chipsets Sean V Kelley
@ 2020-02-14 21:33 ` Sean V Kelley
2020-02-14 21:33 ` [PATCH 2/2] Documentation:PCI: Add background on Boot Interrupts Sean V Kelley
2020-02-15 9:24 ` [PATCH 0/2] Add boot interrupt quirk mechanism for Xeon chipsets Thomas Gleixner
2 siblings, 0 replies; 5+ messages in thread
From: Sean V Kelley @ 2020-02-14 21:33 UTC (permalink / raw)
To: tglx, bhelgaas, corbet, mingo, bp
Cc: x86, linux-pci, linux-kernel, linux-doc, kar.hin.ong, sassmann,
Sean V Kelley
The following was observed by Kar Hin Ong with RT patchset:
Backtrace:
irq 19: nobody cared (try booting with the "irqpoll" option)
CPU: 0 PID: 3329 Comm: irq/34-nipalk Tainted:4.14.87-rt49 #1
Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880,
BIOS 2.1.5f1 01/09/2020
Call Trace:
<IRQ>
? dump_stack+0x46/0x5e
? __report_bad_irq+0x2e/0xb0
? note_interrupt+0x242/0x290
? nNIKAL100_memoryRead16+0x8/0x10 [nikal]
? handle_irq_event_percpu+0x55/0x70
? handle_irq_event+0x4f/0x80
? handle_fasteoi_irq+0x81/0x180
? handle_irq+0x1c/0x30
? do_IRQ+0x41/0xd0
? common_interrupt+0x84/0x84
</IRQ>
...
handlers:
[<ffffffffb3297200>] irq_default_primary_handler threaded
[<ffffffffb3669180>] usb_hcd_irq
Disabling IRQ #19
The problem being that this device is triggering boot interrupts
due to threaded interrupt handling and masking of the IO-APIC. These
boot interrupts are then forwarded on to the legacy PCH's PIRQ lines
where there is no handler present for the device.
Whenever a PCI device is firing interrupt (INTx) to Pin 20 of IOAPIC 2
(GSI 44); the kernel will receives 2 interrupts:
1. Interrupt from Pin 20 of IOAPIC 2 -> Expected
2. Interrupt from Pin 19 of IOAPIC 1 -> UNEXPECTED
Quirks for disabling boot interrupts (preferred) or rerouting the handler
exist but do not address these Xeon chipsets' mechanism:
https://lore.kernel.org/lkml/12131949181903-git-send-email-sassmann@suse.de/
This patch adds a new mechanism via PCI CFG for those chipsets supporting
CIPINTRC register's dis_intx_rout2ich bit.
Reported-by: Kar Hin Ong <kar.hin.ong@ni.com>
Tested-by: Kar Hin Ong <kar.hin.ong@ni.com>
Signed-off-by: Sean V Kelley <sean.v.kelley@linux.intel.com>
---
drivers/pci/quirks.c | 80 ++++++++++++++++++++++++++++++++++++++++----
1 file changed, 73 insertions(+), 7 deletions(-)
diff --git a/drivers/pci/quirks.c b/drivers/pci/quirks.c
index 29f473ebf20f..e8ee670877d1 100644
--- a/drivers/pci/quirks.c
+++ b/drivers/pci/quirks.c
@@ -1970,26 +1970,92 @@ DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_80332_1, quirk
/*
* IO-APIC1 on 6300ESB generates boot interrupts, see Intel order no
* 300641-004US, section 5.7.3.
+ *
+ * Core IO on Xeon E5 1600/2600/4600, see Intel order no 326509-003.
+ * Core IO on Xeon E5 v2, see Intel order no 329188-003.
+ * Core IO on Xeon E7 v2, see Intel order no 329595-002.
+ * Core IO on Xeon E5 v3, see Intel order no 330784-003.
+ * Core IO on Xeon E7 v3, see Intel order no 332315-001US.
+ * Core IO on Xeon E5 v4, see Intel order no 333810-002US.
+ * Core IO on Xeon E7 v4, see Intel order no 332315-001US.
+ * Core IO on Xeon D-1500, see Intel order no 332051-001.
+ * Core IO on Xeon Scalable, see Intel order no 610950.
*/
-#define INTEL_6300_IOAPIC_ABAR 0x40
+#define INTEL_6300_IOAPIC_ABAR 0x40 /* Bus 0, Dev 29, Func 5 */
#define INTEL_6300_DISABLE_BOOT_IRQ (1<<14)
+#define INTEL_CIPINTRC_CFG_OFFSET 0x14C /* Bus 0, Dev 5, Func 0 */
+#define INTEL_CIPINTRC_DIS_INTX_ICH (1<<25)
+
static void quirk_disable_intel_boot_interrupt(struct pci_dev *dev)
{
u16 pci_config_word;
+ u32 pci_config_dword;
if (noioapicquirk)
return;
- pci_read_config_word(dev, INTEL_6300_IOAPIC_ABAR, &pci_config_word);
- pci_config_word |= INTEL_6300_DISABLE_BOOT_IRQ;
- pci_write_config_word(dev, INTEL_6300_IOAPIC_ABAR, pci_config_word);
-
+ switch (dev->device) {
+ case PCI_DEVICE_ID_INTEL_ESB_10:
+ pci_read_config_word(dev, INTEL_6300_IOAPIC_ABAR,
+ &pci_config_word);
+ pci_config_word |= INTEL_6300_DISABLE_BOOT_IRQ;
+ pci_write_config_word(dev, INTEL_6300_IOAPIC_ABAR,
+ pci_config_word);
+ break;
+ case 0x3c28: /* Xeon E5 1600/2600/4600 */
+ case 0x0e28: /* Xeon E5/E7 V2 */
+ case 0x2f28: /* Xeon E5/E7 V3,V4 */
+ case 0x6f28: /* Xeon D-1500 */
+ case 0x2034: /* Xeon Scalable Family */
+ pci_read_config_dword(dev, INTEL_CIPINTRC_CFG_OFFSET,
+ &pci_config_dword);
+ pci_config_dword |= INTEL_CIPINTRC_DIS_INTX_ICH;
+ pci_write_config_dword(dev, INTEL_CIPINTRC_CFG_OFFSET,
+ pci_config_dword);
+ break;
+ default:
+ return;
+ }
pci_info(dev, "disabled boot interrupts on device [%04x:%04x]\n",
dev->vendor, dev->device);
}
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ESB_10, quirk_disable_intel_boot_interrupt);
-DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ESB_10, quirk_disable_intel_boot_interrupt);
+/*
+ * Device 29 Func 5 Device IDs of IOxAPIC
+ * containing ABAR—APIC1 Alternate Base Address Register
+ */
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ESB_10,
+ quirk_disable_intel_boot_interrupt);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, PCI_DEVICE_ID_INTEL_ESB_10,
+ quirk_disable_intel_boot_interrupt);
+
+/*
+ * Device 5 Func 0 Device IDs of IIO or Core modules/hubs
+ * containing Coherent Interface Protocol Interrupt Control
+ *
+ * Device IDs obtained from volume 2 datasheets of commented
+ * families above.
+ */
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x3c28,
+ quirk_disable_intel_boot_interrupt);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x0e28,
+ quirk_disable_intel_boot_interrupt);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x2f28,
+ quirk_disable_intel_boot_interrupt);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x6f28,
+ quirk_disable_intel_boot_interrupt);
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_INTEL, 0x2034,
+ quirk_disable_intel_boot_interrupt);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, 0x3c28,
+ quirk_disable_intel_boot_interrupt);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, 0x0e28,
+ quirk_disable_intel_boot_interrupt);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, 0x2f28,
+ quirk_disable_intel_boot_interrupt);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, 0x6f28,
+ quirk_disable_intel_boot_interrupt);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_INTEL, 0x2034,
+ quirk_disable_intel_boot_interrupt);
/* Disable boot interrupts on HT-1000 */
#define BC_HT1000_FEATURE_REG 0x64
--
2.25.0
^ permalink raw reply related [flat|nested] 5+ messages in thread
* [PATCH 2/2] Documentation:PCI: Add background on Boot Interrupts
2020-02-14 21:33 [PATCH 0/2] Add boot interrupt quirk mechanism for Xeon chipsets Sean V Kelley
2020-02-14 21:33 ` [PATCH 1/2] pci: " Sean V Kelley
@ 2020-02-14 21:33 ` Sean V Kelley
2020-02-15 9:24 ` [PATCH 0/2] Add boot interrupt quirk mechanism for Xeon chipsets Thomas Gleixner
2 siblings, 0 replies; 5+ messages in thread
From: Sean V Kelley @ 2020-02-14 21:33 UTC (permalink / raw)
To: tglx, bhelgaas, corbet, mingo, bp
Cc: x86, linux-pci, linux-kernel, linux-doc, kar.hin.ong, sassmann,
Sean V Kelley
Improve understanding of the PCI quirks for this legacy PCI interrupt
behavior to the benefit of developers and users alike.
Signed-off-by: Sean V Kelley <sean.v.kelley@linux.intel.com>
---
Documentation/PCI/boot-interrupts.rst | 153 ++++++++++++++++++++++++++
Documentation/PCI/index.rst | 1 +
2 files changed, 154 insertions(+)
create mode 100644 Documentation/PCI/boot-interrupts.rst
diff --git a/Documentation/PCI/boot-interrupts.rst b/Documentation/PCI/boot-interrupts.rst
new file mode 100644
index 000000000000..632c080994fc
--- /dev/null
+++ b/Documentation/PCI/boot-interrupts.rst
@@ -0,0 +1,153 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+===============
+Boot Interrupts
+===============
+
+:Author: - Sean V Kelley <sean.v.kelley@linux.intel.com>
+
+Overview
+========
+
+On PCI Express, interrupts are represented with either MSI or inbound interrupt
+messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a given Core
+IO converts the legacy interrupt messages from PCI Express to MSI interrupts.
+If the IO-APIC is disabled (via the mask bits in the IO-APIC table entries),
+the messages are routed to the legacy PCH. This in-band interrupt mechanism was
+traditionally necessary for systems that did not support the IO-APIC and for
+boot. Intel in the past has used the term "boot interrupts" to describe this
+mechanism. Further, the PCI Express protocol describes this in-band legacy
+wire-interrupt INTx mechanism for I/O devices to signal PCI-style level
+interrupts. The subsequent paragraphs describe problems with the Core IO
+handling of INTx message routing to the PCH and mitigation within BIOS and
+the OS.
+
+
+Problem
+=======
+
+When in-band legacy INTx messages are forwarded to the PCH, they in turn
+trigger a new interrupt for which the OS likely lacks a handler. When an
+interrupt goes unhandled over time, they are tracked by the Linux kernel
+as Spurious Interrupts. The IRQ will be disabled by the Linux kernel after
+it reaches a specific count with the error "nobody cared". This disabled
+IRQ now prevents valid usage by an existing interrupt which may happen to
+share the IRQ line.
+
+irq 19: nobody cared (try booting with the "irqpoll" option)
+CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1
+Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020
+Call Trace:
+<IRQ>
+ ? dump_stack+0x46/0x5e
+ ? __report_bad_irq+0x2e/0xb0
+ ? note_interrupt+0x242/0x290
+ ? nNIKAL100_memoryRead16+0x8/0x10 [nikal]
+ ? handle_irq_event_percpu+0x55/0x70
+ ? handle_irq_event+0x4f/0x80
+ ? handle_fasteoi_irq+0x81/0x180
+ ? handle_irq+0x1c/0x30
+ ? do_IRQ+0x41/0xd0
+ ? common_interrupt+0x84/0x84
+</IRQ>
+
+handlers:
+irq_default_primary_handler threaded usb_hcd_irq
+Disabling IRQ #19
+
+
+Conditions
+==========
+
+The use of threaded interrupts is the most likely condition to trigger this
+problem today. Threaded interrupts may not be reenabled after the IRQ handler
+wakes. These "one shot" conditions mean that the threaded interrupt needs to
+keep the interrupt line masked until the threaded handler has run. Especially
+when dealing with high data rate interrupts, the thread needs to run to completion
+otherwise some handlers will end up in stack overflows since the interrupt
+of the issuing device is still active.
+
+Affected Chipsets
+=================
+
+The legacy interrupt forwarding mechansim exists today in a number of devices
+including but not limited to chipsets from AMD/ATI, Broadcom, and Intel. Changes
+made through the mitigations below have been applied to drivers/pci/quirks.c
+
+Starting with ICX there are no longer any IO-APICs in the Core IO's devices.
+IO-APIC is only in the PCH. Devices connected to the Core IO's PCIE Root Ports
+will use native MSI/MSI-X mechanisms.
+
+Mitigations
+===========
+
+The mitigations take the form of PCI quirks. The preference has been to first
+identify and make use of a means to disable the routing to the PCH. In such a
+case a quirk to disable boot interrupt generation can be added.[1]
+
+Intel® 6300ESB I/O Controller Hub
+Alternate Base Address Register:
+ BIE: Boot Interrupt Enable
+ 0 = Boot interrupt is enabled.
+ 1 = Boot interrupt is disabled.
+
+Intel® Sandy Bridge through Sky Lake based Xeon servers:
+Coherent Interface Protocol Interrupt Control
+ dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2:
+ When this bit is set. Local INTx messages received from the
+ Intel® Quick Data DMA/PCI Express ports are not routed to legacy
+ PCH - they are either converted into MSI via the integrated IO-APIC
+ (if the IO-APIC mask bit is clear in the appropriate entries)
+ or cause no further action (when mask bit is set)
+
+In the absence of a way to directly disable the routing, another approach
+has been to make use of PCI Interrupt pin to INTx routing tables for purposes
+of redirecting the interrupt handler to the rerouted interrupt line by default.
+Therefore, on chipsets where this INTx routing cannot be disabled, the
+Linux kernel will reroute the valid interrupt to its legacy interrupt. This
+redirection of the handler will prevent the occurrence of the spurious
+interrupt detection which would ordinarily disable the IRQ line due to
+excessive unhandled counts.[2]
+
+The config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable
+(or disable) the redirection of the interrupt handler to the PCH interrupt
+line. The option can be overridden by either pci=ioapicreroute or
+pci=noioapicreroute.[3]
+
+
+More Documentation
+==================
+
+There is an overview of the legacy interrupt handling mentioned in several
+datasheets (6300ESB and 6700PXH below). While largely the same, it provides
+insight into the evolution of its handling with chipsets.
+
+Example of disabling of the boot interrupt
+------------------------------------------
+
+Intel® 6300ESB I/O Controller Hub (Document # 300641-004US)
+ 2.15.2 PCI Express Legacy INTx Support and Boot Interrupt
+ https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf
+
+Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families
+Datasheet - Volume 2: Registers (Dcument # 330784-003)
+ 6.6.41 cipintrc Coherent Interface Protocol Interrupt Control
+ https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf
+
+Example of handler rerouting
+----------------------------
+
+Intel® 6700PXH 64-bit PCI Hub (Document # 302628)
+ 2.15.2 PCI Express Legacy INTx Support and Boot Interrupt
+ https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf
+
+
+If you have any legacy PCI interrupt questions that aren't answered, email me.
+
+Cheers,
+ Sean V Kelley
+ sean.v.kelley@linux.intel.com
+
+[1] https://lore.kernel.org/lkml/12131949181903-git-send-email-sassmann@suse.de/
+[2] https://lore.kernel.org/lkml/12131949182094-git-send-email-sassmann@suse.de/
+[3] https://lore.kernel.org/lkml/487C8EA7.6020205@suse.de/
diff --git a/Documentation/PCI/index.rst b/Documentation/PCI/index.rst
index 6768305e4c26..8f66feaafd4f 100644
--- a/Documentation/PCI/index.rst
+++ b/Documentation/PCI/index.rst
@@ -16,3 +16,4 @@ Linux PCI Bus Subsystem
pci-error-recovery
pcieaer-howto
endpoint/index
+ boot-interrupts
--
2.25.0
^ permalink raw reply related [flat|nested] 5+ messages in thread