[PATCH 0/2 v2] Fix Broadwell-EP Home Agent & PCU non-compliant BARs

[PATCH 0/2 v2] Fix Broadwell-EP Home Agent & PCU non-compliant BARs

[Prarit Bhargava]

commit b894157145e4 ("x86/PCI: Mark Broadwell-EP Home Agent & PCU as having
non-compliant BARs") marks Home Agent 0 & PCU has having non-compliant
BARs.  commit b84106b4e229 ("PCI: Disable IO/MEM decoding for devices with
non-compliant BARs") introduces a check for non_compliant_bars to avoid
assigning resources to the individual BARs for a device.

Before these commits,

pci 0000:ff:12.0: BAR 2: failed to assign [mem size 0x00000040]
pci 0000:ff:12.0: BAR 4: failed to assign [mem size 0x00000040]
pci 0000:ff:12.4: BAR 2: failed to assign [mem size 0x00000040]
pci 0000:ff:12.4: BAR 4: failed to assign [mem size 0x00000040]
pci 0000:ff:12.0: BAR 1: failed to assign [mem size 0x00000010]
pci 0000:ff:12.0: BAR 3: failed to assign [mem size 0x00000010]
pci 0000:ff:12.0: BAR 5: failed to assign [mem size 0x00000010]
pci 0000:ff:12.4: BAR 1: failed to assign [mem size 0x00000010]
pci 0000:ff:12.4: BAR 3: failed to assign [mem size 0x00000010]
pci 0000:ff:12.4: BAR 5: failed to assign [mem size 0x00000010]
pci 0000:7f:12.0: BAR 2: failed to assign [mem size 0x00000040]
pci 0000:7f:12.0: BAR 4: failed to assign [mem size 0x00000040]
pci 0000:7f:12.4: BAR 2: failed to assign [mem size 0x00000040]
pci 0000:7f:12.4: BAR 4: failed to assign [mem size 0x00000040]
pci 0000:7f:12.0: BAR 1: failed to assign [mem size 0x00000010]
pci 0000:7f:12.0: BAR 3: failed to assign [mem size 0x00000010]
pci 0000:7f:12.0: BAR 5: failed to assign [mem size 0x00000010]
pci 0000:7f:12.4: BAR 1: failed to assign [mem size 0x00000010]
pci 0000:7f:12.4: BAR 3: failed to assign [mem size 0x00000010]
pci 0000:7f:12.4: BAR 5: failed to assign [mem size 0x00000010]

After the commits, there are still "failed to assign" messages,
as well as new "failed to assign" messages for ff:12.0, ff:1e.3,
7f:12.0, and 7f:1e.3.

 pci 0000:ff:12.4: BAR 2: failed to assign [mem size 0x00000040]
 pci 0000:ff:12.4: BAR 4: failed to assign [mem size 0x00000040]
 pci 0000:ff:12.4: BAR 1: failed to assign [mem size 0x00000010]
 pci 0000:ff:12.4: BAR 3: failed to assign [mem size 0x00000010]
 pci 0000:ff:12.4: BAR 5: failed to assign [mem size 0x00000010]
 pci 0000:ff:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
 pci 0000:ff:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]
 pci 0000:7f:12.4: BAR 2: failed to assign [mem size 0x00000040]
 pci 0000:7f:12.4: BAR 4: failed to assign [mem size 0x00000040]
 pci 0000:7f:12.4: BAR 1: failed to assign [mem size 0x00000010]
 pci 0000:7f:12.4: BAR 3: failed to assign [mem size 0x00000010]
 pci 0000:7f:12.4: BAR 5: failed to assign [mem size 0x00000010]
 pci 0000:7f:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
 pci 0000:7f:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]

There are two issues that need to be fixed.  The first is that there is another
device, Home Agent 1 & PCU (device ID 0x6f60) that must also be "blacklisted"
in the same way. After applying the first patch in this patchset the log still
contains these messages:

pci 0000:ff:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:ff:12.4: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:ff:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:7f:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:7f:12.4: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:7f:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]

This second patch fixes these issues by moving the non_compliant_bars_check to
pci_read_bases() where it applies to all of a device's BARs.

v2: Separate into two patches.  Applied non_compliant_bars check to
pci_read_bases().  Added additional explanation to the commit log from
the BDW errata for the Home Agent devices.

Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Prarit Bhargava <prarit@redhat.com>

Prarit Bhargava (2):
  x86/PCI: Add pci_bdwep_bar() quirk entry for Broadwell-EP Home Agent
    1
  x86/PCI: Disable IO/MEM decoding for ROM BARs for devices with
    non-compliant BARs

 arch/x86/pci/fixup.c |    9 +++++++++
 drivers/pci/probe.c  |    6 +++---
 2 files changed, 12 insertions(+), 3 deletions(-)

-- 
1.7.9.3

[PATCH 1/2] x86/PCI: Add pci_bdwep_bar() quirk entry for Broadwell-EP Home Agent 1

[Prarit Bhargava]

commit b894157145e4 ("x86/PCI: Mark Broadwell-EP Home Agent & PCU as having
non-compliant BARs") marks Home Agent 0 & PCU has having non-compliant
BARs.  The commit is missing device 0x6f60 for Home Agent 1.

The problem with these devices is documented in the Xeon v4 specification
update (but is unfortunately missing this ID):

BDF2
PCI BARs in the Home Agent Will Return Non-Zero Values During Enumeration
Problem:
During system initialization the Operating System may access the standard
PCI BARs (Base Address Registers). Due to this erratum, accesses to the
Home Agent BAR registers (Bus 1; Device 18; Function 0,4; Offsets (0x14-0x24)
will return non-zero values.
Implication:
The operating system may issue a warning. Intel has not observed any
functional failures due to this erratum.
Workaround:
None identified.

Link: http://www.intel.com/content/www/us/en/processors/xeon/xeon-e5-v4-spec-update.html
Fixes: b894157145e4 ("x86/PCI: Mark Broadwell-EP Home Agent & PCU as having non-compliant BARs")
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Prarit Bhargava <prarit@redhat.com>

---
 arch/x86/pci/fixup.c |    9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
index b7de1929714b..6d8ab876b732 100644
--- a/arch/x86/pci/fixup.c
+++ b/arch/x86/pci/fixup.c
@@ -552,9 +552,18 @@ static void twinhead_reserve_killing_zone(struct pci_dev *dev)
 }
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_INTEL, 0x27B9, twinhead_reserve_killing_zone);
 
+/*
+ * Broadwell EP Home Agent BARs erroneously return non-zero values
+ * during enumeration which cause bogus warnings during resource
+ * allocation.
+ *
+ *   See: http://www.intel.com/content/www/us/en/processors/xeon/xeon-e5-v4-spec-update.html
+ *   entry BDF2.
+ */
 static void pci_bdwep_bar(struct pci_dev *dev)
 {
 	dev->non_compliant_bars = 1;
 }
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x6f60, pci_bdwep_bar);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x6fa0, pci_bdwep_bar);
 DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_INTEL, 0x6fc0, pci_bdwep_bar);
-- 
1.7.9.3

[PATCH 2/2] x86/PCI: Disable IO/MEM decoding for ROM BARs for devices with non-compliant BARs

[Prarit Bhargava]

After executing quirk patches for the Broadwell-EP Home Agents,
the following errors are in the boot log:

pci 0000:ff:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:ff:12.4: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:ff:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:7f:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:7f:12.4: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:7f:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]

For the PCI ROM device's memory resource (BAR 6), pci_read_bases()
initializes the value of res->flags.  Later in the boot,
pcibios_assign_resources() attempts to assign uninitialized resources, and
calls pcibios_allocate_dev_rom_resource().  That function uses the
res->flags field to determine if the rom resource has been initialized.
The code then attempts to assign a resource to the rom BAR and the "failed
to assign" messages are output to the log.

This patch moves the non_compliant_bars check to pci_read_bases() so it
covers all the BARs of a non-compliant device.

Fixes: b84106b4e229 ("PCI: Disable IO/MEM decoding for devices with non-compliant BARs")
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Prarit Bhargava <prarit@redhat.com>

---
 drivers/pci/probe.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 8004f67c57ec..bf8405fb4ace 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -179,9 +179,6 @@ int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type,
 	u16 orig_cmd;
 	struct pci_bus_region region, inverted_region;
 
-	if (dev->non_compliant_bars)
-		return 0;
-
 	mask = type ? PCI_ROM_ADDRESS_MASK : ~0;
 
 	/* No printks while decoding is disabled! */
@@ -322,6 +319,9 @@ static void pci_read_bases(struct pci_dev *dev, unsigned int howmany, int rom)
 {
 	unsigned int pos, reg;
 
+	if (dev->non_compliant_bars)
+		return;
+
 	for (pos = 0; pos < howmany; pos++) {
 		struct resource *res = &dev->resource[pos];
 		reg = PCI_BASE_ADDRESS_0 + (pos << 2);
-- 
1.7.9.3

Re: [PATCH 0/2 v2] Fix Broadwell-EP Home Agent & PCU non-compliant BARs

[Bjorn Helgaas]

On Wed, May 11, 2016 at 12:27:14PM -0400, Prarit Bhargava wrote:
> commit b894157145e4 ("x86/PCI: Mark Broadwell-EP Home Agent & PCU as having
> non-compliant BARs") marks Home Agent 0 & PCU has having non-compliant
> BARs.  commit b84106b4e229 ("PCI: Disable IO/MEM decoding for devices with
> non-compliant BARs") introduces a check for non_compliant_bars to avoid
> assigning resources to the individual BARs for a device.
> 
> Before these commits,
> 
> pci 0000:ff:12.0: BAR 2: failed to assign [mem size 0x00000040]
> pci 0000:ff:12.0: BAR 4: failed to assign [mem size 0x00000040]
> pci 0000:ff:12.4: BAR 2: failed to assign [mem size 0x00000040]
> pci 0000:ff:12.4: BAR 4: failed to assign [mem size 0x00000040]
> pci 0000:ff:12.0: BAR 1: failed to assign [mem size 0x00000010]
> pci 0000:ff:12.0: BAR 3: failed to assign [mem size 0x00000010]
> pci 0000:ff:12.0: BAR 5: failed to assign [mem size 0x00000010]
> pci 0000:ff:12.4: BAR 1: failed to assign [mem size 0x00000010]
> pci 0000:ff:12.4: BAR 3: failed to assign [mem size 0x00000010]
> pci 0000:ff:12.4: BAR 5: failed to assign [mem size 0x00000010]
> pci 0000:7f:12.0: BAR 2: failed to assign [mem size 0x00000040]
> pci 0000:7f:12.0: BAR 4: failed to assign [mem size 0x00000040]
> pci 0000:7f:12.4: BAR 2: failed to assign [mem size 0x00000040]
> pci 0000:7f:12.4: BAR 4: failed to assign [mem size 0x00000040]
> pci 0000:7f:12.0: BAR 1: failed to assign [mem size 0x00000010]
> pci 0000:7f:12.0: BAR 3: failed to assign [mem size 0x00000010]
> pci 0000:7f:12.0: BAR 5: failed to assign [mem size 0x00000010]
> pci 0000:7f:12.4: BAR 1: failed to assign [mem size 0x00000010]
> pci 0000:7f:12.4: BAR 3: failed to assign [mem size 0x00000010]
> pci 0000:7f:12.4: BAR 5: failed to assign [mem size 0x00000010]
> 
> After the commits, there are still "failed to assign" messages,
> as well as new "failed to assign" messages for ff:12.0, ff:1e.3,
> 7f:12.0, and 7f:1e.3.
> 
>  pci 0000:ff:12.4: BAR 2: failed to assign [mem size 0x00000040]
>  pci 0000:ff:12.4: BAR 4: failed to assign [mem size 0x00000040]
>  pci 0000:ff:12.4: BAR 1: failed to assign [mem size 0x00000010]
>  pci 0000:ff:12.4: BAR 3: failed to assign [mem size 0x00000010]
>  pci 0000:ff:12.4: BAR 5: failed to assign [mem size 0x00000010]
>  pci 0000:ff:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
>  pci 0000:ff:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]
>  pci 0000:7f:12.4: BAR 2: failed to assign [mem size 0x00000040]
>  pci 0000:7f:12.4: BAR 4: failed to assign [mem size 0x00000040]
>  pci 0000:7f:12.4: BAR 1: failed to assign [mem size 0x00000010]
>  pci 0000:7f:12.4: BAR 3: failed to assign [mem size 0x00000010]
>  pci 0000:7f:12.4: BAR 5: failed to assign [mem size 0x00000010]
>  pci 0000:7f:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
>  pci 0000:7f:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]
> 
> There are two issues that need to be fixed.  The first is that there is another
> device, Home Agent 1 & PCU (device ID 0x6f60) that must also be "blacklisted"
> in the same way. After applying the first patch in this patchset the log still
> contains these messages:
> 
> pci 0000:ff:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
> pci 0000:ff:12.4: BAR 6: failed to assign [mem size 0x00000001 pref]
> pci 0000:ff:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]
> pci 0000:7f:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
> pci 0000:7f:12.4: BAR 6: failed to assign [mem size 0x00000001 pref]
> pci 0000:7f:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]
> 
> This second patch fixes these issues by moving the non_compliant_bars_check to
> pci_read_bases() where it applies to all of a device's BARs.
> 
> v2: Separate into two patches.  Applied non_compliant_bars check to
> pci_read_bases().  Added additional explanation to the commit log from
> the BDW errata for the Home Agent devices.
> 
> Cc: Bjorn Helgaas <bhelgaas@google.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: x86@kernel.org
> Cc: linux-kernel@vger.kernel.org
> Cc: Andi Kleen <ak@linux.intel.com>
> Signed-off-by: Prarit Bhargava <prarit@redhat.com>
> 
> Prarit Bhargava (2):
>   x86/PCI: Add pci_bdwep_bar() quirk entry for Broadwell-EP Home Agent
>     1
>   x86/PCI: Disable IO/MEM decoding for ROM BARs for devices with
>     non-compliant BARs
> 
>  arch/x86/pci/fixup.c |    9 +++++++++
>  drivers/pci/probe.c  |    6 +++---
>  2 files changed, 12 insertions(+), 3 deletions(-)

I applied these to pci/resource and intend to merge them for v4.7.

[PATCH 2/2] x86/PCI: Disable IO/MEM decoding for ROM BARs for devices with non-compliant BARs

[Prarit Bhargava]

After executing quirk patches for the Broadwell-EP Home Agents,
the following errors are in the boot log:

pci 0000:ff:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:ff:12.4: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:ff:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:7f:12.0: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:7f:12.4: BAR 6: failed to assign [mem size 0x00000001 pref]
pci 0000:7f:1e.3: BAR 6: failed to assign [mem size 0x00000001 pref]

For the PCI ROM device's memory resource (BAR 6), pci_read_bases()
initializes the value of res->flags.  Later in the boot,
pcibios_assign_resources() attempts to assign uninitialized resources, and
calls pcibios_allocate_dev_rom_resource().  That function uses the
res->flags field to determine if the rom resource has been initialized.
The code then attempts to assign a resource to the rom BAR and the "failed
to assign" messages are output to the log.

This patch moves the non_compliant_bars check to pci_read_bases() so it
covers all the BARs of a non-compliant device.

Fixes: b84106b4e229 ("PCI: Disable IO/MEM decoding for devices with non-compliant BARs")
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: x86@kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: Andi Kleen <ak@linux.intel.com>
Signed-off-by: Prarit Bhargava <prarit@redhat.com>

---
 drivers/pci/probe.c |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c
index 8004f67c57ec..bf8405fb4ace 100644
--- a/drivers/pci/probe.c
+++ b/drivers/pci/probe.c
@@ -179,9 +179,6 @@ int __pci_read_base(struct pci_dev *dev, enum pci_bar_type type,
 	u16 orig_cmd;
 	struct pci_bus_region region, inverted_region;
 
-	if (dev->non_compliant_bars)
-		return 0;
-
 	mask = type ? PCI_ROM_ADDRESS_MASK : ~0;
 
 	/* No printks while decoding is disabled! */
@@ -322,6 +319,9 @@ static void pci_read_bases(struct pci_dev *dev, unsigned int howmany, int rom)
 {
 	unsigned int pos, reg;
 
+	if (dev->non_compliant_bars)
+		return;
+
 	for (pos = 0; pos < howmany; pos++) {
 		struct resource *res = &dev->resource[pos];
 		reg = PCI_BASE_ADDRESS_0 + (pos << 2);
-- 
1.7.9.3