[PATCH V4 0/4] x86/pci Fix numa_node info for AMD hostbridge and misc clean up.

[PATCH V4 0/4] x86/pci Fix numa_node info for AMD hostbridge and misc clean up.

[suravee.suthikulpanit]

From: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>

This in the V4 of the patch set which trys to fixed up numa_node for AMD hostbridges.
This topic was discussed in the following threads:

V1: https://lkml.org/lkml/2014/3/5/898
V2: https://lkml.org/lkml/2014/4/18/623
V3: https://lkml.org/lkml/2014/5/7/588

Change from V3:
Split up the changes in patch1 into two per Robert's suggestions. The second patch has 
no functional change. 

Change from V2:
As requested by Bjorn, the patch set will only add supports for family15h, and 
also deprecating the mechanism, as future platforms should be relying on the ACPI info.
He has also requested to keep the changes to the minimum. Therefore, the V2 patch 2 and 3
has been removed.

Suravee Suthikulpanit (2):
  Patch1: x86/PCI: Fix PCI root numa_node info on AMD family15h
  Patch2: x86/PCI: Clean up and mark early_root_info_init as deprecated

Myron Stowe (2):
  Patch3: ACPI/PCI: Warn if we have to "guess" host bridge node information
  Patch4: X86/PCI: Remove unnecessary 'quirk_amd_nb_node'

 arch/x86/kernel/quirks.c | 58 -------------------------------------
 arch/x86/pci/acpi.c      |  6 +++-
 arch/x86/pci/amd_bus.c   | 75 +++++++++++++++++++++++++++++-------------------
 3 files changed, 51 insertions(+), 88 deletions(-)

-- 
1.9.0

[PATCH V4 1/4] x86/PCI: Fix PCI root numa_node info on AMD family15h

[suravee.suthikulpanit]

From: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>

This patch fixes the numa_node information in sysfs for PCI root
on AMD family15h platforms (currently showing -1) by adding
the hostbridge in the list of probed devices to be used for initializing
pci_root_info structue.

Signed-off-by: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: Myron Stowe <myron.stowe@redhat.com>
Tested-by: Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: Robert Richter <rric@kernel.org>
Cc: Daniel J Blueman <daniel@numascale.com>
Cc: Andreas Herrmann <herrmann.der.user@googlemail.com>
---
 arch/x86/pci/amd_bus.c | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/arch/x86/pci/amd_bus.c b/arch/x86/pci/amd_bus.c
index e88f4c5..330dbfe 100644
--- a/arch/x86/pci/amd_bus.c
+++ b/arch/x86/pci/amd_bus.c
@@ -24,10 +24,11 @@ struct pci_hostbridge_probe {
 };
 
 static struct pci_hostbridge_probe pci_probes[] __initdata = {
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1100 },
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1200 },
-	{ 0xff, 0, PCI_VENDOR_ID_AMD, 0x1200 },
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1300 },
+	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1100 }, /* K8 */
+	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1200 }, /* Fam10h */
+	{ 0xff, 0, PCI_VENDOR_ID_AMD, 0x1200 }, /* Fam10h */
+	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1300 }, /* Fam11h */
+	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1600 }, /* Fam15h */
 };
 
 #define RANGE_NUM 16
@@ -113,6 +114,13 @@ static int __init early_fill_mp_bus_info(void)
 		info = alloc_pci_root_info(min_bus, max_bus, node, link);
 	}
 
+	/*
+	 * The following code is only supported until Fam11h.
+	 * Newer processors will depend on ACPI MCFG table instead.
+	 */
+	if (boot_cpu_data.x86 > 0x11)
+		return 0;
+
 	/* get the default node and link for left over res */
 	reg = read_pci_config(bus, slot, 0, 0x60);
 	def_node = (reg >> 8) & 0x07;
-- 
1.9.0

[PATCH V4 2/4] x86/PCI: Clean up and mark early_root_info_init as deprecated

[suravee.suthikulpanit]

From: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>

early_root_info_init is now deprecated in favor of info in ACPI.
Therefore, this patch adds note stating the deprecation.
Also, adding some clean up.

There is no functional change

Signed-off-by: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
---
 arch/x86/pci/amd_bus.c | 69 ++++++++++++++++++++++++++++----------------------
 1 file changed, 39 insertions(+), 30 deletions(-)

diff --git a/arch/x86/pci/amd_bus.c b/arch/x86/pci/amd_bus.c
index 330dbfe..7c251c2 100644
--- a/arch/x86/pci/amd_bus.c
+++ b/arch/x86/pci/amd_bus.c
@@ -11,28 +11,33 @@
 
 #include "bus_numa.h"
 
-/*
- * This discovers the pcibus <-> node mapping on AMD K8.
- * also get peer root bus resource for io,mmio
- */
+#define AMD_NB_F0_NODE_ID			0x60
+#define AMD_NB_F0_UNIT_ID			0x64
+#define AMD_NB_F1_CONFIG_MAP_REG		0xe0
+
+#define RANGE_NUM				16
+#define AMD_NB_F1_CONFIG_MAP_RANGES		4
 
-struct pci_hostbridge_probe {
+struct amd_hostbridge {
 	u32 bus;
 	u32 slot;
-	u32 vendor;
 	u32 device;
 };
 
-static struct pci_hostbridge_probe pci_probes[] __initdata = {
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1100 }, /* K8 */
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1200 }, /* Fam10h */
-	{ 0xff, 0, PCI_VENDOR_ID_AMD, 0x1200 }, /* Fam10h */
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1300 }, /* Fam11h */
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1600 }, /* Fam15h */
+/*
+ * IMPORTANT NOTE:
+ * hb_probes[] and early_root_info_init() is in maintenance mode.
+ * It only supports K8, Fam10h, Fam11h, and Fam15h_00h-0fh .
+ * Future processor will rely on information in ACPI.
+ */
+static struct amd_hostbridge hb_probes[] __initdata = {
+	{ 0, 0x18, 0x1100 }, /* K8 */
+	{ 0, 0x18, 0x1200 }, /* Family10h */
+	{ 0xff, 0, 0x1200 }, /* Family10h */
+	{ 0, 0x18, 0x1300 }, /* Family11h */
+	{ 0, 0x18, 0x1600 }, /* Family15h */
 };
 
-#define RANGE_NUM 16
-
 static struct pci_root_info __init *find_pci_root_info(int node, int link)
 {
 	struct pci_root_info *info;
@@ -46,12 +51,12 @@ static struct pci_root_info __init *find_pci_root_info(int node, int link)
 }
 
 /**
- * early_fill_mp_bus_to_node()
+ * early_root_info_init()
  * called before pcibios_scan_root and pci_scan_bus
- * fills the mp_bus_to_cpumask array based according to the LDT Bus Number
- * Registers found in the K8 northbridge
+ * fills the mp_bus_to_cpumask array based according
+ * to the LDT Bus Number Registers found in the northbridge.
  */
-static int __init early_fill_mp_bus_info(void)
+static int __init early_root_info_init(void)
 {
 	int i;
 	unsigned bus;
@@ -76,19 +81,21 @@ static int __init early_fill_mp_bus_info(void)
 		return -1;
 
 	found = false;
-	for (i = 0; i < ARRAY_SIZE(pci_probes); i++) {
+	for (i = 0; i < ARRAY_SIZE(hb_probes); i++) {
 		u32 id;
 		u16 device;
 		u16 vendor;
 
-		bus = pci_probes[i].bus;
-		slot = pci_probes[i].slot;
+		bus = hb_probes[i].bus;
+		slot = hb_probes[i].slot;
 		id = read_pci_config(bus, slot, 0, PCI_VENDOR_ID);
-
 		vendor = id & 0xffff;
 		device = (id>>16) & 0xffff;
-		if (pci_probes[i].vendor == vendor &&
-		    pci_probes[i].device == device) {
+
+		if (vendor != PCI_VENDOR_ID_AMD)
+			continue;
+
+		if (hb_probes[i].device == device) {
 			found = true;
 			break;
 		}
@@ -97,10 +104,11 @@ static int __init early_fill_mp_bus_info(void)
 	if (!found)
 		return 0;
 
-	for (i = 0; i < 4; i++) {
+	for (i = 0; i < AMD_NB_F1_CONFIG_MAP_RANGES; i++) {
 		int min_bus;
 		int max_bus;
-		reg = read_pci_config(bus, slot, 1, 0xe0 + (i << 2));
+		reg = read_pci_config(bus, slot, 1,
+				AMD_NB_F1_CONFIG_MAP_REG + (i << 2));
 
 		/* Check if that register is enabled for bus range */
 		if ((reg & 7) != 3)
@@ -122,9 +130,9 @@ static int __init early_fill_mp_bus_info(void)
 		return 0;
 
 	/* get the default node and link for left over res */
-	reg = read_pci_config(bus, slot, 0, 0x60);
+	reg = read_pci_config(bus, slot, 0, AMD_NB_F0_NODE_ID);
 	def_node = (reg >> 8) & 0x07;
-	reg = read_pci_config(bus, slot, 0, 0x64);
+	reg = read_pci_config(bus, slot, 0, AMD_NB_F0_UNIT_ID);
 	def_link = (reg >> 8) & 0x03;
 
 	memset(range, 0, sizeof(range));
@@ -371,7 +379,7 @@ static int __init pci_io_ecs_init(void)
 	int cpu;
 
 	/* assume all cpus from fam10h have IO ECS */
-        if (boot_cpu_data.x86 < 0x10)
+	if (boot_cpu_data.x86 < 0x10)
 		return 0;
 
 	/* Try the PCI method first. */
@@ -395,7 +403,8 @@ static int __init amd_postcore_init(void)
 	if (boot_cpu_data.x86_vendor != X86_VENDOR_AMD)
 		return 0;
 
-	early_fill_mp_bus_info();
+	early_root_info_init();
+
 	pci_io_ecs_init();
 
 	return 0;
-- 
1.9.0

[PATCH V4 3/4] ACPI/PCI: Warn if we have to "guess" host bridge node information

[suravee.suthikulpanit]

From: Myron Stowe <myron.stowe@redhat.com>

The vast majority of platforms are not supplying ACPI _PXM (proximity)
information corresponding to host bridge (PNP0A03/PNP0A08) devices
resulting in sysfs "numa_node" values of -1 (NUMA_NO_NODE) [1]:
  # for i in /sys/devices/pci0000\:00/*/numa_node; do cat $i; done | uniq
  -1

  # find /sys/ -name "numa_node" | while read fname; do cat $fname; \
    done | uniq
  -1

AMD based platforms provide a fall-back for this situation via amd_bus.c.
These platforms snoop out the information by directly reading specific
registers from the Northbridge and caching them via 'alloc_pci_root_info'.

Later during boot processing when host bridges are discovered -
'pci_acpi_scan_root' - the kernel looks for their corresponding ACPI _PXM
method - drivers/acpi/numa.c::acpi_get_node().  If the BIOS supplied a
_PXM method then that node (proximity) value is associated.  If the BIOS
did not supply a _PXM method *and* the platform is AMD based, the
fall-back cached values obtained directly from the Northbridge are used;
otherwise, "NUMA_NO_NODE" is associated.

There are a number of issues with this fall-back mechanism the most
notable being that amd_bus.c extracts a 3-bit number from a CPU register
and uses it as the node number.  The node numbers used by Linux are
logical and there's no reason they need to be identical to settings in the
CPU registers.  So if we have some node information obtained in the normal
way (from _PXM, SLIT, SRAT, etc.) and some from amd_bus.c, there's no
reason to believe they will be compatible.

This patch warns when this situation occurs:
  pci_root PNP0A08:00: [Firmware Bug]: no _PXM; falling back to node 0 from hardware (may be inconsistent with ACPI node numbers)

[1] https://bugzilla.kernel.org/show_bug.cgi?id=72051

Signed-off-by: Myron Stowe <myron.stowe@redhat.com>
Signed-off-by: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
---
 arch/x86/pci/acpi.c | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/x86/pci/acpi.c b/arch/x86/pci/acpi.c
index 01edac6..5075371 100644
--- a/arch/x86/pci/acpi.c
+++ b/arch/x86/pci/acpi.c
@@ -489,8 +489,12 @@ struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root)
 	}
 
 	node = acpi_get_node(device->handle);
-	if (node == NUMA_NO_NODE)
+	if (node == NUMA_NO_NODE) {
 		node = x86_pci_root_bus_node(busnum);
+		if (node != 0 && node != NUMA_NO_NODE)
+			dev_info(&device->dev, FW_BUG "no _PXM; falling back to node %d from hardware (may be inconsistent with ACPI node numbers)\n",
+				node);
+	}
 
 	if (node != NUMA_NO_NODE && !node_online(node))
 		node = NUMA_NO_NODE;
-- 
1.9.0

[PATCH V4 4/4] X86/PCI: Remove unnecessary 'quirk_amd_nb_node'

[suravee.suthikulpanit]

From: Myron Stowe <myron.stowe@redhat.com>

The quirk is used for fixing up the numa_node information in sysfs for AMD hostbridge
devices (i.e. PCI device 00:[18-1f].x).  However, this is currently unused.
and is becoming maintenance burden. Therefore, it is removed.

Signed-off-by: Myron Stowe <myron.stowe@redhat.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: Robert Richter <rric@kernel.org>
Cc: Daniel J Blueman <daniel@numascale.com>
Cc: Andreas Herrmann <herrmann.der.user@googlemail.com>
Signed-off-by: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
---
 arch/x86/kernel/quirks.c | 58 ------------------------------------------------
 1 file changed, 58 deletions(-)

diff --git a/arch/x86/kernel/quirks.c b/arch/x86/kernel/quirks.c
index ff898bb..abbd6bc 100644
--- a/arch/x86/kernel/quirks.c
+++ b/arch/x86/kernel/quirks.c
@@ -514,64 +514,6 @@ DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATI, PCI_DEVICE_ID_ATI_SBX00_SMBUS,
 
 #endif
 
-#if defined(CONFIG_PCI) && defined(CONFIG_NUMA)
-/* Set correct numa_node information for AMD NB functions */
-static void quirk_amd_nb_node(struct pci_dev *dev)
-{
-	struct pci_dev *nb_ht;
-	unsigned int devfn;
-	u32 node;
-	u32 val;
-
-	devfn = PCI_DEVFN(PCI_SLOT(dev->devfn), 0);
-	nb_ht = pci_get_slot(dev->bus, devfn);
-	if (!nb_ht)
-		return;
-
-	pci_read_config_dword(nb_ht, 0x60, &val);
-	node = pcibus_to_node(dev->bus) | (val & 7);
-	/*
-	 * Some hardware may return an invalid node ID,
-	 * so check it first:
-	 */
-	if (node_online(node))
-		set_dev_node(&dev->dev, node);
-	pci_dev_put(nb_ht);
-}
-
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_K8_NB,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_K8_NB_ADDRMAP,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_K8_NB_MEMCTL,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_K8_NB_MISC,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_10H_NB_HT,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_10H_NB_MAP,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_10H_NB_DRAM,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_10H_NB_MISC,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_10H_NB_LINK,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_15H_NB_F0,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_15H_NB_F1,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_15H_NB_F2,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_15H_NB_F3,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_15H_NB_F4,
-			quirk_amd_nb_node);
-DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, PCI_DEVICE_ID_AMD_15H_NB_F5,
-			quirk_amd_nb_node);
-
-#endif
-
 #ifdef CONFIG_PCI
 /*
  * Processor does not ensure DRAM scrub read/write sequence
-- 
1.9.0

Re: [PATCH V4 0/4] x86/pci Fix numa_node info for AMD hostbridge and misc clean up.

[Suravee Suthikulpanit]

Bjorn,

I am following up on this patch set.  Do you have any other concerns?

Thank you,

Suravee

On 05/08/2014 11:44 AM, suravee.suthikulpanit@amd.com wrote:
> From: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
>
> This in the V4 of the patch set which trys to fixed up numa_node for AMD hostbridges.
> This topic was discussed in the following threads:
>
> V1: https://lkml.org/lkml/2014/3/5/898
> V2: https://lkml.org/lkml/2014/4/18/623
> V3: https://lkml.org/lkml/2014/5/7/588
>
> Change from V3:
> Split up the changes in patch1 into two per Robert's suggestions. The second patch has
> no functional change.
>
> Change from V2:
> As requested by Bjorn, the patch set will only add supports for family15h, and
> also deprecating the mechanism, as future platforms should be relying on the ACPI info.
> He has also requested to keep the changes to the minimum. Therefore, the V2 patch 2 and 3
> has been removed.
>
> Suravee Suthikulpanit (2):
>    Patch1: x86/PCI: Fix PCI root numa_node info on AMD family15h
>    Patch2: x86/PCI: Clean up and mark early_root_info_init as deprecated
>
> Myron Stowe (2):
>    Patch3: ACPI/PCI: Warn if we have to "guess" host bridge node information
>    Patch4: X86/PCI: Remove unnecessary 'quirk_amd_nb_node'
>
>   arch/x86/kernel/quirks.c | 58 -------------------------------------
>   arch/x86/pci/acpi.c      |  6 +++-
>   arch/x86/pci/amd_bus.c   | 75 +++++++++++++++++++++++++++++-------------------
>   3 files changed, 51 insertions(+), 88 deletions(-)
>

Re: [PATCH V4 0/4] x86/pci Fix numa_node info for AMD hostbridge and misc clean up.

[Bjorn Helgaas]

On Tue, May 13, 2014 at 11:54 PM, Suravee Suthikulpanit
<suravee.suthikulpanit@amd.com> wrote:
> Bjorn,
>
> I am following up on this patch set.  Do you have any other concerns?

Sorry, I've been busy with some other work and haven't had a chance to
look at this yet.

> On 05/08/2014 11:44 AM, suravee.suthikulpanit@amd.com wrote:
>>
>> From: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
>>
>> This in the V4 of the patch set which trys to fixed up numa_node for AMD
>> hostbridges.
>> This topic was discussed in the following threads:
>>
>> V1: https://lkml.org/lkml/2014/3/5/898
>> V2: https://lkml.org/lkml/2014/4/18/623
>> V3: https://lkml.org/lkml/2014/5/7/588
>>
>> Change from V3:
>> Split up the changes in patch1 into two per Robert's suggestions. The
>> second patch has
>> no functional change.
>>
>> Change from V2:
>> As requested by Bjorn, the patch set will only add supports for family15h,
>> and
>> also deprecating the mechanism, as future platforms should be relying on
>> the ACPI info.
>> He has also requested to keep the changes to the minimum. Therefore, the
>> V2 patch 2 and 3
>> has been removed.
>>
>> Suravee Suthikulpanit (2):
>>    Patch1: x86/PCI: Fix PCI root numa_node info on AMD family15h
>>    Patch2: x86/PCI: Clean up and mark early_root_info_init as deprecated
>>
>> Myron Stowe (2):
>>    Patch3: ACPI/PCI: Warn if we have to "guess" host bridge node
>> information
>>    Patch4: X86/PCI: Remove unnecessary 'quirk_amd_nb_node'
>>
>>   arch/x86/kernel/quirks.c | 58 -------------------------------------
>>   arch/x86/pci/acpi.c      |  6 +++-
>>   arch/x86/pci/amd_bus.c   | 75
>> +++++++++++++++++++++++++++++-------------------
>>   3 files changed, 51 insertions(+), 88 deletions(-)
>>
>

[PATCH V5 0/4] x86/pci Fix numa_node info for AMD hostbridge and misc clean up

[Bjorn Helgaas]

[resending because I forgot to copy the lists, sorry guys]

Hi Suravee,

Sorry it took me so long to get to these patches.  Here's my proposal.  I
reordered them and added some comments in the code and changelogs, but I
think your patches look fine as-is.

So I just need comments on these two significant changes:

  1) I added a patch to stop enabling ECS after Fam16h, because that's
  another case of CPU-dependent code that we should not need to keep
  carrying.  I don't think there are any post-Fam16h CPUs yet, but I
  certainly don't want to do anything that will keep them from working when
  they do arrive.  It would be useful if somebody could test this on
  current platforms by tweaking the patch so we don't enable ECS on Fam15h.

  2) I dropped the quirk_amd_nb_node() removal.  I could be convinced
  otherwise, but I don't really object to the quirk because it is already
  explicitly limited to specific devices, and removing it will change
  things in sysfs.  I think the changes would be harmless as far as the
  kernel is concerned, since there are no drivers for these devices.  But
  Andreas added the quirk because of complaints, so apparently somebody is
  looking at what's in sysfs, and I don't want to get the same complaints
  again by removing it.  However, I will certainly ask questions if I see
  the quirk being extended to more devices.

The AMD BKDG does say the BIOS should provide an MCFG table (sec 2.8 of
42301), so I think it provides guidance matching the intent of my "stop
enabling ECS" patch.  But the BKDG doesn't mention _PXM at all.  Is there
any chance you could squeeze in a mention of that, so BIOS writers know
that they *should* provide it?  I want to avoid more fire-drills in the
future.

Bjorn

---

Bjorn Helgaas (1):
      x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

Myron Stowe (1):
      x86/PCI: Warn if we have to "guess" host bridge node information

Suravee Suthikulpanit (2):
      x86/PCI: Work around AMD Fam15h BIOSes that fail to provide _PXM
      x86/PCI: Clean up and mark early_root_info_init() as deprecated


 arch/x86/pci/acpi.c    |    6 +++
 arch/x86/pci/amd_bus.c |   87 +++++++++++++++++++++++++++++++-----------------
 2 files changed, 62 insertions(+), 31 deletions(-)

[PATCH V5 1/4] x86/PCI: Warn if we have to "guess" host bridge node information

[Bjorn Helgaas]

From: Myron Stowe <myron.stowe@redhat.com>

The vast majority of platforms are not supplying ACPI _PXM (proximity)
information corresponding to host bridge (PNP0A03/PNP0A08) devices
resulting in sysfs "numa_node" values of -1 (NUMA_NO_NODE):

  # for i in /sys/devices/pci0000\:00/*/numa_node; do cat $i; done | uniq
  -1

  # find /sys/ -name "numa_node" | while read fname; do cat $fname; \
    done | uniq
  -1

AMD based platforms provide a fall-back for this situation via amd_bus.c.
These platforms snoop out the information by directly reading specific
registers from the Northbridge and caching them via alloc_pci_root_info().

Later during boot processing when host bridges are discovered -
pci_acpi_scan_root() - the kernel looks for their corresponding ACPI _PXM
method - drivers/acpi/numa.c::acpi_get_node().  If the BIOS supplied a _PXM
method then that node (proximity) value is associated.  If the BIOS did not
supply a _PXM method *and* the platform is AMD-based, the fall-back cached
values obtained directly from the Northbridge are used; otherwise,
"NUMA_NO_NODE" is associated.

There are a number of issues with this fall-back mechanism the most notable
being that amd_bus.c extracts a 3-bit number from a CPU register and uses
it as the node number.  The node numbers used by Linux are logical and
there's no reason they need to be identical to settings in the CPU
registers.  So if we have some node information obtained in the normal way
(from _PXM, SLIT, SRAT, etc.) and some from amd_bus.c, there's no reason to
believe they will be compatible.

This patch warns when this situation occurs:

  pci_root PNP0A08:00: [Firmware Bug]: no _PXM; falling back to node 0 from hardware (may be inconsistent with ACPI node numbers)

Link: https://bugzilla.kernel.org/show_bug.cgi?id=72051
Signed-off-by: Myron Stowe <myron.stowe@redhat.com>
Signed-off-by: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
---
 arch/x86/pci/acpi.c |    6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/x86/pci/acpi.c b/arch/x86/pci/acpi.c
index 01edac6c5e18..5075371ab593 100644
--- a/arch/x86/pci/acpi.c
+++ b/arch/x86/pci/acpi.c
@@ -489,8 +489,12 @@ struct pci_bus *pci_acpi_scan_root(struct acpi_pci_root *root)
 	}
 
 	node = acpi_get_node(device->handle);
-	if (node == NUMA_NO_NODE)
+	if (node == NUMA_NO_NODE) {
 		node = x86_pci_root_bus_node(busnum);
+		if (node != 0 && node != NUMA_NO_NODE)
+			dev_info(&device->dev, FW_BUG "no _PXM; falling back to node %d from hardware (may be inconsistent with ACPI node numbers)\n",
+				node);
+	}
 
 	if (node != NUMA_NO_NODE && !node_online(node))
 		node = NUMA_NO_NODE;

[PATCH V5 2/4] x86/PCI: Work around AMD Fam15h BIOSes that fail to provide _PXM

[Bjorn Helgaas]

From: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>

The BIOS is supposed to provide ACPI _PXM methods for PCI host bridges if
it cares about platform topology.  But some BIOSes do not, so add Fam15h
to the list of CPUs for which we fall back to reading node numbers from the
hardware.

Note that pci_acpi_scan_root() warns about the BIOS bug if we use this
information because (1) the hardware node numbers are not necessarily
compatible with other logical node numbers from ACPI, and (2) the lack of
_PXM forces OS updates that would not otherwise be required.

[bhelgaas: changelog, comments]
Link: https://bugzilla.kernel.org/show_bug.cgi?id=72051
Tested-by: Aravind Gopalakrishnan <Aravind.Gopalakrishnan@amd.com>
Signed-off-by: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: Myron Stowe <myron.stowe@redhat.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: Robert Richter <rric@kernel.org>
Cc: Daniel J Blueman <daniel@numascale.com>
Cc: Andreas Herrmann <herrmann.der.user@googlemail.com>
---
 arch/x86/pci/amd_bus.c |   25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)

diff --git a/arch/x86/pci/amd_bus.c b/arch/x86/pci/amd_bus.c
index e88f4c53d7f6..aa936e3a2019 100644
--- a/arch/x86/pci/amd_bus.c
+++ b/arch/x86/pci/amd_bus.c
@@ -24,10 +24,11 @@ struct pci_hostbridge_probe {
 };
 
 static struct pci_hostbridge_probe pci_probes[] __initdata = {
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1100 },
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1200 },
-	{ 0xff, 0, PCI_VENDOR_ID_AMD, 0x1200 },
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1300 },
+	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1100 }, /* K8 */
+	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1200 }, /* Fam10h */
+	{ 0xff, 0, PCI_VENDOR_ID_AMD, 0x1200 }, /* Fam10h */
+	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1300 }, /* Fam11h */
+	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1600 }, /* Fam15h */
 };
 
 #define RANGE_NUM 16
@@ -96,6 +97,11 @@ static int __init early_fill_mp_bus_info(void)
 	if (!found)
 		return 0;
 
+	/*
+	 * We should learn topology and routing information from _PXM and
+	 * _CRS methods in the ACPI namespace.  We extract node numbers
+	 * here to work around BIOSes that don't supply _PXM.
+	 */
 	for (i = 0; i < 4; i++) {
 		int min_bus;
 		int max_bus;
@@ -113,6 +119,17 @@ static int __init early_fill_mp_bus_info(void)
 		info = alloc_pci_root_info(min_bus, max_bus, node, link);
 	}
 
+	/*
+	 * The following code extracts routing information for use on old
+	 * systems where Linux doesn't automatically use host bridge _CRS
+	 * methods (or when the user specifies "pci=nocrs").
+	 *
+	 * We only do this through Fam11h, because _CRS should be enough on
+	 * newer systems.
+	 */
+	if (boot_cpu_data.x86 > 0x11)
+		return 0;
+
 	/* get the default node and link for left over res */
 	reg = read_pci_config(bus, slot, 0, 0x60);
 	def_node = (reg >> 8) & 0x07;

[PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Bjorn Helgaas]

ECS is an AMD mechanism that allows access to extended PCI config space
(offsets 256-4096) via I/O ports CF8/CFCh.  We normally use ECAM, i.e.,
MMCONFIG, to access that space, but apparently old machines have issues
that meant we couldn't use ECAM.

The solution was to enable ECS so we could use configuration mechanism #1
(I/O ports CF8/CFCh) to access extended config space.  See 831d991821da
("x86: add PCI extended config space access for AMD Barcelona").

New machines should be able to use ECAM, which means they don't need the
CPU-specific code to enable ECS.  This patch leaves ECS the same on all
existing platforms, but stops enabling it on platforms after Fam16h.

Those future platforms should be able to use the standard ACPI MCFG/_CBA
descriptions of the PCIe ECAM mechanism.

Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
CC: Robert Richter <rric@kernel.org>
---
 arch/x86/pci/amd_bus.c |    4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/arch/x86/pci/amd_bus.c b/arch/x86/pci/amd_bus.c
index aa936e3a2019..67dadf179348 100644
--- a/arch/x86/pci/amd_bus.c
+++ b/arch/x86/pci/amd_bus.c
@@ -405,7 +405,9 @@ static int __init amd_postcore_init(void)
 		return 0;
 
 	early_fill_mp_bus_info();
-	pci_io_ecs_init();
+
+	if (boot_cpu_data.x86 <= 0x16)
+		pci_io_ecs_init();
 
 	return 0;
 }

[PATCH V5 4/4] x86/PCI: Clean up and mark early_root_info_init() as deprecated

[Bjorn Helgaas]

From: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>

early_root_info_init() is now deprecated in favor of info in ACPI.  Add a
note to that effect.  Also, clean up the code a bit.

There is no functional change.

Signed-off-by: Suravee Suthikulpanit <Suravee.Suthikulpanit@amd.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
---
 arch/x86/pci/amd_bus.c |   68 +++++++++++++++++++++++++++---------------------
 1 file changed, 38 insertions(+), 30 deletions(-)

diff --git a/arch/x86/pci/amd_bus.c b/arch/x86/pci/amd_bus.c
index 67dadf179348..cdce6ed59e2c 100644
--- a/arch/x86/pci/amd_bus.c
+++ b/arch/x86/pci/amd_bus.c
@@ -11,28 +11,33 @@
 
 #include "bus_numa.h"
 
-/*
- * This discovers the pcibus <-> node mapping on AMD K8.
- * also get peer root bus resource for io,mmio
- */
+#define AMD_NB_F0_NODE_ID			0x60
+#define AMD_NB_F0_UNIT_ID			0x64
+#define AMD_NB_F1_CONFIG_MAP_REG		0xe0
+
+#define RANGE_NUM				16
+#define AMD_NB_F1_CONFIG_MAP_RANGES		4
 
-struct pci_hostbridge_probe {
+struct amd_hostbridge {
 	u32 bus;
 	u32 slot;
-	u32 vendor;
 	u32 device;
 };
 
-static struct pci_hostbridge_probe pci_probes[] __initdata = {
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1100 }, /* K8 */
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1200 }, /* Fam10h */
-	{ 0xff, 0, PCI_VENDOR_ID_AMD, 0x1200 }, /* Fam10h */
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1300 }, /* Fam11h */
-	{ 0, 0x18, PCI_VENDOR_ID_AMD, 0x1600 }, /* Fam15h */
+/*
+ * IMPORTANT NOTE:
+ * hb_probes[] and early_root_info_init() is in maintenance mode.
+ * It only supports K8, Fam10h, Fam11h, and Fam15h_00h-0fh .
+ * Future processor will rely on information in ACPI.
+ */
+static struct amd_hostbridge hb_probes[] __initdata = {
+	{ 0, 0x18, 0x1100 }, /* K8 */
+	{ 0, 0x18, 0x1200 }, /* Family10h */
+	{ 0xff, 0, 0x1200 }, /* Family10h */
+	{ 0, 0x18, 0x1300 }, /* Family11h */
+	{ 0, 0x18, 0x1600 }, /* Family15h */
 };
 
-#define RANGE_NUM 16
-
 static struct pci_root_info __init *find_pci_root_info(int node, int link)
 {
 	struct pci_root_info *info;
@@ -46,12 +51,12 @@ static struct pci_root_info __init *find_pci_root_info(int node, int link)
 }
 
 /**
- * early_fill_mp_bus_to_node()
+ * early_root_info_init()
  * called before pcibios_scan_root and pci_scan_bus
- * fills the mp_bus_to_cpumask array based according to the LDT Bus Number
- * Registers found in the K8 northbridge
+ * fills the mp_bus_to_cpumask array based according
+ * to the LDT Bus Number Registers found in the northbridge.
  */
-static int __init early_fill_mp_bus_info(void)
+static int __init early_root_info_init(void)
 {
 	int i;
 	unsigned bus;
@@ -76,19 +81,21 @@ static int __init early_fill_mp_bus_info(void)
 		return -1;
 
 	found = false;
-	for (i = 0; i < ARRAY_SIZE(pci_probes); i++) {
+	for (i = 0; i < ARRAY_SIZE(hb_probes); i++) {
 		u32 id;
 		u16 device;
 		u16 vendor;
 
-		bus = pci_probes[i].bus;
-		slot = pci_probes[i].slot;
+		bus = hb_probes[i].bus;
+		slot = hb_probes[i].slot;
 		id = read_pci_config(bus, slot, 0, PCI_VENDOR_ID);
-
 		vendor = id & 0xffff;
 		device = (id>>16) & 0xffff;
-		if (pci_probes[i].vendor == vendor &&
-		    pci_probes[i].device == device) {
+
+		if (vendor != PCI_VENDOR_ID_AMD)
+			continue;
+
+		if (hb_probes[i].device == device) {
 			found = true;
 			break;
 		}
@@ -102,10 +109,11 @@ static int __init early_fill_mp_bus_info(void)
 	 * _CRS methods in the ACPI namespace.  We extract node numbers
 	 * here to work around BIOSes that don't supply _PXM.
 	 */
-	for (i = 0; i < 4; i++) {
+	for (i = 0; i < AMD_NB_F1_CONFIG_MAP_RANGES; i++) {
 		int min_bus;
 		int max_bus;
-		reg = read_pci_config(bus, slot, 1, 0xe0 + (i << 2));
+		reg = read_pci_config(bus, slot, 1,
+				AMD_NB_F1_CONFIG_MAP_REG + (i << 2));
 
 		/* Check if that register is enabled for bus range */
 		if ((reg & 7) != 3)
@@ -131,9 +139,9 @@ static int __init early_fill_mp_bus_info(void)
 		return 0;
 
 	/* get the default node and link for left over res */
-	reg = read_pci_config(bus, slot, 0, 0x60);
+	reg = read_pci_config(bus, slot, 0, AMD_NB_F0_NODE_ID);
 	def_node = (reg >> 8) & 0x07;
-	reg = read_pci_config(bus, slot, 0, 0x64);
+	reg = read_pci_config(bus, slot, 0, AMD_NB_F0_UNIT_ID);
 	def_link = (reg >> 8) & 0x03;
 
 	memset(range, 0, sizeof(range));
@@ -380,7 +388,7 @@ static int __init pci_io_ecs_init(void)
 	int cpu;
 
 	/* assume all cpus from fam10h have IO ECS */
-        if (boot_cpu_data.x86 < 0x10)
+	if (boot_cpu_data.x86 < 0x10)
 		return 0;
 
 	/* Try the PCI method first. */
@@ -404,7 +412,7 @@ static int __init amd_postcore_init(void)
 	if (boot_cpu_data.x86_vendor != X86_VENDOR_AMD)
 		return 0;
 
-	early_fill_mp_bus_info();
+	early_root_info_init();
 
 	if (boot_cpu_data.x86 <= 0x16)
 		pci_io_ecs_init();

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Borislav Petkov]

On Wed, May 21, 2014 at 05:18:17PM -0600, Bjorn Helgaas wrote:
> ECS is an AMD mechanism that allows access to extended PCI config space
> (offsets 256-4096) via I/O ports CF8/CFCh.  We normally use ECAM, i.e.,
> MMCONFIG, to access that space, but apparently old machines have issues
> that meant we couldn't use ECAM.
> 
> The solution was to enable ECS so we could use configuration mechanism #1
> (I/O ports CF8/CFCh) to access extended config space.  See 831d991821da
> ("x86: add PCI extended config space access for AMD Barcelona").
> 
> New machines should be able to use ECAM, which means they don't need the
> CPU-specific code to enable ECS.  This patch leaves ECS the same on all
> existing platforms, but stops enabling it on platforms after Fam16h.
> 
> Those future platforms should be able to use the standard ACPI MCFG/_CBA
> descriptions of the PCIe ECAM mechanism.
> 
> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
> CC: Robert Richter <rric@kernel.org>
> ---
>  arch/x86/pci/amd_bus.c |    4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/x86/pci/amd_bus.c b/arch/x86/pci/amd_bus.c
> index aa936e3a2019..67dadf179348 100644
> --- a/arch/x86/pci/amd_bus.c
> +++ b/arch/x86/pci/amd_bus.c
> @@ -405,7 +405,9 @@ static int __init amd_postcore_init(void)
>  		return 0;
>  
>  	early_fill_mp_bus_info();
> -	pci_io_ecs_init();
> +
> +	if (boot_cpu_data.x86 <= 0x16)

Fam 0x16, i.e. hex? I think we talked about fam 0x10, i.e. 16 decimal.

Oh well, that's an AMD call now.

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Bjorn Helgaas]

On Wed, May 21, 2014 at 5:38 PM, Borislav Petkov <bp@alien8.de> wrote:
> On Wed, May 21, 2014 at 05:18:17PM -0600, Bjorn Helgaas wrote:
>> ECS is an AMD mechanism that allows access to extended PCI config space
>> (offsets 256-4096) via I/O ports CF8/CFCh.  We normally use ECAM, i.e.,
>> MMCONFIG, to access that space, but apparently old machines have issues
>> that meant we couldn't use ECAM.
>>
>> The solution was to enable ECS so we could use configuration mechanism #1
>> (I/O ports CF8/CFCh) to access extended config space.  See 831d991821da
>> ("x86: add PCI extended config space access for AMD Barcelona").
>>
>> New machines should be able to use ECAM, which means they don't need the
>> CPU-specific code to enable ECS.  This patch leaves ECS the same on all
>> existing platforms, but stops enabling it on platforms after Fam16h.
>>
>> Those future platforms should be able to use the standard ACPI MCFG/_CBA
>> descriptions of the PCIe ECAM mechanism.
>>
>> Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
>> CC: Robert Richter <rric@kernel.org>
>> ---
>>  arch/x86/pci/amd_bus.c |    4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/x86/pci/amd_bus.c b/arch/x86/pci/amd_bus.c
>> index aa936e3a2019..67dadf179348 100644
>> --- a/arch/x86/pci/amd_bus.c
>> +++ b/arch/x86/pci/amd_bus.c
>> @@ -405,7 +405,9 @@ static int __init amd_postcore_init(void)
>>               return 0;
>>
>>       early_fill_mp_bus_info();
>> -     pci_io_ecs_init();
>> +
>> +     if (boot_cpu_data.x86 <= 0x16)
>
> Fam 0x16, i.e. hex? I think we talked about fam 0x10, i.e. 16 decimal.
>
> Oh well, that's an AMD call now.

I chose Fam16h (0x16) because it looks like that's the newest stuff
that's in the field.  I suspect things would probably work if we
changed this patch to leave ECS disabled on some Fam16h, Fam15h, etc.,
but that would change behavior on existing systems, which obviously
adds some risk.  I didn't think there was much benefit that makes the
risk worthwhile.

My goal is to stop needing CPU-specific changes in the future, not
necessarily to remove the CPU-specific code we already have.

Does that make sense?  I'm not sure whether I understood your real question.

Bjorn

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Borislav Petkov]

On Thu, May 22, 2014 at 11:56:03AM -0600, Bjorn Helgaas wrote:
> I chose Fam16h (0x16) because it looks like that's the newest stuff
> that's in the field. I suspect things would probably work if we
> changed this patch to leave ECS disabled on some Fam16h, Fam15h, etc.,
> but that would change behavior on existing systems, which obviously
> adds some risk. I didn't think there was much benefit that makes the
> risk worthwhile.
>
> My goal is to stop needing CPU-specific changes in the future, not
> necessarily to remove the CPU-specific code we already have.
>
> Does that make sense? I'm not sure whether I understood your real
> question.

No, you got it right. I'm just wondering why only the newest stuff.
MMCONFIG is supposed to work just fine on everything from Fam10h
onwards, I'm not sure all Fam10h supported it though. Maybe Suravee can
verify that...

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Bjorn Helgaas]

On Thu, May 22, 2014 at 1:17 PM, Borislav Petkov <bp@alien8.de> wrote:
> On Thu, May 22, 2014 at 11:56:03AM -0600, Bjorn Helgaas wrote:
>> I chose Fam16h (0x16) because it looks like that's the newest stuff
>> that's in the field. I suspect things would probably work if we
>> changed this patch to leave ECS disabled on some Fam16h, Fam15h, etc.,
>> but that would change behavior on existing systems, which obviously
>> adds some risk. I didn't think there was much benefit that makes the
>> risk worthwhile.
>>
>> My goal is to stop needing CPU-specific changes in the future, not
>> necessarily to remove the CPU-specific code we already have.
>>
>> Does that make sense? I'm not sure whether I understood your real
>> question.
>
> No, you got it right. I'm just wondering why only the newest stuff.
> MMCONFIG is supposed to work just fine on everything from Fam10h
> onwards, I'm not sure all Fam10h supported it though. Maybe Suravee can
> verify that...

Even if MMCONFIG does work fine on everything from Fam10h onwards, we
still depend on the BIOS to provide a correct MCFG table.  I don't
think we can guarantee that changing from ECS to MMCONFIG on a Fam16h
box in the field is safe, because we'd then be using a feature we've
never used before.

Bjorn

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Borislav Petkov]

On Thu, May 22, 2014 at 02:20:04PM -0600, Bjorn Helgaas wrote:
> Even if MMCONFIG does work fine on everything from Fam10h onwards, we
> still depend on the BIOS to provide a correct MCFG table.

It is very hazy memory of mine that starting with F10h, we had an MCFG
table always present but I don't remember anymore. Maybe I'm even
remembering it wrong.

In any case, as Andreas and I pointed out, this is AMD's decision so if
they're fine with F16h, then it is definitely not my place to disagree.

:-)

-- 
Regards/Gruss,
    Boris.

Sent from a fat crate under my desk. Formatting is fine.
--

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Suravee Suthikulanit]

On 5/22/2014 3:20 PM, Bjorn Helgaas wrote:
> On Thu, May 22, 2014 at 1:17 PM, Borislav Petkov <bp@alien8.de> wrote:
>> On Thu, May 22, 2014 at 11:56:03AM -0600, Bjorn Helgaas wrote:
>>> I chose Fam16h (0x16) because it looks like that's the newest stuff
>>> that's in the field. I suspect things would probably work if we
>>> changed this patch to leave ECS disabled on some Fam16h, Fam15h, etc.,
>>> but that would change behavior on existing systems, which obviously
>>> adds some risk. I didn't think there was much benefit that makes the
>>> risk worthwhile.
>>>
>>> My goal is to stop needing CPU-specific changes in the future, not
>>> necessarily to remove the CPU-specific code we already have.
>>>
>>> Does that make sense? I'm not sure whether I understood your real
>>> question.
>>
>> No, you got it right. I'm just wondering why only the newest stuff.
>> MMCONFIG is supposed to work just fine on everything from Fam10h
>> onwards, I'm not sure all Fam10h supported it though. Maybe Suravee can
>> verify that...
>
> Even if MMCONFIG does work fine on everything from Fam10h onwards, we
> still depend on the BIOS to provide a correct MCFG table.  I don't
> think we can guarantee that changing from ECS to MMCONFIG on a Fam16h
> box in the field is safe, because we'd then be using a feature we've
> never used before.
>
> Bjorn
>

At this point, family11h and later (upto 16h which is our most current 
processor) should already have supports for the MCFG. However, we can't 
guarantee that all the systems currently out there would not use the 
ECS. So, I think it is ok to say we won't support it post 16h as Bjorn 
suggests.

Suravee

Re: [PATCH V5 0/4] x86/pci Fix numa_node info for AMD hostbridge and misc clean up

[Suravee Suthikulanit]

On 5/21/2014 6:17 PM, Bjorn Helgaas wrote:
> [resending because I forgot to copy the lists, sorry guys]
>
> Hi Suravee,
>
> Sorry it took me so long to get to these patches.  Here's my proposal.  I
> reordered them and added some comments in the code and changelogs, but I
> think your patches look fine as-is.
>
> So I just need comments on these two significant changes:
>
>    1) I added a patch to stop enabling ECS after Fam16h, because that's
>    another case of CPU-dependent code that we should not need to keep
>    carrying.  I don't think there are any post-Fam16h CPUs yet, but I
>    certainly don't want to do anything that will keep them from working when
>    they do arrive.  It would be useful if somebody could test this on
>    current platforms by tweaking the patch so we don't enable ECS on Fam15h.

I'm okay with this. In my V4 patch, I also tested disabling ECS on 
family15h already and that seems to work fine.  But I would not mind 
deprecate this for post fam16h processors.

>
>    2) I dropped the quirk_amd_nb_node() removal.  I could be convinced
>    otherwise, but I don't really object to the quirk because it is already
>    explicitly limited to specific devices, and removing it will change
>    things in sysfs.  I think the changes would be harmless as far as the
>    kernel is concerned, since there are no drivers for these devices.  But
>    Andreas added the quirk because of complaints, so apparently somebody is
>    looking at what's in sysfs, and I don't want to get the same complaints
>    again by removing it.  However, I will certainly ask questions if I see
>    the quirk being extended to more devices.

I'm okay with keeping this.

>
> The AMD BKDG does say the BIOS should provide an MCFG table (sec 2.8 of
> 42301), so I think it provides guidance matching the intent of my "stop
> enabling ECS" patch.  But the BKDG doesn't mention _PXM at all.  Is there
> any chance you could squeeze in a mention of that, so BIOS writers know
> that they *should* provide it?  I want to avoid more fire-drills in the
> future.

The ship for family15h stuff have sailed and we probably would not be 
able to get them to change.  We are going to have to keep an eye on the 
future platforms to make sure that the _PXM info is documented for 
future platforms.

I have tested this patch set and they seems to be ok now.

Suravee

>
> Bjorn
>
> ---
>
> Bjorn Helgaas (1):
>        x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h
>
> Myron Stowe (1):
>        x86/PCI: Warn if we have to "guess" host bridge node information
>
> Suravee Suthikulpanit (2):
>        x86/PCI: Work around AMD Fam15h BIOSes that fail to provide _PXM
>        x86/PCI: Clean up and mark early_root_info_init() as deprecated
>
>
>   arch/x86/pci/acpi.c    |    6 +++
>   arch/x86/pci/amd_bus.c |   87 +++++++++++++++++++++++++++++++-----------------
>   2 files changed, 62 insertions(+), 31 deletions(-)
>

Re: [PATCH V5 0/4] x86/pci Fix numa_node info for AMD hostbridge and misc clean up

[Suravee Suthikulanit]

[Resending due to mail client crashing]

On 5/21/2014 6:17 PM, Bjorn Helgaas wrote:
> [resending because I forgot to copy the lists, sorry guys]
>
> Hi Suravee,
>
> Sorry it took me so long to get to these patches.  Here's my proposal.  I
> reordered them and added some comments in the code and changelogs, but I
> think your patches look fine as-is.
>
> So I just need comments on these two significant changes:
>
>    1) I added a patch to stop enabling ECS after Fam16h, because that's
>    another case of CPU-dependent code that we should not need to keep
>    carrying.  I don't think there are any post-Fam16h CPUs yet, but I
>    certainly don't want to do anything that will keep them from working when
>    they do arrive.  It would be useful if somebody could test this on
>    current platforms by tweaking the patch so we don't enable ECS on Fam15h.

I'm okay with this. In my V4 patch, I also tested disabling ECS on 
family15h already and that seems to work fine.  But I would not mind 
deprecate this for post fam16h processors.

>
>    2) I dropped the quirk_amd_nb_node() removal.  I could be convinced
>    otherwise, but I don't really object to the quirk because it is already
>    explicitly limited to specific devices, and removing it will change
>    things in sysfs.  I think the changes would be harmless as far as the
>    kernel is concerned, since there are no drivers for these devices.  But
>    Andreas added the quirk because of complaints, so apparently somebody is
>    looking at what's in sysfs, and I don't want to get the same complaints
>    again by removing it.  However, I will certainly ask questions if I see
>    the quirk being extended to more devices.

I'm okay with keeping this.

> The AMD BKDG does say the BIOS should provide an MCFG table (sec 2.8 of
> 42301), so I think it provides guidance matching the intent of my "stop
> enabling ECS" patch.  But the BKDG doesn't mention _PXM at all.  Is there
> any chance you could squeeze in a mention of that, so BIOS writers know
> that they *should* provide it?  I want to avoid more fire-drills in the
> future.
>
> Bjorn

The ship for family15h stuff have sailed and we probably would not be 
able to get them to change.  We are going to have to keep an eye on the 
future platforms to make sure that the _PXM info is documented for 
future platforms.

I have tested this patch set and they seems to be ok now.

Suravee

>
> ---
>
> Bjorn Helgaas (1):
>        x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h
>
> Myron Stowe (1):
>        x86/PCI: Warn if we have to "guess" host bridge node information
>
> Suravee Suthikulpanit (2):
>        x86/PCI: Work around AMD Fam15h BIOSes that fail to provide _PXM
>        x86/PCI: Clean up and mark early_root_info_init() as deprecated
>
>
>   arch/x86/pci/acpi.c    |    6 +++
>   arch/x86/pci/amd_bus.c |   87 +++++++++++++++++++++++++++++++-----------------
>   2 files changed, 62 insertions(+), 31 deletions(-)
>

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Bjorn Helgaas]

On Thu, May 22, 2014 at 5:39 PM, Suravee Suthikulanit
<suravee.suthikulpanit@amd.com> wrote:
> On 5/22/2014 3:20 PM, Bjorn Helgaas wrote:
>> On Thu, May 22, 2014 at 1:17 PM, Borislav Petkov <bp@alien8.de> wrote:
>>> On Thu, May 22, 2014 at 11:56:03AM -0600, Bjorn Helgaas wrote:
>>>>
>>>> I chose Fam16h (0x16) because it looks like that's the newest stuff
>>>> that's in the field. I suspect things would probably work if we
>>>> changed this patch to leave ECS disabled on some Fam16h, Fam15h, etc.,
>>>> but that would change behavior on existing systems, which obviously
>>>> adds some risk. I didn't think there was much benefit that makes the
>>>> risk worthwhile.
>>>>
>>>> My goal is to stop needing CPU-specific changes in the future, not
>>>> necessarily to remove the CPU-specific code we already have.
>>>>
>>>> Does that make sense? I'm not sure whether I understood your real
>>>> question.
>>>
>>> No, you got it right. I'm just wondering why only the newest stuff.
>>> MMCONFIG is supposed to work just fine on everything from Fam10h
>>> onwards, I'm not sure all Fam10h supported it though. Maybe Suravee can
>>> verify that...
>>
>> Even if MMCONFIG does work fine on everything from Fam10h onwards, we
>> still depend on the BIOS to provide a correct MCFG table.  I don't
>> think we can guarantee that changing from ECS to MMCONFIG on a Fam16h
>> box in the field is safe, because we'd then be using a feature we've
>> never used before.
>
> At this point, family11h and later (upto 16h which is our most current
> processor) should already have supports for the MCFG. However, we can't
> guarantee that all the systems currently out there would not use the ECS.
> So, I think it is ok to say we won't support it post 16h as Bjorn suggests.

I think this is more a BIOS question than a hardware question.  I'm
sure all current AMD hardware supports ECAM just fine.  But it's still
up to the BIOS to produce a valid MCFG table, so OEMs could still have
issues, and *that's* what I'm worried about.

I've been poking around for recent dmesg logs that contain "PCI: Using
configuration type 1 for extended access", and there are quite a few.
In most cases there *is* an MCFG table, but apparently we decide not
to use it for some reason (unfortunately we don't print the specific
reason).  One example is at
https://bugzilla.kernel.org/show_bug.cgi?id=68591 .

I'm going to go out on a limb and guess that Windows does not enable
ECS, so it probably uses ECAM.  Therefore, I suspect Linux's parsing
of MCFG is broken in some way, and we probably *could* use ECAM in all
these cases I'm seeing.

It would probably be prudent to figure out why Linux is rejecting
these MCFG tables.  We'll probably see similar tables on Fam17h
systems, and if we continue rejecting them, and we don't turn on ECS,
we won't be able to access extended config space.

I opened a bugzilla for this issue:
https://bugzilla.kernel.org/show_bug.cgi?id=76771

I'm wavering on whether it's a good idea to put this patch in before
understanding the issue.  As much as I'd like to stop fiddling with
ECS, we'd likely end up with a v3.15 where extended config space
doesn't work on some Fam17h systems.

Bjorn

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Robert Richter]

On 22.05.14 20:54:54, Bjorn Helgaas wrote:
> I'm going to go out on a limb and guess that Windows does not enable
> ECS, so it probably uses ECAM.  Therefore, I suspect Linux's parsing
> of MCFG is broken in some way, and we probably *could* use ECAM in all
> these cases I'm seeing.

Even if ECS is not enabled the system should be fine anyway, as ECS is
only used to enable certain features. For family 10h this was
originally the IBS EILVT (extended interrupt local vector table,
needed for hw profiling) setup which need to be set by the OS which
the BIOS didn't right. This should be fixed now and properly set by
the BIOS on 15h+ systems.

I don't remember what was added to 16h where ECS was needed, I think
there was one (Suravee?). Not sure if this is essential.

So using MCFG should be fine, since if in the rare case when it is
broken, the system should work properly anyway without it and only
some special features, if any, do not work anymore (e.g. IBS should
work fine).

-Robert

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Bjorn Helgaas]

On Fri, May 23, 2014 at 5:56 AM, Robert Richter <rric@kernel.org> wrote:
> On 22.05.14 20:54:54, Bjorn Helgaas wrote:
>> I'm going to go out on a limb and guess that Windows does not enable
>> ECS, so it probably uses ECAM.  Therefore, I suspect Linux's parsing
>> of MCFG is broken in some way, and we probably *could* use ECAM in all
>> these cases I'm seeing.
>
> Even if ECS is not enabled the system should be fine anyway, as ECS is
> only used to enable certain features. For family 10h this was
> originally the IBS EILVT (extended interrupt local vector table,
> needed for hw profiling) setup which need to be set by the OS which
> the BIOS didn't right. This should be fixed now and properly set by
> the BIOS on 15h+ systems.
>
> I don't remember what was added to 16h where ECS was needed, I think
> there was one (Suravee?). Not sure if this is essential.
>
> So using MCFG should be fine, since if in the rare case when it is
> broken, the system should work properly anyway without it and only
> some special features, if any, do not work anymore (e.g. IBS should
> work fine).

[I guess I've been using the wrong term here.  I think "ECS" just
refers to the extended config space itself, and I should have been
saying "IO ECS" or "EnableCf8ExtCfg".]

My understanding was that if we don't enable IO ECS and we don't have
MCFG, we will not be able to access extended config space.  The system
can certainly boot without extended config space, but some drivers may
not work correctly, so it would be a regression from the user point of
view.

The case where MCFG is broken may be rare, but I still want to avoid
the regression.  And my guess is that the MCFG is not really broken in
these cases, because I suspect Windows is using it.  Our MCFG parsing
code is a mess, and I suspect that we are claiming MCFG is broken when
it really isn't.

If I understand correctly, IO ECS is also limited to domain 0, so if
we have multiple PCI host bridges, i.e., multiple PNP0A03/PNP0A08
devices, and we want them in different domains, we'd have to use ECAM.

Bjorn

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Robert Richter]

On 23.05.14 07:01:41, Bjorn Helgaas wrote:
> [I guess I've been using the wrong term here.  I think "ECS" just
> refers to the extended config space itself, and I should have been
> saying "IO ECS" or "EnableCf8ExtCfg".]
> 
> My understanding was that if we don't enable IO ECS and we don't have
> MCFG, we will not be able to access extended config space.  The system
> can certainly boot without extended config space, but some drivers may
> not work correctly, so it would be a regression from the user point of
> view.

No, I got you right. If we disable IO ECS there is no fallback if MCFG
fails... and this may cause a regression then.

I might be completely wrong here, but as I remember IO ECS only
affects access to cpu devices (bus 0, slot 0x18-0x1f). Thus, esp. PCIe
extended config space access works since a different host controller
handles this. So only cpu devices would see a regression and thus cpu
bringup code. I don't think ECS (either MCFG or IO ECS) is needed
anymore for cpu bringup (this is true for IBS, but I don't know of
other cpu features requiring ECS, though family 16h might introduced
new ones).

IMHO device drivers are not affected and wont get broken.

-Robert

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Suravee Suthikulanit]

On 5/23/2014 6:56 AM, Robert Richter wrote:
> On 22.05.14 20:54:54, Bjorn Helgaas wrote:
>> I'm going to go out on a limb and guess that Windows does not enable
>> ECS, so it probably uses ECAM.  Therefore, I suspect Linux's parsing
>> of MCFG is broken in some way, and we probably *could* use ECAM in all
>> these cases I'm seeing.
>
> Even if ECS is not enabled the system should be fine anyway, as ECS is
> only used to enable certain features. For family 10h this was
> originally the IBS EILVT (extended interrupt local vector table,
> needed for hw profiling) setup which need to be set by the OS which
> the BIOS didn't right. This should be fixed now and properly set by
> the BIOS on 15h+ systems.
>
> I don't remember what was added to 16h where ECS was needed, I think
> there was one (Suravee?). Not sure if this is essential.

I am not aware of anything specific in the family16h which require 
IO_ECS to be enabled.

Suravee

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Suravee Suthikulanit]

On 5/22/2014 9:54 PM, Bjorn Helgaas wrote:
> I've been poking around for recent dmesg logs that contain "PCI: Using
> configuration type 1 for extended access", and there are quite a few.
> In most cases there*is*  an MCFG table, but apparently we decide not
> to use it for some reason (unfortunately we don't print the specific
> reason).  One example is at
> https://bugzilla.kernel.org/show_bug.cgi?id=68591  .
>
> I'm going to go out on a limb and guess that Windows does not enable
> ECS, so it probably uses ECAM.  Therefore, I suspect Linux's parsing
> of MCFG is broken in some way, and we probably*could*  use ECAM in all
> these cases I'm seeing.
>
> It would probably be prudent to figure out why Linux is rejecting
> these MCFG tables.  We'll probably see similar tables on Fam17h
> systems, and if we continue rejecting them, and we don't turn on ECS,
> we won't be able to access extended config space.
>
> I opened a bugzilla for this issue:
> https://bugzilla.kernel.org/show_bug.cgi?id=76771
>
> I'm wavering on whether it's a good idea to put this patch in before
> understanding the issue.  As much as I'd like to stop fiddling with
> ECS, we'd likely end up with a v3.15 where extended config space
> doesn't work on some Fam17h systems.

So, I have located a system which presents issue with MMCONFIG. Here is 
my investigation:

DEBUG: pci_io_ecs_init: pci_probe = 4000f
ACPI: bus type PCI registered
DEBUG: -----> pci_mmcfg_early_init
DEBUG: pci_parse_mcfg
PCI: MMCONFIG for domain 0000 [bus 00-01] at [mem 0xe0000000-0xe01fffff] 
(base 0xe0000000)
DEBUG: pci_mmcfg_check_reserved
DEBUG: is_mmconf_reserved: method = E820
PCI: not using MMCONFIG
DEBUG: pci_direct_init
PCI: Using configuration type 1 for base access
PCI: Using configuration type 1 for extended access
ACPI: Added _OSI(Module Device)
ACPI: Added _OSI(Processor Device)
ACPI: Added _OSI(3.0 _SCP Extensions)
ACPI: Added _OSI(Processor Aggregator Device)
[Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored
\_SB_:_OSC invalid UUID
_OSC request data:1 1f
ACPI: Interpreter enabled
ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_] 
(20140214/hwxface-580)
ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_] 
(20140214/hwxface-580)
ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_] 
(20140214/hwxface-580)
ACPI: (supports S0 S3 S5)
ACPI: Using IOAPIC for interrupt routing
DEBUG: ----> pci_mmcfg_late_init
DEBUG: pci_parse_mcfg
PCI: MMCONFIG for domain 0000 [bus 00-01] at [mem 0xe0000000-0xe01fffff] 
(base 0xe0000000)
DEBUG: pci_mmcfg_check_reserved
DEBUG: is_mmconf_reserved: method = ACPI motherboard resources
PCI: MMCONFIG at [mem 0xe0000000-0xe01fffff] reserved in ACPI 
motherboard resources

During pci_mmcfg_early_init(), the MMCONFIG failed because the range 
0xe0000000 is not showing as reserved in the E820 mapping.  Here is the 
snippet of E820 mapping from the system:
........
BIOS-e820: [mem 0x00000000c7eb0000-0x00000000c7ec0fff] ACPI data
BIOS-e820: [mem 0x00000000c7ec1000-0x00000000c7ec2fff] ACPI NVS
BIOS-e820: [mem 0x00000000c7ec3000-0x00000000c7efefff] reserved
BIOS-e820: [mem 0x00000000c7f00000-0x00000000c7ffffff] reserved
BIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved

However, during pci_mmcfg_late_init(), the area is reserved in the "ACPI 
motherboard resources", and the pci_mmcfg_check_reserved() does not fail 
here.  But this is too late since we already setup the "raw_pci_ext_ops" 
in the "arch/x86/pci/direct.c: pci_direct_init()" (during to use the IO_ECS.

Suravee

Re: [PATCH V5 3/4] x86/PCI: Stop enabling ECS for AMD CPUs after Fam16h

[Bjorn Helgaas]

On Fri, May 23, 2014 at 6:31 PM, Suravee Suthikulanit
<suravee.suthikulpanit@amd.com> wrote:
> On 5/22/2014 9:54 PM, Bjorn Helgaas wrote:
>>
>> I've been poking around for recent dmesg logs that contain "PCI: Using
>> configuration type 1 for extended access", and there are quite a few.
>> In most cases there*is*  an MCFG table, but apparently we decide not
>>
>> to use it for some reason (unfortunately we don't print the specific
>> reason).  One example is at
>> https://bugzilla.kernel.org/show_bug.cgi?id=68591  .
>>
>> I'm going to go out on a limb and guess that Windows does not enable
>> ECS, so it probably uses ECAM.  Therefore, I suspect Linux's parsing
>> of MCFG is broken in some way, and we probably*could*  use ECAM in all
>>
>> these cases I'm seeing.
>>
>> It would probably be prudent to figure out why Linux is rejecting
>> these MCFG tables.  We'll probably see similar tables on Fam17h
>> systems, and if we continue rejecting them, and we don't turn on ECS,
>> we won't be able to access extended config space.
>>
>> I opened a bugzilla for this issue:
>> https://bugzilla.kernel.org/show_bug.cgi?id=76771
>>
>> I'm wavering on whether it's a good idea to put this patch in before
>> understanding the issue.  As much as I'd like to stop fiddling with
>> ECS, we'd likely end up with a v3.15 where extended config space
>> doesn't work on some Fam17h systems.
>
>
> So, I have located a system which presents issue with MMCONFIG. Here is my
> investigation:
>
> DEBUG: pci_io_ecs_init: pci_probe = 4000f
> ACPI: bus type PCI registered
> DEBUG: -----> pci_mmcfg_early_init
> DEBUG: pci_parse_mcfg
> PCI: MMCONFIG for domain 0000 [bus 00-01] at [mem 0xe0000000-0xe01fffff]
> (base 0xe0000000)
> DEBUG: pci_mmcfg_check_reserved
> DEBUG: is_mmconf_reserved: method = E820
> PCI: not using MMCONFIG
> DEBUG: pci_direct_init
> PCI: Using configuration type 1 for base access
>
> PCI: Using configuration type 1 for extended access
> ACPI: Added _OSI(Module Device)
> ACPI: Added _OSI(Processor Device)
> ACPI: Added _OSI(3.0 _SCP Extensions)
> ACPI: Added _OSI(Processor Aggregator Device)
> [Firmware Bug]: ACPI: BIOS _OSI(Linux) query ignored
> \_SB_:_OSC invalid UUID
> _OSC request data:1 1f
> ACPI: Interpreter enabled
> ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S1_]
> (20140214/hwxface-580)
> ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S2_]
> (20140214/hwxface-580)
> ACPI Exception: AE_NOT_FOUND, While evaluating Sleep State [\_S4_]
> (20140214/hwxface-580)
> ACPI: (supports S0 S3 S5)
> ACPI: Using IOAPIC for interrupt routing
> DEBUG: ----> pci_mmcfg_late_init
> DEBUG: pci_parse_mcfg
> PCI: MMCONFIG for domain 0000 [bus 00-01] at [mem 0xe0000000-0xe01fffff]
> (base 0xe0000000)
> DEBUG: pci_mmcfg_check_reserved
> DEBUG: is_mmconf_reserved: method = ACPI motherboard resources
> PCI: MMCONFIG at [mem 0xe0000000-0xe01fffff] reserved in ACPI motherboard
> resources
>
> During pci_mmcfg_early_init(), the MMCONFIG failed because the range
> 0xe0000000 is not showing as reserved in the E820 mapping.  Here is the
> snippet of E820 mapping from the system:
> ........
> BIOS-e820: [mem 0x00000000c7eb0000-0x00000000c7ec0fff] ACPI data
> BIOS-e820: [mem 0x00000000c7ec1000-0x00000000c7ec2fff] ACPI NVS
> BIOS-e820: [mem 0x00000000c7ec3000-0x00000000c7efefff] reserved
> BIOS-e820: [mem 0x00000000c7f00000-0x00000000c7ffffff] reserved
> BIOS-e820: [mem 0x00000000fec00000-0x00000000fec0ffff] reserved
>
> However, during pci_mmcfg_late_init(), the area is reserved in the "ACPI
> motherboard resources", and the pci_mmcfg_check_reserved() does not fail
> here.  But this is too late since we already setup the "raw_pci_ext_ops" in
> the "arch/x86/pci/direct.c: pci_direct_init()" (during to use the IO_ECS.

Thanks for checking this out.  I'm going to going to drop the IO
ECS-related patch for now, and merge the others for v3.16 (the branch
is here: http://git.kernel.org/cgit/linux/kernel/git/helgaas/pci.git/log/?h=pci/amd-numa).

I don't understand why MCFG init is split into two phases, and I don't
have time to sort all that out before v3.16.

Bjorn