[PATCH 0/3]: Discard reserved PXM bits for SRAT v1

[PATCH 0/3]: Discard reserved PXM bits for SRAT v1

[Kurt Garloff]

[-- Attachment #1: Type: text/plain, Size: 1776 bytes --]

Hi,

ACPI specification says that the OS must disregard reserved bits.
The x86_64 SRAT parser does not discard the upper 24 bits of the
proximity_domain (pxm) in the acpi_srat_mem_affinity entries for
SRAT v1 tables. (v2 has 32 bits wide fields.)
This can lead to problems with poor BIOS implementations that failed
to set resreved bytes to zero. (The ACPI spec is a bit vague here
unfortunately.)

This was also inconsistent: On x86-64 (srat_64.c), the 
_cpu_affinity does only use the low 8 bits of pxm, while the
full 32 bits of _mem_affinity are consumed.
In srat_32.c (x86), only 8bits are used (which is OK, a 32bit system
with >256 PXMs does not seem reasonable at all).
On ia64, the support of more than 8 bits was consistent between
mem and cpu affinity entries, however it dependent on "sn2" platform.

The patch series has the following goals:
* Make the kernel support consistently 8bits or 32bits for the
  proximity domain
* Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.

Overview over the patches:
- [1/3] Store the SRAT table version value in acpi_srat_revision 
- [2/3] x86-64: Discard the upper 24 bits in mem_affinity if rev <= 1
        and use upper 24bits in cpu_affinity if rev >= 2
- [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to
        enable on sn2 as well -- I don't know if sn2 reports v1 or v2
        SRAT) Also add two __init decls in ia64 pxm accessors.

Patch has been tested on x86-64 against an 2.6.27.x kernel.
(Patch is against current git.)

Thanks for James, Greg, Alexey, Norbert for comments, review and testing.

Please review and apply!

Greg, I believe this is a candidate for -stable.
-- 
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[PATCH 1/3]: Store SRAT revision

[Kurt Garloff]

[-- Attachment #1.1: Type: text/plain, Size: 192 bytes --]

Hi,

This patch stores the SRAT table revision for later consumption
by arch specific __init functions.

This is patch 1/3.
-- 
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

[-- Attachment #1.2: srat-pxm-rev-store.diff --]
[-- Type: text/x-patch, Size: 1502 bytes --]

From: Kurt Garloff <garloff@suse.de>
Subject: Store SRAT table revision
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides
32bits for these. The new fields were reserved before.
According to the ACPI spec, the OS must disregrard reserved fields.
In order to know whether or not, we must know what version the SRAT
table has.

This patch stores the SRAT table revision for later consumption
by arch specific __init functions.

This is patch 1/3.

Signed-off-by: Kurt Garloff <garloff@suse.de>

diff -r a4ed8abcc815 drivers/acpi/numa.c
--- a/drivers/acpi/numa.c	Sat May 16 13:41:28 2009 -0700
+++ b/drivers/acpi/numa.c	Mon May 18 10:45:40 2009 +0200
@@ -42,6 +42,8 @@
 static int node_to_pxm_map[MAX_NUMNODES]
 				= { [0 ... MAX_NUMNODES - 1] = PXM_INVAL };
 
+unsigned char acpi_srat_revision __initdata;
+
 int pxm_to_node(int pxm)
 {
 	if (pxm < 0)
@@ -266,6 +268,7 @@
 		return -EINVAL;
 
 	srat = (struct acpi_table_srat *)table;
+	acpi_srat_revision = srat->header.revision;
 
 	return 0;
 }
diff -r a4ed8abcc815 include/acpi/acpi_numa.h
--- a/include/acpi/acpi_numa.h	Sat May 16 13:41:28 2009 -0700
+++ b/include/acpi/acpi_numa.h	Mon May 18 10:45:40 2009 +0200
@@ -15,6 +15,7 @@
 extern int node_to_pxm(int);
 extern void __acpi_map_pxm_to_node(int, int);
 extern int acpi_map_pxm_to_node(int);
+extern unsigned char acpi_srat_revision;
 
 #endif				/* CONFIG_ACPI_NUMA */
 #endif				/* __ACP_NUMA_H */

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 2/3]: x86-64: Handle SRAT v1 and v2 consistently

[Kurt Garloff]

[-- Attachment #1.1: Type: text/plain, Size: 347 bytes --]

Hi,

x86-64 was rather inconsistent prior to this patch; it used 8 bits 
for the pxm field in cpu_affinity, but 32 bits in mem_affinity.
This patch makes it consistent: Either use 8 bits consistently (SRAT
rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

This is patch 2/3.
-- 
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

[-- Attachment #1.2: srat-pxm-rev-x86-64.diff --]
[-- Type: text/x-patch, Size: 1414 bytes --]

From: Kurt Garloff <garloff@suse.de>
Subject: Use SRAT table rev to use 8bit or 32bit PXM fields (x86-64)
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides
32bits for these. The new fields were reserved before.
According to the ACPI spec, the OS must disregrard reserved fields.

x86-64 was rather inconsistent prior to this patch; it used 8 bits 
for the pxm field in cpu_affinity, but 32 bits in mem_affinity.
This patch makes it consistent: Either use 8 bits consistently (SRAT
rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

This is patch 2/3.

Signed-off-by: Kurt Garloff <garloff@suse.de>

diff -r a4ed8abcc815 arch/x86/mm/srat_64.c
--- a/arch/x86/mm/srat_64.c	Sat May 16 13:41:28 2009 -0700
+++ b/arch/x86/mm/srat_64.c	Mon May 18 10:45:40 2009 +0200
@@ -164,6 +164,8 @@
 	if ((pa->flags & ACPI_SRAT_CPU_ENABLED) == 0)
 		return;
 	pxm = pa->proximity_domain_lo;
+	if (acpi_srat_revision >= 2)
+		pxm |= *((unsigned int*)pa->proximity_domain_hi) << 8;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains %x\n", pxm);
@@ -275,6 +277,8 @@
 	start = ma->base_address;
 	end = start + ma->length;
 	pxm = ma->proximity_domain;
+	if (acpi_srat_revision <= 1)
+		pxm &= 0xff;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains.\n");


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[PATCH 3/3]: Consider SRAT rev on ia64

[Kurt Garloff]

[-- Attachment #1.1: Type: text/plain, Size: 290 bytes --]

Hi,

ia64 did handle the PXM fields almost consistently, but depending on 
sgi's sn2 platform. This patch leaves the sn2 logic in, but does also
use 16/32 bits for PXM if the SRAT has rev 2 or higher.

This is patch 3/3.
-- 
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

[-- Attachment #1.2: srat-pxm-rev-ia64.diff --]
[-- Type: text/x-patch, Size: 1936 bytes --]

From: Kurt Garloff <garloff@suse.de>
Subject: Use SRAT table rev to use 8bit or 16/32bit PXM fields (ia64)
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides
32bits for these. The new fields were reserved before.
According to the ACPI spec, the OS must disregrard reserved fields.

ia64 did handle the PXM fields almost consistently, but depending on 
sgi's sn2 platform. This patch leaves the sn2 logic in, but does also
use 16/32 bits for PXM if the SRAT has rev 2 or higher.

The patch also adds __init to the two pxm accessor functions, as they
access __initdata now and are called from an __init function only anyway.

Note that the code only uses 16 bits for the PXM field in the processor
proximity field; the patch does not address this as 16 bits are more than
enough.

This is patch 3/3.

Signed-off-by: Kurt Garloff <garloff@suse.de>

diff -r a4ed8abcc815 arch/ia64/kernel/acpi.c
--- a/arch/ia64/kernel/acpi.c	Sat May 16 13:41:28 2009 -0700
+++ b/arch/ia64/kernel/acpi.c	Mon May 18 10:45:40 2009 +0200
@@ -456,22 +456,24 @@
 static struct acpi_table_slit __initdata *slit_table;
 cpumask_t early_cpu_possible_map = CPU_MASK_NONE;
 
-static int get_processor_proximity_domain(struct acpi_srat_cpu_affinity *pa)
+static int __init
+get_processor_proximity_domain(struct acpi_srat_cpu_affinity *pa)
 {
 	int pxm;
 
 	pxm = pa->proximity_domain_lo;
-	if (ia64_platform_is("sn2"))
+	if (ia64_platform_is("sn2") || acpi_srat_revision >= 2)
 		pxm += pa->proximity_domain_hi[0] << 8;
 	return pxm;
 }
 
-static int get_memory_proximity_domain(struct acpi_srat_mem_affinity *ma)
+static int __init
+get_memory_proximity_domain(struct acpi_srat_mem_affinity *ma)
 {
 	int pxm;
 
 	pxm = ma->proximity_domain;
-	if (!ia64_platform_is("sn2"))
+	if (!ia64_platform_is("sn2") && acpi_srat_revision <= 1)
 		pxm &= 0xff;
 
 	return pxm;


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 0/3]: Discard reserved PXM bits for SRAT v1

[Greg KH]

On Mon, May 18, 2009 at 11:33:52AM +0200, Kurt Garloff wrote:
> Hi,
> 
> ACPI specification says that the OS must disregard reserved bits.
> The x86_64 SRAT parser does not discard the upper 24 bits of the
> proximity_domain (pxm) in the acpi_srat_mem_affinity entries for
> SRAT v1 tables. (v2 has 32 bits wide fields.)
> This can lead to problems with poor BIOS implementations that failed
> to set resreved bytes to zero. (The ACPI spec is a bit vague here
> unfortunately.)
> 
> This was also inconsistent: On x86-64 (srat_64.c), the 
> _cpu_affinity does only use the low 8 bits of pxm, while the
> full 32 bits of _mem_affinity are consumed.
> In srat_32.c (x86), only 8bits are used (which is OK, a 32bit system
> with >256 PXMs does not seem reasonable at all).
> On ia64, the support of more than 8 bits was consistent between
> mem and cpu affinity entries, however it dependent on "sn2" platform.
> 
> The patch series has the following goals:
> * Make the kernel support consistently 8bits or 32bits for the
>   proximity domain
> * Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.
> 
> Overview over the patches:
> - [1/3] Store the SRAT table version value in acpi_srat_revision 
> - [2/3] x86-64: Discard the upper 24 bits in mem_affinity if rev <= 1
>         and use upper 24bits in cpu_affinity if rev >= 2
> - [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to
>         enable on sn2 as well -- I don't know if sn2 reports v1 or v2
>         SRAT) Also add two __init decls in ia64 pxm accessors.
> 
> Patch has been tested on x86-64 against an 2.6.27.x kernel.
> (Patch is against current git.)
> 
> Thanks for James, Greg, Alexey, Norbert for comments, review and testing.
> 
> Please review and apply!
> 
> Greg, I believe this is a candidate for -stable.

Does it meet the rules at described in
Documentation/stable_kernel_rules.txt?  If so, add a
	Cc: stable <stable@kernel.org>
to the signed-off-by: area in the patch, and when it goes into Linus's
tree, it will be automatically queued up to be added to the next -stable
release.

thanks,

greg k-h

Re: [PATCH 2/3]: x86-64: Handle SRAT v1 and v2 consistently

[Yinghai Lu]

On Mon, May 18, 2009 at 2:47 AM, Kurt Garloff <garloff@suse.de> wrote:
> Hi,
>
> x86-64 was rather inconsistent prior to this patch; it used 8 bits
> for the pxm field in cpu_affinity, but 32 bits in mem_affinity.
> This patch makes it consistent: Either use 8 bits consistently (SRAT
> rev 1 or lower) or 32 bits (SRAT rev 2 or higher).
>
> This is patch 2/3.
> --
> Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
>
From: Kurt Garloff <garloff@suse.de>
Subject: Use SRAT table rev to use 8bit or 32bit PXM fields (x86-64)
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides
32bits for these. The new fields were reserved before.
According to the ACPI spec, the OS must disregrard reserved fields.

x86-64 was rather inconsistent prior to this patch; it used 8 bits
for the pxm field in cpu_affinity, but 32 bits in mem_affinity.
This patch makes it consistent: Either use 8 bits consistently (SRAT
rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

This is patch 2/3.

Signed-off-by: Kurt Garloff <garloff@suse.de>

---
 arch/x86/mm/srat_64.c |    4 ++++
 1 file changed, 4 insertions(+)

Index: linux-2.6/arch/x86/mm/srat_64.c
===================================================================
--- linux-2.6.orig/arch/x86/mm/srat_64.c
+++ linux-2.6/arch/x86/mm/srat_64.c
@@ -154,6 +154,8 @@ acpi_numa_processor_affinity_init(struct
 	if ((pa->flags & ACPI_SRAT_CPU_ENABLED) == 0)
 		return;
 	pxm = pa->proximity_domain_lo;
+	if (acpi_srat_revision >= 2)
+		pxm |= *((unsigned int*)pa->proximity_domain_hi) << 8;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains %x\n", pxm);
@@ -255,6 +257,8 @@ acpi_numa_memory_affinity_init(struct ac
 	start = ma->base_address;
 	end = start + ma->length;
 	pxm = ma->proximity_domain;
+	if (acpi_srat_revision <= 1)
+		pxm &= 0xff;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains.\n");


Acked-by: Yinghai Lu <yinghai@kernel.org>

Re: [PATCH 0/3]: Discard reserved PXM bits for SRAT v1

[xb]

Hi Kurt,

We are hitting this problem on some Nehalem based platforms, that 
prevents a correct Numa initialization.
We propsed a patch on linux_acpi to fix it.
We are OK to have your patch go into the mainstream to have this fixed 
quickly.
Thanks.
Xavier

Kurt Garloff wrote:
> Hi,
>
> ACPI specification says that the OS must disregard reserved bits.
> The x86_64 SRAT parser does not discard the upper 24 bits of the
> proximity_domain (pxm) in the acpi_srat_mem_affinity entries for
> SRAT v1 tables. (v2 has 32 bits wide fields.)
> This can lead to problems with poor BIOS implementations that failed
> to set resreved bytes to zero. (The ACPI spec is a bit vague here
> unfortunately.)
>
> This was also inconsistent: On x86-64 (srat_64.c), the 
> _cpu_affinity does only use the low 8 bits of pxm, while the
> full 32 bits of _mem_affinity are consumed.
> In srat_32.c (x86), only 8bits are used (which is OK, a 32bit system
> with >256 PXMs does not seem reasonable at all).
> On ia64, the support of more than 8 bits was consistent between
> mem and cpu affinity entries, however it dependent on "sn2" platform.
>
> The patch series has the following goals:
> * Make the kernel support consistently 8bits or 32bits for the
>   proximity domain
> * Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.
>
> Overview over the patches:
> - [1/3] Store the SRAT table version value in acpi_srat_revision 
> - [2/3] x86-64: Discard the upper 24 bits in mem_affinity if rev <= 1
>         and use upper 24bits in cpu_affinity if rev >= 2
> - [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to
>         enable on sn2 as well -- I don't know if sn2 reports v1 or v2
>         SRAT) Also add two __init decls in ia64 pxm accessors.
>
> Patch has been tested on x86-64 against an 2.6.27.x kernel.
> (Patch is against current git.)
>
> Thanks for James, Greg, Alexey, Norbert for comments, review and testing.
>
> Please review and apply!
>
> Greg, I believe this is a candidate for -stable.
>   

Re: [PATCH 0/3]: Discard reserved PXM bits for SRAT v1

[Erik Jacobson]

I just wanted to post in to the thread that this patch set solved both
the node numbering (where there was a node0 and a node2) and the issue
where all CPUs were showing up one the first node.

I applied the three patches against 2.6.30-git14.

We look forward to seeing the changes integrated.

On this 2-socket Intel 5500 based system (Nehalem), the /sys stuff looks
good:

$ ls /sys/devices/system/node/node?
/sys/devices/system/node/node0:
cpu0  cpu2  cpulist  distance  numastat
cpu1  cpu3  cpumap   meminfo   scan_unevictable_pages

/sys/devices/system/node/node1:
cpu4  cpu6  cpulist  distance  numastat
cpu5  cpu7  cpumap   meminfo   scan_unevictable_pages


Where as, before, it looked like this (non-ordered, all CPUs showing up
in node 0).

# ls -ld /sys/devices/system/node/node*
drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node0
drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node2

[root@cct201 ~]# ls /sys/devices/system/node/node0
cpu0  cpu2  cpu4  cpu6  cpulist  distance  numastat
cpu1  cpu3  cpu5  cpu7  cpumap   meminfo   scan_unevictable_pages


System details:
- SGI Altix XE 270
- Supermicro X8DTN v 1.1 mainboard
- 2 sockets of Xeon X5570


/proc/cpuinfo snip

[root@cct201 erikj]# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
stepping	: 5
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5866.85
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
stepping	: 5
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 8212.05
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
stepping	: 5
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5865.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
stepping	: 5
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 0
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5865.76
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
stepping	: 5
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 0
cpu cores	: 4
apicid		: 16
initial apicid	: 16
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5828.64
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
stepping	: 5
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 1
cpu cores	: 4
apicid		: 18
initial apicid	: 18
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5865.81
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
stepping	: 5
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 2
cpu cores	: 4
apicid		: 20
initial apicid	: 20
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5865.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 26
model name	: Intel(R) Xeon(R) CPU           X5570  @ 2.93GHz
stepping	: 5
cpu MHz		: 1600.000
cache size	: 8192 KB
physical id	: 1
siblings	: 4
core id		: 3
cpu cores	: 4
apicid		: 22
initial apicid	: 22
fpu		: yes
fpu_exception	: yes
cpuid level	: 11
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips	: 5865.80
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

[PATCH 0/3] Resend: Discard reserved PXM bits for SRAT v1

[Kurt Garloff]

[-- Attachment #1: Type: text/plain, Size: 1758 bytes --]

Erik,

On Mon, Jun 22, 2009 at 09:05:22AM -0500, Erik Jacobson wrote:
> I just wanted to post in to the thread that this patch set solved both
> the node numbering (where there was a node0 and a node2) and the issue
> where all CPUs were showing up one the first node.
> 
> I applied the three patches against 2.6.30-git14.
> 
> We look forward to seeing the changes integrated.

So do I.
The patches are obviously already in SUSE kernels, I'd like to see them
in mainline too.

> On this 2-socket Intel 5500 based system (Nehalem), the /sys stuff looks
> good:
> 
> $ ls /sys/devices/system/node/node?
> /sys/devices/system/node/node0:
> cpu0  cpu2  cpulist  distance  numastat
> cpu1  cpu3  cpumap   meminfo   scan_unevictable_pages
> 
> /sys/devices/system/node/node1:
> cpu4  cpu6  cpulist  distance  numastat
> cpu5  cpu7  cpumap   meminfo   scan_unevictable_pages
> 
> 
> Where as, before, it looked like this (non-ordered, all CPUs showing up
> in node 0).
> 
> # ls -ld /sys/devices/system/node/node*
> drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node0
> drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node2
> 
> [root@cct201 ~]# ls /sys/devices/system/node/node0
> cpu0  cpu2  cpu4  cpu6  cpulist  distance  numastat
> cpu1  cpu3  cpu5  cpu7  cpumap   meminfo   scan_unevictable_pages
> 
> 
> System details:
> - SGI Altix XE 270
> - Supermicro X8DTN v 1.1 mainboard
> - 2 sockets of Xeon X5570
 
OK, you really observe the same here as did FZJ.
Thanks for reporting!

So let me resend the patches, rediffed against current 2.6.31rc git
in followup mails.

Len, Linus, please adopt.

Best,
-- 
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[PATCH 1/3] Resend: [PATCH 1/3]: Store SRAT revision

[Kurt Garloff]

[-- Attachment #1.1: Type: text/plain, Size: 535 bytes --]

Hi,

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides
32bits for these. The new fields were reserved before.
According to the ACPI spec, the OS must disregard reserved fields.
In order to know whether to disregard or not, we must know what 
version the SRAT table has.

This patch stores the SRAT table revision for later consumption
by arch specific __init functions.

This is patch 1/3.

Signed-off-by: Kurt Garloff <garloff@suse.de>
-- 
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

[-- Attachment #1.2: 01_srat-pxm-rev-store.diff --]
[-- Type: text/x-patch, Size: 1491 bytes --]

From: Kurt Garloff <garloff@suse.de>
Subject: Store SRAT table revision

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides
32bits for these. The new fields were reserved before.
According to the ACPI spec, the OS must disregard reserved fields.
In order to know whether to disregard or not, we must know what 
version the SRAT table has.

This patch stores the SRAT table revision for later consumption
by arch specific __init functions.

This is patch 1/3.

Signed-off-by: Kurt Garloff <garloff@suse.de>

diff -r a28eb89d1fe1 drivers/acpi/numa.c
--- a/drivers/acpi/numa.c	Tue Jul 14 02:00:45 2009 +0000
+++ b/drivers/acpi/numa.c	Tue Jul 14 16:44:35 2009 +0200
@@ -42,6 +42,8 @@
 static int node_to_pxm_map[MAX_NUMNODES]
 				= { [0 ... MAX_NUMNODES - 1] = PXM_INVAL };
 
+unsigned char acpi_srat_revision __initdata;
+
 int pxm_to_node(int pxm)
 {
 	if (pxm < 0)
@@ -266,6 +268,7 @@
 		return -EINVAL;
 
 	srat = (struct acpi_table_srat *)table;
+	acpi_srat_revision = srat->header.revision;
 
 	return 0;
 }
diff -r a28eb89d1fe1 include/acpi/acpi_numa.h
--- a/include/acpi/acpi_numa.h	Tue Jul 14 02:00:45 2009 +0000
+++ b/include/acpi/acpi_numa.h	Tue Jul 14 16:44:35 2009 +0200
@@ -15,6 +15,7 @@
 extern int node_to_pxm(int);
 extern void __acpi_map_pxm_to_node(int, int);
 extern int acpi_map_pxm_to_node(int);
+extern unsigned char acpi_srat_revision;
 
 #endif				/* CONFIG_ACPI_NUMA */
 #endif				/* __ACP_NUMA_H */

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[PATCH 2/3] Resend: x86-64: Handle SRAT v1 and v2 consistently

[Kurt Garloff]

[-- Attachment #1.1: Type: text/plain, Size: 595 bytes --]

Hi,

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides
32bits for these. The new fields were reserved before.
According to the ACPI spec, the OS must disregard reserved fields.

x86-64 was rather inconsistent prior to this patch; it used 8 bits 
for the pxm field in cpu_affinity, but 32 bits in mem_affinity.
This patch makes it consistent: Either use 8 bits consistently (SRAT
rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

This is patch 2/3.

Signed-off-by: Kurt Garloff <garloff@suse.de>
-- 
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

[-- Attachment #1.2: 02_srat-pxm-rev-x86-64.diff --]
[-- Type: text/x-patch, Size: 1413 bytes --]

From: Kurt Garloff <garloff@suse.de>
Subject: Use SRAT table rev to use 8bit or 32bit PXM fields (x86-64)
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides
32bits for these. The new fields were reserved before.
According to the ACPI spec, the OS must disregard reserved fields.

x86-64 was rather inconsistent prior to this patch; it used 8 bits 
for the pxm field in cpu_affinity, but 32 bits in mem_affinity.
This patch makes it consistent: Either use 8 bits consistently (SRAT
rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

This is patch 2/3.

Signed-off-by: Kurt Garloff <garloff@suse.de>

diff -r a28eb89d1fe1 arch/x86/mm/srat_64.c
--- a/arch/x86/mm/srat_64.c	Tue Jul 14 02:00:45 2009 +0000
+++ b/arch/x86/mm/srat_64.c	Tue Jul 14 16:44:35 2009 +0200
@@ -154,6 +154,8 @@
 	if ((pa->flags & ACPI_SRAT_CPU_ENABLED) == 0)
 		return;
 	pxm = pa->proximity_domain_lo;
+	if (acpi_srat_revision >= 2)
+		pxm |= *((unsigned int*)pa->proximity_domain_hi) << 8;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains %x\n", pxm);
@@ -255,6 +257,8 @@
 	start = ma->base_address;
 	end = start + ma->length;
 	pxm = ma->proximity_domain;
+	if (acpi_srat_revision <= 1)
+		pxm &= 0xff;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains.\n");


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

[PATCH 3/3] Resend: Consider SRAT rev on ia64

[Kurt Garloff]

[-- Attachment #1.1: Type: text/plain, Size: 845 bytes --]

Hi,

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides
32bits for these. The new fields were reserved before.
According to the ACPI spec, the OS must disregard reserved fields.

ia64 did handle the PXM fields almost consistently, but depending on 
sgi's sn2 platform. This patch leaves the sn2 logic in, but does also
use 16/32 bits for PXM if the SRAT has rev 2 or higher.

The patch also adds __init to the two pxm accessor functions, as they
access __initdata now and are called from an __init function only anyway.

Note that the code only uses 16 bits for the PXM field in the processor
proximity field; the patch does not address this as 16 bits are more than
enough.

This is patch 3/3.

Signed-off-by: Kurt Garloff <garloff@suse.de>
-- 
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

[-- Attachment #1.2: 03_srat-pxm-rev-ia64.diff --]
[-- Type: text/x-patch, Size: 1933 bytes --]

From: Kurt Garloff <garloff@suse.de>
Subject: Use SRAT table rev to use 8bit or 16/32bit PXM fields (ia64)
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides
32bits for these. The new fields were reserved before.
According to the ACPI spec, the OS must disregard reserved fields.

ia64 did handle the PXM fields almost consistently, but depending on 
sgi's sn2 platform. This patch leaves the sn2 logic in, but does also
use 16/32 bits for PXM if the SRAT has rev 2 or higher.

The patch also adds __init to the two pxm accessor functions, as they
access __initdata now and are called from an __init function only anyway.

Note that the code only uses 16 bits for the PXM field in the processor
proximity field; the patch does not address this as 16 bits are more than
enough.

This is patch 3/3.

Signed-off-by: Kurt Garloff <garloff@suse.de>

diff -r a28eb89d1fe1 arch/ia64/kernel/acpi.c
--- a/arch/ia64/kernel/acpi.c	Tue Jul 14 02:00:45 2009 +0000
+++ b/arch/ia64/kernel/acpi.c	Tue Jul 14 16:44:35 2009 +0200
@@ -456,22 +456,24 @@
 static struct acpi_table_slit __initdata *slit_table;
 cpumask_t early_cpu_possible_map = CPU_MASK_NONE;
 
-static int get_processor_proximity_domain(struct acpi_srat_cpu_affinity *pa)
+static int __init
+get_processor_proximity_domain(struct acpi_srat_cpu_affinity *pa)
 {
 	int pxm;
 
 	pxm = pa->proximity_domain_lo;
-	if (ia64_platform_is("sn2"))
+	if (ia64_platform_is("sn2") || acpi_srat_revision >= 2)
 		pxm += pa->proximity_domain_hi[0] << 8;
 	return pxm;
 }
 
-static int get_memory_proximity_domain(struct acpi_srat_mem_affinity *ma)
+static int __init
+get_memory_proximity_domain(struct acpi_srat_mem_affinity *ma)
 {
 	int pxm;
 
 	pxm = ma->proximity_domain;
-	if (!ia64_platform_is("sn2"))
+	if (!ia64_platform_is("sn2") && acpi_srat_revision <= 1)
 		pxm &= 0xff;
 
 	return pxm;

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH 2/3] Resend: x86-64: Handle SRAT v1 and v2 consistently

[Yinghai Lu]

On Tue, Jul 14, 2009 at 9:58 AM, Kurt Garloff<garloff@suse.de> wrote:
> Hi,
>
> In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides
> 32bits for these. The new fields were reserved before.
> According to the ACPI spec, the OS must disregard reserved fields.
>
> x86-64 was rather inconsistent prior to this patch; it used 8 bits
> for the pxm field in cpu_affinity, but 32 bits in mem_affinity.
> This patch makes it consistent: Either use 8 bits consistently (SRAT
> rev 1 or lower) or 32 bits (SRAT rev 2 or higher).
>
> This is patch 2/3.
>
> Signed-off-by: Kurt Garloff <garloff@suse.de>
> --

for 1, 2

Acked-by: Yinghai Lu <yinghai@kernel.org>

YH

Re: [PATCH 0/3]: Discard reserved PXM bits for SRAT v1

[Chuck Ebbert]

On Mon, 18 May 2009 11:33:52 +0200
Kurt Garloff <garloff@suse.de> wrote:

> Hi,
> 
> ACPI specification says that the OS must disregard reserved bits.
> The x86_64 SRAT parser does not discard the upper 24 bits of the
> proximity_domain (pxm) in the acpi_srat_mem_affinity entries for
> SRAT v1 tables. (v2 has 32 bits wide fields.)
> This can lead to problems with poor BIOS implementations that failed
> to set resreved bytes to zero. (The ACPI spec is a bit vague here
> unfortunately.)
> 
> This was also inconsistent: On x86-64 (srat_64.c), the 
> _cpu_affinity does only use the low 8 bits of pxm, while the
> full 32 bits of _mem_affinity are consumed.
> In srat_32.c (x86), only 8bits are used (which is OK, a 32bit system
> with >256 PXMs does not seem reasonable at all).
> On ia64, the support of more than 8 bits was consistent between
> mem and cpu affinity entries, however it dependent on "sn2" platform.
> 
> The patch series has the following goals:
> * Make the kernel support consistently 8bits or 32bits for the
>   proximity domain
> * Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.
> 
> Overview over the patches:
> - [1/3] Store the SRAT table version value in acpi_srat_revision 
> - [2/3] x86-64: Discard the upper 24 bits in mem_affinity if rev <= 1
>         and use upper 24bits in cpu_affinity if rev >= 2
> - [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to
>         enable on sn2 as well -- I don't know if sn2 reports v1 or v2
>         SRAT) Also add two __init decls in ia64 pxm accessors.
> 

Did this patchset get lost or something? I don't see it anywhere, not even
in the ACPI 'testing' tree...