* [RFC PATCH v3 00/24] ARM: Add Xen NUMA support
@ 2017-07-18 11:41 vijay.kilari
2017-07-18 11:41 ` [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable vijay.kilari
` (24 more replies)
0 siblings, 25 replies; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
With this RFC patch series, NUMA support is added for ARM platform.
Both DT and ACPI based NUMA support is added.
Only Xen is made aware of NUMA platform. NUMA awareness to DOM0 is not
added.
As part of this series, the code under x86 architecture is
reused by moving into common files.
New files xen/common/numa.c and xen/drivers/acpi/srat.c files are
added.
For ARM specific new folder is added xen/arch/arm/numa and new files
numa.c, dt_numa.c and acpi_numa are introduced under this folder.
DT NUMA: The following major changes are performed
- Dropped numa-node-id information from Dom0 DT.
So that Dom0 devices make allocation from node 0 for
devmalloc requests.
- Memory DT is not deleted by EFI. It is exposed to Xen
to extract numa information.
- On NUMA failure, Fallback to Non-NUMA booting.ACPI_SRAT_TYPE_MEMORY_AFFINITY
Assuming all the memory and CPU's are under node 0.
- CONFIG_NUMA is introduced.
ACPI NUMA:
- MADT is parsed before parsing SRAT table to extract
CPU_ID to MPIDR mapping info. In Linux, while parsing SRAT
table, MADT table is opened and extract MPIDR. This
approach avoids opening ACPI tables recursively.
- SRAT table is parsed for ACPI_SRAT_TYPE_GICC_AFFINITY to extract
proximity info and MPIDR from CPU_ID to MPIDR mapping table.
- Parsing of SRAT table for ACPI_SRAT_TYPE_MEMORY_AFFINITY to extract
memory proximity is reused from x86 arch.
- Re-use SLIT parsing of x86 for node distance information.
- CONFIG_ACPI_NUMA is introduced
This patch is tested on Thunderx platform.
No changes are made to x86 implementation only code is sanitized
and refactored. Hence only compilation tested for x86.
This series is posted as RFC for the reason that it is not tested
on x86. Request some help from community in testing this series on x86.
Code is shared at
https://github.com/vijaykilari/xen-numa/commits/rfc_v3
v3: Major changes
- Rebased to latest staging branch
- Dropped patches 4 & 5 of v2.
- Reused most of the x86 code like numa emulation, acpi
memory node parsing by moving to common code.
- Fixed cpu node parsing with DT.
- Made NR_NODES as configurable
- Dropped hardcoding of memnodemap[] array size
- Segregated new dt functions to single patch 11
- Introduced new patch 10 for NUMA initialization.
v2: Major changes
- Rebased to lastest staging branch
- Reworked on x86 NUMA code and cleanup to possible extent.
Patches 1 to 8 are created for this
- Reworked on DT and ACPI NUMA extracting information
- Reused DT code for memory node processing to extract NUMA info.
- Fixed issues with DT processing
- Added arch specific processing of SRAT
- Reworked on MADT and SRAT processing
- Reworked on node distance
- All ARM changes are moved under folder arch/arm/numa.
- NUMA ACPI common changes are kept in drivers/acpi/srat.c
Vijaya Kumar K (24):
NUMA: Make number of NUMA nodes configurable
x86: NUMA: Clean up: Fix coding styles and drop unused code
x86: NUMA: Fix datatypes and attributes
x86: NUMA: Rename and sanitize memnode shift code
x86: NUMA: Add accessors for nodes[] and node_memblk_range[] structs
x86: NUMA: Rename some generic functions
ARM: NUMA: Add existing ARM numa code under CONFIG_NUMA
NUMA: x86: Move numa code and make it generic
NUMA: x86: Move common code from srat.c
NUMA: Allow numa initialization with DT
ARM: fdt: Export and introduce new fdt functions
ARM: NUMA: DT: Parse CPU NUMA information
ARM: NUMA: DT: Parse memory NUMA information
ARM: NUMA: DT: Parse NUMA distance information
ARM: NUMA: DT: Add CPU NUMA support
ARM: NUMA: Add memory NUMA support
ARM: NUMA: DT: Do not expose numa info to DOM0
ACPI: Refactor acpi SRAT and SLIT table handling code
ARM: NUMA: Extract MPIDR from MADT table
ACPI: Move arch specific SRAT parsing
ARM: NUMA: ACPI: Extract proximity from SRAT table
ARM: NUMA: Initialize ACPI NUMA
NUMA: Move CONFIG_NUMA to common Kconfig
NUMA: Enable ACPI_NUMA config
xen/arch/Kconfig | 7 +
xen/arch/arm/Makefile | 1 +
xen/arch/arm/acpi/boot.c | 2 +
xen/arch/arm/bootfdt.c | 45 ++-
xen/arch/arm/domain_build.c | 9 +
xen/arch/arm/efi/efi-boot.h | 25 --
xen/arch/arm/numa/Makefile | 3 +
xen/arch/arm/numa/acpi_numa.c | 246 +++++++++++++
xen/arch/arm/numa/dt_numa.c | 240 +++++++++++++
xen/arch/arm/numa/numa.c | 188 ++++++++++
xen/arch/arm/setup.c | 6 +
xen/arch/arm/smpboot.c | 25 +-
xen/arch/x86/dom0_build.c | 1 +
xen/arch/x86/mm.c | 2 -
xen/arch/x86/numa.c | 463 +------------------------
xen/arch/x86/physdev.c | 1 +
xen/arch/x86/setup.c | 1 +
xen/arch/x86/smpboot.c | 4 +-
xen/arch/x86/srat.c | 405 ++++------------------
xen/arch/x86/x86_64/mm.c | 3 +-
xen/common/Kconfig | 4 +
xen/common/Makefile | 1 +
xen/common/numa.c | 670 ++++++++++++++++++++++++++++++++++++
xen/drivers/acpi/Kconfig | 5 +-
xen/drivers/acpi/Makefile | 1 +
xen/drivers/acpi/numa.c | 58 +---
xen/drivers/acpi/srat.c | 298 ++++++++++++++++
xen/drivers/passthrough/vtd/iommu.c | 1 +
xen/include/acpi/actbl1.h | 17 +-
xen/include/acpi/srat.h | 24 ++
xen/include/asm-arm/numa.h | 33 +-
xen/include/asm-arm/setup.h | 5 +
xen/include/asm-x86/acpi.h | 4 -
xen/include/asm-x86/config.h | 1 -
xen/include/asm-x86/numa.h | 73 +---
xen/include/asm-x86/setup.h | 1 -
xen/include/xen/acpi.h | 6 +
xen/include/xen/config.h | 1 +
xen/include/xen/nodemask.h | 2 +
xen/include/xen/numa.h | 79 ++++-
40 files changed, 2002 insertions(+), 959 deletions(-)
create mode 100644 xen/arch/arm/numa/Makefile
create mode 100644 xen/arch/arm/numa/acpi_numa.c
create mode 100644 xen/arch/arm/numa/dt_numa.c
create mode 100644 xen/arch/arm/numa/numa.c
create mode 100644 xen/common/numa.c
create mode 100644 xen/drivers/acpi/srat.c
create mode 100644 xen/include/acpi/srat.h
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-18 17:55 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code vijay.kilari
` (23 subsequent siblings)
24 siblings, 2 replies; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Introduce NR_NODES config option to specify number
of NUMA nodes supported. By default value is set at
64 for x86 and 8 for arm. Dropped NODES_SHIFT macro.
Also move NR_NODE_MEMBLKS from asm-x86/acpi.h to xen/numa.h
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
xen/arch/Kconfig | 7 +++++++
xen/include/asm-x86/acpi.h | 1 -
xen/include/asm-x86/numa.h | 2 --
xen/include/xen/config.h | 1 +
xen/include/xen/numa.h | 7 ++-----
5 files changed, 10 insertions(+), 8 deletions(-)
diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig
index cf0acb7..9c2a4e2 100644
--- a/xen/arch/Kconfig
+++ b/xen/arch/Kconfig
@@ -6,3 +6,10 @@ config NR_CPUS
default "128" if ARM
---help---
Specifies the maximum number of physical CPUs which Xen will support.
+
+config NR_NODES
+ int "Maximum number of NUMA nodes"
+ default "64" if X86
+ default "8" if ARM
+ ---help---
+ Specifies the maximum number of NUMA nodes which Xen will support.
diff --git a/xen/include/asm-x86/acpi.h b/xen/include/asm-x86/acpi.h
index 27ecc65..15be784 100644
--- a/xen/include/asm-x86/acpi.h
+++ b/xen/include/asm-x86/acpi.h
@@ -105,7 +105,6 @@ extern void acpi_reserve_bootmem(void);
extern s8 acpi_numa;
extern int acpi_scan_nodes(u64 start, u64 end);
-#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
#ifdef CONFIG_ACPI_SLEEP
diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
index bada2c0..3cf26c2 100644
--- a/xen/include/asm-x86/numa.h
+++ b/xen/include/asm-x86/numa.h
@@ -3,8 +3,6 @@
#include <xen/cpumask.h>
-#define NODES_SHIFT 6
-
typedef u8 nodeid_t;
extern int srat_rev;
diff --git a/xen/include/xen/config.h b/xen/include/xen/config.h
index a1d0f97..0f1a029 100644
--- a/xen/include/xen/config.h
+++ b/xen/include/xen/config.h
@@ -81,6 +81,7 @@
/* allow existing code to work with Kconfig variable */
#define NR_CPUS CONFIG_NR_CPUS
+#define NR_NODES CONFIG_NR_NODES
#ifndef CONFIG_DEBUG
#define NDEBUG
diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
index 7aef1a8..6bba29e 100644
--- a/xen/include/xen/numa.h
+++ b/xen/include/xen/numa.h
@@ -3,14 +3,11 @@
#include <asm/numa.h>
-#ifndef NODES_SHIFT
-#define NODES_SHIFT 0
-#endif
-
#define NUMA_NO_NODE 0xFF
#define NUMA_NO_DISTANCE 0xFF
-#define MAX_NUMNODES (1 << NODES_SHIFT)
+#define MAX_NUMNODES NR_NODES
+#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
#define vcpu_to_node(v) (cpu_to_node((v)->processor))
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
2017-07-18 11:41 ` [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-19 16:23 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 03/24] x86: NUMA: Fix datatypes and attributes vijay.kilari
` (22 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Fix coding style, trailing spaces, tabs in NUMA code.
Also drop unused macros and functions.
There is no functional change.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
---
v3: - Change commit message
- Changed VIRTUAL_BUG_ON to ASSERT
- Dropped useless inner paranthesis for some macros
---
xen/arch/x86/numa.c | 55 +++++++++++++++++++++------------------------
xen/arch/x86/srat.c | 2 +-
xen/include/asm-x86/numa.h | 56 +++++++++++++++++++++++-----------------------
xen/include/xen/numa.h | 3 ---
4 files changed, 54 insertions(+), 62 deletions(-)
diff --git a/xen/arch/x86/numa.c b/xen/arch/x86/numa.c
index d45196fa..444d7ad 100644
--- a/xen/arch/x86/numa.c
+++ b/xen/arch/x86/numa.c
@@ -1,8 +1,8 @@
-/*
+/*
* Generic VM initialization for x86-64 NUMA setups.
* Copyright 2002,2003 Andi Kleen, SuSE Labs.
* Adapted for Xen: Ryan Harper <ryanh@us.ibm.com>
- */
+ */
#include <xen/mm.h>
#include <xen/string.h>
@@ -21,13 +21,6 @@
static int numa_setup(char *s);
custom_param("numa", numa_setup);
-#ifndef Dprintk
-#define Dprintk(x...)
-#endif
-
-/* from proto.h */
-#define round_up(x,y) ((((x)+(y))-1) & (~((y)-1)))
-
struct node_data node_data[MAX_NUMNODES];
/* Mapping from pdx to node id */
@@ -144,8 +137,9 @@ static int __init extract_lsb_from_nodes(const struct node *nodes,
if ( nodes_used <= 1 )
i = BITS_PER_LONG - 1;
else
- i = find_first_bit(&bitfield, sizeof(unsigned long)*8);
+ i = find_first_bit(&bitfield, sizeof(unsigned long) * 8);
memnodemapsize = (memtop >> i) + 1;
+
return i;
}
@@ -173,7 +167,7 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,
}
/* initialize NODE_DATA given nodeid and start/end */
void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
-{
+{
unsigned long start_pfn, end_pfn;
start_pfn = start >> PAGE_SHIFT;
@@ -183,7 +177,7 @@ void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;
node_set_online(nodeid);
-}
+}
void __init numa_init_array(void)
{
@@ -214,7 +208,7 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
{
int i;
struct node nodes[MAX_NUMNODES];
- u64 sz = ((end_pfn - start_pfn)<<PAGE_SHIFT) / numa_fake;
+ u64 sz = ((end_pfn - start_pfn) << PAGE_SHIFT) / numa_fake;
/* Kludge needed for the hash function */
if ( hweight64(sz) > 1 )
@@ -222,21 +216,22 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
u64 x = 1;
while ( (x << 1) < sz )
x <<= 1;
- if ( x < sz/2 )
- printk(KERN_ERR "Numa emulation unbalanced. Complain to maintainer\n");
+ if ( x < sz / 2 )
+ printk(KERN_ERR
+ "Numa emulation unbalanced. Complain to maintainer\n");
sz = x;
}
memset(&nodes,0,sizeof(nodes));
for ( i = 0; i < numa_fake; i++ )
{
- nodes[i].start = (start_pfn<<PAGE_SHIFT) + i*sz;
+ nodes[i].start = (start_pfn << PAGE_SHIFT) + i * sz;
if ( i == numa_fake - 1 )
- sz = (end_pfn<<PAGE_SHIFT) - nodes[i].start;
+ sz = (end_pfn << PAGE_SHIFT) - nodes[i].start;
nodes[i].end = nodes[i].start + sz;
- printk(KERN_INFO "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
- i,
- nodes[i].start, nodes[i].end,
+ printk(KERN_INFO
+ "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
+ i, nodes[i].start, nodes[i].end,
(nodes[i].end - nodes[i].start) >> 20);
node_set_online(i);
}
@@ -256,7 +251,7 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
#endif
void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
-{
+{
int i;
#ifdef CONFIG_NUMA_EMU
@@ -291,7 +286,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
void numa_add_cpu(int cpu)
{
cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
-}
+}
void numa_set_node(int cpu, nodeid_t node)
{
@@ -299,23 +294,23 @@ void numa_set_node(int cpu, nodeid_t node)
}
/* [numa=off] */
-static __init int numa_setup(char *opt)
-{
- if ( !strncmp(opt,"off",3) )
+static __init int numa_setup(char *opt)
+{
+ if ( !strncmp(opt, "off", 3) )
numa_off = true;
- if ( !strncmp(opt,"on",2) )
+ if ( !strncmp(opt, "on", 2) )
numa_off = false;
#ifdef CONFIG_NUMA_EMU
if ( !strncmp(opt, "fake=", 5) )
{
numa_off = false;
- numa_fake = simple_strtoul(opt+5,NULL,0);
+ numa_fake = simple_strtoul(opt + 5, NULL, 0);
if ( numa_fake >= MAX_NUMNODES )
numa_fake = MAX_NUMNODES;
}
#endif
#ifdef CONFIG_ACPI_NUMA
- if ( !strncmp(opt,"noacpi",6) )
+ if ( !strncmp(opt,"noacpi", 6) )
{
numa_off = false;
acpi_numa = -1;
@@ -323,7 +318,7 @@ static __init int numa_setup(char *opt)
#endif
return 1;
-}
+}
/*
* Setup early cpu_to_node.
@@ -385,7 +380,7 @@ static void dump_numa(unsigned char key)
const struct vnuma_info *vnuma;
printk("'%c' pressed -> dumping numa info (now-0x%X:%08X)\n", key,
- (u32)(now>>32), (u32)now);
+ (u32)(now >> 32), (u32)now);
for_each_online_node ( i )
{
diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
index cd1283e..ec08112 100644
--- a/xen/arch/x86/srat.c
+++ b/xen/arch/x86/srat.c
@@ -7,7 +7,7 @@
* Called from acpi_numa_init while reading the SRAT and SLIT tables.
* Assumes all memory regions belonging to a single proximity domain
* are in one chunk. Holes between them will be included in the node.
- *
+ *
* Adapted for Xen: Ryan Harper <ryanh@us.ibm.com>
*/
diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
index 3cf26c2..c0de57b 100644
--- a/xen/include/asm-x86/numa.h
+++ b/xen/include/asm-x86/numa.h
@@ -1,8 +1,11 @@
-#ifndef _ASM_X8664_NUMA_H
+#ifndef _ASM_X8664_NUMA_H
#define _ASM_X8664_NUMA_H 1
#include <xen/cpumask.h>
+#define MAX_NUMNODES NR_NODES
+#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
+
typedef u8 nodeid_t;
extern int srat_rev;
@@ -10,21 +13,21 @@ extern int srat_rev;
extern nodeid_t cpu_to_node[NR_CPUS];
extern cpumask_t node_to_cpumask[];
-#define cpu_to_node(cpu) (cpu_to_node[cpu])
-#define parent_node(node) (node)
+#define cpu_to_node(cpu) (cpu_to_node[cpu])
+#define parent_node(node) (node)
#define node_to_first_cpu(node) (__ffs(node_to_cpumask[node]))
#define node_to_cpumask(node) (node_to_cpumask[node])
-struct node {
- u64 start,end;
+struct node {
+ u64 start;
+ u64 end;
};
extern int compute_hash_shift(struct node *nodes, int numnodes,
- nodeid_t *nodeids);
+ nodeid_t *nodeids);
extern nodeid_t pxm_to_node(unsigned int pxm);
#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
-#define VIRTUAL_BUG_ON(x)
extern void numa_add_cpu(int cpu);
extern void numa_init_array(void);
@@ -40,13 +43,8 @@ extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
extern nodeid_t apicid_to_node[];
extern void init_cpu_to_node(void);
-static inline void clear_node_cpumask(int cpu)
-{
- cpumask_clear_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
-}
-
/* Simple perfect hash to map pdx to node numbers */
-extern int memnode_shift;
+extern int memnode_shift;
extern unsigned long memnodemapsize;
extern u8 *memnodemap;
@@ -57,21 +55,23 @@ struct node_data {
extern struct node_data node_data[];
-static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
-{
- nodeid_t nid;
- VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >= memnodemapsize);
- nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
- VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]);
- return nid;
-}
-
-#define NODE_DATA(nid) (&(node_data[nid]))
-
-#define node_start_pfn(nid) (NODE_DATA(nid)->node_start_pfn)
-#define node_spanned_pages(nid) (NODE_DATA(nid)->node_spanned_pages)
-#define node_end_pfn(nid) (NODE_DATA(nid)->node_start_pfn + \
- NODE_DATA(nid)->node_spanned_pages)
+static inline __attribute_pure__ nodeid_t phys_to_nid(paddr_t addr)
+{
+ nodeid_t nid;
+
+ ASSERT((paddr_to_pdx(addr) >> memnode_shift) < memnodemapsize);
+ nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
+ ASSERT(nid <= MAX_NUMNODES || !node_data[nid].node_start_pfn);
+
+ return nid;
+}
+
+#define NODE_DATA(nid) (&(node_data[nid]))
+
+#define node_start_pfn(nid) NODE_DATA(nid)->node_start_pfn
+#define node_spanned_pages(nid) NODE_DATA(nid)->node_spanned_pages
+#define node_end_pfn(nid) NODE_DATA(nid)->node_start_pfn + \
+ NODE_DATA(nid)->node_spanned_pages
extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
index 6bba29e..3bb4afc 100644
--- a/xen/include/xen/numa.h
+++ b/xen/include/xen/numa.h
@@ -6,9 +6,6 @@
#define NUMA_NO_NODE 0xFF
#define NUMA_NO_DISTANCE 0xFF
-#define MAX_NUMNODES NR_NODES
-#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
-
#define vcpu_to_node(v) (cpu_to_node((v)->processor))
#define domain_to_node(d) \
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 03/24] x86: NUMA: Fix datatypes and attributes
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
2017-07-18 11:41 ` [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable vijay.kilari
2017-07-18 11:41 ` [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-18 11:41 ` [RFC PATCH v3 04/24] x86: NUMA: Rename and sanitize memnode shift code vijay.kilari
` (21 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Change u{8,32,64} to uint{8,32,64}_t, u64 to paddr_t
wherever applicable.
Fix attributes coding styles.
Also changed
- Some variables from int to unsigned int
- Used pfn_to_paddr/paddr_to_pfn whereever required.
- Alloc memnodemap[] of size BITS_PER_LONG.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: - Change unsigned to unsigned int
- Update commit message
- Drop changing memnode_shift as unsigned int
- Used pfn_to_paddr/paddr_to_pfn
- Alloc memnodemap[] of size BITS_PER_LONG
---
xen/arch/x86/numa.c | 54 +++++++++++++++++++++-----------------
xen/arch/x86/srat.c | 64 +++++++++++++++++++++++-----------------------
xen/include/asm-arm/numa.h | 2 +-
xen/include/asm-x86/acpi.h | 2 +-
xen/include/asm-x86/numa.h | 16 ++++++------
5 files changed, 72 insertions(+), 66 deletions(-)
diff --git a/xen/arch/x86/numa.c b/xen/arch/x86/numa.c
index 444d7ad..aa4a7c1 100644
--- a/xen/arch/x86/numa.c
+++ b/xen/arch/x86/numa.c
@@ -25,11 +25,17 @@ struct node_data node_data[MAX_NUMNODES];
/* Mapping from pdx to node id */
int memnode_shift;
-static typeof(*memnodemap) _memnodemap[64];
+
+/*
+ * In case of numa init failure or numa off,
+ * memnode_shift is initialized to BITS_PER_LONG - 1. Hence allocate
+ * memnodemap[] of BITS_PER_LONG.
+ */
+static typeof(*memnodemap) _memnodemap[BITS_PER_LONG];
unsigned long memnodemapsize;
-u8 *memnodemap;
+uint8_t *memnodemap;
-nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
+nodeid_t __read_mostly cpu_to_node[NR_CPUS] = {
[0 ... NR_CPUS-1] = NUMA_NO_NODE
};
/*
@@ -38,7 +44,7 @@ nodeid_t cpu_to_node[NR_CPUS] __read_mostly = {
nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {
[0 ... MAX_LOCAL_APIC-1] = NUMA_NO_NODE
};
-cpumask_t node_to_cpumask[MAX_NUMNODES] __read_mostly;
+cpumask_t __read_mostly node_to_cpumask[MAX_NUMNODES];
nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
@@ -166,12 +172,12 @@ int __init compute_hash_shift(struct node *nodes, int numnodes,
return shift;
}
/* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end)
+void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end)
{
unsigned long start_pfn, end_pfn;
- start_pfn = start >> PAGE_SHIFT;
- end_pfn = end >> PAGE_SHIFT;
+ start_pfn = paddr_to_pfn(start);
+ end_pfn = paddr_to_pfn(end);
NODE_DATA(nodeid)->node_start_pfn = start_pfn;
NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;
@@ -201,19 +207,20 @@ void __init numa_init_array(void)
}
#ifdef CONFIG_NUMA_EMU
-static int numa_fake __initdata = 0;
+static unsigned int __initdata numa_fake;
/* Numa emulation */
-static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
+static int __init numa_emulation(uint64_t start_pfn, uint64_t end_pfn)
{
- int i;
+ unsigned int i;
struct node nodes[MAX_NUMNODES];
- u64 sz = ((end_pfn - start_pfn) << PAGE_SHIFT) / numa_fake;
+ uint64_t sz = ((end_pfn - start_pfn) << PAGE_SHIFT) / numa_fake;
/* Kludge needed for the hash function */
if ( hweight64(sz) > 1 )
{
- u64 x = 1;
+ uint64_t x = 1;
+
while ( (x << 1) < sz )
x <<= 1;
if ( x < sz / 2 )
@@ -225,9 +232,9 @@ static int __init numa_emulation(u64 start_pfn, u64 end_pfn)
memset(&nodes,0,sizeof(nodes));
for ( i = 0; i < numa_fake; i++ )
{
- nodes[i].start = (start_pfn << PAGE_SHIFT) + i * sz;
+ nodes[i].start = pfn_to_paddr(start_pfn) + i * sz;
if ( i == numa_fake - 1 )
- sz = (end_pfn << PAGE_SHIFT) - nodes[i].start;
+ sz = pfn_to_paddr(end_pfn) - nodes[i].start;
nodes[i].end = nodes[i].start + sz;
printk(KERN_INFO
"Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
@@ -260,8 +267,8 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
#endif
#ifdef CONFIG_ACPI_NUMA
- if ( !numa_off && !acpi_scan_nodes((u64)start_pfn << PAGE_SHIFT,
- (u64)end_pfn << PAGE_SHIFT) )
+ if ( !numa_off &&
+ !acpi_scan_nodes(pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn)) )
return;
#endif
@@ -269,8 +276,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
numa_off ? "NUMA turned off" : "No NUMA configuration found");
printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
- (u64)start_pfn << PAGE_SHIFT,
- (u64)end_pfn << PAGE_SHIFT);
+ pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn));
/* setup dummy node covering all memory */
memnode_shift = BITS_PER_LONG - 1;
memnodemap = _memnodemap;
@@ -279,8 +285,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
for ( i = 0; i < nr_cpu_ids; i++ )
numa_set_node(i, 0);
cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
- setup_node_bootmem(0, (u64)start_pfn << PAGE_SHIFT,
- (u64)end_pfn << PAGE_SHIFT);
+ setup_node_bootmem(0, pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn));
}
void numa_add_cpu(int cpu)
@@ -294,7 +299,7 @@ void numa_set_node(int cpu, nodeid_t node)
}
/* [numa=off] */
-static __init int numa_setup(char *opt)
+static int __init numa_setup(char *opt)
{
if ( !strncmp(opt, "off", 3) )
numa_off = true;
@@ -339,7 +344,8 @@ void __init init_cpu_to_node(void)
for ( i = 0; i < nr_cpu_ids; i++ )
{
- u32 apicid = x86_cpu_to_apicid[i];
+ uint32_t apicid = x86_cpu_to_apicid[i];
+
if ( apicid == BAD_APICID )
continue;
node = apicid < MAX_LOCAL_APIC ? apicid_to_node[apicid] : NUMA_NO_NODE;
@@ -380,7 +386,7 @@ static void dump_numa(unsigned char key)
const struct vnuma_info *vnuma;
printk("'%c' pressed -> dumping numa info (now-0x%X:%08X)\n", key,
- (u32)(now >> 32), (u32)now);
+ (uint32_t)(now >> 32), (uint32_t)now);
for_each_online_node ( i )
{
@@ -507,7 +513,7 @@ static void dump_numa(unsigned char key)
rcu_read_unlock(&domlist_read_lock);
}
-static __init int register_numa_trigger(void)
+static int __init register_numa_trigger(void)
{
register_keyhandler('u', dump_numa, "dump NUMA info", 1);
return 0;
diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
index ec08112..209ffc7 100644
--- a/xen/arch/x86/srat.c
+++ b/xen/arch/x86/srat.c
@@ -23,33 +23,33 @@
static struct acpi_table_slit *__read_mostly acpi_slit;
-static nodemask_t memory_nodes_parsed __initdata;
-static nodemask_t processor_nodes_parsed __initdata;
-static struct node nodes[MAX_NUMNODES] __initdata;
+static nodemask_t __initdata memory_nodes_parsed;
+static nodemask_t __initdata processor_nodes_parsed;
+static struct node __initdata nodes[MAX_NUMNODES];
struct pxm2node {
- unsigned pxm;
+ unsigned int pxm;
nodeid_t node;
};
static struct pxm2node __read_mostly pxm2node[MAX_NUMNODES] =
{ [0 ... MAX_NUMNODES - 1] = {.node = NUMA_NO_NODE} };
-static unsigned node_to_pxm(nodeid_t n);
+static unsigned int node_to_pxm(nodeid_t n);
static int num_node_memblks;
static struct node node_memblk_range[NR_NODE_MEMBLKS];
static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
-static inline bool node_found(unsigned idx, unsigned pxm)
+static inline bool node_found(unsigned int idx, unsigned int pxm)
{
return ((pxm2node[idx].pxm == pxm) &&
(pxm2node[idx].node != NUMA_NO_NODE));
}
-nodeid_t pxm_to_node(unsigned pxm)
+nodeid_t pxm_to_node(unsigned int pxm)
{
- unsigned i;
+ unsigned int i;
if ((pxm < ARRAY_SIZE(pxm2node)) && node_found(pxm, pxm))
return pxm2node[pxm].node;
@@ -61,12 +61,12 @@ nodeid_t pxm_to_node(unsigned pxm)
return NUMA_NO_NODE;
}
-nodeid_t setup_node(unsigned pxm)
+nodeid_t setup_node(unsigned int pxm)
{
nodeid_t node;
- unsigned idx;
+ unsigned int idx;
static bool warned;
- static unsigned nodes_found;
+ static unsigned int nodes_found;
BUILD_BUG_ON(MAX_NUMNODES >= NUMA_NO_NODE);
@@ -103,7 +103,7 @@ nodeid_t setup_node(unsigned pxm)
return node;
}
-int valid_numa_range(u64 start, u64 end, nodeid_t node)
+int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
{
int i;
@@ -118,7 +118,7 @@ int valid_numa_range(u64 start, u64 end, nodeid_t node)
return 0;
}
-static __init int conflicting_memblks(u64 start, u64 end)
+static int __init conflicting_memblks(paddr_t start, paddr_t end)
{
int i;
@@ -134,7 +134,7 @@ static __init int conflicting_memblks(u64 start, u64 end)
return -1;
}
-static __init void cutoff_node(int i, u64 start, u64 end)
+static void __init cutoff_node(nodeid_t i, paddr_t start, paddr_t end)
{
struct node *nd = &nodes[i];
if (nd->start < start) {
@@ -149,7 +149,7 @@ static __init void cutoff_node(int i, u64 start, u64 end)
}
}
-static __init void bad_srat(void)
+static void __init bad_srat(void)
{
int i;
printk(KERN_ERR "SRAT: SRAT not used.\n");
@@ -167,13 +167,13 @@ static __init void bad_srat(void)
* distance than the others.
* Do some quick checks here and only use the SLIT if it passes.
*/
-static __init int slit_valid(struct acpi_table_slit *slit)
+static int __init slit_valid(struct acpi_table_slit *slit)
{
int i, j;
int d = slit->locality_count;
for (i = 0; i < d; i++) {
for (j = 0; j < d; j++) {
- u8 val = slit->entry[d*i + j];
+ uint8_t val = slit->entry[d*i + j];
if (i == j) {
if (val != 10)
return 0;
@@ -207,7 +207,7 @@ void __init acpi_numa_slit_init(struct acpi_table_slit *slit)
void __init
acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
{
- unsigned pxm;
+ unsigned int pxm;
nodeid_t node;
if (srat_disabled())
@@ -241,7 +241,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
void __init
acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
{
- unsigned pxm;
+ unsigned int pxm;
nodeid_t node;
if (srat_disabled())
@@ -274,8 +274,8 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
void __init
acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
{
- u64 start, end;
- unsigned pxm;
+ paddr_t start, end;
+ unsigned int pxm;
nodeid_t node;
int i;
@@ -401,7 +401,7 @@ static int __init nodes_cover_memory(void)
void __init acpi_numa_arch_fixup(void) {}
-static u64 __initdata srat_region_mask;
+static uint64_t __initdata srat_region_mask;
static int __init srat_parse_region(struct acpi_subtable_header *header,
const unsigned long end)
@@ -428,9 +428,9 @@ static int __init srat_parse_region(struct acpi_subtable_header *header,
return 0;
}
-void __init srat_parse_regions(u64 addr)
+void __init srat_parse_regions(paddr_t addr)
{
- u64 mask;
+ uint64_t mask;
unsigned int i;
if (acpi_disabled || acpi_numa < 0 ||
@@ -453,9 +453,9 @@ void __init srat_parse_regions(u64 addr)
}
/* Use the information discovered above to actually set up the nodes. */
-int __init acpi_scan_nodes(u64 start, u64 end)
+int __init acpi_scan_nodes(paddr_t start, paddr_t end)
{
- int i;
+ unsigned int i;
nodemask_t all_nodes_parsed;
/* First clean up the node list */
@@ -485,7 +485,7 @@ int __init acpi_scan_nodes(u64 start, u64 end)
/* Finally register nodes */
for_each_node_mask(i, all_nodes_parsed)
{
- u64 size = nodes[i].end - nodes[i].start;
+ uint64_t size = nodes[i].end - nodes[i].start;
if ( size == 0 )
printk(KERN_WARNING "SRAT: Node %u has no memory. "
"BIOS Bug or mis-configured hardware?\n", i);
@@ -502,9 +502,9 @@ int __init acpi_scan_nodes(u64 start, u64 end)
return 0;
}
-static unsigned node_to_pxm(nodeid_t n)
+static unsigned int node_to_pxm(nodeid_t n)
{
- unsigned i;
+ unsigned int i;
if ((n < ARRAY_SIZE(pxm2node)) && (pxm2node[n].node == n))
return pxm2node[n].pxm;
@@ -514,10 +514,10 @@ static unsigned node_to_pxm(nodeid_t n)
return 0;
}
-u8 __node_distance(nodeid_t a, nodeid_t b)
+uint8_t __node_distance(nodeid_t a, nodeid_t b)
{
- unsigned index;
- u8 slit_val;
+ unsigned int index;
+ uint8_t slit_val;
if (!acpi_slit)
return a == b ? 10 : 20;
diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
index a2c1a34..53f99af 100644
--- a/xen/include/asm-arm/numa.h
+++ b/xen/include/asm-arm/numa.h
@@ -1,7 +1,7 @@
#ifndef __ARCH_ARM_NUMA_H
#define __ARCH_ARM_NUMA_H
-typedef u8 nodeid_t;
+typedef uint8_t nodeid_t;
/* Fake one node for now. See also node_online_map. */
#define cpu_to_node(cpu) 0
diff --git a/xen/include/asm-x86/acpi.h b/xen/include/asm-x86/acpi.h
index 15be784..a6fad1e 100644
--- a/xen/include/asm-x86/acpi.h
+++ b/xen/include/asm-x86/acpi.h
@@ -104,7 +104,7 @@ extern void acpi_reserve_bootmem(void);
#define ARCH_HAS_POWER_INIT 1
extern s8 acpi_numa;
-extern int acpi_scan_nodes(u64 start, u64 end);
+extern int acpi_scan_nodes(paddr_t start, paddr_t end);
#ifdef CONFIG_ACPI_SLEEP
diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
index c0de57b..5e8474f 100644
--- a/xen/include/asm-x86/numa.h
+++ b/xen/include/asm-x86/numa.h
@@ -6,7 +6,7 @@
#define MAX_NUMNODES NR_NODES
#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
-typedef u8 nodeid_t;
+typedef uint8_t nodeid_t;
extern int srat_rev;
@@ -19,8 +19,8 @@ extern cpumask_t node_to_cpumask[];
#define node_to_cpumask(node) (node_to_cpumask[node])
struct node {
- u64 start;
- u64 end;
+ paddr_t start;
+ paddr_t end;
};
extern int compute_hash_shift(struct node *nodes, int numnodes,
@@ -39,14 +39,14 @@ extern void numa_set_node(int cpu, nodeid_t node);
extern nodeid_t setup_node(unsigned int pxm);
extern void srat_detect_node(int cpu);
-extern void setup_node_bootmem(nodeid_t nodeid, u64 start, u64 end);
+extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
extern nodeid_t apicid_to_node[];
extern void init_cpu_to_node(void);
/* Simple perfect hash to map pdx to node numbers */
extern int memnode_shift;
extern unsigned long memnodemapsize;
-extern u8 *memnodemap;
+extern uint8_t *memnodemap;
struct node_data {
unsigned long node_start_pfn;
@@ -73,10 +73,10 @@ static inline __attribute_pure__ nodeid_t phys_to_nid(paddr_t addr)
#define node_end_pfn(nid) NODE_DATA(nid)->node_start_pfn + \
NODE_DATA(nid)->node_spanned_pages
-extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
+extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
-void srat_parse_regions(u64 addr);
-extern u8 __node_distance(nodeid_t a, nodeid_t b);
+void srat_parse_regions(paddr_t addr);
+extern uint8_t __node_distance(nodeid_t a, nodeid_t b);
unsigned int arch_get_dma_bitsize(void);
#endif
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 04/24] x86: NUMA: Rename and sanitize memnode shift code
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (2 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 03/24] x86: NUMA: Fix datatypes and attributes vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-19 17:12 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 05/24] x86: NUMA: Add accessors for nodes[] and node_memblk_range[] structs vijay.kilari
` (20 subsequent siblings)
24 siblings, 2 replies; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
memnode_shift variable is changed from int to unsigned int.
With this change, compute_memnode_shift() returns error value
instead of returning shift value. The memnode_shift is updated inside
compute_memnode_shift().
Also, following changes are made
- Rename compute_hash_shift to compute_memnode_shift
- Update int to unsigned int for params in extract_lsb_from_nodes()
- Return values of populate_memnodemap() is changed
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3:
- Update int to unsigned int for params in extract_lsb_from_nodes()
- Return values of populate_memnodemap() is changed
---
xen/arch/x86/numa.c | 53 ++++++++++++++++++++++++----------------------
xen/arch/x86/srat.c | 7 +++---
xen/include/asm-x86/numa.h | 6 +++---
3 files changed, 34 insertions(+), 32 deletions(-)
diff --git a/xen/arch/x86/numa.c b/xen/arch/x86/numa.c
index aa4a7c1..2ea2ec0 100644
--- a/xen/arch/x86/numa.c
+++ b/xen/arch/x86/numa.c
@@ -24,7 +24,7 @@ custom_param("numa", numa_setup);
struct node_data node_data[MAX_NUMNODES];
/* Mapping from pdx to node id */
-int memnode_shift;
+unsigned int memnode_shift;
/*
* In case of numa init failure or numa off,
@@ -59,15 +59,16 @@ int srat_disabled(void)
/*
* Given a shift value, try to populate memnodemap[]
* Returns :
- * 1 if OK
- * 0 if memnodmap[] too small (of shift too small)
- * -1 if node overlap or lost ram (shift too big)
+ * 0 if OK
+ * -ENOSPC if memnodmap[] too small (or shift too small)
+ * -EINVAL if node overlap or lost ram (shift too big)
*/
static int __init populate_memnodemap(const struct node *nodes,
- int numnodes, int shift, nodeid_t *nodeids)
+ unsigned int numnodes, unsigned int shift,
+ nodeid_t *nodeids)
{
unsigned long spdx, epdx;
- int i, res = -1;
+ int i, res = -EINVAL;
memset(memnodemap, NUMA_NO_NODE, memnodemapsize * sizeof(*memnodemap));
for ( i = 0; i < numnodes; i++ )
@@ -77,10 +78,10 @@ static int __init populate_memnodemap(const struct node *nodes,
if ( spdx >= epdx )
continue;
if ( (epdx >> shift) >= memnodemapsize )
- return 0;
+ return -ENOSPC;
do {
if ( memnodemap[spdx >> shift] != NUMA_NO_NODE )
- return -1;
+ return -EINVAL;
if ( !nodeids )
memnodemap[spdx >> shift] = i;
@@ -89,7 +90,7 @@ static int __init populate_memnodemap(const struct node *nodes,
spdx += (1UL << shift);
} while ( spdx < epdx );
- res = 1;
+ res = 0;
}
return res;
@@ -105,7 +106,7 @@ static int __init allocate_cachealigned_memnodemap(void)
printk(KERN_ERR
"NUMA: Unable to allocate Memory to Node hash map\n");
memnodemapsize = 0;
- return -1;
+ return -ENOMEM;
}
memnodemap = mfn_to_virt(mfn);
@@ -122,10 +123,10 @@ static int __init allocate_cachealigned_memnodemap(void)
* The LSB of all start and end addresses in the node map is the value of the
* maximum possible shift.
*/
-static int __init extract_lsb_from_nodes(const struct node *nodes,
- int numnodes)
+static unsigned int __init extract_lsb_from_nodes(const struct node *nodes,
+ unsigned int numnodes)
{
- int i, nodes_used = 0;
+ unsigned int i, nodes_used = 0;
unsigned long spdx, epdx;
unsigned long bitfield = 0, memtop = 0;
@@ -149,27 +150,30 @@ static int __init extract_lsb_from_nodes(const struct node *nodes,
return i;
}
-int __init compute_hash_shift(struct node *nodes, int numnodes,
- nodeid_t *nodeids)
+int __init compute_memnode_shift(struct node *nodes, unsigned int numnodes,
+ nodeid_t *nodeids)
{
- int shift;
+ int ret;
+
+ memnode_shift = extract_lsb_from_nodes(nodes, numnodes);
- shift = extract_lsb_from_nodes(nodes, numnodes);
if ( memnodemapsize <= ARRAY_SIZE(_memnodemap) )
memnodemap = _memnodemap;
else if ( allocate_cachealigned_memnodemap() )
- return -1;
- printk(KERN_DEBUG "NUMA: Using %d for the hash shift.\n", shift);
+ return -ENOMEM;
- if ( populate_memnodemap(nodes, numnodes, shift, nodeids) != 1 )
+ printk(KERN_DEBUG "NUMA: Using %u for the hash shift.\n", memnode_shift);
+
+ ret = populate_memnodemap(nodes, numnodes, memnode_shift, nodeids);
+ if ( ret )
{
printk(KERN_INFO "Your memory is not aligned you need to "
"rebuild your hypervisor with a bigger NODEMAPSIZE "
- "shift=%d\n", shift);
- return -1;
+ "shift=%u\n", memnode_shift);
+ return ret;
}
- return shift;
+ return 0;
}
/* initialize NODE_DATA given nodeid and start/end */
void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end)
@@ -242,8 +246,7 @@ static int __init numa_emulation(uint64_t start_pfn, uint64_t end_pfn)
(nodes[i].end - nodes[i].start) >> 20);
node_set_online(i);
}
- memnode_shift = compute_hash_shift(nodes, numa_fake, NULL);
- if ( memnode_shift < 0 )
+ if ( compute_memnode_shift(nodes, numa_fake, NULL) )
{
memnode_shift = 0;
printk(KERN_ERR "No NUMA hash function found. Emulation disabled.\n");
diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
index 209ffc7..535c9d7 100644
--- a/xen/arch/x86/srat.c
+++ b/xen/arch/x86/srat.c
@@ -470,10 +470,9 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
return -1;
}
- memnode_shift = compute_hash_shift(node_memblk_range, num_node_memblks,
- memblk_nodeid);
-
- if (memnode_shift < 0) {
+ if (compute_memnode_shift(node_memblk_range, num_node_memblks,
+ memblk_nodeid)) {
+ memnode_shift = 0;
printk(KERN_ERR
"SRAT: No NUMA node hash function found. Contact maintainer\n");
bad_srat();
diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
index 5e8474f..1bac25c 100644
--- a/xen/include/asm-x86/numa.h
+++ b/xen/include/asm-x86/numa.h
@@ -23,8 +23,8 @@ struct node {
paddr_t end;
};
-extern int compute_hash_shift(struct node *nodes, int numnodes,
- nodeid_t *nodeids);
+extern int compute_memnode_shift(struct node *nodes, unsigned int numnodes,
+ nodeid_t *nodeids);
extern nodeid_t pxm_to_node(unsigned int pxm);
#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
@@ -44,7 +44,7 @@ extern nodeid_t apicid_to_node[];
extern void init_cpu_to_node(void);
/* Simple perfect hash to map pdx to node numbers */
-extern int memnode_shift;
+extern unsigned int memnode_shift;
extern unsigned long memnodemapsize;
extern uint8_t *memnodemap;
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 05/24] x86: NUMA: Add accessors for nodes[] and node_memblk_range[] structs
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (3 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 04/24] x86: NUMA: Rename and sanitize memnode shift code vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-18 11:41 ` [RFC PATCH v3 06/24] x86: NUMA: Rename some generic functions vijay.kilari
` (19 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Add accessors for nodes[] and other static variables and
use those accessors. These variables are later accessed
outside the file when the code made generic in later
patches. However the coding style is not changed.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: - Changed accessors parameter from int to unsigned int
- Updated commit message
- Fixed wrong indentation
---
xen/arch/x86/srat.c | 106 +++++++++++++++++++++++++++++++++++++++-------------
1 file changed, 81 insertions(+), 25 deletions(-)
diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
index 535c9d7..42cca5a 100644
--- a/xen/arch/x86/srat.c
+++ b/xen/arch/x86/srat.c
@@ -41,6 +41,44 @@ static struct node node_memblk_range[NR_NODE_MEMBLKS];
static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
+static struct node *get_numa_node(unsigned int id)
+{
+ return &nodes[id];
+}
+
+static nodeid_t get_memblk_nodeid(unsigned int id)
+{
+ return memblk_nodeid[id];
+}
+
+static nodeid_t *get_memblk_nodeid_map(void)
+{
+ return &memblk_nodeid[0];
+}
+
+static struct node *get_node_memblk_range(unsigned int memblk)
+{
+ return &node_memblk_range[memblk];
+}
+
+static int get_num_node_memblks(void)
+{
+ return num_node_memblks;
+}
+
+static int __init numa_add_memblk(nodeid_t nodeid, paddr_t start, uint64_t size)
+{
+ if (nodeid >= NR_NODE_MEMBLKS)
+ return -EINVAL;
+
+ node_memblk_range[num_node_memblks].start = start;
+ node_memblk_range[num_node_memblks].end = start + size;
+ memblk_nodeid[num_node_memblks] = nodeid;
+ num_node_memblks++;
+
+ return 0;
+}
+
static inline bool node_found(unsigned int idx, unsigned int pxm)
{
return ((pxm2node[idx].pxm == pxm) &&
@@ -107,11 +145,11 @@ int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
{
int i;
- for (i = 0; i < num_node_memblks; i++) {
- struct node *nd = &node_memblk_range[i];
+ for (i = 0; i < get_num_node_memblks(); i++) {
+ struct node *nd = get_node_memblk_range(i);
if (nd->start <= start && nd->end > end &&
- memblk_nodeid[i] == node )
+ get_memblk_nodeid(i) == node)
return 1;
}
@@ -122,8 +160,8 @@ static int __init conflicting_memblks(paddr_t start, paddr_t end)
{
int i;
- for (i = 0; i < num_node_memblks; i++) {
- struct node *nd = &node_memblk_range[i];
+ for (i = 0; i < get_num_node_memblks(); i++) {
+ struct node *nd = get_node_memblk_range(i);
if (nd->start == nd->end)
continue;
if (nd->end > start && nd->start < end)
@@ -136,7 +174,8 @@ static int __init conflicting_memblks(paddr_t start, paddr_t end)
static void __init cutoff_node(nodeid_t i, paddr_t start, paddr_t end)
{
- struct node *nd = &nodes[i];
+ struct node *nd = get_numa_node(i);
+
if (nd->start < start) {
nd->start = start;
if (nd->end < nd->start)
@@ -278,6 +317,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
unsigned int pxm;
nodeid_t node;
int i;
+ struct node *memblk;
if (srat_disabled())
return;
@@ -288,7 +328,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
return;
- if (num_node_memblks >= NR_NODE_MEMBLKS)
+ if (get_num_node_memblks() >= NR_NODE_MEMBLKS)
{
dprintk(XENLOG_WARNING,
"Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
@@ -310,27 +350,31 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
i = conflicting_memblks(start, end);
if (i < 0)
/* everything fine */;
- else if (memblk_nodeid[i] == node) {
+ else if (get_memblk_nodeid(i) == node) {
bool mismatch = !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) !=
!test_bit(i, memblk_hotplug);
+ memblk = get_node_memblk_range(i);
+
printk("%sSRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
- node_memblk_range[i].start, node_memblk_range[i].end);
+ memblk->start, memblk->end);
if (mismatch) {
bad_srat();
return;
}
} else {
+ memblk = get_node_memblk_range(i);
+
printk(KERN_ERR
"SRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with PXM %u (%"PRIx64"-%"PRIx64")\n",
- pxm, start, end, node_to_pxm(memblk_nodeid[i]),
- node_memblk_range[i].start, node_memblk_range[i].end);
+ pxm, start, end, node_to_pxm(get_memblk_nodeid(i)),
+ memblk->start, memblk->end);
bad_srat();
return;
}
if (!(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE)) {
- struct node *nd = &nodes[node];
+ struct node *nd = get_numa_node(node);
if (!node_test_and_set(node, memory_nodes_parsed)) {
nd->start = start;
@@ -346,15 +390,17 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
node, pxm, start, end,
ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE ? " (hotplug)" : "");
- node_memblk_range[num_node_memblks].start = start;
- node_memblk_range[num_node_memblks].end = end;
- memblk_nodeid[num_node_memblks] = node;
+ if (numa_add_memblk(node, start, ma->length)) {
+ printk(KERN_ERR "SRAT: node-id %u out of range\n", node);
+ bad_srat();
+ return;
+ }
+
if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
- __set_bit(num_node_memblks, memblk_hotplug);
+ __set_bit(get_num_node_memblks(), memblk_hotplug);
if (end > mem_hotplug)
mem_hotplug = end;
}
- num_node_memblks++;
}
/* Sanity check to catch more bad SRATs (they are amazingly common).
@@ -377,17 +423,21 @@ static int __init nodes_cover_memory(void)
do {
found = 0;
for_each_node_mask(j, memory_nodes_parsed)
- if (start < nodes[j].end
- && end > nodes[j].start) {
- if (start >= nodes[j].start) {
- start = nodes[j].end;
+ {
+ struct node *nd = get_numa_node(j);
+
+ if (start < nd->end
+ && end > nd->start) {
+ if (start >= nd->start) {
+ start = nd->end;
found = 1;
}
- if (end <= nodes[j].end) {
- end = nodes[j].start;
+ if (end <= nd->end) {
+ end = nd->start;
found = 1;
}
}
+ }
} while (found && start < end);
if (start < end) {
@@ -457,6 +507,8 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
{
unsigned int i;
nodemask_t all_nodes_parsed;
+ struct node *memblks;
+ nodeid_t *nodeids;
/* First clean up the node list */
for (i = 0; i < MAX_NUMNODES; i++)
@@ -470,6 +522,8 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
return -1;
}
+ memblks = get_node_memblk_range(0);
+ nodeids = get_memblk_nodeid_map();
if (compute_memnode_shift(node_memblk_range, num_node_memblks,
memblk_nodeid)) {
memnode_shift = 0;
@@ -484,12 +538,14 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
/* Finally register nodes */
for_each_node_mask(i, all_nodes_parsed)
{
- uint64_t size = nodes[i].end - nodes[i].start;
+ struct node *nd = get_numa_node(i);
+ uint64_t size = nd->end - nd->start;
+
if ( size == 0 )
printk(KERN_WARNING "SRAT: Node %u has no memory. "
"BIOS Bug or mis-configured hardware?\n", i);
- setup_node_bootmem(i, nodes[i].start, nodes[i].end);
+ setup_node_bootmem(i, nd->start, nd->end);
}
for (i = 0; i < nr_cpu_ids; i++) {
if (cpu_to_node[i] == NUMA_NO_NODE)
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 06/24] x86: NUMA: Rename some generic functions
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (4 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 05/24] x86: NUMA: Add accessors for nodes[] and node_memblk_range[] structs vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-19 17:23 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 07/24] ARM: NUMA: Add existing ARM numa code under CONFIG_NUMA vijay.kilari
` (18 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Rename some function in ACPI code as follow
- Rename setup_node to acpi_setup_node
- Rename bad_srat to numa_failed
- Rename nodes_cover_memory to arch_sanitize_nodes_memory
and changed return type to bool
- Rename acpi_scan_nodes to numa_scan_nodes
Also introduce reset_pxm2node() to reset pxm2node variable.
This avoids exporting pxm2node.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: Changed return type of arch_sanitize_nodes_memory
---
xen/arch/x86/numa.c | 2 +-
xen/arch/x86/smpboot.c | 2 +-
xen/arch/x86/srat.c | 55 ++++++++++++++++++++++++++--------------------
xen/arch/x86/x86_64/mm.c | 2 +-
xen/include/asm-x86/acpi.h | 2 +-
xen/include/asm-x86/numa.h | 2 +-
6 files changed, 36 insertions(+), 29 deletions(-)
diff --git a/xen/arch/x86/numa.c b/xen/arch/x86/numa.c
index 2ea2ec0..44c2e08 100644
--- a/xen/arch/x86/numa.c
+++ b/xen/arch/x86/numa.c
@@ -271,7 +271,7 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
#ifdef CONFIG_ACPI_NUMA
if ( !numa_off &&
- !acpi_scan_nodes(pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn)) )
+ !numa_scan_nodes(pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn)) )
return;
#endif
diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c
index 8d91f6c..78af0d2 100644
--- a/xen/arch/x86/smpboot.c
+++ b/xen/arch/x86/smpboot.c
@@ -957,7 +957,7 @@ int cpu_add(uint32_t apic_id, uint32_t acpi_id, uint32_t pxm)
if ( !srat_disabled() )
{
- nodeid_t node = setup_node(pxm);
+ nodeid_t node = acpi_setup_node(pxm);
if ( node == NUMA_NO_NODE )
{
diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
index 42cca5a..03bc37d 100644
--- a/xen/arch/x86/srat.c
+++ b/xen/arch/x86/srat.c
@@ -85,6 +85,14 @@ static inline bool node_found(unsigned int idx, unsigned int pxm)
(pxm2node[idx].node != NUMA_NO_NODE));
}
+static void reset_pxm2node(void)
+{
+ unsigned int i;
+
+ for (i = 0; i < ARRAY_SIZE(pxm2node); i++)
+ pxm2node[i].node = NUMA_NO_NODE;
+}
+
nodeid_t pxm_to_node(unsigned int pxm)
{
unsigned int i;
@@ -99,7 +107,7 @@ nodeid_t pxm_to_node(unsigned int pxm)
return NUMA_NO_NODE;
}
-nodeid_t setup_node(unsigned int pxm)
+nodeid_t acpi_setup_node(unsigned int pxm)
{
nodeid_t node;
unsigned int idx;
@@ -188,15 +196,14 @@ static void __init cutoff_node(nodeid_t i, paddr_t start, paddr_t end)
}
}
-static void __init bad_srat(void)
+static void __init numa_failed(void)
{
int i;
printk(KERN_ERR "SRAT: SRAT not used.\n");
acpi_numa = -1;
for (i = 0; i < MAX_LOCAL_APIC; i++)
apicid_to_node[i] = NUMA_NO_NODE;
- for (i = 0; i < ARRAY_SIZE(pxm2node); i++)
- pxm2node[i].node = NUMA_NO_NODE;
+ reset_pxm2node();
mem_hotplug = 0;
}
@@ -252,7 +259,7 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
if (srat_disabled())
return;
if (pa->header.length < sizeof(struct acpi_srat_x2apic_cpu_affinity)) {
- bad_srat();
+ numa_failed();
return;
}
if (!(pa->flags & ACPI_SRAT_CPU_ENABLED))
@@ -263,9 +270,9 @@ acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
}
pxm = pa->proximity_domain;
- node = setup_node(pxm);
+ node = acpi_setup_node(pxm);
if (node == NUMA_NO_NODE) {
- bad_srat();
+ numa_failed();
return;
}
@@ -286,7 +293,7 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
if (srat_disabled())
return;
if (pa->header.length != sizeof(struct acpi_srat_cpu_affinity)) {
- bad_srat();
+ numa_failed();
return;
}
if (!(pa->flags & ACPI_SRAT_CPU_ENABLED))
@@ -297,9 +304,9 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
pxm |= pa->proximity_domain_hi[1] << 16;
pxm |= pa->proximity_domain_hi[2] << 24;
}
- node = setup_node(pxm);
+ node = acpi_setup_node(pxm);
if (node == NUMA_NO_NODE) {
- bad_srat();
+ numa_failed();
return;
}
apicid_to_node[pa->apic_id] = node;
@@ -322,7 +329,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
if (srat_disabled())
return;
if (ma->header.length != sizeof(struct acpi_srat_mem_affinity)) {
- bad_srat();
+ numa_failed();
return;
}
if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
@@ -332,7 +339,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
{
dprintk(XENLOG_WARNING,
"Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
- bad_srat();
+ numa_failed();
return;
}
@@ -341,9 +348,9 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
pxm = ma->proximity_domain;
if (srat_rev < 2)
pxm &= 0xff;
- node = setup_node(pxm);
+ node = acpi_setup_node(pxm);
if (node == NUMA_NO_NODE) {
- bad_srat();
+ numa_failed();
return;
}
/* It is fine to add this area to the nodes data it will be used later*/
@@ -360,7 +367,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
memblk->start, memblk->end);
if (mismatch) {
- bad_srat();
+ numa_failed();
return;
}
} else {
@@ -370,7 +377,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
"SRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with PXM %u (%"PRIx64"-%"PRIx64")\n",
pxm, start, end, node_to_pxm(get_memblk_nodeid(i)),
memblk->start, memblk->end);
- bad_srat();
+ numa_failed();
return;
}
if (!(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE)) {
@@ -392,7 +399,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
if (numa_add_memblk(node, start, ma->length)) {
printk(KERN_ERR "SRAT: node-id %u out of range\n", node);
- bad_srat();
+ numa_failed();
return;
}
@@ -405,7 +412,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
/* Sanity check to catch more bad SRATs (they are amazingly common).
Make sure the PXMs cover all memory. */
-static int __init nodes_cover_memory(void)
+static bool __init arch_sanitize_nodes_memory(void)
{
int i;
@@ -443,10 +450,10 @@ static int __init nodes_cover_memory(void)
if (start < end) {
printk(KERN_ERR "SRAT: No PXM for e820 range: "
"%016Lx - %016Lx\n", start, end);
- return 0;
+ return false;
}
}
- return 1;
+ return true;
}
void __init acpi_numa_arch_fixup(void) {}
@@ -503,7 +510,7 @@ void __init srat_parse_regions(paddr_t addr)
}
/* Use the information discovered above to actually set up the nodes. */
-int __init acpi_scan_nodes(paddr_t start, paddr_t end)
+int __init numa_scan_nodes(paddr_t start, paddr_t end)
{
unsigned int i;
nodemask_t all_nodes_parsed;
@@ -517,8 +524,8 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
if (acpi_numa <= 0)
return -1;
- if (!nodes_cover_memory()) {
- bad_srat();
+ if (!arch_sanitize_nodes_memory()) {
+ numa_failed();
return -1;
}
@@ -529,7 +536,7 @@ int __init acpi_scan_nodes(paddr_t start, paddr_t end)
memnode_shift = 0;
printk(KERN_ERR
"SRAT: No NUMA node hash function found. Contact maintainer\n");
- bad_srat();
+ numa_failed();
return -1;
}
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index aa1b94f..a4ffa1f 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -1368,7 +1368,7 @@ int memory_add(unsigned long spfn, unsigned long epfn, unsigned int pxm)
if ( !mem_hotadd_check(spfn, epfn) )
return -EINVAL;
- if ( (node = setup_node(pxm)) == NUMA_NO_NODE )
+ if ( (node = acpi_setup_node(pxm)) == NUMA_NO_NODE )
return -EINVAL;
if ( !valid_numa_range(spfn << PAGE_SHIFT, epfn << PAGE_SHIFT, node) )
diff --git a/xen/include/asm-x86/acpi.h b/xen/include/asm-x86/acpi.h
index a6fad1e..220c2d7 100644
--- a/xen/include/asm-x86/acpi.h
+++ b/xen/include/asm-x86/acpi.h
@@ -104,7 +104,7 @@ extern void acpi_reserve_bootmem(void);
#define ARCH_HAS_POWER_INIT 1
extern s8 acpi_numa;
-extern int acpi_scan_nodes(paddr_t start, paddr_t end);
+extern int numa_scan_nodes(paddr_t start, paddr_t end);
#ifdef CONFIG_ACPI_SLEEP
diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
index 1bac25c..acf509c 100644
--- a/xen/include/asm-x86/numa.h
+++ b/xen/include/asm-x86/numa.h
@@ -36,7 +36,7 @@ extern bool numa_off;
extern int srat_disabled(void);
extern void numa_set_node(int cpu, nodeid_t node);
-extern nodeid_t setup_node(unsigned int pxm);
+extern nodeid_t acpi_setup_node(unsigned int pxm);
extern void srat_detect_node(int cpu);
extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 07/24] ARM: NUMA: Add existing ARM numa code under CONFIG_NUMA
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (5 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 06/24] x86: NUMA: Rename some generic functions vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-18 18:06 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic vijay.kilari
` (17 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Right now CONFIG_NUMA is not enabled for ARM and
existing code in asm-arm/numa.h is for !CONFIG_NUMA.
Hence put this code under #ifndef CONFIG_NUMA.
This help to make this changes work when CONFIG_NUMA
is not enabled. Though CONFIG_NUMA is enabled by default,
manually disabling this option is possible and compilation
should go through. Hence kept the these changes under
!CONFIG_NUMA.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: - Dropped NODE_SHIFT define
---
xen/include/asm-arm/numa.h | 2 ++
1 file changed, 2 insertions(+)
diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
index 53f99af..7f00a36 100644
--- a/xen/include/asm-arm/numa.h
+++ b/xen/include/asm-arm/numa.h
@@ -3,6 +3,7 @@
typedef uint8_t nodeid_t;
+#ifndef CONFIG_NUMA
/* Fake one node for now. See also node_online_map. */
#define cpu_to_node(cpu) 0
#define node_to_cpumask(node) (cpu_online_map)
@@ -16,6 +17,7 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
#define node_spanned_pages(nid) (total_pages)
#define node_start_pfn(nid) (pdx_to_pfn(frametable_base_pdx))
#define __node_distance(a, b) (20)
+#endif /* CONFIG_NUMA */
static inline unsigned int arch_get_dma_bitsize(void)
{
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (6 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 07/24] ARM: NUMA: Add existing ARM numa code under CONFIG_NUMA vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-19 17:41 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 09/24] NUMA: x86: Move common code from srat.c vijay.kilari
` (16 subsequent siblings)
24 siblings, 2 replies; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Move code from xen/arch/x86/numa.c to xen/common/numa.c
so that it can be used by other archs.
The following changes are done:
- Few generic static functions in x86/numa.c is made
non-static common/numa.c
- The generic contents of header file asm-x86/numa.h
are moved to xen/numa.h.
- The header file includes are reordered and externs are
dropped.
- Moved acpi_numa from asm-x86/acpi.h to xen/acpi.h
- Coding style of code moved to commom/numa.c is changed
to Xen style.
- numa_add_cpu() and numa_set_node() and moved to header
file and added inline function in case of CONFIG_NUMA
is not enabled because these functions are called from
generic code with out any config check.
Also the node_online_map is defined in x86/numa.c for x86
and arm/smpboot.c for ARM. For x86 it is moved to x86/smpboot.c
If moved to common code the compilation fails because
common/numa.c is compiled only when NUMA is enabled.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: - Moved acpi_numa variable
- acpi_setup_node declaration move is reverted.
- Dropped extern in header file
- Added inline declaration for numa_add_cpu() and
numa_set_node() function based on CONFIG_NUMA
- Moved numa_initmem_init() to common code
- Moved common code from asm-x86/numa.h to xen/numa.h
- Moved node_online_map from numa.c to smpboot.c
---
xen/arch/x86/numa.c | 459 +----------------------------------------
xen/arch/x86/smpboot.c | 1 +
xen/common/Makefile | 1 +
xen/common/numa.c | 487 ++++++++++++++++++++++++++++++++++++++++++++
xen/include/asm-x86/acpi.h | 1 -
xen/include/asm-x86/numa.h | 56 -----
xen/include/asm-x86/setup.h | 1 -
xen/include/xen/numa.h | 64 ++++++
8 files changed, 561 insertions(+), 509 deletions(-)
diff --git a/xen/arch/x86/numa.c b/xen/arch/x86/numa.c
index 44c2e08..654530b 100644
--- a/xen/arch/x86/numa.c
+++ b/xen/arch/x86/numa.c
@@ -10,323 +10,17 @@
#include <xen/ctype.h>
#include <xen/nodemask.h>
#include <xen/numa.h>
-#include <xen/keyhandler.h>
#include <xen/time.h>
#include <xen/smp.h>
#include <xen/pfn.h>
#include <asm/acpi.h>
-#include <xen/sched.h>
-#include <xen/softirq.h>
-
-static int numa_setup(char *s);
-custom_param("numa", numa_setup);
-
-struct node_data node_data[MAX_NUMNODES];
-
-/* Mapping from pdx to node id */
-unsigned int memnode_shift;
/*
- * In case of numa init failure or numa off,
- * memnode_shift is initialized to BITS_PER_LONG - 1. Hence allocate
- * memnodemap[] of BITS_PER_LONG.
- */
-static typeof(*memnodemap) _memnodemap[BITS_PER_LONG];
-unsigned long memnodemapsize;
-uint8_t *memnodemap;
-
-nodeid_t __read_mostly cpu_to_node[NR_CPUS] = {
- [0 ... NR_CPUS-1] = NUMA_NO_NODE
-};
-/*
* Keep BIOS's CPU2node information, should not be used for memory allocaion
*/
nodeid_t apicid_to_node[MAX_LOCAL_APIC] = {
[0 ... MAX_LOCAL_APIC-1] = NUMA_NO_NODE
};
-cpumask_t __read_mostly node_to_cpumask[MAX_NUMNODES];
-
-nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
-
-bool numa_off;
-s8 acpi_numa = 0;
-
-int srat_disabled(void)
-{
- return numa_off || acpi_numa < 0;
-}
-
-/*
- * Given a shift value, try to populate memnodemap[]
- * Returns :
- * 0 if OK
- * -ENOSPC if memnodmap[] too small (or shift too small)
- * -EINVAL if node overlap or lost ram (shift too big)
- */
-static int __init populate_memnodemap(const struct node *nodes,
- unsigned int numnodes, unsigned int shift,
- nodeid_t *nodeids)
-{
- unsigned long spdx, epdx;
- int i, res = -EINVAL;
-
- memset(memnodemap, NUMA_NO_NODE, memnodemapsize * sizeof(*memnodemap));
- for ( i = 0; i < numnodes; i++ )
- {
- spdx = paddr_to_pdx(nodes[i].start);
- epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
- if ( spdx >= epdx )
- continue;
- if ( (epdx >> shift) >= memnodemapsize )
- return -ENOSPC;
- do {
- if ( memnodemap[spdx >> shift] != NUMA_NO_NODE )
- return -EINVAL;
-
- if ( !nodeids )
- memnodemap[spdx >> shift] = i;
- else
- memnodemap[spdx >> shift] = nodeids[i];
-
- spdx += (1UL << shift);
- } while ( spdx < epdx );
- res = 0;
- }
-
- return res;
-}
-
-static int __init allocate_cachealigned_memnodemap(void)
-{
- unsigned long size = PFN_UP(memnodemapsize * sizeof(*memnodemap));
- unsigned long mfn = alloc_boot_pages(size, 1);
-
- if ( !mfn )
- {
- printk(KERN_ERR
- "NUMA: Unable to allocate Memory to Node hash map\n");
- memnodemapsize = 0;
- return -ENOMEM;
- }
-
- memnodemap = mfn_to_virt(mfn);
- mfn <<= PAGE_SHIFT;
- size <<= PAGE_SHIFT;
- printk(KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n",
- mfn, mfn + size);
- memnodemapsize = size / sizeof(*memnodemap);
-
- return 0;
-}
-
-/*
- * The LSB of all start and end addresses in the node map is the value of the
- * maximum possible shift.
- */
-static unsigned int __init extract_lsb_from_nodes(const struct node *nodes,
- unsigned int numnodes)
-{
- unsigned int i, nodes_used = 0;
- unsigned long spdx, epdx;
- unsigned long bitfield = 0, memtop = 0;
-
- for ( i = 0; i < numnodes; i++ )
- {
- spdx = paddr_to_pdx(nodes[i].start);
- epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
- if ( spdx >= epdx )
- continue;
- bitfield |= spdx;
- nodes_used++;
- if ( epdx > memtop )
- memtop = epdx;
- }
- if ( nodes_used <= 1 )
- i = BITS_PER_LONG - 1;
- else
- i = find_first_bit(&bitfield, sizeof(unsigned long) * 8);
- memnodemapsize = (memtop >> i) + 1;
-
- return i;
-}
-
-int __init compute_memnode_shift(struct node *nodes, unsigned int numnodes,
- nodeid_t *nodeids)
-{
- int ret;
-
- memnode_shift = extract_lsb_from_nodes(nodes, numnodes);
-
- if ( memnodemapsize <= ARRAY_SIZE(_memnodemap) )
- memnodemap = _memnodemap;
- else if ( allocate_cachealigned_memnodemap() )
- return -ENOMEM;
-
- printk(KERN_DEBUG "NUMA: Using %u for the hash shift.\n", memnode_shift);
-
- ret = populate_memnodemap(nodes, numnodes, memnode_shift, nodeids);
- if ( ret )
- {
- printk(KERN_INFO "Your memory is not aligned you need to "
- "rebuild your hypervisor with a bigger NODEMAPSIZE "
- "shift=%u\n", memnode_shift);
- return ret;
- }
-
- return 0;
-}
-/* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end)
-{
- unsigned long start_pfn, end_pfn;
-
- start_pfn = paddr_to_pfn(start);
- end_pfn = paddr_to_pfn(end);
-
- NODE_DATA(nodeid)->node_start_pfn = start_pfn;
- NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;
-
- node_set_online(nodeid);
-}
-
-void __init numa_init_array(void)
-{
- int rr, i;
-
- /* There are unfortunately some poorly designed mainboards around
- that only connect memory to a single CPU. This breaks the 1:1 cpu->node
- mapping. To avoid this fill in the mapping for all possible
- CPUs, as the number of CPUs is not known yet.
- We round robin the existing nodes. */
- rr = first_node(node_online_map);
- for ( i = 0; i < nr_cpu_ids; i++ )
- {
- if ( cpu_to_node[i] != NUMA_NO_NODE )
- continue;
- numa_set_node(i, rr);
- rr = next_node(rr, node_online_map);
- if ( rr == MAX_NUMNODES )
- rr = first_node(node_online_map);
- }
-}
-
-#ifdef CONFIG_NUMA_EMU
-static unsigned int __initdata numa_fake;
-
-/* Numa emulation */
-static int __init numa_emulation(uint64_t start_pfn, uint64_t end_pfn)
-{
- unsigned int i;
- struct node nodes[MAX_NUMNODES];
- uint64_t sz = ((end_pfn - start_pfn) << PAGE_SHIFT) / numa_fake;
-
- /* Kludge needed for the hash function */
- if ( hweight64(sz) > 1 )
- {
- uint64_t x = 1;
-
- while ( (x << 1) < sz )
- x <<= 1;
- if ( x < sz / 2 )
- printk(KERN_ERR
- "Numa emulation unbalanced. Complain to maintainer\n");
- sz = x;
- }
-
- memset(&nodes,0,sizeof(nodes));
- for ( i = 0; i < numa_fake; i++ )
- {
- nodes[i].start = pfn_to_paddr(start_pfn) + i * sz;
- if ( i == numa_fake - 1 )
- sz = pfn_to_paddr(end_pfn) - nodes[i].start;
- nodes[i].end = nodes[i].start + sz;
- printk(KERN_INFO
- "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
- i, nodes[i].start, nodes[i].end,
- (nodes[i].end - nodes[i].start) >> 20);
- node_set_online(i);
- }
- if ( compute_memnode_shift(nodes, numa_fake, NULL) )
- {
- memnode_shift = 0;
- printk(KERN_ERR "No NUMA hash function found. Emulation disabled.\n");
- return -1;
- }
- for_each_online_node ( i )
- setup_node_bootmem(i, nodes[i].start, nodes[i].end);
- numa_init_array();
-
- return 0;
-}
-#endif
-
-void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
-{
- int i;
-
-#ifdef CONFIG_NUMA_EMU
- if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
- return;
-#endif
-
-#ifdef CONFIG_ACPI_NUMA
- if ( !numa_off &&
- !numa_scan_nodes(pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn)) )
- return;
-#endif
-
- printk(KERN_INFO "%s\n",
- numa_off ? "NUMA turned off" : "No NUMA configuration found");
-
- printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
- pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn));
- /* setup dummy node covering all memory */
- memnode_shift = BITS_PER_LONG - 1;
- memnodemap = _memnodemap;
- nodes_clear(node_online_map);
- node_set_online(0);
- for ( i = 0; i < nr_cpu_ids; i++ )
- numa_set_node(i, 0);
- cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
- setup_node_bootmem(0, pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn));
-}
-
-void numa_add_cpu(int cpu)
-{
- cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
-}
-
-void numa_set_node(int cpu, nodeid_t node)
-{
- cpu_to_node[cpu] = node;
-}
-
-/* [numa=off] */
-static int __init numa_setup(char *opt)
-{
- if ( !strncmp(opt, "off", 3) )
- numa_off = true;
- if ( !strncmp(opt, "on", 2) )
- numa_off = false;
-#ifdef CONFIG_NUMA_EMU
- if ( !strncmp(opt, "fake=", 5) )
- {
- numa_off = false;
- numa_fake = simple_strtoul(opt + 5, NULL, 0);
- if ( numa_fake >= MAX_NUMNODES )
- numa_fake = MAX_NUMNODES;
- }
-#endif
-#ifdef CONFIG_ACPI_NUMA
- if ( !strncmp(opt,"noacpi", 6) )
- {
- numa_off = false;
- acpi_numa = -1;
- }
-#endif
-
- return 1;
-}
/*
* Setup early cpu_to_node.
@@ -378,148 +72,11 @@ unsigned int __init arch_get_dma_bitsize(void)
+ PAGE_SHIFT, 32);
}
-static void dump_numa(unsigned char key)
-{
- s_time_t now = NOW();
- unsigned int i, j, n;
- int err;
- struct domain *d;
- struct page_info *page;
- unsigned int page_num_node[MAX_NUMNODES];
- const struct vnuma_info *vnuma;
-
- printk("'%c' pressed -> dumping numa info (now-0x%X:%08X)\n", key,
- (uint32_t)(now >> 32), (uint32_t)now);
-
- for_each_online_node ( i )
- {
- paddr_t pa = pfn_to_paddr(node_start_pfn(i) + 1);
-
- printk("NODE%u start->%lu size->%lu free->%lu\n",
- i, node_start_pfn(i), node_spanned_pages(i),
- avail_node_heap_pages(i));
- /* sanity check phys_to_nid() */
- if ( phys_to_nid(pa) != i )
- printk("phys_to_nid(%"PRIpaddr") -> %d should be %u\n",
- pa, phys_to_nid(pa), i);
- }
-
- j = cpumask_first(&cpu_online_map);
- n = 0;
- for_each_online_cpu ( i )
- {
- if ( i != j + n || cpu_to_node[j] != cpu_to_node[i] )
- {
- if ( n > 1 )
- printk("CPU%u...%u -> NODE%d\n", j, j + n - 1, cpu_to_node[j]);
- else
- printk("CPU%u -> NODE%d\n", j, cpu_to_node[j]);
- j = i;
- n = 1;
- }
- else
- ++n;
- }
- if ( n > 1 )
- printk("CPU%u...%u -> NODE%d\n", j, j + n - 1, cpu_to_node[j]);
- else
- printk("CPU%u -> NODE%d\n", j, cpu_to_node[j]);
-
- rcu_read_lock(&domlist_read_lock);
-
- printk("Memory location of each domain:\n");
- for_each_domain ( d )
- {
- process_pending_softirqs();
-
- printk("Domain %u (total: %u):\n", d->domain_id, d->tot_pages);
-
- for_each_online_node ( i )
- page_num_node[i] = 0;
-
- spin_lock(&d->page_alloc_lock);
- page_list_for_each(page, &d->page_list)
- {
- i = phys_to_nid((paddr_t)page_to_mfn(page) << PAGE_SHIFT);
- page_num_node[i]++;
- }
- spin_unlock(&d->page_alloc_lock);
-
- for_each_online_node ( i )
- printk(" Node %u: %u\n", i, page_num_node[i]);
-
- if ( !read_trylock(&d->vnuma_rwlock) )
- continue;
-
- if ( !d->vnuma )
- {
- read_unlock(&d->vnuma_rwlock);
- continue;
- }
-
- vnuma = d->vnuma;
- printk(" %u vnodes, %u vcpus, guest physical layout:\n",
- vnuma->nr_vnodes, d->max_vcpus);
- for ( i = 0; i < vnuma->nr_vnodes; i++ )
- {
- unsigned int start_cpu = ~0U;
-
- err = snprintf(keyhandler_scratch, 12, "%3u",
- vnuma->vnode_to_pnode[i]);
- if ( err < 0 || vnuma->vnode_to_pnode[i] == NUMA_NO_NODE )
- strlcpy(keyhandler_scratch, "???", sizeof(keyhandler_scratch));
-
- printk(" %3u: pnode %s,", i, keyhandler_scratch);
-
- printk(" vcpus ");
-
- for ( j = 0; j < d->max_vcpus; j++ )
- {
- if ( !(j & 0x3f) )
- process_pending_softirqs();
-
- if ( vnuma->vcpu_to_vnode[j] == i )
- {
- if ( start_cpu == ~0U )
- {
- printk("%d", j);
- start_cpu = j;
- }
- }
- else if ( start_cpu != ~0U )
- {
- if ( j - 1 != start_cpu )
- printk("-%d ", j - 1);
- else
- printk(" ");
- start_cpu = ~0U;
- }
- }
-
- if ( start_cpu != ~0U && start_cpu != j - 1 )
- printk("-%d", j - 1);
-
- printk("\n");
-
- for ( j = 0; j < vnuma->nr_vmemranges; j++ )
- {
- if ( vnuma->vmemrange[j].nid == i )
- printk(" %016"PRIx64" - %016"PRIx64"\n",
- vnuma->vmemrange[j].start,
- vnuma->vmemrange[j].end);
- }
- }
-
- read_unlock(&d->vnuma_rwlock);
- }
-
- rcu_read_unlock(&domlist_read_lock);
-}
-
-static int __init register_numa_trigger(void)
-{
- register_keyhandler('u', dump_numa, "dump NUMA info", 1);
- return 0;
-}
-__initcall(register_numa_trigger);
-
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c
index 78af0d2..168c9d4 100644
--- a/xen/arch/x86/smpboot.c
+++ b/xen/arch/x86/smpboot.c
@@ -58,6 +58,7 @@ DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, cpu_core_mask);
DEFINE_PER_CPU_READ_MOSTLY(cpumask_var_t, scratch_cpumask);
static cpumask_t scratch_cpu0mask;
+nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
cpumask_t cpu_online_map __read_mostly;
EXPORT_SYMBOL(cpu_online_map);
diff --git a/xen/common/Makefile b/xen/common/Makefile
index 26c5a64..c8fdaf7 100644
--- a/xen/common/Makefile
+++ b/xen/common/Makefile
@@ -64,6 +64,7 @@ obj-y += wait.o
obj-bin-y += warning.init.o
obj-$(CONFIG_XENOPROF) += xenoprof.o
obj-y += xmalloc_tlsf.o
+obj-$(CONFIG_NUMA) += numa.o
obj-bin-$(CONFIG_X86) += $(foreach n,decompress bunzip2 unxz unlzma unlzo unlz4 earlycpio,$(n).init.o)
diff --git a/xen/common/numa.c b/xen/common/numa.c
new file mode 100644
index 0000000..0381f1b
--- /dev/null
+++ b/xen/common/numa.c
@@ -0,0 +1,487 @@
+/*
+ * Common NUMA handling functions for x86 and arm.
+ * Original code extracted from arch/x86/numa.c
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/init.h>
+#include <xen/ctype.h>
+#include <xen/sched.h>
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+#include <xen/keyhandler.h>
+#include <xen/time.h>
+#include <xen/smp.h>
+#include <xen/pfn.h>
+#include <xen/mm.h>
+#include <xen/softirq.h>
+#include <xen/string.h>
+#include <asm/acpi.h>
+
+static int numa_setup(char *s);
+custom_param("numa", numa_setup);
+
+struct node_data node_data[MAX_NUMNODES];
+
+/* Mapping from pdx to node id */
+unsigned int memnode_shift;
+
+/*
+ * In case of numa init failure or numa off,
+ * memnode_shift is initialized to BITS_PER_LONG - 1. Hence allocate
+ * memnodemap[] of BITS_PER_LONG.
+ */
+static typeof(*memnodemap) _memnodemap[BITS_PER_LONG];
+unsigned long memnodemapsize;
+uint8_t *memnodemap;
+
+nodeid_t __read_mostly cpu_to_node[NR_CPUS] = {
+ [0 ... NR_CPUS-1] = NUMA_NO_NODE
+};
+
+cpumask_t __read_mostly node_to_cpumask[MAX_NUMNODES];
+
+bool numa_off;
+s8 acpi_numa = 0;
+
+int srat_disabled(void)
+{
+ return numa_off || acpi_numa < 0;
+}
+
+/*
+ * Given a shift value, try to populate memnodemap[]
+ * Returns :
+ * 0 if OK
+ * -ENOSPC if memnodmap[] too small (or shift too small)
+ * -EINVAL if node overlap or lost ram (shift too big)
+ */
+static int __init populate_memnodemap(const struct node *nodes,
+ unsigned int numnodes, unsigned int shift,
+ nodeid_t *nodeids)
+{
+ unsigned long spdx, epdx;
+ int i, res = -EINVAL;
+
+ memset(memnodemap, NUMA_NO_NODE, memnodemapsize * sizeof(*memnodemap));
+ for ( i = 0; i < numnodes; i++ )
+ {
+ spdx = paddr_to_pdx(nodes[i].start);
+ epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
+ if ( spdx >= epdx )
+ continue;
+ if ( (epdx >> shift) >= memnodemapsize )
+ return -ENOSPC;
+ do {
+ if ( memnodemap[spdx >> shift] != NUMA_NO_NODE )
+ return -EINVAL;
+
+ if ( !nodeids )
+ memnodemap[spdx >> shift] = i;
+ else
+ memnodemap[spdx >> shift] = nodeids[i];
+
+ spdx += (1UL << shift);
+ } while ( spdx < epdx );
+ res = 0;
+ }
+
+ return res;
+}
+
+static int __init allocate_cachealigned_memnodemap(void)
+{
+ unsigned long size = PFN_UP(memnodemapsize * sizeof(*memnodemap));
+ unsigned long mfn = alloc_boot_pages(size, 1);
+
+ if ( !mfn )
+ {
+ printk(KERN_ERR
+ "NUMA: Unable to allocate Memory to Node hash map\n");
+ memnodemapsize = 0;
+ return -ENOMEM;
+ }
+
+ memnodemap = mfn_to_virt(mfn);
+ mfn <<= PAGE_SHIFT;
+ size <<= PAGE_SHIFT;
+ printk(KERN_DEBUG "NUMA: Allocated memnodemap from %lx - %lx\n",
+ mfn, mfn + size);
+ memnodemapsize = size / sizeof(*memnodemap);
+
+ return 0;
+}
+
+/*
+ * The LSB of all start and end addresses in the node map is the value of the
+ * maximum possible shift.
+ */
+static unsigned int __init extract_lsb_from_nodes(const struct node *nodes,
+ unsigned int numnodes)
+{
+ unsigned int i, nodes_used = 0;
+ unsigned long spdx, epdx;
+ unsigned long bitfield = 0, memtop = 0;
+
+ for ( i = 0; i < numnodes; i++ )
+ {
+ spdx = paddr_to_pdx(nodes[i].start);
+ epdx = paddr_to_pdx(nodes[i].end - 1) + 1;
+ if ( spdx >= epdx )
+ continue;
+ bitfield |= spdx;
+ nodes_used++;
+ if ( epdx > memtop )
+ memtop = epdx;
+ }
+ if ( nodes_used <= 1 )
+ i = BITS_PER_LONG - 1;
+ else
+ i = find_first_bit(&bitfield, sizeof(unsigned long) * 8);
+ memnodemapsize = (memtop >> i) + 1;
+
+ return i;
+}
+
+int __init compute_memnode_shift(struct node *nodes, unsigned int numnodes,
+ nodeid_t *nodeids)
+{
+ int ret;
+
+ memnode_shift = extract_lsb_from_nodes(nodes, numnodes);
+
+ if ( memnodemapsize <= ARRAY_SIZE(_memnodemap) )
+ memnodemap = _memnodemap;
+ else if ( allocate_cachealigned_memnodemap() )
+ return -ENOMEM;
+
+ printk(KERN_DEBUG "NUMA: Using %u for the hash shift.\n", memnode_shift);
+
+ ret = populate_memnodemap(nodes, numnodes, memnode_shift, nodeids);
+ if ( ret )
+ {
+ printk(KERN_INFO "Your memory is not aligned you need to "
+ "rebuild your hypervisor with a bigger NODEMAPSIZE "
+ "shift=%u\n", memnode_shift);
+ return ret;
+ }
+
+ return 0;
+}
+/* initialize NODE_DATA given nodeid and start/end */
+void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end)
+{
+ unsigned long start_pfn, end_pfn;
+
+ start_pfn = paddr_to_pfn(start);
+ end_pfn = paddr_to_pfn(end);
+
+ NODE_DATA(nodeid)->node_start_pfn = start_pfn;
+ NODE_DATA(nodeid)->node_spanned_pages = end_pfn - start_pfn;
+
+ node_set_online(nodeid);
+}
+
+void __init numa_init_array(void)
+{
+ int rr, i;
+
+ /* There are unfortunately some poorly designed mainboards around
+ that only connect memory to a single CPU. This breaks the 1:1 cpu->node
+ mapping. To avoid this fill in the mapping for all possible
+ CPUs, as the number of CPUs is not known yet.
+ We round robin the existing nodes. */
+ rr = first_node(node_online_map);
+ for ( i = 0; i < nr_cpu_ids; i++ )
+ {
+ if ( cpu_to_node[i] != NUMA_NO_NODE )
+ continue;
+ numa_set_node(i, rr);
+ rr = next_node(rr, node_online_map);
+ if ( rr == MAX_NUMNODES )
+ rr = first_node(node_online_map);
+ }
+}
+
+#ifdef CONFIG_NUMA_EMU
+static unsigned int __initdata numa_fake;
+
+/* Numa emulation */
+static int __init numa_emulation(uint64_t start_pfn, uint64_t end_pfn)
+{
+ unsigned int i;
+ struct node nodes[MAX_NUMNODES];
+ uint64_t sz = ((end_pfn - start_pfn) << PAGE_SHIFT) / numa_fake;
+
+ /* Kludge needed for the hash function */
+ if ( hweight64(sz) > 1 )
+ {
+ uint64_t x = 1;
+
+ while ( (x << 1) < sz )
+ x <<= 1;
+ if ( x < sz / 2 )
+ printk(KERN_ERR
+ "Numa emulation unbalanced. Complain to maintainer\n");
+ sz = x;
+ }
+
+ memset(&nodes,0,sizeof(nodes));
+ for ( i = 0; i < numa_fake; i++ )
+ {
+ nodes[i].start = pfn_to_paddr(start_pfn) + i * sz;
+ if ( i == numa_fake - 1 )
+ sz = pfn_to_paddr(end_pfn) - nodes[i].start;
+ nodes[i].end = nodes[i].start + sz;
+ printk(KERN_INFO
+ "Faking node %d at %"PRIx64"-%"PRIx64" (%"PRIu64"MB)\n",
+ i, nodes[i].start, nodes[i].end,
+ (nodes[i].end - nodes[i].start) >> 20);
+ node_set_online(i);
+ }
+ if ( compute_memnode_shift(nodes, numa_fake, NULL) )
+ {
+ memnode_shift = 0;
+ printk(KERN_ERR "No NUMA hash function found. Emulation disabled.\n");
+ return -1;
+ }
+ for_each_online_node ( i )
+ setup_node_bootmem(i, nodes[i].start, nodes[i].end);
+ numa_init_array();
+
+ return 0;
+}
+#endif
+
+void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
+{
+ int i;
+
+#ifdef CONFIG_NUMA_EMU
+ if ( numa_fake && !numa_emulation(start_pfn, end_pfn) )
+ return;
+#endif
+
+#ifdef CONFIG_ACPI_NUMA
+ if ( !numa_off &&
+ !numa_scan_nodes(pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn)) )
+ return;
+#endif
+
+ printk(KERN_INFO "%s\n",
+ numa_off ? "NUMA turned off" : "No NUMA configuration found");
+
+ printk(KERN_INFO "Faking a node at %016"PRIx64"-%016"PRIx64"\n",
+ pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn));
+ /* setup dummy node covering all memory */
+ memnode_shift = BITS_PER_LONG - 1;
+ memnodemap = _memnodemap;
+ nodes_clear(node_online_map);
+ node_set_online(0);
+ for ( i = 0; i < nr_cpu_ids; i++ )
+ numa_set_node(i, 0);
+ cpumask_copy(&node_to_cpumask[0], cpumask_of(0));
+ setup_node_bootmem(0, pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn));
+}
+
+void numa_add_cpu(int cpu)
+{
+ cpumask_set_cpu(cpu, &node_to_cpumask[cpu_to_node(cpu)]);
+}
+
+void numa_set_node(int cpu, nodeid_t node)
+{
+ cpu_to_node[cpu] = node;
+}
+
+/* [numa=off] */
+static int __init numa_setup(char *opt)
+{
+ if ( !strncmp(opt, "off", 3) )
+ numa_off = true;
+ if ( !strncmp(opt, "on", 2) )
+ numa_off = false;
+#ifdef CONFIG_NUMA_EMU
+ if ( !strncmp(opt, "fake=", 5) )
+ {
+ numa_off = false;
+ numa_fake = simple_strtoul(opt + 5, NULL, 0);
+ if ( numa_fake >= MAX_NUMNODES )
+ numa_fake = MAX_NUMNODES;
+ }
+#endif
+#ifdef CONFIG_ACPI_NUMA
+ if ( !strncmp(opt,"noacpi", 6) )
+ {
+ numa_off = false;
+ acpi_numa = -1;
+ }
+#endif
+
+ return 1;
+}
+
+static void dump_numa(unsigned char key)
+{
+ s_time_t now = NOW();
+ unsigned int i, j, n;
+ int err;
+ struct domain *d;
+ struct page_info *page;
+ unsigned int page_num_node[MAX_NUMNODES];
+ const struct vnuma_info *vnuma;
+
+ printk("'%c' pressed -> dumping numa info (now-0x%X:%08X)\n", key,
+ (uint32_t)(now >> 32), (uint32_t)now);
+
+ for_each_online_node ( i )
+ {
+ paddr_t pa = pfn_to_paddr(node_start_pfn(i) + 1);
+
+ printk("NODE%u start->%lu size->%lu free->%lu\n",
+ i, node_start_pfn(i), node_spanned_pages(i),
+ avail_node_heap_pages(i));
+ /* sanity check phys_to_nid() */
+ if ( phys_to_nid(pa) != i )
+ printk("phys_to_nid(%"PRIpaddr") -> %d should be %u\n",
+ pa, phys_to_nid(pa), i);
+ }
+
+ j = cpumask_first(&cpu_online_map);
+ n = 0;
+ for_each_online_cpu ( i )
+ {
+ if ( i != j + n || cpu_to_node[j] != cpu_to_node[i] )
+ {
+ if ( n > 1 )
+ printk("CPU%u...%u -> NODE%d\n", j, j + n - 1, cpu_to_node[j]);
+ else
+ printk("CPU%u -> NODE%d\n", j, cpu_to_node[j]);
+ j = i;
+ n = 1;
+ }
+ else
+ ++n;
+ }
+ if ( n > 1 )
+ printk("CPU%u...%u -> NODE%d\n", j, j + n - 1, cpu_to_node[j]);
+ else
+ printk("CPU%u -> NODE%d\n", j, cpu_to_node[j]);
+
+ rcu_read_lock(&domlist_read_lock);
+
+ printk("Memory location of each domain:\n");
+ for_each_domain ( d )
+ {
+ process_pending_softirqs();
+
+ printk("Domain %u (total: %u):\n", d->domain_id, d->tot_pages);
+
+ for_each_online_node ( i )
+ page_num_node[i] = 0;
+
+ spin_lock(&d->page_alloc_lock);
+ page_list_for_each(page, &d->page_list)
+ {
+ i = phys_to_nid((paddr_t)page_to_mfn(page) << PAGE_SHIFT);
+ page_num_node[i]++;
+ }
+ spin_unlock(&d->page_alloc_lock);
+
+ for_each_online_node ( i )
+ printk(" Node %u: %u\n", i, page_num_node[i]);
+
+ if ( !read_trylock(&d->vnuma_rwlock) )
+ continue;
+
+ if ( !d->vnuma )
+ {
+ read_unlock(&d->vnuma_rwlock);
+ continue;
+ }
+
+ vnuma = d->vnuma;
+ printk(" %u vnodes, %u vcpus, guest physical layout:\n",
+ vnuma->nr_vnodes, d->max_vcpus);
+ for ( i = 0; i < vnuma->nr_vnodes; i++ )
+ {
+ unsigned int start_cpu = ~0U;
+
+ err = snprintf(keyhandler_scratch, 12, "%3u",
+ vnuma->vnode_to_pnode[i]);
+ if ( err < 0 || vnuma->vnode_to_pnode[i] == NUMA_NO_NODE )
+ strlcpy(keyhandler_scratch, "???", sizeof(keyhandler_scratch));
+
+ printk(" %3u: pnode %s,", i, keyhandler_scratch);
+
+ printk(" vcpus ");
+
+ for ( j = 0; j < d->max_vcpus; j++ )
+ {
+ if ( !(j & 0x3f) )
+ process_pending_softirqs();
+
+ if ( vnuma->vcpu_to_vnode[j] == i )
+ {
+ if ( start_cpu == ~0U )
+ {
+ printk("%d", j);
+ start_cpu = j;
+ }
+ }
+ else if ( start_cpu != ~0U )
+ {
+ if ( j - 1 != start_cpu )
+ printk("-%d ", j - 1);
+ else
+ printk(" ");
+ start_cpu = ~0U;
+ }
+ }
+
+ if ( start_cpu != ~0U && start_cpu != j - 1 )
+ printk("-%d", j - 1);
+
+ printk("\n");
+
+ for ( j = 0; j < vnuma->nr_vmemranges; j++ )
+ {
+ if ( vnuma->vmemrange[j].nid == i )
+ printk(" %016"PRIx64" - %016"PRIx64"\n",
+ vnuma->vmemrange[j].start,
+ vnuma->vmemrange[j].end);
+ }
+ }
+
+ read_unlock(&d->vnuma_rwlock);
+ }
+
+ rcu_read_unlock(&domlist_read_lock);
+}
+
+static int __init register_numa_trigger(void)
+{
+ register_keyhandler('u', dump_numa, "dump NUMA info", 1);
+ return 0;
+}
+__initcall(register_numa_trigger);
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/asm-x86/acpi.h b/xen/include/asm-x86/acpi.h
index 220c2d7..a65c85f 100644
--- a/xen/include/asm-x86/acpi.h
+++ b/xen/include/asm-x86/acpi.h
@@ -103,7 +103,6 @@ extern void acpi_reserve_bootmem(void);
#define ARCH_HAS_POWER_INIT 1
-extern s8 acpi_numa;
extern int numa_scan_nodes(paddr_t start, paddr_t end);
#ifdef CONFIG_ACPI_SLEEP
diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
index acf509c..41bb3ef 100644
--- a/xen/include/asm-x86/numa.h
+++ b/xen/include/asm-x86/numa.h
@@ -3,76 +3,20 @@
#include <xen/cpumask.h>
-#define MAX_NUMNODES NR_NODES
-#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
-
typedef uint8_t nodeid_t;
extern int srat_rev;
-extern nodeid_t cpu_to_node[NR_CPUS];
-extern cpumask_t node_to_cpumask[];
-
-#define cpu_to_node(cpu) (cpu_to_node[cpu])
-#define parent_node(node) (node)
-#define node_to_first_cpu(node) (__ffs(node_to_cpumask[node]))
-#define node_to_cpumask(node) (node_to_cpumask[node])
-
-struct node {
- paddr_t start;
- paddr_t end;
-};
-
-extern int compute_memnode_shift(struct node *nodes, unsigned int numnodes,
- nodeid_t *nodeids);
extern nodeid_t pxm_to_node(unsigned int pxm);
#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
-extern void numa_add_cpu(int cpu);
-extern void numa_init_array(void);
-extern bool numa_off;
-
-
-extern int srat_disabled(void);
-extern void numa_set_node(int cpu, nodeid_t node);
extern nodeid_t acpi_setup_node(unsigned int pxm);
extern void srat_detect_node(int cpu);
-extern void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
extern nodeid_t apicid_to_node[];
extern void init_cpu_to_node(void);
-/* Simple perfect hash to map pdx to node numbers */
-extern unsigned int memnode_shift;
-extern unsigned long memnodemapsize;
-extern uint8_t *memnodemap;
-
-struct node_data {
- unsigned long node_start_pfn;
- unsigned long node_spanned_pages;
-};
-
-extern struct node_data node_data[];
-
-static inline __attribute_pure__ nodeid_t phys_to_nid(paddr_t addr)
-{
- nodeid_t nid;
-
- ASSERT((paddr_to_pdx(addr) >> memnode_shift) < memnodemapsize);
- nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
- ASSERT(nid <= MAX_NUMNODES || !node_data[nid].node_start_pfn);
-
- return nid;
-}
-
-#define NODE_DATA(nid) (&(node_data[nid]))
-
-#define node_start_pfn(nid) NODE_DATA(nid)->node_start_pfn
-#define node_spanned_pages(nid) NODE_DATA(nid)->node_spanned_pages
-#define node_end_pfn(nid) NODE_DATA(nid)->node_start_pfn + \
- NODE_DATA(nid)->node_spanned_pages
-
extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
void srat_parse_regions(paddr_t addr);
diff --git a/xen/include/asm-x86/setup.h b/xen/include/asm-x86/setup.h
index c5b3d4e..cfd83d6 100644
--- a/xen/include/asm-x86/setup.h
+++ b/xen/include/asm-x86/setup.h
@@ -26,7 +26,6 @@ int transmeta_init_cpu(void);
void set_nr_cpu_ids(unsigned int max_cpus);
-void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
void arch_init_memory(void);
void subarch_init_memory(void);
diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
index 3bb4afc..c6bbbdf 100644
--- a/xen/include/xen/numa.h
+++ b/xen/include/xen/numa.h
@@ -1,11 +1,75 @@
#ifndef _XEN_NUMA_H
#define _XEN_NUMA_H
+#include <xen/cpumask.h>
+#include <xen/mm.h>
#include <asm/numa.h>
#define NUMA_NO_NODE 0xFF
#define NUMA_NO_DISTANCE 0xFF
+#define MAX_NUMNODES NR_NODES
+#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
+
+struct node {
+ paddr_t start;
+ paddr_t end;
+};
+
+extern nodeid_t cpu_to_node[NR_CPUS];
+extern cpumask_t node_to_cpumask[];
+/* Simple perfect hash to map pdx to node numbers */
+extern unsigned int memnode_shift;
+extern unsigned long memnodemapsize;
+extern uint8_t *memnodemap;
+extern bool numa_off;
+extern s8 acpi_numa;
+
+void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
+int compute_memnode_shift(struct node *nodes, unsigned int numnodes,
+ nodeid_t *nodeids);
+int srat_disabled(void);
+void numa_init_array(void);
+void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
+
+#ifdef CONFIG_NUMA
+#define cpu_to_node(cpu) (cpu_to_node[cpu])
+#define parent_node(node) (node)
+#define node_to_first_cpu(node) (__ffs(node_to_cpumask[node]))
+#define node_to_cpumask(node) (node_to_cpumask[node])
+
+struct node_data {
+ unsigned long node_start_pfn;
+ unsigned long node_spanned_pages;
+};
+
+extern struct node_data node_data[];
+
+static inline __attribute_pure__ nodeid_t phys_to_nid(paddr_t addr)
+{
+ nodeid_t nid;
+
+ ASSERT((paddr_to_pdx(addr) >> memnode_shift) < memnodemapsize);
+ nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
+ ASSERT(nid <= MAX_NUMNODES || !node_data[nid].node_start_pfn);
+
+ return nid;
+}
+
+#define NODE_DATA(nid) (&(node_data[nid]))
+
+#define node_start_pfn(nid) NODE_DATA(nid)->node_start_pfn
+#define node_spanned_pages(nid) NODE_DATA(nid)->node_spanned_pages
+#define node_end_pfn(nid) NODE_DATA(nid)->node_start_pfn + \
+ NODE_DATA(nid)->node_spanned_pages
+
+void numa_add_cpu(int cpu);
+void numa_set_node(int cpu, nodeid_t node);
+#else
+static inline void numa_add_cpu(int cpu) { }
+static inline void numa_set_node(int cpu, nodeid_t node) { }
+#endif
+
#define vcpu_to_node(v) (cpu_to_node((v)->processor))
#define domain_to_node(d) \
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 09/24] NUMA: x86: Move common code from srat.c
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (7 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-20 11:17 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 10/24] NUMA: Allow numa initialization with DT vijay.kilari
` (15 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Move code from xen/arch/x86/srat.c to xen/common/numa.c
so that it can be used by other archs.
Apart from moving the code the following changes are done
- Coding style of code moved to numa.c is changed to xen style
- {memory,processor}_nodes_parsed are made global and moved
to xen/nodemask.h
- Few generic static functions in x86/srat.c are made
non-static
- Functions moved from x85/srat.c to common/numa.c are made
non-static
- numa_scan_nodes() is made as static function
- compute_memnode_shift() and setup_node_bootmem() are made
static.
Also {memory,processor}_nodes_parsed are made as global.
These are used across multiple code files. Adding helpers
to access these nodemask_t is complex.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: - Move declaration of {memory,processor}_nodes_parsed to header
file
- Drop redundant get_memblk() declaration
- numa_scan_nodes(), setup_node_bootmem(), compute_memnode_shift()
are made as static function
---
xen/arch/x86/srat.c | 151 +----------------------------------------
xen/common/numa.c | 165 +++++++++++++++++++++++++++++++++++++++++++--
xen/include/asm-x86/acpi.h | 2 -
xen/include/asm-x86/numa.h | 2 -
xen/include/xen/nodemask.h | 2 +
xen/include/xen/numa.h | 13 ++--
6 files changed, 174 insertions(+), 161 deletions(-)
diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
index 03bc37d..be2634a 100644
--- a/xen/arch/x86/srat.c
+++ b/xen/arch/x86/srat.c
@@ -23,10 +23,6 @@
static struct acpi_table_slit *__read_mostly acpi_slit;
-static nodemask_t __initdata memory_nodes_parsed;
-static nodemask_t __initdata processor_nodes_parsed;
-static struct node __initdata nodes[MAX_NUMNODES];
-
struct pxm2node {
unsigned int pxm;
nodeid_t node;
@@ -36,49 +32,8 @@ static struct pxm2node __read_mostly pxm2node[MAX_NUMNODES] =
static unsigned int node_to_pxm(nodeid_t n);
-static int num_node_memblks;
-static struct node node_memblk_range[NR_NODE_MEMBLKS];
-static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
-static struct node *get_numa_node(unsigned int id)
-{
- return &nodes[id];
-}
-
-static nodeid_t get_memblk_nodeid(unsigned int id)
-{
- return memblk_nodeid[id];
-}
-
-static nodeid_t *get_memblk_nodeid_map(void)
-{
- return &memblk_nodeid[0];
-}
-
-static struct node *get_node_memblk_range(unsigned int memblk)
-{
- return &node_memblk_range[memblk];
-}
-
-static int get_num_node_memblks(void)
-{
- return num_node_memblks;
-}
-
-static int __init numa_add_memblk(nodeid_t nodeid, paddr_t start, uint64_t size)
-{
- if (nodeid >= NR_NODE_MEMBLKS)
- return -EINVAL;
-
- node_memblk_range[num_node_memblks].start = start;
- node_memblk_range[num_node_memblks].end = start + size;
- memblk_nodeid[num_node_memblks] = nodeid;
- num_node_memblks++;
-
- return 0;
-}
-
static inline bool node_found(unsigned int idx, unsigned int pxm)
{
return ((pxm2node[idx].pxm == pxm) &&
@@ -149,54 +104,7 @@ nodeid_t acpi_setup_node(unsigned int pxm)
return node;
}
-int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
-{
- int i;
-
- for (i = 0; i < get_num_node_memblks(); i++) {
- struct node *nd = get_node_memblk_range(i);
-
- if (nd->start <= start && nd->end > end &&
- get_memblk_nodeid(i) == node)
- return 1;
- }
-
- return 0;
-}
-
-static int __init conflicting_memblks(paddr_t start, paddr_t end)
-{
- int i;
-
- for (i = 0; i < get_num_node_memblks(); i++) {
- struct node *nd = get_node_memblk_range(i);
- if (nd->start == nd->end)
- continue;
- if (nd->end > start && nd->start < end)
- return i;
- if (nd->end == end && nd->start == start)
- return i;
- }
- return -1;
-}
-
-static void __init cutoff_node(nodeid_t i, paddr_t start, paddr_t end)
-{
- struct node *nd = get_numa_node(i);
-
- if (nd->start < start) {
- nd->start = start;
- if (nd->end < nd->start)
- nd->start = nd->end;
- }
- if (nd->end > end) {
- nd->end = end;
- if (nd->start > nd->end)
- nd->start = nd->end;
- }
-}
-
-static void __init numa_failed(void)
+void __init numa_failed(void)
{
int i;
printk(KERN_ERR "SRAT: SRAT not used.\n");
@@ -412,7 +320,7 @@ acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
/* Sanity check to catch more bad SRATs (they are amazingly common).
Make sure the PXMs cover all memory. */
-static bool __init arch_sanitize_nodes_memory(void)
+bool __init arch_sanitize_nodes_memory(void)
{
int i;
@@ -509,61 +417,6 @@ void __init srat_parse_regions(paddr_t addr)
pfn_pdx_hole_setup(mask >> PAGE_SHIFT);
}
-/* Use the information discovered above to actually set up the nodes. */
-int __init numa_scan_nodes(paddr_t start, paddr_t end)
-{
- unsigned int i;
- nodemask_t all_nodes_parsed;
- struct node *memblks;
- nodeid_t *nodeids;
-
- /* First clean up the node list */
- for (i = 0; i < MAX_NUMNODES; i++)
- cutoff_node(i, start, end);
-
- if (acpi_numa <= 0)
- return -1;
-
- if (!arch_sanitize_nodes_memory()) {
- numa_failed();
- return -1;
- }
-
- memblks = get_node_memblk_range(0);
- nodeids = get_memblk_nodeid_map();
- if (compute_memnode_shift(node_memblk_range, num_node_memblks,
- memblk_nodeid)) {
- memnode_shift = 0;
- printk(KERN_ERR
- "SRAT: No NUMA node hash function found. Contact maintainer\n");
- numa_failed();
- return -1;
- }
-
- nodes_or(all_nodes_parsed, memory_nodes_parsed, processor_nodes_parsed);
-
- /* Finally register nodes */
- for_each_node_mask(i, all_nodes_parsed)
- {
- struct node *nd = get_numa_node(i);
- uint64_t size = nd->end - nd->start;
-
- if ( size == 0 )
- printk(KERN_WARNING "SRAT: Node %u has no memory. "
- "BIOS Bug or mis-configured hardware?\n", i);
-
- setup_node_bootmem(i, nd->start, nd->end);
- }
- for (i = 0; i < nr_cpu_ids; i++) {
- if (cpu_to_node[i] == NUMA_NO_NODE)
- continue;
- if (!node_isset(cpu_to_node[i], processor_nodes_parsed))
- numa_set_node(i, NUMA_NO_NODE);
- }
- numa_init_array();
- return 0;
-}
-
static unsigned int node_to_pxm(nodeid_t n)
{
unsigned int i;
diff --git a/xen/common/numa.c b/xen/common/numa.c
index 0381f1b..74c4697 100644
--- a/xen/common/numa.c
+++ b/xen/common/numa.c
@@ -54,12 +54,109 @@ cpumask_t __read_mostly node_to_cpumask[MAX_NUMNODES];
bool numa_off;
s8 acpi_numa = 0;
+nodemask_t __initdata memory_nodes_parsed;
+nodemask_t __initdata processor_nodes_parsed;
+static struct node __initdata nodes[MAX_NUMNODES];
+static int num_node_memblks;
+static struct node node_memblk_range[NR_NODE_MEMBLKS];
+static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
int srat_disabled(void)
{
return numa_off || acpi_numa < 0;
}
+struct node *get_numa_node(unsigned int id)
+{
+ return &nodes[id];
+}
+
+nodeid_t get_memblk_nodeid(unsigned int id)
+{
+ return memblk_nodeid[id];
+}
+
+static nodeid_t *get_memblk_nodeid_map(void)
+{
+ return &memblk_nodeid[0];
+}
+
+struct node *get_node_memblk_range(unsigned int memblk)
+{
+ return &node_memblk_range[memblk];
+}
+
+int get_num_node_memblks(void)
+{
+ return num_node_memblks;
+}
+
+int __init numa_add_memblk(nodeid_t nodeid, paddr_t start, uint64_t size)
+{
+ if ( nodeid >= NR_NODE_MEMBLKS )
+ return -EINVAL;
+
+ node_memblk_range[num_node_memblks].start = start;
+ node_memblk_range[num_node_memblks].end = start + size;
+ memblk_nodeid[num_node_memblks] = nodeid;
+ num_node_memblks++;
+
+ return 0;
+}
+
+int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node)
+{
+ int i;
+
+ for ( i = 0; i < get_num_node_memblks(); i++ )
+ {
+ struct node *nd = get_node_memblk_range(i);
+
+ if ( nd->start <= start && nd->end > end &&
+ get_memblk_nodeid(i) == node )
+ return 1;
+ }
+
+ return 0;
+}
+
+int __init conflicting_memblks(paddr_t start, paddr_t end)
+{
+ int i;
+
+ for ( i = 0; i < get_num_node_memblks(); i++ )
+ {
+ struct node *nd = get_node_memblk_range(i);
+
+ if ( nd->start == nd->end )
+ continue;
+ if ( nd->end > start && nd->start < end )
+ return i;
+ if ( nd->end == end && nd->start == start )
+ return i;
+ }
+
+ return -1;
+}
+
+static void __init cutoff_node(nodeid_t i, paddr_t start, paddr_t end)
+{
+ struct node *nd = get_numa_node(i);
+
+ if ( nd->start < start )
+ {
+ nd->start = start;
+ if ( nd->end < nd->start )
+ nd->start = nd->end;
+ }
+ if ( nd->end > end )
+ {
+ nd->end = end;
+ if ( nd->start > nd->end )
+ nd->start = nd->end;
+ }
+}
+
/*
* Given a shift value, try to populate memnodemap[]
* Returns :
@@ -154,8 +251,8 @@ static unsigned int __init extract_lsb_from_nodes(const struct node *nodes,
return i;
}
-int __init compute_memnode_shift(struct node *nodes, unsigned int numnodes,
- nodeid_t *nodeids)
+static int __init compute_memnode_shift(struct node *nodes, unsigned int numnodes,
+ nodeid_t *nodeids)
{
int ret;
@@ -180,7 +277,8 @@ int __init compute_memnode_shift(struct node *nodes, unsigned int numnodes,
return 0;
}
/* initialize NODE_DATA given nodeid and start/end */
-void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end)
+static void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start,
+ paddr_t end)
{
unsigned long start_pfn, end_pfn;
@@ -193,7 +291,7 @@ void __init setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end)
node_set_online(nodeid);
}
-void __init numa_init_array(void)
+static void __init numa_init_array(void)
{
int rr, i;
@@ -214,6 +312,65 @@ void __init numa_init_array(void)
}
}
+/* Use the information discovered above to actually set up the nodes. */
+static int __init numa_scan_nodes(paddr_t start, paddr_t end)
+{
+ unsigned int i;
+ nodemask_t all_nodes_parsed;
+ struct node *memblks;
+ nodeid_t *nodeids;
+
+ /* First clean up the node list */
+ for ( i = 0; i < MAX_NUMNODES; i++ )
+ cutoff_node(i, start, end);
+
+ if ( acpi_numa <= 0 )
+ return -1;
+
+ if ( !arch_sanitize_nodes_memory() )
+ {
+ numa_failed();
+ return -1;
+ }
+
+ memblks = get_node_memblk_range(0);
+ nodeids = get_memblk_nodeid_map();
+ if ( compute_memnode_shift(node_memblk_range, num_node_memblks,
+ memblk_nodeid) )
+ {
+ memnode_shift = 0;
+ printk(KERN_ERR
+ "SRAT: No NUMA node hash function found. Contact maintainer\n");
+ numa_failed();
+ return -1;
+ }
+
+ nodes_or(all_nodes_parsed, memory_nodes_parsed, processor_nodes_parsed);
+
+ /* Finally register nodes */
+ for_each_node_mask(i, all_nodes_parsed)
+ {
+ struct node *nd = get_numa_node(i);
+ uint64_t size = nd->end - nd->start;
+
+ if ( size == 0 )
+ printk(KERN_WARNING "SRAT: Node %u has no memory. "
+ "BIOS Bug or mis-configured hardware?\n", i);
+
+ setup_node_bootmem(i, nd->start, nd->end);
+ }
+
+ for ( i = 0; i < nr_cpu_ids; i++ )
+ {
+ if (cpu_to_node[i] == NUMA_NO_NODE)
+ continue;
+ if (!node_isset(cpu_to_node[i], processor_nodes_parsed))
+ numa_set_node(i, NUMA_NO_NODE);
+ }
+ numa_init_array();
+ return 0;
+}
+
#ifdef CONFIG_NUMA_EMU
static unsigned int __initdata numa_fake;
diff --git a/xen/include/asm-x86/acpi.h b/xen/include/asm-x86/acpi.h
index a65c85f..59c34e7 100644
--- a/xen/include/asm-x86/acpi.h
+++ b/xen/include/asm-x86/acpi.h
@@ -103,8 +103,6 @@ extern void acpi_reserve_bootmem(void);
#define ARCH_HAS_POWER_INIT 1
-extern int numa_scan_nodes(paddr_t start, paddr_t end);
-
#ifdef CONFIG_ACPI_SLEEP
extern struct acpi_sleep_info acpi_sinfo;
diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
index 41bb3ef..d8a0a44 100644
--- a/xen/include/asm-x86/numa.h
+++ b/xen/include/asm-x86/numa.h
@@ -17,8 +17,6 @@ extern void srat_detect_node(int cpu);
extern nodeid_t apicid_to_node[];
extern void init_cpu_to_node(void);
-extern int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
-
void srat_parse_regions(paddr_t addr);
extern uint8_t __node_distance(nodeid_t a, nodeid_t b);
unsigned int arch_get_dma_bitsize(void);
diff --git a/xen/include/xen/nodemask.h b/xen/include/xen/nodemask.h
index 2a90dc1..c52c110 100644
--- a/xen/include/xen/nodemask.h
+++ b/xen/include/xen/nodemask.h
@@ -80,6 +80,8 @@
typedef struct { DECLARE_BITMAP(bits, MAX_NUMNODES); } nodemask_t;
extern nodemask_t _unused_nodemask_arg_;
+extern nodemask_t processor_nodes_parsed;
+extern nodemask_t memory_nodes_parsed;
#define node_set(node, dst) __node_set((node), &(dst))
static inline void __node_set(int node, volatile nodemask_t *dstp)
diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
index c6bbbdf..110d5dc 100644
--- a/xen/include/xen/numa.h
+++ b/xen/include/xen/numa.h
@@ -26,11 +26,8 @@ extern bool numa_off;
extern s8 acpi_numa;
void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
-int compute_memnode_shift(struct node *nodes, unsigned int numnodes,
- nodeid_t *nodeids);
int srat_disabled(void);
-void numa_init_array(void);
-void setup_node_bootmem(nodeid_t nodeid, paddr_t start, paddr_t end);
+int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
#ifdef CONFIG_NUMA
#define cpu_to_node(cpu) (cpu_to_node[cpu])
@@ -65,6 +62,14 @@ static inline __attribute_pure__ nodeid_t phys_to_nid(paddr_t addr)
void numa_add_cpu(int cpu);
void numa_set_node(int cpu, nodeid_t node);
+int conflicting_memblks(paddr_t start, paddr_t end);
+struct node *get_numa_node(unsigned int id);
+nodeid_t get_memblk_nodeid(unsigned int memblk);
+struct node *get_node_memblk_range(unsigned int memblk);
+int numa_add_memblk(nodeid_t nodeid, paddr_t start, uint64_t size);
+int get_num_node_memblks(void);
+bool arch_sanitize_nodes_memory(void);
+void numa_failed(void);
#else
static inline void numa_add_cpu(int cpu) { }
static inline void numa_set_node(int cpu, nodeid_t node) { }
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 10/24] NUMA: Allow numa initialization with DT
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (8 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 09/24] NUMA: x86: Move common code from srat.c vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-19 17:58 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 11/24] ARM: fdt: Export and introduce new fdt functions vijay.kilari
` (14 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
The common code allows numa initialization only when
ACPI_NUMA config is enabled. Allow initialization when
NUMA config is enabled for DT.
In this patch, along with acpi_numa, check for acpi_disabled
is added.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
xen/common/numa.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/xen/common/numa.c b/xen/common/numa.c
index 74c4697..5e985d2 100644
--- a/xen/common/numa.c
+++ b/xen/common/numa.c
@@ -324,7 +324,7 @@ static int __init numa_scan_nodes(paddr_t start, paddr_t end)
for ( i = 0; i < MAX_NUMNODES; i++ )
cutoff_node(i, start, end);
- if ( acpi_numa <= 0 )
+ if ( !acpi_disabled && acpi_numa <= 0 )
return -1;
if ( !arch_sanitize_nodes_memory() )
@@ -430,11 +430,9 @@ void __init numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn)
return;
#endif
-#ifdef CONFIG_ACPI_NUMA
if ( !numa_off &&
!numa_scan_nodes(pfn_to_paddr(start_pfn), pfn_to_paddr(end_pfn)) )
return;
-#endif
printk(KERN_INFO "%s\n",
numa_off ? "NUMA turned off" : "No NUMA configuration found");
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 11/24] ARM: fdt: Export and introduce new fdt functions
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (9 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 10/24] NUMA: Allow numa initialization with DT vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-18 11:41 ` [RFC PATCH v3 12/24] ARM: NUMA: DT: Parse CPU NUMA information vijay.kilari
` (13 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Introduce new api device_tree_type_matches() to check for
device type. Also export device_tree_get_u32() and
device_tree_node_compatible()
These functions are later used for parsing NUMA information.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: Export device_tree_node_compatible() instead of
device_tree_node_matches()
---
xen/arch/arm/bootfdt.c | 20 ++++++++++++++++----
xen/include/asm-arm/setup.h | 5 +++++
2 files changed, 21 insertions(+), 4 deletions(-)
diff --git a/xen/arch/arm/bootfdt.c b/xen/arch/arm/bootfdt.c
index ea188a0..6e8251b 100644
--- a/xen/arch/arm/bootfdt.c
+++ b/xen/arch/arm/bootfdt.c
@@ -31,8 +31,8 @@ static bool_t __init device_tree_node_matches(const void *fdt, int node,
&& (name[match_len] == '@' || name[match_len] == '\0');
}
-static bool_t __init device_tree_node_compatible(const void *fdt, int node,
- const char *match)
+bool_t __init device_tree_node_compatible(const void *fdt, int node,
+ const char *match)
{
int len, l;
int mlen;
@@ -62,8 +62,20 @@ static void __init device_tree_get_reg(const __be32 **cell, u32 address_cells,
*size = dt_next_cell(size_cells, cell);
}
-static u32 __init device_tree_get_u32(const void *fdt, int node,
- const char *prop_name, u32 dflt)
+bool_t __init device_tree_type_matches(const void *fdt, int node,
+ const char *match)
+{
+ const void *prop;
+
+ prop = fdt_getprop(fdt, node, "device_type", NULL);
+ if ( prop == NULL )
+ return 0;
+
+ return strcmp(prop, match) == 0 ? 1 : 0;
+}
+
+u32 __init device_tree_get_u32(const void *fdt, int node,
+ const char *prop_name, u32 dflt)
{
const struct fdt_property *prop;
diff --git a/xen/include/asm-arm/setup.h b/xen/include/asm-arm/setup.h
index 7ff2c34..fb78478 100644
--- a/xen/include/asm-arm/setup.h
+++ b/xen/include/asm-arm/setup.h
@@ -83,6 +83,11 @@ struct bootmodule *add_boot_module(bootmodule_kind kind,
struct bootmodule *boot_module_find_by_kind(bootmodule_kind kind);
const char * __init boot_module_kind_as_string(bootmodule_kind kind);
+u32 device_tree_get_u32(const void *fdt, int node, const char *prop_name,
+ u32 dflt);
+bool_t device_tree_type_matches(const void *fdt, int node, const char *match);
+bool_t device_tree_node_compatible(const void *fdt, int node,
+ const char *match);
#endif
/*
* Local variables:
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 12/24] ARM: NUMA: DT: Parse CPU NUMA information
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (10 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 11/24] ARM: fdt: Export and introduce new fdt functions vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-19 18:26 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 13/24] ARM: NUMA: DT: Parse memory " vijay.kilari
` (12 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Parse CPU node and fetch numa-node-id information.
For each node-id found, update nodemask_t mask.
Refer to Documentation/devicetree/bindings/numa.txt
in linux kernel.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: - Parse cpu nodes under path /cpus
- Move changes to bootfdt.c as separate patch
- Set numa_off on dt_numa_init() failure
---
xen/arch/arm/Makefile | 1 +
xen/arch/arm/numa/Makefile | 2 ++
xen/arch/arm/numa/dt_numa.c | 77 +++++++++++++++++++++++++++++++++++++++++++++
xen/arch/arm/numa/numa.c | 48 ++++++++++++++++++++++++++++
xen/arch/arm/setup.c | 4 +++
xen/include/asm-arm/numa.h | 10 +++++-
6 files changed, 141 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 49e1fb2..a89be66 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -3,6 +3,7 @@ subdir-$(CONFIG_ARM_64) += arm64
subdir-y += platforms
subdir-$(CONFIG_ARM_64) += efi
subdir-$(CONFIG_ACPI) += acpi
+subdir-$(CONFIG_NUMA) += numa
obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
obj-y += bootfdt.init.o
diff --git a/xen/arch/arm/numa/Makefile b/xen/arch/arm/numa/Makefile
new file mode 100644
index 0000000..3af3aff
--- /dev/null
+++ b/xen/arch/arm/numa/Makefile
@@ -0,0 +1,2 @@
+obj-y += dt_numa.o
+obj-y += numa.o
diff --git a/xen/arch/arm/numa/dt_numa.c b/xen/arch/arm/numa/dt_numa.c
new file mode 100644
index 0000000..963bb40
--- /dev/null
+++ b/xen/arch/arm/numa/dt_numa.c
@@ -0,0 +1,77 @@
+/*
+ * OF NUMA Parsing support.
+ *
+ * Copyright (C) 2015 - 2016 Cavium Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/mm.h>
+#include <xen/nodemask.h>
+#include <xen/libfdt/libfdt.h>
+#include <xen/device_tree.h>
+#include <xen/numa.h>
+#include <asm/setup.h>
+
+/*
+ * Even though we connect cpus to numa domains later in SMP
+ * init, we need to know the node ids now for all cpus.
+ */
+static int __init dt_numa_process_cpu_node(const void *fdt)
+{
+ int node, offset;
+ uint32_t nid;
+
+ offset = fdt_path_offset(fdt, "/cpus");
+ if ( offset < 0 )
+ return -EINVAL;
+
+ node = fdt_first_subnode(fdt, offset);
+ if ( node == -FDT_ERR_NOTFOUND )
+ return -EINVAL;
+
+ do {
+ if ( device_tree_type_matches(fdt, node, "cpu") )
+ {
+ nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
+ if ( nid >= MAX_NUMNODES )
+ printk(XENLOG_WARNING
+ "NUMA: Node id %u exceeds maximum value\n", nid);
+ else
+ node_set(nid, processor_nodes_parsed);
+ }
+
+ offset = node;
+ node = fdt_next_subnode(fdt, offset);
+ } while (node != -FDT_ERR_NOTFOUND);
+
+ return 0;
+}
+
+int __init dt_numa_init(void)
+{
+ int ret;
+
+ ret = dt_numa_process_cpu_node((void *)device_tree_flattened);
+
+ return ret;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
new file mode 100644
index 0000000..45cc418
--- /dev/null
+++ b/xen/arch/arm/numa/numa.c
@@ -0,0 +1,48 @@
+/*
+ * ARM NUMA Implementation
+ *
+ * Copyright (C) 2016 - Cavium Inc.
+ * Vijaya Kumar K <vijaya.kumar@cavium.com>
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms and conditions of the GNU General Public
+ * License, version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/init.h>
+#include <xen/ctype.h>
+#include <xen/nodemask.h>
+#include <xen/numa.h>
+
+void __init numa_init(void)
+{
+ int ret = 0;
+
+ nodes_clear(processor_nodes_parsed);
+ if ( numa_off )
+ goto no_numa;
+
+ ret = dt_numa_init();
+ if ( ret )
+ {
+ numa_off = true;
+ printk(XENLOG_WARNING "DT NUMA init failed\n");
+ }
+
+no_numa:
+ return;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index 3b34855..a6d1499 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -38,6 +38,7 @@
#include <xen/libfdt/libfdt.h>
#include <xen/acpi.h>
#include <asm/alternative.h>
+#include <xen/numa.h>
#include <asm/page.h>
#include <asm/current.h>
#include <asm/setup.h>
@@ -755,6 +756,9 @@ void __init start_xen(unsigned long boot_phys_offset,
/* Parse the ACPI tables for possible boot-time configuration */
acpi_boot_table_init();
+ /* numa_init parses acpi tables. So call after acpi init */
+ numa_init();
+
end_boot_allocator();
vm_init();
diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
index 7f00a36..8f517a2 100644
--- a/xen/include/asm-arm/numa.h
+++ b/xen/include/asm-arm/numa.h
@@ -3,7 +3,15 @@
typedef uint8_t nodeid_t;
-#ifndef CONFIG_NUMA
+#ifdef CONFIG_NUMA
+void numa_init(void);
+int dt_numa_init(void);
+#else
+static inline void numa_init(void)
+{
+ return;
+}
+
/* Fake one node for now. See also node_online_map. */
#define cpu_to_node(cpu) 0
#define node_to_cpumask(node) (cpu_online_map)
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 13/24] ARM: NUMA: DT: Parse memory NUMA information
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (11 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 12/24] ARM: NUMA: DT: Parse CPU NUMA information vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-19 18:39 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 14/24] ARM: NUMA: DT: Parse NUMA distance information vijay.kilari
` (11 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Parse memory node and fetch numa-node-id information.
For each memory range, store in node_memblk_range[]
along with node id.
When booting in UEFI mode, UEFI passes memory information
to Dom0 using EFI memory descriptor table and deletes the
memory nodes from the host DT. However to fetch the memory
numa node id, memory DT node should not be deleted by EFI stub.
With this patch, do not delete memory node from FDT.
NUMA info of memory is extracted from process_memory_node()
instead of parsing the DT again during numa_init().
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: - Set numa_off in numa_failed() and drop dt_numa variable
---
xen/arch/arm/bootfdt.c | 25 +++++++++++++++++++++----
xen/arch/arm/efi/efi-boot.h | 25 -------------------------
xen/arch/arm/numa/dt_numa.c | 32 ++++++++++++++++++++++++++++++++
xen/arch/arm/numa/numa.c | 5 +++++
xen/include/asm-arm/numa.h | 2 ++
5 files changed, 60 insertions(+), 29 deletions(-)
diff --git a/xen/arch/arm/bootfdt.c b/xen/arch/arm/bootfdt.c
index 6e8251b..b3a132c 100644
--- a/xen/arch/arm/bootfdt.c
+++ b/xen/arch/arm/bootfdt.c
@@ -13,6 +13,8 @@
#include <xen/init.h>
#include <xen/device_tree.h>
#include <xen/libfdt/libfdt.h>
+#include <xen/numa.h>
+#include <xen/efi.h>
#include <xsm/xsm.h>
#include <asm/setup.h>
@@ -146,6 +148,9 @@ static void __init process_memory_node(const void *fdt, int node,
const __be32 *cell;
paddr_t start, size;
u32 reg_cells = address_cells + size_cells;
+#ifdef CONFIG_NUMA
+ uint32_t nid;
+#endif
if ( address_cells < 1 || size_cells < 1 )
{
@@ -154,24 +159,36 @@ static void __init process_memory_node(const void *fdt, int node,
return;
}
+#ifdef CONFIG_NUMA
+ nid = device_tree_get_u32(fdt, node, "numa-node-id", NR_NODE_MEMBLKS);
+#endif
prop = fdt_get_property(fdt, node, "reg", NULL);
if ( !prop )
{
printk("fdt: node `%s': missing `reg' property\n", name);
+#ifdef CONFIG_NUMA
+ numa_failed();
+#endif
return;
}
cell = (const __be32 *)prop->data;
banks = fdt32_to_cpu(prop->len) / (reg_cells * sizeof (u32));
- for ( i = 0; i < banks && bootinfo.mem.nr_banks < NR_MEM_BANKS; i++ )
+ for ( i = 0; i < banks; i++ )
{
device_tree_get_reg(&cell, address_cells, size_cells, &start, &size);
if ( !size )
continue;
- bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
- bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
- bootinfo.mem.nr_banks++;
+ if ( !efi_enabled(EFI_BOOT) && bootinfo.mem.nr_banks < NR_MEM_BANKS )
+ {
+ bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
+ bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
+ bootinfo.mem.nr_banks++;
+ }
+#ifdef CONFIG_NUMA
+ dt_numa_process_memory_node(nid, start, size);
+#endif
}
}
diff --git a/xen/arch/arm/efi/efi-boot.h b/xen/arch/arm/efi/efi-boot.h
index 56de26e..a8bde68 100644
--- a/xen/arch/arm/efi/efi-boot.h
+++ b/xen/arch/arm/efi/efi-boot.h
@@ -194,33 +194,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,
int status;
u32 fdt_val32;
u64 fdt_val64;
- int prev;
int num_rsv;
- /*
- * Delete any memory nodes present. The EFI memory map is the only
- * memory description provided to Xen.
- */
- prev = 0;
- for (;;)
- {
- const char *type;
- int len;
-
- node = fdt_next_node(fdt, prev, NULL);
- if ( node < 0 )
- break;
-
- type = fdt_getprop(fdt, node, "device_type", &len);
- if ( type && strncmp(type, "memory", len) == 0 )
- {
- fdt_del_node(fdt, node);
- continue;
- }
-
- prev = node;
- }
-
/*
* Delete all memory reserve map entries. When booting via UEFI,
* kernel will use the UEFI memory map to find reserved regions.
diff --git a/xen/arch/arm/numa/dt_numa.c b/xen/arch/arm/numa/dt_numa.c
index 963bb40..84030e7 100644
--- a/xen/arch/arm/numa/dt_numa.c
+++ b/xen/arch/arm/numa/dt_numa.c
@@ -58,6 +58,38 @@ static int __init dt_numa_process_cpu_node(const void *fdt)
return 0;
}
+void __init dt_numa_process_memory_node(uint32_t nid, paddr_t start,
+ paddr_t size)
+{
+ struct node *nd;
+ int i;
+
+ i = conflicting_memblks(start, start + size);
+ if ( i < 0 )
+ {
+ if ( numa_add_memblk(nid, start, size) )
+ {
+ printk(XENLOG_WARNING "DT: NUMA: node-id %u overflow \n", nid);
+ numa_failed();
+ return;
+ }
+ }
+ else
+ {
+ nd = get_node_memblk_range(i);
+ printk(XENLOG_ERR
+ "NUMA DT: node %u (%"PRIx64"-%"PRIx64") overlaps with %d (%"PRIx64"-%"PRIx64")\n",
+ nid, start, start + size, i, nd->start, nd->end);
+
+ numa_failed();
+ return;
+ }
+
+ node_set(nid, memory_nodes_parsed);
+
+ return;
+}
+
int __init dt_numa_init(void)
{
int ret;
diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
index 45cc418..8227361 100644
--- a/xen/arch/arm/numa/numa.c
+++ b/xen/arch/arm/numa/numa.c
@@ -19,6 +19,11 @@
#include <xen/nodemask.h>
#include <xen/numa.h>
+void numa_failed(void)
+{
+ numa_off = true;
+}
+
void __init numa_init(void)
{
int ret = 0;
diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
index 8f517a2..36cd782 100644
--- a/xen/include/asm-arm/numa.h
+++ b/xen/include/asm-arm/numa.h
@@ -3,6 +3,8 @@
typedef uint8_t nodeid_t;
+void dt_numa_process_memory_node(uint32_t nid, paddr_t start, paddr_t size);
+
#ifdef CONFIG_NUMA
void numa_init(void);
int dt_numa_init(void);
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 14/24] ARM: NUMA: DT: Parse NUMA distance information
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (12 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 13/24] ARM: NUMA: DT: Parse memory " vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-20 13:02 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support vijay.kilari
` (10 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Parse distance-matrix and fetch node distance information.
Store distance information in node_distance[].
Register dt_node_distance() function pointer with
the ARM numa code. This approach can be later used for
ACPI.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: - Moved __node_distance() declaration to common
header file
- Use device_tree_node_compatible() instead of
device_tree_node_matches()
- Dropped xen/errno.h inclusion
---
xen/arch/arm/numa/dt_numa.c | 131 ++++++++++++++++++++++++++++++++++++++++++++
xen/arch/arm/numa/numa.c | 22 ++++++++
xen/include/asm-arm/numa.h | 2 +
xen/include/asm-x86/numa.h | 1 -
xen/include/xen/numa.h | 3 +
5 files changed, 158 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/numa/dt_numa.c b/xen/arch/arm/numa/dt_numa.c
index 84030e7..46c0346 100644
--- a/xen/arch/arm/numa/dt_numa.c
+++ b/xen/arch/arm/numa/dt_numa.c
@@ -23,6 +23,48 @@
#include <xen/numa.h>
#include <asm/setup.h>
+static uint8_t node_distance[MAX_NUMNODES][MAX_NUMNODES];
+
+static uint8_t dt_node_distance(nodeid_t nodea, nodeid_t nodeb)
+{
+ if ( nodea >= MAX_NUMNODES || nodeb >= MAX_NUMNODES )
+ return nodea == nodeb ? LOCAL_DISTANCE : REMOTE_DISTANCE;
+
+ return node_distance[nodea][nodeb];
+}
+
+static int dt_numa_set_distance(uint32_t nodea, uint32_t nodeb,
+ uint32_t distance)
+{
+ /* node_distance is uint8_t. Ensure distance is less than 255 */
+ if ( nodea >= MAX_NUMNODES || nodeb >= MAX_NUMNODES || distance > 255 )
+ return -EINVAL;
+
+ node_distance[nodea][nodeb] = distance;
+
+ return 0;
+}
+
+void init_dt_numa_distance(void)
+{
+ int i, j;
+
+ for ( i = 0; i < MAX_NUMNODES; i++ )
+ {
+ for ( j = 0; j < MAX_NUMNODES; j++ )
+ {
+ /*
+ * Initialize distance 10 for local distance and
+ * 20 for remote distance.
+ */
+ if ( i == j )
+ node_distance[i][j] = LOCAL_DISTANCE;
+ else
+ node_distance[i][j] = REMOTE_DISTANCE;
+ }
+ }
+}
+
/*
* Even though we connect cpus to numa domains later in SMP
* init, we need to know the node ids now for all cpus.
@@ -58,6 +100,76 @@ static int __init dt_numa_process_cpu_node(const void *fdt)
return 0;
}
+static int __init dt_numa_parse_distance_map(const void *fdt, int node,
+ const char *name,
+ uint32_t address_cells,
+ uint32_t size_cells)
+{
+ const struct fdt_property *prop;
+ const __be32 *matrix;
+ int entry_count, len, i;
+
+ printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
+
+ prop = fdt_get_property(fdt, node, "distance-matrix", &len);
+ if ( !prop )
+ {
+ printk(XENLOG_WARNING
+ "NUMA: No distance-matrix property in distance-map\n");
+
+ return -EINVAL;
+ }
+
+ if ( len % sizeof(uint32_t) != 0 )
+ {
+ printk(XENLOG_WARNING
+ "distance-matrix in node is not a multiple of u32\n");
+
+ return -EINVAL;
+ }
+
+ entry_count = len / sizeof(uint32_t);
+ if ( entry_count <= 0 )
+ {
+ printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
+
+ return -EINVAL;
+ }
+
+ matrix = (const __be32 *)prop->data;
+ for ( i = 0; i + 2 < entry_count; i += 3 )
+ {
+ uint32_t nodea, nodeb, distance;
+
+ nodea = dt_read_number(matrix, 1);
+ matrix++;
+ nodeb = dt_read_number(matrix, 1);
+ matrix++;
+ distance = dt_read_number(matrix, 1);
+ matrix++;
+
+ if ( dt_numa_set_distance(nodea, nodeb, distance) )
+ {
+ printk(XENLOG_WARNING
+ "NUMA: node-id out of range in distance matrix for [node%d -> node%d]\n",
+ nodea, nodeb);
+ return -EINVAL;
+
+ }
+ printk(XENLOG_INFO "NUMA: distance[node%d -> node%d] = %d\n",
+ nodea, nodeb, distance);
+
+ /*
+ * Set default distance of node B->A same as A->B.
+ * No need to check for return value of numa_set_distance.
+ */
+ if ( nodeb > nodea )
+ dt_numa_set_distance(nodeb, nodea, distance);
+ }
+
+ return 0;
+}
+
void __init dt_numa_process_memory_node(uint32_t nid, paddr_t start,
paddr_t size)
{
@@ -90,11 +202,30 @@ void __init dt_numa_process_memory_node(uint32_t nid, paddr_t start,
return;
}
+static int __init dt_numa_scan_distance_node(const void *fdt, int node,
+ const char *name, int depth,
+ uint32_t address_cells,
+ uint32_t size_cells, void *data)
+{
+ if ( device_tree_node_compatible(fdt, node, "numa-distance-map-v1") )
+ return dt_numa_parse_distance_map(fdt, node, name, address_cells,
+ size_cells);
+
+ return 0;
+}
+
int __init dt_numa_init(void)
{
int ret;
ret = dt_numa_process_cpu_node((void *)device_tree_flattened);
+ if ( ret )
+ return ret;
+
+ ret = device_tree_for_each_node((void *)device_tree_flattened,
+ dt_numa_scan_distance_node, NULL);
+ if ( !ret )
+ register_node_distance(&dt_node_distance);
return ret;
}
diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
index 8227361..c00b92c 100644
--- a/xen/arch/arm/numa/numa.c
+++ b/xen/arch/arm/numa/numa.c
@@ -18,10 +18,30 @@
#include <xen/ctype.h>
#include <xen/nodemask.h>
#include <xen/numa.h>
+#include <asm/acpi.h>
+
+static uint8_t (*node_distance_fn)(nodeid_t a, nodeid_t b);
void numa_failed(void)
{
numa_off = true;
+ init_dt_numa_distance();
+ node_distance_fn = NULL;
+}
+
+uint8_t __node_distance(nodeid_t a, nodeid_t b)
+{
+ if ( node_distance_fn != NULL);
+ return node_distance_fn(a, b);
+
+ return a == b ? LOCAL_DISTANCE : REMOTE_DISTANCE;
+}
+
+EXPORT_SYMBOL(__node_distance);
+
+void register_node_distance(uint8_t (fn)(nodeid_t a, nodeid_t b))
+{
+ node_distance_fn = fn;
}
void __init numa_init(void)
@@ -29,6 +49,8 @@ void __init numa_init(void)
int ret = 0;
nodes_clear(processor_nodes_parsed);
+ init_dt_numa_distance();
+
if ( numa_off )
goto no_numa;
diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
index 36cd782..d1dc83a 100644
--- a/xen/include/asm-arm/numa.h
+++ b/xen/include/asm-arm/numa.h
@@ -4,6 +4,8 @@
typedef uint8_t nodeid_t;
void dt_numa_process_memory_node(uint32_t nid, paddr_t start, paddr_t size);
+void register_node_distance(uint8_t (fn)(nodeid_t a, nodeid_t b));
+void init_dt_numa_distance(void);
#ifdef CONFIG_NUMA
void numa_init(void);
diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
index d8a0a44..ca0a2a6 100644
--- a/xen/include/asm-x86/numa.h
+++ b/xen/include/asm-x86/numa.h
@@ -18,7 +18,6 @@ extern nodeid_t apicid_to_node[];
extern void init_cpu_to_node(void);
void srat_parse_regions(paddr_t addr);
-extern uint8_t __node_distance(nodeid_t a, nodeid_t b);
unsigned int arch_get_dma_bitsize(void);
#endif
diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
index 110d5dc..10ef4c4 100644
--- a/xen/include/xen/numa.h
+++ b/xen/include/xen/numa.h
@@ -6,6 +6,8 @@
#include <asm/numa.h>
#define NUMA_NO_NODE 0xFF
+#define LOCAL_DISTANCE 10
+#define REMOTE_DISTANCE 20
#define NUMA_NO_DISTANCE 0xFF
#define MAX_NUMNODES NR_NODES
@@ -70,6 +72,7 @@ int numa_add_memblk(nodeid_t nodeid, paddr_t start, uint64_t size);
int get_num_node_memblks(void);
bool arch_sanitize_nodes_memory(void);
void numa_failed(void);
+uint8_t __node_distance(nodeid_t a, nodeid_t b);
#else
static inline void numa_add_cpu(int cpu) { }
static inline void numa_set_node(int cpu, nodeid_t node) { }
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (13 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 14/24] ARM: NUMA: DT: Parse NUMA distance information vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-24 11:24 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 16/24] ARM: NUMA: Add memory " vijay.kilari
` (9 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
For each cpu, update cpu_to_node[] with node id from
the numa-node-id DT property. Also, initialize cpu_to_node[]
with node 0.
Add macros to access cpu_to_node[] information.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: - Dropped numa_add_cpu declaration from asm-arm/numa.h
- Dropped stale declarations
- Call numa_add_cpu for cpu0
---
xen/arch/arm/numa/numa.c | 21 +++++++++++++++++++++
xen/arch/arm/setup.c | 2 ++
xen/arch/arm/smpboot.c | 25 ++++++++++++++++++++++++-
xen/include/asm-arm/numa.h | 7 +++++++
xen/include/asm-x86/numa.h | 1 -
xen/include/xen/numa.h | 1 +
6 files changed, 55 insertions(+), 2 deletions(-)
diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
index c00b92c..dc80aa5 100644
--- a/xen/arch/arm/numa/numa.c
+++ b/xen/arch/arm/numa/numa.c
@@ -22,11 +22,31 @@
static uint8_t (*node_distance_fn)(nodeid_t a, nodeid_t b);
+/*
+ * Setup early cpu_to_node.
+ */
+void __init init_cpu_to_node(void)
+{
+ int i;
+
+ for ( i = 0; i < NR_CPUS; i++ )
+ numa_set_node(i, 0);
+}
+
void numa_failed(void)
{
numa_off = true;
init_dt_numa_distance();
node_distance_fn = NULL;
+ init_cpu_to_node();
+}
+
+void __init numa_set_cpu_node(int cpu, unsigned int nid)
+{
+ if ( !node_isset(nid, processor_nodes_parsed) || nid >= MAX_NUMNODES )
+ nid = 0;
+
+ numa_set_node(cpu, nid);
}
uint8_t __node_distance(nodeid_t a, nodeid_t b)
@@ -49,6 +69,7 @@ void __init numa_init(void)
int ret = 0;
nodes_clear(processor_nodes_parsed);
+ init_cpu_to_node();
init_dt_numa_distance();
if ( numa_off )
diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
index a6d1499..b9c8b0d 100644
--- a/xen/arch/arm/setup.c
+++ b/xen/arch/arm/setup.c
@@ -787,6 +787,8 @@ void __init start_xen(unsigned long boot_phys_offset,
processor_id();
+ numa_add_cpu(0);
+
smp_init_cpus();
cpus = smp_get_max_cpus();
printk(XENLOG_INFO "SMP: Allowing %u CPUs\n", cpus);
diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
index 32e8722..fcf9afc 100644
--- a/xen/arch/arm/smpboot.c
+++ b/xen/arch/arm/smpboot.c
@@ -29,6 +29,7 @@
#include <xen/timer.h>
#include <xen/irq.h>
#include <xen/console.h>
+#include <xen/numa.h>
#include <asm/cpuerrata.h>
#include <asm/gic.h>
#include <asm/psci.h>
@@ -106,6 +107,7 @@ static void __init dt_smp_init_cpus(void)
[0 ... NR_CPUS - 1] = MPIDR_INVALID
};
bool_t bootcpu_valid = 0;
+ nodeid_t *cpu_to_nodemap;
int rc;
mpidr = boot_cpu_data.mpidr.bits & MPIDR_HWID_MASK;
@@ -117,11 +119,18 @@ static void __init dt_smp_init_cpus(void)
return;
}
+ cpu_to_nodemap = xzalloc_array(nodeid_t, NR_CPUS);
+ if ( !cpu_to_nodemap )
+ {
+ printk(XENLOG_WARNING "Failed to allocate memory for cpu_to_nodemap\n");
+ return;
+ }
+
dt_for_each_child_node( cpus, cpu )
{
const __be32 *prop;
u64 addr;
- u32 reg_len;
+ uint32_t reg_len, nid;
register_t hwid;
if ( !dt_device_type_is_equal(cpu, "cpu") )
@@ -146,6 +155,15 @@ static void __init dt_smp_init_cpus(void)
continue;
}
+ if ( !dt_property_read_u32(cpu, "numa-node-id", &nid) )
+ {
+ printk(XENLOG_WARNING "cpu node `%s`: numa-node-id not found\n",
+ dt_node_full_name(cpu));
+ nid = 0;
+ }
+
+ cpu_to_nodemap[cpuidx] = nid;
+
addr = dt_read_number(prop, dt_n_addr_cells(cpu));
hwid = addr;
@@ -224,6 +242,7 @@ static void __init dt_smp_init_cpus(void)
{
printk(XENLOG_WARNING "DT missing boot CPU MPIDR[23:0]\n"
"Using only 1 CPU\n");
+ xfree(cpu_to_nodemap);
return;
}
@@ -233,7 +252,10 @@ static void __init dt_smp_init_cpus(void)
continue;
cpumask_set_cpu(i, &cpu_possible_map);
cpu_logical_map(i) = tmp_map[i];
+ numa_set_cpu_node(i, cpu_to_nodemap[i]);
}
+
+ xfree(cpu_to_nodemap);
}
void __init smp_init_cpus(void)
@@ -313,6 +335,7 @@ void start_secondary(unsigned long boot_phys_offset,
*/
smp_wmb();
+ numa_add_cpu(cpuid);
/* Now report this CPU is up */
cpumask_set_cpu(cpuid, &cpu_online_map);
diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
index d1dc83a..0d3146c 100644
--- a/xen/include/asm-arm/numa.h
+++ b/xen/include/asm-arm/numa.h
@@ -10,12 +10,19 @@ void init_dt_numa_distance(void);
#ifdef CONFIG_NUMA
void numa_init(void);
int dt_numa_init(void);
+void numa_set_cpu_node(int cpu, unsigned int nid);
+
#else
static inline void numa_init(void)
{
return;
}
+static inline void numa_set_cpu_node(int cpu, unsigned int nid)
+{
+ return;
+}
+
/* Fake one node for now. See also node_online_map. */
#define cpu_to_node(cpu) 0
#define node_to_cpumask(node) (cpu_online_map)
diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
index ca0a2a6..fc4747f 100644
--- a/xen/include/asm-x86/numa.h
+++ b/xen/include/asm-x86/numa.h
@@ -15,7 +15,6 @@ extern nodeid_t acpi_setup_node(unsigned int pxm);
extern void srat_detect_node(int cpu);
extern nodeid_t apicid_to_node[];
-extern void init_cpu_to_node(void);
void srat_parse_regions(paddr_t addr);
unsigned int arch_get_dma_bitsize(void);
diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
index 10ef4c4..8a306e7 100644
--- a/xen/include/xen/numa.h
+++ b/xen/include/xen/numa.h
@@ -30,6 +30,7 @@ extern s8 acpi_numa;
void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
int srat_disabled(void);
int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
+void init_cpu_to_node(void);
#ifdef CONFIG_NUMA
#define cpu_to_node(cpu) (cpu_to_node[cpu])
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 16/24] ARM: NUMA: Add memory NUMA support
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (14 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-24 12:43 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 17/24] ARM: NUMA: DT: Do not expose numa info to DOM0 vijay.kilari
` (8 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Implement arch_sanitize_nodes_memory() which looks at all banks
in bootinfo.mem, update nodes[] with corresponding nodeid.
Call numa_scan_nodes() generic function with ram start and
end address, which takes care of further computing memnodeshift
and populating memnodemap[] using generic implementation.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: - Dropped common code from asm-arm/numa.h
- Re-used numa_initmem_init() from common code.
---
xen/arch/arm/numa/numa.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++-
xen/common/numa.c | 14 +++++++++
xen/include/xen/numa.h | 1 +
3 files changed, 91 insertions(+), 1 deletion(-)
diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
index dc80aa5..85352dc 100644
--- a/xen/arch/arm/numa/numa.c
+++ b/xen/arch/arm/numa/numa.c
@@ -18,6 +18,7 @@
#include <xen/ctype.h>
#include <xen/nodemask.h>
#include <xen/numa.h>
+#include <xen/pfn.h>
#include <asm/acpi.h>
static uint8_t (*node_distance_fn)(nodeid_t a, nodeid_t b);
@@ -64,9 +65,66 @@ void register_node_distance(uint8_t (fn)(nodeid_t a, nodeid_t b))
node_distance_fn = fn;
}
+bool __init arch_sanitize_nodes_memory(void)
+{
+ nodemask_t mem_nodes_parsed;
+ int bank, nodeid;
+ struct node *nd;
+ paddr_t start, size, end;
+
+ nodes_clear(mem_nodes_parsed);
+ for ( bank = 0 ; bank < bootinfo.mem.nr_banks; bank++ )
+ {
+ start = bootinfo.mem.bank[bank].start;
+ size = bootinfo.mem.bank[bank].size;
+ end = start + size;
+
+ nodeid = get_mem_nodeid(start, end);
+ if ( nodeid >= NUMA_NO_NODE )
+ {
+ printk(XENLOG_WARNING
+ "NUMA: node for mem bank start 0x%lx - 0x%lx not found\n",
+ start, end);
+
+ return false;
+ }
+
+ nd = get_numa_node(nodeid);
+ if ( !node_test_and_set(nodeid, mem_nodes_parsed) )
+ {
+ nd->start = start;
+ nd->end = end;
+ }
+ else
+ {
+ if ( start < nd->start )
+ nd->start = start;
+ if ( nd->end < end )
+ nd->end = end;
+ }
+ }
+
+ return true;
+}
+
+static void __init numa_reset_numa_nodes(void)
+{
+ int i;
+ struct node *nd;
+
+ for ( i = 0; i < MAX_NUMNODES; i++ )
+ {
+ nd = get_numa_node(i);
+ nd->start = 0;
+ nd->end = 0;
+ }
+}
+
void __init numa_init(void)
{
- int ret = 0;
+ int ret = 0, bank;
+ paddr_t ram_start = ~0;
+ paddr_t ram_end = 0;
nodes_clear(processor_nodes_parsed);
init_cpu_to_node();
@@ -83,6 +141,23 @@ void __init numa_init(void)
}
no_numa:
+ for ( bank = 0 ; bank < bootinfo.mem.nr_banks; bank++ )
+ {
+ paddr_t bank_start = bootinfo.mem.bank[bank].start;
+ paddr_t bank_end = bank_start + bootinfo.mem.bank[bank].size;
+
+ ram_start = min(ram_start, bank_start);
+ ram_end = max(ram_end, bank_end);
+ }
+
+ /*
+ * In arch_sanitize_nodes_memory() we update nodes[] properly.
+ * Hence we reset the nodes[] before calling numa_scan_nodes().
+ */
+ numa_reset_numa_nodes();
+
+ numa_initmem_init(PFN_UP(ram_start), PFN_DOWN(ram_end));
+
return;
}
diff --git a/xen/common/numa.c b/xen/common/numa.c
index 5e985d2..0f79a07 100644
--- a/xen/common/numa.c
+++ b/xen/common/numa.c
@@ -76,6 +76,20 @@ nodeid_t get_memblk_nodeid(unsigned int id)
return memblk_nodeid[id];
}
+int __init get_mem_nodeid(paddr_t start, paddr_t end)
+{
+ unsigned int i;
+
+ for ( i = 0; i < get_num_node_memblks(); i++ )
+ {
+ if ( start >= node_memblk_range[i].start &&
+ end <= node_memblk_range[i].end )
+ return memblk_nodeid[i];
+ }
+
+ return -EINVAL;
+}
+
static nodeid_t *get_memblk_nodeid_map(void)
{
return &memblk_nodeid[0];
diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
index 8a306e7..a541eb7 100644
--- a/xen/include/xen/numa.h
+++ b/xen/include/xen/numa.h
@@ -70,6 +70,7 @@ struct node *get_numa_node(unsigned int id);
nodeid_t get_memblk_nodeid(unsigned int memblk);
struct node *get_node_memblk_range(unsigned int memblk);
int numa_add_memblk(nodeid_t nodeid, paddr_t start, uint64_t size);
+int get_mem_nodeid(paddr_t start, paddr_t end);
int get_num_node_memblks(void);
bool arch_sanitize_nodes_memory(void);
void numa_failed(void);
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 17/24] ARM: NUMA: DT: Do not expose numa info to DOM0
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (15 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 16/24] ARM: NUMA: Add memory " vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-24 20:48 ` Stefano Stabellini
2017-07-26 17:22 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 18/24] ACPI: Refactor acpi SRAT and SLIT table handling code vijay.kilari
` (7 subsequent siblings)
24 siblings, 2 replies; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Delete numa-node-id and distance map from DOM0 DT
so that NUMA information is not exposed to DOM0.
This helps particularly to boot Node 1 devices
as if booting on Node0.
However this approach has limitation where memory allocation
for the devices should be local.
Also, do not expose numa distance node to DOM0.
Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
---
xen/arch/arm/domain_build.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 1bec4fa..a7d6d3a 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -425,6 +425,10 @@ static int write_properties(struct domain *d, struct kernel_info *kinfo,
}
}
+ /* Don't expose the property numa to the guest */
+ if ( dt_property_name_is_equal(prop, "numa-node-id") )
+ continue;
+
/* Don't expose the property "xen,passthrough" to the guest */
if ( dt_property_name_is_equal(prop, "xen,passthrough") )
continue;
@@ -1177,6 +1181,11 @@ static int handle_node(struct domain *d, struct kernel_info *kinfo,
DT_MATCH_TYPE("memory"),
/* The memory mapped timer is not supported by Xen. */
DT_MATCH_COMPATIBLE("arm,armv7-timer-mem"),
+ /*
+ * NUMA info is not exposed to Dom0.
+ * So, skip distance-map infomation
+ */
+ DT_MATCH_COMPATIBLE("numa-distance-map-v1"),
{ /* sentinel */ },
};
static const struct dt_device_match timer_matches[] __initconst =
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 18/24] ACPI: Refactor acpi SRAT and SLIT table handling code
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (16 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 17/24] ARM: NUMA: DT: Do not expose numa info to DOM0 vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-18 15:36 ` Wei Liu
2017-07-18 11:41 ` [RFC PATCH v3 19/24] ARM: NUMA: Extract MPIDR from MADT table vijay.kilari
` (6 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Move SRAT handling code which is common across
architectures is moved to new file xen/drivers/acpi/srat.c
from xen/arch/x86/srat.c file. New header file srat.h is
introduced.
Other major changes are:
- Coding style of code moved is changed.
- Moved struct pxm2node from srat.c to srat.h
- Dropped {memory,processor}_nodes_parsed from x86/srat.c
- Dropped static on node_to_pxm() and moved to beginning of the file.
- Made some static functions as non-static
- acpi_node_distance() is introduced and called from __node_distance()
- Replaced distance constants with LOCAL/REMOTE_DISTANCE defines
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
v3: - Moved common function declarations from asm-x86/srat.h
---
xen/arch/x86/dom0_build.c | 1 +
xen/arch/x86/mm.c | 2 -
xen/arch/x86/physdev.c | 1 +
xen/arch/x86/setup.c | 1 +
xen/arch/x86/smpboot.c | 1 +
xen/arch/x86/srat.c | 246 +----------------------------
xen/arch/x86/x86_64/mm.c | 1 +
xen/drivers/acpi/Makefile | 1 +
xen/drivers/acpi/srat.c | 298 ++++++++++++++++++++++++++++++++++++
xen/drivers/passthrough/vtd/iommu.c | 1 +
xen/include/acpi/srat.h | 24 +++
xen/include/asm-x86/numa.h | 5 -
12 files changed, 331 insertions(+), 251 deletions(-)
diff --git a/xen/arch/x86/dom0_build.c b/xen/arch/x86/dom0_build.c
index 0c125e6..04127e7 100644
--- a/xen/arch/x86/dom0_build.c
+++ b/xen/arch/x86/dom0_build.c
@@ -11,6 +11,7 @@
#include <xen/sched.h>
#include <xen/sched-if.h>
#include <xen/softirq.h>
+#include <acpi/srat.h>
#include <asm/dom0_build.h>
#include <asm/hpet.h>
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 19f672d..5497621 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -135,8 +135,6 @@ l1_pgentry_t __section(".bss.page_aligned") __aligned(PAGE_SIZE)
#define PTE_UPDATE_WITH_CMPXCHG
#endif
-paddr_t __read_mostly mem_hotplug;
-
/* Private domain structs for DOMID_XEN and DOMID_IO. */
struct domain *dom_xen, *dom_io, *dom_cow;
diff --git a/xen/arch/x86/physdev.c b/xen/arch/x86/physdev.c
index 0eb4097..a73a954 100644
--- a/xen/arch/x86/physdev.c
+++ b/xen/arch/x86/physdev.c
@@ -8,6 +8,7 @@
#include <xen/guest_access.h>
#include <xen/iocap.h>
#include <xen/serial.h>
+#include <acpi/srat.h>
#include <asm/current.h>
#include <asm/io_apic.h>
#include <asm/msi.h>
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index db5df69..b957b96 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -27,6 +27,7 @@
#include <xen/tmem_xen.h>
#include <xen/virtual_region.h>
#include <xen/watchdog.h>
+#include <acpi/srat.h>
#include <public/version.h>
#include <compat/platform.h>
#include <compat/xen.h>
diff --git a/xen/arch/x86/smpboot.c b/xen/arch/x86/smpboot.c
index 168c9d4..ff4c7e1 100644
--- a/xen/arch/x86/smpboot.c
+++ b/xen/arch/x86/smpboot.c
@@ -33,6 +33,7 @@
#include <xen/serial.h>
#include <xen/numa.h>
#include <xen/cpu.h>
+#include <acpi/srat.h>
#include <asm/current.h>
#include <asm/mc146818rtc.h>
#include <asm/desc.h>
diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
index be2634a..d5caccf 100644
--- a/xen/arch/x86/srat.c
+++ b/xen/arch/x86/srat.c
@@ -18,92 +18,10 @@
#include <xen/acpi.h>
#include <xen/numa.h>
#include <xen/pfn.h>
+#include <acpi/srat.h>
#include <asm/e820.h>
#include <asm/page.h>
-static struct acpi_table_slit *__read_mostly acpi_slit;
-
-struct pxm2node {
- unsigned int pxm;
- nodeid_t node;
-};
-static struct pxm2node __read_mostly pxm2node[MAX_NUMNODES] =
- { [0 ... MAX_NUMNODES - 1] = {.node = NUMA_NO_NODE} };
-
-static unsigned int node_to_pxm(nodeid_t n);
-
-static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
-
-static inline bool node_found(unsigned int idx, unsigned int pxm)
-{
- return ((pxm2node[idx].pxm == pxm) &&
- (pxm2node[idx].node != NUMA_NO_NODE));
-}
-
-static void reset_pxm2node(void)
-{
- unsigned int i;
-
- for (i = 0; i < ARRAY_SIZE(pxm2node); i++)
- pxm2node[i].node = NUMA_NO_NODE;
-}
-
-nodeid_t pxm_to_node(unsigned int pxm)
-{
- unsigned int i;
-
- if ((pxm < ARRAY_SIZE(pxm2node)) && node_found(pxm, pxm))
- return pxm2node[pxm].node;
-
- for (i = 0; i < ARRAY_SIZE(pxm2node); i++)
- if (node_found(i, pxm))
- return pxm2node[i].node;
-
- return NUMA_NO_NODE;
-}
-
-nodeid_t acpi_setup_node(unsigned int pxm)
-{
- nodeid_t node;
- unsigned int idx;
- static bool warned;
- static unsigned int nodes_found;
-
- BUILD_BUG_ON(MAX_NUMNODES >= NUMA_NO_NODE);
-
- if (pxm < ARRAY_SIZE(pxm2node)) {
- if (node_found(pxm, pxm))
- return pxm2node[pxm].node;
-
- /* Try to maintain indexing of pxm2node by pxm */
- if (pxm2node[pxm].node == NUMA_NO_NODE) {
- idx = pxm;
- goto finish;
- }
- }
-
- for (idx = 0; idx < ARRAY_SIZE(pxm2node); idx++)
- if (pxm2node[idx].node == NUMA_NO_NODE)
- goto finish;
-
- if (!warned) {
- printk(KERN_WARNING "SRAT: Too many proximity domains (%#x)\n",
- pxm);
- warned = true;
- }
-
- return NUMA_NO_NODE;
-
- finish:
- node = nodes_found++;
- if (node >= MAX_NUMNODES)
- return NUMA_NO_NODE;
- pxm2node[idx].pxm = pxm;
- pxm2node[idx].node = node;
-
- return node;
-}
-
void __init numa_failed(void)
{
int i;
@@ -115,48 +33,6 @@ void __init numa_failed(void)
mem_hotplug = 0;
}
-/*
- * A lot of BIOS fill in 10 (= no distance) everywhere. This messes
- * up the NUMA heuristics which wants the local node to have a smaller
- * distance than the others.
- * Do some quick checks here and only use the SLIT if it passes.
- */
-static int __init slit_valid(struct acpi_table_slit *slit)
-{
- int i, j;
- int d = slit->locality_count;
- for (i = 0; i < d; i++) {
- for (j = 0; j < d; j++) {
- uint8_t val = slit->entry[d*i + j];
- if (i == j) {
- if (val != 10)
- return 0;
- } else if (val <= 10)
- return 0;
- }
- }
- return 1;
-}
-
-/* Callback for SLIT parsing */
-void __init acpi_numa_slit_init(struct acpi_table_slit *slit)
-{
- unsigned long mfn;
- if (!slit_valid(slit)) {
- printk(KERN_INFO "ACPI: SLIT table looks invalid. "
- "Not used.\n");
- return;
- }
- mfn = alloc_boot_pages(PFN_UP(slit->header.length), 1);
- if (!mfn) {
- printk(KERN_ERR "ACPI: Unable to allocate memory for "
- "saving ACPI SLIT numa information.\n");
- return;
- }
- acpi_slit = mfn_to_virt(mfn);
- memcpy(acpi_slit, slit, slit->header.length);
-}
-
/* Callback for Proximity Domain -> x2APIC mapping */
void __init
acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *pa)
@@ -224,100 +100,6 @@ acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *pa)
pxm, pa->apic_id, node);
}
-/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
-void __init
-acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
-{
- paddr_t start, end;
- unsigned int pxm;
- nodeid_t node;
- int i;
- struct node *memblk;
-
- if (srat_disabled())
- return;
- if (ma->header.length != sizeof(struct acpi_srat_mem_affinity)) {
- numa_failed();
- return;
- }
- if (!(ma->flags & ACPI_SRAT_MEM_ENABLED))
- return;
-
- if (get_num_node_memblks() >= NR_NODE_MEMBLKS)
- {
- dprintk(XENLOG_WARNING,
- "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
- numa_failed();
- return;
- }
-
- start = ma->base_address;
- end = start + ma->length;
- pxm = ma->proximity_domain;
- if (srat_rev < 2)
- pxm &= 0xff;
- node = acpi_setup_node(pxm);
- if (node == NUMA_NO_NODE) {
- numa_failed();
- return;
- }
- /* It is fine to add this area to the nodes data it will be used later*/
- i = conflicting_memblks(start, end);
- if (i < 0)
- /* everything fine */;
- else if (get_memblk_nodeid(i) == node) {
- bool mismatch = !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) !=
- !test_bit(i, memblk_hotplug);
-
- memblk = get_node_memblk_range(i);
-
- printk("%sSRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
- mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
- memblk->start, memblk->end);
- if (mismatch) {
- numa_failed();
- return;
- }
- } else {
- memblk = get_node_memblk_range(i);
-
- printk(KERN_ERR
- "SRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with PXM %u (%"PRIx64"-%"PRIx64")\n",
- pxm, start, end, node_to_pxm(get_memblk_nodeid(i)),
- memblk->start, memblk->end);
- numa_failed();
- return;
- }
- if (!(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE)) {
- struct node *nd = get_numa_node(node);
-
- if (!node_test_and_set(node, memory_nodes_parsed)) {
- nd->start = start;
- nd->end = end;
- } else {
- if (start < nd->start)
- nd->start = start;
- if (nd->end < end)
- nd->end = end;
- }
- }
- printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIx64"-%"PRIx64"%s\n",
- node, pxm, start, end,
- ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE ? " (hotplug)" : "");
-
- if (numa_add_memblk(node, start, ma->length)) {
- printk(KERN_ERR "SRAT: node-id %u out of range\n", node);
- numa_failed();
- return;
- }
-
- if (ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) {
- __set_bit(get_num_node_memblks(), memblk_hotplug);
- if (end > mem_hotplug)
- mem_hotplug = end;
- }
-}
-
/* Sanity check to catch more bad SRATs (they are amazingly common).
Make sure the PXMs cover all memory. */
bool __init arch_sanitize_nodes_memory(void)
@@ -417,33 +199,9 @@ void __init srat_parse_regions(paddr_t addr)
pfn_pdx_hole_setup(mask >> PAGE_SHIFT);
}
-static unsigned int node_to_pxm(nodeid_t n)
-{
- unsigned int i;
-
- if ((n < ARRAY_SIZE(pxm2node)) && (pxm2node[n].node == n))
- return pxm2node[n].pxm;
- for (i = 0; i < ARRAY_SIZE(pxm2node); i++)
- if (pxm2node[i].node == n)
- return pxm2node[i].pxm;
- return 0;
-}
-
uint8_t __node_distance(nodeid_t a, nodeid_t b)
{
- unsigned int index;
- uint8_t slit_val;
-
- if (!acpi_slit)
- return a == b ? 10 : 20;
- index = acpi_slit->locality_count * node_to_pxm(a);
- slit_val = acpi_slit->entry[index + node_to_pxm(b)];
-
- /* ACPI defines 0xff as an unreachable node and 0-9 are undefined */
- if ((slit_val == 0xff) || (slit_val <= 9))
- return NUMA_NO_DISTANCE;
- else
- return slit_val;
+ return acpi_node_distance(a, b);
}
EXPORT_SYMBOL(__node_distance);
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index a4ffa1f..b6e8eaf 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -27,6 +27,7 @@ asm(".file \"" __FILE__ "\"");
#include <xen/guest_access.h>
#include <xen/hypercall.h>
#include <xen/mem_access.h>
+#include <acpi/srat.h>
#include <asm/current.h>
#include <asm/asm_defns.h>
#include <asm/page.h>
diff --git a/xen/drivers/acpi/Makefile b/xen/drivers/acpi/Makefile
index 444b11d..69edc26 100644
--- a/xen/drivers/acpi/Makefile
+++ b/xen/drivers/acpi/Makefile
@@ -4,6 +4,7 @@ subdir-$(CONFIG_X86) += apei
obj-bin-y += tables.init.o
obj-$(CONFIG_NUMA) += numa.o
+obj-$(CONFIG_NUMA) += srat.o
obj-y += osl.o
obj-$(CONFIG_HAS_CPUFREQ) += pmstat.o
diff --git a/xen/drivers/acpi/srat.c b/xen/drivers/acpi/srat.c
new file mode 100644
index 0000000..19b1647
--- /dev/null
+++ b/xen/drivers/acpi/srat.c
@@ -0,0 +1,298 @@
+/*
+ * ACPI 3.0 based NUMA setup
+ * Copyright 2004 Andi Kleen, SuSE Labs.
+ *
+ * Reads the ACPI SRAT table to figure out what memory belongs to which CPUs.
+ *
+ * Called from acpi_numa_init while reading the SRAT and SLIT tables.
+ * Assumes all memory regions belonging to a single proximity domain
+ * are in one chunk. Holes between them will be included in the node.
+ *
+ * Adapted for Xen: Ryan Harper <ryanh@us.ibm.com>
+ */
+
+#include <xen/init.h>
+#include <xen/mm.h>
+#include <xen/inttypes.h>
+#include <xen/nodemask.h>
+#include <xen/acpi.h>
+#include <xen/numa.h>
+#include <xen/pfn.h>
+#include <acpi/srat.h>
+#include <asm/page.h>
+#include <asm/acpi.h>
+
+paddr_t __read_mostly mem_hotplug;
+static struct acpi_table_slit *__read_mostly acpi_slit;
+static struct pxm2node __read_mostly pxm2node[MAX_NUMNODES] =
+ { [0 ... MAX_NUMNODES - 1] = {.node = NUMA_NO_NODE} };
+
+static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
+
+static inline bool node_found(unsigned int idx, unsigned int pxm)
+{
+ return ((pxm2node[idx].pxm == pxm) &&
+ (pxm2node[idx].node != NUMA_NO_NODE));
+}
+
+void reset_pxm2node(void)
+{
+ unsigned int i;
+
+ for ( i = 0; i < ARRAY_SIZE(pxm2node); i++ )
+ pxm2node[i].node = NUMA_NO_NODE;
+}
+
+unsigned int node_to_pxm(nodeid_t n)
+{
+ unsigned int i;
+
+ if ( (n < ARRAY_SIZE(pxm2node)) && (pxm2node[n].node == n) )
+ return pxm2node[n].pxm;
+
+ for ( i = 0; i < ARRAY_SIZE(pxm2node); i++ )
+ if ( pxm2node[i].node == n )
+ return pxm2node[i].pxm;
+
+ return 0;
+}
+
+nodeid_t pxm_to_node(unsigned int pxm)
+{
+ unsigned int i;
+
+ if ( (pxm < ARRAY_SIZE(pxm2node)) && node_found(pxm, pxm) )
+ return pxm2node[pxm].node;
+
+ for ( i = 0; i < ARRAY_SIZE(pxm2node); i++ )
+ if ( node_found(i, pxm) )
+ return pxm2node[i].node;
+
+ return NUMA_NO_NODE;
+}
+
+nodeid_t acpi_setup_node(unsigned int pxm)
+{
+ nodeid_t node;
+ unsigned int idx;
+ static bool warned;
+ static unsigned int nodes_found;
+
+ BUILD_BUG_ON(MAX_NUMNODES >= NUMA_NO_NODE);
+
+ if ( pxm < ARRAY_SIZE(pxm2node) )
+ {
+ if ( node_found(pxm, pxm) )
+ return pxm2node[pxm].node;
+
+ /* Try to maintain indexing of pxm2node by pxm */
+ if ( pxm2node[pxm].node == NUMA_NO_NODE )
+ {
+ idx = pxm;
+ goto finish;
+ }
+ }
+
+ for ( idx = 0; idx < ARRAY_SIZE(pxm2node); idx++ )
+ if ( pxm2node[idx].node == NUMA_NO_NODE )
+ goto finish;
+
+ if ( !warned )
+ {
+ printk(KERN_WARNING "SRAT: Too many proximity domains (%#x)\n", pxm);
+ warned = true;
+ }
+
+ return NUMA_NO_NODE;
+
+ finish:
+ node = nodes_found++;
+ if ( node >= MAX_NUMNODES )
+ return NUMA_NO_NODE;
+ pxm2node[idx].pxm = pxm;
+ pxm2node[idx].node = node;
+
+ return node;
+}
+
+/*
+ * A lot of BIOS fill in 10 (= no distance) everywhere. This messes
+ * up the NUMA heuristics which wants the local node to have a smaller
+ * distance than the others.
+ * Do some quick checks here and only use the SLIT if it passes.
+ */
+static int __init slit_valid(struct acpi_table_slit *slit)
+{
+ int i, j;
+ int d = slit->locality_count;
+
+ for ( i = 0; i < d; i++ )
+ {
+ for ( j = 0; j < d; j++ )
+ {
+ uint8_t val = slit->entry[d*i + j];
+
+ if ( i == j )
+ {
+ if ( val != LOCAL_DISTANCE )
+ return 0;
+ } else if ( val <= LOCAL_DISTANCE )
+ return 0;
+ }
+ }
+
+ return 1;
+}
+
+/* Callback for SLIT parsing */
+void __init acpi_numa_slit_init(struct acpi_table_slit *slit)
+{
+ unsigned long mfn;
+
+ if ( !slit_valid(slit) )
+ {
+ printk(KERN_INFO "ACPI: SLIT table looks invalid. Not used.\n");
+ return;
+ }
+ mfn = alloc_boot_pages(PFN_UP(slit->header.length), 1);
+ if ( !mfn )
+ {
+ printk(KERN_ERR "ACPI: Unable to allocate memory for "
+ "saving ACPI SLIT numa information.\n");
+ return;
+ }
+ acpi_slit = mfn_to_virt(mfn);
+ memcpy(acpi_slit, slit, slit->header.length);
+}
+
+/* Callback for parsing of the Proximity Domain <-> Memory Area mappings */
+void __init
+acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *ma)
+{
+ paddr_t start, end;
+ unsigned int pxm;
+ nodeid_t node;
+ int i;
+ struct node *memblk;
+
+ if ( srat_disabled() )
+ return;
+ if ( ma->header.length != sizeof(struct acpi_srat_mem_affinity) )
+ {
+ numa_failed();
+ return;
+ }
+ if ( !(ma->flags & ACPI_SRAT_MEM_ENABLED) )
+ return;
+
+ if ( get_num_node_memblks() >= NR_NODE_MEMBLKS )
+ {
+ dprintk(XENLOG_WARNING,
+ "Too many numa entry, try bigger NR_NODE_MEMBLKS \n");
+ numa_failed();
+ return;
+ }
+
+ start = ma->base_address;
+ end = start + ma->length;
+ pxm = ma->proximity_domain;
+ if ( srat_rev < 2 )
+ pxm &= 0xff;
+ node = acpi_setup_node(pxm);
+ if ( node == NUMA_NO_NODE )
+ {
+ numa_failed();
+ return;
+ }
+ /* It is fine to add this area to the nodes data it will be used later*/
+ i = conflicting_memblks(start, end);
+ if ( i < 0 )
+ /* everything fine */;
+ else if ( get_memblk_nodeid(i) == node )
+ {
+ bool mismatch = !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) !=
+ !test_bit(i, memblk_hotplug);
+
+ memblk = get_node_memblk_range(i);
+
+ printk("%sSRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with itself (%"PRIx64"-%"PRIx64")\n",
+ mismatch ? KERN_ERR : KERN_WARNING, pxm, start, end,
+ memblk->start, memblk->end);
+ if ( mismatch )
+ {
+ numa_failed();
+ return;
+ }
+ }
+ else
+ {
+ memblk = get_node_memblk_range(i);
+
+ printk(KERN_ERR
+ "SRAT: PXM %u (%"PRIx64"-%"PRIx64") overlaps with PXM %u (%"PRIx64"-%"PRIx64")\n",
+ pxm, start, end, node_to_pxm(get_memblk_nodeid(i)),
+ memblk->start, memblk->end);
+ numa_failed();
+ return;
+ }
+ if ( !(ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE) )
+ {
+ struct node *nd = get_numa_node(node);
+
+ if ( !node_test_and_set(node, memory_nodes_parsed) )
+ {
+ nd->start = start;
+ nd->end = end;
+ }
+ else
+ {
+ if ( start < nd->start )
+ nd->start = start;
+ if ( nd->end < end )
+ nd->end = end;
+ }
+ }
+ printk(KERN_INFO "SRAT: Node %u PXM %u %"PRIx64"-%"PRIx64"%s\n",
+ node, pxm, start, end,
+ ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE ? " (hotplug)" : "");
+
+ if ( numa_add_memblk(node, start, ma->length) )
+ {
+ printk(KERN_ERR "SRAT: node-id %u out of range\n", node);
+ numa_failed();
+ return;
+ }
+
+ if ( ma->flags & ACPI_SRAT_MEM_HOT_PLUGGABLE )
+ {
+ __set_bit(get_num_node_memblks(), memblk_hotplug);
+ if ( end > mem_hotplug )
+ mem_hotplug = end;
+ }
+}
+
+uint8_t acpi_node_distance(nodeid_t a, nodeid_t b)
+{
+ unsigned int index;
+ uint8_t slit_val;
+
+ if ( !acpi_slit )
+ return a == b ? LOCAL_DISTANCE : REMOTE_DISTANCE;
+ index = acpi_slit->locality_count * node_to_pxm(a);
+ slit_val = acpi_slit->entry[index + node_to_pxm(b)];
+
+ /* ACPI defines 0xff as an unreachable node and 0-9 are undefined */
+ if ( (slit_val == 0xff) || (slit_val <= 9) )
+ return NUMA_NO_DISTANCE;
+ else
+ return slit_val;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/drivers/passthrough/vtd/iommu.c b/xen/drivers/passthrough/vtd/iommu.c
index 19328f6..27317dd 100644
--- a/xen/drivers/passthrough/vtd/iommu.c
+++ b/xen/drivers/passthrough/vtd/iommu.c
@@ -30,6 +30,7 @@
#include <xen/pci.h>
#include <xen/pci_regs.h>
#include <xen/keyhandler.h>
+#include <acpi/srat.h>
#include <asm/msi.h>
#include <asm/irq.h>
#include <asm/hvm/vmx/vmx.h>
diff --git a/xen/include/acpi/srat.h b/xen/include/acpi/srat.h
new file mode 100644
index 0000000..f19b822
--- /dev/null
+++ b/xen/include/acpi/srat.h
@@ -0,0 +1,24 @@
+#ifndef __XEN_SRAT_H__
+#define __XEN_SRAT_H__
+
+extern int srat_rev;
+struct pxm2node {
+ unsigned int pxm;
+ nodeid_t node;
+};
+
+nodeid_t pxm_to_node(unsigned pxm);
+nodeid_t acpi_setup_node(unsigned pxm);
+unsigned int node_to_pxm(nodeid_t n);
+uint8_t acpi_node_distance(nodeid_t a, nodeid_t b);
+void reset_pxm2node(void);
+#endif /* __XEN_SRAT_H__ */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
index fc4747f..4d266c2 100644
--- a/xen/include/asm-x86/numa.h
+++ b/xen/include/asm-x86/numa.h
@@ -5,13 +5,8 @@
typedef uint8_t nodeid_t;
-extern int srat_rev;
-
-extern nodeid_t pxm_to_node(unsigned int pxm);
-
#define ZONE_ALIGN (1UL << (MAX_ORDER+PAGE_SHIFT))
-extern nodeid_t acpi_setup_node(unsigned int pxm);
extern void srat_detect_node(int cpu);
extern nodeid_t apicid_to_node[];
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 19/24] ARM: NUMA: Extract MPIDR from MADT table
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (17 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 18/24] ACPI: Refactor acpi SRAT and SLIT table handling code vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-24 22:17 ` Stefano Stabellini
2017-07-26 18:12 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 20/24] ACPI: Move arch specific SRAT parsing vijay.kilari
` (5 subsequent siblings)
24 siblings, 2 replies; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Parse MADT table and extract MPIDR for all
CPU IDs in MADT ACPI_MADT_TYPE_GENERIC_INTERRUPT entries
and store in cpuid_to_hwid_map[]
This mapping is used by SRAT table parsing to extract MPIDR
of the CPU ID.
MADT table is also parsed in arm/acpi/boot.c during smp boot.
However cannot wait till smp boot as SRAT table is parsed
much before during numa_init. Hence MADT is parsed twice
during boot. Once in numa_init and another in smp init.
Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
---
v3: - acpi_numa is set to -1 on numa failure.
---
xen/arch/arm/numa/Makefile | 1 +
xen/arch/arm/numa/acpi_numa.c | 94 +++++++++++++++++++++++++++++++++++++++++++
xen/arch/arm/numa/numa.c | 6 +++
3 files changed, 101 insertions(+)
diff --git a/xen/arch/arm/numa/Makefile b/xen/arch/arm/numa/Makefile
index 3af3aff..b549459 100644
--- a/xen/arch/arm/numa/Makefile
+++ b/xen/arch/arm/numa/Makefile
@@ -1,2 +1,3 @@
obj-y += dt_numa.o
obj-y += numa.o
+obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o
diff --git a/xen/arch/arm/numa/acpi_numa.c b/xen/arch/arm/numa/acpi_numa.c
new file mode 100644
index 0000000..d9ad547
--- /dev/null
+++ b/xen/arch/arm/numa/acpi_numa.c
@@ -0,0 +1,94 @@
+/*
+ * ACPI based NUMA setup
+ *
+ * Copyright (C) 2016 - Cavium Inc.
+ * Vijaya Kumar K <Vijaya.Kumar@cavium.com>
+ *
+ * Reads the ACPI MADT and SRAT table to setup NUMA information.
+ * Contains Excerpts from x86 implementation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/init.h>
+#include <xen/mm.h>
+#include <xen/inttypes.h>
+#include <xen/nodemask.h>
+#include <xen/acpi.h>
+#include <xen/numa.h>
+#include <xen/pfn.h>
+#include <xen/acpi.h>
+#include <acpi/srat.h>
+#include <asm/page.h>
+
+/* Holds CPUID to MPIDR mapping read from MADT table. */
+struct cpuid_to_hwid {
+ uint32_t cpuid;
+ uint64_t hwid;
+};
+
+#define PHYS_CPUID_INVALID 0xff
+
+/* Holds mapping of CPU id to MPIDR read from MADT */
+static struct cpuid_to_hwid __read_mostly cpuid_to_hwid_map[NR_CPUS] =
+ { [0 ... NR_CPUS - 1] = {PHYS_CPUID_INVALID, MPIDR_INVALID} };
+static unsigned int num_cpuid_to_hwid;
+
+static void __init acpi_map_cpu_to_hwid(uint32_t cpuid, uint64_t mpidr)
+{
+ if ( mpidr == MPIDR_INVALID )
+ {
+ printk("Skip MADT cpu entry with invalid MPIDR\n");
+ numa_failed();
+ return;
+ }
+
+ cpuid_to_hwid_map[num_cpuid_to_hwid].hwid = mpidr;
+ cpuid_to_hwid_map[num_cpuid_to_hwid].cpuid = cpuid;
+ num_cpuid_to_hwid++;
+}
+
+static int __init acpi_parse_madt_handler(struct acpi_subtable_header *header,
+ const unsigned long end)
+{
+ uint64_t mpidr;
+ struct acpi_madt_generic_interrupt *p =
+ container_of(header, struct acpi_madt_generic_interrupt, header);
+
+ if ( BAD_MADT_ENTRY(p, end) )
+ {
+ /* MADT is invalid, we disable NUMA by calling numa_failed() */
+ numa_failed();
+ return -EINVAL;
+ }
+
+ acpi_table_print_madt_entry(header);
+ mpidr = p->arm_mpidr & MPIDR_HWID_MASK;
+ acpi_map_cpu_to_hwid(p->uid, mpidr);
+
+ return 0;
+}
+
+void __init acpi_map_uid_to_mpidr(void)
+{
+ acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT,
+ acpi_parse_madt_handler, NR_CPUS);
+}
+
+void __init acpi_numa_arch_fixup(void) {}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
index 85352dc..26aa4c0 100644
--- a/xen/arch/arm/numa/numa.c
+++ b/xen/arch/arm/numa/numa.c
@@ -19,6 +19,7 @@
#include <xen/nodemask.h>
#include <xen/numa.h>
#include <xen/pfn.h>
+#include <acpi/srat.h>
#include <asm/acpi.h>
static uint8_t (*node_distance_fn)(nodeid_t a, nodeid_t b);
@@ -40,6 +41,11 @@ void numa_failed(void)
init_dt_numa_distance();
node_distance_fn = NULL;
init_cpu_to_node();
+
+#ifdef CONFIG_ACPI_NUMA
+ acpi_numa = -1;
+ reset_pxm2node();
+#endif
}
void __init numa_set_cpu_node(int cpu, unsigned int nid)
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 20/24] ACPI: Move arch specific SRAT parsing
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (18 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 19/24] ARM: NUMA: Extract MPIDR from MADT table vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-24 21:15 ` Stefano Stabellini
2017-07-18 11:41 ` [RFC PATCH v3 21/24] ARM: NUMA: ACPI: Extract proximity from SRAT table vijay.kilari
` (4 subsequent siblings)
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
SRAT's X2APIC_CPU_AFFINITY and CPU_AFFINITY types are not used
by ARM. Hence move handling of this SRAT types to arch specific
file and handle them under arch_table_parse_srat().
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
xen/arch/arm/numa/acpi_numa.c | 5 +++++
xen/arch/x86/srat.c | 44 +++++++++++++++++++++++++++++++++++++++++++
xen/drivers/acpi/numa.c | 43 ++----------------------------------------
xen/include/xen/acpi.h | 6 ++++++
4 files changed, 57 insertions(+), 41 deletions(-)
diff --git a/xen/arch/arm/numa/acpi_numa.c b/xen/arch/arm/numa/acpi_numa.c
index d9ad547..341e20b7 100644
--- a/xen/arch/arm/numa/acpi_numa.c
+++ b/xen/arch/arm/numa/acpi_numa.c
@@ -82,6 +82,11 @@ void __init acpi_map_uid_to_mpidr(void)
acpi_parse_madt_handler, NR_CPUS);
}
+void __init arch_table_parse_srat(void)
+{
+ return;
+}
+
void __init acpi_numa_arch_fixup(void) {}
/*
diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
index d5caccf..a5fdedd 100644
--- a/xen/arch/x86/srat.c
+++ b/xen/arch/x86/srat.c
@@ -205,3 +205,47 @@ uint8_t __node_distance(nodeid_t a, nodeid_t b)
}
EXPORT_SYMBOL(__node_distance);
+
+static int __init
+acpi_parse_x2apic_affinity(struct acpi_subtable_header *header,
+ const unsigned long end)
+{
+ const struct acpi_srat_x2apic_cpu_affinity *processor_affinity
+ = container_of(header, struct acpi_srat_x2apic_cpu_affinity,
+ header);
+
+ if (!header)
+ return -EINVAL;
+
+ acpi_table_print_srat_entry(header);
+
+ /* let architecture-dependent part to do it */
+ acpi_numa_x2apic_affinity_init(processor_affinity);
+
+ return 0;
+}
+
+static int __init
+acpi_parse_processor_affinity(struct acpi_subtable_header *header,
+ const unsigned long end)
+{
+ const struct acpi_srat_cpu_affinity *processor_affinity
+ = container_of(header, struct acpi_srat_cpu_affinity, header);
+
+ if (!header)
+ return -EINVAL;
+
+ acpi_table_print_srat_entry(header);
+
+ acpi_numa_processor_affinity_init(processor_affinity);
+
+ return 0;
+}
+
+void __init arch_table_parse_srat(void)
+{
+ acpi_table_parse_srat(ACPI_SRAT_TYPE_X2APIC_CPU_AFFINITY,
+ acpi_parse_x2apic_affinity, 0);
+ acpi_table_parse_srat(ACPI_SRAT_TYPE_CPU_AFFINITY,
+ acpi_parse_processor_affinity, 0);
+}
diff --git a/xen/drivers/acpi/numa.c b/xen/drivers/acpi/numa.c
index 85f8917..0adc32c 100644
--- a/xen/drivers/acpi/numa.c
+++ b/xen/drivers/acpi/numa.c
@@ -120,43 +120,6 @@ static int __init acpi_parse_slit(struct acpi_table_header *table)
}
static int __init
-acpi_parse_x2apic_affinity(struct acpi_subtable_header *header,
- const unsigned long end)
-{
- const struct acpi_srat_x2apic_cpu_affinity *processor_affinity
- = container_of(header, struct acpi_srat_x2apic_cpu_affinity,
- header);
-
- if (!header)
- return -EINVAL;
-
- acpi_table_print_srat_entry(header);
-
- /* let architecture-dependent part to do it */
- acpi_numa_x2apic_affinity_init(processor_affinity);
-
- return 0;
-}
-
-static int __init
-acpi_parse_processor_affinity(struct acpi_subtable_header *header,
- const unsigned long end)
-{
- const struct acpi_srat_cpu_affinity *processor_affinity
- = container_of(header, struct acpi_srat_cpu_affinity, header);
-
- if (!header)
- return -EINVAL;
-
- acpi_table_print_srat_entry(header);
-
- /* let architecture-dependent part to do it */
- acpi_numa_processor_affinity_init(processor_affinity);
-
- return 0;
-}
-
-static int __init
acpi_parse_memory_affinity(struct acpi_subtable_header *header,
const unsigned long end)
{
@@ -197,13 +160,11 @@ int __init acpi_numa_init(void)
{
/* SRAT: Static Resource Affinity Table */
if (!acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) {
- acpi_table_parse_srat(ACPI_SRAT_TYPE_X2APIC_CPU_AFFINITY,
- acpi_parse_x2apic_affinity, 0);
- acpi_table_parse_srat(ACPI_SRAT_TYPE_CPU_AFFINITY,
- acpi_parse_processor_affinity, 0);
acpi_table_parse_srat(ACPI_SRAT_TYPE_MEMORY_AFFINITY,
acpi_parse_memory_affinity,
NR_NODE_MEMBLKS);
+ /* This call handles architecture dependant SRAT */
+ arch_table_parse_srat();
}
/* SLIT: System Locality Information Table */
diff --git a/xen/include/xen/acpi.h b/xen/include/xen/acpi.h
index 9409350..53795ff 100644
--- a/xen/include/xen/acpi.h
+++ b/xen/include/xen/acpi.h
@@ -95,7 +95,13 @@ void acpi_numa_slit_init (struct acpi_table_slit *slit);
void acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *);
void acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *);
void acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *);
+#ifdef CONFIG_ACPI_NUMA
void acpi_numa_arch_fixup(void);
+void arch_table_parse_srat(void);
+#else
+static inline void acpi_numa_arch_fixup(void) { }
+static inline void arch_table_parse_srat(void) { }
+#endif
#ifdef CONFIG_ACPI_HOTPLUG_CPU
/* Arch dependent functions for cpu hotplug support */
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 21/24] ARM: NUMA: ACPI: Extract proximity from SRAT table
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (19 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 20/24] ACPI: Move arch specific SRAT parsing vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-24 22:17 ` Stefano Stabellini
2017-07-26 18:18 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 22/24] ARM: NUMA: Initialize ACPI NUMA vijay.kilari
` (3 subsequent siblings)
24 siblings, 2 replies; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Register SRAT entry handler for type
ACPI_SRAT_TYPE_GICC_AFFINITY to parse SRAT table
and extract proximity for all CPU IDs.
Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
---
xen/arch/arm/acpi/boot.c | 2 +
xen/arch/arm/numa/acpi_numa.c | 124 +++++++++++++++++++++++++++++++++++++++++-
xen/drivers/acpi/numa.c | 15 +++++
xen/include/acpi/actbl1.h | 17 +++++-
xen/include/asm-arm/numa.h | 9 +++
5 files changed, 165 insertions(+), 2 deletions(-)
diff --git a/xen/arch/arm/acpi/boot.c b/xen/arch/arm/acpi/boot.c
index 889208a..4e28b16 100644
--- a/xen/arch/arm/acpi/boot.c
+++ b/xen/arch/arm/acpi/boot.c
@@ -31,6 +31,7 @@
#include <acpi/actables.h>
#include <xen/mm.h>
#include <xen/device_tree.h>
+#include <xen/numa.h>
#include <asm/acpi.h>
#include <asm/smp.h>
@@ -117,6 +118,7 @@ acpi_map_gic_cpu_interface(struct acpi_madt_generic_interrupt *processor)
return;
}
+ numa_set_cpu_node(enabled_cpus, acpi_get_nodeid(mpidr));
/* map the logical cpu id to cpu MPIDR */
cpu_logical_map(enabled_cpus) = mpidr;
diff --git a/xen/arch/arm/numa/acpi_numa.c b/xen/arch/arm/numa/acpi_numa.c
index 341e20b7..95617f9 100644
--- a/xen/arch/arm/numa/acpi_numa.c
+++ b/xen/arch/arm/numa/acpi_numa.c
@@ -34,13 +34,63 @@ struct cpuid_to_hwid {
uint64_t hwid;
};
+/* Holds NODE to MPIDR mapping. */
+struct node_to_hwid {
+ nodeid_t nodeid;
+ uint64_t hwid;
+};
+
#define PHYS_CPUID_INVALID 0xff
/* Holds mapping of CPU id to MPIDR read from MADT */
static struct cpuid_to_hwid __read_mostly cpuid_to_hwid_map[NR_CPUS] =
{ [0 ... NR_CPUS - 1] = {PHYS_CPUID_INVALID, MPIDR_INVALID} };
+static struct node_to_hwid __read_mostly node_to_hwid_map[NR_CPUS] =
+ { [0 ... NR_CPUS - 1] = {NUMA_NO_NODE, MPIDR_INVALID} };
+static unsigned int cpus_in_srat;
static unsigned int num_cpuid_to_hwid;
+nodeid_t __init acpi_get_nodeid(uint64_t hwid)
+{
+ unsigned int i;
+
+ for ( i = 0; i < cpus_in_srat; i++ )
+ {
+ if ( node_to_hwid_map[i].hwid == hwid )
+ return node_to_hwid_map[i].nodeid;
+ }
+
+ return NUMA_NO_NODE;
+}
+
+static uint64_t acpi_get_cpu_hwid(int cid)
+{
+ unsigned int i;
+
+ for ( i = 0; i < num_cpuid_to_hwid; i++ )
+ {
+ if ( cpuid_to_hwid_map[i].cpuid == cid )
+ return cpuid_to_hwid_map[i].hwid;
+ }
+
+ return MPIDR_INVALID;
+}
+
+static void __init acpi_map_node_to_hwid(nodeid_t nodeid, uint64_t hwid)
+{
+ if ( nodeid >= MAX_NUMNODES )
+ {
+ printk(XENLOG_WARNING
+ "ACPI: NUMA: nodeid out of range %d with MPIDR 0x%lx\n",
+ nodeid, hwid);
+ numa_failed();
+ return;
+ }
+
+ node_to_hwid_map[cpus_in_srat].nodeid = nodeid;
+ node_to_hwid_map[cpus_in_srat].hwid = hwid;
+}
+
static void __init acpi_map_cpu_to_hwid(uint32_t cpuid, uint64_t mpidr)
{
if ( mpidr == MPIDR_INVALID )
@@ -76,15 +126,87 @@ static int __init acpi_parse_madt_handler(struct acpi_subtable_header *header,
return 0;
}
+/* Callback for Proximity Domain -> ACPI processor UID mapping */
+static void __init
+acpi_numa_gicc_affinity_init(const struct acpi_srat_gicc_affinity *pa)
+{
+ int pxm, node;
+ uint64_t mpidr;
+
+ if ( srat_disabled() )
+ return;
+
+ if ( pa->header.length < sizeof(struct acpi_srat_gicc_affinity) )
+ {
+ printk(XENLOG_WARNING "SRAT: Invalid SRAT header length: %d\n",
+ pa->header.length);
+ numa_failed();
+ return;
+ }
+
+ if ( !(pa->flags & ACPI_SRAT_GICC_ENABLED) )
+ return;
+
+ if ( cpus_in_srat >= NR_CPUS )
+ {
+ printk(XENLOG_ERR
+ "SRAT: cpu_to_node_map[%d] is too small to fit all cpus\n",
+ NR_CPUS);
+ return;
+ }
+
+ pxm = pa->proximity_domain;
+ node = acpi_setup_node(pxm);
+ if ( node == NUMA_NO_NODE )
+ {
+ numa_failed();
+ return;
+ }
+
+ mpidr = acpi_get_cpu_hwid(pa->acpi_processor_uid);
+ if ( mpidr == MPIDR_INVALID )
+ {
+ printk(XENLOG_ERR
+ "SRAT: PXM %d with ACPI ID %d has no valid MPIDR in MADT\n",
+ pxm, pa->acpi_processor_uid);
+ numa_failed();
+ return;
+ }
+
+ acpi_map_node_to_hwid(node, mpidr);
+ node_set(node, processor_nodes_parsed);
+ cpus_in_srat++;
+ acpi_numa = 1;
+ printk(XENLOG_INFO "SRAT: PXM %d -> MPIDR 0x%lx -> Node %d\n",
+ pxm, mpidr, node);
+}
+
void __init acpi_map_uid_to_mpidr(void)
{
acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT,
acpi_parse_madt_handler, NR_CPUS);
}
+static int __init
+acpi_parse_gicc_affinity(struct acpi_subtable_header *header,
+ const unsigned long end)
+{
+ const struct acpi_srat_gicc_affinity *processor_affinity
+ = (struct acpi_srat_gicc_affinity *)header;
+
+ if (!processor_affinity)
+ return -EINVAL;
+
+ acpi_table_print_srat_entry(header);
+ acpi_numa_gicc_affinity_init(processor_affinity);
+
+ return 0;
+}
+
void __init arch_table_parse_srat(void)
{
- return;
+ acpi_table_parse_srat(ACPI_SRAT_TYPE_GICC_AFFINITY,
+ acpi_parse_gicc_affinity, NR_CPUS);
}
void __init acpi_numa_arch_fixup(void) {}
diff --git a/xen/drivers/acpi/numa.c b/xen/drivers/acpi/numa.c
index 0adc32c..b48d91d 100644
--- a/xen/drivers/acpi/numa.c
+++ b/xen/drivers/acpi/numa.c
@@ -104,6 +104,21 @@ void __init acpi_table_print_srat_entry(struct acpi_subtable_header * header)
}
#endif /* ACPI_DEBUG_OUTPUT */
break;
+ case ACPI_SRAT_TYPE_GICC_AFFINITY:
+#ifdef ACPI_DEBUG_OUTPUT
+ {
+ struct acpi_srat_gicc_affinity *p =
+ (struct acpi_srat_gicc_affinity *)header;
+ ACPI_DEBUG_PRINT((ACPI_DB_INFO,
+ "SRAT Processor (acpi id[0x%04x]) in"
+ " proximity domain %d %s\n",
+ p->acpi_processor_uid,
+ p->proximity_domain,
+ (p->flags & ACPI_SRAT_GICC_ENABLED) ?
+ "enabled" : "disabled");
+ }
+#endif /* ACPI_DEBUG_OUTPUT */
+ break;
default:
printk(KERN_WARNING PREFIX
"Found unsupported SRAT entry (type = %#x)\n",
diff --git a/xen/include/acpi/actbl1.h b/xen/include/acpi/actbl1.h
index e199136..b84bfba 100644
--- a/xen/include/acpi/actbl1.h
+++ b/xen/include/acpi/actbl1.h
@@ -949,7 +949,8 @@ enum acpi_srat_type {
ACPI_SRAT_TYPE_CPU_AFFINITY = 0,
ACPI_SRAT_TYPE_MEMORY_AFFINITY = 1,
ACPI_SRAT_TYPE_X2APIC_CPU_AFFINITY = 2,
- ACPI_SRAT_TYPE_RESERVED = 3 /* 3 and greater are reserved */
+ ACPI_SRAT_TYPE_GICC_AFFINITY = 3,
+ ACPI_SRAT_TYPE_RESERVED = 4 /* 4 and greater are reserved */
};
/*
@@ -1007,6 +1008,20 @@ struct acpi_srat_x2apic_cpu_affinity {
#define ACPI_SRAT_CPU_ENABLED (1) /* 00: Use affinity structure */
+/* 3: GICC Affinity (ACPI 5.1) */
+
+struct acpi_srat_gicc_affinity {
+ struct acpi_subtable_header header;
+ u32 proximity_domain;
+ u32 acpi_processor_uid;
+ u32 flags;
+ u32 clock_domain;
+};
+
+/* Flags for struct acpi_srat_gicc_affinity */
+
+#define ACPI_SRAT_GICC_ENABLED (1) /* 00: Use affinity structure */
+
/* Reset to default packing */
#pragma pack()
diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
index 0d3146c..f0a50bd 100644
--- a/xen/include/asm-arm/numa.h
+++ b/xen/include/asm-arm/numa.h
@@ -7,6 +7,15 @@ void dt_numa_process_memory_node(uint32_t nid, paddr_t start, paddr_t size);
void register_node_distance(uint8_t (fn)(nodeid_t a, nodeid_t b));
void init_dt_numa_distance(void);
+#ifdef CONFIG_ACPI_NUMA
+nodeid_t acpi_get_nodeid(uint64_t hwid);
+#else
+static inline nodeid_t acpi_get_nodeid(uint64_t hwid)
+{
+ return 0;
+}
+#endif /* CONFIG_ACPI_NUMA */
+
#ifdef CONFIG_NUMA
void numa_init(void);
int dt_numa_init(void);
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 22/24] ARM: NUMA: Initialize ACPI NUMA
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (20 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 21/24] ARM: NUMA: ACPI: Extract proximity from SRAT table vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-24 22:11 ` Stefano Stabellini
2017-07-26 18:23 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 23/24] NUMA: Move CONFIG_NUMA to common Kconfig vijay.kilari
` (2 subsequent siblings)
24 siblings, 2 replies; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Call ACPI NUMA initialization under CONFIG_ACPI_NUMA.
Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
---
xen/arch/arm/numa/acpi_numa.c | 27 ++++++++++++++++++++++++++-
xen/arch/arm/numa/numa.c | 15 +++++++++++++--
xen/common/numa.c | 14 ++++++++++++++
xen/include/asm-arm/numa.h | 1 +
xen/include/xen/numa.h | 1 +
5 files changed, 55 insertions(+), 3 deletions(-)
diff --git a/xen/arch/arm/numa/acpi_numa.c b/xen/arch/arm/numa/acpi_numa.c
index 95617f9..68fff95 100644
--- a/xen/arch/arm/numa/acpi_numa.c
+++ b/xen/arch/arm/numa/acpi_numa.c
@@ -181,7 +181,7 @@ acpi_numa_gicc_affinity_init(const struct acpi_srat_gicc_affinity *pa)
pxm, mpidr, node);
}
-void __init acpi_map_uid_to_mpidr(void)
+static void __init acpi_map_uid_to_mpidr(void)
{
acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT,
acpi_parse_madt_handler, NR_CPUS);
@@ -209,6 +209,31 @@ void __init arch_table_parse_srat(void)
acpi_parse_gicc_affinity, NR_CPUS);
}
+bool_t __init arch_acpi_numa_init(void)
+{
+ int ret;
+
+ if ( !acpi_disabled )
+ {
+ /*
+ * If firmware has DT, process_memory_node() call
+ * would have added memory blocks. So reset it before
+ * ACPI numa init.
+ */
+ numa_clear_memblks();
+ nodes_clear(memory_nodes_parsed);
+ acpi_map_uid_to_mpidr();
+ ret = acpi_numa_init();
+ if ( ret || srat_disabled() )
+ return 1;
+
+ /* Register acpi node_distance handler */
+ register_node_distance(&acpi_node_distance);
+ }
+
+ return 0;
+}
+
void __init acpi_numa_arch_fixup(void) {}
/*
diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
index 26aa4c0..68599c4 100644
--- a/xen/arch/arm/numa/numa.c
+++ b/xen/arch/arm/numa/numa.c
@@ -139,11 +139,22 @@ void __init numa_init(void)
if ( numa_off )
goto no_numa;
- ret = dt_numa_init();
+#ifdef CONFIG_ACPI_NUMA
+ ret = arch_acpi_numa_init();
if ( ret )
{
numa_off = true;
- printk(XENLOG_WARNING "DT NUMA init failed\n");
+ printk(XENLOG_WARNING "ACPI NUMA init failed\n");
+ }
+#endif
+ if ( acpi_disabled )
+ {
+ ret = dt_numa_init();
+ if ( ret )
+ {
+ numa_off = true;
+ printk(XENLOG_WARNING "DT NUMA init failed\n");
+ }
}
no_numa:
diff --git a/xen/common/numa.c b/xen/common/numa.c
index 0f79a07..020bc19 100644
--- a/xen/common/numa.c
+++ b/xen/common/numa.c
@@ -76,6 +76,20 @@ nodeid_t get_memblk_nodeid(unsigned int id)
return memblk_nodeid[id];
}
+void __init numa_clear_memblks(void)
+{
+ unsigned int i;
+
+ for ( i = 0; i < get_num_node_memblks(); i++ )
+ {
+ node_memblk_range[i].start = 0;
+ node_memblk_range[i].end = 0;
+ memblk_nodeid[i] = NUMA_NO_NODE;
+ }
+
+ num_node_memblks = 0;
+}
+
int __init get_mem_nodeid(paddr_t start, paddr_t end)
{
unsigned int i;
diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
index f0a50bd..ff10b31 100644
--- a/xen/include/asm-arm/numa.h
+++ b/xen/include/asm-arm/numa.h
@@ -20,6 +20,7 @@ static inline nodeid_t acpi_get_nodeid(uint64_t hwid)
void numa_init(void);
int dt_numa_init(void);
void numa_set_cpu_node(int cpu, unsigned int nid);
+bool_t arch_acpi_numa_init(void);
#else
static inline void numa_init(void)
diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
index a541eb7..14a7a0c 100644
--- a/xen/include/xen/numa.h
+++ b/xen/include/xen/numa.h
@@ -75,6 +75,7 @@ int get_num_node_memblks(void);
bool arch_sanitize_nodes_memory(void);
void numa_failed(void);
uint8_t __node_distance(nodeid_t a, nodeid_t b);
+void numa_clear_memblks(void);
#else
static inline void numa_add_cpu(int cpu) { }
static inline void numa_set_node(int cpu, nodeid_t node) { }
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 23/24] NUMA: Move CONFIG_NUMA to common Kconfig
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (21 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 22/24] ARM: NUMA: Initialize ACPI NUMA vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-18 16:25 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 24/24] NUMA: Enable ACPI_NUMA config vijay.kilari
2017-07-18 16:18 ` [RFC PATCH v3 00/24] ARM: Add Xen NUMA support Julien Grall
24 siblings, 1 reply; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
CONFIG_NUMA is defined in xen/drivers/acpi/Kconfig.
Move to common/Kconfig and enabled by default.
Also, NUMA feature uses PDX for physical address to
memory node mapping. Hence make HAS_PDX dependent
for NUMA.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
xen/common/Kconfig | 4 ++++
xen/drivers/acpi/Kconfig | 3 ---
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/xen/common/Kconfig b/xen/common/Kconfig
index dc8e876..6e421c7 100644
--- a/xen/common/Kconfig
+++ b/xen/common/Kconfig
@@ -41,6 +41,10 @@ config HAS_GDBSX
config HAS_IOPORTS
bool
+config NUMA
+ def_bool y
+ depends on HAS_PDX
+
config HAS_BUILD_ID
string
option env="XEN_HAS_BUILD_ID"
diff --git a/xen/drivers/acpi/Kconfig b/xen/drivers/acpi/Kconfig
index b64d373..488372f 100644
--- a/xen/drivers/acpi/Kconfig
+++ b/xen/drivers/acpi/Kconfig
@@ -4,6 +4,3 @@ config ACPI
config ACPI_LEGACY_TABLES_LOOKUP
bool
-
-config NUMA
- bool
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* [RFC PATCH v3 24/24] NUMA: Enable ACPI_NUMA config
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (22 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 23/24] NUMA: Move CONFIG_NUMA to common Kconfig vijay.kilari
@ 2017-07-18 11:41 ` vijay.kilari
2017-07-18 16:18 ` [RFC PATCH v3 00/24] ARM: Add Xen NUMA support Julien Grall
24 siblings, 0 replies; 109+ messages in thread
From: vijay.kilari @ 2017-07-18 11:41 UTC (permalink / raw)
To: xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, julien.grall, jbeulich,
Vijaya Kumar K
From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Add CONFIG_ACPI_NUMA to xen/drivers/acpi/Kconfig and
drop CONFIG_ACPI_NUMA set in asm-x86/config.h.
Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
---
xen/drivers/acpi/Kconfig | 4 ++++
xen/include/asm-x86/config.h | 1 -
2 files changed, 4 insertions(+), 1 deletion(-)
diff --git a/xen/drivers/acpi/Kconfig b/xen/drivers/acpi/Kconfig
index 488372f..8e15428 100644
--- a/xen/drivers/acpi/Kconfig
+++ b/xen/drivers/acpi/Kconfig
@@ -4,3 +4,7 @@ config ACPI
config ACPI_LEGACY_TABLES_LOOKUP
bool
+
+config ACPI_NUMA
+ def_bool y
+ depends on ACPI && NUMA
diff --git a/xen/include/asm-x86/config.h b/xen/include/asm-x86/config.h
index dc424f9..3e3cc36 100644
--- a/xen/include/asm-x86/config.h
+++ b/xen/include/asm-x86/config.h
@@ -34,7 +34,6 @@
#define CONFIG_X86_L1_CACHE_SHIFT 7
#define CONFIG_ACPI_SLEEP 1
-#define CONFIG_ACPI_NUMA 1
#define CONFIG_ACPI_SRAT 1
#define CONFIG_ACPI_CSTATE 1
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable
2017-07-18 11:41 ` [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable vijay.kilari
@ 2017-07-18 15:29 ` Wei Liu
2017-07-18 17:52 ` Julien Grall
2017-07-28 10:11 ` Jan Beulich
2017-07-18 17:55 ` Julien Grall
1 sibling, 2 replies; 109+ messages in thread
From: Wei Liu @ 2017-07-18 15:29 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 05:11:23PM +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Introduce NR_NODES config option to specify number
> of NUMA nodes supported. By default value is set at
> 64 for x86 and 8 for arm. Dropped NODES_SHIFT macro.
>
> Also move NR_NODE_MEMBLKS from asm-x86/acpi.h to xen/numa.h
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> xen/arch/Kconfig | 7 +++++++
> xen/include/asm-x86/acpi.h | 1 -
> xen/include/asm-x86/numa.h | 2 --
> xen/include/xen/config.h | 1 +
> xen/include/xen/numa.h | 7 ++-----
> 5 files changed, 10 insertions(+), 8 deletions(-)
>
> diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig
> index cf0acb7..9c2a4e2 100644
> --- a/xen/arch/Kconfig
> +++ b/xen/arch/Kconfig
> @@ -6,3 +6,10 @@ config NR_CPUS
> default "128" if ARM
> ---help---
> Specifies the maximum number of physical CPUs which Xen will support.
> +
> +config NR_NODES
> + int "Maximum number of NUMA nodes"
> + default "64" if X86
> + default "8" if ARM
> + ---help---
> + Specifies the maximum number of NUMA nodes which Xen will support.
Since this can now be specified by user but the definition of
NUMA_NO_NODE is not changed, I think you need to sanitise the value
provided somewhere.
Maybe introduce a build time check? There are some examples in tree. See
cpuid.c:build_assertions.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 03/24] x86: NUMA: Fix datatypes and attributes
2017-07-18 11:41 ` [RFC PATCH v3 03/24] x86: NUMA: Fix datatypes and attributes vijay.kilari
@ 2017-07-18 15:29 ` Wei Liu
0 siblings, 0 replies; 109+ messages in thread
From: Wei Liu @ 2017-07-18 15:29 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 05:11:25PM +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Change u{8,32,64} to uint{8,32,64}_t, u64 to paddr_t
> wherever applicable.
> Fix attributes coding styles.
> Also changed
> - Some variables from int to unsigned int
> - Used pfn_to_paddr/paddr_to_pfn whereever required.
> - Alloc memnodemap[] of size BITS_PER_LONG.
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
FWIW
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 04/24] x86: NUMA: Rename and sanitize memnode shift code
2017-07-18 11:41 ` [RFC PATCH v3 04/24] x86: NUMA: Rename and sanitize memnode shift code vijay.kilari
@ 2017-07-18 15:29 ` Wei Liu
2017-07-19 17:12 ` Julien Grall
1 sibling, 0 replies; 109+ messages in thread
From: Wei Liu @ 2017-07-18 15:29 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 05:11:26PM +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> memnode_shift variable is changed from int to unsigned int.
> With this change, compute_memnode_shift() returns error value
> instead of returning shift value. The memnode_shift is updated inside
> compute_memnode_shift().
>
> Also, following changes are made
> - Rename compute_hash_shift to compute_memnode_shift
> - Update int to unsigned int for params in extract_lsb_from_nodes()
> - Return values of populate_memnodemap() is changed
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Reviewed-by: Wei Liu <wei.liu2@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 05/24] x86: NUMA: Add accessors for nodes[] and node_memblk_range[] structs
2017-07-18 11:41 ` [RFC PATCH v3 05/24] x86: NUMA: Add accessors for nodes[] and node_memblk_range[] structs vijay.kilari
@ 2017-07-18 15:29 ` Wei Liu
2017-07-19 6:40 ` Vijay Kilari
0 siblings, 1 reply; 109+ messages in thread
From: Wei Liu @ 2017-07-18 15:29 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 05:11:27PM +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Add accessors for nodes[] and other static variables and
> use those accessors. These variables are later accessed
> outside the file when the code made generic in later
> patches. However the coding style is not changed.
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> v3: - Changed accessors parameter from int to unsigned int
> - Updated commit message
> - Fixed wrong indentation
> ---
> xen/arch/x86/srat.c | 106 +++++++++++++++++++++++++++++++++++++++-------------
> 1 file changed, 81 insertions(+), 25 deletions(-)
>
> diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
> index 535c9d7..42cca5a 100644
> --- a/xen/arch/x86/srat.c
> +++ b/xen/arch/x86/srat.c
> @@ -41,6 +41,44 @@ static struct node node_memblk_range[NR_NODE_MEMBLKS];
> static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
> static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
>
> +static struct node *get_numa_node(unsigned int id)
> +{
> + return &nodes[id];
> +}
> +
> +static nodeid_t get_memblk_nodeid(unsigned int id)
> +{
> + return memblk_nodeid[id];
> +}
> +
> +static nodeid_t *get_memblk_nodeid_map(void)
> +{
> + return &memblk_nodeid[0];
> +}
> +
> +static struct node *get_node_memblk_range(unsigned int memblk)
> +{
> + return &node_memblk_range[memblk];
> +}
> +
> +static int get_num_node_memblks(void)
> +{
> + return num_node_memblks;
> +}
They should all be inline functions. And maybe at once lift to a header
and add proper prefix since you mention they are going to be used later.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic
2017-07-18 11:41 ` [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic vijay.kilari
@ 2017-07-18 15:29 ` Wei Liu
2017-07-18 18:16 ` Julien Grall
2017-07-19 17:41 ` Julien Grall
1 sibling, 1 reply; 109+ messages in thread
From: Wei Liu @ 2017-07-18 15:29 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 05:11:30PM +0530, vijay.kilari@gmail.com wrote:
[...]
> diff --git a/xen/common/numa.c b/xen/common/numa.c
> new file mode 100644
> index 0000000..0381f1b
> --- /dev/null
> +++ b/xen/common/numa.c
> @@ -0,0 +1,487 @@
> +/*
> + * Common NUMA handling functions for x86 and arm.
> + * Original code extracted from arch/x86/numa.c
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms and conditions of the GNU General Public
> + * License, version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/ctype.h>
> +#include <xen/sched.h>
> +#include <xen/nodemask.h>
> +#include <xen/numa.h>
> +#include <xen/keyhandler.h>
> +#include <xen/time.h>
> +#include <xen/smp.h>
> +#include <xen/pfn.h>
> +#include <xen/mm.h>
> +#include <xen/softirq.h>
> +#include <xen/string.h>
> +#include <asm/acpi.h>
> +
Since you're moving code anyway, please sort the headers alphabetically.
> +static int numa_setup(char *s);
> +custom_param("numa", numa_setup);
> +
> +struct node_data node_data[MAX_NUMNODES];
> +
> +/* Mapping from pdx to node id */
Is this comment applicable to ARM? Does arm has PDX?
> +unsigned int memnode_shift;
> +
> +/*
> + * In case of numa init failure or numa off,
> + * memnode_shift is initialized to BITS_PER_LONG - 1. Hence allocate
> + * memnodemap[] of BITS_PER_LONG.
> + */
> +static typeof(*memnodemap) _memnodemap[BITS_PER_LONG];
> +unsigned long memnodemapsize;
> +uint8_t *memnodemap;
> +
> +nodeid_t __read_mostly cpu_to_node[NR_CPUS] = {
> + [0 ... NR_CPUS-1] = NUMA_NO_NODE
> +};
> +
> +cpumask_t __read_mostly node_to_cpumask[MAX_NUMNODES];
> +
> +bool numa_off;
> +s8 acpi_numa = 0;
> +
> +int srat_disabled(void)
bool here.
Should probably be done in a previous patch.
> +
> +void __init numa_init_array(void)
> +{
> + int rr, i;
> +
> + /* There are unfortunately some poorly designed mainboards around
> + that only connect memory to a single CPU. This breaks the 1:1 cpu->node
> + mapping. To avoid this fill in the mapping for all possible
> + CPUs, as the number of CPUs is not known yet.
> + We round robin the existing nodes. */
Please fix the coding style issue here.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 11/24] ARM: fdt: Export and introduce new fdt functions
2017-07-18 11:41 ` [RFC PATCH v3 11/24] ARM: fdt: Export and introduce new fdt functions vijay.kilari
@ 2017-07-18 15:29 ` Wei Liu
2017-07-18 16:29 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Wei Liu @ 2017-07-18 15:29 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 05:11:33PM +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Introduce new api device_tree_type_matches() to check for
> device type. Also export device_tree_get_u32() and
> device_tree_node_compatible()
>
> These functions are later used for parsing NUMA information.
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> v3: Export device_tree_node_compatible() instead of
> device_tree_node_matches()
> ---
> xen/arch/arm/bootfdt.c | 20 ++++++++++++++++----
> xen/include/asm-arm/setup.h | 5 +++++
> 2 files changed, 21 insertions(+), 4 deletions(-)
>
> diff --git a/xen/arch/arm/bootfdt.c b/xen/arch/arm/bootfdt.c
> index ea188a0..6e8251b 100644
> --- a/xen/arch/arm/bootfdt.c
> +++ b/xen/arch/arm/bootfdt.c
> @@ -31,8 +31,8 @@ static bool_t __init device_tree_node_matches(const void *fdt, int node,
> && (name[match_len] == '@' || name[match_len] == '\0');
> }
>
> -static bool_t __init device_tree_node_compatible(const void *fdt, int node,
> - const char *match)
> +bool_t __init device_tree_node_compatible(const void *fdt, int node,
> + const char *match)
While you're changing the code please change bool_t to bool
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 18/24] ACPI: Refactor acpi SRAT and SLIT table handling code
2017-07-18 11:41 ` [RFC PATCH v3 18/24] ACPI: Refactor acpi SRAT and SLIT table handling code vijay.kilari
@ 2017-07-18 15:36 ` Wei Liu
2017-07-19 6:33 ` Vijay Kilari
0 siblings, 1 reply; 109+ messages in thread
From: Wei Liu @ 2017-07-18 15:36 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 05:11:40PM +0530, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Move SRAT handling code which is common across
> architectures is moved to new file xen/drivers/acpi/srat.c
> from xen/arch/x86/srat.c file. New header file srat.h is
> introduced.
>
> Other major changes are:
> - Coding style of code moved is changed.
> - Moved struct pxm2node from srat.c to srat.h
> - Dropped {memory,processor}_nodes_parsed from x86/srat.c
> - Dropped static on node_to_pxm() and moved to beginning of the file.
> - Made some static functions as non-static
> - acpi_node_distance() is introduced and called from __node_distance()
> - Replaced distance constants with LOCAL/REMOTE_DISTANCE defines
It would be nice if you could break these into individual patches.
[...]
> +
> +/*
> + * A lot of BIOS fill in 10 (= no distance) everywhere. This messes
> + * up the NUMA heuristics which wants the local node to have a smaller
> + * distance than the others.
> + * Do some quick checks here and only use the SLIT if it passes.
> + */
> +static int __init slit_valid(struct acpi_table_slit *slit)
> +{
> + int i, j;
unsigned int
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 00/24] ARM: Add Xen NUMA support
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
` (23 preceding siblings ...)
2017-07-18 11:41 ` [RFC PATCH v3 24/24] NUMA: Enable ACPI_NUMA config vijay.kilari
@ 2017-07-18 16:18 ` Julien Grall
2017-07-19 6:31 ` Vijay Kilari
24 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-18 16:18 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> This patch is tested on Thunderx platform.
> No changes are made to x86 implementation only code is sanitized
> and refactored. Hence only compilation tested for x86.
>
> This series is posted as RFC for the reason that it is not tested
> on x86. Request some help from community in testing this series on x86.
>
> Code is shared at
> https://github.com/vijaykilari/xen-numa/commits/rfc_v3
Few months ago you sent a patch that was a pre-requisite for booting
NUMA (see [1]). It has never been upstreamed, so is it still required?
Cheers,
[1]
https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg03823.html
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 23/24] NUMA: Move CONFIG_NUMA to common Kconfig
2017-07-18 11:41 ` [RFC PATCH v3 23/24] NUMA: Move CONFIG_NUMA to common Kconfig vijay.kilari
@ 2017-07-18 16:25 ` Julien Grall
2017-07-18 18:00 ` Julien Grall
2017-07-28 10:08 ` Jan Beulich
0 siblings, 2 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-18 16:25 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> CONFIG_NUMA is defined in xen/drivers/acpi/Kconfig.
> Move to common/Kconfig and enabled by default.
> Also, NUMA feature uses PDX for physical address to
> memory node mapping. Hence make HAS_PDX dependent
> for NUMA.
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> xen/common/Kconfig | 4 ++++
> xen/drivers/acpi/Kconfig | 3 ---
> 2 files changed, 4 insertions(+), 3 deletions(-)
>
> diff --git a/xen/common/Kconfig b/xen/common/Kconfig
> index dc8e876..6e421c7 100644
> --- a/xen/common/Kconfig
> +++ b/xen/common/Kconfig
> @@ -41,6 +41,10 @@ config HAS_GDBSX
> config HAS_IOPORTS
> bool
>
> +config NUMA
> + def_bool y
> + depends on HAS_PDX
On previous version, Jan asked to remove the dependency on PDX. You said
you will do it... So why it is not done?
> +
> config HAS_BUILD_ID
> string
> option env="XEN_HAS_BUILD_ID"
> diff --git a/xen/drivers/acpi/Kconfig b/xen/drivers/acpi/Kconfig
> index b64d373..488372f 100644
> --- a/xen/drivers/acpi/Kconfig
> +++ b/xen/drivers/acpi/Kconfig
> @@ -4,6 +4,3 @@ config ACPI
>
> config ACPI_LEGACY_TABLES_LOOKUP
> bool
> -
> -config NUMA
> - bool
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 11/24] ARM: fdt: Export and introduce new fdt functions
2017-07-18 15:29 ` Wei Liu
@ 2017-07-18 16:29 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-18 16:29 UTC (permalink / raw)
To: Wei Liu, vijay.kilari
Cc: tim, kevin.tian, sstabellini, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, xen-devel, jbeulich, Vijaya Kumar K
On 18/07/17 16:29, Wei Liu wrote:
> On Tue, Jul 18, 2017 at 05:11:33PM +0530, vijay.kilari@gmail.com wrote:
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> Introduce new api device_tree_type_matches() to check for
>> device type. Also export device_tree_get_u32() and
>> device_tree_node_compatible()
>>
>> These functions are later used for parsing NUMA information.
>>
>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>> ---
>> v3: Export device_tree_node_compatible() instead of
>> device_tree_node_matches()
>> ---
>> xen/arch/arm/bootfdt.c | 20 ++++++++++++++++----
>> xen/include/asm-arm/setup.h | 5 +++++
>> 2 files changed, 21 insertions(+), 4 deletions(-)
>>
>> diff --git a/xen/arch/arm/bootfdt.c b/xen/arch/arm/bootfdt.c
>> index ea188a0..6e8251b 100644
>> --- a/xen/arch/arm/bootfdt.c
>> +++ b/xen/arch/arm/bootfdt.c
>> @@ -31,8 +31,8 @@ static bool_t __init device_tree_node_matches(const void *fdt, int node,
>> && (name[match_len] == '@' || name[match_len] == '\0');
>> }
>>
>> -static bool_t __init device_tree_node_compatible(const void *fdt, int node,
>> - const char *match)
>> +bool_t __init device_tree_node_compatible(const void *fdt, int node,
>> + const char *match)
>
> While you're changing the code please change bool_t to bool
+1.
Also, it is not necessary to CC all the people on every patches. Can you
please use scripts/get_maintainers.pl to only CC relevant maintainers +
people wanted to follow the series on each patch and not everyone
everywhere...
>
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable
2017-07-18 15:29 ` Wei Liu
@ 2017-07-18 17:52 ` Julien Grall
2017-07-19 8:17 ` Wei Liu
2017-07-28 10:11 ` Jan Beulich
1 sibling, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-18 17:52 UTC (permalink / raw)
To: Wei Liu, vijay.kilari
Cc: tim, kevin.tian, sstabellini, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, xen-devel, jbeulich, Vijaya Kumar K
Hi,
On 18/07/17 16:29, Wei Liu wrote:
> On Tue, Jul 18, 2017 at 05:11:23PM +0530, vijay.kilari@gmail.com wrote:
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> Introduce NR_NODES config option to specify number
>> of NUMA nodes supported. By default value is set at
>> 64 for x86 and 8 for arm. Dropped NODES_SHIFT macro.
>>
>> Also move NR_NODE_MEMBLKS from asm-x86/acpi.h to xen/numa.h
>>
>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>> ---
>> xen/arch/Kconfig | 7 +++++++
>> xen/include/asm-x86/acpi.h | 1 -
>> xen/include/asm-x86/numa.h | 2 --
>> xen/include/xen/config.h | 1 +
>> xen/include/xen/numa.h | 7 ++-----
>> 5 files changed, 10 insertions(+), 8 deletions(-)
>>
>> diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig
>> index cf0acb7..9c2a4e2 100644
>> --- a/xen/arch/Kconfig
>> +++ b/xen/arch/Kconfig
>> @@ -6,3 +6,10 @@ config NR_CPUS
>> default "128" if ARM
>> ---help---
>> Specifies the maximum number of physical CPUs which Xen will support.
>> +
>> +config NR_NODES
>> + int "Maximum number of NUMA nodes"
>> + default "64" if X86
>> + default "8" if ARM
>> + ---help---
>> + Specifies the maximum number of NUMA nodes which Xen will support.
>
> Since this can now be specified by user but the definition of
> NUMA_NO_NODE is not changed, I think you need to sanitise the value
> provided somewhere.
>
> Maybe introduce a build time check? There are some examples in tree. See
> cpuid.c:build_assertions.
You can do bound-checking in Kconfig:
range 1 254
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable
2017-07-18 11:41 ` [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable vijay.kilari
2017-07-18 15:29 ` Wei Liu
@ 2017-07-18 17:55 ` Julien Grall
2017-07-19 7:00 ` Vijay Kilari
1 sibling, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-18 17:55 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Introduce NR_NODES config option to specify number
> of NUMA nodes supported. By default value is set at
> 64 for x86 and 8 for arm. Dropped NODES_SHIFT macro.
>
> Also move NR_NODE_MEMBLKS from asm-x86/acpi.h to xen/numa.h
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> xen/arch/Kconfig | 7 +++++++
> xen/include/asm-x86/acpi.h | 1 -
> xen/include/asm-x86/numa.h | 2 --
> xen/include/xen/config.h | 1 +
> xen/include/xen/numa.h | 7 ++-----
> 5 files changed, 10 insertions(+), 8 deletions(-)
>
> diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig
> index cf0acb7..9c2a4e2 100644
> --- a/xen/arch/Kconfig
> +++ b/xen/arch/Kconfig
> @@ -6,3 +6,10 @@ config NR_CPUS
> default "128" if ARM
> ---help---
> Specifies the maximum number of physical CPUs which Xen will support.
> +
> +config NR_NODES
> + int "Maximum number of NUMA nodes"
> + default "64" if X86
> + default "8" if ARM
3rd time I am asking it... Why the difference between x86 and ARM?
Also, you likely want to set to 1 if NUMA is not enabled.
> + ---help---
> + Specifies the maximum number of NUMA nodes which Xen will support.
> diff --git a/xen/include/asm-x86/acpi.h b/xen/include/asm-x86/acpi.h
> index 27ecc65..15be784 100644
> --- a/xen/include/asm-x86/acpi.h
> +++ b/xen/include/asm-x86/acpi.h
> @@ -105,7 +105,6 @@ extern void acpi_reserve_bootmem(void);
>
> extern s8 acpi_numa;
> extern int acpi_scan_nodes(u64 start, u64 end);
> -#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
>
> #ifdef CONFIG_ACPI_SLEEP
>
> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
> index bada2c0..3cf26c2 100644
> --- a/xen/include/asm-x86/numa.h
> +++ b/xen/include/asm-x86/numa.h
> @@ -3,8 +3,6 @@
>
> #include <xen/cpumask.h>
>
> -#define NODES_SHIFT 6
> -
> typedef u8 nodeid_t;
>
> extern int srat_rev;
> diff --git a/xen/include/xen/config.h b/xen/include/xen/config.h
> index a1d0f97..0f1a029 100644
> --- a/xen/include/xen/config.h
> +++ b/xen/include/xen/config.h
> @@ -81,6 +81,7 @@
>
> /* allow existing code to work with Kconfig variable */
> #define NR_CPUS CONFIG_NR_CPUS
> +#define NR_NODES CONFIG_NR_NODES
>
> #ifndef CONFIG_DEBUG
> #define NDEBUG
> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
> index 7aef1a8..6bba29e 100644
> --- a/xen/include/xen/numa.h
> +++ b/xen/include/xen/numa.h
> @@ -3,14 +3,11 @@
>
> #include <asm/numa.h>
>
> -#ifndef NODES_SHIFT
> -#define NODES_SHIFT 0
> -#endif
> -
> #define NUMA_NO_NODE 0xFF
> #define NUMA_NO_DISTANCE 0xFF
>
> -#define MAX_NUMNODES (1 << NODES_SHIFT)
> +#define MAX_NUMNODES NR_NODES
> +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
>
> #define vcpu_to_node(v) (cpu_to_node((v)->processor))
>
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 23/24] NUMA: Move CONFIG_NUMA to common Kconfig
2017-07-18 16:25 ` Julien Grall
@ 2017-07-18 18:00 ` Julien Grall
2017-07-28 10:08 ` Jan Beulich
1 sibling, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-18 18:00 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
On 18/07/17 17:25, Julien Grall wrote:
> Hi,
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> CONFIG_NUMA is defined in xen/drivers/acpi/Kconfig.
>> Move to common/Kconfig and enabled by default.
>> Also, NUMA feature uses PDX for physical address to
>> memory node mapping. Hence make HAS_PDX dependent
>> for NUMA.
>>
>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>> ---
>> xen/common/Kconfig | 4 ++++
>> xen/drivers/acpi/Kconfig | 3 ---
>> 2 files changed, 4 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/common/Kconfig b/xen/common/Kconfig
>> index dc8e876..6e421c7 100644
>> --- a/xen/common/Kconfig
>> +++ b/xen/common/Kconfig
>> @@ -41,6 +41,10 @@ config HAS_GDBSX
>> config HAS_IOPORTS
>> bool
>>
>> +config NUMA
>> + def_bool y
>> + depends on HAS_PDX
>
> On previous version, Jan asked to remove the dependency on PDX. You said
> you will do it... So why it is not done?
>
>> +
>> config HAS_BUILD_ID
>> string
>> option env="XEN_HAS_BUILD_ID"
>> diff --git a/xen/drivers/acpi/Kconfig b/xen/drivers/acpi/Kconfig
>> index b64d373..488372f 100644
>> --- a/xen/drivers/acpi/Kconfig
>> +++ b/xen/drivers/acpi/Kconfig
>> @@ -4,6 +4,3 @@ config ACPI
>>
>> config ACPI_LEGACY_TABLES_LOOKUP
>> bool
>> -
>> -config NUMA
>> - bool
Also, you haven't addressed Jan comments here too:
"This makes clear that so far this is an option which architectures
are expected to select. I think we want it to remain that way, but
if we didn't you should remove the existing select(s)."
I would lean towards keeping selectable from the architecture KConfig.
This would allow a new architecture to not enable NUMA if not supported.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 07/24] ARM: NUMA: Add existing ARM numa code under CONFIG_NUMA
2017-07-18 11:41 ` [RFC PATCH v3 07/24] ARM: NUMA: Add existing ARM numa code under CONFIG_NUMA vijay.kilari
@ 2017-07-18 18:06 ` Julien Grall
2017-07-20 9:31 ` Vijay Kilari
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-18 18:06 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Right now CONFIG_NUMA is not enabled for ARM and
> existing code in asm-arm/numa.h is for !CONFIG_NUMA.
> Hence put this code under #ifndef CONFIG_NUMA.
>
> This help to make this changes work when CONFIG_NUMA
> is not enabled. Though CONFIG_NUMA is enabled by default,
> manually disabling this option is possible and compilation
> should go through. Hence kept the these changes under
> !CONFIG_NUMA.
This is still no true. It is not possible to disable CONFIG_NUMA from
the Kconfig unless you hack it (just tried it)...
As I said on v2, if you always enable NUMA why should we add code in Xen
that get rotten? Either you allow NUMA to be disabled by the user or you
drop this code.
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> v3: - Dropped NODE_SHIFT define
> ---
> xen/include/asm-arm/numa.h | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> index 53f99af..7f00a36 100644
> --- a/xen/include/asm-arm/numa.h
> +++ b/xen/include/asm-arm/numa.h
> @@ -3,6 +3,7 @@
>
> typedef uint8_t nodeid_t;
>
> +#ifndef CONFIG_NUMA
> /* Fake one node for now. See also node_online_map. */
> #define cpu_to_node(cpu) 0
> #define node_to_cpumask(node) (cpu_online_map)
> @@ -16,6 +17,7 @@ static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
> #define node_spanned_pages(nid) (total_pages)
> #define node_start_pfn(nid) (pdx_to_pfn(frametable_base_pdx))
> #define __node_distance(a, b) (20)
> +#endif /* CONFIG_NUMA */
>
> static inline unsigned int arch_get_dma_bitsize(void)
> {
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic
2017-07-18 15:29 ` Wei Liu
@ 2017-07-18 18:16 ` Julien Grall
2017-07-19 6:47 ` Vijay Kilari
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-18 18:16 UTC (permalink / raw)
To: Wei Liu, vijay.kilari
Cc: tim, kevin.tian, sstabellini, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, xen-devel, jbeulich, Vijaya Kumar K
Hi,
On 18/07/17 16:29, Wei Liu wrote:
> On Tue, Jul 18, 2017 at 05:11:30PM +0530, vijay.kilari@gmail.com wrote:
> [...]
>> diff --git a/xen/common/numa.c b/xen/common/numa.c
>> new file mode 100644
>> index 0000000..0381f1b
>> --- /dev/null
>> +++ b/xen/common/numa.c
>> @@ -0,0 +1,487 @@
>> +/*
>> + * Common NUMA handling functions for x86 and arm.
>> + * Original code extracted from arch/x86/numa.c
>> + *
>> + * This program is free software; you can redistribute it and/or
>> + * modify it under the terms and conditions of the GNU General Public
>> + * License, version 2, as published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/init.h>
>> +#include <xen/ctype.h>
>> +#include <xen/sched.h>
>> +#include <xen/nodemask.h>
>> +#include <xen/numa.h>
>> +#include <xen/keyhandler.h>
>> +#include <xen/time.h>
>> +#include <xen/smp.h>
>> +#include <xen/pfn.h>
>> +#include <xen/mm.h>
>> +#include <xen/softirq.h>
>> +#include <xen/string.h>
>> +#include <asm/acpi.h>
>> +
>
> Since you're moving code anyway, please sort the headers alphabetically.
>
>> +static int numa_setup(char *s);
>> +custom_param("numa", numa_setup);
>> +
>> +struct node_data node_data[MAX_NUMNODES];
>> +
>> +/* Mapping from pdx to node id */
>
> Is this comment applicable to ARM? Does arm has PDX?
Yes ARM has PDX. For new architecture we expect the code to provide
dummy helpers if they want to support NUMA.
>
>> +unsigned int memnode_shift;
>> +
>> +/*
>> + * In case of numa init failure or numa off,
>> + * memnode_shift is initialized to BITS_PER_LONG - 1. Hence allocate
>> + * memnodemap[] of BITS_PER_LONG.
>> + */
>> +static typeof(*memnodemap) _memnodemap[BITS_PER_LONG];
>> +unsigned long memnodemapsize;
>> +uint8_t *memnodemap;
>> +
>> +nodeid_t __read_mostly cpu_to_node[NR_CPUS] = {
>> + [0 ... NR_CPUS-1] = NUMA_NO_NODE
>> +};
>> +
>> +cpumask_t __read_mostly node_to_cpumask[MAX_NUMNODES];
>> +
>> +bool numa_off;
>> +s8 acpi_numa = 0;
>> +
>> +int srat_disabled(void)
>
> bool here.
>
> Should probably be done in a previous patch.
Actually, the previous version had srat_disabled return bool. I am aware
that Jan and I requested to keep acpi_numa as int, I didn't find any
request of keep moving srat_disabled to int. So can you explain why??
>
>> +
>> +void __init numa_init_array(void)
>> +{
>> + int rr, i;
>> +
>> + /* There are unfortunately some poorly designed mainboards around
>> + that only connect memory to a single CPU. This breaks the 1:1 cpu->node
>> + mapping. To avoid this fill in the mapping for all possible
>> + CPUs, as the number of CPUs is not known yet.
>> + We round robin the existing nodes. */
>
> Please fix the coding style issue here.
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 00/24] ARM: Add Xen NUMA support
2017-07-18 16:18 ` [RFC PATCH v3 00/24] ARM: Add Xen NUMA support Julien Grall
@ 2017-07-19 6:31 ` Vijay Kilari
2017-07-19 7:18 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-19 6:31 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 9:48 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi,
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>
>> This patch is tested on Thunderx platform.
>> No changes are made to x86 implementation only code is sanitized
>> and refactored. Hence only compilation tested for x86.
>>
>> This series is posted as RFC for the reason that it is not tested
>> on x86. Request some help from community in testing this series on x86.
>>
>> Code is shared at
>> https://github.com/vijaykilari/xen-numa/commits/rfc_v3
>
>
> Few months ago you sent a patch that was a pre-requisite for booting NUMA
> (see [1]). It has never been upstreamed, so is it still required?
yes, it is required and is merged
https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=c6fdc9696a6a6eac59bf9c81121d1f1cd5b88dcd
>
> Cheers,
>
> [1]
> https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg03823.html
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 18/24] ACPI: Refactor acpi SRAT and SLIT table handling code
2017-07-18 15:36 ` Wei Liu
@ 2017-07-19 6:33 ` Vijay Kilari
0 siblings, 0 replies; 109+ messages in thread
From: Vijay Kilari @ 2017-07-19 6:33 UTC (permalink / raw)
To: Wei Liu
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, George Dunlap,
Andrew Cooper, Dario Faggioli, Ian Jackson, xen-devel,
Julien Grall, Jan Beulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 9:06 PM, Wei Liu <wei.liu2@citrix.com> wrote:
> On Tue, Jul 18, 2017 at 05:11:40PM +0530, vijay.kilari@gmail.com wrote:
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> Move SRAT handling code which is common across
>> architectures is moved to new file xen/drivers/acpi/srat.c
>> from xen/arch/x86/srat.c file. New header file srat.h is
>> introduced.
>>
>> Other major changes are:
>> - Coding style of code moved is changed.
>> - Moved struct pxm2node from srat.c to srat.h
>> - Dropped {memory,processor}_nodes_parsed from x86/srat.c
>> - Dropped static on node_to_pxm() and moved to beginning of the file.
>> - Made some static functions as non-static
>> - acpi_node_distance() is introduced and called from __node_distance()
>> - Replaced distance constants with LOCAL/REMOTE_DISTANCE defines
>
> It would be nice if you could break these into individual patches.
Ok. I will split.
>
> [...]
>> +
>> +/*
>> + * A lot of BIOS fill in 10 (= no distance) everywhere. This messes
>> + * up the NUMA heuristics which wants the local node to have a smaller
>> + * distance than the others.
>> + * Do some quick checks here and only use the SLIT if it passes.
>> + */
>> +static int __init slit_valid(struct acpi_table_slit *slit)
>> +{
>> + int i, j;
>
> unsigned int
ok
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 05/24] x86: NUMA: Add accessors for nodes[] and node_memblk_range[] structs
2017-07-18 15:29 ` Wei Liu
@ 2017-07-19 6:40 ` Vijay Kilari
2017-07-19 17:18 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-19 6:40 UTC (permalink / raw)
To: Wei Liu
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, George Dunlap,
Andrew Cooper, Dario Faggioli, Ian Jackson, xen-devel,
Julien Grall, Jan Beulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 8:59 PM, Wei Liu <wei.liu2@citrix.com> wrote:
> On Tue, Jul 18, 2017 at 05:11:27PM +0530, vijay.kilari@gmail.com wrote:
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> Add accessors for nodes[] and other static variables and
>> use those accessors. These variables are later accessed
>> outside the file when the code made generic in later
>> patches. However the coding style is not changed.
>>
>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>> ---
>> v3: - Changed accessors parameter from int to unsigned int
>> - Updated commit message
>> - Fixed wrong indentation
>> ---
>> xen/arch/x86/srat.c | 106 +++++++++++++++++++++++++++++++++++++++-------------
>> 1 file changed, 81 insertions(+), 25 deletions(-)
>>
>> diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
>> index 535c9d7..42cca5a 100644
>> --- a/xen/arch/x86/srat.c
>> +++ b/xen/arch/x86/srat.c
>> @@ -41,6 +41,44 @@ static struct node node_memblk_range[NR_NODE_MEMBLKS];
>> static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
>> static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
>>
>> +static struct node *get_numa_node(unsigned int id)
>> +{
>> + return &nodes[id];
>> +}
>> +
>> +static nodeid_t get_memblk_nodeid(unsigned int id)
>> +{
>> + return memblk_nodeid[id];
>> +}
>> +
>> +static nodeid_t *get_memblk_nodeid_map(void)
>> +{
>> + return &memblk_nodeid[0];
>> +}
>> +
>> +static struct node *get_node_memblk_range(unsigned int memblk)
>> +{
>> + return &node_memblk_range[memblk];
>> +}
>> +
>> +static int get_num_node_memblks(void)
>> +{
>> + return num_node_memblks;
>> +}
>
> They should all be inline functions. And maybe at once lift to a header
> and add proper prefix since you mention they are going to be used later.
Currently these are static variables in x86/srat.c file.
In patch #9 I move them to common/numa.c file and make these functions
non-static.
If I lift them to header file and make inline, then I have to make these as
global variables.
Regards
Vijay
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic
2017-07-18 18:16 ` Julien Grall
@ 2017-07-19 6:47 ` Vijay Kilari
0 siblings, 0 replies; 109+ messages in thread
From: Vijay Kilari @ 2017-07-19 6:47 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 11:46 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi,
>
>
> On 18/07/17 16:29, Wei Liu wrote:
>>
>> On Tue, Jul 18, 2017 at 05:11:30PM +0530, vijay.kilari@gmail.com wrote:
>> [...]
>>>
>>> diff --git a/xen/common/numa.c b/xen/common/numa.c
>>> new file mode 100644
>>> index 0000000..0381f1b
>>> --- /dev/null
>>> +++ b/xen/common/numa.c
>>> @@ -0,0 +1,487 @@
>>> +/*
>>> + * Common NUMA handling functions for x86 and arm.
>>> + * Original code extracted from arch/x86/numa.c
>>> + *
>>> + * This program is free software; you can redistribute it and/or
>>> + * modify it under the terms and conditions of the GNU General Public
>>> + * License, version 2, as published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <xen/init.h>
>>> +#include <xen/ctype.h>
>>> +#include <xen/sched.h>
>>> +#include <xen/nodemask.h>
>>> +#include <xen/numa.h>
>>> +#include <xen/keyhandler.h>
>>> +#include <xen/time.h>
>>> +#include <xen/smp.h>
>>> +#include <xen/pfn.h>
>>> +#include <xen/mm.h>
>>> +#include <xen/softirq.h>
>>> +#include <xen/string.h>
>>> +#include <asm/acpi.h>
>>> +
>>
>>
>> Since you're moving code anyway, please sort the headers alphabetically.
>>
>>> +static int numa_setup(char *s);
>>> +custom_param("numa", numa_setup);
>>> +
>>> +struct node_data node_data[MAX_NUMNODES];
>>> +
>>> +/* Mapping from pdx to node id */
>>
>>
>> Is this comment applicable to ARM? Does arm has PDX?
>
>
> Yes ARM has PDX. For new architecture we expect the code to provide dummy
> helpers if they want to support NUMA.
>
>>
>>> +unsigned int memnode_shift;
>>> +
>>> +/*
>>> + * In case of numa init failure or numa off,
>>> + * memnode_shift is initialized to BITS_PER_LONG - 1. Hence allocate
>>> + * memnodemap[] of BITS_PER_LONG.
>>> + */
>>> +static typeof(*memnodemap) _memnodemap[BITS_PER_LONG];
>>> +unsigned long memnodemapsize;
>>> +uint8_t *memnodemap;
>>> +
>>> +nodeid_t __read_mostly cpu_to_node[NR_CPUS] = {
>>> + [0 ... NR_CPUS-1] = NUMA_NO_NODE
>>> +};
>>> +
>>> +cpumask_t __read_mostly node_to_cpumask[MAX_NUMNODES];
>>> +
>>> +bool numa_off;
>>> +s8 acpi_numa = 0;
>>> +
>>> +int srat_disabled(void)
>>
>>
>> bool here.
>>
>> Should probably be done in a previous patch.
>
>
> Actually, the previous version had srat_disabled return bool. I am aware
> that Jan and I requested to keep acpi_numa as int, I didn't find any request
> of keep moving srat_disabled to int. So can you explain why??
My bad. I dropped patch #4 from v2. But this change was part of patch
#4 and missed it out.
>
>>
>>> +
>>> +void __init numa_init_array(void)
>>> +{
>>> + int rr, i;
>>> +
>>> + /* There are unfortunately some poorly designed mainboards around
>>> + that only connect memory to a single CPU. This breaks the 1:1
>>> cpu->node
>>> + mapping. To avoid this fill in the mapping for all possible
>>> + CPUs, as the number of CPUs is not known yet.
>>> + We round robin the existing nodes. */
>>
>>
>> Please fix the coding style issue here.
>>
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable
2017-07-18 17:55 ` Julien Grall
@ 2017-07-19 7:00 ` Vijay Kilari
2017-07-19 15:55 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-19 7:00 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 11:25 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi,
>
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> Introduce NR_NODES config option to specify number
>> of NUMA nodes supported. By default value is set at
>> 64 for x86 and 8 for arm. Dropped NODES_SHIFT macro.
>>
>> Also move NR_NODE_MEMBLKS from asm-x86/acpi.h to xen/numa.h
>>
>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>> ---
>> xen/arch/Kconfig | 7 +++++++
>> xen/include/asm-x86/acpi.h | 1 -
>> xen/include/asm-x86/numa.h | 2 --
>> xen/include/xen/config.h | 1 +
>> xen/include/xen/numa.h | 7 ++-----
>> 5 files changed, 10 insertions(+), 8 deletions(-)
>>
>> diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig
>> index cf0acb7..9c2a4e2 100644
>> --- a/xen/arch/Kconfig
>> +++ b/xen/arch/Kconfig
>> @@ -6,3 +6,10 @@ config NR_CPUS
>> default "128" if ARM
>> ---help---
>> Specifies the maximum number of physical CPUs which Xen will
>> support.
>> +
>> +config NR_NODES
>> + int "Maximum number of NUMA nodes"
>> + default "64" if X86
>> + default "8" if ARM
>
>
> 3rd time I am asking it... Why the difference between x86 and ARM?
AFAIK, there is no arm platform for now with numa more than 8 nodes.
Thunderx is only 2 nodes.
So kept it low value for ARM to avoid unnecessary memory allocation.
Do you want me to keep same as x86?.
>
> Also, you likely want to set to 1 if NUMA is not enabled.
I don't see any dependency of NR_NODES with NUMA config.
So it is always set to default value. Isn't?
>
>
>> + ---help---
>> + Specifies the maximum number of NUMA nodes which Xen will
>> support.
>> diff --git a/xen/include/asm-x86/acpi.h b/xen/include/asm-x86/acpi.h
>> index 27ecc65..15be784 100644
>> --- a/xen/include/asm-x86/acpi.h
>> +++ b/xen/include/asm-x86/acpi.h
>> @@ -105,7 +105,6 @@ extern void acpi_reserve_bootmem(void);
>>
>> extern s8 acpi_numa;
>> extern int acpi_scan_nodes(u64 start, u64 end);
>> -#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
>>
>> #ifdef CONFIG_ACPI_SLEEP
>>
>> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
>> index bada2c0..3cf26c2 100644
>> --- a/xen/include/asm-x86/numa.h
>> +++ b/xen/include/asm-x86/numa.h
>> @@ -3,8 +3,6 @@
>>
>> #include <xen/cpumask.h>
>>
>> -#define NODES_SHIFT 6
>> -
>> typedef u8 nodeid_t;
>>
>> extern int srat_rev;
>> diff --git a/xen/include/xen/config.h b/xen/include/xen/config.h
>> index a1d0f97..0f1a029 100644
>> --- a/xen/include/xen/config.h
>> +++ b/xen/include/xen/config.h
>> @@ -81,6 +81,7 @@
>>
>> /* allow existing code to work with Kconfig variable */
>> #define NR_CPUS CONFIG_NR_CPUS
>> +#define NR_NODES CONFIG_NR_NODES
>>
>> #ifndef CONFIG_DEBUG
>> #define NDEBUG
>> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
>> index 7aef1a8..6bba29e 100644
>> --- a/xen/include/xen/numa.h
>> +++ b/xen/include/xen/numa.h
>> @@ -3,14 +3,11 @@
>>
>> #include <asm/numa.h>
>>
>> -#ifndef NODES_SHIFT
>> -#define NODES_SHIFT 0
>> -#endif
>> -
>> #define NUMA_NO_NODE 0xFF
>> #define NUMA_NO_DISTANCE 0xFF
>>
>> -#define MAX_NUMNODES (1 << NODES_SHIFT)
>> +#define MAX_NUMNODES NR_NODES
>> +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
>>
>> #define vcpu_to_node(v) (cpu_to_node((v)->processor))
>>
>>
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 00/24] ARM: Add Xen NUMA support
2017-07-19 6:31 ` Vijay Kilari
@ 2017-07-19 7:18 ` Julien Grall
[not found] ` <CALicx6svuo3JXik=8bYuciFzWDu6qmwVi1VXdBgjLp_f_YUhqQ@mail.gmail.com>
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-19 7:18 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K, nd
On 19/07/2017 07:31, Vijay Kilari wrote:
> On Tue, Jul 18, 2017 at 9:48 PM, Julien Grall <julien.grall@arm.com> wrote:
>> Hi,
>>
>> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>>
>>> This patch is tested on Thunderx platform.
>>> No changes are made to x86 implementation only code is sanitized
>>> and refactored. Hence only compilation tested for x86.
>>>
>>> This series is posted as RFC for the reason that it is not tested
>>> on x86. Request some help from community in testing this series on x86.
>>>
>>> Code is shared at
>>> https://github.com/vijaykilari/xen-numa/commits/rfc_v3
>>
>>
>> Few months ago you sent a patch that was a pre-requisite for booting NUMA
>> (see [1]). It has never been upstreamed, so is it still required?
>
> yes, it is required and is merged
Hmmm, somehow I was not able to find it in the tree yesterday. Found it
know.
Cheers,
>
> https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=c6fdc9696a6a6eac59bf9c81121d1f1cd5b88dcd
>
>>
>> Cheers,
>>
>> [1]
>> https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg03823.html
>>
>> --
>> Julien Grall
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable
2017-07-18 17:52 ` Julien Grall
@ 2017-07-19 8:17 ` Wei Liu
2017-07-19 15:48 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Wei Liu @ 2017-07-19 8:17 UTC (permalink / raw)
To: Julien Grall
Cc: kevin.tian, sstabellini, Wei Liu, vijay.kilari, George.Dunlap,
andrew.cooper3, dario.faggioli, tim, xen-devel, jbeulich,
Vijaya Kumar K, ian.jackson
On Tue, Jul 18, 2017 at 06:52:11PM +0100, Julien Grall wrote:
> Hi,
>
> On 18/07/17 16:29, Wei Liu wrote:
> > On Tue, Jul 18, 2017 at 05:11:23PM +0530, vijay.kilari@gmail.com wrote:
> > > From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> > >
> > > Introduce NR_NODES config option to specify number
> > > of NUMA nodes supported. By default value is set at
> > > 64 for x86 and 8 for arm. Dropped NODES_SHIFT macro.
> > >
> > > Also move NR_NODE_MEMBLKS from asm-x86/acpi.h to xen/numa.h
> > >
> > > Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> > > ---
> > > xen/arch/Kconfig | 7 +++++++
> > > xen/include/asm-x86/acpi.h | 1 -
> > > xen/include/asm-x86/numa.h | 2 --
> > > xen/include/xen/config.h | 1 +
> > > xen/include/xen/numa.h | 7 ++-----
> > > 5 files changed, 10 insertions(+), 8 deletions(-)
> > >
> > > diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig
> > > index cf0acb7..9c2a4e2 100644
> > > --- a/xen/arch/Kconfig
> > > +++ b/xen/arch/Kconfig
> > > @@ -6,3 +6,10 @@ config NR_CPUS
> > > default "128" if ARM
> > > ---help---
> > > Specifies the maximum number of physical CPUs which Xen will support.
> > > +
> > > +config NR_NODES
> > > + int "Maximum number of NUMA nodes"
> > > + default "64" if X86
> > > + default "8" if ARM
> > > + ---help---
> > > + Specifies the maximum number of NUMA nodes which Xen will support.
> >
> > Since this can now be specified by user but the definition of
> > NUMA_NO_NODE is not changed, I think you need to sanitise the value
> > provided somewhere.
> >
> > Maybe introduce a build time check? There are some examples in tree. See
> > cpuid.c:build_assertions.
>
> You can do bound-checking in Kconfig:
>
> range 1 254
>
Oh, good to know. Yes this is the way to go.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable
2017-07-19 8:17 ` Wei Liu
@ 2017-07-19 15:48 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-19 15:48 UTC (permalink / raw)
To: Wei Liu
Cc: kevin.tian, sstabellini, vijay.kilari, George.Dunlap,
andrew.cooper3, dario.faggioli, tim, xen-devel, jbeulich,
Vijaya Kumar K, ian.jackson
On 19/07/17 09:17, Wei Liu wrote:
> On Tue, Jul 18, 2017 at 06:52:11PM +0100, Julien Grall wrote:
>> Hi,
>>
>> On 18/07/17 16:29, Wei Liu wrote:
>>> On Tue, Jul 18, 2017 at 05:11:23PM +0530, vijay.kilari@gmail.com wrote:
>>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>>
>>>> Introduce NR_NODES config option to specify number
>>>> of NUMA nodes supported. By default value is set at
>>>> 64 for x86 and 8 for arm. Dropped NODES_SHIFT macro.
>>>>
>>>> Also move NR_NODE_MEMBLKS from asm-x86/acpi.h to xen/numa.h
>>>>
>>>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>> ---
>>>> xen/arch/Kconfig | 7 +++++++
>>>> xen/include/asm-x86/acpi.h | 1 -
>>>> xen/include/asm-x86/numa.h | 2 --
>>>> xen/include/xen/config.h | 1 +
>>>> xen/include/xen/numa.h | 7 ++-----
>>>> 5 files changed, 10 insertions(+), 8 deletions(-)
>>>>
>>>> diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig
>>>> index cf0acb7..9c2a4e2 100644
>>>> --- a/xen/arch/Kconfig
>>>> +++ b/xen/arch/Kconfig
>>>> @@ -6,3 +6,10 @@ config NR_CPUS
>>>> default "128" if ARM
>>>> ---help---
>>>> Specifies the maximum number of physical CPUs which Xen will support.
>>>> +
>>>> +config NR_NODES
>>>> + int "Maximum number of NUMA nodes"
>>>> + default "64" if X86
>>>> + default "8" if ARM
>>>> + ---help---
>>>> + Specifies the maximum number of NUMA nodes which Xen will support.
>>>
>>> Since this can now be specified by user but the definition of
>>> NUMA_NO_NODE is not changed, I think you need to sanitise the value
>>> provided somewhere.
>>>
>>> Maybe introduce a build time check? There are some examples in tree. See
>>> cpuid.c:build_assertions.
>>
>> You can do bound-checking in Kconfig:
>>
>> range 1 254
>>
>
> Oh, good to know. Yes this is the way to go.
(Not directed to you Wei :))
Actually looking again at Xen, we are moving away from a power of 2. So
what is the rationale behind that? Have you looked at why describing the
number of nodes in term of power of 2 was chosen?
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable
2017-07-19 7:00 ` Vijay Kilari
@ 2017-07-19 15:55 ` Julien Grall
2017-07-20 7:30 ` Vijay Kilari
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-19 15:55 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K, nd
Hi Vijay,
On 19/07/2017 08:00, Vijay Kilari wrote:
> On Tue, Jul 18, 2017 at 11:25 PM, Julien Grall <julien.grall@arm.com> wrote:
>> Hi,
>>
>>
>> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>>
>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>
>>> Introduce NR_NODES config option to specify number
>>> of NUMA nodes supported. By default value is set at
>>> 64 for x86 and 8 for arm. Dropped NODES_SHIFT macro.
>>>
>>> Also move NR_NODE_MEMBLKS from asm-x86/acpi.h to xen/numa.h
>>>
>>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>> ---
>>> xen/arch/Kconfig | 7 +++++++
>>> xen/include/asm-x86/acpi.h | 1 -
>>> xen/include/asm-x86/numa.h | 2 --
>>> xen/include/xen/config.h | 1 +
>>> xen/include/xen/numa.h | 7 ++-----
>>> 5 files changed, 10 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig
>>> index cf0acb7..9c2a4e2 100644
>>> --- a/xen/arch/Kconfig
>>> +++ b/xen/arch/Kconfig
>>> @@ -6,3 +6,10 @@ config NR_CPUS
>>> default "128" if ARM
>>> ---help---
>>> Specifies the maximum number of physical CPUs which Xen will
>>> support.
>>> +
>>> +config NR_NODES
>>> + int "Maximum number of NUMA nodes"
>>> + default "64" if X86
>>> + default "8" if ARM
>>
>>
>> 3rd time I am asking it... Why the difference between x86 and ARM?
>
> AFAIK, there is no arm platform for now with numa more than 8 nodes.
> Thunderx is only 2 nodes.
> So kept it low value for ARM to avoid unnecessary memory allocation.
>
> Do you want me to keep same as x86?.
Well, you say it is for saving memory allocation but you don't give any
number on how much you can save by reducing the default from 64 to 8...
Looking at it, MAX_NUMNODES is used for some static allocation and also
for the bitmap nodemask_t.
Because our bitmap is based on unsigned long, you would use the same
quantity of memory for AArch64, for AArch32 the quantity will be divided
by two. Still nodemask_t does not seem to be widely used.
In the case of the static allocation, I spot ~40 bytes per NUMA node. So
8 node will use ~320 bytes and 64 bytes ~2560.
NUMA is likely going to be used in server, don't tell me you are 2k
short in memory? If it is an issue it is better to think how to limit
the number of static variable rather than putting a low limit here.
For Embedded use case, they will likely want to put the default to 1 but
I would not worry about them as they are likely going to tweak the Kconfig.
>
>>
>> Also, you likely want to set to 1 if NUMA is not enabled.
>
> I don't see any dependency of NR_NODES with NUMA config.
> So it is always set to default value. Isn't?
Well, what is the point to allow more than 1 node when NUMA is not
supported?
Not mentioning that this is quite confusing for a user to allow setting
up the maximum number of nodes if the archicture is not supporting numa...
For instance, this is the case today on ARM because, without this
series, we don't support NUMA.
>
>>
>>
>>> + ---help---
>>> + Specifies the maximum number of NUMA nodes which Xen will
>>> support.
>>> diff --git a/xen/include/asm-x86/acpi.h b/xen/include/asm-x86/acpi.h
>>> index 27ecc65..15be784 100644
>>> --- a/xen/include/asm-x86/acpi.h
>>> +++ b/xen/include/asm-x86/acpi.h
>>> @@ -105,7 +105,6 @@ extern void acpi_reserve_bootmem(void);
>>>
>>> extern s8 acpi_numa;
>>> extern int acpi_scan_nodes(u64 start, u64 end);
>>> -#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
>>>
>>> #ifdef CONFIG_ACPI_SLEEP
>>>
>>> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
>>> index bada2c0..3cf26c2 100644
>>> --- a/xen/include/asm-x86/numa.h
>>> +++ b/xen/include/asm-x86/numa.h
>>> @@ -3,8 +3,6 @@
>>>
>>> #include <xen/cpumask.h>
>>>
>>> -#define NODES_SHIFT 6
>>> -
>>> typedef u8 nodeid_t;
>>>
>>> extern int srat_rev;
>>> diff --git a/xen/include/xen/config.h b/xen/include/xen/config.h
>>> index a1d0f97..0f1a029 100644
>>> --- a/xen/include/xen/config.h
>>> +++ b/xen/include/xen/config.h
>>> @@ -81,6 +81,7 @@
>>>
>>> /* allow existing code to work with Kconfig variable */
>>> #define NR_CPUS CONFIG_NR_CPUS
>>> +#define NR_NODES CONFIG_NR_NODES
>>>
>>> #ifndef CONFIG_DEBUG
>>> #define NDEBUG
>>> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
>>> index 7aef1a8..6bba29e 100644
>>> --- a/xen/include/xen/numa.h
>>> +++ b/xen/include/xen/numa.h
>>> @@ -3,14 +3,11 @@
>>>
>>> #include <asm/numa.h>
>>>
>>> -#ifndef NODES_SHIFT
>>> -#define NODES_SHIFT 0
>>> -#endif
>>> -
>>> #define NUMA_NO_NODE 0xFF
>>> #define NUMA_NO_DISTANCE 0xFF
>>>
>>> -#define MAX_NUMNODES (1 << NODES_SHIFT)
>>> +#define MAX_NUMNODES NR_NODES
>>> +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
Also, I don't understand why you move this define from asm-x86/numa.h to
xen/numa.h. At least, this does not seem related to this patch...
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code
2017-07-18 11:41 ` [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code vijay.kilari
@ 2017-07-19 16:23 ` Julien Grall
2017-07-19 16:27 ` Wei Liu
2017-07-20 7:00 ` Vijay Kilari
0 siblings, 2 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-19 16:23 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Fix coding style, trailing spaces, tabs in NUMA code.
> Also drop unused macros and functions.
> There is no functional change.
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> Reviewed-by: Wei Liu <wei.liu2@citrix.com>
> ---
> v3: - Change commit message
> - Changed VIRTUAL_BUG_ON to ASSERT
Looking at the commit message you don't mention any renaming...
> - Dropped useless inner paranthesis for some macros
[...]
> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
> index 3cf26c2..c0de57b 100644
> --- a/xen/include/asm-x86/numa.h
> +++ b/xen/include/asm-x86/numa.h
> @@ -1,8 +1,11 @@
> -#ifndef _ASM_X8664_NUMA_H
> +#ifndef _ASM_X8664_NUMA_H
> #define _ASM_X8664_NUMA_H 1
>
> #include <xen/cpumask.h>
>
> +#define MAX_NUMNODES NR_NODES
> +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
I don't understand why this suddenly appears in the code when you moved
away in patch #1 in xen/numa.h.
[...]
> @@ -57,21 +55,23 @@ struct node_data {
>
> extern struct node_data node_data[];
>
> -static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
> -{
> - nodeid_t nid;
> - VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >= memnodemapsize);
> - nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
> - VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]);
> - return nid;
> -}
> -
> -#define NODE_DATA(nid) (&(node_data[nid]))
> -
> -#define node_start_pfn(nid) (NODE_DATA(nid)->node_start_pfn)
> -#define node_spanned_pages(nid) (NODE_DATA(nid)->node_spanned_pages)
> -#define node_end_pfn(nid) (NODE_DATA(nid)->node_start_pfn + \
> - NODE_DATA(nid)->node_spanned_pages)
> +static inline __attribute_pure__ nodeid_t phys_to_nid(paddr_t addr)
> +{
> + nodeid_t nid;
> +
> + ASSERT((paddr_to_pdx(addr) >> memnode_shift) < memnodemapsize);
> + nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
> + ASSERT(nid <= MAX_NUMNODES || !node_data[nid].node_start_pfn);
> +
> + return nid;
> +}
> +
> +#define NODE_DATA(nid) (&(node_data[nid]))
I understand Jan asked to remove the inner parentheses here. And you
didn't do it. However ...
> +
> +#define node_start_pfn(nid) NODE_DATA(nid)->node_start_pfn
> +#define node_spanned_pages(nid) NODE_DATA(nid)->node_spanned_pages
> +#define node_end_pfn(nid) NODE_DATA(nid)->node_start_pfn + \
> + NODE_DATA(nid)->node_spanned_pages
... here it is totally wrong to remove the parenthesis. Imagine you do:
node_end_pfn(nid) * 2
This will now turned into
NODE_DATA(nid)->node_start_pfn + NODE_DATA(nid)->node_spanned_pages * 2
The parenthesis is not correct anymore and will result to wrong
computation. You should keep the outer parenthesis *everywhere* for
safety and remove only the inner one in NODE_DATA.
This is also more than cosmetics and I think the reviewed-by from Wei
should have been carried.
>
> extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
>
> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
> index 6bba29e..3bb4afc 100644
> --- a/xen/include/xen/numa.h
> +++ b/xen/include/xen/numa.h
> @@ -6,9 +6,6 @@
> #define NUMA_NO_NODE 0xFF
> #define NUMA_NO_DISTANCE 0xFF
>
> -#define MAX_NUMNODES NR_NODES
> -#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
> -
See my comment above.
> #define vcpu_to_node(v) (cpu_to_node((v)->processor))
>
> #define domain_to_node(d) \
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code
2017-07-19 16:23 ` Julien Grall
@ 2017-07-19 16:27 ` Wei Liu
2017-07-19 16:34 ` Julien Grall
2017-07-20 7:00 ` Vijay Kilari
1 sibling, 1 reply; 109+ messages in thread
From: Wei Liu @ 2017-07-19 16:27 UTC (permalink / raw)
To: Julien Grall
Cc: tim, kevin.tian, sstabellini, wei.liu2, vijay.kilari,
George.Dunlap, andrew.cooper3, dario.faggioli, ian.jackson,
xen-devel, jbeulich, Vijaya Kumar K
On Wed, Jul 19, 2017 at 05:23:43PM +0100, Julien Grall wrote:
>
> This is also more than cosmetics and I think the reviewed-by from Wei should
> have been carried.
should *not* have been carried.
And I agree.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code
2017-07-19 16:27 ` Wei Liu
@ 2017-07-19 16:34 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-19 16:34 UTC (permalink / raw)
To: Wei Liu
Cc: tim, kevin.tian, sstabellini, vijay.kilari, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel, jbeulich,
Vijaya Kumar K
Hi,
On 19/07/17 17:27, Wei Liu wrote:
> On Wed, Jul 19, 2017 at 05:23:43PM +0100, Julien Grall wrote:
>>
>> This is also more than cosmetics and I think the reviewed-by from Wei should
>> have been carried.
>
> should *not* have been carried.
That's what I meant but failed to write the not.
>
> And I agree.
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 04/24] x86: NUMA: Rename and sanitize memnode shift code
2017-07-18 11:41 ` [RFC PATCH v3 04/24] x86: NUMA: Rename and sanitize memnode shift code vijay.kilari
2017-07-18 15:29 ` Wei Liu
@ 2017-07-19 17:12 ` Julien Grall
2017-07-20 6:56 ` Vijay Kilari
1 sibling, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-19 17:12 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> memnode_shift variable is changed from int to unsigned int.
> With this change, compute_memnode_shift() returns error value
> instead of returning shift value. The memnode_shift is updated inside
> compute_memnode_shift().
>
> Also, following changes are made
> - Rename compute_hash_shift to compute_memnode_shift
> - Update int to unsigned int for params in extract_lsb_from_nodes()
> - Return values of populate_memnodemap() is changed
I am not sure to understand the rationale behind changing the return
value of populate_memnodemap. Likely this mean a bit more description in
the commit message.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 05/24] x86: NUMA: Add accessors for nodes[] and node_memblk_range[] structs
2017-07-19 6:40 ` Vijay Kilari
@ 2017-07-19 17:18 ` Julien Grall
2017-07-20 7:41 ` Vijay Kilari
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-19 17:18 UTC (permalink / raw)
To: Vijay Kilari, Wei Liu
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, George Dunlap,
Andrew Cooper, Dario Faggioli, Ian Jackson, xen-devel,
Jan Beulich, Vijaya Kumar K
On 19/07/17 07:40, Vijay Kilari wrote:
> On Tue, Jul 18, 2017 at 8:59 PM, Wei Liu <wei.liu2@citrix.com> wrote:
>> On Tue, Jul 18, 2017 at 05:11:27PM +0530, vijay.kilari@gmail.com wrote:
>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>
>>> Add accessors for nodes[] and other static variables and
>>> use those accessors. These variables are later accessed
>>> outside the file when the code made generic in later
>>> patches. However the coding style is not changed.
>>>
>>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>> ---
>>> v3: - Changed accessors parameter from int to unsigned int
>>> - Updated commit message
>>> - Fixed wrong indentation
>>> ---
>>> xen/arch/x86/srat.c | 106 +++++++++++++++++++++++++++++++++++++++-------------
>>> 1 file changed, 81 insertions(+), 25 deletions(-)
>>>
>>> diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
>>> index 535c9d7..42cca5a 100644
>>> --- a/xen/arch/x86/srat.c
>>> +++ b/xen/arch/x86/srat.c
>>> @@ -41,6 +41,44 @@ static struct node node_memblk_range[NR_NODE_MEMBLKS];
>>> static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
>>> static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
>>>
>>> +static struct node *get_numa_node(unsigned int id)
>>> +{
>>> + return &nodes[id];
>>> +}
>>> +
>>> +static nodeid_t get_memblk_nodeid(unsigned int id)
>>> +{
>>> + return memblk_nodeid[id];
>>> +}
>>> +
>>> +static nodeid_t *get_memblk_nodeid_map(void)
>>> +{
>>> + return &memblk_nodeid[0];
>>> +}
>>> +
>>> +static struct node *get_node_memblk_range(unsigned int memblk)
>>> +{
>>> + return &node_memblk_range[memblk];
>>> +}
>>> +
>>> +static int get_num_node_memblks(void)
>>> +{
>>> + return num_node_memblks;
>>> +}
>>
>> They should all be inline functions. And maybe at once lift to a header
>> and add proper prefix since you mention they are going to be used later.
>
> Currently these are static variables in x86/srat.c file.
> In patch #9 I move them to common/numa.c file and make these functions
> non-static.
>
> If I lift them to header file and make inline, then I have to make these as
> global variables.
As I said on v2, I am not sure to understand the usefulness of those
accessors over global variables...
You don't have any kind of sanity check, so they would do exactly the
same job. The global variables would avoid so much churn.
More that you tend to sometimes use global and other time static helpers...
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 06/24] x86: NUMA: Rename some generic functions
2017-07-18 11:41 ` [RFC PATCH v3 06/24] x86: NUMA: Rename some generic functions vijay.kilari
@ 2017-07-19 17:23 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-19 17:23 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Rename some function in ACPI code as follow
> - Rename setup_node to acpi_setup_node
> - Rename bad_srat to numa_failed
> - Rename nodes_cover_memory to arch_sanitize_nodes_memory
> and changed return type to bool
> - Rename acpi_scan_nodes to numa_scan_nodes
>
> Also introduce reset_pxm2node() to reset pxm2node variable.
> This avoids exporting pxm2node.
This does not belong to this patch.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic
2017-07-18 11:41 ` [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic vijay.kilari
2017-07-18 15:29 ` Wei Liu
@ 2017-07-19 17:41 ` Julien Grall
2017-07-20 8:55 ` Vijay Kilari
2017-07-24 20:28 ` Stefano Stabellini
1 sibling, 2 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-19 17:41 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Move code from xen/arch/x86/numa.c to xen/common/numa.c
> so that it can be used by other archs.
>
> The following changes are done:
> - Few generic static functions in x86/numa.c is made
> non-static common/numa.c
> - The generic contents of header file asm-x86/numa.h
> are moved to xen/numa.h.
> - The header file includes are reordered and externs are
> dropped.
> - Moved acpi_numa from asm-x86/acpi.h to xen/acpi.h
> - Coding style of code moved to commom/numa.c is changed
> to Xen style.
> - numa_add_cpu() and numa_set_node() and moved to header
> file and added inline function in case of CONFIG_NUMA
> is not enabled because these functions are called from
> generic code with out any config check.
>
> Also the node_online_map is defined in x86/numa.c for x86
> and arm/smpboot.c for ARM. For x86 it is moved to x86/smpboot.c
> If moved to common code the compilation fails because
> common/numa.c is compiled only when NUMA is enabled.
I would much prefer if this patch does one thing: Moving code. The rest
should be split out to help review and allowing us to easily verify you
only moved code...
> +#define NODE_DATA(nid) (&(node_data[nid]))
> +
> +#define node_start_pfn(nid) NODE_DATA(nid)->node_start_pfn
> +#define node_spanned_pages(nid) NODE_DATA(nid)->node_spanned_pages
> +#define node_end_pfn(nid) NODE_DATA(nid)->node_start_pfn + \
> + NODE_DATA(nid)->node_spanned_pages
> +
> +void numa_add_cpu(int cpu);
> +void numa_set_node(int cpu, nodeid_t node);
> +#else
> +static inline void numa_add_cpu(int cpu) { }
> +static inline void numa_set_node(int cpu, nodeid_t node) { }
I am not sure why you need to define stub at least for numa_set_node...
I can't see use in non-NUMA code. I will comment about the numa_add_cpu
later.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 10/24] NUMA: Allow numa initialization with DT
2017-07-18 11:41 ` [RFC PATCH v3 10/24] NUMA: Allow numa initialization with DT vijay.kilari
@ 2017-07-19 17:58 ` Julien Grall
2017-07-20 10:28 ` Vijay Kilari
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-19 17:58 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> The common code allows numa initialization only when
> ACPI_NUMA config is enabled. Allow initialization when
> NUMA config is enabled for DT.
>
> In this patch, along with acpi_numa, check for acpi_disabled
> is added.
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> xen/common/numa.c | 4 +---
> 1 file changed, 1 insertion(+), 3 deletions(-)
>
> diff --git a/xen/common/numa.c b/xen/common/numa.c
> index 74c4697..5e985d2 100644
> --- a/xen/common/numa.c
> +++ b/xen/common/numa.c
> @@ -324,7 +324,7 @@ static int __init numa_scan_nodes(paddr_t start, paddr_t end)
> for ( i = 0; i < MAX_NUMNODES; i++ )
> cutoff_node(i, start, end);
>
> - if ( acpi_numa <= 0 )
> + if ( !acpi_disabled && acpi_numa <= 0 )
I am struggling to understand this change. Likely you want to similar
variable for DT to say NUMA is available or this has failed.
This also change quite a bit the semantic for x86 because, you will now
continue if acpi_disabled and acpi_numa = 0. The code seems to allow it,
but I don't know if we support it.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 12/24] ARM: NUMA: DT: Parse CPU NUMA information
2017-07-18 11:41 ` [RFC PATCH v3 12/24] ARM: NUMA: DT: Parse CPU NUMA information vijay.kilari
@ 2017-07-19 18:26 ` Julien Grall
2017-07-20 9:20 ` Vijay Kilari
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-19 18:26 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Parse CPU node and fetch numa-node-id information.
> For each node-id found, update nodemask_t mask.
> Refer to Documentation/devicetree/bindings/numa.txt
> in linux kernel.
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> v3: - Parse cpu nodes under path /cpus
> - Move changes to bootfdt.c as separate patch
> - Set numa_off on dt_numa_init() failure
> ---
> xen/arch/arm/Makefile | 1 +
> xen/arch/arm/numa/Makefile | 2 ++
> xen/arch/arm/numa/dt_numa.c | 77 +++++++++++++++++++++++++++++++++++++++++++++
> xen/arch/arm/numa/numa.c | 48 ++++++++++++++++++++++++++++
> xen/arch/arm/setup.c | 4 +++
> xen/include/asm-arm/numa.h | 10 +++++-
> 6 files changed, 141 insertions(+), 1 deletion(-)
>
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 49e1fb2..a89be66 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -3,6 +3,7 @@ subdir-$(CONFIG_ARM_64) += arm64
> subdir-y += platforms
> subdir-$(CONFIG_ARM_64) += efi
> subdir-$(CONFIG_ACPI) += acpi
> +subdir-$(CONFIG_NUMA) += numa
>
> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
> obj-y += bootfdt.init.o
> diff --git a/xen/arch/arm/numa/Makefile b/xen/arch/arm/numa/Makefile
> new file mode 100644
> index 0000000..3af3aff
> --- /dev/null
> +++ b/xen/arch/arm/numa/Makefile
> @@ -0,0 +1,2 @@
> +obj-y += dt_numa.o
> +obj-y += numa.o
> diff --git a/xen/arch/arm/numa/dt_numa.c b/xen/arch/arm/numa/dt_numa.c
> new file mode 100644
> index 0000000..963bb40
> --- /dev/null
> +++ b/xen/arch/arm/numa/dt_numa.c
> @@ -0,0 +1,77 @@
> +/*
> + * OF NUMA Parsing support.
> + *
> + * Copyright (C) 2015 - 2016 Cavium Inc.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/mm.h>
> +#include <xen/nodemask.h>
> +#include <xen/libfdt/libfdt.h>
> +#include <xen/device_tree.h>
Again, this include should not be there as the device tree is not yet
parsed.
> +#include <xen/numa.h>
> +#include <asm/setup.h>
Again, please order in alphabetically the includes...
> +
> +/*
> + * Even though we connect cpus to numa domains later in SMP
> + * init, we need to know the node ids now for all cpus.
> + */
> +static int __init dt_numa_process_cpu_node(const void *fdt)
> +{
> + int node, offset;
> + uint32_t nid;
> +
> + offset = fdt_path_offset(fdt, "/cpus");
> + if ( offset < 0 )
> + return -EINVAL;
> +
> + node = fdt_first_subnode(fdt, offset);
> + if ( node == -FDT_ERR_NOTFOUND )
> + return -EINVAL;
> +
> + do {
> + if ( device_tree_type_matches(fdt, node, "cpu") )
> + {
> + nid = device_tree_get_u32(fdt, node, "numa-node-id", MAX_NUMNODES);
> + if ( nid >= MAX_NUMNODES )
> + printk(XENLOG_WARNING
> + "NUMA: Node id %u exceeds maximum value\n", nid);
> + else
> + node_set(nid, processor_nodes_parsed);
> + }
> +
> + offset = node;
> + node = fdt_next_subnode(fdt, offset);
> + } while (node != -FDT_ERR_NOTFOUND);
> +
> + return 0;
> +}
> +
> +int __init dt_numa_init(void)
> +{
> + int ret;
> +
> + ret = dt_numa_process_cpu_node((void *)device_tree_flattened);
> +
> + return ret;
return dt_numa_process_cpu_node(....);
But I am still not sure to understand why you can't parse the numa node
in directly in bootfdt.c as you do for the memory.
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
> new file mode 100644
> index 0000000..45cc418
> --- /dev/null
> +++ b/xen/arch/arm/numa/numa.c
> @@ -0,0 +1,48 @@
> +/*
> + * ARM NUMA Implementation
> + *
> + * Copyright (C) 2016 - Cavium Inc.
> + * Vijaya Kumar K <vijaya.kumar@cavium.com>
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms and conditions of the GNU General Public
> + * License, version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/ctype.h>
> +#include <xen/nodemask.h>
> +#include <xen/numa.h>
> +
> +void __init numa_init(void)
> +{
> + int ret = 0;
> +
> + nodes_clear(processor_nodes_parsed);
Why do you need to clear processor_nodes_parsed? It should already be
all zeroed.
> + if ( numa_off )
> + goto no_numa;
> +
> + ret = dt_numa_init();
> + if ( ret )
> + {
> + numa_off = true;
> + printk(XENLOG_WARNING "DT NUMA init failed\n");
> + }
> +
> +no_numa:
printk("No NUMA support\n"); or something similar.
And to be honest, this label does not seem really useful...
> + return;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
> index 3b34855..a6d1499 100644
> --- a/xen/arch/arm/setup.c
> +++ b/xen/arch/arm/setup.c
> @@ -38,6 +38,7 @@
> #include <xen/libfdt/libfdt.h>
> #include <xen/acpi.h>
> #include <asm/alternative.h>
> +#include <xen/numa.h>
> #include <asm/page.h>
> #include <asm/current.h>
> #include <asm/setup.h>
> @@ -755,6 +756,9 @@ void __init start_xen(unsigned long boot_phys_offset,
> /* Parse the ACPI tables for possible boot-time configuration */
> acpi_boot_table_init();
>
> + /* numa_init parses acpi tables. So call after acpi init */
> + numa_init();
> +
> end_boot_allocator();
>
> vm_init();
> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> index 7f00a36..8f517a2 100644
> --- a/xen/include/asm-arm/numa.h
> +++ b/xen/include/asm-arm/numa.h
> @@ -3,7 +3,15 @@
>
> typedef uint8_t nodeid_t;
>
> -#ifndef CONFIG_NUMA
> +#ifdef CONFIG_NUMA
> +void numa_init(void);
> +int dt_numa_init(void);
> +#else
> +static inline void numa_init(void)
> +{
> + return;
> +}
> +
> /* Fake one node for now. See also node_online_map. */
> #define cpu_to_node(cpu) 0
> #define node_to_cpumask(node) (cpu_online_map)
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 13/24] ARM: NUMA: DT: Parse memory NUMA information
2017-07-18 11:41 ` [RFC PATCH v3 13/24] ARM: NUMA: DT: Parse memory " vijay.kilari
@ 2017-07-19 18:39 ` Julien Grall
2017-07-20 10:37 ` Vijay Kilari
2017-07-20 11:26 ` Julien Grall
0 siblings, 2 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-19 18:39 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Parse memory node and fetch numa-node-id information.
> For each memory range, store in node_memblk_range[]
> along with node id.
>
> When booting in UEFI mode, UEFI passes memory information
> to Dom0 using EFI memory descriptor table and deletes the
> memory nodes from the host DT. However to fetch the memory
> numa node id, memory DT node should not be deleted by EFI stub.
> With this patch, do not delete memory node from FDT.
>
> NUMA info of memory is extracted from process_memory_node()
> instead of parsing the DT again during numa_init().
This patch does too much and needs to be split. The splitting would be
at least:
- EFI mode change
- Numa change
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> v3: - Set numa_off in numa_failed() and drop dt_numa variable
> ---
> xen/arch/arm/bootfdt.c | 25 +++++++++++++++++++++----
> xen/arch/arm/efi/efi-boot.h | 25 -------------------------
> xen/arch/arm/numa/dt_numa.c | 32 ++++++++++++++++++++++++++++++++
> xen/arch/arm/numa/numa.c | 5 +++++
> xen/include/asm-arm/numa.h | 2 ++
> 5 files changed, 60 insertions(+), 29 deletions(-)
>
> diff --git a/xen/arch/arm/bootfdt.c b/xen/arch/arm/bootfdt.c
> index 6e8251b..b3a132c 100644
> --- a/xen/arch/arm/bootfdt.c
> +++ b/xen/arch/arm/bootfdt.c
> @@ -13,6 +13,8 @@
> #include <xen/init.h>
> #include <xen/device_tree.h>
> #include <xen/libfdt/libfdt.h>
> +#include <xen/numa.h>
> +#include <xen/efi.h>
Please add the headers in alphabetical order.
> #include <xsm/xsm.h>
> #include <asm/setup.h>
>
> @@ -146,6 +148,9 @@ static void __init process_memory_node(const void *fdt, int node,
> const __be32 *cell;
> paddr_t start, size;
> u32 reg_cells = address_cells + size_cells;
> +#ifdef CONFIG_NUMA
> + uint32_t nid;
> +#endif
>
> if ( address_cells < 1 || size_cells < 1 )
> {
> @@ -154,24 +159,36 @@ static void __init process_memory_node(const void *fdt, int node,
> return;
> }
>
> +#ifdef CONFIG_NUMA
> + nid = device_tree_get_u32(fdt, node, "numa-node-id", NR_NODE_MEMBLKS);
Should not you use MAX_NUM_NODES rather than NR_NODE_MEMBLKS?
Also, where is the sanity check?
> +#endif
> prop = fdt_get_property(fdt, node, "reg", NULL);
> if ( !prop )
> {
> printk("fdt: node `%s': missing `reg' property\n", name);
> +#ifdef CONFIG_NUMA
> + numa_failed();
This file is using soft-tab not hard one.
> +#endif
> return;
> }
>
> cell = (const __be32 *)prop->data;
> banks = fdt32_to_cpu(prop->len) / (reg_cells * sizeof (u32));
>
> - for ( i = 0; i < banks && bootinfo.mem.nr_banks < NR_MEM_BANKS; i++ )
> + for ( i = 0; i < banks; i++ )
> {
> device_tree_get_reg(&cell, address_cells, size_cells, &start, &size);
> if ( !size )
> continue;
> - bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
> - bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
> - bootinfo.mem.nr_banks++;
> + if ( !efi_enabled(EFI_BOOT) && bootinfo.mem.nr_banks < NR_MEM_BANKS )
> + {
> + bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
> + bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
> + bootinfo.mem.nr_banks++;
> + }
This change should be split.
> +#ifdef CONFIG_NUMA
> + dt_numa_process_memory_node(nid, start, size);
> +#endif
> }
> }
>
> diff --git a/xen/arch/arm/efi/efi-boot.h b/xen/arch/arm/efi/efi-boot.h
> index 56de26e..a8bde68 100644
> --- a/xen/arch/arm/efi/efi-boot.h
> +++ b/xen/arch/arm/efi/efi-boot.h
> @@ -194,33 +194,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE *sys_table,
> int status;
> u32 fdt_val32;
> u64 fdt_val64;
> - int prev;
> int num_rsv;
>
> - /*
> - * Delete any memory nodes present. The EFI memory map is the only
> - * memory description provided to Xen.
> - */
> - prev = 0;
> - for (;;)
> - {
> - const char *type;
> - int len;
> -
> - node = fdt_next_node(fdt, prev, NULL);
> - if ( node < 0 )
> - break;
> -
> - type = fdt_getprop(fdt, node, "device_type", &len);
> - if ( type && strncmp(type, "memory", len) == 0 )
> - {
> - fdt_del_node(fdt, node);
> - continue;
> - }
> -
> - prev = node;
> - }
> -
That chunk should move to the same patch as the EFI check.
> /*
> * Delete all memory reserve map entries. When booting via UEFI,
> * kernel will use the UEFI memory map to find reserved regions.
> diff --git a/xen/arch/arm/numa/dt_numa.c b/xen/arch/arm/numa/dt_numa.c
> index 963bb40..84030e7 100644
> --- a/xen/arch/arm/numa/dt_numa.c
> +++ b/xen/arch/arm/numa/dt_numa.c
> @@ -58,6 +58,38 @@ static int __init dt_numa_process_cpu_node(const void *fdt)
> return 0;
> }
>
> +void __init dt_numa_process_memory_node(uint32_t nid, paddr_t start,
> + paddr_t size)
> +{
> + struct node *nd;
> + int i;
> +
> + i = conflicting_memblks(start, start + size);
> + if ( i < 0 )
> + {
> + if ( numa_add_memblk(nid, start, size) )
> + {
> + printk(XENLOG_WARNING "DT: NUMA: node-id %u overflow \n", nid);
> + numa_failed();
> + return;
> + }
> + }
> + else
> + {
> + nd = get_node_memblk_range(i);
> + printk(XENLOG_ERR
> + "NUMA DT: node %u (%"PRIx64"-%"PRIx64") overlaps with %d (%"PRIx64"-%"PRIx64")\n",
s/PRIx64/PRI_paddr/
> + nid, start, start + size, i, nd->start, nd->end);
> +
> + numa_failed();
> + return;
> + }
> +
> + node_set(nid, memory_nodes_parsed);
This code looks fairly similar to some bits of
acpi_numa_memory_affinity_init. Is there any way we could introduce a
common helper?
> +
> + return;
> +}
> +
> int __init dt_numa_init(void)
> {
> int ret;
> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
> index 45cc418..8227361 100644
> --- a/xen/arch/arm/numa/numa.c
> +++ b/xen/arch/arm/numa/numa.c
> @@ -19,6 +19,11 @@
> #include <xen/nodemask.h>
> #include <xen/numa.h>
>
> +void numa_failed(void)
> +{
> + numa_off = true;
> +}
> +
> void __init numa_init(void)
> {
> int ret = 0;
> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> index 8f517a2..36cd782 100644
> --- a/xen/include/asm-arm/numa.h
> +++ b/xen/include/asm-arm/numa.h
> @@ -3,6 +3,8 @@
>
> typedef uint8_t nodeid_t;
>
> +void dt_numa_process_memory_node(uint32_t nid, paddr_t start, paddr_t size);
Likely, this should go under CONFIG_NUMA.
> +
> #ifdef CONFIG_NUMA
> void numa_init(void);
> int dt_numa_init(void);
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 04/24] x86: NUMA: Rename and sanitize memnode shift code
2017-07-19 17:12 ` Julien Grall
@ 2017-07-20 6:56 ` Vijay Kilari
0 siblings, 0 replies; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 6:56 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Wed, Jul 19, 2017 at 10:42 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi Vijay,
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> memnode_shift variable is changed from int to unsigned int.
>> With this change, compute_memnode_shift() returns error value
>> instead of returning shift value. The memnode_shift is updated inside
>> compute_memnode_shift().
>>
>> Also, following changes are made
>> - Rename compute_hash_shift to compute_memnode_shift
>> - Update int to unsigned int for params in extract_lsb_from_nodes()
>> - Return values of populate_memnodemap() is changed
>
>
> I am not sure to understand the rationale behind changing the return value
> of populate_memnodemap. Likely this mean a bit more description in the
> commit message.
There is no much rationale behind it. As a part of cleanup, I have made
meaningful return values. Anyway, I will update commit message.
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code
2017-07-19 16:23 ` Julien Grall
2017-07-19 16:27 ` Wei Liu
@ 2017-07-20 7:00 ` Vijay Kilari
2017-07-20 11:00 ` Julien Grall
1 sibling, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 7:00 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Wed, Jul 19, 2017 at 9:53 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi Vijay,
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> Fix coding style, trailing spaces, tabs in NUMA code.
>> Also drop unused macros and functions.
>> There is no functional change.
>>
>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>> Reviewed-by: Wei Liu <wei.liu2@citrix.com>
>> ---
>> v3: - Change commit message
>> - Changed VIRTUAL_BUG_ON to ASSERT
>
>
> Looking at the commit message you don't mention any renaming...
>
>> - Dropped useless inner paranthesis for some macros
>
>
> [...]
>
>> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
>> index 3cf26c2..c0de57b 100644
>> --- a/xen/include/asm-x86/numa.h
>> +++ b/xen/include/asm-x86/numa.h
>> @@ -1,8 +1,11 @@
>> -#ifndef _ASM_X8664_NUMA_H
>> +#ifndef _ASM_X8664_NUMA_H
>> #define _ASM_X8664_NUMA_H 1
>>
>> #include <xen/cpumask.h>
>>
>> +#define MAX_NUMNODES NR_NODES
>> +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
>
>
> I don't understand why this suddenly appears in the code when you moved away
> in patch #1 in xen/numa.h.
Particularly MAX_NUMNODES required by this header file with this
patch changes for compilation.
Though I can include xen/numa.h here but xen/numa.h is including
asm/numa.h back.
I will add separate patch for this defines movement and drop from
this patch.
>
> [...]
>
>
>> @@ -57,21 +55,23 @@ struct node_data {
>>
>> extern struct node_data node_data[];
>>
>> -static inline __attribute__((pure)) nodeid_t phys_to_nid(paddr_t addr)
>> -{
>> - nodeid_t nid;
>> - VIRTUAL_BUG_ON((paddr_to_pdx(addr) >> memnode_shift) >=
>> memnodemapsize);
>> - nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
>> - VIRTUAL_BUG_ON(nid >= MAX_NUMNODES || !node_data[nid]);
>> - return nid;
>> -}
>> -
>> -#define NODE_DATA(nid) (&(node_data[nid]))
>> -
>> -#define node_start_pfn(nid) (NODE_DATA(nid)->node_start_pfn)
>> -#define node_spanned_pages(nid)
>> (NODE_DATA(nid)->node_spanned_pages)
>> -#define node_end_pfn(nid) (NODE_DATA(nid)->node_start_pfn + \
>> - NODE_DATA(nid)->node_spanned_pages)
>> +static inline __attribute_pure__ nodeid_t phys_to_nid(paddr_t addr)
>> +{
>> + nodeid_t nid;
>> +
>> + ASSERT((paddr_to_pdx(addr) >> memnode_shift) < memnodemapsize);
>> + nid = memnodemap[paddr_to_pdx(addr) >> memnode_shift];
>> + ASSERT(nid <= MAX_NUMNODES || !node_data[nid].node_start_pfn);
>> +
>> + return nid;
>> +}
>> +
>> +#define NODE_DATA(nid) (&(node_data[nid]))
>
>
> I understand Jan asked to remove the inner parentheses here. And you didn't
> do it. However ...
>
>> +
>> +#define node_start_pfn(nid) NODE_DATA(nid)->node_start_pfn
>> +#define node_spanned_pages(nid) NODE_DATA(nid)->node_spanned_pages
>> +#define node_end_pfn(nid) NODE_DATA(nid)->node_start_pfn + \
>> + NODE_DATA(nid)->node_spanned_pages
>
>
> ... here it is totally wrong to remove the parenthesis. Imagine you do:
>
> node_end_pfn(nid) * 2
>
> This will now turned into
>
> NODE_DATA(nid)->node_start_pfn + NODE_DATA(nid)->node_spanned_pages * 2
>
> The parenthesis is not correct anymore and will result to wrong computation.
> You should keep the outer parenthesis *everywhere* for safety and remove
> only the inner one in NODE_DATA.
OK.
>
> This is also more than cosmetics and I think the reviewed-by from Wei should
> have been carried.
OK.
>
>>
>> extern int valid_numa_range(u64 start, u64 end, nodeid_t node);
>>
>> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
>> index 6bba29e..3bb4afc 100644
>> --- a/xen/include/xen/numa.h
>> +++ b/xen/include/xen/numa.h
>> @@ -6,9 +6,6 @@
>> #define NUMA_NO_NODE 0xFF
>> #define NUMA_NO_DISTANCE 0xFF
>>
>> -#define MAX_NUMNODES NR_NODES
>> -#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
>> -
>
>
> See my comment above.
>
>> #define vcpu_to_node(v) (cpu_to_node((v)->processor))
>>
>> #define domain_to_node(d) \
>>
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable
2017-07-19 15:55 ` Julien Grall
@ 2017-07-20 7:30 ` Vijay Kilari
2017-07-20 10:57 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 7:30 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K, nd
On Wed, Jul 19, 2017 at 9:25 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi Vijay,
>
>
> On 19/07/2017 08:00, Vijay Kilari wrote:
>>
>> On Tue, Jul 18, 2017 at 11:25 PM, Julien Grall <julien.grall@arm.com>
>> wrote:
>>>
>>> Hi,
>>>
>>>
>>> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>>>
>>>>
>>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>>
>>>> Introduce NR_NODES config option to specify number
>>>> of NUMA nodes supported. By default value is set at
>>>> 64 for x86 and 8 for arm. Dropped NODES_SHIFT macro.
>>>>
>>>> Also move NR_NODE_MEMBLKS from asm-x86/acpi.h to xen/numa.h
>>>>
>>>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>> ---
>>>> xen/arch/Kconfig | 7 +++++++
>>>> xen/include/asm-x86/acpi.h | 1 -
>>>> xen/include/asm-x86/numa.h | 2 --
>>>> xen/include/xen/config.h | 1 +
>>>> xen/include/xen/numa.h | 7 ++-----
>>>> 5 files changed, 10 insertions(+), 8 deletions(-)
>>>>
>>>> diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig
>>>> index cf0acb7..9c2a4e2 100644
>>>> --- a/xen/arch/Kconfig
>>>> +++ b/xen/arch/Kconfig
>>>> @@ -6,3 +6,10 @@ config NR_CPUS
>>>> default "128" if ARM
>>>> ---help---
>>>> Specifies the maximum number of physical CPUs which Xen will
>>>> support.
>>>> +
>>>> +config NR_NODES
>>>> + int "Maximum number of NUMA nodes"
>>>> + default "64" if X86
>>>> + default "8" if ARM
>>>
>>>
>>>
>>> 3rd time I am asking it... Why the difference between x86 and ARM?
>>
>>
>> AFAIK, there is no arm platform for now with numa more than 8 nodes.
>> Thunderx is only 2 nodes.
>> So kept it low value for ARM to avoid unnecessary memory allocation.
>>
>> Do you want me to keep same as x86?.
>
>
> Well, you say it is for saving memory allocation but you don't give any
> number on how much you can save by reducing the default from 64 to 8...
>
> Looking at it, MAX_NUMNODES is used for some static allocation and also for
> the bitmap nodemask_t.
>
> Because our bitmap is based on unsigned long, you would use the same
> quantity of memory for AArch64, for AArch32 the quantity will be divided by
> two. Still nodemask_t does not seem to be widely used.
>
> In the case of the static allocation, I spot ~40 bytes per NUMA node. So 8
> node will use ~320 bytes and 64 bytes ~2560.
>
> NUMA is likely going to be used in server, don't tell me you are 2k short in
> memory? If it is an issue it is better to think how to limit the number of
> static variable rather than putting a low limit here.
>
> For Embedded use case, they will likely want to put the default to 1 but I
> would not worry about them as they are likely going to tweak the Kconfig.
Ok. I will set to 64. same as x86.
>
>>
>>>
>>> Also, you likely want to set to 1 if NUMA is not enabled.
>>
>>
>> I don't see any dependency of NR_NODES with NUMA config.
>> So it is always set to default value. Isn't?
>
>
> Well, what is the point to allow more than 1 node when NUMA is not
> supported?
In such case, I have to make NR_NODES depends on NUMA config
and define this value to 1 if NUMA config is not defined as below.
diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig
index b73d459..a5d40f5 100644
--- a/xen/arch/Kconfig
+++ b/xen/arch/Kconfig
@@ -11,5 +11,6 @@ config NR_NODES
int "Maximum number of NUMA nodes"
+ range 1 254
default "64"
+ depends on NUMA
---help---
Specifies the maximum number of NUMA nodes which Xen will support.
diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
index 604fd6d..eede1c4 100644
--- a/xen/include/asm-x86/numa.h
+++ b/xen/include/asm-x86/numa.h
@@ -10,6 +10,10 @@ extern int srat_rev;
extern nodeid_t cpu_to_node[NR_CPUS];
extern cpumask_t node_to_cpumask[];
+#ifndef CONFIG_NUMA
+#define NR_NODES 1
+#endif
+
#define MAX_NUMNODES NR_NODES
#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
>
> Not mentioning that this is quite confusing for a user to allow setting up
> the maximum number of nodes if the archicture is not supporting numa...
>
> For instance, this is the case today on ARM because, without this series, we
> don't support NUMA.
>
>
>>
>>>
>>>
>>>> + ---help---
>>>> + Specifies the maximum number of NUMA nodes which Xen will
>>>> support.
>>>> diff --git a/xen/include/asm-x86/acpi.h b/xen/include/asm-x86/acpi.h
>>>> index 27ecc65..15be784 100644
>>>> --- a/xen/include/asm-x86/acpi.h
>>>> +++ b/xen/include/asm-x86/acpi.h
>>>> @@ -105,7 +105,6 @@ extern void acpi_reserve_bootmem(void);
>>>>
>>>> extern s8 acpi_numa;
>>>> extern int acpi_scan_nodes(u64 start, u64 end);
>>>> -#define NR_NODE_MEMBLKS (MAX_NUMNODES*2)
>>>>
>>>> #ifdef CONFIG_ACPI_SLEEP
>>>>
>>>> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
>>>> index bada2c0..3cf26c2 100644
>>>> --- a/xen/include/asm-x86/numa.h
>>>> +++ b/xen/include/asm-x86/numa.h
>>>> @@ -3,8 +3,6 @@
>>>>
>>>> #include <xen/cpumask.h>
>>>>
>>>> -#define NODES_SHIFT 6
>>>> -
>>>> typedef u8 nodeid_t;
>>>>
>>>> extern int srat_rev;
>>>> diff --git a/xen/include/xen/config.h b/xen/include/xen/config.h
>>>> index a1d0f97..0f1a029 100644
>>>> --- a/xen/include/xen/config.h
>>>> +++ b/xen/include/xen/config.h
>>>> @@ -81,6 +81,7 @@
>>>>
>>>> /* allow existing code to work with Kconfig variable */
>>>> #define NR_CPUS CONFIG_NR_CPUS
>>>> +#define NR_NODES CONFIG_NR_NODES
>>>>
>>>> #ifndef CONFIG_DEBUG
>>>> #define NDEBUG
>>>> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
>>>> index 7aef1a8..6bba29e 100644
>>>> --- a/xen/include/xen/numa.h
>>>> +++ b/xen/include/xen/numa.h
>>>> @@ -3,14 +3,11 @@
>>>>
>>>> #include <asm/numa.h>
>>>>
>>>> -#ifndef NODES_SHIFT
>>>> -#define NODES_SHIFT 0
>>>> -#endif
>>>> -
>>>> #define NUMA_NO_NODE 0xFF
>>>> #define NUMA_NO_DISTANCE 0xFF
>>>>
>>>> -#define MAX_NUMNODES (1 << NODES_SHIFT)
>>>> +#define MAX_NUMNODES NR_NODES
>>>> +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
>
>
> Also, I don't understand why you move this define from asm-x86/numa.h to
> xen/numa.h. At least, this does not seem related to this patch...
ok. I will drop this change from this patch
>
> Cheers,
>
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 05/24] x86: NUMA: Add accessors for nodes[] and node_memblk_range[] structs
2017-07-19 17:18 ` Julien Grall
@ 2017-07-20 7:41 ` Vijay Kilari
2017-07-20 11:03 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 7:41 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Wed, Jul 19, 2017 at 10:48 PM, Julien Grall <julien.grall@arm.com> wrote:
>
>
> On 19/07/17 07:40, Vijay Kilari wrote:
>>
>> On Tue, Jul 18, 2017 at 8:59 PM, Wei Liu <wei.liu2@citrix.com> wrote:
>>>
>>> On Tue, Jul 18, 2017 at 05:11:27PM +0530, vijay.kilari@gmail.com wrote:
>>>>
>>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>>
>>>> Add accessors for nodes[] and other static variables and
>>>> use those accessors. These variables are later accessed
>>>> outside the file when the code made generic in later
>>>> patches. However the coding style is not changed.
>>>>
>>>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>> ---
>>>> v3: - Changed accessors parameter from int to unsigned int
>>>> - Updated commit message
>>>> - Fixed wrong indentation
>>>> ---
>>>> xen/arch/x86/srat.c | 106
>>>> +++++++++++++++++++++++++++++++++++++++-------------
>>>> 1 file changed, 81 insertions(+), 25 deletions(-)
>>>>
>>>> diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
>>>> index 535c9d7..42cca5a 100644
>>>> --- a/xen/arch/x86/srat.c
>>>> +++ b/xen/arch/x86/srat.c
>>>> @@ -41,6 +41,44 @@ static struct node
>>>> node_memblk_range[NR_NODE_MEMBLKS];
>>>> static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
>>>> static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
>>>>
>>>> +static struct node *get_numa_node(unsigned int id)
>>>> +{
>>>> + return &nodes[id];
>>>> +}
>>>> +
>>>> +static nodeid_t get_memblk_nodeid(unsigned int id)
>>>> +{
>>>> + return memblk_nodeid[id];
>>>> +}
>>>> +
>>>> +static nodeid_t *get_memblk_nodeid_map(void)
>>>> +{
>>>> + return &memblk_nodeid[0];
>>>> +}
>>>> +
>>>> +static struct node *get_node_memblk_range(unsigned int memblk)
>>>> +{
>>>> + return &node_memblk_range[memblk];
>>>> +}
>>>> +
>>>> +static int get_num_node_memblks(void)
>>>> +{
>>>> + return num_node_memblks;
>>>> +}
>>>
>>>
>>> They should all be inline functions. And maybe at once lift to a header
>>> and add proper prefix since you mention they are going to be used later.
>>
>>
>> Currently these are static variables in x86/srat.c file.
>> In patch #9 I move them to common/numa.c file and make these functions
>> non-static.
>>
>> If I lift them to header file and make inline, then I have to make these
>> as
>> global variables.
>
>
> As I said on v2, I am not sure to understand the usefulness of those
> accessors over global variables...
These are static variables. To access across other files (arch specific)
these accessors are added. I have to make them global variables to use
outside of this file.
I am happy to make them global and make these accessors static inline
as suggested by Wei.
>
> You don't have any kind of sanity check, so they would do exactly the same
> job. The global variables would avoid so much churn.
>
> More that you tend to sometimes use global and other time static helpers...
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic
2017-07-19 17:41 ` Julien Grall
@ 2017-07-20 8:55 ` Vijay Kilari
2017-07-20 11:14 ` Julien Grall
2017-07-24 20:28 ` Stefano Stabellini
1 sibling, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 8:55 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Wed, Jul 19, 2017 at 11:11 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi Vijay,
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> Move code from xen/arch/x86/numa.c to xen/common/numa.c
>> so that it can be used by other archs.
>>
>> The following changes are done:
>> - Few generic static functions in x86/numa.c is made
>> non-static common/numa.c
>> - The generic contents of header file asm-x86/numa.h
>> are moved to xen/numa.h.
>> - The header file includes are reordered and externs are
>> dropped.
>> - Moved acpi_numa from asm-x86/acpi.h to xen/acpi.h
>> - Coding style of code moved to commom/numa.c is changed
>> to Xen style.
>> - numa_add_cpu() and numa_set_node() and moved to header
>> file and added inline function in case of CONFIG_NUMA
>> is not enabled because these functions are called from
>> generic code with out any config check.
>>
>> Also the node_online_map is defined in x86/numa.c for x86
>> and arm/smpboot.c for ARM. For x86 it is moved to x86/smpboot.c
>> If moved to common code the compilation fails because
>> common/numa.c is compiled only when NUMA is enabled.
>
>
> I would much prefer if this patch does one thing: Moving code. The rest
> should be split out to help review and allowing us to easily verify you only
> moved code...
Yes, this patch is doing only code movement. Apart from adding inline function
for numa_add_cpu() and numa_set_node().
>
>> +#define NODE_DATA(nid) (&(node_data[nid]))
>> +
>> +#define node_start_pfn(nid) NODE_DATA(nid)->node_start_pfn
>> +#define node_spanned_pages(nid) NODE_DATA(nid)->node_spanned_pages
>> +#define node_end_pfn(nid) NODE_DATA(nid)->node_start_pfn + \
>> + NODE_DATA(nid)->node_spanned_pages
>> +
>> +void numa_add_cpu(int cpu);
>> +void numa_set_node(int cpu, nodeid_t node);
>> +#else
>> +static inline void numa_add_cpu(int cpu) { }
>> +static inline void numa_set_node(int cpu, nodeid_t node) { }
>
>
> I am not sure why you need to define stub at least for numa_set_node... I
> can't see use in non-NUMA code. I will comment about the numa_add_cpu later.
x86 is using from setup.c. yes if we assume that numa is always enabled for x86,
I can drop numa_set_node() inline function.
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 12/24] ARM: NUMA: DT: Parse CPU NUMA information
2017-07-19 18:26 ` Julien Grall
@ 2017-07-20 9:20 ` Vijay Kilari
0 siblings, 0 replies; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 9:20 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Wed, Jul 19, 2017 at 11:56 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi,
>
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> Parse CPU node and fetch numa-node-id information.
>> For each node-id found, update nodemask_t mask.
>> Refer to Documentation/devicetree/bindings/numa.txt
>> in linux kernel.
>>
>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>> ---
>> v3: - Parse cpu nodes under path /cpus
>> - Move changes to bootfdt.c as separate patch
>> - Set numa_off on dt_numa_init() failure
>> ---
>> xen/arch/arm/Makefile | 1 +
>> xen/arch/arm/numa/Makefile | 2 ++
>> xen/arch/arm/numa/dt_numa.c | 77
>> +++++++++++++++++++++++++++++++++++++++++++++
>> xen/arch/arm/numa/numa.c | 48 ++++++++++++++++++++++++++++
>> xen/arch/arm/setup.c | 4 +++
>> xen/include/asm-arm/numa.h | 10 +++++-
>> 6 files changed, 141 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 49e1fb2..a89be66 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -3,6 +3,7 @@ subdir-$(CONFIG_ARM_64) += arm64
>> subdir-y += platforms
>> subdir-$(CONFIG_ARM_64) += efi
>> subdir-$(CONFIG_ACPI) += acpi
>> +subdir-$(CONFIG_NUMA) += numa
>>
>> obj-$(CONFIG_HAS_ALTERNATIVE) += alternative.o
>> obj-y += bootfdt.init.o
>> diff --git a/xen/arch/arm/numa/Makefile b/xen/arch/arm/numa/Makefile
>> new file mode 100644
>> index 0000000..3af3aff
>> --- /dev/null
>> +++ b/xen/arch/arm/numa/Makefile
>> @@ -0,0 +1,2 @@
>> +obj-y += dt_numa.o
>> +obj-y += numa.o
>> diff --git a/xen/arch/arm/numa/dt_numa.c b/xen/arch/arm/numa/dt_numa.c
>> new file mode 100644
>> index 0000000..963bb40
>> --- /dev/null
>> +++ b/xen/arch/arm/numa/dt_numa.c
>> @@ -0,0 +1,77 @@
>> +/*
>> + * OF NUMA Parsing support.
>> + *
>> + * Copyright (C) 2015 - 2016 Cavium Inc.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program. If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/mm.h>
>> +#include <xen/nodemask.h>
>> +#include <xen/libfdt/libfdt.h>
>> +#include <xen/device_tree.h>
>
>
> Again, this include should not be there as the device tree is not yet
> parsed.
I believe that below code needs this header file.
>
>> +#include <xen/numa.h>
>> +#include <asm/setup.h>
>
>
> Again, please order in alphabetically the includes...
>
>
>> +
>> +/*
>> + * Even though we connect cpus to numa domains later in SMP
>> + * init, we need to know the node ids now for all cpus.
>> + */
>> +static int __init dt_numa_process_cpu_node(const void *fdt)
>> +{
>> + int node, offset;
>> + uint32_t nid;
>> +
>> + offset = fdt_path_offset(fdt, "/cpus");
>> + if ( offset < 0 )
>> + return -EINVAL;
>> +
>> + node = fdt_first_subnode(fdt, offset);
>> + if ( node == -FDT_ERR_NOTFOUND )
>> + return -EINVAL;
>> +
>> + do {
>> + if ( device_tree_type_matches(fdt, node, "cpu") )
>> + {
>> + nid = device_tree_get_u32(fdt, node, "numa-node-id",
>> MAX_NUMNODES);
>> + if ( nid >= MAX_NUMNODES )
>> + printk(XENLOG_WARNING
>> + "NUMA: Node id %u exceeds maximum value\n", nid);
>> + else
>> + node_set(nid, processor_nodes_parsed);
>> + }
>> +
>> + offset = node;
>> + node = fdt_next_subnode(fdt, offset);
>> + } while (node != -FDT_ERR_NOTFOUND);
>> +
>> + return 0;
>> +}
>> +
>> +int __init dt_numa_init(void)
>> +{
>> + int ret;
>> +
>> + ret = dt_numa_process_cpu_node((void *)device_tree_flattened);
>> +
>> + return ret;
>
>
> return dt_numa_process_cpu_node(....);
>
> But I am still not sure to understand why you can't parse the numa node in
> directly in bootfdt.c as you do for the memory.
IRC, Initially I was facing issue with this approach. I will re-look into it.
>
>
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
>> new file mode 100644
>> index 0000000..45cc418
>> --- /dev/null
>> +++ b/xen/arch/arm/numa/numa.c
>> @@ -0,0 +1,48 @@
>> +/*
>> + * ARM NUMA Implementation
>> + *
>> + * Copyright (C) 2016 - Cavium Inc.
>> + * Vijaya Kumar K <vijaya.kumar@cavium.com>
>> + *
>> + * This program is free software; you can redistribute it and/or
>> + * modify it under the terms and conditions of the GNU General Public
>> + * License, version 2, as published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#include <xen/init.h>
>> +#include <xen/ctype.h>
>> +#include <xen/nodemask.h>
>> +#include <xen/numa.h>
>> +
>> +void __init numa_init(void)
>> +{
>> + int ret = 0;
>> +
>> + nodes_clear(processor_nodes_parsed);
>
>
> Why do you need to clear processor_nodes_parsed? It should already be all
> zeroed.
OK.
>
>> + if ( numa_off )
>> + goto no_numa;
>> +
>> + ret = dt_numa_init();
>> + if ( ret )
>> + {
>> + numa_off = true;
>> + printk(XENLOG_WARNING "DT NUMA init failed\n");
>> + }
>> +
>> +no_numa:
>
>
> printk("No NUMA support\n"); or something similar.
>
> And to be honest, this label does not seem really useful...
ok
>
>
>> + return;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
>> index 3b34855..a6d1499 100644
>> --- a/xen/arch/arm/setup.c
>> +++ b/xen/arch/arm/setup.c
>> @@ -38,6 +38,7 @@
>> #include <xen/libfdt/libfdt.h>
>> #include <xen/acpi.h>
>> #include <asm/alternative.h>
>> +#include <xen/numa.h>
>> #include <asm/page.h>
>> #include <asm/current.h>
>> #include <asm/setup.h>
>> @@ -755,6 +756,9 @@ void __init start_xen(unsigned long boot_phys_offset,
>> /* Parse the ACPI tables for possible boot-time configuration */
>> acpi_boot_table_init();
>>
>> + /* numa_init parses acpi tables. So call after acpi init */
>> + numa_init();
>> +
>> end_boot_allocator();
>>
>> vm_init();
>> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
>> index 7f00a36..8f517a2 100644
>> --- a/xen/include/asm-arm/numa.h
>> +++ b/xen/include/asm-arm/numa.h
>> @@ -3,7 +3,15 @@
>>
>> typedef uint8_t nodeid_t;
>>
>> -#ifndef CONFIG_NUMA
>> +#ifdef CONFIG_NUMA
>> +void numa_init(void);
>> +int dt_numa_init(void);
>> +#else
>> +static inline void numa_init(void)
>> +{
>> + return;
>> +}
>> +
>> /* Fake one node for now. See also node_online_map. */
>> #define cpu_to_node(cpu) 0
>> #define node_to_cpumask(node) (cpu_online_map)
>>
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 07/24] ARM: NUMA: Add existing ARM numa code under CONFIG_NUMA
2017-07-18 18:06 ` Julien Grall
@ 2017-07-20 9:31 ` Vijay Kilari
2017-07-20 11:10 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 9:31 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Tue, Jul 18, 2017 at 11:36 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi Vijay,
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> Right now CONFIG_NUMA is not enabled for ARM and
>> existing code in asm-arm/numa.h is for !CONFIG_NUMA.
>> Hence put this code under #ifndef CONFIG_NUMA.
>>
>> This help to make this changes work when CONFIG_NUMA
>> is not enabled. Though CONFIG_NUMA is enabled by default,
>> manually disabling this option is possible and compilation
>> should go through. Hence kept the these changes under
>> !CONFIG_NUMA.
>
>
> This is still no true. It is not possible to disable CONFIG_NUMA from the
> Kconfig unless you hack it (just tried it)...
>
> As I said on v2, if you always enable NUMA why should we add code in Xen
> that get rotten? Either you allow NUMA to be disabled by the user or you
> drop this code.
The reason is: The next patch #8, which does the code movement moves
the generic code to common header file xen/numa.h.
If we don't put these *existing* defines in asm-arm/numa.h under
#ifndef CONFIG_NUMA,
the compilation fails for ARM.
Is it ok to removes these defines under separate patch after enabling
NUMA config
at the end of patch series?
Let me know if you have any better approach.
>
>>
>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>> ---
>> v3: - Dropped NODE_SHIFT define
>> ---
>> xen/include/asm-arm/numa.h | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
>> index 53f99af..7f00a36 100644
>> --- a/xen/include/asm-arm/numa.h
>> +++ b/xen/include/asm-arm/numa.h
>> @@ -3,6 +3,7 @@
>>
>> typedef uint8_t nodeid_t;
>>
>> +#ifndef CONFIG_NUMA
>> /* Fake one node for now. See also node_online_map. */
>> #define cpu_to_node(cpu) 0
>> #define node_to_cpumask(node) (cpu_online_map)
>> @@ -16,6 +17,7 @@ static inline __attribute__((pure)) nodeid_t
>> phys_to_nid(paddr_t addr)
>> #define node_spanned_pages(nid) (total_pages)
>> #define node_start_pfn(nid) (pdx_to_pfn(frametable_base_pdx))
>> #define __node_distance(a, b) (20)
>> +#endif /* CONFIG_NUMA */
>>
>> static inline unsigned int arch_get_dma_bitsize(void)
>> {
>>
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 10/24] NUMA: Allow numa initialization with DT
2017-07-19 17:58 ` Julien Grall
@ 2017-07-20 10:28 ` Vijay Kilari
2017-07-20 11:20 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 10:28 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Wed, Jul 19, 2017 at 11:28 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi Vijay,
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> The common code allows numa initialization only when
>> ACPI_NUMA config is enabled. Allow initialization when
>> NUMA config is enabled for DT.
>>
>> In this patch, along with acpi_numa, check for acpi_disabled
>> is added.
>>
>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>> ---
>> xen/common/numa.c | 4 +---
>> 1 file changed, 1 insertion(+), 3 deletions(-)
>>
>> diff --git a/xen/common/numa.c b/xen/common/numa.c
>> index 74c4697..5e985d2 100644
>> --- a/xen/common/numa.c
>> +++ b/xen/common/numa.c
>> @@ -324,7 +324,7 @@ static int __init numa_scan_nodes(paddr_t start,
>> paddr_t end)
>> for ( i = 0; i < MAX_NUMNODES; i++ )
>> cutoff_node(i, start, end);
>>
>> - if ( acpi_numa <= 0 )
>> + if ( !acpi_disabled && acpi_numa <= 0 )
>
>
> I am struggling to understand this change. Likely you want to similar
> variable for DT to say NUMA is available or this has failed.
Yes, without this check for acpi_disabled, when booting with DT, the check
acpi_numa <= 0 is true and does not allow numa initialization.
>
> This also change quite a bit the semantic for x86 because, you will now
> continue if acpi_disabled and acpi_numa = 0. The code seems to allow it, but
> I don't know if we support it.
Yes, but prior to this patch, x86 is assuming that acpi_disabled is
false by checking
only for acpi_numa <=0.
The other solution is create a arch wrapper and call this from here.
Regards
Vijay
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 13/24] ARM: NUMA: DT: Parse memory NUMA information
2017-07-19 18:39 ` Julien Grall
@ 2017-07-20 10:37 ` Vijay Kilari
2017-07-20 11:24 ` Julien Grall
2017-07-20 11:26 ` Julien Grall
1 sibling, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 10:37 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Thu, Jul 20, 2017 at 12:09 AM, Julien Grall <julien.grall@arm.com> wrote:
> Hi Vijay,
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> Parse memory node and fetch numa-node-id information.
>> For each memory range, store in node_memblk_range[]
>> along with node id.
>>
>> When booting in UEFI mode, UEFI passes memory information
>> to Dom0 using EFI memory descriptor table and deletes the
>> memory nodes from the host DT. However to fetch the memory
>> numa node id, memory DT node should not be deleted by EFI stub.
>> With this patch, do not delete memory node from FDT.
>>
>> NUMA info of memory is extracted from process_memory_node()
>> instead of parsing the DT again during numa_init().
>
>
> This patch does too much and needs to be split. The splitting would be at
> least:
>
> - EFI mode change
> - Numa change
OK
>
>>
>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>> ---
>> v3: - Set numa_off in numa_failed() and drop dt_numa variable
>> ---
>> xen/arch/arm/bootfdt.c | 25 +++++++++++++++++++++----
>> xen/arch/arm/efi/efi-boot.h | 25 -------------------------
>> xen/arch/arm/numa/dt_numa.c | 32 ++++++++++++++++++++++++++++++++
>> xen/arch/arm/numa/numa.c | 5 +++++
>> xen/include/asm-arm/numa.h | 2 ++
>> 5 files changed, 60 insertions(+), 29 deletions(-)
>>
>> diff --git a/xen/arch/arm/bootfdt.c b/xen/arch/arm/bootfdt.c
>> index 6e8251b..b3a132c 100644
>> --- a/xen/arch/arm/bootfdt.c
>> +++ b/xen/arch/arm/bootfdt.c
>> @@ -13,6 +13,8 @@
>> #include <xen/init.h>
>> #include <xen/device_tree.h>
>> #include <xen/libfdt/libfdt.h>
>> +#include <xen/numa.h>
>> +#include <xen/efi.h>
>
>
> Please add the headers in alphabetical order.
>
>> #include <xsm/xsm.h>
>> #include <asm/setup.h>
>>
>> @@ -146,6 +148,9 @@ static void __init process_memory_node(const void
>> *fdt, int node,
>> const __be32 *cell;
>> paddr_t start, size;
>> u32 reg_cells = address_cells + size_cells;
>> +#ifdef CONFIG_NUMA
>> + uint32_t nid;
>> +#endif
>>
>> if ( address_cells < 1 || size_cells < 1 )
>> {
>> @@ -154,24 +159,36 @@ static void __init process_memory_node(const void
>> *fdt, int node,
>> return;
>> }
>>
>> +#ifdef CONFIG_NUMA
>> + nid = device_tree_get_u32(fdt, node, "numa-node-id",
>> NR_NODE_MEMBLKS);
>
>
> Should not you use MAX_NUM_NODES rather than NR_NODE_MEMBLKS?
>
> Also, where is the sanity check?
OK
>
>> +#endif
>> prop = fdt_get_property(fdt, node, "reg", NULL);
>> if ( !prop )
>> {
>> printk("fdt: node `%s': missing `reg' property\n", name);
>> +#ifdef CONFIG_NUMA
>> + numa_failed();
>
>
> This file is using soft-tab not hard one.
>
>> +#endif
>> return;
>> }
>>
>> cell = (const __be32 *)prop->data;
>> banks = fdt32_to_cpu(prop->len) / (reg_cells * sizeof (u32));
>>
>> - for ( i = 0; i < banks && bootinfo.mem.nr_banks < NR_MEM_BANKS; i++ )
>> + for ( i = 0; i < banks; i++ )
>> {
>> device_tree_get_reg(&cell, address_cells, size_cells, &start,
>> &size);
>> if ( !size )
>> continue;
>> - bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
>> - bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
>> - bootinfo.mem.nr_banks++;
>> + if ( !efi_enabled(EFI_BOOT) && bootinfo.mem.nr_banks <
>> NR_MEM_BANKS )
>> + {
>> + bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
>> + bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
>> + bootinfo.mem.nr_banks++;
>> + }
>
>
> This change should be split.
>
>
>> +#ifdef CONFIG_NUMA
>> + dt_numa_process_memory_node(nid, start, size);
>> +#endif
>> }
>> }
>>
>> diff --git a/xen/arch/arm/efi/efi-boot.h b/xen/arch/arm/efi/efi-boot.h
>> index 56de26e..a8bde68 100644
>> --- a/xen/arch/arm/efi/efi-boot.h
>> +++ b/xen/arch/arm/efi/efi-boot.h
>> @@ -194,33 +194,8 @@ EFI_STATUS __init fdt_add_uefi_nodes(EFI_SYSTEM_TABLE
>> *sys_table,
>> int status;
>> u32 fdt_val32;
>> u64 fdt_val64;
>> - int prev;
>> int num_rsv;
>>
>> - /*
>> - * Delete any memory nodes present. The EFI memory map is the only
>> - * memory description provided to Xen.
>> - */
>> - prev = 0;
>> - for (;;)
>> - {
>> - const char *type;
>> - int len;
>> -
>> - node = fdt_next_node(fdt, prev, NULL);
>> - if ( node < 0 )
>> - break;
>> -
>> - type = fdt_getprop(fdt, node, "device_type", &len);
>> - if ( type && strncmp(type, "memory", len) == 0 )
>> - {
>> - fdt_del_node(fdt, node);
>> - continue;
>> - }
>> -
>> - prev = node;
>> - }
>> -
>
>
> That chunk should move to the same patch as the EFI check.
>
ok
>
>> /*
>> * Delete all memory reserve map entries. When booting via UEFI,
>> * kernel will use the UEFI memory map to find reserved regions.
>> diff --git a/xen/arch/arm/numa/dt_numa.c b/xen/arch/arm/numa/dt_numa.c
>> index 963bb40..84030e7 100644
>> --- a/xen/arch/arm/numa/dt_numa.c
>> +++ b/xen/arch/arm/numa/dt_numa.c
>> @@ -58,6 +58,38 @@ static int __init dt_numa_process_cpu_node(const void
>> *fdt)
>> return 0;
>> }
>>
>> +void __init dt_numa_process_memory_node(uint32_t nid, paddr_t start,
>> + paddr_t size)
>> +{
>> + struct node *nd;
>> + int i;
>> +
>> + i = conflicting_memblks(start, start + size);
>> + if ( i < 0 )
>> + {
>> + if ( numa_add_memblk(nid, start, size) )
>> + {
>> + printk(XENLOG_WARNING "DT: NUMA: node-id %u overflow \n",
>> nid);
>> + numa_failed();
>> + return;
>> + }
>> + }
>> + else
>> + {
>> + nd = get_node_memblk_range(i);
>> + printk(XENLOG_ERR
>> + "NUMA DT: node %u (%"PRIx64"-%"PRIx64") overlaps with %d
>> (%"PRIx64"-%"PRIx64")\n",
>
>
> s/PRIx64/PRI_paddr/
ok
>
>> + nid, start, start + size, i, nd->start, nd->end);
>> +
>> + numa_failed();
>> + return;
>> + }
>> +
>> + node_set(nid, memory_nodes_parsed);
>
>
> This code looks fairly similar to some bits of
> acpi_numa_memory_affinity_init. Is there any way we could introduce a common
> helper?
Yes some bit of code is similar, But acpi_numa_memory_affinity_init() is stuffed
with some more checks of ACPI data in between the code. So quite complex
to make it common code.
>
>
>> +
>> + return;
>> +}
>> +
>> int __init dt_numa_init(void)
>> {
>> int ret;
>> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
>> index 45cc418..8227361 100644
>> --- a/xen/arch/arm/numa/numa.c
>> +++ b/xen/arch/arm/numa/numa.c
>> @@ -19,6 +19,11 @@
>> #include <xen/nodemask.h>
>> #include <xen/numa.h>
>>
>> +void numa_failed(void)
>> +{
>> + numa_off = true;
>> +}
>> +
>> void __init numa_init(void)
>> {
>> int ret = 0;
>> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
>> index 8f517a2..36cd782 100644
>> --- a/xen/include/asm-arm/numa.h
>> +++ b/xen/include/asm-arm/numa.h
>> @@ -3,6 +3,8 @@
>>
>> typedef uint8_t nodeid_t;
>>
>> +void dt_numa_process_memory_node(uint32_t nid, paddr_t start, paddr_t
>> size);
>
>
> Likely, this should go under CONFIG_NUMA.
ok
>
>> +
>> #ifdef CONFIG_NUMA
>> void numa_init(void);
>> int dt_numa_init(void);
>>
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable
2017-07-20 7:30 ` Vijay Kilari
@ 2017-07-20 10:57 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-20 10:57 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K, nd
Hi Vijay,
On 20/07/17 08:30, Vijay Kilari wrote:
> On Wed, Jul 19, 2017 at 9:25 PM, Julien Grall <julien.grall@arm.com> wrote:
>>>
>>>>
>>>> Also, you likely want to set to 1 if NUMA is not enabled.
>>>
>>>
>>> I don't see any dependency of NR_NODES with NUMA config.
>>> So it is always set to default value. Isn't?
>>
>>
>> Well, what is the point to allow more than 1 node when NUMA is not
>> supported?
>
> In such case, I have to make NR_NODES depends on NUMA config
> and define this value to 1 if NUMA config is not defined as below.
>
> diff --git a/xen/arch/Kconfig b/xen/arch/Kconfig
> index b73d459..a5d40f5 100644
> --- a/xen/arch/Kconfig
> +++ b/xen/arch/Kconfig
> @@ -11,5 +11,6 @@ config NR_NODES
> int "Maximum number of NUMA nodes"
> + range 1 254
> default "64"
> + depends on NUMA
> ---help---
> Specifies the maximum number of NUMA nodes which Xen will support.
> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
> index 604fd6d..eede1c4 100644
> --- a/xen/include/asm-x86/numa.h
> +++ b/xen/include/asm-x86/numa.h
> @@ -10,6 +10,10 @@ extern int srat_rev;
> extern nodeid_t cpu_to_node[NR_CPUS];
> extern cpumask_t node_to_cpumask[];
>
> +#ifndef CONFIG_NUMA
> +#define NR_NODES 1
> +#endif
> +
> #define MAX_NUMNODES NR_NODES
> #define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
Can't Kconfig do it for you? Something like below:
int "Maximum number of NUMA nodes" if NUMA
range 1 254
default "64" if NUMA
default "0" if !NUMA
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code
2017-07-20 7:00 ` Vijay Kilari
@ 2017-07-20 11:00 ` Julien Grall
2017-07-20 12:05 ` Vijay Kilari
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-20 11:00 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
Hi Vijay,
On 20/07/17 08:00, Vijay Kilari wrote:
> On Wed, Jul 19, 2017 at 9:53 PM, Julien Grall <julien.grall@arm.com> wrote:
>> Hi Vijay,
>>
>> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>>
>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>
>>> Fix coding style, trailing spaces, tabs in NUMA code.
>>> Also drop unused macros and functions.
>>> There is no functional change.
>>>
>>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>> Reviewed-by: Wei Liu <wei.liu2@citrix.com>
>>> ---
>>> v3: - Change commit message
>>> - Changed VIRTUAL_BUG_ON to ASSERT
>>
>>
>> Looking at the commit message you don't mention any renaming...
>>
>>> - Dropped useless inner paranthesis for some macros
>>
>>
>> [...]
>>
>>> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
>>> index 3cf26c2..c0de57b 100644
>>> --- a/xen/include/asm-x86/numa.h
>>> +++ b/xen/include/asm-x86/numa.h
>>> @@ -1,8 +1,11 @@
>>> -#ifndef _ASM_X8664_NUMA_H
>>> +#ifndef _ASM_X8664_NUMA_H
>>> #define _ASM_X8664_NUMA_H 1
>>>
>>> #include <xen/cpumask.h>
>>>
>>> +#define MAX_NUMNODES NR_NODES
>>> +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
>>
>>
>> I don't understand why this suddenly appears in the code when you moved away
>> in patch #1 in xen/numa.h.
>
> Particularly MAX_NUMNODES required by this header file with this
> patch changes for compilation.
> Though I can include xen/numa.h here but xen/numa.h is including
> asm/numa.h back.
>
> I will add separate patch for this defines movement and drop from
> this patch.
Why adding a separate patch? The code should not have been moved away in
patch #1 as you did.
But I still don't understand what is the exact error here... If it fails
on this patch, likely this should have failed after applying patch #1.
And *all* patch should be able to build without the rest of the series.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 05/24] x86: NUMA: Add accessors for nodes[] and node_memblk_range[] structs
2017-07-20 7:41 ` Vijay Kilari
@ 2017-07-20 11:03 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-20 11:03 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On 20/07/17 08:41, Vijay Kilari wrote:
> On Wed, Jul 19, 2017 at 10:48 PM, Julien Grall <julien.grall@arm.com> wrote:
>>
>>
>> On 19/07/17 07:40, Vijay Kilari wrote:
>>>
>>> On Tue, Jul 18, 2017 at 8:59 PM, Wei Liu <wei.liu2@citrix.com> wrote:
>>>>
>>>> On Tue, Jul 18, 2017 at 05:11:27PM +0530, vijay.kilari@gmail.com wrote:
>>>>>
>>>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>>>
>>>>> Add accessors for nodes[] and other static variables and
>>>>> use those accessors. These variables are later accessed
>>>>> outside the file when the code made generic in later
>>>>> patches. However the coding style is not changed.
>>>>>
>>>>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>>> ---
>>>>> v3: - Changed accessors parameter from int to unsigned int
>>>>> - Updated commit message
>>>>> - Fixed wrong indentation
>>>>> ---
>>>>> xen/arch/x86/srat.c | 106
>>>>> +++++++++++++++++++++++++++++++++++++++-------------
>>>>> 1 file changed, 81 insertions(+), 25 deletions(-)
>>>>>
>>>>> diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
>>>>> index 535c9d7..42cca5a 100644
>>>>> --- a/xen/arch/x86/srat.c
>>>>> +++ b/xen/arch/x86/srat.c
>>>>> @@ -41,6 +41,44 @@ static struct node
>>>>> node_memblk_range[NR_NODE_MEMBLKS];
>>>>> static nodeid_t memblk_nodeid[NR_NODE_MEMBLKS];
>>>>> static __initdata DECLARE_BITMAP(memblk_hotplug, NR_NODE_MEMBLKS);
>>>>>
>>>>> +static struct node *get_numa_node(unsigned int id)
>>>>> +{
>>>>> + return &nodes[id];
>>>>> +}
>>>>> +
>>>>> +static nodeid_t get_memblk_nodeid(unsigned int id)
>>>>> +{
>>>>> + return memblk_nodeid[id];
>>>>> +}
>>>>> +
>>>>> +static nodeid_t *get_memblk_nodeid_map(void)
>>>>> +{
>>>>> + return &memblk_nodeid[0];
>>>>> +}
>>>>> +
>>>>> +static struct node *get_node_memblk_range(unsigned int memblk)
>>>>> +{
>>>>> + return &node_memblk_range[memblk];
>>>>> +}
>>>>> +
>>>>> +static int get_num_node_memblks(void)
>>>>> +{
>>>>> + return num_node_memblks;
>>>>> +}
>>>>
>>>>
>>>> They should all be inline functions. And maybe at once lift to a header
>>>> and add proper prefix since you mention they are going to be used later.
>>>
>>>
>>> Currently these are static variables in x86/srat.c file.
>>> In patch #9 I move them to common/numa.c file and make these functions
>>> non-static.
>>>
>>> If I lift them to header file and make inline, then I have to make these
>>> as
>>> global variables.
>>
>>
>> As I said on v2, I am not sure to understand the usefulness of those
>> accessors over global variables...
>
> These are static variables. To access across other files (arch specific)
> these accessors are added. I have to make them global variables to use
> outside of this file.
>
> I am happy to make them global and make these accessors static inline
> as suggested by Wei.
I don't think the helpers are useful... They just return a value without
any sanity check. You can directly use the global variable in the code.
I would much prefer if you use plain global.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 07/24] ARM: NUMA: Add existing ARM numa code under CONFIG_NUMA
2017-07-20 9:31 ` Vijay Kilari
@ 2017-07-20 11:10 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-20 11:10 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
Hi Vijay,
On 20/07/17 10:31, Vijay Kilari wrote:
> On Tue, Jul 18, 2017 at 11:36 PM, Julien Grall <julien.grall@arm.com> wrote:
>> Hi Vijay,
>>
>> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>>
>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>
>>> Right now CONFIG_NUMA is not enabled for ARM and
>>> existing code in asm-arm/numa.h is for !CONFIG_NUMA.
>>> Hence put this code under #ifndef CONFIG_NUMA.
>>>
>>> This help to make this changes work when CONFIG_NUMA
>>> is not enabled. Though CONFIG_NUMA is enabled by default,
>>> manually disabling this option is possible and compilation
>>> should go through. Hence kept the these changes under
>>> !CONFIG_NUMA.
>>
>>
>> This is still no true. It is not possible to disable CONFIG_NUMA from the
>> Kconfig unless you hack it (just tried it)...
>>
>> As I said on v2, if you always enable NUMA why should we add code in Xen
>> that get rotten? Either you allow NUMA to be disabled by the user or you
>> drop this code.
>
> The reason is: The next patch #8, which does the code movement moves
> the generic code to common header file xen/numa.h.
> If we don't put these *existing* defines in asm-arm/numa.h under
> #ifndef CONFIG_NUMA,
> the compilation fails for ARM.
>
> Is it ok to removes these defines under separate patch after enabling
> NUMA config
> at the end of patch series?
>
> Let me know if you have any better approach.
The question here is more, do we want to allow the user disabling NUMA
(even if it has to be guarded with EXPERT)? You seem to choose the
approach where NUMA is here by default and can't be disabled.
1) If we decide to let the user configuring NUMA, then this patch is
valid as it is.
2) If not, then you need a patch drop this code at the end and have a
word in this commit message explaining this is temporary...
When the question above is answered, you need to do either 1) or 2)
according to the answer.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic
2017-07-20 8:55 ` Vijay Kilari
@ 2017-07-20 11:14 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-20 11:14 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
Hi Vijay,
On 20/07/17 09:55, Vijay Kilari wrote:
> On Wed, Jul 19, 2017 at 11:11 PM, Julien Grall <julien.grall@arm.com> wrote:
>> Hi Vijay,
>>
>> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>>
>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>
>>> Move code from xen/arch/x86/numa.c to xen/common/numa.c
>>> so that it can be used by other archs.
>>>
>>> The following changes are done:
>>> - Few generic static functions in x86/numa.c is made
>>> non-static common/numa.c
>>> - The generic contents of header file asm-x86/numa.h
>>> are moved to xen/numa.h.
>>> - The header file includes are reordered and externs are
>>> dropped.
>>> - Moved acpi_numa from asm-x86/acpi.h to xen/acpi.h
>>> - Coding style of code moved to commom/numa.c is changed
>>> to Xen style.
>>> - numa_add_cpu() and numa_set_node() and moved to header
>>> file and added inline function in case of CONFIG_NUMA
>>> is not enabled because these functions are called from
>>> generic code with out any config check.
>>>
>>> Also the node_online_map is defined in x86/numa.c for x86
>>> and arm/smpboot.c for ARM. For x86 it is moved to x86/smpboot.c
>>> If moved to common code the compilation fails because
>>> common/numa.c is compiled only when NUMA is enabled.
>>
>>
>> I would much prefer if this patch does one thing: Moving code. The rest
>> should be split out to help review and allowing us to easily verify you only
>> moved code...
>
> Yes, this patch is doing only code movement. Apart from adding inline function
> for numa_add_cpu() and numa_set_node().
The "apart" should then be in a separate patch. I don't want to spend
hours trying to decipher a patch mixing code movement and add code at
the same time.
>
>>
>>> +#define NODE_DATA(nid) (&(node_data[nid]))
>>> +
>>> +#define node_start_pfn(nid) NODE_DATA(nid)->node_start_pfn
>>> +#define node_spanned_pages(nid) NODE_DATA(nid)->node_spanned_pages
>>> +#define node_end_pfn(nid) NODE_DATA(nid)->node_start_pfn + \
>>> + NODE_DATA(nid)->node_spanned_pages
>>> +
>>> +void numa_add_cpu(int cpu);
>>> +void numa_set_node(int cpu, nodeid_t node);
>>> +#else
>>> +static inline void numa_add_cpu(int cpu) { }
>>> +static inline void numa_set_node(int cpu, nodeid_t node) { }
>>
>>
>> I am not sure why you need to define stub at least for numa_set_node... I
>> can't see use in non-NUMA code. I will comment about the numa_add_cpu later.
>
> x86 is using from setup.c. yes if we assume that numa is always enabled for x86,
> I can drop numa_set_node() inline function.
Looking at the code, I don't think there is any way to disable NUMA on
x86 at the moment... So there is no point to keep it.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 09/24] NUMA: x86: Move common code from srat.c
2017-07-18 11:41 ` [RFC PATCH v3 09/24] NUMA: x86: Move common code from srat.c vijay.kilari
@ 2017-07-20 11:17 ` Julien Grall
2017-07-20 11:43 ` Vijay Kilari
2017-07-24 20:35 ` Stefano Stabellini
0 siblings, 2 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-20 11:17 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Move code from xen/arch/x86/srat.c to xen/common/numa.c
> so that it can be used by other archs.
>
> Apart from moving the code the following changes are done
> - Coding style of code moved to numa.c is changed to xen style
> - {memory,processor}_nodes_parsed are made global and moved
> to xen/nodemask.h
> - Few generic static functions in x86/srat.c are made
> non-static
> - Functions moved from x85/srat.c to common/numa.c are made
> non-static
> - numa_scan_nodes() is made as static function
> - compute_memnode_shift() and setup_node_bootmem() are made
> static.
You modify the coding style at the same time as the same time as moving
the code. This makes quite difficult to make sure that a mistake didn't
slip in the new code. Can you please diving this patch in smaller chunk
(i.e moving code in smaller chunk) to ease the review?
We can think of merging all of them when committing it.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 10/24] NUMA: Allow numa initialization with DT
2017-07-20 10:28 ` Vijay Kilari
@ 2017-07-20 11:20 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-20 11:20 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
Hi Vijay,
On 20/07/17 11:28, Vijay Kilari wrote:
> On Wed, Jul 19, 2017 at 11:28 PM, Julien Grall <julien.grall@arm.com> wrote:
>> Hi Vijay,
>>
>> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>>
>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>
>>> The common code allows numa initialization only when
>>> ACPI_NUMA config is enabled. Allow initialization when
>>> NUMA config is enabled for DT.
>>>
>>> In this patch, along with acpi_numa, check for acpi_disabled
>>> is added.
>>>
>>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>> ---
>>> xen/common/numa.c | 4 +---
>>> 1 file changed, 1 insertion(+), 3 deletions(-)
>>>
>>> diff --git a/xen/common/numa.c b/xen/common/numa.c
>>> index 74c4697..5e985d2 100644
>>> --- a/xen/common/numa.c
>>> +++ b/xen/common/numa.c
>>> @@ -324,7 +324,7 @@ static int __init numa_scan_nodes(paddr_t start,
>>> paddr_t end)
>>> for ( i = 0; i < MAX_NUMNODES; i++ )
>>> cutoff_node(i, start, end);
>>>
>>> - if ( acpi_numa <= 0 )
>>> + if ( !acpi_disabled && acpi_numa <= 0 )
>>
>>
>> I am struggling to understand this change. Likely you want to similar
>> variable for DT to say NUMA is available or this has failed.
>
> Yes, without this check for acpi_disabled, when booting with DT, the check
> acpi_numa <= 0 is true and does not allow numa initialization.
>
>>
>> This also change quite a bit the semantic for x86 because, you will now
>> continue if acpi_disabled and acpi_numa = 0. The code seems to allow it, but
>> I don't know if we support it.
>
> Yes, but prior to this patch, x86 is assuming that acpi_disabled is
> false by checking
> only for acpi_numa <=0.
101 of the contributor: make sure the commit message is meaningful to
understand your changes.
>
> The other solution is create a arch wrapper and call this from here.
I suggested another solution but you seem to have ignored it... You can
rename acpi_numa into something more generic and use it also for DT to
detect whether it is possible to use NUMA.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 13/24] ARM: NUMA: DT: Parse memory NUMA information
2017-07-20 10:37 ` Vijay Kilari
@ 2017-07-20 11:24 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-20 11:24 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
Hi Vijay,
On 20/07/17 11:37, Vijay Kilari wrote:
> On Thu, Jul 20, 2017 at 12:09 AM, Julien Grall <julien.grall@arm.com> wrote:
>> This code looks fairly similar to some bits of
>> acpi_numa_memory_affinity_init. Is there any way we could introduce a common
>> helper?
>
> Yes some bit of code is similar, But acpi_numa_memory_affinity_init() is stuffed
> with some more checks of ACPI data in between the code. So quite complex
> to make it common code.
I think it might be possible to rework acpi_numa_memory_affinity_init to
take out common code... But let's keep like that for now.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 13/24] ARM: NUMA: DT: Parse memory NUMA information
2017-07-19 18:39 ` Julien Grall
2017-07-20 10:37 ` Vijay Kilari
@ 2017-07-20 11:26 ` Julien Grall
2017-07-21 11:10 ` Vijay Kilari
1 sibling, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-20 11:26 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
On 19/07/17 19:39, Julien Grall wrote:
>> cell = (const __be32 *)prop->data;
>> banks = fdt32_to_cpu(prop->len) / (reg_cells * sizeof (u32));
>>
>> - for ( i = 0; i < banks && bootinfo.mem.nr_banks < NR_MEM_BANKS;
>> i++ )
>> + for ( i = 0; i < banks; i++ )
>> {
>> device_tree_get_reg(&cell, address_cells, size_cells, &start,
>> &size);
>> if ( !size )
>> continue;
>> - bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
>> - bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
>> - bootinfo.mem.nr_banks++;
>> + if ( !efi_enabled(EFI_BOOT) && bootinfo.mem.nr_banks <
>> NR_MEM_BANKS )
>> + {
>> + bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
>> + bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
>> + bootinfo.mem.nr_banks++;
>> + }
>
> This change should be split.
I thought a bit more about this code during the week. I think it would
be nicer to write:
#ifdef CONFIG_NUMA
dt_numa_process_memory_node(nid, start, size);
#endif
if ( !efi_enabled(EFI_BOOT) )
continue;
if ( bootinfo.mem.nr_banks < NR_MEM_BANKS )
break;
bootinfo.mem.bank[....];
....
Also, you may want to add a stub for dt_numa_process_memory_node rather
than #ifdef in the code.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 09/24] NUMA: x86: Move common code from srat.c
2017-07-20 11:17 ` Julien Grall
@ 2017-07-20 11:43 ` Vijay Kilari
2017-07-24 20:35 ` Stefano Stabellini
1 sibling, 0 replies; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 11:43 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
Hi Julien,
On Thu, Jul 20, 2017 at 4:47 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi Vijay,
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> Move code from xen/arch/x86/srat.c to xen/common/numa.c
>> so that it can be used by other archs.
>>
>> Apart from moving the code the following changes are done
>> - Coding style of code moved to numa.c is changed to xen style
>> - {memory,processor}_nodes_parsed are made global and moved
>> to xen/nodemask.h
>> - Few generic static functions in x86/srat.c are made
>> non-static
>> - Functions moved from x85/srat.c to common/numa.c are made
>> non-static
>> - numa_scan_nodes() is made as static function
>> - compute_memnode_shift() and setup_node_bootmem() are made
>> static.
>
>
> You modify the coding style at the same time as the same time as moving the
> code. This makes quite difficult to make sure that a mistake didn't slip in
> the new code. Can you please diving this patch in smaller chunk (i.e moving
> code in smaller chunk) to ease the review?
OK. I will do so.
>
> We can think of merging all of them when committing it.
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code
2017-07-20 11:00 ` Julien Grall
@ 2017-07-20 12:05 ` Vijay Kilari
2017-07-20 12:09 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 12:05 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Thu, Jul 20, 2017 at 4:30 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi Vijay,
>
>
> On 20/07/17 08:00, Vijay Kilari wrote:
>>
>> On Wed, Jul 19, 2017 at 9:53 PM, Julien Grall <julien.grall@arm.com>
>> wrote:
>>>
>>> Hi Vijay,
>>>
>>> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>>>
>>>>
>>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>>
>>>> Fix coding style, trailing spaces, tabs in NUMA code.
>>>> Also drop unused macros and functions.
>>>> There is no functional change.
>>>>
>>>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>> Reviewed-by: Wei Liu <wei.liu2@citrix.com>
>>>> ---
>>>> v3: - Change commit message
>>>> - Changed VIRTUAL_BUG_ON to ASSERT
>>>
>>>
>>>
>>> Looking at the commit message you don't mention any renaming...
>>>
>>>> - Dropped useless inner paranthesis for some macros
>>>
>>>
>>>
>>> [...]
>>>
>>>> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
>>>> index 3cf26c2..c0de57b 100644
>>>> --- a/xen/include/asm-x86/numa.h
>>>> +++ b/xen/include/asm-x86/numa.h
>>>> @@ -1,8 +1,11 @@
>>>> -#ifndef _ASM_X8664_NUMA_H
>>>> +#ifndef _ASM_X8664_NUMA_H
>>>> #define _ASM_X8664_NUMA_H 1
>>>>
>>>> #include <xen/cpumask.h>
>>>>
>>>> +#define MAX_NUMNODES NR_NODES
>>>> +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
>>>
>>>
>>>
>>> I don't understand why this suddenly appears in the code when you moved
>>> away
>>> in patch #1 in xen/numa.h.
>>
>>
>> Particularly MAX_NUMNODES required by this header file with this
>> patch changes for compilation.
>> Though I can include xen/numa.h here but xen/numa.h is including
>> asm/numa.h back.
>>
>> I will add separate patch for this defines movement and drop from
>> this patch.
>
>
> Why adding a separate patch? The code should not have been moved away in
> patch #1 as you did.
In patch#1 , I have not moved MAX_NUMNODES. It is kept in xen/numa.h file
In this patch, when VIRTUAL_BUG_ON is changed to ASSERT, in asm-x86/numa.h,
it requires MAX_NUMNODES define. So I have moved it from xen/numa.h to
asm-x86/numa.h
So, I was thinking of adding small patch to move both MAX_NUMNODES and
NR_NODE_MEMBLKS to asm-x86/numa.h
And in code movement patch, I will move to xen/numa.h along with ASSERT code.
>
> But I still don't understand what is the exact error here... If it fails on
> this patch, likely this should have failed after applying patch #1. And
> *all* patch should be able to build without the rest of the series.
Yes, all patches are tested for compilation individually.
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code
2017-07-20 12:05 ` Vijay Kilari
@ 2017-07-20 12:09 ` Julien Grall
2017-07-20 12:29 ` Vijay Kilari
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-20 12:09 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On 20/07/17 13:05, Vijay Kilari wrote:
> On Thu, Jul 20, 2017 at 4:30 PM, Julien Grall <julien.grall@arm.com> wrote:
>> Hi Vijay,
>>
>>
>> On 20/07/17 08:00, Vijay Kilari wrote:
>>>
>>> On Wed, Jul 19, 2017 at 9:53 PM, Julien Grall <julien.grall@arm.com>
>>> wrote:
>>>>
>>>> Hi Vijay,
>>>>
>>>> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>>>>
>>>>>
>>>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>>>
>>>>> Fix coding style, trailing spaces, tabs in NUMA code.
>>>>> Also drop unused macros and functions.
>>>>> There is no functional change.
>>>>>
>>>>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>>> Reviewed-by: Wei Liu <wei.liu2@citrix.com>
>>>>> ---
>>>>> v3: - Change commit message
>>>>> - Changed VIRTUAL_BUG_ON to ASSERT
>>>>
>>>>
>>>>
>>>> Looking at the commit message you don't mention any renaming...
>>>>
>>>>> - Dropped useless inner paranthesis for some macros
>>>>
>>>>
>>>>
>>>> [...]
>>>>
>>>>> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
>>>>> index 3cf26c2..c0de57b 100644
>>>>> --- a/xen/include/asm-x86/numa.h
>>>>> +++ b/xen/include/asm-x86/numa.h
>>>>> @@ -1,8 +1,11 @@
>>>>> -#ifndef _ASM_X8664_NUMA_H
>>>>> +#ifndef _ASM_X8664_NUMA_H
>>>>> #define _ASM_X8664_NUMA_H 1
>>>>>
>>>>> #include <xen/cpumask.h>
>>>>>
>>>>> +#define MAX_NUMNODES NR_NODES
>>>>> +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
>>>>
>>>>
>>>>
>>>> I don't understand why this suddenly appears in the code when you moved
>>>> away
>>>> in patch #1 in xen/numa.h.
>>>
>>>
>>> Particularly MAX_NUMNODES required by this header file with this
>>> patch changes for compilation.
>>> Though I can include xen/numa.h here but xen/numa.h is including
>>> asm/numa.h back.
>>>
>>> I will add separate patch for this defines movement and drop from
>>> this patch.
>>
>>
>> Why adding a separate patch? The code should not have been moved away in
>> patch #1 as you did.
>
> In patch#1 , I have not moved MAX_NUMNODES. It is kept in xen/numa.h file
> In this patch, when VIRTUAL_BUG_ON is changed to ASSERT, in asm-x86/numa.h,
> it requires MAX_NUMNODES define. So I have moved it from xen/numa.h to
> asm-x86/numa.h
I am sorry but looked at your patch #1. You moved NR_NODE_MEMBLKS in
patch #1 from asm-x86/numa.h to xen/numa.h. And then you moved it again
here.
>
> So, I was thinking of adding small patch to move both MAX_NUMNODES and
> NR_NODE_MEMBLKS to asm-x86/numa.h
Or better, you can do in xen/numa.h:
#define MAX_NUMNODES ...
#define NR_NODE_...
#include <asm/numa.h>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code
2017-07-20 12:09 ` Julien Grall
@ 2017-07-20 12:29 ` Vijay Kilari
2017-07-20 12:33 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-20 12:29 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On Thu, Jul 20, 2017 at 5:39 PM, Julien Grall <julien.grall@arm.com> wrote:
>
>
> On 20/07/17 13:05, Vijay Kilari wrote:
>>
>> On Thu, Jul 20, 2017 at 4:30 PM, Julien Grall <julien.grall@arm.com>
>> wrote:
>>>
>>> Hi Vijay,
>>>
>>>
>>> On 20/07/17 08:00, Vijay Kilari wrote:
>>>>
>>>>
>>>> On Wed, Jul 19, 2017 at 9:53 PM, Julien Grall <julien.grall@arm.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> Hi Vijay,
>>>>>
>>>>> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>>>>
>>>>>> Fix coding style, trailing spaces, tabs in NUMA code.
>>>>>> Also drop unused macros and functions.
>>>>>> There is no functional change.
>>>>>>
>>>>>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>>>>> Reviewed-by: Wei Liu <wei.liu2@citrix.com>
>>>>>> ---
>>>>>> v3: - Change commit message
>>>>>> - Changed VIRTUAL_BUG_ON to ASSERT
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Looking at the commit message you don't mention any renaming...
>>>>>
>>>>>> - Dropped useless inner paranthesis for some macros
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> [...]
>>>>>
>>>>>> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
>>>>>> index 3cf26c2..c0de57b 100644
>>>>>> --- a/xen/include/asm-x86/numa.h
>>>>>> +++ b/xen/include/asm-x86/numa.h
>>>>>> @@ -1,8 +1,11 @@
>>>>>> -#ifndef _ASM_X8664_NUMA_H
>>>>>> +#ifndef _ASM_X8664_NUMA_H
>>>>>> #define _ASM_X8664_NUMA_H 1
>>>>>>
>>>>>> #include <xen/cpumask.h>
>>>>>>
>>>>>> +#define MAX_NUMNODES NR_NODES
>>>>>> +#define NR_NODE_MEMBLKS (MAX_NUMNODES * 2)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> I don't understand why this suddenly appears in the code when you moved
>>>>> away
>>>>> in patch #1 in xen/numa.h.
>>>>
>>>>
>>>>
>>>> Particularly MAX_NUMNODES required by this header file with this
>>>> patch changes for compilation.
>>>> Though I can include xen/numa.h here but xen/numa.h is including
>>>> asm/numa.h back.
>>>>
>>>> I will add separate patch for this defines movement and drop from
>>>> this patch.
>>>
>>>
>>>
>>> Why adding a separate patch? The code should not have been moved away in
>>> patch #1 as you did.
>>
>>
>> In patch#1 , I have not moved MAX_NUMNODES. It is kept in xen/numa.h file
>> In this patch, when VIRTUAL_BUG_ON is changed to ASSERT, in
>> asm-x86/numa.h,
>> it requires MAX_NUMNODES define. So I have moved it from xen/numa.h to
>> asm-x86/numa.h
>
>
> I am sorry but looked at your patch #1. You moved NR_NODE_MEMBLKS in patch
> #1 from asm-x86/numa.h to xen/numa.h. And then you moved it again here.
>
>>
>> So, I was thinking of adding small patch to move both MAX_NUMNODES and
>> NR_NODE_MEMBLKS to asm-x86/numa.h
>
>
> Or better, you can do in xen/numa.h:
>
> #define MAX_NUMNODES ...
> #define NR_NODE_...
>
> #include <asm/numa.h>
But still compilation issue comes from below code.
where only asm/numa.h is included.
--- a/xen/include/asm-x86/irq.h
+++ b/xen/include/asm-x86/irq.h
@@ -4,7 +4,7 @@
/* (C) 1992, 1993 Linus Torvalds, (C) 1997 Ingo Molnar */
#include <asm/atomic.h>
-#include <asm/numa.h>
+#include <xen/numa.h>
#include <xen/cpumask.h>
#include <xen/smp.h>
#include <xen/hvm/irq.h>
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code
2017-07-20 12:29 ` Vijay Kilari
@ 2017-07-20 12:33 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-20 12:33 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On 20/07/17 13:29, Vijay Kilari wrote:
> On Thu, Jul 20, 2017 at 5:39 PM, Julien Grall <julien.grall@arm.com> wrote:
>
> But still compilation issue comes from below code.
> where only asm/numa.h is included.
>
> --- a/xen/include/asm-x86/irq.h
> +++ b/xen/include/asm-x86/irq.h
> @@ -4,7 +4,7 @@
> /* (C) 1992, 1993 Linus Torvalds, (C) 1997 Ingo Molnar */
>
> #include <asm/atomic.h>
> -#include <asm/numa.h>
> +#include <xen/numa.h>
> #include <xen/cpumask.h>
> #include <xen/smp.h>
> #include <xen/hvm/irq.h>
Send a patch to fix that then.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 14/24] ARM: NUMA: DT: Parse NUMA distance information
2017-07-18 11:41 ` [RFC PATCH v3 14/24] ARM: NUMA: DT: Parse NUMA distance information vijay.kilari
@ 2017-07-20 13:02 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-20 13:02 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Parse distance-matrix and fetch node distance information.
> Store distance information in node_distance[].
>
> Register dt_node_distance() function pointer with
> the ARM numa code. This approach can be later used for
> ACPI.
After my comment on v1, I was expecting to see a link to the binding in
the commit message...
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> v3: - Moved __node_distance() declaration to common
> header file
> - Use device_tree_node_compatible() instead of
> device_tree_node_matches()
> - Dropped xen/errno.h inclusion
> ---
> xen/arch/arm/numa/dt_numa.c | 131 ++++++++++++++++++++++++++++++++++++++++++++
> xen/arch/arm/numa/numa.c | 22 ++++++++
> xen/include/asm-arm/numa.h | 2 +
> xen/include/asm-x86/numa.h | 1 -
> xen/include/xen/numa.h | 3 +
> 5 files changed, 158 insertions(+), 1 deletion(-)
>
> diff --git a/xen/arch/arm/numa/dt_numa.c b/xen/arch/arm/numa/dt_numa.c
> index 84030e7..46c0346 100644
> --- a/xen/arch/arm/numa/dt_numa.c
> +++ b/xen/arch/arm/numa/dt_numa.c
> @@ -23,6 +23,48 @@
> #include <xen/numa.h>
> #include <asm/setup.h>
>
> +static uint8_t node_distance[MAX_NUMNODES][MAX_NUMNODES];
On v1, you said that you will look at allocating node_distance on the
fly. So why it is not done?
You can give a look at alloc_boot_pages(...).
> +
> +static uint8_t dt_node_distance(nodeid_t nodea, nodeid_t nodeb)
> +{
> + if ( nodea >= MAX_NUMNODES || nodeb >= MAX_NUMNODES )
> + return nodea == nodeb ? LOCAL_DISTANCE : REMOTE_DISTANCE;
Do we really expect dt_node_distance to be called with wrong node?
Looking at the ACPI code, they don't check that... So likely this should
be an ASSERT(...).
> +
> + return node_distance[nodea][nodeb];
> +}
> +
> +static int dt_numa_set_distance(uint32_t nodea, uint32_t nodeb,
I think this should be __init.
> + uint32_t distance)
> +{
> + /* node_distance is uint8_t. Ensure distance is less than 255 */
> + if ( nodea >= MAX_NUMNODES || nodeb >= MAX_NUMNODES || distance > 255 )
> + return -EINVAL;
> +
> + node_distance[nodea][nodeb] = distance;
> +
> + return 0;
> +}
> +
> +void init_dt_numa_distance(void)
Ditto.
> +{
> + int i, j;
> +
> + for ( i = 0; i < MAX_NUMNODES; i++ )
> + {
> + for ( j = 0; j < MAX_NUMNODES; j++ )
> + {
> + /*
> + * Initialize distance 10 for local distance and
> + * 20 for remote distance.
> + */
> + if ( i == j )
> + node_distance[i][j] = LOCAL_DISTANCE;
> + else
> + node_distance[i][j] = REMOTE_DISTANCE;
> + }
> + }
> +}
> +
> /*
> * Even though we connect cpus to numa domains later in SMP
> * init, we need to know the node ids now for all cpus.
> @@ -58,6 +100,76 @@ static int __init dt_numa_process_cpu_node(const void *fdt)
> return 0;
> }
>
> +static int __init dt_numa_parse_distance_map(const void *fdt, int node,
> + const char *name,
> + uint32_t address_cells,
> + uint32_t size_cells)
> +{
> + const struct fdt_property *prop;
> + const __be32 *matrix;
> + int entry_count, len, i;
> +
> + printk(XENLOG_INFO "NUMA: parsing numa-distance-map\n");
> +
> + prop = fdt_get_property(fdt, node, "distance-matrix", &len);
> + if ( !prop )
> + {
> + printk(XENLOG_WARNING
s/XENLOG_WARNING/XENLOG_INFO/ because numa-distance-map is not mandatory.
> + "NUMA: No distance-matrix property in distance-map\n");
> +
> + return -EINVAL;
If I am reading correctly the binding, the distance-matrix is not
mandatory. If it is not present, you should use a default matrix. But
here you will disable NUMA completely.
> + }
> +
> + if ( len % sizeof(uint32_t) != 0 )
> + {
> + printk(XENLOG_WARNING
> + "distance-matrix in node is not a multiple of u32\n");
> +
> + return -EINVAL;
> + }
> +
> + entry_count = len / sizeof(uint32_t);
> + if ( entry_count <= 0 )
> + {
> + printk(XENLOG_WARNING "NUMA: Invalid distance-matrix\n");
> +
> + return -EINVAL;
> + }
> +
> + matrix = (const __be32 *)prop->data;
> + for ( i = 0; i + 2 < entry_count; i += 3 )
It would be easier to read if entry_count is the number of triplet. E.g
entry_count = (len / sizeof(uint32_t)) / 3;
for ( i = 0; i < entry_count; i++ )
> + {
> + uint32_t nodea, nodeb, distance;
> +
> + nodea = dt_read_number(matrix, 1);
> + matrix++;
nodea = dt_next_cell(1, &matrix) will do the increment for you.
> + nodeb = dt_read_number(matrix, 1);
> + matrix++;
Ditto.
> + distance = dt_read_number(matrix, 1);
> + matrix++;
Ditto.
> +
> + if ( dt_numa_set_distance(nodea, nodeb, distance) )
> + {
> + printk(XENLOG_WARNING
> + "NUMA: node-id out of range in distance matrix for [node%d -> node%d]\n",
s/%d/%u/
> + nodea, nodeb);
> + return -EINVAL;
> +
> + }
> + printk(XENLOG_INFO "NUMA: distance[node%d -> node%d] = %d\n",
> + nodea, nodeb, distance);
> +
> + /*
> + * Set default distance of node B->A same as A->B.
> + * No need to check for return value of numa_set_distance.
> + */
> + if ( nodeb > nodea )
Mind explaining this if in the comment?
> + dt_numa_set_distance(nodeb, nodea, distance);
> + }
> +
> + return 0;
> +}
> +
> void __init dt_numa_process_memory_node(uint32_t nid, paddr_t start,
> paddr_t size)
> {
> @@ -90,11 +202,30 @@ void __init dt_numa_process_memory_node(uint32_t nid, paddr_t start,
> return;
> }
>
> +static int __init dt_numa_scan_distance_node(const void *fdt, int node,
> + const char *name, int depth,
> + uint32_t address_cells,
> + uint32_t size_cells, void *data)
> +{
> + if ( device_tree_node_compatible(fdt, node, "numa-distance-map-v1") )
> + return dt_numa_parse_distance_map(fdt, node, name, address_cells,
> + size_cells);
> +
> + return 0;
> +}
> +
> int __init dt_numa_init(void)
> {
> int ret;
>
> ret = dt_numa_process_cpu_node((void *)device_tree_flattened);
> + if ( ret )
> + return ret;
> +
> + ret = device_tree_for_each_node((void *)device_tree_flattened,
Why do you need the cast?
> + dt_numa_scan_distance_node, NULL);
> + if ( !ret )
> + register_node_distance(&dt_node_distance);
>
> return ret;
> }
> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
> index 8227361..c00b92c 100644
> --- a/xen/arch/arm/numa/numa.c
> +++ b/xen/arch/arm/numa/numa.c
> @@ -18,10 +18,30 @@
> #include <xen/ctype.h>
> #include <xen/nodemask.h>
> #include <xen/numa.h>
> +#include <asm/acpi.h>
I don't understand why you include asm/acpi.h with no code using ACPI at
the moment...
> +
> +static uint8_t (*node_distance_fn)(nodeid_t a, nodeid_t b);
>
> void numa_failed(void)
> {
> numa_off = true;
> + init_dt_numa_distance();
Why do you need to initialize init_dt_numa_distance when it has failed?
The array will never be used in that case.
> + node_distance_fn = NULL;
> +}
> +
> +uint8_t __node_distance(nodeid_t a, nodeid_t b)
> +{
> + if ( node_distance_fn != NULL);
> + return node_distance_fn(a, b);
> +
> + return a == b ? LOCAL_DISTANCE : REMOTE_DISTANCE;
> +}
> +
> +EXPORT_SYMBOL(__node_distance);
Please drop EXPORT_SYMBOL, this is not used by Xen and only here when
the code is imported from Linux.
> +
> +void register_node_distance(uint8_t (fn)(nodeid_t a, nodeid_t b))
> +{
> + node_distance_fn = fn;
> }
>
> void __init numa_init(void)
> @@ -29,6 +49,8 @@ void __init numa_init(void)
> int ret = 0;
>
> nodes_clear(processor_nodes_parsed);
> + init_dt_numa_distance();
This should go in dt_numa_init and would avoid to export it.
> +
> if ( numa_off )
> goto no_numa;
>
> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> index 36cd782..d1dc83a 100644
> --- a/xen/include/asm-arm/numa.h
> +++ b/xen/include/asm-arm/numa.h
> @@ -4,6 +4,8 @@
> typedef uint8_t nodeid_t;
>
> void dt_numa_process_memory_node(uint32_t nid, paddr_t start, paddr_t size);
> +void register_node_distance(uint8_t (fn)(nodeid_t a, nodeid_t b));
> +void init_dt_numa_distance(void);
>
> #ifdef CONFIG_NUMA
> void numa_init(void);
> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
> index d8a0a44..ca0a2a6 100644
> --- a/xen/include/asm-x86/numa.h
> +++ b/xen/include/asm-x86/numa.h
> @@ -18,7 +18,6 @@ extern nodeid_t apicid_to_node[];
> extern void init_cpu_to_node(void);
>
> void srat_parse_regions(paddr_t addr);
> -extern uint8_t __node_distance(nodeid_t a, nodeid_t b);
> unsigned int arch_get_dma_bitsize(void);
>
> #endif
> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
> index 110d5dc..10ef4c4 100644
> --- a/xen/include/xen/numa.h
> +++ b/xen/include/xen/numa.h
> @@ -6,6 +6,8 @@
> #include <asm/numa.h>
>
> #define NUMA_NO_NODE 0xFF
> +#define LOCAL_DISTANCE 10
> +#define REMOTE_DISTANCE 20
I would add DEFAULT in each name. Probably LOCAL_DEFAULT_DISTANCE and
REMOVE_LOCAL_DISTANCE.
> #define NUMA_NO_DISTANCE 0xFF
>
> #define MAX_NUMNODES NR_NODES
> @@ -70,6 +72,7 @@ int numa_add_memblk(nodeid_t nodeid, paddr_t start, uint64_t size);
> int get_num_node_memblks(void);
> bool arch_sanitize_nodes_memory(void);
> void numa_failed(void);
> +uint8_t __node_distance(nodeid_t a, nodeid_t b);
> #else
> static inline void numa_add_cpu(int cpu) { }
> static inline void numa_set_node(int cpu, nodeid_t node) { }
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 13/24] ARM: NUMA: DT: Parse memory NUMA information
2017-07-20 11:26 ` Julien Grall
@ 2017-07-21 11:10 ` Vijay Kilari
2017-07-21 12:35 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-21 11:10 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
Hi Julien,
On Thu, Jul 20, 2017 at 4:56 PM, Julien Grall <julien.grall@arm.com> wrote:
>
>
> On 19/07/17 19:39, Julien Grall wrote:
>>>
>>> cell = (const __be32 *)prop->data;
>>> banks = fdt32_to_cpu(prop->len) / (reg_cells * sizeof (u32));
>>>
>>> - for ( i = 0; i < banks && bootinfo.mem.nr_banks < NR_MEM_BANKS;
>>> i++ )
>>> + for ( i = 0; i < banks; i++ )
>>> {
>>> device_tree_get_reg(&cell, address_cells, size_cells, &start,
>>> &size);
>>> if ( !size )
>>> continue;
>>> - bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
>>> - bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
>>> - bootinfo.mem.nr_banks++;
>>> + if ( !efi_enabled(EFI_BOOT) && bootinfo.mem.nr_banks <
>>> NR_MEM_BANKS )
>>> + {
>>> + bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
>>> + bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
>>> + bootinfo.mem.nr_banks++;
>>> + }
>>
>>
>> This change should be split.
>
>
> I thought a bit more about this code during the week. I think it would be
> nicer to write:
>
> #ifdef CONFIG_NUMA
> dt_numa_process_memory_node(nid, start, size);
> #endif
>
> if ( !efi_enabled(EFI_BOOT) )
> continue;
Should be if ( efi_enabled(EFI_BOOT) ) ?
>
> if ( bootinfo.mem.nr_banks < NR_MEM_BANKS )
Should be if ( bootinfo.mem.nr_banks >= NR_MEM_BANKS ) ?
> break;
>
> bootinfo.mem.bank[....];
> ....
>
> Also, you may want to add a stub for dt_numa_process_memory_node rather than
> #ifdef in the code.
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 13/24] ARM: NUMA: DT: Parse memory NUMA information
2017-07-21 11:10 ` Vijay Kilari
@ 2017-07-21 12:35 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-21 12:35 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On 21/07/17 12:10, Vijay Kilari wrote:
> Hi Julien,
>
> On Thu, Jul 20, 2017 at 4:56 PM, Julien Grall <julien.grall@arm.com> wrote:
>>
>>
>> On 19/07/17 19:39, Julien Grall wrote:
>>>>
>>>> cell = (const __be32 *)prop->data;
>>>> banks = fdt32_to_cpu(prop->len) / (reg_cells * sizeof (u32));
>>>>
>>>> - for ( i = 0; i < banks && bootinfo.mem.nr_banks < NR_MEM_BANKS;
>>>> i++ )
>>>> + for ( i = 0; i < banks; i++ )
>>>> {
>>>> device_tree_get_reg(&cell, address_cells, size_cells, &start,
>>>> &size);
>>>> if ( !size )
>>>> continue;
>>>> - bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
>>>> - bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
>>>> - bootinfo.mem.nr_banks++;
>>>> + if ( !efi_enabled(EFI_BOOT) && bootinfo.mem.nr_banks <
>>>> NR_MEM_BANKS )
>>>> + {
>>>> + bootinfo.mem.bank[bootinfo.mem.nr_banks].start = start;
>>>> + bootinfo.mem.bank[bootinfo.mem.nr_banks].size = size;
>>>> + bootinfo.mem.nr_banks++;
>>>> + }
>>>
>>>
>>> This change should be split.
>>
>>
>> I thought a bit more about this code during the week. I think it would be
>> nicer to write:
>>
>> #ifdef CONFIG_NUMA
>> dt_numa_process_memory_node(nid, start, size);
>> #endif
>>
>> if ( !efi_enabled(EFI_BOOT) )
>> continue;
>
> Should be if ( efi_enabled(EFI_BOOT) ) ?
>>
>> if ( bootinfo.mem.nr_banks < NR_MEM_BANKS )
>
> Should be if ( bootinfo.mem.nr_banks >= NR_MEM_BANKS ) ?
Yes for both. I wrote too quickly this e-mail.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support
2017-07-18 11:41 ` [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support vijay.kilari
@ 2017-07-24 11:24 ` Julien Grall
2017-07-25 6:47 ` Vijay Kilari
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-24 11:24 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> For each cpu, update cpu_to_node[] with node id from
> the numa-node-id DT property. Also, initialize cpu_to_node[]
> with node 0.
>
> Add macros to access cpu_to_node[] information.
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> v3: - Dropped numa_add_cpu declaration from asm-arm/numa.h
> - Dropped stale declarations
> - Call numa_add_cpu for cpu0
> ---
> xen/arch/arm/numa/numa.c | 21 +++++++++++++++++++++
> xen/arch/arm/setup.c | 2 ++
> xen/arch/arm/smpboot.c | 25 ++++++++++++++++++++++++-
> xen/include/asm-arm/numa.h | 7 +++++++
> xen/include/asm-x86/numa.h | 1 -
> xen/include/xen/numa.h | 1 +
> 6 files changed, 55 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
> index c00b92c..dc80aa5 100644
> --- a/xen/arch/arm/numa/numa.c
> +++ b/xen/arch/arm/numa/numa.c
> @@ -22,11 +22,31 @@
>
> static uint8_t (*node_distance_fn)(nodeid_t a, nodeid_t b);
>
> +/*
> + * Setup early cpu_to_node.
> + */
> +void __init init_cpu_to_node(void)
> +{
> + int i;
> +
> + for ( i = 0; i < NR_CPUS; i++ )
> + numa_set_node(i, 0);
> +}
From the comment: "Setup early cpu_to_node". However this is not how
you are using it.
But I am not sure why it is even here...
> +
> void numa_failed(void)
> {
> numa_off = true;
> init_dt_numa_distance();
> node_distance_fn = NULL;
> + init_cpu_to_node();
> +}
> +
> +void __init numa_set_cpu_node(int cpu, unsigned int nid)
> +{
> + if ( !node_isset(nid, processor_nodes_parsed) || nid >= MAX_NUMNODES )
> + nid = 0;
This looks wrong to me. If the node-id is invalid, why would you blindly
set to 0?
> +
> + numa_set_node(cpu, nid);
> }
>
> uint8_t __node_distance(nodeid_t a, nodeid_t b)
> @@ -49,6 +69,7 @@ void __init numa_init(void)
> int ret = 0;
>
> nodes_clear(processor_nodes_parsed);
> + init_cpu_to_node();
> init_dt_numa_distance();
>
> if ( numa_off )
> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
> index a6d1499..b9c8b0d 100644
> --- a/xen/arch/arm/setup.c
> +++ b/xen/arch/arm/setup.c
> @@ -787,6 +787,8 @@ void __init start_xen(unsigned long boot_phys_offset,
>
> processor_id();
>
> + numa_add_cpu(0);
> +
> smp_init_cpus();
> cpus = smp_get_max_cpus();
> printk(XENLOG_INFO "SMP: Allowing %u CPUs\n", cpus);
> diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
> index 32e8722..fcf9afc 100644
> --- a/xen/arch/arm/smpboot.c
> +++ b/xen/arch/arm/smpboot.c
> @@ -29,6 +29,7 @@
> #include <xen/timer.h>
> #include <xen/irq.h>
> #include <xen/console.h>
> +#include <xen/numa.h>
Please use the alphabetical order.
> #include <asm/cpuerrata.h>
> #include <asm/gic.h>
> #include <asm/psci.h>
> @@ -106,6 +107,7 @@ static void __init dt_smp_init_cpus(void)
> [0 ... NR_CPUS - 1] = MPIDR_INVALID
> };
> bool_t bootcpu_valid = 0;
> + nodeid_t *cpu_to_nodemap;
> int rc;
>
> mpidr = boot_cpu_data.mpidr.bits & MPIDR_HWID_MASK;
> @@ -117,11 +119,18 @@ static void __init dt_smp_init_cpus(void)
> return;
> }
>
> + cpu_to_nodemap = xzalloc_array(nodeid_t, NR_CPUS);
Why do you need to allocate cpu_to_nodemap? Would not it be easier to
put it on the stack as we do for other variable?
> + if ( !cpu_to_nodemap )
> + {
> + printk(XENLOG_WARNING "Failed to allocate memory for cpu_to_nodemap\n");
> + return;
> + }
> +
> dt_for_each_child_node( cpus, cpu )
> {
> const __be32 *prop;
> u64 addr;
> - u32 reg_len;
> + uint32_t reg_len, nid;
> register_t hwid;
>
> if ( !dt_device_type_is_equal(cpu, "cpu") )
> @@ -146,6 +155,15 @@ static void __init dt_smp_init_cpus(void)
> continue;
> }
>
> + if ( !dt_property_read_u32(cpu, "numa-node-id", &nid) )
> + {
> + printk(XENLOG_WARNING "cpu node `%s`: numa-node-id not found\n",
> + dt_node_full_name(cpu));
numa-node-id is not mandatory. So you would print a warning on all
non-NUMA platform. This not what we want.
> + nid = 0;
> + }
> +
> + cpu_to_nodemap[cpuidx] = nid;
> +
> addr = dt_read_number(prop, dt_n_addr_cells(cpu));
>
> hwid = addr;
> @@ -224,6 +242,7 @@ static void __init dt_smp_init_cpus(void)
> {
> printk(XENLOG_WARNING "DT missing boot CPU MPIDR[23:0]\n"
> "Using only 1 CPU\n");
> + xfree(cpu_to_nodemap);
> return;
> }
>
> @@ -233,7 +252,10 @@ static void __init dt_smp_init_cpus(void)
> continue;
> cpumask_set_cpu(i, &cpu_possible_map);
> cpu_logical_map(i) = tmp_map[i];
> + numa_set_cpu_node(i, cpu_to_nodemap[i]);
> }
> +
> + xfree(cpu_to_nodemap);
> }
>
> void __init smp_init_cpus(void)
> @@ -313,6 +335,7 @@ void start_secondary(unsigned long boot_phys_offset,
> */
> smp_wmb();
>
> + numa_add_cpu(cpuid);
Newline here please.
> /* Now report this CPU is up */
> cpumask_set_cpu(cpuid, &cpu_online_map);
>
> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> index d1dc83a..0d3146c 100644
> --- a/xen/include/asm-arm/numa.h
> +++ b/xen/include/asm-arm/numa.h
> @@ -10,12 +10,19 @@ void init_dt_numa_distance(void);
> #ifdef CONFIG_NUMA
> void numa_init(void);
> int dt_numa_init(void);
> +void numa_set_cpu_node(int cpu, unsigned int nid);
> +
> #else
> static inline void numa_init(void)
> {
> return;
> }
>
> +static inline void numa_set_cpu_node(int cpu, unsigned int nid)
> +{
> + return;
> +}
> +
> /* Fake one node for now. See also node_online_map. */
> #define cpu_to_node(cpu) 0
> #define node_to_cpumask(node) (cpu_online_map)
> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
> index ca0a2a6..fc4747f 100644
> --- a/xen/include/asm-x86/numa.h
> +++ b/xen/include/asm-x86/numa.h
> @@ -15,7 +15,6 @@ extern nodeid_t acpi_setup_node(unsigned int pxm);
> extern void srat_detect_node(int cpu);
>
> extern nodeid_t apicid_to_node[];
> -extern void init_cpu_to_node(void);
>
> void srat_parse_regions(paddr_t addr);
> unsigned int arch_get_dma_bitsize(void);
> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
> index 10ef4c4..8a306e7 100644
> --- a/xen/include/xen/numa.h
> +++ b/xen/include/xen/numa.h
> @@ -30,6 +30,7 @@ extern s8 acpi_numa;
> void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
> int srat_disabled(void);
> int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
> +void init_cpu_to_node(void);
You never used this function in common code. So why did you move it in
the common headers?
>
> #ifdef CONFIG_NUMA
> #define cpu_to_node(cpu) (cpu_to_node[cpu])
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 16/24] ARM: NUMA: Add memory NUMA support
2017-07-18 11:41 ` [RFC PATCH v3 16/24] ARM: NUMA: Add memory " vijay.kilari
@ 2017-07-24 12:43 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-24 12:43 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Implement arch_sanitize_nodes_memory() which looks at all banks
> in bootinfo.mem, update nodes[] with corresponding nodeid.
> Call numa_scan_nodes() generic function with ram start and
> end address, which takes care of further computing memnodeshift
> and populating memnodemap[] using generic implementation.
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> ---
> v3: - Dropped common code from asm-arm/numa.h
> - Re-used numa_initmem_init() from common code.
> ---
> xen/arch/arm/numa/numa.c | 77 +++++++++++++++++++++++++++++++++++++++++++++++-
> xen/common/numa.c | 14 +++++++++
> xen/include/xen/numa.h | 1 +
> 3 files changed, 91 insertions(+), 1 deletion(-)
>
> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
> index dc80aa5..85352dc 100644
> --- a/xen/arch/arm/numa/numa.c
> +++ b/xen/arch/arm/numa/numa.c
> @@ -18,6 +18,7 @@
> #include <xen/ctype.h>
> #include <xen/nodemask.h>
> #include <xen/numa.h>
> +#include <xen/pfn.h>
> #include <asm/acpi.h>
>
> static uint8_t (*node_distance_fn)(nodeid_t a, nodeid_t b);
> @@ -64,9 +65,66 @@ void register_node_distance(uint8_t (fn)(nodeid_t a, nodeid_t b))
> node_distance_fn = fn;
> }
>
> +bool __init arch_sanitize_nodes_memory(void)
Likely when I say a function is confusing on a previous version, it
means you have to add more comments in the function...
> +{
> + nodemask_t mem_nodes_parsed;
> + int bank, nodeid;
> + struct node *nd;
> + paddr_t start, size, end;
> +
> + nodes_clear(mem_nodes_parsed);
> + for ( bank = 0 ; bank < bootinfo.mem.nr_banks; bank++ )
> + {
> + start = bootinfo.mem.bank[bank].start;
> + size = bootinfo.mem.bank[bank].size;
> + end = start + size;
> +
> + nodeid = get_mem_nodeid(start, end);
> + if ( nodeid >= NUMA_NO_NODE )
This check is very confusing.
> + {
> + printk(XENLOG_WARNING
> + "NUMA: node for mem bank start 0x%lx - 0x%lx not found\n",
start and end are paddr_t should it should be PRI_paddr.
> + start, end);
> +
> + return false;
> + }
> +
> + nd = get_numa_node(nodeid);
> + if ( !node_test_and_set(nodeid, mem_nodes_parsed) )
> + {
> + nd->start = start;
> + nd->end = end;
> + }
> + else
> + {
> + if ( start < nd->start )
> + nd->start = start;
> + if ( nd->end < end )
> + nd->end = end;
This function is called "sanitize nodes", but here you also update
start/end. Mind explaining why you need this on ARM when it is not
necessary on x86?
> + }
> + }
> +
> + return true;
> +}
> +
> +static void __init numa_reset_numa_nodes(void)
> +{
> + int i;
> + struct node *nd;
> +
> + for ( i = 0; i < MAX_NUMNODES; i++ )
> + {
> + nd = get_numa_node(i);
> + nd->start = 0;
> + nd->end = 0;
> + }
> +}
> +
> void __init numa_init(void)
> {
> - int ret = 0;
> + int ret = 0, bank;
> + paddr_t ram_start = ~0;
> + paddr_t ram_end = 0;
>
> nodes_clear(processor_nodes_parsed);
> init_cpu_to_node();
> @@ -83,6 +141,23 @@ void __init numa_init(void)
> }
>
> no_numa:
> + for ( bank = 0 ; bank < bootinfo.mem.nr_banks; bank++ )
> + {
> + paddr_t bank_start = bootinfo.mem.bank[bank].start;
> + paddr_t bank_end = bank_start + bootinfo.mem.bank[bank].size;
> +
> + ram_start = min(ram_start, bank_start);
> + ram_end = max(ram_end, bank_end);
> + }
> +
> + /*
> + * In arch_sanitize_nodes_memory() we update nodes[] properly.
> + * Hence we reset the nodes[] before calling numa_scan_nodes().
> + */
> + numa_reset_numa_nodes();
> +
> + numa_initmem_init(PFN_UP(ram_start), PFN_DOWN(ram_end));
I might miss something. numa_initmem_init is here to scan the NUMA nodes
and set them up. So why are you calling it here?
> +
> return;
> }
>
> diff --git a/xen/common/numa.c b/xen/common/numa.c
> index 5e985d2..0f79a07 100644
> --- a/xen/common/numa.c
> +++ b/xen/common/numa.c
> @@ -76,6 +76,20 @@ nodeid_t get_memblk_nodeid(unsigned int id)
> return memblk_nodeid[id];
> }
>
> +int __init get_mem_nodeid(paddr_t start, paddr_t end)
> +{
> + unsigned int i;
> +
> + for ( i = 0; i < get_num_node_memblks(); i++ )
> + {
> + if ( start >= node_memblk_range[i].start &&
> + end <= node_memblk_range[i].end )
> + return memblk_nodeid[i];
> + }
> +
> + return -EINVAL;
> +}
> +
> static nodeid_t *get_memblk_nodeid_map(void)
> {
> return &memblk_nodeid[0];
> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
> index 8a306e7..a541eb7 100644
> --- a/xen/include/xen/numa.h
> +++ b/xen/include/xen/numa.h
> @@ -70,6 +70,7 @@ struct node *get_numa_node(unsigned int id);
> nodeid_t get_memblk_nodeid(unsigned int memblk);
> struct node *get_node_memblk_range(unsigned int memblk);
> int numa_add_memblk(nodeid_t nodeid, paddr_t start, uint64_t size);
> +int get_mem_nodeid(paddr_t start, paddr_t end);
> int get_num_node_memblks(void);
> bool arch_sanitize_nodes_memory(void);
> void numa_failed(void);
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic
2017-07-19 17:41 ` Julien Grall
2017-07-20 8:55 ` Vijay Kilari
@ 2017-07-24 20:28 ` Stefano Stabellini
1 sibling, 0 replies; 109+ messages in thread
From: Stefano Stabellini @ 2017-07-24 20:28 UTC (permalink / raw)
To: Julien Grall
Cc: tim, kevin.tian, sstabellini, wei.liu2, vijay.kilari,
George.Dunlap, andrew.cooper3, dario.faggioli, ian.jackson,
xen-devel, jbeulich, Vijaya Kumar K
On Wed, 19 Jul 2017, Julien Grall wrote:
> Hi Vijay,
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> > From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> >
> > Move code from xen/arch/x86/numa.c to xen/common/numa.c
> > so that it can be used by other archs.
> >
> > The following changes are done:
> > - Few generic static functions in x86/numa.c is made
> > non-static common/numa.c
> > - The generic contents of header file asm-x86/numa.h
> > are moved to xen/numa.h.
> > - The header file includes are reordered and externs are
> > dropped.
> > - Moved acpi_numa from asm-x86/acpi.h to xen/acpi.h
> > - Coding style of code moved to commom/numa.c is changed
> > to Xen style.
> > - numa_add_cpu() and numa_set_node() and moved to header
> > file and added inline function in case of CONFIG_NUMA
> > is not enabled because these functions are called from
> > generic code with out any config check.
> >
> > Also the node_online_map is defined in x86/numa.c for x86
> > and arm/smpboot.c for ARM. For x86 it is moved to x86/smpboot.c
> > If moved to common code the compilation fails because
> > common/numa.c is compiled only when NUMA is enabled.
>
> I would much prefer if this patch does one thing: Moving code. The rest should
> be split out to help review and allowing us to easily verify you only moved
> code...
Indeed. However for the sake of making things easier, I did go through
the patch line by line (manually and automatically) to check the code
movement and it is correct.
> > +#define NODE_DATA(nid) (&(node_data[nid]))
> > +
> > +#define node_start_pfn(nid) NODE_DATA(nid)->node_start_pfn
> > +#define node_spanned_pages(nid) NODE_DATA(nid)->node_spanned_pages
> > +#define node_end_pfn(nid) NODE_DATA(nid)->node_start_pfn + \
> > + NODE_DATA(nid)->node_spanned_pages
> > +
> > +void numa_add_cpu(int cpu);
> > +void numa_set_node(int cpu, nodeid_t node);
> > +#else
> > +static inline void numa_add_cpu(int cpu) { }
> > +static inline void numa_set_node(int cpu, nodeid_t node) { }
>
> I am not sure why you need to define stub at least for numa_set_node... I
> can't see use in non-NUMA code. I will comment about the numa_add_cpu later.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 09/24] NUMA: x86: Move common code from srat.c
2017-07-20 11:17 ` Julien Grall
2017-07-20 11:43 ` Vijay Kilari
@ 2017-07-24 20:35 ` Stefano Stabellini
1 sibling, 0 replies; 109+ messages in thread
From: Stefano Stabellini @ 2017-07-24 20:35 UTC (permalink / raw)
To: Julien Grall
Cc: tim, kevin.tian, sstabellini, wei.liu2, vijay.kilari,
George.Dunlap, andrew.cooper3, dario.faggioli, ian.jackson,
xen-devel, jbeulich, Vijaya Kumar K
On Thu, 20 Jul 2017, Julien Grall wrote:
> Hi Vijay,
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> > From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> >
> > Move code from xen/arch/x86/srat.c to xen/common/numa.c
> > so that it can be used by other archs.
> >
> > Apart from moving the code the following changes are done
> > - Coding style of code moved to numa.c is changed to xen style
> > - {memory,processor}_nodes_parsed are made global and moved
> > to xen/nodemask.h
> > - Few generic static functions in x86/srat.c are made
> > non-static
> > - Functions moved from x85/srat.c to common/numa.c are made
> > non-static
> > - numa_scan_nodes() is made as static function
> > - compute_memnode_shift() and setup_node_bootmem() are made
> > static.
>
> You modify the coding style at the same time as the same time as moving the
> code. This makes quite difficult to make sure that a mistake didn't slip in
> the new code. Can you please diving this patch in smaller chunk (i.e moving
> code in smaller chunk) to ease the review?
For your reference and my own, the code movement in this patch is
correct. Splitting it would be good of course.
> We can think of merging all of them when committing it.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 17/24] ARM: NUMA: DT: Do not expose numa info to DOM0
2017-07-18 11:41 ` [RFC PATCH v3 17/24] ARM: NUMA: DT: Do not expose numa info to DOM0 vijay.kilari
@ 2017-07-24 20:48 ` Stefano Stabellini
2017-07-26 17:22 ` Julien Grall
1 sibling, 0 replies; 109+ messages in thread
From: Stefano Stabellini @ 2017-07-24 20:48 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, 18 Jul 2017, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Delete numa-node-id and distance map from DOM0 DT
> so that NUMA information is not exposed to DOM0.
> This helps particularly to boot Node 1 devices
> as if booting on Node0.
>
> However this approach has limitation where memory allocation
> for the devices should be local.
>
> Also, do not expose numa distance node to DOM0.
>
> Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
> xen/arch/arm/domain_build.c | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 1bec4fa..a7d6d3a 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -425,6 +425,10 @@ static int write_properties(struct domain *d, struct kernel_info *kinfo,
> }
> }
>
> + /* Don't expose the property numa to the guest */
> + if ( dt_property_name_is_equal(prop, "numa-node-id") )
> + continue;
> +
> /* Don't expose the property "xen,passthrough" to the guest */
> if ( dt_property_name_is_equal(prop, "xen,passthrough") )
> continue;
> @@ -1177,6 +1181,11 @@ static int handle_node(struct domain *d, struct kernel_info *kinfo,
> DT_MATCH_TYPE("memory"),
> /* The memory mapped timer is not supported by Xen. */
> DT_MATCH_COMPATIBLE("arm,armv7-timer-mem"),
> + /*
> + * NUMA info is not exposed to Dom0.
> + * So, skip distance-map infomation
> + */
> + DT_MATCH_COMPATIBLE("numa-distance-map-v1"),
> { /* sentinel */ },
> };
> static const struct dt_device_match timer_matches[] __initconst =
> --
> 2.7.4
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 20/24] ACPI: Move arch specific SRAT parsing
2017-07-18 11:41 ` [RFC PATCH v3 20/24] ACPI: Move arch specific SRAT parsing vijay.kilari
@ 2017-07-24 21:15 ` Stefano Stabellini
0 siblings, 0 replies; 109+ messages in thread
From: Stefano Stabellini @ 2017-07-24 21:15 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, 18 Jul 2017, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> SRAT's X2APIC_CPU_AFFINITY and CPU_AFFINITY types are not used
> by ARM. Hence move handling of this SRAT types to arch specific
> file and handle them under arch_table_parse_srat().
>
> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>
> ---
> xen/arch/arm/numa/acpi_numa.c | 5 +++++
> xen/arch/x86/srat.c | 44 +++++++++++++++++++++++++++++++++++++++++++
> xen/drivers/acpi/numa.c | 43 ++----------------------------------------
> xen/include/xen/acpi.h | 6 ++++++
> 4 files changed, 57 insertions(+), 41 deletions(-)
>
> diff --git a/xen/arch/arm/numa/acpi_numa.c b/xen/arch/arm/numa/acpi_numa.c
> index d9ad547..341e20b7 100644
> --- a/xen/arch/arm/numa/acpi_numa.c
> +++ b/xen/arch/arm/numa/acpi_numa.c
> @@ -82,6 +82,11 @@ void __init acpi_map_uid_to_mpidr(void)
> acpi_parse_madt_handler, NR_CPUS);
> }
>
> +void __init arch_table_parse_srat(void)
> +{
> + return;
> +}
> +
> void __init acpi_numa_arch_fixup(void) {}
>
> /*
> diff --git a/xen/arch/x86/srat.c b/xen/arch/x86/srat.c
> index d5caccf..a5fdedd 100644
> --- a/xen/arch/x86/srat.c
> +++ b/xen/arch/x86/srat.c
> @@ -205,3 +205,47 @@ uint8_t __node_distance(nodeid_t a, nodeid_t b)
> }
>
> EXPORT_SYMBOL(__node_distance);
> +
> +static int __init
> +acpi_parse_x2apic_affinity(struct acpi_subtable_header *header,
> + const unsigned long end)
> +{
> + const struct acpi_srat_x2apic_cpu_affinity *processor_affinity
> + = container_of(header, struct acpi_srat_x2apic_cpu_affinity,
> + header);
> +
> + if (!header)
> + return -EINVAL;
> +
> + acpi_table_print_srat_entry(header);
> +
> + /* let architecture-dependent part to do it */
> + acpi_numa_x2apic_affinity_init(processor_affinity);
> +
> + return 0;
> +}
> +
> +static int __init
> +acpi_parse_processor_affinity(struct acpi_subtable_header *header,
> + const unsigned long end)
> +{
> + const struct acpi_srat_cpu_affinity *processor_affinity
> + = container_of(header, struct acpi_srat_cpu_affinity, header);
> +
> + if (!header)
> + return -EINVAL;
> +
> + acpi_table_print_srat_entry(header);
> +
> + acpi_numa_processor_affinity_init(processor_affinity);
> +
> + return 0;
> +}
> +
> +void __init arch_table_parse_srat(void)
> +{
> + acpi_table_parse_srat(ACPI_SRAT_TYPE_X2APIC_CPU_AFFINITY,
> + acpi_parse_x2apic_affinity, 0);
> + acpi_table_parse_srat(ACPI_SRAT_TYPE_CPU_AFFINITY,
> + acpi_parse_processor_affinity, 0);
> +}
> diff --git a/xen/drivers/acpi/numa.c b/xen/drivers/acpi/numa.c
> index 85f8917..0adc32c 100644
> --- a/xen/drivers/acpi/numa.c
> +++ b/xen/drivers/acpi/numa.c
> @@ -120,43 +120,6 @@ static int __init acpi_parse_slit(struct acpi_table_header *table)
> }
>
> static int __init
> -acpi_parse_x2apic_affinity(struct acpi_subtable_header *header,
> - const unsigned long end)
> -{
> - const struct acpi_srat_x2apic_cpu_affinity *processor_affinity
> - = container_of(header, struct acpi_srat_x2apic_cpu_affinity,
> - header);
> -
> - if (!header)
> - return -EINVAL;
> -
> - acpi_table_print_srat_entry(header);
> -
> - /* let architecture-dependent part to do it */
> - acpi_numa_x2apic_affinity_init(processor_affinity);
> -
> - return 0;
> -}
> -
> -static int __init
> -acpi_parse_processor_affinity(struct acpi_subtable_header *header,
> - const unsigned long end)
> -{
> - const struct acpi_srat_cpu_affinity *processor_affinity
> - = container_of(header, struct acpi_srat_cpu_affinity, header);
> -
> - if (!header)
> - return -EINVAL;
> -
> - acpi_table_print_srat_entry(header);
> -
> - /* let architecture-dependent part to do it */
> - acpi_numa_processor_affinity_init(processor_affinity);
> -
> - return 0;
> -}
> -
> -static int __init
> acpi_parse_memory_affinity(struct acpi_subtable_header *header,
> const unsigned long end)
> {
> @@ -197,13 +160,11 @@ int __init acpi_numa_init(void)
> {
> /* SRAT: Static Resource Affinity Table */
> if (!acpi_table_parse(ACPI_SIG_SRAT, acpi_parse_srat)) {
> - acpi_table_parse_srat(ACPI_SRAT_TYPE_X2APIC_CPU_AFFINITY,
> - acpi_parse_x2apic_affinity, 0);
> - acpi_table_parse_srat(ACPI_SRAT_TYPE_CPU_AFFINITY,
> - acpi_parse_processor_affinity, 0);
> acpi_table_parse_srat(ACPI_SRAT_TYPE_MEMORY_AFFINITY,
> acpi_parse_memory_affinity,
> NR_NODE_MEMBLKS);
> + /* This call handles architecture dependant SRAT */
> + arch_table_parse_srat();
> }
>
> /* SLIT: System Locality Information Table */
> diff --git a/xen/include/xen/acpi.h b/xen/include/xen/acpi.h
> index 9409350..53795ff 100644
> --- a/xen/include/xen/acpi.h
> +++ b/xen/include/xen/acpi.h
> @@ -95,7 +95,13 @@ void acpi_numa_slit_init (struct acpi_table_slit *slit);
> void acpi_numa_processor_affinity_init(const struct acpi_srat_cpu_affinity *);
> void acpi_numa_x2apic_affinity_init(const struct acpi_srat_x2apic_cpu_affinity *);
> void acpi_numa_memory_affinity_init(const struct acpi_srat_mem_affinity *);
> +#ifdef CONFIG_ACPI_NUMA
> void acpi_numa_arch_fixup(void);
> +void arch_table_parse_srat(void);
> +#else
> +static inline void acpi_numa_arch_fixup(void) { }
> +static inline void arch_table_parse_srat(void) { }
> +#endif
>
> #ifdef CONFIG_ACPI_HOTPLUG_CPU
> /* Arch dependent functions for cpu hotplug support */
> --
> 2.7.4
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 22/24] ARM: NUMA: Initialize ACPI NUMA
2017-07-18 11:41 ` [RFC PATCH v3 22/24] ARM: NUMA: Initialize ACPI NUMA vijay.kilari
@ 2017-07-24 22:11 ` Stefano Stabellini
2017-07-26 18:23 ` Julien Grall
1 sibling, 0 replies; 109+ messages in thread
From: Stefano Stabellini @ 2017-07-24 22:11 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, 18 Jul 2017, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Call ACPI NUMA initialization under CONFIG_ACPI_NUMA.
>
> Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
> ---
> xen/arch/arm/numa/acpi_numa.c | 27 ++++++++++++++++++++++++++-
> xen/arch/arm/numa/numa.c | 15 +++++++++++++--
> xen/common/numa.c | 14 ++++++++++++++
> xen/include/asm-arm/numa.h | 1 +
> xen/include/xen/numa.h | 1 +
> 5 files changed, 55 insertions(+), 3 deletions(-)
>
> diff --git a/xen/arch/arm/numa/acpi_numa.c b/xen/arch/arm/numa/acpi_numa.c
> index 95617f9..68fff95 100644
> --- a/xen/arch/arm/numa/acpi_numa.c
> +++ b/xen/arch/arm/numa/acpi_numa.c
> @@ -181,7 +181,7 @@ acpi_numa_gicc_affinity_init(const struct acpi_srat_gicc_affinity *pa)
> pxm, mpidr, node);
> }
>
> -void __init acpi_map_uid_to_mpidr(void)
> +static void __init acpi_map_uid_to_mpidr(void)
Can we introduce it as static since the beginning?
> {
> acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT,
> acpi_parse_madt_handler, NR_CPUS);
> @@ -209,6 +209,31 @@ void __init arch_table_parse_srat(void)
> acpi_parse_gicc_affinity, NR_CPUS);
> }
>
> +bool_t __init arch_acpi_numa_init(void)
> +{
> + int ret;
> +
> + if ( !acpi_disabled )
if ( acpi_disabled )
return false;
> + {
> + /*
> + * If firmware has DT, process_memory_node() call
> + * would have added memory blocks. So reset it before
> + * ACPI numa init.
> + */
> + numa_clear_memblks();
> + nodes_clear(memory_nodes_parsed);
> + acpi_map_uid_to_mpidr();
> + ret = acpi_numa_init();
> + if ( ret || srat_disabled() )
I don't think we need to check again for srat_disabled: that check is
already done by acpi_numa_init
> + return 1;
return true
> +
> + /* Register acpi node_distance handler */
> + register_node_distance(&acpi_node_distance);
> + }
> +
> + return 0;
> +}
> +
> void __init acpi_numa_arch_fixup(void) {}
>
> /*
> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
> index 26aa4c0..68599c4 100644
> --- a/xen/arch/arm/numa/numa.c
> +++ b/xen/arch/arm/numa/numa.c
> @@ -139,11 +139,22 @@ void __init numa_init(void)
> if ( numa_off )
> goto no_numa;
>
> - ret = dt_numa_init();
> +#ifdef CONFIG_ACPI_NUMA
> + ret = arch_acpi_numa_init();
> if ( ret )
> {
> numa_off = true;
> - printk(XENLOG_WARNING "DT NUMA init failed\n");
> + printk(XENLOG_WARNING "ACPI NUMA init failed\n");
> + }
> +#endif
> + if ( acpi_disabled )
> + {
> + ret = dt_numa_init();
> + if ( ret )
> + {
> + numa_off = true;
> + printk(XENLOG_WARNING "DT NUMA init failed\n");
> + }
> }
>
> no_numa:
> diff --git a/xen/common/numa.c b/xen/common/numa.c
> index 0f79a07..020bc19 100644
> --- a/xen/common/numa.c
> +++ b/xen/common/numa.c
> @@ -76,6 +76,20 @@ nodeid_t get_memblk_nodeid(unsigned int id)
> return memblk_nodeid[id];
> }
>
> +void __init numa_clear_memblks(void)
> +{
> + unsigned int i;
> +
> + for ( i = 0; i < get_num_node_memblks(); i++ )
> + {
> + node_memblk_range[i].start = 0;
> + node_memblk_range[i].end = 0;
> + memblk_nodeid[i] = NUMA_NO_NODE;
> + }
> +
> + num_node_memblks = 0;
> +}
> +
> int __init get_mem_nodeid(paddr_t start, paddr_t end)
> {
> unsigned int i;
> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> index f0a50bd..ff10b31 100644
> --- a/xen/include/asm-arm/numa.h
> +++ b/xen/include/asm-arm/numa.h
> @@ -20,6 +20,7 @@ static inline nodeid_t acpi_get_nodeid(uint64_t hwid)
> void numa_init(void);
> int dt_numa_init(void);
> void numa_set_cpu_node(int cpu, unsigned int nid);
> +bool_t arch_acpi_numa_init(void);
>
> #else
> static inline void numa_init(void)
> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
> index a541eb7..14a7a0c 100644
> --- a/xen/include/xen/numa.h
> +++ b/xen/include/xen/numa.h
> @@ -75,6 +75,7 @@ int get_num_node_memblks(void);
> bool arch_sanitize_nodes_memory(void);
> void numa_failed(void);
> uint8_t __node_distance(nodeid_t a, nodeid_t b);
> +void numa_clear_memblks(void);
> #else
> static inline void numa_add_cpu(int cpu) { }
> static inline void numa_set_node(int cpu, nodeid_t node) { }
> --
> 2.7.4
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 19/24] ARM: NUMA: Extract MPIDR from MADT table
2017-07-18 11:41 ` [RFC PATCH v3 19/24] ARM: NUMA: Extract MPIDR from MADT table vijay.kilari
@ 2017-07-24 22:17 ` Stefano Stabellini
2017-07-26 18:12 ` Julien Grall
1 sibling, 0 replies; 109+ messages in thread
From: Stefano Stabellini @ 2017-07-24 22:17 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, 18 Jul 2017, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Parse MADT table and extract MPIDR for all
> CPU IDs in MADT ACPI_MADT_TYPE_GENERIC_INTERRUPT entries
> and store in cpuid_to_hwid_map[]
>
> This mapping is used by SRAT table parsing to extract MPIDR
> of the CPU ID.
>
> MADT table is also parsed in arm/acpi/boot.c during smp boot.
> However cannot wait till smp boot as SRAT table is parsed
> much before during numa_init. Hence MADT is parsed twice
> during boot. Once in numa_init and another in smp init.
>
> Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
> ---
> v3: - acpi_numa is set to -1 on numa failure.
> ---
> xen/arch/arm/numa/Makefile | 1 +
> xen/arch/arm/numa/acpi_numa.c | 94 +++++++++++++++++++++++++++++++++++++++++++
> xen/arch/arm/numa/numa.c | 6 +++
> 3 files changed, 101 insertions(+)
>
> diff --git a/xen/arch/arm/numa/Makefile b/xen/arch/arm/numa/Makefile
> index 3af3aff..b549459 100644
> --- a/xen/arch/arm/numa/Makefile
> +++ b/xen/arch/arm/numa/Makefile
> @@ -1,2 +1,3 @@
> obj-y += dt_numa.o
> obj-y += numa.o
> +obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o
> diff --git a/xen/arch/arm/numa/acpi_numa.c b/xen/arch/arm/numa/acpi_numa.c
> new file mode 100644
> index 0000000..d9ad547
> --- /dev/null
> +++ b/xen/arch/arm/numa/acpi_numa.c
> @@ -0,0 +1,94 @@
> +/*
> + * ACPI based NUMA setup
> + *
> + * Copyright (C) 2016 - Cavium Inc.
> + * Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> + *
> + * Reads the ACPI MADT and SRAT table to setup NUMA information.
> + * Contains Excerpts from x86 implementation
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/mm.h>
> +#include <xen/inttypes.h>
> +#include <xen/nodemask.h>
> +#include <xen/acpi.h>
> +#include <xen/numa.h>
> +#include <xen/pfn.h>
> +#include <xen/acpi.h>
> +#include <acpi/srat.h>
> +#include <asm/page.h>
> +
> +/* Holds CPUID to MPIDR mapping read from MADT table. */
> +struct cpuid_to_hwid {
> + uint32_t cpuid;
> + uint64_t hwid;
> +};
> +
> +#define PHYS_CPUID_INVALID 0xff
> +
> +/* Holds mapping of CPU id to MPIDR read from MADT */
> +static struct cpuid_to_hwid __read_mostly cpuid_to_hwid_map[NR_CPUS] =
> + { [0 ... NR_CPUS - 1] = {PHYS_CPUID_INVALID, MPIDR_INVALID} };
> +static unsigned int num_cpuid_to_hwid;
> +
> +static void __init acpi_map_cpu_to_hwid(uint32_t cpuid, uint64_t mpidr)
> +{
> + if ( mpidr == MPIDR_INVALID )
> + {
> + printk("Skip MADT cpu entry with invalid MPIDR\n");
> + numa_failed();
> + return;
> + }
> +
> + cpuid_to_hwid_map[num_cpuid_to_hwid].hwid = mpidr;
> + cpuid_to_hwid_map[num_cpuid_to_hwid].cpuid = cpuid;
> + num_cpuid_to_hwid++;
> +}
Isn't cpuid_to_hwid_map the same as cpu_logical_map? What's the
difference between the two?
> +static int __init acpi_parse_madt_handler(struct acpi_subtable_header *header,
> + const unsigned long end)
> +{
> + uint64_t mpidr;
> + struct acpi_madt_generic_interrupt *p =
> + container_of(header, struct acpi_madt_generic_interrupt, header);
> +
> + if ( BAD_MADT_ENTRY(p, end) )
> + {
> + /* MADT is invalid, we disable NUMA by calling numa_failed() */
> + numa_failed();
> + return -EINVAL;
> + }
> +
> + acpi_table_print_madt_entry(header);
> + mpidr = p->arm_mpidr & MPIDR_HWID_MASK;
> + acpi_map_cpu_to_hwid(p->uid, mpidr);
> +
> + return 0;
> +}
> +
> +void __init acpi_map_uid_to_mpidr(void)
> +{
> + acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT,
> + acpi_parse_madt_handler, NR_CPUS);
> +}
> +
> +void __init acpi_numa_arch_fixup(void) {}
Could you please mention in the commit message what is this function
for?
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
> index 85352dc..26aa4c0 100644
> --- a/xen/arch/arm/numa/numa.c
> +++ b/xen/arch/arm/numa/numa.c
> @@ -19,6 +19,7 @@
> #include <xen/nodemask.h>
> #include <xen/numa.h>
> #include <xen/pfn.h>
> +#include <acpi/srat.h>
> #include <asm/acpi.h>
>
> static uint8_t (*node_distance_fn)(nodeid_t a, nodeid_t b);
> @@ -40,6 +41,11 @@ void numa_failed(void)
> init_dt_numa_distance();
> node_distance_fn = NULL;
> init_cpu_to_node();
> +
> +#ifdef CONFIG_ACPI_NUMA
> + acpi_numa = -1;
> + reset_pxm2node();
> +#endif
> }
>
> void __init numa_set_cpu_node(int cpu, unsigned int nid)
> --
> 2.7.4
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 21/24] ARM: NUMA: ACPI: Extract proximity from SRAT table
2017-07-18 11:41 ` [RFC PATCH v3 21/24] ARM: NUMA: ACPI: Extract proximity from SRAT table vijay.kilari
@ 2017-07-24 22:17 ` Stefano Stabellini
2017-07-26 18:18 ` Julien Grall
1 sibling, 0 replies; 109+ messages in thread
From: Stefano Stabellini @ 2017-07-24 22:17 UTC (permalink / raw)
To: vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, xen-devel,
julien.grall, jbeulich, Vijaya Kumar K
On Tue, 18 Jul 2017, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Register SRAT entry handler for type
> ACPI_SRAT_TYPE_GICC_AFFINITY to parse SRAT table
> and extract proximity for all CPU IDs.
>
> Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
> ---
> xen/arch/arm/acpi/boot.c | 2 +
> xen/arch/arm/numa/acpi_numa.c | 124 +++++++++++++++++++++++++++++++++++++++++-
> xen/drivers/acpi/numa.c | 15 +++++
> xen/include/acpi/actbl1.h | 17 +++++-
> xen/include/asm-arm/numa.h | 9 +++
> 5 files changed, 165 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/acpi/boot.c b/xen/arch/arm/acpi/boot.c
> index 889208a..4e28b16 100644
> --- a/xen/arch/arm/acpi/boot.c
> +++ b/xen/arch/arm/acpi/boot.c
> @@ -31,6 +31,7 @@
> #include <acpi/actables.h>
> #include <xen/mm.h>
> #include <xen/device_tree.h>
> +#include <xen/numa.h>
>
> #include <asm/acpi.h>
> #include <asm/smp.h>
> @@ -117,6 +118,7 @@ acpi_map_gic_cpu_interface(struct acpi_madt_generic_interrupt *processor)
> return;
> }
>
> + numa_set_cpu_node(enabled_cpus, acpi_get_nodeid(mpidr));
> /* map the logical cpu id to cpu MPIDR */
> cpu_logical_map(enabled_cpus) = mpidr;
>
> diff --git a/xen/arch/arm/numa/acpi_numa.c b/xen/arch/arm/numa/acpi_numa.c
> index 341e20b7..95617f9 100644
> --- a/xen/arch/arm/numa/acpi_numa.c
> +++ b/xen/arch/arm/numa/acpi_numa.c
> @@ -34,13 +34,63 @@ struct cpuid_to_hwid {
> uint64_t hwid;
> };
>
> +/* Holds NODE to MPIDR mapping. */
> +struct node_to_hwid {
> + nodeid_t nodeid;
> + uint64_t hwid;
> +};
> +
> #define PHYS_CPUID_INVALID 0xff
>
> /* Holds mapping of CPU id to MPIDR read from MADT */
> static struct cpuid_to_hwid __read_mostly cpuid_to_hwid_map[NR_CPUS] =
> { [0 ... NR_CPUS - 1] = {PHYS_CPUID_INVALID, MPIDR_INVALID} };
> +static struct node_to_hwid __read_mostly node_to_hwid_map[NR_CPUS] =
> + { [0 ... NR_CPUS - 1] = {NUMA_NO_NODE, MPIDR_INVALID} };
> +static unsigned int cpus_in_srat;
> static unsigned int num_cpuid_to_hwid;
>
> +nodeid_t __init acpi_get_nodeid(uint64_t hwid)
> +{
> + unsigned int i;
> +
> + for ( i = 0; i < cpus_in_srat; i++ )
> + {
> + if ( node_to_hwid_map[i].hwid == hwid )
> + return node_to_hwid_map[i].nodeid;
> + }
> +
> + return NUMA_NO_NODE;
> +}
> +
> +static uint64_t acpi_get_cpu_hwid(int cid)
> +{
> + unsigned int i;
> +
> + for ( i = 0; i < num_cpuid_to_hwid; i++ )
> + {
> + if ( cpuid_to_hwid_map[i].cpuid == cid )
> + return cpuid_to_hwid_map[i].hwid;
> + }
> +
> + return MPIDR_INVALID;
> +}
> +
> +static void __init acpi_map_node_to_hwid(nodeid_t nodeid, uint64_t hwid)
> +{
> + if ( nodeid >= MAX_NUMNODES )
> + {
> + printk(XENLOG_WARNING
> + "ACPI: NUMA: nodeid out of range %d with MPIDR 0x%lx\n",
> + nodeid, hwid);
> + numa_failed();
> + return;
> + }
> +
> + node_to_hwid_map[cpus_in_srat].nodeid = nodeid;
> + node_to_hwid_map[cpus_in_srat].hwid = hwid;
> +}
> +
> static void __init acpi_map_cpu_to_hwid(uint32_t cpuid, uint64_t mpidr)
> {
> if ( mpidr == MPIDR_INVALID )
> @@ -76,15 +126,87 @@ static int __init acpi_parse_madt_handler(struct acpi_subtable_header *header,
> return 0;
> }
>
> +/* Callback for Proximity Domain -> ACPI processor UID mapping */
> +static void __init
> +acpi_numa_gicc_affinity_init(const struct acpi_srat_gicc_affinity *pa)
> +{
> + int pxm, node;
> + uint64_t mpidr;
> +
> + if ( srat_disabled() )
> + return;
> +
> + if ( pa->header.length < sizeof(struct acpi_srat_gicc_affinity) )
> + {
> + printk(XENLOG_WARNING "SRAT: Invalid SRAT header length: %d\n",
> + pa->header.length);
> + numa_failed();
> + return;
> + }
> +
> + if ( !(pa->flags & ACPI_SRAT_GICC_ENABLED) )
> + return;
> +
> + if ( cpus_in_srat >= NR_CPUS )
> + {
> + printk(XENLOG_ERR
> + "SRAT: cpu_to_node_map[%d] is too small to fit all cpus\n",
> + NR_CPUS);
> + return;
> + }
> +
> + pxm = pa->proximity_domain;
> + node = acpi_setup_node(pxm);
> + if ( node == NUMA_NO_NODE )
> + {
> + numa_failed();
> + return;
> + }
> +
> + mpidr = acpi_get_cpu_hwid(pa->acpi_processor_uid);
> + if ( mpidr == MPIDR_INVALID )
> + {
> + printk(XENLOG_ERR
> + "SRAT: PXM %d with ACPI ID %d has no valid MPIDR in MADT\n",
> + pxm, pa->acpi_processor_uid);
> + numa_failed();
> + return;
> + }
> +
> + acpi_map_node_to_hwid(node, mpidr);
Given that we don't really care about the mapping of nodes to mpidrs, if
not for the sake of setting up later the mapping between cpu ids and
mpidr ids, can't we just skip this step and setup the cpu id to node id
mapping directly?
We have the mpidr here, do we have a cpu id to mpidr mapping already at
this point in the boot process? I think not, because that is done by
acpi_smp_init_cpus which is called after numa_init. Would it be possible
to change the acpi table parsing sequence so that we can create the cpu
id to mpidr mappings first (acpi_map_gic_cpu_interface called before
acpi_numa_gicc_affinity_init)? That way, we wouldn't have to introduce
the array node_to_hwid_map.
> + node_set(node, processor_nodes_parsed);
> + cpus_in_srat++;
I think it would be more consistent with the rest of the code if we
moved "cpus_in_srat++" into acpi_map_node_to_hwid.
> + acpi_numa = 1;
Wait, isn't this function called once per cpu? We should be setting
"acpi_numa = 1" once at boot not once per cpu.
> + printk(XENLOG_INFO "SRAT: PXM %d -> MPIDR 0x%lx -> Node %d\n",
> + pxm, mpidr, node);
> +}
> +
> void __init acpi_map_uid_to_mpidr(void)
> {
> acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT,
> acpi_parse_madt_handler, NR_CPUS);
> }
>
> +static int __init
> +acpi_parse_gicc_affinity(struct acpi_subtable_header *header,
> + const unsigned long end)
> +{
> + const struct acpi_srat_gicc_affinity *processor_affinity
> + = (struct acpi_srat_gicc_affinity *)header;
> +
> + if (!processor_affinity)
> + return -EINVAL;
I don't think is possible for processor_affinity to be NULL here. I
noticed that the existing checks in the srat parsing code on x86 check
for NULL, like you do here, but I don't think NULL can happen. However
the length of the header might be smaller than required so I think it
would be more appropriate to have a check like BAD_MADT_ENTRY, but that
it already done by acpi_numa_gicc_affinity_init.
> + acpi_table_print_srat_entry(header);
> + acpi_numa_gicc_affinity_init(processor_affinity);
> +
> + return 0;
> +}
> +
> void __init arch_table_parse_srat(void)
> {
> - return;
> + acpi_table_parse_srat(ACPI_SRAT_TYPE_GICC_AFFINITY,
> + acpi_parse_gicc_affinity, NR_CPUS);
> }
>
> void __init acpi_numa_arch_fixup(void) {}
> diff --git a/xen/drivers/acpi/numa.c b/xen/drivers/acpi/numa.c
> index 0adc32c..b48d91d 100644
> --- a/xen/drivers/acpi/numa.c
> +++ b/xen/drivers/acpi/numa.c
> @@ -104,6 +104,21 @@ void __init acpi_table_print_srat_entry(struct acpi_subtable_header * header)
> }
> #endif /* ACPI_DEBUG_OUTPUT */
> break;
> + case ACPI_SRAT_TYPE_GICC_AFFINITY:
> +#ifdef ACPI_DEBUG_OUTPUT
> + {
> + struct acpi_srat_gicc_affinity *p =
> + (struct acpi_srat_gicc_affinity *)header;
> + ACPI_DEBUG_PRINT((ACPI_DB_INFO,
> + "SRAT Processor (acpi id[0x%04x]) in"
> + " proximity domain %d %s\n",
> + p->acpi_processor_uid,
> + p->proximity_domain,
> + (p->flags & ACPI_SRAT_GICC_ENABLED) ?
> + "enabled" : "disabled");
> + }
> +#endif /* ACPI_DEBUG_OUTPUT */
> + break;
> default:
> printk(KERN_WARNING PREFIX
> "Found unsupported SRAT entry (type = %#x)\n",
> diff --git a/xen/include/acpi/actbl1.h b/xen/include/acpi/actbl1.h
> index e199136..b84bfba 100644
> --- a/xen/include/acpi/actbl1.h
> +++ b/xen/include/acpi/actbl1.h
> @@ -949,7 +949,8 @@ enum acpi_srat_type {
> ACPI_SRAT_TYPE_CPU_AFFINITY = 0,
> ACPI_SRAT_TYPE_MEMORY_AFFINITY = 1,
> ACPI_SRAT_TYPE_X2APIC_CPU_AFFINITY = 2,
> - ACPI_SRAT_TYPE_RESERVED = 3 /* 3 and greater are reserved */
> + ACPI_SRAT_TYPE_GICC_AFFINITY = 3,
> + ACPI_SRAT_TYPE_RESERVED = 4 /* 4 and greater are reserved */
> };
>
> /*
> @@ -1007,6 +1008,20 @@ struct acpi_srat_x2apic_cpu_affinity {
>
> #define ACPI_SRAT_CPU_ENABLED (1) /* 00: Use affinity structure */
>
> +/* 3: GICC Affinity (ACPI 5.1) */
> +
> +struct acpi_srat_gicc_affinity {
> + struct acpi_subtable_header header;
> + u32 proximity_domain;
> + u32 acpi_processor_uid;
> + u32 flags;
> + u32 clock_domain;
> +};
> +
> +/* Flags for struct acpi_srat_gicc_affinity */
> +
> +#define ACPI_SRAT_GICC_ENABLED (1) /* 00: Use affinity structure */
> +
> /* Reset to default packing */
>
> #pragma pack()
> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> index 0d3146c..f0a50bd 100644
> --- a/xen/include/asm-arm/numa.h
> +++ b/xen/include/asm-arm/numa.h
> @@ -7,6 +7,15 @@ void dt_numa_process_memory_node(uint32_t nid, paddr_t start, paddr_t size);
> void register_node_distance(uint8_t (fn)(nodeid_t a, nodeid_t b));
> void init_dt_numa_distance(void);
>
> +#ifdef CONFIG_ACPI_NUMA
> +nodeid_t acpi_get_nodeid(uint64_t hwid);
> +#else
> +static inline nodeid_t acpi_get_nodeid(uint64_t hwid)
> +{
> + return 0;
> +}
> +#endif /* CONFIG_ACPI_NUMA */
> +
> #ifdef CONFIG_NUMA
> void numa_init(void);
> int dt_numa_init(void);
> --
> 2.7.4
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support
2017-07-24 11:24 ` Julien Grall
@ 2017-07-25 6:47 ` Vijay Kilari
2017-07-25 18:38 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Vijay Kilari @ 2017-07-25 6:47 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
Hi Julien,
On Mon, Jul 24, 2017 at 4:54 PM, Julien Grall <julien.grall@arm.com> wrote:
> Hi Vijay,
>
>
> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>>
>> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>>
>> For each cpu, update cpu_to_node[] with node id from
>> the numa-node-id DT property. Also, initialize cpu_to_node[]
>> with node 0.
>>
>> Add macros to access cpu_to_node[] information.
>>
>> Signed-off-by: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>> ---
>> v3: - Dropped numa_add_cpu declaration from asm-arm/numa.h
>> - Dropped stale declarations
>> - Call numa_add_cpu for cpu0
>> ---
>> xen/arch/arm/numa/numa.c | 21 +++++++++++++++++++++
>> xen/arch/arm/setup.c | 2 ++
>> xen/arch/arm/smpboot.c | 25 ++++++++++++++++++++++++-
>> xen/include/asm-arm/numa.h | 7 +++++++
>> xen/include/asm-x86/numa.h | 1 -
>> xen/include/xen/numa.h | 1 +
>> 6 files changed, 55 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
>> index c00b92c..dc80aa5 100644
>> --- a/xen/arch/arm/numa/numa.c
>> +++ b/xen/arch/arm/numa/numa.c
>> @@ -22,11 +22,31 @@
>>
>> static uint8_t (*node_distance_fn)(nodeid_t a, nodeid_t b);
>>
>> +/*
>> + * Setup early cpu_to_node.
>> + */
>> +void __init init_cpu_to_node(void)
>> +{
>> + int i;
>> +
>> + for ( i = 0; i < NR_CPUS; i++ )
>> + numa_set_node(i, 0);
>> +}
>
>
> From the comment: "Setup early cpu_to_node". However this is not how you are
> using it.
Ok. I will update the comment.
>
> But I am not sure why it is even here...
>
>> +
>> void numa_failed(void)
>> {
>> numa_off = true;
>> init_dt_numa_distance();
>> node_distance_fn = NULL;
>> + init_cpu_to_node();
>> +}
>> +
>> +void __init numa_set_cpu_node(int cpu, unsigned int nid)
>> +{
>> + if ( !node_isset(nid, processor_nodes_parsed) || nid >= MAX_NUMNODES
>> )
>> + nid = 0;
>
>
> This looks wrong to me. If the node-id is invalid, why would you blindly set
> to 0?
Generally this check will not pass. I will make this function return
error code in case
of wrong nid.
>
>
>> +
>> + numa_set_node(cpu, nid);
>> }
>>
>> uint8_t __node_distance(nodeid_t a, nodeid_t b)
>> @@ -49,6 +69,7 @@ void __init numa_init(void)
>> int ret = 0;
>>
>> nodes_clear(processor_nodes_parsed);
>> + init_cpu_to_node();
>> init_dt_numa_distance();
>>
>> if ( numa_off )
>> diff --git a/xen/arch/arm/setup.c b/xen/arch/arm/setup.c
>> index a6d1499..b9c8b0d 100644
>> --- a/xen/arch/arm/setup.c
>> +++ b/xen/arch/arm/setup.c
>> @@ -787,6 +787,8 @@ void __init start_xen(unsigned long boot_phys_offset,
>>
>> processor_id();
>>
>> + numa_add_cpu(0);
>> +
>> smp_init_cpus();
>> cpus = smp_get_max_cpus();
>> printk(XENLOG_INFO "SMP: Allowing %u CPUs\n", cpus);
>> diff --git a/xen/arch/arm/smpboot.c b/xen/arch/arm/smpboot.c
>> index 32e8722..fcf9afc 100644
>> --- a/xen/arch/arm/smpboot.c
>> +++ b/xen/arch/arm/smpboot.c
>> @@ -29,6 +29,7 @@
>> #include <xen/timer.h>
>> #include <xen/irq.h>
>> #include <xen/console.h>
>> +#include <xen/numa.h>
>
>
> Please use the alphabetical order.
>
>> #include <asm/cpuerrata.h>
>> #include <asm/gic.h>
>> #include <asm/psci.h>
>> @@ -106,6 +107,7 @@ static void __init dt_smp_init_cpus(void)
>> [0 ... NR_CPUS - 1] = MPIDR_INVALID
>> };
>> bool_t bootcpu_valid = 0;
>> + nodeid_t *cpu_to_nodemap;
>> int rc;
>>
>> mpidr = boot_cpu_data.mpidr.bits & MPIDR_HWID_MASK;
>> @@ -117,11 +119,18 @@ static void __init dt_smp_init_cpus(void)
>> return;
>> }
>>
>> + cpu_to_nodemap = xzalloc_array(nodeid_t, NR_CPUS);
>
>
> Why do you need to allocate cpu_to_nodemap? Would not it be easier to put it
> on the stack as we do for other variable?
This array holds nodemap indexed by cpuid once for all the cpus.
Later while setting the logical cpu id mapping, the node mapping is set
by calling numa_set_cpu_node().
>
>> + if ( !cpu_to_nodemap )
>> + {
>> + printk(XENLOG_WARNING "Failed to allocate memory for
>> cpu_to_nodemap\n");
>> + return;
>> + }
>> +
>> dt_for_each_child_node( cpus, cpu )
>> {
>> const __be32 *prop;
>> u64 addr;
>> - u32 reg_len;
>> + uint32_t reg_len, nid;
>> register_t hwid;
>>
>> if ( !dt_device_type_is_equal(cpu, "cpu") )
>> @@ -146,6 +155,15 @@ static void __init dt_smp_init_cpus(void)
>> continue;
>> }
>>
>> + if ( !dt_property_read_u32(cpu, "numa-node-id", &nid) )
>> + {
>> + printk(XENLOG_WARNING "cpu node `%s`: numa-node-id not
>> found\n",
>> + dt_node_full_name(cpu));
>
>
> numa-node-id is not mandatory. So you would print a warning on all non-NUMA
> platform. This not what we want.
ok. I will drop this warning.
>
>> + nid = 0;
>> + }
>> +
>> + cpu_to_nodemap[cpuidx] = nid;
>> +
>> addr = dt_read_number(prop, dt_n_addr_cells(cpu));
>>
>> hwid = addr;
>> @@ -224,6 +242,7 @@ static void __init dt_smp_init_cpus(void)
>> {
>> printk(XENLOG_WARNING "DT missing boot CPU MPIDR[23:0]\n"
>> "Using only 1 CPU\n");
>> + xfree(cpu_to_nodemap);
>> return;
>> }
>>
>> @@ -233,7 +252,10 @@ static void __init dt_smp_init_cpus(void)
>> continue;
>> cpumask_set_cpu(i, &cpu_possible_map);
>> cpu_logical_map(i) = tmp_map[i];
>> + numa_set_cpu_node(i, cpu_to_nodemap[i]);
>> }
>> +
>> + xfree(cpu_to_nodemap);
>> }
>>
>> void __init smp_init_cpus(void)
>> @@ -313,6 +335,7 @@ void start_secondary(unsigned long boot_phys_offset,
>> */
>> smp_wmb();
>>
>> + numa_add_cpu(cpuid);
>
>
> Newline here please.
>
>
>> /* Now report this CPU is up */
>> cpumask_set_cpu(cpuid, &cpu_online_map);
>>
>> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
>> index d1dc83a..0d3146c 100644
>> --- a/xen/include/asm-arm/numa.h
>> +++ b/xen/include/asm-arm/numa.h
>> @@ -10,12 +10,19 @@ void init_dt_numa_distance(void);
>> #ifdef CONFIG_NUMA
>> void numa_init(void);
>> int dt_numa_init(void);
>> +void numa_set_cpu_node(int cpu, unsigned int nid);
>> +
>> #else
>> static inline void numa_init(void)
>> {
>> return;
>> }
>>
>> +static inline void numa_set_cpu_node(int cpu, unsigned int nid)
>> +{
>> + return;
>> +}
>> +
>> /* Fake one node for now. See also node_online_map. */
>> #define cpu_to_node(cpu) 0
>> #define node_to_cpumask(node) (cpu_online_map)
>> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
>> index ca0a2a6..fc4747f 100644
>> --- a/xen/include/asm-x86/numa.h
>> +++ b/xen/include/asm-x86/numa.h
>> @@ -15,7 +15,6 @@ extern nodeid_t acpi_setup_node(unsigned int pxm);
>> extern void srat_detect_node(int cpu);
>>
>> extern nodeid_t apicid_to_node[];
>> -extern void init_cpu_to_node(void);
>>
>> void srat_parse_regions(paddr_t addr);
>> unsigned int arch_get_dma_bitsize(void);
>> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
>> index 10ef4c4..8a306e7 100644
>> --- a/xen/include/xen/numa.h
>> +++ b/xen/include/xen/numa.h
>> @@ -30,6 +30,7 @@ extern s8 acpi_numa;
>> void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
>> int srat_disabled(void);
>> int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
>> +void init_cpu_to_node(void);
>
>
> You never used this function in common code. So why did you move it in the
> common headers?
Same was defined for x86 as well. So I have moved to common header file.
>
>>
>> #ifdef CONFIG_NUMA
>> #define cpu_to_node(cpu) (cpu_to_node[cpu])
>>
>
> Cheers,
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support
2017-07-25 6:47 ` Vijay Kilari
@ 2017-07-25 18:38 ` Julien Grall
2017-07-25 18:48 ` Stefano Stabellini
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-25 18:38 UTC (permalink / raw)
To: Vijay Kilari
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
George Dunlap, Andrew Cooper, Dario Faggioli, Ian Jackson,
xen-devel, Jan Beulich, Vijaya Kumar K
On 25/07/17 07:47, Vijay Kilari wrote:
>>> void numa_failed(void)
>>> {
>>> numa_off = true;
>>> init_dt_numa_distance();
>>> node_distance_fn = NULL;
>>> + init_cpu_to_node();
>>> +}
>>> +
>>> +void __init numa_set_cpu_node(int cpu, unsigned int nid)
>>> +{
>>> + if ( !node_isset(nid, processor_nodes_parsed) || nid >= MAX_NUMNODES
>>> )
>>> + nid = 0;
>>
>>
>> This looks wrong to me. If the node-id is invalid, why would you blindly set
>> to 0?
>
> Generally this check will not pass. I will make this function return
> error code in case
> of wrong nid.
I don't really want to see error code and error handling everywhere in
the initialization code. I would assume that if the NUMA bindings are
wrong we should just crash Xen rather continuing with NUMA disabled.
Stefano do you have any opinion here?
[...]
>>> #include <asm/cpuerrata.h>
>>> #include <asm/gic.h>
>>> #include <asm/psci.h>
>>> @@ -106,6 +107,7 @@ static void __init dt_smp_init_cpus(void)
>>> [0 ... NR_CPUS - 1] = MPIDR_INVALID
>>> };
>>> bool_t bootcpu_valid = 0;
>>> + nodeid_t *cpu_to_nodemap;
>>> int rc;
>>>
>>> mpidr = boot_cpu_data.mpidr.bits & MPIDR_HWID_MASK;
>>> @@ -117,11 +119,18 @@ static void __init dt_smp_init_cpus(void)
>>> return;
>>> }
>>>
>>> + cpu_to_nodemap = xzalloc_array(nodeid_t, NR_CPUS);
>>
>>
>> Why do you need to allocate cpu_to_nodemap? Would not it be easier to put it
>> on the stack as we do for other variable?
>
> This array holds nodemap indexed by cpuid once for all the cpus.
> Later while setting the logical cpu id mapping, the node mapping is set
> by calling numa_set_cpu_node().
This does not answer question... Please read it again and explain why
you can't do:
nodeid_t cpu_to_nodemap[NR_CPUS];
>>> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
>>> index d1dc83a..0d3146c 100644
>>> --- a/xen/include/asm-arm/numa.h
>>> +++ b/xen/include/asm-arm/numa.h
>>> @@ -10,12 +10,19 @@ void init_dt_numa_distance(void);
>>> #ifdef CONFIG_NUMA
>>> void numa_init(void);
>>> int dt_numa_init(void);
>>> +void numa_set_cpu_node(int cpu, unsigned int nid);
>>> +
>>> #else
>>> static inline void numa_init(void)
>>> {
>>> return;
>>> }
>>>
>>> +static inline void numa_set_cpu_node(int cpu, unsigned int nid)
>>> +{
>>> + return;
>>> +}
>>> +
>>> /* Fake one node for now. See also node_online_map. */
>>> #define cpu_to_node(cpu) 0
>>> #define node_to_cpumask(node) (cpu_online_map)
>>> diff --git a/xen/include/asm-x86/numa.h b/xen/include/asm-x86/numa.h
>>> index ca0a2a6..fc4747f 100644
>>> --- a/xen/include/asm-x86/numa.h
>>> +++ b/xen/include/asm-x86/numa.h
>>> @@ -15,7 +15,6 @@ extern nodeid_t acpi_setup_node(unsigned int pxm);
>>> extern void srat_detect_node(int cpu);
>>>
>>> extern nodeid_t apicid_to_node[];
>>> -extern void init_cpu_to_node(void);
>>>
>>> void srat_parse_regions(paddr_t addr);
>>> unsigned int arch_get_dma_bitsize(void);
>>> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
>>> index 10ef4c4..8a306e7 100644
>>> --- a/xen/include/xen/numa.h
>>> +++ b/xen/include/xen/numa.h
>>> @@ -30,6 +30,7 @@ extern s8 acpi_numa;
>>> void numa_initmem_init(unsigned long start_pfn, unsigned long end_pfn);
>>> int srat_disabled(void);
>>> int valid_numa_range(paddr_t start, paddr_t end, nodeid_t node);
>>> +void init_cpu_to_node(void);
>>
>>
>> You never used this function in common code. So why did you move it in the
>> common headers?
>
> Same was defined for x86 as well. So I have moved to common header file.
You should make common only functions that will be called from code
common or are part of common code.
In this particular case, the name might be the same but the behavior is
completely different and the way to use it too...
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support
2017-07-25 18:38 ` Julien Grall
@ 2017-07-25 18:48 ` Stefano Stabellini
2017-07-25 18:51 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Stefano Stabellini @ 2017-07-25 18:48 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
Vijay Kilari, George Dunlap, Andrew Cooper, Dario Faggioli,
Ian Jackson, xen-devel, Jan Beulich, Vijaya Kumar K
On Tue, 25 Jul 2017, Julien Grall wrote:
> On 25/07/17 07:47, Vijay Kilari wrote:
> > > > void numa_failed(void)
> > > > {
> > > > numa_off = true;
> > > > init_dt_numa_distance();
> > > > node_distance_fn = NULL;
> > > > + init_cpu_to_node();
> > > > +}
> > > > +
> > > > +void __init numa_set_cpu_node(int cpu, unsigned int nid)
> > > > +{
> > > > + if ( !node_isset(nid, processor_nodes_parsed) || nid >=
> > > > MAX_NUMNODES
> > > > )
> > > > + nid = 0;
> > >
> > >
> > > This looks wrong to me. If the node-id is invalid, why would you blindly
> > > set
> > > to 0?
> >
> > Generally this check will not pass. I will make this function return
> > error code in case
> > of wrong nid.
>
> I don't really want to see error code and error handling everywhere in the
> initialization code. I would assume that if the NUMA bindings are wrong we
> should just crash Xen rather continuing with NUMA disabled.
>
> Stefano do you have any opinion here?
Yes, I noticed that there is an overabundance of error checks in the
patches. I have pointed out in other cases that some of these checks are
duplicates.
I am OK with some checks but we should not do the same check over and
over.
To answer the question: do we need any checks at all?
I am fine with no checks on the device tree or ACPI bindings themselves.
I am also OK with some checks in few places to check that the
information passed by the firmware is in the right shape (for example we
check for the ACPI header length before accessing any ACPI tables). That
is good. But I am not OK with repeating the same check multiple times
uselessly or checking for conditions that cannot happen (like a NULL
pointer in the ACPI header case again).
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support
2017-07-25 18:48 ` Stefano Stabellini
@ 2017-07-25 18:51 ` Julien Grall
2017-07-25 19:06 ` Stefano Stabellini
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-25 18:51 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Tim Deegan, kevin.tian, Wei Liu, Vijay Kilari, George Dunlap,
Andrew Cooper, Dario Faggioli, Ian Jackson, xen-devel,
Jan Beulich, Vijaya Kumar K
On 25/07/17 19:48, Stefano Stabellini wrote:
> On Tue, 25 Jul 2017, Julien Grall wrote:
>> On 25/07/17 07:47, Vijay Kilari wrote:
>>>>> void numa_failed(void)
>>>>> {
>>>>> numa_off = true;
>>>>> init_dt_numa_distance();
>>>>> node_distance_fn = NULL;
>>>>> + init_cpu_to_node();
>>>>> +}
>>>>> +
>>>>> +void __init numa_set_cpu_node(int cpu, unsigned int nid)
>>>>> +{
>>>>> + if ( !node_isset(nid, processor_nodes_parsed) || nid >=
>>>>> MAX_NUMNODES
>>>>> )
>>>>> + nid = 0;
>>>>
>>>>
>>>> This looks wrong to me. If the node-id is invalid, why would you blindly
>>>> set
>>>> to 0?
>>>
>>> Generally this check will not pass. I will make this function return
>>> error code in case
>>> of wrong nid.
>>
>> I don't really want to see error code and error handling everywhere in the
>> initialization code. I would assume that if the NUMA bindings are wrong we
>> should just crash Xen rather continuing with NUMA disabled.
>>
>> Stefano do you have any opinion here?
>
> Yes, I noticed that there is an overabundance of error checks in the
> patches. I have pointed out in other cases that some of these checks are
> duplicates.
>
> I am OK with some checks but we should not do the same check over and
> over.
>
> To answer the question: do we need any checks at all?
>
> I am fine with no checks on the device tree or ACPI bindings themselves.
> I am also OK with some checks in few places to check that the
> information passed by the firmware is in the right shape (for example we
> check for the ACPI header length before accessing any ACPI tables). That
> is good. But I am not OK with repeating the same check multiple times
> uselessly or checking for conditions that cannot happen (like a NULL
> pointer in the ACPI header case again).
I would prefer to keep the check on the DT bindings and ACPI bindings. I
hit some problem in the past that were quite annoying to debug without them.
But I was wondering if we should just panic/BUG_ON directly. Rather than
returning an error.
For any redundant check, we could just turn to an ASSERT.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support
2017-07-25 18:51 ` Julien Grall
@ 2017-07-25 19:06 ` Stefano Stabellini
2017-07-26 17:18 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: Stefano Stabellini @ 2017-07-25 19:06 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
Vijay Kilari, George Dunlap, Andrew Cooper, Dario Faggioli,
Ian Jackson, xen-devel, Jan Beulich, Vijaya Kumar K
On Tue, 25 Jul 2017, Julien Grall wrote:
> On 25/07/17 19:48, Stefano Stabellini wrote:
> > On Tue, 25 Jul 2017, Julien Grall wrote:
> > > On 25/07/17 07:47, Vijay Kilari wrote:
> > > > > > void numa_failed(void)
> > > > > > {
> > > > > > numa_off = true;
> > > > > > init_dt_numa_distance();
> > > > > > node_distance_fn = NULL;
> > > > > > + init_cpu_to_node();
> > > > > > +}
> > > > > > +
> > > > > > +void __init numa_set_cpu_node(int cpu, unsigned int nid)
> > > > > > +{
> > > > > > + if ( !node_isset(nid, processor_nodes_parsed) || nid >=
> > > > > > MAX_NUMNODES
> > > > > > )
> > > > > > + nid = 0;
> > > > >
> > > > >
> > > > > This looks wrong to me. If the node-id is invalid, why would you
> > > > > blindly
> > > > > set
> > > > > to 0?
> > > >
> > > > Generally this check will not pass. I will make this function return
> > > > error code in case
> > > > of wrong nid.
> > >
> > > I don't really want to see error code and error handling everywhere in the
> > > initialization code. I would assume that if the NUMA bindings are wrong we
> > > should just crash Xen rather continuing with NUMA disabled.
> > >
> > > Stefano do you have any opinion here?
> >
> > Yes, I noticed that there is an overabundance of error checks in the
> > patches. I have pointed out in other cases that some of these checks are
> > duplicates.
> >
> > I am OK with some checks but we should not do the same check over and
> > over.
> >
> > To answer the question: do we need any checks at all?
> >
> > I am fine with no checks on the device tree or ACPI bindings themselves.
> > I am also OK with some checks in few places to check that the
> > information passed by the firmware is in the right shape (for example we
> > check for the ACPI header length before accessing any ACPI tables). That
> > is good. But I am not OK with repeating the same check multiple times
> > uselessly or checking for conditions that cannot happen (like a NULL
> > pointer in the ACPI header case again).
>
> I would prefer to keep the check on the DT bindings and ACPI bindings. I hit
> some problem in the past that were quite annoying to debug without them.
>
> But I was wondering if we should just panic/BUG_ON directly. Rather than
> returning an error.
I think BUG_ON is fine, but it would be best if we also printed a
useful message before crashing Xen. At least the user would know that
the problem is a broken device_tree/ACPI.
> For any redundant check, we could just turn to an ASSERT.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support
2017-07-25 19:06 ` Stefano Stabellini
@ 2017-07-26 17:18 ` Julien Grall
2017-07-26 17:21 ` Stefano Stabellini
0 siblings, 1 reply; 109+ messages in thread
From: Julien Grall @ 2017-07-26 17:18 UTC (permalink / raw)
To: Stefano Stabellini
Cc: Tim Deegan, kevin.tian, Wei Liu, Vijay Kilari, George Dunlap,
Andrew Cooper, Dario Faggioli, Ian Jackson, xen-devel,
Jan Beulich, Vijaya Kumar K
Hi Stefano,
On 25/07/17 20:06, Stefano Stabellini wrote:
> On Tue, 25 Jul 2017, Julien Grall wrote:
>> On 25/07/17 19:48, Stefano Stabellini wrote:
>>> On Tue, 25 Jul 2017, Julien Grall wrote:
>>>> On 25/07/17 07:47, Vijay Kilari wrote:
>>>>>>> void numa_failed(void)
>>>>>>> {
>>>>>>> numa_off = true;
>>>>>>> init_dt_numa_distance();
>>>>>>> node_distance_fn = NULL;
>>>>>>> + init_cpu_to_node();
>>>>>>> +}
>>>>>>> +
>>>>>>> +void __init numa_set_cpu_node(int cpu, unsigned int nid)
>>>>>>> +{
>>>>>>> + if ( !node_isset(nid, processor_nodes_parsed) || nid >=
>>>>>>> MAX_NUMNODES
>>>>>>> )
>>>>>>> + nid = 0;
>>>>>>
>>>>>>
>>>>>> This looks wrong to me. If the node-id is invalid, why would you
>>>>>> blindly
>>>>>> set
>>>>>> to 0?
>>>>>
>>>>> Generally this check will not pass. I will make this function return
>>>>> error code in case
>>>>> of wrong nid.
>>>>
>>>> I don't really want to see error code and error handling everywhere in the
>>>> initialization code. I would assume that if the NUMA bindings are wrong we
>>>> should just crash Xen rather continuing with NUMA disabled.
>>>>
>>>> Stefano do you have any opinion here?
>>>
>>> Yes, I noticed that there is an overabundance of error checks in the
>>> patches. I have pointed out in other cases that some of these checks are
>>> duplicates.
>>>
>>> I am OK with some checks but we should not do the same check over and
>>> over.
>>>
>>> To answer the question: do we need any checks at all?
>>>
>>> I am fine with no checks on the device tree or ACPI bindings themselves.
>>> I am also OK with some checks in few places to check that the
>>> information passed by the firmware is in the right shape (for example we
>>> check for the ACPI header length before accessing any ACPI tables). That
>>> is good. But I am not OK with repeating the same check multiple times
>>> uselessly or checking for conditions that cannot happen (like a NULL
>>> pointer in the ACPI header case again).
>>
>> I would prefer to keep the check on the DT bindings and ACPI bindings. I hit
>> some problem in the past that were quite annoying to debug without them.
>>
>> But I was wondering if we should just panic/BUG_ON directly. Rather than
>> returning an error.
>
> I think BUG_ON is fine, but it would be best if we also printed a
> useful message before crashing Xen. At least the user would know that
> the problem is a broken device_tree/ACPI.
I was suggesting to use panic because you can get a nice message :).
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support
2017-07-26 17:18 ` Julien Grall
@ 2017-07-26 17:21 ` Stefano Stabellini
0 siblings, 0 replies; 109+ messages in thread
From: Stefano Stabellini @ 2017-07-26 17:21 UTC (permalink / raw)
To: Julien Grall
Cc: Tim Deegan, kevin.tian, Stefano Stabellini, Wei Liu,
Vijay Kilari, George Dunlap, Andrew Cooper, Dario Faggioli,
Ian Jackson, xen-devel, Jan Beulich, Vijaya Kumar K
On Wed, 26 Jul 2017, Julien Grall wrote:
> Hi Stefano,
>
> On 25/07/17 20:06, Stefano Stabellini wrote:
> > On Tue, 25 Jul 2017, Julien Grall wrote:
> > > On 25/07/17 19:48, Stefano Stabellini wrote:
> > > > On Tue, 25 Jul 2017, Julien Grall wrote:
> > > > > On 25/07/17 07:47, Vijay Kilari wrote:
> > > > > > > > void numa_failed(void)
> > > > > > > > {
> > > > > > > > numa_off = true;
> > > > > > > > init_dt_numa_distance();
> > > > > > > > node_distance_fn = NULL;
> > > > > > > > + init_cpu_to_node();
> > > > > > > > +}
> > > > > > > > +
> > > > > > > > +void __init numa_set_cpu_node(int cpu, unsigned int nid)
> > > > > > > > +{
> > > > > > > > + if ( !node_isset(nid, processor_nodes_parsed) || nid >=
> > > > > > > > MAX_NUMNODES
> > > > > > > > )
> > > > > > > > + nid = 0;
> > > > > > >
> > > > > > >
> > > > > > > This looks wrong to me. If the node-id is invalid, why would you
> > > > > > > blindly
> > > > > > > set
> > > > > > > to 0?
> > > > > >
> > > > > > Generally this check will not pass. I will make this function return
> > > > > > error code in case
> > > > > > of wrong nid.
> > > > >
> > > > > I don't really want to see error code and error handling everywhere in
> > > > > the
> > > > > initialization code. I would assume that if the NUMA bindings are
> > > > > wrong we
> > > > > should just crash Xen rather continuing with NUMA disabled.
> > > > >
> > > > > Stefano do you have any opinion here?
> > > >
> > > > Yes, I noticed that there is an overabundance of error checks in the
> > > > patches. I have pointed out in other cases that some of these checks are
> > > > duplicates.
> > > >
> > > > I am OK with some checks but we should not do the same check over and
> > > > over.
> > > >
> > > > To answer the question: do we need any checks at all?
> > > >
> > > > I am fine with no checks on the device tree or ACPI bindings themselves.
> > > > I am also OK with some checks in few places to check that the
> > > > information passed by the firmware is in the right shape (for example we
> > > > check for the ACPI header length before accessing any ACPI tables). That
> > > > is good. But I am not OK with repeating the same check multiple times
> > > > uselessly or checking for conditions that cannot happen (like a NULL
> > > > pointer in the ACPI header case again).
> > >
> > > I would prefer to keep the check on the DT bindings and ACPI bindings. I
> > > hit
> > > some problem in the past that were quite annoying to debug without them.
> > >
> > > But I was wondering if we should just panic/BUG_ON directly. Rather than
> > > returning an error.
> >
> > I think BUG_ON is fine, but it would be best if we also printed a
> > useful message before crashing Xen. At least the user would know that
> > the problem is a broken device_tree/ACPI.
>
> I was suggesting to use panic because you can get a nice message :).
panic is great, somehow BUG_ON grabbed all my attention when I read your
email :-)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 17/24] ARM: NUMA: DT: Do not expose numa info to DOM0
2017-07-18 11:41 ` [RFC PATCH v3 17/24] ARM: NUMA: DT: Do not expose numa info to DOM0 vijay.kilari
2017-07-24 20:48 ` Stefano Stabellini
@ 2017-07-26 17:22 ` Julien Grall
1 sibling, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-26 17:22 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Delete numa-node-id and distance map from DOM0 DT
> so that NUMA information is not exposed to DOM0.
> This helps particularly to boot Node 1 devices
> as if booting on Node0.
>
> However this approach has limitation where memory allocation
> for the devices should be local.
>
> Also, do not expose numa distance node to DOM0.
>
> Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
> ---
> xen/arch/arm/domain_build.c | 9 +++++++++
> 1 file changed, 9 insertions(+)
>
> diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
> index 1bec4fa..a7d6d3a 100644
> --- a/xen/arch/arm/domain_build.c
> +++ b/xen/arch/arm/domain_build.c
> @@ -425,6 +425,10 @@ static int write_properties(struct domain *d, struct kernel_info *kinfo,
> }
> }
>
> + /* Don't expose the property numa to the guest */
s/numa/NUMA/ and missing full stop.
> + if ( dt_property_name_is_equal(prop, "numa-node-id") )
> + continue;
> +
> /* Don't expose the property "xen,passthrough" to the guest */
> if ( dt_property_name_is_equal(prop, "xen,passthrough") )
> continue;
> @@ -1177,6 +1181,11 @@ static int handle_node(struct domain *d, struct kernel_info *kinfo,
> DT_MATCH_TYPE("memory"),
> /* The memory mapped timer is not supported by Xen. */
> DT_MATCH_COMPATIBLE("arm,armv7-timer-mem"),
> + /*
> + * NUMA info is not exposed to Dom0.
Please use the term "hardware domain" and not Dom0 in new code.
> + * So, skip distance-map infomation
s/infomation/information/ and missing full stop.
> + */
> + DT_MATCH_COMPATIBLE("numa-distance-map-v1"),
> { /* sentinel */ },
> };
> static const struct dt_device_match timer_matches[] __initconst =
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 19/24] ARM: NUMA: Extract MPIDR from MADT table
2017-07-18 11:41 ` [RFC PATCH v3 19/24] ARM: NUMA: Extract MPIDR from MADT table vijay.kilari
2017-07-24 22:17 ` Stefano Stabellini
@ 2017-07-26 18:12 ` Julien Grall
1 sibling, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-26 18:12 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Parse MADT table and extract MPIDR for all
> CPU IDs in MADT ACPI_MADT_TYPE_GENERIC_INTERRUPT entries
> and store in cpuid_to_hwid_map[]
>
> This mapping is used by SRAT table parsing to extract MPIDR
> of the CPU ID.
>
> MADT table is also parsed in arm/acpi/boot.c during smp boot.
> However cannot wait till smp boot as SRAT table is parsed
> much before during numa_init. Hence MADT is parsed twice
> during boot. Once in numa_init and another in smp init.
The mapping CPU IDs <-> MIDR is not NUMA specific. I still can't see why
you didn't rework the code as I asked in v1.
I would really appreciate if you try to add the support of NUMA on ARM
using other things than a hammer. This will not be sustainable to
maintain in long-term.
>
> Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
> ---
> v3: - acpi_numa is set to -1 on numa failure.
> ---
> xen/arch/arm/numa/Makefile | 1 +
> xen/arch/arm/numa/acpi_numa.c | 94 +++++++++++++++++++++++++++++++++++++++++++
> xen/arch/arm/numa/numa.c | 6 +++
> 3 files changed, 101 insertions(+)
>
> diff --git a/xen/arch/arm/numa/Makefile b/xen/arch/arm/numa/Makefile
> index 3af3aff..b549459 100644
> --- a/xen/arch/arm/numa/Makefile
> +++ b/xen/arch/arm/numa/Makefile
> @@ -1,2 +1,3 @@
> obj-y += dt_numa.o
> obj-y += numa.o
> +obj-$(CONFIG_ACPI_NUMA) += acpi_numa.o
> diff --git a/xen/arch/arm/numa/acpi_numa.c b/xen/arch/arm/numa/acpi_numa.c
> new file mode 100644
> index 0000000..d9ad547
> --- /dev/null
> +++ b/xen/arch/arm/numa/acpi_numa.c
> @@ -0,0 +1,94 @@
> +/*
> + * ACPI based NUMA setup
> + *
> + * Copyright (C) 2016 - Cavium Inc.
> + * Vijaya Kumar K <Vijaya.Kumar@cavium.com>
> + *
> + * Reads the ACPI MADT and SRAT table to setup NUMA information.
> + * Contains Excerpts from x86 implementation
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <xen/init.h>
> +#include <xen/mm.h>
> +#include <xen/inttypes.h>
> +#include <xen/nodemask.h>
> +#include <xen/acpi.h>
> +#include <xen/numa.h>
> +#include <xen/pfn.h>
> +#include <xen/acpi.h>
> +#include <acpi/srat.h>
> +#include <asm/page.h>
> +
> +/* Holds CPUID to MPIDR mapping read from MADT table. */
> +struct cpuid_to_hwid {
> + uint32_t cpuid;
> + uint64_t hwid;
> +};
> +
> +#define PHYS_CPUID_INVALID 0xff
> +
> +/* Holds mapping of CPU id to MPIDR read from MADT */
> +static struct cpuid_to_hwid __read_mostly cpuid_to_hwid_map[NR_CPUS] =
> + { [0 ... NR_CPUS - 1] = {PHYS_CPUID_INVALID, MPIDR_INVALID} };
> +static unsigned int num_cpuid_to_hwid;
> +
> +static void __init acpi_map_cpu_to_hwid(uint32_t cpuid, uint64_t mpidr)
> +{
> + if ( mpidr == MPIDR_INVALID )
> + {
> + printk("Skip MADT cpu entry with invalid MPIDR\n");
> + numa_failed();
> + return;
> + }
> +
> + cpuid_to_hwid_map[num_cpuid_to_hwid].hwid = mpidr;
> + cpuid_to_hwid_map[num_cpuid_to_hwid].cpuid = cpuid;
> + num_cpuid_to_hwid++;
> +}
> +
> +static int __init acpi_parse_madt_handler(struct acpi_subtable_header *header,
> + const unsigned long end)
> +{
> + uint64_t mpidr;
> + struct acpi_madt_generic_interrupt *p =
> + container_of(header, struct acpi_madt_generic_interrupt, header);
> +
> + if ( BAD_MADT_ENTRY(p, end) )
> + {
> + /* MADT is invalid, we disable NUMA by calling numa_failed() */
> + numa_failed();
> + return -EINVAL;
> + }
> +
> + acpi_table_print_madt_entry(header);
Why do you always print the MADT entry?
> + mpidr = p->arm_mpidr & MPIDR_HWID_MASK;
> + acpi_map_cpu_to_hwid(p->uid, mpidr);
> +
> + return 0;
> +}
> +
> +void __init acpi_map_uid_to_mpidr(void)
> +{
> + acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT,
> + acpi_parse_madt_handler, NR_CPUS);
> +}
> +
> +void __init acpi_numa_arch_fixup(void) {}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
> index 85352dc..26aa4c0 100644
> --- a/xen/arch/arm/numa/numa.c
> +++ b/xen/arch/arm/numa/numa.c
> @@ -19,6 +19,7 @@
> #include <xen/nodemask.h>
> #include <xen/numa.h>
> #include <xen/pfn.h>
> +#include <acpi/srat.h>
> #include <asm/acpi.h>
>
> static uint8_t (*node_distance_fn)(nodeid_t a, nodeid_t b);
> @@ -40,6 +41,11 @@ void numa_failed(void)
> init_dt_numa_distance();
> node_distance_fn = NULL;
> init_cpu_to_node();
> +
> +#ifdef CONFIG_ACPI_NUMA
> + acpi_numa = -1;
> + reset_pxm2node();
> +#endif
ARM support both DT and ACPI, so it sounds a bit weird to always do that.
> }
>
> void __init numa_set_cpu_node(int cpu, unsigned int nid)
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 21/24] ARM: NUMA: ACPI: Extract proximity from SRAT table
2017-07-18 11:41 ` [RFC PATCH v3 21/24] ARM: NUMA: ACPI: Extract proximity from SRAT table vijay.kilari
2017-07-24 22:17 ` Stefano Stabellini
@ 2017-07-26 18:18 ` Julien Grall
1 sibling, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-26 18:18 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Register SRAT entry handler for type
> ACPI_SRAT_TYPE_GICC_AFFINITY to parse SRAT table
> and extract proximity for all CPU IDs.
>
> Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
Please split this patch in multiple chunk. The addition in the ACPI
common code should at least be separated.
[...]
> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> index 0d3146c..f0a50bd 100644
> --- a/xen/include/asm-arm/numa.h
> +++ b/xen/include/asm-arm/numa.h
> @@ -7,6 +7,15 @@ void dt_numa_process_memory_node(uint32_t nid, paddr_t start, paddr_t size);
> void register_node_distance(uint8_t (fn)(nodeid_t a, nodeid_t b));
> void init_dt_numa_distance(void);
>
> +#ifdef CONFIG_ACPI_NUMA
> +nodeid_t acpi_get_nodeid(uint64_t hwid);
> +#else
> +static inline nodeid_t acpi_get_nodeid(uint64_t hwid)
> +{
> + return 0;
> +}
> +#endif /* CONFIG_ACPI_NUMA */
Do you expect CONFIG_ACPI_NUMA to be disabled when ACPI is supported on ARM?
> +
> #ifdef CONFIG_NUMA
> void numa_init(void);
> int dt_numa_init(void);
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 22/24] ARM: NUMA: Initialize ACPI NUMA
2017-07-18 11:41 ` [RFC PATCH v3 22/24] ARM: NUMA: Initialize ACPI NUMA vijay.kilari
2017-07-24 22:11 ` Stefano Stabellini
@ 2017-07-26 18:23 ` Julien Grall
1 sibling, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-07-26 18:23 UTC (permalink / raw)
To: vijay.kilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, tim, jbeulich, Vijaya Kumar K
Hi Vijay,
On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> From: Vijaya Kumar K <Vijaya.Kumar@cavium.com>
>
> Call ACPI NUMA initialization under CONFIG_ACPI_NUMA.
>
> Signed-off-by: Vijaya Kumar <Vijaya.Kumar@cavium.com>
> ---
> xen/arch/arm/numa/acpi_numa.c | 27 ++++++++++++++++++++++++++-
> xen/arch/arm/numa/numa.c | 15 +++++++++++++--
> xen/common/numa.c | 14 ++++++++++++++
> xen/include/asm-arm/numa.h | 1 +
> xen/include/xen/numa.h | 1 +
> 5 files changed, 55 insertions(+), 3 deletions(-)
>
> diff --git a/xen/arch/arm/numa/acpi_numa.c b/xen/arch/arm/numa/acpi_numa.c
> index 95617f9..68fff95 100644
> --- a/xen/arch/arm/numa/acpi_numa.c
> +++ b/xen/arch/arm/numa/acpi_numa.c
> @@ -181,7 +181,7 @@ acpi_numa_gicc_affinity_init(const struct acpi_srat_gicc_affinity *pa)
> pxm, mpidr, node);
> }
>
> -void __init acpi_map_uid_to_mpidr(void)
> +static void __init acpi_map_uid_to_mpidr(void)
> {
> acpi_table_parse_madt(ACPI_MADT_TYPE_GENERIC_INTERRUPT,
> acpi_parse_madt_handler, NR_CPUS);
> @@ -209,6 +209,31 @@ void __init arch_table_parse_srat(void)
> acpi_parse_gicc_affinity, NR_CPUS);
> }
>
> +bool_t __init arch_acpi_numa_init(void)
When I read arch_* I expect this to be called in common code. This is
actually not the case. Please rename it in arm_acpi_numa_init or
something else.
Also, as it has been repeated a numerous amount of time, you should use
bool and not bool_t.
> +{
> + int ret;
> +
> + if ( !acpi_disabled )
> + {
> + /*
> + * If firmware has DT, process_memory_node() call
> + * would have added memory blocks. So reset it before
> + * ACPI numa init.
> + */
Likely something is wrong in the handling path if it is happening.
> + numa_clear_memblks();
> + nodes_clear(memory_nodes_parsed);
> + acpi_map_uid_to_mpidr();
> + ret = acpi_numa_init();
> + if ( ret || srat_disabled() )
> + return 1;
> +
> + /* Register acpi node_distance handler */
> + register_node_distance(&acpi_node_distance);
> + }
> +
> + return 0;
> +}
> +
> void __init acpi_numa_arch_fixup(void) {}
>
> /*
> diff --git a/xen/arch/arm/numa/numa.c b/xen/arch/arm/numa/numa.c
> index 26aa4c0..68599c4 100644
> --- a/xen/arch/arm/numa/numa.c
> +++ b/xen/arch/arm/numa/numa.c
> @@ -139,11 +139,22 @@ void __init numa_init(void)
> if ( numa_off )
> goto no_numa;
>
> - ret = dt_numa_init();
> +#ifdef CONFIG_ACPI_NUMA
if ( !acpi_disabled )
{
....
}
And you can avoid #ifdef CONFIG_ACPI_NUMA by introducing stubs as you do
in other places.
> + ret = arch_acpi_numa_init();
> if ( ret )
> {
> numa_off = true;
> - printk(XENLOG_WARNING "DT NUMA init failed\n");
> + printk(XENLOG_WARNING "ACPI NUMA init failed\n");
> + }
> +#endif
> + if ( acpi_disabled )
> + {
> + ret = dt_numa_init();
> + if ( ret )
> + {
> + numa_off = true;
> + printk(XENLOG_WARNING "DT NUMA init failed\n");
> + }
> }
>
> no_numa:
> diff --git a/xen/common/numa.c b/xen/common/numa.c
> index 0f79a07..020bc19 100644
> --- a/xen/common/numa.c
> +++ b/xen/common/numa.c
> @@ -76,6 +76,20 @@ nodeid_t get_memblk_nodeid(unsigned int id)
> return memblk_nodeid[id];
> }
>
> +void __init numa_clear_memblks(void)
> +{
> + unsigned int i;
> +
> + for ( i = 0; i < get_num_node_memblks(); i++ )
> + {
> + node_memblk_range[i].start = 0;
> + node_memblk_range[i].end = 0;
> + memblk_nodeid[i] = NUMA_NO_NODE;
> + }
> +
> + num_node_memblks = 0;
> +}
> +
> int __init get_mem_nodeid(paddr_t start, paddr_t end)
> {
> unsigned int i;
> diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> index f0a50bd..ff10b31 100644
> --- a/xen/include/asm-arm/numa.h
> +++ b/xen/include/asm-arm/numa.h
> @@ -20,6 +20,7 @@ static inline nodeid_t acpi_get_nodeid(uint64_t hwid)
> void numa_init(void);
> int dt_numa_init(void);
> void numa_set_cpu_node(int cpu, unsigned int nid);
> +bool_t arch_acpi_numa_init(void);
>
> #else
> static inline void numa_init(void)
> diff --git a/xen/include/xen/numa.h b/xen/include/xen/numa.h
> index a541eb7..14a7a0c 100644
> --- a/xen/include/xen/numa.h
> +++ b/xen/include/xen/numa.h
> @@ -75,6 +75,7 @@ int get_num_node_memblks(void);
> bool arch_sanitize_nodes_memory(void);
> void numa_failed(void);
> uint8_t __node_distance(nodeid_t a, nodeid_t b);
> +void numa_clear_memblks(void);
> #else
> static inline void numa_add_cpu(int cpu) { }
> static inline void numa_set_node(int cpu, nodeid_t node) { }
>
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 23/24] NUMA: Move CONFIG_NUMA to common Kconfig
2017-07-18 16:25 ` Julien Grall
2017-07-18 18:00 ` Julien Grall
@ 2017-07-28 10:08 ` Jan Beulich
1 sibling, 0 replies; 109+ messages in thread
From: Jan Beulich @ 2017-07-28 10:08 UTC (permalink / raw)
To: julien.grall, vijay.kilari
Cc: tim, kevin.tian, sstabellini, wei.liu2, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, Vijaya.Kumar,
xen-devel
>>> Julien Grall <julien.grall@arm.com> 07/18/17 6:25 PM >>>
>On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
>> --- a/xen/common/Kconfig
>> +++ b/xen/common/Kconfig
>> @@ -41,6 +41,10 @@ config HAS_GDBSX
>> config HAS_IOPORTS
>> bool
>>
>> +config NUMA
>> + def_bool y
>> + depends on HAS_PDX
>
>On previous version, Jan asked to remove the dependency on PDX. You said
>you will do it... So why it is not done?
Plus it should remain a simple bool with select-s in the arch specific Kconfig-s,
as I'm pretty sure I had said back then too.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable
2017-07-18 15:29 ` Wei Liu
2017-07-18 17:52 ` Julien Grall
@ 2017-07-28 10:11 ` Jan Beulich
1 sibling, 0 replies; 109+ messages in thread
From: Jan Beulich @ 2017-07-28 10:11 UTC (permalink / raw)
To: wei.liu2, vijay.kilari
Cc: tim, kevin.tian, sstabellini, George.Dunlap, andrew.cooper3,
dario.faggioli, ian.jackson, Vijaya.Kumar, julien.grall,
xen-devel
>>> Wei Liu <wei.liu2@citrix.com> 07/18/17 5:30 PM >>>
>On Tue, Jul 18, 2017 at 05:11:23PM +0530, vijay.kilari@gmail.com wrote:
>> --- a/xen/arch/Kconfig
>> +++ b/xen/arch/Kconfig
>> @@ -6,3 +6,10 @@ config NR_CPUS
>> default "128" if ARM
>> ---help---
>> Specifies the maximum number of physical CPUs which Xen will support.
>> +
>> +config NR_NODES
>> + int "Maximum number of NUMA nodes"
>> + default "64" if X86
>> + default "8" if ARM
>> + ---help---
>> + Specifies the maximum number of NUMA nodes which Xen will support.
>
>Since this can now be specified by user but the definition of
>NUMA_NO_NODE is not changed, I think you need to sanitise the value
>provided somewhere.
>
>Maybe introduce a build time check? There are some examples in tree. See
>cpuid.c:build_assertions.
I think the option above should obtain a "range" line, at once excluding non-
positive values. A BUILD_BUG_ON() would then still be desirable, though.
Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 00/24] ARM: Add Xen NUMA support
[not found] ` <CALicx6svuo3JXik=8bYuciFzWDu6qmwVi1VXdBgjLp_f_YUhqQ@mail.gmail.com>
@ 2017-10-06 17:09 ` vkilari
2017-10-06 17:30 ` Julien Grall
0 siblings, 1 reply; 109+ messages in thread
From: vkilari @ 2017-10-06 17:09 UTC (permalink / raw)
To: julien.grall, xen-devel
Cc: vkilari, kevin.tian, sstabellini, wei.liu2, konrad.wilk,
George.Dunlap, andrew.cooper3, dario.faggioli, ian.jackson, tim,
jbeulich, sgoel, nd, shankerd
Hi Julien,
Sorry for not being active on this topic for some time. I was on vacation and have left cavium.
Note: My email has been changed
After going through the comments and fixing some of them, I see that clean up and code movement
patches are ~20+. So I would like to split this series into 2 patch series.
1) Separate patch series with x86 clean up and code movement.
2) Separate patch series for ARM NUMA patches
This helps to ease in review and ease in rebasing the cleanup and code movement patches.
Let me know if this ok. I will work and post the patches accordingly.
Regards
Vijay
> -----Original Message-----
> From: Vijay Kilari [mailto:vijay.kilari@gmail.com]
> Sent: Friday, October 6, 2017 7:33 PM
> To: vkilari@codeaurora.org
> Subject: Fwd: [RFC PATCH v3 00/24] ARM: Add Xen NUMA support
>
> ---------- Forwarded message ----------
> From: Julien Grall <julien.grall@arm.com>
> Date: Wed, Jul 19, 2017 at 12:48 PM
> Subject: Re: [RFC PATCH v3 00/24] ARM: Add Xen NUMA support
> To: Vijay Kilari <vijay.kilari@gmail.com>
> Cc: nd@arm.com, "xen-devel@lists.xen.org" <xen-devel@lists.xen.org>,
> Andrew Cooper <andrew.cooper3@citrix.com>, George Dunlap
> <George.Dunlap@eu.citrix.com>, Ian Jackson <ian.jackson@eu.citrix.com>, Jan
> Beulich <jbeulich@suse.com>, Konrad Rzeszutek Wilk
> <konrad.wilk@oracle.com>, Stefano Stabellini <sstabellini@kernel.org>, Tim
> Deegan <tim@xen.org>, Wei Liu <wei.liu2@citrix.com>, Dario Faggioli
> <dario.faggioli@citrix.com>, kevin.tian@intel.com, Vijaya Kumar K
> <Vijaya.Kumar@cavium.com>
>
>
>
>
> On 19/07/2017 07:31, Vijay Kilari wrote:
> >
> > On Tue, Jul 18, 2017 at 9:48 PM, Julien Grall <julien.grall@arm.com> wrote:
> >>
> >> Hi,
> >>
> >> On 18/07/17 12:41, vijay.kilari@gmail.com wrote:
> >>>
> >>>
> >>> This patch is tested on Thunderx platform.
> >>> No changes are made to x86 implementation only code is sanitized and
> >>> refactored. Hence only compilation tested for x86.
> >>>
> >>> This series is posted as RFC for the reason that it is not tested on
> >>> x86. Request some help from community in testing this series on x86.
> >>>
> >>> Code is shared at
> >>> https://github.com/vijaykilari/xen-numa/commits/rfc_v3
> >>
> >>
> >>
> >> Few months ago you sent a patch that was a pre-requisite for booting
> >> NUMA (see [1]). It has never been upstreamed, so is it still required?
> >
> >
> > yes, it is required and is merged
>
>
> Hmmm, somehow I was not able to find it in the tree yesterday. Found it know.
>
> Cheers,
>
>
> >
> > https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=c6fdc9696a6a6eac5
> > 9bf9c81121d1f1cd5b88dcd
> >
> >>
> >> Cheers,
> >>
> >> [1]
> >> https://lists.xenproject.org/archives/html/xen-devel/2017-03/msg03823
> >> .html
> >>
> >> --
> >> Julien Grall
>
>
> --
> Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
* Re: [RFC PATCH v3 00/24] ARM: Add Xen NUMA support
2017-10-06 17:09 ` vkilari
@ 2017-10-06 17:30 ` Julien Grall
0 siblings, 0 replies; 109+ messages in thread
From: Julien Grall @ 2017-10-06 17:30 UTC (permalink / raw)
To: vkilari, xen-devel
Cc: kevin.tian, sstabellini, wei.liu2, konrad.wilk, George.Dunlap,
andrew.cooper3, dario.faggioli, ian.jackson, tim, jbeulich,
sgoel, nd, shankerd
On 06/10/17 18:09, vkilari@codeaurora.org wrote:
> Hi Julien,
Hello,
>
> Sorry for not being active on this topic for some time. I was on vacation and have left cavium.
> Note: My email has been changed
>
> After going through the comments and fixing some of them, I see that clean up and code movement
> patches are ~20+. So I would like to split this series into 2 patch series.
>
> 1) Separate patch series with x86 clean up and code movement.
> 2) Separate patch series for ARM NUMA patches
>
> This helps to ease in review and ease in rebasing the cleanup and code movement patches.
TBH, it makes sense to have the x86 clean up in a separate series. I am
not so sure about the code movement.
I think having the ARM changes would help to know whether the code
movement is good.
Others may have a different opinion.
Cheers,
--
Julien Grall
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 109+ messages in thread
end of thread, other threads:[~2017-10-06 17:30 UTC | newest]
Thread overview: 109+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-07-18 11:41 [RFC PATCH v3 00/24] ARM: Add Xen NUMA support vijay.kilari
2017-07-18 11:41 ` [RFC PATCH v3 01/24] NUMA: Make number of NUMA nodes configurable vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-18 17:52 ` Julien Grall
2017-07-19 8:17 ` Wei Liu
2017-07-19 15:48 ` Julien Grall
2017-07-28 10:11 ` Jan Beulich
2017-07-18 17:55 ` Julien Grall
2017-07-19 7:00 ` Vijay Kilari
2017-07-19 15:55 ` Julien Grall
2017-07-20 7:30 ` Vijay Kilari
2017-07-20 10:57 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 02/24] x86: NUMA: Clean up: Fix coding styles and drop unused code vijay.kilari
2017-07-19 16:23 ` Julien Grall
2017-07-19 16:27 ` Wei Liu
2017-07-19 16:34 ` Julien Grall
2017-07-20 7:00 ` Vijay Kilari
2017-07-20 11:00 ` Julien Grall
2017-07-20 12:05 ` Vijay Kilari
2017-07-20 12:09 ` Julien Grall
2017-07-20 12:29 ` Vijay Kilari
2017-07-20 12:33 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 03/24] x86: NUMA: Fix datatypes and attributes vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-18 11:41 ` [RFC PATCH v3 04/24] x86: NUMA: Rename and sanitize memnode shift code vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-19 17:12 ` Julien Grall
2017-07-20 6:56 ` Vijay Kilari
2017-07-18 11:41 ` [RFC PATCH v3 05/24] x86: NUMA: Add accessors for nodes[] and node_memblk_range[] structs vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-19 6:40 ` Vijay Kilari
2017-07-19 17:18 ` Julien Grall
2017-07-20 7:41 ` Vijay Kilari
2017-07-20 11:03 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 06/24] x86: NUMA: Rename some generic functions vijay.kilari
2017-07-19 17:23 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 07/24] ARM: NUMA: Add existing ARM numa code under CONFIG_NUMA vijay.kilari
2017-07-18 18:06 ` Julien Grall
2017-07-20 9:31 ` Vijay Kilari
2017-07-20 11:10 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 08/24] NUMA: x86: Move numa code and make it generic vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-18 18:16 ` Julien Grall
2017-07-19 6:47 ` Vijay Kilari
2017-07-19 17:41 ` Julien Grall
2017-07-20 8:55 ` Vijay Kilari
2017-07-20 11:14 ` Julien Grall
2017-07-24 20:28 ` Stefano Stabellini
2017-07-18 11:41 ` [RFC PATCH v3 09/24] NUMA: x86: Move common code from srat.c vijay.kilari
2017-07-20 11:17 ` Julien Grall
2017-07-20 11:43 ` Vijay Kilari
2017-07-24 20:35 ` Stefano Stabellini
2017-07-18 11:41 ` [RFC PATCH v3 10/24] NUMA: Allow numa initialization with DT vijay.kilari
2017-07-19 17:58 ` Julien Grall
2017-07-20 10:28 ` Vijay Kilari
2017-07-20 11:20 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 11/24] ARM: fdt: Export and introduce new fdt functions vijay.kilari
2017-07-18 15:29 ` Wei Liu
2017-07-18 16:29 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 12/24] ARM: NUMA: DT: Parse CPU NUMA information vijay.kilari
2017-07-19 18:26 ` Julien Grall
2017-07-20 9:20 ` Vijay Kilari
2017-07-18 11:41 ` [RFC PATCH v3 13/24] ARM: NUMA: DT: Parse memory " vijay.kilari
2017-07-19 18:39 ` Julien Grall
2017-07-20 10:37 ` Vijay Kilari
2017-07-20 11:24 ` Julien Grall
2017-07-20 11:26 ` Julien Grall
2017-07-21 11:10 ` Vijay Kilari
2017-07-21 12:35 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 14/24] ARM: NUMA: DT: Parse NUMA distance information vijay.kilari
2017-07-20 13:02 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 15/24] ARM: NUMA: DT: Add CPU NUMA support vijay.kilari
2017-07-24 11:24 ` Julien Grall
2017-07-25 6:47 ` Vijay Kilari
2017-07-25 18:38 ` Julien Grall
2017-07-25 18:48 ` Stefano Stabellini
2017-07-25 18:51 ` Julien Grall
2017-07-25 19:06 ` Stefano Stabellini
2017-07-26 17:18 ` Julien Grall
2017-07-26 17:21 ` Stefano Stabellini
2017-07-18 11:41 ` [RFC PATCH v3 16/24] ARM: NUMA: Add memory " vijay.kilari
2017-07-24 12:43 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 17/24] ARM: NUMA: DT: Do not expose numa info to DOM0 vijay.kilari
2017-07-24 20:48 ` Stefano Stabellini
2017-07-26 17:22 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 18/24] ACPI: Refactor acpi SRAT and SLIT table handling code vijay.kilari
2017-07-18 15:36 ` Wei Liu
2017-07-19 6:33 ` Vijay Kilari
2017-07-18 11:41 ` [RFC PATCH v3 19/24] ARM: NUMA: Extract MPIDR from MADT table vijay.kilari
2017-07-24 22:17 ` Stefano Stabellini
2017-07-26 18:12 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 20/24] ACPI: Move arch specific SRAT parsing vijay.kilari
2017-07-24 21:15 ` Stefano Stabellini
2017-07-18 11:41 ` [RFC PATCH v3 21/24] ARM: NUMA: ACPI: Extract proximity from SRAT table vijay.kilari
2017-07-24 22:17 ` Stefano Stabellini
2017-07-26 18:18 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 22/24] ARM: NUMA: Initialize ACPI NUMA vijay.kilari
2017-07-24 22:11 ` Stefano Stabellini
2017-07-26 18:23 ` Julien Grall
2017-07-18 11:41 ` [RFC PATCH v3 23/24] NUMA: Move CONFIG_NUMA to common Kconfig vijay.kilari
2017-07-18 16:25 ` Julien Grall
2017-07-18 18:00 ` Julien Grall
2017-07-28 10:08 ` Jan Beulich
2017-07-18 11:41 ` [RFC PATCH v3 24/24] NUMA: Enable ACPI_NUMA config vijay.kilari
2017-07-18 16:18 ` [RFC PATCH v3 00/24] ARM: Add Xen NUMA support Julien Grall
2017-07-19 6:31 ` Vijay Kilari
2017-07-19 7:18 ` Julien Grall
[not found] ` <CALicx6svuo3JXik=8bYuciFzWDu6qmwVi1VXdBgjLp_f_YUhqQ@mail.gmail.com>
2017-10-06 17:09 ` vkilari
2017-10-06 17:30 ` Julien Grall
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.