[PATCH v4 00/14] fix some type infos and bugs for arm64/of numa

[PATCH v4 00/14] fix some type infos and bugs for arm64/of numa

[Zhen Lei]

v3 -> v4:
1. Packed three patches of Kefeng Wang, patch6-8.
2. Add 6 new patches(9-15) to enhance the numa on arm64.

v2 -> v3:
1. Adjust patch2 and patch5 according to Matthias Brugger's advice, to make the
   patches looks more well. The final code have no change. 

v1 -> v2:
1. Base on https://lkml.org/lkml/2016/5/24/679
2. Rewrote of_numa_parse_memory_nodes according to Rob Herring's advice. So that it looks more clear.
3. Rewrote patch 5 because some scenes were not considered before.

Kefeng Wang (3):
  of_numa: Use of_get_next_parent to simplify code
  of_numa: Use pr_fmt()
  arm64: numa: Use pr_fmt()

Zhen Lei (11):
  of/numa: remove a duplicated pr_debug information
  of/numa: fix a memory@ node can only contains one memory block
  arm64/numa: add nid check for memory block
  of/numa: remove a duplicated warning
  arm64/numa: avoid inconsistent information to be printed
  arm64/numa: support HAVE_SETUP_PER_CPU_AREA
  arm64/numa: define numa_distance as array to simplify code
  arm64/numa: support HAVE_MEMORYLESS_NODES
  arm64/numa: remove some useless code
  of/numa: remove the constraint on the distances of node pairs
  Documentation: remove the constraint on the distances of node pairs

 Documentation/devicetree/bindings/numa.txt |   1 -
 arch/arm64/Kconfig                         |  12 ++
 arch/arm64/include/asm/numa.h              |   1 -
 arch/arm64/kernel/smp.c                    |   1 +
 arch/arm64/mm/numa.c                       | 228 ++++++++++++++++-------------
 drivers/of/of_numa.c                       |  87 +++++------
 6 files changed, 180 insertions(+), 150 deletions(-)

--
2.5.0

[PATCH v4 01/14] of/numa: remove a duplicated pr_debug information

[Zhen Lei]

This information will be printed in the subfunction numa_add_memblk.
They are not the same, but very similar.

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
 drivers/of/of_numa.c | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
index ed5a097..fb71b4e 100644
--- a/drivers/of/of_numa.c
+++ b/drivers/of/of_numa.c
@@ -88,10 +88,6 @@ static int __init of_numa_parse_memory_nodes(void)
 			break;
 		}

-		pr_debug("NUMA:  base = %llx len = %llx, node = %u\n",
-			 rsrc.start, rsrc.end - rsrc.start + 1, nid);
-
-
 		r = numa_add_memblk(nid, rsrc.start, rsrc.end + 1);
 		if (r)
 			break;
--
2.5.0

[PATCH v4 02/14] of/numa: fix a memory@ node can only contains one memory block

[Zhen Lei]

For a normal memory@ devicetree node, its reg property can contains more
memory blocks.

Because we don't known how many memory blocks maybe contained, so we try
from index=0, increase 1 until error returned(the end).

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
 drivers/of/of_numa.c | 29 ++++++++++-------------------
 1 file changed, 10 insertions(+), 19 deletions(-)

diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
index fb71b4e..7b3fbdc 100644
--- a/drivers/of/of_numa.c
+++ b/drivers/of/of_numa.c
@@ -63,13 +63,9 @@ static int __init of_numa_parse_memory_nodes(void)
 	struct device_node *np = NULL;
 	struct resource rsrc;
 	u32 nid;
-	int r = 0;
-
-	for (;;) {
-		np = of_find_node_by_type(np, "memory");
-		if (!np)
-			break;
+	int i, r;

+	for_each_node_by_type(np, "memory") {
 		r = of_property_read_u32(np, "numa-node-id", &nid);
 		if (r == -EINVAL)
 			/*
@@ -78,23 +74,18 @@ static int __init of_numa_parse_memory_nodes(void)
 			 * "numa-node-id" property
 			 */
 			continue;
-		else if (r)
-			/* some other error */
-			break;

-		r = of_address_to_resource(np, 0, &rsrc);
-		if (r) {
-			pr_err("NUMA: bad reg property in memory node\n");
-			break;
-		}
+		for (i = 0; !r && !of_address_to_resource(np, i, &rsrc); i++)
+			r = numa_add_memblk(nid, rsrc.start, rsrc.end + 1);

-		r = numa_add_memblk(nid, rsrc.start, rsrc.end + 1);
-		if (r)
-			break;
+		if (!i || r) {
+			of_node_put(np);
+			pr_err("NUMA: bad property in memory node\n");
+			return r ? : -EINVAL;
+		}
 	}
-	of_node_put(np);

-	return r;
+	return 0;
 }

 static int __init of_numa_parse_distance_map_v1(struct device_node *map)
--
2.5.0

[PATCH v4 03/14] arm64/numa: add nid check for memory block

[Zhen Lei]

Use the same tactic to cpu and numa-distance nodes.

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
 arch/arm64/mm/numa.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
index c7fe3ec..2601660 100644
--- a/arch/arm64/mm/numa.c
+++ b/arch/arm64/mm/numa.c
@@ -141,6 +141,11 @@ int __init numa_add_memblk(int nid, u64 start, u64 end)
 {
 	int ret;

+	if (nid >= MAX_NUMNODES) {
+		pr_warn("NUMA: Node id %u exceeds maximum value\n", nid);
+		return -EINVAL;
+	}
+
 	ret = memblock_set_node(start, (end - start), &memblock.memory, nid);
 	if (ret < 0) {
 		pr_err("NUMA: memblock [0x%llx - 0x%llx] failed to add on node %d\n",
--
2.5.0

[PATCH v4 04/14] of/numa: remove a duplicated warning

[Zhen Lei]

This warning has been printed in of_numa_parse_cpu_nodes before.

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
 drivers/of/of_numa.c | 9 ++-------
 1 file changed, 2 insertions(+), 7 deletions(-)

diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
index 7b3fbdc..3157130 100644
--- a/drivers/of/of_numa.c
+++ b/drivers/of/of_numa.c
@@ -174,13 +174,8 @@ int of_node_to_nid(struct device_node *device)
 			np->name);
 	of_node_put(np);

-	if (!r) {
-		if (nid >= MAX_NUMNODES)
-			pr_warn("NUMA: Node id %u exceeds maximum value\n",
-				nid);
-		else
-			return nid;
-	}
+	if (!r)
+		return nid;

 	return NUMA_NO_NODE;
 }
--
2.5.0

[PATCH v4 05/14] arm64/numa: avoid inconsistent information to be printed

[Zhen Lei]

numa_init(of_numa_init) may returned error because of numa configuration
error. So "No NUMA configuration found" is inaccurate. In fact, specific
configuration error information should be immediately printed by the
testing branch.

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
 arch/arm64/mm/numa.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
index 2601660..1b9622c 100644
--- a/arch/arm64/mm/numa.c
+++ b/arch/arm64/mm/numa.c
@@ -338,8 +338,10 @@ static int __init numa_init(int (*init_func)(void))
 	if (ret < 0)
 		return ret;

-	if (nodes_empty(numa_nodes_parsed))
+	if (nodes_empty(numa_nodes_parsed)) {
+		pr_info("No NUMA configuration found\n");
 		return -EINVAL;
+	}

 	ret = numa_register_nodes();
 	if (ret < 0)
@@ -370,8 +372,6 @@ static int __init dummy_numa_init(void)

 	if (numa_off)
 		pr_info("NUMA disabled\n"); /* Forced off on command line. */
-	else
-		pr_info("No NUMA configuration found\n");
 	pr_info("NUMA: Faking a node at [mem %#018Lx-%#018Lx]\n",
 	       0LLU, PFN_PHYS(max_pfn) - 1);

--
2.5.0

[PATCH v4 06/14] of_numa: Use of_get_next_parent to simplify code

[Zhen Lei]

From: Kefeng Wang <wangkefeng.wang@huawei.com>

Use of_get_next_parent() instead of open-code.

Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
---
 drivers/of/of_numa.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
index 3157130..ed7bd22 100644
--- a/drivers/of/of_numa.c
+++ b/drivers/of/of_numa.c
@@ -153,8 +153,6 @@ int of_node_to_nid(struct device_node *device)
 	np = of_node_get(device);

 	while (np) {
-		struct device_node *parent;
-
 		r = of_property_read_u32(np, "numa-node-id", &nid);
 		/*
 		 * -EINVAL indicates the property was not found, and
@@ -165,9 +163,7 @@ int of_node_to_nid(struct device_node *device)
 		if (r != -EINVAL)
 			break;

-		parent = of_get_parent(np);
-		of_node_put(np);
-		np = parent;
+		np = of_get_next_parent(np);
 	}
 	if (np && r)
 		pr_warn("NUMA: Invalid \"numa-node-id\" property in node %s\n",
--
2.5.0

[PATCH v4 07/14] of_numa: Use pr_fmt()

[Zhen Lei]

From: Kefeng Wang <wangkefeng.wang@huawei.com>

Use pr_fmt to prefix kernel output.

Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
---
 drivers/of/of_numa.c | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
index ed7bd22..019738f 100644
--- a/drivers/of/of_numa.c
+++ b/drivers/of/of_numa.c
@@ -16,6 +16,8 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */

+#define pr_fmt(fmt) "OF: NUMA: " fmt
+
 #include <linux/of.h>
 #include <linux/of_address.h>
 #include <linux/nodemask.h>
@@ -49,10 +51,9 @@ static void __init of_numa_parse_cpu_nodes(void)
 		if (r)
 			continue;

-		pr_debug("NUMA: CPU on %u\n", nid);
+		pr_debug("CPU on %u\n", nid);
 		if (nid >= MAX_NUMNODES)
-			pr_warn("NUMA: Node id %u exceeds maximum value\n",
-				nid);
+			pr_warn("Node id %u exceeds maximum value\n", nid);
 		else
 			node_set(nid, numa_nodes_parsed);
 	}
@@ -80,7 +81,7 @@ static int __init of_numa_parse_memory_nodes(void)

 		if (!i || r) {
 			of_node_put(np);
-			pr_err("NUMA: bad property in memory node\n");
+			pr_err("bad property in memory node\n");
 			return r ? : -EINVAL;
 		}
 	}
@@ -94,17 +95,17 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map)
 	int entry_count;
 	int i;

-	pr_info("NUMA: parsing numa-distance-map-v1\n");
+	pr_info("parsing numa-distance-map-v1\n");

 	matrix = of_get_property(map, "distance-matrix", NULL);
 	if (!matrix) {
-		pr_err("NUMA: No distance-matrix property in distance-map\n");
+		pr_err("No distance-matrix property in distance-map\n");
 		return -EINVAL;
 	}

 	entry_count = of_property_count_u32_elems(map, "distance-matrix");
 	if (entry_count <= 0) {
-		pr_err("NUMA: Invalid distance-matrix\n");
+		pr_err("Invalid distance-matrix\n");
 		return -EINVAL;
 	}

@@ -119,7 +120,7 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map)
 		matrix++;

 		numa_set_distance(nodea, nodeb, distance);
-		pr_debug("NUMA:  distance[node%d -> node%d] = %d\n",
+		pr_debug("distance[node%d -> node%d] = %d\n",
 			 nodea, nodeb, distance);

 		/* Set default distance of node B->A same as A->B */
@@ -166,7 +167,7 @@ int of_node_to_nid(struct device_node *device)
 		np = of_get_next_parent(np);
 	}
 	if (np && r)
-		pr_warn("NUMA: Invalid \"numa-node-id\" property in node %s\n",
+		pr_warn("Invalid \"numa-node-id\" property in node %s\n",
 			np->name);
 	of_node_put(np);

--
2.5.0

[PATCH v4 08/14] arm64: numa: Use pr_fmt()

[Zhen Lei]

From: Kefeng Wang <wangkefeng.wang@huawei.com>

Use pr_fmt to prefix kernel output, and remove duplicated msg
of NUMA turned off.

Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
---
 arch/arm64/mm/numa.c | 42 +++++++++++++++++++++---------------------
 1 file changed, 21 insertions(+), 21 deletions(-)

diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
index 1b9622c..8be5ba3 100644
--- a/arch/arm64/mm/numa.c
+++ b/arch/arm64/mm/numa.c
@@ -17,6 +17,8 @@
  * along with this program.  If not, see <http://www.gnu.org/licenses/>.
  */

+#define pr_fmt(fmt) "numa: " fmt
+
 #include <linux/acpi.h>
 #include <linux/bootmem.h>
 #include <linux/memblock.h>
@@ -36,10 +38,9 @@ static __init int numa_parse_early_param(char *opt)
 {
 	if (!opt)
 		return -EINVAL;
-	if (!strncmp(opt, "off", 3)) {
-		pr_info("%s\n", "NUMA turned off");
+	if (!strncmp(opt, "off", 3))
 		numa_off = true;
-	}
+
 	return 0;
 }
 early_param("numa", numa_parse_early_param);
@@ -108,7 +109,7 @@ static void __init setup_node_to_cpumask_map(void)
 		set_cpu_numa_node(cpu, NUMA_NO_NODE);

 	/* cpumask_of_node() will now work */
-	pr_debug("NUMA: Node to cpumask map for %d nodes\n", nr_node_ids);
+	pr_debug("Node to cpumask map for %d nodes\n", nr_node_ids);
 }

 /*
@@ -142,19 +143,19 @@ int __init numa_add_memblk(int nid, u64 start, u64 end)
 	int ret;

 	if (nid >= MAX_NUMNODES) {
-		pr_warn("NUMA: Node id %u exceeds maximum value\n", nid);
+		pr_warn("Node id %u exceeds maximum value\n", nid);
 		return -EINVAL;
 	}

 	ret = memblock_set_node(start, (end - start), &memblock.memory, nid);
 	if (ret < 0) {
-		pr_err("NUMA: memblock [0x%llx - 0x%llx] failed to add on node %d\n",
+		pr_err("memblock [0x%llx - 0x%llx] failed to add on node %d\n",
 			start, (end - 1), nid);
 		return ret;
 	}

 	node_set(nid, numa_nodes_parsed);
-	pr_info("NUMA: Adding memblock [0x%llx - 0x%llx] on node %d\n",
+	pr_info("Adding memblock [0x%llx - 0x%llx] on node %d\n",
 			start, (end - 1), nid);
 	return ret;
 }
@@ -169,19 +170,18 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
 	void *nd;
 	int tnid;

-	pr_info("NUMA: Initmem setup node %d [mem %#010Lx-%#010Lx]\n",
-			nid, start_pfn << PAGE_SHIFT,
-			(end_pfn << PAGE_SHIFT) - 1);
+	pr_info("Initmem setup node %d [mem %#010Lx-%#010Lx]\n",
+		nid, start_pfn << PAGE_SHIFT, (end_pfn << PAGE_SHIFT) - 1);

 	nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
 	nd = __va(nd_pa);

 	/* report and initialize */
-	pr_info("NUMA: NODE_DATA [mem %#010Lx-%#010Lx]\n",
+	pr_info("  NODE_DATA [mem %#010Lx-%#010Lx]\n",
 		nd_pa, nd_pa + nd_size - 1);
 	tnid = early_pfn_to_nid(nd_pa >> PAGE_SHIFT);
 	if (tnid != nid)
-		pr_info("NUMA: NODE_DATA(%d) on node %d\n", nid, tnid);
+		pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);

 	node_data[nid] = nd;
 	memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
@@ -238,8 +238,7 @@ static int __init numa_alloc_distance(void)
 			numa_distance[i * numa_distance_cnt + j] = i == j ?
 				LOCAL_DISTANCE : REMOTE_DISTANCE;

-	pr_debug("NUMA: Initialized distance table, cnt=%d\n",
-			numa_distance_cnt);
+	pr_debug("Initialized distance table, cnt=%d\n", numa_distance_cnt);

 	return 0;
 }
@@ -260,20 +259,20 @@ static int __init numa_alloc_distance(void)
 void __init numa_set_distance(int from, int to, int distance)
 {
 	if (!numa_distance) {
-		pr_warn_once("NUMA: Warning: distance table not allocated yet\n");
+		pr_warn_once("Warning: distance table not allocated yet\n");
 		return;
 	}

 	if (from >= numa_distance_cnt || to >= numa_distance_cnt ||
 			from < 0 || to < 0) {
-		pr_warn_once("NUMA: Warning: node ids are out of bound, from=%d to=%d distance=%d\n",
+		pr_warn_once("Warning: node ids are out of bound, from=%d to=%d distance=%d\n",
 			    from, to, distance);
 		return;
 	}

 	if ((u8)distance != distance ||
 	    (from == to && distance != LOCAL_DISTANCE)) {
-		pr_warn_once("NUMA: Warning: invalid distance parameter, from=%d to=%d distance=%d\n",
+		pr_warn_once("Warning: invalid distance parameter, from=%d to=%d distance=%d\n",
 			     from, to, distance);
 		return;
 	}
@@ -300,7 +299,7 @@ static int __init numa_register_nodes(void)
 	/* Check that valid nid is set to memblks */
 	for_each_memblock(memory, mblk)
 		if (mblk->nid == NUMA_NO_NODE || mblk->nid >= MAX_NUMNODES) {
-			pr_warn("NUMA: Warning: invalid memblk node %d [mem %#010Lx-%#010Lx]\n",
+			pr_warn("Warning: invalid memblk node %d [mem %#010Lx-%#010Lx]\n",
 				mblk->nid, mblk->base,
 				mblk->base + mblk->size - 1);
 			return -EINVAL;
@@ -371,9 +370,10 @@ static int __init dummy_numa_init(void)
 	struct memblock_region *mblk;

 	if (numa_off)
-		pr_info("NUMA disabled\n"); /* Forced off on command line. */
-	pr_info("NUMA: Faking a node at [mem %#018Lx-%#018Lx]\n",
-	       0LLU, PFN_PHYS(max_pfn) - 1);
+		pr_warn("NUMA turned off by user\n"); /* Forced off on command line. */
+
+	pr_info("Faking a node at [mem %#018Lx-%#018Lx]\n",
+	        0LLU, PFN_PHYS(max_pfn) - 1);

 	for_each_memblock(memory, mblk) {
 		ret = numa_add_memblk(0, mblk->base, mblk->base + mblk->size);
--
2.5.0

[PATCH v4 09/14] arm64/numa: support HAVE_SETUP_PER_CPU_AREA

[Zhen Lei]

To make each percpu area allocated from its local numa node. Without this
patch, all percpu areas will be allocated from the node which cpu0 belongs
to.

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
 arch/arm64/Kconfig   |  8 ++++++++
 arch/arm64/mm/numa.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 64 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 76747d9..05c1bf1 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -573,6 +573,14 @@ config USE_PERCPU_NUMA_NODE_ID
 	def_bool y
 	depends on NUMA

+config HAVE_SETUP_PER_CPU_AREA
+	def_bool y
+	depends on NUMA
+
+config NEED_PER_CPU_EMBED_FIRST_CHUNK
+	def_bool y
+	depends on NUMA
+
 source kernel/Kconfig.preempt
 source kernel/Kconfig.hz

diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
index 8be5ba3..99401aa 100644
--- a/arch/arm64/mm/numa.c
+++ b/arch/arm64/mm/numa.c
@@ -25,6 +25,8 @@
 #include <linux/module.h>
 #include <linux/of.h>

+#include <asm/sections.h>
+
 struct pglist_data *node_data[MAX_NUMNODES] __read_mostly;
 EXPORT_SYMBOL(node_data);
 nodemask_t numa_nodes_parsed __initdata;
@@ -129,6 +131,60 @@ void __init early_map_cpu_to_node(unsigned int cpu, int nid)
 	cpu_to_node_map[cpu] = nid;
 }

+#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
+unsigned long __per_cpu_offset[NR_CPUS] __read_mostly;
+EXPORT_SYMBOL(__per_cpu_offset);
+
+static int __init early_cpu_to_node(int cpu)
+{
+	return cpu_to_node_map[cpu];
+}
+
+static int __init pcpu_cpu_distance(unsigned int from, unsigned int to)
+{
+	if (early_cpu_to_node(from) == early_cpu_to_node(to))
+		return LOCAL_DISTANCE;
+	else
+		return REMOTE_DISTANCE;
+}
+
+static void * __init pcpu_fc_alloc(unsigned int cpu, size_t size,
+				       size_t align)
+{
+	int nid = early_cpu_to_node(cpu);
+
+	return  memblock_virt_alloc_try_nid(size, align,
+			__pa(MAX_DMA_ADDRESS), MEMBLOCK_ALLOC_ACCESSIBLE, nid);
+}
+
+static void __init pcpu_fc_free(void *ptr, size_t size)
+{
+	memblock_free_early(__pa(ptr), size);
+}
+
+void __init setup_per_cpu_areas(void)
+{
+	unsigned long delta;
+	unsigned int cpu;
+	int rc;
+
+	/*
+	 * Always reserve area for module percpu variables.  That's
+	 * what the legacy allocator did.
+	 */
+	rc = pcpu_embed_first_chunk(PERCPU_MODULE_RESERVE,
+				    PERCPU_DYNAMIC_RESERVE, PAGE_SIZE,
+				    pcpu_cpu_distance,
+				    pcpu_fc_alloc, pcpu_fc_free);
+	if (rc < 0)
+		panic("Failed to initialize percpu areas.");
+
+	delta = (unsigned long)pcpu_base_addr - (unsigned long)__per_cpu_start;
+	for_each_possible_cpu(cpu)
+		__per_cpu_offset[cpu] = delta + pcpu_unit_offsets[cpu];
+}
+#endif
+
 /**
  * numa_add_memblk - Set node id to memblk
  * @nid: NUMA node ID of the new memblk
--
2.5.0

[PATCH v4 10/14] arm64/numa: define numa_distance as array to simplify code

[Zhen Lei]

1. MAX_NUMNODES is base on CONFIG_NODES_SHIFT, the default value of the
   latter is very small now.
2. Suppose the default value of MAX_NUMNODES is enlarged to 64, so the
   size of numa_distance is 4K, it's still acceptable if run the Image
   on other processors.
3. It will make function __node_distance quicker than before.

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
 arch/arm64/include/asm/numa.h |  1 -
 arch/arm64/mm/numa.c          | 74 +++----------------------------------------
 2 files changed, 5 insertions(+), 70 deletions(-)

diff --git a/arch/arm64/include/asm/numa.h b/arch/arm64/include/asm/numa.h
index 600887e..9b6cc38 100644
--- a/arch/arm64/include/asm/numa.h
+++ b/arch/arm64/include/asm/numa.h
@@ -32,7 +32,6 @@ static inline const struct cpumask *cpumask_of_node(int node)
 void __init arm64_numa_init(void);
 int __init numa_add_memblk(int nodeid, u64 start, u64 end);
 void __init numa_set_distance(int from, int to, int distance);
-void __init numa_free_distance(void);
 void __init early_map_cpu_to_node(unsigned int cpu, int nid);
 void numa_store_cpu_info(unsigned int cpu);

diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
index 99401aa..df5c842 100644
--- a/arch/arm64/mm/numa.c
+++ b/arch/arm64/mm/numa.c
@@ -32,8 +32,7 @@ EXPORT_SYMBOL(node_data);
 nodemask_t numa_nodes_parsed __initdata;
 static int cpu_to_node_map[NR_CPUS] = { [0 ... NR_CPUS-1] = NUMA_NO_NODE };

-static int numa_distance_cnt;
-static u8 *numa_distance;
+static u8 numa_distance[MAX_NUMNODES][MAX_NUMNODES];
 static bool numa_off;

 static __init int numa_parse_early_param(char *opt)
@@ -247,59 +246,6 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
 }

 /**
- * numa_free_distance
- *
- * The current table is freed.
- */
-void __init numa_free_distance(void)
-{
-	size_t size;
-
-	if (!numa_distance)
-		return;
-
-	size = numa_distance_cnt * numa_distance_cnt *
-		sizeof(numa_distance[0]);
-
-	memblock_free(__pa(numa_distance), size);
-	numa_distance_cnt = 0;
-	numa_distance = NULL;
-}
-
-/**
- *
- * Create a new NUMA distance table.
- *
- */
-static int __init numa_alloc_distance(void)
-{
-	size_t size;
-	u64 phys;
-	int i, j;
-
-	size = nr_node_ids * nr_node_ids * sizeof(numa_distance[0]);
-	phys = memblock_find_in_range(0, PFN_PHYS(max_pfn),
-				      size, PAGE_SIZE);
-	if (WARN_ON(!phys))
-		return -ENOMEM;
-
-	memblock_reserve(phys, size);
-
-	numa_distance = __va(phys);
-	numa_distance_cnt = nr_node_ids;
-
-	/* fill with the default distances */
-	for (i = 0; i < numa_distance_cnt; i++)
-		for (j = 0; j < numa_distance_cnt; j++)
-			numa_distance[i * numa_distance_cnt + j] = i == j ?
-				LOCAL_DISTANCE : REMOTE_DISTANCE;
-
-	pr_debug("Initialized distance table, cnt=%d\n", numa_distance_cnt);
-
-	return 0;
-}
-
-/**
  * numa_set_distance - Set inter node NUMA distance from node to node.
  * @from: the 'from' node to set distance
  * @to: the 'to'  node to set distance
@@ -314,12 +260,7 @@ static int __init numa_alloc_distance(void)
  */
 void __init numa_set_distance(int from, int to, int distance)
 {
-	if (!numa_distance) {
-		pr_warn_once("Warning: distance table not allocated yet\n");
-		return;
-	}
-
-	if (from >= numa_distance_cnt || to >= numa_distance_cnt ||
+	if (from >= MAX_NUMNODES || to >= MAX_NUMNODES ||
 			from < 0 || to < 0) {
 		pr_warn_once("Warning: node ids are out of bound, from=%d to=%d distance=%d\n",
 			    from, to, distance);
@@ -333,7 +274,7 @@ void __init numa_set_distance(int from, int to, int distance)
 		return;
 	}

-	numa_distance[from * numa_distance_cnt + to] = distance;
+	numa_distance[from][to] = distance;
 }

 /**
@@ -341,9 +282,9 @@ void __init numa_set_distance(int from, int to, int distance)
  */
 int __node_distance(int from, int to)
 {
-	if (from >= numa_distance_cnt || to >= numa_distance_cnt)
+	if (from >= MAX_NUMNODES || to >= MAX_NUMNODES)
 		return from == to ? LOCAL_DISTANCE : REMOTE_DISTANCE;
-	return numa_distance[from * numa_distance_cnt + to];
+	return numa_distance[from][to];
 }
 EXPORT_SYMBOL(__node_distance);

@@ -383,11 +324,6 @@ static int __init numa_init(int (*init_func)(void))
 	nodes_clear(numa_nodes_parsed);
 	nodes_clear(node_possible_map);
 	nodes_clear(node_online_map);
-	numa_free_distance();
-
-	ret = numa_alloc_distance();
-	if (ret < 0)
-		return ret;

 	ret = init_func();
 	if (ret < 0)
--
2.5.0

[PATCH v4 11/14] arm64/numa: support HAVE_MEMORYLESS_NODES

[Zhen Lei]

Some numa nodes may have no memory. For example:
1. cpu0 on node0
2. cpu1 on node1
3. device0 access the momory from node0 and node1 take the same time.

So, we can not simply classify device0 to node0 or node1, but we can
define a node2 which distances to node0 and node1 are the same.

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
 arch/arm64/Kconfig      |  4 ++++
 arch/arm64/kernel/smp.c |  1 +
 arch/arm64/mm/numa.c    | 43 +++++++++++++++++++++++++++++++++++++++++--
 3 files changed, 46 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 05c1bf1..5904a62 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -581,6 +581,10 @@ config NEED_PER_CPU_EMBED_FIRST_CHUNK
 	def_bool y
 	depends on NUMA

+config HAVE_MEMORYLESS_NODES
+	def_bool y
+	depends on NUMA
+
 source kernel/Kconfig.preempt
 source kernel/Kconfig.hz

diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index d099306..9e15297 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -620,6 +620,7 @@ static void __init of_parse_and_init_cpus(void)
 			}

 			bootcpu_valid = true;
+			early_map_cpu_to_node(0, of_node_to_nid(dn));

 			/*
 			 * cpu_logical_map has already been
diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
index df5c842..d73b0a0 100644
--- a/arch/arm64/mm/numa.c
+++ b/arch/arm64/mm/numa.c
@@ -128,6 +128,14 @@ void __init early_map_cpu_to_node(unsigned int cpu, int nid)
 		nid = 0;

 	cpu_to_node_map[cpu] = nid;
+
+	/*
+	 * We should set the numa node of cpu0 as soon as possible, because it
+	 * has already been set up online before. cpu_to_node(0) will soon be
+	 * called.
+	 */
+	if (!cpu)
+		set_cpu_numa_node(cpu, nid);
 }

 #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
@@ -215,6 +223,35 @@ int __init numa_add_memblk(int nid, u64 start, u64 end)
 	return ret;
 }

+static u64 __init alloc_node_data_from_nearest_node(int nid, const size_t size)
+{
+	int i, best_nid, distance;
+	u64 pa;
+	DECLARE_BITMAP(nodes_map, MAX_NUMNODES);
+
+	bitmap_zero(nodes_map, MAX_NUMNODES);
+	bitmap_set(nodes_map, nid, 1);
+
+find_nearest_node:
+	best_nid = NUMA_NO_NODE;
+	distance = INT_MAX;
+
+	for_each_clear_bit(i, nodes_map, MAX_NUMNODES)
+		if (numa_distance[nid][i] < distance) {
+			best_nid = i;
+			distance = numa_distance[nid][i];
+		}
+
+	pa = memblock_alloc_nid(size, SMP_CACHE_BYTES, best_nid);
+	if (!pa) {
+		BUG_ON(best_nid == NUMA_NO_NODE);
+		bitmap_set(nodes_map, best_nid, 1);
+		goto find_nearest_node;
+	}
+
+	return pa;
+}
+
 /**
  * Initialize NODE_DATA for a node on the local memory
  */
@@ -228,7 +265,9 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
 	pr_info("Initmem setup node %d [mem %#010Lx-%#010Lx]\n",
 		nid, start_pfn << PAGE_SHIFT, (end_pfn << PAGE_SHIFT) - 1);

-	nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
+	nd_pa = memblock_alloc_nid(nd_size, SMP_CACHE_BYTES, nid);
+	if (!nd_pa)
+		nd_pa = alloc_node_data_from_nearest_node(nid, nd_size);
 	nd = __va(nd_pa);

 	/* report and initialize */
@@ -238,7 +277,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
 	if (tnid != nid)
 		pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);

-	node_data[nid] = nd;
+	NODE_DATA(nid) = nd;
 	memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
 	NODE_DATA(nid)->node_id = nid;
 	NODE_DATA(nid)->node_start_pfn = start_pfn;
--
2.5.0

Re: [PATCH v4 11/14] arm64/numa: support HAVE_MEMORYLESS_NODES

[Ganapatrao Kulkarni]

On Tue, Jun 7, 2016 at 1:38 PM, Zhen Lei <thunder.leizhen@huawei.com> wrote:
> Some numa nodes may have no memory. For example:
> 1. cpu0 on node0
> 2. cpu1 on node1
> 3. device0 access the momory from node0 and node1 take the same time.

i am wondering, if access to both nodes is same, then why you need numa.
the example you are quoting is against the basic principle of "numa"
what is device0 here? cpu?
>
> So, we can not simply classify device0 to node0 or node1, but we can
> define a node2 which distances to node0 and node1 are the same.
>
> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
> ---
>  arch/arm64/Kconfig      |  4 ++++
>  arch/arm64/kernel/smp.c |  1 +
>  arch/arm64/mm/numa.c    | 43 +++++++++++++++++++++++++++++++++++++++++--
>  3 files changed, 46 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 05c1bf1..5904a62 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -581,6 +581,10 @@ config NEED_PER_CPU_EMBED_FIRST_CHUNK
>         def_bool y
>         depends on NUMA
>
> +config HAVE_MEMORYLESS_NODES
> +       def_bool y
> +       depends on NUMA
> +
>  source kernel/Kconfig.preempt
>  source kernel/Kconfig.hz
>
> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
> index d099306..9e15297 100644
> --- a/arch/arm64/kernel/smp.c
> +++ b/arch/arm64/kernel/smp.c
> @@ -620,6 +620,7 @@ static void __init of_parse_and_init_cpus(void)
>                         }
>
>                         bootcpu_valid = true;
> +                       early_map_cpu_to_node(0, of_node_to_nid(dn));
>
>                         /*
>                          * cpu_logical_map has already been
> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
> index df5c842..d73b0a0 100644
> --- a/arch/arm64/mm/numa.c
> +++ b/arch/arm64/mm/numa.c
> @@ -128,6 +128,14 @@ void __init early_map_cpu_to_node(unsigned int cpu, int nid)
>                 nid = 0;
>
>         cpu_to_node_map[cpu] = nid;
> +
> +       /*
> +        * We should set the numa node of cpu0 as soon as possible, because it
> +        * has already been set up online before. cpu_to_node(0) will soon be
> +        * called.
> +        */
> +       if (!cpu)
> +               set_cpu_numa_node(cpu, nid);
>  }
>
>  #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
> @@ -215,6 +223,35 @@ int __init numa_add_memblk(int nid, u64 start, u64 end)
>         return ret;
>  }
>
> +static u64 __init alloc_node_data_from_nearest_node(int nid, const size_t size)
> +{
> +       int i, best_nid, distance;
> +       u64 pa;
> +       DECLARE_BITMAP(nodes_map, MAX_NUMNODES);
> +
> +       bitmap_zero(nodes_map, MAX_NUMNODES);
> +       bitmap_set(nodes_map, nid, 1);
> +
> +find_nearest_node:
> +       best_nid = NUMA_NO_NODE;
> +       distance = INT_MAX;
> +
> +       for_each_clear_bit(i, nodes_map, MAX_NUMNODES)
> +               if (numa_distance[nid][i] < distance) {
> +                       best_nid = i;
> +                       distance = numa_distance[nid][i];
> +               }
> +
> +       pa = memblock_alloc_nid(size, SMP_CACHE_BYTES, best_nid);
> +       if (!pa) {
> +               BUG_ON(best_nid == NUMA_NO_NODE);
> +               bitmap_set(nodes_map, best_nid, 1);
> +               goto find_nearest_node;
> +       }
> +
> +       return pa;
> +}
> +
>  /**
>   * Initialize NODE_DATA for a node on the local memory
>   */
> @@ -228,7 +265,9 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>         pr_info("Initmem setup node %d [mem %#010Lx-%#010Lx]\n",
>                 nid, start_pfn << PAGE_SHIFT, (end_pfn << PAGE_SHIFT) - 1);
>
> -       nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
> +       nd_pa = memblock_alloc_nid(nd_size, SMP_CACHE_BYTES, nid);
> +       if (!nd_pa)
> +               nd_pa = alloc_node_data_from_nearest_node(nid, nd_size);
>         nd = __va(nd_pa);
>
>         /* report and initialize */
> @@ -238,7 +277,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>         if (tnid != nid)
>                 pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);
>
> -       node_data[nid] = nd;
> +       NODE_DATA(nid) = nd;
>         memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
>         NODE_DATA(nid)->node_id = nid;
>         NODE_DATA(nid)->node_start_pfn = start_pfn;
> --
> 2.5.0
>
>
Ganapat
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v4 11/14] arm64/numa: support HAVE_MEMORYLESS_NODES

[Leizhen (ThunderTown)]

On 2016/6/7 16:31, Ganapatrao Kulkarni wrote:
> On Tue, Jun 7, 2016 at 1:38 PM, Zhen Lei <thunder.leizhen@huawei.com> wrote:
>> Some numa nodes may have no memory. For example:
>> 1. cpu0 on node0
>> 2. cpu1 on node1
>> 3. device0 access the momory from node0 and node1 take the same time.
> 
> i am wondering, if access to both nodes is same, then why you need numa.
> the example you are quoting is against the basic principle of "numa"
> what is device0 here? cpu?
The device0 can also be a cpu. I drew a simple diagram:

  cpu0     cpu1        cpu2/device0
    |        |              |
    |        |              |
   DDR0     DDR1    No DIMM slots or no DIMM plugged
 (node0)  (node1)         (node2)

>>
>> So, we can not simply classify device0 to node0 or node1, but we can
>> define a node2 which distances to node0 and node1 are the same.
>>
>> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
>> ---
>>  arch/arm64/Kconfig      |  4 ++++
>>  arch/arm64/kernel/smp.c |  1 +
>>  arch/arm64/mm/numa.c    | 43 +++++++++++++++++++++++++++++++++++++++++--
>>  3 files changed, 46 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>> index 05c1bf1..5904a62 100644
>> --- a/arch/arm64/Kconfig
>> +++ b/arch/arm64/Kconfig
>> @@ -581,6 +581,10 @@ config NEED_PER_CPU_EMBED_FIRST_CHUNK
>>         def_bool y
>>         depends on NUMA
>>
>> +config HAVE_MEMORYLESS_NODES
>> +       def_bool y
>> +       depends on NUMA
>> +
>>  source kernel/Kconfig.preempt
>>  source kernel/Kconfig.hz
>>
>> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
>> index d099306..9e15297 100644
>> --- a/arch/arm64/kernel/smp.c
>> +++ b/arch/arm64/kernel/smp.c
>> @@ -620,6 +620,7 @@ static void __init of_parse_and_init_cpus(void)
>>                         }
>>
>>                         bootcpu_valid = true;
>> +                       early_map_cpu_to_node(0, of_node_to_nid(dn));
>>
>>                         /*
>>                          * cpu_logical_map has already been
>> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
>> index df5c842..d73b0a0 100644
>> --- a/arch/arm64/mm/numa.c
>> +++ b/arch/arm64/mm/numa.c
>> @@ -128,6 +128,14 @@ void __init early_map_cpu_to_node(unsigned int cpu, int nid)
>>                 nid = 0;
>>
>>         cpu_to_node_map[cpu] = nid;
>> +
>> +       /*
>> +        * We should set the numa node of cpu0 as soon as possible, because it
>> +        * has already been set up online before. cpu_to_node(0) will soon be
>> +        * called.
>> +        */
>> +       if (!cpu)
>> +               set_cpu_numa_node(cpu, nid);
>>  }
>>
>>  #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
>> @@ -215,6 +223,35 @@ int __init numa_add_memblk(int nid, u64 start, u64 end)
>>         return ret;
>>  }
>>
>> +static u64 __init alloc_node_data_from_nearest_node(int nid, const size_t size)
>> +{
>> +       int i, best_nid, distance;
>> +       u64 pa;
>> +       DECLARE_BITMAP(nodes_map, MAX_NUMNODES);
>> +
>> +       bitmap_zero(nodes_map, MAX_NUMNODES);
>> +       bitmap_set(nodes_map, nid, 1);
>> +
>> +find_nearest_node:
>> +       best_nid = NUMA_NO_NODE;
>> +       distance = INT_MAX;
>> +
>> +       for_each_clear_bit(i, nodes_map, MAX_NUMNODES)
>> +               if (numa_distance[nid][i] < distance) {
>> +                       best_nid = i;
>> +                       distance = numa_distance[nid][i];
>> +               }
>> +
>> +       pa = memblock_alloc_nid(size, SMP_CACHE_BYTES, best_nid);
>> +       if (!pa) {
>> +               BUG_ON(best_nid == NUMA_NO_NODE);
>> +               bitmap_set(nodes_map, best_nid, 1);
>> +               goto find_nearest_node;
>> +       }
>> +
>> +       return pa;
>> +}
>> +
>>  /**
>>   * Initialize NODE_DATA for a node on the local memory
>>   */
>> @@ -228,7 +265,9 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>>         pr_info("Initmem setup node %d [mem %#010Lx-%#010Lx]\n",
>>                 nid, start_pfn << PAGE_SHIFT, (end_pfn << PAGE_SHIFT) - 1);
>>
>> -       nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
>> +       nd_pa = memblock_alloc_nid(nd_size, SMP_CACHE_BYTES, nid);
>> +       if (!nd_pa)
>> +               nd_pa = alloc_node_data_from_nearest_node(nid, nd_size);
>>         nd = __va(nd_pa);
>>
>>         /* report and initialize */
>> @@ -238,7 +277,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>>         if (tnid != nid)
>>                 pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);
>>
>> -       node_data[nid] = nd;
>> +       NODE_DATA(nid) = nd;
>>         memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
>>         NODE_DATA(nid)->node_id = nid;
>>         NODE_DATA(nid)->node_start_pfn = start_pfn;
>> --
>> 2.5.0
>>
>>
> Ganapat
>>
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 
> .
> 

Re: [PATCH v4 11/14] arm64/numa: support HAVE_MEMORYLESS_NODES

[Ganapatrao Kulkarni]

On Tue, Jun 7, 2016 at 6:27 PM, Leizhen (ThunderTown)
<thunder.leizhen@huawei.com> wrote:
>
>
> On 2016/6/7 16:31, Ganapatrao Kulkarni wrote:
>> On Tue, Jun 7, 2016 at 1:38 PM, Zhen Lei <thunder.leizhen@huawei.com> wrote:
>>> Some numa nodes may have no memory. For example:
>>> 1. cpu0 on node0
>>> 2. cpu1 on node1
>>> 3. device0 access the momory from node0 and node1 take the same time.
>>
>> i am wondering, if access to both nodes is same, then why you need numa.
>> the example you are quoting is against the basic principle of "numa"
>> what is device0 here? cpu?
> The device0 can also be a cpu. I drew a simple diagram:
>
>   cpu0     cpu1        cpu2/device0
>     |        |              |
>     |        |              |
>    DDR0     DDR1    No DIMM slots or no DIMM plugged
>  (node0)  (node1)         (node2)
>

thanks for the clarification. your example is for 3 node system, where
third node is memory less node.
do you see any issue in supporting this topology with existing code?
I think, this use case should be supported with present code.

>>>
>>> So, we can not simply classify device0 to node0 or node1, but we can
>>> define a node2 which distances to node0 and node1 are the same.
>>>
>>> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
>>> ---
>>>  arch/arm64/Kconfig      |  4 ++++
>>>  arch/arm64/kernel/smp.c |  1 +
>>>  arch/arm64/mm/numa.c    | 43 +++++++++++++++++++++++++++++++++++++++++--
>>>  3 files changed, 46 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>>> index 05c1bf1..5904a62 100644
>>> --- a/arch/arm64/Kconfig
>>> +++ b/arch/arm64/Kconfig
>>> @@ -581,6 +581,10 @@ config NEED_PER_CPU_EMBED_FIRST_CHUNK
>>>         def_bool y
>>>         depends on NUMA
>>>
>>> +config HAVE_MEMORYLESS_NODES
>>> +       def_bool y
>>> +       depends on NUMA
>>> +
>>>  source kernel/Kconfig.preempt
>>>  source kernel/Kconfig.hz
>>>
>>> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
>>> index d099306..9e15297 100644
>>> --- a/arch/arm64/kernel/smp.c
>>> +++ b/arch/arm64/kernel/smp.c
>>> @@ -620,6 +620,7 @@ static void __init of_parse_and_init_cpus(void)
>>>                         }
>>>
>>>                         bootcpu_valid = true;
>>> +                       early_map_cpu_to_node(0, of_node_to_nid(dn));
>>>
>>>                         /*
>>>                          * cpu_logical_map has already been
>>> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
>>> index df5c842..d73b0a0 100644
>>> --- a/arch/arm64/mm/numa.c
>>> +++ b/arch/arm64/mm/numa.c
>>> @@ -128,6 +128,14 @@ void __init early_map_cpu_to_node(unsigned int cpu, int nid)
>>>                 nid = 0;
>>>
>>>         cpu_to_node_map[cpu] = nid;
>>> +
>>> +       /*
>>> +        * We should set the numa node of cpu0 as soon as possible, because it
>>> +        * has already been set up online before. cpu_to_node(0) will soon be
>>> +        * called.
>>> +        */
>>> +       if (!cpu)
>>> +               set_cpu_numa_node(cpu, nid);
>>>  }
>>>
>>>  #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
>>> @@ -215,6 +223,35 @@ int __init numa_add_memblk(int nid, u64 start, u64 end)
>>>         return ret;
>>>  }
>>>
>>> +static u64 __init alloc_node_data_from_nearest_node(int nid, const size_t size)
>>> +{
>>> +       int i, best_nid, distance;
>>> +       u64 pa;
>>> +       DECLARE_BITMAP(nodes_map, MAX_NUMNODES);
>>> +
>>> +       bitmap_zero(nodes_map, MAX_NUMNODES);
>>> +       bitmap_set(nodes_map, nid, 1);
>>> +
>>> +find_nearest_node:
>>> +       best_nid = NUMA_NO_NODE;
>>> +       distance = INT_MAX;
>>> +
>>> +       for_each_clear_bit(i, nodes_map, MAX_NUMNODES)
>>> +               if (numa_distance[nid][i] < distance) {
>>> +                       best_nid = i;
>>> +                       distance = numa_distance[nid][i];
>>> +               }
>>> +
>>> +       pa = memblock_alloc_nid(size, SMP_CACHE_BYTES, best_nid);
>>> +       if (!pa) {
>>> +               BUG_ON(best_nid == NUMA_NO_NODE);
>>> +               bitmap_set(nodes_map, best_nid, 1);
>>> +               goto find_nearest_node;
>>> +       }
>>> +
>>> +       return pa;
>>> +}
>>> +

why do we need this function in arch specific code.
dont you think common code will take care of this? when you define
HAVE_MEMORYLESS_NODES

>>>  /**
>>>   * Initialize NODE_DATA for a node on the local memory
>>>   */
>>> @@ -228,7 +265,9 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>>>         pr_info("Initmem setup node %d [mem %#010Lx-%#010Lx]\n",
>>>                 nid, start_pfn << PAGE_SHIFT, (end_pfn << PAGE_SHIFT) - 1);
>>>
>>> -       nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);

this function try to allocate from a nid, if fails, it allocates from
node 0(local node).
this is ok for memory less node i guess.

>>> +       nd_pa = memblock_alloc_nid(nd_size, SMP_CACHE_BYTES, nid);
>>> +       if (!nd_pa)
>>> +               nd_pa = alloc_node_data_from_nearest_node(nid, nd_size);
>>>         nd = __va(nd_pa);
>>>
>>>         /* report and initialize */
>>> @@ -238,7 +277,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>>>         if (tnid != nid)
>>>                 pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);
>>>
>>> -       node_data[nid] = nd;
>>> +       NODE_DATA(nid) = nd;
>>>         memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
>>>         NODE_DATA(nid)->node_id = nid;
>>>         NODE_DATA(nid)->node_start_pfn = start_pfn;
>>> --
>>> 2.5.0
>>>
>>>
>> Ganapat
>>>
>>> _______________________________________________
>>> linux-arm-kernel mailing list
>>> linux-arm-kernel@lists.infradead.org
>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>
>> .
>>
>

Re: [PATCH v4 11/14] arm64/numa: support HAVE_MEMORYLESS_NODES

[Leizhen (ThunderTown)]

On 2016/6/7 22:01, Ganapatrao Kulkarni wrote:
> On Tue, Jun 7, 2016 at 6:27 PM, Leizhen (ThunderTown)
> <thunder.leizhen@huawei.com> wrote:
>>
>>
>> On 2016/6/7 16:31, Ganapatrao Kulkarni wrote:
>>> On Tue, Jun 7, 2016 at 1:38 PM, Zhen Lei <thunder.leizhen@huawei.com> wrote:
>>>> Some numa nodes may have no memory. For example:
>>>> 1. cpu0 on node0
>>>> 2. cpu1 on node1
>>>> 3. device0 access the momory from node0 and node1 take the same time.
>>>
>>> i am wondering, if access to both nodes is same, then why you need numa.
>>> the example you are quoting is against the basic principle of "numa"
>>> what is device0 here? cpu?
>> The device0 can also be a cpu. I drew a simple diagram:
>>
>>   cpu0     cpu1        cpu2/device0
>>     |        |              |
>>     |        |              |
>>    DDR0     DDR1    No DIMM slots or no DIMM plugged
>>  (node0)  (node1)         (node2)
>>
> 
> thanks for the clarification. your example is for 3 node system, where
> third node is memory less node.
> do you see any issue in supporting this topology with existing code?
If opened HAVE_MEMORYLESS_NODES, it will pick the nearest node for the cpus on
memoryless node.

For example, in include/linux/topology.h
#ifdef CONFIG_HAVE_MEMORYLESS_NODES
...
static inline int cpu_to_mem(int cpu)
{
	return per_cpu(_numa_mem_, cpu);
}
...
#else
...
static inline int cpu_to_mem(int cpu)
{
	return cpu_to_node(cpu);
}
...
#endif

> I think, this use case should be supported with present code.
> 
>>>>
>>>> So, we can not simply classify device0 to node0 or node1, but we can
>>>> define a node2 which distances to node0 and node1 are the same.
>>>>
>>>> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
>>>> ---
>>>>  arch/arm64/Kconfig      |  4 ++++
>>>>  arch/arm64/kernel/smp.c |  1 +
>>>>  arch/arm64/mm/numa.c    | 43 +++++++++++++++++++++++++++++++++++++++++--
>>>>  3 files changed, 46 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>>>> index 05c1bf1..5904a62 100644
>>>> --- a/arch/arm64/Kconfig
>>>> +++ b/arch/arm64/Kconfig
>>>> @@ -581,6 +581,10 @@ config NEED_PER_CPU_EMBED_FIRST_CHUNK
>>>>         def_bool y
>>>>         depends on NUMA
>>>>
>>>> +config HAVE_MEMORYLESS_NODES
>>>> +       def_bool y
>>>> +       depends on NUMA
>>>> +
>>>>  source kernel/Kconfig.preempt
>>>>  source kernel/Kconfig.hz
>>>>
>>>> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
>>>> index d099306..9e15297 100644
>>>> --- a/arch/arm64/kernel/smp.c
>>>> +++ b/arch/arm64/kernel/smp.c
>>>> @@ -620,6 +620,7 @@ static void __init of_parse_and_init_cpus(void)
>>>>                         }
>>>>
>>>>                         bootcpu_valid = true;
>>>> +                       early_map_cpu_to_node(0, of_node_to_nid(dn));
>>>>
>>>>                         /*
>>>>                          * cpu_logical_map has already been
>>>> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
>>>> index df5c842..d73b0a0 100644
>>>> --- a/arch/arm64/mm/numa.c
>>>> +++ b/arch/arm64/mm/numa.c
>>>> @@ -128,6 +128,14 @@ void __init early_map_cpu_to_node(unsigned int cpu, int nid)
>>>>                 nid = 0;
>>>>
>>>>         cpu_to_node_map[cpu] = nid;
>>>> +
>>>> +       /*
>>>> +        * We should set the numa node of cpu0 as soon as possible, because it
>>>> +        * has already been set up online before. cpu_to_node(0) will soon be
>>>> +        * called.
>>>> +        */
>>>> +       if (!cpu)
>>>> +               set_cpu_numa_node(cpu, nid);
>>>>  }
>>>>
>>>>  #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
>>>> @@ -215,6 +223,35 @@ int __init numa_add_memblk(int nid, u64 start, u64 end)
>>>>         return ret;
>>>>  }
>>>>
>>>> +static u64 __init alloc_node_data_from_nearest_node(int nid, const size_t size)
>>>> +{
>>>> +       int i, best_nid, distance;
>>>> +       u64 pa;
>>>> +       DECLARE_BITMAP(nodes_map, MAX_NUMNODES);
>>>> +
>>>> +       bitmap_zero(nodes_map, MAX_NUMNODES);
>>>> +       bitmap_set(nodes_map, nid, 1);
>>>> +
>>>> +find_nearest_node:
>>>> +       best_nid = NUMA_NO_NODE;
>>>> +       distance = INT_MAX;
>>>> +
>>>> +       for_each_clear_bit(i, nodes_map, MAX_NUMNODES)
>>>> +               if (numa_distance[nid][i] < distance) {
>>>> +                       best_nid = i;
>>>> +                       distance = numa_distance[nid][i];
>>>> +               }
>>>> +
>>>> +       pa = memblock_alloc_nid(size, SMP_CACHE_BYTES, best_nid);
>>>> +       if (!pa) {
>>>> +               BUG_ON(best_nid == NUMA_NO_NODE);
>>>> +               bitmap_set(nodes_map, best_nid, 1);
>>>> +               goto find_nearest_node;
>>>> +       }
>>>> +
>>>> +       return pa;
>>>> +}
>>>> +
> 
> why do we need this function in arch specific code.
I also considered put these code(include HAVE_SETUP_PER_CPU_AREA) into drivers/of/of_numa.c,
but if I do that, it will make acpi numa dependent on of numa.

> dont you think common code will take care of this? when you define
> HAVE_MEMORYLESS_NODES

I have searched CONFIG_HAVE_MEMORYLESS_NODES in *.c, but did not find the relevant content.
So maybe other ARCHs also missed this.

> 
>>>>  /**
>>>>   * Initialize NODE_DATA for a node on the local memory
>>>>   */
>>>> @@ -228,7 +265,9 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>>>>         pr_info("Initmem setup node %d [mem %#010Lx-%#010Lx]\n",
>>>>                 nid, start_pfn << PAGE_SHIFT, (end_pfn << PAGE_SHIFT) - 1);
>>>>
>>>> -       nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
> 
> this function try to allocate from a nid, if fails, it allocates from
> node 0(local node).
> this is ok for memory less node i guess.
Yes, the function is OK, but the performance is not.

Suppose there are 3 nodes:
1. cpu0 on node0, cpu1 on node1, cpu2 on node2.
2. cpu2 access the memory on node1 take 1us, but access the memory on node1 take 5us.
   That is, distance[2,1] is shorter than distance[2,0].
3. And node2 is a memoryless node.

So if NODE_DATA(2) allocated from node0, it will take more time than allocted from node1 at run time.
Because NODE_DATA will be accessed at run time.

> 
>>>> +       nd_pa = memblock_alloc_nid(nd_size, SMP_CACHE_BYTES, nid);
>>>> +       if (!nd_pa)
>>>> +               nd_pa = alloc_node_data_from_nearest_node(nid, nd_size);
>>>>         nd = __va(nd_pa);
>>>>
>>>>         /* report and initialize */
>>>> @@ -238,7 +277,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>>>>         if (tnid != nid)
>>>>                 pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);
>>>>
>>>> -       node_data[nid] = nd;
>>>> +       NODE_DATA(nid) = nd;
>>>>         memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
>>>>         NODE_DATA(nid)->node_id = nid;
>>>>         NODE_DATA(nid)->node_start_pfn = start_pfn;
>>>> --
>>>> 2.5.0
>>>>
>>>>
>>> Ganapat
>>>>
>>>> _______________________________________________
>>>> linux-arm-kernel mailing list
>>>> linux-arm-kernel@lists.infradead.org
>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>
>>> .
>>>
>>
> 
> .
> 

Re: [PATCH v4 11/14] arm64/numa: support HAVE_MEMORYLESS_NODES

[Ganapatrao Kulkarni]

On Wed, Jun 8, 2016 at 7:46 AM, Leizhen (ThunderTown)
<thunder.leizhen@huawei.com> wrote:
>
>
> On 2016/6/7 22:01, Ganapatrao Kulkarni wrote:
>> On Tue, Jun 7, 2016 at 6:27 PM, Leizhen (ThunderTown)
>> <thunder.leizhen@huawei.com> wrote:
>>>
>>>
>>> On 2016/6/7 16:31, Ganapatrao Kulkarni wrote:
>>>> On Tue, Jun 7, 2016 at 1:38 PM, Zhen Lei <thunder.leizhen@huawei.com> wrote:
>>>>> Some numa nodes may have no memory. For example:
>>>>> 1. cpu0 on node0
>>>>> 2. cpu1 on node1
>>>>> 3. device0 access the momory from node0 and node1 take the same time.
>>>>
>>>> i am wondering, if access to both nodes is same, then why you need numa.
>>>> the example you are quoting is against the basic principle of "numa"
>>>> what is device0 here? cpu?
>>> The device0 can also be a cpu. I drew a simple diagram:
>>>
>>>   cpu0     cpu1        cpu2/device0
>>>     |        |              |
>>>     |        |              |
>>>    DDR0     DDR1    No DIMM slots or no DIMM plugged
>>>  (node0)  (node1)         (node2)
>>>
>>
>> thanks for the clarification. your example is for 3 node system, where
>> third node is memory less node.
>> do you see any issue in supporting this topology with existing code?
> If opened HAVE_MEMORYLESS_NODES, it will pick the nearest node for the cpus on
> memoryless node.

i see couple of arch enabled HAVE_MEMORYLESS_NODES, but i don't see
any code in arch specific numa code for this.
is that means the core code will take care of this?

>
> For example, in include/linux/topology.h
> #ifdef CONFIG_HAVE_MEMORYLESS_NODES
> ...
> static inline int cpu_to_mem(int cpu)
> {
>         return per_cpu(_numa_mem_, cpu);
> }
> ...
> #else
> ...
> static inline int cpu_to_mem(int cpu)
> {
>         return cpu_to_node(cpu);
> }
> ...
> #endif
>
>> I think, this use case should be supported with present code.
>>
>>>>>
>>>>> So, we can not simply classify device0 to node0 or node1, but we can
>>>>> define a node2 which distances to node0 and node1 are the same.
>>>>>
>>>>> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
>>>>> ---
>>>>>  arch/arm64/Kconfig      |  4 ++++
>>>>>  arch/arm64/kernel/smp.c |  1 +
>>>>>  arch/arm64/mm/numa.c    | 43 +++++++++++++++++++++++++++++++++++++++++--
>>>>>  3 files changed, 46 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>>>>> index 05c1bf1..5904a62 100644
>>>>> --- a/arch/arm64/Kconfig
>>>>> +++ b/arch/arm64/Kconfig
>>>>> @@ -581,6 +581,10 @@ config NEED_PER_CPU_EMBED_FIRST_CHUNK
>>>>>         def_bool y
>>>>>         depends on NUMA
>>>>>
>>>>> +config HAVE_MEMORYLESS_NODES
>>>>> +       def_bool y
>>>>> +       depends on NUMA
>>>>> +
>>>>>  source kernel/Kconfig.preempt
>>>>>  source kernel/Kconfig.hz
>>>>>
>>>>> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
>>>>> index d099306..9e15297 100644
>>>>> --- a/arch/arm64/kernel/smp.c
>>>>> +++ b/arch/arm64/kernel/smp.c
>>>>> @@ -620,6 +620,7 @@ static void __init of_parse_and_init_cpus(void)
>>>>>                         }
>>>>>
>>>>>                         bootcpu_valid = true;
>>>>> +                       early_map_cpu_to_node(0, of_node_to_nid(dn));
>>>>>
>>>>>                         /*
>>>>>                          * cpu_logical_map has already been
>>>>> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
>>>>> index df5c842..d73b0a0 100644
>>>>> --- a/arch/arm64/mm/numa.c
>>>>> +++ b/arch/arm64/mm/numa.c
>>>>> @@ -128,6 +128,14 @@ void __init early_map_cpu_to_node(unsigned int cpu, int nid)
>>>>>                 nid = 0;
>>>>>
>>>>>         cpu_to_node_map[cpu] = nid;
>>>>> +
>>>>> +       /*
>>>>> +        * We should set the numa node of cpu0 as soon as possible, because it
>>>>> +        * has already been set up online before. cpu_to_node(0) will soon be
>>>>> +        * called.
>>>>> +        */
>>>>> +       if (!cpu)
>>>>> +               set_cpu_numa_node(cpu, nid);
>>>>>  }
>>>>>
>>>>>  #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
>>>>> @@ -215,6 +223,35 @@ int __init numa_add_memblk(int nid, u64 start, u64 end)
>>>>>         return ret;
>>>>>  }
>>>>>
>>>>> +static u64 __init alloc_node_data_from_nearest_node(int nid, const size_t size)
>>>>> +{
>>>>> +       int i, best_nid, distance;
>>>>> +       u64 pa;
>>>>> +       DECLARE_BITMAP(nodes_map, MAX_NUMNODES);
>>>>> +
>>>>> +       bitmap_zero(nodes_map, MAX_NUMNODES);
>>>>> +       bitmap_set(nodes_map, nid, 1);
>>>>> +
>>>>> +find_nearest_node:
>>>>> +       best_nid = NUMA_NO_NODE;
>>>>> +       distance = INT_MAX;
>>>>> +
>>>>> +       for_each_clear_bit(i, nodes_map, MAX_NUMNODES)
>>>>> +               if (numa_distance[nid][i] < distance) {
>>>>> +                       best_nid = i;
>>>>> +                       distance = numa_distance[nid][i];
>>>>> +               }
>>>>> +
>>>>> +       pa = memblock_alloc_nid(size, SMP_CACHE_BYTES, best_nid);
>>>>> +       if (!pa) {
>>>>> +               BUG_ON(best_nid == NUMA_NO_NODE);
>>>>> +               bitmap_set(nodes_map, best_nid, 1);
>>>>> +               goto find_nearest_node;
>>>>> +       }
>>>>> +
>>>>> +       return pa;
>>>>> +}
>>>>> +
>>
>> why do we need this function in arch specific code.
> I also considered put these code(include HAVE_SETUP_PER_CPU_AREA) into drivers/of/of_numa.c,
> but if I do that, it will make acpi numa dependent on of numa.

numa core/common code is mainly in directory mm/
drivers/of/of_numa.c implements only device tree numa binding.

>
>> dont you think common code will take care of this? when you define
>> HAVE_MEMORYLESS_NODES
>
> I have searched CONFIG_HAVE_MEMORYLESS_NODES in *.c, but did not find the relevant content.
> So maybe other ARCHs also missed this.

as mentioned above, arch code may not need any changes for this.
>
>>
>>>>>  /**
>>>>>   * Initialize NODE_DATA for a node on the local memory
>>>>>   */
>>>>> @@ -228,7 +265,9 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>>>>>         pr_info("Initmem setup node %d [mem %#010Lx-%#010Lx]\n",
>>>>>                 nid, start_pfn << PAGE_SHIFT, (end_pfn << PAGE_SHIFT) - 1);
>>>>>
>>>>> -       nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
>>
>> this function try to allocate from a nid, if fails, it allocates from
>> node 0(local node).
>> this is ok for memory less node i guess.
> Yes, the function is OK, but the performance is not.
>
> Suppose there are 3 nodes:
> 1. cpu0 on node0, cpu1 on node1, cpu2 on node2.
> 2. cpu2 access the memory on node1 take 1us, but access the memory on node1 take 5us.
>    That is, distance[2,1] is shorter than distance[2,0].
> 3. And node2 is a memoryless node.
>
> So if NODE_DATA(2) allocated from node0, it will take more time than allocted from node1 at run time.
> Because NODE_DATA will be accessed at run time.
>
>>
>>>>> +       nd_pa = memblock_alloc_nid(nd_size, SMP_CACHE_BYTES, nid);
>>>>> +       if (!nd_pa)
>>>>> +               nd_pa = alloc_node_data_from_nearest_node(nid, nd_size);
>>>>>         nd = __va(nd_pa);
>>>>>
>>>>>         /* report and initialize */
>>>>> @@ -238,7 +277,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>>>>>         if (tnid != nid)
>>>>>                 pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);
>>>>>
>>>>> -       node_data[nid] = nd;
>>>>> +       NODE_DATA(nid) = nd;
>>>>>         memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
>>>>>         NODE_DATA(nid)->node_id = nid;
>>>>>         NODE_DATA(nid)->node_start_pfn = start_pfn;
>>>>> --
>>>>> 2.5.0
>>>>>
>>>>>
>>>> Ganapat
>>>>>
>>>>> _______________________________________________
>>>>> linux-arm-kernel mailing list
>>>>> linux-arm-kernel@lists.infradead.org
>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>>
>>>> .
>>>>
>>>
>>
>> .
>>
>

thanks
Ganapat

Re: [PATCH v4 11/14] arm64/numa: support HAVE_MEMORYLESS_NODES

[Leizhen (ThunderTown)]

On 2016/6/8 12:45, Ganapatrao Kulkarni wrote:
> On Wed, Jun 8, 2016 at 7:46 AM, Leizhen (ThunderTown)
> <thunder.leizhen@huawei.com> wrote:
>>
>>
>> On 2016/6/7 22:01, Ganapatrao Kulkarni wrote:
>>> On Tue, Jun 7, 2016 at 6:27 PM, Leizhen (ThunderTown)
>>> <thunder.leizhen@huawei.com> wrote:
>>>>
>>>>
>>>> On 2016/6/7 16:31, Ganapatrao Kulkarni wrote:
>>>>> On Tue, Jun 7, 2016 at 1:38 PM, Zhen Lei <thunder.leizhen@huawei.com> wrote:
>>>>>> Some numa nodes may have no memory. For example:
>>>>>> 1. cpu0 on node0
>>>>>> 2. cpu1 on node1
>>>>>> 3. device0 access the momory from node0 and node1 take the same time.
>>>>>
>>>>> i am wondering, if access to both nodes is same, then why you need numa.
>>>>> the example you are quoting is against the basic principle of "numa"
>>>>> what is device0 here? cpu?
>>>> The device0 can also be a cpu. I drew a simple diagram:
>>>>
>>>>   cpu0     cpu1        cpu2/device0
>>>>     |        |              |
>>>>     |        |              |
>>>>    DDR0     DDR1    No DIMM slots or no DIMM plugged
>>>>  (node0)  (node1)         (node2)
>>>>
>>>
>>> thanks for the clarification. your example is for 3 node system, where
>>> third node is memory less node.
>>> do you see any issue in supporting this topology with existing code?
>> If opened HAVE_MEMORYLESS_NODES, it will pick the nearest node for the cpus on
>> memoryless node.
> 
> i see couple of arch enabled HAVE_MEMORYLESS_NODES, but i don't see
> any code in arch specific numa code for this
> is that means the core code will take care of this?
I just spent some time to read the implementation code of HAVE_MEMORYLESS_NODES on PPC and IA64.
For NODE_DATA initialization, it's similar to mine on IA64. But PPC have no special process, it's
similar to yours. I think the developers of PPC need to fix it.

I picked the code on IA64 as below:
static void __init *memory_less_node_alloc(int nid, unsigned long pernodesize)
{
        void *ptr = NULL;
        u8 best = 0xff;
        int bestnode = -1, node, anynode = 0;

        for_each_online_node(node) {
                if (node_isset(node, memory_less_mask))
                        continue;
                else if (node_distance(nid, node) < best) {
                        best = node_distance(nid, node);
                        bestnode = node;
                }
                anynode = node;
        }

        if (bestnode == -1)
                bestnode = anynode;

        ptr = __alloc_bootmem_node(pgdat_list[bestnode], pernodesize,
                PERCPU_PAGE_SIZE, __pa(MAX_DMA_ADDRESS));

        return ptr;
}

/**
 * memory_less_nodes - allocate and initialize CPU only nodes pernode
 *      information.
 */
static void __init memory_less_nodes(void)
{
        unsigned long pernodesize;
        void *pernode;
        int node;

        for_each_node_mask(node, memory_less_mask) {
                pernodesize = compute_pernodesize(node);
                pernode = memory_less_node_alloc(node, pernodesize);
                fill_pernode(node, __pa(pernode), pernodesize);
        }

        return;
}



> 
>>
>> For example, in include/linux/topology.h
>> #ifdef CONFIG_HAVE_MEMORYLESS_NODES
>> ...
>> static inline int cpu_to_mem(int cpu)
>> {
>>         return per_cpu(_numa_mem_, cpu);
>> }
>> ...
>> #else
>> ...
>> static inline int cpu_to_mem(int cpu)
>> {
>>         return cpu_to_node(cpu);
>> }
>> ...
>> #endif
>>
>>> I think, this use case should be supported with present code.
>>>
>>>>>>
>>>>>> So, we can not simply classify device0 to node0 or node1, but we can
>>>>>> define a node2 which distances to node0 and node1 are the same.
>>>>>>
>>>>>> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
>>>>>> ---
>>>>>>  arch/arm64/Kconfig      |  4 ++++
>>>>>>  arch/arm64/kernel/smp.c |  1 +
>>>>>>  arch/arm64/mm/numa.c    | 43 +++++++++++++++++++++++++++++++++++++++++--
>>>>>>  3 files changed, 46 insertions(+), 2 deletions(-)
>>>>>>
>>>>>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>>>>>> index 05c1bf1..5904a62 100644
>>>>>> --- a/arch/arm64/Kconfig
>>>>>> +++ b/arch/arm64/Kconfig
>>>>>> @@ -581,6 +581,10 @@ config NEED_PER_CPU_EMBED_FIRST_CHUNK
>>>>>>         def_bool y
>>>>>>         depends on NUMA
>>>>>>
>>>>>> +config HAVE_MEMORYLESS_NODES
>>>>>> +       def_bool y
>>>>>> +       depends on NUMA
>>>>>> +
>>>>>>  source kernel/Kconfig.preempt
>>>>>>  source kernel/Kconfig.hz
>>>>>>
>>>>>> diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
>>>>>> index d099306..9e15297 100644
>>>>>> --- a/arch/arm64/kernel/smp.c
>>>>>> +++ b/arch/arm64/kernel/smp.c
>>>>>> @@ -620,6 +620,7 @@ static void __init of_parse_and_init_cpus(void)
>>>>>>                         }
>>>>>>
>>>>>>                         bootcpu_valid = true;
>>>>>> +                       early_map_cpu_to_node(0, of_node_to_nid(dn));
>>>>>>
>>>>>>                         /*
>>>>>>                          * cpu_logical_map has already been
>>>>>> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
>>>>>> index df5c842..d73b0a0 100644
>>>>>> --- a/arch/arm64/mm/numa.c
>>>>>> +++ b/arch/arm64/mm/numa.c
>>>>>> @@ -128,6 +128,14 @@ void __init early_map_cpu_to_node(unsigned int cpu, int nid)
>>>>>>                 nid = 0;
>>>>>>
>>>>>>         cpu_to_node_map[cpu] = nid;
>>>>>> +
>>>>>> +       /*
>>>>>> +        * We should set the numa node of cpu0 as soon as possible, because it
>>>>>> +        * has already been set up online before. cpu_to_node(0) will soon be
>>>>>> +        * called.
>>>>>> +        */
>>>>>> +       if (!cpu)
>>>>>> +               set_cpu_numa_node(cpu, nid);
>>>>>>  }
>>>>>>
>>>>>>  #ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
>>>>>> @@ -215,6 +223,35 @@ int __init numa_add_memblk(int nid, u64 start, u64 end)
>>>>>>         return ret;
>>>>>>  }
>>>>>>
>>>>>> +static u64 __init alloc_node_data_from_nearest_node(int nid, const size_t size)
>>>>>> +{
>>>>>> +       int i, best_nid, distance;
>>>>>> +       u64 pa;
>>>>>> +       DECLARE_BITMAP(nodes_map, MAX_NUMNODES);
>>>>>> +
>>>>>> +       bitmap_zero(nodes_map, MAX_NUMNODES);
>>>>>> +       bitmap_set(nodes_map, nid, 1);
>>>>>> +
>>>>>> +find_nearest_node:
>>>>>> +       best_nid = NUMA_NO_NODE;
>>>>>> +       distance = INT_MAX;
>>>>>> +
>>>>>> +       for_each_clear_bit(i, nodes_map, MAX_NUMNODES)
>>>>>> +               if (numa_distance[nid][i] < distance) {
>>>>>> +                       best_nid = i;
>>>>>> +                       distance = numa_distance[nid][i];
>>>>>> +               }
>>>>>> +
>>>>>> +       pa = memblock_alloc_nid(size, SMP_CACHE_BYTES, best_nid);
>>>>>> +       if (!pa) {
>>>>>> +               BUG_ON(best_nid == NUMA_NO_NODE);
>>>>>> +               bitmap_set(nodes_map, best_nid, 1);
>>>>>> +               goto find_nearest_node;
>>>>>> +       }
>>>>>> +
>>>>>> +       return pa;
>>>>>> +}
>>>>>> +
>>>
>>> why do we need this function in arch specific code.
>> I also considered put these code(include HAVE_SETUP_PER_CPU_AREA) into drivers/of/of_numa.c,
>> but if I do that, it will make acpi numa dependent on of numa.
> 
> numa core/common code is mainly in directory mm/
> drivers/of/of_numa.c implements only device tree numa binding.

As above, IA64 also have a similar implementation under arch/ia64 directory.
And it seems the implementation of IA64 and mine cann't be merged into one.
So I suggest that currently stay these code here.

> 
>>
>>> dont you think common code will take care of this? when you define
>>> HAVE_MEMORYLESS_NODES
>>
>> I have searched CONFIG_HAVE_MEMORYLESS_NODES in *.c, but did not find the relevant content.
>> So maybe other ARCHs also missed this.
> 
> as mentioned above, arch code may not need any changes for this.
>>
>>>
>>>>>>  /**
>>>>>>   * Initialize NODE_DATA for a node on the local memory
>>>>>>   */
>>>>>> @@ -228,7 +265,9 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>>>>>>         pr_info("Initmem setup node %d [mem %#010Lx-%#010Lx]\n",
>>>>>>                 nid, start_pfn << PAGE_SHIFT, (end_pfn << PAGE_SHIFT) - 1);
>>>>>>
>>>>>> -       nd_pa = memblock_alloc_try_nid(nd_size, SMP_CACHE_BYTES, nid);
>>>
>>> this function try to allocate from a nid, if fails, it allocates from
>>> node 0(local node).
>>> this is ok for memory less node i guess.
>> Yes, the function is OK, but the performance is not.
>>
>> Suppose there are 3 nodes:
>> 1. cpu0 on node0, cpu1 on node1, cpu2 on node2.
>> 2. cpu2 access the memory on node1 take 1us, but access the memory on node1 take 5us.
>>    That is, distance[2,1] is shorter than distance[2,0].
>> 3. And node2 is a memoryless node.
>>
>> So if NODE_DATA(2) allocated from node0, it will take more time than allocted from node1 at run time.
>> Because NODE_DATA will be accessed at run time.
>>
>>>
>>>>>> +       nd_pa = memblock_alloc_nid(nd_size, SMP_CACHE_BYTES, nid);
>>>>>> +       if (!nd_pa)
>>>>>> +               nd_pa = alloc_node_data_from_nearest_node(nid, nd_size);
>>>>>>         nd = __va(nd_pa);
>>>>>>
>>>>>>         /* report and initialize */
>>>>>> @@ -238,7 +277,7 @@ static void __init setup_node_data(int nid, u64 start_pfn, u64 end_pfn)
>>>>>>         if (tnid != nid)
>>>>>>                 pr_info("    NODE_DATA(%d) on node %d\n", nid, tnid);
>>>>>>
>>>>>> -       node_data[nid] = nd;
>>>>>> +       NODE_DATA(nid) = nd;
>>>>>>         memset(NODE_DATA(nid), 0, sizeof(pg_data_t));
>>>>>>         NODE_DATA(nid)->node_id = nid;
>>>>>>         NODE_DATA(nid)->node_start_pfn = start_pfn;
>>>>>> --
>>>>>> 2.5.0
>>>>>>
>>>>>>
>>>>> Ganapat
>>>>>>
>>>>>> _______________________________________________
>>>>>> linux-arm-kernel mailing list
>>>>>> linux-arm-kernel@lists.infradead.org
>>>>>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
>>>>>
>>>>> .
>>>>>
>>>>
>>>
>>> .
>>>
>>
> 
> thanks
> Ganapat
> 
> .
> 

[PATCH v4 12/14] arm64/numa: remove some useless code

[Zhen Lei]

1. Currently only cpu0 set on cpu_possible_mask and percpu areas have not
   been initialized.
2. No reason to limit cpu0 must belongs to node0.

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
 arch/arm64/mm/numa.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
index d73b0a0..92b1692 100644
--- a/arch/arm64/mm/numa.c
+++ b/arch/arm64/mm/numa.c
@@ -93,7 +93,6 @@ void numa_clear_node(unsigned int cpu)
  */
 static void __init setup_node_to_cpumask_map(void)
 {
-	unsigned int cpu;
 	int node;

 	/* setup nr_node_ids if not done yet */
@@ -106,9 +105,6 @@ static void __init setup_node_to_cpumask_map(void)
 		cpumask_clear(node_to_cpumask_map[node]);
 	}

-	for_each_possible_cpu(cpu)
-		set_cpu_numa_node(cpu, NUMA_NO_NODE);
-
 	/* cpumask_of_node() will now work */
 	pr_debug("Node to cpumask map for %d nodes\n", nr_node_ids);
 }
@@ -379,10 +375,6 @@ static int __init numa_init(int (*init_func)(void))

 	setup_node_to_cpumask_map();

-	/* init boot processor */
-	cpu_to_node_map[0] = 0;
-	map_cpu_to_node(0, 0);
-
 	return 0;
 }

--
2.5.0

Re: [PATCH v4 12/14] arm64/numa: remove some useless code

[Ganapatrao Kulkarni]

On Tue, Jun 7, 2016 at 1:38 PM, Zhen Lei <thunder.leizhen@huawei.com> wrote:
> 1. Currently only cpu0 set on cpu_possible_mask and percpu areas have not
>    been initialized.
> 2. No reason to limit cpu0 must belongs to node0.

even smp init assumes cpu0/boot processor.
is this patch tested on any hardware?
can you describe your testing hardware?
>
> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
> ---
>  arch/arm64/mm/numa.c | 8 --------
>  1 file changed, 8 deletions(-)
>
> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
> index d73b0a0..92b1692 100644
> --- a/arch/arm64/mm/numa.c
> +++ b/arch/arm64/mm/numa.c
> @@ -93,7 +93,6 @@ void numa_clear_node(unsigned int cpu)
>   */
>  static void __init setup_node_to_cpumask_map(void)
>  {
> -       unsigned int cpu;
>         int node;
>
>         /* setup nr_node_ids if not done yet */
> @@ -106,9 +105,6 @@ static void __init setup_node_to_cpumask_map(void)
>                 cpumask_clear(node_to_cpumask_map[node]);
>         }
>
> -       for_each_possible_cpu(cpu)
> -               set_cpu_numa_node(cpu, NUMA_NO_NODE);
> -

do you see this init of setting node id to NUMA_NO_NODE  for each cpu
happening any where else?
otherwise, better to have initialised node id/NUMA_NO_NODE to every
cpu otherwise default  node id will be shown as zero
which is not correct.

>         /* cpumask_of_node() will now work */
>         pr_debug("Node to cpumask map for %d nodes\n", nr_node_ids);
>  }
> @@ -379,10 +375,6 @@ static int __init numa_init(int (*init_func)(void))
>
>         setup_node_to_cpumask_map();
>
> -       /* init boot processor */
> -       cpu_to_node_map[0] = 0;
> -       map_cpu_to_node(0, 0);
> -

otherwise, how you set numa info for cpu0/boot-processor?

thanks
Ganapat
>         return 0;
>  }
>
> --
> 2.5.0
>
>
>

thanks
ganapat

> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v4 12/14] arm64/numa: remove some useless code

[Leizhen (ThunderTown)]

On 2016/6/7 16:28, Ganapatrao Kulkarni wrote:
> On Tue, Jun 7, 2016 at 1:38 PM, Zhen Lei <thunder.leizhen@huawei.com> wrote:
>> 1. Currently only cpu0 set on cpu_possible_mask and percpu areas have not
>>    been initialized.
>> 2. No reason to limit cpu0 must belongs to node0.
> 
> even smp init assumes cpu0/boot processor.
Yes, we define boot cpu as cpu0. But we can not force cpu0 must belongs to node0.
For example, we use the same Image and dtb run on two boards. On the first board,
BIOS choose cpu-A as boot cpu. But on the other board, BIOS may choose CPU-B as
boot cpu. Although this case is difficult to appear, but we can not sure that it
will not appear.

> is this patch tested on any hardware?
Yes, I tested it on our D02 board.

> can you describe your testing hardware?
Althoug D02 only contains one hardware numa node. But the implementation of numa
software is hardware independent, so I define some logical numa nodes. For example:
treat each core as a numa node, and subdivide momory.

>>
>> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
>> ---
>>  arch/arm64/mm/numa.c | 8 --------
>>  1 file changed, 8 deletions(-)
>>
>> diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
>> index d73b0a0..92b1692 100644
>> --- a/arch/arm64/mm/numa.c
>> +++ b/arch/arm64/mm/numa.c
>> @@ -93,7 +93,6 @@ void numa_clear_node(unsigned int cpu)
>>   */
>>  static void __init setup_node_to_cpumask_map(void)
>>  {
>> -       unsigned int cpu;
>>         int node;
>>
>>         /* setup nr_node_ids if not done yet */
>> @@ -106,9 +105,6 @@ static void __init setup_node_to_cpumask_map(void)
>>                 cpumask_clear(node_to_cpumask_map[node]);
>>         }
>>
>> -       for_each_possible_cpu(cpu)
>> -               set_cpu_numa_node(cpu, NUMA_NO_NODE);
>> -
> 
> do you see this init of setting node id to NUMA_NO_NODE  for each cpu
> happening any where else?
I have used below code to verify my judgement, it's only "cpu=0" printed.

for_each_possible_cpu(cpu)
	pr_info("setup_node_to_cpumask_map: cpu=%d\n", cpu);

Actually, the execution sequence is as below:
1. setup_arch
   1) bootmem_init();   -->arm64_numa_init
   2) smp_init_cpus();	-->smp_cpu_setup --> set_cpu_possible(cpu, true);

So that, the above deleted code only set cpu0 to NUMA_NO_NODE. And the below
deleted code set cpu0 to nid0. In fact, the default value of cpu_to_node(0)
is also zero. So I said these code take no effect.

> otherwise, better to have initialised node id/NUMA_NO_NODE to every
> cpu otherwise default  node id will be shown as zero
> which is not correct.
> 
>>         /* cpumask_of_node() will now work */
>>         pr_debug("Node to cpumask map for %d nodes\n", nr_node_ids);
>>  }
>> @@ -379,10 +375,6 @@ static int __init numa_init(int (*init_func)(void))
>>
>>         setup_node_to_cpumask_map();
>>
>> -       /* init boot processor */
>> -       cpu_to_node_map[0] = 0;
>> -       map_cpu_to_node(0, 0);
>> -
> 
> otherwise, how you set numa info for cpu0/boot-processor?
I have done it in the previous patch.

diff --git a/arch/arm64/kernel/smp.c b/arch/arm64/kernel/smp.c
index d099306..9e15297 100644
--- a/arch/arm64/kernel/smp.c
+++ b/arch/arm64/kernel/smp.c
@@ -620,6 +620,7 @@ static void __init of_parse_and_init_cpus(void)
 			}

 			bootcpu_valid = true;
+			early_map_cpu_to_node(0, of_node_to_nid(dn));

 			/*
 			 * cpu_logical_map has already been
diff --git a/arch/arm64/mm/numa.c b/arch/arm64/mm/numa.c
index df5c842..d73b0a0 100644
--- a/arch/arm64/mm/numa.c
+++ b/arch/arm64/mm/numa.c
@@ -128,6 +128,14 @@ void __init early_map_cpu_to_node(unsigned int cpu, int nid)
 		nid = 0;

 	cpu_to_node_map[cpu] = nid;
+
+	/*
+	 * We should set the numa node of cpu0 as soon as possible, because it
+	 * has already been set up online before. cpu_to_node(0) will soon be
+	 * called.
+	 */
+	if (!cpu)
+		set_cpu_numa_node(cpu, nid);
 }

> 
> thanks
> Ganapat
>>         return 0;
>>  }
>>
>> --
>> 2.5.0
>>
>>
>>
> 
> thanks
> ganapat
> 
>> _______________________________________________
>> linux-arm-kernel mailing list
>> linux-arm-kernel@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
> 
> .
> 

[PATCH v4 13/14] of/numa: remove the constraint on the distances of node pairs

[Zhen Lei]

At present, the distances must equal in both direction for each node
pairs. For example: the distance of node B->A must the same to A->B.
But we really don't have to do this.

End up fill default distances as below:
1. If both direction specified, keep no change.
2. If only one direction specified, assign it to the other direction.
3. If none of the two direction specified, both are assigned to
   REMOTE_DISTANCE.

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
 drivers/of/of_numa.c | 22 +++++++++++++++++-----
 1 file changed, 17 insertions(+), 5 deletions(-)

diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
index 019738f..ce782f9 100644
--- a/drivers/of/of_numa.c
+++ b/drivers/of/of_numa.c
@@ -122,15 +122,25 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map)
 		numa_set_distance(nodea, nodeb, distance);
 		pr_debug("distance[node%d -> node%d] = %d\n",
 			 nodea, nodeb, distance);
-
-		/* Set default distance of node B->A same as A->B */
-		if (nodeb > nodea)
-			numa_set_distance(nodeb, nodea, distance);
 	}

 	return 0;
 }

+static void __init fill_default_distances(void)
+{
+	int i, j;
+
+	for (i = 0; i < nr_node_ids; i++)
+		for (j = 0; j < nr_node_ids; j++)
+			if (i == j)
+				numa_set_distance(i, j, LOCAL_DISTANCE);
+			else if (!node_distance(i, j))
+				numa_set_distance(i, j,
+				    node_distance(j, i) ? : REMOTE_DISTANCE);
+
+}
+
 static int __init of_numa_parse_distance_map(void)
 {
 	int ret = 0;
@@ -140,8 +150,10 @@ static int __init of_numa_parse_distance_map(void)
 				     "numa-distance-map-v1");
 	if (np)
 		ret = of_numa_parse_distance_map_v1(np);
-
 	of_node_put(np);
+
+	fill_default_distances();
+
 	return ret;
 }

--
2.5.0

[PATCH v4 14/14] Documentation: remove the constraint on the distances of node pairs

[Zhen Lei]

Update documentation. This limit is unneccessary.

Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
---
 Documentation/devicetree/bindings/numa.txt | 1 -
 1 file changed, 1 deletion(-)

diff --git a/Documentation/devicetree/bindings/numa.txt b/Documentation/devicetree/bindings/numa.txt
index 21b3505..c0ea4a7 100644
--- a/Documentation/devicetree/bindings/numa.txt
+++ b/Documentation/devicetree/bindings/numa.txt
@@ -48,7 +48,6 @@ distance (memory latency) between all numa nodes.

   Note:
 	1. Each entry represents distance from first node to second node.
-	The distances are equal in either direction.
 	2. The distance from a node to self (local distance) is represented
 	with value 10 and all internode distance should be represented with
 	a value greater than 10.
--
2.5.0

Re: [PATCH v4 14/14] Documentation: remove the constraint on the distances of node pairs

[Rob Herring]

On Tue, Jun 07, 2016 at 04:08:18PM +0800, Zhen Lei wrote:
> Update documentation. This limit is unneccessary.
> 
> Signed-off-by: Zhen Lei <thunder.leizhen@huawei.com>
> ---
>  Documentation/devicetree/bindings/numa.txt | 1 -
>  1 file changed, 1 deletion(-)

Acked-by: Rob Herring <robh@kernel.org>

Re: [PATCH v4 00/14] fix some type infos and bugs for arm64/of numa

[Will Deacon]

On Tue, Jun 07, 2016 at 04:08:04PM +0800, Zhen Lei wrote:
> v3 -> v4:
> 1. Packed three patches of Kefeng Wang, patch6-8.
> 2. Add 6 new patches(9-15) to enhance the numa on arm64.
> 
> v2 -> v3:
> 1. Adjust patch2 and patch5 according to Matthias Brugger's advice, to make the
>    patches looks more well. The final code have no change. 
> 
> v1 -> v2:
> 1. Base on https://lkml.org/lkml/2016/5/24/679

If you want bug fixes to land in 4.7, you'll need to base them on a
mainline kernel.

Will

Re: [PATCH v4 00/14] fix some type infos and bugs for arm64/of numa

[Leizhen (ThunderTown)]

On 2016/6/7 21:58, Will Deacon wrote:
> On Tue, Jun 07, 2016 at 04:08:04PM +0800, Zhen Lei wrote:
>> v3 -> v4:
>> 1. Packed three patches of Kefeng Wang, patch6-8.
>> 2. Add 6 new patches(9-15) to enhance the numa on arm64.
>>
>> v2 -> v3:
>> 1. Adjust patch2 and patch5 according to Matthias Brugger's advice, to make the
>>    patches looks more well. The final code have no change. 
>>
>> v1 -> v2:
>> 1. Base on https://lkml.org/lkml/2016/5/24/679
> 
> If you want bug fixes to land in 4.7, you'll need to base them on a
> mainline kernel.

I heared that David Daney's acpi numa patch series was accepted and put into next branch(Linux 4.8).
Otherwise I will suggest him sending his patch6-7 to mainline first. So that, only a very small conflict
will be exist.

I also tested that:
1. git am David Daney's patch6-7, then git am all of my patches on a branch, named branch A.
2. git am David Daney's patch6-7 on another branch, named branch B.
3. when I git merge B into branch A, it's still conflict. So I guess git merge is based on source code, rather than patches.

So at present, unless the maintainers are willing to resolve the conflict, otherwise I update my patches will not work.

Fortunately, these patches are not particularly urgent. So I think I can wait until Linux 4.8 start, then send these patches again.
But I'm not sure whether these patches can be merged into Linux 4.8, I really hope.

> 
> Will
> 
> .
> 

Re: [PATCH v4 00/14] fix some type infos and bugs for arm64/of numa

[Catalin Marinas]

On Wed, Jun 08, 2016 at 04:59:03PM +0800, Leizhen (ThunderTown) wrote:
> On 2016/6/7 21:58, Will Deacon wrote:
> > On Tue, Jun 07, 2016 at 04:08:04PM +0800, Zhen Lei wrote:
> >> v3 -> v4:
> >> 1. Packed three patches of Kefeng Wang, patch6-8.
> >> 2. Add 6 new patches(9-15) to enhance the numa on arm64.
> >>
> >> v2 -> v3:
> >> 1. Adjust patch2 and patch5 according to Matthias Brugger's advice, to make the
> >>    patches looks more well. The final code have no change. 
> >>
> >> v1 -> v2:
> >> 1. Base on https://lkml.org/lkml/2016/5/24/679
> > 
> > If you want bug fixes to land in 4.7, you'll need to base them on a
> > mainline kernel.
> 
> I heared that David Daney's acpi numa patch series was accepted and
> put into next branch(Linux 4.8).
> Otherwise I will suggest him sending his patch6-7 to mainline first.
> So that, only a very small conflict will be exist.
> 
> I also tested that:
> 1. git am David Daney's patch6-7, then git am all of my patches on a
> branch, named branch A.
> 2. git am David Daney's patch6-7 on another branch, named branch B.
> 3. when I git merge B into branch A, it's still conflict. So I guess
> git merge is based on source code, rather than patches.
> 
> So at present, unless the maintainers are willing to resolve the
> conflict, otherwise I update my patches will not work.

It usually depends on how complex the conflict is and whether your
patches functionally depend on the other patches. I have no idea what
the dependency is here since I haven't tried applying them to mainline.

> Fortunately, these patches are not particularly urgent. So I think I
> can wait until Linux 4.8 start, then send these patches again. But I'm
> not sure whether these patches can be merged into Linux 4.8, I really
> hope.

If there are fixes to the arm64 ACPI NUMA patches that Rafael queued
into linux-next, they should be sent to him and potentially being queued
on top ahead of the 4.8 merging window or shortly after 4.8-rc1.
Non-ACPI NUMA patches (as I can see, most of these patches are DT
specific) could be merged independently.

So how many patches do you have in each category below:

1. NUMA fixes against current mainline (4.7-rc3)
2. NUMA fixes against the arm64 ACPI NUMA patches queued by Rafael
3. New functionality or clean-up. Are these against mainline or ACPI
   NUMA patches?

-- 
Catalin

Re: [PATCH v4 00/14] fix some type infos and bugs for arm64/of numa

[Leizhen (ThunderTown)]

On 2016/6/14 22:22, Catalin Marinas wrote:
> On Wed, Jun 08, 2016 at 04:59:03PM +0800, Leizhen (ThunderTown) wrote:
>> On 2016/6/7 21:58, Will Deacon wrote:
>>> On Tue, Jun 07, 2016 at 04:08:04PM +0800, Zhen Lei wrote:
>>>> v3 -> v4:
>>>> 1. Packed three patches of Kefeng Wang, patch6-8.
>>>> 2. Add 6 new patches(9-15) to enhance the numa on arm64.
>>>>
>>>> v2 -> v3:
>>>> 1. Adjust patch2 and patch5 according to Matthias Brugger's advice, to make the
>>>>    patches looks more well. The final code have no change. 
>>>>
>>>> v1 -> v2:
>>>> 1. Base on https://lkml.org/lkml/2016/5/24/679
>>>
>>> If you want bug fixes to land in 4.7, you'll need to base them on a
>>> mainline kernel.
>>
>> I heared that David Daney's acpi numa patch series was accepted and
>> put into next branch(Linux 4.8).
>> Otherwise I will suggest him sending his patch6-7 to mainline first.
>> So that, only a very small conflict will be exist.
>>
>> I also tested that:
>> 1. git am David Daney's patch6-7, then git am all of my patches on a
>> branch, named branch A.
>> 2. git am David Daney's patch6-7 on another branch, named branch B.
>> 3. when I git merge B into branch A, it's still conflict. So I guess
>> git merge is based on source code, rather than patches.
>>
>> So at present, unless the maintainers are willing to resolve the
>> conflict, otherwise I update my patches will not work.
> 
> It usually depends on how complex the conflict is and whether your
> patches functionally depend on the other patches. I have no idea what
> the dependency is here since I haven't tried applying them to mainline.
> 
>> Fortunately, these patches are not particularly urgent. So I think I
>> can wait until Linux 4.8 start, then send these patches again. But I'm
>> not sure whether these patches can be merged into Linux 4.8, I really
>> hope.
> 
> If there are fixes to the arm64 ACPI NUMA patches that Rafael queued
> into linux-next, they should be sent to him and potentially being queued
> on top ahead of the 4.8 merging window or shortly after 4.8-rc1.
> Non-ACPI NUMA patches (as I can see, most of these patches are DT
> specific) could be merged independently.
> 
> So how many patches do you have in each category below:
> 
> 1. NUMA fixes against current mainline (4.7-rc3)
> 2. NUMA fixes against the arm64 ACPI NUMA patches queued by Rafael
My patches have not fixed any bugs for ACPI NUMA, but just based on it.
There are only three related patches:
[PATCH v7 06_15] arm64, numa  rework numa_add_memblk()
[PATCH v7 07_15] arm64, numa  Cleanup NUMA disabled messages.
[PATCH v7 14_15] arm64, acpi, numa  NUMA support based on SRAT and SLIT

arch/arm64/mm/numa.c          |  28 ++++--
drivers/of/of_numa.c          |   4 +-

My patches 1-5, 8, 11 will confict with it.

> 3. New functionality or clean-up. Are these against mainline or ACPI
>    NUMA patches?
Hi, Catalin
I'm sorry to reply this email too late. Because I have been thinking if
there are any other solutions.

I try to adjust the sequence of my patches as below:
1. New functionality 		//queued in your branch  (my patches 9-14, and 6, 6 is clean-up)
2. 4.8-rc1			//apci numa series and my new functionality had been merged
3. bug fixes			//other 4.8-rc versions	 (my patches 1-5)
4. clean-up (pr_fmt)		//queued in 4.9		 (my patches 7-8)

And there only one confliction exist:
++<<<<<<< HEAD
 +static u8 numa_distance[MAX_NUMNODES][MAX_NUMNODES];			//choose this
 +static int numa_off;
++=======
+ static int numa_distance_cnt;
+ static u8 *numa_distance;
+ static bool numa_off;							//choose this
++>>>>>>> acpi

> 

Re: [PATCH v4 00/14] fix some type infos and bugs for arm64/of numa

[Leizhen (ThunderTown)]

On 2016/6/20 14:39, Leizhen (ThunderTown) wrote:
> 
> 
> On 2016/6/14 22:22, Catalin Marinas wrote:
>> On Wed, Jun 08, 2016 at 04:59:03PM +0800, Leizhen (ThunderTown) wrote:
>>> On 2016/6/7 21:58, Will Deacon wrote:
>>>> On Tue, Jun 07, 2016 at 04:08:04PM +0800, Zhen Lei wrote:
>>>>> v3 -> v4:
>>>>> 1. Packed three patches of Kefeng Wang, patch6-8.
>>>>> 2. Add 6 new patches(9-15) to enhance the numa on arm64.
>>>>>
>>>>> v2 -> v3:
>>>>> 1. Adjust patch2 and patch5 according to Matthias Brugger's advice, to make the
>>>>>    patches looks more well. The final code have no change. 
>>>>>
>>>>> v1 -> v2:
>>>>> 1. Base on https://lkml.org/lkml/2016/5/24/679
>>>>
>>>> If you want bug fixes to land in 4.7, you'll need to base them on a
>>>> mainline kernel.
>>>
>>> I heared that David Daney's acpi numa patch series was accepted and
>>> put into next branch(Linux 4.8).
>>> Otherwise I will suggest him sending his patch6-7 to mainline first.
>>> So that, only a very small conflict will be exist.
>>>
>>> I also tested that:
>>> 1. git am David Daney's patch6-7, then git am all of my patches on a
>>> branch, named branch A.
>>> 2. git am David Daney's patch6-7 on another branch, named branch B.
>>> 3. when I git merge B into branch A, it's still conflict. So I guess
>>> git merge is based on source code, rather than patches.
>>>
>>> So at present, unless the maintainers are willing to resolve the
>>> conflict, otherwise I update my patches will not work.
>>
>> It usually depends on how complex the conflict is and whether your
>> patches functionally depend on the other patches. I have no idea what
>> the dependency is here since I haven't tried applying them to mainline.
>>
>>> Fortunately, these patches are not particularly urgent. So I think I
>>> can wait until Linux 4.8 start, then send these patches again. But I'm
>>> not sure whether these patches can be merged into Linux 4.8, I really
>>> hope.
>>
>> If there are fixes to the arm64 ACPI NUMA patches that Rafael queued
>> into linux-next, they should be sent to him and potentially being queued
>> on top ahead of the 4.8 merging window or shortly after 4.8-rc1.
>> Non-ACPI NUMA patches (as I can see, most of these patches are DT
>> specific) could be merged independently.
>>
>> So how many patches do you have in each category below:
>>
>> 1. NUMA fixes against current mainline (4.7-rc3)
>> 2. NUMA fixes against the arm64 ACPI NUMA patches queued by Rafael
> My patches have not fixed any bugs for ACPI NUMA, but just based on it.
> There are only three related patches:
> [PATCH v7 06_15] arm64, numa  rework numa_add_memblk()
> [PATCH v7 07_15] arm64, numa  Cleanup NUMA disabled messages.
> [PATCH v7 14_15] arm64, acpi, numa  NUMA support based on SRAT and SLIT
> 
> arch/arm64/mm/numa.c          |  28 ++++--
> drivers/of/of_numa.c          |   4 +-
> 
> My patches 1-5, 8, 11 will confict with it.
> 
>> 3. New functionality or clean-up. Are these against mainline or ACPI
>>    NUMA patches?
> Hi, Catalin
> I'm sorry to reply this email too late. Because I have been thinking if
> there are any other solutions.
> 
> I try to adjust the sequence of my patches as below:
> 1. New functionality 		//queued in your branch  (my patches 9-14, and 6, 6 is clean-up)
> 2. 4.8-rc1			//apci numa series and my new functionality had been merged
> 3. bug fixes			//other 4.8-rc versions	 (my patches 1-5)
> 4. clean-up (pr_fmt)		//queued in 4.9		 (my patches 7-8)

Hi, Catalin
  What about your opinion? Are you agree?

> 
> And there only one confliction exist:
> ++<<<<<<< HEAD
>  +static u8 numa_distance[MAX_NUMNODES][MAX_NUMNODES];			//choose this
>  +static int numa_off;
> ++=======
> + static int numa_distance_cnt;
> + static u8 *numa_distance;
> + static bool numa_off;							//choose this
> ++>>>>>>> acpi
> 
>>

Re: [PATCH v4 00/14] fix some type infos and bugs for arm64/of numa

[Hanjun Guo]

Hi Will,

On 2016/6/7 21:58, Will Deacon wrote:
> On Tue, Jun 07, 2016 at 04:08:04PM +0800, Zhen Lei wrote:
>> v3 -> v4:
>> 1. Packed three patches of Kefeng Wang, patch6-8.
>> 2. Add 6 new patches(9-15) to enhance the numa on arm64.
>>
>> v2 -> v3:
>> 1. Adjust patch2 and patch5 according to Matthias Brugger's advice, to make the
>>    patches looks more well. The final code have no change. 
>>
>> v1 -> v2:
>> 1. Base on https://lkml.org/lkml/2016/5/24/679
> If you want bug fixes to land in 4.7, you'll need to base them on a
> mainline kernel.
>

Rafael queued ARM64 ACPI NUMA support for 4.8, and this patch set is based
on that with no urgent bugfixes, can this patch set be queued for 4.8?

Thanks
Hanjun

Re: [PATCH v4 00/14] fix some type infos and bugs for arm64/of numa

[Will Deacon]

On Sun, Jun 12, 2016 at 03:09:46PM +0800, Hanjun Guo wrote:
> On 2016/6/7 21:58, Will Deacon wrote:
> > On Tue, Jun 07, 2016 at 04:08:04PM +0800, Zhen Lei wrote:
> >> v3 -> v4:
> >> 1. Packed three patches of Kefeng Wang, patch6-8.
> >> 2. Add 6 new patches(9-15) to enhance the numa on arm64.
> >>
> >> v2 -> v3:
> >> 1. Adjust patch2 and patch5 according to Matthias Brugger's advice, to make the
> >>    patches looks more well. The final code have no change. 
> >>
> >> v1 -> v2:
> >> 1. Base on https://lkml.org/lkml/2016/5/24/679
> > If you want bug fixes to land in 4.7, you'll need to base them on a
> > mainline kernel.
> >
> 
> Rafael queued ARM64 ACPI NUMA support for 4.8, and this patch set is based
> on that with no urgent bugfixes, can this patch set be queued for 4.8?

Up to Catalin, since he's handling the 4.8 merge window. It would be really
nice if you could give us the heads up about dependencies like this in the
future, preferably *before* the base part has already been merged. That way,
it's easier to create shared topic branches and gives us more options if
we think that conflicts are likely to occur.

Will

Re: [PATCH v4 00/14] fix some type infos and bugs for arm64/of numa

[Hanjun Guo]

On 2016/6/13 18:12, Will Deacon wrote:
> On Sun, Jun 12, 2016 at 03:09:46PM +0800, Hanjun Guo wrote:
>> On 2016/6/7 21:58, Will Deacon wrote:
>>> On Tue, Jun 07, 2016 at 04:08:04PM +0800, Zhen Lei wrote:
>>>> v3 -> v4:
>>>> 1. Packed three patches of Kefeng Wang, patch6-8.
>>>> 2. Add 6 new patches(9-15) to enhance the numa on arm64.
>>>>
>>>> v2 -> v3:
>>>> 1. Adjust patch2 and patch5 according to Matthias Brugger's advice, to make the
>>>>    patches looks more well. The final code have no change. 
>>>>
>>>> v1 -> v2:
>>>> 1. Base on https://lkml.org/lkml/2016/5/24/679
>>> If you want bug fixes to land in 4.7, you'll need to base them on a
>>> mainline kernel.
>>>
>> Rafael queued ARM64 ACPI NUMA support for 4.8, and this patch set is based
>> on that with no urgent bugfixes, can this patch set be queued for 4.8?
> Up to Catalin, since he's handling the 4.8 merge window. It would be really
> nice if you could give us the heads up about dependencies like this in the
> future, preferably *before* the base part has already been merged. That way,
> it's easier to create shared topic branches and gives us more options if
> we think that conflicts are likely to occur.

Understood the maintenance difficulty will have, will do that next time.
Much appreciate if it can be merged for 4.8.

Thanks
Hanjun

Re: [PATCH v4 00/14] fix some type infos and bugs for arm64/of numa

[Rob Herring]

On Tue, Jun 7, 2016 at 3:08 AM, Zhen Lei <thunder.leizhen@huawei.com> wrote:
> v3 -> v4:
> 1. Packed three patches of Kefeng Wang, patch6-8.
> 2. Add 6 new patches(9-15) to enhance the numa on arm64.
>
> v2 -> v3:
> 1. Adjust patch2 and patch5 according to Matthias Brugger's advice, to make the
>    patches looks more well. The final code have no change.
>
> v1 -> v2:
> 1. Base on https://lkml.org/lkml/2016/5/24/679
> 2. Rewrote of_numa_parse_memory_nodes according to Rob Herring's advice. So that it looks more clear.
> 3. Rewrote patch 5 because some scenes were not considered before.
>
> Kefeng Wang (3):
>   of_numa: Use of_get_next_parent to simplify code
>   of_numa: Use pr_fmt()
>   arm64: numa: Use pr_fmt()
>
> Zhen Lei (11):
>   of/numa: remove a duplicated pr_debug information
>   of/numa: fix a memory@ node can only contains one memory block
>   arm64/numa: add nid check for memory block
>   of/numa: remove a duplicated warning
>   arm64/numa: avoid inconsistent information to be printed
>   arm64/numa: support HAVE_SETUP_PER_CPU_AREA
>   arm64/numa: define numa_distance as array to simplify code
>   arm64/numa: support HAVE_MEMORYLESS_NODES
>   arm64/numa: remove some useless code
>   of/numa: remove the constraint on the distances of node pairs
>   Documentation: remove the constraint on the distances of node pairs
>
>  Documentation/devicetree/bindings/numa.txt |   1 -
>  arch/arm64/Kconfig                         |  12 ++
>  arch/arm64/include/asm/numa.h              |   1 -
>  arch/arm64/kernel/smp.c                    |   1 +
>  arch/arm64/mm/numa.c                       | 228 ++++++++++++++++-------------
>  drivers/of/of_numa.c                       |  87 +++++------
>  6 files changed, 180 insertions(+), 150 deletions(-)

For patches 1, 2, 4, 6, 7, and 13:

Acked-by: Rob Herring <robh@kernel.org>

Rob