Linux-arch Archive on lore.kernel.org
 help / color / Atom feed
* [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V
@ 2020-10-06  0:17 Atish Patra
  2020-10-06  0:17 ` [PATCH v4 1/5] numa: Move numa implementation to common code Atish Patra
                   ` (5 more replies)
  0 siblings, 6 replies; 11+ messages in thread
From: Atish Patra @ 2020-10-06  0:17 UTC (permalink / raw)
  To: linux-kernel
  Cc: Atish Patra, Jonathan Cameron, Albert Ou, Andrew Morton,
	Anshuman Khandual, Anup Patel, Arnd Bergmann, Catalin Marinas,
	David Hildenbrand, Greentime Hu, Greg Kroah-Hartman, Jia He,
	Kefeng Wang, linux-arch, linux-riscv, Mike Rapoport,
	Nicolas Saenz Julienne, Palmer Dabbelt, Paul Walmsley,
	Rafael J. Wysocki, Steven Price, Will Deacon, Zong Li,
	linux-arm-kernel

This series attempts to move the ARM64 numa implementation to common
code so that RISC-V can leverage that as well instead of reimplementing
it again.

RISC-V specific bits are based on initial work done by Greentime Hu [1] but
modified to reuse the common implementation to avoid duplication.

[1] https://lkml.org/lkml/2020/1/10/233

This series has been tested on qemu with numa enabled for both RISC-V & ARM64.
It would be great if somebody can test it on numa capable ARM64 hardware platforms.
This patch series doesn't modify the maintainers list for the common code (arch_numa)
as I am not sure if somebody from ARM64 community or Greg should take up the
maintainership. Ganapatrao was the original author of the arm64 version.
I would be happy to update that in the next revision once it is decided.

# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3
node 0 size: 486 MB
node 0 free: 470 MB
node 1 cpus: 4 5 6 7
node 1 size: 424 MB
node 1 free: 408 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10 
# numactl -show
policy: default
preferred node: current
physcpubind: 0 1 2 3 4 5 6 7 
cpubind: 0 1 
nodebind: 0 1 
membind: 0 1 

The patches are also available at
https://github.com/atishp04/linux/tree/5.10_numa_unified_v4

For RISC-V, the following qemu series is a pre-requisite(already available in upstream)
https://patchwork.kernel.org/project/qemu-devel/list/?series=303313

Testing:
RISC-V:
Tested in Qemu and 2 socket OmniXtend FPGA.

ARM64:
2 socket kunpeng920 (4 nodes around 250G a node)
Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>

There may be some minor conflicts with Mike's cleanup series [2] depending on the
order in which these two series are being accepted. I can rebase on top his series
if required.

[2] https://lkml.org/lkml/2020/8/18/754

Changes from v3->v4:
1. Removed redundant duplicate header.
2. Added Reviewed-by tags.

Changes from v2->v3:
1. Added Acked-by/Reviewed-by tags.
2. Replaced asm/acpi.h with linux/acpi.h
3. Defined arch_acpi_numa_init as static.

Changes from v1->v2:
1. Replaced ARM64 specific compile time protection with ACPI specific ones.
2. Dropped common pcibus_to_node changes. Added required changes in RISC-V. 
3. Fixed few typos.

Atish Patra (4):
numa: Move numa implementation to common code
arm64, numa: Change the numa init functions name to be generic
riscv: Separate memory init from paging init
riscv: Add numa support for riscv64 platform

Greentime Hu (1):
riscv: Add support pte_protnone and pmd_protnone if
CONFIG_NUMA_BALANCING

arch/arm64/Kconfig                            |  1 +
arch/arm64/include/asm/numa.h                 | 45 +----------------
arch/arm64/kernel/acpi_numa.c                 | 13 -----
arch/arm64/mm/Makefile                        |  1 -
arch/arm64/mm/init.c                          |  4 +-
arch/riscv/Kconfig                            | 31 +++++++++++-
arch/riscv/include/asm/mmzone.h               | 13 +++++
arch/riscv/include/asm/numa.h                 |  8 +++
arch/riscv/include/asm/pci.h                  | 14 ++++++
arch/riscv/include/asm/pgtable.h              | 21 ++++++++
arch/riscv/kernel/setup.c                     | 11 ++++-
arch/riscv/kernel/smpboot.c                   | 12 ++++-
arch/riscv/mm/init.c                          | 10 +++-
drivers/base/Kconfig                          |  6 +++
drivers/base/Makefile                         |  1 +
.../mm/numa.c => drivers/base/arch_numa.c     | 30 ++++++++++--
include/asm-generic/numa.h                    | 49 +++++++++++++++++++
17 files changed, 199 insertions(+), 71 deletions(-)
create mode 100644 arch/riscv/include/asm/mmzone.h
create mode 100644 arch/riscv/include/asm/numa.h
rename arch/arm64/mm/numa.c => drivers/base/arch_numa.c (95%)
create mode 100644 include/asm-generic/numa.h

--
2.25.1


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH v4 1/5] numa: Move numa implementation to common code
  2020-10-06  0:17 [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V Atish Patra
@ 2020-10-06  0:17 ` Atish Patra
  2020-10-06  0:17 ` [PATCH v4 2/5] arm64, numa: Change the numa init functions name to be generic Atish Patra
                   ` (4 subsequent siblings)
  5 siblings, 0 replies; 11+ messages in thread
From: Atish Patra @ 2020-10-06  0:17 UTC (permalink / raw)
  To: linux-kernel
  Cc: Atish Patra, Jonathan Cameron, Albert Ou, Andrew Morton,
	Anshuman Khandual, Anup Patel, Arnd Bergmann, Catalin Marinas,
	David Hildenbrand, Greentime Hu, Greg Kroah-Hartman, Jia He,
	Kefeng Wang, linux-arch, linux-riscv, Mike Rapoport,
	Nicolas Saenz Julienne, Palmer Dabbelt, Paul Walmsley,
	Rafael J. Wysocki, Steven Price, Will Deacon, Zong Li,
	linux-arm-kernel

ARM64 numa implementation is generic enough that RISC-V can reuse that
implementation with very minor cosmetic changes. This will help both
ARM64 and RISC-V in terms of maintanace and feature improvement

Move the numa implementation code to common directory so that both ISAs
can reuse this. This doesn't introduce any function changes for ARM64.

Signed-off-by: Atish Patra <atish.patra@wdc.com>
Acked-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 arch/arm64/Kconfig                            |  1 +
 arch/arm64/include/asm/numa.h                 | 45 +----------------
 arch/arm64/mm/Makefile                        |  1 -
 drivers/base/Kconfig                          |  6 +++
 drivers/base/Makefile                         |  1 +
 .../mm/numa.c => drivers/base/arch_numa.c     |  0
 include/asm-generic/numa.h                    | 49 +++++++++++++++++++
 7 files changed, 58 insertions(+), 45 deletions(-)
 rename arch/arm64/mm/numa.c => drivers/base/arch_numa.c (100%)
 create mode 100644 include/asm-generic/numa.h

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 6d232837cbee..955a0cf75b16 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -960,6 +960,7 @@ config HOTPLUG_CPU
 # Common NUMA Features
 config NUMA
 	bool "NUMA Memory Allocation and Scheduler Support"
+	select GENERIC_ARCH_NUMA
 	select ACPI_NUMA if ACPI
 	select OF_NUMA
 	help
diff --git a/arch/arm64/include/asm/numa.h b/arch/arm64/include/asm/numa.h
index 626ad01e83bf..8c8cf4297cc3 100644
--- a/arch/arm64/include/asm/numa.h
+++ b/arch/arm64/include/asm/numa.h
@@ -3,49 +3,6 @@
 #define __ASM_NUMA_H
 
 #include <asm/topology.h>
-
-#ifdef CONFIG_NUMA
-
-#define NR_NODE_MEMBLKS		(MAX_NUMNODES * 2)
-
-int __node_distance(int from, int to);
-#define node_distance(a, b) __node_distance(a, b)
-
-extern nodemask_t numa_nodes_parsed __initdata;
-
-extern bool numa_off;
-
-/* Mappings between node number and cpus on that node. */
-extern cpumask_var_t node_to_cpumask_map[MAX_NUMNODES];
-void numa_clear_node(unsigned int cpu);
-
-#ifdef CONFIG_DEBUG_PER_CPU_MAPS
-const struct cpumask *cpumask_of_node(int node);
-#else
-/* Returns a pointer to the cpumask of CPUs on Node 'node'. */
-static inline const struct cpumask *cpumask_of_node(int node)
-{
-	return node_to_cpumask_map[node];
-}
-#endif
-
-void __init arm64_numa_init(void);
-int __init numa_add_memblk(int nodeid, u64 start, u64 end);
-void __init numa_set_distance(int from, int to, int distance);
-void __init numa_free_distance(void);
-void __init early_map_cpu_to_node(unsigned int cpu, int nid);
-void numa_store_cpu_info(unsigned int cpu);
-void numa_add_cpu(unsigned int cpu);
-void numa_remove_cpu(unsigned int cpu);
-
-#else	/* CONFIG_NUMA */
-
-static inline void numa_store_cpu_info(unsigned int cpu) { }
-static inline void numa_add_cpu(unsigned int cpu) { }
-static inline void numa_remove_cpu(unsigned int cpu) { }
-static inline void arm64_numa_init(void) { }
-static inline void early_map_cpu_to_node(unsigned int cpu, int nid) { }
-
-#endif	/* CONFIG_NUMA */
+#include <asm-generic/numa.h>
 
 #endif	/* __ASM_NUMA_H */
diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile
index d91030f0ffee..928c308b044b 100644
--- a/arch/arm64/mm/Makefile
+++ b/arch/arm64/mm/Makefile
@@ -6,7 +6,6 @@ obj-y				:= dma-mapping.o extable.o fault.o init.o \
 obj-$(CONFIG_HUGETLB_PAGE)	+= hugetlbpage.o
 obj-$(CONFIG_PTDUMP_CORE)	+= dump.o
 obj-$(CONFIG_PTDUMP_DEBUGFS)	+= ptdump_debugfs.o
-obj-$(CONFIG_NUMA)		+= numa.o
 obj-$(CONFIG_DEBUG_VIRTUAL)	+= physaddr.o
 KASAN_SANITIZE_physaddr.o	+= n
 
diff --git a/drivers/base/Kconfig b/drivers/base/Kconfig
index 8d7001712062..c5956c8845cc 100644
--- a/drivers/base/Kconfig
+++ b/drivers/base/Kconfig
@@ -210,4 +210,10 @@ config GENERIC_ARCH_TOPOLOGY
 	  appropriate scaling, sysfs interface for reading capacity values at
 	  runtime.
 
+config GENERIC_ARCH_NUMA
+	bool
+	help
+	  Enable support for generic NUMA implementation. Currently, RISC-V
+	  and ARM64 uses it.
+
 endmenu
diff --git a/drivers/base/Makefile b/drivers/base/Makefile
index 157452080f3d..c3d02c644222 100644
--- a/drivers/base/Makefile
+++ b/drivers/base/Makefile
@@ -23,6 +23,7 @@ obj-$(CONFIG_PINCTRL) += pinctrl.o
 obj-$(CONFIG_DEV_COREDUMP) += devcoredump.o
 obj-$(CONFIG_GENERIC_MSI_IRQ_DOMAIN) += platform-msi.o
 obj-$(CONFIG_GENERIC_ARCH_TOPOLOGY) += arch_topology.o
+obj-$(CONFIG_GENERIC_ARCH_NUMA) += arch_numa.o
 
 obj-y			+= test/
 
diff --git a/arch/arm64/mm/numa.c b/drivers/base/arch_numa.c
similarity index 100%
rename from arch/arm64/mm/numa.c
rename to drivers/base/arch_numa.c
diff --git a/include/asm-generic/numa.h b/include/asm-generic/numa.h
new file mode 100644
index 000000000000..2718d5a6ff03
--- /dev/null
+++ b/include/asm-generic/numa.h
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_GENERIC_NUMA_H
+#define __ASM_GENERIC_NUMA_H
+
+#ifdef CONFIG_NUMA
+
+#define NR_NODE_MEMBLKS		(MAX_NUMNODES * 2)
+
+int __node_distance(int from, int to);
+#define node_distance(a, b) __node_distance(a, b)
+
+extern nodemask_t numa_nodes_parsed __initdata;
+
+extern bool numa_off;
+
+/* Mappings between node number and cpus on that node. */
+extern cpumask_var_t node_to_cpumask_map[MAX_NUMNODES];
+void numa_clear_node(unsigned int cpu);
+
+#ifdef CONFIG_DEBUG_PER_CPU_MAPS
+const struct cpumask *cpumask_of_node(int node);
+#else
+/* Returns a pointer to the cpumask of CPUs on Node 'node'. */
+static inline const struct cpumask *cpumask_of_node(int node)
+{
+	return node_to_cpumask_map[node];
+}
+#endif
+
+void __init arm64_numa_init(void);
+int __init numa_add_memblk(int nodeid, u64 start, u64 end);
+void __init numa_set_distance(int from, int to, int distance);
+void __init numa_free_distance(void);
+void __init early_map_cpu_to_node(unsigned int cpu, int nid);
+void numa_store_cpu_info(unsigned int cpu);
+void numa_add_cpu(unsigned int cpu);
+void numa_remove_cpu(unsigned int cpu);
+
+#else	/* CONFIG_NUMA */
+
+static inline void numa_store_cpu_info(unsigned int cpu) { }
+static inline void numa_add_cpu(unsigned int cpu) { }
+static inline void numa_remove_cpu(unsigned int cpu) { }
+static inline void arm64_numa_init(void) { }
+static inline void early_map_cpu_to_node(unsigned int cpu, int nid) { }
+
+#endif	/* CONFIG_NUMA */
+
+#endif	/* __ASM_GENERIC_NUMA_H */
-- 
2.25.1


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH v4 2/5] arm64, numa: Change the numa init functions name to be generic
  2020-10-06  0:17 [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V Atish Patra
  2020-10-06  0:17 ` [PATCH v4 1/5] numa: Move numa implementation to common code Atish Patra
@ 2020-10-06  0:17 ` Atish Patra
  2020-10-06  0:17 ` [PATCH v4 3/5] riscv: Separate memory init from paging init Atish Patra
                   ` (3 subsequent siblings)
  5 siblings, 0 replies; 11+ messages in thread
From: Atish Patra @ 2020-10-06  0:17 UTC (permalink / raw)
  To: linux-kernel
  Cc: Atish Patra, Jonathan Cameron, Albert Ou, Andrew Morton,
	Anshuman Khandual, Anup Patel, Arnd Bergmann, Catalin Marinas,
	David Hildenbrand, Greentime Hu, Greg Kroah-Hartman, Jia He,
	Kefeng Wang, linux-arch, linux-riscv, Mike Rapoport,
	Nicolas Saenz Julienne, Palmer Dabbelt, Paul Walmsley,
	Rafael J. Wysocki, Steven Price, Will Deacon, Zong Li,
	linux-arm-kernel

As we are using generic numa implementation code, modify the acpi & numa
init functions name to indicate that generic implementation.

Signed-off-by: Atish Patra <atish.patra@wdc.com>
Reviewed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
 arch/arm64/kernel/acpi_numa.c | 13 -------------
 arch/arm64/mm/init.c          |  4 ++--
 drivers/base/arch_numa.c      | 30 +++++++++++++++++++++++++-----
 include/asm-generic/numa.h    |  4 ++--
 4 files changed, 29 insertions(+), 22 deletions(-)

diff --git a/arch/arm64/kernel/acpi_numa.c b/arch/arm64/kernel/acpi_numa.c
index 7ff800045434..96502ff92af5 100644
--- a/arch/arm64/kernel/acpi_numa.c
+++ b/arch/arm64/kernel/acpi_numa.c
@@ -117,16 +117,3 @@ void __init acpi_numa_gicc_affinity_init(struct acpi_srat_gicc_affinity *pa)
 
 	node_set(node, numa_nodes_parsed);
 }
-
-int __init arm64_acpi_numa_init(void)
-{
-	int ret;
-
-	ret = acpi_numa_init();
-	if (ret) {
-		pr_info("Failed to initialise from firmware\n");
-		return ret;
-	}
-
-	return srat_disabled() ? -EINVAL : 0;
-}
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 481d22c32a2e..93b660229e1d 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -418,10 +418,10 @@ void __init bootmem_init(void)
 	max_pfn = max_low_pfn = max;
 	min_low_pfn = min;
 
-	arm64_numa_init();
+	arch_numa_init();
 
 	/*
-	 * must be done after arm64_numa_init() which calls numa_init() to
+	 * must be done after arch_numa_init() which calls numa_init() to
 	 * initialize node_online_map that gets used in hugetlb_cma_reserve()
 	 * while allocating required CMA size across online nodes.
 	 */
diff --git a/drivers/base/arch_numa.c b/drivers/base/arch_numa.c
index 73f8b49d485c..74b4f2ddad70 100644
--- a/drivers/base/arch_numa.c
+++ b/drivers/base/arch_numa.c
@@ -13,7 +13,6 @@
 #include <linux/module.h>
 #include <linux/of.h>
 
-#include <asm/acpi.h>
 #include <asm/sections.h>
 
 struct pglist_data *node_data[MAX_NUMNODES] __read_mostly;
@@ -444,16 +443,37 @@ static int __init dummy_numa_init(void)
 	return 0;
 }
 
+#ifdef CONFIG_ACPI_NUMA
+static int __init arch_acpi_numa_init(void)
+{
+	int ret;
+
+	ret = acpi_numa_init();
+	if (ret) {
+		pr_info("Failed to initialise from firmware\n");
+		return ret;
+	}
+
+	return srat_disabled() ? -EINVAL : 0;
+}
+#else
+static int __init arch_acpi_numa_init(void)
+{
+	return -EOPNOTSUPP;
+}
+
+#endif
+
 /**
- * arm64_numa_init() - Initialize NUMA
+ * arch_numa_init() - Initialize NUMA
  *
  * Try each configured NUMA initialization method until one succeeds. The
- * last fallback is dummy single node config encomapssing whole memory.
+ * last fallback is dummy single node config encompassing whole memory.
  */
-void __init arm64_numa_init(void)
+void __init arch_numa_init(void)
 {
 	if (!numa_off) {
-		if (!acpi_disabled && !numa_init(arm64_acpi_numa_init))
+		if (!acpi_disabled && !numa_init(arch_acpi_numa_init))
 			return;
 		if (acpi_disabled && !numa_init(of_numa_init))
 			return;
diff --git a/include/asm-generic/numa.h b/include/asm-generic/numa.h
index 2718d5a6ff03..e7962db4ba44 100644
--- a/include/asm-generic/numa.h
+++ b/include/asm-generic/numa.h
@@ -27,7 +27,7 @@ static inline const struct cpumask *cpumask_of_node(int node)
 }
 #endif
 
-void __init arm64_numa_init(void);
+void __init arch_numa_init(void);
 int __init numa_add_memblk(int nodeid, u64 start, u64 end);
 void __init numa_set_distance(int from, int to, int distance);
 void __init numa_free_distance(void);
@@ -41,7 +41,7 @@ void numa_remove_cpu(unsigned int cpu);
 static inline void numa_store_cpu_info(unsigned int cpu) { }
 static inline void numa_add_cpu(unsigned int cpu) { }
 static inline void numa_remove_cpu(unsigned int cpu) { }
-static inline void arm64_numa_init(void) { }
+static inline void arch_numa_init(void) { }
 static inline void early_map_cpu_to_node(unsigned int cpu, int nid) { }
 
 #endif	/* CONFIG_NUMA */
-- 
2.25.1


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH v4 3/5] riscv: Separate memory init from paging init
  2020-10-06  0:17 [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V Atish Patra
  2020-10-06  0:17 ` [PATCH v4 1/5] numa: Move numa implementation to common code Atish Patra
  2020-10-06  0:17 ` [PATCH v4 2/5] arm64, numa: Change the numa init functions name to be generic Atish Patra
@ 2020-10-06  0:17 ` Atish Patra
  2020-10-22  5:37   ` Anup Patel
  2020-10-06  0:17 ` [PATCH v4 4/5] riscv: Add support pte_protnone and pmd_protnone if CONFIG_NUMA_BALANCING Atish Patra
                   ` (2 subsequent siblings)
  5 siblings, 1 reply; 11+ messages in thread
From: Atish Patra @ 2020-10-06  0:17 UTC (permalink / raw)
  To: linux-kernel
  Cc: Atish Patra, Greentime Hu, Albert Ou, Andrew Morton,
	Anshuman Khandual, Anup Patel, Arnd Bergmann, Catalin Marinas,
	David Hildenbrand, Greg Kroah-Hartman, Jia He, Jonathan Cameron,
	Kefeng Wang, linux-arch, linux-riscv, Mike Rapoport,
	Nicolas Saenz Julienne, Palmer Dabbelt, Paul Walmsley,
	Rafael J. Wysocki, Steven Price, Will Deacon, Zong Li,
	linux-arm-kernel

Currently, we perform some memory init functions in paging init. But,
that will be an issue for NUMA support where DT needs to be flattened
before numa initialization and memblock_present can only be called
after numa initialization.

Move memory initialization related functions to a separate function.

Signed-off-by: Atish Patra <atish.patra@wdc.com>
Reviewed-by: Greentime Hu <greentime.hu@sifive.com>
---
 arch/riscv/include/asm/pgtable.h | 1 +
 arch/riscv/kernel/setup.c        | 1 +
 arch/riscv/mm/init.c             | 6 +++++-
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index eaea1f717010..515b42f98d34 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -466,6 +466,7 @@ static inline void __kernel_map_pages(struct page *page, int numpages, int enabl
 extern void *dtb_early_va;
 void setup_bootmem(void);
 void paging_init(void);
+void misc_mem_init(void);
 
 #define FIRST_USER_ADDRESS  0
 
diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c
index 2c6dd329312b..07fa6d13367e 100644
--- a/arch/riscv/kernel/setup.c
+++ b/arch/riscv/kernel/setup.c
@@ -78,6 +78,7 @@ void __init setup_arch(char **cmdline_p)
 #else
 	unflatten_device_tree();
 #endif
+	misc_mem_init();
 
 #ifdef CONFIG_SWIOTLB
 	swiotlb_init(1);
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index ed6e83871112..114c3966aadb 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -565,8 +565,12 @@ static void __init resource_init(void)
 void __init paging_init(void)
 {
 	setup_vm_final();
-	sparse_init();
 	setup_zero_page();
+}
+
+void __init misc_mem_init(void)
+{
+	sparse_init();
 	zone_sizes_init();
 	resource_init();
 }
-- 
2.25.1


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH v4 4/5] riscv: Add support pte_protnone and pmd_protnone if CONFIG_NUMA_BALANCING
  2020-10-06  0:17 [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V Atish Patra
                   ` (2 preceding siblings ...)
  2020-10-06  0:17 ` [PATCH v4 3/5] riscv: Separate memory init from paging init Atish Patra
@ 2020-10-06  0:17 ` Atish Patra
  2020-10-22  5:38   ` Anup Patel
  2020-10-06  0:17 ` [PATCH v4 5/5] riscv: Add numa support for riscv64 platform Atish Patra
  2020-10-13 20:19 ` [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V Atish Patra
  5 siblings, 1 reply; 11+ messages in thread
From: Atish Patra @ 2020-10-06  0:17 UTC (permalink / raw)
  To: linux-kernel
  Cc: Greentime Hu, Albert Ou, Andrew Morton, Anshuman Khandual,
	Anup Patel, Arnd Bergmann, Atish Patra, Catalin Marinas,
	David Hildenbrand, Greg Kroah-Hartman, Jia He, Jonathan Cameron,
	Kefeng Wang, linux-arch, linux-riscv, Mike Rapoport,
	Nicolas Saenz Julienne, Palmer Dabbelt, Paul Walmsley,
	Rafael J. Wysocki, Steven Price, Will Deacon, Zong Li,
	linux-arm-kernel

From: Greentime Hu <greentime.hu@sifive.com>

These two functions are used to distinguish between PROT_NONENUMA
protections and hinting fault protections.

Signed-off-by: Greentime Hu <greentime.hu@sifive.com>
---
 arch/riscv/include/asm/pgtable.h | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
index 515b42f98d34..2751110675e6 100644
--- a/arch/riscv/include/asm/pgtable.h
+++ b/arch/riscv/include/asm/pgtable.h
@@ -183,6 +183,11 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
 	return (unsigned long)pfn_to_virt(pmd_val(pmd) >> _PAGE_PFN_SHIFT);
 }
 
+static inline pte_t pmd_pte(pmd_t pmd)
+{
+	return __pte(pmd_val(pmd));
+}
+
 /* Yields the page frame number (PFN) of a page table entry */
 static inline unsigned long pte_pfn(pte_t pte)
 {
@@ -286,6 +291,21 @@ static inline pte_t pte_mkhuge(pte_t pte)
 	return pte;
 }
 
+#ifdef CONFIG_NUMA_BALANCING
+/*
+ * See the comment in include/asm-generic/pgtable.h
+ */
+static inline int pte_protnone(pte_t pte)
+{
+	return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)) == _PAGE_PROT_NONE;
+}
+
+static inline int pmd_protnone(pmd_t pmd)
+{
+	return pte_protnone(pmd_pte(pmd));
+}
+#endif
+
 /* Modify page protection bits */
 static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
 {
-- 
2.25.1


^ permalink raw reply	[flat|nested] 11+ messages in thread

* [PATCH v4 5/5] riscv: Add numa support for riscv64 platform
  2020-10-06  0:17 [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V Atish Patra
                   ` (3 preceding siblings ...)
  2020-10-06  0:17 ` [PATCH v4 4/5] riscv: Add support pte_protnone and pmd_protnone if CONFIG_NUMA_BALANCING Atish Patra
@ 2020-10-06  0:17 ` Atish Patra
  2020-10-22  5:40   ` Anup Patel
  2020-10-13 20:19 ` [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V Atish Patra
  5 siblings, 1 reply; 11+ messages in thread
From: Atish Patra @ 2020-10-06  0:17 UTC (permalink / raw)
  To: linux-kernel
  Cc: Atish Patra, Greentime Hu, Albert Ou, Andrew Morton,
	Anshuman Khandual, Anup Patel, Arnd Bergmann, Catalin Marinas,
	David Hildenbrand, Greg Kroah-Hartman, Jia He, Jonathan Cameron,
	Kefeng Wang, linux-arch, linux-riscv, Mike Rapoport,
	Nicolas Saenz Julienne, Palmer Dabbelt, Paul Walmsley,
	Rafael J. Wysocki, Steven Price, Will Deacon, Zong Li,
	linux-arm-kernel

Use the generic numa implementation to add NUMA support for RISC-V.
This is based on Greentime's patch[1] but modified to use generic NUMA
implementation and few more fixes.

[1] https://lkml.org/lkml/2020/1/10/233

Co-developed-by: Greentime Hu <greentime.hu@sifive.com>
Signed-off-by: Greentime Hu <greentime.hu@sifive.com>
Signed-off-by: Atish Patra <atish.patra@wdc.com>
---
 arch/riscv/Kconfig              | 31 ++++++++++++++++++++++++++++++-
 arch/riscv/include/asm/mmzone.h | 13 +++++++++++++
 arch/riscv/include/asm/numa.h   |  8 ++++++++
 arch/riscv/include/asm/pci.h    | 14 ++++++++++++++
 arch/riscv/kernel/setup.c       | 10 ++++++++--
 arch/riscv/kernel/smpboot.c     | 12 +++++++++++-
 arch/riscv/mm/init.c            |  4 +++-
 7 files changed, 87 insertions(+), 5 deletions(-)
 create mode 100644 arch/riscv/include/asm/mmzone.h
 create mode 100644 arch/riscv/include/asm/numa.h

diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index df18372861d8..7beb6ddb6eb1 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -137,7 +137,7 @@ config PAGE_OFFSET
 	default 0xffffffe000000000 if 64BIT && MAXPHYSMEM_128GB
 
 config ARCH_FLATMEM_ENABLE
-	def_bool y
+	def_bool !NUMA
 
 config ARCH_SPARSEMEM_ENABLE
 	def_bool y
@@ -295,6 +295,35 @@ config TUNE_GENERIC
 
 endchoice
 
+# Common NUMA Features
+config NUMA
+	bool "NUMA Memory Allocation and Scheduler Support"
+	select GENERIC_ARCH_NUMA
+	select OF_NUMA
+	select ARCH_SUPPORTS_NUMA_BALANCING
+	help
+	  Enable NUMA (Non-Uniform Memory Access) support.
+
+	  The kernel will try to allocate memory used by a CPU on the
+	  local memory of the CPU and add some more NUMA awareness to the kernel.
+
+config NODES_SHIFT
+	int "Maximum NUMA Nodes (as a power of 2)"
+	range 1 10
+	default "2"
+	depends on NEED_MULTIPLE_NODES
+	help
+	  Specify the maximum number of NUMA Nodes available on the target
+	  system.  Increases memory reserved to accommodate various tables.
+
+config USE_PERCPU_NUMA_NODE_ID
+	def_bool y
+	depends on NUMA
+
+config NEED_PER_CPU_EMBED_FIRST_CHUNK
+	def_bool y
+	depends on NUMA
+
 config RISCV_ISA_C
 	bool "Emit compressed instructions when building Linux"
 	default y
diff --git a/arch/riscv/include/asm/mmzone.h b/arch/riscv/include/asm/mmzone.h
new file mode 100644
index 000000000000..fa17e01d9ab2
--- /dev/null
+++ b/arch/riscv/include/asm/mmzone.h
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_MMZONE_H
+#define __ASM_MMZONE_H
+
+#ifdef CONFIG_NUMA
+
+#include <asm/numa.h>
+
+extern struct pglist_data *node_data[];
+#define NODE_DATA(nid)		(node_data[(nid)])
+
+#endif /* CONFIG_NUMA */
+#endif /* __ASM_MMZONE_H */
diff --git a/arch/riscv/include/asm/numa.h b/arch/riscv/include/asm/numa.h
new file mode 100644
index 000000000000..8c8cf4297cc3
--- /dev/null
+++ b/arch/riscv/include/asm/numa.h
@@ -0,0 +1,8 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_NUMA_H
+#define __ASM_NUMA_H
+
+#include <asm/topology.h>
+#include <asm-generic/numa.h>
+
+#endif	/* __ASM_NUMA_H */
diff --git a/arch/riscv/include/asm/pci.h b/arch/riscv/include/asm/pci.h
index 1c473a1bd986..658e112c3ce7 100644
--- a/arch/riscv/include/asm/pci.h
+++ b/arch/riscv/include/asm/pci.h
@@ -32,6 +32,20 @@ static inline int pci_proc_domain(struct pci_bus *bus)
 	/* always show the domain in /proc */
 	return 1;
 }
+
+#ifdef	CONFIG_NUMA
+
+static inline int pcibus_to_node(struct pci_bus *bus)
+{
+	return dev_to_node(&bus->dev);
+}
+#ifndef cpumask_of_pcibus
+#define cpumask_of_pcibus(bus)	(pcibus_to_node(bus) == -1 ?		\
+				 cpu_all_mask :				\
+				 cpumask_of_node(pcibus_to_node(bus)))
+#endif
+#endif	/* CONFIG_NUMA */
+
 #endif  /* CONFIG_PCI */
 
 #endif  /* _ASM_RISCV_PCI_H */
diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c
index 07fa6d13367e..53a806a9cbaf 100644
--- a/arch/riscv/kernel/setup.c
+++ b/arch/riscv/kernel/setup.c
@@ -101,13 +101,19 @@ void __init setup_arch(char **cmdline_p)
 
 static int __init topology_init(void)
 {
-	int i;
+	int i, ret;
+
+	for_each_online_node(i)
+		register_one_node(i);
 
 	for_each_possible_cpu(i) {
 		struct cpu *cpu = &per_cpu(cpu_devices, i);
 
 		cpu->hotpluggable = cpu_has_hotplug(i);
-		register_cpu(cpu, i);
+		ret = register_cpu(cpu, i);
+		if (unlikely(ret))
+			pr_warn("Warning: %s: register_cpu %d failed (%d)\n",
+			       __func__, i, ret);
 	}
 
 	return 0;
diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c
index 96167d55ed98..5e276c25646f 100644
--- a/arch/riscv/kernel/smpboot.c
+++ b/arch/riscv/kernel/smpboot.c
@@ -27,6 +27,7 @@
 #include <asm/cpu_ops.h>
 #include <asm/irq.h>
 #include <asm/mmu_context.h>
+#include <asm/numa.h>
 #include <asm/tlbflush.h>
 #include <asm/sections.h>
 #include <asm/sbi.h>
@@ -45,13 +46,18 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
 {
 	int cpuid;
 	int ret;
+	unsigned int curr_cpuid;
+
+	curr_cpuid = smp_processor_id();
+	numa_store_cpu_info(curr_cpuid);
+	numa_add_cpu(curr_cpuid);
 
 	/* This covers non-smp usecase mandated by "nosmp" option */
 	if (max_cpus == 0)
 		return;
 
 	for_each_possible_cpu(cpuid) {
-		if (cpuid == smp_processor_id())
+		if (cpuid == curr_cpuid)
 			continue;
 		if (cpu_ops[cpuid]->cpu_prepare) {
 			ret = cpu_ops[cpuid]->cpu_prepare(cpuid);
@@ -59,6 +65,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
 				continue;
 		}
 		set_cpu_present(cpuid, true);
+		numa_store_cpu_info(cpuid);
 	}
 }
 
@@ -79,6 +86,7 @@ void __init setup_smp(void)
 		if (hart == cpuid_to_hartid_map(0)) {
 			BUG_ON(found_boot_cpu);
 			found_boot_cpu = 1;
+			early_map_cpu_to_node(0, of_node_to_nid(dn));
 			continue;
 		}
 		if (cpuid >= NR_CPUS) {
@@ -88,6 +96,7 @@ void __init setup_smp(void)
 		}
 
 		cpuid_to_hartid_map(cpuid) = hart;
+		early_map_cpu_to_node(cpuid, of_node_to_nid(dn));
 		cpuid++;
 	}
 
@@ -153,6 +162,7 @@ asmlinkage __visible void smp_callin(void)
 	current->active_mm = mm;
 
 	notify_cpu_starting(curr_cpuid);
+	numa_add_cpu(curr_cpuid);
 	update_siblings_masks(curr_cpuid);
 	set_cpu_online(curr_cpuid, 1);
 
diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
index 114c3966aadb..c4046e11d264 100644
--- a/arch/riscv/mm/init.c
+++ b/arch/riscv/mm/init.c
@@ -20,6 +20,7 @@
 #include <asm/soc.h>
 #include <asm/io.h>
 #include <asm/ptdump.h>
+#include <asm/numa.h>
 
 #include "../kernel/head.h"
 
@@ -185,7 +186,6 @@ void __init setup_bootmem(void)
 
 	early_init_fdt_scan_reserved_mem();
 	memblock_allow_resize();
-	memblock_dump_all();
 
 	for_each_memblock(memory, reg) {
 		unsigned long start_pfn = memblock_region_memory_base_pfn(reg);
@@ -570,9 +570,11 @@ void __init paging_init(void)
 
 void __init misc_mem_init(void)
 {
+	arch_numa_init();
 	sparse_init();
 	zone_sizes_init();
 	resource_init();
+	memblock_dump_all();
 }
 
 #ifdef CONFIG_SPARSEMEM_VMEMMAP
-- 
2.25.1


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V
  2020-10-06  0:17 [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V Atish Patra
                   ` (4 preceding siblings ...)
  2020-10-06  0:17 ` [PATCH v4 5/5] riscv: Add numa support for riscv64 platform Atish Patra
@ 2020-10-13 20:19 ` Atish Patra
  2020-10-19 21:29   ` Palmer Dabbelt
  5 siblings, 1 reply; 11+ messages in thread
From: Atish Patra @ 2020-10-13 20:19 UTC (permalink / raw)
  To: Atish Patra
  Cc: linux-kernel@vger.kernel.org List, Kefeng Wang,
	David Hildenbrand, Catalin Marinas, Jonathan Cameron, Zong Li,
	linux-riscv, Will Deacon, linux-arch, Jia He, Anup Patel,
	Rafael J. Wysocki, Steven Price, Greentime Hu, Albert Ou,
	Arnd Bergmann, Anshuman Khandual, Paul Walmsley,
	linux-arm-kernel, Greg Kroah-Hartman, Palmer Dabbelt,
	Mike Rapoport, Andrew Morton, Nicolas Saenz Julienne

On Mon, Oct 5, 2020 at 5:18 PM Atish Patra <atish.patra@wdc.com> wrote:
>
> This series attempts to move the ARM64 numa implementation to common
> code so that RISC-V can leverage that as well instead of reimplementing
> it again.
>
> RISC-V specific bits are based on initial work done by Greentime Hu [1] but
> modified to reuse the common implementation to avoid duplication.
>
> [1] https://lkml.org/lkml/2020/1/10/233
>
> This series has been tested on qemu with numa enabled for both RISC-V & ARM64.
> It would be great if somebody can test it on numa capable ARM64 hardware platforms.
> This patch series doesn't modify the maintainers list for the common code (arch_numa)
> as I am not sure if somebody from ARM64 community or Greg should take up the
> maintainership. Ganapatrao was the original author of the arm64 version.
> I would be happy to update that in the next revision once it is decided.
>
> # numactl --hardware
> available: 2 nodes (0-1)
> node 0 cpus: 0 1 2 3
> node 0 size: 486 MB
> node 0 free: 470 MB
> node 1 cpus: 4 5 6 7
> node 1 size: 424 MB
> node 1 free: 408 MB
> node distances:
> node   0   1
>   0:  10  20
>   1:  20  10
> # numactl -show
> policy: default
> preferred node: current
> physcpubind: 0 1 2 3 4 5 6 7
> cpubind: 0 1
> nodebind: 0 1
> membind: 0 1
>
> The patches are also available at
> https://github.com/atishp04/linux/tree/5.10_numa_unified_v4
>
> For RISC-V, the following qemu series is a pre-requisite(already available in upstream)
> https://patchwork.kernel.org/project/qemu-devel/list/?series=303313
>
> Testing:
> RISC-V:
> Tested in Qemu and 2 socket OmniXtend FPGA.
>
> ARM64:
> 2 socket kunpeng920 (4 nodes around 250G a node)
> Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>
> There may be some minor conflicts with Mike's cleanup series [2] depending on the
> order in which these two series are being accepted. I can rebase on top his series
> if required.
>
> [2] https://lkml.org/lkml/2020/8/18/754
>
> Changes from v3->v4:
> 1. Removed redundant duplicate header.
> 2. Added Reviewed-by tags.
>
> Changes from v2->v3:
> 1. Added Acked-by/Reviewed-by tags.
> 2. Replaced asm/acpi.h with linux/acpi.h
> 3. Defined arch_acpi_numa_init as static.
>
> Changes from v1->v2:
> 1. Replaced ARM64 specific compile time protection with ACPI specific ones.
> 2. Dropped common pcibus_to_node changes. Added required changes in RISC-V.
> 3. Fixed few typos.
>
> Atish Patra (4):
> numa: Move numa implementation to common code
> arm64, numa: Change the numa init functions name to be generic
> riscv: Separate memory init from paging init
> riscv: Add numa support for riscv64 platform
>
> Greentime Hu (1):
> riscv: Add support pte_protnone and pmd_protnone if
> CONFIG_NUMA_BALANCING
>
> arch/arm64/Kconfig                            |  1 +
> arch/arm64/include/asm/numa.h                 | 45 +----------------
> arch/arm64/kernel/acpi_numa.c                 | 13 -----
> arch/arm64/mm/Makefile                        |  1 -
> arch/arm64/mm/init.c                          |  4 +-
> arch/riscv/Kconfig                            | 31 +++++++++++-
> arch/riscv/include/asm/mmzone.h               | 13 +++++
> arch/riscv/include/asm/numa.h                 |  8 +++
> arch/riscv/include/asm/pci.h                  | 14 ++++++
> arch/riscv/include/asm/pgtable.h              | 21 ++++++++
> arch/riscv/kernel/setup.c                     | 11 ++++-
> arch/riscv/kernel/smpboot.c                   | 12 ++++-
> arch/riscv/mm/init.c                          | 10 +++-
> drivers/base/Kconfig                          |  6 +++
> drivers/base/Makefile                         |  1 +
> .../mm/numa.c => drivers/base/arch_numa.c     | 30 ++++++++++--
> include/asm-generic/numa.h                    | 49 +++++++++++++++++++
> 17 files changed, 199 insertions(+), 71 deletions(-)
> create mode 100644 arch/riscv/include/asm/mmzone.h
> create mode 100644 arch/riscv/include/asm/numa.h
> rename arch/arm64/mm/numa.c => drivers/base/arch_numa.c (95%)
> create mode 100644 include/asm-generic/numa.h
>
> --
> 2.25.1
>
>
> _______________________________________________
> linux-riscv mailing list
> linux-riscv@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-riscv

Ping ?

-- 
Regards,
Atish

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V
  2020-10-13 20:19 ` [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V Atish Patra
@ 2020-10-19 21:29   ` Palmer Dabbelt
  0 siblings, 0 replies; 11+ messages in thread
From: Palmer Dabbelt @ 2020-10-19 21:29 UTC (permalink / raw)
  To: atishp
  Cc: Atish Patra, linux-kernel, wangkefeng.wang, david,
	catalin.marinas, Jonathan.Cameron, zong.li, linux-riscv, will,
	linux-arch, justin.he, anup, rafael, steven.price, greentime.hu,
	aou, Arnd Bergmann, anshuman.khandual, Paul Walmsley,
	linux-arm-kernel, Greg KH, rppt, akpm, nsaenzjulienne

On Tue, 13 Oct 2020 13:19:35 PDT (-0700), atishp@atishpatra.org wrote:
> On Mon, Oct 5, 2020 at 5:18 PM Atish Patra <atish.patra@wdc.com> wrote:
>>
>> This series attempts to move the ARM64 numa implementation to common
>> code so that RISC-V can leverage that as well instead of reimplementing
>> it again.
>>
>> RISC-V specific bits are based on initial work done by Greentime Hu [1] but
>> modified to reuse the common implementation to avoid duplication.
>>
>> [1] https://lkml.org/lkml/2020/1/10/233
>>
>> This series has been tested on qemu with numa enabled for both RISC-V & ARM64.
>> It would be great if somebody can test it on numa capable ARM64 hardware platforms.
>> This patch series doesn't modify the maintainers list for the common code (arch_numa)
>> as I am not sure if somebody from ARM64 community or Greg should take up the
>> maintainership. Ganapatrao was the original author of the arm64 version.
>> I would be happy to update that in the next revision once it is decided.
>>
>> # numactl --hardware
>> available: 2 nodes (0-1)
>> node 0 cpus: 0 1 2 3
>> node 0 size: 486 MB
>> node 0 free: 470 MB
>> node 1 cpus: 4 5 6 7
>> node 1 size: 424 MB
>> node 1 free: 408 MB
>> node distances:
>> node   0   1
>>   0:  10  20
>>   1:  20  10
>> # numactl -show
>> policy: default
>> preferred node: current
>> physcpubind: 0 1 2 3 4 5 6 7
>> cpubind: 0 1
>> nodebind: 0 1
>> membind: 0 1
>>
>> The patches are also available at
>> https://github.com/atishp04/linux/tree/5.10_numa_unified_v4
>>
>> For RISC-V, the following qemu series is a pre-requisite(already available in upstream)
>> https://patchwork.kernel.org/project/qemu-devel/list/?series=303313
>>
>> Testing:
>> RISC-V:
>> Tested in Qemu and 2 socket OmniXtend FPGA.
>>
>> ARM64:
>> 2 socket kunpeng920 (4 nodes around 250G a node)
>> Tested-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>>
>> There may be some minor conflicts with Mike's cleanup series [2] depending on the
>> order in which these two series are being accepted. I can rebase on top his series
>> if required.
>>
>> [2] https://lkml.org/lkml/2020/8/18/754
>>
>> Changes from v3->v4:
>> 1. Removed redundant duplicate header.
>> 2. Added Reviewed-by tags.
>>
>> Changes from v2->v3:
>> 1. Added Acked-by/Reviewed-by tags.
>> 2. Replaced asm/acpi.h with linux/acpi.h
>> 3. Defined arch_acpi_numa_init as static.
>>
>> Changes from v1->v2:
>> 1. Replaced ARM64 specific compile time protection with ACPI specific ones.
>> 2. Dropped common pcibus_to_node changes. Added required changes in RISC-V.
>> 3. Fixed few typos.
>>
>> Atish Patra (4):
>> numa: Move numa implementation to common code
>> arm64, numa: Change the numa init functions name to be generic
>> riscv: Separate memory init from paging init
>> riscv: Add numa support for riscv64 platform
>>
>> Greentime Hu (1):
>> riscv: Add support pte_protnone and pmd_protnone if
>> CONFIG_NUMA_BALANCING
>>
>> arch/arm64/Kconfig                            |  1 +
>> arch/arm64/include/asm/numa.h                 | 45 +----------------
>> arch/arm64/kernel/acpi_numa.c                 | 13 -----
>> arch/arm64/mm/Makefile                        |  1 -
>> arch/arm64/mm/init.c                          |  4 +-
>> arch/riscv/Kconfig                            | 31 +++++++++++-
>> arch/riscv/include/asm/mmzone.h               | 13 +++++
>> arch/riscv/include/asm/numa.h                 |  8 +++
>> arch/riscv/include/asm/pci.h                  | 14 ++++++
>> arch/riscv/include/asm/pgtable.h              | 21 ++++++++
>> arch/riscv/kernel/setup.c                     | 11 ++++-
>> arch/riscv/kernel/smpboot.c                   | 12 ++++-
>> arch/riscv/mm/init.c                          | 10 +++-
>> drivers/base/Kconfig                          |  6 +++
>> drivers/base/Makefile                         |  1 +
>> .../mm/numa.c => drivers/base/arch_numa.c     | 30 ++++++++++--
>> include/asm-generic/numa.h                    | 49 +++++++++++++++++++
>> 17 files changed, 199 insertions(+), 71 deletions(-)
>> create mode 100644 arch/riscv/include/asm/mmzone.h
>> create mode 100644 arch/riscv/include/asm/numa.h
>> rename arch/arm64/mm/numa.c => drivers/base/arch_numa.c (95%)
>> create mode 100644 include/asm-generic/numa.h
>>
>> --
>> 2.25.1
>>
>>
>> _______________________________________________
>> linux-riscv mailing list
>> linux-riscv@lists.infradead.org
>> http://lists.infradead.org/mailman/listinfo/linux-riscv
>
> Ping ?

This has been at the top of my inbox for a week or two now, I just haven't
gotten around to taking a look yet because of all the other fires going on.
Sorry.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH v4 3/5] riscv: Separate memory init from paging init
  2020-10-06  0:17 ` [PATCH v4 3/5] riscv: Separate memory init from paging init Atish Patra
@ 2020-10-22  5:37   ` Anup Patel
  0 siblings, 0 replies; 11+ messages in thread
From: Anup Patel @ 2020-10-22  5:37 UTC (permalink / raw)
  To: Atish Patra
  Cc: linux-kernel@vger.kernel.org List, Greentime Hu, Albert Ou,
	Andrew Morton, Anshuman Khandual, Arnd Bergmann, Catalin Marinas,
	David Hildenbrand, Greg Kroah-Hartman, Jia He, Jonathan Cameron,
	Kefeng Wang, linux-arch, linux-riscv, Mike Rapoport,
	Nicolas Saenz Julienne, Palmer Dabbelt, Paul Walmsley,
	Rafael J. Wysocki, Steven Price, Will Deacon, Zong Li,
	linux-arm-kernel

On Tue, Oct 6, 2020 at 5:48 AM Atish Patra <atish.patra@wdc.com> wrote:
>
> Currently, we perform some memory init functions in paging init. But,
> that will be an issue for NUMA support where DT needs to be flattened
> before numa initialization and memblock_present can only be called
> after numa initialization.
>
> Move memory initialization related functions to a separate function.
>
> Signed-off-by: Atish Patra <atish.patra@wdc.com>
> Reviewed-by: Greentime Hu <greentime.hu@sifive.com>
> ---
>  arch/riscv/include/asm/pgtable.h | 1 +
>  arch/riscv/kernel/setup.c        | 1 +
>  arch/riscv/mm/init.c             | 6 +++++-
>  3 files changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
> index eaea1f717010..515b42f98d34 100644
> --- a/arch/riscv/include/asm/pgtable.h
> +++ b/arch/riscv/include/asm/pgtable.h
> @@ -466,6 +466,7 @@ static inline void __kernel_map_pages(struct page *page, int numpages, int enabl
>  extern void *dtb_early_va;
>  void setup_bootmem(void);
>  void paging_init(void);
> +void misc_mem_init(void);
>
>  #define FIRST_USER_ADDRESS  0
>
> diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c
> index 2c6dd329312b..07fa6d13367e 100644
> --- a/arch/riscv/kernel/setup.c
> +++ b/arch/riscv/kernel/setup.c
> @@ -78,6 +78,7 @@ void __init setup_arch(char **cmdline_p)
>  #else
>         unflatten_device_tree();
>  #endif
> +       misc_mem_init();
>
>  #ifdef CONFIG_SWIOTLB
>         swiotlb_init(1);
> diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
> index ed6e83871112..114c3966aadb 100644
> --- a/arch/riscv/mm/init.c
> +++ b/arch/riscv/mm/init.c
> @@ -565,8 +565,12 @@ static void __init resource_init(void)
>  void __init paging_init(void)
>  {
>         setup_vm_final();
> -       sparse_init();
>         setup_zero_page();
> +}
> +
> +void __init misc_mem_init(void)
> +{
> +       sparse_init();
>         zone_sizes_init();
>         resource_init();
>  }
> --
> 2.25.1
>

Looks good to me.

Reviewed-by: Anup Patel <anup@brainfault.org>

Regards,
Anup

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH v4 4/5] riscv: Add support pte_protnone and pmd_protnone if CONFIG_NUMA_BALANCING
  2020-10-06  0:17 ` [PATCH v4 4/5] riscv: Add support pte_protnone and pmd_protnone if CONFIG_NUMA_BALANCING Atish Patra
@ 2020-10-22  5:38   ` Anup Patel
  0 siblings, 0 replies; 11+ messages in thread
From: Anup Patel @ 2020-10-22  5:38 UTC (permalink / raw)
  To: Atish Patra
  Cc: linux-kernel@vger.kernel.org List, Greentime Hu, Albert Ou,
	Andrew Morton, Anshuman Khandual, Arnd Bergmann, Catalin Marinas,
	David Hildenbrand, Greg Kroah-Hartman, Jia He, Jonathan Cameron,
	Kefeng Wang, linux-arch, linux-riscv, Mike Rapoport,
	Nicolas Saenz Julienne, Palmer Dabbelt, Paul Walmsley,
	Rafael J. Wysocki, Steven Price, Will Deacon, Zong Li,
	linux-arm-kernel

On Tue, Oct 6, 2020 at 5:48 AM Atish Patra <atish.patra@wdc.com> wrote:
>
> From: Greentime Hu <greentime.hu@sifive.com>
>
> These two functions are used to distinguish between PROT_NONENUMA
> protections and hinting fault protections.
>
> Signed-off-by: Greentime Hu <greentime.hu@sifive.com>
> ---
>  arch/riscv/include/asm/pgtable.h | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
>
> diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h
> index 515b42f98d34..2751110675e6 100644
> --- a/arch/riscv/include/asm/pgtable.h
> +++ b/arch/riscv/include/asm/pgtable.h
> @@ -183,6 +183,11 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
>         return (unsigned long)pfn_to_virt(pmd_val(pmd) >> _PAGE_PFN_SHIFT);
>  }
>
> +static inline pte_t pmd_pte(pmd_t pmd)
> +{
> +       return __pte(pmd_val(pmd));
> +}
> +
>  /* Yields the page frame number (PFN) of a page table entry */
>  static inline unsigned long pte_pfn(pte_t pte)
>  {
> @@ -286,6 +291,21 @@ static inline pte_t pte_mkhuge(pte_t pte)
>         return pte;
>  }
>
> +#ifdef CONFIG_NUMA_BALANCING
> +/*
> + * See the comment in include/asm-generic/pgtable.h
> + */
> +static inline int pte_protnone(pte_t pte)
> +{
> +       return (pte_val(pte) & (_PAGE_PRESENT | _PAGE_PROT_NONE)) == _PAGE_PROT_NONE;
> +}
> +
> +static inline int pmd_protnone(pmd_t pmd)
> +{
> +       return pte_protnone(pmd_pte(pmd));
> +}
> +#endif
> +
>  /* Modify page protection bits */
>  static inline pte_t pte_modify(pte_t pte, pgprot_t newprot)
>  {
> --
> 2.25.1
>

Looks good to me.

Reviewed-by: Anup Patel <anup@brainfault.org>

Regards,
Anup

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH v4 5/5] riscv: Add numa support for riscv64 platform
  2020-10-06  0:17 ` [PATCH v4 5/5] riscv: Add numa support for riscv64 platform Atish Patra
@ 2020-10-22  5:40   ` Anup Patel
  0 siblings, 0 replies; 11+ messages in thread
From: Anup Patel @ 2020-10-22  5:40 UTC (permalink / raw)
  To: Atish Patra
  Cc: linux-kernel@vger.kernel.org List, Greentime Hu, Albert Ou,
	Andrew Morton, Anshuman Khandual, Arnd Bergmann, Catalin Marinas,
	David Hildenbrand, Greg Kroah-Hartman, Jia He, Jonathan Cameron,
	Kefeng Wang, linux-arch, linux-riscv, Mike Rapoport,
	Nicolas Saenz Julienne, Palmer Dabbelt, Paul Walmsley,
	Rafael J. Wysocki, Steven Price, Will Deacon, Zong Li,
	linux-arm-kernel

On Tue, Oct 6, 2020 at 5:48 AM Atish Patra <atish.patra@wdc.com> wrote:
>
> Use the generic numa implementation to add NUMA support for RISC-V.
> This is based on Greentime's patch[1] but modified to use generic NUMA
> implementation and few more fixes.
>
> [1] https://lkml.org/lkml/2020/1/10/233
>
> Co-developed-by: Greentime Hu <greentime.hu@sifive.com>
> Signed-off-by: Greentime Hu <greentime.hu@sifive.com>
> Signed-off-by: Atish Patra <atish.patra@wdc.com>
> ---
>  arch/riscv/Kconfig              | 31 ++++++++++++++++++++++++++++++-
>  arch/riscv/include/asm/mmzone.h | 13 +++++++++++++
>  arch/riscv/include/asm/numa.h   |  8 ++++++++
>  arch/riscv/include/asm/pci.h    | 14 ++++++++++++++
>  arch/riscv/kernel/setup.c       | 10 ++++++++--
>  arch/riscv/kernel/smpboot.c     | 12 +++++++++++-
>  arch/riscv/mm/init.c            |  4 +++-
>  7 files changed, 87 insertions(+), 5 deletions(-)
>  create mode 100644 arch/riscv/include/asm/mmzone.h
>  create mode 100644 arch/riscv/include/asm/numa.h
>
> diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
> index df18372861d8..7beb6ddb6eb1 100644
> --- a/arch/riscv/Kconfig
> +++ b/arch/riscv/Kconfig
> @@ -137,7 +137,7 @@ config PAGE_OFFSET
>         default 0xffffffe000000000 if 64BIT && MAXPHYSMEM_128GB
>
>  config ARCH_FLATMEM_ENABLE
> -       def_bool y
> +       def_bool !NUMA
>
>  config ARCH_SPARSEMEM_ENABLE
>         def_bool y
> @@ -295,6 +295,35 @@ config TUNE_GENERIC
>
>  endchoice
>
> +# Common NUMA Features
> +config NUMA
> +       bool "NUMA Memory Allocation and Scheduler Support"
> +       select GENERIC_ARCH_NUMA
> +       select OF_NUMA
> +       select ARCH_SUPPORTS_NUMA_BALANCING
> +       help
> +         Enable NUMA (Non-Uniform Memory Access) support.
> +
> +         The kernel will try to allocate memory used by a CPU on the
> +         local memory of the CPU and add some more NUMA awareness to the kernel.
> +
> +config NODES_SHIFT
> +       int "Maximum NUMA Nodes (as a power of 2)"
> +       range 1 10
> +       default "2"
> +       depends on NEED_MULTIPLE_NODES
> +       help
> +         Specify the maximum number of NUMA Nodes available on the target
> +         system.  Increases memory reserved to accommodate various tables.
> +
> +config USE_PERCPU_NUMA_NODE_ID
> +       def_bool y
> +       depends on NUMA
> +
> +config NEED_PER_CPU_EMBED_FIRST_CHUNK
> +       def_bool y
> +       depends on NUMA
> +
>  config RISCV_ISA_C
>         bool "Emit compressed instructions when building Linux"
>         default y
> diff --git a/arch/riscv/include/asm/mmzone.h b/arch/riscv/include/asm/mmzone.h
> new file mode 100644
> index 000000000000..fa17e01d9ab2
> --- /dev/null
> +++ b/arch/riscv/include/asm/mmzone.h
> @@ -0,0 +1,13 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef __ASM_MMZONE_H
> +#define __ASM_MMZONE_H
> +
> +#ifdef CONFIG_NUMA
> +
> +#include <asm/numa.h>
> +
> +extern struct pglist_data *node_data[];
> +#define NODE_DATA(nid)         (node_data[(nid)])
> +
> +#endif /* CONFIG_NUMA */
> +#endif /* __ASM_MMZONE_H */
> diff --git a/arch/riscv/include/asm/numa.h b/arch/riscv/include/asm/numa.h
> new file mode 100644
> index 000000000000..8c8cf4297cc3
> --- /dev/null
> +++ b/arch/riscv/include/asm/numa.h
> @@ -0,0 +1,8 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +#ifndef __ASM_NUMA_H
> +#define __ASM_NUMA_H
> +
> +#include <asm/topology.h>
> +#include <asm-generic/numa.h>
> +
> +#endif /* __ASM_NUMA_H */
> diff --git a/arch/riscv/include/asm/pci.h b/arch/riscv/include/asm/pci.h
> index 1c473a1bd986..658e112c3ce7 100644
> --- a/arch/riscv/include/asm/pci.h
> +++ b/arch/riscv/include/asm/pci.h
> @@ -32,6 +32,20 @@ static inline int pci_proc_domain(struct pci_bus *bus)
>         /* always show the domain in /proc */
>         return 1;
>  }
> +
> +#ifdef CONFIG_NUMA
> +
> +static inline int pcibus_to_node(struct pci_bus *bus)
> +{
> +       return dev_to_node(&bus->dev);
> +}
> +#ifndef cpumask_of_pcibus
> +#define cpumask_of_pcibus(bus) (pcibus_to_node(bus) == -1 ?            \
> +                                cpu_all_mask :                         \
> +                                cpumask_of_node(pcibus_to_node(bus)))
> +#endif
> +#endif /* CONFIG_NUMA */
> +
>  #endif  /* CONFIG_PCI */
>
>  #endif  /* _ASM_RISCV_PCI_H */
> diff --git a/arch/riscv/kernel/setup.c b/arch/riscv/kernel/setup.c
> index 07fa6d13367e..53a806a9cbaf 100644
> --- a/arch/riscv/kernel/setup.c
> +++ b/arch/riscv/kernel/setup.c
> @@ -101,13 +101,19 @@ void __init setup_arch(char **cmdline_p)
>
>  static int __init topology_init(void)
>  {
> -       int i;
> +       int i, ret;
> +
> +       for_each_online_node(i)
> +               register_one_node(i);
>
>         for_each_possible_cpu(i) {
>                 struct cpu *cpu = &per_cpu(cpu_devices, i);
>
>                 cpu->hotpluggable = cpu_has_hotplug(i);
> -               register_cpu(cpu, i);
> +               ret = register_cpu(cpu, i);
> +               if (unlikely(ret))
> +                       pr_warn("Warning: %s: register_cpu %d failed (%d)\n",
> +                              __func__, i, ret);
>         }
>
>         return 0;
> diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c
> index 96167d55ed98..5e276c25646f 100644
> --- a/arch/riscv/kernel/smpboot.c
> +++ b/arch/riscv/kernel/smpboot.c
> @@ -27,6 +27,7 @@
>  #include <asm/cpu_ops.h>
>  #include <asm/irq.h>
>  #include <asm/mmu_context.h>
> +#include <asm/numa.h>
>  #include <asm/tlbflush.h>
>  #include <asm/sections.h>
>  #include <asm/sbi.h>
> @@ -45,13 +46,18 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
>  {
>         int cpuid;
>         int ret;
> +       unsigned int curr_cpuid;
> +
> +       curr_cpuid = smp_processor_id();
> +       numa_store_cpu_info(curr_cpuid);
> +       numa_add_cpu(curr_cpuid);
>
>         /* This covers non-smp usecase mandated by "nosmp" option */
>         if (max_cpus == 0)
>                 return;
>
>         for_each_possible_cpu(cpuid) {
> -               if (cpuid == smp_processor_id())
> +               if (cpuid == curr_cpuid)
>                         continue;
>                 if (cpu_ops[cpuid]->cpu_prepare) {
>                         ret = cpu_ops[cpuid]->cpu_prepare(cpuid);
> @@ -59,6 +65,7 @@ void __init smp_prepare_cpus(unsigned int max_cpus)
>                                 continue;
>                 }
>                 set_cpu_present(cpuid, true);
> +               numa_store_cpu_info(cpuid);
>         }
>  }
>
> @@ -79,6 +86,7 @@ void __init setup_smp(void)
>                 if (hart == cpuid_to_hartid_map(0)) {
>                         BUG_ON(found_boot_cpu);
>                         found_boot_cpu = 1;
> +                       early_map_cpu_to_node(0, of_node_to_nid(dn));
>                         continue;
>                 }
>                 if (cpuid >= NR_CPUS) {
> @@ -88,6 +96,7 @@ void __init setup_smp(void)
>                 }
>
>                 cpuid_to_hartid_map(cpuid) = hart;
> +               early_map_cpu_to_node(cpuid, of_node_to_nid(dn));
>                 cpuid++;
>         }
>
> @@ -153,6 +162,7 @@ asmlinkage __visible void smp_callin(void)
>         current->active_mm = mm;
>
>         notify_cpu_starting(curr_cpuid);
> +       numa_add_cpu(curr_cpuid);
>         update_siblings_masks(curr_cpuid);
>         set_cpu_online(curr_cpuid, 1);
>
> diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c
> index 114c3966aadb..c4046e11d264 100644
> --- a/arch/riscv/mm/init.c
> +++ b/arch/riscv/mm/init.c
> @@ -20,6 +20,7 @@
>  #include <asm/soc.h>
>  #include <asm/io.h>
>  #include <asm/ptdump.h>
> +#include <asm/numa.h>
>
>  #include "../kernel/head.h"
>
> @@ -185,7 +186,6 @@ void __init setup_bootmem(void)
>
>         early_init_fdt_scan_reserved_mem();
>         memblock_allow_resize();
> -       memblock_dump_all();
>
>         for_each_memblock(memory, reg) {
>                 unsigned long start_pfn = memblock_region_memory_base_pfn(reg);
> @@ -570,9 +570,11 @@ void __init paging_init(void)
>
>  void __init misc_mem_init(void)
>  {
> +       arch_numa_init();
>         sparse_init();
>         zone_sizes_init();
>         resource_init();
> +       memblock_dump_all();
>  }
>
>  #ifdef CONFIG_SPARSEMEM_VMEMMAP
> --
> 2.25.1
>

Looks good to me.

Reviewed-by: Anup Patel <anup@brainfault.org>

Regards,
Anup

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, back to index

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-10-06  0:17 [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V Atish Patra
2020-10-06  0:17 ` [PATCH v4 1/5] numa: Move numa implementation to common code Atish Patra
2020-10-06  0:17 ` [PATCH v4 2/5] arm64, numa: Change the numa init functions name to be generic Atish Patra
2020-10-06  0:17 ` [PATCH v4 3/5] riscv: Separate memory init from paging init Atish Patra
2020-10-22  5:37   ` Anup Patel
2020-10-06  0:17 ` [PATCH v4 4/5] riscv: Add support pte_protnone and pmd_protnone if CONFIG_NUMA_BALANCING Atish Patra
2020-10-22  5:38   ` Anup Patel
2020-10-06  0:17 ` [PATCH v4 5/5] riscv: Add numa support for riscv64 platform Atish Patra
2020-10-22  5:40   ` Anup Patel
2020-10-13 20:19 ` [PATCH v4 0/5] Unify NUMA implementation between ARM64 & RISC-V Atish Patra
2020-10-19 21:29   ` Palmer Dabbelt

Linux-arch Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-arch/0 linux-arch/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-arch linux-arch/ https://lore.kernel.org/linux-arch \
		linux-arch@vger.kernel.org
	public-inbox-index linux-arch

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-arch


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git