Re: [patch 1/2] x86: add numa=possible command line option

Re: [patch 1/2] x86: add numa=possible command line option

[Shaohui Zheng]

> Adds a numa=possible=<N> command line option to set an additional N nodes
> as being possible for memory hotplug.  This set of possible nodes
> controls nr_node_ids and the sizes of several dynamically allocated node
> arrays.
> 
> This allows memory hotplug to create new nodes for newly added memory
> rather than binding it to existing nodes.
> 
> The first use-case for this will be node hotplug emulation which will use
> these possible nodes to create new nodes to test the memory hotplug
> callbacks and surrounding memory hotplug code.

It is the improved solution from thread http://lkml.org/lkml/2010/11/18/3,
our draft patch set all the nodes as possbile node, it wastes a lot of memory,
the command line numa=possible=<N> seems to be an acceptable, and it is a optimization
for our patch.

I like your active work attitude for the patch reviewing, it is real helpful to 
improve the patch quality.

the NUMA Hotplug Emulator is an overall solution for Node/CPU/Memory hotplug
emulation, it includes a lot of engineers' review comments and feedbacks in the
mailling list. We review and test each version very carefully, and make sure
each version is very stable and it includes in most of the feedbacks, so we deliver
each version very slow.

we do NOT want to split the emulator into many parts, and want to maintain them together.
How about add this patch into our emulator patchset, and add your sign-off?. Thanks.

Re: [patch 1/2] x86: add numa=possible command line option

[Shaohui Zheng]

> Adds a numa=possible=<N> command line option to set an additional N nodes
> as being possible for memory hotplug.  This set of possible nodes
> controls nr_node_ids and the sizes of several dynamically allocated node
> arrays.
> 
> This allows memory hotplug to create new nodes for newly added memory
> rather than binding it to existing nodes.
> 
> The first use-case for this will be node hotplug emulation which will use
> these possible nodes to create new nodes to test the memory hotplug
> callbacks and surrounding memory hotplug code.

It is the improved solution from thread http://lkml.org/lkml/2010/11/18/3,
our draft patch set all the nodes as possbile node, it wastes a lot of memory,
the command line numa=possible=<N> seems to be an acceptable, and it is a optimization
for our patch.

I like your active work attitude for the patch reviewing, it is real helpful to 
improve the patch quality.

the NUMA Hotplug Emulator is an overall solution for Node/CPU/Memory hotplug
emulation, it includes a lot of engineers' review comments and feedbacks in the
mailling list. We review and test each version very carefully, and make sure
each version is very stable and it includes in most of the feedbacks, so we deliver
each version very slow.

we do NOT want to split the emulator into many parts, and want to maintain them together.
How about add this patch into our emulator patchset, and add your sign-off?. Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 1/2] x86: add numa=possible command line option

[David Rientjes]

On Mon, 22 Nov 2010, Shaohui Zheng wrote:

> It is the improved solution from thread http://lkml.org/lkml/2010/11/18/3,
> our draft patch set all the nodes as possbile node, it wastes a lot of memory,
> the command line numa=possible=<N> seems to be an acceptable, and it is a optimization
> for our patch.
> 

node_possible_map is a generic nodemask used throughout the kernel, but 
its handling is highly dependent on the arch.  This patch enables the 
support for x86, I'd encourage anyone else interested in other archs to 
look into adding the support for it as well (perhaps you can do it for 
powerpc?).

> I like your active work attitude for the patch reviewing, it is real helpful to 
> improve the patch quality.
> 

Thanks, I'm happy to be involved!

Re: [patch 1/2] x86: add numa=possible command line option

[David Rientjes]

On Mon, 22 Nov 2010, Shaohui Zheng wrote:

> It is the improved solution from thread http://lkml.org/lkml/2010/11/18/3,
> our draft patch set all the nodes as possbile node, it wastes a lot of memory,
> the command line numa=possible=<N> seems to be an acceptable, and it is a optimization
> for our patch.
> 

node_possible_map is a generic nodemask used throughout the kernel, but 
its handling is highly dependent on the arch.  This patch enables the 
support for x86, I'd encourage anyone else interested in other archs to 
look into adding the support for it as well (perhaps you can do it for 
powerpc?).

> I like your active work attitude for the patch reviewing, it is real helpful to 
> improve the patch quality.
> 

Thanks, I'm happy to be involved!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 1/2] x86: add numa=possible command line option

[Américo Wang]

On Sun, Nov 21, 2010 at 01:46:07PM -0800, David Rientjes wrote:
>On Sun, 21 Nov 2010, Américo Wang wrote:
>> Also, numa=possible= is not as clear as numa=max=, for me at least.
>> 
>
>I like name, but it requires that you know how many nodes that system 
>already has.  In other words, numa=possible=4 allows you to specify that 4 
>additional nodes will be possible, but initially offline, for this or 
>other purposes.  numa=max=4 would be no-op if the system actually had 4 
>nodes.
>
>I chose numa=possible over numa=additional because it is more clearly tied 
>to node_possible_map, which is the only thing it modifies.

Okay, I thought "possible" means "max", but "possible" means "addtional" here.
It's clear for me now.

Thanks!

Re: [patch 1/2] x86: add numa=possible command line option

[Américo Wang]

On Sun, Nov 21, 2010 at 01:46:07PM -0800, David Rientjes wrote:
>On Sun, 21 Nov 2010, AmA(C)rico Wang wrote:
>> Also, numa=possible= is not as clear as numa=max=, for me at least.
>> 
>
>I like name, but it requires that you know how many nodes that system 
>already has.  In other words, numa=possible=4 allows you to specify that 4 
>additional nodes will be possible, but initially offline, for this or 
>other purposes.  numa=max=4 would be no-op if the system actually had 4 
>nodes.
>
>I chose numa=possible over numa=additional because it is more clearly tied 
>to node_possible_map, which is the only thing it modifies.

Okay, I thought "possible" means "max", but "possible" means "addtional" here.
It's clear for me now.

Thanks!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [patch 1/2] x86: add numa=possible command line option

[David Rientjes]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 805 bytes --]

On Sun, 21 Nov 2010, Américo Wang wrote:

> I am not sure how much value of making this dynamic,
> for CPU, we do this at compile time, i.e. NR_CPUS,
> so how about NR_NODES?
> 

This is outside the scope of node hotplug emulation, it needs to be built 
on top of whatever the kernel implements.

> Also, numa=possible= is not as clear as numa=max=, for me at least.
> 

I like name, but it requires that you know how many nodes that system 
already has.  In other words, numa=possible=4 allows you to specify that 4 
additional nodes will be possible, but initially offline, for this or 
other purposes.  numa=max=4 would be no-op if the system actually had 4 
nodes.

I chose numa=possible over numa=additional because it is more clearly tied 
to node_possible_map, which is the only thing it modifies.

Re: [patch 1/2] x86: add numa=possible command line option

[David Rientjes]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 807 bytes --]

On Sun, 21 Nov 2010, AmA(C)rico Wang wrote:

> I am not sure how much value of making this dynamic,
> for CPU, we do this at compile time, i.e. NR_CPUS,
> so how about NR_NODES?
> 

This is outside the scope of node hotplug emulation, it needs to be built 
on top of whatever the kernel implements.

> Also, numa=possible= is not as clear as numa=max=, for me at least.
> 

I like name, but it requires that you know how many nodes that system 
already has.  In other words, numa=possible=4 allows you to specify that 4 
additional nodes will be possible, but initially offline, for this or 
other purposes.  numa=max=4 would be no-op if the system actually had 4 
nodes.

I chose numa=possible over numa=additional because it is more clearly tied 
to node_possible_map, which is the only thing it modifies.

Re: [patch 1/2] x86: add numa=possible command line option

[Américo Wang]

Hi, David

On Sat, Nov 20, 2010 at 06:28:31PM -0800, David Rientjes wrote:
>Adds a numa=possible=<N> command line option to set an additional N nodes
>as being possible for memory hotplug.  This set of possible nodes
>controls nr_node_ids and the sizes of several dynamically allocated node
>arrays.
>
>This allows memory hotplug to create new nodes for newly added memory
>rather than binding it to existing nodes.
>
>The first use-case for this will be node hotplug emulation which will use
>these possible nodes to create new nodes to test the memory hotplug
>callbacks and surrounding memory hotplug code.
>


I am not sure how much value of making this dynamic,
for CPU, we do this at compile time, i.e. NR_CPUS,
so how about NR_NODES?

Also, numa=possible= is not as clear as numa=max=, for me at least.

Thanks.

Re: [patch 1/2] x86: add numa=possible command line option

[Américo Wang]

Hi, David

On Sat, Nov 20, 2010 at 06:28:31PM -0800, David Rientjes wrote:
>Adds a numa=possible=<N> command line option to set an additional N nodes
>as being possible for memory hotplug.  This set of possible nodes
>controls nr_node_ids and the sizes of several dynamically allocated node
>arrays.
>
>This allows memory hotplug to create new nodes for newly added memory
>rather than binding it to existing nodes.
>
>The first use-case for this will be node hotplug emulation which will use
>these possible nodes to create new nodes to test the memory hotplug
>callbacks and surrounding memory hotplug code.
>


I am not sure how much value of making this dynamic,
for CPU, we do this at compile time, i.e. NR_CPUS,
so how about NR_NODES?

Also, numa=possible= is not as clear as numa=max=, for me at least.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

[patch 1/2] x86: add numa=possible command line option

[David Rientjes]

Adds a numa=possible=<N> command line option to set an additional N nodes
as being possible for memory hotplug.  This set of possible nodes
controls nr_node_ids and the sizes of several dynamically allocated node
arrays.

This allows memory hotplug to create new nodes for newly added memory
rather than binding it to existing nodes.

The first use-case for this will be node hotplug emulation which will use
these possible nodes to create new nodes to test the memory hotplug
callbacks and surrounding memory hotplug code.

Signed-off-by: David Rientjes <rientjes@google.com>
---
 Documentation/x86/x86_64/boot-options.txt |    4 ++++
 arch/x86/mm/numa_64.c                     |   18 +++++++++++++++---
 2 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/Documentation/x86/x86_64/boot-options.txt b/Documentation/x86/x86_64/boot-options.txt
--- a/Documentation/x86/x86_64/boot-options.txt
+++ b/Documentation/x86/x86_64/boot-options.txt
@@ -174,6 +174,10 @@ NUMA
 		If given as an integer, fills all system RAM with N fake nodes
 		interleaved over physical nodes.
 
+  numa=possible=<N>
+		Sets an additional N nodes as being possible for memory
+		hotplug.
+
 ACPI
 
   acpi=off	Don't enable ACPI
diff --git a/arch/x86/mm/numa_64.c b/arch/x86/mm/numa_64.c
--- a/arch/x86/mm/numa_64.c
+++ b/arch/x86/mm/numa_64.c
@@ -33,6 +33,7 @@ s16 apicid_to_node[MAX_LOCAL_APIC] __cpuinitdata = {
 int numa_off __initdata;
 static unsigned long __initdata nodemap_addr;
 static unsigned long __initdata nodemap_size;
+static unsigned long __initdata numa_possible_nodes;
 
 /*
  * Map cpu index to node index
@@ -611,7 +612,7 @@ void __init initmem_init(unsigned long start_pfn, unsigned long last_pfn,
 
 #ifdef CONFIG_NUMA_EMU
 	if (cmdline && !numa_emulation(start_pfn, last_pfn, acpi, k8))
-		return;
+		goto out;
 	nodes_clear(node_possible_map);
 	nodes_clear(node_online_map);
 #endif
@@ -619,14 +620,14 @@ void __init initmem_init(unsigned long start_pfn, unsigned long last_pfn,
 #ifdef CONFIG_ACPI_NUMA
 	if (!numa_off && acpi && !acpi_scan_nodes(start_pfn << PAGE_SHIFT,
 						  last_pfn << PAGE_SHIFT))
-		return;
+		goto out;
 	nodes_clear(node_possible_map);
 	nodes_clear(node_online_map);
 #endif
 
 #ifdef CONFIG_K8_NUMA
 	if (!numa_off && k8 && !k8_scan_nodes())
-		return;
+		goto out;
 	nodes_clear(node_possible_map);
 	nodes_clear(node_online_map);
 #endif
@@ -646,6 +647,15 @@ void __init initmem_init(unsigned long start_pfn, unsigned long last_pfn,
 		numa_set_node(i, 0);
 	memblock_x86_register_active_regions(0, start_pfn, last_pfn);
 	setup_node_bootmem(0, start_pfn << PAGE_SHIFT, last_pfn << PAGE_SHIFT);
+out: __maybe_unused
+	for (i = 0; i < numa_possible_nodes; i++) {
+		int nid;
+
+		nid = first_unset_node(node_possible_map);
+		if (nid == MAX_NUMNODES)
+			break;
+		node_set(nid, node_possible_map);
+	}
 }
 
 unsigned long __init numa_free_all_bootmem(void)
@@ -675,6 +685,8 @@ static __init int numa_setup(char *opt)
 	if (!strncmp(opt, "noacpi", 6))
 		acpi_numa = -1;
 #endif
+	if (!strncmp(opt, "possible=", 9))
+		numa_possible_nodes = simple_strtoul(opt + 9, NULL, 0);
 	return 0;
 }
 early_param("numa", numa_setup);

[patch 1/2] x86: add numa=possible command line option

[David Rientjes]

Adds a numa=possible=<N> command line option to set an additional N nodes
as being possible for memory hotplug.  This set of possible nodes
controls nr_node_ids and the sizes of several dynamically allocated node
arrays.

This allows memory hotplug to create new nodes for newly added memory
rather than binding it to existing nodes.

The first use-case for this will be node hotplug emulation which will use
these possible nodes to create new nodes to test the memory hotplug
callbacks and surrounding memory hotplug code.

Signed-off-by: David Rientjes <rientjes@google.com>
---
 Documentation/x86/x86_64/boot-options.txt |    4 ++++
 arch/x86/mm/numa_64.c                     |   18 +++++++++++++++---
 2 files changed, 19 insertions(+), 3 deletions(-)

diff --git a/Documentation/x86/x86_64/boot-options.txt b/Documentation/x86/x86_64/boot-options.txt
--- a/Documentation/x86/x86_64/boot-options.txt
+++ b/Documentation/x86/x86_64/boot-options.txt
@@ -174,6 +174,10 @@ NUMA
 		If given as an integer, fills all system RAM with N fake nodes
 		interleaved over physical nodes.
 
+  numa=possible=<N>
+		Sets an additional N nodes as being possible for memory
+		hotplug.
+
 ACPI
 
   acpi=off	Don't enable ACPI
diff --git a/arch/x86/mm/numa_64.c b/arch/x86/mm/numa_64.c
--- a/arch/x86/mm/numa_64.c
+++ b/arch/x86/mm/numa_64.c
@@ -33,6 +33,7 @@ s16 apicid_to_node[MAX_LOCAL_APIC] __cpuinitdata = {
 int numa_off __initdata;
 static unsigned long __initdata nodemap_addr;
 static unsigned long __initdata nodemap_size;
+static unsigned long __initdata numa_possible_nodes;
 
 /*
  * Map cpu index to node index
@@ -611,7 +612,7 @@ void __init initmem_init(unsigned long start_pfn, unsigned long last_pfn,
 
 #ifdef CONFIG_NUMA_EMU
 	if (cmdline && !numa_emulation(start_pfn, last_pfn, acpi, k8))
-		return;
+		goto out;
 	nodes_clear(node_possible_map);
 	nodes_clear(node_online_map);
 #endif
@@ -619,14 +620,14 @@ void __init initmem_init(unsigned long start_pfn, unsigned long last_pfn,
 #ifdef CONFIG_ACPI_NUMA
 	if (!numa_off && acpi && !acpi_scan_nodes(start_pfn << PAGE_SHIFT,
 						  last_pfn << PAGE_SHIFT))
-		return;
+		goto out;
 	nodes_clear(node_possible_map);
 	nodes_clear(node_online_map);
 #endif
 
 #ifdef CONFIG_K8_NUMA
 	if (!numa_off && k8 && !k8_scan_nodes())
-		return;
+		goto out;
 	nodes_clear(node_possible_map);
 	nodes_clear(node_online_map);
 #endif
@@ -646,6 +647,15 @@ void __init initmem_init(unsigned long start_pfn, unsigned long last_pfn,
 		numa_set_node(i, 0);
 	memblock_x86_register_active_regions(0, start_pfn, last_pfn);
 	setup_node_bootmem(0, start_pfn << PAGE_SHIFT, last_pfn << PAGE_SHIFT);
+out: __maybe_unused
+	for (i = 0; i < numa_possible_nodes; i++) {
+		int nid;
+
+		nid = first_unset_node(node_possible_map);
+		if (nid == MAX_NUMNODES)
+			break;
+		node_set(nid, node_possible_map);
+	}
 }
 
 unsigned long __init numa_free_all_bootmem(void)
@@ -675,6 +685,8 @@ static __init int numa_setup(char *opt)
 	if (!strncmp(opt, "noacpi", 6))
 		acpi_numa = -1;
 #endif
+	if (!strncmp(opt, "possible=", 9))
+		numa_possible_nodes = simple_strtoul(opt + 9, NULL, 0);
 	return 0;
 }
 early_param("numa", numa_setup);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom policy in Canada: sign http://dissolvethecrtc.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>