[PATCH 1/1] memhp: Add hot-added memory ranges to memblock before allocate node_data for a node.

[PATCH 1/1] memhp: Add hot-added memory ranges to memblock before allocate node_data for a node.

[Tang Chen]

The commit below adds hot-added memory range to memblock, after
creating pgdat for new node.

commit f9126ab9241f66562debf69c2c9d8fee32ddcc53
Author: Xishi Qiu <qiuxishi@huawei.com>
Date:   Fri Aug 14 15:35:16 2015 -0700

    memory-hotplug: fix wrong edge when hot add a new node

But there is a problem:

add_memory()
|--> hotadd_new_pgdat()
     |--> free_area_init_node()
          |--> get_pfn_range_for_nid()
               |--> find start_pfn and end_pfn in memblock
|--> ......
|--> memblock_add_node(start, size, nid)    --------    Here, just too late.

get_pfn_range_for_nid() will find that start_pfn and end_pfn are both 0.
As a result, when adding memory, dmesg will give the following wrong message.

[ 2007.577000] Initmem setup node 5 [mem 0x0000000000000000-0xffffffffffffffff]
[ 2007.584000] On node 5 totalpages: 0
[ 2007.585000] Built 5 zonelists in Node order, mobility grouping on.  Total pages: 32588823
[ 2007.594000] Policy zone: Normal
[ 2007.598000] init_memory_mapping: [mem 0x60000000000-0x607ffffffff]

The solution is simple, just add the memory range to memblock a little earlier,
before hotadd_new_pgdat().

Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
---
 mm/memory_hotplug.c | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 6da82bc..9b78aff 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1248,6 +1248,14 @@ int __ref add_memory(int nid, u64 start, u64 size)
 
 	mem_hotplug_begin();
 
+	/*
+	 * Add new range to memblock so that when hotadd_new_pgdat() is called to
+	 * allocate new pgdat, get_pfn_range_for_nid() will be able to find this
+	 * new range and calculate total pages correctly. The range will be remove
+	 * at hot-remove time.
+	 */
+	memblock_add_node(start, size, nid);
+
 	new_node = !node_online(nid);
 	if (new_node) {
 		pgdat = hotadd_new_pgdat(nid, start);
@@ -1277,7 +1285,6 @@ int __ref add_memory(int nid, u64 start, u64 size)
 
 	/* create new memmap entry */
 	firmware_map_add_hotplug(start, start + size, "System RAM");
-	memblock_add_node(start, size, nid);
 
 	goto out;
 
@@ -1286,6 +1293,7 @@ error:
 	if (new_pgdat)
 		rollback_node_hotadd(nid, pgdat);
 	release_memory_resource(res);
+	memblock_remove(start, size);
 
 out:
 	mem_hotplug_done();
-- 
1.8.3.1

Re: [PATCH 1/1] memhp: Add hot-added memory ranges to memblock before allocate node_data for a node.

[Xishi Qiu]

On 2015/8/24 1:06, Tang Chen wrote:

> The commit below adds hot-added memory range to memblock, after
> creating pgdat for new node.
> 
> commit f9126ab9241f66562debf69c2c9d8fee32ddcc53
> Author: Xishi Qiu <qiuxishi@huawei.com>
> Date:   Fri Aug 14 15:35:16 2015 -0700
> 
>     memory-hotplug: fix wrong edge when hot add a new node
> 
> But there is a problem:
> 
> add_memory()
> |--> hotadd_new_pgdat()
>      |--> free_area_init_node()
>           |--> get_pfn_range_for_nid()
>                |--> find start_pfn and end_pfn in memblock
> |--> ......
> |--> memblock_add_node(start, size, nid)    --------    Here, just too late.
> 
> get_pfn_range_for_nid() will find that start_pfn and end_pfn are both 0.
> As a result, when adding memory, dmesg will give the following wrong message.
> 
> [ 2007.577000] Initmem setup node 5 [mem 0x0000000000000000-0xffffffffffffffff]
> [ 2007.584000] On node 5 totalpages: 0
> [ 2007.585000] Built 5 zonelists in Node order, mobility grouping on.  Total pages: 32588823
> [ 2007.594000] Policy zone: Normal
> [ 2007.598000] init_memory_mapping: [mem 0x60000000000-0x607ffffffff]
> 
> The solution is simple, just add the memory range to memblock a little earlier,
> before hotadd_new_pgdat().
> 
> Signed-off-by: Tang Chen <tangchen@cn.fujitsu.com>
> ---
>  mm/memory_hotplug.c | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index 6da82bc..9b78aff 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1248,6 +1248,14 @@ int __ref add_memory(int nid, u64 start, u64 size)
>  
>  	mem_hotplug_begin();
>  
> +	/*
> +	 * Add new range to memblock so that when hotadd_new_pgdat() is called to
> +	 * allocate new pgdat, get_pfn_range_for_nid() will be able to find this
> +	 * new range and calculate total pages correctly. The range will be remove
> +	 * at hot-remove time.
> +	 */
> +	memblock_add_node(start, size, nid);
> +

Hi Tang,

Looks fine to me.

If we add memblock_add_node() here, we should reset the managed pages and present pages,
so please revert my patch which Andrew has already merged into mm-tree.
"[PATCH 2/2] memory-hotplug: remove reset_node_managed_pages() and reset_node_managed_pages() in hotadd_new_pgdat()"

Thanks,
Xishi Qiu

>  	new_node = !node_online(nid);
>  	if (new_node) {
>  		pgdat = hotadd_new_pgdat(nid, start);
> @@ -1277,7 +1285,6 @@ int __ref add_memory(int nid, u64 start, u64 size)
>  
>  	/* create new memmap entry */
>  	firmware_map_add_hotplug(start, start + size, "System RAM");
> -	memblock_add_node(start, size, nid);
>  
>  	goto out;
>  
> @@ -1286,6 +1293,7 @@ error:
>  	if (new_pgdat)
>  		rollback_node_hotadd(nid, pgdat);
>  	release_memory_resource(res);
> +	memblock_remove(start, size);
>  
>  out:
>  	mem_hotplug_done();

Re: [PATCH 1/1] memhp: Add hot-added memory ranges to memblock before allocate node_data for a node.

[Xishi Qiu]

On 2015/8/24 17:22, Xishi Qiu wrote:

> On 2015/8/24 1:06, Tang Chen wrote:
> 
>> The commit below adds hot-added memory range to memblock, after
>> creating pgdat for new node.
>>
>> commit f9126ab9241f66562debf69c2c9d8fee32ddcc53
>> Author: Xishi Qiu <qiuxishi@huawei.com>
>> Date:   Fri Aug 14 15:35:16 2015 -0700
>>
>>     memory-hotplug: fix wrong edge when hot add a new node
>>
>> But there is a problem:
>>
>> add_memory()
>> |--> hotadd_new_pgdat()
>>      |--> free_area_init_node()
>>           |--> get_pfn_range_for_nid()
>>                |--> find start_pfn and end_pfn in memblock
>> |--> ......
>> |--> memblock_add_node(start, size, nid)    --------    Here, just too late.
>>
>> get_pfn_range_for_nid() will find that start_pfn and end_pfn are both 0.
>> As a result, when adding memory, dmesg will give the following wrong message.
>>

Hi Tang,

Another question, if we add cpu first, there will be print error too.

cpu_up()
	try_online_node()
		hotadd_new_pgdat()

So how about just skip the print if the size is empty or just print 
"node xx is empty now, will update when online memory"? 

Thanks,
Xishi Qiu

>> [ 2007.577000] Initmem setup node 5 [mem 0x0000000000000000-0xffffffffffffffff]
>> [ 2007.584000] On node 5 totalpages: 0
>> [ 2007.585000] Built 5 zonelists in Node order, mobility grouping on.  Total pages: 32588823
>> [ 2007.594000] Policy zone: Normal
>> [ 2007.598000] init_memory_mapping: [mem 0x60000000000-0x607ffffffff]
>>

Re: [PATCH 1/1] memhp: Add hot-added memory ranges to memblock before allocate node_data for a node.

[Tang Chen]

On 08/25/2015 09:09 AM, Xishi Qiu wrote:
> On 2015/8/24 17:22, Xishi Qiu wrote:
>
>> On 2015/8/24 1:06, Tang Chen wrote:
>>
>>> The commit below adds hot-added memory range to memblock, after
>>> creating pgdat for new node.
>>>
>>> commit f9126ab9241f66562debf69c2c9d8fee32ddcc53
>>> Author: Xishi Qiu <qiuxishi@huawei.com>
>>> Date:   Fri Aug 14 15:35:16 2015 -0700
>>>
>>>      memory-hotplug: fix wrong edge when hot add a new node
>>>
>>> But there is a problem:
>>>
>>> add_memory()
>>> |--> hotadd_new_pgdat()
>>>       |--> free_area_init_node()
>>>            |--> get_pfn_range_for_nid()
>>>                 |--> find start_pfn and end_pfn in memblock
>>> |--> ......
>>> |--> memblock_add_node(start, size, nid)    --------    Here, just too late.
>>>
>>> get_pfn_range_for_nid() will find that start_pfn and end_pfn are both 0.
>>> As a result, when adding memory, dmesg will give the following wrong message.
>>>
> Hi Tang,
>
> Another question, if we add cpu first, there will be print error too.
>
> cpu_up()
> 	try_online_node()
> 		hotadd_new_pgdat()
>
> So how about just skip the print if the size is empty or just print
> "node xx is empty now, will update when online memory"?

As Liu Jiang said, memory-less node is not supported on x86 now.
And he is working on it.

Please refer to https://lkml.org/lkml/2015/8/16/130.

About your question, now, node could only be onlined when it has some 
memory.
So the printed message is also about memory, and sis put in 
hotadd_new_pgdat() .
I think the author of the code didn't think about online a node when a 
CPU is up.

But now, memory-less will be supported. So, I think, as you said, the 
message should
be modified.

But how it will go, I think we should refer to Liu Jiang's patch, and 
make a decision.

Thanks.

>
> Thanks,
> Xishi Qiu
>
>>> [ 2007.577000] Initmem setup node 5 [mem 0x0000000000000000-0xffffffffffffffff]
>>> [ 2007.584000] On node 5 totalpages: 0
>>> [ 2007.585000] Built 5 zonelists in Node order, mobility grouping on.  Total pages: 32588823
>>> [ 2007.594000] Policy zone: Normal
>>> [ 2007.598000] init_memory_mapping: [mem 0x60000000000-0x607ffffffff]
>>>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> .
>