[RFC PATCH 0/2] arm64: memory-hotplug: Add Memory Hotplug support

[RFC PATCH 0/2] arm64: memory-hotplug: Add Memory Hotplug support

[Scott Branden]

This patchset is sent for comment to add memory hotplug support for ARM64
based platforms.  It follows hotplug code added for other architectures
in the linux kernel.

I tried testing the memory hotplug feature following documentation from
Documentation/memory-hotplug.txt.  I don't think it is working as expected
- see below:

To add memory to the system I did the following:
echo 0x400000000 > /sys/devices/system/memory/probe

The memory is displayed as system ram:
cat /proc/iomem:
74000000-77ffffff : System RAM
  74080000-748dffff : Kernel code
  74950000-749d2fff : Kernel data
400000000-43fffffff : System RAM

But does not seem to be added to the kernel memory.
/proc/meminfo did not change.

What else needs to be done so the memory is added to the kernel memory
pool for normal allocation?

Scott Branden (2):
  arm64: memory-hotplug: Add MEMORY_HOTPLUG, MEMORY_HOTREMOVE,
    MEMORY_PROBE
  arm64: defconfig: enable MEMORY_HOTPLUG config options

 arch/arm64/Kconfig           | 10 ++++++++++
 arch/arm64/configs/defconfig |  3 +++
 arch/arm64/mm/init.c         | 42 ++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 55 insertions(+)

-- 
2.5.0

[RFC PATCH 1/2] arm64: memory-hotplug: Add MEMORY_HOTPLUG, MEMORY_HOTREMOVE, MEMORY_PROBE

[Scott Branden]

Add memory-hotplug support for ARM64 platform.

This requires addition of
ARCH_ENABLE_MEMORY_HOTPLUG and ARCH_ENABLE_MEMORY_HOTREMOVE config options.

MEMORY_PROBE config option is added to support
/sys/devices/system/memory/probe functionality.

In addition architecture specific arch_add_memory and
arch_remove memory management functions are added.

Signed-off-by: Scott Branden <scott.branden@broadcom.com>
---
 arch/arm64/Kconfig   | 10 ++++++++++
 arch/arm64/mm/init.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 52 insertions(+)

diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 969ef88..2482fdd 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -576,6 +576,12 @@ config HOTPLUG_CPU
 	  Say Y here to experiment with turning CPUs off and on.  CPUs
 	  can be controlled through /sys/devices/system/cpu.
 
+config ARCH_ENABLE_MEMORY_HOTPLUG
+	def_bool y
+
+config ARCH_ENABLE_MEMORY_HOTREMOVE
+	def_bool y
+
 # Common NUMA Features
 config NUMA
 	bool "Numa Memory Allocation and Scheduler Support"
@@ -646,6 +652,10 @@ config ARCH_HAS_CACHE_LINE_SIZE
 
 source "mm/Kconfig"
 
+config ARCH_MEMORY_PROBE
+	def_bool y
+	depends on MEMORY_HOTPLUG
+
 config SECCOMP
 	bool "Enable seccomp to safely compute untrusted bytecode"
 	---help---
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 212c4d1..687d087 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -536,3 +536,45 @@ static int __init register_mem_limit_dumper(void)
 	return 0;
 }
 __initcall(register_mem_limit_dumper);
+
+#ifdef CONFIG_MEMORY_HOTPLUG
+int arch_add_memory(int nid, u64 start, u64 size, bool for_device)
+{
+	pg_data_t *pgdat;
+	struct zone *zone;
+	unsigned long start_pfn = start >> PAGE_SHIFT;
+	unsigned long nr_pages = size >> PAGE_SHIFT;
+	int ret;
+
+	pgdat = NODE_DATA(nid);
+
+	zone = pgdat->node_zones +
+		zone_for_memory(nid, start, size, ZONE_NORMAL, for_device);
+	ret = __add_pages(nid, zone, start_pfn, nr_pages);
+
+	if (ret)
+		pr_warn("%s: Problem encountered in __add_pages() ret=%d\n",
+			__func__, ret);
+
+	return ret;
+}
+
+#ifdef CONFIG_MEMORY_HOTREMOVE
+int arch_remove_memory(u64 start, u64 size)
+{
+	unsigned long start_pfn = start >> PAGE_SHIFT;
+	unsigned long nr_pages = size >> PAGE_SHIFT;
+	struct zone *zone;
+	int ret;
+
+	zone = page_zone(pfn_to_page(start_pfn));
+	ret = __remove_pages(zone, start_pfn, nr_pages);
+	if (ret)
+		pr_warn("%s: Problem encountered in __remove_pages() ret=%d\n",
+			__func__, ret);
+
+	return ret;
+}
+#endif
+#endif
+
-- 
2.5.0

[RFC PATCH 2/2] arm64: defconfig: enable MEMORY_HOTPLUG config options

[Scott Branden]

Enable memory hotplug config options to add/remove memory.

Signed-off-by: Scott Branden <scott.branden@broadcom.com>
---
 arch/arm64/configs/defconfig | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index dab2cb0..e801b37 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -73,6 +73,9 @@ CONFIG_PCIE_ARMADA_8K=y
 CONFIG_ARM64_VA_BITS_48=y
 CONFIG_SCHED_MC=y
 CONFIG_PREEMPT=y
+CONFIG_MEMORY_HOTPLUG=y
+CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE=y
+CONFIG_MEMORY_HOTREMOVE=y
 CONFIG_KSM=y
 CONFIG_TRANSPARENT_HUGEPAGE=y
 CONFIG_CMA=y
-- 
2.5.0

Re: [RFC PATCH 0/2] arm64: memory-hotplug: Add Memory Hotplug support

[Xishi Qiu]

On 2016/12/2 8:19, Scott Branden wrote:

> This patchset is sent for comment to add memory hotplug support for ARM64
> based platforms.  It follows hotplug code added for other architectures
> in the linux kernel.
> 
> I tried testing the memory hotplug feature following documentation from
> Documentation/memory-hotplug.txt.  I don't think it is working as expected
> - see below:
> 
> To add memory to the system I did the following:
> echo 0x400000000 > /sys/devices/system/memory/probe
> 
> The memory is displayed as system ram:
> cat /proc/iomem:
> 74000000-77ffffff : System RAM
>   74080000-748dffff : Kernel code
>   74950000-749d2fff : Kernel data
> 400000000-43fffffff : System RAM
> 
> But does not seem to be added to the kernel memory.
> /proc/meminfo did not change.
> 
> What else needs to be done so the memory is added to the kernel memory
> pool for normal allocation?
> 

Hi Scott,

Do you mean it still don't support hod-add after apply this patchset?

Thanks,
Xishi Qiu

> Scott Branden (2):
>   arm64: memory-hotplug: Add MEMORY_HOTPLUG, MEMORY_HOTREMOVE,
>     MEMORY_PROBE
>   arm64: defconfig: enable MEMORY_HOTPLUG config options
> 
>  arch/arm64/Kconfig           | 10 ++++++++++
>  arch/arm64/configs/defconfig |  3 +++
>  arch/arm64/mm/init.c         | 42 ++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 55 insertions(+)
> 

Re: [RFC PATCH 0/2] arm64: memory-hotplug: Add Memory Hotplug support

[Scott Branden]

Hi Xishi,

Thanks for the reply - please see comments below.

On 16-12-01 05:49 PM, Xishi Qiu wrote:
> On 2016/12/2 8:19, Scott Branden wrote:
>
>> This patchset is sent for comment to add memory hotplug support for ARM64
>> based platforms.  It follows hotplug code added for other architectures
>> in the linux kernel.
>>
>> I tried testing the memory hotplug feature following documentation from
>> Documentation/memory-hotplug.txt.  I don't think it is working as expected
>> - see below:
>>
>> To add memory to the system I did the following:
>> echo 0x400000000 > /sys/devices/system/memory/probe
>>
>> The memory is displayed as system ram:
>> cat /proc/iomem:
>> 74000000-77ffffff : System RAM
>>   74080000-748dffff : Kernel code
>>   74950000-749d2fff : Kernel data
>> 400000000-43fffffff : System RAM
>>
>> But does not seem to be added to the kernel memory.
>> /proc/meminfo did not change.
>>
>> What else needs to be done so the memory is added to the kernel memory
>> pool for normal allocation?
>>
>
> Hi Scott,
>
> Do you mean it still don't support hod-add after apply this patchset?

After applying the patch it appears to partially support hot-add. 
Please let me know if you think it is working as expected?

The memory probe functions in that the memory is registered with the 
system and shows up in /proc/iomem.  But, the memory is not available in 
/proc/meminfo.  Do you think something else needs to be adjusted for 
ARM64 to hotadd the memory

I just found another clue:
under /sys/devices/system/memory I only see one memory entry (before or 
after I try to hotadd additional memory).

/sys/devices/system/memory # ls
auto_online_blocks  memory0            uevent
block_size_bytes    probe

In arch/arm64/include/asm/sparsemem.h if I change SECTION_SIZE_BITS from 
30 to 28 and recompile I get the following:
/sys/devices/system/memory # ls
auto_online_blocks  memory7            uevent
block_size_bytes    probe


In arch/arm64/include/asm/sparsemem.h if I change SECTION_SIZE_BITS from 
30 to 27 and recompile I get the following:
/sys/devices/system/memory # ls
auto_online_blocks  memory14            uevent
block_size_bytes    probe

If looks to me like something is not working properly in the ARM64 
implementation.  I should expect to see multiple memoryX entries under 
/sys/devices/system/memory?



>
> Thanks,
> Xishi Qiu
>
>> Scott Branden (2):
>>   arm64: memory-hotplug: Add MEMORY_HOTPLUG, MEMORY_HOTREMOVE,
>>     MEMORY_PROBE
>>   arm64: defconfig: enable MEMORY_HOTPLUG config options
>>
>>  arch/arm64/Kconfig           | 10 ++++++++++
>>  arch/arm64/configs/defconfig |  3 +++
>>  arch/arm64/mm/init.c         | 42 ++++++++++++++++++++++++++++++++++++++++++
>>  3 files changed, 55 insertions(+)
>>
>
>
>

Re: [RFC PATCH 0/2] arm64: memory-hotplug: Add Memory Hotplug support

[Xishi Qiu]

On 2016/12/2 10:38, Scott Branden wrote:

> Hi Xishi,
> 
> Thanks for the reply - please see comments below.
> 
> On 16-12-01 05:49 PM, Xishi Qiu wrote:
>> On 2016/12/2 8:19, Scott Branden wrote:
>>
>>> This patchset is sent for comment to add memory hotplug support for ARM64
>>> based platforms.  It follows hotplug code added for other architectures
>>> in the linux kernel.
>>>
>>> I tried testing the memory hotplug feature following documentation from
>>> Documentation/memory-hotplug.txt.  I don't think it is working as expected
>>> - see below:
>>>
>>> To add memory to the system I did the following:
>>> echo 0x400000000 > /sys/devices/system/memory/probe
>>>
>>> The memory is displayed as system ram:
>>> cat /proc/iomem:
>>> 74000000-77ffffff : System RAM
>>>   74080000-748dffff : Kernel code
>>>   74950000-749d2fff : Kernel data
>>> 400000000-43fffffff : System RAM
>>>
>>> But does not seem to be added to the kernel memory.
>>> /proc/meminfo did not change.
>>>
>>> What else needs to be done so the memory is added to the kernel memory
>>> pool for normal allocation?
>>>
>>
>> Hi Scott,
>>
>> Do you mean it still don't support hod-add after apply this patchset?
> 
> After applying the patch it appears to partially support hot-add. Please let me know if you think it is working as expected?
> 
> The memory probe functions in that the memory is registered with the system and shows up in /proc/iomem.  But, the memory is not available in /proc/meminfo.  Do you think something else needs to be adjusted for ARM64 to hotadd the memory
> 
> I just found another clue:
> under /sys/devices/system/memory I only see one memory entry (before or after I try to hotadd additional memory).
> 
> /sys/devices/system/memory # ls
> auto_online_blocks  memory0            uevent
> block_size_bytes    probe
> 
> In arch/arm64/include/asm/sparsemem.h if I change SECTION_SIZE_BITS from 30 to 28 and recompile I get the following:
> /sys/devices/system/memory # ls
> auto_online_blocks  memory7            uevent
> block_size_bytes    probe
> 
> 
> In arch/arm64/include/asm/sparsemem.h if I change SECTION_SIZE_BITS from 30 to 27 and recompile I get the following:
> /sys/devices/system/memory # ls
> auto_online_blocks  memory14            uevent
> block_size_bytes    probe
> 
> If looks to me like something is not working properly in the ARM64 implementation.  I should expect to see multiple memoryX entries under /sys/devices/system/memory?
> 

Hi Scott,

1. Do you enable the following configs?
CONFIG_SPARSEMEM
MEMORY_HOTPLUG
CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE

2. I find you missed create mapping in arch_add_memory(), and x86 has it.

3. We will add memblock first, so pfn_valid() maybe always return true(in the
following function), and this will lead __add_section() failed. Please check
it.

int pfn_valid(unsigned long pfn)
{
	return (pfn & PFN_MASK) == pfn && memblock_is_memory(pfn << PAGE_SHIFT);
}

add_memory
  add_memory_resource
    memblock_add_node
      arch_add_memory
        __add_pages
          __add_section
            pfn_valid

Thanks,
Xishi Qiu

> 
> 
>>
>> Thanks,
>> Xishi Qiu
>>
>>> Scott Branden (2):
>>>   arm64: memory-hotplug: Add MEMORY_HOTPLUG, MEMORY_HOTREMOVE,
>>>     MEMORY_PROBE
>>>   arm64: defconfig: enable MEMORY_HOTPLUG config options
>>>
>>>  arch/arm64/Kconfig           | 10 ++++++++++
>>>  arch/arm64/configs/defconfig |  3 +++
>>>  arch/arm64/mm/init.c         | 42 ++++++++++++++++++++++++++++++++++++++++++
>>>  3 files changed, 55 insertions(+)
>>>
>>
>>
>>
> 
> .
> 

Re: [RFC PATCH 0/2] arm64: memory-hotplug: Add Memory Hotplug support

[Maciej Bielski]

Hello,

Recently we have announced our effort on that:
https://lkml.org/lkml/2016/11/17/49

For now we have a working solution for hotplug and we are performing
code cleanup to push the patches soon.

BR,

On 02/12/2016 01:19, Scott Branden wrote:
> This patchset is sent for comment to add memory hotplug support for ARM64
> based platforms.  It follows hotplug code added for other architectures
> in the linux kernel.
> 
> I tried testing the memory hotplug feature following documentation from
> Documentation/memory-hotplug.txt.  I don't think it is working as expected
> - see below:
> 
> To add memory to the system I did the following:
> echo 0x400000000 > /sys/devices/system/memory/probe
> 
> The memory is displayed as system ram:
> cat /proc/iomem:
> 74000000-77ffffff : System RAM
>   74080000-748dffff : Kernel code
>   74950000-749d2fff : Kernel data
> 400000000-43fffffff : System RAM
> 
> But does not seem to be added to the kernel memory.
> /proc/meminfo did not change.
> 
> What else needs to be done so the memory is added to the kernel memory
> pool for normal allocation?
> 
> Scott Branden (2):
>   arm64: memory-hotplug: Add MEMORY_HOTPLUG, MEMORY_HOTREMOVE,
>     MEMORY_PROBE
>   arm64: defconfig: enable MEMORY_HOTPLUG config options
> 
>  arch/arm64/Kconfig           | 10 ++++++++++
>  arch/arm64/configs/defconfig |  3 +++
>  arch/arm64/mm/init.c         | 42 ++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 55 insertions(+)
> 

-- 
Maciej Bielski

Re: [RFC PATCH 0/2] arm64: memory-hotplug: Add Memory Hotplug support

[Will Deacon]

On Fri, Dec 02, 2016 at 10:13:43AM +0100, Maciej Bielski wrote:
> Recently we have announced our effort on that:
> https://lkml.org/lkml/2016/11/17/49
> 
> For now we have a working solution for hotplug and we are performing
> code cleanup to push the patches soon.

Are these intended to replace or extend Scott's patches? If the former,
please work with Scott's stuff as a base rather than posting a competing
series.

Will

Re: [RFC PATCH 0/2] arm64: memory-hotplug: Add Memory Hotplug support

[Maciej Bielski]

On 02/12/2016 11:49, Will Deacon wrote:
> On Fri, Dec 02, 2016 at 10:13:43AM +0100, Maciej Bielski wrote:
>> Recently we have announced our effort on that:
>> https://lkml.org/lkml/2016/11/17/49
>>
>> For now we have a working solution for hotplug and we are performing
>> code cleanup to push the patches soon.
> Are these intended to replace or extend Scott's patches? If the former,
> please work with Scott's stuff as a base rather than posting a competing
> series.
In the piece of code provided by Scott I have seen similar steps to what
is done by us but our work went further since we have the mapping
created and everything is working via the sysfs interface. I am now
having closer look and comparing them.
>
> Will

-- 
Maciej Bielski

Re: [RFC PATCH 0/2] arm64: memory-hotplug: Add Memory Hotplug support

[Scott Branden]

Hi Maciej,

On 16-12-02 02:55 AM, Maciej Bielski wrote:
>
>
> On 02/12/2016 11:49, Will Deacon wrote:
>> On Fri, Dec 02, 2016 at 10:13:43AM +0100, Maciej Bielski wrote:
>>> Recently we have announced our effort on that:
>>> https://lkml.org/lkml/2016/11/17/49
>>>
>>> For now we have a working solution for hotplug and we are performing
>>> code cleanup to push the patches soon.
>> Are these intended to replace or extend Scott's patches? If the former,
>> please work with Scott's stuff as a base rather than posting a competing
>> series.
> In the piece of code provided by Scott I have seen similar steps to what
> is done by us but our work went further since we have the mapping
> created and everything is working via the sysfs interface. I am now
> having closer look and comparing them.

I would love to see the missing section mapping step and any other 
additions to test out.  Please send additions as soon as you have a chance.

Regards,
  Scott

>>
>> Will
>

Re: [RFC PATCH 0/2] arm64: memory-hotplug: Add Memory Hotplug support

[Scott Branden]

Hi Xishi,

I followed you suggestions and found pfn_valid is always true.  Answers 
to your questions inline.

I could keep debugging this but hope Marcin sends out some code - I'm 
quite willing to test and help clean up the patchset.

On 16-12-01 07:11 PM, Xishi Qiu wrote:
> On 2016/12/2 10:38, Scott Branden wrote:
>
>> Hi Xishi,
>>
>> Thanks for the reply - please see comments below.
>>
>> On 16-12-01 05:49 PM, Xishi Qiu wrote:
>>> On 2016/12/2 8:19, Scott Branden wrote:
>>>
>>>> This patchset is sent for comment to add memory hotplug support for ARM64
>>>> based platforms.  It follows hotplug code added for other architectures
>>>> in the linux kernel.
>>>>
>>>> I tried testing the memory hotplug feature following documentation from
>>>> Documentation/memory-hotplug.txt.  I don't think it is working as expected
>>>> - see below:
>>>>
>>>> To add memory to the system I did the following:
>>>> echo 0x400000000 > /sys/devices/system/memory/probe
>>>>
>>>> The memory is displayed as system ram:
>>>> cat /proc/iomem:
>>>> 74000000-77ffffff : System RAM
>>>>   74080000-748dffff : Kernel code
>>>>   74950000-749d2fff : Kernel data
>>>> 400000000-43fffffff : System RAM
>>>>
>>>> But does not seem to be added to the kernel memory.
>>>> /proc/meminfo did not change.
>>>>
>>>> What else needs to be done so the memory is added to the kernel memory
>>>> pool for normal allocation?
>>>>
>>>
>>> Hi Scott,
>>>
>>> Do you mean it still don't support hod-add after apply this patchset?
>>
>> After applying the patch it appears to partially support hot-add. Please let me know if you think it is working as expected?
>>
>> The memory probe functions in that the memory is registered with the system and shows up in /proc/iomem.  But, the memory is not available in /proc/meminfo.  Do you think something else needs to be adjusted for ARM64 to hotadd the memory
>>
>> I just found another clue:
>> under /sys/devices/system/memory I only see one memory entry (before or after I try to hotadd additional memory).
>>
>> /sys/devices/system/memory # ls
>> auto_online_blocks  memory0            uevent
>> block_size_bytes    probe
>>
>> In arch/arm64/include/asm/sparsemem.h if I change SECTION_SIZE_BITS from 30 to 28 and recompile I get the following:
>> /sys/devices/system/memory # ls
>> auto_online_blocks  memory7            uevent
>> block_size_bytes    probe
>>
>>
>> In arch/arm64/include/asm/sparsemem.h if I change SECTION_SIZE_BITS from 30 to 27 and recompile I get the following:
>> /sys/devices/system/memory # ls
>> auto_online_blocks  memory14            uevent
>> block_size_bytes    probe
>>
>> If looks to me like something is not working properly in the ARM64 implementation.  I should expect to see multiple memoryX entries under /sys/devices/system/memory?
>>
>
> Hi Scott,
>
> 1. Do you enable the following configs?
> CONFIG_SPARSEMEM
> MEMORY_HOTPLUG
> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE
Yes, these configs are enabled
>
> 2. I find you missed create mapping in arch_add_memory(), and x86 has it.
Could you please explain this further?  The patch I submitted hass 
arch_add_memory identical to the ia64 implementation.
>
> 3. We will add memblock first, so pfn_valid() maybe always return true(in the
> following function), and this will lead __add_section() failed. Please check
> it.
You are correct - pfn_valid always returns true.  The function is in 
arch/arm64/mm/init.c and different than the one you indicated below:

#ifdef CONFIG_HAVE_ARCH_PFN_VALID
int pfn_valid(unsigned long pfn)
{
	return memblock_is_map_memory(pfn << PAGE_SHIFT);
}
EXPORT_SYMBOL(pfn_valid);
#endif

>
> int pfn_valid(unsigned long pfn)
> {
> 	return (pfn & PFN_MASK) == pfn && memblock_is_memory(pfn << PAGE_SHIFT);
> }
>
> add_memory
>   add_memory_resource
>     memblock_add_node
>       arch_add_memory
>         __add_pages
>           __add_section
>             pfn_valid
>
> Thanks,
> Xishi Qiu
>
>>
>>
>>>
>>> Thanks,
>>> Xishi Qiu
>>>
>>>> Scott Branden (2):
>>>>   arm64: memory-hotplug: Add MEMORY_HOTPLUG, MEMORY_HOTREMOVE,
>>>>     MEMORY_PROBE
>>>>   arm64: defconfig: enable MEMORY_HOTPLUG config options
>>>>
>>>>  arch/arm64/Kconfig           | 10 ++++++++++
>>>>  arch/arm64/configs/defconfig |  3 +++
>>>>  arch/arm64/mm/init.c         | 42 ++++++++++++++++++++++++++++++++++++++++++
>>>>  3 files changed, 55 insertions(+)
>>>>
>>>
>>>
>>>
>>
>> .
>>
>
>
>

Re: [RFC PATCH 0/2] arm64: memory-hotplug: Add Memory Hotplug support

[Xishi Qiu]

On 2016/12/7 16:43, Scott Branden wrote:

> Hi Xishi,
> 
> I followed you suggestions and found pfn_valid is always true.  Answers to your questions inline.
> 
> I could keep debugging this but hope Marcin sends out some code - I'm quite willing to test and help clean up the patchset.
> 
> On 16-12-01 07:11 PM, Xishi Qiu wrote:
>> On 2016/12/2 10:38, Scott Branden wrote:
>>
>>> Hi Xishi,
>>>
>>> Thanks for the reply - please see comments below.
>>>
>>> On 16-12-01 05:49 PM, Xishi Qiu wrote:
>>>> On 2016/12/2 8:19, Scott Branden wrote:
>>>>
>>>>> This patchset is sent for comment to add memory hotplug support for ARM64
>>>>> based platforms.  It follows hotplug code added for other architectures
>>>>> in the linux kernel.
>>>>>
>>>>> I tried testing the memory hotplug feature following documentation from
>>>>> Documentation/memory-hotplug.txt.  I don't think it is working as expected
>>>>> - see below:
>>>>>
>>>>> To add memory to the system I did the following:
>>>>> echo 0x400000000 > /sys/devices/system/memory/probe
>>>>>
>>>>> The memory is displayed as system ram:
>>>>> cat /proc/iomem:
>>>>> 74000000-77ffffff : System RAM
>>>>>   74080000-748dffff : Kernel code
>>>>>   74950000-749d2fff : Kernel data
>>>>> 400000000-43fffffff : System RAM
>>>>>
>>>>> But does not seem to be added to the kernel memory.
>>>>> /proc/meminfo did not change.
>>>>>
>>>>> What else needs to be done so the memory is added to the kernel memory
>>>>> pool for normal allocation?
>>>>>
>>>>
>>>> Hi Scott,
>>>>
>>>> Do you mean it still don't support hod-add after apply this patchset?
>>>
>>> After applying the patch it appears to partially support hot-add. Please let me know if you think it is working as expected?
>>>
>>> The memory probe functions in that the memory is registered with the system and shows up in /proc/iomem.  But, the memory is not available in /proc/meminfo.  Do you think something else needs to be adjusted for ARM64 to hotadd the memory
>>>
>>> I just found another clue:
>>> under /sys/devices/system/memory I only see one memory entry (before or after I try to hotadd additional memory).
>>>
>>> /sys/devices/system/memory # ls
>>> auto_online_blocks  memory0            uevent
>>> block_size_bytes    probe
>>>
>>> In arch/arm64/include/asm/sparsemem.h if I change SECTION_SIZE_BITS from 30 to 28 and recompile I get the following:
>>> /sys/devices/system/memory # ls
>>> auto_online_blocks  memory7            uevent
>>> block_size_bytes    probe
>>>
>>>
>>> In arch/arm64/include/asm/sparsemem.h if I change SECTION_SIZE_BITS from 30 to 27 and recompile I get the following:
>>> /sys/devices/system/memory # ls
>>> auto_online_blocks  memory14            uevent
>>> block_size_bytes    probe
>>>
>>> If looks to me like something is not working properly in the ARM64 implementation.  I should expect to see multiple memoryX entries under /sys/devices/system/memory?
>>>
>>
>> Hi Scott,
>>
>> 1. Do you enable the following configs?
>> CONFIG_SPARSEMEM
>> MEMORY_HOTPLUG
>> CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE
> Yes, these configs are enabled
>>
>> 2. I find you missed create mapping in arch_add_memory(), and x86 has it.
> Could you please explain this further?  The patch I submitted hass arch_add_memory identical to the ia64 implementation.

Hi Scott,

I think we should create page table first for the new hotadd memory.
e.g. create_mapping_late(start, __phys_to_virt(start), size, PAGE_KERNEL);

I don't know why ia64 don't have this step.
CC Tony

>>
>> 3. We will add memblock first, so pfn_valid() maybe always return true(in the
>> following function), and this will lead __add_section() failed. Please check
>> it.
> You are correct - pfn_valid always returns true.  The function is in arch/arm64/mm/init.c and different than the one you indicated below:
> 

> #ifdef CONFIG_HAVE_ARCH_PFN_VALID
> int pfn_valid(unsigned long pfn)
> {
>     return memblock_is_map_memory(pfn << PAGE_SHIFT);
> }
> EXPORT_SYMBOL(pfn_valid);
> #endif
> 
>>
>> int pfn_valid(unsigned long pfn)
>> {
>>     return (pfn & PFN_MASK) == pfn && memblock_is_memory(pfn << PAGE_SHIFT);
>> }
>>
>> add_memory
>>   add_memory_resource
>>     memblock_add_node
>>       arch_add_memory
>>         __add_pages
>>           __add_section
>>             pfn_valid
>>
>> Thanks,
>> Xishi Qiu
>>
>>>
>>>
>>>>
>>>> Thanks,
>>>> Xishi Qiu
>>>>
>>>>> Scott Branden (2):
>>>>>   arm64: memory-hotplug: Add MEMORY_HOTPLUG, MEMORY_HOTREMOVE,
>>>>>     MEMORY_PROBE
>>>>>   arm64: defconfig: enable MEMORY_HOTPLUG config options
>>>>>
>>>>>  arch/arm64/Kconfig           | 10 ++++++++++
>>>>>  arch/arm64/configs/defconfig |  3 +++
>>>>>  arch/arm64/mm/init.c         | 42 ++++++++++++++++++++++++++++++++++++++++++
>>>>>  3 files changed, 55 insertions(+)
>>>>>
>>>>
>>>>
>>>>
>>>
>>> .
>>>
>>
>>
>>
> 
> .
>