Re: [PATCH v6 00/15] memory-hotplug: hot-remove physical memory

[Tang Chen <tangchen@cn.fujitsu.com>]

On 01/30/2013 06:15 PM, Tang Chen wrote:
> Hi Simon,
>
> Please see below. :)
>
> On 01/29/2013 08:52 PM, Simon Jeons wrote:
>> Hi Tang,
>>
>> On Wed, 2013-01-09 at 17:32 +0800, Tang Chen wrote:
>>> Here is the physical memory hot-remove patch-set based on 3.8rc-2.
>>
>> Some questions ask you, not has relationship with this patchset, but is
>> memory hotplug stuff.
>>
>> 1. In function node_states_check_changes_online:
>>
>> comments:
>> * If we don't have HIGHMEM nor movable node,
>> * node_states[N_NORMAL_MEMORY] contains nodes which have zones of
>> * 0...ZONE_MOVABLE, set zone_last to ZONE_MOVABLE.
>>
>> How to understand it? Why we don't have HIGHMEM nor movable node and
>> node_staes[N_NORMAL_MEMORY] contains 0...ZONE_MOVABLE, IIUC,
>> N_NORMAL_MEMORY only means the node has regular memory.
>>
>
> First of all, I think we need to understand why we need N_MEMORY.
>
> In order to support movable node, which has only ZONE_MOVABLE (the last
> zone),
> we introduce N_MEMORY to represent the node has normal, highmem and
> movable memory.
>
> Here, "we have movable node" means you configured CONFIG_MOVABLE_NODE.

Sorry, should be "we don't have movable node" means you didn't 
configured CONFIG_MOVABLE_NODE.

> This config option doesn't mean we don't have movable pages, (NO)
> it means we don't have a node which has only movable pages (only have
> ZONE_MOVABLE). (YES)
>
> Here, if we don't have CONFIG_MOVABLE_NODE (we don't have movable node),
> we don't need a
> separate node_states[] element to represent a particular node because we
> won't have a node
> which has only ZONE_MOVABLE.
>
> So,
> 1) if we don't have highmem nor movable node, N_MEMORY == N_HIGH_MEMORY
> == N_NORMAL_MEMORY,
> which means N_NORMAL_MEMORY effects as N_MEMORY. If we online pages as
> movable, we need
> to update node_states[N_NORMAL_MEMORY].
>
> Please refer to the definition of enum zone_type, if we don't have
> CONFIG_HIGHMEM, we won't
> have ZONE_HIGHMEM, but ZONE_NORMAL and ZONE_MOVABLE will always there.
> So we can have movable
> pages, and the zone_last should be ZONE_MOVABLE.
>
> Again, because we won't have a node only having ZONE_MOVABLE, so we just
> need to update
> node_states[N_NORMAL_MEMORY].
>
>> * If we don't have movable node, node_states[N_NORMAL_MEMORY]
>> * contains nodes which have zones of 0...ZONE_MOVABLE,
>> * set zone_last to ZONE_MOVABLE.
>>
>> How to understand?
>
> 2) this code is in #ifdef CONFIG_HIGHMEM, which means we have highmem,
> so if we don't have
> movable node, N_MEMORY == N_HIGH_MEMORY, and N_HIGH_MEMORY effects as
> N_MEMORY. If we
> online pages as movable, we need to update node_states[N_NORMAL_MEMORY].
>
>>
>> 2. In function move_pfn_range_left, why end<= z2->zone_start_pfn is not
>> correct? The comments said that must include/overlap, why?
>>
>
> This one is easy, if I understand you correctly.
> move_pfn_range_left() is used to move the left most part [start_pfn,
> end_pfn) of z2 to z1.
> So if end_pfn<= z2->zone_start_pfn, it means [start_pfn, end_pfn) is not
> part of z2.
> Then it fails.
>
>> 3. In function online_pages, the normal case(w/o online_kenrel,
>> online_movable), why not check if the new zone is overlap with adjacent
>> zones?
>>
>
> Can a zone overlap with the others ? I don't think so.
>
> One pfn could only be in one zone,
> zone = page_zone(pfn_to_page(pfn));
>
> it could overlap with others, I think. :)
>
> But maybe I misunderstand you. :)
>
>> 4. Could you summarize the difference implementation between hot-add and
>> logic-add, hot-remove and logic-remove?
>
> Sorry, I don't quite understand what do you mean by logic-add/remove.
> Would you please explain more ?
>
> If you meant the sys fs interfaces, I think they are just another set of
> entrances
> of memory hotplug.
>
> Thanks. :)
>
>>
>>
>>>
>>> This patch-set aims to implement physical memory hot-removing.
>>>
>>> The patches can free/remove the following things:
>>>
>>> - /sys/firmware/memmap/X/{end, start, type} : [PATCH 4/15]
>>> - memmap of sparse-vmemmap : [PATCH 6,7,8,10/15]
>>> - page table of removed memory : [RFC PATCH 7,8,10/15]
>>> - node and related sysfs files : [RFC PATCH 13-15/15]
>>>
>>>
>>> Existing problem:
>>> If CONFIG_MEMCG is selected, we will allocate memory to store page
>>> cgroup
>>> when we online pages.
>>>
>>> For example: there is a memory device on node 1. The address range
>>> is [1G, 1.5G). You will find 4 new directories memory8, memory9,
>>> memory10,
>>> and memory11 under the directory /sys/devices/system/memory/.
>>>
>>> If CONFIG_MEMCG is selected, when we online memory8, the memory
>>> stored page
>>> cgroup is not provided by this memory device. But when we online
>>> memory9, the
>>> memory stored page cgroup may be provided by memory8. So we can't
>>> offline
>>> memory8 now. We should offline the memory in the reversed order.
>>>
>>> When the memory device is hotremoved, we will auto offline memory
>>> provided
>>> by this memory device. But we don't know which memory is onlined
>>> first, so
>>> offlining memory may fail.
>>>
>>> In patch1, we provide a solution which is not good enough:
>>> Iterate twice to offline the memory.
>>> 1st iterate: offline every non primary memory block.
>>> 2nd iterate: offline primary (i.e. first added) memory block.
>>>
>>> And a new idea from Wen Congyang<wency@cn.fujitsu.com> is:
>>> allocate the memory from the memory block they are describing.
>>>
>>> But we are not sure if it is OK to do so because there is not
>>> existing API
>>> to do so, and we need to move page_cgroup memory allocation from
>>> MEM_GOING_ONLINE
>>> to MEM_ONLINE. And also, it may interfere the hugepage.
>>>
>>>
>>>
>>> How to test this patchset?
>>> 1. apply this patchset and build the kernel. MEMORY_HOTPLUG,
>>> MEMORY_HOTREMOVE,
>>> ACPI_HOTPLUG_MEMORY must be selected.
>>> 2. load the module acpi_memhotplug
>>> 3. hotplug the memory device(it depends on your hardware)
>>> You will see the memory device under the directory
>>> /sys/bus/acpi/devices/.
>>> Its name is PNP0C80:XX.
>>> 4. online/offline pages provided by this memory device
>>> You can write online/offline to
>>> /sys/devices/system/memory/memoryX/state to
>>> online/offline pages provided by this memory device
>>> 5. hotremove the memory device
>>> You can hotremove the memory device by the hardware, or writing 1 to
>>> /sys/bus/acpi/devices/PNP0C80:XX/eject.
>>
>> Is there a similar knode to hot-add the memory device?
>>
>>>
>>>
>>> Note: if the memory provided by the memory device is used by the
>>> kernel, it
>>> can't be offlined. It is not a bug.
>>>
>>>
>>> Changelogs from v5 to v6:
>>> Patch3: Add some more comments to explain memory hot-remove.
>>> Patch4: Remove bootmem member in struct firmware_map_entry.
>>> Patch6: Repeatedly register bootmem pages when using hugepage.
>>> Patch8: Repeatedly free bootmem pages when using hugepage.
>>> Patch14: Don't free pgdat when offlining a node, just reset it to 0.
>>> Patch15: New patch, pgdat is not freed in patch14, so don't allocate
>>> a new
>>> one when online a node.
>>>
>>> Changelogs from v4 to v5:
>>> Patch7: new patch, move pgdat_resize_lock into
>>> sparse_remove_one_section() to
>>> avoid disabling irq because we need flush tlb when free pagetables.
>>> Patch8: new patch, pick up some common APIs that are used to free
>>> direct mapping
>>> and vmemmap pagetables.
>>> Patch9: free direct mapping pagetables on x86_64 arch.
>>> Patch10: free vmemmap pagetables.
>>> Patch11: since freeing memmap with vmemmap has been implemented, the
>>> config
>>> macro CONFIG_SPARSEMEM_VMEMMAP when defining __remove_section() is
>>> no longer needed.
>>> Patch13: no need to modify acpi_memory_disable_device() since it was
>>> removed,
>>> and add nid parameter when calling remove_memory().
>>>
>>> Changelogs from v3 to v4:
>>> Patch7: remove unused codes.
>>> Patch8: fix nr_pages that is passed to free_map_bootmem()
>>>
>>> Changelogs from v2 to v3:
>>> Patch9: call sync_global_pgds() if pgd is changed
>>> Patch10: fix a problem int the patch
>>>
>>> Changelogs from v1 to v2:
>>> Patch1: new patch, offline memory twice. 1st iterate: offline every
>>> non primary
>>> memory block. 2nd iterate: offline primary (i.e. first added) memory
>>> block.
>>>
>>> Patch3: new patch, no logical change, just remove reduntant codes.
>>>
>>> Patch9: merge the patch from wujianguo into this patch. flush tlb on
>>> all cpu
>>> after the pagetable is changed.
>>>
>>> Patch12: new patch, free node_data when a node is offlined.
>>>
>>>
>>> Tang Chen (6):
>>> memory-hotplug: move pgdat_resize_lock into
>>> sparse_remove_one_section()
>>> memory-hotplug: remove page table of x86_64 architecture
>>> memory-hotplug: remove memmap of sparse-vmemmap
>>> memory-hotplug: Integrated __remove_section() of
>>> CONFIG_SPARSEMEM_VMEMMAP.
>>> memory-hotplug: remove sysfs file of node
>>> memory-hotplug: Do not allocate pdgat if it was not freed when
>>> offline.
>>>
>>> Wen Congyang (5):
>>> memory-hotplug: try to offline the memory twice to avoid dependence
>>> memory-hotplug: remove redundant codes
>>> memory-hotplug: introduce new function arch_remove_memory() for
>>> removing page table depends on architecture
>>> memory-hotplug: Common APIs to support page tables hot-remove
>>> memory-hotplug: free node_data when a node is offlined
>>>
>>> Yasuaki Ishimatsu (4):
>>> memory-hotplug: check whether all memory blocks are offlined or not
>>> when removing memory
>>> memory-hotplug: remove /sys/firmware/memmap/X sysfs
>>> memory-hotplug: implement register_page_bootmem_info_section of
>>> sparse-vmemmap
>>> memory-hotplug: memory_hotplug: clear zone when removing the memory
>>>
>>> arch/arm64/mm/mmu.c | 3 +
>>> arch/ia64/mm/discontig.c | 10 +
>>> arch/ia64/mm/init.c | 18 ++
>>> arch/powerpc/mm/init_64.c | 10 +
>>> arch/powerpc/mm/mem.c | 12 +
>>> arch/s390/mm/init.c | 12 +
>>> arch/s390/mm/vmem.c | 10 +
>>> arch/sh/mm/init.c | 17 ++
>>> arch/sparc/mm/init_64.c | 10 +
>>> arch/tile/mm/init.c | 8 +
>>> arch/x86/include/asm/pgtable_types.h | 1 +
>>> arch/x86/mm/init_32.c | 12 +
>>> arch/x86/mm/init_64.c | 390 +++++++++++++++++++++++++++++
>>> arch/x86/mm/pageattr.c | 47 ++--
>>> drivers/acpi/acpi_memhotplug.c | 8 +-
>>> drivers/base/memory.c | 6 +
>>> drivers/firmware/memmap.c | 96 +++++++-
>>> include/linux/bootmem.h | 1 +
>>> include/linux/firmware-map.h | 6 +
>>> include/linux/memory_hotplug.h | 15 +-
>>> include/linux/mm.h | 4 +-
>>> mm/memory_hotplug.c | 459 +++++++++++++++++++++++++++++++---
>>> mm/sparse.c | 8 +-
>>> 23 files changed, 1094 insertions(+), 69 deletions(-)
>>>
>>> --
>>> To unsubscribe, send a message with 'unsubscribe linux-mm' in
>>> the body to majordomo@kvack.org. For more info on Linux MM,
>>> see: http://www.linux-mm.org/ .
>>> Don't email:<a href=mailto:"dont@kvack.org"> email@kvack.org</a>
>>
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>