[PATCH 0/2] mm/memory_hotplug, arm64: allow certain bootmem sections to be offlinable

[PATCH 0/2] mm/memory_hotplug, arm64: allow certain bootmem sections to be offlinable

[Sudarshan Rajagopalan]

In the patch that enables memory hot-remove (commit bbd6ec605c0f ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier put in place that prevents boot memory from being offlined and removed. The commit text mentions that boot memory on arm64 cannot be removed. But x86 and other archs doesn’t seem to do this prevention.

The current logic is that only “new” memory blocks which are hot-added can later be offlined and removed. The memory that system booted up with cannot be offlined and removed. But there could be many usercases such as inter-VM memory sharing where a primary VM could offline and hot-remove a block/section of memory and lend it to secondary VM where it could hot-add it. And after usecase is done, the reverse happens where secondary VM hot-removes and gives it back to primary which can hot-add it back. In such cases, the present logic for arm64 doesn’t allow this hot-remove in primary to happen.

Also, on systems with movable zone that sort of guarantees pages to be migrated and isolated so that blocks can be offlined, this logic also defeats the purpose of having a movable zone which system can rely on memory hot-plugging, which say virt-io mem also relies on for fully plugged memory blocks.

This patch tries to solve by introducing a new section mem map sit 'SECTION_MARK_HOTPLUGGABLE' which allows the concerned module drivers be able
to mark requried sections as "hotpluggable" by setting this bit. Also this marking is only allowed for sections which are in movable zone and have unmovable pages. The arm64 mmu code on receiving the MEM_GOING_OFFLINE notification, we disallow offlining of any boot memory by checking if section_early or not. With the introduction of SECTION_MARK_HOTPLUGGABLE, we allow boot mem sections that are marked as hotpluggable with this bit set to be offlined and removed. Thereby allowing required bootmem sections to be offlinable.

Sudarshan Rajagopalan (2):
  mm/memory_hotplug: allow marking of memory sections as hotpluggable
  arm64: allow hotpluggable sections to be offlined

 arch/arm64/mm/mmu.c            |  2 +-
 include/linux/memory_hotplug.h |  1 +
 include/linux/mmzone.h         |  9 ++++++++-
 mm/memory_hotplug.c            | 20 ++++++++++++++++++++
 mm/sparse.c                    | 31 +++++++++++++++++++++++++++++++
 5 files changed, 61 insertions(+), 2 deletions(-)

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

[PATCH 1/2] mm/memory_hotplug: allow marking of memory sections as hotpluggable

[Sudarshan Rajagopalan]

Certain architectures such as arm64 doesn't allow boot memory to be
offlined and removed. Distinguish certain memory sections as
"hotpluggable" which can be marked by module drivers stating to memory
hotplug layer that these sections can be offlined and then removed.
This is done by using a separate section memory mab bit and setting it,
rather than clearing the existing SECTION_IS_EARLY bit.
This patch introduces SECTION_MARK_HOTPLUGGABLE bit into section mem map.
Only the allowed sections which are in movable zone and have unmovable
pages are allowed to be set with this new bit.

Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Steven Price <steven.price@arm.com>
Cc: Logan Gunthorpe <logang@deltatee.com>
Cc: Suren Baghdasaryan <surenb@google.com>
---
 include/linux/memory_hotplug.h |  1 +
 include/linux/mmzone.h         |  9 ++++++++-
 mm/memory_hotplug.c            | 20 ++++++++++++++++++++
 mm/sparse.c                    | 31 +++++++++++++++++++++++++++++++
 4 files changed, 60 insertions(+), 1 deletion(-)

diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 375515803cd8..81df45b582c8 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -319,6 +319,7 @@ extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
 extern int remove_memory(int nid, u64 start, u64 size);
 extern void __remove_memory(int nid, u64 start, u64 size);
 extern int offline_and_remove_memory(int nid, u64 start, u64 size);
+extern int mark_memory_hotpluggable(unsigned long start, unsigned long end);
 
 #else
 static inline void try_offline_node(int nid) {}
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 8379432f4f2f..3df3a4975236 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1247,7 +1247,8 @@ extern size_t mem_section_usage_size(void);
 #define SECTION_HAS_MEM_MAP	(1UL<<1)
 #define SECTION_IS_ONLINE	(1UL<<2)
 #define SECTION_IS_EARLY	(1UL<<3)
-#define SECTION_MAP_LAST_BIT	(1UL<<4)
+#define SECTION_MARK_HOTPLUGGABLE	(1UL<<4)
+#define SECTION_MAP_LAST_BIT	(1UL<<5)
 #define SECTION_MAP_MASK	(~(SECTION_MAP_LAST_BIT-1))
 #define SECTION_NID_SHIFT	3
 
@@ -1278,6 +1279,11 @@ static inline int early_section(struct mem_section *section)
 	return (section && (section->section_mem_map & SECTION_IS_EARLY));
 }
 
+static inline int removable_section(struct mem_section *section)
+{
+	return (section && (section->section_mem_map & SECTION_MARK_HOTPLUGGABLE));
+}
+
 static inline int valid_section_nr(unsigned long nr)
 {
 	return valid_section(__nr_to_section(nr));
@@ -1297,6 +1303,7 @@ static inline int online_section_nr(unsigned long nr)
 void online_mem_sections(unsigned long start_pfn, unsigned long end_pfn);
 #ifdef CONFIG_MEMORY_HOTREMOVE
 void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn);
+int section_mark_hotpluggable(struct mem_section *ms);
 #endif
 #endif
 
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index e9d5ab5d3ca0..503b0de489a0 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1860,4 +1860,24 @@ int offline_and_remove_memory(int nid, u64 start, u64 size)
 	return rc;
 }
 EXPORT_SYMBOL_GPL(offline_and_remove_memory);
+
+int mark_memory_hotpluggable(unsigned long start_pfn, unsigned long end_pfn)
+{
+	struct mem_section *ms;
+	unsigned long nr;
+	int rc = -EINVAL;
+
+	if (end_pfn < start_pfn)
+		return rc;
+
+	for (nr = start_pfn; nr <= end_pfn; nr++) {
+		ms = __pfn_to_section(nr);
+		rc = section_mark_hotpluggable(ms);
+		if (!rc)
+			break;
+	}
+
+	return rc;
+}
+EXPORT_SYMBOL_GPL(mark_memory_hotpluggable);
 #endif /* CONFIG_MEMORY_HOTREMOVE */
diff --git a/mm/sparse.c b/mm/sparse.c
index fcc3d176f1ea..cc21c23e2f1d 100644
--- a/mm/sparse.c
+++ b/mm/sparse.c
@@ -13,6 +13,7 @@
 #include <linux/vmalloc.h>
 #include <linux/swap.h>
 #include <linux/swapops.h>
+#include <linux/page-isolation.h>
 
 #include "internal.h"
 #include <asm/dma.h>
@@ -644,6 +645,36 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn)
 		ms->section_mem_map &= ~SECTION_IS_ONLINE;
 	}
 }
+
+int section_mark_hotpluggable(struct mem_section *ms)
+{
+	unsigned long section_nr, pfn;
+	bool unmovable;
+	struct page *page;
+
+	/* section needs to be both valid and present to be marked */
+	if (WARN_ON(!valid_section(ms)) || !present_section(ms))
+		return -EINVAL;
+
+	/*
+	 * now check if this section is removable. This can be done by checking
+	 * if section has unmovable pages or not.
+	 */
+	section_nr = __section_nr(ms);
+	pfn = section_nr_to_pfn(section_nr);
+	page = pfn_to_page(pfn);
+	unmovable = has_unmovable_pages(page_zone(page), page,
+					MIGRATE_MOVABLE, MEMORY_OFFLINE | REPORT_FAILURE);
+	if (unmovable) {
+		pr_info("section %lu has unmovable pages. Cannot be marked as hotpluggable\n");
+		return -EINVAL;
+	}
+
+	/* all good! mark section as hotpluggable */
+	ms->section_mem_map |= SECTION_MARK_HOTPLUGGABLE;
+
+	return 0;
+}
 #endif
 
 #ifdef CONFIG_SPARSEMEM_VMEMMAP
-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

[PATCH 2/2] arm64: allow hotpluggable sections to be offlined

[Sudarshan Rajagopalan]

On receiving the MEM_GOING_OFFLINE notification, we disallow offlining of
any boot memory by checking if section_early or not. With the introduction
of SECTION_MARK_HOTPLUGGABLE, allow boot mem sections that are marked as
hotpluggable with this bit set to be offlined and removed. This now allows
certain boot mem sections to be offlined.

Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Gavin Shan <gshan@redhat.com>
Cc: Logan Gunthorpe <logang@deltatee.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Steven Price <steven.price@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
---
 arch/arm64/mm/mmu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 75df62fea1b6..fb8878698672 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -1487,7 +1487,7 @@ static int prevent_bootmem_remove_notifier(struct notifier_block *nb,
 
 	for (; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
 		ms = __pfn_to_section(pfn);
-		if (early_section(ms))
+		if (early_section(ms) && !removable_section(ms))
 			return NOTIFY_BAD;
 	}
 	return NOTIFY_OK;
-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [PATCH 2/2] arm64: allow hotpluggable sections to be offlined

[David Hildenbrand]

> Am 17.10.2020 um 04:03 schrieb Sudarshan Rajagopalan <sudaraja@codeaurora.org>:
> 
> On receiving the MEM_GOING_OFFLINE notification, we disallow offlining of
> any boot memory by checking if section_early or not. With the introduction
> of SECTION_MARK_HOTPLUGGABLE, allow boot mem sections that are marked as
> hotpluggable with this bit set to be offlined and removed. This now allows
> certain boot mem sections to be offlined.
> 

The check (notifier) is in arm64 code. I don‘t see why you cannot make such decisions completely in arm64 code? Why would you have to mark sections?

Also, I think I am missing from *where* the code that marks sections removable is even called? Who makes such decisions?

This feels wrong. 

> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Anshuman Khandual <anshuman.khandual@arm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Gavin Shan <gshan@redhat.com>
> Cc: Logan Gunthorpe <logang@deltatee.com>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Steven Price <steven.price@arm.com>
> Cc: Suren Baghdasaryan <surenb@google.com>
> ---
> arch/arm64/mm/mmu.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> index 75df62fea1b6..fb8878698672 100644
> --- a/arch/arm64/mm/mmu.c
> +++ b/arch/arm64/mm/mmu.c
> @@ -1487,7 +1487,7 @@ static int prevent_bootmem_remove_notifier(struct notifier_block *nb,
> 
>    for (; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
>        ms = __pfn_to_section(pfn);
> -        if (early_section(ms))
> +        if (early_section(ms) && !removable_section(ms))
>            return NOTIFY_BAD;
>    }
>    return NOTIFY_OK;
> -- 
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> a Linux Foundation Collaborative Project
> 

Re: [PATCH 1/2] mm/memory_hotplug: allow marking of memory sections as hotpluggable

[Mike Rapoport]

On Fri, Oct 16, 2020 at 07:02:23PM -0700, Sudarshan Rajagopalan wrote:
> Certain architectures such as arm64 doesn't allow boot memory to be
> offlined and removed. Distinguish certain memory sections as
> "hotpluggable" which can be marked by module drivers stating to memory
> hotplug layer that these sections can be offlined and then removed.

I don't quite follow why marking sections as hotpluggable or not should
be done by a device driver. Can you describe in more details your
use-case and why there is a need to add a flag to the memory map?


> This is done by using a separate section memory mab bit and setting it,
> rather than clearing the existing SECTION_IS_EARLY bit.
> This patch introduces SECTION_MARK_HOTPLUGGABLE bit into section mem map.
> Only the allowed sections which are in movable zone and have unmovable
> pages are allowed to be set with this new bit.
> 
> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
> Cc: Catalin Marinas <catalin.marinas@arm.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Mike Rapoport <rppt@kernel.org>
> Cc: Anshuman Khandual <anshuman.khandual@arm.com>
> Cc: David Hildenbrand <david@redhat.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Steven Price <steven.price@arm.com>
> Cc: Logan Gunthorpe <logang@deltatee.com>
> Cc: Suren Baghdasaryan <surenb@google.com>
> ---
>  include/linux/memory_hotplug.h |  1 +
>  include/linux/mmzone.h         |  9 ++++++++-
>  mm/memory_hotplug.c            | 20 ++++++++++++++++++++
>  mm/sparse.c                    | 31 +++++++++++++++++++++++++++++++
>  4 files changed, 60 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
> index 375515803cd8..81df45b582c8 100644
> --- a/include/linux/memory_hotplug.h
> +++ b/include/linux/memory_hotplug.h
> @@ -319,6 +319,7 @@ extern int offline_pages(unsigned long start_pfn, unsigned long nr_pages);
>  extern int remove_memory(int nid, u64 start, u64 size);
>  extern void __remove_memory(int nid, u64 start, u64 size);
>  extern int offline_and_remove_memory(int nid, u64 start, u64 size);
> +extern int mark_memory_hotpluggable(unsigned long start, unsigned long end);
>  
>  #else
>  static inline void try_offline_node(int nid) {}
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 8379432f4f2f..3df3a4975236 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -1247,7 +1247,8 @@ extern size_t mem_section_usage_size(void);
>  #define SECTION_HAS_MEM_MAP	(1UL<<1)
>  #define SECTION_IS_ONLINE	(1UL<<2)
>  #define SECTION_IS_EARLY	(1UL<<3)
> -#define SECTION_MAP_LAST_BIT	(1UL<<4)
> +#define SECTION_MARK_HOTPLUGGABLE	(1UL<<4)
> +#define SECTION_MAP_LAST_BIT	(1UL<<5)
>  #define SECTION_MAP_MASK	(~(SECTION_MAP_LAST_BIT-1))
>  #define SECTION_NID_SHIFT	3
>  
> @@ -1278,6 +1279,11 @@ static inline int early_section(struct mem_section *section)
>  	return (section && (section->section_mem_map & SECTION_IS_EARLY));
>  }
>  
> +static inline int removable_section(struct mem_section *section)
> +{
> +	return (section && (section->section_mem_map & SECTION_MARK_HOTPLUGGABLE));
> +}
> +
>  static inline int valid_section_nr(unsigned long nr)
>  {
>  	return valid_section(__nr_to_section(nr));
> @@ -1297,6 +1303,7 @@ static inline int online_section_nr(unsigned long nr)
>  void online_mem_sections(unsigned long start_pfn, unsigned long end_pfn);
>  #ifdef CONFIG_MEMORY_HOTREMOVE
>  void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn);
> +int section_mark_hotpluggable(struct mem_section *ms);
>  #endif
>  #endif
>  
> diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
> index e9d5ab5d3ca0..503b0de489a0 100644
> --- a/mm/memory_hotplug.c
> +++ b/mm/memory_hotplug.c
> @@ -1860,4 +1860,24 @@ int offline_and_remove_memory(int nid, u64 start, u64 size)
>  	return rc;
>  }
>  EXPORT_SYMBOL_GPL(offline_and_remove_memory);
> +
> +int mark_memory_hotpluggable(unsigned long start_pfn, unsigned long end_pfn)
> +{
> +	struct mem_section *ms;
> +	unsigned long nr;
> +	int rc = -EINVAL;
> +
> +	if (end_pfn < start_pfn)
> +		return rc;
> +
> +	for (nr = start_pfn; nr <= end_pfn; nr++) {
> +		ms = __pfn_to_section(nr);
> +		rc = section_mark_hotpluggable(ms);
> +		if (!rc)
> +			break;
> +	}
> +
> +	return rc;
> +}
> +EXPORT_SYMBOL_GPL(mark_memory_hotpluggable);
>  #endif /* CONFIG_MEMORY_HOTREMOVE */
> diff --git a/mm/sparse.c b/mm/sparse.c
> index fcc3d176f1ea..cc21c23e2f1d 100644
> --- a/mm/sparse.c
> +++ b/mm/sparse.c
> @@ -13,6 +13,7 @@
>  #include <linux/vmalloc.h>
>  #include <linux/swap.h>
>  #include <linux/swapops.h>
> +#include <linux/page-isolation.h>
>  
>  #include "internal.h"
>  #include <asm/dma.h>
> @@ -644,6 +645,36 @@ void offline_mem_sections(unsigned long start_pfn, unsigned long end_pfn)
>  		ms->section_mem_map &= ~SECTION_IS_ONLINE;
>  	}
>  }
> +
> +int section_mark_hotpluggable(struct mem_section *ms)
> +{
> +	unsigned long section_nr, pfn;
> +	bool unmovable;
> +	struct page *page;
> +
> +	/* section needs to be both valid and present to be marked */
> +	if (WARN_ON(!valid_section(ms)) || !present_section(ms))
> +		return -EINVAL;
> +
> +	/*
> +	 * now check if this section is removable. This can be done by checking
> +	 * if section has unmovable pages or not.
> +	 */
> +	section_nr = __section_nr(ms);
> +	pfn = section_nr_to_pfn(section_nr);
> +	page = pfn_to_page(pfn);
> +	unmovable = has_unmovable_pages(page_zone(page), page,
> +					MIGRATE_MOVABLE, MEMORY_OFFLINE | REPORT_FAILURE);
> +	if (unmovable) {
> +		pr_info("section %lu has unmovable pages. Cannot be marked as hotpluggable\n");
> +		return -EINVAL;
> +	}
> +
> +	/* all good! mark section as hotpluggable */
> +	ms->section_mem_map |= SECTION_MARK_HOTPLUGGABLE;
> +
> +	return 0;
> +}
>  #endif
>  
>  #ifdef CONFIG_SPARSEMEM_VMEMMAP
> -- 
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
> a Linux Foundation Collaborative Project
> 

-- 
Sincerely yours,
Mike.

Re: [PATCH 1/2] mm/memory_hotplug: allow marking of memory sections as hotpluggable

[David Hildenbrand]

On 17.10.20 10:26, Mike Rapoport wrote:
> On Fri, Oct 16, 2020 at 07:02:23PM -0700, Sudarshan Rajagopalan wrote:
>> Certain architectures such as arm64 doesn't allow boot memory to be
>> offlined and removed. Distinguish certain memory sections as
>> "hotpluggable" which can be marked by module drivers stating to memory
>> hotplug layer that these sections can be offlined and then removed.
> 
> I don't quite follow why marking sections as hotpluggable or not should
> be done by a device driver. Can you describe in more details your
> use-case and why there is a need to add a flag to the memory map?
> 

This seems to be related to

https://lkml.kernel.org/r/de8388df2fbc5a6a33aab95831ba7db4@codeaurora.org

After reading how the driver is trying to abuse memory hot(un)plug
infrastructure, my tentative

Nacked-by: David Hildenbrand <david@redhat.com>

-- 
Thanks,

David / dhildenb

Re: [PATCH 0/2] mm/memory_hotplug, arm64: allow certain bootmem sections to be offlinable

[Anshuman Khandual]

Hello Sudarshan,

On 10/17/2020 07:32 AM, Sudarshan Rajagopalan wrote:
> In the patch that enables memory hot-remove (commit bbd6ec605c0f ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier put in place that prevents boot memory from being offlined and removed. The commit text mentions that boot memory on arm64 cannot be removed. But x86 and other archs doesn’t seem to do this prevention.
> 
> The current logic is that only “new” memory blocks which are hot-added can later be offlined and removed. The memory that system booted up with cannot be offlined and removed. But there could be many usercases such as inter-VM memory sharing where a primary VM could offline and hot-remove a block/section of memory and lend it to secondary VM where it could hot-add it. And after usecase is done, the reverse happens where secondary VM hot-removes and gives it back to primary which can hot-add it back. In such cases, the present logic for arm64 doesn’t allow this hot-remove in primary to happen.
> 
> Also, on systems with movable zone that sort of guarantees pages to be migrated and isolated so that blocks can be offlined, this logic also defeats the purpose of having a movable zone which system can rely on memory hot-plugging, which say virt-io mem also relies on for fully plugged memory blocks.
> 
> This patch tries to solve by introducing a new section mem map sit 'SECTION_MARK_HOTPLUGGABLE' which allows the concerned module drivers be able
> to mark requried sections as "hotpluggable" by setting this bit. Also this marking is only allowed for sections which are in movable zone and have unmovable pages. The arm64 mmu code on receiving the MEM_GOING_OFFLINE notification, we disallow offlining of any boot memory by checking if section_early or not. With the introduction of SECTION_MARK_HOTPLUGGABLE, we allow boot mem sections that are marked as hotpluggable with this bit set to be offlined and removed. Thereby allowing required bootmem sections to be offlinable.

This series was posted right after another thread you initiated in this regard
but without even waiting for it to conclude in any manner.

https://lore.kernel.org/linux-arm-kernel/de8388df2fbc5a6a33aab95831ba7db4@codeaurora.org/ 

Inter-VM memory migration could be solved in other methods as David has mentioned.
Boot memory cannot be removed and hence offlined on arm64 due to multiple reasons
including making kexec non-functional afterwards. Besides these intrusive core MM
changes are not really required.

- Anshuman

Re: [PATCH 2/2] arm64: allow hotpluggable sections to be offlined

[Anshuman Khandual]

On 10/17/2020 01:04 PM, David Hildenbrand wrote:
> 
>> Am 17.10.2020 um 04:03 schrieb Sudarshan Rajagopalan <sudaraja@codeaurora.org>:
>>
>> On receiving the MEM_GOING_OFFLINE notification, we disallow offlining of
>> any boot memory by checking if section_early or not. With the introduction
>> of SECTION_MARK_HOTPLUGGABLE, allow boot mem sections that are marked as
>> hotpluggable with this bit set to be offlined and removed. This now allows
>> certain boot mem sections to be offlined.
>>
> 
> The check (notifier) is in arm64 code. I don‘t see why you cannot make such decisions completely in arm64 code? Why would you have to mark sections?
> 
> Also, I think I am missing from *where* the code that marks sections removable is even called? Who makes such decisions?

From the previous patch.

+EXPORT_SYMBOL_GPL(mark_memory_hotpluggable);

> 
> This feels wrong. 
> 
>> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
>> Cc: Catalin Marinas <catalin.marinas@arm.com>
>> Cc: Will Deacon <will@kernel.org>
>> Cc: Anshuman Khandual <anshuman.khandual@arm.com>
>> Cc: Mark Rutland <mark.rutland@arm.com>
>> Cc: Gavin Shan <gshan@redhat.com>
>> Cc: Logan Gunthorpe <logang@deltatee.com>
>> Cc: David Hildenbrand <david@redhat.com>
>> Cc: Andrew Morton <akpm@linux-foundation.org>
>> Cc: Steven Price <steven.price@arm.com>
>> Cc: Suren Baghdasaryan <surenb@google.com>
>> ---
>> arch/arm64/mm/mmu.c | 2 +-
>> 1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
>> index 75df62fea1b6..fb8878698672 100644
>> --- a/arch/arm64/mm/mmu.c
>> +++ b/arch/arm64/mm/mmu.c
>> @@ -1487,7 +1487,7 @@ static int prevent_bootmem_remove_notifier(struct notifier_block *nb,
>>
>>    for (; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
>>        ms = __pfn_to_section(pfn);
>> -        if (early_section(ms))
>> +        if (early_section(ms) && !removable_section(ms))

Till challenges related to boot memory removal on arm64 platform get
resolved, no portion of boot memory can be offlined. Let alone via a
driver making such decisions.

>>            return NOTIFY_BAD;
>>    }
>>    return NOTIFY_OK;
>> -- 
>> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
>> a Linux Foundation Collaborative Project
>>
> 
> 

Re: [PATCH 1/2] mm/memory_hotplug: allow marking of memory sections as hotpluggable

[Mike Rapoport]

On Sat, Oct 17, 2020 at 11:36:39AM +0200, David Hildenbrand wrote:
> On 17.10.20 10:26, Mike Rapoport wrote:
> > On Fri, Oct 16, 2020 at 07:02:23PM -0700, Sudarshan Rajagopalan wrote:
> >> Certain architectures such as arm64 doesn't allow boot memory to be
> >> offlined and removed. Distinguish certain memory sections as
> >> "hotpluggable" which can be marked by module drivers stating to memory
> >> hotplug layer that these sections can be offlined and then removed.
> > 
> > I don't quite follow why marking sections as hotpluggable or not should
> > be done by a device driver. Can you describe in more details your
> > use-case and why there is a need to add a flag to the memory map?
> > 
> 
> This seems to be related to
> 
> https://lkml.kernel.org/r/de8388df2fbc5a6a33aab95831ba7db4@codeaurora.org

Thanks for the pointer.

> After reading how the driver is trying to abuse memory hot(un)plug
> infrastructure, my tentative
> 
> Nacked-by: David Hildenbrand <david@redhat.com>

I also don't think we would want to let drivers play with the memory
map.

> -- 
> Thanks,
> 
> David / dhildenb
> 

-- 
Sincerely yours,
Mike.