[PATCH V2] arm64/mm/hotplug: Warn when memory limit has been reduced

[PATCH V2] arm64/mm/hotplug: Warn when memory limit has been reduced

[Anshuman Khandual]

If the max memory limit has been reduced with 'mem=' kernel command line
option, there might be UEFI memory map described memory beyond that limit
which could be hot removed. This might be problematic for subsequent kexec
kernel which could just access such removed memory.

Memory offline notifier exists because there is no other way to block the
removal of boot memory, only the offlining (which isn't actually a problem)
But with 'mem=', there is no chance to stop such boot memory being offlined
as it where never in use by the kernel. As 'mem=' is a debug only option on
arm64 platform, just warn for such a situation and move on.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: James Morse <james.morse@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
This applies on v5.16-rc8

Changes in V2:

- Moved has_mem_limit_reduced() inside hotplug notifier per Catalin
- Replaced pr_warn() with pr_warn_once() per Catalin

Changes in V1:

https://lore.kernel.org/linux-arm-kernel/1631602270-29215-1-git-send-email-anshuman.khandual@arm.com/

 arch/arm64/include/asm/memory.h |  1 +
 arch/arm64/mm/init.c            |  9 +++++++--
 arch/arm64/mm/mmu.c             | 12 ++++++++++++
 3 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index 1b9a1e242612..dba58cf529e4 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -354,6 +354,7 @@ static inline void *phys_to_virt(phys_addr_t x)
 })
 
 void dump_mem_limit(void);
+bool has_mem_limit_reduced(void);
 #endif /* !ASSEMBLY */
 
 /*
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index a8834434af99..4cd3b2311199 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -174,6 +174,11 @@ EXPORT_SYMBOL(pfn_is_map_memory);
 
 static phys_addr_t memory_limit = PHYS_ADDR_MAX;
 
+bool has_mem_limit_reduced(void)
+{
+	return memory_limit != PHYS_ADDR_MAX;
+}
+
 /*
  * Limit the memory size that was specified via FDT.
  */
@@ -248,7 +253,7 @@ void __init arm64_memblock_init(void)
 	 * high up in memory, add back the kernel region that must be accessible
 	 * via the linear mapping.
 	 */
-	if (memory_limit != PHYS_ADDR_MAX) {
+	if (has_mem_limit_reduced()) {
 		memblock_mem_limit_remove_map(memory_limit);
 		memblock_add(__pa_symbol(_text), (u64)(_end - _text));
 	}
@@ -422,7 +427,7 @@ void free_initmem(void)
 
 void dump_mem_limit(void)
 {
-	if (memory_limit != PHYS_ADDR_MAX) {
+	if (has_mem_limit_reduced()) {
 		pr_emerg("Memory Limit: %llu MB\n", memory_limit >> 20);
 	} else {
 		pr_emerg("Memory Limit: none\n");
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index acfae9b41cc8..9288cae136a1 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -1537,6 +1537,18 @@ static int prevent_bootmem_remove_notifier(struct notifier_block *nb,
 	if ((action != MEM_GOING_OFFLINE) && (action != MEM_OFFLINE))
 		return NOTIFY_OK;
 
+	if (has_mem_limit_reduced()) {
+		/*
+		 * Physical memory limit has been reduced via the 'mem=' kernel
+		 * command line option. Memory beyond reduced limit could now be
+		 * removed and reassigned (guest ?) transparently to the kernel.
+		 * This might cause subsequent kexec kernel to crash or at least
+		 * corrupt the memory when accessing UEFI memory map enumerated
+		 * boot memory which might have been repurposed.
+		 */
+		pr_warn_once("Memory limit reduced, kexec might be problematic\n");
+	}
+
 	for (; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
 		unsigned long start = PFN_PHYS(pfn);
 		unsigned long end = start + (1UL << PA_SECTION_SHIFT);
-- 
2.20.1

[PATCH V2] arm64/mm/hotplug: Warn when memory limit has been reduced

[Anshuman Khandual]

If the max memory limit has been reduced with 'mem=' kernel command line
option, there might be UEFI memory map described memory beyond that limit
which could be hot removed. This might be problematic for subsequent kexec
kernel which could just access such removed memory.

Memory offline notifier exists because there is no other way to block the
removal of boot memory, only the offlining (which isn't actually a problem)
But with 'mem=', there is no chance to stop such boot memory being offlined
as it where never in use by the kernel. As 'mem=' is a debug only option on
arm64 platform, just warn for such a situation and move on.

Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Will Deacon <will@kernel.org>
Cc: James Morse <james.morse@arm.com>
Cc: linux-arm-kernel@lists.infradead.org
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Anshuman Khandual <anshuman.khandual@arm.com>
---
This applies on v5.16-rc8

Changes in V2:

- Moved has_mem_limit_reduced() inside hotplug notifier per Catalin
- Replaced pr_warn() with pr_warn_once() per Catalin

Changes in V1:

https://lore.kernel.org/linux-arm-kernel/1631602270-29215-1-git-send-email-anshuman.khandual@arm.com/

 arch/arm64/include/asm/memory.h |  1 +
 arch/arm64/mm/init.c            |  9 +++++++--
 arch/arm64/mm/mmu.c             | 12 ++++++++++++
 3 files changed, 20 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/memory.h b/arch/arm64/include/asm/memory.h
index 1b9a1e242612..dba58cf529e4 100644
--- a/arch/arm64/include/asm/memory.h
+++ b/arch/arm64/include/asm/memory.h
@@ -354,6 +354,7 @@ static inline void *phys_to_virt(phys_addr_t x)
 })
 
 void dump_mem_limit(void);
+bool has_mem_limit_reduced(void);
 #endif /* !ASSEMBLY */
 
 /*
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index a8834434af99..4cd3b2311199 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -174,6 +174,11 @@ EXPORT_SYMBOL(pfn_is_map_memory);
 
 static phys_addr_t memory_limit = PHYS_ADDR_MAX;
 
+bool has_mem_limit_reduced(void)
+{
+	return memory_limit != PHYS_ADDR_MAX;
+}
+
 /*
  * Limit the memory size that was specified via FDT.
  */
@@ -248,7 +253,7 @@ void __init arm64_memblock_init(void)
 	 * high up in memory, add back the kernel region that must be accessible
 	 * via the linear mapping.
 	 */
-	if (memory_limit != PHYS_ADDR_MAX) {
+	if (has_mem_limit_reduced()) {
 		memblock_mem_limit_remove_map(memory_limit);
 		memblock_add(__pa_symbol(_text), (u64)(_end - _text));
 	}
@@ -422,7 +427,7 @@ void free_initmem(void)
 
 void dump_mem_limit(void)
 {
-	if (memory_limit != PHYS_ADDR_MAX) {
+	if (has_mem_limit_reduced()) {
 		pr_emerg("Memory Limit: %llu MB\n", memory_limit >> 20);
 	} else {
 		pr_emerg("Memory Limit: none\n");
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index acfae9b41cc8..9288cae136a1 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -1537,6 +1537,18 @@ static int prevent_bootmem_remove_notifier(struct notifier_block *nb,
 	if ((action != MEM_GOING_OFFLINE) && (action != MEM_OFFLINE))
 		return NOTIFY_OK;
 
+	if (has_mem_limit_reduced()) {
+		/*
+		 * Physical memory limit has been reduced via the 'mem=' kernel
+		 * command line option. Memory beyond reduced limit could now be
+		 * removed and reassigned (guest ?) transparently to the kernel.
+		 * This might cause subsequent kexec kernel to crash or at least
+		 * corrupt the memory when accessing UEFI memory map enumerated
+		 * boot memory which might have been repurposed.
+		 */
+		pr_warn_once("Memory limit reduced, kexec might be problematic\n");
+	}
+
 	for (; pfn < end_pfn; pfn += PAGES_PER_SECTION) {
 		unsigned long start = PFN_PHYS(pfn);
 		unsigned long end = start + (1UL << PA_SECTION_SHIFT);
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH V2] arm64/mm/hotplug: Warn when memory limit has been reduced

[David Hildenbrand]

On 03.01.22 07:21, Anshuman Khandual wrote:
> If the max memory limit has been reduced with 'mem=' kernel command line
> option, there might be UEFI memory map described memory beyond that limit
> which could be hot removed. This might be problematic for subsequent kexec
> kernel which could just access such removed memory.
> 
> Memory offline notifier exists because there is no other way to block the
> removal of boot memory, only the offlining (which isn't actually a problem)
> But with 'mem=', there is no chance to stop such boot memory being offlined
> as it where never in use by the kernel. As 'mem=' is a debug only option on
> arm64 platform, just warn for such a situation and move on.
> 

Hi,

but why warn on the offlining path?

IIUC, you'd have a layout like


[ boot memory ][ unused memory ]
               ^memory_limit

Let's assume "unused memory" corresponds to exactly one DIMM that can
get hotunplugged. The complete DIMM isn't added to Linux and not online,
so the offline notifier will never trigger.

Via which mechanism would the unplug of that memory happen? On arm64,
this should only be possible via ACPI, when unplugging a DIMM that was
available since boot.

But won't acpi_memory_enable_device() try adding that memory while
ignoring the memory limit? And adding should work, no?

Can you share some details on how to trigger this on arm64?

-- 
Thanks,

David / dhildenb

Re: [PATCH V2] arm64/mm/hotplug: Warn when memory limit has been reduced

[David Hildenbrand]

On 03.01.22 07:21, Anshuman Khandual wrote:
> If the max memory limit has been reduced with 'mem=' kernel command line
> option, there might be UEFI memory map described memory beyond that limit
> which could be hot removed. This might be problematic for subsequent kexec
> kernel which could just access such removed memory.
> 
> Memory offline notifier exists because there is no other way to block the
> removal of boot memory, only the offlining (which isn't actually a problem)
> But with 'mem=', there is no chance to stop such boot memory being offlined
> as it where never in use by the kernel. As 'mem=' is a debug only option on
> arm64 platform, just warn for such a situation and move on.
> 

Hi,

but why warn on the offlining path?

IIUC, you'd have a layout like


[ boot memory ][ unused memory ]
               ^memory_limit

Let's assume "unused memory" corresponds to exactly one DIMM that can
get hotunplugged. The complete DIMM isn't added to Linux and not online,
so the offline notifier will never trigger.

Via which mechanism would the unplug of that memory happen? On arm64,
this should only be possible via ACPI, when unplugging a DIMM that was
available since boot.

But won't acpi_memory_enable_device() try adding that memory while
ignoring the memory limit? And adding should work, no?

Can you share some details on how to trigger this on arm64?

-- 
Thanks,

David / dhildenb


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH V2] arm64/mm/hotplug: Warn when memory limit has been reduced

[Anshuman Khandual]

On 1/3/22 3:25 PM, David Hildenbrand wrote:
> On 03.01.22 07:21, Anshuman Khandual wrote:
>> If the max memory limit has been reduced with 'mem=' kernel command line
>> option, there might be UEFI memory map described memory beyond that limit
>> which could be hot removed. This might be problematic for subsequent kexec
>> kernel which could just access such removed memory.
>>
>> Memory offline notifier exists because there is no other way to block the
>> removal of boot memory, only the offlining (which isn't actually a problem)
>> But with 'mem=', there is no chance to stop such boot memory being offlined
>> as it where never in use by the kernel. As 'mem=' is a debug only option on
>> arm64 platform, just warn for such a situation and move on.
>>
> 
> Hi,
> 
> but why warn on the offlining path?

The proposed change is just a one shot warning which is triggered during
the first memory offlining attempt, explaining potential kexec problems
as the kernel is already running on a trimmed boot memory.

> 
> IIUC, you'd have a layout like
> 
> 
> [ boot memory ][ unused memory ]
>                ^memory_limit

s/unused memory/unused boot memory/ instead as it is present during boot.

> 
> Let's assume "unused memory" corresponds to exactly one DIMM that can
> get hotunplugged. The complete DIMM isn't added to Linux and not online,
> so the offline notifier will never trigger.

Right, it will not trigger the warning. This proposal does not cover such
scenarios, where "unused memory" can just be "hotunplugged" without the
kernel knowing about it. But as the warning is not really dependent on
which exact memory is being offlined, user would still come to know if
there is an offlining attempt for any memory.

We probably might require another patch adding an warning when the boot
memory is trimmed with "mem=" cmdline, irrespective of hotplug support
in the kernel. It would ensure that the user is still warned about any
potential kexec problems, above scenario (i.e "unused memory" getting
"hotunplugged" without kernel knowing about it) might cause.

This patch just adds an warning in case there is an offlining attempt
on a hot-remove enabled kernel.

> 
> Via which mechanism would the unplug of that memory happen? On arm64,
> this should only be possible via ACPI, when unplugging a DIMM that was
> available since boot.
> 
> But won't acpi_memory_enable_device() try adding that memory while
> ignoring the memory limit? And adding should work, no?

Adding that memory via hotplug into the kernel first ? In that case
removal would still go via the kernel and user would know about it.

> 
> Can you share some details on how to trigger this on arm64?

The primary scenario this proposal is targeted towards is when boot
memory is set aside from the host, hot-plugged back into the kernel
and repurposed (via hotplug-hotremove path) for guest kernel usage.
This new warning would reassert that "mem=" cmdline option is debug
only on arm64 platform, and should not be used for production.

Re: [PATCH V2] arm64/mm/hotplug: Warn when memory limit has been reduced

[Anshuman Khandual]

On 1/3/22 3:25 PM, David Hildenbrand wrote:
> On 03.01.22 07:21, Anshuman Khandual wrote:
>> If the max memory limit has been reduced with 'mem=' kernel command line
>> option, there might be UEFI memory map described memory beyond that limit
>> which could be hot removed. This might be problematic for subsequent kexec
>> kernel which could just access such removed memory.
>>
>> Memory offline notifier exists because there is no other way to block the
>> removal of boot memory, only the offlining (which isn't actually a problem)
>> But with 'mem=', there is no chance to stop such boot memory being offlined
>> as it where never in use by the kernel. As 'mem=' is a debug only option on
>> arm64 platform, just warn for such a situation and move on.
>>
> 
> Hi,
> 
> but why warn on the offlining path?

The proposed change is just a one shot warning which is triggered during
the first memory offlining attempt, explaining potential kexec problems
as the kernel is already running on a trimmed boot memory.

> 
> IIUC, you'd have a layout like
> 
> 
> [ boot memory ][ unused memory ]
>                ^memory_limit

s/unused memory/unused boot memory/ instead as it is present during boot.

> 
> Let's assume "unused memory" corresponds to exactly one DIMM that can
> get hotunplugged. The complete DIMM isn't added to Linux and not online,
> so the offline notifier will never trigger.

Right, it will not trigger the warning. This proposal does not cover such
scenarios, where "unused memory" can just be "hotunplugged" without the
kernel knowing about it. But as the warning is not really dependent on
which exact memory is being offlined, user would still come to know if
there is an offlining attempt for any memory.

We probably might require another patch adding an warning when the boot
memory is trimmed with "mem=" cmdline, irrespective of hotplug support
in the kernel. It would ensure that the user is still warned about any
potential kexec problems, above scenario (i.e "unused memory" getting
"hotunplugged" without kernel knowing about it) might cause.

This patch just adds an warning in case there is an offlining attempt
on a hot-remove enabled kernel.

> 
> Via which mechanism would the unplug of that memory happen? On arm64,
> this should only be possible via ACPI, when unplugging a DIMM that was
> available since boot.
> 
> But won't acpi_memory_enable_device() try adding that memory while
> ignoring the memory limit? And adding should work, no?

Adding that memory via hotplug into the kernel first ? In that case
removal would still go via the kernel and user would know about it.

> 
> Can you share some details on how to trigger this on arm64?

The primary scenario this proposal is targeted towards is when boot
memory is set aside from the host, hot-plugged back into the kernel
and repurposed (via hotplug-hotremove path) for guest kernel usage.
This new warning would reassert that "mem=" cmdline option is debug
only on arm64 platform, and should not be used for production.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH V2] arm64/mm/hotplug: Warn when memory limit has been reduced

[David Hildenbrand]

>> Via which mechanism would the unplug of that memory happen? On arm64,
>> this should only be possible via ACPI, when unplugging a DIMM that was
>> available since boot.
>>
>> But won't acpi_memory_enable_device() try adding that memory while
>> ignoring the memory limit? And adding should work, no?
> 
> Adding that memory via hotplug into the kernel first ? In that case
> removal would still go via the kernel and user would know about it.

Can we please add details on how to actually trigger it (below) to the
patch description? Otherwise it's really hard to get about which senario
we do care, and about which we don't care.

> 
>>
>> Can you share some details on how to trigger this on arm64?
> 
> The primary scenario this proposal is targeted towards is when boot
> memory is set aside from the host, hot-plugged back into the kernel
> and repurposed (via hotplug-hotremove path) for guest kernel usage.
> This new warning would reassert that "mem=" cmdline option is debug
> only on arm64 platform, and should not be used for production.
Let me get this straight:

1. Restrict physical memory to use via "mem="

-> Some boot memory is !present and, therefore !early

2. Hotplug that memory to the kernel

-> How?

a) dax/kmem? Does not apply I think.
b) DIMM? Does not apply I think.
c) CONFIG_ARCH_MEMORY_PROBE ?

3. Trigger physical hotunplug and actually remove the memory

-> How?

4. kexec; will try using hotunplugged memory


-- 
Thanks,

David / dhildenb

Re: [PATCH V2] arm64/mm/hotplug: Warn when memory limit has been reduced

[David Hildenbrand]

>> Via which mechanism would the unplug of that memory happen? On arm64,
>> this should only be possible via ACPI, when unplugging a DIMM that was
>> available since boot.
>>
>> But won't acpi_memory_enable_device() try adding that memory while
>> ignoring the memory limit? And adding should work, no?
> 
> Adding that memory via hotplug into the kernel first ? In that case
> removal would still go via the kernel and user would know about it.

Can we please add details on how to actually trigger it (below) to the
patch description? Otherwise it's really hard to get about which senario
we do care, and about which we don't care.

> 
>>
>> Can you share some details on how to trigger this on arm64?
> 
> The primary scenario this proposal is targeted towards is when boot
> memory is set aside from the host, hot-plugged back into the kernel
> and repurposed (via hotplug-hotremove path) for guest kernel usage.
> This new warning would reassert that "mem=" cmdline option is debug
> only on arm64 platform, and should not be used for production.
Let me get this straight:

1. Restrict physical memory to use via "mem="

-> Some boot memory is !present and, therefore !early

2. Hotplug that memory to the kernel

-> How?

a) dax/kmem? Does not apply I think.
b) DIMM? Does not apply I think.
c) CONFIG_ARCH_MEMORY_PROBE ?

3. Trigger physical hotunplug and actually remove the memory

-> How?

4. kexec; will try using hotunplugged memory


-- 
Thanks,

David / dhildenb


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH V2] arm64/mm/hotplug: Warn when memory limit has been reduced

[Anshuman Khandual]

On 1/3/22 6:27 PM, David Hildenbrand wrote:
> 
>>> Via which mechanism would the unplug of that memory happen? On arm64,
>>> this should only be possible via ACPI, when unplugging a DIMM that was
>>> available since boot.
>>>
>>> But won't acpi_memory_enable_device() try adding that memory while
>>> ignoring the memory limit? And adding should work, no?
>>
>> Adding that memory via hotplug into the kernel first ? In that case
>> removal would still go via the kernel and user would know about it.
> 
> Can we please add details on how to actually trigger it (below) to the
> patch description? Otherwise it's really hard to get about which senario
> we do care, and about which we don't care.

Sure, will try and add those details in the commit message.

> 
>>
>>>
>>> Can you share some details on how to trigger this on arm64?
>>
>> The primary scenario this proposal is targeted towards is when boot
>> memory is set aside from the host, hot-plugged back into the kernel
>> and repurposed (via hotplug-hotremove path) for guest kernel usage.
>> This new warning would reassert that "mem=" cmdline option is debug
>> only on arm64 platform, and should not be used for production.
> Let me get this straight:
> 
> 1. Restrict physical memory to use via "mem="
> 
> -> Some boot memory is !present and, therefore !early
> 
> 2. Hotplug that memory to the kernel
> 
> -> How?
> 
> a) dax/kmem? Does not apply I think.
> b) DIMM? Does not apply I think.
> c) CONFIG_ARCH_MEMORY_PROBE ?
> 
> 3. Trigger physical hotunplug and actually remove the memory
> 
> -> How?
> 
> 4. kexec; will try using hotunplugged memory
> 
> 

Re: [PATCH V2] arm64/mm/hotplug: Warn when memory limit has been reduced

[Anshuman Khandual]

On 1/3/22 6:27 PM, David Hildenbrand wrote:
> 
>>> Via which mechanism would the unplug of that memory happen? On arm64,
>>> this should only be possible via ACPI, when unplugging a DIMM that was
>>> available since boot.
>>>
>>> But won't acpi_memory_enable_device() try adding that memory while
>>> ignoring the memory limit? And adding should work, no?
>>
>> Adding that memory via hotplug into the kernel first ? In that case
>> removal would still go via the kernel and user would know about it.
> 
> Can we please add details on how to actually trigger it (below) to the
> patch description? Otherwise it's really hard to get about which senario
> we do care, and about which we don't care.

Sure, will try and add those details in the commit message.

> 
>>
>>>
>>> Can you share some details on how to trigger this on arm64?
>>
>> The primary scenario this proposal is targeted towards is when boot
>> memory is set aside from the host, hot-plugged back into the kernel
>> and repurposed (via hotplug-hotremove path) for guest kernel usage.
>> This new warning would reassert that "mem=" cmdline option is debug
>> only on arm64 platform, and should not be used for production.
> Let me get this straight:
> 
> 1. Restrict physical memory to use via "mem="
> 
> -> Some boot memory is !present and, therefore !early
> 
> 2. Hotplug that memory to the kernel
> 
> -> How?
> 
> a) dax/kmem? Does not apply I think.
> b) DIMM? Does not apply I think.
> c) CONFIG_ARCH_MEMORY_PROBE ?
> 
> 3. Trigger physical hotunplug and actually remove the memory
> 
> -> How?
> 
> 4. kexec; will try using hotunplugged memory
> 
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel