[PATCH v5 0/5] introduce mirrored memory support for arm64

[PATCH v5 0/5] introduce mirrored memory support for arm64

[Wupeng Ma]

From: Ma Wupeng <mawupeng1@huawei.com>

Commit b05b9f5f9dcf ("x86, mirror: x86 enabling - find mirrored memory ranges")
introduced mirrored memory support for x86. This support rely on UEFI to
report mirrored memory address ranges.  See UEFI 2.5 spec pages 157-158:

  http://www.uefi.org/sites/default/files/resources/UEFI%202_5.pdf

Memory mirroring is a technique used to separate memory into two separate
channels, usually on a memory device, like a server. In memory mirroring,
one channel is copied to another to create redundancy. This method makes
input/output (I/O) registers and memory appear with more than one address
range because the same physical byte is accessible at more than one
address. Using memory mirroring, higher memory reliability and a higher
level of memory consolidation are possible.

These EFI memory regions have various attributes, and the "mirrored"
attribute is one of them. The physical memory region whose descriptors
in EFI memory map has EFI_MEMORY_MORE_RELIABLE attribute (bit: 16) are
mirrored. The address range mirroring feature of the kernel arranges such
mirrored regions into normal zones and other regions into movable zones.

Arm64 can support this too. So mirrored memory support is added to support
arm64.

The main purpose of this patch set is to introduce mirrored support for
arm64 and we have already fixed the problems we had which is shown in
patch #5 to patch #8 and try to bring total isolation in patch #9 which
will disable mirror feature if kernelcore is not specified.

In order to test this support in arm64:
- patch this patch set
- add kernelcore=mirror in kernel parameter
- start you kernel

Patch #1 introduce mirrored memory support form arm64.
Patch #2-#4 fix some bugs for arm64 if memory reliable is enabled.
Patch #5 disable mirror feature if kernelcore is not specified.

Thanks to Ard Biesheuvel's hard work [1], now kernel will perfer mirrored
memory if kaslr is enabled.

[1] https://lore.kernel.org/linux-arm-kernel/CAMj1kXEPVEzMgOM4+Yj6PxHA-jFuDOAUdDJSiSxy_XaP4P7LSw@mail.gmail.com/T/

Changelog since v4:
- merge the first two patches into one
- change __initdata to __initdata_memblock in patch #5

Changelog since v3:
- limit warning message in vmemmap_verify via pr_warn_once()
- only clear memblock_nomap flags rather than bring the mirrored flag back
- disable mirrored feature in memblock_mark_mirror()

Changelog since v2:
- remove efi_fake_mem support
- remove Commit ("remove some redundant code in ia64 efi_init") since
  efi_print_memmap() is not public
- add mirror flag back on initrd memory

Changelog since v1:
- update changelog in cover letter
- use PHYS_PFN in patch #7

Ma Wupeng (5):
  efi: arm64: Introduce ability to find mirrored memory ranges
  mm: Ratelimited mirrored memory related warning messages
  mm: Limit warning message in vmemmap_verify() to once
  arm64: mm: Only remove nomap flag for initrd
  memblock: Disable mirror feature if kernelcore is not specified

 arch/arm64/mm/init.c            |  2 +-
 arch/x86/include/asm/efi.h      |  4 ----
 arch/x86/platform/efi/efi.c     | 23 -----------------------
 drivers/firmware/efi/efi-init.c |  1 +
 drivers/firmware/efi/efi.c      | 23 +++++++++++++++++++++++
 include/linux/efi.h             |  3 +++
 mm/internal.h                   |  2 ++
 mm/memblock.c                   |  7 +++++--
 mm/page_alloc.c                 |  2 +-
 mm/sparse-vmemmap.c             |  2 +-
 10 files changed, 37 insertions(+), 32 deletions(-)

-- 
2.25.1

[PATCH v5 1/5] efi: arm64: Introduce ability to find mirrored memory ranges

[Wupeng Ma]

From: Ma Wupeng <mawupeng1@huawei.com>

Commit b05b9f5f9dcf ("x86, mirror: x86 enabling - find mirrored memory
ranges") introduce the efi_find_mirror() function on x86. In order to reuse
the API we make it public.

Arm64 can support mirrored memory too, so function efi_find_mirror() is added to
efi_init() to this support for arm64.

Since efi_init() is shared by ARM, arm64 and riscv, this patch will bring
mirror memory support for these architectures, but this support is only tested
in arm64.

Signed-off-by: Ma Wupeng <mawupeng1@huawei.com>
---
 arch/x86/include/asm/efi.h      |  4 ----
 arch/x86/platform/efi/efi.c     | 23 -----------------------
 drivers/firmware/efi/efi-init.c |  1 +
 drivers/firmware/efi/efi.c      | 23 +++++++++++++++++++++++
 include/linux/efi.h             |  3 +++
 5 files changed, 27 insertions(+), 27 deletions(-)

diff --git a/arch/x86/include/asm/efi.h b/arch/x86/include/asm/efi.h
index 71943dce691e..eb90206eae80 100644
--- a/arch/x86/include/asm/efi.h
+++ b/arch/x86/include/asm/efi.h
@@ -383,7 +383,6 @@ static inline bool efi_is_64bit(void)
 extern bool efi_reboot_required(void);
 extern bool efi_is_table_address(unsigned long phys_addr);
 
-extern void efi_find_mirror(void);
 extern void efi_reserve_boot_services(void);
 #else
 static inline void parse_efi_setup(u64 phys_addr, u32 data_len) {}
@@ -395,9 +394,6 @@ static inline  bool efi_is_table_address(unsigned long phys_addr)
 {
 	return false;
 }
-static inline void efi_find_mirror(void)
-{
-}
 static inline void efi_reserve_boot_services(void)
 {
 }
diff --git a/arch/x86/platform/efi/efi.c b/arch/x86/platform/efi/efi.c
index 1591d67e0bcd..6e598bd78eef 100644
--- a/arch/x86/platform/efi/efi.c
+++ b/arch/x86/platform/efi/efi.c
@@ -108,29 +108,6 @@ static int __init setup_add_efi_memmap(char *arg)
 }
 early_param("add_efi_memmap", setup_add_efi_memmap);
 
-void __init efi_find_mirror(void)
-{
-	efi_memory_desc_t *md;
-	u64 mirror_size = 0, total_size = 0;
-
-	if (!efi_enabled(EFI_MEMMAP))
-		return;
-
-	for_each_efi_memory_desc(md) {
-		unsigned long long start = md->phys_addr;
-		unsigned long long size = md->num_pages << EFI_PAGE_SHIFT;
-
-		total_size += size;
-		if (md->attribute & EFI_MEMORY_MORE_RELIABLE) {
-			memblock_mark_mirror(start, size);
-			mirror_size += size;
-		}
-	}
-	if (mirror_size)
-		pr_info("Memory: %lldM/%lldM mirrored memory\n",
-			mirror_size>>20, total_size>>20);
-}
-
 /*
  * Tell the kernel about the EFI memory map.  This might include
  * more than the max 128 entries that can fit in the passed in e820
diff --git a/drivers/firmware/efi/efi-init.c b/drivers/firmware/efi/efi-init.c
index b2c829e95bd1..8e3a0d14dac5 100644
--- a/drivers/firmware/efi/efi-init.c
+++ b/drivers/firmware/efi/efi-init.c
@@ -241,6 +241,7 @@ void __init efi_init(void)
 	 */
 	early_init_dt_check_for_usable_mem_range();
+	efi_find_mirror();
 	efi_esrt_init();
 	efi_mokvar_table_init();
 
 	memblock_reserve(data.phys_map & PAGE_MASK,
diff --git a/drivers/firmware/efi/efi.c b/drivers/firmware/efi/efi.c
index 860534bcfdac..79c232e07de7 100644
--- a/drivers/firmware/efi/efi.c
+++ b/drivers/firmware/efi/efi.c
@@ -446,6 +446,29 @@ static int __init efisubsys_init(void)
 
 subsys_initcall(efisubsys_init);
 
+void __init efi_find_mirror(void)
+{
+	efi_memory_desc_t *md;
+	u64 mirror_size = 0, total_size = 0;
+
+	if (!efi_enabled(EFI_MEMMAP))
+		return;
+
+	for_each_efi_memory_desc(md) {
+		unsigned long long start = md->phys_addr;
+		unsigned long long size = md->num_pages << EFI_PAGE_SHIFT;
+
+		total_size += size;
+		if (md->attribute & EFI_MEMORY_MORE_RELIABLE) {
+			memblock_mark_mirror(start, size);
+			mirror_size += size;
+		}
+	}
+	if (mirror_size)
+		pr_info("Memory: %lldM/%lldM mirrored memory\n",
+			mirror_size>>20, total_size>>20);
+}
+
 /*
  * Find the efi memory descriptor for a given physical address.  Given a
  * physical address, determine if it exists within an EFI Memory Map entry,
diff --git a/include/linux/efi.h b/include/linux/efi.h
index 7d9b0bb47eb3..53f64c14a525 100644
--- a/include/linux/efi.h
+++ b/include/linux/efi.h
@@ -872,6 +872,7 @@ static inline bool efi_rt_services_supported(unsigned int mask)
 {
 	return (efi.runtime_supported_mask & mask) == mask;
 }
+extern void efi_find_mirror(void);
 #else
 static inline bool efi_enabled(int feature)
 {
@@ -889,6 +890,8 @@ static inline bool efi_rt_services_supported(unsigned int mask)
 {
 	return false;
 }
+
+static inline void efi_find_mirror(void) {}
 #endif
 
 extern int efi_status_to_err(efi_status_t status);
-- 
2.25.1

[PATCH v5 2/5] mm: Ratelimited mirrored memory related warning messages

[Wupeng Ma]

From: Ma Wupeng <mawupeng1@huawei.com>

If system has mirrored memory, memblock will try to allocate mirrored
memory firstly and fallback to non-mirrored memory when fails, but if with
limited mirrored memory or some numa node without mirrored memory, lots of
warning message about memblock allocation will occur.

This patch ratelimit the warning message to avoid a very long print during
bootup.

Signed-off-by: Ma Wupeng <mawupeng1@huawei.com>
Reviewed-by: David Hildenbrand <david@redhat.com>
Acked-by: Mike Rapoport <rppt@linux.ibm.com>
Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>
Reviewed-by: Kefeng Wang <wangkefeng.wang@huawei.com>
---
 mm/memblock.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/memblock.c b/mm/memblock.c
index e4f03a6e8e56..b1d2a0009733 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -327,7 +327,7 @@ static phys_addr_t __init_memblock memblock_find_in_range(phys_addr_t start,
 					    NUMA_NO_NODE, flags);
 
 	if (!ret && (flags & MEMBLOCK_MIRROR)) {
-		pr_warn("Could not allocate %pap bytes of mirrored memory\n",
+		pr_warn_ratelimited("Could not allocate %pap bytes of mirrored memory\n",
 			&size);
 		flags &= ~MEMBLOCK_MIRROR;
 		goto again;
@@ -1384,7 +1384,7 @@ phys_addr_t __init memblock_alloc_range_nid(phys_addr_t size,
 
 	if (flags & MEMBLOCK_MIRROR) {
 		flags &= ~MEMBLOCK_MIRROR;
-		pr_warn("Could not allocate %pap bytes of mirrored memory\n",
+		pr_warn_ratelimited("Could not allocate %pap bytes of mirrored memory\n",
 			&size);
 		goto again;
 	}
-- 
2.25.1

[PATCH v5 3/5] mm: Limit warning message in vmemmap_verify() to once

[Wupeng Ma]

From: Ma Wupeng <mawupeng1@huawei.com>

For a system only have limited mirrored memory or some numa node without
mirrored memory, the per node vmemmap page_structs prefer to allocate
memory from mirrored region, which will lead to vmemmap_verify() in
vmemmap_populate_basepages() report lots of warning message.

This patch change the frequency of "potential offnode page_structs" warning
messages to only once to avoid a very long print during bootup.

Signed-off-by: Ma Wupeng <mawupeng1@huawei.com>
Acked-by: David Hildenbrand <david@redhat.com>
---
 mm/sparse-vmemmap.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index f4fa61dbbee3..f34c6889b0a6 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -528,7 +528,7 @@ void __meminit vmemmap_verify(pte_t *pte, int node,
 	int actual_node = early_pfn_to_nid(pfn);
 
 	if (node_distance(actual_node, node) > LOCAL_DISTANCE)
-		pr_warn("[%lx-%lx] potential offnode page_structs\n",
+		pr_warn_once("[%lx-%lx] potential offnode page_structs\n",
 			start, end - 1);
 }
 
-- 
2.25.1

[PATCH v5 4/5] arm64: mm: Only remove nomap flag for initrd

[Wupeng Ma]

From: Ma Wupeng <mawupeng1@huawei.com>

Commit 177e15f0c144 ("arm64: add the initrd region to the linear mapping explicitly")
remove all the flags of the memory used by initrd. This is fine since
MEMBLOCK_MIRROR is not used in arm64.

However with mirrored feature introduced to arm64, this will clear the mirrored
flag used by initrd, which will lead to error log printed by
find_zone_movable_pfns_for_nodes() if the lower 4G range has some non-mirrored
memory.

To solve this problem, only MEMBLOCK_NOMAP flag will be removed via
memblock_clear_nomap().

Signed-off-by: Ma Wupeng <mawupeng1@huawei.com>
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
---
 arch/arm64/mm/init.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 339ee84e5a61..8456dbae9441 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -350,8 +350,8 @@ void __init arm64_memblock_init(void)
 			"initrd not fully accessible via the linear mapping -- please check your bootloader ...\n")) {
 			phys_initrd_size = 0;
 		} else {
-			memblock_remove(base, size); /* clear MEMBLOCK_ flags */
 			memblock_add(base, size);
+			memblock_clear_nomap(base, size);
 			memblock_reserve(base, size);
 		}
 	}
-- 
2.25.1

[PATCH v5 5/5] memblock: Disable mirror feature if kernelcore is not specified

[Wupeng Ma]

From: Ma Wupeng <mawupeng1@huawei.com>

If system have some mirrored memory and mirrored feature is not specified
in boot parameter, the basic mirrored feature will be enabled and this will
lead to the following situations:

- memblock memory allocation prefers mirrored region. This may have some
  unexpected influence on numa affinity.

- contiguous memory will be split into several parts if parts of them
  is mirrored memory via memblock_mark_mirror().

To fix this, variable mirrored_kernelcore will be checked in
memblock_mark_mirror(). Mark mirrored memory with flag MEMBLOCK_MIRROR iff
kernelcore=mirror is added in the kernel parameters.

Signed-off-by: Ma Wupeng <mawupeng1@huawei.com>
Acked-by: Ard Biesheuvel <ardb@kernel.org>
---
 mm/internal.h   | 2 ++
 mm/memblock.c   | 3 +++
 mm/page_alloc.c | 2 +-
 3 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/mm/internal.h b/mm/internal.h
index c0f8fbe0445b..ddd2d6a46f1b 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -861,4 +861,6 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags);
 
 DECLARE_PER_CPU(struct per_cpu_nodestat, boot_nodestats);
 
+extern bool mirrored_kernelcore;
+
 #endif	/* __MM_INTERNAL_H */
diff --git a/mm/memblock.c b/mm/memblock.c
index b1d2a0009733..a9f18b988b7f 100644
--- a/mm/memblock.c
+++ b/mm/memblock.c
@@ -924,6 +924,9 @@ int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size)
  */
 int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size)
 {
+	if (!mirrored_kernelcore)
+		return 0;
+
 	system_has_some_mirror = true;
 
 	return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR);
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index e008a3df0485..10dc35ec7479 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -356,7 +356,7 @@ static unsigned long required_kernelcore_percent __initdata;
 static unsigned long required_movablecore __initdata;
 static unsigned long required_movablecore_percent __initdata;
 static unsigned long zone_movable_pfn[MAX_NUMNODES] __initdata;
-static bool mirrored_kernelcore __meminitdata;
+bool mirrored_kernelcore __initdata_memblock;
 
 /* movable_zone is the "real" zone pages in ZONE_MOVABLE are taken from */
 int movable_zone;
-- 
2.25.1

Re: [PATCH v5 5/5] memblock: Disable mirror feature if kernelcore is not specified

[Baoquan He]

On 06/14/22 at 05:21pm, Wupeng Ma wrote:
> From: Ma Wupeng <mawupeng1@huawei.com>
> 
> If system have some mirrored memory and mirrored feature is not specified
> in boot parameter, the basic mirrored feature will be enabled and this will
> lead to the following situations:
> 
> - memblock memory allocation prefers mirrored region. This may have some
>   unexpected influence on numa affinity.
> 
> - contiguous memory will be split into several parts if parts of them
>   is mirrored memory via memblock_mark_mirror().
> 
> To fix this, variable mirrored_kernelcore will be checked in
> memblock_mark_mirror(). Mark mirrored memory with flag MEMBLOCK_MIRROR iff
> kernelcore=mirror is added in the kernel parameters.
> 
> Signed-off-by: Ma Wupeng <mawupeng1@huawei.com>
> Acked-by: Ard Biesheuvel <ardb@kernel.org>
> ---
>  mm/internal.h   | 2 ++
>  mm/memblock.c   | 3 +++
>  mm/page_alloc.c | 2 +-
>  3 files changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/mm/internal.h b/mm/internal.h
> index c0f8fbe0445b..ddd2d6a46f1b 100644
> --- a/mm/internal.h
> +++ b/mm/internal.h
> @@ -861,4 +861,6 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags);
>  
>  DECLARE_PER_CPU(struct per_cpu_nodestat, boot_nodestats);
>  
> +extern bool mirrored_kernelcore;
> +
>  #endif	/* __MM_INTERNAL_H */
> diff --git a/mm/memblock.c b/mm/memblock.c
> index b1d2a0009733..a9f18b988b7f 100644
> --- a/mm/memblock.c
> +++ b/mm/memblock.c
> @@ -924,6 +924,9 @@ int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size)
>   */
>  int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size)
>  {
> +	if (!mirrored_kernelcore)
> +		return 0;

memblock_mark_mirror() is just a wrapper, maybe we should check this in
efi_find_mirror(). Otherwise, how do we explain the message printed out
at below in boot log if we don't mark mirror memory at all?

void __init efi_find_mirror(void)
{
......
	if (mirror_size)
                pr_info("Memory: %lldM/%lldM mirrored memory\n",
                        mirror_size>>20, total_size>>20);
}

> +
>  	system_has_some_mirror = true;
>  
>  	return memblock_setclr_flag(base, size, 1, MEMBLOCK_MIRROR);
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index e008a3df0485..10dc35ec7479 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -356,7 +356,7 @@ static unsigned long required_kernelcore_percent __initdata;
>  static unsigned long required_movablecore __initdata;
>  static unsigned long required_movablecore_percent __initdata;
>  static unsigned long zone_movable_pfn[MAX_NUMNODES] __initdata;
> -static bool mirrored_kernelcore __meminitdata;
> +bool mirrored_kernelcore __initdata_memblock;
>  
>  /* movable_zone is the "real" zone pages in ZONE_MOVABLE are taken from */
>  int movable_zone;
> -- 
> 2.25.1
> 
> 

Re: [PATCH v5 5/5] memblock: Disable mirror feature if kernelcore is not specified

[Ard Biesheuvel]

On Tue, 14 Jun 2022 at 12:20, Baoquan He <bhe@redhat.com> wrote:
>
> On 06/14/22 at 05:21pm, Wupeng Ma wrote:
> > From: Ma Wupeng <mawupeng1@huawei.com>
> >
> > If system have some mirrored memory and mirrored feature is not specified
> > in boot parameter, the basic mirrored feature will be enabled and this will
> > lead to the following situations:
> >
> > - memblock memory allocation prefers mirrored region. This may have some
> >   unexpected influence on numa affinity.
> >
> > - contiguous memory will be split into several parts if parts of them
> >   is mirrored memory via memblock_mark_mirror().
> >
> > To fix this, variable mirrored_kernelcore will be checked in
> > memblock_mark_mirror(). Mark mirrored memory with flag MEMBLOCK_MIRROR iff
> > kernelcore=mirror is added in the kernel parameters.
> >
> > Signed-off-by: Ma Wupeng <mawupeng1@huawei.com>
> > Acked-by: Ard Biesheuvel <ardb@kernel.org>
> > ---
> >  mm/internal.h   | 2 ++
> >  mm/memblock.c   | 3 +++
> >  mm/page_alloc.c | 2 +-
> >  3 files changed, 6 insertions(+), 1 deletion(-)
> >
> > diff --git a/mm/internal.h b/mm/internal.h
> > index c0f8fbe0445b..ddd2d6a46f1b 100644
> > --- a/mm/internal.h
> > +++ b/mm/internal.h
> > @@ -861,4 +861,6 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags);
> >
> >  DECLARE_PER_CPU(struct per_cpu_nodestat, boot_nodestats);
> >
> > +extern bool mirrored_kernelcore;
> > +
> >  #endif       /* __MM_INTERNAL_H */
> > diff --git a/mm/memblock.c b/mm/memblock.c
> > index b1d2a0009733..a9f18b988b7f 100644
> > --- a/mm/memblock.c
> > +++ b/mm/memblock.c
> > @@ -924,6 +924,9 @@ int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size)
> >   */
> >  int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size)
> >  {
> > +     if (!mirrored_kernelcore)
> > +             return 0;
>
> memblock_mark_mirror() is just a wrapper, maybe we should check this in
> efi_find_mirror(). Otherwise, how do we explain the message printed out
> at below in boot log if we don't mark mirror memory at all?
>
> void __init efi_find_mirror(void)
> {
> ......
>         if (mirror_size)
>                 pr_info("Memory: %lldM/%lldM mirrored memory\n",
>                         mirror_size>>20, total_size>>20);
> }
>

EFI does not care about *how* mirrored memory is being used or not, it
just reports what the firmware provided. So EFI is not the appropriate
level to take kernelcore=mirror into account.

I already mentioned that memblock_mark_mirror() is also the wrong
place IMO, but Kefeng explained that doing it elsewhere is
problematic.

Re: [PATCH v5 5/5] memblock: Disable mirror feature if kernelcore is not specified

[Baoquan He]

On 06/14/22 at 12:27pm, Ard Biesheuvel wrote:
> On Tue, 14 Jun 2022 at 12:20, Baoquan He <bhe@redhat.com> wrote:
> >
> > On 06/14/22 at 05:21pm, Wupeng Ma wrote:
> > > From: Ma Wupeng <mawupeng1@huawei.com>
> > >
> > > If system have some mirrored memory and mirrored feature is not specified
> > > in boot parameter, the basic mirrored feature will be enabled and this will
> > > lead to the following situations:
> > >
> > > - memblock memory allocation prefers mirrored region. This may have some
> > >   unexpected influence on numa affinity.
> > >
> > > - contiguous memory will be split into several parts if parts of them
> > >   is mirrored memory via memblock_mark_mirror().
> > >
> > > To fix this, variable mirrored_kernelcore will be checked in
> > > memblock_mark_mirror(). Mark mirrored memory with flag MEMBLOCK_MIRROR iff
> > > kernelcore=mirror is added in the kernel parameters.
> > >
> > > Signed-off-by: Ma Wupeng <mawupeng1@huawei.com>
> > > Acked-by: Ard Biesheuvel <ardb@kernel.org>
> > > ---
> > >  mm/internal.h   | 2 ++
> > >  mm/memblock.c   | 3 +++
> > >  mm/page_alloc.c | 2 +-
> > >  3 files changed, 6 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/mm/internal.h b/mm/internal.h
> > > index c0f8fbe0445b..ddd2d6a46f1b 100644
> > > --- a/mm/internal.h
> > > +++ b/mm/internal.h
> > > @@ -861,4 +861,6 @@ struct folio *try_grab_folio(struct page *page, int refs, unsigned int flags);
> > >
> > >  DECLARE_PER_CPU(struct per_cpu_nodestat, boot_nodestats);
> > >
> > > +extern bool mirrored_kernelcore;
> > > +
> > >  #endif       /* __MM_INTERNAL_H */
> > > diff --git a/mm/memblock.c b/mm/memblock.c
> > > index b1d2a0009733..a9f18b988b7f 100644
> > > --- a/mm/memblock.c
> > > +++ b/mm/memblock.c
> > > @@ -924,6 +924,9 @@ int __init_memblock memblock_clear_hotplug(phys_addr_t base, phys_addr_t size)
> > >   */
> > >  int __init_memblock memblock_mark_mirror(phys_addr_t base, phys_addr_t size)
> > >  {
> > > +     if (!mirrored_kernelcore)
> > > +             return 0;
> >
> > memblock_mark_mirror() is just a wrapper, maybe we should check this in
> > efi_find_mirror(). Otherwise, how do we explain the message printed out
> > at below in boot log if we don't mark mirror memory at all?
> >
> > void __init efi_find_mirror(void)
> > {
> > ......
> >         if (mirror_size)
> >                 pr_info("Memory: %lldM/%lldM mirrored memory\n",
> >                         mirror_size>>20, total_size>>20);
> > }
> >
> 
> EFI does not care about *how* mirrored memory is being used or not, it
> just reports what the firmware provided. So EFI is not the appropriate
> level to take kernelcore=mirror into account.
> 
> I already mentioned that memblock_mark_mirror() is also the wrong
> place IMO, but Kefeng explained that doing it elsewhere is
> problematic.

OK, seems we have no better choice other than these two. 

> 

Re: [PATCH v5 0/5] introduce mirrored memory support for arm64

[Kefeng Wang]

On 2022/6/14 17:21, Wupeng Ma wrote:
> From: Ma Wupeng <mawupeng1@huawei.com>
>
> Commit b05b9f5f9dcf ("x86, mirror: x86 enabling - find mirrored memory ranges")
> introduced mirrored memory support for x86. This support rely on UEFI to
> report mirrored memory address ranges.  See UEFI 2.5 spec pages 157-158:
>
>    http://www.uefi.org/sites/default/files/resources/UEFI%202_5.pdf
>
> Memory mirroring is a technique used to separate memory into two separate
> channels, usually on a memory device, like a server. In memory mirroring,
> one channel is copied to another to create redundancy. This method makes
> input/output (I/O) registers and memory appear with more than one address
> range because the same physical byte is accessible at more than one
> address. Using memory mirroring, higher memory reliability and a higher
> level of memory consolidation are possible.
>
> These EFI memory regions have various attributes, and the "mirrored"
> attribute is one of them. The physical memory region whose descriptors
> in EFI memory map has EFI_MEMORY_MORE_RELIABLE attribute (bit: 16) are
> mirrored. The address range mirroring feature of the kernel arranges such
> mirrored regions into normal zones and other regions into movable zones.
>
> Arm64 can support this too. So mirrored memory support is added to support
> arm64.
>
> The main purpose of this patch set is to introduce mirrored support for
> arm64 and we have already fixed the problems we had which is shown in
> patch #5 to patch #8 and try to bring total isolation in patch #9 which
> will disable mirror feature if kernelcore is not specified.
>
> In order to test this support in arm64:
> - patch this patch set
> - add kernelcore=mirror in kernel parameter
> - start you kernel
>
> Patch #1 introduce mirrored memory support form arm64.
> Patch #2-#4 fix some bugs for arm64 if memory reliable is enabled.
> Patch #5 disable mirror feature if kernelcore is not specified.
>
> Thanks to Ard Biesheuvel's hard work [1], now kernel will perfer mirrored
> memory if kaslr is enabled.
>
> [1] https://lore.kernel.org/linux-arm-kernel/CAMj1kXEPVEzMgOM4+Yj6PxHA-jFuDOAUdDJSiSxy_XaP4P7LSw@mail.gmail.com/T/

For series,

Reviewed-by: Kefeng Wang<wangkefeng.wang@huawei.com>

>
> Changelog since v4:
> - merge the first two patches into one
> - change __initdata to __initdata_memblock in patch #5
>
> Changelog since v3:
> - limit warning message in vmemmap_verify via pr_warn_once()
> - only clear memblock_nomap flags rather than bring the mirrored flag back
> - disable mirrored feature in memblock_mark_mirror()
>
> Changelog since v2:
> - remove efi_fake_mem support
> - remove Commit ("remove some redundant code in ia64 efi_init") since
>    efi_print_memmap() is not public
> - add mirror flag back on initrd memory
>
> Changelog since v1:
> - update changelog in cover letter
> - use PHYS_PFN in patch #7
>
> Ma Wupeng (5):
>    efi: arm64: Introduce ability to find mirrored memory ranges
>    mm: Ratelimited mirrored memory related warning messages
>    mm: Limit warning message in vmemmap_verify() to once
>    arm64: mm: Only remove nomap flag for initrd
>    memblock: Disable mirror feature if kernelcore is not specified
>
>   arch/arm64/mm/init.c            |  2 +-
>   arch/x86/include/asm/efi.h      |  4 ----
>   arch/x86/platform/efi/efi.c     | 23 -----------------------
>   drivers/firmware/efi/efi-init.c |  1 +
>   drivers/firmware/efi/efi.c      | 23 +++++++++++++++++++++++
>   include/linux/efi.h             |  3 +++
>   mm/internal.h                   |  2 ++
>   mm/memblock.c                   |  7 +++++--
>   mm/page_alloc.c                 |  2 +-
>   mm/sparse-vmemmap.c             |  2 +-
>   10 files changed, 37 insertions(+), 32 deletions(-)
>

Re: [PATCH v5 0/5] introduce mirrored memory support for arm64

[Mike Rapoport]

On Tue, Jun 14, 2022 at 05:21:51PM +0800, Wupeng Ma wrote:
> From: Ma Wupeng <mawupeng1@huawei.com>
> 
> Patch #1 introduce mirrored memory support form arm64.
> Patch #2-#4 fix some bugs for arm64 if memory reliable is enabled.
> Patch #5 disable mirror feature if kernelcore is not specified.
> 
> Thanks to Ard Biesheuvel's hard work [1], now kernel will perfer mirrored
> memory if kaslr is enabled.
> 
> [1] https://lore.kernel.org/linux-arm-kernel/CAMj1kXEPVEzMgOM4+Yj6PxHA-jFuDOAUdDJSiSxy_XaP4P7LSw@mail.gmail.com/T/
> 
> 
> Ma Wupeng (5):
>   efi: arm64: Introduce ability to find mirrored memory ranges
>   mm: Ratelimited mirrored memory related warning messages
>   mm: Limit warning message in vmemmap_verify() to once
>   arm64: mm: Only remove nomap flag for initrd
>   memblock: Disable mirror feature if kernelcore is not specified
> 
>  arch/arm64/mm/init.c            |  2 +-
>  arch/x86/include/asm/efi.h      |  4 ----
>  arch/x86/platform/efi/efi.c     | 23 -----------------------
>  drivers/firmware/efi/efi-init.c |  1 +
>  drivers/firmware/efi/efi.c      | 23 +++++++++++++++++++++++
>  include/linux/efi.h             |  3 +++
>  mm/internal.h                   |  2 ++
>  mm/memblock.c                   |  7 +++++--
>  mm/page_alloc.c                 |  2 +-
>  mm/sparse-vmemmap.c             |  2 +-
>  10 files changed, 37 insertions(+), 32 deletions(-)

For the series: Acked-by: Mike Rapoport <rppt@linux.ibm.com>

> -- 
> 2.25.1
> 

-- 
Sincerely yours,
Mike.

Re: [PATCH v5 1/5] efi: arm64: Introduce ability to find mirrored memory ranges

[Catalin Marinas]

On Tue, Jun 14, 2022 at 05:21:52PM +0800, Wupeng Ma wrote:
> From: Ma Wupeng <mawupeng1@huawei.com>
> 
> Commit b05b9f5f9dcf ("x86, mirror: x86 enabling - find mirrored memory
> ranges") introduce the efi_find_mirror() function on x86. In order to reuse
> the API we make it public.
> 
> Arm64 can support mirrored memory too, so function efi_find_mirror() is added to
> efi_init() to this support for arm64.
> 
> Since efi_init() is shared by ARM, arm64 and riscv, this patch will bring
> mirror memory support for these architectures, but this support is only tested
> in arm64.
> 
> Signed-off-by: Ma Wupeng <mawupeng1@huawei.com>
> ---
>  arch/x86/include/asm/efi.h      |  4 ----
>  arch/x86/platform/efi/efi.c     | 23 -----------------------
>  drivers/firmware/efi/efi-init.c |  1 +
>  drivers/firmware/efi/efi.c      | 23 +++++++++++++++++++++++
>  include/linux/efi.h             |  3 +++

The subject prefix says "efi: arm64: " but it looks like x86 diffstat
here. You may want to get an ack from the x86 maintainers.

-- 
Catalin

Re: [PATCH v5 1/5] efi: arm64: Introduce ability to find mirrored memory ranges

[Ard Biesheuvel]

On Wed, 15 Jun 2022 at 12:02, Catalin Marinas <catalin.marinas@arm.com> wrote:
>
> On Tue, Jun 14, 2022 at 05:21:52PM +0800, Wupeng Ma wrote:
> > From: Ma Wupeng <mawupeng1@huawei.com>
> >
> > Commit b05b9f5f9dcf ("x86, mirror: x86 enabling - find mirrored memory
> > ranges") introduce the efi_find_mirror() function on x86. In order to reuse
> > the API we make it public.
> >
> > Arm64 can support mirrored memory too, so function efi_find_mirror() is added to
> > efi_init() to this support for arm64.
> >
> > Since efi_init() is shared by ARM, arm64 and riscv, this patch will bring
> > mirror memory support for these architectures, but this support is only tested
> > in arm64.
> >
> > Signed-off-by: Ma Wupeng <mawupeng1@huawei.com>
> > ---
> >  arch/x86/include/asm/efi.h      |  4 ----
> >  arch/x86/platform/efi/efi.c     | 23 -----------------------
> >  drivers/firmware/efi/efi-init.c |  1 +
> >  drivers/firmware/efi/efi.c      | 23 +++++++++++++++++++++++
> >  include/linux/efi.h             |  3 +++
>
> The subject prefix says "efi: arm64: " but it looks like x86 diffstat
> here. You may want to get an ack from the x86 maintainers.
>

This just moves efi code around, and this is going through the EFI
tree, so that should be fine. I'll fix the subject in any case,
though.

Re: [PATCH v5 4/5] arm64: mm: Only remove nomap flag for initrd

[Catalin Marinas]

On Tue, Jun 14, 2022 at 05:21:55PM +0800, Wupeng Ma wrote:
> From: Ma Wupeng <mawupeng1@huawei.com>
> 
> Commit 177e15f0c144 ("arm64: add the initrd region to the linear mapping explicitly")
> remove all the flags of the memory used by initrd. This is fine since
> MEMBLOCK_MIRROR is not used in arm64.
> 
> However with mirrored feature introduced to arm64, this will clear the mirrored
> flag used by initrd, which will lead to error log printed by
> find_zone_movable_pfns_for_nodes() if the lower 4G range has some non-mirrored
> memory.
> 
> To solve this problem, only MEMBLOCK_NOMAP flag will be removed via
> memblock_clear_nomap().
> 
> Signed-off-by: Ma Wupeng <mawupeng1@huawei.com>
> Reviewed-by: Ard Biesheuvel <ardb@kernel.org>

Acked-by: Catalin Marinas <catalin.marinas@arm.com>

Re: [PATCH v5 0/5] introduce mirrored memory support for arm64

[Ard Biesheuvel]

On Wed, 15 Jun 2022 at 09:54, Mike Rapoport <rppt@kernel.org> wrote:
>
> On Tue, Jun 14, 2022 at 05:21:51PM +0800, Wupeng Ma wrote:
> > From: Ma Wupeng <mawupeng1@huawei.com>
> >
> > Patch #1 introduce mirrored memory support form arm64.
> > Patch #2-#4 fix some bugs for arm64 if memory reliable is enabled.
> > Patch #5 disable mirror feature if kernelcore is not specified.
> >
> > Thanks to Ard Biesheuvel's hard work [1], now kernel will perfer mirrored
> > memory if kaslr is enabled.
> >
> > [1] https://lore.kernel.org/linux-arm-kernel/CAMj1kXEPVEzMgOM4+Yj6PxHA-jFuDOAUdDJSiSxy_XaP4P7LSw@mail.gmail.com/T/
> >
> >
> > Ma Wupeng (5):
> >   efi: arm64: Introduce ability to find mirrored memory ranges
> >   mm: Ratelimited mirrored memory related warning messages
> >   mm: Limit warning message in vmemmap_verify() to once
> >   arm64: mm: Only remove nomap flag for initrd
> >   memblock: Disable mirror feature if kernelcore is not specified
> >
> >  arch/arm64/mm/init.c            |  2 +-
> >  arch/x86/include/asm/efi.h      |  4 ----
> >  arch/x86/platform/efi/efi.c     | 23 -----------------------
> >  drivers/firmware/efi/efi-init.c |  1 +
> >  drivers/firmware/efi/efi.c      | 23 +++++++++++++++++++++++
> >  include/linux/efi.h             |  3 +++
> >  mm/internal.h                   |  2 ++
> >  mm/memblock.c                   |  7 +++++--
> >  mm/page_alloc.c                 |  2 +-
> >  mm/sparse-vmemmap.c             |  2 +-
> >  10 files changed, 37 insertions(+), 32 deletions(-)
>
> For the series: Acked-by: Mike Rapoport <rppt@linux.ibm.com>
>


Thanks all, I've queued these up now.