[PATCH 1/2] arm64: mm: apply __ro_after_init to memory_limit

[PATCH 1/2] arm64: mm: apply __ro_after_init to memory_limit

[Peng Fan (OSS)]

From: Peng Fan <peng.fan@nxp.com>

This variable is only set during initialization, so mark with
__ro_after_init.

Signed-off-by: Peng Fan <peng.fan@nxp.com>
---
 arch/arm64/mm/init.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index a8834434af99..db63cc885771 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -172,7 +172,7 @@ int pfn_is_map_memory(unsigned long pfn)
 }
 EXPORT_SYMBOL(pfn_is_map_memory);
 
-static phys_addr_t memory_limit = PHYS_ADDR_MAX;
+static phys_addr_t memory_limit __ro_after_init = PHYS_ADDR_MAX;
 
 /*
  * Limit the memory size that was specified via FDT.
-- 
2.25.1

[PATCH 2/2] arm64: mm: support bootparam max_addr

[Peng Fan (OSS)]

From: Peng Fan <peng.fan@nxp.com>

There is a "mem=[x]" boot parameter, but when there is a whole reserved
by secure TEE, the continuous DRAM area is split with two memblocks.

For example, DRAM area [0x40000000, 0xffffffff], when TEE uses
[0x50000000, 0x51000000), the memblock will be split into
[0x40000000, 0x50000000) and [0x51000000, 0xffffffff].

If pass "mem=1024MB", the actually max addr will be 0x81000000.
However if need the max addr be 0x80000000, mem=1008MB should be used.

There also might be multiple other holes that no visible to Linux, when
we wanna to limit the max addr usable by Linux, using "max_addr=[X]" is
much easier than "mem=[X]"

Signed-off-by: Peng Fan <peng.fan@nxp.com>
---
 arch/arm64/mm/init.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index db63cc885771..3364b5e7a7fe 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -173,6 +173,7 @@ int pfn_is_map_memory(unsigned long pfn)
 EXPORT_SYMBOL(pfn_is_map_memory);
 
 static phys_addr_t memory_limit __ro_after_init = PHYS_ADDR_MAX;
+static phys_addr_t max_addr __ro_after_init = PHYS_ADDR_MAX;
 
 /*
  * Limit the memory size that was specified via FDT.
@@ -189,6 +190,18 @@ static int __init early_mem(char *p)
 }
 early_param("mem", early_mem);
 
+static int __init set_max_addr(char *p)
+{
+	if (!p)
+		return 1;
+
+	max_addr = memparse(p, &p) & PAGE_MASK;
+	pr_notice("Memory max addr set to 0x%llx\n", max_addr);
+
+	return 0;
+}
+early_param("max_addr", set_max_addr);
+
 void __init arm64_memblock_init(void)
 {
 	s64 linear_region_size = PAGE_END - _PAGE_OFFSET(vabits_actual);
@@ -253,6 +266,9 @@ void __init arm64_memblock_init(void)
 		memblock_add(__pa_symbol(_text), (u64)(_end - _text));
 	}
 
+	if (max_addr != PHYS_ADDR_MAX)
+		memblock_cap_memory_range(0, max_addr);
+
 	if (IS_ENABLED(CONFIG_BLK_DEV_INITRD) && phys_initrd_size) {
 		/*
 		 * Add back the memory we just removed if it results in the
@@ -427,4 +443,9 @@ void dump_mem_limit(void)
 	} else {
 		pr_emerg("Memory Limit: none\n");
 	}
+
+	if (max_addr != PHYS_ADDR_MAX)
+		pr_emerg("Max addr: 0x%llx\n", max_addr);
+	else
+		pr_emerg("Max addr: none\n");
 }
-- 
2.25.1

Re: [PATCH 1/2] arm64: mm: apply __ro_after_init to memory_limit

[Ard Biesheuvel]

On Wed, 15 Dec 2021 at 07:55, Peng Fan (OSS) <peng.fan@oss.nxp.com> wrote:
>
> From: Peng Fan <peng.fan@nxp.com>
>
> This variable is only set during initialization, so mark with
> __ro_after_init.
>
> Signed-off-by: Peng Fan <peng.fan@nxp.com>

Acked-by: Ard Biesheuvel <ardb@kernel.org>

> ---
>  arch/arm64/mm/init.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index a8834434af99..db63cc885771 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -172,7 +172,7 @@ int pfn_is_map_memory(unsigned long pfn)
>  }
>  EXPORT_SYMBOL(pfn_is_map_memory);
>
> -static phys_addr_t memory_limit = PHYS_ADDR_MAX;
> +static phys_addr_t memory_limit __ro_after_init = PHYS_ADDR_MAX;
>
>  /*
>   * Limit the memory size that was specified via FDT.
> --
> 2.25.1
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 2/2] arm64: mm: support bootparam max_addr

[Ard Biesheuvel]

On Wed, 15 Dec 2021 at 07:56, Peng Fan (OSS) <peng.fan@oss.nxp.com> wrote:
>
> From: Peng Fan <peng.fan@nxp.com>
>
> There is a "mem=[x]" boot parameter, but when there is a whole reserved
> by secure TEE, the continuous DRAM area is split with two memblocks.
>
> For example, DRAM area [0x40000000, 0xffffffff], when TEE uses
> [0x50000000, 0x51000000), the memblock will be split into
> [0x40000000, 0x50000000) and [0x51000000, 0xffffffff].
>
> If pass "mem=1024MB", the actually max addr will be 0x81000000.
> However if need the max addr be 0x80000000, mem=1008MB should be used.
>
> There also might be multiple other holes that no visible to Linux, when
> we wanna to limit the max addr usable by Linux, using "max_addr=[X]" is
> much easier than "mem=[X]"
>
> Signed-off-by: Peng Fan <peng.fan@nxp.com>

mem= is a hack already, please don't add another one. Limiting the
memory like this is far too tricky, given that the kernel itself and
the initrd could end up in memory that is excluded, and we have to go
and fix things up if that happens.


> ---
>  arch/arm64/mm/init.c | 21 +++++++++++++++++++++
>  1 file changed, 21 insertions(+)
>
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index db63cc885771..3364b5e7a7fe 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -173,6 +173,7 @@ int pfn_is_map_memory(unsigned long pfn)
>  EXPORT_SYMBOL(pfn_is_map_memory);
>
>  static phys_addr_t memory_limit __ro_after_init = PHYS_ADDR_MAX;
> +static phys_addr_t max_addr __ro_after_init = PHYS_ADDR_MAX;
>
>  /*
>   * Limit the memory size that was specified via FDT.
> @@ -189,6 +190,18 @@ static int __init early_mem(char *p)
>  }
>  early_param("mem", early_mem);
>
> +static int __init set_max_addr(char *p)
> +{
> +       if (!p)
> +               return 1;
> +
> +       max_addr = memparse(p, &p) & PAGE_MASK;
> +       pr_notice("Memory max addr set to 0x%llx\n", max_addr);
> +
> +       return 0;
> +}
> +early_param("max_addr", set_max_addr);
> +
>  void __init arm64_memblock_init(void)
>  {
>         s64 linear_region_size = PAGE_END - _PAGE_OFFSET(vabits_actual);
> @@ -253,6 +266,9 @@ void __init arm64_memblock_init(void)
>                 memblock_add(__pa_symbol(_text), (u64)(_end - _text));
>         }
>
> +       if (max_addr != PHYS_ADDR_MAX)
> +               memblock_cap_memory_range(0, max_addr);
> +
>         if (IS_ENABLED(CONFIG_BLK_DEV_INITRD) && phys_initrd_size) {
>                 /*
>                  * Add back the memory we just removed if it results in the
> @@ -427,4 +443,9 @@ void dump_mem_limit(void)
>         } else {
>                 pr_emerg("Memory Limit: none\n");
>         }
> +
> +       if (max_addr != PHYS_ADDR_MAX)
> +               pr_emerg("Max addr: 0x%llx\n", max_addr);
> +       else
> +               pr_emerg("Max addr: none\n");
>  }
> --
> 2.25.1
>
>
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

RE: [PATCH 2/2] arm64: mm: support bootparam max_addr

[Peng Fan]

> Subject: Re: [PATCH 2/2] arm64: mm: support bootparam max_addr
> 
> On Wed, 15 Dec 2021 at 07:56, Peng Fan (OSS) <peng.fan@oss.nxp.com>
> wrote:
> >
> > From: Peng Fan <peng.fan@nxp.com>
> >
> > There is a "mem=[x]" boot parameter, but when there is a whole
> > reserved by secure TEE, the continuous DRAM area is split with two
> memblocks.
> >
> > For example, DRAM area [0x40000000, 0xffffffff], when TEE uses
> > [0x50000000, 0x51000000), the memblock will be split into [0x40000000,
> > 0x50000000) and [0x51000000, 0xffffffff].
> >
> > If pass "mem=1024MB", the actually max addr will be 0x81000000.
> > However if need the max addr be 0x80000000, mem=1008MB should be
> used.
> >
> > There also might be multiple other holes that no visible to Linux,
> > when we wanna to limit the max addr usable by Linux, using
> > "max_addr=[X]" is much easier than "mem=[X]"
> >
> > Signed-off-by: Peng Fan <peng.fan@nxp.com>
> 
> mem= is a hack already, please don't add another one. Limiting the memory
> like this is far too tricky, given that the kernel itself and the initrd could end up
> in memory that is excluded, and we have to go and fix things up if that
> happens.

We wanna to use the reserved memory with request_mem_region, but with
commit 86588296acbfb1 ("fdt: Properly handle "no-map" field in the memory region ")

request_mem_region will fail, because the reserved memory are now as
kernel memory.

So we use "mem=X" to work around the issue, but "mem=X" is not user friendly
compared with "max_addr=" when there are multiple holes used by others.

Thanks,
Peng.

> 
> 
> > ---
> >  arch/arm64/mm/init.c | 21 +++++++++++++++++++++
> >  1 file changed, 21 insertions(+)
> >
> > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index
> > db63cc885771..3364b5e7a7fe 100644
> > --- a/arch/arm64/mm/init.c
> > +++ b/arch/arm64/mm/init.c
> > @@ -173,6 +173,7 @@ int pfn_is_map_memory(unsigned long pfn)
> > EXPORT_SYMBOL(pfn_is_map_memory);
> >
> >  static phys_addr_t memory_limit __ro_after_init = PHYS_ADDR_MAX;
> > +static phys_addr_t max_addr __ro_after_init = PHYS_ADDR_MAX;
> >
> >  /*
> >   * Limit the memory size that was specified via FDT.
> > @@ -189,6 +190,18 @@ static int __init early_mem(char *p)  }
> > early_param("mem", early_mem);
> >
> > +static int __init set_max_addr(char *p) {
> > +       if (!p)
> > +               return 1;
> > +
> > +       max_addr = memparse(p, &p) & PAGE_MASK;
> > +       pr_notice("Memory max addr set to 0x%llx\n", max_addr);
> > +
> > +       return 0;
> > +}
> > +early_param("max_addr", set_max_addr);
> > +
> >  void __init arm64_memblock_init(void)  {
> >         s64 linear_region_size = PAGE_END -
> > _PAGE_OFFSET(vabits_actual); @@ -253,6 +266,9 @@ void __init
> arm64_memblock_init(void)
> >                 memblock_add(__pa_symbol(_text), (u64)(_end -
> _text));
> >         }
> >
> > +       if (max_addr != PHYS_ADDR_MAX)
> > +               memblock_cap_memory_range(0, max_addr);
> > +
> >         if (IS_ENABLED(CONFIG_BLK_DEV_INITRD) && phys_initrd_size)
> {
> >                 /*
> >                  * Add back the memory we just removed if it results
> > in the @@ -427,4 +443,9 @@ void dump_mem_limit(void)
> >         } else {
> >                 pr_emerg("Memory Limit: none\n");
> >         }
> > +
> > +       if (max_addr != PHYS_ADDR_MAX)
> > +               pr_emerg("Max addr: 0x%llx\n", max_addr);
> > +       else
> > +               pr_emerg("Max addr: none\n");
> >  }
> > --
> > 2.25.1
> >
> >
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists
> > .infradead.org%2Fmailman%2Flistinfo%2Flinux-arm-kernel&amp;data=04%
> 7C0
> >
> 1%7Cpeng.fan%40nxp.com%7C3ad0ef697ad64542556208d9bf9d1e1f%7C68
> 6ea1d3bc
> >
> 2b4c6fa92cd99c5c301635%7C0%7C0%7C637751503805222488%7CUnknow
> n%7CTWFpbG
> >
> Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> Mn0%
> >
> 3D%7C3000&amp;sdata=iKVO4PUPnaRr%2B5gHcXxaaRxBt%2BK%2Fjytg8eQ
> dCqgqh5o%
> > 3D&amp;reserved=0

Re: [PATCH 2/2] arm64: mm: support bootparam max_addr

[Mike Rapoport]

On Wed, Dec 15, 2021 at 07:59:45AM +0000, Peng Fan wrote:
> > Subject: Re: [PATCH 2/2] arm64: mm: support bootparam max_addr
> > 
> > On Wed, 15 Dec 2021 at 07:56, Peng Fan (OSS) <peng.fan@oss.nxp.com>
> > wrote:
> > >
> > > From: Peng Fan <peng.fan@nxp.com>
> > >
> > > There is a "mem=[x]" boot parameter, but when there is a whole
> > > reserved by secure TEE, the continuous DRAM area is split with two
> > memblocks.
> > >
> > > For example, DRAM area [0x40000000, 0xffffffff], when TEE uses
> > > [0x50000000, 0x51000000), the memblock will be split into [0x40000000,
> > > 0x50000000) and [0x51000000, 0xffffffff].
> > >
> > > If pass "mem=1024MB", the actually max addr will be 0x81000000.
> > > However if need the max addr be 0x80000000, mem=1008MB should be
> > used.
> > >
> > > There also might be multiple other holes that no visible to Linux,
> > > when we wanna to limit the max addr usable by Linux, using
> > > "max_addr=[X]" is much easier than "mem=[X]"
> > >
> > > Signed-off-by: Peng Fan <peng.fan@nxp.com>
> > 
> > mem= is a hack already, please don't add another one. Limiting the memory
> > like this is far too tricky, given that the kernel itself and the initrd could end up
> > in memory that is excluded, and we have to go and fix things up if that
> > happens.
> 
> We wanna to use the reserved memory with request_mem_region, but with
> commit 86588296acbfb1 ("fdt: Properly handle "no-map" field in the memory region ")
> 
> request_mem_region will fail, because the reserved memory are now as
> kernel memory.
 
request_mem_region() is for MMIO.  Why do you want to use it for RAM?

> So we use "mem=X" to work around the issue, but "mem=X" is not user friendly
> compared with "max_addr=" when there are multiple holes used by others.
> 
> Thanks,
> Peng.
> 
> > 
> > 
> > > ---
> > >  arch/arm64/mm/init.c | 21 +++++++++++++++++++++
> > >  1 file changed, 21 insertions(+)
> > >
> > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index
> > > db63cc885771..3364b5e7a7fe 100644
> > > --- a/arch/arm64/mm/init.c
> > > +++ b/arch/arm64/mm/init.c
> > > @@ -173,6 +173,7 @@ int pfn_is_map_memory(unsigned long pfn)
> > > EXPORT_SYMBOL(pfn_is_map_memory);
> > >
> > >  static phys_addr_t memory_limit __ro_after_init = PHYS_ADDR_MAX;
> > > +static phys_addr_t max_addr __ro_after_init = PHYS_ADDR_MAX;
> > >
> > >  /*
> > >   * Limit the memory size that was specified via FDT.
> > > @@ -189,6 +190,18 @@ static int __init early_mem(char *p)  }
> > > early_param("mem", early_mem);
> > >
> > > +static int __init set_max_addr(char *p) {
> > > +       if (!p)
> > > +               return 1;
> > > +
> > > +       max_addr = memparse(p, &p) & PAGE_MASK;
> > > +       pr_notice("Memory max addr set to 0x%llx\n", max_addr);
> > > +
> > > +       return 0;
> > > +}
> > > +early_param("max_addr", set_max_addr);
> > > +
> > >  void __init arm64_memblock_init(void)  {
> > >         s64 linear_region_size = PAGE_END -
> > > _PAGE_OFFSET(vabits_actual); @@ -253,6 +266,9 @@ void __init
> > arm64_memblock_init(void)
> > >                 memblock_add(__pa_symbol(_text), (u64)(_end -
> > _text));
> > >         }
> > >
> > > +       if (max_addr != PHYS_ADDR_MAX)
> > > +               memblock_cap_memory_range(0, max_addr);
> > > +
> > >         if (IS_ENABLED(CONFIG_BLK_DEV_INITRD) && phys_initrd_size)
> > {
> > >                 /*
> > >                  * Add back the memory we just removed if it results
> > > in the @@ -427,4 +443,9 @@ void dump_mem_limit(void)
> > >         } else {
> > >                 pr_emerg("Memory Limit: none\n");
> > >         }
> > > +
> > > +       if (max_addr != PHYS_ADDR_MAX)
> > > +               pr_emerg("Max addr: 0x%llx\n", max_addr);
> > > +       else
> > > +               pr_emerg("Max addr: none\n");
> > >  }
> > > --
> > > 2.25.1
> > >
> > >
> > > _______________________________________________
> > > linux-arm-kernel mailing list
> > > linux-arm-kernel@lists.infradead.org
> > > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists
> > > .infradead.org%2Fmailman%2Flistinfo%2Flinux-arm-kernel&amp;data=04%
> > 7C0
> > >
> > 1%7Cpeng.fan%40nxp.com%7C3ad0ef697ad64542556208d9bf9d1e1f%7C68
> > 6ea1d3bc
> > >
> > 2b4c6fa92cd99c5c301635%7C0%7C0%7C637751503805222488%7CUnknow
> > n%7CTWFpbG
> > >
> > Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> > Mn0%
> > >
> > 3D%7C3000&amp;sdata=iKVO4PUPnaRr%2B5gHcXxaaRxBt%2BK%2Fjytg8eQ
> > dCqgqh5o%
> > > 3D&amp;reserved=0

-- 
Sincerely yours,
Mike.

RE: [PATCH 2/2] arm64: mm: support bootparam max_addr

[Peng Fan]

> Subject: Re: [PATCH 2/2] arm64: mm: support bootparam max_addr
> 
> On Wed, Dec 15, 2021 at 07:59:45AM +0000, Peng Fan wrote:
> > > Subject: Re: [PATCH 2/2] arm64: mm: support bootparam max_addr
> > >
> > > On Wed, 15 Dec 2021 at 07:56, Peng Fan (OSS) <peng.fan@oss.nxp.com>
> > > wrote:
> > > >
> > > > From: Peng Fan <peng.fan@nxp.com>
> > > >
> > > > There is a "mem=[x]" boot parameter, but when there is a whole
> > > > reserved by secure TEE, the continuous DRAM area is split with two
> > > memblocks.
> > > >
> > > > For example, DRAM area [0x40000000, 0xffffffff], when TEE uses
> > > > [0x50000000, 0x51000000), the memblock will be split into
> > > > [0x40000000,
> > > > 0x50000000) and [0x51000000, 0xffffffff].
> > > >
> > > > If pass "mem=1024MB", the actually max addr will be 0x81000000.
> > > > However if need the max addr be 0x80000000, mem=1008MB should be
> > > used.
> > > >
> > > > There also might be multiple other holes that no visible to Linux,
> > > > when we wanna to limit the max addr usable by Linux, using
> > > > "max_addr=[X]" is much easier than "mem=[X]"
> > > >
> > > > Signed-off-by: Peng Fan <peng.fan@nxp.com>
> > >
> > > mem= is a hack already, please don't add another one. Limiting the
> > > memory like this is far too tricky, given that the kernel itself and
> > > the initrd could end up in memory that is excluded, and we have to
> > > go and fix things up if that happens.
> >
> > We wanna to use the reserved memory with request_mem_region, but with
> > commit 86588296acbfb1 ("fdt: Properly handle "no-map" field in the
> > memory region ")
> >
> > request_mem_region will fail, because the reserved memory are now as
> > kernel memory.
> 
> request_mem_region() is for MMIO.  Why do you want to use it for RAM?

+ Jan, the jailhouse hypervisor owner.

There is an out of tree driver
https://github.com/siemens/jailhouse/blob/master/driver/main.c#L466

The hypervisor jailhouse is loaded after linux boot up, and the hypervisor
bin file needs to be loaded into DRAM that reserved in our device
tree with node with no map property.

And the hypervisor use virtual pci for communication between VMs,
The virtual pci use part of the reserved DRAM area as PCI MMIO space.

Maybe I should use /memreserve, but not node with no-map property.

Thanks,
Peng.

> 
> > So we use "mem=X" to work around the issue, but "mem=X" is not user
> > friendly compared with "max_addr=" when there are multiple holes used by
> others.
> >
> > Thanks,
> > Peng.
> >
> > >
> > >
> > > > ---
> > > >  arch/arm64/mm/init.c | 21 +++++++++++++++++++++
> > > >  1 file changed, 21 insertions(+)
> > > >
> > > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index
> > > > db63cc885771..3364b5e7a7fe 100644
> > > > --- a/arch/arm64/mm/init.c
> > > > +++ b/arch/arm64/mm/init.c
> > > > @@ -173,6 +173,7 @@ int pfn_is_map_memory(unsigned long pfn)
> > > > EXPORT_SYMBOL(pfn_is_map_memory);
> > > >
> > > >  static phys_addr_t memory_limit __ro_after_init = PHYS_ADDR_MAX;
> > > > +static phys_addr_t max_addr __ro_after_init = PHYS_ADDR_MAX;
> > > >
> > > >  /*
> > > >   * Limit the memory size that was specified via FDT.
> > > > @@ -189,6 +190,18 @@ static int __init early_mem(char *p)  }
> > > > early_param("mem", early_mem);
> > > >
> > > > +static int __init set_max_addr(char *p) {
> > > > +       if (!p)
> > > > +               return 1;
> > > > +
> > > > +       max_addr = memparse(p, &p) & PAGE_MASK;
> > > > +       pr_notice("Memory max addr set to 0x%llx\n", max_addr);
> > > > +
> > > > +       return 0;
> > > > +}
> > > > +early_param("max_addr", set_max_addr);
> > > > +
> > > >  void __init arm64_memblock_init(void)  {
> > > >         s64 linear_region_size = PAGE_END -
> > > > _PAGE_OFFSET(vabits_actual); @@ -253,6 +266,9 @@ void __init
> > > arm64_memblock_init(void)
> > > >                 memblock_add(__pa_symbol(_text), (u64)(_end -
> > > _text));
> > > >         }
> > > >
> > > > +       if (max_addr != PHYS_ADDR_MAX)
> > > > +               memblock_cap_memory_range(0, max_addr);
> > > > +
> > > >         if (IS_ENABLED(CONFIG_BLK_DEV_INITRD) &&
> phys_initrd_size)
> > > {
> > > >                 /*
> > > >                  * Add back the memory we just removed if it
> > > > results in the @@ -427,4 +443,9 @@ void dump_mem_limit(void)
> > > >         } else {
> > > >                 pr_emerg("Memory Limit: none\n");
> > > >         }
> > > > +
> > > > +       if (max_addr != PHYS_ADDR_MAX)
> > > > +               pr_emerg("Max addr: 0x%llx\n", max_addr);
> > > > +       else
> > > > +               pr_emerg("Max addr: none\n");
> > > >  }
> > > > --
> > > > 2.25.1
> > > >
> > > >
> > > > _______________________________________________
> > > > linux-arm-kernel mailing list
> > > > linux-arm-kernel@lists.infradead.org
> > > > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fl
> > > > ists
> > > > .infradead.org%2Fmailman%2Flistinfo%2Flinux-arm-kernel&amp;data=0
> 4
> > > > %
> > > 7C0
> > > >
> > >
> 1%7Cpeng.fan%40nxp.com%7C3ad0ef697ad64542556208d9bf9d1e1f%7C68
> > > 6ea1d3bc
> > > >
> > >
> 2b4c6fa92cd99c5c301635%7C0%7C0%7C637751503805222488%7CUnknow
> > > n%7CTWFpbG
> > > >
> > >
> Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> > > Mn0%
> > > >
> > >
> 3D%7C3000&amp;sdata=iKVO4PUPnaRr%2B5gHcXxaaRxBt%2BK%2Fjytg8eQ
> > > dCqgqh5o%
> > > > 3D&amp;reserved=0
> 
> --
> Sincerely yours,
> Mike.

Re: [PATCH 2/2] arm64: mm: support bootparam max_addr

[Mike Rapoport]

On Wed, Dec 15, 2021 at 09:30:36AM +0000, Peng Fan wrote:
> > Subject: Re: [PATCH 2/2] arm64: mm: support bootparam max_addr
> > 
> > On Wed, Dec 15, 2021 at 07:59:45AM +0000, Peng Fan wrote:
> > > > Subject: Re: [PATCH 2/2] arm64: mm: support bootparam max_addr
> > > >
> > > > On Wed, 15 Dec 2021 at 07:56, Peng Fan (OSS) <peng.fan@oss.nxp.com>
> > > > wrote:
> > > > >
> > > > > From: Peng Fan <peng.fan@nxp.com>
> > > > >
> > > > > There is a "mem=[x]" boot parameter, but when there is a whole
> > > > > reserved by secure TEE, the continuous DRAM area is split with two
> > > > memblocks.
> > > > >
> > > > > For example, DRAM area [0x40000000, 0xffffffff], when TEE uses
> > > > > [0x50000000, 0x51000000), the memblock will be split into
> > > > > [0x40000000,
> > > > > 0x50000000) and [0x51000000, 0xffffffff].
> > > > >
> > > > > If pass "mem=1024MB", the actually max addr will be 0x81000000.
> > > > > However if need the max addr be 0x80000000, mem=1008MB should be
> > > > used.
> > > > >
> > > > > There also might be multiple other holes that no visible to Linux,
> > > > > when we wanna to limit the max addr usable by Linux, using
> > > > > "max_addr=[X]" is much easier than "mem=[X]"
> > > > >
> > > > > Signed-off-by: Peng Fan <peng.fan@nxp.com>
> > > >
> > > > mem= is a hack already, please don't add another one. Limiting the
> > > > memory like this is far too tricky, given that the kernel itself and
> > > > the initrd could end up in memory that is excluded, and we have to
> > > > go and fix things up if that happens.
> > >
> > > We wanna to use the reserved memory with request_mem_region, but with
> > > commit 86588296acbfb1 ("fdt: Properly handle "no-map" field in the
> > > memory region ")
> > >
> > > request_mem_region will fail, because the reserved memory are now as
> > > kernel memory.
> > 
> > request_mem_region() is for MMIO.  Why do you want to use it for RAM?
> 
> + Jan, the jailhouse hypervisor owner.
> 
> There is an out of tree driver
> https://github.com/siemens/jailhouse/blob/master/driver/main.c#L466
> 
> The hypervisor jailhouse is loaded after linux boot up, and the hypervisor
> bin file needs to be loaded into DRAM that reserved in our device
> tree with node with no map property.
> 
> And the hypervisor use virtual pci for communication between VMs,
> The virtual pci use part of the reserved DRAM area as PCI MMIO space.
> 
> Maybe I should use /memreserve, but not node with no-map property.

So, my understanding is that you need a chunk of memory that Linux does not
use and does not map into the kernel page tables.
In that case /memreserve + nomap in the device tree could be a better
solution than mem=X.
 
> Thanks,
> Peng.
> 
> > 
> > > So we use "mem=X" to work around the issue, but "mem=X" is not user
> > > friendly compared with "max_addr=" when there are multiple holes used by
> > others.
> > >
> > > Thanks,
> > > Peng.
> > >
> > > >
> > > >
> > > > > ---
> > > > >  arch/arm64/mm/init.c | 21 +++++++++++++++++++++
> > > > >  1 file changed, 21 insertions(+)
> > > > >
> > > > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index
> > > > > db63cc885771..3364b5e7a7fe 100644
> > > > > --- a/arch/arm64/mm/init.c
> > > > > +++ b/arch/arm64/mm/init.c
> > > > > @@ -173,6 +173,7 @@ int pfn_is_map_memory(unsigned long pfn)
> > > > > EXPORT_SYMBOL(pfn_is_map_memory);
> > > > >
> > > > >  static phys_addr_t memory_limit __ro_after_init = PHYS_ADDR_MAX;
> > > > > +static phys_addr_t max_addr __ro_after_init = PHYS_ADDR_MAX;
> > > > >
> > > > >  /*
> > > > >   * Limit the memory size that was specified via FDT.
> > > > > @@ -189,6 +190,18 @@ static int __init early_mem(char *p)  }
> > > > > early_param("mem", early_mem);
> > > > >
> > > > > +static int __init set_max_addr(char *p) {
> > > > > +       if (!p)
> > > > > +               return 1;
> > > > > +
> > > > > +       max_addr = memparse(p, &p) & PAGE_MASK;
> > > > > +       pr_notice("Memory max addr set to 0x%llx\n", max_addr);
> > > > > +
> > > > > +       return 0;
> > > > > +}
> > > > > +early_param("max_addr", set_max_addr);
> > > > > +
> > > > >  void __init arm64_memblock_init(void)  {
> > > > >         s64 linear_region_size = PAGE_END -
> > > > > _PAGE_OFFSET(vabits_actual); @@ -253,6 +266,9 @@ void __init
> > > > arm64_memblock_init(void)
> > > > >                 memblock_add(__pa_symbol(_text), (u64)(_end -
> > > > _text));
> > > > >         }
> > > > >
> > > > > +       if (max_addr != PHYS_ADDR_MAX)
> > > > > +               memblock_cap_memory_range(0, max_addr);
> > > > > +
> > > > >         if (IS_ENABLED(CONFIG_BLK_DEV_INITRD) &&
> > phys_initrd_size)
> > > > {
> > > > >                 /*
> > > > >                  * Add back the memory we just removed if it
> > > > > results in the @@ -427,4 +443,9 @@ void dump_mem_limit(void)
> > > > >         } else {
> > > > >                 pr_emerg("Memory Limit: none\n");
> > > > >         }
> > > > > +
> > > > > +       if (max_addr != PHYS_ADDR_MAX)
> > > > > +               pr_emerg("Max addr: 0x%llx\n", max_addr);
> > > > > +       else
> > > > > +               pr_emerg("Max addr: none\n");
> > > > >  }
> > > > > --
> > > > > 2.25.1
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > linux-arm-kernel mailing list
> > > > > linux-arm-kernel@lists.infradead.org
> > > > > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fl
> > > > > ists
> > > > > .infradead.org%2Fmailman%2Flistinfo%2Flinux-arm-kernel&amp;data=0
> > 4
> > > > > %
> > > > 7C0
> > > > >
> > > >
> > 1%7Cpeng.fan%40nxp.com%7C3ad0ef697ad64542556208d9bf9d1e1f%7C68
> > > > 6ea1d3bc
> > > > >
> > > >
> > 2b4c6fa92cd99c5c301635%7C0%7C0%7C637751503805222488%7CUnknow
> > > > n%7CTWFpbG
> > > > >
> > > >
> > Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> > > > Mn0%
> > > > >
> > > >
> > 3D%7C3000&amp;sdata=iKVO4PUPnaRr%2B5gHcXxaaRxBt%2BK%2Fjytg8eQ
> > > > dCqgqh5o%
> > > > > 3D&amp;reserved=0
> > 
> > --
> > Sincerely yours,
> > Mike.

-- 
Sincerely yours,
Mike.

Re: [PATCH 1/2] arm64: mm: apply __ro_after_init to memory_limit

[David Hildenbrand]

On 15.12.21 07:45, Peng Fan (OSS) wrote:
> From: Peng Fan <peng.fan@nxp.com>
> 
> This variable is only set during initialization, so mark with
> __ro_after_init.
> 
> Signed-off-by: Peng Fan <peng.fan@nxp.com>
> ---
>  arch/arm64/mm/init.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index a8834434af99..db63cc885771 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -172,7 +172,7 @@ int pfn_is_map_memory(unsigned long pfn)
>  }
>  EXPORT_SYMBOL(pfn_is_map_memory);
>  
> -static phys_addr_t memory_limit = PHYS_ADDR_MAX;
> +static phys_addr_t memory_limit __ro_after_init = PHYS_ADDR_MAX;
>  
>  /*
>   * Limit the memory size that was specified via FDT.
> 

Reviewed-by: David Hildenbrand <david@redhat.com>

-- 
Thanks,

David / dhildenb

RE: [PATCH 2/2] arm64: mm: support bootparam max_addr

[Peng Fan]

> Subject: Re: [PATCH 2/2] arm64: mm: support bootparam max_addr
> 
> On Wed, Dec 15, 2021 at 09:30:36AM +0000, Peng Fan wrote:
> > > Subject: Re: [PATCH 2/2] arm64: mm: support bootparam max_addr
> > >
> > > On Wed, Dec 15, 2021 at 07:59:45AM +0000, Peng Fan wrote:
> > > > > Subject: Re: [PATCH 2/2] arm64: mm: support bootparam max_addr
> > > > >
> > > > > On Wed, 15 Dec 2021 at 07:56, Peng Fan (OSS)
> > > > > <peng.fan@oss.nxp.com>
> > > > > wrote:
> > > > > >
> > > > > > From: Peng Fan <peng.fan@nxp.com>
> > > > > >
> > > > > > There is a "mem=[x]" boot parameter, but when there is a whole
> > > > > > reserved by secure TEE, the continuous DRAM area is split with
> > > > > > two
> > > > > memblocks.
> > > > > >
> > > > > > For example, DRAM area [0x40000000, 0xffffffff], when TEE uses
> > > > > > [0x50000000, 0x51000000), the memblock will be split into
> > > > > > [0x40000000,
> > > > > > 0x50000000) and [0x51000000, 0xffffffff].
> > > > > >
> > > > > > If pass "mem=1024MB", the actually max addr will be 0x81000000.
> > > > > > However if need the max addr be 0x80000000, mem=1008MB
> should
> > > > > > be
> > > > > used.
> > > > > >
> > > > > > There also might be multiple other holes that no visible to
> > > > > > Linux, when we wanna to limit the max addr usable by Linux,
> > > > > > using "max_addr=[X]" is much easier than "mem=[X]"
> > > > > >
> > > > > > Signed-off-by: Peng Fan <peng.fan@nxp.com>
> > > > >
> > > > > mem= is a hack already, please don't add another one. Limiting
> > > > > the memory like this is far too tricky, given that the kernel
> > > > > itself and the initrd could end up in memory that is excluded,
> > > > > and we have to go and fix things up if that happens.
> > > >
> > > > We wanna to use the reserved memory with request_mem_region, but
> > > > with commit 86588296acbfb1 ("fdt: Properly handle "no-map" field
> > > > in the memory region ")
> > > >
> > > > request_mem_region will fail, because the reserved memory are now
> > > > as kernel memory.
> > >
> > > request_mem_region() is for MMIO.  Why do you want to use it for
> RAM?
> >
> > + Jan, the jailhouse hypervisor owner.
> >
> > There is an out of tree driver
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
> >
> ub.com%2Fsiemens%2Fjailhouse%2Fblob%2Fmaster%2Fdriver%2Fmain.c%23
> L466&
> >
> amp;data=04%7C01%7Cpeng.fan%40nxp.com%7C0540c0c55ed24688f9f308d
> 9bfb0b7
> >
> 7d%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6377515879788
> 01324%7CU
> >
> nknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI
> 6Ik1ha
> >
> WwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=owLVa3T4jnTPvohHwdEnP2%2F
> uMiUJmYeTc
> > 1TwM0i9iCE%3D&amp;reserved=0
> >
> > The hypervisor jailhouse is loaded after linux boot up, and the
> > hypervisor bin file needs to be loaded into DRAM that reserved in our
> > device tree with node with no map property.
> >
> > And the hypervisor use virtual pci for communication between VMs, The
> > virtual pci use part of the reserved DRAM area as PCI MMIO space.
> >
> > Maybe I should use /memreserve, but not node with no-map property.
> 
> So, my understanding is that you need a chunk of memory that Linux does not
> use and does not map into the kernel page tables.
> In that case /memreserve + nomap in the device tree could be a better
> solution than mem=X.

nomap not work now since commit 
86588296acbfb1 ("fdt: Properly handle "no-map" field in the memory region ")

I need try /memreserve

BTW, do you think max_addr would be an option be added to memblock
common code mm/memblock.c?

Thanks,
Peng.

> 
> > Thanks,
> > Peng.
> >
> > >
> > > > So we use "mem=X" to work around the issue, but "mem=X" is not
> > > > user friendly compared with "max_addr=" when there are multiple
> > > > holes used by
> > > others.
> > > >
> > > > Thanks,
> > > > Peng.
> > > >
> > > > >
> > > > >
> > > > > > ---
> > > > > >  arch/arm64/mm/init.c | 21 +++++++++++++++++++++
> > > > > >  1 file changed, 21 insertions(+)
> > > > > >
> > > > > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c index
> > > > > > db63cc885771..3364b5e7a7fe 100644
> > > > > > --- a/arch/arm64/mm/init.c
> > > > > > +++ b/arch/arm64/mm/init.c
> > > > > > @@ -173,6 +173,7 @@ int pfn_is_map_memory(unsigned long pfn)
> > > > > > EXPORT_SYMBOL(pfn_is_map_memory);
> > > > > >
> > > > > >  static phys_addr_t memory_limit __ro_after_init =
> > > > > > PHYS_ADDR_MAX;
> > > > > > +static phys_addr_t max_addr __ro_after_init = PHYS_ADDR_MAX;
> > > > > >
> > > > > >  /*
> > > > > >   * Limit the memory size that was specified via FDT.
> > > > > > @@ -189,6 +190,18 @@ static int __init early_mem(char *p)  }
> > > > > > early_param("mem", early_mem);
> > > > > >
> > > > > > +static int __init set_max_addr(char *p) {
> > > > > > +       if (!p)
> > > > > > +               return 1;
> > > > > > +
> > > > > > +       max_addr = memparse(p, &p) & PAGE_MASK;
> > > > > > +       pr_notice("Memory max addr set to 0x%llx\n",
> > > > > > + max_addr);
> > > > > > +
> > > > > > +       return 0;
> > > > > > +}
> > > > > > +early_param("max_addr", set_max_addr);
> > > > > > +
> > > > > >  void __init arm64_memblock_init(void)  {
> > > > > >         s64 linear_region_size = PAGE_END -
> > > > > > _PAGE_OFFSET(vabits_actual); @@ -253,6 +266,9 @@ void __init
> > > > > arm64_memblock_init(void)
> > > > > >                 memblock_add(__pa_symbol(_text), (u64)(_end
> -
> > > > > _text));
> > > > > >         }
> > > > > >
> > > > > > +       if (max_addr != PHYS_ADDR_MAX)
> > > > > > +               memblock_cap_memory_range(0, max_addr);
> > > > > > +
> > > > > >         if (IS_ENABLED(CONFIG_BLK_DEV_INITRD) &&
> > > phys_initrd_size)
> > > > > {
> > > > > >                 /*
> > > > > >                  * Add back the memory we just removed if it
> > > > > > results in the @@ -427,4 +443,9 @@ void dump_mem_limit(void)
> > > > > >         } else {
> > > > > >                 pr_emerg("Memory Limit: none\n");
> > > > > >         }
> > > > > > +
> > > > > > +       if (max_addr != PHYS_ADDR_MAX)
> > > > > > +               pr_emerg("Max addr: 0x%llx\n", max_addr);
> > > > > > +       else
> > > > > > +               pr_emerg("Max addr: none\n");
> > > > > >  }
> > > > > > --
> > > > > > 2.25.1
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > linux-arm-kernel mailing list
> > > > > > linux-arm-kernel@lists.infradead.org
> > > > > > https://eur01.safelinks.protection.outlook.com/?url=http%3A%2F
> > > > > > %2Fl
> > > > > > ists
> > > > > > .infradead.org%2Fmailman%2Flistinfo%2Flinux-arm-kernel&amp;dat
> > > > > > a=0
> > > 4
> > > > > > %
> > > > > 7C0
> > > > > >
> > > > >
> > >
> 1%7Cpeng.fan%40nxp.com%7C3ad0ef697ad64542556208d9bf9d1e1f%7C68
> > > > > 6ea1d3bc
> > > > > >
> > > > >
> > >
> 2b4c6fa92cd99c5c301635%7C0%7C0%7C637751503805222488%7CUnknow
> > > > > n%7CTWFpbG
> > > > > >
> > > > >
> > >
> Zsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6
> > > > > Mn0%
> > > > > >
> > > > >
> > >
> 3D%7C3000&amp;sdata=iKVO4PUPnaRr%2B5gHcXxaaRxBt%2BK%2Fjytg8eQ
> > > > > dCqgqh5o%
> > > > > > 3D&amp;reserved=0
> > >
> > > --
> > > Sincerely yours,
> > > Mike.
> 
> --
> Sincerely yours,
> Mike.

Re: [PATCH 2/2] arm64: mm: support bootparam max_addr

[Mike Rapoport]

On Wed, Dec 15, 2021 at 12:05:36PM +0000, Peng Fan wrote:
> > Subject: Re: [PATCH 2/2] arm64: mm: support bootparam max_addr
> > 
> > > > > > > If pass "mem=1024MB", the actually max addr will be
> > > > > > > 0x81000000.  However if need the max addr be 0x80000000,
> > > > > > > mem=1008MB should be used.
> > > > > > >
> 
> BTW, do you think max_addr would be an option be added to memblock
> common code mm/memblock.c?

You have a working solution with mem=1008MB, I don't see a need for
additional kernel parameter.

-- 
Sincerely yours,
Mike.

Re: (subset) [PATCH 1/2] arm64: mm: apply __ro_after_init to memory_limit

[Catalin Marinas]

On Wed, 15 Dec 2021 14:45:58 +0800, Peng Fan (OSS) wrote:
> From: Peng Fan <peng.fan@nxp.com>
> 
> This variable is only set during initialization, so mark with
> __ro_after_init.
> 
> 

Applied to arm64 (for-next/core), thanks!

[1/2] arm64: mm: apply __ro_after_init to memory_limit
      https://git.kernel.org/arm64/c/bb425a759847

-- 
Catalin

Re: [PATCH 1/2] arm64: mm: apply __ro_after_init to memory_limit

[Anshuman Khandual]

On 12/15/21 12:15 PM, Peng Fan (OSS) wrote:
> From: Peng Fan <peng.fan@nxp.com>
> 
> This variable is only set during initialization, so mark with
> __ro_after_init.
> 
> Signed-off-by: Peng Fan <peng.fan@nxp.com>
> ---
>  arch/arm64/mm/init.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index a8834434af99..db63cc885771 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -172,7 +172,7 @@ int pfn_is_map_memory(unsigned long pfn)
>  }
>  EXPORT_SYMBOL(pfn_is_map_memory);
>  
> -static phys_addr_t memory_limit = PHYS_ADDR_MAX;
> +static phys_addr_t memory_limit __ro_after_init = PHYS_ADDR_MAX;
>  
>  /*
>   * Limit the memory size that was specified via FDT.
> 

Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>