bug in identity map for 4KB pages?

bug in identity map for 4KB pages?

[Stuart Yoder]

Our system has RAM at a high address, and previously required using 48-bit VA
in order to have an idmap that covered all of RAM.

In testing on 4.2-rc4, which now contains support for the increased VA range
of the idmap without using 48-bit VA, I'm finding that things work for
64KB pages, but do not for 4KB pages.

Is there any known limitation here with 4KB pages?  Any ideas?

Kernel crash dump I'm getting is below.

Thanks,
Stuart

------------------------------------------------------------------------------

Booting Linux on physical CPU 0x0
Initializing cgroup subsys cpu
Linux version 4.2.0-rc4+ (b08248 at right) (gcc version 4.8.3 20131202 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2013.12 - Linaro GCC 2013.11) ) #180 SMP PREEMPT Tue Jul 28 20:51:47 CDT 2015
CPU: AArch64 Processor [411fd071] revision 1
Detected PIPT I-cache on CPU0
alternatives: enabling workaround for ARM erratum 832075
alternatives: detected feature GIC system register CPU interface
earlycon: Early serial console at MMIO 0x21c0600 (options '115200')
bootconsole [uart0] enabled
efi: Getting EFI parameters from FDT:
efi: UEFI not found.
cma: Reserved 16 MiB at 0x00000000fec00000
BUG: not creating mapping for 0x0000008080000000 at 0x0000004000000000 - outside kernel range
Unable to handle kernel paging request at virtual address 435ffff000
pgd = ffffffc000924000
[435ffff000] *pgd=0000000000000000, *pud=0000000000000000
Internal error: Oops: 96000045 [#1] PREEMPT SMP
Modules linked in:
CPU: 0 PID: 0 Comm: swapper Not tainted 4.2.0-rc4+ #180
Hardware name: Freescale Layerscape 2085a software Simulator model (DT)
task: ffffffc00088cc70 ti: ffffffc000880000 task.ti: ffffffc000880000
PC is at memset+0x16c/0x1c0
LR is at early_alloc+0x70/0x80
pc : [<ffffffc0003250ac>] lr : [<ffffffc00080de50>] pstate: 400002c5
sp : ffffffc000883ea0
x29: ffffffc000883ea0 x28: 00000000a00000dc
x27: ffffffc000081198 x26: 00000083e0000000
x25: ffffffc000000000 x24: ffffffc00089a000
x23: 0000000000000018 x22: 00000000c0000000
x21: 0000008080000000 x20: 000000435ffff000
x19: 0000000000001000 x18: 0000000000000000
x17: 0000000000000006 x16: 00000083e0000000
x15: 0000000000000018 x14: 0000008080000000
x13: ffffffc000900dd0 x12: 0000000000000578
x11: 0000000600000000 x10: 00000083dffff000
x9 : 0000000000000000 x8 : 000000435ffff000
x7 : 0000000000000000 x6 : 000000000000003f
x5 : 0000000000000040 x4 : 0000000000000000
x3 : 0000000000000004 x2 : 0000000000000fc0
x1 : 0000000000000000 x0 : 000000435ffff000

Process swapper (pid: 0, stack limit = 0xffffffc000880020)
Stack: (0xffffffc000883ea0 to 0xffffffc000884000)
3ea0: 00883ec0 ffffffc0 0080df60 ffffffc0 00900200 ffffffc0 00886000 ffffffc0
3ec0: 00883f20 ffffffc0 0080b7ac ffffffc0 00080000 ffffffc0 fa9fa000 ffffffbf
3ee0: 00000040 00000000 00000000 00000000 00886000 ffffffc0 80000000 00000000
3f00: 80921000 00000000 80924000 00000000 00000040 00000000 fa9fa000 ffffffbf
3f20: 00883fa0 ffffffc0 00809690 ffffffc0 00000001 00000000 0083c120 ffffffc0
3f40: 9fffa000 00000000 00000000 00000000 00886000 ffffffc0 80000000 00000000
3f60: 80921000 00000000 80924000 00000000 00000001 00000000 9fffa000 00000000
3f80: 00000000 00000000 00000000 80000000 00000080 00000000 64667467 ff616b73
3fa0: 00000000 00000000 805d3000 00000000 fff9fa40 00000000 00000e12 00000000
3fc0: 9fffa000 00000000 00000000 00000000 00000000 00000000 80000000 00000000
3fe0: 00000000 00000000 0083c948 ffffffc0 00000000 00000000 00000000 00000000
Call trace:
[<ffffffc0003250ac>] memset+0x16c/0x1c0
[<ffffffc00080df5c>] paging_init+0xd0/0x174
[<ffffffc00080b7a8>] setup_arch+0x28c/0x5c4
[<ffffffc00080968c>] start_kernel+0x94/0x3d0
Code: 91010108 54ffff4a 8b040108 cb050042 (d50b7428)
---[ end trace cb88537fdc8fa200 ]---
Kernel panic - not syncing: Attempted to kill the idle task!
---[ end Kernel panic - not syncing: Attempted to kill the idle task!

bug in identity map for 4KB pages?

[Ard Biesheuvel]

On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> Our system has RAM at a high address, and previously required using 48-bit VA
> in order to have an idmap that covered all of RAM.
>
> In testing on 4.2-rc4, which now contains support for the increased VA range
> of the idmap without using 48-bit VA, I'm finding that things work for
> 64KB pages, but do not for 4KB pages.
>
> Is there any known limitation here with 4KB pages?  Any ideas?
>

You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
So the physical arrangement still requires more than the 39 bits of
virtual address space you get with 3 levels, even if the ID map can
cope now. That is why you get the 0x40_0000_0000 virtual address:
__phys_to_virt() just wraps to a positive number.

-- 
Ard.


> ------------------------------------------------------------------------------
>
> Booting Linux on physical CPU 0x0
> Initializing cgroup subsys cpu
> Linux version 4.2.0-rc4+ (b08248 at right) (gcc version 4.8.3 20131202 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2013.12 - Linaro GCC 2013.11) ) #180 SMP PREEMPT Tue Jul 28 20:51:47 CDT 2015
> CPU: AArch64 Processor [411fd071] revision 1
> Detected PIPT I-cache on CPU0
> alternatives: enabling workaround for ARM erratum 832075
> alternatives: detected feature GIC system register CPU interface
> earlycon: Early serial console at MMIO 0x21c0600 (options '115200')
> bootconsole [uart0] enabled
> efi: Getting EFI parameters from FDT:
> efi: UEFI not found.
> cma: Reserved 16 MiB at 0x00000000fec00000
> BUG: not creating mapping for 0x0000008080000000 at 0x0000004000000000 - outside kernel range
> Unable to handle kernel paging request at virtual address 435ffff000
> pgd = ffffffc000924000
> [435ffff000] *pgd=0000000000000000, *pud=0000000000000000
> Internal error: Oops: 96000045 [#1] PREEMPT SMP
> Modules linked in:
> CPU: 0 PID: 0 Comm: swapper Not tainted 4.2.0-rc4+ #180
> Hardware name: Freescale Layerscape 2085a software Simulator model (DT)
> task: ffffffc00088cc70 ti: ffffffc000880000 task.ti: ffffffc000880000
> PC is at memset+0x16c/0x1c0
> LR is at early_alloc+0x70/0x80
> pc : [<ffffffc0003250ac>] lr : [<ffffffc00080de50>] pstate: 400002c5
> sp : ffffffc000883ea0
> x29: ffffffc000883ea0 x28: 00000000a00000dc
> x27: ffffffc000081198 x26: 00000083e0000000
> x25: ffffffc000000000 x24: ffffffc00089a000
> x23: 0000000000000018 x22: 00000000c0000000
> x21: 0000008080000000 x20: 000000435ffff000
> x19: 0000000000001000 x18: 0000000000000000
> x17: 0000000000000006 x16: 00000083e0000000
> x15: 0000000000000018 x14: 0000008080000000
> x13: ffffffc000900dd0 x12: 0000000000000578
> x11: 0000000600000000 x10: 00000083dffff000
> x9 : 0000000000000000 x8 : 000000435ffff000
> x7 : 0000000000000000 x6 : 000000000000003f
> x5 : 0000000000000040 x4 : 0000000000000000
> x3 : 0000000000000004 x2 : 0000000000000fc0
> x1 : 0000000000000000 x0 : 000000435ffff000
>
> Process swapper (pid: 0, stack limit = 0xffffffc000880020)
> Stack: (0xffffffc000883ea0 to 0xffffffc000884000)
> 3ea0: 00883ec0 ffffffc0 0080df60 ffffffc0 00900200 ffffffc0 00886000 ffffffc0
> 3ec0: 00883f20 ffffffc0 0080b7ac ffffffc0 00080000 ffffffc0 fa9fa000 ffffffbf
> 3ee0: 00000040 00000000 00000000 00000000 00886000 ffffffc0 80000000 00000000
> 3f00: 80921000 00000000 80924000 00000000 00000040 00000000 fa9fa000 ffffffbf
> 3f20: 00883fa0 ffffffc0 00809690 ffffffc0 00000001 00000000 0083c120 ffffffc0
> 3f40: 9fffa000 00000000 00000000 00000000 00886000 ffffffc0 80000000 00000000
> 3f60: 80921000 00000000 80924000 00000000 00000001 00000000 9fffa000 00000000
> 3f80: 00000000 00000000 00000000 80000000 00000080 00000000 64667467 ff616b73
> 3fa0: 00000000 00000000 805d3000 00000000 fff9fa40 00000000 00000e12 00000000
> 3fc0: 9fffa000 00000000 00000000 00000000 00000000 00000000 80000000 00000000
> 3fe0: 00000000 00000000 0083c948 ffffffc0 00000000 00000000 00000000 00000000
> Call trace:
> [<ffffffc0003250ac>] memset+0x16c/0x1c0
> [<ffffffc00080df5c>] paging_init+0xd0/0x174
> [<ffffffc00080b7a8>] setup_arch+0x28c/0x5c4
> [<ffffffc00080968c>] start_kernel+0x94/0x3d0
> Code: 91010108 54ffff4a 8b040108 cb050042 (d50b7428)
> ---[ end trace cb88537fdc8fa200 ]---
> Kernel panic - not syncing: Attempted to kill the idle task!
> ---[ end Kernel panic - not syncing: Attempted to kill the idle task!

bug in identity map for 4KB pages?

[Mark Rutland]

On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
> On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> > Our system has RAM at a high address, and previously required using 48-bit VA
> > in order to have an idmap that covered all of RAM.
> >
> > In testing on 4.2-rc4, which now contains support for the increased VA range
> > of the idmap without using 48-bit VA, I'm finding that things work for
> > 64KB pages, but do not for 4KB pages.
> >
> > Is there any known limitation here with 4KB pages?  Any ideas?
> >
> 
> You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
> So the physical arrangement still requires more than the 39 bits of
> virtual address space you get with 3 levels, even if the ID map can
> cope now. That is why you get the 0x40_0000_0000 virtual address:
> __phys_to_virt() just wraps to a positive number.

Ah, I see. So it's the linear mapping rather than the idmap which is the
problem.

We cut memory which would fall below the start of the linear map in
early_init_dt_add_memory_arch, by cutting memory below phys_offset.

It looks like we already have the logic for cutting memory beyond the
end of the linear map, so we should just need to override the limit.

Stuart, does the below patch prevent the panic you see?

Thanks,
Mark.

---->8----
>From 1bdea78e4427ae6af20050967f03ca7efb1d58fa Mon Sep 17 00:00:00 2001
From: Mark Rutland <mark.rutland@arm.com>
Date: Wed, 29 Jul 2015 12:35:35 +0100
Subject: [PATCH] arm64/fdt: cut out memory which cannot be mapped

Currently early_init_dt_add_memory_arch only cuts out memory above
((phys_addr_t)~0), while the end of memory an architecture can map may
be lower than this, and will likely be some fixed offset from
phys_offset.

This patch adds a new (optional) ARCH_VA_RANGE that architectures can
override to define such a limit, and plumbs it in for arch/arm64.

Signed-off-by: Mark Rutland <mark.rutland@arm.com>
---
 arch/arm64/include/asm/page.h |  2 ++
 drivers/of/fdt.c              | 14 +++++++++-----
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/include/asm/page.h b/arch/arm64/include/asm/page.h
index 7d9c7e4..91eb057 100644
--- a/arch/arm64/include/asm/page.h
+++ b/arch/arm64/include/asm/page.h
@@ -67,6 +67,8 @@ extern int pfn_valid(unsigned long);
 
 #include <asm/memory.h>
 
+#define ARCH_VA_RANGE (1UL << VA_BITS)
+
 #endif /* !__ASSEMBLY__ */
 
 #define VM_DATA_DEFAULT_FLAGS \
diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
index 07496560..643e37c 100644
--- a/drivers/of/fdt.c
+++ b/drivers/of/fdt.c
@@ -967,11 +967,15 @@ int __init early_init_dt_scan_chosen(unsigned long node, const char *uname,
 }
 
 #ifdef CONFIG_HAVE_MEMBLOCK
-#define MAX_PHYS_ADDR	((phys_addr_t)~0)
 
 void __init __weak early_init_dt_add_memory_arch(u64 base, u64 size)
 {
 	const u64 phys_offset = __pa(PAGE_OFFSET);
+#ifdef ARCH_VA_RANGE
+	const u64 phys_end = phys_offset + ARCH_VA_RANGE - 1;
+#else
+	const u64 phys_end = ((phys_addr_t)~0);
+#endif
 
 	if (!PAGE_ALIGNED(base)) {
 		if (size < PAGE_SIZE - (base & ~PAGE_MASK)) {
@@ -984,16 +988,16 @@ void __init __weak early_init_dt_add_memory_arch(u64 base, u64 size)
 	}
 	size &= PAGE_MASK;
 
-	if (base > MAX_PHYS_ADDR) {
+	if (base > phys_end) {
 		pr_warning("Ignoring memory block 0x%llx - 0x%llx\n",
 				base, base + size);
 		return;
 	}
 
-	if (base + size - 1 > MAX_PHYS_ADDR) {
+	if (base + size - 1 > phys_end) {
 		pr_warning("Ignoring memory range 0x%llx - 0x%llx\n",
-				((u64)MAX_PHYS_ADDR) + 1, base + size);
-		size = MAX_PHYS_ADDR - base + 1;
+				((u64)phys_end) + 1, base + size);
+		size = phys_end - base + 1;
 	}
 
 	if (base + size < phys_offset) {
-- 
1.9.1

bug in identity map for 4KB pages?

[Ard Biesheuvel]

On 29 July 2015 at 13:42, Mark Rutland <mark.rutland@arm.com> wrote:
> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
>> On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>> > Our system has RAM at a high address, and previously required using 48-bit VA
>> > in order to have an idmap that covered all of RAM.
>> >
>> > In testing on 4.2-rc4, which now contains support for the increased VA range
>> > of the idmap without using 48-bit VA, I'm finding that things work for
>> > 64KB pages, but do not for 4KB pages.
>> >
>> > Is there any known limitation here with 4KB pages?  Any ideas?
>> >
>>
>> You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
>> So the physical arrangement still requires more than the 39 bits of
>> virtual address space you get with 3 levels, even if the ID map can
>> cope now. That is why you get the 0x40_0000_0000 virtual address:
>> __phys_to_virt() just wraps to a positive number.
>
> Ah, I see. So it's the linear mapping rather than the idmap which is the
> problem.
>
> We cut memory which would fall below the start of the linear map in
> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
>
> It looks like we already have the logic for cutting memory beyond the
> end of the linear map, so we should just need to override the limit.
>
> Stuart, does the below patch prevent the panic you see?
>

Wouldn't something like this make more sense?

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index 597831bdddf3..64480b65ef17 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -158,6 +158,15 @@ early_param("mem", early_mem);

 void __init arm64_memblock_init(void)
 {
+       /*
+        * Remove the memory that we will not be able to cover
+        * with the linear mapping.
+        */
+       const s64 linear_region_size = -(s64)PAGE_OFFSET;
+
+       memblock_remove(0, memstart_addr);
+       memblock_remove(memstart_addr + linear_region_size, ULLONG_MAX);
+
        memblock_enforce_memory_limit(memory_limit);

        /*
"""

(taken from my patch 'arm64: override early_init_dt_add_memory_arch()'
sent to the list a while ago as part of the linear mapping decoupling
series)

-- 
Ard.

bug in identity map for 4KB pages?

[Ard Biesheuvel]

On 29 July 2015 at 13:49, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> On 29 July 2015 at 13:42, Mark Rutland <mark.rutland@arm.com> wrote:
>> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
>>> On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>>> > Our system has RAM at a high address, and previously required using 48-bit VA
>>> > in order to have an idmap that covered all of RAM.
>>> >
>>> > In testing on 4.2-rc4, which now contains support for the increased VA range
>>> > of the idmap without using 48-bit VA, I'm finding that things work for
>>> > 64KB pages, but do not for 4KB pages.
>>> >
>>> > Is there any known limitation here with 4KB pages?  Any ideas?
>>> >
>>>
>>> You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
>>> So the physical arrangement still requires more than the 39 bits of
>>> virtual address space you get with 3 levels, even if the ID map can
>>> cope now. That is why you get the 0x40_0000_0000 virtual address:
>>> __phys_to_virt() just wraps to a positive number.
>>
>> Ah, I see. So it's the linear mapping rather than the idmap which is the
>> problem.
>>
>> We cut memory which would fall below the start of the linear map in
>> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
>>
>> It looks like we already have the logic for cutting memory beyond the
>> end of the linear map, so we should just need to override the limit.
>>
>> Stuart, does the below patch prevent the panic you see?
>>
>
> Wouldn't something like this make more sense?
>
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index 597831bdddf3..64480b65ef17 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -158,6 +158,15 @@ early_param("mem", early_mem);
>
>  void __init arm64_memblock_init(void)
>  {
> +       /*
> +        * Remove the memory that we will not be able to cover
> +        * with the linear mapping.
> +        */
> +       const s64 linear_region_size = -(s64)PAGE_OFFSET;
> +
> +       memblock_remove(0, memstart_addr);
> +       memblock_remove(memstart_addr + linear_region_size, ULLONG_MAX);
> +
>         memblock_enforce_memory_limit(memory_limit);
>
>         /*
> """
>
> (taken from my patch 'arm64: override early_init_dt_add_memory_arch()'
> sent to the list a while ago as part of the linear mapping decoupling
> series)
>

(replying to self) but actually, either solution still means that, of
the ~16 GB this platform seems to have installed, only 2 GB is made
available.

I suppose there is little we can do about it, since we cannot ignore
the lower 2GB if the kernel resides there ...

-- 
Ard.

bug in identity map for 4KB pages?

[Will Deacon]

On Wed, Jul 29, 2015 at 12:58:07PM +0100, Ard Biesheuvel wrote:
> On 29 July 2015 at 13:49, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> > Wouldn't something like this make more sense?
> >
> > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > index 597831bdddf3..64480b65ef17 100644
> > --- a/arch/arm64/mm/init.c
> > +++ b/arch/arm64/mm/init.c
> > @@ -158,6 +158,15 @@ early_param("mem", early_mem);
> >
> >  void __init arm64_memblock_init(void)
> >  {
> > +       /*
> > +        * Remove the memory that we will not be able to cover
> > +        * with the linear mapping.
> > +        */
> > +       const s64 linear_region_size = -(s64)PAGE_OFFSET;
> > +
> > +       memblock_remove(0, memstart_addr);
> > +       memblock_remove(memstart_addr + linear_region_size, ULLONG_MAX);
> > +
> >         memblock_enforce_memory_limit(memory_limit);
> >
> >         /*
> > """
> >
> > (taken from my patch 'arm64: override early_init_dt_add_memory_arch()'
> > sent to the list a while ago as part of the linear mapping decoupling
> > series)
> >
> 
> (replying to self) but actually, either solution still means that, of
> the ~16 GB this platform seems to have installed, only 2 GB is made
> available.
> 
> I suppose there is little we can do about it, since we cannot ignore
> the lower 2GB if the kernel resides there ...

Printing a message to the console isn't a bad idea, though.

Will

bug in identity map for 4KB pages?

[Mark Rutland]

> (replying to self) but actually, either solution still means that, of
> the ~16 GB this platform seems to have installed, only 2 GB is made
> available.
> 
> I suppose there is little we can do about it, since we cannot ignore
> the lower 2GB if the kernel resides there ...

Yup.

We need to cap memory for the general case anyway (e.g. 48 bits worth of
contiguous memory when the kernel only supports 39), and changes to
spasemem behaviour are orthogonal to that.

Mark.

bug in identity map for 4KB pages?

[Mark Rutland]

On Wed, Jul 29, 2015 at 01:30:02PM +0100, Will Deacon wrote:
> On Wed, Jul 29, 2015 at 12:58:07PM +0100, Ard Biesheuvel wrote:
> > On 29 July 2015 at 13:49, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> > > Wouldn't something like this make more sense?
> > >
> > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > > index 597831bdddf3..64480b65ef17 100644
> > > --- a/arch/arm64/mm/init.c
> > > +++ b/arch/arm64/mm/init.c
> > > @@ -158,6 +158,15 @@ early_param("mem", early_mem);
> > >
> > >  void __init arm64_memblock_init(void)
> > >  {
> > > +       /*
> > > +        * Remove the memory that we will not be able to cover
> > > +        * with the linear mapping.
> > > +        */
> > > +       const s64 linear_region_size = -(s64)PAGE_OFFSET;
> > > +
> > > +       memblock_remove(0, memstart_addr);
> > > +       memblock_remove(memstart_addr + linear_region_size, ULLONG_MAX);
> > > +
> > >         memblock_enforce_memory_limit(memory_limit);
> > >
> > >         /*
> > > """
> > >
> > > (taken from my patch 'arm64: override early_init_dt_add_memory_arch()'
> > > sent to the list a while ago as part of the linear mapping decoupling
> > > series)
> > >
> > 
> > (replying to self) but actually, either solution still means that, of
> > the ~16 GB this platform seems to have installed, only 2 GB is made
> > available.
> > 
> > I suppose there is little we can do about it, since we cannot ignore
> > the lower 2GB if the kernel resides there ...
> 
> Printing a message to the console isn't a bad idea, though.

The logic in early_init_dt_add_memory_arch does that, provided the end
is set up correctly.

We already rely on that for warnings when memory below phys_offset is
cut out, which is why I wanted the logic there.

That said, my patch got the limit wrong, so I'll respin that.

Mark.

bug in identity map for 4KB pages?

[Mark Rutland]

On Wed, Jul 29, 2015 at 12:49:53PM +0100, Ard Biesheuvel wrote:
> On 29 July 2015 at 13:42, Mark Rutland <mark.rutland@arm.com> wrote:
> > On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
> >> On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >> > Our system has RAM at a high address, and previously required using 48-bit VA
> >> > in order to have an idmap that covered all of RAM.
> >> >
> >> > In testing on 4.2-rc4, which now contains support for the increased VA range
> >> > of the idmap without using 48-bit VA, I'm finding that things work for
> >> > 64KB pages, but do not for 4KB pages.
> >> >
> >> > Is there any known limitation here with 4KB pages?  Any ideas?
> >> >
> >>
> >> You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
> >> So the physical arrangement still requires more than the 39 bits of
> >> virtual address space you get with 3 levels, even if the ID map can
> >> cope now. That is why you get the 0x40_0000_0000 virtual address:
> >> __phys_to_virt() just wraps to a positive number.
> >
> > Ah, I see. So it's the linear mapping rather than the idmap which is the
> > problem.
> >
> > We cut memory which would fall below the start of the linear map in
> > early_init_dt_add_memory_arch, by cutting memory below phys_offset.
> >
> > It looks like we already have the logic for cutting memory beyond the
> > end of the linear map, so we should just need to override the limit.
> >
> > Stuart, does the below patch prevent the panic you see?
> >
> 
> Wouldn't something like this make more sense?
> 
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index 597831bdddf3..64480b65ef17 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -158,6 +158,15 @@ early_param("mem", early_mem);
> 
>  void __init arm64_memblock_init(void)
>  {
> +       /*
> +        * Remove the memory that we will not be able to cover
> +        * with the linear mapping.
> +        */
> +       const s64 linear_region_size = -(s64)PAGE_OFFSET;
> +
> +       memblock_remove(0, memstart_addr);
> +       memblock_remove(memstart_addr + linear_region_size, ULLONG_MAX);

The maths is more correct, certainly.

I'd prefer to handle the truncation in early_init_dt_add_memory_arch
because it already has the requisite logic, and keeping the truncation
in one place should keep things less confusing.

To that end I'll respin my patch using PAGE_OFFSET to determine the
physical memory limit.

Unless there's something I've missed?

Thanks,
Mark.

bug in identity map for 4KB pages?

[Ard Biesheuvel]

On 29 July 2015 at 14:53, Mark Rutland <mark.rutland@arm.com> wrote:
> On Wed, Jul 29, 2015 at 12:49:53PM +0100, Ard Biesheuvel wrote:
>> On 29 July 2015 at 13:42, Mark Rutland <mark.rutland@arm.com> wrote:
>> > On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
>> >> On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>> >> > Our system has RAM at a high address, and previously required using 48-bit VA
>> >> > in order to have an idmap that covered all of RAM.
>> >> >
>> >> > In testing on 4.2-rc4, which now contains support for the increased VA range
>> >> > of the idmap without using 48-bit VA, I'm finding that things work for
>> >> > 64KB pages, but do not for 4KB pages.
>> >> >
>> >> > Is there any known limitation here with 4KB pages?  Any ideas?
>> >> >
>> >>
>> >> You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
>> >> So the physical arrangement still requires more than the 39 bits of
>> >> virtual address space you get with 3 levels, even if the ID map can
>> >> cope now. That is why you get the 0x40_0000_0000 virtual address:
>> >> __phys_to_virt() just wraps to a positive number.
>> >
>> > Ah, I see. So it's the linear mapping rather than the idmap which is the
>> > problem.
>> >
>> > We cut memory which would fall below the start of the linear map in
>> > early_init_dt_add_memory_arch, by cutting memory below phys_offset.
>> >
>> > It looks like we already have the logic for cutting memory beyond the
>> > end of the linear map, so we should just need to override the limit.
>> >
>> > Stuart, does the below patch prevent the panic you see?
>> >
>>
>> Wouldn't something like this make more sense?
>>
>> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
>> index 597831bdddf3..64480b65ef17 100644
>> --- a/arch/arm64/mm/init.c
>> +++ b/arch/arm64/mm/init.c
>> @@ -158,6 +158,15 @@ early_param("mem", early_mem);
>>
>>  void __init arm64_memblock_init(void)
>>  {
>> +       /*
>> +        * Remove the memory that we will not be able to cover
>> +        * with the linear mapping.
>> +        */
>> +       const s64 linear_region_size = -(s64)PAGE_OFFSET;
>> +
>> +       memblock_remove(0, memstart_addr);
>> +       memblock_remove(memstart_addr + linear_region_size, ULLONG_MAX);
>
> The maths is more correct, certainly.
>
> I'd prefer to handle the truncation in early_init_dt_add_memory_arch
> because it already has the requisite logic, and keeping the truncation
> in one place should keep things less confusing.
>
> To that end I'll respin my patch using PAGE_OFFSET to determine the
> physical memory limit.
>
> Unless there's something I've missed?
>

I don't think so. But perhaps it is just as easy just to override
early_init_dt_add_memory_arch() now since I will need to do that
anyway if the linear mapping changes ever land.

http://article.gmane.org/gmane.linux.kernel.efi/5744

bug in identity map for 4KB pages?

[Mark Rutland]

On Wed, Jul 29, 2015 at 01:57:01PM +0100, Ard Biesheuvel wrote:
> On 29 July 2015 at 14:53, Mark Rutland <mark.rutland@arm.com> wrote:
> > On Wed, Jul 29, 2015 at 12:49:53PM +0100, Ard Biesheuvel wrote:
> >> On 29 July 2015 at 13:42, Mark Rutland <mark.rutland@arm.com> wrote:
> >> > On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
> >> >> On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >> >> > Our system has RAM at a high address, and previously required using 48-bit VA
> >> >> > in order to have an idmap that covered all of RAM.
> >> >> >
> >> >> > In testing on 4.2-rc4, which now contains support for the increased VA range
> >> >> > of the idmap without using 48-bit VA, I'm finding that things work for
> >> >> > 64KB pages, but do not for 4KB pages.
> >> >> >
> >> >> > Is there any known limitation here with 4KB pages?  Any ideas?
> >> >> >
> >> >>
> >> >> You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
> >> >> So the physical arrangement still requires more than the 39 bits of
> >> >> virtual address space you get with 3 levels, even if the ID map can
> >> >> cope now. That is why you get the 0x40_0000_0000 virtual address:
> >> >> __phys_to_virt() just wraps to a positive number.
> >> >
> >> > Ah, I see. So it's the linear mapping rather than the idmap which is the
> >> > problem.
> >> >
> >> > We cut memory which would fall below the start of the linear map in
> >> > early_init_dt_add_memory_arch, by cutting memory below phys_offset.
> >> >
> >> > It looks like we already have the logic for cutting memory beyond the
> >> > end of the linear map, so we should just need to override the limit.
> >> >
> >> > Stuart, does the below patch prevent the panic you see?
> >> >
> >>
> >> Wouldn't something like this make more sense?
> >>
> >> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> >> index 597831bdddf3..64480b65ef17 100644
> >> --- a/arch/arm64/mm/init.c
> >> +++ b/arch/arm64/mm/init.c
> >> @@ -158,6 +158,15 @@ early_param("mem", early_mem);
> >>
> >>  void __init arm64_memblock_init(void)
> >>  {
> >> +       /*
> >> +        * Remove the memory that we will not be able to cover
> >> +        * with the linear mapping.
> >> +        */
> >> +       const s64 linear_region_size = -(s64)PAGE_OFFSET;
> >> +
> >> +       memblock_remove(0, memstart_addr);
> >> +       memblock_remove(memstart_addr + linear_region_size, ULLONG_MAX);
> >
> > The maths is more correct, certainly.
> >
> > I'd prefer to handle the truncation in early_init_dt_add_memory_arch
> > because it already has the requisite logic, and keeping the truncation
> > in one place should keep things less confusing.
> >
> > To that end I'll respin my patch using PAGE_OFFSET to determine the
> > physical memory limit.
> >
> > Unless there's something I've missed?
> >
> 
> I don't think so. But perhaps it is just as easy just to override
> early_init_dt_add_memory_arch() now since I will need to do that
> anyway if the linear mapping changes ever land.
> 
> http://article.gmane.org/gmane.linux.kernel.efi/5744

That probably makes sense, yes.

We'll need to log when we actually truncate things, though. If we can do
that, then overriding early_init_dt_add_memory_arch (and having
arm64_memblock_init do the truncation) makes sense to me.

Mark.

bug in identity map for 4KB pages?

[Stuart Yoder]

> -----Original Message-----
> From: Mark Rutland [mailto:mark.rutland at arm.com]
> Sent: Wednesday, July 29, 2015 6:43 AM
> To: Ard Biesheuvel; Yoder Stuart-B08248
> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-kernel at lists.infradead.org
> Subject: Re: bug in identity map for 4KB pages?
> 
> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> > > Our system has RAM at a high address, and previously required using 48-bit VA
> > > in order to have an idmap that covered all of RAM.
> > >
> > > In testing on 4.2-rc4, which now contains support for the increased VA range
> > > of the idmap without using 48-bit VA, I'm finding that things work for
> > > 64KB pages, but do not for 4KB pages.
> > >
> > > Is there any known limitation here with 4KB pages?  Any ideas?
> > >
> >
> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?

Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at 
0x80_8000_0000.

> > So the physical arrangement still requires more than the 39 bits of
> > virtual address space you get with 3 levels, even if the ID map can
> > cope now. That is why you get the 0x40_0000_0000 virtual address:
> > __phys_to_virt() just wraps to a positive number.
> 
> Ah, I see. So it's the linear mapping rather than the idmap which is the
> problem.
> 
> We cut memory which would fall below the start of the linear map in
> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
> 
> It looks like we already have the logic for cutting memory beyond the
> end of the linear map, so we should just need to override the limit.

What is the effect of cutting that memory?  ...are we losing something?

> Stuart, does the below patch prevent the panic you see?

I will try it and let you know...can try the suggested patch by
Ard as well.

Thanks,
Stuart

bug in identity map for 4KB pages?

[Ard Biesheuvel]

On 29 July 2015 at 17:45, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>
>
>> -----Original Message-----
>> From: Mark Rutland [mailto:mark.rutland at arm.com]
>> Sent: Wednesday, July 29, 2015 6:43 AM
>> To: Ard Biesheuvel; Yoder Stuart-B08248
>> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-kernel at lists.infradead.org
>> Subject: Re: bug in identity map for 4KB pages?
>>
>> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
>> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>> > > Our system has RAM at a high address, and previously required using 48-bit VA
>> > > in order to have an idmap that covered all of RAM.
>> > >
>> > > In testing on 4.2-rc4, which now contains support for the increased VA range
>> > > of the idmap without using 48-bit VA, I'm finding that things work for
>> > > 64KB pages, but do not for 4KB pages.
>> > >
>> > > Is there any known limitation here with 4KB pages?  Any ideas?
>> > >
>> >
>> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
>
> Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at
> 0x80_8000_0000.
>
>> > So the physical arrangement still requires more than the 39 bits of
>> > virtual address space you get with 3 levels, even if the ID map can
>> > cope now. That is why you get the 0x40_0000_0000 virtual address:
>> > __phys_to_virt() just wraps to a positive number.
>>
>> Ah, I see. So it's the linear mapping rather than the idmap which is the
>> problem.
>>
>> We cut memory which would fall below the start of the linear map in
>> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
>>
>> It looks like we already have the logic for cutting memory beyond the
>> end of the linear map, so we should just need to override the limit.
>
> What is the effect of cutting that memory?  ...are we losing something?
>

Yes, you will lose everything above 2 GB in the current configuration.

>> Stuart, does the below patch prevent the panic you see?
>
> I will try it and let you know...can try the suggested patch by
> Ard as well.
>

Is it possible to load the kernel at 0x80_0000_0000? I think your
system will just boot fine in that case, and you will only lose the
lower 2 GB in that case.

Re patch: I am working on a more comprehensive patch atm. Could you
perhaps try that instead (~15 min)?

Thanks,
Ard.

bug in identity map for 4KB pages?

[Stuart Yoder]

> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> Sent: Wednesday, July 29, 2015 10:48 AM
> To: Yoder Stuart-B08248
> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
> kernel at lists.infradead.org
> Subject: Re: bug in identity map for 4KB pages?
> 
> On 29 July 2015 at 17:45, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: Mark Rutland [mailto:mark.rutland at arm.com]
> >> Sent: Wednesday, July 29, 2015 6:43 AM
> >> To: Ard Biesheuvel; Yoder Stuart-B08248
> >> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-kernel at lists.infradead.org
> >> Subject: Re: bug in identity map for 4KB pages?
> >>
> >> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
> >> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >> > > Our system has RAM at a high address, and previously required using 48-bit VA
> >> > > in order to have an idmap that covered all of RAM.
> >> > >
> >> > > In testing on 4.2-rc4, which now contains support for the increased VA range
> >> > > of the idmap without using 48-bit VA, I'm finding that things work for
> >> > > 64KB pages, but do not for 4KB pages.
> >> > >
> >> > > Is there any known limitation here with 4KB pages?  Any ideas?
> >> > >
> >> >
> >> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
> >
> > Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at
> > 0x80_8000_0000.
> >
> >> > So the physical arrangement still requires more than the 39 bits of
> >> > virtual address space you get with 3 levels, even if the ID map can
> >> > cope now. That is why you get the 0x40_0000_0000 virtual address:
> >> > __phys_to_virt() just wraps to a positive number.
> >>
> >> Ah, I see. So it's the linear mapping rather than the idmap which is the
> >> problem.
> >>
> >> We cut memory which would fall below the start of the linear map in
> >> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
> >>
> >> It looks like we already have the logic for cutting memory beyond the
> >> end of the linear map, so we should just need to override the limit.
> >
> > What is the effect of cutting that memory?  ...are we losing something?
> >
> 
> Yes, you will lose everything above 2 GB in the current configuration.

Hmmm...so you really mean that a 16 GB system will have only 2GB
total available for both kernel/user space?   That won't be workable
as a long term solution.
 
> >> Stuart, does the below patch prevent the panic you see?
> >
> > I will try it and let you know...can try the suggested patch by
> > Ard as well.
> >
> 
> Is it possible to load the kernel at 0x80_0000_0000? I think your
> system will just boot fine in that case, and you will only lose the
> lower 2 GB in that case.

So, it's like that lower 2 GB does not exist at all? ...that is going to
be hard for our customers to swallow I think.

With 48-bit VA enabled, we get all 16GB (and beyond) of memory.

If it's true that due to our physical memory map that we will not
be able to access all RAM with a standard defconfig, then makes
me wonder whether we need to request our hardware designers to change
that physical map.  (but, IIRC this physical map is that recommended
by ARM)

> Re patch: I am working on a more comprehensive patch atm. Could you
> perhaps try that instead (~15 min)?

Sure, will wait for your patch and will try it.

Thanks,
Stuart

bug in identity map for 4KB pages?

[Ard Biesheuvel]

On 29 July 2015 at 17:58, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>
>> -----Original Message-----
>> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> Sent: Wednesday, July 29, 2015 10:48 AM
>> To: Yoder Stuart-B08248
>> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
>> kernel at lists.infradead.org
>> Subject: Re: bug in identity map for 4KB pages?
>>
>> On 29 July 2015 at 17:45, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>> >
>> >
>> >> -----Original Message-----
>> >> From: Mark Rutland [mailto:mark.rutland at arm.com]
>> >> Sent: Wednesday, July 29, 2015 6:43 AM
>> >> To: Ard Biesheuvel; Yoder Stuart-B08248
>> >> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-kernel at lists.infradead.org
>> >> Subject: Re: bug in identity map for 4KB pages?
>> >>
>> >> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
>> >> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>> >> > > Our system has RAM at a high address, and previously required using 48-bit VA
>> >> > > in order to have an idmap that covered all of RAM.
>> >> > >
>> >> > > In testing on 4.2-rc4, which now contains support for the increased VA range
>> >> > > of the idmap without using 48-bit VA, I'm finding that things work for
>> >> > > 64KB pages, but do not for 4KB pages.
>> >> > >
>> >> > > Is there any known limitation here with 4KB pages?  Any ideas?
>> >> > >
>> >> >
>> >> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
>> >
>> > Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at
>> > 0x80_8000_0000.
>> >
>> >> > So the physical arrangement still requires more than the 39 bits of
>> >> > virtual address space you get with 3 levels, even if the ID map can
>> >> > cope now. That is why you get the 0x40_0000_0000 virtual address:
>> >> > __phys_to_virt() just wraps to a positive number.
>> >>
>> >> Ah, I see. So it's the linear mapping rather than the idmap which is the
>> >> problem.
>> >>
>> >> We cut memory which would fall below the start of the linear map in
>> >> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
>> >>
>> >> It looks like we already have the logic for cutting memory beyond the
>> >> end of the linear map, so we should just need to override the limit.
>> >
>> > What is the effect of cutting that memory?  ...are we losing something?
>> >
>>
>> Yes, you will lose everything above 2 GB in the current configuration.
>
> Hmmm...so you really mean that a 16 GB system will have only 2GB
> total available for both kernel/user space?   That won't be workable
> as a long term solution.
>

I agree. So that essentially means that 3 level page table kernels are
unsuitable for such systems.

>> >> Stuart, does the below patch prevent the panic you see?
>> >
>> > I will try it and let you know...can try the suggested patch by
>> > Ard as well.
>> >
>>
>> Is it possible to load the kernel at 0x80_0000_0000? I think your
>> system will just boot fine in that case, and you will only lose the
>> lower 2 GB in that case.
>
> So, it's like that lower 2 GB does not exist at all? ...that is going to
> be hard for our customers to swallow I think.
>
> With 48-bit VA enabled, we get all 16GB (and beyond) of memory.
>

Indeed.

> If it's true that due to our physical memory map that we will not
> be able to access all RAM with a standard defconfig, then makes
> me wonder whether we need to request our hardware designers to change
> that physical map.  (but, IIRC this physical map is that recommended
> by ARM)
>

Well, I am aware of an ARM document that explains the placement at
0x8000_0000, 0x80_8000_0000 etc.
But as far as I know, this is not a general recommendation for
hardware, but an internal guideline for the reference platforms. Of
course, you need some memory below 4 GB if you want to boot 32-bit
kernels without jumping through a lot of hoops, i.e., you won't be
able to boot a 32-bit ARM kernel on AMD Seattle which has all of its
memory at 0x80_0000_0000

>> Re patch: I am working on a more comprehensive patch atm. Could you
>> perhaps try that instead (~15 min)?
>
> Sure, will wait for your patch and will try it.
>

Cheers,
Ard.

bug in identity map for 4KB pages?

[Ard Biesheuvel]

On 29 July 2015 at 18:12, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> On 29 July 2015 at 17:58, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>>
>>> -----Original Message-----
>>> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>>> Sent: Wednesday, July 29, 2015 10:48 AM
>>> To: Yoder Stuart-B08248
>>> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
>>> kernel at lists.infradead.org
>>> Subject: Re: bug in identity map for 4KB pages?
>>>
>>> On 29 July 2015 at 17:45, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>>> >
>>> >
>>> >> -----Original Message-----
>>> >> From: Mark Rutland [mailto:mark.rutland at arm.com]
>>> >> Sent: Wednesday, July 29, 2015 6:43 AM
>>> >> To: Ard Biesheuvel; Yoder Stuart-B08248
>>> >> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-kernel at lists.infradead.org
>>> >> Subject: Re: bug in identity map for 4KB pages?
>>> >>
>>> >> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
>>> >> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>>> >> > > Our system has RAM at a high address, and previously required using 48-bit VA
>>> >> > > in order to have an idmap that covered all of RAM.
>>> >> > >
>>> >> > > In testing on 4.2-rc4, which now contains support for the increased VA range
>>> >> > > of the idmap without using 48-bit VA, I'm finding that things work for
>>> >> > > 64KB pages, but do not for 4KB pages.
>>> >> > >
>>> >> > > Is there any known limitation here with 4KB pages?  Any ideas?
>>> >> > >
>>> >> >
>>> >> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
>>> >
>>> > Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at
>>> > 0x80_8000_0000.
>>> >
>>> >> > So the physical arrangement still requires more than the 39 bits of
>>> >> > virtual address space you get with 3 levels, even if the ID map can
>>> >> > cope now. That is why you get the 0x40_0000_0000 virtual address:
>>> >> > __phys_to_virt() just wraps to a positive number.
>>> >>
>>> >> Ah, I see. So it's the linear mapping rather than the idmap which is the
>>> >> problem.
>>> >>
>>> >> We cut memory which would fall below the start of the linear map in
>>> >> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
>>> >>
>>> >> It looks like we already have the logic for cutting memory beyond the
>>> >> end of the linear map, so we should just need to override the limit.
>>> >
>>> > What is the effect of cutting that memory?  ...are we losing something?
>>> >
>>>
>>> Yes, you will lose everything above 2 GB in the current configuration.
>>
>> Hmmm...so you really mean that a 16 GB system will have only 2GB
>> total available for both kernel/user space?   That won't be workable
>> as a long term solution.
>>
>
> I agree. So that essentially means that 3 level page table kernels are
> unsuitable for such systems.
>
>>> >> Stuart, does the below patch prevent the panic you see?
>>> >
>>> > I will try it and let you know...can try the suggested patch by
>>> > Ard as well.
>>> >
>>>
>>> Is it possible to load the kernel at 0x80_0000_0000? I think your
>>> system will just boot fine in that case, and you will only lose the
>>> lower 2 GB in that case.
>>
>> So, it's like that lower 2 GB does not exist at all? ...that is going to
>> be hard for our customers to swallow I think.
>>
>> With 48-bit VA enabled, we get all 16GB (and beyond) of memory.
>>
>
> Indeed.
>
>> If it's true that due to our physical memory map that we will not
>> be able to access all RAM with a standard defconfig, then makes
>> me wonder whether we need to request our hardware designers to change
>> that physical map.  (but, IIRC this physical map is that recommended
>> by ARM)
>>
>
> Well, I am aware of an ARM document that explains the placement at
> 0x8000_0000, 0x80_8000_0000 etc.
> But as far as I know, this is not a general recommendation for
> hardware, but an internal guideline for the reference platforms. Of
> course, you need some memory below 4 GB if you want to boot 32-bit
> kernels without jumping through a lot of hoops, i.e., you won't be
> able to boot a 32-bit ARM kernel on AMD Seattle which has all of its
> memory at 0x80_0000_0000
>

I found the doc in question:
http://infocenter.arm.com/help/topic/com.arm.doc.den0001c/DEN0001C_principles_of_arm_memory_maps.pdf

However, it recommends the following layout for up to 512 GB of DRAM:
2 GB at 0x8000_0000
30 GB at 0x8_8000_0000
480 GB at 0x88_0000_0000

bug in identity map for 4KB pages?

[Stuart Yoder]

> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> Sent: Wednesday, July 29, 2015 11:12 AM
> To: Yoder Stuart-B08248
> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
> kernel at lists.infradead.org
> Subject: Re: bug in identity map for 4KB pages?
> 
> On 29 July 2015 at 17:58, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >
> >> -----Original Message-----
> >> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> >> Sent: Wednesday, July 29, 2015 10:48 AM
> >> To: Yoder Stuart-B08248
> >> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
> >> kernel at lists.infradead.org
> >> Subject: Re: bug in identity map for 4KB pages?
> >>
> >> On 29 July 2015 at 17:45, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >> >
> >> >
> >> >> -----Original Message-----
> >> >> From: Mark Rutland [mailto:mark.rutland at arm.com]
> >> >> Sent: Wednesday, July 29, 2015 6:43 AM
> >> >> To: Ard Biesheuvel; Yoder Stuart-B08248
> >> >> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
> kernel at lists.infradead.org
> >> >> Subject: Re: bug in identity map for 4KB pages?
> >> >>
> >> >> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
> >> >> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >> >> > > Our system has RAM at a high address, and previously required using 48-bit VA
> >> >> > > in order to have an idmap that covered all of RAM.
> >> >> > >
> >> >> > > In testing on 4.2-rc4, which now contains support for the increased VA range
> >> >> > > of the idmap without using 48-bit VA, I'm finding that things work for
> >> >> > > 64KB pages, but do not for 4KB pages.
> >> >> > >
> >> >> > > Is there any known limitation here with 4KB pages?  Any ideas?
> >> >> > >
> >> >> >
> >> >> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
> >> >
> >> > Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at
> >> > 0x80_8000_0000.
> >> >
> >> >> > So the physical arrangement still requires more than the 39 bits of
> >> >> > virtual address space you get with 3 levels, even if the ID map can
> >> >> > cope now. That is why you get the 0x40_0000_0000 virtual address:
> >> >> > __phys_to_virt() just wraps to a positive number.
> >> >>
> >> >> Ah, I see. So it's the linear mapping rather than the idmap which is the
> >> >> problem.
> >> >>
> >> >> We cut memory which would fall below the start of the linear map in
> >> >> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
> >> >>
> >> >> It looks like we already have the logic for cutting memory beyond the
> >> >> end of the linear map, so we should just need to override the limit.
> >> >
> >> > What is the effect of cutting that memory?  ...are we losing something?
> >> >
> >>
> >> Yes, you will lose everything above 2 GB in the current configuration.
> >
> > Hmmm...so you really mean that a 16 GB system will have only 2GB
> > total available for both kernel/user space?   That won't be workable
> > as a long term solution.
> >
> 
> I agree. So that essentially means that 3 level page table kernels are
> unsuitable for such systems.
> 
> >> >> Stuart, does the below patch prevent the panic you see?
> >> >
> >> > I will try it and let you know...can try the suggested patch by
> >> > Ard as well.
> >> >
> >>
> >> Is it possible to load the kernel at 0x80_0000_0000? I think your
> >> system will just boot fine in that case, and you will only lose the
> >> lower 2 GB in that case.
> >
> > So, it's like that lower 2 GB does not exist at all? ...that is going to
> > be hard for our customers to swallow I think.
> >
> > With 48-bit VA enabled, we get all 16GB (and beyond) of memory.
> >
> 
> Indeed.
> 
> > If it's true that due to our physical memory map that we will not
> > be able to access all RAM with a standard defconfig, then makes
> > me wonder whether we need to request our hardware designers to change
> > that physical map.  (but, IIRC this physical map is that recommended
> > by ARM)
> >
> 
> Well, I am aware of an ARM document that explains the placement at
> 0x8000_0000, 0x80_8000_0000 etc.
> But as far as I know, this is not a general recommendation for
> hardware, but an internal guideline for the reference platforms. Of
> course, you need some memory below 4 GB if you want to boot 32-bit
> kernels without jumping through a lot of hoops, i.e., you won't be
> able to boot a 32-bit ARM kernel on AMD Seattle which has all of its
> memory at 0x80_0000_0000
> 
> >> Re patch: I am working on a more comprehensive patch atm. Could you
> >> perhaps try that instead (~15 min)?
> >
> > Sure, will wait for your patch and will try it.

Thanks. Tried your patch and the kernel boots (leaving 2GB of memory available).

[cut]
earlycon: Early serial console at MMIO 0x21c0600 (options '115200')
bootconsole [uart0] enabled
efi: Getting EFI parameters from FDT:
efi: UEFI not found.
Ignoring memory outside of linear range (0x004080000000 - 0x0083dfffffff)
cma: Reserved 16 MiB at 0x00000000fec00000
[cut]

So, this still leaves us unable to access all of memory without
48-bit VA or 64KB pages (neither of which are on by default).
(I had started a thread on 48-bit VA last week, and will continue
the discussion there)

Stuart

bug in identity map for 4KB pages?

[Stuart Yoder]

> -----Original Message-----
> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> Sent: Wednesday, July 29, 2015 1:41 PM
> To: Yoder Stuart-B08248
> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
> kernel at lists.infradead.org
> Subject: Re: bug in identity map for 4KB pages?
> 
> On 29 July 2015 at 18:12, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> > On 29 July 2015 at 17:58, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >>
> >>> -----Original Message-----
> >>> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
> >>> Sent: Wednesday, July 29, 2015 10:48 AM
> >>> To: Yoder Stuart-B08248
> >>> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
> >>> kernel at lists.infradead.org
> >>> Subject: Re: bug in identity map for 4KB pages?
> >>>
> >>> On 29 July 2015 at 17:45, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >>> >
> >>> >
> >>> >> -----Original Message-----
> >>> >> From: Mark Rutland [mailto:mark.rutland at arm.com]
> >>> >> Sent: Wednesday, July 29, 2015 6:43 AM
> >>> >> To: Ard Biesheuvel; Yoder Stuart-B08248
> >>> >> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
> kernel at lists.infradead.org
> >>> >> Subject: Re: bug in identity map for 4KB pages?
> >>> >>
> >>> >> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
> >>> >> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
> >>> >> > > Our system has RAM at a high address, and previously required using 48-bit VA
> >>> >> > > in order to have an idmap that covered all of RAM.
> >>> >> > >
> >>> >> > > In testing on 4.2-rc4, which now contains support for the increased VA range
> >>> >> > > of the idmap without using 48-bit VA, I'm finding that things work for
> >>> >> > > 64KB pages, but do not for 4KB pages.
> >>> >> > >
> >>> >> > > Is there any known limitation here with 4KB pages?  Any ideas?
> >>> >> > >
> >>> >> >
> >>> >> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
> >>> >
> >>> > Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at
> >>> > 0x80_8000_0000.
> >>> >
> >>> >> > So the physical arrangement still requires more than the 39 bits of
> >>> >> > virtual address space you get with 3 levels, even if the ID map can
> >>> >> > cope now. That is why you get the 0x40_0000_0000 virtual address:
> >>> >> > __phys_to_virt() just wraps to a positive number.
> >>> >>
> >>> >> Ah, I see. So it's the linear mapping rather than the idmap which is the
> >>> >> problem.
> >>> >>
> >>> >> We cut memory which would fall below the start of the linear map in
> >>> >> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
> >>> >>
> >>> >> It looks like we already have the logic for cutting memory beyond the
> >>> >> end of the linear map, so we should just need to override the limit.
> >>> >
> >>> > What is the effect of cutting that memory?  ...are we losing something?
> >>> >
> >>>
> >>> Yes, you will lose everything above 2 GB in the current configuration.
> >>
> >> Hmmm...so you really mean that a 16 GB system will have only 2GB
> >> total available for both kernel/user space?   That won't be workable
> >> as a long term solution.
> >>
> >
> > I agree. So that essentially means that 3 level page table kernels are
> > unsuitable for such systems.
> >
> >>> >> Stuart, does the below patch prevent the panic you see?
> >>> >
> >>> > I will try it and let you know...can try the suggested patch by
> >>> > Ard as well.
> >>> >
> >>>
> >>> Is it possible to load the kernel at 0x80_0000_0000? I think your
> >>> system will just boot fine in that case, and you will only lose the
> >>> lower 2 GB in that case.
> >>
> >> So, it's like that lower 2 GB does not exist at all? ...that is going to
> >> be hard for our customers to swallow I think.
> >>
> >> With 48-bit VA enabled, we get all 16GB (and beyond) of memory.
> >>
> >
> > Indeed.
> >
> >> If it's true that due to our physical memory map that we will not
> >> be able to access all RAM with a standard defconfig, then makes
> >> me wonder whether we need to request our hardware designers to change
> >> that physical map.  (but, IIRC this physical map is that recommended
> >> by ARM)
> >>
> >
> > Well, I am aware of an ARM document that explains the placement at
> > 0x8000_0000, 0x80_8000_0000 etc.
> > But as far as I know, this is not a general recommendation for
> > hardware, but an internal guideline for the reference platforms. Of
> > course, you need some memory below 4 GB if you want to boot 32-bit
> > kernels without jumping through a lot of hoops, i.e., you won't be
> > able to boot a 32-bit ARM kernel on AMD Seattle which has all of its
> > memory at 0x80_0000_0000
> >
> 
> I found the doc in question:
> http://infocenter.arm.com/help/topic/com.arm.doc.den0001c/DEN0001C_principles_of_arm_memory_maps.pdf
> 
> However, it recommends the following layout for up to 512 GB of DRAM:
> 2 GB at 0x8000_0000
> 30 GB at 0x8_8000_0000
> 480 GB at 0x88_0000_0000

Thanks, I found that as well.  Not sure why the recommendation was not followed
in our case.

Stuart

bug in identity map for 4KB pages?

[Ard Biesheuvel]

On 29 July 2015 at 21:19, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>
>
>> -----Original Message-----
>> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> Sent: Wednesday, July 29, 2015 11:12 AM
>> To: Yoder Stuart-B08248
>> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
>> kernel at lists.infradead.org
>> Subject: Re: bug in identity map for 4KB pages?
>>
>> On 29 July 2015 at 17:58, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>> >
>> >> -----Original Message-----
>> >> From: Ard Biesheuvel [mailto:ard.biesheuvel at linaro.org]
>> >> Sent: Wednesday, July 29, 2015 10:48 AM
>> >> To: Yoder Stuart-B08248
>> >> Cc: Mark Rutland; Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
>> >> kernel at lists.infradead.org
>> >> Subject: Re: bug in identity map for 4KB pages?
>> >>
>> >> On 29 July 2015 at 17:45, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>> >> >
>> >> >
>> >> >> -----Original Message-----
>> >> >> From: Mark Rutland [mailto:mark.rutland at arm.com]
>> >> >> Sent: Wednesday, July 29, 2015 6:43 AM
>> >> >> To: Ard Biesheuvel; Yoder Stuart-B08248
>> >> >> Cc: Marc Zyngier; Catalin Marinas; Newton Peter-RA3823; Will Deacon; linux-arm-
>> kernel at lists.infradead.org
>> >> >> Subject: Re: bug in identity map for 4KB pages?
>> >> >>
>> >> >> On Wed, Jul 29, 2015 at 08:47:10AM +0100, Ard Biesheuvel wrote:
>> >> >> > On 29 July 2015 at 04:37, Stuart Yoder <stuart.yoder@freescale.com> wrote:
>> >> >> > > Our system has RAM at a high address, and previously required using 48-bit VA
>> >> >> > > in order to have an idmap that covered all of RAM.
>> >> >> > >
>> >> >> > > In testing on 4.2-rc4, which now contains support for the increased VA range
>> >> >> > > of the idmap without using 48-bit VA, I'm finding that things work for
>> >> >> > > 64KB pages, but do not for 4KB pages.
>> >> >> > >
>> >> >> > > Is there any known limitation here with 4KB pages?  Any ideas?
>> >> >> > >
>> >> >> >
>> >> >> > You probably have memory at 0x8000_0000 and at 0x80_8000_0000, right?
>> >> >
>> >> > Yes.  We have 2GB of memory at 0x8000_0000 and the remainder at
>> >> > 0x80_8000_0000.
>> >> >
>> >> >> > So the physical arrangement still requires more than the 39 bits of
>> >> >> > virtual address space you get with 3 levels, even if the ID map can
>> >> >> > cope now. That is why you get the 0x40_0000_0000 virtual address:
>> >> >> > __phys_to_virt() just wraps to a positive number.
>> >> >>
>> >> >> Ah, I see. So it's the linear mapping rather than the idmap which is the
>> >> >> problem.
>> >> >>
>> >> >> We cut memory which would fall below the start of the linear map in
>> >> >> early_init_dt_add_memory_arch, by cutting memory below phys_offset.
>> >> >>
>> >> >> It looks like we already have the logic for cutting memory beyond the
>> >> >> end of the linear map, so we should just need to override the limit.
>> >> >
>> >> > What is the effect of cutting that memory?  ...are we losing something?
>> >> >
>> >>
>> >> Yes, you will lose everything above 2 GB in the current configuration.
>> >
>> > Hmmm...so you really mean that a 16 GB system will have only 2GB
>> > total available for both kernel/user space?   That won't be workable
>> > as a long term solution.
>> >
>>
>> I agree. So that essentially means that 3 level page table kernels are
>> unsuitable for such systems.
>>
>> >> >> Stuart, does the below patch prevent the panic you see?
>> >> >
>> >> > I will try it and let you know...can try the suggested patch by
>> >> > Ard as well.
>> >> >
>> >>
>> >> Is it possible to load the kernel at 0x80_0000_0000? I think your
>> >> system will just boot fine in that case, and you will only lose the
>> >> lower 2 GB in that case.
>> >
>> > So, it's like that lower 2 GB does not exist at all? ...that is going to
>> > be hard for our customers to swallow I think.
>> >
>> > With 48-bit VA enabled, we get all 16GB (and beyond) of memory.
>> >
>>
>> Indeed.
>>
>> > If it's true that due to our physical memory map that we will not
>> > be able to access all RAM with a standard defconfig, then makes
>> > me wonder whether we need to request our hardware designers to change
>> > that physical map.  (but, IIRC this physical map is that recommended
>> > by ARM)
>> >
>>
>> Well, I am aware of an ARM document that explains the placement at
>> 0x8000_0000, 0x80_8000_0000 etc.
>> But as far as I know, this is not a general recommendation for
>> hardware, but an internal guideline for the reference platforms. Of
>> course, you need some memory below 4 GB if you want to boot 32-bit
>> kernels without jumping through a lot of hoops, i.e., you won't be
>> able to boot a 32-bit ARM kernel on AMD Seattle which has all of its
>> memory at 0x80_0000_0000
>>
>> >> Re patch: I am working on a more comprehensive patch atm. Could you
>> >> perhaps try that instead (~15 min)?
>> >
>> > Sure, will wait for your patch and will try it.
>
> Thanks. Tried your patch and the kernel boots (leaving 2GB of memory available).
>

OK. Thanks for confirming.

-- 
Ard.


> [cut]
> earlycon: Early serial console at MMIO 0x21c0600 (options '115200')
> bootconsole [uart0] enabled
> efi: Getting EFI parameters from FDT:
> efi: UEFI not found.
> Ignoring memory outside of linear range (0x004080000000 - 0x0083dfffffff)
> cma: Reserved 16 MiB at 0x00000000fec00000
> [cut]
>
> So, this still leaves us unable to access all of memory without
> 48-bit VA or 64KB pages (neither of which are on by default).
> (I had started a thread on 48-bit VA last week, and will continue
> the discussion there)
>
> Stuart