SPARSEMEM memory needs?

SPARSEMEM memory needs?

[Joakim Tjernlund]

I am trying to reduce RAM used by the kernel and enabled memblock debug and found these(annotated with BT here):

[    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
[    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
[    0.000000] ------------[ cut here ]------------
[    0.000000] Call trace:
[    0.000000]  vmemmap_alloc_block+0xc4/0xe8
[    0.000000]  vmemmap_pud_populate+0x24/0xb8
[    0.000000]  vmemmap_populate+0xa4/0x180
[    0.000000]  __populate_section_memmap+0x50/0x70
[    0.000000]  sparse_init_nid+0x164/0x1d4
[    0.000000]  sparse_init+0xb0/0x224
[    0.000000]  bootmem_init+0x40/0x80
[    0.000000]  setup_arch+0x244/0x540
[    0.000000]  start_kernel+0x60/0x804
[    0.000000]  __primary_switched+0xa0/0xa8
[    0.000000] ---[ end trace 0000000000000000 ]---
[    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
[    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
[    0.000000] ------------[ cut here ]------------
[    0.000000] Call trace:
[    0.000000]  vmemmap_alloc_block+0xc4/0xe8
[    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
[    0.000000]  vmemmap_populate+0xd0/0x180
[    0.000000]  __populate_section_memmap+0x50/0x70
[    0.000000]  sparse_init_nid+0x164/0x1d4
[    0.000000]  sparse_init+0xb0/0x224
[    0.000000]  bootmem_init+0x40/0x80
[    0.000000]  setup_arch+0x244/0x540
[    0.000000]  start_kernel+0x60/0x804
[    0.000000]  __primary_switched+0xa0/0xa8
[    0.000000] ---[ end trace f68728a0d3053b53 ]---
[    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
[    0.000000] memblock_reserve: [0x0000000061800000-0x00000000619fffff] memblock_alloc_range_nid+0xc8/0x134

It seems AARCH64 SPARSEMEM uses some RAM and I wonder if this use can be somewhat reduced?
Looking at this code I have not yet figured out how these allocations work.

If SPARSEMEM really needs all that RAM, could a simpler memory model like FLATMEM be supported?

 Jocke
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Ard Biesheuvel]

Hello Joakim,

On Mon, 6 Jun 2022 at 17:20, Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
>
> I am trying to reduce RAM used by the kernel and enabled memblock debug and found these(annotated with BT here):
>
> [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> [    0.000000] ------------[ cut here ]------------
> [    0.000000] Call trace:
> [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> [    0.000000]  vmemmap_populate+0xa4/0x180
> [    0.000000]  __populate_section_memmap+0x50/0x70
> [    0.000000]  sparse_init_nid+0x164/0x1d4
> [    0.000000]  sparse_init+0xb0/0x224
> [    0.000000]  bootmem_init+0x40/0x80
> [    0.000000]  setup_arch+0x244/0x540
> [    0.000000]  start_kernel+0x60/0x804
> [    0.000000]  __primary_switched+0xa0/0xa8
> [    0.000000] ---[ end trace 0000000000000000 ]---
> [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134

I'm not sure which backtrace belongs with which memblock debug
message, but something looks wrong here.  vmemmap_pud_populate() does
an allocation of PAGE_SIZE, but your kernel is allocating 2 megabytes
here.

Which kernel version are you using? Do you have any out of tree
changes related to memory management? I assume you are using the
standard 4k page size, right?


> [    0.000000] ------------[ cut here ]------------
> [    0.000000] Call trace:
> [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> [    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
> [    0.000000]  vmemmap_populate+0xd0/0x180
> [    0.000000]  __populate_section_memmap+0x50/0x70
> [    0.000000]  sparse_init_nid+0x164/0x1d4
> [    0.000000]  sparse_init+0xb0/0x224
> [    0.000000]  bootmem_init+0x40/0x80
> [    0.000000]  setup_arch+0x244/0x540
> [    0.000000]  start_kernel+0x60/0x804
> [    0.000000]  __primary_switched+0xa0/0xa8
> [    0.000000] ---[ end trace f68728a0d3053b53 ]---
> [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> [    0.000000] memblock_reserve: [0x0000000061800000-0x00000000619fffff] memblock_alloc_range_nid+0xc8/0x134
>
> It seems AARCH64 SPARSEMEM uses some RAM and I wonder if this use can be somewhat reduced?
> Looking at this code I have not yet figured out how these allocations work.
>
> If SPARSEMEM really needs all that RAM, could a simpler memory model like FLATMEM be supported?
>
>  Jocke
> _______________________________________________
> linux-arm-kernel mailing list
> linux-arm-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Joakim Tjernlund]

On Mon, 2022-06-06 at 21:10 +0200, Ard Biesheuvel wrote:
> Hello Joakim,
> 
> On Mon, 6 Jun 2022 at 17:20, Joakim Tjernlund
> <Joakim.Tjernlund@infinera.com> wrote:
> > 
> > I am trying to reduce RAM used by the kernel and enabled memblock debug and found these(annotated with BT here):
> > 
> > [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> > [    0.000000] ------------[ cut here ]------------
> > [    0.000000] Call trace:
> > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> > [    0.000000]  vmemmap_populate+0xa4/0x180
> > [    0.000000]  __populate_section_memmap+0x50/0x70
> > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > [    0.000000]  sparse_init+0xb0/0x224
> > [    0.000000]  bootmem_init+0x40/0x80
> > [    0.000000]  setup_arch+0x244/0x540
> > [    0.000000]  start_kernel+0x60/0x804
> > [    0.000000]  __primary_switched+0xa0/0xa8
> > [    0.000000] ---[ end trace 0000000000000000 ]---
> > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
> 
> I'm not sure which backtrace belongs with which memblock debug
> message, but something looks wrong here.  vmemmap_pud_populate() does
> an allocation of PAGE_SIZE, but your kernel is allocating 2 megabytes
> here.

I have added this to get a BT, I trimmed the regs ways and just kept the BT
diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
index bdce883f9286..f8326e1295ed 100644
--- a/mm/sparse-vmemmap.c
+++ b/mm/sparse-vmemmap.c
@@ -418,9 +418,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
                        warned = true;
                }
                return NULL;
-       } else
+       } else {
+               WARN_ON(1);
                return __earlyonly_bootmem_alloc(node, size, size,
                                __pa(MAX_DMA_ADDRESS));
+       }
 }


I guess I may have trimmed the log a bit too much. How does this look?

[    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
[    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
[    0.000000] Call trace:
[    0.000000]  vmemmap_alloc_block+0xc4/0xe8
[    0.000000]  vmemmap_pud_populate+0x24/0xb8
[    0.000000]  vmemmap_populate+0xa4/0x180
[    0.000000]  __populate_section_memmap+0x50/0x70
[    0.000000]  sparse_init_nid+0x164/0x1d4
[    0.000000]  sparse_init+0xb0/0x224
[    0.000000]  bootmem_init+0x40/0x80
[    0.000000]  setup_arch+0x244/0x540
[    0.000000]  start_kernel+0x60/0x804
[    0.000000]  __primary_switched+0xa0/0xa8
[    0.000000] ---[ end trace 0000000000000000 ]---
[    0.000000] memblock_alloc_try_nid_raw: 4096 bytes align=0x1000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
[    0.000000] memblock_reserve: [0x00000000703e7000-0x00000000703e7fff] memblock_alloc_range_nid+0xc8/0x134
[    0.000000] Call trace:
[    0.000000]  vmemmap_alloc_block+0xc4/0xe8
[    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
[    0.000000]  vmemmap_populate+0xd0/0x180
[    0.000000]  __populate_section_memmap+0x50/0x70
[    0.000000]  sparse_init_nid+0x164/0x1d4
[    0.000000]  sparse_init+0xb0/0x224
[    0.000000]  bootmem_init+0x40/0x80
[    0.000000]  setup_arch+0x244/0x540
[    0.000000]  start_kernel+0x60/0x804
[    0.000000]  __primary_switched+0xa0/0xa8
[    0.000000] ---[ end trace f68728a0d3053b52 ]---
[    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
[    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
[    0.000000] Call trace:
[    0.000000]  vmemmap_alloc_block+0xc4/0xe8
[    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
[    0.000000]  vmemmap_populate+0xd0/0x180
[    0.000000]  __populate_section_memmap+0x50/0x70
[    0.000000]  sparse_init_nid+0x164/0x1d4
[    0.000000]  sparse_init+0xb0/0x224
[    0.000000]  bootmem_init+0x40/0x80
[    0.000000]  setup_arch+0x244/0x540
[    0.000000]  start_kernel+0x60/0x804
[    0.000000]  __primary_switched+0xa0/0xa8
[    0.000000] ---[ end trace f68728a0d3053b53 ]---
[    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
[    0.000000] memblock_reserve: [0x0000000061800000-0x00000000619fffff] memblock_alloc_range_nid+0xc8/0x134
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x0000000060000000-0x00000000703fffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000060000000-0x0000000061ffffff]
[    0.000000]   node   0: [mem 0x0000000070000000-0x0000000070000fff]
[    0.000000]   node   0: [mem 0x0000000070001000-0x00000000703fffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000060000000-0x00000000703fffff]
[    0.000000] MEMBLOCK configuration:
[    0.000000]  memory size = 0x0000000002400000 reserved size = 0x0000000000de991f
[    0.000000]  memory.cnt  = 0x3
[    0.000000]  memory[0x0]     [0x0000000060000000-0x0000000061ffffff], 0x0000000002000000 bytes flags: 0x0
[    0.000000]  memory[0x1]     [0x0000000070000000-0x0000000070000fff], 0x0000000000001000 bytes flags: 0x4
[    0.000000]  memory[0x2]     [0x0000000070001000-0x00000000703fffff], 0x00000000003ff000 bytes flags: 0x0
[    0.000000]  reserved.cnt  = 0x9
[    0.000000]  reserved[0x0]   [0x0000000060010000-0x00000000605cefff], 0x00000000005bf000 bytes flags: 0x0
[    0.000000]  reserved[0x1]   [0x00000000605d1000-0x00000000605dffff], 0x000000000000f000 bytes flags: 0x0
[    0.000000]  reserved[0x2]   [0x0000000061800000-0x0000000061ffffff], 0x0000000000800000 bytes flags: 0x0
[    0.000000]  reserved[0x3]   [0x0000000070377000-0x0000000070379fff], 0x0000000000003000 bytes flags: 0x0
[    0.000000]  reserved[0x4]   [0x00000000703e7000-0x00000000703e7fff], 0x0000000000001000 bytes flags: 0x0
[    0.000000]  reserved[0x5]   [0x00000000703e86c0-0x00000000703e86ef], 0x0000000000000030 bytes flags: 0x0
[    0.000000]  reserved[0x6]   [0x00000000703e8700-0x00000000703e97ff], 0x0000000000001100 bytes flags: 0x0
[    0.000000]  reserved[0x7]   [0x00000000703e9810-0x00000000703e983e], 0x000000000000002f bytes flags: 0x0
[    0.000000]  reserved[0x8]   [0x00000000703e9840-0x00000000703fffff], 0x00000000000167c0 bytes flags: 0x0


> 
> Which kernel version are you using? Do you have any out of tree
> changes related to memory management? I assume you are using the
> standard 4k page size, right?

5.15.26, no MM changes and using 4KB. Oh a small MM change that didn't do much I had added to this run:
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -941,6 +941,9 @@ choice
 config ARM64_PA_BITS_48
        bool "48-bit"
 
+config ARM64_PA_BITS_40
+       bool "40-bit"
+
 config ARM64_PA_BITS_52
        bool "52-bit (ARMv8.2)"
        depends on ARM64_64K_PAGES
@@ -957,6 +960,7 @@ endchoice
 
 config ARM64_PA_BITS
        int
+       default 40 if ARM64_PA_BITS_40
        default 48 if ARM64_PA_BITS_48
        default 52 if ARM64_PA_BITS_52

> 
> 
> > [    0.000000] ------------[ cut here ]------------
> > [    0.000000] Call trace:
> > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > [    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
> > [    0.000000]  vmemmap_populate+0xd0/0x180
> > [    0.000000]  __populate_section_memmap+0x50/0x70
> > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > [    0.000000]  sparse_init+0xb0/0x224
> > [    0.000000]  bootmem_init+0x40/0x80
> > [    0.000000]  setup_arch+0x244/0x540
> > [    0.000000]  start_kernel+0x60/0x804
> > [    0.000000]  __primary_switched+0xa0/0xa8
> > [    0.000000] ---[ end trace f68728a0d3053b53 ]---
> > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > [    0.000000] memblock_reserve: [0x0000000061800000-0x00000000619fffff] memblock_alloc_range_nid+0xc8/0x134
> > 
> > It seems AARCH64 SPARSEMEM uses some RAM and I wonder if this use can be somewhat reduced?
> > Looking at this code I have not yet figured out how these allocations work.
> > 
> > If SPARSEMEM really needs all that RAM, could a simpler memory model like FLATMEM be supported?
> > 
> >  Jocke
> > _______________________________________________
> > linux-arm-kernel mailing list
> > linux-arm-kernel@lists.infradead.org
> > https://nam11.safelinks.protection.outlook.com/?url=http%3A%2F%2Flists.infradead.org%2Fmailman%2Flistinfo%2Flinux-arm-kernel&amp;data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C7fe98adbbd1f4582486608da47f05071%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637901394719615340%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=irSjvXlqRsP%2FoIKjcZjC%2Bg5hfMJZBkWaJbkJTLAQk%2Bs%3D&amp;reserved=0

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Ard Biesheuvel]

On Mon, 6 Jun 2022 at 22:37, Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
>
> On Mon, 2022-06-06 at 21:10 +0200, Ard Biesheuvel wrote:
> > Hello Joakim,
> >
> > On Mon, 6 Jun 2022 at 17:20, Joakim Tjernlund
> > <Joakim.Tjernlund@infinera.com> wrote:
> > >
> > > I am trying to reduce RAM used by the kernel and enabled memblock debug and found these(annotated with BT here):
> > >
> > > [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > > [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> > > [    0.000000] ------------[ cut here ]------------
> > > [    0.000000] Call trace:
> > > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > > [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> > > [    0.000000]  vmemmap_populate+0xa4/0x180
> > > [    0.000000]  __populate_section_memmap+0x50/0x70
> > > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > > [    0.000000]  sparse_init+0xb0/0x224
> > > [    0.000000]  bootmem_init+0x40/0x80
> > > [    0.000000]  setup_arch+0x244/0x540
> > > [    0.000000]  start_kernel+0x60/0x804
> > > [    0.000000]  __primary_switched+0xa0/0xa8
> > > [    0.000000] ---[ end trace 0000000000000000 ]---
> > > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > > [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
> >
> > I'm not sure which backtrace belongs with which memblock debug
> > message, but something looks wrong here.  vmemmap_pud_populate() does
> > an allocation of PAGE_SIZE, but your kernel is allocating 2 megabytes
> > here.
>
> I have added this to get a BT, I trimmed the regs ways and just kept the BT
> diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
> index bdce883f9286..f8326e1295ed 100644
> --- a/mm/sparse-vmemmap.c
> +++ b/mm/sparse-vmemmap.c
> @@ -418,9 +418,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
>                         warned = true;
>                 }
>                 return NULL;
> -       } else
> +       } else {
> +               WARN_ON(1);
>                 return __earlyonly_bootmem_alloc(node, size, size,
>                                 __pa(MAX_DMA_ADDRESS));
> +       }
>  }
>
>
> I guess I may have trimmed the log a bit too much. How does this look?
>
> [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134

OK, so this one is unaccounted for.

> [    0.000000] Call trace:
> [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> [    0.000000]  vmemmap_populate+0xa4/0x180
> [    0.000000]  __populate_section_memmap+0x50/0x70
> [    0.000000]  sparse_init_nid+0x164/0x1d4
> [    0.000000]  sparse_init+0xb0/0x224
> [    0.000000]  bootmem_init+0x40/0x80
> [    0.000000]  setup_arch+0x244/0x540
> [    0.000000]  start_kernel+0x60/0x804
> [    0.000000]  __primary_switched+0xa0/0xa8
> [    0.000000] ---[ end trace 0000000000000000 ]---
> [    0.000000] memblock_alloc_try_nid_raw: 4096 bytes align=0x1000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> [    0.000000] memblock_reserve: [0x00000000703e7000-0x00000000703e7fff] memblock_alloc_range_nid+0xc8/0x134

This is a single page for the vmemmap PUD

> [    0.000000] Call trace:
> [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> [    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
> [    0.000000]  vmemmap_populate+0xd0/0x180
> [    0.000000]  __populate_section_memmap+0x50/0x70
> [    0.000000]  sparse_init_nid+0x164/0x1d4
> [    0.000000]  sparse_init+0xb0/0x224
> [    0.000000]  bootmem_init+0x40/0x80
> [    0.000000]  setup_arch+0x244/0x540
> [    0.000000]  start_kernel+0x60/0x804
> [    0.000000]  __primary_switched+0xa0/0xa8
> [    0.000000] ---[ end trace f68728a0d3053b52 ]---
> [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134

This allocates 2 MB to cover the struct page[] array for the first
section of memory (32 MB), starting at address 0x6000_0000

> [    0.000000] Call trace:
> [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> [    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
> [    0.000000]  vmemmap_populate+0xd0/0x180
> [    0.000000]  __populate_section_memmap+0x50/0x70
> [    0.000000]  sparse_init_nid+0x164/0x1d4
> [    0.000000]  sparse_init+0xb0/0x224
> [    0.000000]  bootmem_init+0x40/0x80
> [    0.000000]  setup_arch+0x244/0x540
> [    0.000000]  start_kernel+0x60/0x804
> [    0.000000]  __primary_switched+0xa0/0xa8
> [    0.000000] ---[ end trace f68728a0d3053b53 ]---
> [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> [    0.000000] memblock_reserve: [0x0000000061800000-0x00000000619fffff] memblock_alloc_range_nid+0xc8/0x134

This allocates 2 MB to cover the struct page[] array for the second
section of memory (4 MB), starting at 0x7000_0000

So the problem is that you only have 36 MB of DRAM, with a large hole
in the middle. Sparsemem was actually designed for that (hence the
name), and flatmem would make things much worse.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Joakim Tjernlund]

On Mon, 2022-06-06 at 23:50 +0200, Ard Biesheuvel wrote:
> On Mon, 6 Jun 2022 at 22:37, Joakim Tjernlund
> <Joakim.Tjernlund@infinera.com> wrote:
> > 
> > On Mon, 2022-06-06 at 21:10 +0200, Ard Biesheuvel wrote:
> > > Hello Joakim,
> > > 
> > > On Mon, 6 Jun 2022 at 17:20, Joakim Tjernlund
> > > <Joakim.Tjernlund@infinera.com> wrote:
> > > > 
> > > > I am trying to reduce RAM used by the kernel and enabled memblock debug and found these(annotated with BT here):
> > > > 
> > > > [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > > > [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> > > > [    0.000000] ------------[ cut here ]------------
> > > > [    0.000000] Call trace:
> > > > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > > > [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> > > > [    0.000000]  vmemmap_populate+0xa4/0x180
> > > > [    0.000000]  __populate_section_memmap+0x50/0x70
> > > > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > > > [    0.000000]  sparse_init+0xb0/0x224
> > > > [    0.000000]  bootmem_init+0x40/0x80
> > > > [    0.000000]  setup_arch+0x244/0x540
> > > > [    0.000000]  start_kernel+0x60/0x804
> > > > [    0.000000]  __primary_switched+0xa0/0xa8
> > > > [    0.000000] ---[ end trace 0000000000000000 ]---
> > > > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > > > [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
> > > 
> > > I'm not sure which backtrace belongs with which memblock debug
> > > message, but something looks wrong here.  vmemmap_pud_populate() does
> > > an allocation of PAGE_SIZE, but your kernel is allocating 2 megabytes
> > > here.
> > 
> > I have added this to get a BT, I trimmed the regs ways and just kept the BT
> > diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
> > index bdce883f9286..f8326e1295ed 100644
> > --- a/mm/sparse-vmemmap.c
> > +++ b/mm/sparse-vmemmap.c
> > @@ -418,9 +418,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
> >                         warned = true;
> >                 }
> >                 return NULL;
> > -       } else
> > +       } else {
> > +               WARN_ON(1);
> >                 return __earlyonly_bootmem_alloc(node, size, size,
> >                                 __pa(MAX_DMA_ADDRESS));
> > +       }
> >  }
> > 
> > 
> > I guess I may have trimmed the log a bit too much. How does this look?
> > 
> > [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> 
> OK, so this one is unaccounted for.

Need to find out where this is coming from then, thanks.

> 
> > [    0.000000] Call trace:
> > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> > [    0.000000]  vmemmap_populate+0xa4/0x180
> > [    0.000000]  __populate_section_memmap+0x50/0x70
> > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > [    0.000000]  sparse_init+0xb0/0x224
> > [    0.000000]  bootmem_init+0x40/0x80
> > [    0.000000]  setup_arch+0x244/0x540
> > [    0.000000]  start_kernel+0x60/0x804
> > [    0.000000]  __primary_switched+0xa0/0xa8
> > [    0.000000] ---[ end trace 0000000000000000 ]---
> > [    0.000000] memblock_alloc_try_nid_raw: 4096 bytes align=0x1000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > [    0.000000] memblock_reserve: [0x00000000703e7000-0x00000000703e7fff] memblock_alloc_range_nid+0xc8/0x134
> 
> This is a single page for the vmemmap PUD
> 
> > [    0.000000] Call trace:
> > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > [    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
> > [    0.000000]  vmemmap_populate+0xd0/0x180
> > [    0.000000]  __populate_section_memmap+0x50/0x70
> > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > [    0.000000]  sparse_init+0xb0/0x224
> > [    0.000000]  bootmem_init+0x40/0x80
> > [    0.000000]  setup_arch+0x244/0x540
> > [    0.000000]  start_kernel+0x60/0x804
> > [    0.000000]  __primary_switched+0xa0/0xa8
> > [    0.000000] ---[ end trace f68728a0d3053b52 ]---
> > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
> 
> This allocates 2 MB to cover the struct page[] array for the first
> section of memory (32 MB), starting at address 0x6000_0000
> 
> > [    0.000000] Call trace:
> > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > [    0.000000]  vmemmap_alloc_block_buf+0xfc/0x100
> > [    0.000000]  vmemmap_populate+0xd0/0x180
> > [    0.000000]  __populate_section_memmap+0x50/0x70
> > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > [    0.000000]  sparse_init+0xb0/0x224
> > [    0.000000]  bootmem_init+0x40/0x80
> > [    0.000000]  setup_arch+0x244/0x540
> > [    0.000000]  start_kernel+0x60/0x804
> > [    0.000000]  __primary_switched+0xa0/0xa8
> > [    0.000000] ---[ end trace f68728a0d3053b53 ]---
> > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > [    0.000000] memblock_reserve: [0x0000000061800000-0x00000000619fffff] memblock_alloc_range_nid+0xc8/0x134
> 
> This allocates 2 MB to cover the struct page[] array for the second
> section of memory (4 MB), starting at 0x7000_0000

OK, so every section of RAM costs 2MB to administer. I guess there is nothing one can do about that?
The one think that his my mind is that we would be happy with ARM64_PA_BITS_32, we don't have any
addresses above that in this small system.

> 
> So the problem is that you only have 36 MB of DRAM, with a large hole
> in the middle. Sparsemem was actually designed for that (hence the
> name), and flatmem would make things much worse.

Yes, that hole is not ideal.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Ard Biesheuvel]

On Tue, 7 Jun 2022 at 08:36, Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
>
> On Mon, 2022-06-06 at 23:50 +0200, Ard Biesheuvel wrote:
> > On Mon, 6 Jun 2022 at 22:37, Joakim Tjernlund
> > <Joakim.Tjernlund@infinera.com> wrote:
> > >
> > > On Mon, 2022-06-06 at 21:10 +0200, Ard Biesheuvel wrote:
> > > > Hello Joakim,
> > > >
> > > > On Mon, 6 Jun 2022 at 17:20, Joakim Tjernlund
> > > > <Joakim.Tjernlund@infinera.com> wrote:
> > > > >
> > > > > I am trying to reduce RAM used by the kernel and enabled memblock debug and found these(annotated with BT here):
> > > > >
> > > > > [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > > > > [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> > > > > [    0.000000] ------------[ cut here ]------------
> > > > > [    0.000000] Call trace:
> > > > > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > > > > [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> > > > > [    0.000000]  vmemmap_populate+0xa4/0x180
> > > > > [    0.000000]  __populate_section_memmap+0x50/0x70
> > > > > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > > > > [    0.000000]  sparse_init+0xb0/0x224
> > > > > [    0.000000]  bootmem_init+0x40/0x80
> > > > > [    0.000000]  setup_arch+0x244/0x540
> > > > > [    0.000000]  start_kernel+0x60/0x804
> > > > > [    0.000000]  __primary_switched+0xa0/0xa8
> > > > > [    0.000000] ---[ end trace 0000000000000000 ]---
> > > > > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > > > > [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
> > > >
> > > > I'm not sure which backtrace belongs with which memblock debug
> > > > message, but something looks wrong here.  vmemmap_pud_populate() does
> > > > an allocation of PAGE_SIZE, but your kernel is allocating 2 megabytes
> > > > here.
> > >
> > > I have added this to get a BT, I trimmed the regs ways and just kept the BT
> > > diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
> > > index bdce883f9286..f8326e1295ed 100644
> > > --- a/mm/sparse-vmemmap.c
> > > +++ b/mm/sparse-vmemmap.c
> > > @@ -418,9 +418,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
> > >                         warned = true;
> > >                 }
> > >                 return NULL;
> > > -       } else
> > > +       } else {
> > > +               WARN_ON(1);
> > >                 return __earlyonly_bootmem_alloc(node, size, size,
> > >                                 __pa(MAX_DMA_ADDRESS));
> > > +       }
> > >  }
> > >
> > >
> > > I guess I may have trimmed the log a bit too much. How does this look?
> > >
> > > [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > > [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> >
> > OK, so this one is unaccounted for.
>
> Need to find out where this is coming from then, thanks.
>
...
> OK, so every section of RAM costs 2MB to administer. I guess there is nothing one can do about that?
> The one think that his my mind is that we would be happy with ARM64_PA_BITS_32, we don't have any
> addresses above that in this small system.
>

I don't see how that is going to help.

> >
> > So the problem is that you only have 36 MB of DRAM, with a large hole
> > in the middle. Sparsemem was actually designed for that (hence the
> > name), and flatmem would make things much worse.
>
> Yes, that hole is not ideal.

What you might try is changing the section size to 32 MB and mapping
the vmemmap region down to pages. That way, the vmemmap region should
only take up
- 512 KiB for the struct page array[] itself
- 4 KiB for the page table that replaces the 2 MB block mapping

You could try the below and see if it makes any difference?

diff --git a/arch/arm64/include/asm/sparsemem.h
b/arch/arm64/include/asm/sparsemem.h
index 4b73463423c3..a008f4342532 100644
--- a/arch/arm64/include/asm/sparsemem.h
+++ b/arch/arm64/include/asm/sparsemem.h
@@ -23,7 +23,7 @@
  * entries could not be created for vmemmap mappings.
  * 16K follows 4K for simplicity.
  */
-#define SECTION_SIZE_BITS 27
+#define SECTION_SIZE_BITS 25
 #endif /* CONFIG_ARM64_64K_PAGES */

 #endif
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 5b1946f1805c..d25560a53a67 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
addr, unsigned long end,
 }
 #endif

-#if !ARM64_KERNEL_USES_PMD_MAPS
+#if 1// !ARM64_KERNEL_USES_PMD_MAPS
 int __meminit vmemmap_populate(unsigned long start, unsigned long
end, int node,
                struct vmem_altmap *altmap)
 {

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Joakim Tjernlund]

On Tue, 2022-06-07 at 09:15 +0200, Ard Biesheuvel wrote:
> On Tue, 7 Jun 2022 at 08:36, Joakim Tjernlund
> <Joakim.Tjernlund@infinera.com> wrote:
> > 
> > On Mon, 2022-06-06 at 23:50 +0200, Ard Biesheuvel wrote:
> > > On Mon, 6 Jun 2022 at 22:37, Joakim Tjernlund
> > > <Joakim.Tjernlund@infinera.com> wrote:
> > > > 
> > > > On Mon, 2022-06-06 at 21:10 +0200, Ard Biesheuvel wrote:
> > > > > Hello Joakim,
> > > > > 
> > > > > On Mon, 6 Jun 2022 at 17:20, Joakim Tjernlund
> > > > > <Joakim.Tjernlund@infinera.com> wrote:
> > > > > > 
> > > > > > I am trying to reduce RAM used by the kernel and enabled memblock debug and found these(annotated with BT here):
> > > > > > 
> > > > > > [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > > > > > [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> > > > > > [    0.000000] ------------[ cut here ]------------
> > > > > > [    0.000000] Call trace:
> > > > > > [    0.000000]  vmemmap_alloc_block+0xc4/0xe8
> > > > > > [    0.000000]  vmemmap_pud_populate+0x24/0xb8
> > > > > > [    0.000000]  vmemmap_populate+0xa4/0x180
> > > > > > [    0.000000]  __populate_section_memmap+0x50/0x70
> > > > > > [    0.000000]  sparse_init_nid+0x164/0x1d4
> > > > > > [    0.000000]  sparse_init+0xb0/0x224
> > > > > > [    0.000000]  bootmem_init+0x40/0x80
> > > > > > [    0.000000]  setup_arch+0x244/0x540
> > > > > > [    0.000000]  start_kernel+0x60/0x804
> > > > > > [    0.000000]  __primary_switched+0xa0/0xa8
> > > > > > [    0.000000] ---[ end trace 0000000000000000 ]---
> > > > > > [    0.000000] memblock_alloc_try_nid_raw: 2097152 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 __earlyonly_bootmem_alloc+0x20/0x28
> > > > > > [    0.000000] memblock_reserve: [0x0000000061a00000-0x0000000061bfffff] memblock_alloc_range_nid+0xc8/0x134
> > > > > 
> > > > > I'm not sure which backtrace belongs with which memblock debug
> > > > > message, but something looks wrong here.  vmemmap_pud_populate() does
> > > > > an allocation of PAGE_SIZE, but your kernel is allocating 2 megabytes
> > > > > here.
> > > > 
> > > > I have added this to get a BT, I trimmed the regs ways and just kept the BT
> > > > diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c
> > > > index bdce883f9286..f8326e1295ed 100644
> > > > --- a/mm/sparse-vmemmap.c
> > > > +++ b/mm/sparse-vmemmap.c
> > > > @@ -418,9 +418,11 @@ void * __meminit vmemmap_alloc_block(unsigned long size, int node)
> > > >                         warned = true;
> > > >                 }
> > > >                 return NULL;
> > > > -       } else
> > > > +       } else {
> > > > +               WARN_ON(1);
> > > >                 return __earlyonly_bootmem_alloc(node, size, size,
> > > >                                 __pa(MAX_DMA_ADDRESS));
> > > > +       }
> > > >  }
> > > > 
> > > > 
> > > > I guess I may have trimmed the log a bit too much. How does this look?
> > > > 
> > > > [    0.000000] memblock_alloc_exact_nid_raw: 4194304 bytes align=0x200000 nid=0 from=0x0000000040000000 max_addr=0x0000000000000000 memmap_alloc+0x1c/0x2c
> > > > [    0.000000] memblock_reserve: [0x0000000061c00000-0x0000000061ffffff] memblock_alloc_range_nid+0xc8/0x134
> > > 
> > > OK, so this one is unaccounted for.
> > 
> > Need to find out where this is coming from then, thanks.
> > 
> ...
> > OK, so every section of RAM costs 2MB to administer. I guess there is nothing one can do about that?
> > The one think that his my mind is that we would be happy with ARM64_PA_BITS_32, we don't have any
> > addresses above that in this small system.
> > 
> 
> I don't see how that is going to help.
> 
> > > 
> > > So the problem is that you only have 36 MB of DRAM, with a large hole
> > > in the middle. Sparsemem was actually designed for that (hence the
> > > name), and flatmem would make things much worse.
> > 
> > Yes, that hole is not ideal.
> 
> What you might try is changing the section size to 32 MB and mapping
> the vmemmap region down to pages. That way, the vmemmap region should
> only take up
> - 512 KiB for the struct page array[] itself
> - 4 KiB for the page table that replaces the 2 MB block mapping
> 
> You could try the below and see if it makes any difference?
> 
> diff --git a/arch/arm64/include/asm/sparsemem.h
> b/arch/arm64/include/asm/sparsemem.h
> index 4b73463423c3..a008f4342532 100644
> --- a/arch/arm64/include/asm/sparsemem.h
> +++ b/arch/arm64/include/asm/sparsemem.h
> @@ -23,7 +23,7 @@
>   * entries could not be created for vmemmap mappings.
>   * 16K follows 4K for simplicity.
>   */
> -#define SECTION_SIZE_BITS 27
> +#define SECTION_SIZE_BITS 25
>  #endif /* CONFIG_ARM64_64K_PAGES */
> 
>  #endif
> diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> index 5b1946f1805c..d25560a53a67 100644
> --- a/arch/arm64/mm/mmu.c
> +++ b/arch/arm64/mm/mmu.c
> @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
> addr, unsigned long end,
>  }
>  #endif
> 
> -#if !ARM64_KERNEL_USES_PMD_MAPS
> +#if 1// !ARM64_KERNEL_USES_PMD_MAPS
>  int __meminit vmemmap_populate(unsigned long start, unsigned long
> end, int node,
>                 struct vmem_altmap *altmap)
>  {

That was a really good idea, now I have:
 Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma-reserved)

Reserved dropped from 14+MB to 7+MB :)

Should I look into something particular before testing this in a bigger scale?

 Jocke
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Ard Biesheuvel]

On Tue, 7 Jun 2022 at 09:39, Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
>
> On Tue, 2022-06-07 at 09:15 +0200, Ard Biesheuvel wrote:
..
> >
> > What you might try is changing the section size to 32 MB and mapping
> > the vmemmap region down to pages. That way, the vmemmap region should
> > only take up
> > - 512 KiB for the struct page array[] itself
> > - 4 KiB for the page table that replaces the 2 MB block mapping
> >
> > You could try the below and see if it makes any difference?
> >
> > diff --git a/arch/arm64/include/asm/sparsemem.h
> > b/arch/arm64/include/asm/sparsemem.h
> > index 4b73463423c3..a008f4342532 100644
> > --- a/arch/arm64/include/asm/sparsemem.h
> > +++ b/arch/arm64/include/asm/sparsemem.h
> > @@ -23,7 +23,7 @@
> >   * entries could not be created for vmemmap mappings.
> >   * 16K follows 4K for simplicity.
> >   */
> > -#define SECTION_SIZE_BITS 27
> > +#define SECTION_SIZE_BITS 25
> >  #endif /* CONFIG_ARM64_64K_PAGES */
> >
> >  #endif
> > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > index 5b1946f1805c..d25560a53a67 100644
> > --- a/arch/arm64/mm/mmu.c
> > +++ b/arch/arm64/mm/mmu.c
> > @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
> > addr, unsigned long end,
> >  }
> >  #endif
> >
> > -#if !ARM64_KERNEL_USES_PMD_MAPS
> > +#if 1// !ARM64_KERNEL_USES_PMD_MAPS
> >  int __meminit vmemmap_populate(unsigned long start, unsigned long
> > end, int node,
> >                 struct vmem_altmap *altmap)
> >  {
>
> That was a really good idea, now I have:
>  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma-reserved)
>
> Reserved dropped from 14+MB to 7+MB :)
>

Glad to hear that.

> Should I look into something particular before testing this in a bigger scale?
>

A bit of stress testing would be useful, although I wouldn't be able
to recommend anything that will be able to run with that little
memory.

Just be aware that this is not a change that is likely to be accepted
by the arm64 maintainers - it is something you will have to carry in
your downstream.

Out of curiosity, what type of system is this? What user space are you using?

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Joakim Tjernlund]

On Tue, 2022-06-07 at 09:48 +0200, Ard Biesheuvel wrote:
> On Tue, 7 Jun 2022 at 09:39, Joakim Tjernlund
> <Joakim.Tjernlund@infinera.com> wrote:
> > 
> > On Tue, 2022-06-07 at 09:15 +0200, Ard Biesheuvel wrote:
> ..
> > > 
> > > What you might try is changing the section size to 32 MB and mapping
> > > the vmemmap region down to pages. That way, the vmemmap region should
> > > only take up
> > > - 512 KiB for the struct page array[] itself
> > > - 4 KiB for the page table that replaces the 2 MB block mapping
> > > 
> > > You could try the below and see if it makes any difference?
> > > 
> > > diff --git a/arch/arm64/include/asm/sparsemem.h
> > > b/arch/arm64/include/asm/sparsemem.h
> > > index 4b73463423c3..a008f4342532 100644
> > > --- a/arch/arm64/include/asm/sparsemem.h
> > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > @@ -23,7 +23,7 @@
> > >   * entries could not be created for vmemmap mappings.
> > >   * 16K follows 4K for simplicity.
> > >   */
> > > -#define SECTION_SIZE_BITS 27
> > > +#define SECTION_SIZE_BITS 25
> > >  #endif /* CONFIG_ARM64_64K_PAGES */
> > > 
> > >  #endif
> > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > > index 5b1946f1805c..d25560a53a67 100644
> > > --- a/arch/arm64/mm/mmu.c
> > > +++ b/arch/arm64/mm/mmu.c
> > > @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
> > > addr, unsigned long end,
> > >  }
> > >  #endif
> > > 
> > > -#if !ARM64_KERNEL_USES_PMD_MAPS
> > > +#if 1// !ARM64_KERNEL_USES_PMD_MAPS
> > >  int __meminit vmemmap_populate(unsigned long start, unsigned long
> > > end, int node,
> > >                 struct vmem_altmap *altmap)
> > >  {
> > 
> > That was a really good idea, now I have:
> >  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma-reserved)
> > 
> > Reserved dropped from 14+MB to 7+MB :)
> > 
> 
> Glad to hear that.
> 
> > Should I look into something particular before testing this in a bigger scale?
> > 
> 
> A bit of stress testing would be useful, although I wouldn't be able
> to recommend anything that will be able to run with that little
> memory.
> 
> Just be aware that this is not a change that is likely to be accepted
> by the arm64 maintainers - it is something you will have to carry in
> your downstream.

In its current form, agreed.
If it could be generalized maybe, as is today aarch64 mandates lots of RAM
but your hack shows promise so maybe something will come of it.

> 
> Out of curiosity, what type of system is this? What user space are you using?
This is a very fancy custom ASIC, user space is a custom Yocto in 32 bits musl, mainly busybox
for utilities.

 Jocke 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Ard Biesheuvel]

On Tue, 7 Jun 2022 at 10:45, Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
>
> On Tue, 2022-06-07 at 09:48 +0200, Ard Biesheuvel wrote:
> > On Tue, 7 Jun 2022 at 09:39, Joakim Tjernlund
> > <Joakim.Tjernlund@infinera.com> wrote:
> > >
> > > On Tue, 2022-06-07 at 09:15 +0200, Ard Biesheuvel wrote:
> > ..
> > > >
> > > > What you might try is changing the section size to 32 MB and mapping
> > > > the vmemmap region down to pages. That way, the vmemmap region should
> > > > only take up
> > > > - 512 KiB for the struct page array[] itself
> > > > - 4 KiB for the page table that replaces the 2 MB block mapping
> > > >
> > > > You could try the below and see if it makes any difference?
> > > >
> > > > diff --git a/arch/arm64/include/asm/sparsemem.h
> > > > b/arch/arm64/include/asm/sparsemem.h
> > > > index 4b73463423c3..a008f4342532 100644
> > > > --- a/arch/arm64/include/asm/sparsemem.h
> > > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > > @@ -23,7 +23,7 @@
> > > >   * entries could not be created for vmemmap mappings.
> > > >   * 16K follows 4K for simplicity.
> > > >   */
> > > > -#define SECTION_SIZE_BITS 27
> > > > +#define SECTION_SIZE_BITS 25
> > > >  #endif /* CONFIG_ARM64_64K_PAGES */
> > > >
> > > >  #endif
> > > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > > > index 5b1946f1805c..d25560a53a67 100644
> > > > --- a/arch/arm64/mm/mmu.c
> > > > +++ b/arch/arm64/mm/mmu.c
> > > > @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
> > > > addr, unsigned long end,
> > > >  }
> > > >  #endif
> > > >
> > > > -#if !ARM64_KERNEL_USES_PMD_MAPS
> > > > +#if 1// !ARM64_KERNEL_USES_PMD_MAPS
> > > >  int __meminit vmemmap_populate(unsigned long start, unsigned long
> > > > end, int node,
> > > >                 struct vmem_altmap *altmap)
> > > >  {
> > >
> > > That was a really good idea, now I have:
> > >  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma-reserved)
> > >
> > > Reserved dropped from 14+MB to 7+MB :)
> > >
> >
> > Glad to hear that.
> >
> > > Should I look into something particular before testing this in a bigger scale?
> > >
> >
> > A bit of stress testing would be useful, although I wouldn't be able
> > to recommend anything that will be able to run with that little
> > memory.
> >
> > Just be aware that this is not a change that is likely to be accepted
> > by the arm64 maintainers - it is something you will have to carry in
> > your downstream.
>
> In its current form, agreed.
> If it could be generalized maybe, as is today aarch64 mandates lots of RAM
> but your hack shows promise so maybe something will come of it.
>

I'm not sure - using PMD maps actually uses 4 KiB less for every 128
MiB of memory, and it is the rounding up to multiples of 128 MiB that
is biting you. IOW, on a system that has a multiple of 128 MiB which
is laid out straight-forwardly, this change results in more memory to
be used (the number of sections also increases, which means more
per-section bookkeeping)

As Arnd explained in the other thread, 128 MiB of DRAM is already
considered less than a reasonable minimum even on 32-bit architectures
(if I am paraphrasing that correctly), and so getting anyone to care
about running the arm64 kernel on a device with 36 MB of DRAM is going
to be very difficult.

> >
> > Out of curiosity, what type of system is this? What user space are you using?
> This is a very fancy custom ASIC, user space is a custom Yocto in 32 bits musl, mainly busybox
> for utilities.
>

OK, so even if I am not a fan of this practice [0], running a 32-bit
kernel would likely be an option for you as well. Have you considered
this?

[0] https://security.googleblog.com/2022/02/mitigating-kernel-risks-on-32-bit-arm.html

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Joakim Tjernlund]

On Tue, 2022-06-07 at 11:02 +0200, Ard Biesheuvel wrote:
> On Tue, 7 Jun 2022 at 10:45, Joakim Tjernlund
> <Joakim.Tjernlund@infinera.com> wrote:
> > 
> > On Tue, 2022-06-07 at 09:48 +0200, Ard Biesheuvel wrote:
> > > On Tue, 7 Jun 2022 at 09:39, Joakim Tjernlund
> > > <Joakim.Tjernlund@infinera.com> wrote:
> > > > 
> > > > On Tue, 2022-06-07 at 09:15 +0200, Ard Biesheuvel wrote:
> > > ..
> > > > > 
> > > > > What you might try is changing the section size to 32 MB and mapping
> > > > > the vmemmap region down to pages. That way, the vmemmap region should
> > > > > only take up
> > > > > - 512 KiB for the struct page array[] itself
> > > > > - 4 KiB for the page table that replaces the 2 MB block mapping
> > > > > 
> > > > > You could try the below and see if it makes any difference?
> > > > > 
> > > > > diff --git a/arch/arm64/include/asm/sparsemem.h
> > > > > b/arch/arm64/include/asm/sparsemem.h
> > > > > index 4b73463423c3..a008f4342532 100644
> > > > > --- a/arch/arm64/include/asm/sparsemem.h
> > > > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > > > @@ -23,7 +23,7 @@
> > > > >   * entries could not be created for vmemmap mappings.
> > > > >   * 16K follows 4K for simplicity.
> > > > >   */
> > > > > -#define SECTION_SIZE_BITS 27
> > > > > +#define SECTION_SIZE_BITS 25
> > > > >  #endif /* CONFIG_ARM64_64K_PAGES */
> > > > > 
> > > > >  #endif
> > > > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > > > > index 5b1946f1805c..d25560a53a67 100644
> > > > > --- a/arch/arm64/mm/mmu.c
> > > > > +++ b/arch/arm64/mm/mmu.c
> > > > > @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
> > > > > addr, unsigned long end,
> > > > >  }
> > > > >  #endif
> > > > > 
> > > > > -#if !ARM64_KERNEL_USES_PMD_MAPS
> > > > > +#if 1// !ARM64_KERNEL_USES_PMD_MAPS
> > > > >  int __meminit vmemmap_populate(unsigned long start, unsigned long
> > > > > end, int node,
> > > > >                 struct vmem_altmap *altmap)
> > > > >  {
> > > > 
> > > > That was a really good idea, now I have:
> > > >  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma-reserved)
> > > > 
> > > > Reserved dropped from 14+MB to 7+MB :)
> > > > 
> > > 
> > > Glad to hear that.
> > > 
> > > > Should I look into something particular before testing this in a bigger scale?
> > > > 
> > > 
> > > A bit of stress testing would be useful, although I wouldn't be able
> > > to recommend anything that will be able to run with that little
> > > memory.
> > > 
> > > Just be aware that this is not a change that is likely to be accepted
> > > by the arm64 maintainers - it is something you will have to carry in
> > > your downstream.
> > 
> > In its current form, agreed.
> > If it could be generalized maybe, as is today aarch64 mandates lots of RAM
> > but your hack shows promise so maybe something will come of it.
> > 
> 
> I'm not sure - using PMD maps actually uses 4 KiB less for every 128
> MiB of memory, and it is the rounding up to multiples of 128 MiB that
> is biting you. IOW, on a system that has a multiple of 128 MiB which
> is laid out straight-forwardly, this change results in more memory to
> be used (the number of sections also increases, which means more
> per-section bookkeeping)

hmm, I don't quite understand this but seems to me that there something more
going on as I recovered about 7MB with your hack.
In any case, I figure exact memory model should be a build option and one
will have to choose what fits your system best. 

> 
> As Arnd explained in the other thread, 128 MiB of DRAM is already
> considered less than a reasonable minimum even on 32-bit architectures
> (if I am paraphrasing that correctly), and so getting anyone to care
> about running the arm64 kernel on a device with 36 MB of DRAM is going
> to be very difficult.

That is a lot of RAM in the embedded space.

> 
> > > 
> > > Out of curiosity, what type of system is this? What user space are you using?
> > This is a very fancy custom ASIC, user space is a custom Yocto in 32 bits musl, mainly busybox
> > for utilities.
> > 
> 
> OK, so even if I am not a fan of this practice [0], running a 32-bit
> kernel would likely be an option for you as well. Have you considered
> this?
> 
> [0] https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsecurity.googleblog.com%2F2022%2F02%2Fmitigating-kernel-risks-on-32-bit-arm.html&amp;data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7Ce1561709c32a446e957708da48647e8b%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637901893702410819%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&amp;sdata=Mwus3JktdxLJ2znXumuVGI8k5hNTkniMgbr7CKwjvrE%3D&amp;reserved=0

Briefly considered 32 bits kernel but as I understand it, it is even more unsupported so I have opted
for 64 bits kernel for now. I may have to revisit that though. ARM is still new to me.
Thanks for the link, good read.

     Jocke
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Ard Biesheuvel]

On Tue, 7 Jun 2022 at 11:29, Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
>
> On Tue, 2022-06-07 at 11:02 +0200, Ard Biesheuvel wrote:
> > On Tue, 7 Jun 2022 at 10:45, Joakim Tjernlund
> > <Joakim.Tjernlund@infinera.com> wrote:
> > >
> > > On Tue, 2022-06-07 at 09:48 +0200, Ard Biesheuvel wrote:
> > > > On Tue, 7 Jun 2022 at 09:39, Joakim Tjernlund
> > > > <Joakim.Tjernlund@infinera.com> wrote:
> > > > >
> > > > > On Tue, 2022-06-07 at 09:15 +0200, Ard Biesheuvel wrote:
> > > > ..
> > > > > >
> > > > > > What you might try is changing the section size to 32 MB and mapping
> > > > > > the vmemmap region down to pages. That way, the vmemmap region should
> > > > > > only take up
> > > > > > - 512 KiB for the struct page array[] itself
> > > > > > - 4 KiB for the page table that replaces the 2 MB block mapping
> > > > > >
> > > > > > You could try the below and see if it makes any difference?
> > > > > >
> > > > > > diff --git a/arch/arm64/include/asm/sparsemem.h
> > > > > > b/arch/arm64/include/asm/sparsemem.h
> > > > > > index 4b73463423c3..a008f4342532 100644
> > > > > > --- a/arch/arm64/include/asm/sparsemem.h
> > > > > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > > > > @@ -23,7 +23,7 @@
> > > > > >   * entries could not be created for vmemmap mappings.
> > > > > >   * 16K follows 4K for simplicity.
> > > > > >   */
> > > > > > -#define SECTION_SIZE_BITS 27
> > > > > > +#define SECTION_SIZE_BITS 25
> > > > > >  #endif /* CONFIG_ARM64_64K_PAGES */
> > > > > >
> > > > > >  #endif
> > > > > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > > > > > index 5b1946f1805c..d25560a53a67 100644
> > > > > > --- a/arch/arm64/mm/mmu.c
> > > > > > +++ b/arch/arm64/mm/mmu.c
> > > > > > @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
> > > > > > addr, unsigned long end,
> > > > > >  }
> > > > > >  #endif
> > > > > >
> > > > > > -#if !ARM64_KERNEL_USES_PMD_MAPS
> > > > > > +#if 1// !ARM64_KERNEL_USES_PMD_MAPS
> > > > > >  int __meminit vmemmap_populate(unsigned long start, unsigned long
> > > > > > end, int node,
> > > > > >                 struct vmem_altmap *altmap)
> > > > > >  {
> > > > >
> > > > > That was a really good idea, now I have:
> > > > >  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma-reserved)
> > > > >
> > > > > Reserved dropped from 14+MB to 7+MB :)
> > > > >
> > > >
> > > > Glad to hear that.
> > > >
> > > > > Should I look into something particular before testing this in a bigger scale?
> > > > >
> > > >
> > > > A bit of stress testing would be useful, although I wouldn't be able
> > > > to recommend anything that will be able to run with that little
> > > > memory.
> > > >
> > > > Just be aware that this is not a change that is likely to be accepted
> > > > by the arm64 maintainers - it is something you will have to carry in
> > > > your downstream.
> > >
> > > In its current form, agreed.
> > > If it could be generalized maybe, as is today aarch64 mandates lots of RAM
> > > but your hack shows promise so maybe something will come of it.
> > >
> >
> > I'm not sure - using PMD maps actually uses 4 KiB less for every 128
> > MiB of memory, and it is the rounding up to multiples of 128 MiB that
> > is biting you. IOW, on a system that has a multiple of 128 MiB which
> > is laid out straight-forwardly, this change results in more memory to
> > be used (the number of sections also increases, which means more
> > per-section bookkeeping)
>
> hmm, I don't quite understand this but seems to me that there something more
> going on as I recovered about 7MB with your hack.

Yeah, I noticed that. I guess the rounding up to 128 MiB results in
some overhead somewhere else as well.

> In any case, I figure exact memory model should be a build option and one
> will have to choose what fits your system best.
>

We used removed support for everything except SPARSEMEM_VMEMMAP, and I
don't think it will be coming back. Note that half of the overhead you
are observing is a result of the peculiar physical placement of DRAM,
and the current arrangement is fine except for extreme cases like
yours.

But of course, feel free to propose a change along these lines: I am
not a arm64 maintainer, and so it is not up to me ultimately.

> >
> > As Arnd explained in the other thread, 128 MiB of DRAM is already
> > considered less than a reasonable minimum even on 32-bit architectures
> > (if I am paraphrasing that correctly), and so getting anyone to care
> > about running the arm64 kernel on a device with 36 MB of DRAM is going
> > to be very difficult.
>
> That is a lot of RAM in the embedded space.
>

Of course. But this also means that you have no need for a kernel that
uses 64-bit pointers, each of which wastes 4 bytes of memory too.

> >
> > > >
> > > > Out of curiosity, what type of system is this? What user space are you using?
> > > This is a very fancy custom ASIC, user space is a custom Yocto in 32 bits musl, mainly busybox
> > > for utilities.
> > >
> >
> > OK, so even if I am not a fan of this practice [0], running a 32-bit
> > kernel would likely be an option for you as well. Have you considered
> > this?
> >
..
>
> Briefly considered 32 bits kernel but as I understand it, it is even more unsupported so I have opted
> for 64 bits kernel for now. I may have to revisit that though. ARM is still new to me.

It is a bad idea for many reasons, but you may not have a choice.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Ard Biesheuvel]

On Tue, 7 Jun 2022 at 11:38, Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Tue, 7 Jun 2022 at 11:29, Joakim Tjernlund
> <Joakim.Tjernlund@infinera.com> wrote:
> >
> > On Tue, 2022-06-07 at 11:02 +0200, Ard Biesheuvel wrote:
> > > On Tue, 7 Jun 2022 at 10:45, Joakim Tjernlund
> > > <Joakim.Tjernlund@infinera.com> wrote:
> > > >
> > > > On Tue, 2022-06-07 at 09:48 +0200, Ard Biesheuvel wrote:
> > > > > On Tue, 7 Jun 2022 at 09:39, Joakim Tjernlund
> > > > > <Joakim.Tjernlund@infinera.com> wrote:
> > > > > >
> > > > > > On Tue, 2022-06-07 at 09:15 +0200, Ard Biesheuvel wrote:
> > > > > ..
> > > > > > >
> > > > > > > What you might try is changing the section size to 32 MB and mapping
> > > > > > > the vmemmap region down to pages. That way, the vmemmap region should
> > > > > > > only take up
> > > > > > > - 512 KiB for the struct page array[] itself
> > > > > > > - 4 KiB for the page table that replaces the 2 MB block mapping
> > > > > > >
> > > > > > > You could try the below and see if it makes any difference?
> > > > > > >
> > > > > > > diff --git a/arch/arm64/include/asm/sparsemem.h
> > > > > > > b/arch/arm64/include/asm/sparsemem.h
> > > > > > > index 4b73463423c3..a008f4342532 100644
> > > > > > > --- a/arch/arm64/include/asm/sparsemem.h
> > > > > > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > > > > > @@ -23,7 +23,7 @@
> > > > > > >   * entries could not be created for vmemmap mappings.
> > > > > > >   * 16K follows 4K for simplicity.
> > > > > > >   */
> > > > > > > -#define SECTION_SIZE_BITS 27
> > > > > > > +#define SECTION_SIZE_BITS 25
> > > > > > >  #endif /* CONFIG_ARM64_64K_PAGES */
> > > > > > >
> > > > > > >  #endif
> > > > > > > diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
> > > > > > > index 5b1946f1805c..d25560a53a67 100644
> > > > > > > --- a/arch/arm64/mm/mmu.c
> > > > > > > +++ b/arch/arm64/mm/mmu.c
> > > > > > > @@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
> > > > > > > addr, unsigned long end,
> > > > > > >  }
> > > > > > >  #endif
> > > > > > >
> > > > > > > -#if !ARM64_KERNEL_USES_PMD_MAPS
> > > > > > > +#if 1// !ARM64_KERNEL_USES_PMD_MAPS
> > > > > > >  int __meminit vmemmap_populate(unsigned long start, unsigned long
> > > > > > > end, int node,
> > > > > > >                 struct vmem_altmap *altmap)
> > > > > > >  {
> > > > > >
> > > > > > That was a really good idea, now I have:
> > > > > >  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma-reserved)
> > > > > >
> > > > > > Reserved dropped from 14+MB to 7+MB :)
> > > > > >
> > > > >
> > > > > Glad to hear that.
> > > > >
> > > > > > Should I look into something particular before testing this in a bigger scale?
> > > > > >
> > > > >
> > > > > A bit of stress testing would be useful, although I wouldn't be able
> > > > > to recommend anything that will be able to run with that little
> > > > > memory.
> > > > >
> > > > > Just be aware that this is not a change that is likely to be accepted
> > > > > by the arm64 maintainers - it is something you will have to carry in
> > > > > your downstream.
> > > >
> > > > In its current form, agreed.
> > > > If it could be generalized maybe, as is today aarch64 mandates lots of RAM
> > > > but your hack shows promise so maybe something will come of it.
> > > >
> > >
> > > I'm not sure - using PMD maps actually uses 4 KiB less for every 128
> > > MiB of memory, and it is the rounding up to multiples of 128 MiB that
> > > is biting you. IOW, on a system that has a multiple of 128 MiB which
> > > is laid out straight-forwardly, this change results in more memory to
> > > be used (the number of sections also increases, which means more
> > > per-section bookkeeping)
> >
> > hmm, I don't quite understand this but seems to me that there something more
> > going on as I recovered about 7MB with your hack.
>
> Yeah, I noticed that. I guess the rounding up to 128 MiB results in
> some overhead somewhere else as well.
>
> > In any case, I figure exact memory model should be a build option and one
> > will have to choose what fits your system best.
> >
>
> We used

'just' - no idea how I manage to typo that ...

> removed support for everything except SPARSEMEM_VMEMMAP,

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Arnd Bergmann]

On Tue, Jun 7, 2022 at 8:36 AM Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
> On Mon, 2022-06-06 at 23:50 +0200, Ard Biesheuvel wrote:
> > So the problem is that you only have 36 MB of DRAM, with a large hole
> > in the middle. Sparsemem was actually designed for that (hence the
> > name), and flatmem would make things much worse.
>
> Yes, that hole is not ideal.

Is there a reason for this hole? Are both sections actually required to be
normal RAM, rather than something else? If the 4MB chunk is actually
some fast SRAM, then having a special-purpose device driver or an XIP
capable file system for it would let you some critical data on it and not
actually require page pointers.

        Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Arnd Bergmann]

On Tue, Jun 7, 2022 at 11:38 AM Ard Biesheuvel <ardb@kernel.org> wrote:
> On Tue, 7 Jun 2022 at 11:29, Joakim Tjernlund <Joakim.Tjernlund@infinera.com> wrote:
> > >
> > > OK, so even if I am not a fan of this practice [0], running a 32-bit
> > > kernel would likely be an option for you as well. Have you considered
> > > this?
> > >
> ..
> >
> > Briefly considered 32 bits kernel but as I understand it, it is even more unsupported so I have opted
> > for 64 bits kernel for now. I may have to revisit that though. ARM is still new to me.
>
> It is a bad idea for many reasons, but you may not have a choice.

They are clearly both bad on this hardware ;-)

If we can improve support for a 36MB configuration on arm64, that
would likely also help
those users with actual DDR3 ram (128MB to 512MB typically) that you see more
commonly, and that's certainly a good thing.

Another alternative to using a 32-bit kernel would be to not use Linux
at all. There are
enough RTOS variants in the world that have a growing feature set and that may
be sufficient for Joakim's workload but use a lot less RAM.
If there is only one application running, there may not be a no need
for the complexity
in our file systems, memory protection or network stack either.

        Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Ard Biesheuvel]

On Tue, 7 Jun 2022 at 11:53, Arnd Bergmann <arnd@arndb.de> wrote:
>
> On Tue, Jun 7, 2022 at 11:38 AM Ard Biesheuvel <ardb@kernel.org> wrote:
> > On Tue, 7 Jun 2022 at 11:29, Joakim Tjernlund <Joakim.Tjernlund@infinera.com> wrote:
> > > >
> > > > OK, so even if I am not a fan of this practice [0], running a 32-bit
> > > > kernel would likely be an option for you as well. Have you considered
> > > > this?
> > > >
> > ..
> > >
> > > Briefly considered 32 bits kernel but as I understand it, it is even more unsupported so I have opted
> > > for 64 bits kernel for now. I may have to revisit that though. ARM is still new to me.
> >
> > It is a bad idea for many reasons, but you may not have a choice.
>
> They are clearly both bad on this hardware ;-)
>
> If we can improve support for a 36MB configuration on arm64, that
> would likely also help
> those users with actual DDR3 ram (128MB to 512MB typically) that you see more
> commonly, and that's certainly a good thing.
>

I agree with that in principle, but as I explained, playing with the
section sizes and vmemmap granularity does not help on systems where
the DRAM is a 128 MiB aligned multiple of 128 MiB; if anything, we'll
use slightly more memory.

> Another alternative to using a 32-bit kernel would be to not use Linux
> at all. There are
> enough RTOS variants in the world that have a growing feature set and that may
> be sufficient for Joakim's workload but use a lot less RAM.
> If there is only one application running, there may not be a no need
> for the complexity
> in our file systems, memory protection or network stack either.
>

Nicolas Pitre did some work a few years ago to drastically reduce the
kernel's memory footprint, but I don't think there was a lot of
support for the more intrusive changes that he suggested. Linux is
just not as suited for true embedded use anymore, and the arm64 kernel
in particular was just never intended for that.

So yes to using less memory in the general case, but intrusive changes
such as the one I proposed are unlikely to be adopted (and the arm64
kernel, unlike the 32-bit one, tries really hard to only have a single
image and a single defconfig)

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Arnd Bergmann]

On Tue, Jun 7, 2022 at 12:03 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> On Tue, 7 Jun 2022 at 11:53, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Tue, Jun 7, 2022 at 11:38 AM Ard Biesheuvel <ardb@kernel.org> wrote:
> > > On Tue, 7 Jun 2022 at 11:29, Joakim Tjernlund <Joakim.Tjernlund@infinera.com> wrote:
> > > > >
> > > > > OK, so even if I am not a fan of this practice [0], running a 32-bit
> > > > > kernel would likely be an option for you as well. Have you considered
> > > > > this?
> > > > >
> > > ..
> > > >
> > > > Briefly considered 32 bits kernel but as I understand it, it is even more unsupported so I have opted
> > > > for 64 bits kernel for now. I may have to revisit that though. ARM is still new to me.
> > >
> > > It is a bad idea for many reasons, but you may not have a choice.
> >
> > They are clearly both bad on this hardware ;-)
> >
> > If we can improve support for a 36MB configuration on arm64, that
> > would likely also help
> > those users with actual DDR3 ram (128MB to 512MB typically) that you see more
> > commonly, and that's certainly a good thing.
> >
>
> I agree with that in principle, but as I explained, playing with the
> section sizes and vmemmap granularity does not help on systems where
> the DRAM is a 128 MiB aligned multiple of 128 MiB; if anything, we'll
> use slightly more memory.

Right, that part would not help, but maybe there are some other easy
optimizations that would.

> > Another alternative to using a 32-bit kernel would be to not use Linux
> > at all. There are
> > enough RTOS variants in the world that have a growing feature set and that may
> > be sufficient for Joakim's workload but use a lot less RAM.
> > If there is only one application running, there may not be a no need
> > for the complexity
> > in our file systems, memory protection or network stack either.
> >
>
> Nicolas Pitre did some work a few years ago to drastically reduce the
> kernel's memory footprint, but I don't think there was a lot of
> support for the more intrusive changes that he suggested. Linux is
> just not as suited for true embedded use anymore, and the arm64 kernel
> in particular was just never intended for that.

I think what happened with Nico's work is that the particular use case
for the MMU-less or DRAM-less systems just disappeared in 2017 when
Zephyr, FreeRTOS, mbed OS and others all received a lot of funding
and took over that particular market from Linux.

On the other hand, there is clearly still a market for small embedded
systems running Linux, it's just that "small" is no longer as small
as it used to be, with DDR3 (128MB to 512MB) and soon LPDDR4
(512MB and up) DRAM now as the dominant technology for low-end
devices, while DDR2 and earlier are no longer economically viable.

I think about half the platforms in arch/arm64 fit into this embedded
space (four or fewer A35/A53/A55 cores, under 1GB of RAM) already,
and I expect this to become more important over time as 32-bit cores
get less common in new chips.

> So yes to using less memory in the general case, but intrusive changes
> such as the one I proposed are unlikely to be adopted (and the arm64
> kernel, unlike the 32-bit one, tries really hard to only have a single
> image and a single defconfig)

I don't see a configurable section size (possibly under CONFIG_EXPERT)
as too intrusive. The arm64 defconfig clearly is clearly required to work on
any kind of hardware, but there is enough precedence for Kconfig options
that make machine specific trade-offs (e.g. page size and VA size), or that
completely disable support for certain SoC platforms.

       Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Joakim Tjernlund]

On Tue, 2022-06-07 at 12:56 +0200, Arnd Bergmann wrote:
> On Tue, Jun 7, 2022 at 12:03 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> > On Tue, 7 Jun 2022 at 11:53, Arnd Bergmann <arnd@arndb.de> wrote:
> > > On Tue, Jun 7, 2022 at 11:38 AM Ard Biesheuvel <ardb@kernel.org> wrote:
> > > > On Tue, 7 Jun 2022 at 11:29, Joakim Tjernlund <Joakim.Tjernlund@infinera.com> wrote:
> > > > > > 
> 

SNIP

> > So yes to using less memory in the general case, but intrusive changes
> > such as the one I proposed are unlikely to be adopted (and the arm64
> > kernel, unlike the 32-bit one, tries really hard to only have a single
> > image and a single defconfig)
> 
> I don't see a configurable section size (possibly under CONFIG_EXPERT)
> as too intrusive. The arm64 defconfig clearly is clearly required to work on
> any kind of hardware, but there is enough precedence for Kconfig options
> that make machine specific trade-offs (e.g. page size and VA size), or that
> completely disable support for certain SoC platforms.
> 
>        Arnd

Yes, that would be very appreciated.

 Jocke 
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: SPARSEMEM memory needs?

[Ard Biesheuvel]

On Tue, 7 Jun 2022 at 13:44, Joakim Tjernlund
<Joakim.Tjernlund@infinera.com> wrote:
>
> On Tue, 2022-06-07 at 12:56 +0200, Arnd Bergmann wrote:
> > On Tue, Jun 7, 2022 at 12:03 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> > > On Tue, 7 Jun 2022 at 11:53, Arnd Bergmann <arnd@arndb.de> wrote:
> > > > On Tue, Jun 7, 2022 at 11:38 AM Ard Biesheuvel <ardb@kernel.org> wrote:
> > > > > On Tue, 7 Jun 2022 at 11:29, Joakim Tjernlund <Joakim.Tjernlund@infinera.com> wrote:
> > > > > > >
> >
>
> SNIP
>
> > > So yes to using less memory in the general case, but intrusive changes
> > > such as the one I proposed are unlikely to be adopted (and the arm64
> > > kernel, unlike the 32-bit one, tries really hard to only have a single
> > > image and a single defconfig)
> >
> > I don't see a configurable section size (possibly under CONFIG_EXPERT)
> > as too intrusive. The arm64 defconfig clearly is clearly required to work on
> > any kind of hardware, but there is enough precedence for Kconfig options
> > that make machine specific trade-offs (e.g. page size and VA size), or that
> > completely disable support for certain SoC platforms.
> >

Sure. But if we introduce such a knob, it will now be the job of
developers working on the MM code to ensure that any changes don't
break this niche configuration.

Perhaps we could put it under CONFIG_EXPERT and make it a best-effort thing?

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel