Finding kernel RAM consumers ?

Finding kernel RAM consumers ?

[Joakim Tjernlund]

We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K:
Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved)

I want to track down were most of this RAM is consumed so I can trim away some MBs
but I am having a hard time finding may way.
Is there some tool/kernel config that can help me with that?

 Jocke

Re: Finding kernel RAM consumers ?

[Matthew Wilcox]

On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote:
> We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K:
> Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved)
> 
> I want to track down were most of this RAM is consumed so I can trim away some MBs
> but I am having a hard time finding may way.
> Is there some tool/kernel config that can help me with that?

You may find this series of articles interesting:

https://lwn.net/Articles/741494/
https://lwn.net/Articles/744507/
https://lwn.net/Articles/746780/
https://lwn.net/Articles/748198/

While they're a little old and they're targetting a much smaller system
than yours, they may give you some ideas of things you can try and tools
you can use.

Re: Finding kernel RAM consumers ?

[Joakim Tjernlund]

On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote:
> On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote:
> > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K:
> > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved)
> > 
> > I want to track down were most of this RAM is consumed so I can trim away some MBs
> > but I am having a hard time finding may way.
> > Is there some tool/kernel config that can help me with that?
> 
> You may find this series of articles interesting:
> 
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F741494%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zP2oazSeoVKiTrbPONqmQXEhzyIIL3pQdvsKfN8nOA0%3D&reserved=0
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F744507%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9kNmXDhuQ24tWlECGrYQOqaaTsc0sOO4TozcQBk5aYc%3D&reserved=0
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F746780%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZFu0q4tXX677JwkgBqK73M8WxDrhylw0XnuEw5Y8mA4%3D&reserved=0
> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F748198%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ws5Rlrhts80ndzr8aEv3xTu6EmCnND3L%2FmGXHLDaygw%3D&reserved=0
> 
> While they're a little old and they're targetting a much smaller system
> than yours, they may give you some ideas of things you can try and tools
> you can use.

Those are interesting, thanks.
In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data
is 5+MB but the total need is 14MB, 9MB is buffer and similar.

 Jocke

Re: Finding kernel RAM consumers ?

[Joakim Tjernlund]

On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote:
> On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote:
> > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote:
> > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K:
> > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved)
> > > 
> > > I want to track down were most of this RAM is consumed so I can trim away some MBs
> > > but I am having a hard time finding may way.
> > > Is there some tool/kernel config that can help me with that?
> > 
> > You may find this series of articles interesting:
> > 
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F741494%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zP2oazSeoVKiTrbPONqmQXEhzyIIL3pQdvsKfN8nOA0%3D&reserved=0
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F744507%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=9kNmXDhuQ24tWlECGrYQOqaaTsc0sOO4TozcQBk5aYc%3D&reserved=0
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F746780%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ZFu0q4tXX677JwkgBqK73M8WxDrhylw0XnuEw5Y8mA4%3D&reserved=0
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flwn.net%2FArticles%2F748198%2F&data=05%7C01%7CJoakim.Tjernlund%40infinera.com%7C509eaa2f688b4612945208da44d40494%7C285643de5f5b4b03a1530ae2dc8aaf77%7C1%7C0%7C637897974658412129%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ws5Rlrhts80ndzr8aEv3xTu6EmCnND3L%2FmGXHLDaygw%3D&reserved=0
> > 
> > While they're a little old and they're targetting a much smaller system
> > than yours, they may give you some ideas of things you can try and tools
> > you can use.
> 
> Those are interesting, thanks.
> In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data
> is 5+MB but the total need is 14MB, 9MB is buffer and similar.
> 
>  Jocke

Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory.
Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ? 

Re: Finding kernel RAM consumers ?

[Matthew Wilcox]

On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote:
> On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote:
> > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote:
> > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote:
> > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K:
> > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved)
> > > > 
> > > > I want to track down were most of this RAM is consumed so I can trim away some MBs
> > > > but I am having a hard time finding may way.
> > > > Is there some tool/kernel config that can help me with that?
> > 
> > Those are interesting, thanks.
> > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data
> > is 5+MB but the total need is 14MB, 9MB is buffer and similar.
> > 
> >  Jocke
> 
> Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory.
> Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ? 
That sounds like a question for the arm64 maintainers

Re: Finding kernel RAM consumers ?

[Matthew Wilcox]

On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote:
> On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote:
> > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote:
> > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote:
> > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K:
> > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved)
> > > > 
> > > > I want to track down were most of this RAM is consumed so I can trim away some MBs
> > > > but I am having a hard time finding may way.
> > > > Is there some tool/kernel config that can help me with that?
> > 
> > Those are interesting, thanks.
> > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data
> > is 5+MB but the total need is 14MB, 9MB is buffer and similar.
> > 
> >  Jocke
> 
> Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory.
> Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ? 
That sounds like a question for the arm64 maintainers

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Finding kernel RAM consumers ?

[Arnd Bergmann]

On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote:
> On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote:
> > On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote:
> > > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote:
> > > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote:
> > > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K:
> > > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved)
> > > > >
> > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs
> > > > > but I am having a hard time finding may way.
> > > > > Is there some tool/kernel config that can help me with that?
> > >
> > > Those are interesting, thanks.
> > > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data
> > > is 5+MB but the total need is 14MB, 9MB is buffer and similar.

I'm fairly sure you are in new territory here. The arm64 kernel is
just not really optimized
for small memory configurations. Even for 32-bit Arm, 32MB is
considered too small for
most workloads these days, though we do have some very limited support
for machines
with as little as 2MB of SRAM. The smallest arm64 machines that I can
think of being
supported in the kernel today have at least 256MB.

> > Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory.
> > Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ?
>
> That sounds like a question for the arm64 maintainers

From what I understand, the 'static' variant uses a static allocation
in .bss, so that
may end up consuming most of your RAM even if it works normally. If all memory
is contiguous, then using flatmem may be more appropriate. You may also be able
to save some memory for sparsemem by reducing the size of the physical address
space. CONFIG_ARM64_PA_BITS is normally '48', but reducing it to '40' may
help (provided you can get that to boot). Most of the smaller CPUs only support
a 40 bit physical address space, so having another option for this is
probably sensible.

I think this is a case of "patches welcome". Nobody has really needed
this so far, but
as even the smaller machines are slowly migrating from 32-bit to
64-bit cores, optimizing
this will get interesting for more developers. There are probably
other low-hanging
fruit that you can address after figuring out.

One observation I made is that modern kernels don't seem to deal as
well as older
ones with low-memory situations, so even if you manage to free up most of your
32MB, it might still not work reliably.

       Arnd

Re: Finding kernel RAM consumers ?

[Arnd Bergmann]

On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote:
> On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote:
> > On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote:
> > > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote:
> > > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote:
> > > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K:
> > > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved)
> > > > >
> > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs
> > > > > but I am having a hard time finding may way.
> > > > > Is there some tool/kernel config that can help me with that?
> > >
> > > Those are interesting, thanks.
> > > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data
> > > is 5+MB but the total need is 14MB, 9MB is buffer and similar.

I'm fairly sure you are in new territory here. The arm64 kernel is
just not really optimized
for small memory configurations. Even for 32-bit Arm, 32MB is
considered too small for
most workloads these days, though we do have some very limited support
for machines
with as little as 2MB of SRAM. The smallest arm64 machines that I can
think of being
supported in the kernel today have at least 256MB.

> > Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory.
> > Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ?
>
> That sounds like a question for the arm64 maintainers

From what I understand, the 'static' variant uses a static allocation
in .bss, so that
may end up consuming most of your RAM even if it works normally. If all memory
is contiguous, then using flatmem may be more appropriate. You may also be able
to save some memory for sparsemem by reducing the size of the physical address
space. CONFIG_ARM64_PA_BITS is normally '48', but reducing it to '40' may
help (provided you can get that to boot). Most of the smaller CPUs only support
a 40 bit physical address space, so having another option for this is
probably sensible.

I think this is a case of "patches welcome". Nobody has really needed
this so far, but
as even the smaller machines are slowly migrating from 32-bit to
64-bit cores, optimizing
this will get interesting for more developers. There are probably
other low-hanging
fruit that you can address after figuring out.

One observation I made is that modern kernels don't seem to deal as
well as older
ones with low-memory situations, so even if you manage to free up most of your
32MB, it might still not work reliably.

       Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Finding kernel RAM consumers ?

[Alexander Dahl]

Hei hei,

Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann:
> On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote:
> > On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote:
> > > On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote:
> > > > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote:
> > > > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote:
> > > > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K:
> > > > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved)
> > > > > >
> > > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs
> > > > > > but I am having a hard time finding may way.
> > > > > > Is there some tool/kernel config that can help me with that?
> > > >
> > > > Those are interesting, thanks.
> > > > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data
> > > > is 5+MB but the total need is 14MB, 9MB is buffer and similar.
> 
> I'm fairly sure you are in new territory here. The arm64 kernel is
> just not really optimized
> for small memory configurations. Even for 32-bit Arm, 32MB is
> considered too small for
> most workloads these days, though we do have some very limited support
> for machines
> with as little as 2MB of SRAM. The smallest arm64 machines that I can
> think of being
> supported in the kernel today have at least 256MB.
> 
> > > Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory.
> > > Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ?
> >
> > That sounds like a question for the arm64 maintainers
> 
> From what I understand, the 'static' variant uses a static allocation
> in .bss, so that
> may end up consuming most of your RAM even if it works normally. If all memory
> is contiguous, then using flatmem may be more appropriate. You may also be able
> to save some memory for sparsemem by reducing the size of the physical address
> space. CONFIG_ARM64_PA_BITS is normally '48', but reducing it to '40' may
> help (provided you can get that to boot). Most of the smaller CPUs only support
> a 40 bit physical address space, so having another option for this is
> probably sensible.
> 
> I think this is a case of "patches welcome". Nobody has really needed
> this so far, but
> as even the smaller machines are slowly migrating from 32-bit to
> 64-bit cores, optimizing
> this will get interesting for more developers. There are probably
> other low-hanging
> fruit that you can address after figuring out.

The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with
64 MiB or 128 MiB, and given the latter is a new SoC announced only
two or three years ago, requiring at least 256 MiB would be at best
unfortunate.  Given those SoCs are used in industrial applications
with very long support times, I think 32bit ARM will stay for years,
even with new products.

> One observation I made is that modern kernels don't seem to deal as
> well as older
> ones with low-memory situations, so even if you manage to free up most of your
> 32MB, it might still not work reliably.

Since we are using the SAMA5D27C-D5M in production, I would also be
interested in support for running 32 bit ARM with recent kernels on
systems with 64 MiB or even 32 MiB of memory.  If there are specific
things to test, you can let me know.

Greets
Alex

Re: Finding kernel RAM consumers ?

[Alexander Dahl]

Hei hei,

Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann:
> On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote:
> > On Fri, Jun 03, 2022 at 05:26:33PM +0000, Joakim Tjernlund wrote:
> > > On Fri, 2022-06-03 at 08:49 +0200, Joakim Tjernlund wrote:
> > > > On Thu, 2022-06-02 at 21:11 +0100, Matthew Wilcox wrote:
> > > > > On Thu, Jun 02, 2022 at 07:24:23PM +0000, Joakim Tjernlund wrote:
> > > > > > We have this small embedded target(aarch64) with 32 MB of RAM where the kernel consumes 14420K:
> > > > > > Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma-reserved)
> > > > > >
> > > > > > I want to track down were most of this RAM is consumed so I can trim away some MBs
> > > > > > but I am having a hard time finding may way.
> > > > > > Is there some tool/kernel config that can help me with that?
> > > >
> > > > Those are interesting, thanks.
> > > > In my case it is the amount of work space RAM allocated that is a bit much. The kernel code/data
> > > > is 5+MB but the total need is 14MB, 9MB is buffer and similar.
> 
> I'm fairly sure you are in new territory here. The arm64 kernel is
> just not really optimized
> for small memory configurations. Even for 32-bit Arm, 32MB is
> considered too small for
> most workloads these days, though we do have some very limited support
> for machines
> with as little as 2MB of SRAM. The smallest arm64 machines that I can
> think of being
> supported in the kernel today have at least 256MB.
> 
> > > Found something, arm64 only supports mem model SPARSEMEM_EXTREME and it uses most of my memory.
> > > Tried to force SPARSEMEM_STATIC but that didn't boot, is STATIC even cheaper ?
> >
> > That sounds like a question for the arm64 maintainers
> 
> From what I understand, the 'static' variant uses a static allocation
> in .bss, so that
> may end up consuming most of your RAM even if it works normally. If all memory
> is contiguous, then using flatmem may be more appropriate. You may also be able
> to save some memory for sparsemem by reducing the size of the physical address
> space. CONFIG_ARM64_PA_BITS is normally '48', but reducing it to '40' may
> help (provided you can get that to boot). Most of the smaller CPUs only support
> a 40 bit physical address space, so having another option for this is
> probably sensible.
> 
> I think this is a case of "patches welcome". Nobody has really needed
> this so far, but
> as even the smaller machines are slowly migrating from 32-bit to
> 64-bit cores, optimizing
> this will get interesting for more developers. There are probably
> other low-hanging
> fruit that you can address after figuring out.

The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with
64 MiB or 128 MiB, and given the latter is a new SoC announced only
two or three years ago, requiring at least 256 MiB would be at best
unfortunate.  Given those SoCs are used in industrial applications
with very long support times, I think 32bit ARM will stay for years,
even with new products.

> One observation I made is that modern kernels don't seem to deal as
> well as older
> ones with low-memory situations, so even if you manage to free up most of your
> 32MB, it might still not work reliably.

Since we are using the SAMA5D27C-D5M in production, I would also be
interested in support for running 32 bit ARM with recent kernels on
systems with 64 MiB or even 32 MiB of memory.  If there are specific
things to test, you can let me know.

Greets
Alex

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Finding kernel RAM consumers ?

[Arnd Bergmann]

On Tue, Jun 7, 2022 at 7:41 AM Alexander Dahl <ada@thorsis.com> wrote:
> Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann:
> > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote:
> >
> > I think this is a case of "patches welcome". Nobody has really needed
> > this so far, but as even the smaller machines are slowly migrating from
> > 32-bit to 64-bit cores, optimizing this will get interesting for more
> >  developers. There are probably other low-hanging
> > fruit that you can address after figuring out.
>
> The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with
> 64 MiB or 128 MiB, and given the latter is a new SoC announced only
> two or three years ago, requiring at least 256 MiB would be at best
> unfortunate.  Given those SoCs are used in industrial applications
> with very long support times, I think 32bit ARM will stay for years,
> even with new products.

Yes, of course, and there is nothing wrong with that. We already see
Cortex-A7 cores down to 7nm, all running Linux, and I expect there
will likely be another 5 to 10 years of new 32-bit chips, and then another
10 years of people putting the existing chips into production, and after
that a slow decline of users updating their kernels before supporting
32-bit hardware becomes too expensive to support in the kernel.

On hardware that can run both 32-bit and 64-bit kernels, there are
pretty much only upsides to running 64-bit kernels (with 32-bit
thumb user space), but there is a memory overhead for the kernel
itself, usually some 20 to 30 MB. Reducing this difference can hopefully
help get fewer users stuck on 32-bit kernels by the time that they
get too painful to use.

> > One observation I made is that modern kernels don't seem to deal as
> > well as older ones with low-memory situations, so even if you manage
> > to free up most of your 32MB, it might still not work reliably.
>
> Since we are using the SAMA5D27C-D5M in production, I would also be
> interested in support for running 32 bit ARM with recent kernels on
> systems with 64 MiB or even 32 MiB of memory.  If there are specific
> things to test, you can let me know.

I don't have anything specific, just the general feeling that there is
something wrong about memory reclaim in smaller configurations.
A system that is fine after bootup can run for a long time without running
out of memory, but one thing I've observed in the past is that after a
process manages to consume all RAM and swap space, killing that one
task does not restore the overall system back into the same state
as before, and it remains sluggish.

Another issue is that I think we have more broken error handling
for failed in-kernel memory allocations. After you see the kernel
fail an allocation, I would usually recommend a reboot, and my
feeling is that this used to be better in the past.

Both of these could of course just be side-effects of kernel
bloat, where a particular workload is now worse than in the
past because it now needs more RAM to do the same thing.

        Arnd

Re: Finding kernel RAM consumers ?

[Arnd Bergmann]

On Tue, Jun 7, 2022 at 7:41 AM Alexander Dahl <ada@thorsis.com> wrote:
> Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann:
> > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote:
> >
> > I think this is a case of "patches welcome". Nobody has really needed
> > this so far, but as even the smaller machines are slowly migrating from
> > 32-bit to 64-bit cores, optimizing this will get interesting for more
> >  developers. There are probably other low-hanging
> > fruit that you can address after figuring out.
>
> The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with
> 64 MiB or 128 MiB, and given the latter is a new SoC announced only
> two or three years ago, requiring at least 256 MiB would be at best
> unfortunate.  Given those SoCs are used in industrial applications
> with very long support times, I think 32bit ARM will stay for years,
> even with new products.

Yes, of course, and there is nothing wrong with that. We already see
Cortex-A7 cores down to 7nm, all running Linux, and I expect there
will likely be another 5 to 10 years of new 32-bit chips, and then another
10 years of people putting the existing chips into production, and after
that a slow decline of users updating their kernels before supporting
32-bit hardware becomes too expensive to support in the kernel.

On hardware that can run both 32-bit and 64-bit kernels, there are
pretty much only upsides to running 64-bit kernels (with 32-bit
thumb user space), but there is a memory overhead for the kernel
itself, usually some 20 to 30 MB. Reducing this difference can hopefully
help get fewer users stuck on 32-bit kernels by the time that they
get too painful to use.

> > One observation I made is that modern kernels don't seem to deal as
> > well as older ones with low-memory situations, so even if you manage
> > to free up most of your 32MB, it might still not work reliably.
>
> Since we are using the SAMA5D27C-D5M in production, I would also be
> interested in support for running 32 bit ARM with recent kernels on
> systems with 64 MiB or even 32 MiB of memory.  If there are specific
> things to test, you can let me know.

I don't have anything specific, just the general feeling that there is
something wrong about memory reclaim in smaller configurations.
A system that is fine after bootup can run for a long time without running
out of memory, but one thing I've observed in the past is that after a
process manages to consume all RAM and swap space, killing that one
task does not restore the overall system back into the same state
as before, and it remains sluggish.

Another issue is that I think we have more broken error handling
for failed in-kernel memory allocations. After you see the kernel
fail an allocation, I would usually recommend a reboot, and my
feeling is that this used to be better in the past.

Both of these could of course just be side-effects of kernel
bloat, where a particular workload is now worse than in the
past because it now needs more RAM to do the same thing.

        Arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Finding kernel RAM consumers ?

[Joakim Tjernlund]

On Tue, 2022-06-07 at 10:38 +0200, Arnd Bergmann wrote:
> On Tue, Jun 7, 2022 at 7:41 AM Alexander Dahl <ada@thorsis.com> wrote:
> > Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann:
> > > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote:
> > > 
> > > I think this is a case of "patches welcome". Nobody has really needed
> > > this so far, but as even the smaller machines are slowly migrating from
> > > 32-bit to 64-bit cores, optimizing this will get interesting for more
> > >  developers. There are probably other low-hanging
> > > fruit that you can address after figuring out.
> > 
> > The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with
> > 64 MiB or 128 MiB, and given the latter is a new SoC announced only
> > two or three years ago, requiring at least 256 MiB would be at best
> > unfortunate.  Given those SoCs are used in industrial applications
> > with very long support times, I think 32bit ARM will stay for years,
> > even with new products.
> 
> Yes, of course, and there is nothing wrong with that. We already see
> Cortex-A7 cores down to 7nm, all running Linux, and I expect there
> will likely be another 5 to 10 years of new 32-bit chips, and then another
> 10 years of people putting the existing chips into production, and after
> that a slow decline of users updating their kernels before supporting
> 32-bit hardware becomes too expensive to support in the kernel.

My aarch64 system with 36MB RAM just dropped from:
  Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma)
to 
  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma)
with a small hack by Ard Biesheuvel <ardb@kernel.org>
-----------------------------------------------------------------
What you might try is changing the section size to 32 MB and mapping
the vmemmap region down to pages. That way, the vmemmap region should
only take up
- 512 KiB for the struct page array[] itself
- 4 KiB for the page table that replaces the 2 MB block mapping

You could try the below and see if it makes any difference?

diff --git a/arch/arm64/include/asm/sparsemem.h
b/arch/arm64/include/asm/sparsemem.h
index 4b73463423c3..a008f4342532 100644
--- a/arch/arm64/include/asm/sparsemem.h
+++ b/arch/arm64/include/asm/sparsemem.h
@@ -23,7 +23,7 @@
  * entries could not be created for vmemmap mappings.
  * 16K follows 4K for simplicity.
  */
-#define SECTION_SIZE_BITS 27
+#define SECTION_SIZE_BITS 25
 #endif /* CONFIG_ARM64_64K_PAGES */

 #endif
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 5b1946f1805c..d25560a53a67 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
addr, unsigned long end,
 }
 #endif

-#if !ARM64_KERNEL_USES_PMD_MAPS
+#if 1// !ARM64_KERNEL_USES_PMD_MAPS
 int __meminit vmemmap_populate(unsigned long start, unsigned long
end, int node,
                struct vmem_altmap *altmap)
 {

So there is hope for systems with little RAM

 Jocke

Re: Finding kernel RAM consumers ?

[Joakim Tjernlund]

On Tue, 2022-06-07 at 10:38 +0200, Arnd Bergmann wrote:
> On Tue, Jun 7, 2022 at 7:41 AM Alexander Dahl <ada@thorsis.com> wrote:
> > Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann:
> > > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote:
> > > 
> > > I think this is a case of "patches welcome". Nobody has really needed
> > > this so far, but as even the smaller machines are slowly migrating from
> > > 32-bit to 64-bit cores, optimizing this will get interesting for more
> > >  developers. There are probably other low-hanging
> > > fruit that you can address after figuring out.
> > 
> > The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with
> > 64 MiB or 128 MiB, and given the latter is a new SoC announced only
> > two or three years ago, requiring at least 256 MiB would be at best
> > unfortunate.  Given those SoCs are used in industrial applications
> > with very long support times, I think 32bit ARM will stay for years,
> > even with new products.
> 
> Yes, of course, and there is nothing wrong with that. We already see
> Cortex-A7 cores down to 7nm, all running Linux, and I expect there
> will likely be another 5 to 10 years of new 32-bit chips, and then another
> 10 years of people putting the existing chips into production, and after
> that a slow decline of users updating their kernels before supporting
> 32-bit hardware becomes too expensive to support in the kernel.

My aarch64 system with 36MB RAM just dropped from:
  Memory: 22444K/36864K available (3584K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 14420K reserved, 0K cma)
to 
  Memory: 29732K/36864K available (3648K kernel code, 698K rwdata, 936K rodata, 320K init, 255K bss, 7132K reserved, 0K cma)
with a small hack by Ard Biesheuvel <ardb@kernel.org>
-----------------------------------------------------------------
What you might try is changing the section size to 32 MB and mapping
the vmemmap region down to pages. That way, the vmemmap region should
only take up
- 512 KiB for the struct page array[] itself
- 4 KiB for the page table that replaces the 2 MB block mapping

You could try the below and see if it makes any difference?

diff --git a/arch/arm64/include/asm/sparsemem.h
b/arch/arm64/include/asm/sparsemem.h
index 4b73463423c3..a008f4342532 100644
--- a/arch/arm64/include/asm/sparsemem.h
+++ b/arch/arm64/include/asm/sparsemem.h
@@ -23,7 +23,7 @@
  * entries could not be created for vmemmap mappings.
  * 16K follows 4K for simplicity.
  */
-#define SECTION_SIZE_BITS 27
+#define SECTION_SIZE_BITS 25
 #endif /* CONFIG_ARM64_64K_PAGES */

 #endif
diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c
index 5b1946f1805c..d25560a53a67 100644
--- a/arch/arm64/mm/mmu.c
+++ b/arch/arm64/mm/mmu.c
@@ -1196,7 +1196,7 @@ static void free_empty_tables(unsigned long
addr, unsigned long end,
 }
 #endif

-#if !ARM64_KERNEL_USES_PMD_MAPS
+#if 1// !ARM64_KERNEL_USES_PMD_MAPS
 int __meminit vmemmap_populate(unsigned long start, unsigned long
end, int node,
                struct vmem_altmap *altmap)
 {

So there is hope for systems with little RAM

 Jocke


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Finding kernel RAM consumers ?

[Russell King (Oracle)]

On Tue, Jun 07, 2022 at 10:38:54AM +0200, Arnd Bergmann wrote:
> On Tue, Jun 7, 2022 at 7:41 AM Alexander Dahl <ada@thorsis.com> wrote:
> > Am Fri, Jun 03, 2022 at 08:11:31PM +0200 schrieb Arnd Bergmann:
> > > On Fri, Jun 3, 2022 at 7:29 PM Matthew Wilcox <willy@infradead.org> wrote:
> > >
> > > I think this is a case of "patches welcome". Nobody has really needed
> > > this so far, but as even the smaller machines are slowly migrating from
> > > 32-bit to 64-bit cores, optimizing this will get interesting for more
> > >  developers. There are probably other low-hanging
> > > fruit that you can address after figuring out.
> >
> > The SiP variants of at91 SAMA5D2 (armv7) or SAM9x60 (armv5) come with
> > 64 MiB or 128 MiB, and given the latter is a new SoC announced only
> > two or three years ago, requiring at least 256 MiB would be at best
> > unfortunate.  Given those SoCs are used in industrial applications
> > with very long support times, I think 32bit ARM will stay for years,
> > even with new products.
> 
> Yes, of course, and there is nothing wrong with that. We already see
> Cortex-A7 cores down to 7nm, all running Linux, and I expect there
> will likely be another 5 to 10 years of new 32-bit chips, and then another
> 10 years of people putting the existing chips into production, and after
> that a slow decline of users updating their kernels before supporting
> 32-bit hardware becomes too expensive to support in the kernel.

It should be noted that 20 years puts us past the 2038 32-bit time_t
wrap problem - and although there's been work to address that in the
UAPI, that doesn't mean that userspace will cope.

Anyone deploying a system that is expected to still be live beyond
the end of 32-bit time_t had better be testing their userspace for
that event now!

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last!

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Finding kernel RAM consumers ?

[Arnd Bergmann]

On Tue, Jun 7, 2022 at 11:44 AM Russell King (Oracle)
<linux@armlinux.org.uk> wrote:
> On Tue, Jun 07, 2022 at 10:38:54AM +0200, Arnd Bergmann wrote:
> >
> > Yes, of course, and there is nothing wrong with that. We already see
> > Cortex-A7 cores down to 7nm, all running Linux, and I expect there
> > will likely be another 5 to 10 years of new 32-bit chips, and then another
> > 10 years of people putting the existing chips into production, and after
> > that a slow decline of users updating their kernels before supporting
> > 32-bit hardware becomes too expensive to support in the kernel.
>
> It should be noted that 20 years puts us past the 2038 32-bit time_t
> wrap problem - and although there's been work to address that in the
> UAPI, that doesn't mean that userspace will cope.
>
> Anyone deploying a system that is expected to still be live beyond
> the end of 32-bit time_t had better be testing their userspace for
> that event now!

That is absolutely true, but it's also independent of what kernel is
being used. I assume that anyone who is this memory constrained
is using 32-bit thumb userspace even on 64-bit kernels.

The base distro support in embedded systems using openembedded
with musl-1.2.x usually works fine beyond y2038, but of course each
system should be tested for this before shipping as there are still
bugs in less common code.

         arnd

Re: Finding kernel RAM consumers ?

[Arnd Bergmann]

On Tue, Jun 7, 2022 at 11:44 AM Russell King (Oracle)
<linux@armlinux.org.uk> wrote:
> On Tue, Jun 07, 2022 at 10:38:54AM +0200, Arnd Bergmann wrote:
> >
> > Yes, of course, and there is nothing wrong with that. We already see
> > Cortex-A7 cores down to 7nm, all running Linux, and I expect there
> > will likely be another 5 to 10 years of new 32-bit chips, and then another
> > 10 years of people putting the existing chips into production, and after
> > that a slow decline of users updating their kernels before supporting
> > 32-bit hardware becomes too expensive to support in the kernel.
>
> It should be noted that 20 years puts us past the 2038 32-bit time_t
> wrap problem - and although there's been work to address that in the
> UAPI, that doesn't mean that userspace will cope.
>
> Anyone deploying a system that is expected to still be live beyond
> the end of 32-bit time_t had better be testing their userspace for
> that event now!

That is absolutely true, but it's also independent of what kernel is
being used. I assume that anyone who is this memory constrained
is using 32-bit thumb userspace even on 64-bit kernels.

The base distro support in embedded systems using openembedded
with musl-1.2.x usually works fine beyond y2038, but of course each
system should be tested for this before shipping as there are still
bugs in less common code.

         arnd

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel