5.11 "debug" missing 20-30% memory, and exceptionally high reserved

5.11 "debug" missing 20-30% memory, and exceptionally high reserved

[Chris Murphy]

Fedora is seeing a significantly reduced /proc/meminfo MemTotal  with
"debug" 5.11 series kernels. The problem isn't happening with
"nodebug" kernels. The other possible instigator is GCC 11.0.0 is
being used (all 5.10 series, nodebug and debug, are built with GCC
10.2.1). We're still doing some tests to narrow it down.

The problem reproduces on x86_64 (no info on other archs yet) both
baremetal and qemu-kvm. And first manifests with this line:

5.11.0-0.rc5.134.fc34.x86_64+debug
Jan 28 13:56:08 flap.local kernel: Memory: 6547356K/8276392K available
(43019K kernel code, 11036K rwdata, 27184K rodata, 5036K init, 31780K
bss, 1728780K reserved, 0K cma-reserved)

5.11.0-0.rc5.134.fc34.x86_64
Jan 28 13:54:19 flap.local kernel: Memory: 7915056K/8276392K available
(16393K kernel code, 3469K rwdata, 26796K rodata, 2544K init, 5484K
bss, 361076K reserved, 0K cma-reserved)

Reserved memory seems quite high in the debug case.

Downstream bug report has dmesg for
5.11.0-0.rc5.134.fc34.x86_64+debug, and kernel config attachments.

Bug 1921923 - Kernel under-reporting or hiding about a third of
available memory in "debug" 5.11 series
https://bugzilla.redhat.com/show_bug.cgi?id=1921923

-- 
Chris Murphy

Re: 5.11 "debug" missing 20-30% memory, and exceptionally high reserved

[Chris Murphy]

On Sat, Jan 30, 2021 at 11:48 AM Chris Murphy <lists@colorremedies.com> wrote:

> Bug 1921923 - Kernel under-reporting or hiding about a third of
> available memory in "debug" 5.11 series
> https://bugzilla.redhat.com/show_bug.cgi?id=1921923

Looks like it's newy enabled CONFIG_KASAN options. The issue doesn't
happen when these are disabled:

-CONFIG_KASAN_SHADOW_OFFSET=0xdffffc0000000000
-CONFIG_KASAN=y
-CONFIG_KASAN_GENERIC=y
-CONFIG_KASAN_INLINE=y
-CONFIG_KASAN_STACK=1
-CONFIG_KASAN_VMALLOC=y

-- 
Chris Murphy

Re: 5.11 "debug" missing 20-30% memory, and exceptionally high reserved

[David Hildenbrand]

On 31.01.21 00:21, Chris Murphy wrote:
> On Sat, Jan 30, 2021 at 11:48 AM Chris Murphy <lists@colorremedies.com> wrote:
> 
>> Bug 1921923 - Kernel under-reporting or hiding about a third of
>> available memory in "debug" 5.11 series
>> https://bugzilla.redhat.com/show_bug.cgi?id=1921923
> 
> Looks like it's newy enabled CONFIG_KASAN options. The issue doesn't
> happen when these are disabled:
> 
> -CONFIG_KASAN_SHADOW_OFFSET=0xdffffc0000000000
> -CONFIG_KASAN=y
> -CONFIG_KASAN_GENERIC=y
> -CONFIG_KASAN_INLINE=y
> -CONFIG_KASAN_STACK=1
> -CONFIG_KASAN_VMALLOC=y

https://static.lwn.net/kerneldoc/dev-tools/kasan.html

"The state of each 8 aligned bytes of memory is encoded in one shadow 
byte." - Memory overhead of 1/8, which matches what you report.

As that memory is allocated early during boot, it's never managed by the 
buddy and therefore not accounted towards MemTotal.

One could adjust MemTotal accounting in the kernel using 
adjust_managed_pages() from kasan code, but why should we really bother 
about fixing up accounting for debug kernels.

This debug option comes with a big price tag on it. Be careful when 
enabling it.

-- 
Thanks,

David / dhildenb