Re: Deferred Memory Init: How to bring rest of memory online after limiting it with `mem=XG`?

[Feng Tang <feng.tang@intel.com>]

Hi Paul,

On Thu, Dec 03, 2020 at 11:51:58AM +0100, Paul Menzel wrote:
> Dear Feng,
> 
> 
> I am trying to reduce the startup time of Debian’s Linux 5.9.9 on a Intel
> Kaby Lake system with 32 GB of memory (TUXEDO Book BU1406 (Clevo N240BU)).
> On your Linux Plumbers Conference 2019 slides of your talk *Linux Kernel
> Fastboot On the Way* [1], you mention *Deferred Memory Init*:
> 
> >Deferred Memory Init
> >
> >•   8GB RAM’s initialization costs 100+ ms
> >•   In early boot phase, we don’t need that much memory
> >•   Utilize the memory hotplug feature
> >    •   “mem=4096m” in cmdline to only init 2 GB
> >    •   Use systemd service to add rest memory in parallel
> 
> Starting Linux with `mem=2G` indeed reduces the startup time, but I am
> unable to get the rest of the memory online. Comparing it with a boot
> without `mem=2G` the `memoryX`  devices under `/sys/devices/system/memory/`
> are missing.

[...]

> 
> Can the deferred memory initialization be done with the upstream Linux
> kernel, or were you using patches on top?

Yes, it should be able to work with upstream kernel. And you need to
do a 'probe' operation to create 'memoryX' for those deferred memroy.

When you use "mem=2G", there are a bunch of memory from e820 map will
be chopped off, so you see a few memory devices left in /sys/devices/system/memory/

And to see those missing devices, you can check your dmesg log to find
the physical memory address range which is chopped off. 

Followin is a quick test I did on qemu:

	[    0.000000] BIOS-provided physical RAM map:
	[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
	[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
	[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
	[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007ffdffff] usable  --> real E820 map
	...
	[    0.000000] user-defined physical RAM map:
	[    0.000000] user: [mem 0x0000000000000000-0x000000000009fbff] usable
	[    0.000000] user: [mem 0x000000000009fc00-0x000000000009ffff] reserved
	[    0.000000] user: [mem 0x00000000000f0000-0x00000000000fffff] reserved
	[    0.000000] user: [mem 0x0000000000100000-0x000000005dbfffff] usable	      --> with "mem=xxx" option

We can see the "user-defined" version has much less "useable" memory,
figue out the physical address, for this

	echo 0x60000000 > /sys/devices/system/memory/probe

will create a new "memoryX" in /sys/devices/system/memory/, and echo
"online" to the 'state' inside its folder will make it online.

There may be some more stuff to care, like the memory block size and
alignment, you can read the related code and document.

Thanks,
Feng