From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F20E4C71156 for ; Thu, 3 Dec 2020 10:52:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CA389208A9 for ; Thu, 3 Dec 2020 10:52:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CA389208A9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=molgen.mpg.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2565E6B0068; Thu, 3 Dec 2020 05:52:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 206566B006E; Thu, 3 Dec 2020 05:52:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11D2D6B0070; Thu, 3 Dec 2020 05:52:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0125.hostedemail.com [216.40.44.125]) by kanga.kvack.org (Postfix) with ESMTP id E7F9B6B0068 for ; Thu, 3 Dec 2020 05:52:02 -0500 (EST) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id B239D180AD807 for ; Thu, 3 Dec 2020 10:52:02 +0000 (UTC) X-FDA: 77551656084.02.tank98_2c07c9f273bb Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin02.hostedemail.com (Postfix) with ESMTP id 8AD7210097AA0 for ; Thu, 3 Dec 2020 10:52:02 +0000 (UTC) X-HE-Tag: tank98_2c07c9f273bb X-Filterd-Recvd-Size: 3892 Received: from mx1.molgen.mpg.de (mx3.molgen.mpg.de [141.14.17.11]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Thu, 3 Dec 2020 10:52:01 +0000 (UTC) Received: from [192.168.0.4] (ip5f5af442.dynamic.kabel-deutschland.de [95.90.244.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: pmenzel) by mx.molgen.mpg.de (Postfix) with ESMTPSA id 57ED920647B73; Thu, 3 Dec 2020 11:51:59 +0100 (CET) To: Feng Tang Cc: linux-mm@kvack.org, Arjan van de Ven From: Paul Menzel Subject: Deferred Memory Init: How to bring rest of memory online after limiting it with `mem=XG`? Message-ID: Date: Thu, 3 Dec 2020 11:51:58 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.174314, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Dear Feng, I am trying to reduce the startup time of Debian=E2=80=99s Linux 5.9.9 on= a=20 Intel Kaby Lake system with 32 GB of memory (TUXEDO Book BU1406 (Clevo=20 N240BU)). On your Linux Plumbers Conference 2019 slides of your talk=20 *Linux Kernel Fastboot On the Way* [1], you mention *Deferred Memory Init= *: > Deferred Memory Init >=20 > =E2=80=A2 8GB RAM=E2=80=99s initialization costs 100+ ms > =E2=80=A2 In early boot phase, we don=E2=80=99t need that much memory > =E2=80=A2 Utilize the memory hotplug feature > =E2=80=A2 =E2=80=9Cmem=3D4096m=E2=80=9D in cmdline to only init 2= GB > =E2=80=A2 Use systemd service to add rest memory in parallel Starting Linux with `mem=3D2G` indeed reduces the startup time, but I am=20 unable to get the rest of the memory online. Comparing it with a boot=20 without `mem=3D2G` the `memoryX` devices under=20 `/sys/devices/system/memory/` are missing. With `mem=3D2G`: $ lsmem --output-all RANGE SIZE STATE REMOVABLE BLOCK NODE=20 ZONES 0x0000000000000000-0x0000000007ffffff 128M online yes 0 0=20 None 0x0000000008000000-0x000000007fffffff 1,9G online yes 1-15 0=20 DMA32 Memory block size: 128M Total online memory: 2G Total offline memory: 0B $ ls -d /sys/devices/system/memory/memory* /sys/devices/system/memory/memory0 /sys/devices/system/memory/memory2 /sys/devices/system/memory/memory1 /sys/devices/system/memory/memory3 /sys/devices/system/memory/memory10 /sys/devices/system/memory/memory4 /sys/devices/system/memory/memory11 /sys/devices/system/memory/memory5 /sys/devices/system/memory/memory12 /sys/devices/system/memory/memory6 /sys/devices/system/memory/memory13 /sys/devices/system/memory/memory7 /sys/devices/system/memory/memory14 /sys/devices/system/memory/memory8 /sys/devices/system/memory/memory15 /sys/devices/system/memory/memory9 ``` Without `mem=3D2G`: ``` $ lsmem --output-all RANGE SIZE STATE REMOVABLE BLOCK NODE=20 ZONES 0x0000000000000000-0x0000000007ffffff 128M online yes 0 0=20 None 0x0000000008000000-0x0000000087ffffff 2G online yes 1-16 0=20 DMA32 0x0000000088000000-0x000000008fffffff 128M online yes 17 0=20 None 0x0000000100000000-0x0000000867ffffff 29,6G online yes 32-268 0=20 Normal 0x0000000868000000-0x000000086fffffff 128M online yes 269 0=20 None Memory block size: 128M Total online memory: 32G Total offline memory: 0B ``` Can the deferred memory initialization be done with the upstream Linux=20 kernel, or were you using patches on top? Kind regards, Paul [1]:=20 https://www.linuxplumbersconf.org/event/4/contributions/281/attachments/2= 16/617/LPC_2019_kernel_fastboot_on_the_way.pdf