From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 636BCC4361A for ; Fri, 4 Dec 2020 08:06:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B004022581 for ; Fri, 4 Dec 2020 08:06:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B004022581 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CDB876B0036; Fri, 4 Dec 2020 03:06:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C8A366B005C; Fri, 4 Dec 2020 03:06:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BA1436B0068; Fri, 4 Dec 2020 03:06:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0129.hostedemail.com [216.40.44.129]) by kanga.kvack.org (Postfix) with ESMTP id A3C396B0036 for ; Fri, 4 Dec 2020 03:06:06 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 65454180AD80F for ; Fri, 4 Dec 2020 08:06:06 +0000 (UTC) X-FDA: 77554866732.01.angle00_560189e273c2 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 3DDEF1004F63D for ; Fri, 4 Dec 2020 08:06:06 +0000 (UTC) X-HE-Tag: angle00_560189e273c2 X-Filterd-Recvd-Size: 4451 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Fri, 4 Dec 2020 08:06:04 +0000 (UTC) IronPort-SDR: K3gAfXODf0JVAlBrQ9NGp/uJOql1q94z9XC5ekM5oeftYFrfRTTefP8xFxIvEFjClDc/rbQxS6 WelXrcfPQ+LA== X-IronPort-AV: E=McAfee;i="6000,8403,9824"; a="152596018" X-IronPort-AV: E=Sophos;i="5.78,392,1599548400"; d="scan'208";a="152596018" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Dec 2020 00:05:59 -0800 IronPort-SDR: lEhyC/nzafYmb8H3Rq5WQpriCxNQ9jIBryAg4HStlsq2k21dMG1mKTmRTeuZNew3PqpO+x0Sqe vLTIepPVsWaw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,392,1599548400"; d="scan'208";a="336293955" Received: from shbuild999.sh.intel.com (HELO localhost) ([10.239.147.98]) by orsmga006.jf.intel.com with ESMTP; 04 Dec 2020 00:05:57 -0800 Date: Fri, 4 Dec 2020 16:05:56 +0800 From: Feng Tang To: Paul Menzel Cc: linux-mm@kvack.org, Arjan van de Ven Subject: Re: Deferred Memory Init: How to bring rest of memory online after limiting it with `mem=XG`? Message-ID: <20201204080556.GA66896@shbuild999.sh.intel.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Paul, On Thu, Dec 03, 2020 at 11:51:58AM +0100, Paul Menzel wrote: > Dear Feng, >=20 >=20 > I am trying to reduce the startup time of Debian=E2=80=99s Linux 5.9.9 = on a Intel > Kaby Lake system with 32 GB of memory (TUXEDO Book BU1406 (Clevo N240BU= )). > On your Linux Plumbers Conference 2019 slides of your talk *Linux Kerne= l > Fastboot On the Way* [1], you mention *Deferred Memory Init*: >=20 > >Deferred Memory Init > > > >=E2=80=A2 8GB RAM=E2=80=99s initialization costs 100+ ms > >=E2=80=A2 In early boot phase, we don=E2=80=99t need that much memor= y > >=E2=80=A2 Utilize the memory hotplug feature > > =E2=80=A2 =E2=80=9Cmem=3D4096m=E2=80=9D in cmdline to only init = 2 GB > > =E2=80=A2 Use systemd service to add rest memory in parallel >=20 > Starting Linux with `mem=3D2G` indeed reduces the startup time, but I a= m > unable to get the rest of the memory online. Comparing it with a boot > without `mem=3D2G` the `memoryX` devices under `/sys/devices/system/me= mory/` > are missing. [...] >=20 > Can the deferred memory initialization be done with the upstream Linux > kernel, or were you using patches on top? Yes, it should be able to work with upstream kernel. And you need to do a 'probe' operation to create 'memoryX' for those deferred memroy. When you use "mem=3D2G", there are a bunch of memory from e820 map will be chopped off, so you see a few memory devices left in /sys/devices/syst= em/memory/ And to see those missing devices, you can check your dmesg log to find the physical memory address range which is chopped off.=20 Followin is a quick test I did on qemu: [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] us= able [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] re= served [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] re= served [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x000000007ffdffff] us= able --> real E820 map ... [ 0.000000] user-defined physical RAM map: [ 0.000000] user: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] user: [mem 0x000000000009fc00-0x000000000009ffff] reserve= d [ 0.000000] user: [mem 0x00000000000f0000-0x00000000000fffff] reserve= d [ 0.000000] user: [mem 0x0000000000100000-0x000000005dbfffff] usable = --> with "mem=3Dxxx" option We can see the "user-defined" version has much less "useable" memory, figue out the physical address, for this echo 0x60000000 > /sys/devices/system/memory/probe will create a new "memoryX" in /sys/devices/system/memory/, and echo "online" to the 'state' inside its folder will make it online. There may be some more stuff to care, like the memory block size and alignment, you can read the related code and document. Thanks, Feng