From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B532C433E0 for ; Thu, 25 Feb 2021 22:44:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7321264EF5 for ; Thu, 25 Feb 2021 22:44:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7321264EF5 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E43A86B0006; Thu, 25 Feb 2021 17:44:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DF3366B006C; Thu, 25 Feb 2021 17:44:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D09D66B006E; Thu, 25 Feb 2021 17:44:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0224.hostedemail.com [216.40.44.224]) by kanga.kvack.org (Postfix) with ESMTP id BC3FC6B0006 for ; Thu, 25 Feb 2021 17:44:22 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 8B4AF824556B for ; Thu, 25 Feb 2021 22:44:22 +0000 (UTC) X-FDA: 77858270364.11.732931D Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf09.hostedemail.com (Postfix) with ESMTP id EBBA1600249C for ; Thu, 25 Feb 2021 22:44:17 +0000 (UTC) Received: by mail.kernel.org (Postfix) with ESMTPSA id 3E7C864EDC; Thu, 25 Feb 2021 22:44:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1614293061; bh=detox8qlMqSyvNouatLGwPyUgUdnIhtkxS0rmzHuamk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RoRJYEBkwZxYpXCh1znRdEHVmML/cqDpMbtkIolK8fbPGp2APtSCFUW5zmNipyO1Q nBEpkv4rBhsqepxLrvLEmZcxqj61BG7J90uivn4H5fRhIxSDCHfdFQjZsUl3iVzWqT UeyIOiSTkGlpdaOoPV6KJRtBhoyQazsGfeDhgrJmNYydjCYHdYOPIqGwQbdSm7BT5N ksuqwWrV3uI/3hYFpNJU+dOW1tkTC7KZbiqvfsBpPnuttaof0rKinMFK+uuddOKva4 b0pWHOTjA+alCybOPQ1AcXPVaii0EVX56qBivUO7FEWCj0syKfcx4dYp+QJtM63wzS Ovr3gjKEte62w== From: Mike Rapoport To: Andrew Morton Cc: Andrea Arcangeli , Baoquan He , Borislav Petkov , Chris Wilson , David Hildenbrand , "H. Peter Anvin" , Ingo Molnar , Linus Torvalds , =?UTF-8?q?=C5=81ukasz=20Majczak?= , Mel Gorman , Michal Hocko , Mike Rapoport , Mike Rapoport , Qian Cai , "Sarvela, Tomi P" , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mm@kvack.org, stable@vger.kernel.org, x86@kernel.org Subject: [PATCH v8 1/1] mm/page_alloc.c: refactor initialization of struct page for holes in memory layout Date: Fri, 26 Feb 2021 00:43:51 +0200 Message-Id: <20210225224351.7356-2-rppt@kernel.org> X-Mailer: git-send-email 2.28.0 In-Reply-To: <20210225224351.7356-1-rppt@kernel.org> References: <20210225224351.7356-1-rppt@kernel.org> MIME-Version: 1.0 X-Stat-Signature: bqmo5iqizxw9ttswaxk6tqup67os5zwm X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: EBBA1600249C Received-SPF: none (kernel.org>: No applicable sender policy available) receiver=imf09; identity=mailfrom; envelope-from=""; helo=mail.kernel.org; client-ip=198.145.29.99 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614293057-881937 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Mike Rapoport There could be struct pages that are not backed by actual physical memory= . This can happen when the actual memory bank is not a multiple of SECTION_SIZE or when an architecture does not register memory holes reserved by the firmware as memblock.memory. Such pages are currently initialized using init_unavailable_mem() functio= n that iterates through PFNs in holes in memblock.memory and if there is a struct page corresponding to a PFN, the fields of this page are set to default values and it is marked as Reserved. init_unavailable_mem() does not take into account zone and node the page belongs to and sets both zone and node links in struct page to zero. Before commit 73a6e474cb37 ("mm: memmap_init: iterate over memblock regio= ns rather that check each PFN") the holes inside a zone were re-initialized during memmap_init() and got their zone/node links right. However, after that commit nothing updates the struct pages representing such holes. On a system that has firmware reserved holes in a zone above ZONE_DMA, fo= r instance in a configuration below: # grep -A1 E820 /proc/iomem 7a17b000-7a216fff : Unknown E820 type 7a217000-7bffffff : System RAM unset zone link in struct page will trigger VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page); in set_pfnblock_flags_mask() when called with a struct page from a range other than E820_TYPE_RAM because there are pages in the range of ZONE_DMA= 32 but the unset zone link in struct page makes them appear as a part of ZONE_DMA. Interleave initialization of the unavailable pages with the normal initialization of memory map, so that zone and node information will be properly set on struct pages that are not backed by the actual memory. With this change the pages for holes inside a zone will get proper zone/node links and the pages that are not spanned by any node will get links to the adjacent zone/node. The holes between nodes will be prepende= d to the zone/node above the hole and the trailing pages in the last sectio= n that will be appended to the zone/node below. Fixes: 73a6e474cb37 ("mm: memmap_init: iterate over memblock regions rath= er that check each PFN") Signed-off-by: Mike Rapoport Reported-by: Qian Cai Reported-by: Andrea Arcangeli Reviewed-by: Baoquan He Acked-by: Vlastimil Babka --- mm/page_alloc.c | 158 +++++++++++++++++++++++------------------------- 1 file changed, 75 insertions(+), 83 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3e93f8b29bae..b6b8b9396c1e 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -6280,12 +6280,65 @@ static void __meminit zone_init_free_lists(struct= zone *zone) } } =20 +#if !defined(CONFIG_FLAT_NODE_MEM_MAP) +/* + * Only struct pages that correspond to ranges defined by memblock.memor= y + * are zeroed and initialized by going through __init_single_page() duri= ng + * memmap_init_zone(). + * + * But, there could be struct pages that correspond to holes in + * memblock.memory. This can happen because of the following reasons: + * - physical memory bank size is not necessarily the exact multiple of = the + * arbitrary section size + * - early reserved memory may not be listed in memblock.memory + * - memory layouts defined with memmap=3D kernel parameter may not alig= n + * nicely with memmap sections + * + * Explicitly initialize those struct pages so that: + * - PG_Reserved is set + * - zone and node links point to zone and node that span the page if th= e + * hole is in the middle of a zone + * - zone and node links point to adjacent zone/node if the hole falls o= n + * the zone boundary; the pages in such holes will be prepended to the + * zone/node above the hole except for the trailing pages in the last + * section that will be appended to the zone/node below. + */ +static u64 __meminit init_unavailable_range(unsigned long spfn, + unsigned long epfn, + int zone, int node) +{ + unsigned long pfn; + u64 pgcnt =3D 0; + + for (pfn =3D spfn; pfn < epfn; pfn++) { + if (!pfn_valid(ALIGN_DOWN(pfn, pageblock_nr_pages))) { + pfn =3D ALIGN_DOWN(pfn, pageblock_nr_pages) + + pageblock_nr_pages - 1; + continue; + } + __init_single_page(pfn_to_page(pfn), pfn, zone, node); + __SetPageReserved(pfn_to_page(pfn)); + pgcnt++; + } + + return pgcnt; +} +#else +static inline u64 init_unavailable_range(unsigned long spfn, unsigned lo= ng epfn, + int zone, int node) +{ + return 0; +} +#endif + void __meminit __weak memmap_init_zone(struct zone *zone) { unsigned long zone_start_pfn =3D zone->zone_start_pfn; unsigned long zone_end_pfn =3D zone_start_pfn + zone->spanned_pages; int i, nid =3D zone_to_nid(zone), zone_id =3D zone_idx(zone); + static unsigned long hole_pfn =3D 0; unsigned long start_pfn, end_pfn; + u64 pgcnt =3D 0; =20 for_each_mem_pfn_range(i, nid, &start_pfn, &end_pfn, NULL) { start_pfn =3D clamp(start_pfn, zone_start_pfn, zone_end_pfn); @@ -6295,7 +6348,29 @@ void __meminit __weak memmap_init_zone(struct zone= *zone) memmap_init_range(end_pfn - start_pfn, nid, zone_id, start_pfn, zone_end_pfn, MEMINIT_EARLY, NULL, MIGRATE_MOVABLE); + + if (hole_pfn < start_pfn) + pgcnt +=3D init_unavailable_range(hole_pfn, start_pfn, + zone_id, nid); + hole_pfn =3D end_pfn; } + +#ifdef CONFIG_SPARSEMEM + /* + * Initialize the hole in the range [zone_end_pfn, section_end]. + * If zone boundary falls in the middle of a section, this hole + * will be re-initialized during the call to this function for the + * higher zone. + */ + end_pfn =3D round_up(zone_end_pfn, PAGES_PER_SECTION); + if (hole_pfn < end_pfn) + pgcnt +=3D init_unavailable_range(hole_pfn, end_pfn, + zone_id, nid); +#endif + + if (pgcnt) + pr_info(" %s zone: %lld pages in unavailable ranges\n", + zone->name, pgcnt); } =20 static int zone_batchsize(struct zone *zone) @@ -7092,88 +7167,6 @@ void __init free_area_init_memoryless_node(int nid= ) free_area_init_node(nid); } =20 -#if !defined(CONFIG_FLAT_NODE_MEM_MAP) -/* - * Initialize all valid struct pages in the range [spfn, epfn) and mark = them - * PageReserved(). Return the number of struct pages that were initializ= ed. - */ -static u64 __init init_unavailable_range(unsigned long spfn, unsigned lo= ng epfn) -{ - unsigned long pfn; - u64 pgcnt =3D 0; - - for (pfn =3D spfn; pfn < epfn; pfn++) { - if (!pfn_valid(ALIGN_DOWN(pfn, pageblock_nr_pages))) { - pfn =3D ALIGN_DOWN(pfn, pageblock_nr_pages) - + pageblock_nr_pages - 1; - continue; - } - /* - * Use a fake node/zone (0) for now. Some of these pages - * (in memblock.reserved but not in memblock.memory) will - * get re-initialized via reserve_bootmem_region() later. - */ - __init_single_page(pfn_to_page(pfn), pfn, 0, 0); - __SetPageReserved(pfn_to_page(pfn)); - pgcnt++; - } - - return pgcnt; -} - -/* - * Only struct pages that are backed by physical memory are zeroed and - * initialized by going through __init_single_page(). But, there are som= e - * struct pages which are reserved in memblock allocator and their field= s - * may be accessed (for example page_to_pfn() on some configuration acce= sses - * flags). We must explicitly initialize those struct pages. - * - * This function also addresses a similar issue where struct pages are l= eft - * uninitialized because the physical address range is not covered by - * memblock.memory or memblock.reserved. That could happen when memblock - * layout is manually configured via memmap=3D, or when the highest phys= ical - * address (max_pfn) does not end on a section boundary. - */ -static void __init init_unavailable_mem(void) -{ - phys_addr_t start, end; - u64 i, pgcnt; - phys_addr_t next =3D 0; - - /* - * Loop through unavailable ranges not covered by memblock.memory. - */ - pgcnt =3D 0; - for_each_mem_range(i, &start, &end) { - if (next < start) - pgcnt +=3D init_unavailable_range(PFN_DOWN(next), - PFN_UP(start)); - next =3D end; - } - - /* - * Early sections always have a fully populated memmap for the whole - * section - see pfn_valid(). If the last section has holes at the - * end and that section is marked "online", the memmap will be - * considered initialized. Make sure that memmap has a well defined - * state. - */ - pgcnt +=3D init_unavailable_range(PFN_DOWN(next), - round_up(max_pfn, PAGES_PER_SECTION)); - - /* - * Struct pages that do not have backing memory. This could be because - * firmware is using some of this memory, or for some other reasons. - */ - if (pgcnt) - pr_info("Zeroed struct page in unavailable ranges: %lld pages", pgcnt)= ; -} -#else -static inline void __init init_unavailable_mem(void) -{ -} -#endif /* !CONFIG_FLAT_NODE_MEM_MAP */ - #if MAX_NUMNODES > 1 /* * Figure out the number of possible node ids. @@ -7597,7 +7590,6 @@ void __init free_area_init(unsigned long *max_zone_= pfn) /* Initialise every node */ mminit_verify_pageflags_layout(); setup_nr_node_ids(); - init_unavailable_mem(); for_each_online_node(nid) { pg_data_t *pgdat =3D NODE_DATA(nid); free_area_init_node(nid); --=20 2.28.0