From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15071C433FE for ; Thu, 27 Jan 2022 09:41:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A23DB6B0071; Thu, 27 Jan 2022 04:41:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D32F6B0072; Thu, 27 Jan 2022 04:41:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C19B6B0073; Thu, 27 Jan 2022 04:41:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0206.hostedemail.com [216.40.44.206]) by kanga.kvack.org (Postfix) with ESMTP id 7C4786B0071 for ; Thu, 27 Jan 2022 04:41:50 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 15C168249980 for ; Thu, 27 Jan 2022 09:41:50 +0000 (UTC) X-FDA: 79075575180.16.71F0A11 Received: from mail-pg1-f176.google.com (mail-pg1-f176.google.com [209.85.215.176]) by imf10.hostedemail.com (Postfix) with ESMTP id 1DA85C0003 for ; Thu, 27 Jan 2022 09:41:48 +0000 (UTC) Received: by mail-pg1-f176.google.com with SMTP id j10so1807084pgc.6 for ; Thu, 27 Jan 2022 01:41:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ajou.ac.kr; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=FwGYXa/4Ke/nl7w6rwwFT93Jqd/KRsceKw3bmpTizd0=; b=c7f7zsHi9kFYEI3DLIcZJNkTZ6mwWpT5jJzA3XquJWSc3nVDPyVjgD2LDaCRl2f49L E8jlbenDE0bp4V7semQXNIXUMfqgcbPy4QzrW1xm3E1yq4f80G0IKO2ZnQJGUKgAEqKN VXG0hltwq4W/ZVrupfCfQ+uI3sCzmcdEzRr4E= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=FwGYXa/4Ke/nl7w6rwwFT93Jqd/KRsceKw3bmpTizd0=; b=1fb1OA+CQbY6O9Nk/tOdIL4cVJf5V1tYDKryj6sfyeFYA53Oa3m6m/GjUsyHLVJavD g9C7nbwT5ycYo7g58HbELSFurz3uuIJfmymO1mmjBdUVN6FqqdkI6qwmJswLdPttoik5 zlMnJQ+q4oDOgirC8EWFXrNqAghe90WiiuavaOIpX+5UYMOQSvHiwDJ3BXfHetfUpbuu WkH0rfUKv7BGHhtD1daxaJqKSvPHyx6X6U2me7Kpo0pDIfkOuMFtcnvZao7arzwneGct 6gTGm90ezys3gp6uiajQGUPNs5q+D/SS4QwbrGDC3tHnK19qcUzHEP2t/EnnpyukOIqg rhJg== X-Gm-Message-State: AOAM530W2erUiH22UZq+8I8b76BmTAM5uRtjRX0DWwLS+l67YR+CKbrt AJKUDKGVFeM9TtS2seN1vCuZdQ== X-Google-Smtp-Source: ABdhPJyqApfUm7sSZ0dQD2FVycwOL7CDbEG0tcN66TiQouXhJL6YT0Y3ygdGR6HPOBbUNgizsWukrw== X-Received: by 2002:a63:d54:: with SMTP id 20mr2276891pgn.442.1643276507407; Thu, 27 Jan 2022 01:41:47 -0800 (PST) Received: from swarm08 ([210.107.197.32]) by smtp.gmail.com with ESMTPSA id t17sm5695302pgm.69.2022.01.27.01.41.44 (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 27 Jan 2022 01:41:46 -0800 (PST) Date: Thu, 27 Jan 2022 18:41:42 +0900 From: Jonghyeon Kim To: David Hildenbrand Cc: dan.j.williams@intel.com, vishal.l.verma@intel.com, dave.jiang@intel.com, akpm@linux-foundation.org, nvdimm@lists.linux.dev, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH 1/2] mm/memory_hotplug: Export shrink span functions for zone and node Message-ID: <20220127094142.GA31409@swarm08> References: <20220126170002.19754-1-tome01@ajou.ac.kr> <5d02ea0e-aca6-a64b-23de-bc9307572d17@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5d02ea0e-aca6-a64b-23de-bc9307572d17@redhat.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-Rspam-User: nil X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 1DA85C0003 X-Stat-Signature: 6m3d5udafmf3yucm9ambi8g13b5seoq3 Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=ajou.ac.kr header.s=google header.b=c7f7zsHi; spf=pass (imf10.hostedemail.com: domain of tome01@ajou.ac.kr designates 209.85.215.176 as permitted sender) smtp.mailfrom=tome01@ajou.ac.kr; dmarc=pass (policy=reject) header.from=ajou.ac.kr X-HE-Tag: 1643276508-945127 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jan 26, 2022 at 06:04:50PM +0100, David Hildenbrand wrote: > On 26.01.22 18:00, Jonghyeon Kim wrote: > > Export shrink_zone_span() and update_pgdat_span() functions to head > > file. We need to update real number of spanned pages for NUMA nodes and > > zones when we add memory device node such as device dax memory. > > > > Can you elaborate a bit more what you intend to fix? > > Memory onlining/offlining is reponsible for updating the node/zone span, > and that's triggered when the dax/kmem mamory gets onlined/offlined. > Sure, sorry for the lack of explanation of the intended fix. Before onlining nvdimm memory using dax(devdax or fsdax), these memory belong to cpu NUMA nodes, which extends span pages of node/zone as a ZONE_DEVICE. So there is no problem because node/zone contain these additional non-visible memory devices to the system. But, if we online dax-memory, zone[ZONE_DEVICE] of CPU NUMA node is hot-plugged to new NUMA node(but CPU-less). I think there is no need to hold zone[ZONE_DEVICE] pages on the original node. Additionally, spanned pages are also used to calculate the end pfn of a node. Thus, it is needed to maintain accurate page stats for node/zone. My machine contains two CPU-socket consisting of DRAM and Intel DCPMM (DC persistent memory modules) with App-Direct mode. Below are my test results. Before memory onlining: # ndctl create-namespace --mode=devdax # ndctl create-namespace --mode=devdax # cat /proc/zoneinfo | grep -E "Node|spanned" | paste - - Node 0, zone DMA spanned 4095 Node 0, zone DMA32 spanned 1044480 Node 0, zone Normal spanned 7864320 Node 0, zone Movable spanned 0 Node 0, zone Device spanned 66060288 Node 1, zone DMA spanned 0 Node 1, zone DMA32 spanned 0 Node 1, zone Normal spanned 8388608 Node 1, zone Movable spanned 0 Node 1, zone Device spanned 66060288 After memory onlining: # daxctl reconfigure-device --mode=system-ram --no-online dax0.0 # daxctl reconfigure-device --mode=system-ram --no-online dax1.0 # cat /proc/zoneinfo | grep -E "Node|spanned" | paste - - Node 0, zone DMA spanned 4095 Node 0, zone DMA32 spanned 1044480 Node 0, zone Normal spanned 7864320 Node 0, zone Movable spanned 0 Node 0, zone Device spanned 66060288 Node 1, zone DMA spanned 0 Node 1, zone DMA32 spanned 0 Node 1, zone Normal spanned 8388608 Node 1, zone Movable spanned 0 Node 1, zone Device spanned 66060288 Node 2, zone DMA spanned 0 Node 2, zone DMA32 spanned 0 Node 2, zone Normal spanned 65011712 Node 2, zone Movable spanned 0 Node 2, zone Device spanned 0 Node 3, zone DMA spanned 0 Node 3, zone DMA32 spanned 0 Node 3, zone Normal spanned 65011712 Node 3, zone Movable spanned 0 Node 3, zone Device spanned 0 As we can see, Node 0 and 1 still have zone_device pages after memory onlining. This causes problem that Node 0 and Node 2 have same end of pfn values, also Node 1 and Node 3 have same problem. > > Signed-off-by: Jonghyeon Kim > > --- > > include/linux/memory_hotplug.h | 3 +++ > > mm/memory_hotplug.c | 6 ++++-- > > 2 files changed, 7 insertions(+), 2 deletions(-) > > > > diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h > > index be48e003a518..25c7f60c317e 100644 > > --- a/include/linux/memory_hotplug.h > > +++ b/include/linux/memory_hotplug.h > > @@ -337,6 +337,9 @@ extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn, > > extern void remove_pfn_range_from_zone(struct zone *zone, > > unsigned long start_pfn, > > unsigned long nr_pages); > > +extern void shrink_zone_span(struct zone *zone, unsigned long start_pfn, > > + unsigned long end_pfn); > > +extern void update_pgdat_span(struct pglist_data *pgdat); > > extern bool is_memblock_offlined(struct memory_block *mem); > > extern int sparse_add_section(int nid, unsigned long pfn, > > unsigned long nr_pages, struct vmem_altmap *altmap); > > diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c > > index 2a9627dc784c..38f46a9ef853 100644 > > --- a/mm/memory_hotplug.c > > +++ b/mm/memory_hotplug.c > > @@ -389,7 +389,7 @@ static unsigned long find_biggest_section_pfn(int nid, struct zone *zone, > > return 0; > > } > > > > -static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, > > +void shrink_zone_span(struct zone *zone, unsigned long start_pfn, > > unsigned long end_pfn) > > { > > unsigned long pfn; > > @@ -428,8 +428,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, > > } > > } > > } > > +EXPORT_SYMBOL_GPL(shrink_zone_span); > > Exporting both as symbols feels very wrong. This is memory > onlining/offlining internal stuff. I agree with you that your comment. I will find another approach to avoid directly using onlining/offlining internal stuff while updating node/zone span. Thanks, Jonghyeon > > > > -- > Thanks, > > David / dhildenb >