From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5775DC47087 for ; Fri, 28 May 2021 12:37:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EC268610A0 for ; Fri, 28 May 2021 12:37:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EC268610A0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6F4EE6B0036; Fri, 28 May 2021 08:37:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CA7C6B006E; Fri, 28 May 2021 08:37:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 593806B0070; Fri, 28 May 2021 08:37:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0131.hostedemail.com [216.40.44.131]) by kanga.kvack.org (Postfix) with ESMTP id 21DC16B0036 for ; Fri, 28 May 2021 08:37:27 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 241843AA7 for ; Fri, 28 May 2021 12:37:26 +0000 (UTC) X-FDA: 78190590492.34.5FE8F1B Received: from outbound-smtp01.blacknight.com (outbound-smtp01.blacknight.com [81.17.249.7]) by imf25.hostedemail.com (Postfix) with ESMTP id 2C3A360002CC for ; Fri, 28 May 2021 12:37:16 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail06.blacknight.ie [81.17.255.152]) by outbound-smtp01.blacknight.com (Postfix) with ESMTPS id A5A35C4A37 for ; Fri, 28 May 2021 13:37:23 +0100 (IST) Received: (qmail 8040 invoked from network); 28 May 2021 12:37:23 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[84.203.23.168]) by 81.17.254.9 with ESMTPSA (AES256-SHA encrypted, authenticated); 28 May 2021 12:37:23 -0000 Date: Fri, 28 May 2021 13:37:22 +0100 From: Mel Gorman To: Vlastimil Babka Cc: Dave Hansen , Andrew Morton , Hillf Danton , Dave Hansen , Michal Hocko , LKML , Linux-MM , "Tang, Feng" Subject: Re: [PATCH 0/6 v2] Calculate pcp->high based on zone sizes and active CPUs Message-ID: <20210528123721.GO30378@techsingularity.net> References: <20210525080119.5455-1-mgorman@techsingularity.net> <7177f59b-dc05-daff-7dc6-5815b539a790@intel.com> <20210528085545.GJ30378@techsingularity.net> <416f39e7-704a-86d0-8261-dc27366336ab@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <416f39e7-704a-86d0-8261-dc27366336ab@suse.cz> User-Agent: Mutt/1.10.1 (2018-07-13) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf25.hostedemail.com: domain of mgorman@techsingularity.net designates 81.17.249.7 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-Stat-Signature: tprwkmmkdi31yaj3nakcumyenw6mzejq X-Rspamd-Queue-Id: 2C3A360002CC X-Rspamd-Server: rspam02 X-HE-Tag: 1622205436-54286 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, May 28, 2021 at 02:12:09PM +0200, Vlastimil Babka wrote: > > mm/page_alloc: Split pcp->high across all online CPUs for cpuless nodes > > > > Dave Hansen reported the following about Feng Tang's tests on a machine > > with persistent memory onlined as a DRAM-like device. > > > > Feng Tang tossed these on a "Cascade Lake" system with 96 threads and > > ~512G of persistent memory and 128G of DRAM. The PMEM is in "volatile > > use" mode and being managed via the buddy just like the normal RAM. > > > > The PMEM zones are big ones: > > > > present 65011712 = 248 G > > high 134595 = 525 M > > > > The PMEM nodes, of course, don't have any CPUs in them. > > > > With your series, the pcp->high value per-cpu is 69584 pages or about > > 270MB per CPU. Scaled up by the 96 CPU threads, that's ~26GB of > > worst-case memory in the pcps per zone, or roughly 10% of the size of > > the zone. > > > > This should not cause a problem as such although it could trigger reclaim > > due to pages being stored on per-cpu lists for CPUs remote to a node. It > > is not possible to treat cpuless nodes exactly the same as normal nodes > > but the worst-case scenario can be mitigated by splitting pcp->high across > > all online CPUs for cpuless memory nodes. > > > > Suggested-by: Dave Hansen > > Signed-off-by: Mel Gorman > > Acked-by: Vlastimil Babka > Thanks. > Maybe we should even consider distinguishing high limits for local-to-cpu zones > vs remote, for example for the local-to-cpu zones we would divide by the number > of local cpus, for remote-to-cpu zones we would divide by all cpus. > > Because we can expect cpus to allocate mostly from local zones, so leaving more > pages on percpu for those zones can be beneficial. > I did think about whether the ratios should be different but failed to conclude that it was necessary or useful so I kept it simple. > But as the motivation here was to reduce lock contention on freeing, that's less > clear. We probably can't expect the cpu to be freeing mostly local pages (in > case of e.g. a large process exiting), because no mechanism works towards that, > or does it? In case of cpu freeing to remote zone, the lower high limit could hurt. > This is the major issue. Even if an application was NUMA aware and heavily threaded, the process exiting is potentially freeing remote memory and there is nothing wrong about that. The remote memory will be partially drained by pcp->high being reached and the remaining memory will be cleaned up by vmstat. It's a similar problem if a process is truncating a large file with page cache allocated on a remote node. Hence I decided to do nothing fancy with the ratios until a practical problem was identified that could be alleviated by adjusting pcp->high based on whether the CPU is remote or local to memory. -- Mel Gorman SUSE Labs