From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=3.0 tests=BAYES_00, FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAE1BC433E0 for ; Fri, 5 Feb 2021 07:53:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CC5FA64F8C for ; Fri, 5 Feb 2021 07:53:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CC5FA64F8C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=sina.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3C69B6B0005; Fri, 5 Feb 2021 02:53:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 350C16B006C; Fri, 5 Feb 2021 02:53:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23EC86B006E; Fri, 5 Feb 2021 02:53:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0064.hostedemail.com [216.40.44.64]) by kanga.kvack.org (Postfix) with ESMTP id 0B68B6B0005 for ; Fri, 5 Feb 2021 02:53:38 -0500 (EST) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id BEE2D180AD81D for ; Fri, 5 Feb 2021 07:53:37 +0000 (UTC) X-FDA: 77783449674.26.help84_45179a6275e3 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 9D4041804B669 for ; Fri, 5 Feb 2021 07:53:37 +0000 (UTC) X-HE-Tag: help84_45179a6275e3 X-Filterd-Recvd-Size: 4894 Received: from mail3-164.sinamail.sina.com.cn (mail3-164.sinamail.sina.com.cn [202.108.3.164]) by imf21.hostedemail.com (Postfix) with SMTP for ; Fri, 5 Feb 2021 07:53:35 +0000 (UTC) Received: from unknown (HELO localhost.localdomain)([103.193.190.174]) by sina.com (172.16.97.35) with ESMTP id 601CF9700001E91E; Fri, 5 Feb 2021 15:53:28 +0800 (CST) X-Sender: hdanton@sina.com X-Auth-ID: hdanton@sina.com X-SMAIL-MID: 1369915073431 From: Hillf Danton To: Huang Ying Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Mel Gorman , Dave Hansen Subject: Re: [RFC -V5 1/6] NUMA balancing: optimize page placement for memory tiering system Date: Fri, 5 Feb 2021 15:53:12 +0800 Message-Id: <20210205075312.2515-1-hdanton@sina.com> In-Reply-To: <20210204101056.89336-2-ying.huang@intel.com> References: <20210204101056.89336-1-ying.huang@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, 4 Feb 2021 18:10:51 +0800 Huang Ying wrote: > With the advent of various new memory types, some machines will have > multiple types of memory, e.g. DRAM and PMEM (persistent memory). The > memory subsystem of these machines can be called memory tiering > system, because the performance of the different types of memory are > usually different. >=20 > In such system, because of the memory accessing pattern changing etc, > some pages in the slow memory may become hot globally. So in this > patch, the NUMA balancing mechanism is enhanced to optimize the page > placement among the different memory types according to hot/cold > dynamically. >=20 > In a typical memory tiering system, there are CPUs, fast memory and > slow memory in each physical NUMA node. The CPUs and the fast memory > will be put in one logical node (called fast memory node), while the > slow memory will be put in another (faked) logical node (called slow > memory node). That is, the fast memory is regarded as local while the > slow memory is regarded as remote. So it's possible for the recently > accessed pages in the slow memory node to be promoted to the fast > memory node via the existing NUMA balancing mechanism. >=20 > The original NUMA balancing mechanism will stop to migrate pages if the= free > memory of the target node will become below the high watermark. This > is a reasonable policy if there's only one memory type. But this > makes the original NUMA balancing mechanism almost not work to optimize= page > placement among different memory types. Details are as follows. >=20 > It's the common cases that the working-set size of the workload is > larger than the size of the fast memory nodes. Otherwise, it's > unnecessary to use the slow memory at all. So in the common cases, > there are almost always no enough free pages in the fast memory nodes, > so that the globally hot pages in the slow memory node cannot be In assumption like 1/ the workload's working set size is 1.5x larger than one DRAM node, 2/ PMEM is 10x (or 5x) larger than DRAM, what difference is it going to make if the spinning hard disk swap can be replaced with PMEM? With PMEM swap, the page demotion is swapout and we will pay nothing for page promotion. > promoted to the fast memory node. To solve the issue, we have 2 > choices as follows, >=20 > a. Ignore the free pages watermark checking when promoting hot pages > from the slow memory node to the fast memory node. This will > create some memory pressure in the fast memory node, thus trigger > the memory reclaiming. So that, the cold pages in the fast memory > node will be demoted to the slow memory node. >=20 > b. Make kswapd of the fast memory node to reclaim pages until the free > pages are a little more (about 10MB) than the high watermark. Then, > if the free pages of the fast memory node reaches high watermark, an= d > some hot pages need to be promoted, kswapd of the fast memory node > will be waken up to demote some cold pages in the fast memory node t= o > the slow memory node. This will free some extra space in the fast > memory node, so the hot pages in the slow memory node can be > promoted to the fast memory node. >=20 > The choice "a" will create the memory pressure in the fast memory > node. If the memory pressure of the workload is high, the memory > pressure may become so high that the memory allocation latency of the > workload is influenced, e.g. the direct reclaiming may be triggered. >=20 > The choice "b" works much better at this aspect. If the memory > pressure of the workload is high, the hot pages promotion will stop > earlier because its allocation watermark is higher than that of the > normal memory allocation. So in this patch, choice "b" is > implemented. >=20