From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FCE7C43460 for ; Fri, 9 Apr 2021 05:32:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ED3A1610C8 for ; Fri, 9 Apr 2021 05:32:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org ED3A1610C8 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7EDBE6B006C; Fri, 9 Apr 2021 01:32:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7C43C6B006E; Fri, 9 Apr 2021 01:32:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 664FE6B0070; Fri, 9 Apr 2021 01:32:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0091.hostedemail.com [216.40.44.91]) by kanga.kvack.org (Postfix) with ESMTP id 4591A6B006C for ; Fri, 9 Apr 2021 01:32:37 -0400 (EDT) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id EA4C1814A for ; Fri, 9 Apr 2021 05:32:36 +0000 (UTC) X-FDA: 78011708712.24.06D9223 Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) by imf22.hostedemail.com (Postfix) with ESMTP id 26249C0001FA for ; Fri, 9 Apr 2021 05:32:34 +0000 (UTC) Received: by mail-io1-f53.google.com with SMTP id d12so562441iod.12 for ; Thu, 08 Apr 2021 22:32:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=AxWZg8GIxRLBho/ausojU/sJO0xe78oHCVyalpIaIOk=; b=QOL4OZ1urFgNXve/8IvwqfMO5TWDKsDZuKsv0cLeHRxQyvM5/H4eSPB7H9TSVuZy6G E5sXuqnmOKmTV+VYoFJgdQXPIp5RQuO8h8skw+Y1c2PelpZzGSwT71Uh+rp+v0MQcUbx WwsdJ8csxIXJiCqm2kpIE+xTk/qQDgtrNVW4cWa5Yygf5dXW12/srme+G9amHoPUz8bL czFasIpo//7/39H//fsS0emmjm1VRSD7684CUr5WdEjn/VEAvO1BA1OhAUxdZY69eldC wbZFI19VWRNox6ayVJSN8RgRyO7NOCwctosx2swPNngNcQXwVQF1AeSKbPPea8hqvm97 mStw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=AxWZg8GIxRLBho/ausojU/sJO0xe78oHCVyalpIaIOk=; b=rJY073PzcLEu5gd0DxXSIfSxIyPe1KYBGF7z3vEXZdksAdUXXzuvEQu87QfIVQvSzy j2cdbei2qtjgHyRv1k3ILEkE9bO1NxnOfCbGKJYrgbCqqDnJZhX3IOOIkVYXwukBtZRD N9xr2LVg0DX7DfbWJcPX0VVUZJq50/jYGno0nYd5RfeR9DCgU6SZbORRvXpDEnpDO8bB dOALy6zanedHSNsell6hSHUdwmDdAH0qmsod9wKYsHkLbvN7cl0qPVqZz20WzIMLh2R+ 5RCTrNMUUU/juHci8c6Z9O/NThfUV2Mwxo3qqdP0sYleP9QZAEFb0NXS7y/21Y4d7AL0 7Now== X-Gm-Message-State: AOAM531dVhZVuuPEXF/01V9DcgoYDMGe/t991hE32wZNsIOfZG8tE4YV eGomRB5yX4BDUG7bmoUyq+nyFj+5fAPxt6rmo7xmNA== X-Google-Smtp-Source: ABdhPJweUM9c8DnJ7OuLnPeNxaLs4UVsS28KDeOhqoFpLF8v3R5OvLaSNa6cMuV1cBMLcryj+VtdUOzku111YltYcTI= X-Received: by 2002:a6b:661a:: with SMTP id a26mr9823930ioc.124.1617946355681; Thu, 08 Apr 2021 22:32:35 -0700 (PDT) MIME-Version: 1.0 References: <20210401183216.443C4443@viggo.jf.intel.com> <20210401183218.E7C9CE24@viggo.jf.intel.com> In-Reply-To: <20210401183218.E7C9CE24@viggo.jf.intel.com> From: Wei Xu Date: Thu, 8 Apr 2021 22:32:24 -0700 Message-ID: Subject: Re: [PATCH 01/10] mm/numa: node demotion data structure and lookup To: Dave Hansen Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, shy828301@gmail.com, David Rientjes , ying.huang@intel.com, Dan Williams , david@redhat.com, osalvador@suse.de Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 26249C0001FA X-Stat-Signature: cpnkjn9whtd9w1zhmtkfgezeh4d5stt9 Received-SPF: none (google.com>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=mail-io1-f53.google.com; client-ip=209.85.166.53 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617946354-873725 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It makes sense to start with a simple node tiering model like this change, which looks good to me. I do want to mention a likely usage scenario that motivates the need for a list of nodes as the demotion target of a source node. Access to a cross-socket DRAM node is still fast enough. So to minimize memory stranding, job processes can be allowed to fall back to allocate pages from a remote DRAM node. But cross-socket access to PMEM nodes (the slower tier) can be slow, especially for random writes. It is then desirable not to demote the pages of a process to a remote PMEM node, even when the pages are on a remote DRAM node, which has the remote PMEM node as its demotion target. At the same time, it is also desirable to still be able to demote such pages when they become cold so that the more precious DRAM occupied by these pages can be used for more active data. To support such use cases, we need to be able to specify a list of demotion target nodes for the remote DRAM node, which should include the PMEM node closer to the process. Certainly, we will also need an ability to limit the demotion target nodes of a process (or a cgroup) to ensure that only local PMEM nodes are eligible as the actual demotion target. Note that demoting a page to a remote PMEM node is more acceptable than a process accesses the same remote PMEM node because demotion is one-time, sequential access, and can also use non-temporal stores to reduce the access overheads and bypass caches. Reviewed-by: Wei Xu On Thu, Apr 1, 2021 at 11:35 AM Dave Hansen wrote: > > > From: Dave Hansen > > Prepare for the kernel to auto-migrate pages to other memory nodes > with a user defined node migration table. This allows creating single > migration target for each NUMA node to enable the kernel to do NUMA > page migrations instead of simply reclaiming colder pages. A node > with no target is a "terminal node", so reclaim acts normally there. > The migration target does not fundamentally _need_ to be a single node, > but this implementation starts there to limit complexity. > > If you consider the migration path as a graph, cycles (loops) in the > graph are disallowed. This avoids wasting resources by constantly > migrating (A->B, B->A, A->B ...). The expectation is that cycles will > never be allowed. > > Signed-off-by: Dave Hansen > Reviewed-by: Yang Shi > Cc: Wei Xu > Cc: David Rientjes > Cc: Huang Ying > Cc: Dan Williams > Cc: David Hildenbrand > Cc: osalvador > > -- > > changes since 20200122: > * Make node_demotion[] __read_mostly > > changes in July 2020: > - Remove loop from next_demotion_node() and get_online_mems(). > This means that the node returned by next_demotion_node() > might now be offline, but the worst case is that the > allocation fails. That's fine since it is transient. > --- > > b/mm/migrate.c | 17 +++++++++++++++++ > 1 file changed, 17 insertions(+) > > diff -puN mm/migrate.c~0006-node-Define-and-export-memory-migration-path mm/migrate.c > --- a/mm/migrate.c~0006-node-Define-and-export-memory-migration-path 2021-03-31 15:17:10.734000264 -0700 > +++ b/mm/migrate.c 2021-03-31 15:17:10.742000264 -0700 > @@ -1163,6 +1163,23 @@ out: > return rc; > } > > +static int node_demotion[MAX_NUMNODES] __read_mostly = > + {[0 ... MAX_NUMNODES - 1] = NUMA_NO_NODE}; > + > +/** > + * next_demotion_node() - Get the next node in the demotion path > + * @node: The starting node to lookup the next node > + * > + * @returns: node id for next memory node in the demotion path hierarchy > + * from @node; NUMA_NO_NODE if @node is terminal. This does not keep > + * @node online or guarantee that it *continues* to be the next demotion > + * target. > + */ > +int next_demotion_node(int node) > +{ > + return node_demotion[node]; > +} > + > /* > * Obtain the lock on page, remove all ptes and migrate the page > * to the newly allocated page in newpage. > _