From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E61B4C433F5 for ; Fri, 6 May 2022 19:05:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 73CCA6B0071; Fri, 6 May 2022 15:05:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6EAFA6B0073; Fri, 6 May 2022 15:05:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58CE36B0074; Fri, 6 May 2022 15:05:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4952D6B0071 for ; Fri, 6 May 2022 15:05:30 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0C87F20D7A for ; Fri, 6 May 2022 19:05:30 +0000 (UTC) X-FDA: 79436246820.21.03D73E8 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf04.hostedemail.com (Postfix) with ESMTP id 582774003A for ; Fri, 6 May 2022 19:05:21 +0000 (UTC) Received: by mail-pl1-f169.google.com with SMTP id c11so8289483plg.13 for ; Fri, 06 May 2022 12:05:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=XY52lokW3Gz896dORyk3XbXtZ36Z+nGvhDg4ZfkIyOw=; b=nrZYCL9vNGmBRbVEBTZhmaePhgNibPPAvI1Ko/g0AOz0bOG4W9sIEtGysQh3cEmzBs zXGxN0uMR+Orc2tDBLFiEfS/yb48EqYdNmEE6CwZHTJpjavlfVk9w4BiZRSR+0VzdKBK 28QbwtVSMjLavuMay9vGRa81DfVb3cv8CgOl6vdELBAyk7WWjv7fBq8I3tWaoRzvDgJk pZ95y0n6N4qBhzWRiTQnMFQv+ZTFXbFjxAb1CNW7Db4sSyEiNS13b6kPAkrH+ReLf+g8 +YQybLKwPlyDZ00e6pYM00rWFSOaf2FHX1h8c0jLr0wNxhYjn8UyQ2nszC6707Kej2Gh GzxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=XY52lokW3Gz896dORyk3XbXtZ36Z+nGvhDg4ZfkIyOw=; b=ZdPHHnBPfN4n4G6e3GizczYGyxAkDrnfYrMOaOBRolo9cSE3lSsWJrt2O6O/VcIzMt nYuGDqyI+MtpnTqyVS4HtlJyclZg6PFyi1bUf8crytZ5ichLk4FCIvbeP82CoqBibThT 8+f4wLuTuGrpQmj/A1cjI5jxurrURZRJ2JE+6AibNeZZa0bZion7CFd0PiI1U7EjmACe O3jKnyM8rXvIWbouKZLJgZJpHOWINcazCfwnrYyjpgXTYrprGyRZIgrx9CI90pOBqFW+ 3KzsG107L5vMvy+LIKno2LqCB3ZUGXiwLnCshXyBiokwDEq231xXsSAWr6BN0r1MXz8M eD3A== X-Gm-Message-State: AOAM530mtGsJyKBtyHRp5YtjJe+iZ5jUpyzeMT9SMYvapc1iBkhdDo0R x6YYFGhSy4sg1U8X2sntH7oiGnjV2f1lFnRxngLOkmUa X-Google-Smtp-Source: ABdhPJyd+cRv1GOJhxyYaGhWmVSwV+3xawEPq8StjvWp2gy3AptjgQf6GqaQLBqehGf7+oR/DcvFTkbbqoE8JKCoGXY= X-Received: by 2002:a17:902:e851:b0:15e:93ac:41db with SMTP id t17-20020a170902e85100b0015e93ac41dbmr5065327plg.26.1651863928520; Fri, 06 May 2022 12:05:28 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Yang Shi Date: Fri, 6 May 2022 12:05:16 -0700 Message-ID: Subject: Re: RFC: Memory Tiering Kernel Interfaces To: Dan Williams Cc: Wei Xu , Andrew Morton , Dave Hansen , Huang Ying , Linux MM , Greg Thelen , "Aneesh Kumar K.V" , Jagdish Gediya , Linux Kernel Mailing List , Alistair Popple , Davidlohr Bueso , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Jonathan Cameron Content-Type: text/plain; charset="UTF-8" Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=nrZYCL9v; spf=pass (imf04.hostedemail.com: domain of shy828301@gmail.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 582774003A X-Stat-Signature: ydgisgxpw5nig97zuoh598n6jbtkmkic X-HE-Tag: 1651863921-443523 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sun, May 1, 2022 at 11:35 AM Dan Williams wrote: > > On Fri, Apr 29, 2022 at 8:59 PM Yang Shi wrote: > > > > Hi Wei, > > > > Thanks for the nice writing. Please see the below inline comments. > > > > On Fri, Apr 29, 2022 at 7:10 PM Wei Xu wrote: > > > > > > The current kernel has the basic memory tiering support: Inactive > > > pages on a higher tier NUMA node can be migrated (demoted) to a lower > > > tier NUMA node to make room for new allocations on the higher tier > > > NUMA node. Frequently accessed pages on a lower tier NUMA node can be > > > migrated (promoted) to a higher tier NUMA node to improve the > > > performance. > > > > > > A tiering relationship between NUMA nodes in the form of demotion path > > > is created during the kernel initialization and updated when a NUMA > > > node is hot-added or hot-removed. The current implementation puts all > > > nodes with CPU into the top tier, and then builds the tiering hierarchy > > > tier-by-tier by establishing the per-node demotion targets based on > > > the distances between nodes. > > > > > > The current memory tiering interface needs to be improved to address > > > several important use cases: > > > > > > * The current tiering initialization code always initializes > > > each memory-only NUMA node into a lower tier. But a memory-only > > > NUMA node may have a high performance memory device (e.g. a DRAM > > > device attached via CXL.mem or a DRAM-backed memory-only node on > > > a virtual machine) and should be put into the top tier. > > > > > > * The current tiering hierarchy always puts CPU nodes into the top > > > tier. But on a system with HBM (e.g. GPU memory) devices, these > > > memory-only HBM NUMA nodes should be in the top tier, and DRAM nodes > > > with CPUs are better to be placed into the next lower tier. > > > > > > * Also because the current tiering hierarchy always puts CPU nodes > > > into the top tier, when a CPU is hot-added (or hot-removed) and > > > triggers a memory node from CPU-less into a CPU node (or vice > > > versa), the memory tiering hierarchy gets changed, even though no > > > memory node is added or removed. This can make the tiering > > > hierarchy much less stable. > > > > I'd prefer the firmware builds up tiers topology then passes it to > > kernel so that kernel knows what nodes are in what tiers. No matter > > what nodes are hot-removed/hot-added they always stay in their tiers > > defined by the firmware. I think this is important information like > > numa distances. NUMA distance alone can't satisfy all the usecases > > IMHO. > > Just want to note here that the platform firmware can only describe > the tiers of static memory present at boot. CXL hotplug breaks this > model and the kernel is left to dynamically determine the device's > performance characteristics and the performance of the topology to > reach that device. Now, the platform firmware does set expectations > for the perfomance class of different memory ranges, but there is no > way to know in advance the performance of devices that will be asked > to be physically or logically added to the memory configuration. That > said, it's probably still too early to define ABI for those > exceptional cases where the kernel needs to make a policy decision > about a device that does not fit into the firmware's performance > expectations, but just note that there are limits to the description > that platform firmware can provide. Thanks, Dan. I don't know too much about CXL. Is it possible to make it static? For example, put it into a default tier (for example, the lowest tier) as long as CXL is available regardless of whether there is any device connected or not? Then the kernel driver could probe some information and move it to the proper tier once the device is hot plugged. Anyway, just off the top of my head. > > I agree that NUMA distance alone is inadequate and the kernel needs to > make better use of data like ACPI HMAT to determine the default > tiering order. Yeah, we are on the same page.