From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B031C433EF for ; Sun, 1 May 2022 18:35:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FC6A6B0072; Sun, 1 May 2022 14:35:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4AB946B0073; Sun, 1 May 2022 14:35:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3741E6B0074; Sun, 1 May 2022 14:35:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 277046B0072 for ; Sun, 1 May 2022 14:35:15 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 036DB1163 for ; Sun, 1 May 2022 18:35:14 +0000 (UTC) X-FDA: 79418026590.29.459A389 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf30.hostedemail.com (Postfix) with ESMTP id 2D1588007D for ; Sun, 1 May 2022 18:35:03 +0000 (UTC) Received: by mail-pj1-f51.google.com with SMTP id r9so11072000pjo.5 for ; Sun, 01 May 2022 11:35:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel-com.20210112.gappssmtp.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gEKLzHDmrjf1i9KSUaOozCHBh8PDFjk1Tl8GhCKHgdo=; b=GxjehPULVcq/fzKtlONcVsDXlSml+007HBNIVAwS5tLyal/+nGIbMJGqjftZQyrLM6 mew7Gtz7zhmlquBaV/NkgJdIGMHy5wyStoFxN7mP7wPcGWm0MtIlqYpFF0dfoX9Mu8KX XravQI9uPa5r5MKHo8eRYgVrvYeB1d5TbQOvU/7G+UYwEBn8EW6qXUTjBqk43FOQeAY/ S38y1+uf4Wby0yDtmPgP5aulnqnGMJfT5cDnJQMqEJTRuPEtzw5i+2bHj+Kq2ANPcig5 2hfPFka0LVMcH5y5cE9zac5UYwHiA2HWn6OdAtYIrD6WeyzopQubvMyeVKa4iSncTY8R 1VQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gEKLzHDmrjf1i9KSUaOozCHBh8PDFjk1Tl8GhCKHgdo=; b=oE2VPLX+KaY/0J6bbUdFx1mAg2HODYQ5iJd6Am870sx9G6QoehTrS/HvZ5k9tSrg2x Ey9iwSJEohbe/y49EDe8BDfplXRl8RDtqSnBdUU2sowe/BhmimduUfVnaDUDIQFwctK1 3QBiqSYmjEQaMk7+LA11U8CYFlRDjvOynbS7RAzkoRCXziFUhV+Wpl3FJLPa/uQMdujl kgLh9Vi7H9VMusGr0HXOOqQNC+XFPxsf3G3JthzYvuP6cwKyUKv6bn9sxNTNr393bcka ShQTE9aUHHiEKvATLbhn5tC5zfJpcPU0gGyEt2EzE3wn1gkO+JouU3a8Al98qy9AcIaq Nygw== X-Gm-Message-State: AOAM5322WIUraXyg0AD+NoNTNmb5zMlpUrus5QLxD85HRohtDnZvgiTk Pct5lGhY7T+8Cbe9ylyx5u7YTESdsl6kSxwEfNRvMA== X-Google-Smtp-Source: ABdhPJzvrBGis6Hl4oe4MkPWYo2Y5jmHdVB2zp8hDYW2UDFqwPAHi080eN8/7jY9cZKpmq6SOt/oJFoLclRRU05e/UE= X-Received: by 2002:a17:90b:4b01:b0:1d2:abf5:c83f with SMTP id lx1-20020a17090b4b0100b001d2abf5c83fmr9201434pjb.93.1651430112699; Sun, 01 May 2022 11:35:12 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Dan Williams Date: Sun, 1 May 2022 11:35:01 -0700 Message-ID: Subject: Re: RFC: Memory Tiering Kernel Interfaces To: Yang Shi Cc: Wei Xu , Andrew Morton , Dave Hansen , Huang Ying , Linux MM , Greg Thelen , "Aneesh Kumar K.V" , Jagdish Gediya , Linux Kernel Mailing List , Alistair Popple , Davidlohr Bueso , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Jonathan Cameron Content-Type: text/plain; charset="UTF-8" Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=intel-com.20210112.gappssmtp.com header.s=20210112 header.b=GxjehPUL; spf=none (imf30.hostedemail.com: domain of dan.j.williams@intel.com has no SPF policy when checking 209.85.216.51) smtp.mailfrom=dan.j.williams@intel.com; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=intel.com (policy=none) X-Rspam-User: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 2D1588007D X-Stat-Signature: 6mcop4karqy6zp6sp7pztkmpezfq65e1 X-HE-Tag: 1651430103-36365 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Apr 29, 2022 at 8:59 PM Yang Shi wrote: > > Hi Wei, > > Thanks for the nice writing. Please see the below inline comments. > > On Fri, Apr 29, 2022 at 7:10 PM Wei Xu wrote: > > > > The current kernel has the basic memory tiering support: Inactive > > pages on a higher tier NUMA node can be migrated (demoted) to a lower > > tier NUMA node to make room for new allocations on the higher tier > > NUMA node. Frequently accessed pages on a lower tier NUMA node can be > > migrated (promoted) to a higher tier NUMA node to improve the > > performance. > > > > A tiering relationship between NUMA nodes in the form of demotion path > > is created during the kernel initialization and updated when a NUMA > > node is hot-added or hot-removed. The current implementation puts all > > nodes with CPU into the top tier, and then builds the tiering hierarchy > > tier-by-tier by establishing the per-node demotion targets based on > > the distances between nodes. > > > > The current memory tiering interface needs to be improved to address > > several important use cases: > > > > * The current tiering initialization code always initializes > > each memory-only NUMA node into a lower tier. But a memory-only > > NUMA node may have a high performance memory device (e.g. a DRAM > > device attached via CXL.mem or a DRAM-backed memory-only node on > > a virtual machine) and should be put into the top tier. > > > > * The current tiering hierarchy always puts CPU nodes into the top > > tier. But on a system with HBM (e.g. GPU memory) devices, these > > memory-only HBM NUMA nodes should be in the top tier, and DRAM nodes > > with CPUs are better to be placed into the next lower tier. > > > > * Also because the current tiering hierarchy always puts CPU nodes > > into the top tier, when a CPU is hot-added (or hot-removed) and > > triggers a memory node from CPU-less into a CPU node (or vice > > versa), the memory tiering hierarchy gets changed, even though no > > memory node is added or removed. This can make the tiering > > hierarchy much less stable. > > I'd prefer the firmware builds up tiers topology then passes it to > kernel so that kernel knows what nodes are in what tiers. No matter > what nodes are hot-removed/hot-added they always stay in their tiers > defined by the firmware. I think this is important information like > numa distances. NUMA distance alone can't satisfy all the usecases > IMHO. Just want to note here that the platform firmware can only describe the tiers of static memory present at boot. CXL hotplug breaks this model and the kernel is left to dynamically determine the device's performance characteristics and the performance of the topology to reach that device. Now, the platform firmware does set expectations for the perfomance class of different memory ranges, but there is no way to know in advance the performance of devices that will be asked to be physically or logically added to the memory configuration. That said, it's probably still too early to define ABI for those exceptional cases where the kernel needs to make a policy decision about a device that does not fit into the firmware's performance expectations, but just note that there are limits to the description that platform firmware can provide. I agree that NUMA distance alone is inadequate and the kernel needs to make better use of data like ACPI HMAT to determine the default tiering order.