From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CAE0C433EF for ; Tue, 3 May 2022 06:06:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 454566B0071; Tue, 3 May 2022 02:06:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4046E6B0073; Tue, 3 May 2022 02:06:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CB5F6B0074; Tue, 3 May 2022 02:06:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.25]) by kanga.kvack.org (Postfix) with ESMTP id 1B0E96B0071 for ; Tue, 3 May 2022 02:06:38 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D9C252BDA9 for ; Tue, 3 May 2022 06:06:37 +0000 (UTC) X-FDA: 79423397634.05.C0C6BC3 Received: from mail-vk1-f179.google.com (mail-vk1-f179.google.com [209.85.221.179]) by imf25.hostedemail.com (Postfix) with ESMTP id 6CB3CA0081 for ; Tue, 3 May 2022 06:06:25 +0000 (UTC) Received: by mail-vk1-f179.google.com with SMTP id bc42so7507954vkb.12 for ; Mon, 02 May 2022 23:06:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=7N4xc/wMzuulecI04es1Ttu3ntRturBloFa549S0Y3Q=; b=NEVrVzg9Pe7LgLukiVkco8UDTV6L59oHrngUV4vIQi1UdW9T5lIL78M4V4Q8qQKIFJ DLHLChcTxDhzAcdGXIh5iSm1rZ7eJRQXd9My+REEAGmqWRKjHa2lyUXfN3+2N4UVX8Fq GeujgEZzSMb6mfbE2Wf/objpyHuRUunvrk/NmpQq9qySmCUdy7VVudW3tWxLPdB4zlwo S8V+p2TevVjqVcURU/nIg9m8fDH7xx7B/vg+nv5XR7UFULjcv7aWGiBl8oGFuVc1ZkqL voxnEbtK36gSGYTkwYM92/5J0q9xcferwd12sIh4tg08NKsnaQ2SoD7+ougi/LEF3T59 56yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=7N4xc/wMzuulecI04es1Ttu3ntRturBloFa549S0Y3Q=; b=WtByNFjOBNUEG9rveHLmGWb8iD9Dyp6jfmnxlvAtgIOqrKhOcMkrJpm9+u425zycTN 3tzgS8JdqEqhHMj3sNhK65WqW+SIuelgZW/J7dPIYkju+UCij/lcvhU7UKXzHk83QUeI ePUu+mEAFOWS7o+iBl+xZPr8Fuhov5lwgpvR5owuEDnX7G7xa4uw4ZfQxZn4/rq0DyVU 9xW3mk5Jx3SF3azzWZJs2oG4umnwWkNmmJ07kpdzZ4WUGr2VS+FUyiPGckDTySsJ8V5v pvGDyZhsiSV5ACo0IGI8qKKssSDDUd/SEGDNPd4jNyEcFqS+12s0W6YLn2kZ9T4rYbgk ZAtQ== X-Gm-Message-State: AOAM5319thGoh99TfA+ked8K6A1L9gpK59t5rPXQKphP2YUVsGtazj3a c6vViXzSaINsf6vQgTsofbicx5tQzrUxhDWcuSFZRQ== X-Google-Smtp-Source: ABdhPJybo+d0kbBuduaYftUo3seNL4tZkqegmoofOoOd+SP9ml8MIN+ga2yGL+UEog18KotV0y6a+Lszzi+VSenrjx0= X-Received: by 2002:a1f:9b85:0:b0:32d:4d56:cf64 with SMTP id d127-20020a1f9b85000000b0032d4d56cf64mr4410925vke.31.1651557996716; Mon, 02 May 2022 23:06:36 -0700 (PDT) MIME-Version: 1.0 References: <20220501175813.tvytoosygtqlh3nn@offworld> In-Reply-To: <20220501175813.tvytoosygtqlh3nn@offworld> From: Wei Xu Date: Mon, 2 May 2022 23:06:25 -0700 Message-ID: Subject: Re: RFC: Memory Tiering Kernel Interfaces To: Wei Xu , Andrew Morton , Dave Hansen , Huang Ying , Dan Williams , Yang Shi , Linux MM , Greg Thelen , "Aneesh Kumar K.V" , Jagdish Gediya , Linux Kernel Mailing List , Alistair Popple , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Jonathan.Cameron@huawei.com Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: 5t1b7sgi97fed48ffr4t5gk77kmohwwu Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=NEVrVzg9; spf=pass (imf25.hostedemail.com: domain of weixugc@google.com designates 209.85.221.179 as permitted sender) smtp.mailfrom=weixugc@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 6CB3CA0081 X-HE-Tag: 1651557985-279234 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Sun, May 1, 2022 at 11:09 AM Davidlohr Bueso wrote: > > Nice summary, thanks. I don't know who of the interested parties will be > at lsfmm, but fyi we have a couple of sessions on memory tiering Tuesday > at 14:00 and 15:00. > > On Fri, 29 Apr 2022, Wei Xu wrote: > > >The current kernel has the basic memory tiering support: Inactive > >pages on a higher tier NUMA node can be migrated (demoted) to a lower > >tier NUMA node to make room for new allocations on the higher tier > >NUMA node. Frequently accessed pages on a lower tier NUMA node can be > >migrated (promoted) to a higher tier NUMA node to improve the > >performance. > > Regardless of the promotion algorithm, at some point I see the NUMA hinting > fault mechanism being in the way of performance. It would be nice if hardware > began giving us page "heatmaps" instead of having to rely on faulting or > sampling based ways to identify hot memory. I agree with your comments on both NUMA hinting faults and hardware-assisted "heatmaps". > >A tiering relationship between NUMA nodes in the form of demotion path > >is created during the kernel initialization and updated when a NUMA > >node is hot-added or hot-removed. The current implementation puts all > >nodes with CPU into the top tier, and then builds the tiering hierarchy > >tier-by-tier by establishing the per-node demotion targets based on > >the distances between nodes. > > > >The current memory tiering interface needs to be improved to address > >several important use cases: > > > >* The current tiering initialization code always initializes > > each memory-only NUMA node into a lower tier. But a memory-only > > NUMA node may have a high performance memory device (e.g. a DRAM > > device attached via CXL.mem or a DRAM-backed memory-only node on > > a virtual machine) and should be put into the top tier. > > At least the CXL memory (volatile or not) will still be slower than > regular DRAM, so I think that we'd not want this to be top-tier. But > in general, yes I agree that defining top tier as whether or not the > node has a CPU a bit limiting, as you've detailed here. > > >Tiering Hierarchy Initialization > >================================ > > > >By default, all memory nodes are in the top tier (N_TOPTIER_MEMORY). > > > >A device driver can remove its memory nodes from the top tier, e.g. > >a dax driver can remove PMEM nodes from the top tier. > > > >The kernel builds the memory tiering hierarchy and per-node demotion > >order tier-by-tier starting from N_TOPTIER_MEMORY. For a node N, the > >best distance nodes in the next lower tier are assigned to > >node_demotion[N].preferred and all the nodes in the next lower tier > >are assigned to node_demotion[N].allowed. > > > >node_demotion[N].preferred can be empty if no preferred demotion node > >is available for node N. > > Upon cases where there more than one possible demotion node (with equal > cost), I'm wondering if we want to do something better than choosing > randomly, like we do now - perhaps round robin? Of course anything > like this will require actual performance data, something I have seen > very little of. I'd prefer that the demotion node selection follows the way how the kernel selects the node/zone for normal allocations. If we want to group several demotion nodes with equal cost together (e.g. to better utilize the bandwidth from these nodes), we'd better to improve such an optimization in __alloc_pages_nodemask() to benefit normal allocations as well. > >Memory tiering hierarchy is rebuilt upon hot-add or hot-remove of a > >memory node, but is NOT rebuilt upon hot-add or hot-remove of a CPU > >node. > > I think this makes sense. > > Thanks, > Davidlohr