From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DE5BC433F5 for ; Wed, 11 May 2022 05:42:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9CE386B0073; Wed, 11 May 2022 01:42:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9568F6B0075; Wed, 11 May 2022 01:42:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F6296B0078; Wed, 11 May 2022 01:42:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6D9006B0073 for ; Wed, 11 May 2022 01:42:43 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3F3F121E68 for ; Wed, 11 May 2022 05:42:43 +0000 (UTC) X-FDA: 79452367806.04.D4BC0F1 Received: from mail-ua1-f45.google.com (mail-ua1-f45.google.com [209.85.222.45]) by imf15.hostedemail.com (Postfix) with ESMTP id 9EB15A009A for ; Wed, 11 May 2022 05:42:30 +0000 (UTC) Received: by mail-ua1-f45.google.com with SMTP id z15so373144uad.7 for ; Tue, 10 May 2022 22:42:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=vGxPmJDM+cvx1KDtqTZS+UTdKLlUMP0zI7c+wSVmsgY=; b=MYGn0ZQ3a43eA+HINVZpZPpyE4SoDYREVL72759FD4eYh5rz+qlhLlfCmgCWrpcmBi evBpYgzn2guWEhSpqz6CJ98/Ltf4fUmuFSfsW6iDEMBcu8gvPaRykbJmS19i78/kBMAR uU22AwG7xDYnFhs+qFjmYzrgMM8gBmdaGmY9llctP9UQpK5dE9uQlFxkMDWj1TbPgeBM InNWP1EFZTG6Ech3/sS5J/POD/aDZ8JWWP/HBAS5KrHoTTRapVEawmHJrWAq8crPfzAn +e/mi1ounO+JP2PYmP74INHWma6QVm8Ay92juLa3gq7ne6QxfS8gE5NVXCMhh3XDto8V VD1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=vGxPmJDM+cvx1KDtqTZS+UTdKLlUMP0zI7c+wSVmsgY=; b=kn+T94h9Dk1uvnD5FYdQuJsZ+b47borxkcBkyBEYStyK/xhakJMHhD5Z9kcFUep16L r52sHuefypXUM/N/svVBQ/uj8HCoBPskoCkqnC8bI+usBdp1loP0UIk4p/tPsWxHXqc+ WJLbIEH5yxUnlub0cluhk2T/Aw72Bawuj0ZgyT9as9HAW+BmIbtpdKVdVwvZHSyuxTg0 L6hH2Ya3Tv98Wmv2WuLyO/igKjd45yJogYYo8Rw/MkekEE1OGhwXD7FmsSjVlGcDq5rd MwB9f9xrBhjXHW1O0HVFAVmUnmNuXcx1jPd9noVDhuZ24gC6qJZB/OXpSZPF3XzJanII wS3g== X-Gm-Message-State: AOAM5324LdzmZX3ktJj/nOGOteM2BwB46/hm1eRtLMApA+FjFwk17l3j rtF7d4hxyIBu4NvcDiPPa2fA2J2OsAezZWhXLgp+TQ== X-Google-Smtp-Source: ABdhPJzFPFltZtUFX4vhtGVjtORhdG+McAS0oa6PaWPDH5huwZItLsFCY18KX/JXOqEtg34hf3vp8BO60wHDIsdtDg4= X-Received: by 2002:ab0:349a:0:b0:35c:b898:a733 with SMTP id c26-20020ab0349a000000b0035cb898a733mr12723212uar.85.1652247761972; Tue, 10 May 2022 22:42:41 -0700 (PDT) MIME-Version: 1.0 References: <1642ab64-7957-e1e6-71c5-ceab9c23bf41@huawei.com> In-Reply-To: From: Wei Xu Date: Tue, 10 May 2022 22:42:30 -0700 Message-ID: Subject: Re: RFC: Memory Tiering Kernel Interfaces To: Aneesh Kumar K V Cc: Hesham Almatary , Yang Shi , Andrew Morton , Dave Hansen , Huang Ying , Dan Williams , Linux MM , Greg Thelen , Jagdish Gediya , Linux Kernel Mailing List , Alistair Popple , Davidlohr Bueso , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Tim Chen Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9EB15A009A X-Stat-Signature: jo84m3grgfozbsgwb9jcj9rgqz1jkj3d X-Rspam-User: Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=MYGn0ZQ3; spf=pass (imf15.hostedemail.com: domain of weixugc@google.com designates 209.85.222.45 as permitted sender) smtp.mailfrom=weixugc@google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1652247750-935675 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, May 10, 2022 at 5:10 AM Aneesh Kumar K V wrote: > > On 5/10/22 3:29 PM, Hesham Almatary wrote: > > Hello Yang, > > > > On 5/10/2022 4:24 AM, Yang Shi wrote: > >> On Mon, May 9, 2022 at 7:32 AM Hesham Almatary > >> wrote: > > > ... > > >>> > >>> node 0 has a CPU and DDR memory in tier 0, node 1 has GPU and DDR memory > >>> in tier 0, > >>> node 2 has NVMM memory in tier 1, node 3 has some sort of bigger memory > >>> (could be a bigger DDR or something) in tier 2. The distances are as > >>> follows: > >>> > >>> -------------- -------------- > >>> | Node 0 | | Node 1 | > >>> | ------- | | ------- | > >>> | | DDR | | | | DDR | | > >>> | ------- | | ------- | > >>> | | | | > >>> -------------- -------------- > >>> | 20 | 120 | > >>> v v | > >>> ---------------------------- | > >>> | Node 2 PMEM | | 100 > >>> ---------------------------- | > >>> | 100 | > >>> v v > >>> -------------------------------------- > >>> | Node 3 Large mem | > >>> -------------------------------------- > >>> > >>> node distances: > >>> node 0 1 2 3 > >>> 0 10 20 20 120 > >>> 1 20 10 120 100 > >>> 2 20 120 10 100 > >>> 3 120 100 100 10 > >>> > >>> /sys/devices/system/node/memory_tiers > >>> 0-1 > >>> 2 > >>> 3 > >>> > >>> N_TOPTIER_MEMORY: 0-1 > >>> > >>> > >>> In this case, we want to be able to "skip" the demotion path from Node 1 > >>> to Node 2, > >>> > >>> and make demotion go directely to Node 3 as it is closer, distance wise. > >>> How can > >>> > >>> we accommodate this scenario (or at least not rule it out as future > >>> work) with the > >>> > >>> current RFC? > >> If I remember correctly NUMA distance is hardcoded in SLIT by the > >> firmware, it is supposed to reflect the latency. So I suppose it is > >> the firmware's responsibility to have correct information. And the RFC > >> assumes higher tier memory has better performance than lower tier > >> memory (latency, bandwidth, throughput, etc), so it sounds like a > >> buggy firmware to have lower tier memory with shorter distance than > >> higher tier memory IMHO. > > > > You are correct if you're assuming the topology is all hierarchically > > > > symmetric, but unfortuantely, in real hardware (e.g., my example above) > > > > it is not. The distance/latency between two nodes in the same tier > > > > and a third node, is different. The firmware still provides the correct > > > > latency, but putting a node in a tier is up to the kernel/user, and > > > > is relative: e.g., Node 3 could belong to tier 1 from Node 1's > > > > perspective, but to tier 2 from Node 0's. > > > > > > A more detailed example (building on my previous one) is when having > > > > the GPU connected to a switch: > > > > ---------------------------- > > | Node 2 PMEM | > > ---------------------------- > > ^ > > | > > -------------- -------------- > > | Node 0 | | Node 1 | > > | ------- | | ------- | > > | | DDR | | | | DDR | | > > | ------- | | ------- | > > | CPU | | GPU | > > -------------- -------------- > > | | > > v v > > ---------------------------- > > | Switch | > > ---------------------------- > > | > > v > > -------------------------------------- > > | Node 3 Large mem | > > -------------------------------------- > > > > Here, demoting from Node 1 to Node 3 directly would be faster as > > > > it only has to go through one hub, compared to demoting from Node 1 > > > > to Node 2, where it goes through two hubs. I hope that example > > > > clarifies things a little bit. > > > > Alistair mentioned that we want to consider GPU memory to be expensive > and want to demote from GPU to regular DRAM. In that case for the above > case we should end up with > > > tier 0 - > Node3 > tier 1 -> Node0, Node1 > tier 2 -> Node2 > > Hence > > node 0: allowed=2 > node 1: allowed=2 > node 2: allowed = empty > node 3: allowed = 0-1 , based on fallback order 1, 0 If we have 3 tiers as defined above, then we'd better to have: node 0: allowed = 2 node 1: allowed = 2 node 2: allowed = empty node 3: allowed = 0-2, based on fallback order: 1,0,2 The firmware should provide the node distance values to reflect that PMEM is slowest and should have the largest distance away from node 3. > -aneesh > >