From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6AE1C433F5 for ; Thu, 12 May 2022 04:49:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A31E6B0074; Thu, 12 May 2022 00:49:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 452506B0075; Thu, 12 May 2022 00:49:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 31AAC6B0078; Thu, 12 May 2022 00:49:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2001B6B0074 for ; Thu, 12 May 2022 00:49:28 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E1F32306B6 for ; Thu, 12 May 2022 04:49:27 +0000 (UTC) X-FDA: 79455862374.22.42D0F9A Received: from mail-ua1-f45.google.com (mail-ua1-f45.google.com [209.85.222.45]) by imf25.hostedemail.com (Postfix) with ESMTP id 85252A000E for ; Thu, 12 May 2022 04:49:08 +0000 (UTC) Received: by mail-ua1-f45.google.com with SMTP id x11so1581399uao.2 for ; Wed, 11 May 2022 21:49:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=dZTDnUWLhC6cjTXpBhDBsxqrgSnxn4/8Ogucz8EU94s=; b=D4JCCyKL+1RD+XHpB+jmPd5St1jxQqq2hTlRXQmLvp/mQplKm2rSHkWSxjslOs/aws DY+FuY8f/UNVG4vnlsFn8uGi6Zly00KYEZ0vGfHnh0zSQcfDdf1Oog+L/d7Lyjkc1YSE gKD5dy6CpL5eamPmyQiuMWPEyvZzOukSrH2t680iP4M9f6idiCrNLgvVct1wNxJGwOzw wuDtLo/aqerLTd5w9SylSJXEukAl0ZLagVCifcsK9rphdylVwfq32iOP5bsgVBWEnBYo xm+8vQEoAHeS2zp2c4gayV1oOyXsPDKdJYTE2mqdA08FGEbcFJ2+Fgbux8asFHByR9MY 6iYw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dZTDnUWLhC6cjTXpBhDBsxqrgSnxn4/8Ogucz8EU94s=; b=FZCWDNSqQRENwcZ3+uIOcZssYuadp0eGsKsFs9sUWhocgkV+PcVyqfcj8xG9IqMEIY 3ldVUvYyeQ4NMwtJGsEy8V2Zedb8MT5W9x0h1NY3CKAjM+7F4ISfuBVLXOBERy9iLMB9 3k1p6VKFou2mlMC8oOYVb/0HSSA9PPOcvYrPLDgAno+aGB288U2QqJR8WuQTkMo/PvNU hoSid8ApU3hfVCxhnOtIHUoj1fk1GHHPi3AgqadaIOfGkZ+VKWYnkrlF4t4XkRBSDLQt CE8oo2XA99GA6QehCTTx6Bh/tNbEcteO1/L8WxT6Ujmoxiua5v8iEWQKBzcCbxn6e0BT gEjw== X-Gm-Message-State: AOAM532P+hcXZ0AN4iRDcZbprzk/KsBdm+upfRUuvuPvtZS/czHtJx/q 47/9L/FxdlyDM06wMELBIasQcaeUjXvPNHhG2wMsZw== X-Google-Smtp-Source: ABdhPJyQe63+/h17kay1cN4u1mpqQ7Ry8hCjFIfni3c4e/cCLhNVI8Z2+1O3/9TnmtB3OBOySV9qicWJyZxF/e4c4Ps= X-Received: by 2002:ab0:e14:0:b0:360:e13:e5d7 with SMTP id g20-20020ab00e14000000b003600e13e5d7mr15545369uak.95.1652330964925; Wed, 11 May 2022 21:49:24 -0700 (PDT) MIME-Version: 1.0 References: <1642ab64-7957-e1e6-71c5-ceab9c23bf41@huawei.com> <87o804r08w.fsf@nvdebian.thelocal> In-Reply-To: From: Wei Xu Date: Wed, 11 May 2022 21:49:13 -0700 Message-ID: Subject: Re: RFC: Memory Tiering Kernel Interfaces To: Aneesh Kumar K V Cc: Alistair Popple , Hesham Almatary , Yang Shi , Andrew Morton , Dave Hansen , Huang Ying , Dan Williams , Linux MM , Greg Thelen , Jagdish Gediya , Linux Kernel Mailing List , Davidlohr Bueso , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Tim Chen Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 85252A000E X-Stat-Signature: 1wptgj8unt44u171z6w3i6kt1q3ehgho Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=D4JCCyKL; spf=pass (imf25.hostedemail.com: domain of weixugc@google.com designates 209.85.222.45 as permitted sender) smtp.mailfrom=weixugc@google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1652330948-759647 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, May 11, 2022 at 9:40 PM Aneesh Kumar K V wrote: > > On 5/11/22 12:42 PM, Alistair Popple wrote: > > > > Wei Xu writes: > > > >> On Tue, May 10, 2022 at 5:10 AM Aneesh Kumar K V > >> wrote: > >>> > >>> On 5/10/22 3:29 PM, Hesham Almatary wrote: > >>>> Hello Yang, > >>>> > >>>> On 5/10/2022 4:24 AM, Yang Shi wrote: > >>>>> On Mon, May 9, 2022 at 7:32 AM Hesham Almatary > >>>>> wrote: > >>> > >>> > >>> ... > >>> > >>>>>> > >>>>>> node 0 has a CPU and DDR memory in tier 0, node 1 has GPU and DDR memory > >>>>>> in tier 0, > >>>>>> node 2 has NVMM memory in tier 1, node 3 has some sort of bigger memory > >>>>>> (could be a bigger DDR or something) in tier 2. The distances are as > >>>>>> follows: > >>>>>> > >>>>>> -------------- -------------- > >>>>>> | Node 0 | | Node 1 | > >>>>>> | ------- | | ------- | > >>>>>> | | DDR | | | | DDR | | > >>>>>> | ------- | | ------- | > >>>>>> | | | | > >>>>>> -------------- -------------- > >>>>>> | 20 | 120 | > >>>>>> v v | > >>>>>> ---------------------------- | > >>>>>> | Node 2 PMEM | | 100 > >>>>>> ---------------------------- | > >>>>>> | 100 | > >>>>>> v v > >>>>>> -------------------------------------- > >>>>>> | Node 3 Large mem | > >>>>>> -------------------------------------- > >>>>>> > >>>>>> node distances: > >>>>>> node 0 1 2 3 > >>>>>> 0 10 20 20 120 > >>>>>> 1 20 10 120 100 > >>>>>> 2 20 120 10 100 > >>>>>> 3 120 100 100 10 > >>>>>> > >>>>>> /sys/devices/system/node/memory_tiers > >>>>>> 0-1 > >>>>>> 2 > >>>>>> 3 > >>>>>> > >>>>>> N_TOPTIER_MEMORY: 0-1 > >>>>>> > >>>>>> > >>>>>> In this case, we want to be able to "skip" the demotion path from Node 1 > >>>>>> to Node 2, > >>>>>> > >>>>>> and make demotion go directely to Node 3 as it is closer, distance wise. > >>>>>> How can > >>>>>> > >>>>>> we accommodate this scenario (or at least not rule it out as future > >>>>>> work) with the > >>>>>> > >>>>>> current RFC? > >>>>> If I remember correctly NUMA distance is hardcoded in SLIT by the > >>>>> firmware, it is supposed to reflect the latency. So I suppose it is > >>>>> the firmware's responsibility to have correct information. And the RFC > >>>>> assumes higher tier memory has better performance than lower tier > >>>>> memory (latency, bandwidth, throughput, etc), so it sounds like a > >>>>> buggy firmware to have lower tier memory with shorter distance than > >>>>> higher tier memory IMHO. > >>>> > >>>> You are correct if you're assuming the topology is all hierarchically > >>>> > >>>> symmetric, but unfortuantely, in real hardware (e.g., my example above) > >>>> > >>>> it is not. The distance/latency between two nodes in the same tier > >>>> > >>>> and a third node, is different. The firmware still provides the correct > >>>> > >>>> latency, but putting a node in a tier is up to the kernel/user, and > >>>> > >>>> is relative: e.g., Node 3 could belong to tier 1 from Node 1's > >>>> > >>>> perspective, but to tier 2 from Node 0's. > >>>> > >>>> > >>>> A more detailed example (building on my previous one) is when having > >>>> > >>>> the GPU connected to a switch: > >>>> > >>>> ---------------------------- > >>>> | Node 2 PMEM | > >>>> ---------------------------- > >>>> ^ > >>>> | > >>>> -------------- -------------- > >>>> | Node 0 | | Node 1 | > >>>> | ------- | | ------- | > >>>> | | DDR | | | | DDR | | > >>>> | ------- | | ------- | > >>>> | CPU | | GPU | > >>>> -------------- -------------- > >>>> | | > >>>> v v > >>>> ---------------------------- > >>>> | Switch | > >>>> ---------------------------- > >>>> | > >>>> v > >>>> -------------------------------------- > >>>> | Node 3 Large mem | > >>>> -------------------------------------- > >>>> > >>>> Here, demoting from Node 1 to Node 3 directly would be faster as > >>>> > >>>> it only has to go through one hub, compared to demoting from Node 1 > >>>> > >>>> to Node 2, where it goes through two hubs. I hope that example > >>>> > >>>> clarifies things a little bit. > >>>> > >>> > >>> Alistair mentioned that we want to consider GPU memory to be expensive > >>> and want to demote from GPU to regular DRAM. In that case for the above > >>> case we should end up with > >>> > >>> > >>> tier 0 - > Node3 > >>> tier 1 -> Node0, Node1 > >>> tier 2 -> Node2 > > > > I'm a little bit confused by the tiering here as I don't think it's > > quite what we want. As pointed out GPU memory is expensive and therefore > > we don't want anything demoting to it. That implies it should be in the > > top tier: > > > > > I didn't look closely at the topology and assumed that Node3 is the GPU > connected to the switch. Hence all the confusion. > > > > tier 0 -> Node1 > > tier 1 -> Node0, Node3 > > tier 2 -> Node2 > > > > Hence: > > > > node 0: allowed=2 > > node 1: allowed=0,3,2 > > node 2: allowed=empty > > node 3: allowed=2 > > looks good to be default and simple. > > > > > Alternatively Node3 could be put in tier 2 which would prevent demotion > > to PMEM via the switch/CPU: > > > > tier 0 -> Node1 > > tier 1 -> Node0 > > tier 2 -> Node2, Node3 > > > > node 0: allowed=2,3 > > node 1: allowed=0,3,2 > > node 2: allowed=empty > > node 3: allowed=empty > > > > and this can be configured via userspace? The per-node tier customization interface that I just mentioned should support such reconfigurations. > > Both of these would be an improvement over the current situation > > upstream, which demotes everything to GPU memory and doesn't support > > demoting from the GPU (meaning reclaim on GPU memory pages everything to > > disk). > > > >>> > >>> Hence > >>> > >>> node 0: allowed=2 > >>> node 1: allowed=2 > >>> node 2: allowed = empty > >>> node 3: allowed = 0-1 , based on fallback order 1, 0 > >> > >> If we have 3 tiers as defined above, then we'd better to have: > >> > >> node 0: allowed = 2 > >> node 1: allowed = 2 > >> node 2: allowed = empty > >> node 3: allowed = 0-2, based on fallback order: 1,0,2 > >> > >> The firmware should provide the node distance values to reflect that > >> PMEM is slowest and should have the largest distance away from node 3. > > > > Right. In my above example firmware would have to provide reasonable > > distance values to ensure optimal fallback order. > > > >>> -aneesh > >>> > >>> >