From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43AF5C433EF for ; Wed, 11 May 2022 09:05:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A97946B0073; Wed, 11 May 2022 05:05:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A48016B0075; Wed, 11 May 2022 05:05:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90F2D6B0078; Wed, 11 May 2022 05:05:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 7F8796B0073 for ; Wed, 11 May 2022 05:05:33 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3892831C58 for ; Wed, 11 May 2022 09:05:33 +0000 (UTC) X-FDA: 79452878946.09.1C693F1 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf26.hostedemail.com (Postfix) with ESMTP id 5DE8014009B for ; Wed, 11 May 2022 09:05:29 +0000 (UTC) Received: from fraeml739-chm.china.huawei.com (unknown [172.18.147.206]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4Kyplv4QX8z67mCt; Wed, 11 May 2022 17:00:43 +0800 (CST) Received: from lhreml751-chm.china.huawei.com (10.201.108.201) by fraeml739-chm.china.huawei.com (10.206.15.220) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Wed, 11 May 2022 11:05:29 +0200 Received: from localhost (10.202.227.118) by lhreml751-chm.china.huawei.com (10.201.108.201) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Wed, 11 May 2022 10:05:28 +0100 Date: Wed, 11 May 2022 10:05:27 +0100 From: Hesham Almatary To: Alistair Popple CC: Wei Xu , Aneesh Kumar K V , Yang Shi , Andrew Morton , Dave Hansen , "Huang Ying" , Dan Williams , "Linux MM" , Greg Thelen , Jagdish Gediya , Linux Kernel Mailing List , Davidlohr Bueso , "Michal Hocko" , Baolin Wang , "Brice Goglin" , Feng Tang , Tim Chen Subject: Re: RFC: Memory Tiering Kernel Interfaces Message-ID: <20220511100527.00007bc2@huawei.com> In-Reply-To: <87o804r08w.fsf@nvdebian.thelocal> References: <1642ab64-7957-e1e6-71c5-ceab9c23bf41@huawei.com> <87o804r08w.fsf@nvdebian.thelocal> Organization: Huawei UK R&D X-Mailer: Claws Mail 3.18.0 (GTK+ 2.24.33; x86_64-w64-mingw32) MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.202.227.118] X-ClientProxiedBy: lhreml727-chm.china.huawei.com (10.201.108.78) To lhreml751-chm.china.huawei.com (10.201.108.201) X-CFilter-Loop: Reflected X-Rspamd-Queue-Id: 5DE8014009B X-Stat-Signature: p5iq7e783ks5pemhknu86fu3a6hyftcb X-Rspam-User: Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=huawei.com; spf=pass (imf26.hostedemail.com: domain of hesham.almatary@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=hesham.almatary@huawei.com X-Rspamd-Server: rspam09 X-HE-Tag: 1652259929-741804 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 11 May 2022 17:12:34 +1000 Alistair Popple wrote: > > Wei Xu writes: > > > On Tue, May 10, 2022 at 5:10 AM Aneesh Kumar K V > > wrote: > >> > >> On 5/10/22 3:29 PM, Hesham Almatary wrote: > >> > Hello Yang, > >> > > >> > On 5/10/2022 4:24 AM, Yang Shi wrote: > >> >> On Mon, May 9, 2022 at 7:32 AM Hesham Almatary > >> >> wrote: > >> > >> > >> ... > >> > >> >>> > >> >>> node 0 has a CPU and DDR memory in tier 0, node 1 has GPU and > >> >>> DDR memory in tier 0, > >> >>> node 2 has NVMM memory in tier 1, node 3 has some sort of > >> >>> bigger memory (could be a bigger DDR or something) in tier 2. > >> >>> The distances are as follows: > >> >>> > >> >>> -------------- -------------- > >> >>> | Node 0 | | Node 1 | > >> >>> | ------- | | ------- | > >> >>> | | DDR | | | | DDR | | > >> >>> | ------- | | ------- | > >> >>> | | | | > >> >>> -------------- -------------- > >> >>> | 20 | 120 | > >> >>> v v | > >> >>> ---------------------------- | > >> >>> | Node 2 PMEM | | 100 > >> >>> ---------------------------- | > >> >>> | 100 | > >> >>> v v > >> >>> -------------------------------------- > >> >>> | Node 3 Large mem | > >> >>> -------------------------------------- > >> >>> > >> >>> node distances: > >> >>> node 0 1 2 3 > >> >>> 0 10 20 20 120 > >> >>> 1 20 10 120 100 > >> >>> 2 20 120 10 100 > >> >>> 3 120 100 100 10 > >> >>> > >> >>> /sys/devices/system/node/memory_tiers > >> >>> 0-1 > >> >>> 2 > >> >>> 3 > >> >>> > >> >>> N_TOPTIER_MEMORY: 0-1 > >> >>> > >> >>> > >> >>> In this case, we want to be able to "skip" the demotion path > >> >>> from Node 1 to Node 2, > >> >>> > >> >>> and make demotion go directely to Node 3 as it is closer, > >> >>> distance wise. How can > >> >>> > >> >>> we accommodate this scenario (or at least not rule it out as > >> >>> future work) with the > >> >>> > >> >>> current RFC? > >> >> If I remember correctly NUMA distance is hardcoded in SLIT by > >> >> the firmware, it is supposed to reflect the latency. So I > >> >> suppose it is the firmware's responsibility to have correct > >> >> information. And the RFC assumes higher tier memory has better > >> >> performance than lower tier memory (latency, bandwidth, > >> >> throughput, etc), so it sounds like a buggy firmware to have > >> >> lower tier memory with shorter distance than higher tier memory > >> >> IMHO. > >> > > >> > You are correct if you're assuming the topology is all > >> > hierarchically > >> > > >> > symmetric, but unfortuantely, in real hardware (e.g., my example > >> > above) > >> > > >> > it is not. The distance/latency between two nodes in the same > >> > tier > >> > > >> > and a third node, is different. The firmware still provides the > >> > correct > >> > > >> > latency, but putting a node in a tier is up to the kernel/user, > >> > and > >> > > >> > is relative: e.g., Node 3 could belong to tier 1 from Node 1's > >> > > >> > perspective, but to tier 2 from Node 0's. > >> > > >> > > >> > A more detailed example (building on my previous one) is when > >> > having > >> > > >> > the GPU connected to a switch: > >> > > >> > ---------------------------- > >> > | Node 2 PMEM | > >> > ---------------------------- > >> > ^ > >> > | > >> > -------------- -------------- > >> > | Node 0 | | Node 1 | > >> > | ------- | | ------- | > >> > | | DDR | | | | DDR | | > >> > | ------- | | ------- | > >> > | CPU | | GPU | > >> > -------------- -------------- > >> > | | > >> > v v > >> > ---------------------------- > >> > | Switch | > >> > ---------------------------- > >> > | > >> > v > >> > -------------------------------------- > >> > | Node 3 Large mem | > >> > -------------------------------------- > >> > > >> > Here, demoting from Node 1 to Node 3 directly would be faster as > >> > > >> > it only has to go through one hub, compared to demoting from > >> > Node 1 > >> > > >> > to Node 2, where it goes through two hubs. I hope that example > >> > > >> > clarifies things a little bit. > >> > > >> > >> Alistair mentioned that we want to consider GPU memory to be > >> expensive and want to demote from GPU to regular DRAM. In that > >> case for the above case we should end up with > >> > >> > >> tier 0 - > Node3 > >> tier 1 -> Node0, Node1 > >> tier 2 -> Node2 > > I'm a little bit confused by the tiering here as I don't think it's > quite what we want. As pointed out GPU memory is expensive and > therefore we don't want anything demoting to it. That implies it > should be in the top tier: > > tier 0 -> Node1 > tier 1 -> Node0, Node3 > tier 2 -> Node2 > > Hence: > > node 0: allowed=2 > node 1: allowed=0,3,2 > node 2: allowed=empty > node 3: allowed=2 > > Alternatively Node3 could be put in tier 2 which would prevent > demotion to PMEM via the switch/CPU: > > tier 0 -> Node1 > tier 1 -> Node0 > tier 2 -> Node2, Node3 > > node 0: allowed=2,3 > node 1: allowed=0,3,2 > node 2: allowed=empty > node 3: allowed=empty > Indeed. The scenario I described here is where the GPU can't/don't demote to PMEM, but the CPU can. In this case it would work fine if we put the GPU (Node 1) in tier 0, and rely on the fallback order. > Both of these would be an improvement over the current situation > upstream, which demotes everything to GPU memory and doesn't support > demoting from the GPU (meaning reclaim on GPU memory pages everything > to disk). > > >> > >> Hence > >> > >> node 0: allowed=2 > >> node 1: allowed=2 > >> node 2: allowed = empty > >> node 3: allowed = 0-1 , based on fallback order 1, 0 > > > > If we have 3 tiers as defined above, then we'd better to have: > > > > node 0: allowed = 2 > > node 1: allowed = 2 > > node 2: allowed = empty > > node 3: allowed = 0-2, based on fallback order: 1,0,2 > > > > The firmware should provide the node distance values to reflect that > > PMEM is slowest and should have the largest distance away from node > > 3. > > Right. In my above example firmware would have to provide reasonable > distance values to ensure optimal fallback order. > > >> -aneesh > >> > >>