From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADA77C433F5 for ; Thu, 12 May 2022 03:37:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 281A26B0074; Wed, 11 May 2022 23:37:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 230FA6B0075; Wed, 11 May 2022 23:37:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D1EF8D0001; Wed, 11 May 2022 23:37:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EEC3D6B0074 for ; Wed, 11 May 2022 23:37:26 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B952D30817 for ; Thu, 12 May 2022 03:37:26 +0000 (UTC) X-FDA: 79455680892.08.5EF8645 Received: from mail-vs1-f48.google.com (mail-vs1-f48.google.com [209.85.217.48]) by imf23.hostedemail.com (Postfix) with ESMTP id 640BF1400A4 for ; Thu, 12 May 2022 03:37:12 +0000 (UTC) Received: by mail-vs1-f48.google.com with SMTP id u205so3914968vsu.6 for ; Wed, 11 May 2022 20:37:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=Yi/Z75PNHk1ys10Xd95MwCUa0CwRgya3lUvptT8pN84=; b=EBzA3KZDwe8GWuCM37qaNGOlYoPBwCtZ6W3PJIIJva2D0X6crlO7eEtvvMJs3EZCrB IqF7LSwqR3pxIqmepA0+SLCqqcEIzuwqWJqqqzwk5NCXaH5DTHLvwMC1tDH9W1oGhBin VzpckuHvHJSAA6sYwiKem2g5KWFl8qwfSKO0s/jn8On6YMlxQ5ni/ln1abdvb/Ic+vwm oxXoy0YXfpqutKwd7kJ/B5I5abiMZdOD9yiS2QBJ3A7TjO2mtZQPHo2YHpgfee6p6066 peycm4azgAW9nnie6GYiUesOQTkYytrH9sA4GnDWEPHAdk6k9GbqICEaD3aQZylpDXQ7 pKcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Yi/Z75PNHk1ys10Xd95MwCUa0CwRgya3lUvptT8pN84=; b=SBZ1rjY8Ax8+3jDqpCq75QnevhU9ymmTxGvQgKzsyRaFGlwx093nsUAAYdPnuIfxUT bd7vTNV0P2onADUYKzgSLSEFgwjiaBYjnGFi/ZSLZosNNwprh/nHIcv78wxulO4N/8Vw 60pwEmewXIiOnTorSLeZMSJ4BgnzNFiVT9RZ8MTunwiaDoYX1XaOPamOqW2+rCBmadNR PoqExugE+vy99EZ5NQxCt6/Sk3Qo8hrT9b+PlcBVJ97xT1OYoj8mwv3S/nWcq6jDJ7iu ikQNYBgFmA3JgJv6YyhjT0cvPzd/pgpjzlwkfvd2C7oFzaSwZbL9qMl3M0ZrMKSQoaM2 AHdg== X-Gm-Message-State: AOAM531wMAalMD8ewkfEyULvHOzmOW7+Q07E1V9VIRQEevuA+Ql+pjSI JRJ/Sla2uJ34vQsbGHgm/A5FCp918/MzpyZFvGwoLg== X-Google-Smtp-Source: ABdhPJxJAM6v9BkR77OE9dm6XqaSjp3t/bp9j6I2Wb6XgS/lK5UtdmARV40uKZLqOuIB+VgWTdvgEPyRhda+zklaZgo= X-Received: by 2002:a67:f343:0:b0:32c:c4b7:e238 with SMTP id p3-20020a67f343000000b0032cc4b7e238mr14295223vsm.77.1652326645358; Wed, 11 May 2022 20:37:25 -0700 (PDT) MIME-Version: 1.0 References: <87tua3h5r1.fsf@nvdebian.thelocal> <875ymerl81.fsf@nvdebian.thelocal> <87fslhhb2l.fsf@linux.ibm.com> <68333b21a58604f3fd0e660f1a39921ae22849d8.camel@intel.com> <0a92d0040edb3b74ac259062d241b8cd28924edf.camel@intel.com> In-Reply-To: From: Wei Xu Date: Wed, 11 May 2022 20:37:14 -0700 Message-ID: Subject: Re: RFC: Memory Tiering Kernel Interfaces To: "ying.huang@intel.com" Cc: "Aneesh Kumar K.V" , Alistair Popple , Yang Shi , Andrew Morton , Dave Hansen , Dan Williams , Linux MM , Greg Thelen , Jagdish Gediya , Linux Kernel Mailing List , Davidlohr Bueso , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Jonathan Cameron , Tim Chen Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 640BF1400A4 X-Stat-Signature: u3336zjris1e1br79ohnbno7979rtzo1 Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=EBzA3KZD; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of weixugc@google.com designates 209.85.217.48 as permitted sender) smtp.mailfrom=weixugc@google.com X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1652326632-862195 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, May 11, 2022 at 8:14 PM ying.huang@intel.com wrote: > > On Wed, 2022-05-11 at 19:39 -0700, Wei Xu wrote: > > On Wed, May 11, 2022 at 6:42 PM ying.huang@intel.com > > wrote: > > > > > > On Wed, 2022-05-11 at 10:07 -0700, Wei Xu wrote: > > > > On Wed, May 11, 2022 at 12:49 AM ying.huang@intel.com > > > > wrote: > > > > > > > > > > On Tue, 2022-05-10 at 22:30 -0700, Wei Xu wrote: > > > > > > On Tue, May 10, 2022 at 4:38 AM Aneesh Kumar K.V > > > > > > wrote: > > > > > > > > > > > > > > Alistair Popple writes: > > > > > > > > > > > > > > > Wei Xu writes: > > > > > > > > > > > > > > > > > On Thu, May 5, 2022 at 5:19 PM Alistair Popple wrote: > > > > > > > > > > > > > > > > > > > > Wei Xu writes: > > > > > > > > > > > > > > > > > > > > [...] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Tiering Hierarchy Initialization > > > > > > > > > > > > > `==============================' > > > > > > > > > > > > > > > > > > > > > > > > > > By default, all memory nodes are in the top tier (N_TOPTIER_MEMORY). > > > > > > > > > > > > > > > > > > > > > > > > > > A device driver can remove its memory nodes from the top tier, e.g. > > > > > > > > > > > > > a dax driver can remove PMEM nodes from the top tier. > > > > > > > > > > > > > > > > > > > > > > > > With the topology built by firmware we should not need this. > > > > > > > > > > > > > > > > > > > > I agree that in an ideal world the hierarchy should be built by firmware based > > > > > > > > > > on something like the HMAT. But I also think being able to override this will be > > > > > > > > > > useful in getting there. Therefore a way of overriding the generated hierarchy > > > > > > > > > > would be good, either via sysfs or kernel boot parameter if we don't want to > > > > > > > > > > commit to a particular user interface now. > > > > > > > > > > > > > > > > > > > > However I'm less sure letting device-drivers override this is a good idea. How > > > > > > > > > > for example would a GPU driver make sure it's node is in the top tier? By moving > > > > > > > > > > every node that the driver does not know about out of N_TOPTIER_MEMORY? That > > > > > > > > > > could get messy if say there were two drivers both of which wanted their node to > > > > > > > > > > be in the top tier. > > > > > > > > > > > > > > > > > > The suggestion is to allow a device driver to opt out its memory > > > > > > > > > devices from the top-tier, not the other way around. > > > > > > > > > > > > > > > > So how would demotion work in the case of accelerators then? In that > > > > > > > > case we would want GPU memory to demote to DRAM, but that won't happen > > > > > > > > if both DRAM and GPU memory are in N_TOPTIER_MEMORY and it seems the > > > > > > > > only override available with this proposal would move GPU memory into a > > > > > > > > lower tier, which is the opposite of what's needed there. > > > > > > > > > > > > > > How about we do 3 tiers now. dax kmem devices can be registered to > > > > > > > tier 3. By default all numa nodes can be registered at tier 2 and HBM or > > > > > > > GPU can be enabled to register at tier 1. ? > > > > > > > > > > > > This makes sense. I will send an updated RFC based on the discussions so far. > > > > > > > > > > Are these tier number fixed? If so, it appears strange that the > > > > > smallest tier number is 0 on some machines, but 1 on some other > > > > > machines. > > > > > > > > When the kernel is configured to allow 3 tiers, we can always show all > > > > the 3 tiers. It is just that some tiers (e.g. tier 0) may be empty on > > > > some machines. > > > > > > I still think that it's better to have no empty tiers for auto-generated > > > memory tiers by kernel. Yes, the tier number will be not absolutely > > > stable, but that only happens during system bootup in practice, so it's > > > not a big issue IMHO. > > > > It should not be hard to hide empty tiers (e.g. tier-0) if we prefer. > > But even if tier-0 is empty, we should still keep this tier in the > > kernel and not move DRAM nodes into this tier. One reason is that a > > HBM node might be hot-added into tier-0 at a later time. > > > > Yes. The in-kernel representation and the user space interface could be > different. > > I have thought something like below. We always make the main memory > (DRAM here, CPU local) as tier 0. Then the slower memory will be > positive, tier 1, 2, 3, ..., and the faster memory will be negative, > tier -1, -2, -3, .... Then, GPU driver can regesiter its memory as tier > -1. And the tier number could be more stable. But I'm not sure whether > users will be happy with negtive tier number. Given that we have agreed that the tier id itself should not carry any specific meaning to the userspace and what matters is the relative tier order, I think it is better to avoid negative tier numbers. > > > And, I still think it's better to make only N-1 tiers writable for > > > totally N tiers (or even readable). Considering "tier0" is written, how > > > to deal with nodes in "tier0" before but not after writing? One > > > possible way is to put them into "tierN". And during a user customize > > > the tiers, the union of "N tiers" may be not complete. > > > > The sysfs interfaces that I have in mind now are: > > > > * /sys/devices/system/memtier/memtierN/nodelist (N=0, 1, 2) > > > > This is read-only to list the memory nodes for a specific tier. > > > > * /sys/devices/system/node/nodeN/memtier. (N=0, 1, ...,) > > > > This is a read-write interface. When written, the kernel moves the > > node into the user-specified tier. No other nodes are affected. > > > > This interface should be able to avoid the above issue. > > Yes. This works too. > > Best Regards, > Huang, Ying > > > > > BTW, the userspace should not assume a specific meaning of a > > > > particular tier id because it can change depending on the number of > > > > tiers that the kernel is configured with. For example, the userspace > > > > should not assume that tier-2 always means PMEM nodes. In a system > > > > with 4 tiers, PMEM nodes may be in tier-3, not tier-2. > > > > > > Yes. This sounds good. > > > > > > Best Regards, > > > Huang, Ying > > > > >