From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF206C433F5 for ; Wed, 25 May 2022 15:32:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B3E898D0003; Wed, 25 May 2022 11:32:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AEC8E8D0002; Wed, 25 May 2022 11:32:46 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D6408D0003; Wed, 25 May 2022 11:32:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 905338D0002 for ; Wed, 25 May 2022 11:32:46 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 1FD961213C2 for ; Wed, 25 May 2022 15:32:46 +0000 (UTC) X-FDA: 79504657932.15.C7757A9 Received: from mail-vs1-f52.google.com (mail-vs1-f52.google.com [209.85.217.52]) by imf09.hostedemail.com (Postfix) with ESMTP id E0212140036 for ; Wed, 25 May 2022 15:32:32 +0000 (UTC) Received: by mail-vs1-f52.google.com with SMTP id 68so14174119vse.11 for ; Wed, 25 May 2022 08:32:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=V6fNtaXrpXPdwdoGVh0BHVRTyE3sTb/7nc/NM05nCe0=; b=oQsvSC8oEHuqSFLW0whM02wczKQrPEEyRwIDRWwKX2XQlCFjJgF1jeOh3dWrRdAhxz 5QNtkOQPmwuEEU4WUXD6nT8XpGKYuGjk+lxGtK+D+45zA4+MJqiez/ARh+JvFvf3/sqO X7qsNSRhDYk71RwEdVZunX92MESwEbYDHgHjBUDmVlq4/jQkfMvHEJltFjx15HUWOQms nyHD80H+FKu+bcFqd4WGgRowmWb4MrB7vtNFuMp0w0xQq85Xx6egFuyCJ768m8KyqjiQ pm+TrxzQWv5tK1z/uLYkU9kG7wE2Ikibo0FMRjl1dDZJpmBpvnyW6ZNniXd3QO+RQfa1 UJ3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=V6fNtaXrpXPdwdoGVh0BHVRTyE3sTb/7nc/NM05nCe0=; b=1kYo+X3rnAzcv8zl4VNPljpu3kZNviKtsi9ZWtXEASPKv9/qLxxinwyh2ALAIXUfNM e66XPnvUhFp5qBbJ+yV9SSGp/+aAW/nenYA/MeXGo6qah3fT2Y3ZupM0iAb6GEv4rq/Z RmIDoIBBsIJYBcblL4wWk2V8CWP7Ezewk5cxaanOHqa/EnYckj/nTKkvJ8u5dqswXPh4 3G13pXf8B1DKFWa7Kb0aHH5DyHYsw0P/i8wrbtavhNsVqF9rHo6vExPBro3q9CvW/X4N QkUOfJmohEJdforewp7zQZNbfMicSiZjzPNCoVZcttp/57PJreVzYCCCvj/OjLLtmYPu 8wUw== X-Gm-Message-State: AOAM531ADtjTdm91ge7fif4gPRbUPrOfJEGAsHbVMw0mUtCqrHuCE4ev p25s082SXHytYMrwwfbJ7lhC/uWvU+TdZaXTKQsuSQ== X-Google-Smtp-Source: ABdhPJxDq3R3pD+aS09hMGojTfBgBObu+GY+RmXYJHjBNu/MchSvwaaOQN5cX9hIB9bfhPbd9+MIazpOvNFY2jmTTb0= X-Received: by 2002:a67:ed88:0:b0:328:27d9:1381 with SMTP id d8-20020a67ed88000000b0032827d91381mr13563848vsp.12.1653492764752; Wed, 25 May 2022 08:32:44 -0700 (PDT) MIME-Version: 1.0 References: <20220512160010.00005bc4@Huawei.com> <20220518130037.00001cce@Huawei.com> <8735gzdpsx.fsf@linux.ibm.com> <87h75ef3y5.fsf@nvdebian.thelocal> <20220525124847.00007a16@Huawei.com> In-Reply-To: <20220525124847.00007a16@Huawei.com> From: Wei Xu Date: Wed, 25 May 2022 08:32:33 -0700 Message-ID: Subject: Re: RFC: Memory Tiering Kernel Interfaces (v2) To: Jonathan Cameron Cc: Alistair Popple , "Aneesh Kumar K.V" , Dave Hansen , Huang Ying , Andrew Morton , Greg Thelen , Yang Shi , Linux Kernel Mailing List , Jagdish Gediya , Michal Hocko , Tim C Chen , Baolin Wang , Feng Tang , Davidlohr Bueso , Dan Williams , David Rientjes , Linux MM , Brice Goglin , Hesham Almatary Content-Type: text/plain; charset="UTF-8" Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=oQsvSC8o; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf09.hostedemail.com: domain of weixugc@google.com designates 209.85.217.52 as permitted sender) smtp.mailfrom=weixugc@google.com X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: E0212140036 X-Stat-Signature: cg38y6rpg75isjwikmq8ndii3goox8yi X-HE-Tag: 1653492752-995726 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, May 25, 2022 at 4:48 AM Jonathan Cameron wrote: > > On Wed, 25 May 2022 17:47:33 +1000 > Alistair Popple wrote: > > > Wei Xu writes: > > > > > On Tue, May 24, 2022 at 6:27 AM Aneesh Kumar K.V > > > wrote: > > >> > > >> Wei Xu writes: > > >> > > >> > On Wed, May 18, 2022 at 5:00 AM Jonathan Cameron > > >> > wrote: > > >> >> > > >> >> On Wed, 18 May 2022 00:09:48 -0700 > > >> >> Wei Xu wrote: > > >> > > >> ... > > >> > > >> > Nice :) > > >> >> > > >> >> Initially I thought this was over complicated when compared to just leaving space, but > > >> >> after a chat with Hesham just now you have us both convinced that this is an elegant solution. > > >> >> > > >> >> Few corners probably need fleshing out: > > >> >> * Use of an allocator for new tiers. Flat number at startup, or new one on write of unique > > >> >> value to set_memtier perhaps? Also whether to allow drivers to allocate (I think > > >> >> we should). > > >> >> * Multiple tiers with same rank. My assumption is from demotion path point of view you > > >> >> fuse them (treat them as if they were a single tier), but keep them expressed > > >> >> separately in the sysfs interface so that the rank can be changed independently. > > >> >> * Some guidance on what values make sense for given rank default that might be set by > > >> >> a driver. If we have multiple GPU vendors, and someone mixes them in a system we > > >> >> probably don't want the default values they use to result in demotion between them. > > >> >> This might well be a guidance DOC or appropriate set of #define > > >> > > > >> > All of these are good ideas, though I am afraid that these can make > > >> > tier management too complex for what it's worth. > > >> > > > >> > How about an alternative tier numbering scheme that uses major.minor > > >> > device IDs? For simplicity, we can just start with 3 major tiers. > > >> > New tiers can be inserted in-between using minor tier IDs. > > >> > > >> > > >> What drives the creation of a new memory tier here? Jonathan was > > >> suggesting we could do something similar to writing to set_memtier for > > >> creating a new memory tier. > > >> > > >> $ echo "memtier128" > sys/devices/system/node/node1/set_memtier > > >> > > >> But I am wondering whether we should implement that now. If we keep > > >> "rank" concept and detach tier index (memtier0 is the memory tier with > > >> index 0) separate from rank, I assume we have enough flexibility for a > > >> future extension that will allow us to create a memory tier from userspace > > >> and assigning it a rank value that helps the device to be placed before or > > >> after DRAM in demotion order. > > >> > > >> ie, For now we will only have memtier0, memtier1, memtier2. We won't add > > >> dynamic creation of memory tiers and the above memory tiers will have > > >> rank value 0, 1, 2 according with demotion order 0 -> 1 -> 2. > > > > > > Great. So the consensus is to go with the "rank" approach. The above > > > sounds good to me as a starting point. > > > > The rank approach seems good to me too. > > Rank is good, but I do slightly worry about accidentally defining ABI > that people care about with the particular numbers used for the initial ranks. > > Maybe just x100 on all of them to allow things in between with no change to > this initial set of 3? So 0, 100, 200 I strongly support this, which is also my original intention for rank values. I'd suggest to even remove 0 to avoid it becoming a special value that userspace depends on. > Jonathan > > > > > - Alistair > > > > >> -aneesh >