From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0E54C4332F for ; Thu, 5 May 2022 14:24:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1D6116B0071; Thu, 5 May 2022 10:24:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 184386B0073; Thu, 5 May 2022 10:24:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0255F6B0074; Thu, 5 May 2022 10:24:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E836B6B0071 for ; Thu, 5 May 2022 10:24:12 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id A6CB7121282 for ; Thu, 5 May 2022 14:24:12 +0000 (UTC) X-FDA: 79431909144.23.3A392C3 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf10.hostedemail.com (Postfix) with ESMTP id 162DDC0087 for ; Thu, 5 May 2022 14:23:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1651760651; x=1683296651; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=OhcxbBgmutkoElcmxsM0ZTVXHgDmWx9IZ8gbNt7Pzfc=; b=TsxM5S4L8qUi4l2glGKtT7vAwq2nAssdRWymRoFDTXhX6wUErKEPaE8R HZoylvZJe/3jQppgSlQ2qjbYIWIqzLhqsz2Gx/E1nDFiNulRWI9sCnCS8 LolAPXvE2LBIfFaLrFt7E7TX5q2pODjcstf/6nl0GUlqQwAaFlwv3dnLS AVvVVk+Gsq8tkZtWfQ7isDHQRslQyKbCIO/NLRouUu2Oq5XVaWtCADtQL Clqm0AmaEamlODZ2yT9ArYVVuCYNvRfx98Kt9tKeRt+ooDdlz0yy0Z9/M humOHOhRjRZYfWkd5kv8j6JYiNi6MdlGH2MNqkjaFlkGsa8PPBVWqQ5wY g==; X-IronPort-AV: E=McAfee;i="6400,9594,10338"; a="248031285" X-IronPort-AV: E=Sophos;i="5.91,201,1647327600"; d="scan'208";a="248031285" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 May 2022 07:24:08 -0700 X-IronPort-AV: E=Sophos;i="5.91,201,1647327600"; d="scan'208";a="537354032" Received: from evegaag-mobl1.amr.corp.intel.com (HELO [10.209.187.127]) ([10.209.187.127]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 05 May 2022 07:24:06 -0700 Message-ID: Date: Thu, 5 May 2022 07:24:26 -0700 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: RFC: Memory Tiering Kernel Interfaces Content-Language: en-US To: Wei Xu Cc: Alistair Popple , Davidlohr Bueso , Andrew Morton , Dave Hansen , Huang Ying , Dan Williams , Yang Shi , Linux MM , Greg Thelen , "Aneesh Kumar K.V" , Jagdish Gediya , Linux Kernel Mailing List , Michal Hocko , Baolin Wang , Brice Goglin , Feng Tang , Jonathan Cameron References: <20220501175813.tvytoosygtqlh3nn@offworld> <87o80eh65f.fsf@nvdebian.thelocal> <87mtfygoxs.fsf@nvdebian.thelocal> <9fb22767-54de-d316-7e6b-5aac375c9c49@intel.com> <52541497-c097-5a51-4718-feed13660255@intel.com> From: Dave Hansen In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 162DDC0087 X-Stat-Signature: xy7jd1dddriuzqaf6ttj6bdta4fc9jxd X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=TsxM5S4L; spf=none (imf10.hostedemail.com: domain of dave.hansen@intel.com has no SPF policy when checking 192.55.52.136) smtp.mailfrom=dave.hansen@intel.com; dmarc=pass (policy=none) header.from=intel.com X-HE-Tag: 1651760633-132424 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 5/4/22 23:35, Wei Xu wrote: > On Wed, May 4, 2022 at 10:02 AM Dave Hansen wrote: >> That means a lot of page table and EPT walks to map those linear >> addresses back to physical. That adds to the inefficiency. > > That's true if the tracking is purely based on physical pages. For > hot page tracking from PEBS, we can consider tracking in > virtual/linear addresses. We don't need to maintain the history for > all linear page addresses nor for an indefinite amount of time. After > all, we just need to identify pages accessed frequently recently and > promote them. Except that you don't want to promote on *every* access. That might lead to too much churn. You're also assuming that all accesses to a physical page are via a single linear address, which ignores shared memory mapped at different linear addresses. Our (maybe wrong) assumption has been that shared memory is important enough to manage that it can't be ignored. >> In the end, you get big PEBS buffers with lots of irrelevant data that >> needs significant post-processing to make sense of it. > > I am curious about what are "lots of irrelevant data" if PEBS data is > filtered on data sources (e.g. DRAM vs PMEM) by hardware. If we need > to have different policies for the pages from the same data source, > then I agree that the software has to do a lot of filtering work. Perhaps "irrelevant" was a bad term to use. I meant that you can't just take the PEBS data and act directly on it. It has to be post-processed and you will see things in there like lots of adjacent accesses to a page. Those additional accesses can be interesting but at some point you have all the weight you need to promote the page and the _rest_ are irrelevant. >> The folks at Intel that tried this really struggled to take this mess and turn it into a successful hot-page tracking. >> >> Maybe someone else will find a better way to do it, but we tried and >> gave up. > > It might be challenging to use PEBS as the only and universal hot page > tracking hardware mechanism. For example, there are challenges to use > PEBS to sample KVM guest accesses from the host. Yep, agreed. This aspect of the hardware is very painful at the moment. > On the other hand, PEBS with hardware-based data source filtering can > be a useful mechanism to improve hot page tracking in conjunction > with other techniques. Rather than "can", I'd say: "might". Backing up to what I said originally: > So, in practice, these events (PEBS) weren't very useful > for driving memory tiering. By "driving" I really meant solely driving. Like, can PEBS be used as the one and only mechanism? We couldn't make it work. But, the hardware _is_ sitting there mostly unused. It might be great to augment what is there, and nobody should be discouraged from looking at it again.