From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7439C433F5 for ; Fri, 18 Feb 2022 02:21:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1ED106B0075; Thu, 17 Feb 2022 21:21:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 19D646B0078; Thu, 17 Feb 2022 21:21:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08BC96B007B; Thu, 17 Feb 2022 21:21:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0204.hostedemail.com [216.40.44.204]) by kanga.kvack.org (Postfix) with ESMTP id EF3C36B0075 for ; Thu, 17 Feb 2022 21:21:33 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A81FF8249980 for ; Fri, 18 Feb 2022 02:21:33 +0000 (UTC) X-FDA: 79154299266.20.B6D64DD Received: from out30-45.freemail.mail.aliyun.com (out30-45.freemail.mail.aliyun.com [115.124.30.45]) by imf09.hostedemail.com (Postfix) with ESMTP id 3D62F140006 for ; Fri, 18 Feb 2022 02:21:31 +0000 (UTC) X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04357;MF=xhao@linux.alibaba.com;NM=1;PH=DS;RN=7;SR=0;TI=SMTPD_---0V4nBQe7_1645150887; Received: from B-X3VXMD6M-2058.local(mailfrom:xhao@linux.alibaba.com fp:SMTPD_---0V4nBQe7_1645150887) by smtp.aliyun-inc.com(127.0.0.1); Fri, 18 Feb 2022 10:21:28 +0800 From: Xin Hao Reply-To: xhao@linux.alibaba.com Subject: Re: [RFC PATCH V1 0/5] mm/damon: Add NUMA access statistics function support To: SeongJae Park Cc: rongwei.wang@linux.alibaba.com, akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, rientjes@google.com, linux-damon@amazon.com References: <20220217082939.2850-1-sj@kernel.org> Message-ID: <503fa0b1-be20-a17e-72f0-14b38c0dc719@linux.alibaba.com> Date: Fri, 18 Feb 2022 10:21:27 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:78.0) Gecko/20100101 Thunderbird/78.14.0 MIME-Version: 1.0 In-Reply-To: <20220217082939.2850-1-sj@kernel.org> Content-Type: text/plain; charset=utf-8; format=flowed X-Rspamd-Queue-Id: 3D62F140006 X-Rspam-User: Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf09.hostedemail.com: domain of xhao@linux.alibaba.com designates 115.124.30.45 as permitted sender) smtp.mailfrom=xhao@linux.alibaba.com X-Stat-Signature: yz3ny74giq5deoqghgnttafr3qnoofkw X-Rspamd-Server: rspam11 X-HE-Tag: 1645150891-411909 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi SeongJae: On 2/17/22 4:29 PM, SeongJae Park wrote: > + David Rientjes, who has shown interest[1] in this topic. > > [1] https://lore.kernel.org/linux-mm/bcc8d9a0-81d-5f34-5e4-fcc28eb7ce@g= oogle.com/ > > --- > > Hi Xin, > > > Thank you always for great patches! > > On Wed, 16 Feb 2022 16:30:36 +0800 Xin Hao wro= te: > >> On today's cloud computing service scenario, NUMA (non uniform memory = access) >> architecture server has been applied on a large scale. Using Damon fun= ction, >> it can easily and lightweight identify hot and cold memory, but it can= not >> display the situation of locale and remote NUMA memory access. >> >> The purpose of these serie patches is to identify the situation of NUM= A access >> in combination with DAMON, especially for remote NUMA access in hot me= mory. >> We hope to detect this situation in the data center and use page migra= tion or >> multi backup page technology to optimize the behavior of memory access= . >> >> So next, we will further improve Damon NUMA function: >> 1. Support hugtlbfs NUMA access statistics. >> 2. Add the DAMO tool to parse NUMA local & remote in "damon_region" su= pport. >> 3. For hot memory remote NUMA access, support page migration or multi = backup page. >> >> About DAMON correctness of numa access statistics >> We wrote a test case, allocate about 1G memory, and use numa_alloc(), = set 512M in >> NUMA node0 and 512M in NUMA node1, and The test case alternately acces= ses the 1G of memory. >> >> We used "perf record -e damon:damon_aggregated" and "perf script" >> cmd to obtain data, like this: >> kdamond.0 target_id=3D0 nr_regions=3D10 281473056325632-2814731279646= 72:: 12 0 5243 5513 >> kdamond.0 target_id=3D0 nr_regions=3D10 281473127964672-2814732380282= 88: 8 1 5427 5399 >> ... >> kdamond.0 target_id=3D0 nr_regions=3D10 281473056325632-281473127964= 672: 9 3 7669 7632 >> kdamond.0 target_id=3D0 nr_regions=3D10 281473127964672-28147323802= 8288: 7 2 7913 7892 >> >> And compared with numastat like this: >> Per-node process memory usage (in MBs) for PID 111676 (lt-numademo) >> Node 0 Node 1 Node 2 >> --------------- --------------- --------------- >> Huge 0.00 0.00 0.00 >> Heap 0.02 0.00 0.00 >> Stack 0.01 0.00 0.00 >> Private 565.24 564.00 0.00 >> ---------------- --------------- --------------- --------------- >> Total 565.27 564.00 0.00 >> This comparison can determine the accuracy of Damon NUMA memory access= statistics. >> >> About the impact of DAMON NUMA access on Performance >> During the benchmakr test, we found that the MBW benchmark memcpy tes= t item >> will cause about 3% performance degradation, and there is no performan= ce degradation >> in other benchmarks. >> So we added "numa_stat" switch in DAMON dbgfs interface, turn on this = switch when NUMA access >> statistics is required. >> >> >> Xin Hao (5): >> mm/damon: Add NUMA local and remote variables in 'damon_region' >> mm/damon: Add 'damon_region' NUMA fault simulation support >> mm/damon: Add 'damon_region' NUMA access statistics core >> implementation >> mm/damon/dbgfs: Add numa simulate switch >> mm/damon/tracepoint: Add 'damon_region' NUMA access statistics supp= ort >> >> include/linux/damon.h | 25 ++++++++++ >> include/trace/events/damon.h | 9 +++- >> mm/damon/core.c | 94 ++++++++++++++++++++++++++++++++++= +- >> mm/damon/dbgfs.c | 70 ++++++++++++++++++++++++--- >> mm/damon/paddr.c | 25 ++++++++-- >> mm/damon/prmtv-common.c | 44 +++++++++++++++++ >> mm/damon/prmtv-common.h | 3 ++ >> mm/damon/vaddr.c | 45 ++++++++++------- >> mm/huge_memory.c | 5 ++ >> mm/memory.c | 5 ++ >> 10 files changed, 292 insertions(+), 33 deletions(-) > I'd like to comment on the high level design at the moment. To my > understanding, this patchset extends DAMON core and monitoring operatio= ns for > virtual address spaces (vaddr) and the physical address space (paddr) t= o > monitor NUMA-local/remote accesses via PROT_NONE and page faults mechan= ism. > > The underlying mechanism for NUMA-local/remote accesses (PROT_NONE and = page > fault) looks ok to me. But, changes to the core and vaddr/paddr operat= ions > looks unnecessary, to me. That's also not for general use cases. You are right, adding NUMA access statistics does make the PA & VA codes=20 look confusing=E3=80=82 > > I think it would be simpler to implment more monitoring operations for = NUMA > monitoring use case (one for NUMA-local accesses accounting and another= one for > NUMA-remote accesses accounting), alongside vaddr and paddr. Then, use= rs could > configure DAMON to have three monitoring contexts (one with vaddr ops, = second > one with numa-local ops, and third one with numa-remote ops), run those > concurrently, then show the three results and make some decisions like > migrations. Thanks for your advice, I will implement these in the next version, But=20 from my understanding or maybe I didn't get what you were thinking, I think only one monitor context is=20 needed for NUMA Local & Remote, Do not need a separate implementation like "numa_local_ops" and=20 "numa_remote_ops", just set "numa_access_ops" is ok. > > One additional advantage of this approach is that the accuracy for > NUMA-local/remote accessed could be better, because the contexts config= ured to > use NUMA-local/remote monitoring ops will do the regions adjustment wit= h > NUMA-local/remote accesses (to my understanding, this patchset let regi= ons have > NUMA-local/remote accesses counter in addition to the total one, but st= ill use > only the total one for the regions adjustment). > > If I'm missing something, please let me know. > > > Thanks, > SJ > >> -- >> 2.27.0 --=20 Best Regards! Xin Hao