From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 484F9C4167B for ; Tue, 26 Apr 2022 09:08:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B807F6B0073; Tue, 26 Apr 2022 05:08:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE4156B0074; Tue, 26 Apr 2022 05:08:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 95C816B0075; Tue, 26 Apr 2022 05:08:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 80A236B0073 for ; Tue, 26 Apr 2022 05:08:05 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 47896120C03 for ; Tue, 26 Apr 2022 09:08:05 +0000 (UTC) X-FDA: 79398453330.22.5E4B81C Received: from mx0b-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by imf11.hostedemail.com (Postfix) with ESMTP id 2DCAC40049 for ; Tue, 26 Apr 2022 09:08:01 +0000 (UTC) Received: from pps.filterd (m0127361.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 23Q81LhC010635; Tue, 26 Apr 2022 09:08:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=message-id : date : mime-version : subject : to : cc : references : from : in-reply-to : content-type : content-transfer-encoding; s=pp1; bh=PIxu2E71rfXmR81PBhoZhEk7raaOGzgfbCl1jKf0/Ww=; b=i8He8qGGONPEZeZEInotHUaxIq8zKJ0K/CyExeXLF/R5M9MfVYLrVjvMf/NEOf5Eu/dE PoFVXVO5ljTjFA4q5wwkhhCw6DghgGmgS9CSl+Fi8Gz8GATthVuUWSSQkOU8ZjNIfDV2 13TAyLpct8WTBytC9uO+tD78J4StBs9Vep1o3seNTJHatBkyhl30cbFgftJt7MXubI/j NHysMXJaPkvfhPp9LzU/5mLN6hjckyBFe+VzpoiIbBZMX7HJ/IAp2FnE5002IvnbllUT uYTPxhF2D5fnP8YPfsbk1hWXFtQO2kg2Jg6p5R28onqpGqokvi4qNf+fGPUC3F2PlVeT Bg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3fpcv3h6gp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 26 Apr 2022 09:08:02 +0000 Received: from m0127361.ppops.net (m0127361.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 23Q90YHb006620; Tue, 26 Apr 2022 09:08:02 GMT Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3fpcv3h6g5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 26 Apr 2022 09:08:02 +0000 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 23Q92PQd021384; Tue, 26 Apr 2022 09:08:00 GMT Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by ppma03ams.nl.ibm.com with ESMTP id 3fm938v371-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 26 Apr 2022 09:08:00 +0000 Received: from d06av21.portsmouth.uk.ibm.com (d06av21.portsmouth.uk.ibm.com [9.149.105.232]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 23Q97v7t47972818 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 26 Apr 2022 09:07:57 GMT Received: from d06av21.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B3FA952052; Tue, 26 Apr 2022 09:07:57 +0000 (GMT) Received: from [9.109.205.170] (unknown [9.109.205.170]) by d06av21.portsmouth.uk.ibm.com (Postfix) with ESMTP id 9BB7A5204E; Tue, 26 Apr 2022 09:07:55 +0000 (GMT) Message-ID: <63b0c4ab-861c-3d07-4912-e6cd842d0bfd@linux.ibm.com> Date: Tue, 26 Apr 2022 14:37:54 +0530 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.8.0 Subject: Re: [PATCH v3 0/7] mm: demotion: Introduce new node state N_DEMOTION_TARGETS Content-Language: en-US To: "ying.huang@intel.com" , Jagdish Gediya Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, baolin.wang@linux.alibaba.com, dave.hansen@linux.intel.com, shy828301@gmail.com, weixugc@google.com, gthelen@google.com, dan.j.williams@intel.com References: <20220422195516.10769-1-jvgediya@linux.ibm.com> <4b986b46afb2fe888c127d8758221d0f0d3ec55f.camel@intel.com> <085260285e48093f48d889994aaa500a78577bf2.camel@intel.com> From: Aneesh Kumar K V In-Reply-To: <085260285e48093f48d889994aaa500a78577bf2.camel@intel.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 0hQfjNpTbWfuiBtKpDO6Ix7kQ8pHlbac X-Proofpoint-ORIG-GUID: YyvYEz6fG-GOM4uwq7y66KJooeJnqEVE X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.858,Hydra:6.0.486,FMLib:17.11.64.514 definitions=2022-04-26_02,2022-04-25_03,2022-02-23_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1015 priorityscore=1501 malwarescore=0 bulkscore=0 mlxscore=0 impostorscore=0 adultscore=0 lowpriorityscore=0 mlxlogscore=849 spamscore=0 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2204260060 X-Stat-Signature: cpw6qbssbqzgcujueijaht3fe61optpz X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 2DCAC40049 X-Rspam-User: Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=i8He8qGG; dmarc=pass (policy=none) header.from=ibm.com; spf=pass (imf11.hostedemail.com: domain of aneesh.kumar@linux.ibm.com designates 148.163.158.5 as permitted sender) smtp.mailfrom=aneesh.kumar@linux.ibm.com X-HE-Tag: 1650964081-499266 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 4/26/22 1:25 PM, ying.huang@intel.com wrote: > On Mon, 2022-04-25 at 16:45 +0530, Jagdish Gediya wrote: >> On Sun, Apr 24, 2022 at 11:19:53AM +0800, ying.huang@intel.com wrote: >>> On Sat, 2022-04-23 at 01:25 +0530, Jagdish Gediya wrote: >>>> Some systems(e.g. PowerVM) can have both DRAM(fast memory) only >>>> NUMA node which are N_MEMORY and slow memory(persistent memory) >>>> only NUMA node which are also N_MEMORY. As the current demotion >>>> target finding algorithm works based on N_MEMORY and best distance, >>>> it will choose DRAM only NUMA node as demotion target instead of >>>> persistent memory node on such systems. If DRAM only NUMA node is >>>> filled with demoted pages then at some point new allocations can >>>> start falling to persistent memory, so basically cold pages are in >>>> fast memor (due to demotion) and new pages are in slow memory, this >>>> is why persistent memory nodes should be utilized for demotion and >>>> dram node should be avoided for demotion so that they can be used >>>> for new allocations. >>>> >>>> Current implementation can work fine on the system where the memory >>>> only numa nodes are possible only for persistent/slow memory but it >>>> is not suitable for the like of systems mentioned above. >>> >>> Can you share the NUMA topology information of your machine? And the >>> demotion order before and after your change? >>> >>> Whether it's good to use the PMEM nodes as the demotion targets of the >>> DRAM-only node too? >> >> $ numactl -H >> available: 2 nodes (0-1) >> node 0 cpus: 0 1 2 3 4 5 6 7 >> node 0 size: 14272 MB >> node 0 free: 13392 MB >> node 1 cpus: >> node 1 size: 2028 MB >> node 1 free: 1971 MB >> node distances: >> node 0 1 >>   0: 10 40 >>   1: 40 10 >> >> 1) without N_DEMOTION_TARGETS patch series, 1 is demotion target >>    for 0 even when 1 is DRAM node and there is no demotion targets for 1. >> >> $ cat /sys/bus/nd/devices/dax0.0/target_node >> 2 >> $ >> # cd /sys/bus/dax/drivers/ >> :/sys/bus/dax/drivers# ls >> device_dax kmem >> :/sys/bus/dax/drivers# cd device_dax/ >> :/sys/bus/dax/drivers/device_dax# echo dax0.0 > unbind >> :/sys/bus/dax/drivers/device_dax# echo dax0.0 > ../kmem/new_id >> :/sys/bus/dax/drivers/device_dax# numactl -H >> available: 3 nodes (0-2) >> node 0 cpus: 0 1 2 3 4 5 6 7 >> node 0 size: 14272 MB >> node 0 free: 13380 MB >> node 1 cpus: >> node 1 size: 2028 MB >> node 1 free: 1961 MB >> node 2 cpus: >> node 2 size: 0 MB >> node 2 free: 0 MB >> node distances: >> node 0 1 2 >>   0: 10 40 80 >>   1: 40 10 80 >>   2: 80 80 10 >> > > This looks like a virtual machine, not a real machine. That's > unfortunate. I am looking forward to a real issue, not a theoritical > possible issue. > This is the source of confusion i guess. A large class of ppc64 systems are virtualized. The firmware include a hypervisor (PowerVM) and end user creates guest (aka LPAR) on them. That is the way end user will use this system. There is no baremetal access on this (There is an openpower variant, but all new systems built by IBM these days do have PowerVM on them). So this is not a theoretical possibility. -aneesh