From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B432C4742C for ; Wed, 4 Nov 2020 05:37:08 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AAC0D22277 for ; Wed, 4 Nov 2020 05:37:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AAC0D22277 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BC5ED6B0036; Wed, 4 Nov 2020 00:37:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B4EFF6B005D; Wed, 4 Nov 2020 00:37:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A16FD6B0068; Wed, 4 Nov 2020 00:37:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0036.hostedemail.com [216.40.44.36]) by kanga.kvack.org (Postfix) with ESMTP id 714F46B0036 for ; Wed, 4 Nov 2020 00:37:06 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id EF24D180AD802 for ; Wed, 4 Nov 2020 05:37:05 +0000 (UTC) X-FDA: 77445627210.07.waves00_1e03b28272be Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id D7A141803F9A5 for ; Wed, 4 Nov 2020 05:37:05 +0000 (UTC) X-HE-Tag: waves00_1e03b28272be X-Filterd-Recvd-Size: 6135 Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Wed, 4 Nov 2020 05:37:04 +0000 (UTC) IronPort-SDR: dzNakk/ULVsHV8yJg1ZtFAplCc1DoIL4LjWQCjVUD/59QluxRFGG9CPri/mNfcW9wd+en5HcAh KRxklgiuEhDA== X-IronPort-AV: E=McAfee;i="6000,8403,9794"; a="156944141" X-IronPort-AV: E=Sophos;i="5.77,450,1596524400"; d="scan'208";a="156944141" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Nov 2020 21:37:02 -0800 IronPort-SDR: JBmoy5Ga3Tv7yI/iYiNFRnrmamXSlr95HUuxW8mrShOJ9hOCP5H+/0OLKwJYCiCQuy+9dQD5Si T/ABmZUIAaIw== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.77,450,1596524400"; d="scan'208";a="325484137" Received: from yhuang-dev.sh.intel.com (HELO yhuang-dev) ([10.239.159.65]) by orsmga006.jf.intel.com with ESMTP; 03 Nov 2020 21:36:59 -0800 From: "Huang\, Ying" To: Mel Gorman Cc: Peter Zijlstra , , , Andrew Morton , Ingo Molnar , Rik van Riel , "Johannes Weiner" , "Matthew Wilcox \(Oracle\)" , Dave Hansen , Andi Kleen , Michal Hocko , David Rientjes Subject: Re: [PATCH -V2 2/2] autonuma: Migrate on fault among multiple bound nodes References: <20201028023411.15045-1-ying.huang@intel.com> <20201028023411.15045-3-ying.huang@intel.com> <20201102111717.GB3306@suse.de> Date: Wed, 04 Nov 2020 13:36:58 +0800 In-Reply-To: <20201102111717.GB3306@suse.de> (Mel Gorman's message of "Mon, 2 Nov 2020 11:17:17 +0000") Message-ID: <87eel9wumd.fsf@yhuang-dev.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, Mel, Thanks for comments! Mel Gorman writes: > On Wed, Oct 28, 2020 at 10:34:11AM +0800, Huang Ying wrote: >> Now, AutoNUMA can only optimize the page placement among the NUMA nodes if the >> default memory policy is used. Because the memory policy specified explicitly >> should take precedence. But this seems too strict in some situations. For >> example, on a system with 4 NUMA nodes, if the memory of an application is bound >> to the node 0 and 1, AutoNUMA can potentially migrate the pages between the node >> 0 and 1 to reduce cross-node accessing without breaking the explicit memory >> binding policy. >> >> So in this patch, if mbind(.mode=MPOL_BIND, .flags=MPOL_MF_LAZY) is used to bind >> the memory of the application to multiple nodes, and in the hint page fault >> handler both the faulting page node and the accessing node are in the policy >> nodemask, the page will be tried to be migrated to the accessing node to reduce >> the cross-node accessing. >> >> [Peter Zijlstra: provided the simplified implementation method.] >> >> Questions: >> >> Sysctl knob kernel.numa_balancing can enable/disable AutoNUMA optimizing >> globally. But for the memory areas that are bound to multiple NUMA nodes, even >> if the AutoNUMA is enabled globally via the sysctl knob, we still need to enable >> AutoNUMA again with a special flag. Why not just optimize the page placement if >> possible as long as AutoNUMA is enabled globally? The interface would look >> simpler with that. >> >> Signed-off-by: "Huang, Ying" > > I've no specific objection to the patch or the name change. I can't > remember exactly why I picked the name, it was 8 years ago but I think it > was because the policy represented the most basic possible approach that > could be done without any attempt at being intelligent and established > a baseline. The intent was that anything built on top had to be better > than the most basic policy imaginable. The name reflected the dictionary > definition at the time and happened to match the acronym closely enough > and I wanted to make it absolutely clear to reviewers that the policy > was not good enough (ruling out MPOL_BASIC or variants thereof) even if > it happened to work for some workload and there was no intent to report > it to the userspace API. > > The only hazard with the patch is that applications that use MPOL_BIND > on multiple nodes may now incur some NUMA balancing overhead due to > trapping faults and migrations. For this specific version of patch, I don't think this will happen. Because now, MPOL_F_MOF need to be set in struct mempolicy, for MPOL_BIND, only if mbind() syscall is called with MPOL_MF_LAZY, that will be the case. So I think most workloads will not be affected by this patch. The feature is opt-in. But from another point of view, I suggest to remove the constraints of MPOL_F_MOF in the future. If the overhead of AutoNUMA isn't acceptable, why not just disable AutoNUMA globally via sysctl knob? > It might still end up being better but I was not aware of a > *realistic* workload that binds to multiple nodes > deliberately. Generally I expect if an application is binding, it's > binding to one local node. Yes. It's not popular configuration for now. But for the memory tiering system with both DRAM and PMEM, the DRAM and PMEM in one socket will become 2 NUMA nodes. To avoid too much cross-socket memory accessing, but take advantage of both the DRAM and PMEM, the workload can be bound to 2 NUMA nodes (DRAM and PMEM). > If it shows up in regressions, it'll be interesting to get a detailed > description of the workload. Pay particular attention to if THP is > disabled as I learned relatively recently that NUMA balancing with THP > disabled has higher overhead (which is hardly surprising). Got it! > Lacking data or a specific objection > > Acked-by: Mel Gorman Thanks! Best Regards, Huang, Ying