From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 033C1C433B4 for ; Thu, 1 Apr 2021 08:38:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7C6BC60190 for ; Thu, 1 Apr 2021 08:38:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7C6BC60190 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 08DC06B0081; Thu, 1 Apr 2021 04:38:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0511E6B0082; Thu, 1 Apr 2021 04:38:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5BF16B0083; Thu, 1 Apr 2021 04:38:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0071.hostedemail.com [216.40.44.71]) by kanga.kvack.org (Postfix) with ESMTP id CBCCF6B0081 for ; Thu, 1 Apr 2021 04:38:13 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 84AA08134 for ; Thu, 1 Apr 2021 08:38:13 +0000 (UTC) X-FDA: 77983146066.28.5102333 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf04.hostedemail.com (Postfix) with ESMTP id 2EFC313A for ; Thu, 1 Apr 2021 08:38:12 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id CA413AE92; Thu, 1 Apr 2021 08:38:11 +0000 (UTC) Date: Thu, 1 Apr 2021 09:38:09 +0100 From: Mel Gorman To: Nadav Amit Cc: "Huang, Ying" , Linux-MM , Andrew Morton , LKML , Peter Zijlstra , Peter Xu , Johannes Weiner , Vlastimil Babka , Matthew Wilcox , Will Deacon , Michel Lespinasse , Arjun Roy , "Kirill A. Shutemov" Subject: Re: [RFC] NUMA balancing: reduce TLB flush via delaying mapping on hint page fault Message-ID: <20210401083809.GX15768@suse.de> References: <20210329062651.2487905-1-ying.huang@intel.com> <20210330133310.GT15768@suse.de> <87a6qj8t92.fsf@yhuang6-desk1.ccr.corp.intel.com> <20210331131658.GV15768@suse.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="8GpibOaaTibBMecb" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.10.1 (2018-07-13) X-Stat-Signature: qaoakq3smqbdtcejnxbwrrhh8uythucr X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2EFC313A Received-SPF: none (suse.de>: No applicable sender policy available) receiver=imf04; identity=mailfrom; envelope-from=""; helo=mx2.suse.de; client-ip=195.135.220.15 X-HE-DKIM-Result: none/none X-HE-Tag: 1617266292-207489 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --8GpibOaaTibBMecb Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 31, 2021 at 09:36:04AM -0700, Nadav Amit wrote: >=20 >=20 > > On Mar 31, 2021, at 6:16 AM, Mel Gorman wrote: > >=20 > > On Wed, Mar 31, 2021 at 07:20:09PM +0800, Huang, Ying wrote: > >> Mel Gorman writes: > >>=20 > >>> On Mon, Mar 29, 2021 at 02:26:51PM +0800, Huang Ying wrote: > >>>> For NUMA balancing, in hint page fault handler, the faulting page wi= ll > >>>> be migrated to the accessing node if necessary. During the migratio= n, > >>>> TLB will be shot down on all CPUs that the process has run on > >>>> recently. Because in the hint page fault handler, the PTE will be > >>>> made accessible before the migration is tried. The overhead of TLB > >>>> shooting down is high, so it's better to be avoided if possible. In > >>>> fact, if we delay mapping the page in PTE until migration, that can = be > >>>> avoided. This is what this patch doing. > >>>>=20 > >>>=20 > >>> Why would the overhead be high? It was previously inaccessibly so it's > >>> only parallel accesses making forward progress that trigger the need > >>> for a flush. > >>=20 > >> Sorry, I don't understand this. Although the page is inaccessible, the > >> threads may access other pages, so TLB flushing is still necessary. > >>=20 > >=20 > > You assert the overhead of TLB shootdown is high and yes, it can be > > very high but you also said "the benchmark score has no visible changes" > > indicating the TLB shootdown cost is not a major problem for the worklo= ad. > > It does not mean we should ignore it though. >=20 > If you are looking for a benchmark that is negatively affected by NUMA > balancing, then IIRC Parsec???s dedup is such a workload. [1] >=20 Few questions; Is Parsec imparied due to NUMA balancing in general or due to TLB shootdowns specifically? Are you using "gcc-pthreads" for parallelisation and the "native" size for Parsec? Is there any specific thread count that matters either in absolute terms or as a precentage of online CPUs? --=20 Mel Gorman SUSE Labs --8GpibOaaTibBMecb Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEElcbIJ2qkxLDKryriKjSY26pIcMkFAmBlhnEACgkQKjSY26pI cMkZqwf/eRNLVsSneirGm6M0609dSk2Vg3k6VeqVCLyJCJu4kVFS3HRiG61MyUsv S0edPNITHFcJFpql9Tf7U7Z1tjQQ8oUZR2KJjIw/lrPBqJu+rW4hu1F1VYfeJHqG Q6zMR2rsFBNe6kR0hW64e3I8A1rUFVzLrc+HyPQN33UkOjuLZzptUOhXOgVC+80s KBIPdyAhmDpk80MHy0hwKufbyCH3PcJa1pQ6XIKDQ/na+eTPOgliZayQtKGPs4iV yghAaPdSNeYftM0NJ4M3WaizOgRTQGQFA6mTaOC+MMuN5h4fEPTcQgwxWbEhQBh4 4mOTvXRbxvhpkcirFa7DuOhsEO+mMw== =gTdL -----END PGP SIGNATURE----- --8GpibOaaTibBMecb--