From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAC34C433B4 for ; Wed, 31 Mar 2021 16:36:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4E13661008 for ; Wed, 31 Mar 2021 16:36:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4E13661008 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C5B4C6B007E; Wed, 31 Mar 2021 12:36:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C32506B0080; Wed, 31 Mar 2021 12:36:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A5F316B0081; Wed, 31 Mar 2021 12:36:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0089.hostedemail.com [216.40.44.89]) by kanga.kvack.org (Postfix) with ESMTP id 87F146B007E for ; Wed, 31 Mar 2021 12:36:10 -0400 (EDT) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2F60C180B3079 for ; Wed, 31 Mar 2021 16:36:10 +0000 (UTC) X-FDA: 77980721700.14.5AEFA18 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf21.hostedemail.com (Postfix) with ESMTP id 52C03E000409 for ; Wed, 31 Mar 2021 16:36:09 +0000 (UTC) Received: by mail-pl1-f169.google.com with SMTP id e14so8185302plj.2 for ; Wed, 31 Mar 2021 09:36:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=MjlfWtKKNtvhcQxErgPdfEHfy5pj5T+5lDeGsL21dU0=; b=od6Qn6Sb+/X1QJjdXBdBOuWwoHtBuZuAB0iA9sjOV0bN1QYx3fSQlQlXATlCwKNaP6 moHey2gAwy/uJ/kWV6SDRwA6eOXv91z0arenc/54XPkg2uYfOLDQcQUz4gpAJqF/gpqS C1mFc/y+WHWcmQKQjrM/wtQGKPKFlrTbaQjQmRdK3fSfJVpYWmpcwYKlQqJAgQ4PCpz2 ATDdU8zVi7TT6IMDtAy2mKRcUrvg2H1EnU1VbHMq+83kom0IceX7x0ECvwfVHrzqinv3 jZLrWNGCz4wkoxfs83j6fzh+OWJ5RTl2199wGzUG8vJvCE2hLgtGmYd8iYQFVSR1qcdl h8jw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=MjlfWtKKNtvhcQxErgPdfEHfy5pj5T+5lDeGsL21dU0=; b=lqf2QlWJS4v2B0b24e55l4PbojaFnX55v2VAGtfe1/dDQas/NA52tFclWe8wgC8lFr 3ed+PNYMHOe5gDuDbZ+tiFGL9XFyKdlGYAUfqywAv3pyvuSCG1QKGXCJLBmwZf1nRJ+Q aqApccE6/7Qo6g0aaLsAcRAMxVJjeO1/VroD78SMhCyoZMS4xn+8LIx6+HrcmMFuOnMS +MiMLPmpw5hiy+TREiflpTgWdaukjcDZuPzeny6WLzONDri0NbRp8p26jF8PTaIJbod8 D6Ye9Rh833zg/C3LtmvcKut6ZBC/XmNAc9NNen1rRuT2MCkdsUHY88tsP7LAS96sEkOn 3vzg== X-Gm-Message-State: AOAM530Qmnm/NjMPBvPr/vQr1GsFpfpa8ohJO0ktSmahc5e0fFq8jC5Y 8dD8md7cWM4XQu0CQV0jVj0= X-Google-Smtp-Source: ABdhPJx3c8WHn6bdxe5Bt63IFgNZSfHCKpFVP5drDSjfY+QBjOKt6lfi1az0cSeKWaG2PM8eK3In3A== X-Received: by 2002:a17:90a:5d09:: with SMTP id s9mr4245708pji.228.1617208568531; Wed, 31 Mar 2021 09:36:08 -0700 (PDT) Received: from [192.168.88.245] (c-24-6-216-183.hsd1.ca.comcast.net. [24.6.216.183]) by smtp.gmail.com with ESMTPSA id 205sm3061907pfc.201.2021.03.31.09.36.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 31 Mar 2021 09:36:07 -0700 (PDT) From: Nadav Amit Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_F337632E-750D-4B46-8772-8021968CA3B8"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.60.0.2.21\)) Subject: Re: [RFC] NUMA balancing: reduce TLB flush via delaying mapping on hint page fault Date: Wed, 31 Mar 2021 09:36:04 -0700 In-Reply-To: <20210331131658.GV15768@suse.de> Cc: Linux-MM , Andrew Morton , LKML , Peter Zijlstra , Peter Xu , Johannes Weiner , Vlastimil Babka , Matthew Wilcox , Will Deacon , Michel Lespinasse , Arjun Roy , "Kirill A. Shutemov" , Mel Gorman To: "Huang, Ying" References: <20210329062651.2487905-1-ying.huang@intel.com> <20210330133310.GT15768@suse.de> <87a6qj8t92.fsf@yhuang6-desk1.ccr.corp.intel.com> <20210331131658.GV15768@suse.de> X-Mailer: Apple Mail (2.3654.60.0.2.21) X-Rspamd-Queue-Id: 52C03E000409 X-Stat-Signature: zg3nwuuk7tgr3et539jqghnpw788m4du X-Rspamd-Server: rspam02 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf21; identity=mailfrom; envelope-from=""; helo=mail-pl1-f169.google.com; client-ip=209.85.214.169 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1617208569-336017 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: --Apple-Mail=_F337632E-750D-4B46-8772-8021968CA3B8 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Mar 31, 2021, at 6:16 AM, Mel Gorman wrote: >=20 > On Wed, Mar 31, 2021 at 07:20:09PM +0800, Huang, Ying wrote: >> Mel Gorman writes: >>=20 >>> On Mon, Mar 29, 2021 at 02:26:51PM +0800, Huang Ying wrote: >>>> For NUMA balancing, in hint page fault handler, the faulting page = will >>>> be migrated to the accessing node if necessary. During the = migration, >>>> TLB will be shot down on all CPUs that the process has run on >>>> recently. Because in the hint page fault handler, the PTE will be >>>> made accessible before the migration is tried. The overhead of TLB >>>> shooting down is high, so it's better to be avoided if possible. = In >>>> fact, if we delay mapping the page in PTE until migration, that can = be >>>> avoided. This is what this patch doing. >>>>=20 >>>=20 >>> Why would the overhead be high? It was previously inaccessibly so = it's >>> only parallel accesses making forward progress that trigger the need >>> for a flush. >>=20 >> Sorry, I don't understand this. Although the page is inaccessible, = the >> threads may access other pages, so TLB flushing is still necessary. >>=20 >=20 > You assert the overhead of TLB shootdown is high and yes, it can be > very high but you also said "the benchmark score has no visible = changes" > indicating the TLB shootdown cost is not a major problem for the = workload. > It does not mean we should ignore it though. If you are looking for a benchmark that is negatively affected by NUMA balancing, then IIRC Parsec=E2=80=99s dedup is such a workload. [1] [1] https://parsec.cs.princeton.edu/ --Apple-Mail=_F337632E-750D-4B46-8772-8021968CA3B8 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEESJL3osl5Ymx/w9I1HaAqSabaD1oFAmBkpPQACgkQHaAqSaba D1pAvBAAjzpeqjamVJvvm0jAmkjIyImbd6zd8mCnwDXmKVaGuhPVi329I8qadJ+u /eq0a+xVd1Rm2FbhEi4LT7LoccrIDe8959f7thT0TDIdaUGNuO83qJRIHQQZiyEL dxwFWIfwHHvoXjGgRZZ/yHttCGNhbyh/J9RToOrEURQJm8EniBo1Ik2X2BdP2bMc eLWzHMiXn8flthpiIrNXwKz9oThlVXg9U3VJEVzNOP5y8L9yTkg1iwhzDxSOZlNH 78/4EXTYx8PS0UD+0KLlO3d3J2sFiYnPRmBAD/oBxDBYGfprcV9CHsaJUlEh6F8V +joOQfMvpKZNah74gw1A4JsE/8fEJGubfzZ3g+QQV40dA03xP7+mQIX5llo2h2a1 +u3rLDygT1wLzZK8jsmG6GEFcL+4JhCEMOlBq/CEtb2nAourfi9einvbWanS9bCq Ytw5sMsc/At6KWg4uQiPuM75edvPdUjgokaE/dVZRQwesUt98dj9mjLNR44mu8N2 QI5fU5J4IDr1jR2SQrAfMRYgQQU3eftaOVf6Quvh63h2rXRv9YpmG/C2pqZcYF/Q CA0PiqV1jB6Z+yxvTLoUuJXnCC0pl5vEhdPR3BmEDdbkDuTNmJkTiD9Pk5jEWTuR 5qnebPt+zsIjgWrO9sppk8GuYHOVAtRQuE73Nhj/d4Bxi42m4GE= =i6Am -----END PGP SIGNATURE----- --Apple-Mail=_F337632E-750D-4B46-8772-8021968CA3B8--