From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.7 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0787DC433E1 for ; Fri, 19 Jun 2020 16:25:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CA34D217BA for ; Fri, 19 Jun 2020 16:25:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CA34D217BA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 261D98D00EC; Fri, 19 Jun 2020 12:24:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EC318D00E9; Fri, 19 Jun 2020 12:24:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 068218D00EC; Fri, 19 Jun 2020 12:24:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0197.hostedemail.com [216.40.44.197]) by kanga.kvack.org (Postfix) with ESMTP id DEE338D00E9 for ; Fri, 19 Jun 2020 12:24:52 -0400 (EDT) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 9FBEF15BEC2 for ; Fri, 19 Jun 2020 16:24:52 +0000 (UTC) X-FDA: 76946485224.13.steam84_1b046b326e1a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id A245B181411C5 for ; Fri, 19 Jun 2020 16:24:36 +0000 (UTC) X-HE-Tag: steam84_1b046b326e1a X-Filterd-Recvd-Size: 6540 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Fri, 19 Jun 2020 16:24:35 +0000 (UTC) IronPort-SDR: 8vMb2O253zWB3TDqVyfe/+SGIn62EZZi7EmVvURHzpHTo9M6V60S13Ki4ysUAQMnrmV1kBqmrD +EP6zvYa2SEQ== X-IronPort-AV: E=McAfee;i="6000,8403,9657"; a="141280165" X-IronPort-AV: E=Sophos;i="5.75,255,1589266800"; d="scan'208";a="141280165" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jun 2020 09:24:31 -0700 IronPort-SDR: nx13Rsqul0tYp3FcKnRS1dWP4vZKJT58vvpM/bohsOEdFL+zQS2ZIkYmwpbgXMW/7b1OeF2JQx utbzQgMz4NPQ== X-IronPort-AV: E=Sophos;i="5.75,255,1589266800"; d="scan'208";a="264368298" Received: from sjiang-mobl2.ccr.corp.intel.com (HELO bwidawsk-mobl5.local) ([10.252.131.131]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jun 2020 09:24:30 -0700 From: Ben Widawsky To: linux-mm Cc: Ben Widawsky , Andrew Morton , Dave Hansen , Michal Hocko Subject: [PATCH 11/18] mm: Extract THP hugepage allocation Date: Fri, 19 Jun 2020 09:24:18 -0700 Message-Id: <20200619162425.1052382-12-ben.widawsky@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200619162425.1052382-1-ben.widawsky@intel.com> References: <20200619162425.1052382-1-ben.widawsky@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A245B181411C5 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The next patch is going to rework this code to support MPOL_PREFERRED_MANY. This refactor makes the that change much more readable. After the extraction, the resulting code makes it apparent that this can be converted to a simple if ladder and thus allows removing the goto. There is not meant to be any functional or behavioral changes. Note that still at this point MPOL_PREFERRED_MANY isn't specially handled for huge pages. Cc: Andrew Morton Cc: Dave Hansen Cc: Michal Hocko Signed-off-by: Ben Widawsky --- mm/mempolicy.c | 96 ++++++++++++++++++++++++++------------------------ 1 file changed, 49 insertions(+), 47 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 408ba78c8424..3ce2354fed44 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2232,6 +2232,48 @@ static struct page *alloc_page_interleave(gfp_t gf= p, unsigned order, return page; } =20 +static struct page *alloc_pages_vma_thp(gfp_t gfp, struct mempolicy *pol= , + int order, int node) +{ + nodemask_t *nmask; + struct page *page; + int hpage_node =3D node; + + /* + * For hugepage allocation and non-interleave policy which allows the + * current node (or other explicitly preferred node) we only try to + * allocate from the current/preferred node and don't fall back to othe= r + * nodes, as the cost of remote accesses would likely offset THP + * benefits. + * + * If the policy is interleave or multiple preferred nodes, or does not + * allow the current node in its nodemask, we allocate the standard way= . + */ + if (pol->mode =3D=3D MPOL_PREFERRED && !(pol->flags & MPOL_F_LOCAL)) + hpage_node =3D first_node(pol->v.preferred_nodes); + + nmask =3D policy_nodemask(gfp, pol); + + /* + * First, try to allocate THP only on local node, but don't reclaim + * unnecessarily, just compact. + */ + page =3D __alloc_pages_nodemask(gfp | __GFP_THISNODE | __GFP_NORETRY, + order, hpage_node, nmask); + + /* + * If hugepage allocations are configured to always synchronous compact + * or the vma has been madvised to prefer hugepage backing, retry + * allowing remote memory with both reclaim and compact as well. + */ + if (!page && (gfp & __GFP_DIRECT_RECLAIM)) + page =3D __alloc_pages_nodemask(gfp, order, hpage_node, nmask); + + VM_BUG_ON(page && nmask && !node_isset(page_to_nid(page), *nmask)); + + return page; +} + /** * alloc_pages_vma - Allocate a page for a VMA. * @@ -2272,57 +2314,17 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_a= rea_struct *vma, nid =3D interleave_nid(pol, vma, addr, PAGE_SHIFT + order); mpol_cond_put(pol); page =3D alloc_page_interleave(gfp, order, nid); - goto out; - } - - if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && hugepage)) { - int hpage_node =3D node; - - /* - * For hugepage allocation and non-interleave policy which - * allows the current node (or other explicitly preferred - * node) we only try to allocate from the current/preferred - * node and don't fall back to other nodes, as the cost of - * remote accesses would likely offset THP benefits. - * - * If the policy is interleave or multiple preferred nodes, or - * does not allow the current node in its nodemask, we allocate - * the standard way. - */ - if (pol->mode =3D=3D MPOL_PREFERRED && !(pol->flags & MPOL_F_LOCAL)) - hpage_node =3D first_node(pol->v.preferred_nodes); - + } else if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && + hugepage)) { + page =3D alloc_pages_vma_thp(gfp, pol, order, node); + mpol_cond_put(pol); + } else { nmask =3D policy_nodemask(gfp, pol); + preferred_nid =3D policy_node(gfp, pol, node); + page =3D __alloc_pages_nodemask(gfp, order, preferred_nid, nmask); mpol_cond_put(pol); - - /* - * First, try to allocate THP only on local node, but - * don't reclaim unnecessarily, just compact. - */ - page =3D __alloc_pages_nodemask(gfp | __GFP_THISNODE | - __GFP_NORETRY, - order, hpage_node, nmask); - - /* - * If hugepage allocations are configured to always synchronous - * compact or the vma has been madvised to prefer hugepage - * backing, retry allowing remote memory with both reclaim and - * compact as well. - */ - if (!page && (gfp & __GFP_DIRECT_RECLAIM)) - page =3D __alloc_pages_nodemask(gfp, order, hpage_node, - nmask); - - VM_BUG_ON(page && nmask && - !node_isset(page_to_nid(page), *nmask)); - goto out; } =20 - nmask =3D policy_nodemask(gfp, pol); - preferred_nid =3D policy_node(gfp, pol, node); - page =3D __alloc_pages_nodemask(gfp, order, preferred_nid, nmask); - mpol_cond_put(pol); -out: return page; } EXPORT_SYMBOL(alloc_pages_vma); --=20 2.27.0