From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.7 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D36EC433E0 for ; Fri, 19 Jun 2020 16:24:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 28AF721707 for ; Fri, 19 Jun 2020 16:24:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 28AF721707 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6F5628D00D0; Fri, 19 Jun 2020 12:24:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6808C8D00D3; Fri, 19 Jun 2020 12:24:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D1D78D00D0; Fri, 19 Jun 2020 12:24:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0245.hostedemail.com [216.40.44.245]) by kanga.kvack.org (Postfix) with ESMTP id 2F3BB8D00D3 for ; Fri, 19 Jun 2020 12:24:33 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id E3C0E5F367 for ; Fri, 19 Jun 2020 16:24:32 +0000 (UTC) X-FDA: 76946484384.03.thumb27_0c0207926e1a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id BC74128A4EB for ; Fri, 19 Jun 2020 16:24:32 +0000 (UTC) X-HE-Tag: thumb27_0c0207926e1a X-Filterd-Recvd-Size: 8671 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Fri, 19 Jun 2020 16:24:31 +0000 (UTC) IronPort-SDR: FHHzQ8tFY1+ltRvJ657bVVSBb1+387ogcLSHtue9/DkCSMBytZz0RAYhdD5tW96dEBurXWP4oE fJORtiy1I1bw== X-IronPort-AV: E=McAfee;i="6000,8403,9657"; a="141280144" X-IronPort-AV: E=Sophos;i="5.75,255,1589266800"; d="scan'208";a="141280144" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jun 2020 09:24:29 -0700 IronPort-SDR: LhtlEfdA+55J0z77uTKRZq8AJOwmbJI8XUREPdp65EdQN7ZV09vct7JT+LcL0hAvuKbb68ZSen cU19Mhj5mDAw== X-IronPort-AV: E=Sophos;i="5.75,255,1589266800"; d="scan'208";a="264368066" Received: from sjiang-mobl2.ccr.corp.intel.com (HELO bwidawsk-mobl5.local) ([10.252.131.131]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jun 2020 09:24:28 -0700 From: Ben Widawsky To: linux-mm Cc: Ben Widawsky , Andi Kleen , Andrew Morton , Dave Hansen , Johannes Weiner , Mel Gorman , Michal Hocko , Vlastimil Babka Subject: [PATCH 04/18] mm/page_alloc: add preferred pass to page allocation Date: Fri, 19 Jun 2020 09:24:11 -0700 Message-Id: <20200619162425.1052382-5-ben.widawsky@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200619162425.1052382-1-ben.widawsky@intel.com> References: <20200619162425.1052382-1-ben.widawsky@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BC74128A4EB X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch updates the core part of page allocation (pulling from the free list) to take preferred nodes into account first. If an allocation from a preferred node cannot be found, the remaining nodes in the nodemask are checked. Intentionally not handled in this patch are OOM node scanning and reclaim scanning. I am very open and receptive on comments as to whether it is worth handling those cases with a preferred node ordering. In this patch the code first scans the preferred nodes to make the allocation, and then takes the subset of nodes in the remaining bound nodes (often this is NULL aka all nodes) - potentially two passes. Actually, the code was already two passes as it tries not to fragment on the first pass, so now it's up to 4 passes. Consider a 3 node system (0-2) passed the following masks: Preferred: 100 Bound: 110 pass 1: node 2 no fragmentation pass 2: node 1 no fragmentation pass 3: node 2 w/fragmentation pass 4: node 1 w/fragmentation Cc: Andi Kleen Cc: Andrew Morton Cc: Dave Hansen Cc: Johannes Weiner Cc: Mel Gorman Cc: Michal Hocko Cc: Vlastimil Babka Signed-off-by: Ben Widawsky --- mm/internal.h | 1 + mm/page_alloc.c | 108 +++++++++++++++++++++++++++++++++++------------- 2 files changed, 80 insertions(+), 29 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 9886db20d94f..8d16229c6cbb 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -138,6 +138,7 @@ extern pmd_t *mm_find_pmd(struct mm_struct *mm, unsig= ned long address); struct alloc_context { struct zonelist *zonelist; nodemask_t *nodemask; + nodemask_t *prefmask; struct zoneref *preferred_zoneref; int migratetype; =20 diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 280ca85dc4d8..3cf44b6c31ae 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3675,6 +3675,69 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gf= p_mask) return alloc_flags; } =20 +#ifdef CONFIG_NUMA +static void set_pref_bind_mask(nodemask_t *out, const nodemask_t *prefma= sk, + const nodemask_t *bindmask) +{ + bool has_pref, has_bind; + + has_pref =3D prefmask && !nodes_empty(*prefmask); + has_bind =3D bindmask && !nodes_empty(*bindmask); + + if (has_pref && has_bind) + nodes_and(*out, *prefmask, *bindmask); + else if (has_pref && !has_bind) + *out =3D *prefmask; + else if (!has_pref && has_bind) + *out =3D *bindmask; + else if (!has_pref && !has_bind) + *out =3D NODE_MASK_ALL; + else + unreachable(); +} +#else +#define set_pref_bind_mask(out, pref, bind) = \ + { = \ + (out)->bits[0] =3D 1UL \ + } +#endif + +/* Helper to generate the preferred and fallback nodelists */ +static void __nodemask_for_freelist_scan(const struct alloc_context *ac, + bool preferred, nodemask_t *outnodes) +{ + bool has_pref; + bool has_bind; + + if (preferred) { + set_pref_bind_mask(outnodes, ac->prefmask, ac->nodemask); + return; + } + + has_pref =3D ac->prefmask && !nodes_empty(*ac->prefmask); + has_bind =3D ac->nodemask && !nodes_empty(*ac->nodemask); + + if (!has_bind && !has_pref) { + /* + * If no preference, we already tried the full nodemask, + * so we have to bail. + */ + nodes_clear(*outnodes); + } else if (!has_bind && has_pref) { + /* We tried preferred nodes only before. Invert that. */ + nodes_complement(*outnodes, *ac->prefmask); + } else if (has_bind && !has_pref) { + /* + * If preferred was empty, we've tried all bound nodes, + * and there nothing further we can do. + */ + nodes_clear(*outnodes); + } else if (has_bind && has_pref) { + /* Try the bound nodes that weren't tried before. */ + nodes_andnot(*outnodes, *ac->nodemask, *ac->prefmask); + } +} + /* * get_page_from_freelist goes through the zonelist trying to allocate * a page. @@ -3686,7 +3749,10 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned in= t order, int alloc_flags, struct zoneref *z; struct zone *zone; struct pglist_data *last_pgdat_dirty_limit =3D NULL; - bool no_fallback; + nodemask_t nodes; + bool no_fallback, preferred_nodes_exhausted =3D false; + + __nodemask_for_freelist_scan(ac, true, &nodes); =20 retry: /* @@ -3696,7 +3762,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int= order, int alloc_flags, no_fallback =3D alloc_flags & ALLOC_NOFRAGMENT; z =3D ac->preferred_zoneref; for_next_zone_zonelist_nodemask(zone, z, ac->zonelist, - ac->highest_zoneidx, ac->nodemask) { + ac->highest_zoneidx, &nodes) + { struct page *page; unsigned long mark; =20 @@ -3816,12 +3883,20 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned i= nt order, int alloc_flags, } } =20 + if (!preferred_nodes_exhausted) { + __nodemask_for_freelist_scan(ac, false, &nodes); + preferred_nodes_exhausted =3D true; + goto retry; + } + /* * It's possible on a UMA machine to get through all zones that are * fragmented. If avoiding fragmentation, reset and try again. */ if (no_fallback) { alloc_flags &=3D ~ALLOC_NOFRAGMENT; + __nodemask_for_freelist_scan(ac, true, &nodes); + preferred_nodes_exhausted =3D false; goto retry; } =20 @@ -4763,33 +4838,6 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned in= t order, return page; } =20 -#ifndef CONFIG_NUMA -#define set_pref_bind_mask(out, pref, bind) = \ - { = \ - (out)->bits[0] =3D 1UL \ - } -#else -static void set_pref_bind_mask(nodemask_t *out, const nodemask_t *prefma= sk, - const nodemask_t *bindmask) -{ - bool has_pref, has_bind; - - has_pref =3D prefmask && !nodes_empty(*prefmask); - has_bind =3D bindmask && !nodes_empty(*bindmask); - - if (has_pref && has_bind) - nodes_and(*out, *prefmask, *bindmask); - else if (has_pref && !has_bind) - *out =3D *prefmask; - else if (!has_pref && has_bind) - *out =3D *bindmask; - else if (!has_pref && !has_bind) - unreachable(); /* Handled above */ - else - unreachable(); -} -#endif - /* * Find a zonelist from a preferred node. Here is a truth table example = using 2 * different masks. The choices are, NULL mask, empty mask, two masks wi= th an @@ -4945,6 +4993,8 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int= order, int preferred_nid, &alloc_mask, &alloc_flags)) return NULL; =20 + ac.prefmask =3D &prefmask; + finalise_ac(gfp_mask, &ac); =20 /* --=20 2.27.0