[PATCH 1/2] mm, page_alloc: remove redundant checks from alloc fastpath

[PATCH 1/2] mm, page_alloc: remove redundant checks from alloc fastpath

[Vlastimil Babka]

The allocation fast path contains two similar checks for zoneref->zone being
NULL, where zoneref points either to the first zone in the zonelist, or to the
preferred zone. These can be NULL either due to empty zonelist, or no zone
being compatible with given nodemask or task's cpuset.

These checks are unnecessary, because the zonelist walks in
first_zones_zonelist() and get_page_from_freelist() handle a NULL starting
zoneref->zone or preferred_zoneref->zone safely. It's safe to fallback to
__alloc_pages_slowpath() where we also have the check early enough.

Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
---
This and the following patch is on top of Mel's bulk cpu work.

 mm/page_alloc.c | 18 ------------------
 1 file changed, 18 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 9288175e57e3..05068adf9007 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3912,14 +3912,6 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
 	if (should_fail_alloc_page(gfp_mask, order))
 		return false;
 
-	/*
-	 * Check the zones suitable for the gfp_mask contain at least one
-	 * valid zone. It's possible to have an empty zonelist as a result
-	 * of __GFP_THISNODE and a memoryless node
-	 */
-	if (unlikely(!ac->zonelist->_zonerefs->zone))
-		return false;
-
 	if (IS_ENABLED(CONFIG_CMA) && ac->migratetype == MIGRATE_MOVABLE)
 		*alloc_flags |= ALLOC_CMA;
 
@@ -3959,22 +3951,12 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,
 		return NULL;
 
 	finalise_ac(gfp_mask, order, &ac);
-	if (!ac.preferred_zoneref->zone) {
-		page = NULL;
-		/*
-		 * This might be due to race with cpuset_current_mems_allowed
-		 * update, so make sure we retry with original nodemask in the
-		 * slow path.
-		 */
-		goto no_zone;
-	}
 
 	/* First allocation attempt */
 	page = get_page_from_freelist(alloc_mask, order, alloc_flags, &ac);
 	if (likely(page))
 		goto out;
 
-no_zone:
 	/*
 	 * Runtime PM, block IO and its error handling path can deadlock
 	 * because I/O on the device might not complete.
-- 
2.11.0

[PATCH 2/2] mm, page_alloc: don't check cpuset allowed twice in fast-path

[Vlastimil Babka]

Since commit 682a3385e773 ("mm, page_alloc: inline the fast path of the
zonelist iterator") we replace a NULL nodemask with cpuset_current_mems_allowed
in the fast path, so that get_page_from_freelist() filters nodes allowed by the
cpuset via for_next_zone_zonelist_nodemask(). In that case it's pointless to
additionaly check __cpuset_zone_allowed() in each iteration, which we can avoid
by not adding ALLOC_CPUSET to alloc_flags in that scenario.

This saves some cycles in the allocator fast path on systems with one or more
non-root cpuset configured. In the slow path, ALLOC_CPUSET is reset according
to __alloc_pages_slowpath(). Without configured cpusets, this code is disabled
by a static key.

Signed-off-by: Vlastimil Babka <vbabka@suse.cz>
Acked-by: Michal Hocko <mhocko@suse.com>
Reviewed-by: Anshuman Khandual <khandual@linux.vnet.ibm.com>
---
 mm/page_alloc.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 05068adf9007..407e5d89ad2e 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3900,9 +3900,10 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
 
 	if (cpusets_enabled()) {
 		*alloc_mask |= __GFP_HARDWALL;
-		*alloc_flags |= ALLOC_CPUSET;
 		if (!ac->nodemask)
 			ac->nodemask = &cpuset_current_mems_allowed;
+		else
+			*alloc_flags |= ALLOC_CPUSET;
 	}
 
 	lockdep_trace_alloc(gfp_mask);
-- 
2.11.0