[PATCH 00/18] multiple preferred nodes

[PATCH 00/18] multiple preferred nodes

[Ben Widawsky]

This patch series introduces the concept of the MPOL_PREFERRED_MANY mempolicy.
This mempolicy mode can be used with either the set_mempolicy(2) or mbind(2)
interfaces. Like the MPOL_PREFERRED interface, it allows an application to set a
preference for nodes which will fulfil memory allocation requests. Like the
MPOL_BIND interface, it works over a set of nodes.

Summary:
1-2: Random fixes I found along the way
3-4: Logic to handle many preferred nodes in page allocation
5-9: Plumbing to allow multiple preferred nodes in mempolicy
10-13: Teach page allocation APIs about nodemasks
14: Provide a helper to generate preferred nodemasks
15: Have page allocation callers generate preferred nodemasks
16-17: Flip the switch to have __alloc_pages_nodemask take preferred mask.
18: Expose the new uapi

Along with these patches are patches for libnuma, numactl, numademo, and memhog.
They still need some polish, but can be found here:
https://gitlab.com/bwidawsk/numactl/-/tree/prefer-many
It allows new usage: `numactl -P 0,3,4`

The goal of the new mode is to enable some use-cases when using tiered memory
usage models which I've lovingly named.
1a. The Hare - The interconnect is fast enough to meet bandwidth and latency
requirements allowing preference to be given to all nodes with "fast" memory.
1b. The Indiscriminate Hare - An application knows it wants fast memory (or
perhaps slow memory), but doesn't care which node it runs on. The application
can prefer a set of nodes and then xpu bind to the local node (cpu, accelerator,
etc). This reverses the nodes are chosen today where the kernel attempts to use
local memory to the CPU whenever possible. This will attempt to use the local
accelerator to the memory.
2. The Tortoise - The administrator (or the application itself) is aware it only
needs slow memory, and so can prefer that.

Much of this is almost achievable with the bind interface, but the bind
interface suffers from an inability to fallback to another set of nodes if
binding fails to all nodes in the nodemask.

Like MPOL_BIND a nodemask is given. Inherently this removes ordering from the
preference.

> /* Set first two nodes as preferred in an 8 node system. */
> const unsigned long nodes = 0x3
> set_mempolicy(MPOL_PREFER_MANY, &nodes, 8);

> /* Mimic interleave policy, but have fallback *.
> const unsigned long nodes = 0xaa
> set_mempolicy(MPOL_PREFER_MANY, &nodes, 8);

Some internal discussion took place around the interface. There are two
alternatives which we have discussed, plus one I stuck in:
1. Ordered list of nodes. Currently it's believed that the added complexity is
   nod needed for expected usecases.
2. A flag for bind to allow falling back to other nodes. This confuses the
   notion of binding and is less flexible than the current solution.
3. Create flags or new modes that helps with some ordering. This offers both a
   friendlier API as well as a solution for more customized usage. It's unknown
   if it's worth the complexity to support this. Here is sample code for how
   this might work:

> // Default
> set_mempolicy(MPOL_PREFER_MANY | MPOL_F_PREFER_ORDER_SOCKET, NULL, 0);
> // which is the same as
> set_mempolicy(MPOL_DEFAULT, NULL, 0);
>
> // The Hare
> set_mempolicy(MPOL_PREFER_MANY | MPOL_F_PREFER_ORDER_TYPE, NULL, 0);
>
> // The Tortoise
> set_mempolicy(MPOL_PREFER_MANY | MPOL_F_PREFER_ORDER_TYPE_REV, NULL, 0);
>
> // Prefer the fast memory of the first two sockets
> set_mempolicy(MPOL_PREFER_MANY | MPOL_F_PREFER_ORDER_TYPE, -1, 2);
>
> // Prefer specific nodes for some something wacky
> set_mempolicy(MPOL_PREFER_MANY | MPOL_F_PREFER_ORDER_TYPE_CUSTOM, 0x17c, 1024);

---

Cc: Andi Kleen <ak@linux.intel.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Christoph Lameter <cl@linux.com>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: David Rientjes <rientjes@google.com>
Cc: Jason Gunthorpe <jgg@ziepe.ca>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: Li Xinhai <lixinhai.lxh@gmail.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Mike Kravetz <mike.kravetz@oracle.com>
Cc: Mina Almasry <almasrymina@google.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: Vlastimil Babka <vbabka@suse.cz>

Ben Widawsky (14):
  mm/mempolicy: Add comment for missing LOCAL
  mm/mempolicy: Use node_mem_id() instead of node_id()
  mm/page_alloc: start plumbing multi preferred node
  mm/page_alloc: add preferred pass to page allocation
  mm: Finish handling MPOL_PREFERRED_MANY
  mm: clean up alloc_pages_vma (thp)
  mm: Extract THP hugepage allocation
  mm/mempolicy: Use __alloc_page_node for interleaved
  mm: kill __alloc_pages
  mm/mempolicy: Introduce policy_preferred_nodes()
  mm: convert callers of __alloc_pages_nodemask to pmask
  alloc_pages_nodemask: turn preferred nid into a nodemask
  mm: Use less stack for page allocations
  mm/mempolicy: Advertise new MPOL_PREFERRED_MANY

Dave Hansen (4):
  mm/mempolicy: convert single preferred_node to full nodemask
  mm/mempolicy: Add MPOL_PREFERRED_MANY for multiple preferred nodes
  mm/mempolicy: allow preferred code to take a nodemask
  mm/mempolicy: refactor rebind code for PREFERRED_MANY

 .../admin-guide/mm/numa_memory_policy.rst     |  22 +-
 include/linux/gfp.h                           |  19 +-
 include/linux/mempolicy.h                     |   4 +-
 include/linux/migrate.h                       |   4 +-
 include/linux/mmzone.h                        |   3 +
 include/uapi/linux/mempolicy.h                |   6 +-
 mm/hugetlb.c                                  |  10 +-
 mm/internal.h                                 |   1 +
 mm/mempolicy.c                                | 271 +++++++++++++-----
 mm/page_alloc.c                               | 179 +++++++++++-
 10 files changed, 403 insertions(+), 116 deletions(-)


-- 
2.27.0

[PATCH 01/18] mm/mempolicy: Add comment for missing LOCAL

[Ben Widawsky]

MPOL_LOCAL is a bit weird because it is simply a different name for an
existing behavior (preferred policy with no node mask). It has been this
way since it was added here:
commit 479e2802d09f ("mm: mempolicy: Make MPOL_LOCAL a real policy")

It is so similar to MPOL_PREFERRED in fact that when the policy is
created in mpol_new, the mode is set as PREFERRED, and an internal state
representing LOCAL doesn't exist.

To prevent future explorers from scratching their head as to why
MPOL_LOCAL isn't defined in the mpol_ops table, add a small comment
explaining the situations.

Cc: Christoph Lameter <cl@linux.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: David Rientjes <rientjes@google.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 mm/mempolicy.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 381320671677..36ee3267c25f 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -427,6 +427,7 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = {
 		.create = mpol_new_bind,
 		.rebind = mpol_rebind_nodemask,
 	},
+	/* MPOL_LOCAL is converted to MPOL_PREFERRED on policy creation */
 };
 
 static int migrate_page_add(struct page *page, struct list_head *pagelist,
-- 
2.27.0

[PATCH 02/18] mm/mempolicy: Use node_mem_id() instead of node_id()

[Ben Widawsky]

Calling out some distinctions first as I understand it, and the
reasoning of the patch:
numa_node_id() - The node id for the currently running CPU.
numa_mem_id() - The node id for the closest memory node.

The case where they are not the same is CONFIG_HAVE_MEMORYLESS_NODES.
Only ia64 and powerpc support this option, so it is perhaps not a very
interesting situation to most.

The question is, when you do want which? numa_node_id() is definitely
what's desired if MPOL_PREFERRED, or MPOL_LOCAL were used, since the ABI
states "This mode specifies "local allocation"; the memory is allocated
on the node of the CPU that triggered the allocation (the "local
node")." It would be weird, though not impossible to set this policy on
a CPU that has memoryless nodes. A more likely way to hit this is with
interleaving. The current interfaces will return some equally weird
thing, but at least it's symmetric. Therefore, in cases where the node
is being queried for the currently running process, it probably makes
sense to use numa_node_id(). For other cases however, when CPU is trying
to obtain the "local" memory, numa_mem_id() already contains this and
should be used instead.

This really should only effect configurations where
CONFIG_HAVE_MEMORYLESS_NODES=y, and even on those machines it's quite
possible the ultimate behavior would be identical.

Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 mm/mempolicy.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 36ee3267c25f..99e0f3f9c4a6 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1991,7 +1991,7 @@ static unsigned offset_il_node(struct mempolicy *pol, unsigned long n)
 	int nid;
 
 	if (!nnodes)
-		return numa_node_id();
+		return numa_mem_id();
 	target = (unsigned int)n % nnodes;
 	nid = first_node(pol->v.nodes);
 	for (i = 0; i < target; i++)
@@ -2049,7 +2049,7 @@ int huge_node(struct vm_area_struct *vma, unsigned long addr, gfp_t gfp_flags,
 		nid = interleave_nid(*mpol, vma, addr,
 					huge_page_shift(hstate_vma(vma)));
 	} else {
-		nid = policy_node(gfp_flags, *mpol, numa_node_id());
+		nid = policy_node(gfp_flags, *mpol, numa_mem_id());
 		if ((*mpol)->mode == MPOL_BIND)
 			*nodemask = &(*mpol)->v.nodes;
 	}
-- 
2.27.0

[PATCH 03/18] mm/page_alloc: start plumbing multi preferred node

[Ben Widawsky]

In preparation for supporting multiple preferred nodes, we need the
internals to switch from taking a nid to a nodemask.

As mentioned in the code as a comment, __alloc_pages_nodemask() is the
heart of the page allocator. It takes a single node as a preferred node
to try to obtain a zonelist from first. This patch leaves that internal
interface in place, but changes the guts of the function to consider a
list of preferred nodes.

The local node is always most preferred. If the local node is either
restricted because of preference or binding, then the closest node that
meets both the binding and preference criteria is used. If the
intersection of binding and preference is the empty set, then fall back
to the first node the meets binding criteria.

As of this patch, multiple preferred nodes aren't actually supported as
it might seem initially. As an example, suppose your preferred nodes
are 0, and 1. Node 0's fallback zone list may have zones from nodes
ordered 0->2->1. If this code were to pick 0's zonelist, and all zones
from node 0 were full, you'd get a zone from node 2 instead of 1. As
multiple nodes aren't yet supported anyway, this is okay just as a
preparatory patch.

v2:
Fixed memory hotplug handling (Ben)

Cc: Andi Kleen <ak@linux.intel.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Michal Hocko <mhocko@kernel.org>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 mm/page_alloc.c | 125 +++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 119 insertions(+), 6 deletions(-)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 48eb0f1410d4..280ca85dc4d8 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -129,6 +129,10 @@ nodemask_t node_states[NR_NODE_STATES] __read_mostly = {
 };
 EXPORT_SYMBOL(node_states);
 
+#ifdef CONFIG_NUMA
+static int find_next_best_node(int node, nodemask_t *used_node_mask);
+#endif
+
 atomic_long_t _totalram_pages __read_mostly;
 EXPORT_SYMBOL(_totalram_pages);
 unsigned long totalreserve_pages __read_mostly;
@@ -4759,13 +4763,118 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	return page;
 }
 
-static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
-		int preferred_nid, nodemask_t *nodemask,
-		struct alloc_context *ac, gfp_t *alloc_mask,
-		unsigned int *alloc_flags)
+#ifndef CONFIG_NUMA
+#define set_pref_bind_mask(out, pref, bind)                                    \
+	{                                                                      \
+		(out)->bits[0] = 1UL                                           \
+	}
+#else
+static void set_pref_bind_mask(nodemask_t *out, const nodemask_t *prefmask,
+			       const nodemask_t *bindmask)
+{
+	bool has_pref, has_bind;
+
+	has_pref = prefmask && !nodes_empty(*prefmask);
+	has_bind = bindmask && !nodes_empty(*bindmask);
+
+	if (has_pref && has_bind)
+		nodes_and(*out, *prefmask, *bindmask);
+	else if (has_pref && !has_bind)
+		*out = *prefmask;
+	else if (!has_pref && has_bind)
+		*out = *bindmask;
+	else if (!has_pref && !has_bind)
+		unreachable(); /* Handled above */
+	else
+		unreachable();
+}
+#endif
+
+/*
+ * Find a zonelist from a preferred node. Here is a truth table example using 2
+ * different masks. The choices are, NULL mask, empty mask, two masks with an
+ * intersection, and two masks with no intersection. If the local node is in the
+ * intersection, it is used, otherwise the first set node is used.
+ *
+ * +----------+----------+------------+
+ * | bindmask | prefmask |  zonelist  |
+ * +----------+----------+------------+
+ * | NULL/0   | NULL/0   | local node |
+ * | NULL/0   | 0x2      | 0x2        |
+ * | NULL/0   | 0x4      | 0x4        |
+ * | 0x2      | NULL/0   | 0x2        |
+ * | 0x2      | 0x2      | 0x2        |
+ * | 0x2      | 0x4      | local*     |
+ * | 0x4      | NULL/0   | 0x4        |
+ * | 0x4      | 0x2      | local*     |
+ * | 0x4      | 0x4      | 0x4        |
+ * +----------+----------+------------+
+ *
+ * NB: That zonelist will have *all* zones in the fallback case, and not all of
+ * those zones will belong to preferred nodes.
+ */
+static struct zonelist *preferred_zonelist(gfp_t gfp_mask,
+					   const nodemask_t *prefmask,
+					   const nodemask_t *bindmask)
+{
+	nodemask_t pref;
+	int nid, local_node = numa_mem_id();
+
+	/* Multi nodes not supported yet */
+	VM_BUG_ON(prefmask && nodes_weight(*prefmask) != 1);
+
+#define _isset(mask, node)                                                     \
+	(!(mask) || nodes_empty(*(mask)) ? 1 : node_isset(node, *(mask)))
+	/*
+	 * This will handle NULL masks, empty masks, and when the local node
+	 * match all constraints. It does most of the magic here.
+	 */
+	if (_isset(prefmask, local_node) && _isset(bindmask, local_node))
+		return node_zonelist(local_node, gfp_mask);
+#undef _isset
+
+	VM_BUG_ON(!prefmask && !bindmask);
+
+	set_pref_bind_mask(&pref, prefmask, bindmask);
+
+	/*
+	 * It is possible that the caller may ask for a preferred set that isn't
+	 * available. One such case is memory hotplug. Memory hotplug code tries
+	 * to do some allocations from the target node (what will be local to
+	 * the new node) before the pages are onlined (N_MEMORY).
+	 */
+	for_each_node_mask(nid, pref) {
+		if (!node_state(nid, N_MEMORY))
+			node_clear(nid, pref);
+	}
+
+	/*
+	 * If we couldn't manage to get anything reasonable, let later code
+	 * clean up our mess. Local node will be the best approximation for what
+	 * is desired, just use it.
+	 */
+	if (unlikely(nodes_empty(pref)))
+		return node_zonelist(local_node, gfp_mask);
+
+	/* Try to find the "closest" node in the list. */
+	nodes_complement(pref, pref);
+	while ((nid = find_next_best_node(local_node, &pref)) != NUMA_NO_NODE)
+		return node_zonelist(nid, gfp_mask);
+
+	/*
+	 * find_next_best_node() will have to have found something if the
+	 * node list isn't empty and so it isn't possible to get here unless
+	 * find_next_best_node() is modified and this function isn't updated.
+	 */
+	BUG();
+}
+
+static inline bool
+prepare_alloc_pages(gfp_t gfp_mask, unsigned int order, nodemask_t *prefmask,
+		    nodemask_t *nodemask, struct alloc_context *ac,
+		    gfp_t *alloc_mask, unsigned int *alloc_flags)
 {
 	ac->highest_zoneidx = gfp_zone(gfp_mask);
-	ac->zonelist = node_zonelist(preferred_nid, gfp_mask);
 	ac->nodemask = nodemask;
 	ac->migratetype = gfp_migratetype(gfp_mask);
 
@@ -4777,6 +4886,8 @@ static inline bool prepare_alloc_pages(gfp_t gfp_mask, unsigned int order,
 			*alloc_flags |= ALLOC_CPUSET;
 	}
 
+	ac->zonelist = preferred_zonelist(gfp_mask, prefmask, ac->nodemask);
+
 	fs_reclaim_acquire(gfp_mask);
 	fs_reclaim_release(gfp_mask);
 
@@ -4817,6 +4928,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
 	unsigned int alloc_flags = ALLOC_WMARK_LOW;
 	gfp_t alloc_mask; /* The gfp_t that was actually used for allocation */
 	struct alloc_context ac = { };
+	nodemask_t prefmask = nodemask_of_node(preferred_nid);
 
 	/*
 	 * There are several places where we assume that the order value is sane
@@ -4829,7 +4941,8 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
 
 	gfp_mask &= gfp_allowed_mask;
 	alloc_mask = gfp_mask;
-	if (!prepare_alloc_pages(gfp_mask, order, preferred_nid, nodemask, &ac, &alloc_mask, &alloc_flags))
+	if (!prepare_alloc_pages(gfp_mask, order, &prefmask, nodemask, &ac,
+				 &alloc_mask, &alloc_flags))
 		return NULL;
 
 	finalise_ac(gfp_mask, &ac);
-- 
2.27.0

[PATCH 04/18] mm/page_alloc: add preferred pass to page allocation

[Ben Widawsky]

This patch updates the core part of page allocation (pulling from the
free list) to take preferred nodes into account first. If an allocation
from a preferred node cannot be found, the remaining nodes in the
nodemask are checked.

Intentionally not handled in this patch are OOM node scanning and
reclaim scanning. I am very open and receptive on comments as to whether
it is worth handling those cases with a preferred node ordering.

In this patch the code first scans the preferred nodes to make the
allocation, and then takes the subset of nodes in the remaining bound
nodes (often this is NULL aka all nodes) - potentially two passes.
Actually, the code was already two passes as it tries not to fragment on
the first pass, so now it's up to 4 passes.

Consider a 3 node system (0-2) passed the following masks:
Preferred: 	100
Bound:		110

pass 1: node 2 no fragmentation
pass 2: node 1 no fragmentation
pass 3: node 2 w/fragmentation
pass 4: node 1 w/fragmentation

Cc: Andi Kleen <ak@linux.intel.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Michal Hocko <mhocko@kernel.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 mm/internal.h   |   1 +
 mm/page_alloc.c | 108 +++++++++++++++++++++++++++++++++++-------------
 2 files changed, 80 insertions(+), 29 deletions(-)

diff --git a/mm/internal.h b/mm/internal.h
index 9886db20d94f..8d16229c6cbb 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -138,6 +138,7 @@ extern pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address);
 struct alloc_context {
 	struct zonelist *zonelist;
 	nodemask_t *nodemask;
+	nodemask_t *prefmask;
 	struct zoneref *preferred_zoneref;
 	int migratetype;
 
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 280ca85dc4d8..3cf44b6c31ae 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3675,6 +3675,69 @@ alloc_flags_nofragment(struct zone *zone, gfp_t gfp_mask)
 	return alloc_flags;
 }
 
+#ifdef CONFIG_NUMA
+static void set_pref_bind_mask(nodemask_t *out, const nodemask_t *prefmask,
+			       const nodemask_t *bindmask)
+{
+	bool has_pref, has_bind;
+
+	has_pref = prefmask && !nodes_empty(*prefmask);
+	has_bind = bindmask && !nodes_empty(*bindmask);
+
+	if (has_pref && has_bind)
+		nodes_and(*out, *prefmask, *bindmask);
+	else if (has_pref && !has_bind)
+		*out = *prefmask;
+	else if (!has_pref && has_bind)
+		*out = *bindmask;
+	else if (!has_pref && !has_bind)
+		*out = NODE_MASK_ALL;
+	else
+		unreachable();
+}
+#else
+#define set_pref_bind_mask(out, pref, bind)                                    \
+	{                                                                      \
+		(out)->bits[0] = 1UL                                           \
+	}
+#endif
+
+/* Helper to generate the preferred and fallback nodelists */
+static void __nodemask_for_freelist_scan(const struct alloc_context *ac,
+					 bool preferred, nodemask_t *outnodes)
+{
+	bool has_pref;
+	bool has_bind;
+
+	if (preferred) {
+		set_pref_bind_mask(outnodes, ac->prefmask, ac->nodemask);
+		return;
+	}
+
+	has_pref = ac->prefmask && !nodes_empty(*ac->prefmask);
+	has_bind = ac->nodemask && !nodes_empty(*ac->nodemask);
+
+	if (!has_bind && !has_pref) {
+		/*
+		 * If no preference, we already tried the full nodemask,
+		 * so we have to bail.
+		 */
+		nodes_clear(*outnodes);
+	} else if (!has_bind && has_pref) {
+		/* We tried preferred nodes only before. Invert that. */
+		nodes_complement(*outnodes, *ac->prefmask);
+	} else if (has_bind && !has_pref) {
+		/*
+		 * If preferred was empty, we've tried all bound nodes,
+		 * and there nothing further we can do.
+		 */
+		nodes_clear(*outnodes);
+	} else if (has_bind && has_pref) {
+		/* Try the bound nodes that weren't tried before. */
+		nodes_andnot(*outnodes, *ac->nodemask, *ac->prefmask);
+	}
+}
+
 /*
  * get_page_from_freelist goes through the zonelist trying to allocate
  * a page.
@@ -3686,7 +3749,10 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 	struct zoneref *z;
 	struct zone *zone;
 	struct pglist_data *last_pgdat_dirty_limit = NULL;
-	bool no_fallback;
+	nodemask_t nodes;
+	bool no_fallback, preferred_nodes_exhausted = false;
+
+	__nodemask_for_freelist_scan(ac, true, &nodes);
 
 retry:
 	/*
@@ -3696,7 +3762,8 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 	no_fallback = alloc_flags & ALLOC_NOFRAGMENT;
 	z = ac->preferred_zoneref;
 	for_next_zone_zonelist_nodemask(zone, z, ac->zonelist,
-					ac->highest_zoneidx, ac->nodemask) {
+					ac->highest_zoneidx, &nodes)
+	{
 		struct page *page;
 		unsigned long mark;
 
@@ -3816,12 +3883,20 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags,
 		}
 	}
 
+	if (!preferred_nodes_exhausted) {
+		__nodemask_for_freelist_scan(ac, false, &nodes);
+		preferred_nodes_exhausted = true;
+		goto retry;
+	}
+
 	/*
 	 * It's possible on a UMA machine to get through all zones that are
 	 * fragmented. If avoiding fragmentation, reset and try again.
 	 */
 	if (no_fallback) {
 		alloc_flags &= ~ALLOC_NOFRAGMENT;
+		__nodemask_for_freelist_scan(ac, true, &nodes);
+		preferred_nodes_exhausted = false;
 		goto retry;
 	}
 
@@ -4763,33 +4838,6 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	return page;
 }
 
-#ifndef CONFIG_NUMA
-#define set_pref_bind_mask(out, pref, bind)                                    \
-	{                                                                      \
-		(out)->bits[0] = 1UL                                           \
-	}
-#else
-static void set_pref_bind_mask(nodemask_t *out, const nodemask_t *prefmask,
-			       const nodemask_t *bindmask)
-{
-	bool has_pref, has_bind;
-
-	has_pref = prefmask && !nodes_empty(*prefmask);
-	has_bind = bindmask && !nodes_empty(*bindmask);
-
-	if (has_pref && has_bind)
-		nodes_and(*out, *prefmask, *bindmask);
-	else if (has_pref && !has_bind)
-		*out = *prefmask;
-	else if (!has_pref && has_bind)
-		*out = *bindmask;
-	else if (!has_pref && !has_bind)
-		unreachable(); /* Handled above */
-	else
-		unreachable();
-}
-#endif
-
 /*
  * Find a zonelist from a preferred node. Here is a truth table example using 2
  * different masks. The choices are, NULL mask, empty mask, two masks with an
@@ -4945,6 +4993,8 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
 				 &alloc_mask, &alloc_flags))
 		return NULL;
 
+	ac.prefmask = &prefmask;
+
 	finalise_ac(gfp_mask, &ac);
 
 	/*
-- 
2.27.0

[PATCH 05/18] mm/mempolicy: convert single preferred_node to full nodemask

[Ben Widawsky]

From: Dave Hansen <dave.hansen@linux.intel.com>

The NUMA APIs currently allow passing in a "preferred node" as a
single bit set in a nodemask.  If more than one bit it set, bits
after the first are ignored.  Internally, this is implemented as
a single integer: mempolicy->preferred_node.

This single node is generally OK for location-based NUMA where
memory being allocated will eventually be operated on by a single
CPU.  However, in systems with multiple memory types, folks want
to target a *type* of memory instead of a location.  For instance,
someone might want some high-bandwidth memory but do not care about
the CPU next to which it is allocated.  Or, they want a cheap,
high capacity allocation and want to target all NUMA nodes which
have persistent memory in volatile mode.  In both of these cases,
the application wants to target a *set* of nodes, but does not
want strict MPOL_BIND behavior.

To get that behavior, a MPOL_PREFERRED mode is desirable, but one
that honors multiple nodes to be set in the nodemask.

The first step in that direction is to be able to internally store
multiple preferred nodes, which is implemented in this patch.

This should not have any function changes and just switches the
internal representation of mempolicy->preferred_node from an
integer to a nodemask called 'mempolicy->preferred_nodes'.

This is not a pie-in-the-sky dream for an API.  This was a response to a
specific ask of more than one group at Intel.  Specifically:

1. There are existing libraries that target memory types such as
   https://github.com/memkind/memkind.  These are known to suffer
   from SIGSEGV's when memory is low on targeted memory "kinds" that
   span more than one node.  The MCDRAM on a Xeon Phi in "Cluster on
   Die" mode is an example of this.
2. Volatile-use persistent memory users want to have a memory policy
   which is targeted at either "cheap and slow" (PMEM) or "expensive and
   fast" (DRAM).  However, they do not want to experience allocation
   failures when the targeted type is unavailable.
3. Allocate-then-run.  Generally, we let the process scheduler decide
   on which physical CPU to run a task.  That location provides a
   default allocation policy, and memory availability is not generally
   considered when placing tasks.  For situations where memory is
   valuable and constrained, some users want to allocate memory first,
   *then* allocate close compute resources to the allocation.  This is
   the reverse of the normal (CPU) model.  Accelerators such as GPUs
   that operate on core-mm-managed memory are interested in this model.

v2:
Fix spelling errors in commit message. (Ben)
clang-format. (Ben)
Integrated bit from another patch. (Ben)
Update the docs to reflect the internal data structure change (Ben)
Don't advertise MPOL_PREFERRED_MANY in UAPI until we can handle it (Ben)
Added more to the commit message (Dave)

Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com> (v2)
Co-developed-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 .../admin-guide/mm/numa_memory_policy.rst     |  6 +--
 include/linux/mempolicy.h                     |  4 +-
 mm/mempolicy.c                                | 40 ++++++++++---------
 3 files changed, 27 insertions(+), 23 deletions(-)

diff --git a/Documentation/admin-guide/mm/numa_memory_policy.rst b/Documentation/admin-guide/mm/numa_memory_policy.rst
index 067a90a1499c..1ad020c459b8 100644
--- a/Documentation/admin-guide/mm/numa_memory_policy.rst
+++ b/Documentation/admin-guide/mm/numa_memory_policy.rst
@@ -205,9 +205,9 @@ MPOL_PREFERRED
 	of increasing distance from the preferred node based on
 	information provided by the platform firmware.
 
-	Internally, the Preferred policy uses a single node--the
-	preferred_node member of struct mempolicy.  When the internal
-	mode flag MPOL_F_LOCAL is set, the preferred_node is ignored
+	Internally, the Preferred policy uses a nodemask--the
+	preferred_nodes member of struct mempolicy.  When the internal
+	mode flag MPOL_F_LOCAL is set, the preferred_nodes are ignored
 	and the policy is interpreted as local allocation.  "Local"
 	allocation policy can be viewed as a Preferred policy that
 	starts at the node containing the cpu where the allocation
diff --git a/include/linux/mempolicy.h b/include/linux/mempolicy.h
index ea9c15b60a96..c66ea9f4c61e 100644
--- a/include/linux/mempolicy.h
+++ b/include/linux/mempolicy.h
@@ -47,8 +47,8 @@ struct mempolicy {
 	unsigned short mode; 	/* See MPOL_* above */
 	unsigned short flags;	/* See set_mempolicy() MPOL_F_* above */
 	union {
-		short 		 preferred_node; /* preferred */
-		nodemask_t	 nodes;		/* interleave/bind */
+		nodemask_t preferred_nodes; /* preferred */
+		nodemask_t nodes; /* interleave/bind */
 		/* undefined for default */
 	} v;
 	union {
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 99e0f3f9c4a6..e0b576838e57 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -205,7 +205,7 @@ static int mpol_new_preferred(struct mempolicy *pol, const nodemask_t *nodes)
 	else if (nodes_empty(*nodes))
 		return -EINVAL;			/*  no allowed nodes */
 	else
-		pol->v.preferred_node = first_node(*nodes);
+		pol->v.preferred_nodes = nodemask_of_node(first_node(*nodes));
 	return 0;
 }
 
@@ -345,22 +345,26 @@ static void mpol_rebind_preferred(struct mempolicy *pol,
 						const nodemask_t *nodes)
 {
 	nodemask_t tmp;
+	nodemask_t preferred_node;
+
+	/* MPOL_PREFERRED uses only the first node in the mask */
+	preferred_node = nodemask_of_node(first_node(*nodes));
 
 	if (pol->flags & MPOL_F_STATIC_NODES) {
 		int node = first_node(pol->w.user_nodemask);
 
 		if (node_isset(node, *nodes)) {
-			pol->v.preferred_node = node;
+			pol->v.preferred_nodes = nodemask_of_node(node);
 			pol->flags &= ~MPOL_F_LOCAL;
 		} else
 			pol->flags |= MPOL_F_LOCAL;
 	} else if (pol->flags & MPOL_F_RELATIVE_NODES) {
 		mpol_relative_nodemask(&tmp, &pol->w.user_nodemask, nodes);
-		pol->v.preferred_node = first_node(tmp);
+		pol->v.preferred_nodes = tmp;
 	} else if (!(pol->flags & MPOL_F_LOCAL)) {
-		pol->v.preferred_node = node_remap(pol->v.preferred_node,
-						   pol->w.cpuset_mems_allowed,
-						   *nodes);
+		nodes_remap(tmp, pol->v.preferred_nodes,
+			    pol->w.cpuset_mems_allowed, preferred_node);
+		pol->v.preferred_nodes = tmp;
 		pol->w.cpuset_mems_allowed = *nodes;
 	}
 }
@@ -913,7 +917,7 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes)
 		break;
 	case MPOL_PREFERRED:
 		if (!(p->flags & MPOL_F_LOCAL))
-			node_set(p->v.preferred_node, *nodes);
+			*nodes = p->v.preferred_nodes;
 		/* else return empty node mask for local allocation */
 		break;
 	default:
@@ -1906,9 +1910,9 @@ static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy)
 static int policy_node(gfp_t gfp, struct mempolicy *policy,
 								int nd)
 {
-	if (policy->mode == MPOL_PREFERRED && !(policy->flags & MPOL_F_LOCAL))
-		nd = policy->v.preferred_node;
-	else {
+	if (policy->mode == MPOL_PREFERRED && !(policy->flags & MPOL_F_LOCAL)) {
+		nd = first_node(policy->v.preferred_nodes);
+	} else {
 		/*
 		 * __GFP_THISNODE shouldn't even be used with the bind policy
 		 * because we might easily break the expectation to stay on the
@@ -1953,7 +1957,7 @@ unsigned int mempolicy_slab_node(void)
 		/*
 		 * handled MPOL_F_LOCAL above
 		 */
-		return policy->v.preferred_node;
+		return first_node(policy->v.preferred_nodes);
 
 	case MPOL_INTERLEAVE:
 		return interleave_nodes(policy);
@@ -2087,7 +2091,7 @@ bool init_nodemask_of_mempolicy(nodemask_t *mask)
 		if (mempolicy->flags & MPOL_F_LOCAL)
 			nid = numa_node_id();
 		else
-			nid = mempolicy->v.preferred_node;
+			nid = first_node(mempolicy->v.preferred_nodes);
 		init_nodemask_of_node(mask, nid);
 		break;
 
@@ -2225,7 +2229,7 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma,
 		 * node in its nodemask, we allocate the standard way.
 		 */
 		if (pol->mode == MPOL_PREFERRED && !(pol->flags & MPOL_F_LOCAL))
-			hpage_node = pol->v.preferred_node;
+			hpage_node = first_node(pol->v.preferred_nodes);
 
 		nmask = policy_nodemask(gfp, pol);
 		if (!nmask || node_isset(hpage_node, *nmask)) {
@@ -2364,7 +2368,7 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b)
 		/* a's ->flags is the same as b's */
 		if (a->flags & MPOL_F_LOCAL)
 			return true;
-		return a->v.preferred_node == b->v.preferred_node;
+		return nodes_equal(a->v.preferred_nodes, b->v.preferred_nodes);
 	default:
 		BUG();
 		return false;
@@ -2508,7 +2512,7 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 		if (pol->flags & MPOL_F_LOCAL)
 			polnid = numa_node_id();
 		else
-			polnid = pol->v.preferred_node;
+			polnid = first_node(pol->v.preferred_nodes);
 		break;
 
 	case MPOL_BIND:
@@ -2825,7 +2829,7 @@ void __init numa_policy_init(void)
 			.refcnt = ATOMIC_INIT(1),
 			.mode = MPOL_PREFERRED,
 			.flags = MPOL_F_MOF | MPOL_F_MORON,
-			.v = { .preferred_node = nid, },
+			.v = { .preferred_nodes = nodemask_of_node(nid), },
 		};
 	}
 
@@ -2991,7 +2995,7 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
 	if (mode != MPOL_PREFERRED)
 		new->v.nodes = nodes;
 	else if (nodelist)
-		new->v.preferred_node = first_node(nodes);
+		new->v.preferred_nodes = nodemask_of_node(first_node(nodes));
 	else
 		new->flags |= MPOL_F_LOCAL;
 
@@ -3044,7 +3048,7 @@ void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol)
 		if (flags & MPOL_F_LOCAL)
 			mode = MPOL_LOCAL;
 		else
-			node_set(pol->v.preferred_node, nodes);
+			nodes_or(nodes, nodes, pol->v.preferred_nodes);
 		break;
 	case MPOL_BIND:
 	case MPOL_INTERLEAVE:
-- 
2.27.0

[PATCH 06/18] mm/mempolicy: Add MPOL_PREFERRED_MANY for multiple preferred nodes

[Ben Widawsky]

From: Dave Hansen <dave.hansen@linux.intel.com>

MPOL_PREFERRED honors only a single node set in the nodemask.  Add the
bare define for a new mode which will allow more than one.

The patch does all the plumbing without actually adding the new policy
type.

v2:
Plumb most MPOL_PREFERRED_MANY without exposing UAPI (Ben)
Fixes for checkpatch (Ben)

Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Co-developed-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 mm/mempolicy.c | 42 +++++++++++++++++++++++++++++++++++++-----
 1 file changed, 37 insertions(+), 5 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index e0b576838e57..6c7301cefeb6 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -31,6 +31,9 @@
  *                but useful to set in a VMA when you have a non default
  *                process policy.
  *
+ * preferred many Try a set of nodes first before normal fallback. This is
+ *                similar to preferred without the special case.
+ *
  * default        Allocate on the local node first, or when on a VMA
  *                use the process policy. This is what Linux always did
  *		  in a NUMA aware kernel and still does by, ahem, default.
@@ -105,6 +108,8 @@
 
 #include "internal.h"
 
+#define MPOL_PREFERRED_MANY MPOL_MAX
+
 /* Internal flags */
 #define MPOL_MF_DISCONTIG_OK (MPOL_MF_INTERNAL << 0)	/* Skip checks for continuous vmas */
 #define MPOL_MF_INVERT (MPOL_MF_INTERNAL << 1)		/* Invert check for nodemask */
@@ -175,7 +180,7 @@ struct mempolicy *get_task_policy(struct task_struct *p)
 static const struct mempolicy_operations {
 	int (*create)(struct mempolicy *pol, const nodemask_t *nodes);
 	void (*rebind)(struct mempolicy *pol, const nodemask_t *nodes);
-} mpol_ops[MPOL_MAX];
+} mpol_ops[MPOL_MAX + 1];
 
 static inline int mpol_store_user_nodemask(const struct mempolicy *pol)
 {
@@ -415,7 +420,7 @@ void mpol_rebind_mm(struct mm_struct *mm, nodemask_t *new)
 	mmap_write_unlock(mm);
 }
 
-static const struct mempolicy_operations mpol_ops[MPOL_MAX] = {
+static const struct mempolicy_operations mpol_ops[MPOL_MAX + 1] = {
 	[MPOL_DEFAULT] = {
 		.rebind = mpol_rebind_default,
 	},
@@ -432,6 +437,10 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = {
 		.rebind = mpol_rebind_nodemask,
 	},
 	/* MPOL_LOCAL is converted to MPOL_PREFERRED on policy creation */
+	[MPOL_PREFERRED_MANY] = {
+		.create = NULL,
+		.rebind = NULL,
+	},
 };
 
 static int migrate_page_add(struct page *page, struct list_head *pagelist,
@@ -915,6 +924,9 @@ static void get_policy_nodemask(struct mempolicy *p, nodemask_t *nodes)
 	case MPOL_INTERLEAVE:
 		*nodes = p->v.nodes;
 		break;
+	case MPOL_PREFERRED_MANY:
+		*nodes = p->v.preferred_nodes;
+		break;
 	case MPOL_PREFERRED:
 		if (!(p->flags & MPOL_F_LOCAL))
 			*nodes = p->v.preferred_nodes;
@@ -1910,7 +1922,9 @@ static nodemask_t *policy_nodemask(gfp_t gfp, struct mempolicy *policy)
 static int policy_node(gfp_t gfp, struct mempolicy *policy,
 								int nd)
 {
-	if (policy->mode == MPOL_PREFERRED && !(policy->flags & MPOL_F_LOCAL)) {
+	if ((policy->mode == MPOL_PREFERRED ||
+	     policy->mode == MPOL_PREFERRED_MANY) &&
+	    !(policy->flags & MPOL_F_LOCAL)) {
 		nd = first_node(policy->v.preferred_nodes);
 	} else {
 		/*
@@ -1953,6 +1967,7 @@ unsigned int mempolicy_slab_node(void)
 		return node;
 
 	switch (policy->mode) {
+	case MPOL_PREFERRED_MANY:
 	case MPOL_PREFERRED:
 		/*
 		 * handled MPOL_F_LOCAL above
@@ -2087,6 +2102,9 @@ bool init_nodemask_of_mempolicy(nodemask_t *mask)
 	task_lock(current);
 	mempolicy = current->mempolicy;
 	switch (mempolicy->mode) {
+	case MPOL_PREFERRED_MANY:
+		*mask = mempolicy->v.preferred_nodes;
+		break;
 	case MPOL_PREFERRED:
 		if (mempolicy->flags & MPOL_F_LOCAL)
 			nid = numa_node_id();
@@ -2141,6 +2159,9 @@ bool mempolicy_nodemask_intersects(struct task_struct *tsk,
 		 * nodes in mask.
 		 */
 		break;
+	case MPOL_PREFERRED_MANY:
+		ret = nodes_intersects(mempolicy->v.preferred_nodes, *mask);
+		break;
 	case MPOL_BIND:
 	case MPOL_INTERLEAVE:
 		ret = nodes_intersects(mempolicy->v.nodes, *mask);
@@ -2225,8 +2246,9 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma,
 		 * node and don't fall back to other nodes, as the cost of
 		 * remote accesses would likely offset THP benefits.
 		 *
-		 * If the policy is interleave, or does not allow the current
-		 * node in its nodemask, we allocate the standard way.
+		 * If the policy is interleave or multiple preferred nodes, or
+		 * does not allow the current node in its nodemask, we allocate
+		 * the standard way.
 		 */
 		if (pol->mode == MPOL_PREFERRED && !(pol->flags & MPOL_F_LOCAL))
 			hpage_node = first_node(pol->v.preferred_nodes);
@@ -2364,6 +2386,9 @@ bool __mpol_equal(struct mempolicy *a, struct mempolicy *b)
 	case MPOL_BIND:
 	case MPOL_INTERLEAVE:
 		return !!nodes_equal(a->v.nodes, b->v.nodes);
+	case MPOL_PREFERRED_MANY:
+		return !!nodes_equal(a->v.preferred_nodes,
+				     b->v.preferred_nodes);
 	case MPOL_PREFERRED:
 		/* a's ->flags is the same as b's */
 		if (a->flags & MPOL_F_LOCAL)
@@ -2532,6 +2557,8 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 		polnid = zone_to_nid(z->zone);
 		break;
 
+	/* case MPOL_PREFERRED_MANY: */
+
 	default:
 		BUG();
 	}
@@ -2883,6 +2910,7 @@ static const char * const policy_modes[] =
 	[MPOL_BIND]       = "bind",
 	[MPOL_INTERLEAVE] = "interleave",
 	[MPOL_LOCAL]      = "local",
+	[MPOL_PREFERRED_MANY]  = "prefer (many)",
 };
 
 
@@ -2962,6 +2990,7 @@ int mpol_parse_str(char *str, struct mempolicy **mpol)
 		if (!nodelist)
 			err = 0;
 		goto out;
+	case MPOL_PREFERRED_MANY:
 	case MPOL_BIND:
 		/*
 		 * Insist on a nodelist
@@ -3044,6 +3073,9 @@ void mpol_to_str(char *buffer, int maxlen, struct mempolicy *pol)
 	switch (mode) {
 	case MPOL_DEFAULT:
 		break;
+	case MPOL_PREFERRED_MANY:
+		WARN_ON(flags & MPOL_F_LOCAL);
+		fallthrough;
 	case MPOL_PREFERRED:
 		if (flags & MPOL_F_LOCAL)
 			mode = MPOL_LOCAL;
-- 
2.27.0

[PATCH 07/18] mm/mempolicy: allow preferred code to take a nodemask

[Ben Widawsky]

From: Dave Hansen <dave.hansen@linux.intel.com>

Create a helper function (mpol_new_preferred_many()) which is usable
both by the old, single-node MPOL_PREFERRED and the new
MPOL_PREFERRED_MANY.

Enforce the old single-node MPOL_PREFERRED behavior in the "new"
version of mpol_new_preferred() which calls mpol_new_preferred_many().

Cc: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 mm/mempolicy.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 6c7301cefeb6..541675a5b947 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -203,17 +203,30 @@ static int mpol_new_interleave(struct mempolicy *pol, const nodemask_t *nodes)
 	return 0;
 }
 
-static int mpol_new_preferred(struct mempolicy *pol, const nodemask_t *nodes)
+static int mpol_new_preferred_many(struct mempolicy *pol,
+				   const nodemask_t *nodes)
 {
 	if (!nodes)
 		pol->flags |= MPOL_F_LOCAL;	/* local allocation */
 	else if (nodes_empty(*nodes))
 		return -EINVAL;			/*  no allowed nodes */
 	else
-		pol->v.preferred_nodes = nodemask_of_node(first_node(*nodes));
+		pol->v.preferred_nodes = *nodes;
 	return 0;
 }
 
+static int mpol_new_preferred(struct mempolicy *pol, const nodemask_t *nodes)
+{
+	if (nodes) {
+		/* MPOL_PREFERRED can only take a single node: */
+		nodemask_t tmp = nodemask_of_node(first_node(*nodes));
+
+		return mpol_new_preferred_many(pol, &tmp);
+	}
+
+	return mpol_new_preferred_many(pol, NULL);
+}
+
 static int mpol_new_bind(struct mempolicy *pol, const nodemask_t *nodes)
 {
 	if (nodes_empty(*nodes))
-- 
2.27.0

[PATCH 08/18] mm/mempolicy: refactor rebind code for PREFERRED_MANY

[Ben Widawsky]

From: Dave Hansen <dave.hansen@linux.intel.com>

Again, this extracts the "only one node must be set" behavior of
MPOL_PREFERRED.  It retains virtually all of the existing code so it can
be used by MPOL_PREFERRED_MANY as well.

v2:
Fixed typos in commit message. (Ben)
Merged bits from other patches. (Ben)
annotate mpol_rebind_preferred_many as unused (Ben)

Signed-off-by: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 mm/mempolicy.c | 29 ++++++++++++++++++++++-------
 1 file changed, 22 insertions(+), 7 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 541675a5b947..bfc4ef2af90d 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -359,14 +359,11 @@ static void mpol_rebind_nodemask(struct mempolicy *pol, const nodemask_t *nodes)
 	pol->v.nodes = tmp;
 }
 
-static void mpol_rebind_preferred(struct mempolicy *pol,
-						const nodemask_t *nodes)
+static void mpol_rebind_preferred_common(struct mempolicy *pol,
+					 const nodemask_t *preferred_nodes,
+					 const nodemask_t *nodes)
 {
 	nodemask_t tmp;
-	nodemask_t preferred_node;
-
-	/* MPOL_PREFERRED uses only the first node in the mask */
-	preferred_node = nodemask_of_node(first_node(*nodes));
 
 	if (pol->flags & MPOL_F_STATIC_NODES) {
 		int node = first_node(pol->w.user_nodemask);
@@ -381,12 +378,30 @@ static void mpol_rebind_preferred(struct mempolicy *pol,
 		pol->v.preferred_nodes = tmp;
 	} else if (!(pol->flags & MPOL_F_LOCAL)) {
 		nodes_remap(tmp, pol->v.preferred_nodes,
-			    pol->w.cpuset_mems_allowed, preferred_node);
+			    pol->w.cpuset_mems_allowed, *preferred_nodes);
 		pol->v.preferred_nodes = tmp;
 		pol->w.cpuset_mems_allowed = *nodes;
 	}
 }
 
+/* MPOL_PREFERRED_MANY allows multiple nodes to be set in 'nodes' */
+static void __maybe_unused mpol_rebind_preferred_many(struct mempolicy *pol,
+						      const nodemask_t *nodes)
+{
+	mpol_rebind_preferred_common(pol, nodes, nodes);
+}
+
+static void mpol_rebind_preferred(struct mempolicy *pol,
+				  const nodemask_t *nodes)
+{
+	nodemask_t preferred_node;
+
+	/* MPOL_PREFERRED uses only the first node in 'nodes' */
+	preferred_node = nodemask_of_node(first_node(*nodes));
+
+	mpol_rebind_preferred_common(pol, &preferred_node, nodes);
+}
+
 /*
  * mpol_rebind_policy - Migrate a policy to a different set of nodes
  *
-- 
2.27.0

[PATCH 09/18] mm: Finish handling MPOL_PREFERRED_MANY

[Ben Widawsky]

Now that there is a function to generate the preferred zonelist given a
preferred mask, bindmask, and flags it is possible to support
MPOL_PREFERRED_MANY policy easily in more places.

This patch was developed on top of Dave's original work. When Dave wrote
his patches there was no clean way to implement MPOL_PREFERRED_MANY. Now
that the other bits are in place, this is easy to drop on top.

Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dan Williams <dan.j.williams@intel.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 include/linux/mmzone.h |  3 +++
 mm/mempolicy.c         | 20 ++++++++++++++++++--
 mm/page_alloc.c        |  5 ++---
 3 files changed, 23 insertions(+), 5 deletions(-)

diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index c4c37fd12104..6b62ee98bb96 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -1001,6 +1001,9 @@ struct zoneref *__next_zones_zonelist(struct zoneref *z,
 					enum zone_type highest_zoneidx,
 					nodemask_t *nodes);
 
+struct zonelist *preferred_zonelist(gfp_t gfp_mask, const nodemask_t *prefmask,
+				    const nodemask_t *bindmask);
+
 /**
  * next_zones_zonelist - Returns the next zone at or below highest_zoneidx within the allowed nodemask using a cursor within a zonelist as a starting point
  * @z - The cursor used as a starting point for the search
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index bfc4ef2af90d..90bc9c93b1b9 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1995,7 +1995,6 @@ unsigned int mempolicy_slab_node(void)
 		return node;
 
 	switch (policy->mode) {
-	case MPOL_PREFERRED_MANY:
 	case MPOL_PREFERRED:
 		/*
 		 * handled MPOL_F_LOCAL above
@@ -2020,6 +2019,18 @@ unsigned int mempolicy_slab_node(void)
 		return z->zone ? zone_to_nid(z->zone) : node;
 	}
 
+	case MPOL_PREFERRED_MANY: {
+		struct zoneref *z;
+		struct zonelist *zonelist;
+		enum zone_type highest_zoneidx = gfp_zone(GFP_KERNEL);
+
+		zonelist = preferred_zonelist(GFP_KERNEL,
+					      &policy->v.preferred_nodes, NULL);
+		z = first_zones_zonelist(zonelist, highest_zoneidx,
+					 &policy->v.nodes);
+		return z->zone ? zone_to_nid(z->zone) : node;
+	}
+
 	default:
 		BUG();
 	}
@@ -2585,7 +2596,12 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long
 		polnid = zone_to_nid(z->zone);
 		break;
 
-	/* case MPOL_PREFERRED_MANY: */
+	case MPOL_PREFERRED_MANY:
+		z = first_zones_zonelist(preferred_zonelist(GFP_HIGHUSER,
+							    &pol->v.preferred_nodes, NULL),
+					 gfp_zone(GFP_HIGHUSER), &pol->v.preferred_nodes);
+		polnid = zone_to_nid(z->zone);
+		break;
 
 	default:
 		BUG();
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 3cf44b6c31ae..c6f8f112a5d4 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -4861,9 +4861,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
  * NB: That zonelist will have *all* zones in the fallback case, and not all of
  * those zones will belong to preferred nodes.
  */
-static struct zonelist *preferred_zonelist(gfp_t gfp_mask,
-					   const nodemask_t *prefmask,
-					   const nodemask_t *bindmask)
+struct zonelist *preferred_zonelist(gfp_t gfp_mask, const nodemask_t *prefmask,
+				    const nodemask_t *bindmask)
 {
 	nodemask_t pref;
 	int nid, local_node = numa_mem_id();
-- 
2.27.0

[PATCH 10/18] mm: clean up alloc_pages_vma (thp)

[Ben Widawsky]

__alloc_pages_nodemask() already does the right thing for a preferred
node and bind nodemask. Calling it directly allows us to simplify much
of this. The handling occurs in prepare_alloc_pages()

A VM assertion is added to prove correctness.

Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 mm/mempolicy.c | 40 +++++++++++++++++++++-------------------
 1 file changed, 21 insertions(+), 19 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 90bc9c93b1b9..408ba78c8424 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2293,27 +2293,29 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma,
 			hpage_node = first_node(pol->v.preferred_nodes);
 
 		nmask = policy_nodemask(gfp, pol);
-		if (!nmask || node_isset(hpage_node, *nmask)) {
-			mpol_cond_put(pol);
-			/*
-			 * First, try to allocate THP only on local node, but
-			 * don't reclaim unnecessarily, just compact.
-			 */
-			page = __alloc_pages_node(hpage_node,
-				gfp | __GFP_THISNODE | __GFP_NORETRY, order);
+		mpol_cond_put(pol);
 
-			/*
-			 * If hugepage allocations are configured to always
-			 * synchronous compact or the vma has been madvised
-			 * to prefer hugepage backing, retry allowing remote
-			 * memory with both reclaim and compact as well.
-			 */
-			if (!page && (gfp & __GFP_DIRECT_RECLAIM))
-				page = __alloc_pages_node(hpage_node,
-								gfp, order);
+		/*
+		 * First, try to allocate THP only on local node, but
+		 * don't reclaim unnecessarily, just compact.
+		 */
+		page = __alloc_pages_nodemask(gfp | __GFP_THISNODE |
+						      __GFP_NORETRY,
+					      order, hpage_node, nmask);
 
-			goto out;
-		}
+		/*
+		 * If hugepage allocations are configured to always synchronous
+		 * compact or the vma has been madvised to prefer hugepage
+		 * backing, retry allowing remote memory with both reclaim and
+		 * compact as well.
+		 */
+		if (!page && (gfp & __GFP_DIRECT_RECLAIM))
+			page = __alloc_pages_nodemask(gfp, order, hpage_node,
+						      nmask);
+
+		VM_BUG_ON(page && nmask &&
+			  !node_isset(page_to_nid(page), *nmask));
+		goto out;
 	}
 
 	nmask = policy_nodemask(gfp, pol);
-- 
2.27.0

[PATCH 11/18] mm: Extract THP hugepage allocation

[Ben Widawsky]

The next patch is going to rework this code to support
MPOL_PREFERRED_MANY. This refactor makes the that change much more
readable.

After the extraction, the resulting code makes it apparent that this can
be converted to a simple if ladder and thus allows removing the goto.

There is not meant to be any functional or behavioral changes.

Note that still at this point MPOL_PREFERRED_MANY isn't specially
handled for huge pages.

Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Michal Hocko <mhocko@kernel.org>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 mm/mempolicy.c | 96 ++++++++++++++++++++++++++------------------------
 1 file changed, 49 insertions(+), 47 deletions(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 408ba78c8424..3ce2354fed44 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2232,6 +2232,48 @@ static struct page *alloc_page_interleave(gfp_t gfp, unsigned order,
 	return page;
 }
 
+static struct page *alloc_pages_vma_thp(gfp_t gfp, struct mempolicy *pol,
+					int order, int node)
+{
+	nodemask_t *nmask;
+	struct page *page;
+	int hpage_node = node;
+
+	/*
+	 * For hugepage allocation and non-interleave policy which allows the
+	 * current node (or other explicitly preferred node) we only try to
+	 * allocate from the current/preferred node and don't fall back to other
+	 * nodes, as the cost of remote accesses would likely offset THP
+	 * benefits.
+	 *
+	 * If the policy is interleave or multiple preferred nodes, or does not
+	 * allow the current node in its nodemask, we allocate the standard way.
+	 */
+	if (pol->mode == MPOL_PREFERRED && !(pol->flags & MPOL_F_LOCAL))
+		hpage_node = first_node(pol->v.preferred_nodes);
+
+	nmask = policy_nodemask(gfp, pol);
+
+	/*
+	 * First, try to allocate THP only on local node, but don't reclaim
+	 * unnecessarily, just compact.
+	 */
+	page = __alloc_pages_nodemask(gfp | __GFP_THISNODE | __GFP_NORETRY,
+				      order, hpage_node, nmask);
+
+	/*
+	 * If hugepage allocations are configured to always synchronous compact
+	 * or the vma has been madvised to prefer hugepage backing, retry
+	 * allowing remote memory with both reclaim and compact as well.
+	 */
+	if (!page && (gfp & __GFP_DIRECT_RECLAIM))
+		page = __alloc_pages_nodemask(gfp, order, hpage_node, nmask);
+
+	VM_BUG_ON(page && nmask && !node_isset(page_to_nid(page), *nmask));
+
+	return page;
+}
+
 /**
  * 	alloc_pages_vma	- Allocate a page for a VMA.
  *
@@ -2272,57 +2314,17 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma,
 		nid = interleave_nid(pol, vma, addr, PAGE_SHIFT + order);
 		mpol_cond_put(pol);
 		page = alloc_page_interleave(gfp, order, nid);
-		goto out;
-	}
-
-	if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && hugepage)) {
-		int hpage_node = node;
-
-		/*
-		 * For hugepage allocation and non-interleave policy which
-		 * allows the current node (or other explicitly preferred
-		 * node) we only try to allocate from the current/preferred
-		 * node and don't fall back to other nodes, as the cost of
-		 * remote accesses would likely offset THP benefits.
-		 *
-		 * If the policy is interleave or multiple preferred nodes, or
-		 * does not allow the current node in its nodemask, we allocate
-		 * the standard way.
-		 */
-		if (pol->mode == MPOL_PREFERRED && !(pol->flags & MPOL_F_LOCAL))
-			hpage_node = first_node(pol->v.preferred_nodes);
-
+	} else if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) &&
+			    hugepage)) {
+		page = alloc_pages_vma_thp(gfp, pol, order, node);
+		mpol_cond_put(pol);
+	} else {
 		nmask = policy_nodemask(gfp, pol);
+		preferred_nid = policy_node(gfp, pol, node);
+		page = __alloc_pages_nodemask(gfp, order, preferred_nid, nmask);
 		mpol_cond_put(pol);
-
-		/*
-		 * First, try to allocate THP only on local node, but
-		 * don't reclaim unnecessarily, just compact.
-		 */
-		page = __alloc_pages_nodemask(gfp | __GFP_THISNODE |
-						      __GFP_NORETRY,
-					      order, hpage_node, nmask);
-
-		/*
-		 * If hugepage allocations are configured to always synchronous
-		 * compact or the vma has been madvised to prefer hugepage
-		 * backing, retry allowing remote memory with both reclaim and
-		 * compact as well.
-		 */
-		if (!page && (gfp & __GFP_DIRECT_RECLAIM))
-			page = __alloc_pages_nodemask(gfp, order, hpage_node,
-						      nmask);
-
-		VM_BUG_ON(page && nmask &&
-			  !node_isset(page_to_nid(page), *nmask));
-		goto out;
 	}
 
-	nmask = policy_nodemask(gfp, pol);
-	preferred_nid = policy_node(gfp, pol, node);
-	page = __alloc_pages_nodemask(gfp, order, preferred_nid, nmask);
-	mpol_cond_put(pol);
-out:
 	return page;
 }
 EXPORT_SYMBOL(alloc_pages_vma);
-- 
2.27.0

[PATCH 12/18] mm/mempolicy: Use __alloc_page_node for interleaved

[Ben Widawsky]

This helps reduce the consumers of the interface and get us in better
shape to clean up some of the low level page allocation routines. The
goal in doing that is to eventually limit the places we'll need to
declare nodemask_t variables on the stack (more on that later).

Currently the only distinction between __alloc_pages_node and
__alloc_pages is that the former does sanity checks on the gfp flags and
the nid. In the case of interleave nodes, this isn't necessary because
the caller has already figured out the right nid and flags with
interleave_nodes(),

This kills the only real user of __alloc_pages, which can then be
removed later.

Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Vlastimil Babka <vbabka@suse.cz>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 mm/mempolicy.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 3ce2354fed44..eb2520d68a04 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -2220,7 +2220,7 @@ static struct page *alloc_page_interleave(gfp_t gfp, unsigned order,
 {
 	struct page *page;
 
-	page = __alloc_pages(gfp, order, nid);
+	page = __alloc_pages_node(nid, gfp, order);
 	/* skip NUMA_INTERLEAVE_HIT counter update if numa stats is disabled */
 	if (!static_branch_likely(&vm_numa_stat_key))
 		return page;
-- 
2.27.0

[PATCH 13/18] mm: kill __alloc_pages

[Ben Widawsky]

IMPORTANT NOTE: It's unclear how safe it is to declare nodemask_t on the
stack, when nodemask_t can be relatively large in huge NUMA systems.
Upcoming patches will try to limit this.

The primary purpose of this patch is to clear up which interfaces should
be used for page allocation.

There are several attributes in page allocation after the obvious gfp
and order:
1. node mask: set of nodes to try to allocate from, fail if unavailable
2. preferred nid: a preferred node to try to allocate from, falling back
to node mask if unavailable
3. (soon) preferred mask: like preferred nid, but multiple nodes.

Here's a summary of the existing interfaces, and which they cover
*alloc_pages: 		()
*alloc_pages_node:	(2)
__alloc_pages_nodemask: (1,2,3)

I am instead proposing instead the following interfaces as a reasonable
set. Generally node binding isn't used by kernel code, it's only used
for mempolicy. On the other hand, the kernel does have preferred nodes
(today it's only one), and that is why those interfaces exist while an
interface to specify binding does not.

alloc_pages: () I don't care, give me pages.
alloc_pages_node: (2) I want pages from this particular node first
alloc_pages_nodes: (3) I want pages from *these* nodes first
__alloc_pages_nodemask: (1,2,3) I'm picky about my pages

Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Michal Hocko <mhocko@kernel.org>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 include/linux/gfp.h | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/include/linux/gfp.h b/include/linux/gfp.h
index 67a0774e080b..9ab5c07579bd 100644
--- a/include/linux/gfp.h
+++ b/include/linux/gfp.h
@@ -504,9 +504,10 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid,
 							nodemask_t *nodemask);
 
 static inline struct page *
-__alloc_pages(gfp_t gfp_mask, unsigned int order, int preferred_nid)
+__alloc_pages_nodes(nodemask_t *nodes, gfp_t gfp_mask, unsigned int order)
 {
-	return __alloc_pages_nodemask(gfp_mask, order, preferred_nid, NULL);
+	return __alloc_pages_nodemask(gfp_mask, order, first_node(*nodes),
+				      NULL);
 }
 
 /*
@@ -516,10 +517,12 @@ __alloc_pages(gfp_t gfp_mask, unsigned int order, int preferred_nid)
 static inline struct page *
 __alloc_pages_node(int nid, gfp_t gfp_mask, unsigned int order)
 {
+	nodemask_t tmp;
 	VM_BUG_ON(nid < 0 || nid >= MAX_NUMNODES);
 	VM_WARN_ON((gfp_mask & __GFP_THISNODE) && !node_online(nid));
 
-	return __alloc_pages(gfp_mask, order, nid);
+	tmp = nodemask_of_node(nid);
+	return __alloc_pages_nodes(&tmp, gfp_mask, order);
 }
 
 /*
-- 
2.27.0

Re: [PATCH 00/18] multiple preferred nodes

[Ben Widawsky]

Please ignore this. I --suppress-cc=all set.

Sorry for the noise.

Re: [PATCH 01/18] mm/mempolicy: Add comment for missing LOCAL

[Michal Hocko]

On Fri 19-06-20 09:24:08, Ben Widawsky wrote:
> MPOL_LOCAL is a bit weird because it is simply a different name for an
> existing behavior (preferred policy with no node mask). It has been this
> way since it was added here:
> commit 479e2802d09f ("mm: mempolicy: Make MPOL_LOCAL a real policy")
> 
> It is so similar to MPOL_PREFERRED in fact that when the policy is
> created in mpol_new, the mode is set as PREFERRED, and an internal state
> representing LOCAL doesn't exist.
> 
> To prevent future explorers from scratching their head as to why
> MPOL_LOCAL isn't defined in the mpol_ops table, add a small comment
> explaining the situations.

Agreed. MPOL_LOCAL can be really confusing for whoever looks at the code
the first time.
 
> Cc: Christoph Lameter <cl@linux.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: David Rientjes <rientjes@google.com>
> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>

Acked-by: Michal Hocko <mhocko@suse.com>

> ---
>  mm/mempolicy.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/mm/mempolicy.c b/mm/mempolicy.c
> index 381320671677..36ee3267c25f 100644
> --- a/mm/mempolicy.c
> +++ b/mm/mempolicy.c
> @@ -427,6 +427,7 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = {
>  		.create = mpol_new_bind,
>  		.rebind = mpol_rebind_nodemask,
>  	},
> +	/* MPOL_LOCAL is converted to MPOL_PREFERRED on policy creation */

I would just add. See mpol_new()

>  };
>  
>  static int migrate_page_add(struct page *page, struct list_head *pagelist,
> -- 
> 2.27.0
> 

-- 
Michal Hocko
SUSE Labs

[PATCH 01/18] mm/mempolicy: Add comment for missing LOCAL

[Ben Widawsky]

MPOL_LOCAL is a bit weird because it is simply a different name for an
existing behavior (preferred policy with no node mask). It has been this
way since it was added here:
commit 479e2802d09f ("mm: mempolicy: Make MPOL_LOCAL a real policy")

It is so similar to MPOL_PREFERRED in fact that when the policy is
created in mpol_new, the mode is set as PREFERRED, and an internal state
representing LOCAL doesn't exist.

To prevent future explorers from scratching their head as to why
MPOL_LOCAL isn't defined in the mpol_ops table, add a small comment
explaining the situations.

Cc: Christoph Lameter <cl@linux.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: David Rientjes <rientjes@google.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
 mm/mempolicy.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 381320671677..36ee3267c25f 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -427,6 +427,7 @@ static const struct mempolicy_operations mpol_ops[MPOL_MAX] = {
 		.create = mpol_new_bind,
 		.rebind = mpol_rebind_nodemask,
 	},
+	/* MPOL_LOCAL is converted to MPOL_PREFERRED on policy creation */
 };
 
 static int migrate_page_add(struct page *page, struct list_head *pagelist,
-- 
2.27.0