[PATCH v2 0/3] memory-hotplug.rst: document the "auto-movable" online policy

[PATCH v2 0/3] memory-hotplug.rst: document the "auto-movable" online policy

[David Hildenbrand]

As promised, now that the memory-hotplug.rst overhaul is upstream, proper
documentation for the "auto-movable" online policy, documenting all new
toggles and options. Along, two fixes for the original overhaul.

v1 -> v2:
- "memory-hotplug.rst: document the "auto-movable" online policy"
-- Add a pointer to generic command line parameter documentation
-- Explain relationship of "auto-movable" and "movable_node" command
   line parameter, also indicating in which setups "auto-movable" doesn't
   really make sense.

Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: linux-doc@vger.kernel.org
Cc: linux-mm@kvack.org

David Hildenbrand (3):
  memory-hotplug.rst: fix two instances of "movablecore" that should be
    "movable_node"
  memory-hotplug.rst: fix wrong /sys/module/memory_hotplug/parameters/
    path
  memory-hotplug.rst: document the "auto-movable" online policy

 .../admin-guide/mm/memory-hotplug.rst         | 143 +++++++++++++++---
 1 file changed, 122 insertions(+), 21 deletions(-)


base-commit: 64570fbc14f8d7cb3fe3995f20e26bc25ce4b2cc
-- 
2.31.1

[PATCH v2 1/3] memory-hotplug.rst: fix two instances of "movablecore" that should be "movable_node"

[David Hildenbrand]

We really want to refer to the "movable_node" kernel command line
parameter here.

Fixes: ac3332c44767 ("memory-hotplug.rst: complete admin-guide overhaul")
Signed-off-by: David Hildenbrand <david@redhat.com>
---
 Documentation/admin-guide/mm/memory-hotplug.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
index 03dfbc925252..27d748cb6ee0 100644
--- a/Documentation/admin-guide/mm/memory-hotplug.rst
+++ b/Documentation/admin-guide/mm/memory-hotplug.rst
@@ -166,7 +166,7 @@ Or alternatively::
 	% echo 1 > /sys/devices/system/memory/memoryXXX/online
 
 The kernel will select the target zone automatically, usually defaulting to
-``ZONE_NORMAL`` unless ``movablecore=1`` has been specified on the kernel
+``ZONE_NORMAL`` unless ``movable_node`` has been specified on the kernel
 command line or if the memory block would intersect the ZONE_MOVABLE already.
 
 One can explicitly request to associate an offline memory block with
@@ -393,7 +393,7 @@ command line parameters are relevant:
 ======================== =======================================================
 ``memhp_default_state``	 configure auto-onlining by essentially setting
                          ``/sys/devices/system/memory/auto_online_blocks``.
-``movablecore``		 configure automatic zone selection of the kernel. When
+``movable_node``	 configure automatic zone selection in the kernel. When
 			 set, the kernel will default to ZONE_MOVABLE, unless
 			 other zones can be kept contiguous.
 ======================== =======================================================
-- 
2.31.1

[PATCH v2 2/3] memory-hotplug.rst: fix wrong /sys/module/memory_hotplug/parameters/ path

[David Hildenbrand]

We accidentially added a superfluous "s".

Fixes: ac3332c44767 ("memory-hotplug.rst: complete admin-guide overhaul")
Signed-off-by: David Hildenbrand <david@redhat.com>
---
 Documentation/admin-guide/mm/memory-hotplug.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
index 27d748cb6ee0..ee00b70dedde 100644
--- a/Documentation/admin-guide/mm/memory-hotplug.rst
+++ b/Documentation/admin-guide/mm/memory-hotplug.rst
@@ -410,7 +410,7 @@ them with ``memory_hotplug.`` such as::
 
 and they can be observed (and some even modified at runtime) via::
 
-	/sys/modules/memory_hotplug/parameters/
+	/sys/module/memory_hotplug/parameters/
 
 The following module parameters are currently defined:
 
-- 
2.31.1

[PATCH v2 3/3] memory-hotplug.rst: document the "auto-movable" online policy

[David Hildenbrand]

In commit e83a437faa62 ("mm/memory_hotplug: introduce "auto-movable" online
policy") we introduced a new memory online policy to automatically
select a zone for memory blocks to be onlined. We added a way to
set the active online policy and tunables for the auto-movable online
policy. In follow-up commits we tweaked the "auto-movable" policy to also
consider memory device details when selecting zones for memory blocks to
be onlined.

Let's document the new toggles and how the two online policies we have
work.

Signed-off-by: David Hildenbrand <david@redhat.com>
---
 .../admin-guide/mm/memory-hotplug.rst         | 141 +++++++++++++++---
 1 file changed, 121 insertions(+), 20 deletions(-)

diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
index ee00b70dedde..0f56ecd8ac05 100644
--- a/Documentation/admin-guide/mm/memory-hotplug.rst
+++ b/Documentation/admin-guide/mm/memory-hotplug.rst
@@ -165,9 +165,8 @@ Or alternatively::
 
 	% echo 1 > /sys/devices/system/memory/memoryXXX/online
 
-The kernel will select the target zone automatically, usually defaulting to
-``ZONE_NORMAL`` unless ``movable_node`` has been specified on the kernel
-command line or if the memory block would intersect the ZONE_MOVABLE already.
+The kernel will select the target zone automatically, depending on the
+configured ``online_policy``.
 
 One can explicitly request to associate an offline memory block with
 ZONE_MOVABLE by::
@@ -198,6 +197,9 @@ Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or
 
 	% echo online > /sys/devices/system/memory/auto_online_blocks
 
+Similarly to manual onlining, with ``online`` the kernel will select the
+target zone automatically, depending on the configured ``online_policy``.
+
 Modifying the auto-online behavior will only affect all subsequently added
 memory blocks only.
 
@@ -393,11 +395,16 @@ command line parameters are relevant:
 ======================== =======================================================
 ``memhp_default_state``	 configure auto-onlining by essentially setting
                          ``/sys/devices/system/memory/auto_online_blocks``.
-``movable_node``	 configure automatic zone selection in the kernel. When
-			 set, the kernel will default to ZONE_MOVABLE, unless
-			 other zones can be kept contiguous.
+``movable_node``	 configure automatic zone selection in the kernel when
+			 using the ``contig-zones`` online policy. When
+			 set, the kernel will default to ZONE_MOVABLE when
+			 onlining a memory block, unless other zones can be kept
+			 contiguous.
 ======================== =======================================================
 
+See Documentation/admin-guide/kernel-parameters.txt for a more generic
+description of these command line parameters.
+
 Module Parameters
 ------------------
 
@@ -414,20 +421,114 @@ and they can be observed (and some even modified at runtime) via::
 
 The following module parameters are currently defined:
 
-======================== =======================================================
-``memmap_on_memory``	 read-write: Allocate memory for the memmap from the
-			 added memory block itself. Even if enabled, actual
-			 support depends on various other system properties and
-			 should only be regarded as a hint whether the behavior
-			 would be desired.
-
-			 While allocating the memmap from the memory block
-			 itself makes memory hotplug less likely to fail and
-			 keeps the memmap on the same NUMA node in any case, it
-			 can fragment physical memory in a way that huge pages
-			 in bigger granularity cannot be formed on hotplugged
-			 memory.
-======================== =======================================================
+================================ ===============================================
+``memmap_on_memory``		 read-write: Allocate memory for the memmap from
+				 the added memory block itself. Even if enabled,
+				 actual support depends on various other system
+				 properties and should only be regarded as a
+				 hint whether the behavior would be desired.
+
+				 While allocating the memmap from the memory
+				 block itself makes memory hotplug less likely
+				 to fail and keeps the memmap on the same NUMA
+				 node in any case, it can fragment physical
+				 memory in a way that huge pages in bigger
+				 granularity cannot be formed on hotplugged
+				 memory.
+``online_policy``		 read-write: Set the basic policy used for
+				 automatic zone selection when onlining memory
+				 blocks without specifying a target zone.
+				 ``contig-zones`` has been the kernel default
+				 before this parameter was added. After an
+				 online policy was configured and memory was
+				 online, the policy should not be changed
+				 anymore.
+
+				 When set to ``contig-zones``, the kernel will
+				 try keeping zones contiguous. If a memory block
+				 intersects multiple zones or no zone, the
+				 behavior depends on the ``movable_node`` kernel
+				 command line parameter: default to ZONE_MOVABLE
+				 if set, default to the applicable kernel zone
+				 (usually ZONE_NORMAL) if not set.
+
+				 When set to ``auto-movable``, the kernel will
+				 try onlining memory blocks to ZONE_MOVABLE if
+				 possible according to the configuration and
+				 memory device details. With this policy, one
+				 can avoid zone imbalances when eventually
+				 hotplugging a lot of memory later and still
+				 wanting to be able to hotunplug as much as
+				 possible reliably, very desirable in
+				 virtualized environments. This policy ignores
+				 the ``movable_node`` kernel command line
+				 parameter and isn't really applicable in
+				 environments that require it (e.g., bare metal
+				 with hotunpluggable nodes) where hotplugged
+				 memory might be exposed via the
+				 firmware-provided memory map early during boot
+				 to the system instead of getting detected,
+				 added and onlined  later during boot (such as
+				 done by virtio-mem or by some hypervisors
+				 implementing emulated DIMMs). As one example, a
+				 hotplugged DIMM will be onlined either
+				 completely to ZONE_MOVABLE or completely to
+				 ZONE_NORMAL, not a mixture.
+				 As another example, as many memory blocks
+				 belonging to a virtio-mem device will be
+				 onlined to ZONE_MOVABLE as possible,
+				 special-casing units of memory blocks that can
+				 only get hotunplugged together. *This policy
+				 does not protect from setups that are
+				 problematic with ZONE_MOVABLE and does not
+				 change the zone of memory blocks dynamically
+				 after they were onlined.*
+``auto_movable_ratio``		 read-write: Set the maximum MOVABLE:KERNEL
+				 memory ratio in % for the ``auto-movable``
+				 online policy. Whether the ratio applies only
+				 for the system across all NUMA nodes or also
+				 per NUMA nodes depends on the
+				 ``auto_movable_numa_aware`` configuration.
+
+				 All accounting is based on present memory pages
+				 in the zones combined with accounting per
+				 memory device. Memory dedicated to the CMA
+				 allocator is accounted as MOVABLE, although
+				 residing on one of the kernel zones. The
+				 possible ratio depends on the actual workload.
+				 The kernel default is "301" %, for example,
+				 allowing for hotplugging 24 GiB to a 8 GiB VM
+				 and automatically onlining all hotplugged
+				 memory to ZONE_MOVABLE in many setups. The
+				 additional 1% deals with some pages being not
+				 present, for example, because of some firmware
+				 allocations.
+
+				 Note that ZONE_NORMAL memory provided by one
+				 memory device does not allow for more
+				 ZONE_MOVABLE memory for a different memory
+				 device. As one example, onlining memory of a
+				 hotplugged DIMM to ZONE_NORMAL will not allow
+				 for another hotplugged DIMM to get onlined to
+				 ZONE_MOVABLE automatically. In contrast, memory
+				 hotplugged by a virtio-mem device that got
+				 onlined to ZONE_NORMAL will allow for more
+				 ZONE_MOVABLE memory within *the same*
+				 virtio-mem device.
+``auto_movable_numa_aware``	 read-write: Configure whether the
+				 ``auto_movable_ratio`` in the ``auto-movable``
+				 online policy also applies per NUMA
+				 node in addition to the whole system across all
+				 NUMA nodes. The kernel default is "Y".
+
+				 Disabling NUMA awareness can be helpful when
+				 dealing with NUMA nodes that should be
+				 completely hotunpluggable, onlining the memory
+				 completely to ZONE_MOVABLE automatically if
+				 possible.
+
+				 Parameter availability depends on CONFIG_NUMA.
+================================ ===============================================
 
 ZONE_MOVABLE
 ============
-- 
2.31.1

Re: [PATCH v2 1/3] memory-hotplug.rst: fix two instances of "movablecore" that should be "movable_node"

[Mike Rapoport]

On Mon, Oct 11, 2021 at 10:20:56AM +0200, David Hildenbrand wrote:
> We really want to refer to the "movable_node" kernel command line
> parameter here.
> 
> Fixes: ac3332c44767 ("memory-hotplug.rst: complete admin-guide overhaul")
> Signed-off-by: David Hildenbrand <david@redhat.com>

Acked-by: Mike Rapoport <rppt@linux.ibm.com>

> ---
>  Documentation/admin-guide/mm/memory-hotplug.rst | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
> index 03dfbc925252..27d748cb6ee0 100644
> --- a/Documentation/admin-guide/mm/memory-hotplug.rst
> +++ b/Documentation/admin-guide/mm/memory-hotplug.rst
> @@ -166,7 +166,7 @@ Or alternatively::
>  	% echo 1 > /sys/devices/system/memory/memoryXXX/online
>  
>  The kernel will select the target zone automatically, usually defaulting to
> -``ZONE_NORMAL`` unless ``movablecore=1`` has been specified on the kernel
> +``ZONE_NORMAL`` unless ``movable_node`` has been specified on the kernel
>  command line or if the memory block would intersect the ZONE_MOVABLE already.
>  
>  One can explicitly request to associate an offline memory block with
> @@ -393,7 +393,7 @@ command line parameters are relevant:
>  ======================== =======================================================
>  ``memhp_default_state``	 configure auto-onlining by essentially setting
>                           ``/sys/devices/system/memory/auto_online_blocks``.
> -``movablecore``		 configure automatic zone selection of the kernel. When
> +``movable_node``	 configure automatic zone selection in the kernel. When
>  			 set, the kernel will default to ZONE_MOVABLE, unless
>  			 other zones can be kept contiguous.
>  ======================== =======================================================
> -- 
> 2.31.1
> 

-- 
Sincerely yours,
Mike.

Re: [PATCH v2 2/3] memory-hotplug.rst: fix wrong /sys/module/memory_hotplug/parameters/ path

[Mike Rapoport]

On Mon, Oct 11, 2021 at 10:20:57AM +0200, David Hildenbrand wrote:
> We accidentially added a superfluous "s".
> 
> Fixes: ac3332c44767 ("memory-hotplug.rst: complete admin-guide overhaul")
> Signed-off-by: David Hildenbrand <david@redhat.com>

Acked-by: Mike Rapoport <rppt@linux.ibm.com>

> ---
>  Documentation/admin-guide/mm/memory-hotplug.rst | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
> index 27d748cb6ee0..ee00b70dedde 100644
> --- a/Documentation/admin-guide/mm/memory-hotplug.rst
> +++ b/Documentation/admin-guide/mm/memory-hotplug.rst
> @@ -410,7 +410,7 @@ them with ``memory_hotplug.`` such as::
>  
>  and they can be observed (and some even modified at runtime) via::
>  
> -	/sys/modules/memory_hotplug/parameters/
> +	/sys/module/memory_hotplug/parameters/
>  
>  The following module parameters are currently defined:
>  
> -- 
> 2.31.1
> 

-- 
Sincerely yours,
Mike.

Re: [PATCH v2 3/3] memory-hotplug.rst: document the "auto-movable" online policy

[Mike Rapoport]

On Mon, Oct 11, 2021 at 10:20:58AM +0200, David Hildenbrand wrote:
> In commit e83a437faa62 ("mm/memory_hotplug: introduce "auto-movable" online
> policy") we introduced a new memory online policy to automatically
> select a zone for memory blocks to be onlined. We added a way to
> set the active online policy and tunables for the auto-movable online
> policy. In follow-up commits we tweaked the "auto-movable" policy to also
> consider memory device details when selecting zones for memory blocks to
> be onlined.
> 
> Let's document the new toggles and how the two online policies we have
> work.
> 
> Signed-off-by: David Hildenbrand <david@redhat.com>

Acked-by: Mike Rapoport <rppt@linux.ibm.com>

> ---
>  .../admin-guide/mm/memory-hotplug.rst         | 141 +++++++++++++++---
>  1 file changed, 121 insertions(+), 20 deletions(-)
> 
> diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
> index ee00b70dedde..0f56ecd8ac05 100644
> --- a/Documentation/admin-guide/mm/memory-hotplug.rst
> +++ b/Documentation/admin-guide/mm/memory-hotplug.rst
> @@ -165,9 +165,8 @@ Or alternatively::
>  
>  	% echo 1 > /sys/devices/system/memory/memoryXXX/online
>  
> -The kernel will select the target zone automatically, usually defaulting to
> -``ZONE_NORMAL`` unless ``movable_node`` has been specified on the kernel
> -command line or if the memory block would intersect the ZONE_MOVABLE already.
> +The kernel will select the target zone automatically, depending on the
> +configured ``online_policy``.
>  
>  One can explicitly request to associate an offline memory block with
>  ZONE_MOVABLE by::
> @@ -198,6 +197,9 @@ Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or
>  
>  	% echo online > /sys/devices/system/memory/auto_online_blocks
>  
> +Similarly to manual onlining, with ``online`` the kernel will select the
> +target zone automatically, depending on the configured ``online_policy``.
> +
>  Modifying the auto-online behavior will only affect all subsequently added
>  memory blocks only.
>  
> @@ -393,11 +395,16 @@ command line parameters are relevant:
>  ======================== =======================================================
>  ``memhp_default_state``	 configure auto-onlining by essentially setting
>                           ``/sys/devices/system/memory/auto_online_blocks``.
> -``movable_node``	 configure automatic zone selection in the kernel. When
> -			 set, the kernel will default to ZONE_MOVABLE, unless
> -			 other zones can be kept contiguous.
> +``movable_node``	 configure automatic zone selection in the kernel when
> +			 using the ``contig-zones`` online policy. When
> +			 set, the kernel will default to ZONE_MOVABLE when
> +			 onlining a memory block, unless other zones can be kept
> +			 contiguous.
>  ======================== =======================================================
>  
> +See Documentation/admin-guide/kernel-parameters.txt for a more generic
> +description of these command line parameters.
> +
>  Module Parameters
>  ------------------
>  
> @@ -414,20 +421,114 @@ and they can be observed (and some even modified at runtime) via::
>  
>  The following module parameters are currently defined:
>  
> -======================== =======================================================
> -``memmap_on_memory``	 read-write: Allocate memory for the memmap from the
> -			 added memory block itself. Even if enabled, actual
> -			 support depends on various other system properties and
> -			 should only be regarded as a hint whether the behavior
> -			 would be desired.
> -
> -			 While allocating the memmap from the memory block
> -			 itself makes memory hotplug less likely to fail and
> -			 keeps the memmap on the same NUMA node in any case, it
> -			 can fragment physical memory in a way that huge pages
> -			 in bigger granularity cannot be formed on hotplugged
> -			 memory.
> -======================== =======================================================
> +================================ ===============================================
> +``memmap_on_memory``		 read-write: Allocate memory for the memmap from
> +				 the added memory block itself. Even if enabled,
> +				 actual support depends on various other system
> +				 properties and should only be regarded as a
> +				 hint whether the behavior would be desired.
> +
> +				 While allocating the memmap from the memory
> +				 block itself makes memory hotplug less likely
> +				 to fail and keeps the memmap on the same NUMA
> +				 node in any case, it can fragment physical
> +				 memory in a way that huge pages in bigger
> +				 granularity cannot be formed on hotplugged
> +				 memory.
> +``online_policy``		 read-write: Set the basic policy used for
> +				 automatic zone selection when onlining memory
> +				 blocks without specifying a target zone.
> +				 ``contig-zones`` has been the kernel default
> +				 before this parameter was added. After an
> +				 online policy was configured and memory was
> +				 online, the policy should not be changed
> +				 anymore.
> +
> +				 When set to ``contig-zones``, the kernel will
> +				 try keeping zones contiguous. If a memory block
> +				 intersects multiple zones or no zone, the
> +				 behavior depends on the ``movable_node`` kernel
> +				 command line parameter: default to ZONE_MOVABLE
> +				 if set, default to the applicable kernel zone
> +				 (usually ZONE_NORMAL) if not set.
> +
> +				 When set to ``auto-movable``, the kernel will
> +				 try onlining memory blocks to ZONE_MOVABLE if
> +				 possible according to the configuration and
> +				 memory device details. With this policy, one
> +				 can avoid zone imbalances when eventually
> +				 hotplugging a lot of memory later and still
> +				 wanting to be able to hotunplug as much as
> +				 possible reliably, very desirable in
> +				 virtualized environments. This policy ignores
> +				 the ``movable_node`` kernel command line
> +				 parameter and isn't really applicable in
> +				 environments that require it (e.g., bare metal
> +				 with hotunpluggable nodes) where hotplugged
> +				 memory might be exposed via the
> +				 firmware-provided memory map early during boot
> +				 to the system instead of getting detected,
> +				 added and onlined  later during boot (such as
> +				 done by virtio-mem or by some hypervisors
> +				 implementing emulated DIMMs). As one example, a
> +				 hotplugged DIMM will be onlined either
> +				 completely to ZONE_MOVABLE or completely to
> +				 ZONE_NORMAL, not a mixture.
> +				 As another example, as many memory blocks
> +				 belonging to a virtio-mem device will be
> +				 onlined to ZONE_MOVABLE as possible,
> +				 special-casing units of memory blocks that can
> +				 only get hotunplugged together. *This policy
> +				 does not protect from setups that are
> +				 problematic with ZONE_MOVABLE and does not
> +				 change the zone of memory blocks dynamically
> +				 after they were onlined.*
> +``auto_movable_ratio``		 read-write: Set the maximum MOVABLE:KERNEL
> +				 memory ratio in % for the ``auto-movable``
> +				 online policy. Whether the ratio applies only
> +				 for the system across all NUMA nodes or also
> +				 per NUMA nodes depends on the
> +				 ``auto_movable_numa_aware`` configuration.
> +
> +				 All accounting is based on present memory pages
> +				 in the zones combined with accounting per
> +				 memory device. Memory dedicated to the CMA
> +				 allocator is accounted as MOVABLE, although
> +				 residing on one of the kernel zones. The
> +				 possible ratio depends on the actual workload.
> +				 The kernel default is "301" %, for example,
> +				 allowing for hotplugging 24 GiB to a 8 GiB VM
> +				 and automatically onlining all hotplugged
> +				 memory to ZONE_MOVABLE in many setups. The
> +				 additional 1% deals with some pages being not
> +				 present, for example, because of some firmware
> +				 allocations.
> +
> +				 Note that ZONE_NORMAL memory provided by one
> +				 memory device does not allow for more
> +				 ZONE_MOVABLE memory for a different memory
> +				 device. As one example, onlining memory of a
> +				 hotplugged DIMM to ZONE_NORMAL will not allow
> +				 for another hotplugged DIMM to get onlined to
> +				 ZONE_MOVABLE automatically. In contrast, memory
> +				 hotplugged by a virtio-mem device that got
> +				 onlined to ZONE_NORMAL will allow for more
> +				 ZONE_MOVABLE memory within *the same*
> +				 virtio-mem device.
> +``auto_movable_numa_aware``	 read-write: Configure whether the
> +				 ``auto_movable_ratio`` in the ``auto-movable``
> +				 online policy also applies per NUMA
> +				 node in addition to the whole system across all
> +				 NUMA nodes. The kernel default is "Y".
> +
> +				 Disabling NUMA awareness can be helpful when
> +				 dealing with NUMA nodes that should be
> +				 completely hotunpluggable, onlining the memory
> +				 completely to ZONE_MOVABLE automatically if
> +				 possible.
> +
> +				 Parameter availability depends on CONFIG_NUMA.
> +================================ ===============================================
>  
>  ZONE_MOVABLE
>  ============
> -- 
> 2.31.1
> 

-- 
Sincerely yours,
Mike.