[PATCH v1 0/3] memory-hotplug.rst: document the "auto-movable" online policy

[PATCH v1 0/3] memory-hotplug.rst: document the "auto-movable" online policy

[David Hildenbrand]

As promised, now that the memory-hotplug.rst overhaul is upstream, proper
documentation for the "auto-movable" online policy, documenting all new
toggles and options. Along, two fixes for the original overhaul.

Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Oscar Salvador <osalvador@suse.de>
Cc: Mike Rapoport <rppt@kernel.org>
Cc: linux-doc@vger.kernel.org
Cc: linux-mm@kvack.org

David Hildenbrand (3):
  memory-hotplug.rst: fix two instances of "movablecore" that should be
    "movable_node"
  memory-hotplug.rst: fix wrong /sys/module/memory_hotplug/parameters/
    path
  memory-hotplug.rst: document the "auto-movable" online policy

 .../admin-guide/mm/memory-hotplug.rst         | 130 +++++++++++++++---
 1 file changed, 109 insertions(+), 21 deletions(-)

-- 
2.31.1

[PATCH v1 1/3] memory-hotplug.rst: fix two instances of "movablecore" that should be "movable_node"

[David Hildenbrand]

We really want to refer to the "movable_node" kernel command line
parameter here.

Fixes: ac3332c44767 ("memory-hotplug.rst: complete admin-guide overhaul")
Signed-off-by: David Hildenbrand <david@redhat.com>
---
 Documentation/admin-guide/mm/memory-hotplug.rst | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
index 03dfbc925252..27d748cb6ee0 100644
--- a/Documentation/admin-guide/mm/memory-hotplug.rst
+++ b/Documentation/admin-guide/mm/memory-hotplug.rst
@@ -166,7 +166,7 @@ Or alternatively::
 	% echo 1 > /sys/devices/system/memory/memoryXXX/online
 
 The kernel will select the target zone automatically, usually defaulting to
-``ZONE_NORMAL`` unless ``movablecore=1`` has been specified on the kernel
+``ZONE_NORMAL`` unless ``movable_node`` has been specified on the kernel
 command line or if the memory block would intersect the ZONE_MOVABLE already.
 
 One can explicitly request to associate an offline memory block with
@@ -393,7 +393,7 @@ command line parameters are relevant:
 ======================== =======================================================
 ``memhp_default_state``	 configure auto-onlining by essentially setting
                          ``/sys/devices/system/memory/auto_online_blocks``.
-``movablecore``		 configure automatic zone selection of the kernel. When
+``movable_node``	 configure automatic zone selection in the kernel. When
 			 set, the kernel will default to ZONE_MOVABLE, unless
 			 other zones can be kept contiguous.
 ======================== =======================================================
-- 
2.31.1

[PATCH v1 2/3] memory-hotplug.rst: fix wrong /sys/module/memory_hotplug/parameters/ path

[David Hildenbrand]

We accidentially added a superfluous "s".

Fixes: ac3332c44767 ("memory-hotplug.rst: complete admin-guide overhaul")
Signed-off-by: David Hildenbrand <david@redhat.com>
---
 Documentation/admin-guide/mm/memory-hotplug.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
index 27d748cb6ee0..ee00b70dedde 100644
--- a/Documentation/admin-guide/mm/memory-hotplug.rst
+++ b/Documentation/admin-guide/mm/memory-hotplug.rst
@@ -410,7 +410,7 @@ them with ``memory_hotplug.`` such as::
 
 and they can be observed (and some even modified at runtime) via::
 
-	/sys/modules/memory_hotplug/parameters/
+	/sys/module/memory_hotplug/parameters/
 
 The following module parameters are currently defined:
 
-- 
2.31.1

[PATCH v1 3/3] memory-hotplug.rst: document the "auto-movable" online policy

[David Hildenbrand]

In commit e83a437faa62 ("mm/memory_hotplug: introduce "auto-movable" online
policy") we introduced a new memory online policy to automatically
select a zone for memory blocks to be onlined. We added a way to
set the active online policy and tunables for the auto-movable online
policy. In follow-up commits we tweaked the "auto-movable" policy to also
consider memory device details when selecting zones for memory blocks to
be onlined.

Let's document the new toggles and how the two online policies we have
work.

Signed-off-by: David Hildenbrand <david@redhat.com>
---
 .../admin-guide/mm/memory-hotplug.rst         | 128 +++++++++++++++---
 1 file changed, 108 insertions(+), 20 deletions(-)

diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
index ee00b70dedde..c20a2c0031cf 100644
--- a/Documentation/admin-guide/mm/memory-hotplug.rst
+++ b/Documentation/admin-guide/mm/memory-hotplug.rst
@@ -165,9 +165,8 @@ Or alternatively::
 
 	% echo 1 > /sys/devices/system/memory/memoryXXX/online
 
-The kernel will select the target zone automatically, usually defaulting to
-``ZONE_NORMAL`` unless ``movable_node`` has been specified on the kernel
-command line or if the memory block would intersect the ZONE_MOVABLE already.
+The kernel will select the target zone automatically, depending on the
+configured ``online_policy``.
 
 One can explicitly request to associate an offline memory block with
 ZONE_MOVABLE by::
@@ -198,6 +197,9 @@ Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or
 
 	% echo online > /sys/devices/system/memory/auto_online_blocks
 
+Similarly to manual onlining, with ``online`` the kernel will select the
+target zone automatically, depending on the configured ``online_policy``.
+
 Modifying the auto-online behavior will only affect all subsequently added
 memory blocks only.
 
@@ -393,9 +395,11 @@ command line parameters are relevant:
 ======================== =======================================================
 ``memhp_default_state``	 configure auto-onlining by essentially setting
                          ``/sys/devices/system/memory/auto_online_blocks``.
-``movable_node``	 configure automatic zone selection in the kernel. When
-			 set, the kernel will default to ZONE_MOVABLE, unless
-			 other zones can be kept contiguous.
+``movable_node``	 configure automatic zone selection in the kernel when
+			 using the ``contig-zones`` online policy. When
+			 set, the kernel will default to ZONE_MOVABLE when
+			 onlining a memory block, unless other zones can be kept
+			 contiguous.
 ======================== =======================================================
 
 Module Parameters
@@ -414,20 +418,104 @@ and they can be observed (and some even modified at runtime) via::
 
 The following module parameters are currently defined:
 
-======================== =======================================================
-``memmap_on_memory``	 read-write: Allocate memory for the memmap from the
-			 added memory block itself. Even if enabled, actual
-			 support depends on various other system properties and
-			 should only be regarded as a hint whether the behavior
-			 would be desired.
-
-			 While allocating the memmap from the memory block
-			 itself makes memory hotplug less likely to fail and
-			 keeps the memmap on the same NUMA node in any case, it
-			 can fragment physical memory in a way that huge pages
-			 in bigger granularity cannot be formed on hotplugged
-			 memory.
-======================== =======================================================
+================================ ===============================================
+``memmap_on_memory``		 read-write: Allocate memory for the memmap from
+				 the added memory block itself. Even if enabled,
+				 actual support depends on various other system
+				 properties and should only be regarded as a
+				 hint whether the behavior would be desired.
+
+				 While allocating the memmap from the memory
+				 block itself makes memory hotplug less likely
+				 to fail and keeps the memmap on the same NUMA
+				 node in any case, it can fragment physical
+				 memory in a way that huge pages in bigger
+				 granularity cannot be formed on hotplugged
+				 memory.
+``online_policy``		 read-write: Set the basic policy used for
+				 automatic zone selection when onlining memory
+				 blocks without specifying a target zone.
+				 ``contig-zones`` has been the kernel default
+				 before this parameter was added. After an
+				 online policy was configured and memory was
+				 online, the policy should not be changed
+				 anymore.
+
+				 When set to ``contig-zones``, the kernel will
+				 try keeping zones contiguous. If a memory block
+				 intersects multiple zones or no zone, the
+				 behavior depends on the ``movable_node`` kernel
+				 command line parameter: default to ZONE_MOVABLE
+				 if set, default to the applicable kernel zone
+				 (usually ZONE_NORMAL) if not set.
+
+				 When set to ``auto-movable``, the kernel will
+				 try onlining memory blocks to ZONE_MOVABLE if
+				 possible according to the configuration and
+				 memory device details. With this policy, one
+				 can avoid zone imbalances when eventually
+				 hotplugging a lot of memory later and still
+				 wanting to be able to hotunplug as much as
+				 possible reliably, very desirable in
+				 virtualized environments. As one example, a
+				 hotplugged DIMM will be onlined either
+				 completely to ZONE_MOVABLE or completely to
+				 ZONE_NORMAL, not a mixture.
+				 As another example, as many memory blocks
+				 belonging to a virtio-mem device will be
+				 onlined to ZONE_MOVABLE as possible,
+				 special-casing units of memory blocks that can
+				 only get hotunplugged together. *This policy
+				 does not protect from setups that are
+				 problematic with ZONE_MOVABLE and does not
+				 change the zone of memory blocks dynamically
+				 after they were onlined.*
+``auto_movable_ratio``		 read-write: Set the maximum MOVABLE:KERNEL
+				 memory ratio in % for the ``auto-movable``
+				 online policy. Whether the ratio applies only
+				 for the system across all NUMA nodes or also
+				 per NUMA nodes depends on the
+				 ``auto_movable_numa_aware`` configuration.
+
+				 All accounting is based on present memory pages
+				 in the zones combined with accounting per
+				 memory device. Memory dedicated to the CMA
+				 allocator is accounted as MOVABLE, although
+				 residing on one of the kernel zones. The
+				 possible ratio depends on the actual workload.
+				 The kernel default is "301" %, for example,
+				 allowing for hotplugging 24 GiB to a 8 GiB VM
+				 and automatically onlining all hotplugged
+				 memory to ZONE_MOVABLE in many setups. The
+				 additional 1% deals with some pages being not
+				 present, for example, because of some firmware
+				 allocations.
+
+				 Note that ZONE_NORMAL memory provided by one
+				 memory device does not allow for more
+				 ZONE_MOVABLE memory for a different memory
+				 device. As one example, onlining memory of a
+				 hotplugged DIMM to ZONE_NORMAL will not allow
+				 for another hotplugged DIMM to get onlined to
+				 ZONE_MOVABLE automatically. In contrast, memory
+				 hotplugged by a virtio-mem device that got
+				 onlined to ZONE_NORMAL will allow for more
+				 ZONE_MOVABLE memory within *the same*
+				 virtio-mem device.
+``auto_movable_numa_aware``	 read-write: Configure whether the
+				 ``auto_movable_ratio`` in the ``auto-movable``
+				 online policy also applies per NUMA
+				 node in addition to the whole system across all
+				 NUMA nodes. The kernel default is "Y".
+
+				 Disabling NUMA awareness can be helpful when
+				 dealing with NUMA nodes that should be
+				 completely hotunpluggable, onlining the memory
+				 completely to ZONE_MOVABLE automatically if
+				 possible.
+
+				 Parameter availability depends on CONFIG_NUMA.
+================================ ===============================================
 
 ZONE_MOVABLE
 ============
-- 
2.31.1

Re: [PATCH v1 3/3] memory-hotplug.rst: document the "auto-movable" online policy

[Mike Rapoport]

On Thu, Sep 30, 2021 at 04:41:17PM +0200, David Hildenbrand wrote:
> In commit e83a437faa62 ("mm/memory_hotplug: introduce "auto-movable" online
> policy") we introduced a new memory online policy to automatically
> select a zone for memory blocks to be onlined. We added a way to
> set the active online policy and tunables for the auto-movable online
> policy. In follow-up commits we tweaked the "auto-movable" policy to also
> consider memory device details when selecting zones for memory blocks to
> be onlined.
> 
> Let's document the new toggles and how the two online policies we have
> work.
> 
> Signed-off-by: David Hildenbrand <david@redhat.com>
> ---
>  .../admin-guide/mm/memory-hotplug.rst         | 128 +++++++++++++++---
>  1 file changed, 108 insertions(+), 20 deletions(-)
> 
> diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
> index ee00b70dedde..c20a2c0031cf 100644
> --- a/Documentation/admin-guide/mm/memory-hotplug.rst
> +++ b/Documentation/admin-guide/mm/memory-hotplug.rst
> @@ -165,9 +165,8 @@ Or alternatively::
>  
>  	% echo 1 > /sys/devices/system/memory/memoryXXX/online
>  
> -The kernel will select the target zone automatically, usually defaulting to
> -``ZONE_NORMAL`` unless ``movable_node`` has been specified on the kernel
> -command line or if the memory block would intersect the ZONE_MOVABLE already.
> +The kernel will select the target zone automatically, depending on the
> +configured ``online_policy``.
>  
>  One can explicitly request to associate an offline memory block with
>  ZONE_MOVABLE by::
> @@ -198,6 +197,9 @@ Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or
>  
>  	% echo online > /sys/devices/system/memory/auto_online_blocks
>  
> +Similarly to manual onlining, with ``online`` the kernel will select the
> +target zone automatically, depending on the configured ``online_policy``.
> +
>  Modifying the auto-online behavior will only affect all subsequently added
>  memory blocks only.
>  
> @@ -393,9 +395,11 @@ command line parameters are relevant:
>  ======================== =======================================================
>  ``memhp_default_state``	 configure auto-onlining by essentially setting
>                           ``/sys/devices/system/memory/auto_online_blocks``.
> -``movable_node``	 configure automatic zone selection in the kernel. When
> -			 set, the kernel will default to ZONE_MOVABLE, unless
> -			 other zones can be kept contiguous.
> +``movable_node``	 configure automatic zone selection in the kernel when
> +			 using the ``contig-zones`` online policy. When
> +			 set, the kernel will default to ZONE_MOVABLE when
> +			 onlining a memory block, unless other zones can be kept
> +			 contiguous.

The movable_node main purpose is to allow unplugging an entire node. Zone
selection is a consequence of this. You may want to cite the description of
movable_node in kernel-paramenters.txt here.

And, pardon my ignorance, how movable_node will play with auto-movable
policy?

>  ======================== =======================================================
>  
>  Module Parameters
> @@ -414,20 +418,104 @@ and they can be observed (and some even modified at runtime) via::
>  
>  The following module parameters are currently defined:
>  
> -======================== =======================================================
> -``memmap_on_memory``	 read-write: Allocate memory for the memmap from the
> -			 added memory block itself. Even if enabled, actual
> -			 support depends on various other system properties and
> -			 should only be regarded as a hint whether the behavior
> -			 would be desired.
> -
> -			 While allocating the memmap from the memory block
> -			 itself makes memory hotplug less likely to fail and
> -			 keeps the memmap on the same NUMA node in any case, it
> -			 can fragment physical memory in a way that huge pages
> -			 in bigger granularity cannot be formed on hotplugged
> -			 memory.
> -======================== =======================================================
> +================================ ===============================================
> +``memmap_on_memory``		 read-write: Allocate memory for the memmap from
> +				 the added memory block itself. Even if enabled,
> +				 actual support depends on various other system
> +				 properties and should only be regarded as a
> +				 hint whether the behavior would be desired.
> +
> +				 While allocating the memmap from the memory
> +				 block itself makes memory hotplug less likely
> +				 to fail and keeps the memmap on the same NUMA
> +				 node in any case, it can fragment physical
> +				 memory in a way that huge pages in bigger
> +				 granularity cannot be formed on hotplugged
> +				 memory.
> +``online_policy``		 read-write: Set the basic policy used for
> +				 automatic zone selection when onlining memory
> +				 blocks without specifying a target zone.
> +				 ``contig-zones`` has been the kernel default
> +				 before this parameter was added. After an
> +				 online policy was configured and memory was
> +				 online, the policy should not be changed
> +				 anymore.
> +
> +				 When set to ``contig-zones``, the kernel will
> +				 try keeping zones contiguous. If a memory block
> +				 intersects multiple zones or no zone, the
> +				 behavior depends on the ``movable_node`` kernel
> +				 command line parameter: default to ZONE_MOVABLE
> +				 if set, default to the applicable kernel zone
> +				 (usually ZONE_NORMAL) if not set.
> +
> +				 When set to ``auto-movable``, the kernel will
> +				 try onlining memory blocks to ZONE_MOVABLE if
> +				 possible according to the configuration and
> +				 memory device details. With this policy, one
> +				 can avoid zone imbalances when eventually
> +				 hotplugging a lot of memory later and still
> +				 wanting to be able to hotunplug as much as
> +				 possible reliably, very desirable in
> +				 virtualized environments. As one example, a
> +				 hotplugged DIMM will be onlined either
> +				 completely to ZONE_MOVABLE or completely to
> +				 ZONE_NORMAL, not a mixture.
> +				 As another example, as many memory blocks
> +				 belonging to a virtio-mem device will be
> +				 onlined to ZONE_MOVABLE as possible,
> +				 special-casing units of memory blocks that can
> +				 only get hotunplugged together. *This policy
> +				 does not protect from setups that are
> +				 problematic with ZONE_MOVABLE and does not
> +				 change the zone of memory blocks dynamically
> +				 after they were onlined.*
> +``auto_movable_ratio``		 read-write: Set the maximum MOVABLE:KERNEL
> +				 memory ratio in % for the ``auto-movable``
> +				 online policy. Whether the ratio applies only
> +				 for the system across all NUMA nodes or also
> +				 per NUMA nodes depends on the
> +				 ``auto_movable_numa_aware`` configuration.
> +
> +				 All accounting is based on present memory pages
> +				 in the zones combined with accounting per
> +				 memory device. Memory dedicated to the CMA
> +				 allocator is accounted as MOVABLE, although
> +				 residing on one of the kernel zones. The
> +				 possible ratio depends on the actual workload.
> +				 The kernel default is "301" %, for example,
> +				 allowing for hotplugging 24 GiB to a 8 GiB VM
> +				 and automatically onlining all hotplugged
> +				 memory to ZONE_MOVABLE in many setups. The
> +				 additional 1% deals with some pages being not
> +				 present, for example, because of some firmware
> +				 allocations.
> +
> +				 Note that ZONE_NORMAL memory provided by one
> +				 memory device does not allow for more
> +				 ZONE_MOVABLE memory for a different memory
> +				 device. As one example, onlining memory of a
> +				 hotplugged DIMM to ZONE_NORMAL will not allow
> +				 for another hotplugged DIMM to get onlined to
> +				 ZONE_MOVABLE automatically. In contrast, memory
> +				 hotplugged by a virtio-mem device that got
> +				 onlined to ZONE_NORMAL will allow for more
> +				 ZONE_MOVABLE memory within *the same*
> +				 virtio-mem device.
> +``auto_movable_numa_aware``	 read-write: Configure whether the
> +				 ``auto_movable_ratio`` in the ``auto-movable``
> +				 online policy also applies per NUMA
> +				 node in addition to the whole system across all
> +				 NUMA nodes. The kernel default is "Y".
> +
> +				 Disabling NUMA awareness can be helpful when
> +				 dealing with NUMA nodes that should be
> +				 completely hotunpluggable, onlining the memory
> +				 completely to ZONE_MOVABLE automatically if
> +				 possible.
> +
> +				 Parameter availability depends on CONFIG_NUMA.
> +================================ ===============================================
>  
>  ZONE_MOVABLE
>  ============
> -- 
> 2.31.1
> 

-- 
Sincerely yours,
Mike.

Re: [PATCH v1 3/3] memory-hotplug.rst: document the "auto-movable" online policy

[David Hildenbrand]

On 06.10.21 02:35, Mike Rapoport wrote:
> On Thu, Sep 30, 2021 at 04:41:17PM +0200, David Hildenbrand wrote:
>> In commit e83a437faa62 ("mm/memory_hotplug: introduce "auto-movable" online
>> policy") we introduced a new memory online policy to automatically
>> select a zone for memory blocks to be onlined. We added a way to
>> set the active online policy and tunables for the auto-movable online
>> policy. In follow-up commits we tweaked the "auto-movable" policy to also
>> consider memory device details when selecting zones for memory blocks to
>> be onlined.
>>
>> Let's document the new toggles and how the two online policies we have
>> work.
>>
>> Signed-off-by: David Hildenbrand <david@redhat.com>
>> ---
>>   .../admin-guide/mm/memory-hotplug.rst         | 128 +++++++++++++++---
>>   1 file changed, 108 insertions(+), 20 deletions(-)
>>
>> diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
>> index ee00b70dedde..c20a2c0031cf 100644
>> --- a/Documentation/admin-guide/mm/memory-hotplug.rst
>> +++ b/Documentation/admin-guide/mm/memory-hotplug.rst
>> @@ -165,9 +165,8 @@ Or alternatively::
>>   
>>   	% echo 1 > /sys/devices/system/memory/memoryXXX/online
>>   
>> -The kernel will select the target zone automatically, usually defaulting to
>> -``ZONE_NORMAL`` unless ``movable_node`` has been specified on the kernel
>> -command line or if the memory block would intersect the ZONE_MOVABLE already.
>> +The kernel will select the target zone automatically, depending on the
>> +configured ``online_policy``.
>>   
>>   One can explicitly request to associate an offline memory block with
>>   ZONE_MOVABLE by::
>> @@ -198,6 +197,9 @@ Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or
>>   
>>   	% echo online > /sys/devices/system/memory/auto_online_blocks
>>   
>> +Similarly to manual onlining, with ``online`` the kernel will select the
>> +target zone automatically, depending on the configured ``online_policy``.
>> +
>>   Modifying the auto-online behavior will only affect all subsequently added
>>   memory blocks only.
>>   
>> @@ -393,9 +395,11 @@ command line parameters are relevant:
>>   ======================== =======================================================
>>   ``memhp_default_state``	 configure auto-onlining by essentially setting
>>                            ``/sys/devices/system/memory/auto_online_blocks``.
>> -``movable_node``	 configure automatic zone selection in the kernel. When
>> -			 set, the kernel will default to ZONE_MOVABLE, unless
>> -			 other zones can be kept contiguous.
>> +``movable_node``	 configure automatic zone selection in the kernel when
>> +			 using the ``contig-zones`` online policy. When
>> +			 set, the kernel will default to ZONE_MOVABLE when
>> +			 onlining a memory block, unless other zones can be kept
>> +			 contiguous.
> 
> The movable_node main purpose is to allow unplugging an entire node. Zone
> selection is a consequence of this. You may want to cite the description of
> movable_node in kernel-paramenters.txt here.

Right, I only document the effects of these parameters on memory 
hot(un)plug.

What about:

diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst 
b/Documentation/admin-guide/mm/memory-hotplug.rst
index c20a2c0031cf..f8976ded0863 100644
--- a/Documentation/admin-guide/mm/memory-hotplug.rst
+++ b/Documentation/admin-guide/mm/memory-hotplug.rst
@@ -402,6 +402,9 @@ command line parameters are relevant:
                          contiguous.
  ======================== 
=======================================================

+See Documentation/admin-guide/kernel-parameters.txt for a more generic
+description of these command line parameters.
+
  Module Parameters
  ------------------


> 
> And, pardon my ignorance, how movable_node will play with auto-movable
> policy?

It's essentially ignored with the auto-movable policy for memory 
hotplugged after boot (!MEMBLOCK_HOTPLUG). That's why only the 
description of "contig-zones" below describes how it interacts with the 
``movable_node``, and we make it clear here that it's restricted to the 
"contig-zones" policy as well.

<details>
Bare metal, where we care about reliably unplugging hotplugged memory 
usually configures auto-onlining to "online_movable": for example, 
that's the case on RHEL. auto-movable doesn't make too much sense for 
bare metal: the nature of "movable_node" is to essentially online 
anything that might get hotunplugged MOVABLE, especially after 
hotplugging memory and rebooting: that is highly dangerous especially in 
virtualized environments.

"auto-movable" is valuable in virtualized environments, where we add 
memory via:
* add_memory_driver_managed() like virtio-mem, whereby such memory is
   never part of the firmware provided memory-map, so the policy is
   always in control even after a reboot.
* Hotplugged virtual DIMMs, such as provided by x86-64/arm64, whereby we
   don't include these DIMMs in the firmware-provided memory map, but
   ACPI code adds them after early boot, making it behave similar to
   add_memory_driver_managed() -- the policy is always in control even
   after a reboot.
</details>


Thanks!

-- 
Thanks,

David / dhildenb

Re: [PATCH v1 3/3] memory-hotplug.rst: document the "auto-movable" online policy

[Mike Rapoport]

On Wed, Oct 06, 2021 at 10:01:39AM +0200, David Hildenbrand wrote:
> On 06.10.21 02:35, Mike Rapoport wrote:
> > On Thu, Sep 30, 2021 at 04:41:17PM +0200, David Hildenbrand wrote:
> > > In commit e83a437faa62 ("mm/memory_hotplug: introduce "auto-movable" online
> > > policy") we introduced a new memory online policy to automatically
> > > select a zone for memory blocks to be onlined. We added a way to
> > > set the active online policy and tunables for the auto-movable online
> > > policy. In follow-up commits we tweaked the "auto-movable" policy to also
> > > consider memory device details when selecting zones for memory blocks to
> > > be onlined.
> > > 
> > > Let's document the new toggles and how the two online policies we have
> > > work.
> > > 
> > > Signed-off-by: David Hildenbrand <david@redhat.com>
> > > ---
> > >   .../admin-guide/mm/memory-hotplug.rst         | 128 +++++++++++++++---
> > >   1 file changed, 108 insertions(+), 20 deletions(-)
> > > 
> > > diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst b/Documentation/admin-guide/mm/memory-hotplug.rst
> > > index ee00b70dedde..c20a2c0031cf 100644
> > > --- a/Documentation/admin-guide/mm/memory-hotplug.rst
> > > +++ b/Documentation/admin-guide/mm/memory-hotplug.rst
> > > @@ -165,9 +165,8 @@ Or alternatively::
> > >   	% echo 1 > /sys/devices/system/memory/memoryXXX/online
> > > -The kernel will select the target zone automatically, usually defaulting to
> > > -``ZONE_NORMAL`` unless ``movable_node`` has been specified on the kernel
> > > -command line or if the memory block would intersect the ZONE_MOVABLE already.
> > > +The kernel will select the target zone automatically, depending on the
> > > +configured ``online_policy``.
> > >   One can explicitly request to associate an offline memory block with
> > >   ZONE_MOVABLE by::
> > > @@ -198,6 +197,9 @@ Auto-onlining can be enabled by writing ``online``, ``online_kernel`` or
> > >   	% echo online > /sys/devices/system/memory/auto_online_blocks
> > > +Similarly to manual onlining, with ``online`` the kernel will select the
> > > +target zone automatically, depending on the configured ``online_policy``.
> > > +
> > >   Modifying the auto-online behavior will only affect all subsequently added
> > >   memory blocks only.
> > > @@ -393,9 +395,11 @@ command line parameters are relevant:
> > >   ======================== =======================================================
> > >   ``memhp_default_state``	 configure auto-onlining by essentially setting
> > >                            ``/sys/devices/system/memory/auto_online_blocks``.
> > > -``movable_node``	 configure automatic zone selection in the kernel. When
> > > -			 set, the kernel will default to ZONE_MOVABLE, unless
> > > -			 other zones can be kept contiguous.
> > > +``movable_node``	 configure automatic zone selection in the kernel when
> > > +			 using the ``contig-zones`` online policy. When
> > > +			 set, the kernel will default to ZONE_MOVABLE when
> > > +			 onlining a memory block, unless other zones can be kept
> > > +			 contiguous.
> > 
> > The movable_node main purpose is to allow unplugging an entire node. Zone
> > selection is a consequence of this. You may want to cite the description of
> > movable_node in kernel-paramenters.txt here.
> 
> Right, I only document the effects of these parameters on memory
> hot(un)plug.
> 
> What about:
> 
> diff --git a/Documentation/admin-guide/mm/memory-hotplug.rst
> b/Documentation/admin-guide/mm/memory-hotplug.rst
> index c20a2c0031cf..f8976ded0863 100644
> --- a/Documentation/admin-guide/mm/memory-hotplug.rst
> +++ b/Documentation/admin-guide/mm/memory-hotplug.rst
> @@ -402,6 +402,9 @@ command line parameters are relevant:
>                          contiguous.
>  ========================
> =======================================================
> 
> +See Documentation/admin-guide/kernel-parameters.txt for a more generic
> +description of these command line parameters.
> +

Ok.

>  Module Parameters
>  ------------------
> 
> 
> > 
> > And, pardon my ignorance, how movable_node will play with auto-movable
> > policy?
> 
> It's essentially ignored with the auto-movable policy for memory hotplugged
> after boot (!MEMBLOCK_HOTPLUG). That's why only the description of
> "contig-zones" below describes how it interacts with the ``movable_node``,
> and we make it clear here that it's restricted to the "contig-zones" policy
> as well.
> 
> <details>
> Bare metal, where we care about reliably unplugging hotplugged memory
> usually configures auto-onlining to "online_movable": for example, that's
> the case on RHEL. auto-movable doesn't make too much sense for bare metal:
> the nature of "movable_node" is to essentially online anything that might
> get hotunplugged MOVABLE, especially after hotplugging memory and rebooting:
> that is highly dangerous especially in virtualized environments.
> 
> "auto-movable" is valuable in virtualized environments, where we add memory
> via:
> * add_memory_driver_managed() like virtio-mem, whereby such memory is
>   never part of the firmware provided memory-map, so the policy is
>   always in control even after a reboot.
> * Hotplugged virtual DIMMs, such as provided by x86-64/arm64, whereby we
>   don't include these DIMMs in the firmware-provided memory map, but
>   ACPI code adds them after early boot, making it behave similar to
>   add_memory_driver_managed() -- the policy is always in control even
>   after a reboot.
> </details>
 
Do you want to put it somewhere in Documentation/ ?
It's already written anyway ;-)

-- 
Sincerely yours,
Mike.

Re: [PATCH v1 3/3] memory-hotplug.rst: document the "auto-movable" online policy

[David Hildenbrand]

>> It's essentially ignored with the auto-movable policy for memory hotplugged
>> after boot (!MEMBLOCK_HOTPLUG). That's why only the description of
>> "contig-zones" below describes how it interacts with the ``movable_node``,
>> and we make it clear here that it's restricted to the "contig-zones" policy
>> as well.
>>
>> <details>
>> Bare metal, where we care about reliably unplugging hotplugged memory
>> usually configures auto-onlining to "online_movable": for example, that's
>> the case on RHEL. auto-movable doesn't make too much sense for bare metal:
>> the nature of "movable_node" is to essentially online anything that might
>> get hotunplugged MOVABLE, especially after hotplugging memory and rebooting:
>> that is highly dangerous especially in virtualized environments.
>>
>> "auto-movable" is valuable in virtualized environments, where we add memory
>> via:
>> * add_memory_driver_managed() like virtio-mem, whereby such memory is
>>    never part of the firmware provided memory-map, so the policy is
>>    always in control even after a reboot.
>> * Hotplugged virtual DIMMs, such as provided by x86-64/arm64, whereby we
>>    don't include these DIMMs in the firmware-provided memory map, but
>>    ACPI code adds them after early boot, making it behave similar to
>>    add_memory_driver_managed() -- the policy is always in control even
>>    after a reboot.
>> </details>
>   
> Do you want to put it somewhere in Documentation/ ?
> It's already written anyway ;-)
> 

I'll add to the "auto-movable" description:

"This policy ignores the ``movable_node`` kernel command line parameter 
and isn't really applicable in environments that require it (e.g., bare 
metal with hotunpluggable nodes) where hotplugged memory might be 
exposed via the firmware-provided memory map early during boot to the 
system instead of getting detected, added and onlined later during boot 
(such as done by virito-mem or by some hypervisors implementing emulated 
DIMMs)."

Thanks Mike!

-- 
Thanks,

David / dhildenb