Re: [PATCH v1 5/5] mm/memory_hotplug: allow to specify a default online_type

[David Hildenbrand <david@redhat.com>]

On 11.03.20 17:55, Vitaly Kuznetsov wrote:
> David Hildenbrand <david@redhat.com> writes:
> 
>> For now, distributions implement advanced udev rules to essentially
>> - Don't online any hotplugged memory (s390x)
>> - Online all memory to ZONE_NORMAL (e.g., most virt environments like
>>   hyperv)
>> - Online all memory to ZONE_MOVABLE in case the zone imbalance is taken
>>   care of (e.g., bare metal, special virt environments)
>>
>> In summary: All memory is usually onlined the same way, however, the
>> kernel always has to ask userspace to come up with the same answer.
>> E.g., HyperV always waits for a memory block to get onlined before
>> continuing, otherwise it might end up adding memory faster than
>> hotplugging it, which can result in strange OOM situations.
>>
>> Let's allow to specify a default online_type, not just "online" and
>> "offline". This allows distributions to configure the default online_type
>> when booting up and be done with it.
>>
>> We can now specify "offline", "online", "online_movable" and
>> "online_kernel" via
>> - "memhp_default_state=" on the kernel cmdline
>> - /sys/devices/systemn/memory/auto_online_blocks
>> just like we are able to specify for a single memory block via
>> /sys/devices/systemn/memory/memoryX/state
>>
> 
> Thank you for picking this up! 
> 
> It's been awhile since I've added CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE
> but I vaguely recall one problem: memory hotplug may happen *very* early
> (just because some memory is presented to a VM as hotplug memory, it is
> not in e820). It happens way before we launch userspace (including
> udev). The question is -- which ZONE will this memory be assigned too?

If it's added via add_memory() ("hot/cold plugged memory") like ACPI
DIMMs not part of e820, Hyper-V balloon added memory, XEN balloon added
memory, s390x standby memory etc. the memory will be onlined as
configured via memhp_default_online_type. Assume that one is set to
"offline".

*If* userspace changes memhp_default_online_type (as in my script in the
cover letter), userspace has to online all memory that has been added
before userspace was active itself (again, as done in my script).

Memory not added via add_memory() is considered "initial memory" and not
as hot/cold plugged memory.

Same handling as for now using udev rules. (once userspace is up, udev
rules for all early added memory is triggered as well)

> 
> 'memhp_default_state=' resolves the issue but nobody likes additional
> kernel parameters for anything but
> debug. CONFIG_MEMORY_HOTPLUG_DEFAULT_ONLINE was supposed to help, but it
> is binary and distro-wide (so *all* deployments will get the same
> default and as you validly stated we want it differently).
> 
> We could've added something like your example onlining script to the
> kernel itself but this is likely going to be hard to sell: "policies
> belong to userspace!" will likely be the answer. 

Exactly my thought.

-- 
Thanks,

David / dhildenb