Re: arm64: dropping prevent_bootmem_remove_notifier

[Sudarshan Rajagopalan <sudaraja@codeaurora.org>]

Hi Anshuman, David,

Thanks for all the detailed explanations for the reasoning to have 
bootmem protected from being removed. Also, I do agree drivers being 
able to mark memory sections isn't the right thing to do.

We went ahead with the approach of using "mem=" as you suggested to 
limit the bootmem and add remaining blocks using 
add_memory_driver_managed() so that driver has ownership of these 
blocks.

We do have some follow-up questions regarding this - will initiate a 
discussion soon.


On 2020-10-18 22:37, Anshuman Khandual wrote:
> Hello Sudarshan,
> 
> On 10/17/2020 04:41 AM, Sudarshan Rajagopalan wrote:
>> 
>> Hello Anshuman,
>> 
>> In the patch that enables memory hot-remove (commit bbd6ec605c0f 
>> ("arm64/mm: Enable memory hot remove")) for arm64, there’s a notifier 
>> put in place that prevents boot memory from being offlined and 
>> removed. Also commit text mentions that boot memory on arm64 cannot be 
>> removed. We wanted to understand more about the reasoning for this. 
>> X86 and other archs doesn’t seem to do this prevention. There’s also 
>> comment in the code that this notifier could be dropped in future if 
>> and when boot memory can be removed.
> 
> Right and till then the notifier cannot be dropped. There was a lot of
> discussions
> around this topic during multiple iterations of memory hot remove
> series. Hence, I
> would just request you to please go through them first. This list here
> is from one
> such series (https://lwn.net/Articles/809179/) but might not be 
> exhaustive.
> 
> -----------------
> On arm64 platform, it is essential to ensure that the boot time 
> discovered
> memory couldn't be hot-removed so that,
> 
> 1. FW data structures used across kexec are idempotent
>    e.g. the EFI memory map.
> 
> 2. linear map or vmemmap would not have to be dynamically split, and 
> can
>    map boot memory at a large granularity
> 
> 3. Avoid penalizing paths that have to walk page tables, where we can 
> be
>    certain that the memory is not hot-removable
> -----------------
> 
> The primary reason being kexec which would need substantial rework 
> otherwise.
> 
>> 
>> The current logic is that only “new” memory blocks which are hot-added 
>> can later be offlined and removed. The memory that system booted up 
>> with cannot be offlined and removed. But there could be many usercases 
>> such as inter-VM memory sharing where a primary VM could offline and 
>> hot-remove a block/section of memory and lend it to secondary VM where 
>> it could hot-add it. And after usecase is done, the reverse happens 
>> where secondary VM hot-removes and gives it back to primary which can 
>> hot-add it back. In such cases, the present logic for arm64 doesn’t 
>> allow this hot-remove in primary to happen.
> 
> That is not true. Each VM could just boot with a minimum boot memory 
> which can
> not be offlined or removed but then a possible larger portion of memory 
> can be
> hot added during the boot process itself, making them available for any 
> future
> inter VM sharing purpose. Hence this problem could easily be solved in 
> the user
> space itself.
> 
>> 
>> Also, on systems with movable zone that sort of guarantees pages to be 
>> migrated and isolated so that blocks can be offlined, this logic also 
>> defeats the purpose of having a movable zone which system can rely on 
>> memory hot-plugging, which say virt-io mem also relies on for fully 
>> plugged memory blocks.
> ZONE_MOVABLE does not really guarantee migration, isolation and 
> removal. There
> are reasons an offline request might just fail. I agree that those 
> reasons are
> normally not platform related but core memory gives platform an 
> opportunity to
> decline an offlining request via a notifier. Hence ZONE_MOVABLE offline 
> can be
> denied. Semantics wise we are still okay.
> 
> This might look bit inconsistent that 
> movablecore/kernelcore/movable_node with
> firmware sending in 'hot pluggable' memory (IIRC arm64 does not really 
> support
> this yet), the system might end up with ZONE_MOVABLE marked boot memory 
> which
> cannot be offlined or removed. But an offline notifier action is 
> orthogonal.
> Hence did not block those kernel command line paths that creates 
> ZONE_MOVABLE
> during boot to preserve existing behavior.
> 
>> 
>> I understand that some region of boot RAM shouldn’t be allowed to be 
>> removed, but such regions won’t be allowed to be offlined in first 
>> place since pages cannot be migrated and isolated, example reserved 
>> pages.
>> 
>> So we’re trying to understand the reasoning for such a prevention put 
>> in place for arm64 arch alone.
> 
> Primary reason being kexec. During kexec on arm64, next kernel's memory 
> map is
> derived from firmware and not from current running kernel. So the next 
> kernel
> will crash if it would access memory that might have been removed in 
> running
> kernel. Until kexec on arm64 changes substantially and takes into 
> account the
> real available memory on the current kernel, boot memory cannot be 
> removed.
> 
>> 
>> One possible way to solve this is by marking the required sections as 
>> “non-early” by removing the SECTION_IS_EARLY bit in its 
>> section_mem_map.
> 
> That is too intrusive from core memory perspective.
> 
>  This puts these sections in the context of “memory hotpluggable”
> which can be offlined-removed and added-onlined which are part of boot
> RAM itself and doesn’t need any extra blocks to be hot added. This way
> of marking certain sections as “non-early” could be exported so that
> module drivers can set the required number of sections as “memory
> hotpluggable”. This could have certain checks put in place to see
> which sections are allowed, example only movable zone sections can be
> marked as “non-early”.
> 
> Giving modules the right to mark memory hotpluggable ? That is too 
> intrusive
> and would still not solve the problem with kexec.
> 
>> 
>> Your thoughts on this? We are also looking for different ways to solve 
>> the problem without having to completely dropping this notifier, but 
>> just putting out the concern here about the notifier logic that is 
>> breaking our usecase which is a generic memory sharing usecase using 
>> memory hotplug feature.
> 
> Completely preventing boot memory offline and removal is essential for 
> kexec
> to work as expected afterwards. As suggested previously, splitting the 
> VM
> memory into boot and non boot chunks during init can help work around 
> this
> restriction effectively in userspace itself and would not require any 
> kernel
> changes.
> 
> - Anshuman


Sudarshan

--
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a 
Linux Foundation Collaborative Project