RE: FW: [LSF/MM/BPF TOPIC] BoF VM live migration over CXL memory

[Kyungsan Kim <ks0204.kim@samsung.com>]

>> Gregory Price <gregory.price@memverge.com> writes:
>>
>>> On Tue, Apr 11, 2023 at 02:37:50PM +0800, Huang, Ying wrote:
>>>> Gregory Price <gregory.price@memverge.com> writes:
>>>>
>>>> [snip]
>>>>
>>>>> 2. During the migration process, the memory needs to be forced not to be
>>>>>     migrated to another node by other means (tiering software, swap,
>>>>>     etc).  The obvious way of doing this would be to migrate and
>>>>>     temporarily pin the page... but going back to problem #1 we see that
>>>>>     ZONE_MOVABLE and Pinning are mutually exclusive.  So that's
>>>>>     troublesome.
>>>>
>>>> Can we use memory policy (cpusets, mbind(), set_mempolicy(), etc.) to
>>>> avoid move pages out of CXL.mem node?  Now, there are gaps in tiering,
>>>> but I think it is fixable.
>>>>
>>>> Best Regards,
>>>> Huang, Ying
>>>>
>>>> [snip]
>>>
>>> That feels like a hack/bodge rather than a proper solution to me.
>>>
>>> Maybe this is an affirmative argument for the creation of an EXMEM
>>> zone.
>>
>> Let's start with requirements.  What is the requirements for a new zone
>> type?
>
>I'm stills scratching my head regarding this. I keep hearing all
>different kind of statements that just add more confusions "we want it
>to be hotunpluggable" "we want to allow for long-term pinning memory"
>"but we still want it to be movable" "we want to place some unmovable
>allocations on it". Huh?
>
>Just to clarify: ZONE_MOVABLE allows for pinning. It just doesn't allow
>for long-term pinning of memory.
>
>For good reason, because long-term pinning of memory is just the worst
>(memory waste, fragmentation, overcommit) and instead of finding new
>ways to *avoid* long-term pinnings, we're coming up with advanced
>concepts to work-around the fundamental property of long-term pinnings.
>
>We want all memory to be long-term pinnable and we want all memory to be
>movable/hotunpluggable. That's not going to work.

Looks there is misunderstanding about ZONE_EXMEM argument.
Pinning and plubbability is mutual exclusive so it can not happen at the same time.
What we argue is ZONE_EXMEM does not "confine movability". an allocation context can determine the movability attribute.
Even one unmovable allocation will make the entire CXL DRAM unpluggable. 
When you see ZONE_EXMEM just on movable/unmoable aspect, we think it is the same with ZONE_NORMAL,
but ZONE_EXMEM works on an extended memory, as of now CXL DRAM.

Then why ZONE_EXMEM is, ZONE_EXMEM considers not only the pluggability aspect, but CXL identifier for user/kenelspace API, 
the abstraction of multiple CXL DRAM channels, and zone unit algorithm for CXL HW characteristics.
The last one is potential at the moment, though.

As mentioned in ZONE_EXMEM thread, we are preparing slides to explain experiences and proposals.
It it not final version now[1].
[1] https://github.com/OpenMPDK/SMDK/wiki/93.-%5BLSF-MM-BPF-TOPIC%5D-SMDK-inspired-MM-changes-for-CXL

>If you'd ask me today, my prediction is that ZONE_EXMEM is not going to
>happen.
>
>--
>Thanks,
>
>David / dhildenb