Re: [LSF/MM/BPF TOPIC] BoF VM live migration over CXL memory​

[James Bottomley <James.Bottomley@HansenPartnership.com>]

On Wed, 2023-04-12 at 10:38 +0200, David Hildenbrand wrote:
> On 12.04.23 04:54, Huang, Ying wrote:
> > Gregory Price <gregory.price@memverge.com> writes:
[...]
> > > That feels like a hack/bodge rather than a proper solution to me.
> > > 
> > > Maybe this is an affirmative argument for the creation of an
> > > EXMEM zone.
> > 
> > Let's start with requirements.  What is the requirements for a new
> > zone type?
> 
> I'm stills scratching my head regarding this. I keep hearing all 
> different kind of statements that just add more confusions "we want
> it to be hotunpluggable" "we want to allow for long-term pinning
> memory" "but we still want it to be movable" "we want to place some
> unmovable allocations on it". Huh?

This is the essential question about CXL memory itself: what would its
killer app be?  The CXL people (or at least the ones I've talked to)
don't exactly know.  Within IBM I've seen lots of ideas but no actual
concrete applications.  Given the rates at which memory density in
systems is increasing, I'm a bit dubious of the extensible system pool
argument.   Providing extensible memory to VMs sounds a bit more
plausible, particularly as it solves a big part of the local overcommit
problem (although you still have a global one).  I'm not really sure I
buy the VM migration use case: iterative transfer works fine with small
down times so transferring memory seems to be the least of problems
with the VM migration use case (it's mostly about problems with
attached devices).  CXL 3.0 is adding sharing primitives for memory so
now we have to ask if there are any multi-node shared memory use cases
for this, but most of us have already been burned by multi-node shared
clusters once in our career and are a bit leery of a second go around.

Is there a use case I left out (or needs expanding)?

James