Re: [LSF/MM/BPF TOPIC] BoF VM live migration over CXL memory

From: Dave Hansen <dave.hansen@intel.com>
To: Dragan Stancevic <dragan@stancevic.com>,
	lsf-pc@lists.linux-foundation.org
Cc: nil-migration@lists.linux.dev, linux-cxl@vger.kernel.org,
	linux-mm@kvack.org
Subject: Re: [LSF/MM/BPF TOPIC] BoF VM live migration over CXL memory
Date: Tue, 11 Apr 2023 11:00:02 -0700	[thread overview]
Message-ID: <14a601ea-8cf8-bb9c-a87a-63567c5aba5b@intel.com> (raw)
In-Reply-To: <5d1156eb-02ae-a6cc-54bb-db3df3ca0597@stancevic.com>

On 4/7/23 14:05, Dragan Stancevic wrote:
> I'd be interested in doing a small BoF session with some slides and get
> into a discussion/brainstorming with other people that deal with VM/LM
> cloud loads. Among other things to discuss would be page migrations over
> switched CXL memory, shared in-memory ABI to allow VM hand-off between
> hypervisors, etc...

How would 'struct page' or other kernel metadata be handled?

I assume you'd want a really big CXL memory device with as many hosts
connected to it as is feasible.  But, in order to hand the memory off
from one host to another, both would need to have metadata for it at
_some_ point.

So, do all hosts have metadata for the whole CXL memory device all the
time?  Or, would they create the metadata (hotplug) when a VM is
migrated in and destroy it (hot unplug) when a VM is migrated out?

That gets back to the granularity question discussed elsewhere in the
thread.  How would the metadata allocation granularity interact with the
page allocation granularity?  How would fragmentation be avoided so that
hosts don't eat up all their RAM with unused metadata?