RE: Questions about CXL device (type 3 memory) hotplug

From: Dan Williams <dan.j.williams@intel.com>
To: "Yasunori Gotou (Fujitsu)" <y-goto@fujitsu.com>,
	'Dan Williams' <dan.j.williams@intel.com>,
	"linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>
Subject: RE: Questions about CXL device (type 3 memory) hotplug
Date: Thu, 8 Jun 2023 11:37:07 -0700	[thread overview]
Message-ID: <64821fd31d563_1433ac294a3@dwillia2-xfh.jf.intel.com.notmuch> (raw)
In-Reply-To: <TYWPR01MB10082CE5542BCFCEF127CDD399050A@TYWPR01MB10082.jpnprd01.prod.outlook.com>

Yasunori Gotou (Fujitsu) wrote:
> 
> > > Ah, sorry... My description of question was not good.
> > > I understand that PCIe hotremove is not suitable for trigger of CXL memory.
> > >
> > > What I would like to ask is "Are there any agent or daemon which gets
> > > a hotremove request from outside of server and executes offline and
> > > cxl disable region without users operation?"
> > > I suppose such memory pool manager (or others) would like to ask the
> > > agent to execute such operation.
> > > (Probably, the agent need to get the request by REST API.)
> > 
> > No, there's no coordination between the kernel and userspace when the
> > attention button is pressed. So any coordinated removal must be handled
> > before the removal is attempted. I think it would be useful to have a mode of
> > operation where pressing the attention button just notifies userspace and it
> > handles the coordinated shutdown of the device.
> > 
> > If the question is having a management API to trigger removal I am not aware of
> > any work in this space.
> 
> Hmmmm, my question is NOT about attention button now. 
> (I noticed that my quote of previous mail may be not good. Sorry).
> I would like to confirm what component/method is still necessary for memory pool.
> 
> I imagine that there is(will be) a total memory pool manager which manages
> a lot of Linux systems on each servers which have CXL memory.
> And I think that an agent/daemon will be necessary in each Linux system to hotplug operation
> like offline/online memory blocks, and/or requesting configuration of Dynamic Capacity Device 
> to a Fabric Manager.
> I guess such agent/daemon is not created yet for memory pool feature.
> 
> Here is my current understanding of an example of steps for memory pool. 
> This does not include attention button, and use DCD.
> 
> 1) The pool manager will request hot remove to a daemon in a Linux system by REST API, ssl, or somekind of
>    network interface.
> 2) Then the daemon will execute memory offline some memory blocks.
> 3) It will correct offlined memory blocks until requested amount, and request 
>   configuration of DCD to Fabric Manager.
> 4) Fabric Manager will configure Dynamic Capacity Device.
> 5) The memory pool manager will detect finish of its configuration somehow, and request hot add to a daemon in
>   another Linux system.
> 6) The daemon will request configuration of DCD to FM
> 7) The daemon will detect new area somehow and online blocks for them.
> 
> If I still misunderstand something, please let me know.

Yes, I agree that a daemon like that is needed, I am not aware of any
current work in this space.

However, I wonder if there is existing virtio-mem-like management
infrastructure that could be repurposed for coordinating host-mem
dynamic memory adjustment.  In other words coordinating a shared pool of
memory across multiple kernel instances is a problem that has been
solved before, CXL just makes it so that the coordination is across
physical hosts rather than virtual.