RE: Questions about CXL device (type 3 memory) hotplug

From: Dan Williams <dan.j.williams@intel.com>
To: "Yasunori Gotou (Fujitsu)" <y-goto@fujitsu.com>,
	"linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>
Cc: 'Dan Williams' <dan.j.williams@intel.com>
Subject: RE: Questions about CXL device (type 3 memory) hotplug
Date: Mon, 22 May 2023 17:11:39 -0700	[thread overview]
Message-ID: <646c04bbbd96_33fb32944b@dwillia2-xfh.jf.intel.com.notmuch> (raw)
In-Reply-To: <TYWPR01MB10082116669BA002263DF132A90439@TYWPR01MB10082.jpnprd01.prod.outlook.com>

Yasunori Gotou (Fujitsu) wrote:
> Hello,
> 
> I have some questions about CXL device hotplug (especially type 3 memory device).
> 
> Though my team members still need to work for a remain issue of Filesystem-DAX,
> or RDMA for persistent memory yet, I would like to move some of them for
> CXL type 3 memory device after finish the above works.
> This is preparation for it. I would like to confirm current status of CXL
> type 3 memory hotplug.
> 
> Q1) Can PCIe hotplug driver detect and call CXL driver?
> 
>     CXL specification says as follows
>     "9.9 Hot-Plug"
>     "CXL leverages PCIe Hot-plug model and Hot-plug elements as defined in PCIe
>     BaseSpecification and the applicable form-factor specifications."
> 
>     At a glance, PCIe hotplug driver seems to be able to detect any PCIe
>     hotpluged device, and call its suitable probe/attach function. But I'm new
>     around here, and  I'm not sure it can actually call the suitable CXL driver.
> 
>     Can PCIe hotplug driver recognize a CXL device and call its driver?

Yes.

The cxl_pci driver (drivers/cxl/pci.c) is just a typical PCI driver as
far as the PCI hotplug driver is concerned. So add/remove events of a
CXL card get turned into probe()/remove() events on the driver.

> 
> Q2) Can QEMU/KVM emulate CXL device hotplug?
> 
>    I heard that QEMU/KVM has PCIe device hotplug emulation, but I'm not sure
>    it can hotplug CXL device.

It can, but as far as the driver is concerned you can achieve the same
by:

echo $devname > /sys/bus/pci/drivers/cxl_pci/unbind

...that exercises the same software flows as physical unplug.

> 
> Q3) After called CXL driver and detected a CXL type 3 memory device,
>     what sequence is/will be executed?
> 
>     IIRC, kernel created /sys/devices/system/memory/memoryNNN directories when
>     memory device is recognized. Then, online operation was executed by 
>     a user or an application.
>     
>     However, CXL specification seems to require more configuration like
>     interleave, region, and namespace by Fabric Manager (cxl command?)
>     after device detection before memory online.
> 
>     So, my understanding is that the above configuration must be executed
>     after device detection, and before memory online. Is it correct?

Correct, after the device is added and the driver attaches there is
still a step needed to configure a CXL region.

For now that step is to manually run:

cxl create-region

...later we might consider some udev rules to automatically assemble
regions from discovered capacity.

> Q4) Current CXL drivers/tools support Hot-removal request from PCIe?
> 
>     CXL specification says "In a managed Hot-Remove flow, software is
>     notified of a hot removal request."

Currently there is a requirement that:

cxl disable-memdev

...is run before the device can be removed. There is no warning from the
PCI hotplug driver. Which means that if end user does the wrong sequence
they can crash the kernel / remove memory that may still be in active
use.

>     I think that CXL drivers/tools need to find which sections belongs to the
>     requested device, and execute offline them at least. In addition,
>     Fabric Manager may need to prepare removing the device due to configuration
>     change.
> 
>     Does current CXL drivers/tools can execute them?
>     Otherwise, does it need to be implemented yet?

Currently the 'cxl disable-memdev' command is not smart about
determining when the device is in active use it just claims that it is
always in use. That is in progress to be improved.

> Q5) How CXL driver treat region/namespace size against section size?
>     Current x86-64 section size can be 2Gbyte, but CXL region size may be
>     able to smaller than it. 

The section size is still 128MB, the hotplug memory block size is what
expands to 2GB. That size limits what can be onlined via the dax_kmem
driver.