RE: Questions about CXL device (type 3 memory) hotplug

From: "Yasunori Gotou (Fujitsu)" <y-goto@fujitsu.com>
To: 'Dan Williams' <dan.j.williams@intel.com>,
	"linux-cxl@vger.kernel.org" <linux-cxl@vger.kernel.org>
Subject: RE: Questions about CXL device (type 3 memory) hotplug
Date: Tue, 23 May 2023 08:31:27 +0000	[thread overview]
Message-ID: <TYWPR01MB10082C8CB891827C6438D311D90409@TYWPR01MB10082.jpnprd01.prod.outlook.com> (raw)
In-Reply-To: <646c04bbbd96_33fb32944b@dwillia2-xfh.jf.intel.com.notmuch>

Thank you for your answer!
Its progress seems to be better than I thought.

I would like to ask more questions.

> Yasunori Gotou (Fujitsu) wrote:
> > Hello,
> >
> > I have some questions about CXL device hotplug (especially type 3 memory
> device).
> >
> > Though my team members still need to work for a remain issue of
> > Filesystem-DAX, or RDMA for persistent memory yet, I would like to
> > move some of them for CXL type 3 memory device after finish the above
> works.
> > This is preparation for it. I would like to confirm current status of
> > CXL type 3 memory hotplug.
> >
> > Q1) Can PCIe hotplug driver detect and call CXL driver?
> >
> >     CXL specification says as follows
> >     "9.9 Hot-Plug"
> >     "CXL leverages PCIe Hot-plug model and Hot-plug elements as defined
> in PCIe
> >     BaseSpecification and the applicable form-factor specifications."
> >
> >     At a glance, PCIe hotplug driver seems to be able to detect any PCIe
> >     hotpluged device, and call its suitable probe/attach function. But I'm
> new
> >     around here, and  I'm not sure it can actually call the suitable CXL
> driver.
> >
> >     Can PCIe hotplug driver recognize a CXL device and call its driver?
> 
> Yes.
> 
> The cxl_pci driver (drivers/cxl/pci.c) is just a typical PCI driver as far as the PCI
> hotplug driver is concerned. So add/remove events of a CXL card get turned
> into probe()/remove() events on the driver.

Sounds good!

> >
> > Q2) Can QEMU/KVM emulate CXL device hotplug?
> >
> >    I heard that QEMU/KVM has PCIe device hotplug emulation, but I'm not
> sure
> >    it can hotplug CXL device.
> 
> It can,

Ok, then, are there any test-set for CXL device hotplug with QEMU/KVM emulation?
It will be more helpful when actual CXL memory device will be released.

> but as far as the driver is concerned you can achieve the same
> by:
> 
> echo $devname > /sys/bus/pci/drivers/cxl_pci/unbind
> 
> ...that exercises the same software flows as physical unplug.

Ok. I see.

> >
> > Q3) After called CXL driver and detected a CXL type 3 memory device,
> >     what sequence is/will be executed?
> >
> >     IIRC, kernel created /sys/devices/system/memory/memoryNNN
> directories when
> >     memory device is recognized. Then, online operation was executed by
> >     a user or an application.
> >
> >     However, CXL specification seems to require more configuration like
> >     interleave, region, and namespace by Fabric Manager (cxl command?)
> >     after device detection before memory online.
> >
> >     So, my understanding is that the above configuration must be executed
> >     after device detection, and before memory online. Is it correct?
> 
> Correct, after the device is added and the driver attaches there is still a step
> needed to configure a CXL region.
> 
> For now that step is to manually run:
> 
> cxl create-region
> 
> ...later we might consider some udev rules to automatically assemble regions
> from discovered capacity.

Hmm, I suppose 2 types of udev rules may be necessary.
The first one is for notify new CXL device is detected, and cxl-command assemble 
a region automatically.
The second one is for notify region is configured, online is execute for each
memory block on the region by the notification, and rollback when one of the block
fails hotadd If necessary.

> 
> > Q4) Current CXL drivers/tools support Hot-removal request from PCIe?
> >
> >     CXL specification says "In a managed Hot-Remove flow, software is
> >     notified of a hot removal request."
> 
> Currently there is a requirement that:
> 
> cxl disable-memdev
> 
> ...is run before the device can be removed. There is no warning from the PCI
> hotplug driver. Which means that if end user does the wrong sequence they
> can crash the kernel / remove memory that may still be in active use.

Ok.
Though "Surprising remove" is not guaranteed by specification, I think
"managed hot-removed flow" should be realized.
I'll chase more what should we do about it.

> 
> >     I think that CXL drivers/tools need to find which sections belongs to the
> >     requested device, and execute offline them at least. In addition,
> >     Fabric Manager may need to prepare removing the device due to
> configuration
> >     change.
> >
> >     Does current CXL drivers/tools can execute them?
> >     Otherwise, does it need to be implemented yet?
> 
> Currently the 'cxl disable-memdev' command is not smart about determining
> when the device is in active use it just claims that it is always in use. That is in
> progress to be improved.

Ok. I see.

> 
> > Q5) How CXL driver treat region/namespace size against section size?
> >     Current x86-64 section size can be 2Gbyte, but CXL region size may be
> >     able to smaller than it.
> 
> The section size is still 128MB, the hotplug memory block size is what expands
> to 2GB. That size limits what can be onlined via the dax_kmem driver.

Oops. 
OK, I understand I should change my word "section" to "hotplug memory block".

One of the background of this question is "rollback".
"If memory hotadd or hotremove for a memory block fails, is rollback available?".

If a block hotadd sequence fails in the device for some reasons, its user wants to remove
the device for the moment, and may want to retry hotadd again or try other device.
To achieve it, already onlined blocks before failed block should be offlined again.

If a block hotremove sequence fails in the device, its user would like to keep the device
online to postpone replacing it or select other device for device pooling. (vice vesa).
I don't find which component handle this situation.

I noticed that current users prefer online after device detection immediately, and kernel
supports it. Though it is natural for some use-case, I feel it may be obstacle for rollback of
CXL device hotplug failure.

Thanks,