From: Yuquan Wang <wangyuquan1236@phytium.com.cn>
To: dan.j.williams@intel.com, jonathan.cameron@huawei.com,
y-goto@fujitsu.com
Cc: linux-cxl@vger.kernel.org, linux-kernel@vger.kernel.org,
qemu-devel@nongnu.org
Subject: RE: Questions about CXL device (type 3 memory) hotplug
Date: Wed, 27 Mar 2024 15:18:12 +0800 [thread overview]
Message-ID: <20240327071812.955794-1-wangyuquan1236@phytium.com.cn> (raw)
In-Reply-To: <646c04bbbd96_33fb32944b@dwillia2-xfh.jf.intel.com.notmuch>
On Mon, May 22, 2023 at 05:11:39PM -0700, Dan Williams wrote:
> Yasunori Gotou (Fujitsu) wrote:
[...]
Hi, all
There was some confusions about CXL device hotplug when I recently
tried to use Qemu to emulate CXL device hotplug and verify the relevant
functions of kernel.
> > Q1) Can PCIe hotplug driver detect and call CXL driver?
[...]
>
> Yes.
>
> The cxl_pci driver (drivers/cxl/pci.c) is just a typical PCI driver as
> far as the PCI hotplug driver is concerned. So add/remove events of a
> CXL card get turned into probe()/remove() events on the driver.
>
1. Can we divide steps of CXL device hotplug into two parts(PCI hotplug & Memory Hotplug)?
PCI Hotplug: the same as the native PCIe hotplug, including initializing cxl.io,
assigning PCIe BARs, allocating interrupts, etc. And the cxl_pci driver
is responsible for this part.
Memory Hotplug: focusing on enabling CXL memory including discovering and Configuring HDM,
extracting NUMA info from device, notifying memory management, etc.
> >
> > Q2) Can QEMU/KVM emulate CXL device hotplug?
> >
> > I heard that QEMU/KVM has PCIe device hotplug emulation, but I'm not sure
> > it can hotplug CXL device.
>
> It can, but as far as the driver is concerned you can achieve the same
> by:
>
> echo $devname > /sys/bus/pci/drivers/cxl_pci/unbind
>
> ...that exercises the same software flows as physical unplug.
>
2. What is the difference between "echo $devname > /sys/bus/pci/drivers/cxl_pci/unbind" and
"(qemu) device_del cxl-mem0" ?
According to the test, I found that "(qemu) device_del cxl-mem0" would directly
unplug the device and cause the interrupts on the cxl root port. It seems like this
operation would not only trigger cxl_pci driver but also pcieport driver.
The kernel dmesg is like below:
(qemu) device_del cxl-mem0
# dmesg
[ 699.057907] pcieport 0000:0c:00.0: pciehp: pending interrupts 0x0001 from Slot Status
[ 699.058929] pcieport 0000:0c:00.0: pciehp: Slot(0): Button press: will power off in 5 sec
[ 699.059986] pcieport 0000:0c:00.0: pciehp: pending interrupts 0x0010 from Slot Status
[ 699.060099] pcieport 0000:0c:00.0: pciehp: pciehp_set_indicators: SLOTCTRL 90 write cmd 2c0
Then I also tried "echo $devname > /sys/bus/pci/drivers/cxl_pci/unbind"
to check the behaviour of kernel. The kernel dmesg is like below:
# echo 0000:0d:00.0 > /sys/bus/pci/drivers/cxl_pci/unbind
# dmesg
[70387.978931] cxl_pci 0000:0d:00.0: vgaarb: pci_notify
[70388.021476] cxl_mem mem0: disconnect mem0 from port1
[70388.033099] pci 0000:0d:00.0: vgaarb: pci_notify
It seems like this operation would just unbind the cxl_pci driver from the cxl device.
Is my understanding about these two method correct?
3) Can I just use "ndctl/test/cxl-topology.sh" to test the cxl hotplug functions of kernel?
IIUC, cxl-topology.sh would utilize cxl_test (tools/testing/cxl) which is for regression
testing the kernel-user ABI.
PS: My qemu command line:
qemu-system-x86_64 \
-M q35,nvdimm=on,cxl=on \
-m 4G \
-smp 4 \
-object memory-backend-ram,size=2G,id=mem0 \
-numa node,nodeid=0,cpus=0-1,memdev=mem0 \
-object memory-backend-ram,size=2G,id=mem1 \
-numa node,nodeid=1,cpus=2-3,memdev=mem1 \
-object memory-backend-ram,size=256M,id=cxl-mem0 \
-device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
-device cxl-rp,port=0,bus=cxl.1,id=root_port0,chassis=0,slot=0 \
-device cxl-type3,bus=root_port0,volatile-memdev=cxl-mem0,id=cxl-mem0 \
-M cxl-fmw.0.targets.0=cxl.1,cxl-fmw.0.size=4G,cxl-fmw.0.interleave-granularity=4k \
-hda ../disk/ubuntu_x86_test_new.qcow2 \
-nographic \
Qemu version: 8.2.50, the lastest commit of branch cxl-2024-03-05 in "https://gitlab.com/jic23/qemu"
Kernel version: 6.8.0-rc6
Many thanks
Yuquan
prev parent reply other threads:[~2024-03-27 7:18 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-22 8:06 Questions about CXL device (type 3 memory) hotplug Yasunori Gotou (Fujitsu)
2023-05-23 0:11 ` Dan Williams
2023-05-23 8:31 ` Yasunori Gotou (Fujitsu)
2023-05-23 17:36 ` Dan Williams
2023-05-24 11:12 ` Yasunori Gotou (Fujitsu)
2023-05-24 20:51 ` Dan Williams
2023-05-25 10:32 ` Yasunori Gotou (Fujitsu)
2023-05-26 8:05 ` Yasunori Gotou (Fujitsu)
2023-05-26 14:48 ` Dan Williams
2023-05-29 8:07 ` Yasunori Gotou (Fujitsu)
2023-06-06 17:58 ` Dan Williams
2023-06-08 7:39 ` Yasunori Gotou (Fujitsu)
2023-06-08 18:37 ` Dan Williams
2023-06-09 1:02 ` Yasunori Gotou (Fujitsu)
2023-05-23 13:34 ` Vikram Sethi
2023-05-23 18:40 ` Dan Williams
2023-05-24 0:02 ` Vikram Sethi
2023-05-24 4:03 ` Dan Williams
2023-05-24 14:47 ` Vikram Sethi
2023-05-24 21:20 ` Dan Williams
2023-05-31 4:25 ` Vikram Sethi
2023-06-06 20:54 ` Dan Williams
2023-06-07 1:06 ` Vikram Sethi
2023-06-07 15:12 ` Jonathan Cameron
2023-06-07 18:44 ` Vikram Sethi
2023-06-08 15:19 ` Jonathan Cameron
2023-06-08 18:41 ` Dan Williams
2024-03-27 7:10 ` Yuquan Wang
2024-03-27 7:18 ` Yuquan Wang [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240327071812.955794-1-wangyuquan1236@phytium.com.cn \
--to=wangyuquan1236@phytium.com.cn \
--cc=dan.j.williams@intel.com \
--cc=jonathan.cameron@huawei.com \
--cc=linux-cxl@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=qemu-devel@nongnu.org \
--cc=y-goto@fujitsu.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.