* Re: [RFC PATCH 00/25] Introduce CXL 2.0 Emulation
@ 2020-12-04 5:08 Chris Browy
2020-12-04 5:55 ` Dan Williams
0 siblings, 1 reply; 9+ messages in thread
From: Chris Browy @ 2020-12-04 5:08 UTC (permalink / raw)
To: ben.widawsky; +Cc: vishal.l.verma, dan.j.williams, qemu-devel
[-- Attachment #1: Type: text/plain, Size: 5474 bytes --]
Hi Ben,
Trying to bring up the environment using the latest developments as follows:
1. Linux kernel baseline version is cloned using
git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
Using master branch. Merged the 9 CXL linux kernel patches manually and built kernel
2. QEMU baseline version is cloned using
git clone https://gitlab.com/bwidawsk/qemu.git <https://gitlab.com/bwidawsk/qemu.git>
3. UEFI baseline is cloned using
git clone https://github.com/tianocore/edk2.git <https://github.com/tianocore/edk2.git>
Using master and built
4. Now can run qemu as follows:
The qcow2 we use is based on Ubuntu 20.10 with updated with kernel from 1) above
QEMU command:
sudo qemu-system-x86_64 -nic \
user,hostfwd=tcp::2222-:22,hostfwd=tcp::1234-:1234 -machine \
type=pc-q35-4.0,hmat=on,accel=kvm -enable-kvm -cpu host -smp \
6,cores=6,threads=1,sockets=1 -m 8G -boot order=d -k 'en-us' -vga virtio \
-drive file=/home/chris/Downloads/AQCXL/ubuntu_20.qcow,format=qcow2 -drive \
if=pflash,format=raw,readonly,file=/home/chris/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/home/chris/OVMF_VARS.fd \
-object memory-backend-file,id=cxl-mem1,share,mem-path=/tmp/cxl-test/cxl,size=512M \
-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52,uid=0,len-window-base=1,\
window-base[0]=0x4c0000000,memdev[0]=cxl-mem1 \
-device cxl-rp,id=rp0,bus=cxl.0,addr=0.0,chassis=0,slot=0 \
-device cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0,size=256M 2>&1 | tee -a \
/home/chris/Downloads/AQCXL/log/qemu.log
The qemu options are derived from looking at the tests/qtests/cxl-test.c
along with the -hmat=on which seemed to make sense.
The system boots and lspci -vvv shows the CXL device is enumerated. But
no DOE capability register for CDAT access though (see below). Otherwise the
DVSEC registers are present.
acpidump indicates the CXL0 and CXLM devices but no SRAT or HMAT tables are
in the dump which is curious.
35:00.0 Memory controller [0502]: Intel Corporation Device 0d93 (rev 01) (prog-if 10)
Subsystem: Red Hat, Inc. Device 1100
Physical Slot: 0
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
Region 0: Memory at c0a00000 (64-bit, non-prefetchable) [size=64K]
Region 2: Memory at c0a10000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s, Exit Latency L0s <64ns
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (ok), Width x1 (ok)
TrErr- Train- SlotClk- DLActive+ BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, NROPrPrP-, LTR-
10BitTagComp-, 10BitTagReq-, OBFF Not Supported, ExtFmt+, EETLPPrefix+, MaxEETLPPrefixes 4
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS-, TPHComp-, ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
Capabilities: [100 v1] Designated Vendor-Specific <?>
Capabilities: [138 v1] Designated Vendor-Specific <?>
Kernel driver in use: cxl_mem
Questions/Comments:
-------------------
1. Linux
a. Is there a gitlab for the linux kernel patches for CXL? This would
facilitate review and code modifications.
2. UEFI (edk2 from tianocore)
a. seems to only support CXL 1.1 which means only method #1 (Device
option ROM) of Coherent Device Attribute Table_1.02 spec
for CDAT handling is possible now.
Does device option ROM need to be added to QEMU CXL setup?
Can we add a CXL 1.1 emulated device?
b. lspci doesn’t show the existence of the DOE extended capability register
in the CXL CT3D (needed to support method #2). Are there more patches?
3. Do you have example user programs to share or better yet the CXL 2.0
Sec 14.3.6.1 Application Layer/ Transaction layer test for CXL.mem?
4. What are the userspace system APIs for targeting CXL HDM address domain?
Usually you can mmap a SPA if you know how to look it up.
Best Regards,
Chris Browy
[-- Attachment #2: Type: text/html, Size: 14535 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 00/25] Introduce CXL 2.0 Emulation
2020-12-04 5:08 [RFC PATCH 00/25] Introduce CXL 2.0 Emulation Chris Browy
@ 2020-12-04 5:55 ` Dan Williams
0 siblings, 0 replies; 9+ messages in thread
From: Dan Williams @ 2020-12-04 5:55 UTC (permalink / raw)
To: Chris Browy; +Cc: Ben Widawsky, Qemu Developers, Vishal L Verma, linux-cxl
[ add linux-cxl for the Linux driver question ]
On Thu, Dec 3, 2020 at 9:10 PM Chris Browy <cbrowy@avery-design.com> wrote:
[..]
I'll let Ben address the other questions...
> 4. What are the userspace system APIs for targeting CXL HDM address domain?
> Usually you can mmap a SPA if you know how to look it up.
tl;dr: /proc/iomem and sysfs
There are a lot of moving pieces to this question. Once a CXL.mem
device is mapped the driver will arrange for it to appear in
/proc/iomem. If it is persistent memory it will be accessed through a
/dev/pmem or /dev/dax device, and if it is volatile it will show up as
just another "System RAM" range. The System RAM range may end up with
a unique Linux NUMA node id. Alternatively, if the BIOS maps the
device then the BIOS will set up the EFI memory map + ACPI
SRAT/SLIT/HMAT for Linux to interpret.
Once something shows up in /proc/iomem it was traditionally accessible
via /dev/mem. These days though /dev/mem is slowly being deprecated
due to security (see CONFIG_LOCK_DOWN_KERNEL*) and stability (see
CONFIG_IO_STRICT_DEVMEM) concerns, so depending on your kernel
configuration tools may not be able to use /dev/mem to map the SPA
directly and will need to go through the driver provided interface.
The driver needs to be able coordinate coherent HDM settings across
multiple devices in an interleave, all switches in the path, and the
root bridge(s). So I am expecting a sysfs interface similar to the
nvdimm-subsystem and device-dax-subsystem where there will be
sysfs-files to enumerate the mappings and a provisioning mechanism to
reconfigure HDMs with various interleave settings. For a flavor of
what this might look like see the description of NVDIMM "Region"
devices:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/driver-api/nvdimm/nvdimm.rst#n469
The difference will be that unlike the nvdimm case where regions are
platform-firmware-defined and static, CXL regions are runtime dynamic.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 00/25] Introduce CXL 2.0 Emulation
@ 2020-12-04 5:55 ` Dan Williams
0 siblings, 0 replies; 9+ messages in thread
From: Dan Williams @ 2020-12-04 5:55 UTC (permalink / raw)
To: Chris Browy; +Cc: Vishal L Verma, Ben Widawsky, Qemu Developers, linux-cxl
[ add linux-cxl for the Linux driver question ]
On Thu, Dec 3, 2020 at 9:10 PM Chris Browy <cbrowy@avery-design.com> wrote:
[..]
I'll let Ben address the other questions...
> 4. What are the userspace system APIs for targeting CXL HDM address domain?
> Usually you can mmap a SPA if you know how to look it up.
tl;dr: /proc/iomem and sysfs
There are a lot of moving pieces to this question. Once a CXL.mem
device is mapped the driver will arrange for it to appear in
/proc/iomem. If it is persistent memory it will be accessed through a
/dev/pmem or /dev/dax device, and if it is volatile it will show up as
just another "System RAM" range. The System RAM range may end up with
a unique Linux NUMA node id. Alternatively, if the BIOS maps the
device then the BIOS will set up the EFI memory map + ACPI
SRAT/SLIT/HMAT for Linux to interpret.
Once something shows up in /proc/iomem it was traditionally accessible
via /dev/mem. These days though /dev/mem is slowly being deprecated
due to security (see CONFIG_LOCK_DOWN_KERNEL*) and stability (see
CONFIG_IO_STRICT_DEVMEM) concerns, so depending on your kernel
configuration tools may not be able to use /dev/mem to map the SPA
directly and will need to go through the driver provided interface.
The driver needs to be able coordinate coherent HDM settings across
multiple devices in an interleave, all switches in the path, and the
root bridge(s). So I am expecting a sysfs interface similar to the
nvdimm-subsystem and device-dax-subsystem where there will be
sysfs-files to enumerate the mappings and a provisioning mechanism to
reconfigure HDMs with various interleave settings. For a flavor of
what this might look like see the description of NVDIMM "Region"
devices:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/driver-api/nvdimm/nvdimm.rst#n469
The difference will be that unlike the nvdimm case where regions are
platform-firmware-defined and static, CXL regions are runtime dynamic.
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 00/25] Introduce CXL 2.0 Emulation
2020-11-11 5:46 Ben Widawsky
2020-11-16 17:21 ` Jonathan Cameron
@ 2020-12-04 14:27 ` Daniel P. Berrangé
1 sibling, 0 replies; 9+ messages in thread
From: Daniel P. Berrangé @ 2020-12-04 14:27 UTC (permalink / raw)
To: Ben Widawsky, Michael S. Tsirkin, Marcel Apfelbaum
Cc: Vishal Verma, Dan Williams, qemu-devel
Just copying in the two primary QEMU maintainers for the PCI subsystem
to bring it to their attention.
On Tue, Nov 10, 2020 at 09:46:59PM -0800, Ben Widawsky wrote:
> Introduce emulation of Compute Express Link 2.0, which was released
> today at https://www.computeexpresslink.org/.
>
> I've pushed a branch here: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0
>
> The emulation has been critical to get the Linux enabling started
> (https://lore.kernel.org/linux-cxl/), it would be an ideal place to land
> regression tests for different topology handling, and there may be applications
> for this emulation as a way for a guest to manipulate its address space relative
> to different performance memories. I am new to QEMU development, so please
> forgive and point me in the right direction if I severely misinterpreted where a
> piece of infrastructure belongs.
>
> Three of the five CXL component types are emulated with some level of functionality:
> host bridge, root port, and memory device. Upstream ports and downstream ports
> aren't implemented (the two components needed to make up a switch).
>
> CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the
> implementation utilizes existing PCI paradigms. To implement the host bridge,
> I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural
> fit even though it doesn't directly map to how hardware will work. For
> persistent capacity of the memory device, I utilized the memory subsystem
> (hw/mem).
>
> We have 3 reasons why this work is valuable:
> 1. OS driver development and testing
> 2. OS driver regression testing
> 3. Possible guest support for HDMs
>
> As mentioned above there are three benefits to carrying this enabling in
> upstream QEMU:
>
> 1. Linux driver feature development benefits from emulation both due to
> a lack of initial hardware availability, but also, as is seen with
> NVDIMM/PMEM emulation, there is value in being able to share
> topologies with system-software developers even after hardware is
> available.
>
> 2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake
> resources via custom modules (nfit_test). In retrospect a QEMU emulation of
> nfit_test capabilities would have made the test environment more portable, and
> allowed for easier community contributions of example configurations.
>
> 3. This is still being fleshed out, but in short it provides a standardized
> mechanism for the guest to provide feedback to the host about size and placement
> needs of the memory. After the host gives the guest a physical window mapping to
> the CXL device, the emulated HDM decoders allow the guest a way to tell the host
> how much it wants and where. There are likely simpler ways to do this, but
> they'd require inventing a new interface and you'd need to have diverging driver
> code in the guest programming of the HDM decoder vs. the host. Since we've
> already done this work, why not use it?
>
> There is quite a long list of work to do for full spec compliance, but I don't
> believe that any of it precludes merging. Off the top of my head:
> - Main host bridge support (WIP)
> - Interleaving
> - Better Tests
> - Huge swaths of firmware functionality
> - Hot plug support
> - Emulating volatile capacity
>
> The flow of the patches in general is to define all the data structures and
> registers associated with the various components in a top down manner. Host
> bridge, component, ports, devices. Then, the actual implementation is done in
> the same order.
>
> The summary is:
> 1-8: Put infrastructure in place for emulation of the components.
> 9-11: Create the concept of a CXL bus and plumb into PXB
> 12-16: Implement host bridges
> 17: Implement a root port
> 18: Implement a memory device
> 19: Implement HDM decoders
> 20-24: ACPI bits
> 25: Start working on enabling the main host bridge
>
> Ben Widawsky (23):
> hw/pci/cxl: Add a CXL component type (interface)
> hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
> hw/cxl/device: Introduce a CXL device (8.2.8)
> hw/cxl/device: Implement the CAP array (8.2.8.1-2)
> hw/cxl/device: Add device status (8.2.8.3)
> hw/cxl/device: Implement basic mailbox (8.2.8.4)
> hw/cxl/device: Add memory devices (8.2.8.5)
> hw/pxb: Use a type for realizing expanders
> hw/pci/cxl: Create a CXL bus type
> hw/pxb: Allow creation of a CXL PXB (host bridge)
> acpi/pci: Consolidate host bridge setup
> hw/pci: Plumb _UID through host bridges
> hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
> acpi/pxb/cxl: Reserve host bridge MMIO
> hw/pxb/cxl: Add "windows" for host bridges
> hw/cxl/rp: Add a root port
> hw/cxl/device: Add a memory device (8.2.8.5)
> hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
> acpi/cxl: Add _OSC implementation (9.14.2)
> acpi/cxl: Create the CEDT (9.14.1)
> Temp: acpi/cxl: Add ACPI0017 (CEDT awareness)
> WIP: i386/cxl: Initialize a host bridge
> qtest/cxl: Add very basic sanity tests
>
> Jonathan Cameron (1):
> Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.
>
> Vishal Verma (1):
> acpi/cxl: Introduce a compat-driver UUID for CXL _OSC
>
> MAINTAINERS | 6 +
> hw/Kconfig | 1 +
> hw/acpi/Kconfig | 5 +
> hw/acpi/cxl.c | 198 +++++++++++++
> hw/acpi/meson.build | 1 +
> hw/arm/virt.c | 1 +
> hw/core/machine.c | 26 ++
> hw/core/numa.c | 3 +
> hw/cxl/Kconfig | 3 +
> hw/cxl/cxl-component-utils.c | 192 +++++++++++++
> hw/cxl/cxl-device-utils.c | 293 +++++++++++++++++++
> hw/cxl/cxl-mailbox-utils.c | 139 +++++++++
> hw/cxl/meson.build | 5 +
> hw/i386/acpi-build.c | 87 +++++-
> hw/i386/microvm.c | 1 +
> hw/i386/pc.c | 2 +
> hw/mem/Kconfig | 5 +
> hw/mem/cxl_type3.c | 334 ++++++++++++++++++++++
> hw/mem/meson.build | 1 +
> hw/meson.build | 1 +
> hw/pci-bridge/Kconfig | 5 +
> hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++
> hw/pci-bridge/meson.build | 1 +
> hw/pci-bridge/pci_expander_bridge.c | 209 +++++++++++++-
> hw/pci-bridge/pcie_root_port.c | 6 +-
> hw/pci/pci.c | 32 ++-
> hw/pci/pcie.c | 30 ++
> hw/ppc/spapr.c | 2 +
> include/hw/acpi/cxl.h | 27 ++
> include/hw/boards.h | 2 +
> include/hw/cxl/cxl.h | 30 ++
> include/hw/cxl/cxl_component.h | 181 ++++++++++++
> include/hw/cxl/cxl_device.h | 199 +++++++++++++
> include/hw/cxl/cxl_pci.h | 155 ++++++++++
> include/hw/pci/pci.h | 15 +
> include/hw/pci/pci_bridge.h | 25 ++
> include/hw/pci/pci_bus.h | 8 +
> include/hw/pci/pci_ids.h | 1 +
> include/standard-headers/linux/pci_regs.h | 1 +
> monitor/hmp-cmds.c | 15 +
> qapi/machine.json | 1 +
> tests/qtest/cxl-test.c | 93 ++++++
> tests/qtest/meson.build | 4 +
> 43 files changed, 2547 insertions(+), 30 deletions(-)
> create mode 100644 hw/acpi/cxl.c
> create mode 100644 hw/cxl/Kconfig
> create mode 100644 hw/cxl/cxl-component-utils.c
> create mode 100644 hw/cxl/cxl-device-utils.c
> create mode 100644 hw/cxl/cxl-mailbox-utils.c
> create mode 100644 hw/cxl/meson.build
> create mode 100644 hw/mem/cxl_type3.c
> create mode 100644 hw/pci-bridge/cxl_root_port.c
> create mode 100644 include/hw/acpi/cxl.h
> create mode 100644 include/hw/cxl/cxl.h
> create mode 100644 include/hw/cxl/cxl_component.h
> create mode 100644 include/hw/cxl/cxl_device.h
> create mode 100644 include/hw/cxl/cxl_pci.h
> create mode 100644 tests/qtest/cxl-test.c
>
> --
> 2.29.2
>
>
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 00/25] Introduce CXL 2.0 Emulation
2020-11-17 14:09 ` Jonathan Cameron
@ 2020-11-25 18:29 ` Ben Widawsky
0 siblings, 0 replies; 9+ messages in thread
From: Ben Widawsky @ 2020-11-25 18:29 UTC (permalink / raw)
To: Jonathan Cameron
Cc: Vishal Verma, Dan Williams, qemu-devel, Philippe Mathieu-Daudé
On 20-11-17 14:09:14, Jonathan Cameron wrote:
[snip]
>
> Agreed, it was the intermediate state that I wasn't keen on of structures defined
> but then given 0 size. I'd rather just look at them all once. If that sometimes
> means introducing a file that isn't even referenced for a few patches, that's
> fine by me.
I've pushed v2 which hopefully addressed most of the feedback from you (it also
should fix some of the BIOS table failures from v1). My next plan is to
implement a few more firmware commands, and work on supporting interleaving
using the work from Phil. I will repost to the list after that.
https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0v2
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 00/25] Introduce CXL 2.0 Emulation
2020-11-16 18:06 ` Ben Widawsky
@ 2020-11-17 14:09 ` Jonathan Cameron
2020-11-25 18:29 ` Ben Widawsky
0 siblings, 1 reply; 9+ messages in thread
From: Jonathan Cameron @ 2020-11-17 14:09 UTC (permalink / raw)
To: Ben Widawsky; +Cc: Vishal Verma, Dan Williams, qemu-devel
On Mon, 16 Nov 2020 10:06:26 -0800
Ben Widawsky <ben.widawsky@intel.com> wrote:
> On 20-11-16 17:21:07, Jonathan Cameron wrote:
> > On Tue, 10 Nov 2020 21:46:59 -0800
> > Ben Widawsky <ben.widawsky@intel.com> wrote:
> >
> > > Introduce emulation of Compute Express Link 2.0, which was released
> > > today at https://www.computeexpresslink.org/.
> > >
> > > I've pushed a branch here: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0
> > >
> > > The emulation has been critical to get the Linux enabling started
> > > (https://lore.kernel.org/linux-cxl/), it would be an ideal place to land
> > > regression tests for different topology handling, and there may be applications
> > > for this emulation as a way for a guest to manipulate its address space relative
> > > to different performance memories. I am new to QEMU development, so please
> > > forgive and point me in the right direction if I severely misinterpreted where a
> > > piece of infrastructure belongs.
> > >
> > > Three of the five CXL component types are emulated with some level of functionality:
> > > host bridge, root port, and memory device. Upstream ports and downstream ports
> > > aren't implemented (the two components needed to make up a switch).
> > >
> > > CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the
> > > implementation utilizes existing PCI paradigms. To implement the host bridge,
> > > I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural
> > > fit even though it doesn't directly map to how hardware will work. For
> > > persistent capacity of the memory device, I utilized the memory subsystem
> > > (hw/mem).
> > >
> > > We have 3 reasons why this work is valuable:
> > > 1. OS driver development and testing
> > > 2. OS driver regression testing
> > > 3. Possible guest support for HDMs
> > >
> > > As mentioned above there are three benefits to carrying this enabling in
> > > upstream QEMU:
> > >
> > > 1. Linux driver feature development benefits from emulation both due to
> > > a lack of initial hardware availability, but also, as is seen with
> > > NVDIMM/PMEM emulation, there is value in being able to share
> > > topologies with system-software developers even after hardware is
> > > available.
> > >
> > > 2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake
> > > resources via custom modules (nfit_test). In retrospect a QEMU emulation of
> > > nfit_test capabilities would have made the test environment more portable, and
> > > allowed for easier community contributions of example configurations.
> > >
> > > 3. This is still being fleshed out, but in short it provides a standardized
> > > mechanism for the guest to provide feedback to the host about size and placement
> > > needs of the memory. After the host gives the guest a physical window mapping to
> > > the CXL device, the emulated HDM decoders allow the guest a way to tell the host
> > > how much it wants and where. There are likely simpler ways to do this, but
> > > they'd require inventing a new interface and you'd need to have diverging driver
> > > code in the guest programming of the HDM decoder vs. the host. Since we've
> > > already done this work, why not use it?
> > >
> > > There is quite a long list of work to do for full spec compliance, but I don't
> > > believe that any of it precludes merging. Off the top of my head:
> > > - Main host bridge support (WIP)
> > > - Interleaving
> > > - Better Tests
> > > - Huge swaths of firmware functionality
> > > - Hot plug support
> > > - Emulating volatile capacity
> > >
> > > The flow of the patches in general is to define all the data structures and
> > > registers associated with the various components in a top down manner. Host
> > > bridge, component, ports, devices. Then, the actual implementation is done in
> > > the same order.
> > >
> > > The summary is:
> > > 1-8: Put infrastructure in place for emulation of the components.
> > > 9-11: Create the concept of a CXL bus and plumb into PXB
> > > 12-16: Implement host bridges
> > > 17: Implement a root port
> > > 18: Implement a memory device
> > > 19: Implement HDM decoders
> > > 20-24: ACPI bits
> > > 25: Start working on enabling the main host bridge
> >
> > Hi Ben,
> >
> > I've take a look at the whole series and offered a few comments in things that
> > stood out. Unfortunately I'm playing catchup on CXL 2.0 and my qemu knowledge
> > is not what I'd like it to be.
> >
> > Having said that, this feels like a good start to me. Please clean up
> > the few patch handling issues before a v2. Code that appears, disappears and
> > reappears is a bit distracting :)
> >
> > Next up, the kernel side.
> >
> > Thanks,
> >
> > Jonathan
>
> Thanks very much for taking the time Jonathan. I saw your CCIX series early on
> and it was definitely helpful to me, so thanks for that as well. As you can
> probably tell, this series has been rebased to hell and back and you caught some
> of that in the code churn. I'll work on fixing those. I foolishly did a pretty
> major refactor just before submission.
>
> I wanted to discuss the 'dump all the defines in a patch and use them later'
> style I went for. In general, I don't do this and I leave feedback on patches
> that do this. I had two reasons for doing it here:
> 1. I wanted to separate a, 'go read the spec review' from actual functionality.
> I hope some of the issues you spotted were because of that.
An aim I can definitely get behind. However, at the moment it feels like a half
way stage. Some sections are fully defined, others not. Mind you I don't know
about how the qemu community feels about large definition sets that aren't going
to get used for a 'while'.
> 2. Since I decided to make all the helper libraries first, many defines are
> needed for that.
>
> For v2, I'll make sure there are no #define only patches, but I would still like
> to introduce the helper libraries first which will leave some unused functions
> and defines for a few patches.
Agreed, it was the intermediate state that I wasn't keen on of structures defined
but then given 0 size. I'd rather just look at them all once. If that sometimes
means introducing a file that isn't even referenced for a few patches, that's
fine by me.
Jonathan
>
> Ben
>
> >
> > >
> > > Ben Widawsky (23):
> > > hw/pci/cxl: Add a CXL component type (interface)
> > > hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
> > > hw/cxl/device: Introduce a CXL device (8.2.8)
> > > hw/cxl/device: Implement the CAP array (8.2.8.1-2)
> > > hw/cxl/device: Add device status (8.2.8.3)
> > > hw/cxl/device: Implement basic mailbox (8.2.8.4)
> > > hw/cxl/device: Add memory devices (8.2.8.5)
> > > hw/pxb: Use a type for realizing expanders
> > > hw/pci/cxl: Create a CXL bus type
> > > hw/pxb: Allow creation of a CXL PXB (host bridge)
> > > acpi/pci: Consolidate host bridge setup
> > > hw/pci: Plumb _UID through host bridges
> > > hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
> > > acpi/pxb/cxl: Reserve host bridge MMIO
> > > hw/pxb/cxl: Add "windows" for host bridges
> > > hw/cxl/rp: Add a root port
> > > hw/cxl/device: Add a memory device (8.2.8.5)
> > > hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
> > > acpi/cxl: Add _OSC implementation (9.14.2)
> > > acpi/cxl: Create the CEDT (9.14.1)
> > > Temp: acpi/cxl: Add ACPI0017 (CEDT awareness)
> > > WIP: i386/cxl: Initialize a host bridge
> > > qtest/cxl: Add very basic sanity tests
> > >
> > > Jonathan Cameron (1):
> > > Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.
> > >
> > > Vishal Verma (1):
> > > acpi/cxl: Introduce a compat-driver UUID for CXL _OSC
> > >
> > > MAINTAINERS | 6 +
> > > hw/Kconfig | 1 +
> > > hw/acpi/Kconfig | 5 +
> > > hw/acpi/cxl.c | 198 +++++++++++++
> > > hw/acpi/meson.build | 1 +
> > > hw/arm/virt.c | 1 +
> > > hw/core/machine.c | 26 ++
> > > hw/core/numa.c | 3 +
> > > hw/cxl/Kconfig | 3 +
> > > hw/cxl/cxl-component-utils.c | 192 +++++++++++++
> > > hw/cxl/cxl-device-utils.c | 293 +++++++++++++++++++
> > > hw/cxl/cxl-mailbox-utils.c | 139 +++++++++
> > > hw/cxl/meson.build | 5 +
> > > hw/i386/acpi-build.c | 87 +++++-
> > > hw/i386/microvm.c | 1 +
> > > hw/i386/pc.c | 2 +
> > > hw/mem/Kconfig | 5 +
> > > hw/mem/cxl_type3.c | 334 ++++++++++++++++++++++
> > > hw/mem/meson.build | 1 +
> > > hw/meson.build | 1 +
> > > hw/pci-bridge/Kconfig | 5 +
> > > hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++
> > > hw/pci-bridge/meson.build | 1 +
> > > hw/pci-bridge/pci_expander_bridge.c | 209 +++++++++++++-
> > > hw/pci-bridge/pcie_root_port.c | 6 +-
> > > hw/pci/pci.c | 32 ++-
> > > hw/pci/pcie.c | 30 ++
> > > hw/ppc/spapr.c | 2 +
> > > include/hw/acpi/cxl.h | 27 ++
> > > include/hw/boards.h | 2 +
> > > include/hw/cxl/cxl.h | 30 ++
> > > include/hw/cxl/cxl_component.h | 181 ++++++++++++
> > > include/hw/cxl/cxl_device.h | 199 +++++++++++++
> > > include/hw/cxl/cxl_pci.h | 155 ++++++++++
> > > include/hw/pci/pci.h | 15 +
> > > include/hw/pci/pci_bridge.h | 25 ++
> > > include/hw/pci/pci_bus.h | 8 +
> > > include/hw/pci/pci_ids.h | 1 +
> > > include/standard-headers/linux/pci_regs.h | 1 +
> > > monitor/hmp-cmds.c | 15 +
> > > qapi/machine.json | 1 +
> > > tests/qtest/cxl-test.c | 93 ++++++
> > > tests/qtest/meson.build | 4 +
> > > 43 files changed, 2547 insertions(+), 30 deletions(-)
> > > create mode 100644 hw/acpi/cxl.c
> > > create mode 100644 hw/cxl/Kconfig
> > > create mode 100644 hw/cxl/cxl-component-utils.c
> > > create mode 100644 hw/cxl/cxl-device-utils.c
> > > create mode 100644 hw/cxl/cxl-mailbox-utils.c
> > > create mode 100644 hw/cxl/meson.build
> > > create mode 100644 hw/mem/cxl_type3.c
> > > create mode 100644 hw/pci-bridge/cxl_root_port.c
> > > create mode 100644 include/hw/acpi/cxl.h
> > > create mode 100644 include/hw/cxl/cxl.h
> > > create mode 100644 include/hw/cxl/cxl_component.h
> > > create mode 100644 include/hw/cxl/cxl_device.h
> > > create mode 100644 include/hw/cxl/cxl_pci.h
> > > create mode 100644 tests/qtest/cxl-test.c
> > >
> >
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 00/25] Introduce CXL 2.0 Emulation
2020-11-16 17:21 ` Jonathan Cameron
@ 2020-11-16 18:06 ` Ben Widawsky
2020-11-17 14:09 ` Jonathan Cameron
0 siblings, 1 reply; 9+ messages in thread
From: Ben Widawsky @ 2020-11-16 18:06 UTC (permalink / raw)
To: Jonathan Cameron; +Cc: Vishal Verma, Dan Williams, qemu-devel
On 20-11-16 17:21:07, Jonathan Cameron wrote:
> On Tue, 10 Nov 2020 21:46:59 -0800
> Ben Widawsky <ben.widawsky@intel.com> wrote:
>
> > Introduce emulation of Compute Express Link 2.0, which was released
> > today at https://www.computeexpresslink.org/.
> >
> > I've pushed a branch here: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0
> >
> > The emulation has been critical to get the Linux enabling started
> > (https://lore.kernel.org/linux-cxl/), it would be an ideal place to land
> > regression tests for different topology handling, and there may be applications
> > for this emulation as a way for a guest to manipulate its address space relative
> > to different performance memories. I am new to QEMU development, so please
> > forgive and point me in the right direction if I severely misinterpreted where a
> > piece of infrastructure belongs.
> >
> > Three of the five CXL component types are emulated with some level of functionality:
> > host bridge, root port, and memory device. Upstream ports and downstream ports
> > aren't implemented (the two components needed to make up a switch).
> >
> > CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the
> > implementation utilizes existing PCI paradigms. To implement the host bridge,
> > I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural
> > fit even though it doesn't directly map to how hardware will work. For
> > persistent capacity of the memory device, I utilized the memory subsystem
> > (hw/mem).
> >
> > We have 3 reasons why this work is valuable:
> > 1. OS driver development and testing
> > 2. OS driver regression testing
> > 3. Possible guest support for HDMs
> >
> > As mentioned above there are three benefits to carrying this enabling in
> > upstream QEMU:
> >
> > 1. Linux driver feature development benefits from emulation both due to
> > a lack of initial hardware availability, but also, as is seen with
> > NVDIMM/PMEM emulation, there is value in being able to share
> > topologies with system-software developers even after hardware is
> > available.
> >
> > 2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake
> > resources via custom modules (nfit_test). In retrospect a QEMU emulation of
> > nfit_test capabilities would have made the test environment more portable, and
> > allowed for easier community contributions of example configurations.
> >
> > 3. This is still being fleshed out, but in short it provides a standardized
> > mechanism for the guest to provide feedback to the host about size and placement
> > needs of the memory. After the host gives the guest a physical window mapping to
> > the CXL device, the emulated HDM decoders allow the guest a way to tell the host
> > how much it wants and where. There are likely simpler ways to do this, but
> > they'd require inventing a new interface and you'd need to have diverging driver
> > code in the guest programming of the HDM decoder vs. the host. Since we've
> > already done this work, why not use it?
> >
> > There is quite a long list of work to do for full spec compliance, but I don't
> > believe that any of it precludes merging. Off the top of my head:
> > - Main host bridge support (WIP)
> > - Interleaving
> > - Better Tests
> > - Huge swaths of firmware functionality
> > - Hot plug support
> > - Emulating volatile capacity
> >
> > The flow of the patches in general is to define all the data structures and
> > registers associated with the various components in a top down manner. Host
> > bridge, component, ports, devices. Then, the actual implementation is done in
> > the same order.
> >
> > The summary is:
> > 1-8: Put infrastructure in place for emulation of the components.
> > 9-11: Create the concept of a CXL bus and plumb into PXB
> > 12-16: Implement host bridges
> > 17: Implement a root port
> > 18: Implement a memory device
> > 19: Implement HDM decoders
> > 20-24: ACPI bits
> > 25: Start working on enabling the main host bridge
>
> Hi Ben,
>
> I've take a look at the whole series and offered a few comments in things that
> stood out. Unfortunately I'm playing catchup on CXL 2.0 and my qemu knowledge
> is not what I'd like it to be.
>
> Having said that, this feels like a good start to me. Please clean up
> the few patch handling issues before a v2. Code that appears, disappears and
> reappears is a bit distracting :)
>
> Next up, the kernel side.
>
> Thanks,
>
> Jonathan
Thanks very much for taking the time Jonathan. I saw your CCIX series early on
and it was definitely helpful to me, so thanks for that as well. As you can
probably tell, this series has been rebased to hell and back and you caught some
of that in the code churn. I'll work on fixing those. I foolishly did a pretty
major refactor just before submission.
I wanted to discuss the 'dump all the defines in a patch and use them later'
style I went for. In general, I don't do this and I leave feedback on patches
that do this. I had two reasons for doing it here:
1. I wanted to separate a, 'go read the spec review' from actual functionality.
I hope some of the issues you spotted were because of that.
2. Since I decided to make all the helper libraries first, many defines are
needed for that.
For v2, I'll make sure there are no #define only patches, but I would still like
to introduce the helper libraries first which will leave some unused functions
and defines for a few patches.
Ben
>
> >
> > Ben Widawsky (23):
> > hw/pci/cxl: Add a CXL component type (interface)
> > hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
> > hw/cxl/device: Introduce a CXL device (8.2.8)
> > hw/cxl/device: Implement the CAP array (8.2.8.1-2)
> > hw/cxl/device: Add device status (8.2.8.3)
> > hw/cxl/device: Implement basic mailbox (8.2.8.4)
> > hw/cxl/device: Add memory devices (8.2.8.5)
> > hw/pxb: Use a type for realizing expanders
> > hw/pci/cxl: Create a CXL bus type
> > hw/pxb: Allow creation of a CXL PXB (host bridge)
> > acpi/pci: Consolidate host bridge setup
> > hw/pci: Plumb _UID through host bridges
> > hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
> > acpi/pxb/cxl: Reserve host bridge MMIO
> > hw/pxb/cxl: Add "windows" for host bridges
> > hw/cxl/rp: Add a root port
> > hw/cxl/device: Add a memory device (8.2.8.5)
> > hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
> > acpi/cxl: Add _OSC implementation (9.14.2)
> > acpi/cxl: Create the CEDT (9.14.1)
> > Temp: acpi/cxl: Add ACPI0017 (CEDT awareness)
> > WIP: i386/cxl: Initialize a host bridge
> > qtest/cxl: Add very basic sanity tests
> >
> > Jonathan Cameron (1):
> > Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.
> >
> > Vishal Verma (1):
> > acpi/cxl: Introduce a compat-driver UUID for CXL _OSC
> >
> > MAINTAINERS | 6 +
> > hw/Kconfig | 1 +
> > hw/acpi/Kconfig | 5 +
> > hw/acpi/cxl.c | 198 +++++++++++++
> > hw/acpi/meson.build | 1 +
> > hw/arm/virt.c | 1 +
> > hw/core/machine.c | 26 ++
> > hw/core/numa.c | 3 +
> > hw/cxl/Kconfig | 3 +
> > hw/cxl/cxl-component-utils.c | 192 +++++++++++++
> > hw/cxl/cxl-device-utils.c | 293 +++++++++++++++++++
> > hw/cxl/cxl-mailbox-utils.c | 139 +++++++++
> > hw/cxl/meson.build | 5 +
> > hw/i386/acpi-build.c | 87 +++++-
> > hw/i386/microvm.c | 1 +
> > hw/i386/pc.c | 2 +
> > hw/mem/Kconfig | 5 +
> > hw/mem/cxl_type3.c | 334 ++++++++++++++++++++++
> > hw/mem/meson.build | 1 +
> > hw/meson.build | 1 +
> > hw/pci-bridge/Kconfig | 5 +
> > hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++
> > hw/pci-bridge/meson.build | 1 +
> > hw/pci-bridge/pci_expander_bridge.c | 209 +++++++++++++-
> > hw/pci-bridge/pcie_root_port.c | 6 +-
> > hw/pci/pci.c | 32 ++-
> > hw/pci/pcie.c | 30 ++
> > hw/ppc/spapr.c | 2 +
> > include/hw/acpi/cxl.h | 27 ++
> > include/hw/boards.h | 2 +
> > include/hw/cxl/cxl.h | 30 ++
> > include/hw/cxl/cxl_component.h | 181 ++++++++++++
> > include/hw/cxl/cxl_device.h | 199 +++++++++++++
> > include/hw/cxl/cxl_pci.h | 155 ++++++++++
> > include/hw/pci/pci.h | 15 +
> > include/hw/pci/pci_bridge.h | 25 ++
> > include/hw/pci/pci_bus.h | 8 +
> > include/hw/pci/pci_ids.h | 1 +
> > include/standard-headers/linux/pci_regs.h | 1 +
> > monitor/hmp-cmds.c | 15 +
> > qapi/machine.json | 1 +
> > tests/qtest/cxl-test.c | 93 ++++++
> > tests/qtest/meson.build | 4 +
> > 43 files changed, 2547 insertions(+), 30 deletions(-)
> > create mode 100644 hw/acpi/cxl.c
> > create mode 100644 hw/cxl/Kconfig
> > create mode 100644 hw/cxl/cxl-component-utils.c
> > create mode 100644 hw/cxl/cxl-device-utils.c
> > create mode 100644 hw/cxl/cxl-mailbox-utils.c
> > create mode 100644 hw/cxl/meson.build
> > create mode 100644 hw/mem/cxl_type3.c
> > create mode 100644 hw/pci-bridge/cxl_root_port.c
> > create mode 100644 include/hw/acpi/cxl.h
> > create mode 100644 include/hw/cxl/cxl.h
> > create mode 100644 include/hw/cxl/cxl_component.h
> > create mode 100644 include/hw/cxl/cxl_device.h
> > create mode 100644 include/hw/cxl/cxl_pci.h
> > create mode 100644 tests/qtest/cxl-test.c
> >
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [RFC PATCH 00/25] Introduce CXL 2.0 Emulation
2020-11-11 5:46 Ben Widawsky
@ 2020-11-16 17:21 ` Jonathan Cameron
2020-11-16 18:06 ` Ben Widawsky
2020-12-04 14:27 ` Daniel P. Berrangé
1 sibling, 1 reply; 9+ messages in thread
From: Jonathan Cameron @ 2020-11-16 17:21 UTC (permalink / raw)
To: Ben Widawsky; +Cc: Vishal Verma, Dan Williams, qemu-devel
On Tue, 10 Nov 2020 21:46:59 -0800
Ben Widawsky <ben.widawsky@intel.com> wrote:
> Introduce emulation of Compute Express Link 2.0, which was released
> today at https://www.computeexpresslink.org/.
>
> I've pushed a branch here: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0
>
> The emulation has been critical to get the Linux enabling started
> (https://lore.kernel.org/linux-cxl/), it would be an ideal place to land
> regression tests for different topology handling, and there may be applications
> for this emulation as a way for a guest to manipulate its address space relative
> to different performance memories. I am new to QEMU development, so please
> forgive and point me in the right direction if I severely misinterpreted where a
> piece of infrastructure belongs.
>
> Three of the five CXL component types are emulated with some level of functionality:
> host bridge, root port, and memory device. Upstream ports and downstream ports
> aren't implemented (the two components needed to make up a switch).
>
> CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the
> implementation utilizes existing PCI paradigms. To implement the host bridge,
> I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural
> fit even though it doesn't directly map to how hardware will work. For
> persistent capacity of the memory device, I utilized the memory subsystem
> (hw/mem).
>
> We have 3 reasons why this work is valuable:
> 1. OS driver development and testing
> 2. OS driver regression testing
> 3. Possible guest support for HDMs
>
> As mentioned above there are three benefits to carrying this enabling in
> upstream QEMU:
>
> 1. Linux driver feature development benefits from emulation both due to
> a lack of initial hardware availability, but also, as is seen with
> NVDIMM/PMEM emulation, there is value in being able to share
> topologies with system-software developers even after hardware is
> available.
>
> 2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake
> resources via custom modules (nfit_test). In retrospect a QEMU emulation of
> nfit_test capabilities would have made the test environment more portable, and
> allowed for easier community contributions of example configurations.
>
> 3. This is still being fleshed out, but in short it provides a standardized
> mechanism for the guest to provide feedback to the host about size and placement
> needs of the memory. After the host gives the guest a physical window mapping to
> the CXL device, the emulated HDM decoders allow the guest a way to tell the host
> how much it wants and where. There are likely simpler ways to do this, but
> they'd require inventing a new interface and you'd need to have diverging driver
> code in the guest programming of the HDM decoder vs. the host. Since we've
> already done this work, why not use it?
>
> There is quite a long list of work to do for full spec compliance, but I don't
> believe that any of it precludes merging. Off the top of my head:
> - Main host bridge support (WIP)
> - Interleaving
> - Better Tests
> - Huge swaths of firmware functionality
> - Hot plug support
> - Emulating volatile capacity
>
> The flow of the patches in general is to define all the data structures and
> registers associated with the various components in a top down manner. Host
> bridge, component, ports, devices. Then, the actual implementation is done in
> the same order.
>
> The summary is:
> 1-8: Put infrastructure in place for emulation of the components.
> 9-11: Create the concept of a CXL bus and plumb into PXB
> 12-16: Implement host bridges
> 17: Implement a root port
> 18: Implement a memory device
> 19: Implement HDM decoders
> 20-24: ACPI bits
> 25: Start working on enabling the main host bridge
Hi Ben,
I've take a look at the whole series and offered a few comments in things that
stood out. Unfortunately I'm playing catchup on CXL 2.0 and my qemu knowledge
is not what I'd like it to be.
Having said that, this feels like a good start to me. Please clean up
the few patch handling issues before a v2. Code that appears, disappears and
reappears is a bit distracting :)
Next up, the kernel side.
Thanks,
Jonathan
>
> Ben Widawsky (23):
> hw/pci/cxl: Add a CXL component type (interface)
> hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
> hw/cxl/device: Introduce a CXL device (8.2.8)
> hw/cxl/device: Implement the CAP array (8.2.8.1-2)
> hw/cxl/device: Add device status (8.2.8.3)
> hw/cxl/device: Implement basic mailbox (8.2.8.4)
> hw/cxl/device: Add memory devices (8.2.8.5)
> hw/pxb: Use a type for realizing expanders
> hw/pci/cxl: Create a CXL bus type
> hw/pxb: Allow creation of a CXL PXB (host bridge)
> acpi/pci: Consolidate host bridge setup
> hw/pci: Plumb _UID through host bridges
> hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
> acpi/pxb/cxl: Reserve host bridge MMIO
> hw/pxb/cxl: Add "windows" for host bridges
> hw/cxl/rp: Add a root port
> hw/cxl/device: Add a memory device (8.2.8.5)
> hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
> acpi/cxl: Add _OSC implementation (9.14.2)
> acpi/cxl: Create the CEDT (9.14.1)
> Temp: acpi/cxl: Add ACPI0017 (CEDT awareness)
> WIP: i386/cxl: Initialize a host bridge
> qtest/cxl: Add very basic sanity tests
>
> Jonathan Cameron (1):
> Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.
>
> Vishal Verma (1):
> acpi/cxl: Introduce a compat-driver UUID for CXL _OSC
>
> MAINTAINERS | 6 +
> hw/Kconfig | 1 +
> hw/acpi/Kconfig | 5 +
> hw/acpi/cxl.c | 198 +++++++++++++
> hw/acpi/meson.build | 1 +
> hw/arm/virt.c | 1 +
> hw/core/machine.c | 26 ++
> hw/core/numa.c | 3 +
> hw/cxl/Kconfig | 3 +
> hw/cxl/cxl-component-utils.c | 192 +++++++++++++
> hw/cxl/cxl-device-utils.c | 293 +++++++++++++++++++
> hw/cxl/cxl-mailbox-utils.c | 139 +++++++++
> hw/cxl/meson.build | 5 +
> hw/i386/acpi-build.c | 87 +++++-
> hw/i386/microvm.c | 1 +
> hw/i386/pc.c | 2 +
> hw/mem/Kconfig | 5 +
> hw/mem/cxl_type3.c | 334 ++++++++++++++++++++++
> hw/mem/meson.build | 1 +
> hw/meson.build | 1 +
> hw/pci-bridge/Kconfig | 5 +
> hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++
> hw/pci-bridge/meson.build | 1 +
> hw/pci-bridge/pci_expander_bridge.c | 209 +++++++++++++-
> hw/pci-bridge/pcie_root_port.c | 6 +-
> hw/pci/pci.c | 32 ++-
> hw/pci/pcie.c | 30 ++
> hw/ppc/spapr.c | 2 +
> include/hw/acpi/cxl.h | 27 ++
> include/hw/boards.h | 2 +
> include/hw/cxl/cxl.h | 30 ++
> include/hw/cxl/cxl_component.h | 181 ++++++++++++
> include/hw/cxl/cxl_device.h | 199 +++++++++++++
> include/hw/cxl/cxl_pci.h | 155 ++++++++++
> include/hw/pci/pci.h | 15 +
> include/hw/pci/pci_bridge.h | 25 ++
> include/hw/pci/pci_bus.h | 8 +
> include/hw/pci/pci_ids.h | 1 +
> include/standard-headers/linux/pci_regs.h | 1 +
> monitor/hmp-cmds.c | 15 +
> qapi/machine.json | 1 +
> tests/qtest/cxl-test.c | 93 ++++++
> tests/qtest/meson.build | 4 +
> 43 files changed, 2547 insertions(+), 30 deletions(-)
> create mode 100644 hw/acpi/cxl.c
> create mode 100644 hw/cxl/Kconfig
> create mode 100644 hw/cxl/cxl-component-utils.c
> create mode 100644 hw/cxl/cxl-device-utils.c
> create mode 100644 hw/cxl/cxl-mailbox-utils.c
> create mode 100644 hw/cxl/meson.build
> create mode 100644 hw/mem/cxl_type3.c
> create mode 100644 hw/pci-bridge/cxl_root_port.c
> create mode 100644 include/hw/acpi/cxl.h
> create mode 100644 include/hw/cxl/cxl.h
> create mode 100644 include/hw/cxl/cxl_component.h
> create mode 100644 include/hw/cxl/cxl_device.h
> create mode 100644 include/hw/cxl/cxl_pci.h
> create mode 100644 tests/qtest/cxl-test.c
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH 00/25] Introduce CXL 2.0 Emulation
@ 2020-11-11 5:46 Ben Widawsky
2020-11-16 17:21 ` Jonathan Cameron
2020-12-04 14:27 ` Daniel P. Berrangé
0 siblings, 2 replies; 9+ messages in thread
From: Ben Widawsky @ 2020-11-11 5:46 UTC (permalink / raw)
To: qemu-devel; +Cc: Vishal Verma, Dan Williams, Ben Widawsky
Introduce emulation of Compute Express Link 2.0, which was released
today at https://www.computeexpresslink.org/.
I've pushed a branch here: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0
The emulation has been critical to get the Linux enabling started
(https://lore.kernel.org/linux-cxl/), it would be an ideal place to land
regression tests for different topology handling, and there may be applications
for this emulation as a way for a guest to manipulate its address space relative
to different performance memories. I am new to QEMU development, so please
forgive and point me in the right direction if I severely misinterpreted where a
piece of infrastructure belongs.
Three of the five CXL component types are emulated with some level of functionality:
host bridge, root port, and memory device. Upstream ports and downstream ports
aren't implemented (the two components needed to make up a switch).
CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the
implementation utilizes existing PCI paradigms. To implement the host bridge,
I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural
fit even though it doesn't directly map to how hardware will work. For
persistent capacity of the memory device, I utilized the memory subsystem
(hw/mem).
We have 3 reasons why this work is valuable:
1. OS driver development and testing
2. OS driver regression testing
3. Possible guest support for HDMs
As mentioned above there are three benefits to carrying this enabling in
upstream QEMU:
1. Linux driver feature development benefits from emulation both due to
a lack of initial hardware availability, but also, as is seen with
NVDIMM/PMEM emulation, there is value in being able to share
topologies with system-software developers even after hardware is
available.
2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake
resources via custom modules (nfit_test). In retrospect a QEMU emulation of
nfit_test capabilities would have made the test environment more portable, and
allowed for easier community contributions of example configurations.
3. This is still being fleshed out, but in short it provides a standardized
mechanism for the guest to provide feedback to the host about size and placement
needs of the memory. After the host gives the guest a physical window mapping to
the CXL device, the emulated HDM decoders allow the guest a way to tell the host
how much it wants and where. There are likely simpler ways to do this, but
they'd require inventing a new interface and you'd need to have diverging driver
code in the guest programming of the HDM decoder vs. the host. Since we've
already done this work, why not use it?
There is quite a long list of work to do for full spec compliance, but I don't
believe that any of it precludes merging. Off the top of my head:
- Main host bridge support (WIP)
- Interleaving
- Better Tests
- Huge swaths of firmware functionality
- Hot plug support
- Emulating volatile capacity
The flow of the patches in general is to define all the data structures and
registers associated with the various components in a top down manner. Host
bridge, component, ports, devices. Then, the actual implementation is done in
the same order.
The summary is:
1-8: Put infrastructure in place for emulation of the components.
9-11: Create the concept of a CXL bus and plumb into PXB
12-16: Implement host bridges
17: Implement a root port
18: Implement a memory device
19: Implement HDM decoders
20-24: ACPI bits
25: Start working on enabling the main host bridge
Ben Widawsky (23):
hw/pci/cxl: Add a CXL component type (interface)
hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
hw/cxl/device: Introduce a CXL device (8.2.8)
hw/cxl/device: Implement the CAP array (8.2.8.1-2)
hw/cxl/device: Add device status (8.2.8.3)
hw/cxl/device: Implement basic mailbox (8.2.8.4)
hw/cxl/device: Add memory devices (8.2.8.5)
hw/pxb: Use a type for realizing expanders
hw/pci/cxl: Create a CXL bus type
hw/pxb: Allow creation of a CXL PXB (host bridge)
acpi/pci: Consolidate host bridge setup
hw/pci: Plumb _UID through host bridges
hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
acpi/pxb/cxl: Reserve host bridge MMIO
hw/pxb/cxl: Add "windows" for host bridges
hw/cxl/rp: Add a root port
hw/cxl/device: Add a memory device (8.2.8.5)
hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
acpi/cxl: Add _OSC implementation (9.14.2)
acpi/cxl: Create the CEDT (9.14.1)
Temp: acpi/cxl: Add ACPI0017 (CEDT awareness)
WIP: i386/cxl: Initialize a host bridge
qtest/cxl: Add very basic sanity tests
Jonathan Cameron (1):
Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.
Vishal Verma (1):
acpi/cxl: Introduce a compat-driver UUID for CXL _OSC
MAINTAINERS | 6 +
hw/Kconfig | 1 +
hw/acpi/Kconfig | 5 +
hw/acpi/cxl.c | 198 +++++++++++++
hw/acpi/meson.build | 1 +
hw/arm/virt.c | 1 +
hw/core/machine.c | 26 ++
hw/core/numa.c | 3 +
hw/cxl/Kconfig | 3 +
hw/cxl/cxl-component-utils.c | 192 +++++++++++++
hw/cxl/cxl-device-utils.c | 293 +++++++++++++++++++
hw/cxl/cxl-mailbox-utils.c | 139 +++++++++
hw/cxl/meson.build | 5 +
hw/i386/acpi-build.c | 87 +++++-
hw/i386/microvm.c | 1 +
hw/i386/pc.c | 2 +
hw/mem/Kconfig | 5 +
hw/mem/cxl_type3.c | 334 ++++++++++++++++++++++
hw/mem/meson.build | 1 +
hw/meson.build | 1 +
hw/pci-bridge/Kconfig | 5 +
hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++
hw/pci-bridge/meson.build | 1 +
hw/pci-bridge/pci_expander_bridge.c | 209 +++++++++++++-
hw/pci-bridge/pcie_root_port.c | 6 +-
hw/pci/pci.c | 32 ++-
hw/pci/pcie.c | 30 ++
hw/ppc/spapr.c | 2 +
include/hw/acpi/cxl.h | 27 ++
include/hw/boards.h | 2 +
include/hw/cxl/cxl.h | 30 ++
include/hw/cxl/cxl_component.h | 181 ++++++++++++
include/hw/cxl/cxl_device.h | 199 +++++++++++++
include/hw/cxl/cxl_pci.h | 155 ++++++++++
include/hw/pci/pci.h | 15 +
include/hw/pci/pci_bridge.h | 25 ++
include/hw/pci/pci_bus.h | 8 +
include/hw/pci/pci_ids.h | 1 +
include/standard-headers/linux/pci_regs.h | 1 +
monitor/hmp-cmds.c | 15 +
qapi/machine.json | 1 +
tests/qtest/cxl-test.c | 93 ++++++
tests/qtest/meson.build | 4 +
43 files changed, 2547 insertions(+), 30 deletions(-)
create mode 100644 hw/acpi/cxl.c
create mode 100644 hw/cxl/Kconfig
create mode 100644 hw/cxl/cxl-component-utils.c
create mode 100644 hw/cxl/cxl-device-utils.c
create mode 100644 hw/cxl/cxl-mailbox-utils.c
create mode 100644 hw/cxl/meson.build
create mode 100644 hw/mem/cxl_type3.c
create mode 100644 hw/pci-bridge/cxl_root_port.c
create mode 100644 include/hw/acpi/cxl.h
create mode 100644 include/hw/cxl/cxl.h
create mode 100644 include/hw/cxl/cxl_component.h
create mode 100644 include/hw/cxl/cxl_device.h
create mode 100644 include/hw/cxl/cxl_pci.h
create mode 100644 tests/qtest/cxl-test.c
--
2.29.2
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2020-12-04 14:28 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-12-04 5:08 [RFC PATCH 00/25] Introduce CXL 2.0 Emulation Chris Browy
2020-12-04 5:55 ` Dan Williams
2020-12-04 5:55 ` Dan Williams
-- strict thread matches above, loose matches on Subject: below --
2020-11-11 5:46 Ben Widawsky
2020-11-16 17:21 ` Jonathan Cameron
2020-11-16 18:06 ` Ben Widawsky
2020-11-17 14:09 ` Jonathan Cameron
2020-11-25 18:29 ` Ben Widawsky
2020-12-04 14:27 ` Daniel P. Berrangé
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.