QEMU-Devel Archive on lore.kernel.org
 help / color / Atom feed
From: Ben Widawsky <ben.widawsky@intel.com>
To: qemu-devel@nongnu.org
Cc: "Ben Widawsky" <ben.widawsky@intel.com>,
	"David Hildenbrand" <david@redhat.com>,
	"Vishal Verma" <vishal.l.verma@intel.com>,
	"John Groves (jgroves)" <jgroves@micron.com>,
	"Chris Browy" <cbrowy@avery-design.com>,
	"Markus Armbruster" <armbru@redhat.com>,
	linux-cxl@vger.kernel.org,
	"Philippe Mathieu-Daudé" <f4bug@amsat.org>,
	"Michael S. Tsirkin" <mst@redhat.com>,
	"Jonathan Cameron" <Jonathan.Cameron@Huawei.com>,
	"Igor Mammedov" <imammedo@redhat.com>,
	"Dan Williams" <dan.j.williams@intel.com>,
	"Ira Weiny" <ira.weiny@intel.com>
Subject: [RFC PATCH v3 00/31] CXL 2.0 Support
Date: Mon,  1 Feb 2021 16:59:17 -0800
Message-ID: <20210202005948.241655-1-ben.widawsky@intel.com> (raw)

Major changes since v2 [1]:
 * Removed all register endian/alignment/size checking. Using core functionality
   instead. This untested on big endian hosts, but Should Work(tm).
 * Fix component capability header generation (off by 1).
 * Fixed HDM programming (multiple issues).
 * Fixed timestamp command implementations.
 * Added commands: GET_FIRMWARE_UPDATE_INFO, GET_PARTITION_INFO, GET_LSA, SET_LSA

Things have remained fairly stable since since v2. The biggest change here is
definitely the HDM programming which has received limited (but not 0) testing in
the Linux driver.

Jonathan Cameron has gotten this patch series working on ARM [2], and added some
much sought after functionality [3].

---

I've started #cxl on OFTC IRC for discussion. Please feel free to use that
channel for questions or suggestions in addition to #qemu.

---

Introduce emulation of Compute Express Link 2.0
(https://www.computeexpresslink.org/). Specifically, add support for Type 3
memory expanders with persistent memory.

The emulation has been critical to get the Linux enabling started [4], it would
be an ideal place to land regression tests for different topology handling, and
there may be applications for this emulation as a way for a guest to manipulate
its address space relative to different performance memories.

Three of the five CXL component types are emulated with some level of
functionality: host bridge, root port, and memory device. All components and
devices implement basic MMIO. Devices/memory devices implement the mailbo
interface. Basic ACPI support is also included. Upstream ports and downstream
ports aren't implemented (the two components needed to make up a switch).

CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the
implementation utilizes existing PCI paradigms. To implement the host bridge,
I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural
fit even though it doesn't directly map to how hardware will work. For
persistent capacity of the memory device, I utilized the memory subsystem
(hw/mem).

We have 3 reasons why this work is valuable:
1. Linux driver feature development benefits from emulation both due to a lack
   of initial hardware availability, but also, as is seen with NVDIMM/PMEM
   emulation, there is value in being able to share topologies with
   system-software developers even after hardware is available.

2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake
   resources via custom modules (nfit_test). In retrospect a QEMU emulation of
   nfit_test capabilities would have made the test environment more portable,
   and allowed for easier community contributions of example configurations.

3. This is still being fleshed out, but in short it provides a standardized
   mechanism for the guest to provide feedback to the host about size and
   placement needs of the memory. After the host gives the guest a physical
   window mapping to the CXL device, the emulated HDM decoders allow the guest a
   way to tell the host how much it wants and where. There are likely simpler
   ways to do this, but they'd require inventing a new interface and you'd need
   to have diverging driver code in the guest programming of the HDM decoder vs.
   the host. Since we've already done this work, why not use it?

There is quite a long list of work to do for full spec compliance, but I don't
believe that any of it precludes merging. Off the top of my head:
- Main host bridge support (WIP)
- Interleaving
- Better Tests
- Hot plug support
- Emulating volatile capacity
- CDAT emulation [3]

The flow of the patches in general is to define all the data structures and
registers associated with the various components in a top down manner. Host
bridge, component, ports, devices. Then, the actual implementation is done in
the same order.

The summary is:
1-5: Infrastructure for component and device emulation
6-9: Basic mailbox command implementations
10-19: Implement CXL host bridges as PXB devices
20: Implement a root port
21-22: Implement a memory device
23-26: ACPI bits
27-29: Add some more advanced mailbox command implementations
30: Start working on enabling the main host bridge
31: Basic test case

---

[1]: https://lore.kernel.org/qemu-devel/20210105165323.783725-1-ben.widawsky@intel.com/
[2]: https://lore.kernel.org/qemu-devel/20210201152655.31027-1-Jonathan.Cameron@huawei.com/
[3]: https://lore.kernel.org/qemu-devel/20210201151629.29656-1-Jonathan.Cameron@huawei.com/
[4]: https://lore.kernel.org/linux-cxl/20210130002438.1872527-1-ben.widawsky@intel.com/

---

Ben Widawsky (31):
  hw/pci/cxl: Add a CXL component type (interface)
  hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
  hw/cxl/device: Introduce a CXL device (8.2.8)
  hw/cxl/device: Implement the CAP array (8.2.8.1-2)
  hw/cxl/device: Implement basic mailbox (8.2.8.4)
  hw/cxl/device: Add memory device utilities
  hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1)
  hw/cxl/device: Timestamp implementation (8.2.9.3)
  hw/cxl/device: Add log commands (8.2.9.4) + CEL
  hw/pxb: Use a type for realizing expanders
  hw/pci/cxl: Create a CXL bus type
  hw/pxb: Allow creation of a CXL PXB (host bridge)
  qtest: allow DSDT acpi table changes
  acpi/pci: Consolidate host bridge setup
  tests/acpi: remove stale allowed tables
  hw/pci: Plumb _UID through host bridges
  hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
  acpi/pxb/cxl: Reserve host bridge MMIO
  hw/pxb/cxl: Add "windows" for host bridges
  hw/cxl/rp: Add a root port
  hw/cxl/device: Add a memory device (8.2.8.5)
  hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
  acpi/cxl: Add _OSC implementation (9.14.2)
  tests/acpi: allow CEDT table addition
  acpi/cxl: Create the CEDT (9.14.1)
  tests/acpi: Add new CEDT files
  hw/cxl/device: Add some trivial commands
  hw/cxl/device: Plumb real LSA sizing
  hw/cxl/device: Implement get/set LSA
  qtest/cxl: Add very basic sanity tests
  WIP: i386/cxl: Initialize a host bridge

 MAINTAINERS                         |   6 +
 hw/Kconfig                          |   1 +
 hw/acpi/Kconfig                     |   5 +
 hw/acpi/cxl.c                       | 173 ++++++++++
 hw/acpi/meson.build                 |   1 +
 hw/arm/virt.c                       |   1 +
 hw/core/machine.c                   |  26 ++
 hw/core/numa.c                      |   3 +
 hw/cxl/Kconfig                      |   3 +
 hw/cxl/cxl-component-utils.c        | 208 ++++++++++++
 hw/cxl/cxl-device-utils.c           | 264 +++++++++++++++
 hw/cxl/cxl-mailbox-utils.c          | 498 ++++++++++++++++++++++++++++
 hw/cxl/meson.build                  |   5 +
 hw/i386/acpi-build.c                |  87 ++++-
 hw/i386/microvm.c                   |   1 +
 hw/i386/pc.c                        |   2 +
 hw/mem/Kconfig                      |   5 +
 hw/mem/cxl_type3.c                  | 405 ++++++++++++++++++++++
 hw/mem/meson.build                  |   1 +
 hw/meson.build                      |   1 +
 hw/pci-bridge/Kconfig               |   5 +
 hw/pci-bridge/cxl_root_port.c       | 231 +++++++++++++
 hw/pci-bridge/meson.build           |   1 +
 hw/pci-bridge/pci_expander_bridge.c | 209 +++++++++++-
 hw/pci-bridge/pcie_root_port.c      |   6 +-
 hw/pci/pci.c                        |  32 +-
 hw/pci/pcie.c                       |  30 ++
 hw/ppc/spapr.c                      |   2 +
 include/hw/acpi/cxl.h               |  27 ++
 include/hw/boards.h                 |   2 +
 include/hw/cxl/cxl.h                |  29 ++
 include/hw/cxl/cxl_component.h      | 187 +++++++++++
 include/hw/cxl/cxl_device.h         | 255 ++++++++++++++
 include/hw/cxl/cxl_pci.h            | 160 +++++++++
 include/hw/pci/pci.h                |  15 +
 include/hw/pci/pci_bridge.h         |  25 ++
 include/hw/pci/pci_bus.h            |   8 +
 include/hw/pci/pci_ids.h            |   1 +
 monitor/hmp-cmds.c                  |  15 +
 qapi/machine.json                   |   1 +
 tests/data/acpi/pc/CEDT             | Bin 0 -> 36 bytes
 tests/data/acpi/pc/DSDT             | Bin 5065 -> 5065 bytes
 tests/data/acpi/pc/DSDT.acpihmat    | Bin 6390 -> 6390 bytes
 tests/data/acpi/pc/DSDT.bridge      | Bin 6924 -> 6924 bytes
 tests/data/acpi/pc/DSDT.cphp        | Bin 5529 -> 5529 bytes
 tests/data/acpi/pc/DSDT.dimmpxm     | Bin 6719 -> 6719 bytes
 tests/data/acpi/pc/DSDT.hpbridge    | Bin 5026 -> 5026 bytes
 tests/data/acpi/pc/DSDT.hpbrroot    | Bin 3084 -> 3084 bytes
 tests/data/acpi/pc/DSDT.ipmikcs     | Bin 5137 -> 5137 bytes
 tests/data/acpi/pc/DSDT.memhp       | Bin 6424 -> 6424 bytes
 tests/data/acpi/pc/DSDT.numamem     | Bin 5071 -> 5071 bytes
 tests/data/acpi/pc/DSDT.roothp      | Bin 5261 -> 5261 bytes
 tests/data/acpi/q35/CEDT            | Bin 0 -> 36 bytes
 tests/data/acpi/q35/DSDT            | Bin 7801 -> 7801 bytes
 tests/data/acpi/q35/DSDT.acpihmat   | Bin 9126 -> 9126 bytes
 tests/data/acpi/q35/DSDT.bridge     | Bin 7819 -> 7819 bytes
 tests/data/acpi/q35/DSDT.cphp       | Bin 8265 -> 8265 bytes
 tests/data/acpi/q35/DSDT.dimmpxm    | Bin 9455 -> 9455 bytes
 tests/data/acpi/q35/DSDT.ipmibt     | Bin 7876 -> 7876 bytes
 tests/data/acpi/q35/DSDT.memhp      | Bin 9160 -> 9160 bytes
 tests/data/acpi/q35/DSDT.mmio64     | Bin 8932 -> 8932 bytes
 tests/data/acpi/q35/DSDT.numamem    | Bin 7807 -> 7807 bytes
 tests/qtest/cxl-test.c              |  93 ++++++
 tests/qtest/meson.build             |   4 +
 64 files changed, 3004 insertions(+), 30 deletions(-)
 create mode 100644 hw/acpi/cxl.c
 create mode 100644 hw/cxl/Kconfig
 create mode 100644 hw/cxl/cxl-component-utils.c
 create mode 100644 hw/cxl/cxl-device-utils.c
 create mode 100644 hw/cxl/cxl-mailbox-utils.c
 create mode 100644 hw/cxl/meson.build
 create mode 100644 hw/mem/cxl_type3.c
 create mode 100644 hw/pci-bridge/cxl_root_port.c
 create mode 100644 include/hw/acpi/cxl.h
 create mode 100644 include/hw/cxl/cxl.h
 create mode 100644 include/hw/cxl/cxl_component.h
 create mode 100644 include/hw/cxl/cxl_device.h
 create mode 100644 include/hw/cxl/cxl_pci.h
 create mode 100644 tests/data/acpi/pc/CEDT
 create mode 100644 tests/data/acpi/q35/CEDT
 create mode 100644 tests/qtest/cxl-test.c

-- 
2.30.0



             reply index

Thread overview: 64+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-02-02  0:59 Ben Widawsky [this message]
2021-02-02  0:59 ` [RFC PATCH v3 01/31] hw/pci/cxl: Add a CXL component type (interface) Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 02/31] hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) Ben Widawsky
2021-02-02 11:48   ` Jonathan Cameron
2021-02-17 18:36     ` Ben Widawsky
2021-02-11 17:08   ` Jonathan Cameron
2021-02-17 16:40     ` Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 03/31] hw/cxl/device: Introduce a CXL device (8.2.8) Ben Widawsky
2021-02-02 12:03   ` Jonathan Cameron
2021-02-02  0:59 ` [RFC PATCH v3 04/31] hw/cxl/device: Implement the CAP array (8.2.8.1-2) Ben Widawsky
2021-02-02 12:23   ` Jonathan Cameron
2021-02-17 22:15     ` Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 05/31] hw/cxl/device: Implement basic mailbox (8.2.8.4) Ben Widawsky
2021-02-02 14:58   ` Jonathan Cameron
2021-02-11 17:46     ` Jonathan Cameron
2021-02-18  0:55       ` Ben Widawsky
2021-02-18 16:50         ` Jonathan Cameron
2021-02-11 18:09   ` Jonathan Cameron
2021-02-02  0:59 ` [RFC PATCH v3 06/31] hw/cxl/device: Add memory device utilities Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 07/31] hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) Ben Widawsky
2021-02-02 13:44   ` Jonathan Cameron
2021-02-11 17:59   ` Jonathan Cameron
2021-02-02  0:59 ` [RFC PATCH v3 08/31] hw/cxl/device: Timestamp implementation (8.2.9.3) Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 09/31] hw/cxl/device: Add log commands (8.2.9.4) + CEL Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 10/31] hw/pxb: Use a type for realizing expanders Ben Widawsky
2021-02-02 13:50   ` Jonathan Cameron
2021-02-02  0:59 ` [RFC PATCH v3 11/31] hw/pci/cxl: Create a CXL bus type Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 12/31] hw/pxb: Allow creation of a CXL PXB (host bridge) Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 13/31] qtest: allow DSDT acpi table changes Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 14/31] acpi/pci: Consolidate host bridge setup Ben Widawsky
2021-02-02 13:56   ` Jonathan Cameron
2021-02-02  0:59 ` [RFC PATCH v3 15/31] tests/acpi: remove stale allowed tables Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 16/31] hw/pci: Plumb _UID through host bridges Ben Widawsky
2021-02-02 15:00   ` Jonathan Cameron
2021-02-02 15:24     ` Michael S. Tsirkin
2021-02-02 15:42       ` Ben Widawsky
2021-02-02 15:51         ` Michael S. Tsirkin
2021-02-02 16:20           ` Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 17/31] hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) Ben Widawsky
2021-02-02 19:21   ` Jonathan Cameron
2021-02-02 19:45     ` Ben Widawsky
2021-02-02 20:43       ` Jonathan Cameron
2021-02-02 21:03         ` Ben Widawsky
2021-02-02 22:06           ` Jonathan Cameron
2021-02-02  0:59 ` [RFC PATCH v3 18/31] acpi/pxb/cxl: Reserve host bridge MMIO Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 19/31] hw/pxb/cxl: Add "windows" for host bridges Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 20/31] hw/cxl/rp: Add a root port Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 21/31] hw/cxl/device: Add a memory device (8.2.8.5) Ben Widawsky
2021-02-02 14:26   ` Eric Blake
2021-02-02 15:06     ` Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 22/31] hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 23/31] acpi/cxl: Add _OSC implementation (9.14.2) Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 24/31] tests/acpi: allow CEDT table addition Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 25/31] acpi/cxl: Create the CEDT (9.14.1) Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 26/31] tests/acpi: Add new CEDT files Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 27/31] hw/cxl/device: Add some trivial commands Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 28/31] hw/cxl/device: Plumb real LSA sizing Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 29/31] hw/cxl/device: Implement get/set LSA Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 30/31] qtest/cxl: Add very basic sanity tests Ben Widawsky
2021-02-02  0:59 ` [RFC PATCH v3 31/31] WIP: i386/cxl: Initialize a host bridge Ben Widawsky
2021-02-02  1:33 ` [RFC PATCH v3 00/31] CXL 2.0 Support no-reply
2021-02-03 17:42 ` Ben Widawsky
2021-02-11 18:51   ` Jonathan Cameron
2021-03-11 23:27 ` [RFC PATCH] hw/mem/cxl_type3: Go back to subregions Ben Widawsky

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20210202005948.241655-1-ben.widawsky@intel.com \
    --to=ben.widawsky@intel.com \
    --cc=Jonathan.Cameron@Huawei.com \
    --cc=armbru@redhat.com \
    --cc=cbrowy@avery-design.com \
    --cc=dan.j.williams@intel.com \
    --cc=david@redhat.com \
    --cc=f4bug@amsat.org \
    --cc=imammedo@redhat.com \
    --cc=ira.weiny@intel.com \
    --cc=jgroves@micron.com \
    --cc=linux-cxl@vger.kernel.org \
    --cc=mst@redhat.com \
    --cc=qemu-devel@nongnu.org \
    --cc=vishal.l.verma@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

QEMU-Devel Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/qemu-devel/0 qemu-devel/git/0.git
	git clone --mirror https://lore.kernel.org/qemu-devel/1 qemu-devel/git/1.git
	git clone --mirror https://lore.kernel.org/qemu-devel/2 qemu-devel/git/2.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 qemu-devel qemu-devel/ https://lore.kernel.org/qemu-devel \
		qemu-devel@nongnu.org
	public-inbox-index qemu-devel

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.nongnu.qemu-devel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git