From: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
To: Ben Widawsky <ben.widawsky@intel.com>
Cc: Vishal Verma <vishal.l.verma@intel.com>,
Dan Williams <dan.j.williams@intel.com>,
qemu-devel@nongnu.org
Subject: Re: [RFC PATCH 00/25] Introduce CXL 2.0 Emulation
Date: Tue, 17 Nov 2020 14:09:14 +0000 [thread overview]
Message-ID: <20201117140914.000076d1@Huawei.com> (raw)
In-Reply-To: <20201116180626.g7swvwu5jhgzwc6o@intel.com>
On Mon, 16 Nov 2020 10:06:26 -0800
Ben Widawsky <ben.widawsky@intel.com> wrote:
> On 20-11-16 17:21:07, Jonathan Cameron wrote:
> > On Tue, 10 Nov 2020 21:46:59 -0800
> > Ben Widawsky <ben.widawsky@intel.com> wrote:
> >
> > > Introduce emulation of Compute Express Link 2.0, which was released
> > > today at https://www.computeexpresslink.org/.
> > >
> > > I've pushed a branch here: https://gitlab.com/bwidawsk/qemu/-/tree/cxl-2.0
> > >
> > > The emulation has been critical to get the Linux enabling started
> > > (https://lore.kernel.org/linux-cxl/), it would be an ideal place to land
> > > regression tests for different topology handling, and there may be applications
> > > for this emulation as a way for a guest to manipulate its address space relative
> > > to different performance memories. I am new to QEMU development, so please
> > > forgive and point me in the right direction if I severely misinterpreted where a
> > > piece of infrastructure belongs.
> > >
> > > Three of the five CXL component types are emulated with some level of functionality:
> > > host bridge, root port, and memory device. Upstream ports and downstream ports
> > > aren't implemented (the two components needed to make up a switch).
> > >
> > > CXL 2.0 is built on top of PCIe (see spec for details). As a result, much of the
> > > implementation utilizes existing PCI paradigms. To implement the host bridge,
> > > I've chosen to use PXB (PCI Expander Bridge). It seemed to be the most natural
> > > fit even though it doesn't directly map to how hardware will work. For
> > > persistent capacity of the memory device, I utilized the memory subsystem
> > > (hw/mem).
> > >
> > > We have 3 reasons why this work is valuable:
> > > 1. OS driver development and testing
> > > 2. OS driver regression testing
> > > 3. Possible guest support for HDMs
> > >
> > > As mentioned above there are three benefits to carrying this enabling in
> > > upstream QEMU:
> > >
> > > 1. Linux driver feature development benefits from emulation both due to
> > > a lack of initial hardware availability, but also, as is seen with
> > > NVDIMM/PMEM emulation, there is value in being able to share
> > > topologies with system-software developers even after hardware is
> > > available.
> > >
> > > 2. The Linux kernel's unit test suite for NVDIMM/PMEM ended up injecting fake
> > > resources via custom modules (nfit_test). In retrospect a QEMU emulation of
> > > nfit_test capabilities would have made the test environment more portable, and
> > > allowed for easier community contributions of example configurations.
> > >
> > > 3. This is still being fleshed out, but in short it provides a standardized
> > > mechanism for the guest to provide feedback to the host about size and placement
> > > needs of the memory. After the host gives the guest a physical window mapping to
> > > the CXL device, the emulated HDM decoders allow the guest a way to tell the host
> > > how much it wants and where. There are likely simpler ways to do this, but
> > > they'd require inventing a new interface and you'd need to have diverging driver
> > > code in the guest programming of the HDM decoder vs. the host. Since we've
> > > already done this work, why not use it?
> > >
> > > There is quite a long list of work to do for full spec compliance, but I don't
> > > believe that any of it precludes merging. Off the top of my head:
> > > - Main host bridge support (WIP)
> > > - Interleaving
> > > - Better Tests
> > > - Huge swaths of firmware functionality
> > > - Hot plug support
> > > - Emulating volatile capacity
> > >
> > > The flow of the patches in general is to define all the data structures and
> > > registers associated with the various components in a top down manner. Host
> > > bridge, component, ports, devices. Then, the actual implementation is done in
> > > the same order.
> > >
> > > The summary is:
> > > 1-8: Put infrastructure in place for emulation of the components.
> > > 9-11: Create the concept of a CXL bus and plumb into PXB
> > > 12-16: Implement host bridges
> > > 17: Implement a root port
> > > 18: Implement a memory device
> > > 19: Implement HDM decoders
> > > 20-24: ACPI bits
> > > 25: Start working on enabling the main host bridge
> >
> > Hi Ben,
> >
> > I've take a look at the whole series and offered a few comments in things that
> > stood out. Unfortunately I'm playing catchup on CXL 2.0 and my qemu knowledge
> > is not what I'd like it to be.
> >
> > Having said that, this feels like a good start to me. Please clean up
> > the few patch handling issues before a v2. Code that appears, disappears and
> > reappears is a bit distracting :)
> >
> > Next up, the kernel side.
> >
> > Thanks,
> >
> > Jonathan
>
> Thanks very much for taking the time Jonathan. I saw your CCIX series early on
> and it was definitely helpful to me, so thanks for that as well. As you can
> probably tell, this series has been rebased to hell and back and you caught some
> of that in the code churn. I'll work on fixing those. I foolishly did a pretty
> major refactor just before submission.
>
> I wanted to discuss the 'dump all the defines in a patch and use them later'
> style I went for. In general, I don't do this and I leave feedback on patches
> that do this. I had two reasons for doing it here:
> 1. I wanted to separate a, 'go read the spec review' from actual functionality.
> I hope some of the issues you spotted were because of that.
An aim I can definitely get behind. However, at the moment it feels like a half
way stage. Some sections are fully defined, others not. Mind you I don't know
about how the qemu community feels about large definition sets that aren't going
to get used for a 'while'.
> 2. Since I decided to make all the helper libraries first, many defines are
> needed for that.
>
> For v2, I'll make sure there are no #define only patches, but I would still like
> to introduce the helper libraries first which will leave some unused functions
> and defines for a few patches.
Agreed, it was the intermediate state that I wasn't keen on of structures defined
but then given 0 size. I'd rather just look at them all once. If that sometimes
means introducing a file that isn't even referenced for a few patches, that's
fine by me.
Jonathan
>
> Ben
>
> >
> > >
> > > Ben Widawsky (23):
> > > hw/pci/cxl: Add a CXL component type (interface)
> > > hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
> > > hw/cxl/device: Introduce a CXL device (8.2.8)
> > > hw/cxl/device: Implement the CAP array (8.2.8.1-2)
> > > hw/cxl/device: Add device status (8.2.8.3)
> > > hw/cxl/device: Implement basic mailbox (8.2.8.4)
> > > hw/cxl/device: Add memory devices (8.2.8.5)
> > > hw/pxb: Use a type for realizing expanders
> > > hw/pci/cxl: Create a CXL bus type
> > > hw/pxb: Allow creation of a CXL PXB (host bridge)
> > > acpi/pci: Consolidate host bridge setup
> > > hw/pci: Plumb _UID through host bridges
> > > hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
> > > acpi/pxb/cxl: Reserve host bridge MMIO
> > > hw/pxb/cxl: Add "windows" for host bridges
> > > hw/cxl/rp: Add a root port
> > > hw/cxl/device: Add a memory device (8.2.8.5)
> > > hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
> > > acpi/cxl: Add _OSC implementation (9.14.2)
> > > acpi/cxl: Create the CEDT (9.14.1)
> > > Temp: acpi/cxl: Add ACPI0017 (CEDT awareness)
> > > WIP: i386/cxl: Initialize a host bridge
> > > qtest/cxl: Add very basic sanity tests
> > >
> > > Jonathan Cameron (1):
> > > Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy.
> > >
> > > Vishal Verma (1):
> > > acpi/cxl: Introduce a compat-driver UUID for CXL _OSC
> > >
> > > MAINTAINERS | 6 +
> > > hw/Kconfig | 1 +
> > > hw/acpi/Kconfig | 5 +
> > > hw/acpi/cxl.c | 198 +++++++++++++
> > > hw/acpi/meson.build | 1 +
> > > hw/arm/virt.c | 1 +
> > > hw/core/machine.c | 26 ++
> > > hw/core/numa.c | 3 +
> > > hw/cxl/Kconfig | 3 +
> > > hw/cxl/cxl-component-utils.c | 192 +++++++++++++
> > > hw/cxl/cxl-device-utils.c | 293 +++++++++++++++++++
> > > hw/cxl/cxl-mailbox-utils.c | 139 +++++++++
> > > hw/cxl/meson.build | 5 +
> > > hw/i386/acpi-build.c | 87 +++++-
> > > hw/i386/microvm.c | 1 +
> > > hw/i386/pc.c | 2 +
> > > hw/mem/Kconfig | 5 +
> > > hw/mem/cxl_type3.c | 334 ++++++++++++++++++++++
> > > hw/mem/meson.build | 1 +
> > > hw/meson.build | 1 +
> > > hw/pci-bridge/Kconfig | 5 +
> > > hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++
> > > hw/pci-bridge/meson.build | 1 +
> > > hw/pci-bridge/pci_expander_bridge.c | 209 +++++++++++++-
> > > hw/pci-bridge/pcie_root_port.c | 6 +-
> > > hw/pci/pci.c | 32 ++-
> > > hw/pci/pcie.c | 30 ++
> > > hw/ppc/spapr.c | 2 +
> > > include/hw/acpi/cxl.h | 27 ++
> > > include/hw/boards.h | 2 +
> > > include/hw/cxl/cxl.h | 30 ++
> > > include/hw/cxl/cxl_component.h | 181 ++++++++++++
> > > include/hw/cxl/cxl_device.h | 199 +++++++++++++
> > > include/hw/cxl/cxl_pci.h | 155 ++++++++++
> > > include/hw/pci/pci.h | 15 +
> > > include/hw/pci/pci_bridge.h | 25 ++
> > > include/hw/pci/pci_bus.h | 8 +
> > > include/hw/pci/pci_ids.h | 1 +
> > > include/standard-headers/linux/pci_regs.h | 1 +
> > > monitor/hmp-cmds.c | 15 +
> > > qapi/machine.json | 1 +
> > > tests/qtest/cxl-test.c | 93 ++++++
> > > tests/qtest/meson.build | 4 +
> > > 43 files changed, 2547 insertions(+), 30 deletions(-)
> > > create mode 100644 hw/acpi/cxl.c
> > > create mode 100644 hw/cxl/Kconfig
> > > create mode 100644 hw/cxl/cxl-component-utils.c
> > > create mode 100644 hw/cxl/cxl-device-utils.c
> > > create mode 100644 hw/cxl/cxl-mailbox-utils.c
> > > create mode 100644 hw/cxl/meson.build
> > > create mode 100644 hw/mem/cxl_type3.c
> > > create mode 100644 hw/pci-bridge/cxl_root_port.c
> > > create mode 100644 include/hw/acpi/cxl.h
> > > create mode 100644 include/hw/cxl/cxl.h
> > > create mode 100644 include/hw/cxl/cxl_component.h
> > > create mode 100644 include/hw/cxl/cxl_device.h
> > > create mode 100644 include/hw/cxl/cxl_pci.h
> > > create mode 100644 tests/qtest/cxl-test.c
> > >
> >
next prev parent reply other threads:[~2020-11-17 14:11 UTC|newest]
Thread overview: 66+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-11 5:46 [RFC PATCH 00/25] Introduce CXL 2.0 Emulation Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 01/25] Temp: Add the PCI_EXT_ID_DVSEC definition to the qemu pci_regs.h copy Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 02/25] hw/pci/cxl: Add a CXL component type (interface) Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 03/25] hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) Ben Widawsky
2020-11-16 12:03 ` Jonathan Cameron
2020-11-16 19:19 ` Ben Widawsky
2020-11-17 12:29 ` Jonathan Cameron
2020-11-24 23:09 ` Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 04/25] hw/cxl/device: Introduce a CXL device (8.2.8) Ben Widawsky
2020-11-16 13:07 ` Jonathan Cameron
2020-11-16 21:11 ` Ben Widawsky
2020-11-17 14:21 ` Jonathan Cameron
2020-11-11 5:47 ` [RFC PATCH 05/25] hw/cxl/device: Implement the CAP array (8.2.8.1-2) Ben Widawsky
2020-11-16 13:11 ` Jonathan Cameron
2020-11-16 18:08 ` Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 06/25] hw/cxl/device: Add device status (8.2.8.3) Ben Widawsky
2020-11-16 13:16 ` Jonathan Cameron
2020-11-16 21:18 ` Ben Widawsky
2020-11-17 14:24 ` Jonathan Cameron
2020-11-11 5:47 ` [RFC PATCH 07/25] hw/cxl/device: Implement basic mailbox (8.2.8.4) Ben Widawsky
2020-11-16 13:46 ` Jonathan Cameron
2020-11-16 21:42 ` Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 08/25] hw/cxl/device: Add memory devices (8.2.8.5) Ben Widawsky
2020-11-16 16:37 ` Jonathan Cameron
2020-11-16 21:45 ` Ben Widawsky
2020-11-17 14:31 ` Jonathan Cameron
2020-11-11 5:47 ` [RFC PATCH 09/25] hw/pxb: Use a type for realizing expanders Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 10/25] hw/pci/cxl: Create a CXL bus type Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 11/25] hw/pxb: Allow creation of a CXL PXB (host bridge) Ben Widawsky
2020-11-16 16:44 ` Jonathan Cameron
2020-11-16 22:01 ` Ben Widawsky
2020-11-17 14:33 ` Jonathan Cameron
2020-11-11 5:47 ` [RFC PATCH 12/25] acpi/pci: Consolidate host bridge setup Ben Widawsky
2020-11-12 17:46 ` Ben Widawsky
2020-11-16 16:45 ` Jonathan Cameron
2020-11-11 5:47 ` [RFC PATCH 13/25] hw/pci: Plumb _UID through host bridges Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 14/25] hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 15/25] acpi/pxb/cxl: Reserve host bridge MMIO Ben Widawsky
2020-11-16 16:54 ` Jonathan Cameron
2020-11-11 5:47 ` [RFC PATCH 16/25] hw/pxb/cxl: Add "windows" for host bridges Ben Widawsky
2020-11-13 0:49 ` Ben Widawsky
2020-11-23 19:12 ` Philippe Mathieu-Daudé
2020-11-11 5:47 ` [RFC PATCH 17/25] hw/cxl/rp: Add a root port Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 18/25] hw/cxl/device: Add a memory device (8.2.8.5) Ben Widawsky
2020-11-12 18:37 ` Eric Blake
2020-11-13 7:47 ` Markus Armbruster
2020-11-25 16:53 ` Ben Widawsky
2020-11-26 6:36 ` Markus Armbruster
2020-11-30 17:07 ` Ben Widawsky
2020-12-01 17:06 ` Markus Armbruster
2020-11-11 5:47 ` [RFC PATCH 19/25] hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 20/25] acpi/cxl: Add _OSC implementation (9.14.2) Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 21/25] acpi/cxl: Introduce a compat-driver UUID for CXL _OSC Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 22/25] acpi/cxl: Create the CEDT (9.14.1) Ben Widawsky
2020-11-16 17:15 ` Jonathan Cameron
2020-11-16 22:05 ` Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 23/25] Temp: acpi/cxl: Add ACPI0017 (CEDT awareness) Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 24/25] WIP: i386/cxl: Initialize a host bridge Ben Widawsky
2020-11-11 5:47 ` [RFC PATCH 25/25] qtest/cxl: Add very basic sanity tests Ben Widawsky
2020-11-16 17:21 ` [RFC PATCH 00/25] Introduce CXL 2.0 Emulation Jonathan Cameron
2020-11-16 18:06 ` Ben Widawsky
2020-11-17 14:09 ` Jonathan Cameron [this message]
2020-11-25 18:29 ` Ben Widawsky
2020-12-04 14:27 ` Daniel P. Berrangé
2020-12-04 5:08 Chris Browy
2020-12-04 5:55 ` Dan Williams
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201117140914.000076d1@Huawei.com \
--to=jonathan.cameron@huawei.com \
--cc=ben.widawsky@intel.com \
--cc=dan.j.williams@intel.com \
--cc=qemu-devel@nongnu.org \
--cc=vishal.l.verma@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).