* [PATCH v5 01/43] hw/pci/cxl: Add a CXL component type (interface)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
@ 2022-02-02 14:09 ` Jonathan Cameron
2022-02-02 14:09 ` [PATCH v5 02/43] hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) Jonathan Cameron
` (43 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:09 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
A CXL component is a hardware entity that implements CXL component
registers from the CXL 2.0 spec (8.2.3). Currently these represent 3
general types.
1. Host Bridge
2. Ports (root, upstream, downstream)
3. Devices (memory, other)
A CXL component can be conceptually thought of as a PCIe device with
extra functionality when enumerated and enabled. For this reason, CXL
does here, and will continue to add on to existing PCI code paths.
Host bridges will typically need to be handled specially and so they can
implement this newly introduced interface or not. All other components
should implement this interface. Implementing this interface allows the
core PCI code to treat these devices as special where appropriate.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
hw/pci/pci.c | 10 ++++++++++
include/hw/pci/pci.h | 8 ++++++++
2 files changed, 18 insertions(+)
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 5d30f9ca60..474ea98c1d 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -201,6 +201,11 @@ static const TypeInfo pci_bus_info = {
.class_init = pci_bus_class_init,
};
+static const TypeInfo cxl_interface_info = {
+ .name = INTERFACE_CXL_DEVICE,
+ .parent = TYPE_INTERFACE,
+};
+
static const TypeInfo pcie_interface_info = {
.name = INTERFACE_PCIE_DEVICE,
.parent = TYPE_INTERFACE,
@@ -2128,6 +2133,10 @@ static void pci_qdev_realize(DeviceState *qdev, Error **errp)
pci_dev->cap_present |= QEMU_PCI_CAP_EXPRESS;
}
+ if (object_class_dynamic_cast(klass, INTERFACE_CXL_DEVICE)) {
+ pci_dev->cap_present |= QEMU_PCIE_CAP_CXL;
+ }
+
pci_dev = do_pci_register_device(pci_dev,
object_get_typename(OBJECT(qdev)),
pci_dev->devfn, errp);
@@ -2884,6 +2893,7 @@ static void pci_register_types(void)
type_register_static(&pci_bus_info);
type_register_static(&pcie_bus_info);
type_register_static(&conventional_pci_interface_info);
+ type_register_static(&cxl_interface_info);
type_register_static(&pcie_interface_info);
type_register_static(&pci_device_type_info);
}
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index 023abc0f79..908896ebe8 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -195,6 +195,8 @@ enum {
QEMU_PCIE_LNKSTA_DLLLA = (1 << QEMU_PCIE_LNKSTA_DLLLA_BITNR),
#define QEMU_PCIE_EXTCAP_INIT_BITNR 9
QEMU_PCIE_EXTCAP_INIT = (1 << QEMU_PCIE_EXTCAP_INIT_BITNR),
+#define QEMU_PCIE_CXL_BITNR 10
+ QEMU_PCIE_CAP_CXL = (1 << QEMU_PCIE_CXL_BITNR),
};
#define TYPE_PCI_DEVICE "pci-device"
@@ -202,6 +204,12 @@ typedef struct PCIDeviceClass PCIDeviceClass;
DECLARE_OBJ_CHECKERS(PCIDevice, PCIDeviceClass,
PCI_DEVICE, TYPE_PCI_DEVICE)
+/*
+ * Implemented by devices that can be plugged on CXL buses. In the spec, this is
+ * actually a "CXL Component, but we name it device to match the PCI naming.
+ */
+#define INTERFACE_CXL_DEVICE "cxl-device"
+
/* Implemented by devices that can be plugged on PCI Express buses */
#define INTERFACE_PCIE_DEVICE "pci-express-device"
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 02/43] hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
2022-02-02 14:09 ` [PATCH v5 01/43] hw/pci/cxl: Add a CXL component type (interface) Jonathan Cameron
@ 2022-02-02 14:09 ` Jonathan Cameron
2022-02-02 14:09 ` [PATCH v5 03/43] MAINTAINERS: Add entry for Compute Express Link Emulation Jonathan Cameron
` (42 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:09 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
A CXL 2.0 component is any entity in the CXL topology. All components
have a analogous function in PCIe. Except for the CXL host bridge, all
have a PCIe config space that is accessible via the common PCIe
mechanisms. CXL components are enumerated via DVSEC fields in the
extended PCIe header space. CXL components will minimally implement some
subset of CXL.mem and CXL.cache registers defined in 8.2.5 of the CXL
2.0 specification. Two headers and a utility library are introduced to
support the minimum functionality needed to enumerate components.
The cxl_pci header manages bits associated with PCI, specifically the
DVSEC and related fields. The cxl_component.h variant has data
structures and APIs that are useful for drivers implementing any of the
CXL 2.0 components. The library takes care of making use of the DVSEC
bits and the CXL.[mem|cache] registers. Per spec, the registers are
little endian.
None of the mechanisms required to enumerate a CXL capable hostbridge
are introduced at this point.
Note that the CXL.mem and CXL.cache registers used are always 4B wide.
It's possible in the future that this constraint will not hold.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
v5:
Alex pointed out the odd handling of 8 byte accesses.
That lead to discovering a kernel bug around access to
the Cap Array Header for which a fix is no on the linux-cxl@vger.kernel.org
list.
We don't currently implement any of the 8 byte registers, so for
now this logs UNIMP and read 0, write ignored.
hw/Kconfig | 1 +
hw/cxl/Kconfig | 3 +
hw/cxl/cxl-component-utils.c | 219 +++++++++++++++++++++++++++++++++
hw/cxl/meson.build | 4 +
hw/meson.build | 1 +
include/hw/cxl/cxl.h | 16 +++
include/hw/cxl/cxl_component.h | 196 +++++++++++++++++++++++++++++
include/hw/cxl/cxl_pci.h | 138 +++++++++++++++++++++
8 files changed, 578 insertions(+)
create mode 100644 hw/cxl/Kconfig
create mode 100644 hw/cxl/cxl-component-utils.c
create mode 100644 hw/cxl/meson.build
create mode 100644 include/hw/cxl/cxl.h
create mode 100644 include/hw/cxl/cxl_component.h
create mode 100644 include/hw/cxl/cxl_pci.h
diff --git a/hw/Kconfig b/hw/Kconfig
index ad20cce0a9..50e0952889 100644
--- a/hw/Kconfig
+++ b/hw/Kconfig
@@ -6,6 +6,7 @@ source audio/Kconfig
source block/Kconfig
source char/Kconfig
source core/Kconfig
+source cxl/Kconfig
source display/Kconfig
source dma/Kconfig
source gpio/Kconfig
diff --git a/hw/cxl/Kconfig b/hw/cxl/Kconfig
new file mode 100644
index 0000000000..8e67519b16
--- /dev/null
+++ b/hw/cxl/Kconfig
@@ -0,0 +1,3 @@
+config CXL
+ bool
+ default y if PCI_EXPRESS
diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
new file mode 100644
index 0000000000..07297b3bbe
--- /dev/null
+++ b/hw/cxl/cxl-component-utils.c
@@ -0,0 +1,219 @@
+/*
+ * CXL Utility library for components
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "hw/pci/pci.h"
+#include "hw/cxl/cxl.h"
+
+static uint64_t cxl_cache_mem_read_reg(void *opaque, hwaddr offset,
+ unsigned size)
+{
+ CXLComponentState *cxl_cstate = opaque;
+ ComponentRegisters *cregs = &cxl_cstate->crb;
+
+ if (size == 8) {
+ qemu_log_mask(LOG_UNIMP,
+ "CXL 8 byte cache mem registers not implemented\n");
+ return 0;
+ }
+
+ if (cregs->special_ops && cregs->special_ops->read) {
+ return cregs->special_ops->read(cxl_cstate, offset, size);
+ } else {
+ return cregs->cache_mem_registers[offset / 4];
+ }
+}
+
+static void cxl_cache_mem_write_reg(void *opaque, hwaddr offset, uint64_t value,
+ unsigned size)
+{
+ CXLComponentState *cxl_cstate = opaque;
+ ComponentRegisters *cregs = &cxl_cstate->crb;
+
+ if (size == 8) {
+ qemu_log_mask(LOG_UNIMP,
+ "CXL 8 byte cache mem registers not implemented\n");
+ return;
+ }
+ if (cregs->special_ops && cregs->special_ops->write) {
+ cregs->special_ops->write(cxl_cstate, offset, value, size);
+ } else {
+ cregs->cache_mem_registers[offset / 4] = value;
+ }
+}
+
+/*
+ * 8.2.3
+ * The access restrictions specified in Section 8.2.2 also apply to CXL 2.0
+ * Component Registers.
+ *
+ * 8.2.2
+ * • A 32 bit register shall be accessed as a 4 Bytes quantity. Partial
+ * reads are not permitted.
+ * • A 64 bit register shall be accessed as a 8 Bytes quantity. Partial
+ * reads are not permitted.
+ *
+ * As of the spec defined today, only 4 byte registers exist.
+ */
+static const MemoryRegionOps cache_mem_ops = {
+ .read = cxl_cache_mem_read_reg,
+ .write = cxl_cache_mem_write_reg,
+ .endianness = DEVICE_LITTLE_ENDIAN,
+ .valid = {
+ .min_access_size = 4,
+ .max_access_size = 8,
+ .unaligned = false,
+ },
+ .impl = {
+ .min_access_size = 4,
+ .max_access_size = 8,
+ },
+};
+
+void cxl_component_register_block_init(Object *obj,
+ CXLComponentState *cxl_cstate,
+ const char *type)
+{
+ ComponentRegisters *cregs = &cxl_cstate->crb;
+
+ memory_region_init(&cregs->component_registers, obj, type,
+ CXL2_COMPONENT_BLOCK_SIZE);
+
+ /* io registers controls link which we don't care about in QEMU */
+ memory_region_init_io(&cregs->io, obj, NULL, cregs, ".io",
+ CXL2_COMPONENT_IO_REGION_SIZE);
+ memory_region_init_io(&cregs->cache_mem, obj, &cache_mem_ops, cregs,
+ ".cache_mem", CXL2_COMPONENT_CM_REGION_SIZE);
+
+ memory_region_add_subregion(&cregs->component_registers, 0, &cregs->io);
+ memory_region_add_subregion(&cregs->component_registers,
+ CXL2_COMPONENT_IO_REGION_SIZE,
+ &cregs->cache_mem);
+}
+
+static void ras_init_common(uint32_t *reg_state)
+{
+ reg_state[R_CXL_RAS_UNC_ERR_STATUS] = 0;
+ reg_state[R_CXL_RAS_UNC_ERR_MASK] = 0x1cfff;
+ reg_state[R_CXL_RAS_UNC_ERR_SEVERITY] = 0x1cfff;
+ reg_state[R_CXL_RAS_COR_ERR_STATUS] = 0;
+ reg_state[R_CXL_RAS_COR_ERR_MASK] = 0x3f;
+
+ /* CXL switches and devices must set */
+ reg_state[R_CXL_RAS_ERR_CAP_CTRL] = 0;
+}
+
+static void hdm_init_common(uint32_t *reg_state)
+{
+ ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, DECODER_COUNT, 0);
+ ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, TARGET_COUNT, 1);
+ ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_GLOBAL_CONTROL,
+ HDM_DECODER_ENABLE, 0);
+}
+
+void cxl_component_register_init_common(uint32_t *reg_state, enum reg_type type)
+{
+ int caps = 0;
+ switch (type) {
+ case CXL2_DOWNSTREAM_PORT:
+ case CXL2_DEVICE:
+ /* CAP, RAS, Link */
+ caps = 2;
+ break;
+ case CXL2_UPSTREAM_PORT:
+ case CXL2_TYPE3_DEVICE:
+ case CXL2_LOGICAL_DEVICE:
+ /* + HDM */
+ caps = 3;
+ break;
+ case CXL2_ROOT_PORT:
+ /* + Extended Security, + Snoop */
+ caps = 5;
+ break;
+ default:
+ abort();
+ }
+
+ memset(reg_state, 0, CXL2_COMPONENT_CM_REGION_SIZE);
+
+ /* CXL Capability Header Register */
+ ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ID, 1);
+ ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, VERSION, 1);
+ ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, CACHE_MEM_VERSION, 1);
+ ARRAY_FIELD_DP32(reg_state, CXL_CAPABILITY_HEADER, ARRAY_SIZE, caps);
+
+
+#define init_cap_reg(reg, id, version) \
+ _Static_assert(CXL_##reg##_REGISTERS_OFFSET != 0, "Invalid cap offset\n");\
+ do { \
+ int which = R_CXL_##reg##_CAPABILITY_HEADER; \
+ reg_state[which] = FIELD_DP32(reg_state[which], \
+ CXL_##reg##_CAPABILITY_HEADER, ID, id); \
+ reg_state[which] = \
+ FIELD_DP32(reg_state[which], CXL_##reg##_CAPABILITY_HEADER, \
+ VERSION, version); \
+ reg_state[which] = \
+ FIELD_DP32(reg_state[which], CXL_##reg##_CAPABILITY_HEADER, PTR, \
+ CXL_##reg##_REGISTERS_OFFSET); \
+ } while (0)
+
+ init_cap_reg(RAS, 2, 1);
+ ras_init_common(reg_state);
+
+ init_cap_reg(LINK, 4, 2);
+
+ if (caps < 3) {
+ return;
+ }
+
+ init_cap_reg(HDM, 5, 1);
+ hdm_init_common(reg_state);
+
+ if (caps < 5) {
+ return;
+ }
+
+ init_cap_reg(EXTSEC, 6, 1);
+ init_cap_reg(SNOOP, 8, 1);
+
+#undef init_cap_reg
+}
+
+/*
+ * Helper to creates a DVSEC header for a CXL entity. The caller is responsible
+ * for tracking the valid offset.
+ *
+ * This function will build the DVSEC header on behalf of the caller and then
+ * copy in the remaining data for the vendor specific bits.
+ */
+void cxl_component_create_dvsec(CXLComponentState *cxl, uint16_t length,
+ uint16_t type, uint8_t rev, uint8_t *body)
+{
+ PCIDevice *pdev = cxl->pdev;
+ uint16_t offset = cxl->dvsec_offset;
+
+ assert(offset >= PCI_CFG_SPACE_SIZE &&
+ ((offset + length) < PCI_CFG_SPACE_EXP_SIZE));
+ assert((length & 0xf000) == 0);
+ assert((rev & ~0xf) == 0);
+
+ /* Create the DVSEC in the MCFG space */
+ pcie_add_capability(pdev, PCI_EXT_CAP_ID_DVSEC, 1, offset, length);
+ pci_set_long(pdev->config + offset + PCIE_DVSEC_HEADER1_OFFSET,
+ (length << 20) | (rev << 16) | CXL_VENDOR_ID);
+ pci_set_word(pdev->config + offset + PCIE_DVSEC_ID_OFFSET, type);
+ memcpy(pdev->config + offset + sizeof(struct dvsec_header),
+ body + sizeof(struct dvsec_header),
+ length - sizeof(struct dvsec_header));
+
+ /* Update state for future DVSEC additions */
+ range_init_nofail(&cxl->dvsecs[type], cxl->dvsec_offset, length);
+ cxl->dvsec_offset += length;
+}
diff --git a/hw/cxl/meson.build b/hw/cxl/meson.build
new file mode 100644
index 0000000000..3231b5de1e
--- /dev/null
+++ b/hw/cxl/meson.build
@@ -0,0 +1,4 @@
+softmmu_ss.add(when: 'CONFIG_CXL',
+ if_true: files(
+ 'cxl-component-utils.c',
+ ))
diff --git a/hw/meson.build b/hw/meson.build
index b3366c888e..9992c5101e 100644
--- a/hw/meson.build
+++ b/hw/meson.build
@@ -6,6 +6,7 @@ subdir('block')
subdir('char')
subdir('core')
subdir('cpu')
+subdir('cxl')
subdir('display')
subdir('dma')
subdir('gpio')
diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
new file mode 100644
index 0000000000..8c738c7a2b
--- /dev/null
+++ b/include/hw/cxl/cxl.h
@@ -0,0 +1,16 @@
+/*
+ * QEMU CXL Support
+ *
+ * Copyright (c) 2020 Intel
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#ifndef CXL_H
+#define CXL_H
+
+#include "cxl_pci.h"
+#include "cxl_component.h"
+
+#endif
diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
new file mode 100644
index 0000000000..33aeab9b99
--- /dev/null
+++ b/include/hw/cxl/cxl_component.h
@@ -0,0 +1,196 @@
+/*
+ * QEMU CXL Component
+ *
+ * Copyright (c) 2020 Intel
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#ifndef CXL_COMPONENT_H
+#define CXL_COMPONENT_H
+
+/* CXL 2.0 - 8.2.4 */
+#define CXL2_COMPONENT_IO_REGION_SIZE 0x1000
+#define CXL2_COMPONENT_CM_REGION_SIZE 0x1000
+#define CXL2_COMPONENT_BLOCK_SIZE 0x10000
+
+#include "qemu/range.h"
+#include "qemu/typedefs.h"
+#include "hw/register.h"
+
+enum reg_type {
+ CXL2_DEVICE,
+ CXL2_TYPE3_DEVICE,
+ CXL2_LOGICAL_DEVICE,
+ CXL2_ROOT_PORT,
+ CXL2_UPSTREAM_PORT,
+ CXL2_DOWNSTREAM_PORT
+};
+
+/*
+ * Capability registers are defined at the top of the CXL.cache/mem region and
+ * are packed. For our purposes we will always define the caps in the same
+ * order.
+ * CXL 2.0 - 8.2.5 Table 142 for details.
+ */
+
+/* CXL 2.0 - 8.2.5.1 */
+REG32(CXL_CAPABILITY_HEADER, 0)
+ FIELD(CXL_CAPABILITY_HEADER, ID, 0, 16)
+ FIELD(CXL_CAPABILITY_HEADER, VERSION, 16, 4)
+ FIELD(CXL_CAPABILITY_HEADER, CACHE_MEM_VERSION, 20, 4)
+ FIELD(CXL_CAPABILITY_HEADER, ARRAY_SIZE, 24, 8)
+
+#define CXLx_CAPABILITY_HEADER(type, offset) \
+ REG32(CXL_##type##_CAPABILITY_HEADER, offset) \
+ FIELD(CXL_##type##_CAPABILITY_HEADER, ID, 0, 16) \
+ FIELD(CXL_##type##_CAPABILITY_HEADER, VERSION, 16, 4) \
+ FIELD(CXL_##type##_CAPABILITY_HEADER, PTR, 20, 12)
+CXLx_CAPABILITY_HEADER(RAS, 0x4)
+CXLx_CAPABILITY_HEADER(LINK, 0x8)
+CXLx_CAPABILITY_HEADER(HDM, 0xc)
+CXLx_CAPABILITY_HEADER(EXTSEC, 0x10)
+CXLx_CAPABILITY_HEADER(SNOOP, 0x14)
+
+/*
+ * Capability structures contain the actual registers that the CXL component
+ * implements. Some of these are specific to certain types of components, but
+ * this implementation leaves enough space regardless.
+ */
+/* 8.2.5.9 - CXL RAS Capability Structure */
+
+/* Give ample space for caps before this */
+#define CXL_RAS_REGISTERS_OFFSET 0x80
+#define CXL_RAS_REGISTERS_SIZE 0x58
+REG32(CXL_RAS_UNC_ERR_STATUS, CXL_RAS_REGISTERS_OFFSET)
+REG32(CXL_RAS_UNC_ERR_MASK, CXL_RAS_REGISTERS_OFFSET + 0x4)
+REG32(CXL_RAS_UNC_ERR_SEVERITY, CXL_RAS_REGISTERS_OFFSET + 0x8)
+REG32(CXL_RAS_COR_ERR_STATUS, CXL_RAS_REGISTERS_OFFSET + 0xc)
+REG32(CXL_RAS_COR_ERR_MASK, CXL_RAS_REGISTERS_OFFSET + 0x10)
+REG32(CXL_RAS_ERR_CAP_CTRL, CXL_RAS_REGISTERS_OFFSET + 0x14)
+/* Offset 0x18 - 0x58 reserved for RAS logs */
+
+/* 8.2.5.10 - CXL Security Capability Structure */
+#define CXL_SEC_REGISTERS_OFFSET \
+ (CXL_RAS_REGISTERS_OFFSET + CXL_RAS_REGISTERS_SIZE)
+#define CXL_SEC_REGISTERS_SIZE 0 /* We don't implement 1.1 downstream ports */
+
+/* 8.2.5.11 - CXL Link Capability Structure */
+#define CXL_LINK_REGISTERS_OFFSET \
+ (CXL_SEC_REGISTERS_OFFSET + CXL_SEC_REGISTERS_SIZE)
+#define CXL_LINK_REGISTERS_SIZE 0x38
+
+/* 8.2.5.12 - CXL HDM Decoder Capability Structure */
+#define HDM_DECODE_MAX 10 /* 8.2.5.12.1 */
+#define CXL_HDM_REGISTERS_OFFSET \
+ (CXL_LINK_REGISTERS_OFFSET + CXL_LINK_REGISTERS_SIZE)
+#define CXL_HDM_REGISTERS_SIZE (0x20 + HDM_DECODE_MAX + 10)
+#define HDM_DECODER_INIT(n) \
+ REG32(CXL_HDM_DECODER##n##_BASE_LO, \
+ CXL_HDM_REGISTERS_OFFSET + (0x20 * n) + 0x10) \
+ FIELD(CXL_HDM_DECODER##n##_BASE_LO, L, 28, 4) \
+ REG32(CXL_HDM_DECODER##n##_BASE_HI, \
+ CXL_HDM_REGISTERS_OFFSET + (0x20 * n) + 0x14) \
+ REG32(CXL_HDM_DECODER##n##_SIZE_LO, \
+ CXL_HDM_REGISTERS_OFFSET + (0x20 * n) + 0x18) \
+ REG32(CXL_HDM_DECODER##n##_SIZE_HI, \
+ CXL_HDM_REGISTERS_OFFSET + (0x20 * n) + 0x1C) \
+ REG32(CXL_HDM_DECODER##n##_CTRL, \
+ CXL_HDM_REGISTERS_OFFSET + (0x20 * n) + 0x20) \
+ FIELD(CXL_HDM_DECODER##n##_CTRL, IG, 0, 4) \
+ FIELD(CXL_HDM_DECODER##n##_CTRL, IW, 4, 4) \
+ FIELD(CXL_HDM_DECODER##n##_CTRL, LOCK_ON_COMMIT, 8, 1) \
+ FIELD(CXL_HDM_DECODER##n##_CTRL, COMMIT, 9, 1) \
+ FIELD(CXL_HDM_DECODER##n##_CTRL, COMMITTED, 10, 1) \
+ FIELD(CXL_HDM_DECODER##n##_CTRL, ERR, 11, 1) \
+ FIELD(CXL_HDM_DECODER##n##_CTRL, TYPE, 12, 1) \
+ REG32(CXL_HDM_DECODER##n##_TARGET_LIST_LO, \
+ CXL_HDM_REGISTERS_OFFSET + (0x20 * n) + 0x24) \
+ REG32(CXL_HDM_DECODER##n##_TARGET_LIST_HI, \
+ CXL_HDM_REGISTERS_OFFSET + (0x20 * n) + 0x28)
+
+REG32(CXL_HDM_DECODER_CAPABILITY, CXL_HDM_REGISTERS_OFFSET)
+ FIELD(CXL_HDM_DECODER_CAPABILITY, DECODER_COUNT, 0, 4)
+ FIELD(CXL_HDM_DECODER_CAPABILITY, TARGET_COUNT, 4, 4)
+ FIELD(CXL_HDM_DECODER_CAPABILITY, INTERLEAVE_256B, 8, 1)
+ FIELD(CXL_HDM_DECODER_CAPABILITY, INTELEAVE_4K, 9, 1)
+ FIELD(CXL_HDM_DECODER_CAPABILITY, POISON_ON_ERR_CAP, 10, 1)
+REG32(CXL_HDM_DECODER_GLOBAL_CONTROL, CXL_HDM_REGISTERS_OFFSET + 4)
+ FIELD(CXL_HDM_DECODER_GLOBAL_CONTROL, POISON_ON_ERR_EN, 0, 1)
+ FIELD(CXL_HDM_DECODER_GLOBAL_CONTROL, HDM_DECODER_ENABLE, 1, 1)
+
+HDM_DECODER_INIT(0);
+
+/* 8.2.5.13 - CXL Extended Security Capability Structure (Root complex only) */
+#define EXTSEC_ENTRY_MAX 256
+#define CXL_EXTSEC_REGISTERS_OFFSET \
+ (CXL_HDM_REGISTERS_OFFSET + CXL_HDM_REGISTERS_SIZE)
+#define CXL_EXTSEC_REGISTERS_SIZE (8 * EXTSEC_ENTRY_MAX + 4)
+
+/* 8.2.5.14 - CXL IDE Capability Structure */
+#define CXL_IDE_REGISTERS_OFFSET \
+ (CXL_EXTSEC_REGISTERS_OFFSET + CXL_EXTSEC_REGISTERS_SIZE)
+#define CXL_IDE_REGISTERS_SIZE 0x20
+
+/* 8.2.5.15 - CXL Snoop Filter Capability Structure */
+#define CXL_SNOOP_REGISTERS_OFFSET \
+ (CXL_IDE_REGISTERS_OFFSET + CXL_IDE_REGISTERS_SIZE)
+#define CXL_SNOOP_REGISTERS_SIZE 0x8
+
+_Static_assert((CXL_SNOOP_REGISTERS_OFFSET + CXL_SNOOP_REGISTERS_SIZE) < 0x1000,
+ "No space for registers");
+
+typedef struct component_registers {
+ /*
+ * Main memory region to be registered with QEMU core.
+ */
+ MemoryRegion component_registers;
+
+ /*
+ * 8.2.4 Table 141:
+ * 0x0000 - 0x0fff CXL.io registers
+ * 0x1000 - 0x1fff CXL.cache and CXL.mem
+ * 0x2000 - 0xdfff Implementation specific
+ * 0xe000 - 0xe3ff CXL ARB/MUX registers
+ * 0xe400 - 0xffff RSVD
+ */
+ uint32_t io_registers[CXL2_COMPONENT_IO_REGION_SIZE >> 2];
+ MemoryRegion io;
+
+ uint32_t cache_mem_registers[CXL2_COMPONENT_CM_REGION_SIZE >> 2];
+ MemoryRegion cache_mem;
+
+ MemoryRegion impl_specific;
+ MemoryRegion arb_mux;
+ MemoryRegion rsvd;
+
+ /* special_ops is used for any component that needs any specific handling */
+ MemoryRegionOps *special_ops;
+} ComponentRegisters;
+
+/*
+ * A CXL component represents all entities in a CXL hierarchy. This includes,
+ * host bridges, root ports, upstream/downstream switch ports, and devices
+ */
+typedef struct cxl_component {
+ ComponentRegisters crb;
+ union {
+ struct {
+ Range dvsecs[CXL20_MAX_DVSEC];
+ uint16_t dvsec_offset;
+ struct PCIDevice *pdev;
+ };
+ };
+} CXLComponentState;
+
+void cxl_component_register_block_init(Object *obj,
+ CXLComponentState *cxl_cstate,
+ const char *type);
+void cxl_component_register_init_common(uint32_t *reg_state,
+ enum reg_type type);
+
+void cxl_component_create_dvsec(CXLComponentState *cxl_cstate, uint16_t length,
+ uint16_t type, uint8_t rev, uint8_t *body);
+
+#endif
diff --git a/include/hw/cxl/cxl_pci.h b/include/hw/cxl/cxl_pci.h
new file mode 100644
index 0000000000..40c7329afe
--- /dev/null
+++ b/include/hw/cxl/cxl_pci.h
@@ -0,0 +1,138 @@
+/*
+ * QEMU CXL PCI interfaces
+ *
+ * Copyright (c) 2020 Intel
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#ifndef CXL_PCI_H
+#define CXL_PCI_H
+
+#include "hw/pci/pci.h"
+#include "hw/pci/pcie.h"
+
+#define CXL_VENDOR_ID 0x1e98
+
+#define PCIE_DVSEC_HEADER1_OFFSET 0x4 /* Offset from start of extend cap */
+#define PCIE_DVSEC_ID_OFFSET 0x8
+
+#define PCIE_CXL_DEVICE_DVSEC_LENGTH 0x38
+#define PCIE_CXL1_DEVICE_DVSEC_REVID 0
+#define PCIE_CXL2_DEVICE_DVSEC_REVID 1
+
+#define EXTENSIONS_PORT_DVSEC_LENGTH 0x28
+#define EXTENSIONS_PORT_DVSEC_REVID 0
+
+#define GPF_PORT_DVSEC_LENGTH 0x10
+#define GPF_PORT_DVSEC_REVID 0
+
+#define PCIE_FLEXBUS_PORT_DVSEC_LENGTH_2_0 0x14
+#define PCIE_FLEXBUS_PORT_DVSEC_REVID_2_0 1
+
+#define REG_LOC_DVSEC_LENGTH 0x24
+#define REG_LOC_DVSEC_REVID 0
+
+enum {
+ PCIE_CXL_DEVICE_DVSEC = 0,
+ NON_CXL_FUNCTION_MAP_DVSEC = 2,
+ EXTENSIONS_PORT_DVSEC = 3,
+ GPF_PORT_DVSEC = 4,
+ GPF_DEVICE_DVSEC = 5,
+ PCIE_FLEXBUS_PORT_DVSEC = 7,
+ REG_LOC_DVSEC = 8,
+ MLD_DVSEC = 9,
+ CXL20_MAX_DVSEC
+};
+
+struct dvsec_header {
+ uint32_t cap_hdr;
+ uint32_t dv_hdr1;
+ uint16_t dv_hdr2;
+} __attribute__((__packed__));
+_Static_assert(sizeof(struct dvsec_header) == 10,
+ "dvsec header size incorrect");
+
+/*
+ * CXL 2.0 devices must implement certain DVSEC IDs, and can [optionally]
+ * implement others.
+ *
+ * CXL 2.0 Device: 0, [2], 5, 8
+ * CXL 2.0 RP: 3, 4, 7, 8
+ * CXL 2.0 Upstream Port: [2], 7, 8
+ * CXL 2.0 Downstream Port: 3, 4, 7, 8
+ */
+
+/* CXL 2.0 - 8.1.5 (ID 0003) */
+struct cxl_dvsec_port_extensions {
+ struct dvsec_header hdr;
+ uint16_t status;
+ uint16_t control;
+ uint8_t alt_bus_base;
+ uint8_t alt_bus_limit;
+ uint16_t alt_memory_base;
+ uint16_t alt_memory_limit;
+ uint16_t alt_prefetch_base;
+ uint16_t alt_prefetch_limit;
+ uint32_t alt_prefetch_base_high;
+ uint32_t alt_prefetch_base_low;
+ uint32_t rcrb_base;
+ uint32_t rcrb_base_high;
+};
+_Static_assert(sizeof(struct cxl_dvsec_port_extensions) == 0x28,
+ "extensions dvsec port size incorrect");
+#define PORT_CONTROL_OFFSET 0xc
+#define PORT_CONTROL_UNMASK_SBR 1
+#define PORT_CONTROL_ALT_MEMID_EN 4
+
+/* CXL 2.0 - 8.1.6 GPF DVSEC (ID 0004) */
+struct cxl_dvsec_port_gpf {
+ struct dvsec_header hdr;
+ uint16_t rsvd;
+ uint16_t phase1_ctrl;
+ uint16_t phase2_ctrl;
+};
+_Static_assert(sizeof(struct cxl_dvsec_port_gpf) == 0x10,
+ "dvsec port GPF size incorrect");
+
+/* CXL 2.0 - 8.1.8/8.2.1.3 Flexbus DVSEC (ID 0007) */
+struct cxl_dvsec_port_flexbus {
+ struct dvsec_header hdr;
+ uint16_t cap;
+ uint16_t ctrl;
+ uint16_t status;
+ uint32_t rcvd_mod_ts_data_phase1;
+};
+_Static_assert(sizeof(struct cxl_dvsec_port_flexbus) == 0x14,
+ "dvsec port flexbus size incorrect");
+
+/* CXL 2.0 - 8.1.9 Register Locator DVSEC (ID 0008) */
+struct cxl_dvsec_register_locator {
+ struct dvsec_header hdr;
+ uint16_t rsvd;
+ uint32_t reg0_base_lo;
+ uint32_t reg0_base_hi;
+ uint32_t reg1_base_lo;
+ uint32_t reg1_base_hi;
+ uint32_t reg2_base_lo;
+ uint32_t reg2_base_hi;
+};
+_Static_assert(sizeof(struct cxl_dvsec_register_locator) == 0x24,
+ "dvsec register locator size incorrect");
+
+/* BAR Equivalence Indicator */
+#define BEI_BAR_10H 0
+#define BEI_BAR_14H 1
+#define BEI_BAR_18H 2
+#define BEI_BAR_1cH 3
+#define BEI_BAR_20H 4
+#define BEI_BAR_24H 5
+
+/* Register Block Identifier */
+#define RBI_EMPTY 0
+#define RBI_COMPONENT_REG (1 << 8)
+#define RBI_BAR_VIRT_ACL (2 << 8)
+#define RBI_CXL_DEVICE_REG (3 << 8)
+
+#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 03/43] MAINTAINERS: Add entry for Compute Express Link Emulation
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
2022-02-02 14:09 ` [PATCH v5 01/43] hw/pci/cxl: Add a CXL component type (interface) Jonathan Cameron
2022-02-02 14:09 ` [PATCH v5 02/43] hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5) Jonathan Cameron
@ 2022-02-02 14:09 ` Jonathan Cameron
2022-02-02 14:09 ` [PATCH v5 04/43] hw/cxl/device: Introduce a CXL device (8.2.8) Jonathan Cameron
` (41 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:09 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
The CXL emulation will be jointly maintained by Ben Widawsky
and Jonathan Cameron. Broken out as a separate patch
to improve visibility.
Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
MAINTAINERS | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index b43344fa98..930f04c6c2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2524,6 +2524,13 @@ F: qapi/block*.json
F: qapi/transaction.json
T: git https://repo.or.cz/qemu/armbru.git block-next
+Compute Express Link
+M: Ben Widawsky <ben.widawsky@intel.com>
+M: Jonathan Cameron <jonathan.cameron@huawei.com>
+S: Supported
+F: hw/cxl/
+F: include/hw/cxl/
+
Dirty Bitmaps
M: Eric Blake <eblake@redhat.com>
M: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 04/43] hw/cxl/device: Introduce a CXL device (8.2.8)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (2 preceding siblings ...)
2022-02-02 14:09 ` [PATCH v5 03/43] MAINTAINERS: Add entry for Compute Express Link Emulation Jonathan Cameron
@ 2022-02-02 14:09 ` Jonathan Cameron
2022-02-02 14:09 ` [PATCH v5 05/43] hw/cxl/device: Implement the CAP array (8.2.8.1-2) Jonathan Cameron
` (40 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:09 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
A CXL device is a type of CXL component. Conceptually, a CXL device
would be a leaf node in a CXL topology. From an emulation perspective,
CXL devices are the most complex and so the actual implementation is
reserved for discrete commits.
This new device type is specifically catered towards the eventual
implementation of a Type3 CXL.mem device, 8.2.8.5 in the CXL 2.0
specification.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
v5:
Include the impacts of the published CXL 2.0 Errata F4 which clarified
access permissions.
- Documentation updates.
- The 48 bit registers is gone.
include/hw/cxl/cxl.h | 1 +
include/hw/cxl/cxl_device.h | 165 ++++++++++++++++++++++++++++++++++++
2 files changed, 166 insertions(+)
create mode 100644 include/hw/cxl/cxl_device.h
diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
index 8c738c7a2b..b9d1ac3fad 100644
--- a/include/hw/cxl/cxl.h
+++ b/include/hw/cxl/cxl.h
@@ -12,5 +12,6 @@
#include "cxl_pci.h"
#include "cxl_component.h"
+#include "cxl_device.h"
#endif
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
new file mode 100644
index 0000000000..b2416e45bf
--- /dev/null
+++ b/include/hw/cxl/cxl_device.h
@@ -0,0 +1,165 @@
+/*
+ * QEMU CXL Devices
+ *
+ * Copyright (c) 2020 Intel
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#ifndef CXL_DEVICE_H
+#define CXL_DEVICE_H
+
+#include "hw/register.h"
+
+/*
+ * The following is how a CXL device's MMIO space is laid out. The only
+ * requirement from the spec is that the capabilities array and the capability
+ * headers start at offset 0 and are contiguously packed. The headers themselves
+ * provide offsets to the register fields. For this emulation, registers will
+ * start at offset 0x80 (m == 0x80). No secondary mailbox is implemented which
+ * means that n = m + sizeof(mailbox registers) + sizeof(device registers).
+ *
+ * This is roughly described in 8.2.8 Figure 138 of the CXL 2.0 spec.
+ *
+ * +---------------------------------+
+ * | |
+ * | Memory Device Registers |
+ * | |
+ * n + PAYLOAD_SIZE_MAX -----------------------------------
+ * ^ | |
+ * | | |
+ * | | |
+ * | | |
+ * | | |
+ * | | Mailbox Payload |
+ * | | |
+ * | | |
+ * | | |
+ * | -----------------------------------
+ * | | Mailbox Registers |
+ * | | |
+ * n -----------------------------------
+ * ^ | |
+ * | | Device Registers |
+ * | | |
+ * m ---------------------------------->
+ * ^ | Memory Device Capability Header|
+ * | -----------------------------------
+ * | | Mailbox Capability Header |
+ * | -------------- --------------------
+ * | | Device Capability Header |
+ * | -----------------------------------
+ * | | |
+ * | | |
+ * | | Device Cap Array[0..n] |
+ * | | |
+ * | | |
+ * | |
+ * 0 +---------------------------------+
+ *
+ */
+
+#define CXL_DEVICE_CAP_HDR1_OFFSET 0x10 /* Figure 138 */
+#define CXL_DEVICE_CAP_REG_SIZE 0x10 /* 8.2.8.2 */
+#define CXL_DEVICE_CAPS_MAX 4 /* 8.2.8.2.1 + 8.2.8.5 */
+
+#define CXL_DEVICE_REGISTERS_OFFSET 0x80 /* Read comment above */
+#define CXL_DEVICE_REGISTERS_LENGTH 0x8 /* 8.2.8.3.1 */
+
+#define CXL_MAILBOX_REGISTERS_OFFSET \
+ (CXL_DEVICE_REGISTERS_OFFSET + CXL_DEVICE_REGISTERS_LENGTH)
+#define CXL_MAILBOX_REGISTERS_SIZE 0x20 /* 8.2.8.4, Figure 139 */
+#define CXL_MAILBOX_PAYLOAD_SHIFT 11
+#define CXL_MAILBOX_MAX_PAYLOAD_SIZE (1 << CXL_MAILBOX_PAYLOAD_SHIFT)
+#define CXL_MAILBOX_REGISTERS_LENGTH \
+ (CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE)
+
+typedef struct cxl_device_state {
+ MemoryRegion device_registers;
+
+ /* mmio for device capabilities array - 8.2.8.2 */
+ MemoryRegion device;
+ MemoryRegion caps;
+
+ /* mmio for the mailbox registers 8.2.8.4 */
+ MemoryRegion mailbox;
+
+ /* memory region for persistent memory, HDM */
+ uint64_t pmem_size;
+} CXLDeviceState;
+
+/* Initialize the register block for a device */
+void cxl_device_register_block_init(Object *obj, CXLDeviceState *dev);
+
+/* Set up default values for the register block */
+void cxl_device_register_init_common(CXLDeviceState *dev);
+
+/*
+ * CXL 2.0 - 8.2.8.1 including errata F4
+ * Documented as a 128 bit register, but 64 bit accesses and the second
+ * 64 bits are currently reserved.
+ */
+REG64(CXL_DEV_CAP_ARRAY, 0) /* Documented as 128 bit register but 64 byte accesses */
+ FIELD(CXL_DEV_CAP_ARRAY, CAP_ID, 0, 16)
+ FIELD(CXL_DEV_CAP_ARRAY, CAP_VERSION, 16, 8)
+ FIELD(CXL_DEV_CAP_ARRAY, CAP_COUNT, 32, 16)
+
+/*
+ * Helper macro to initialize capability headers for CXL devices.
+ *
+ * In the 8.2.8.2, this is listed as a 128b register, but in 8.2.8, it says:
+ * > No registers defined in Section 8.2.8 are larger than 64-bits wide so that
+ * > is the maximum access size allowed for these registers. If this rule is not
+ * > followed, the behavior is undefined
+ *
+ * CXL 2.0 Errata F4 states futher that the layouts in the specification are
+ * shown as greater than 128 bits, but implementations are expected to
+ * use any size of access up to 64 bits.
+ *
+ * Here we've chosen to make it 4 dwords. The spec allows any pow2 multiple
+ * access to be used for a register up to 64 bits.
+ */
+#define CXL_DEVICE_CAPABILITY_HEADER_REGISTER(n, offset) \
+ REG32(CXL_DEV_##n##_CAP_HDR0, offset) \
+ FIELD(CXL_DEV_##n##_CAP_HDR0, CAP_ID, 0, 16) \
+ FIELD(CXL_DEV_##n##_CAP_HDR0, CAP_VERSION, 16, 8) \
+ REG32(CXL_DEV_##n##_CAP_HDR1, offset + 4) \
+ FIELD(CXL_DEV_##n##_CAP_HDR1, CAP_OFFSET, 0, 32) \
+ REG32(CXL_DEV_##n##_CAP_HDR2, offset + 8) \
+ FIELD(CXL_DEV_##n##_CAP_HDR2, CAP_LENGTH, 0, 32)
+
+CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)
+CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MAILBOX, CXL_DEVICE_CAP_HDR1_OFFSET + \
+ CXL_DEVICE_CAP_REG_SIZE)
+
+REG32(CXL_DEV_MAILBOX_CAP, 0)
+ FIELD(CXL_DEV_MAILBOX_CAP, PAYLOAD_SIZE, 0, 5)
+ FIELD(CXL_DEV_MAILBOX_CAP, INT_CAP, 5, 1)
+ FIELD(CXL_DEV_MAILBOX_CAP, BG_INT_CAP, 6, 1)
+ FIELD(CXL_DEV_MAILBOX_CAP, MSI_N, 7, 4)
+
+REG32(CXL_DEV_MAILBOX_CTRL, 4)
+ FIELD(CXL_DEV_MAILBOX_CTRL, DOORBELL, 0, 1)
+ FIELD(CXL_DEV_MAILBOX_CTRL, INT_EN, 1, 1)
+ FIELD(CXL_DEV_MAILBOX_CTRL, BG_INT_EN, 2, 1)
+
+REG64(CXL_DEV_MAILBOX_CMD, 8)
+ FIELD(CXL_DEV_MAILBOX_CMD, COMMAND, 0, 8)
+ FIELD(CXL_DEV_MAILBOX_CMD, COMMAND_SET, 8, 8)
+ FIELD(CXL_DEV_MAILBOX_CMD, LENGTH, 16, 20)
+
+REG64(CXL_DEV_MAILBOX_STS, 0x10)
+ FIELD(CXL_DEV_MAILBOX_STS, BG_OP, 0, 1)
+ FIELD(CXL_DEV_MAILBOX_STS, ERRNO, 32, 16)
+ FIELD(CXL_DEV_MAILBOX_STS, VENDOR_ERRNO, 48, 16)
+
+REG64(CXL_DEV_BG_CMD_STS, 0x18)
+ FIELD(CXL_DEV_BG_CMD_STS, BG, 0, 16)
+ FIELD(CXL_DEV_BG_CMD_STS, DONE, 16, 7)
+ FIELD(CXL_DEV_BG_CMD_STS, ERRNO, 32, 16)
+ FIELD(CXL_DEV_BG_CMD_STS, VENDOR_ERRNO, 48, 16)
+
+REG32(CXL_DEV_CMD_PAYLOAD, 0x20)
+
+#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 05/43] hw/cxl/device: Implement the CAP array (8.2.8.1-2)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (3 preceding siblings ...)
2022-02-02 14:09 ` [PATCH v5 04/43] hw/cxl/device: Introduce a CXL device (8.2.8) Jonathan Cameron
@ 2022-02-02 14:09 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 06/43] hw/cxl/device: Implement basic mailbox (8.2.8.4) Jonathan Cameron
` (39 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:09 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
This implements all device MMIO up to the first capability. That
includes the CXL Device Capabilities Array Register, as well as all of
the CXL Device Capability Header Registers. The latter are filled in as
they are implemented in the following patches.
Endianness and alignment are managed by softmmu memory core.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
hw/cxl/cxl-device-utils.c | 109 ++++++++++++++++++++++++++++++++++++
hw/cxl/meson.build | 1 +
include/hw/cxl/cxl_device.h | 31 +++++++++-
3 files changed, 140 insertions(+), 1 deletion(-)
create mode 100644 hw/cxl/cxl-device-utils.c
diff --git a/hw/cxl/cxl-device-utils.c b/hw/cxl/cxl-device-utils.c
new file mode 100644
index 0000000000..0895b9d78b
--- /dev/null
+++ b/hw/cxl/cxl-device-utils.c
@@ -0,0 +1,109 @@
+/*
+ * CXL Utility library for devices
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "hw/cxl/cxl.h"
+
+/*
+ * Device registers have no restrictions per the spec, and so fall back to the
+ * default memory mapped register rules in 8.2:
+ * Software shall use CXL.io Memory Read and Write to access memory mapped
+ * register defined in this section. Unless otherwise specified, software
+ * shall restrict the accesses width based on the following:
+ * • A 32 bit register shall be accessed as a 1 Byte, 2 Bytes or 4 Bytes
+ * quantity.
+ * • A 64 bit register shall be accessed as a 1 Byte, 2 Bytes, 4 Bytes or 8
+ * Bytes
+ * • The address shall be a multiple of the access width, e.g. when
+ * accessing a register as a 4 Byte quantity, the address shall be
+ * multiple of 4.
+ * • The accesses shall map to contiguous bytes.If these rules are not
+ * followed, the behavior is undefined
+ */
+
+static uint64_t caps_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+ CXLDeviceState *cxl_dstate = opaque;
+
+ if (size == 4) {
+ return cxl_dstate->caps_reg_state32[offset / 4];
+ } else {
+ return cxl_dstate->caps_reg_state64[offset / 8];
+ }
+}
+
+static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+ return 0;
+}
+
+static const MemoryRegionOps dev_ops = {
+ .read = dev_reg_read,
+ .write = NULL, /* status register is read only */
+ .endianness = DEVICE_LITTLE_ENDIAN,
+ .valid = {
+ .min_access_size = 1,
+ .max_access_size = 8,
+ .unaligned = false,
+ },
+ .impl = {
+ .min_access_size = 1,
+ .max_access_size = 8,
+ },
+};
+
+static const MemoryRegionOps caps_ops = {
+ .read = caps_reg_read,
+ .write = NULL, /* caps registers are read only */
+ .endianness = DEVICE_LITTLE_ENDIAN,
+ .valid = {
+ .min_access_size = 1,
+ .max_access_size = 8,
+ .unaligned = false,
+ },
+ .impl = {
+ .min_access_size = 4,
+ .max_access_size = 8,
+ },
+};
+
+void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
+{
+ /* This will be a BAR, so needs to be rounded up to pow2 for PCI spec */
+ memory_region_init(&cxl_dstate->device_registers, obj, "device-registers",
+ pow2ceil(CXL_MMIO_SIZE));
+
+ memory_region_init_io(&cxl_dstate->caps, obj, &caps_ops, cxl_dstate,
+ "cap-array", CXL_CAPS_SIZE);
+ memory_region_init_io(&cxl_dstate->device, obj, &dev_ops, cxl_dstate,
+ "device-status", CXL_DEVICE_REGISTERS_LENGTH);
+
+ memory_region_add_subregion(&cxl_dstate->device_registers, 0,
+ &cxl_dstate->caps);
+ memory_region_add_subregion(&cxl_dstate->device_registers,
+ CXL_DEVICE_REGISTERS_OFFSET,
+ &cxl_dstate->device);
+}
+
+static void device_reg_init_common(CXLDeviceState *cxl_dstate) { }
+
+void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
+{
+ uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64;
+ const int cap_count = 1;
+
+ /* CXL Device Capabilities Array Register */
+ ARRAY_FIELD_DP64(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
+ ARRAY_FIELD_DP64(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_VERSION, 1);
+ ARRAY_FIELD_DP64(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_COUNT, cap_count);
+
+ cxl_device_cap_init(cxl_dstate, DEVICE, 1);
+ device_reg_init_common(cxl_dstate);
+}
diff --git a/hw/cxl/meson.build b/hw/cxl/meson.build
index 3231b5de1e..dd7c6f8e5a 100644
--- a/hw/cxl/meson.build
+++ b/hw/cxl/meson.build
@@ -1,4 +1,5 @@
softmmu_ss.add(when: 'CONFIG_CXL',
if_true: files(
'cxl-component-utils.c',
+ 'cxl-device-utils.c',
))
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index b2416e45bf..1ac0dcd97e 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -63,6 +63,8 @@
#define CXL_DEVICE_CAP_HDR1_OFFSET 0x10 /* Figure 138 */
#define CXL_DEVICE_CAP_REG_SIZE 0x10 /* 8.2.8.2 */
#define CXL_DEVICE_CAPS_MAX 4 /* 8.2.8.2.1 + 8.2.8.5 */
+#define CXL_CAPS_SIZE \
+ (CXL_DEVICE_CAP_REG_SIZE * (CXL_DEVICE_CAPS_MAX + 1)) /* +1 for header */
#define CXL_DEVICE_REGISTERS_OFFSET 0x80 /* Read comment above */
#define CXL_DEVICE_REGISTERS_LENGTH 0x8 /* 8.2.8.3.1 */
@@ -75,12 +77,22 @@
#define CXL_MAILBOX_REGISTERS_LENGTH \
(CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE)
+#define CXL_MMIO_SIZE \
+ (CXL_DEVICE_CAP_REG_SIZE + CXL_DEVICE_REGISTERS_LENGTH + \
+ CXL_MAILBOX_REGISTERS_LENGTH)
+
typedef struct cxl_device_state {
MemoryRegion device_registers;
/* mmio for device capabilities array - 8.2.8.2 */
MemoryRegion device;
- MemoryRegion caps;
+ struct {
+ MemoryRegion caps;
+ union {
+ uint32_t caps_reg_state32[CXL_CAPS_SIZE / 4];
+ uint64_t caps_reg_state64[CXL_CAPS_SIZE / 8];
+ };
+ };
/* mmio for the mailbox registers 8.2.8.4 */
MemoryRegion mailbox;
@@ -133,6 +145,23 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)
CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MAILBOX, CXL_DEVICE_CAP_HDR1_OFFSET + \
CXL_DEVICE_CAP_REG_SIZE)
+#define cxl_device_cap_init(dstate, reg, cap_id) \
+ do { \
+ uint32_t *cap_hdrs = dstate->caps_reg_state32; \
+ int which = R_CXL_DEV_##reg##_CAP_HDR0; \
+ cap_hdrs[which] = \
+ FIELD_DP32(cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, \
+ CAP_ID, cap_id); \
+ cap_hdrs[which] = FIELD_DP32( \
+ cap_hdrs[which], CXL_DEV_##reg##_CAP_HDR0, CAP_VERSION, 1); \
+ cap_hdrs[which + 1] = \
+ FIELD_DP32(cap_hdrs[which + 1], CXL_DEV_##reg##_CAP_HDR1, \
+ CAP_OFFSET, CXL_##reg##_REGISTERS_OFFSET); \
+ cap_hdrs[which + 2] = \
+ FIELD_DP32(cap_hdrs[which + 2], CXL_DEV_##reg##_CAP_HDR2, \
+ CAP_LENGTH, CXL_##reg##_REGISTERS_LENGTH); \
+ } while (0)
+
REG32(CXL_DEV_MAILBOX_CAP, 0)
FIELD(CXL_DEV_MAILBOX_CAP, PAYLOAD_SIZE, 0, 5)
FIELD(CXL_DEV_MAILBOX_CAP, INT_CAP, 5, 1)
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 06/43] hw/cxl/device: Implement basic mailbox (8.2.8.4)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (4 preceding siblings ...)
2022-02-02 14:09 ` [PATCH v5 05/43] hw/cxl/device: Implement the CAP array (8.2.8.1-2) Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 07/43] hw/cxl/device: Add memory device utilities Jonathan Cameron
` (38 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
This is the beginning of implementing mailbox support for CXL 2.0
devices. The implementation recognizes when the doorbell is rung,
handles the command/payload, clears the doorbell while returning error
codes and data.
Generally the mailbox mechanism is designed to permit communication
between the host OS and the firmware running on the device. For our
purposes, we emulate both the firmware, implemented primarily in
cxl-mailbox-utils.c, and the hardware.
No commands are implemented yet.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
v5: Responses to Alex Bennée review.
- Fix the invalid write case Alex noted and return early.
- Drop the RCU_READ_LOCK as it was pointless and I don't think
we need to lock at all until we introduce other write paths
(second mailbox or background commands).
- Missing static on cel_uuid
- Documentation of where cel_uuid value comes from (the CXL spec)
- Drop a check that can't fail and hence get rid of a confusing
LOG_UNIMP.
- Move some small code rearrangement back to earlier patch.
- Reorder the mailbox handler code and update the docs, as first
part of removing many of the macros from this code.
- Upper case remaining defines + drop the define_mailbox_handler_const()
as it is never used.
hw/cxl/cxl-device-utils.c | 128 ++++++++++++++++++++++++++-
hw/cxl/cxl-mailbox-utils.c | 171 ++++++++++++++++++++++++++++++++++++
hw/cxl/meson.build | 1 +
include/hw/cxl/cxl.h | 3 +
include/hw/cxl/cxl_device.h | 19 +++-
5 files changed, 320 insertions(+), 2 deletions(-)
create mode 100644 hw/cxl/cxl-mailbox-utils.c
diff --git a/hw/cxl/cxl-device-utils.c b/hw/cxl/cxl-device-utils.c
index 0895b9d78b..39011468ef 100644
--- a/hw/cxl/cxl-device-utils.c
+++ b/hw/cxl/cxl-device-utils.c
@@ -44,6 +44,114 @@ static uint64_t dev_reg_read(void *opaque, hwaddr offset, unsigned size)
return 0;
}
+static uint64_t mailbox_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+ CXLDeviceState *cxl_dstate = opaque;
+
+ switch (size) {
+ case 1:
+ return cxl_dstate->mbox_reg_state[offset];
+ case 2:
+ return cxl_dstate->mbox_reg_state16[offset / 2];
+ case 4:
+ return cxl_dstate->mbox_reg_state32[offset / 4];
+ case 8:
+ return cxl_dstate->mbox_reg_state64[offset / 8];
+ default:
+ g_assert_not_reached();
+ }
+}
+
+static void mailbox_mem_writel(uint32_t *reg_state, hwaddr offset,
+ uint64_t value)
+{
+ switch (offset) {
+ case A_CXL_DEV_MAILBOX_CTRL:
+ /* fallthrough */
+ case A_CXL_DEV_MAILBOX_CAP:
+ /* RO register */
+ break;
+ default:
+ qemu_log_mask(LOG_UNIMP,
+ "%s Unexpected 32-bit access to 0x%" PRIx64 " (WI)\n",
+ __func__, offset);
+ return;
+ }
+
+ reg_state[offset / 4] = value;
+}
+
+static void mailbox_mem_writeq(uint64_t *reg_state, hwaddr offset,
+ uint64_t value)
+{
+ switch (offset) {
+ case A_CXL_DEV_MAILBOX_CMD:
+ break;
+ case A_CXL_DEV_BG_CMD_STS:
+ /* BG not supported */
+ /* fallthrough */
+ case A_CXL_DEV_MAILBOX_STS:
+ /* Read only register, will get updated by the state machine */
+ return;
+ default:
+ qemu_log_mask(LOG_UNIMP,
+ "%s Unexpected 64-bit access to 0x%" PRIx64 " (WI)\n",
+ __func__, offset);
+ return;
+ }
+
+
+ reg_state[offset / 8] = value;
+}
+
+static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,
+ unsigned size)
+{
+ CXLDeviceState *cxl_dstate = opaque;
+
+ if (offset >= A_CXL_DEV_CMD_PAYLOAD) {
+ memcpy(cxl_dstate->mbox_reg_state + offset, &value, size);
+ return;
+ }
+
+ /*
+ * Lock is needed to prevent concurrent writes as well as to
+ * prevent writes coming in while the firmware is processing.
+ * Until background commands or the second mailbox are implemented
+ * memory access is synchronized at a higher level (per memory region).
+ */
+
+ switch (size) {
+ case 4:
+ mailbox_mem_writel(cxl_dstate->mbox_reg_state32, offset, value);
+ break;
+ case 8:
+ mailbox_mem_writeq(cxl_dstate->mbox_reg_state64, offset, value);
+ break;
+ default:
+ g_assert_not_reached();
+ }
+
+ if (ARRAY_FIELD_EX32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
+ DOORBELL))
+ cxl_process_mailbox(cxl_dstate);
+}
+
+static const MemoryRegionOps mailbox_ops = {
+ .read = mailbox_reg_read,
+ .write = mailbox_reg_write,
+ .endianness = DEVICE_LITTLE_ENDIAN,
+ .valid = {
+ .min_access_size = 1,
+ .max_access_size = 8,
+ .unaligned = false,
+ },
+ .impl = {
+ .min_access_size = 1,
+ .max_access_size = 8,
+ },
+};
+
static const MemoryRegionOps dev_ops = {
.read = dev_reg_read,
.write = NULL, /* status register is read only */
@@ -84,20 +192,33 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
"cap-array", CXL_CAPS_SIZE);
memory_region_init_io(&cxl_dstate->device, obj, &dev_ops, cxl_dstate,
"device-status", CXL_DEVICE_REGISTERS_LENGTH);
+ memory_region_init_io(&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate,
+ "mailbox", CXL_MAILBOX_REGISTERS_LENGTH);
memory_region_add_subregion(&cxl_dstate->device_registers, 0,
&cxl_dstate->caps);
memory_region_add_subregion(&cxl_dstate->device_registers,
CXL_DEVICE_REGISTERS_OFFSET,
&cxl_dstate->device);
+ memory_region_add_subregion(&cxl_dstate->device_registers,
+ CXL_MAILBOX_REGISTERS_OFFSET,
+ &cxl_dstate->mailbox);
}
static void device_reg_init_common(CXLDeviceState *cxl_dstate) { }
+static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate)
+{
+ /* 2048 payload size, with no interrupt or background support */
+ ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CAP,
+ PAYLOAD_SIZE, CXL_MAILBOX_PAYLOAD_SHIFT);
+ cxl_dstate->payload_size = CXL_MAILBOX_MAX_PAYLOAD_SIZE;
+}
+
void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
{
uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64;
- const int cap_count = 1;
+ const int cap_count = 2;
/* CXL Device Capabilities Array Register */
ARRAY_FIELD_DP64(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
@@ -106,4 +227,9 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
cxl_device_cap_init(cxl_dstate, DEVICE, 1);
device_reg_init_common(cxl_dstate);
+
+ cxl_device_cap_init(cxl_dstate, MAILBOX, 2);
+ mailbox_reg_init_common(cxl_dstate);
+
+ assert(cxl_initialize_mailbox(cxl_dstate) == 0);
}
diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
new file mode 100644
index 0000000000..d497ec50a6
--- /dev/null
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -0,0 +1,171 @@
+/*
+ * CXL Utility library for mailbox interface
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See the
+ * COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "hw/cxl/cxl.h"
+#include "hw/pci/pci.h"
+#include "qemu/log.h"
+#include "qemu/uuid.h"
+
+/*
+ * How to add a new command, example. The command set FOO, with cmd BAR.
+ * 1. Add the command set and cmd to the enum.
+ * FOO = 0x7f,
+ * #define BAR 0
+ * 2. Implement the handler
+ * static ret_code cmd_foo_bar(struct cxl_cmd *cmd,
+ * CXLDeviceState *cxl_dstate, uint16_t *len)
+ * 3. Add the command to the cxl_cmd_set[][]
+ * [FOO][BAR] = { "FOO_BAR", cmd_foo_bar, x, y },
+ * 4. Implement your handler
+ * define_mailbox_handler(FOO_BAR) { ... return CXL_MBOX_SUCCESS; }
+ *
+ *
+ * Writing the handler:
+ * The handler will provide the &struct cxl_cmd, the &CXLDeviceState, and the
+ * in/out length of the payload. The handler is responsible for consuming the
+ * payload from cmd->payload and operating upon it as necessary. It must then
+ * fill the output data into cmd->payload (overwriting what was there),
+ * setting the length, and returning a valid return code.
+ *
+ * XXX: The handler need not worry about endianess. The payload is read out of
+ * a register interface that already deals with it.
+ */
+
+/* 8.2.8.4.5.1 Command Return Codes */
+typedef enum {
+ CXL_MBOX_SUCCESS = 0x0,
+ CXL_MBOX_BG_STARTED = 0x1,
+ CXL_MBOX_INVALID_INPUT = 0x2,
+ CXL_MBOX_UNSUPPORTED = 0x3,
+ CXL_MBOX_INTERNAL_ERROR = 0x4,
+ CXL_MBOX_RETRY_REQUIRED = 0x5,
+ CXL_MBOX_BUSY = 0x6,
+ CXL_MBOX_MEDIA_DISABLED = 0x7,
+ CXL_MBOX_FW_XFER_IN_PROGRESS = 0x8,
+ CXL_MBOX_FW_XFER_OUT_OF_ORDER = 0x9,
+ CXL_MBOX_FW_AUTH_FAILED = 0xa,
+ CXL_MBOX_FW_INVALID_SLOT = 0xb,
+ CXL_MBOX_FW_ROLLEDBACK = 0xc,
+ CXL_MBOX_FW_REST_REQD = 0xd,
+ CXL_MBOX_INVALID_HANDLE = 0xe,
+ CXL_MBOX_INVALID_PA = 0xf,
+ CXL_MBOX_INJECT_POISON_LIMIT = 0x10,
+ CXL_MBOX_PERMANENT_MEDIA_FAILURE = 0x11,
+ CXL_MBOX_ABORTED = 0x12,
+ CXL_MBOX_INVALID_SECURITY_STATE = 0x13,
+ CXL_MBOX_INCORRECT_PASSPHRASE = 0x14,
+ CXL_MBOX_UNSUPPORTED_MAILBOX = 0x15,
+ CXL_MBOX_INVALID_PAYLOAD_LENGTH = 0x16,
+ CXL_MBOX_MAX = 0x17
+} ret_code;
+
+struct cxl_cmd;
+typedef ret_code (*opcode_handler)(struct cxl_cmd *cmd,
+ CXLDeviceState *cxl_dstate, uint16_t *len);
+struct cxl_cmd {
+ const char *name;
+ opcode_handler handler;
+ ssize_t in;
+ uint16_t effect; /* Reported in CEL */
+ uint8_t *payload;
+};
+
+#define DEFINE_MAILBOX_HANDLER_ZEROED(name, size) \
+ uint16_t __zero##name = size; \
+ static ret_code cmd_##name(struct cxl_cmd *cmd, \
+ CXLDeviceState *cxl_dstate, uint16_t *len) \
+ { \
+ *len = __zero##name; \
+ memset(cmd->payload, 0, *len); \
+ return CXL_MBOX_SUCCESS; \
+ }
+#define DEFINE_MAILBOX_HANDLER_NOP(name) \
+ static ret_code cmd_##name(struct cxl_cmd *cmd, \
+ CXLDeviceState *cxl_dstate, uint16_t *len) \
+ { \
+ return CXL_MBOX_SUCCESS; \
+ }
+
+static QemuUUID cel_uuid;
+
+static struct cxl_cmd cxl_cmd_set[256][256] = {};
+
+void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
+{
+ uint16_t ret = CXL_MBOX_SUCCESS;
+ struct cxl_cmd *cxl_cmd;
+ uint64_t status_reg;
+ opcode_handler h;
+
+ /*
+ * current state of mailbox interface
+ * mbox_cap_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CAP];
+ * mbox_ctrl_reg = cxl_dstate->reg_state32[R_CXL_DEV_MAILBOX_CTRL];
+ * status_reg = *(uint64_t *)&cxl_dstate->reg_state[A_CXL_DEV_MAILBOX_STS];
+ */
+ uint64_t command_reg = cxl_dstate->mbox_reg_state64[R_CXL_DEV_MAILBOX_CMD];
+
+ uint8_t set = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET);
+ uint8_t cmd = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND);
+ uint16_t len = FIELD_EX64(command_reg, CXL_DEV_MAILBOX_CMD, LENGTH);
+ cxl_cmd = &cxl_cmd_set[set][cmd];
+ h = cxl_cmd->handler;
+ if (!h) {
+ qemu_log_mask(LOG_UNIMP, "Command %04xh not implemented\n",
+ set << 8 | cmd);
+ goto handled;
+ }
+
+ if (len != cxl_cmd->in) {
+ ret = CXL_MBOX_INVALID_PAYLOAD_LENGTH;
+ }
+
+ cxl_cmd->payload = cxl_dstate->mbox_reg_state + A_CXL_DEV_CMD_PAYLOAD;
+ ret = (*h)(cxl_cmd, cxl_dstate, &len);
+ assert(len <= cxl_dstate->payload_size);
+
+handled:
+ /* Set the return code */
+ status_reg = FIELD_DP64(0, CXL_DEV_MAILBOX_STS, ERRNO, ret);
+
+ /* Set the return length */
+ command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND_SET, 0);
+ command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, COMMAND, 0);
+ command_reg = FIELD_DP64(command_reg, CXL_DEV_MAILBOX_CMD, LENGTH, len);
+
+ cxl_dstate->mbox_reg_state64[R_CXL_DEV_MAILBOX_CMD] = command_reg;
+ cxl_dstate->mbox_reg_state64[R_CXL_DEV_MAILBOX_STS] = status_reg;
+
+ /* Tell the host we're done */
+ ARRAY_FIELD_DP32(cxl_dstate->mbox_reg_state32, CXL_DEV_MAILBOX_CTRL,
+ DOORBELL, 0);
+}
+
+int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate)
+{
+ /* CXL 2.0: Table 169 Get Supported Logs Log Entry */
+ const char *cel_uuidstr = "0da9c0b5-bf41-4b78-8f79-96b1623b3f17";
+
+ for (int set = 0; set < 256; set++) {
+ for (int cmd = 0; cmd < 256; cmd++) {
+ if (cxl_cmd_set[set][cmd].handler) {
+ struct cxl_cmd *c = &cxl_cmd_set[set][cmd];
+ struct cel_log *log =
+ &cxl_dstate->cel_log[cxl_dstate->cel_size];
+
+ log->opcode = (set << 8) | cmd;
+ log->effect = c->effect;
+ cxl_dstate->cel_size++;
+ }
+ }
+ }
+
+ return qemu_uuid_parse(cel_uuidstr, &cel_uuid);
+}
diff --git a/hw/cxl/meson.build b/hw/cxl/meson.build
index dd7c6f8e5a..e68eea2358 100644
--- a/hw/cxl/meson.build
+++ b/hw/cxl/meson.build
@@ -2,4 +2,5 @@ softmmu_ss.add(when: 'CONFIG_CXL',
if_true: files(
'cxl-component-utils.c',
'cxl-device-utils.c',
+ 'cxl-mailbox-utils.c',
))
diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
index b9d1ac3fad..554ad93b6b 100644
--- a/include/hw/cxl/cxl.h
+++ b/include/hw/cxl/cxl.h
@@ -14,4 +14,7 @@
#include "cxl_component.h"
#include "cxl_device.h"
+#define CXL_COMPONENT_REG_BAR_IDX 0
+#define CXL_DEVICE_REG_BAR_IDX 2
+
#endif
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index 1ac0dcd97e..49dcca7e44 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -95,7 +95,21 @@ typedef struct cxl_device_state {
};
/* mmio for the mailbox registers 8.2.8.4 */
- MemoryRegion mailbox;
+ struct {
+ MemoryRegion mailbox;
+ uint16_t payload_size;
+ union {
+ uint8_t mbox_reg_state[CXL_MAILBOX_REGISTERS_LENGTH];
+ uint16_t mbox_reg_state16[CXL_MAILBOX_REGISTERS_LENGTH / 2];
+ uint32_t mbox_reg_state32[CXL_MAILBOX_REGISTERS_LENGTH / 4];
+ uint64_t mbox_reg_state64[CXL_MAILBOX_REGISTERS_LENGTH / 8];
+ };
+ struct cel_log {
+ uint16_t opcode;
+ uint16_t effect;
+ } cel_log[1 << 16];
+ size_t cel_size;
+ };
/* memory region for persistent memory, HDM */
uint64_t pmem_size;
@@ -145,6 +159,9 @@ CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)
CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MAILBOX, CXL_DEVICE_CAP_HDR1_OFFSET + \
CXL_DEVICE_CAP_REG_SIZE)
+int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate);
+void cxl_process_mailbox(CXLDeviceState *cxl_dstate);
+
#define cxl_device_cap_init(dstate, reg, cap_id) \
do { \
uint32_t *cap_hdrs = dstate->caps_reg_state32; \
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 07/43] hw/cxl/device: Add memory device utilities
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (5 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 06/43] hw/cxl/device: Implement basic mailbox (8.2.8.4) Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 08/43] hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) Jonathan Cameron
` (37 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
Memory devices implement extra capabilities on top of CXL devices. This
adds support for that.
A large part of memory devices is the mailbox/command interface. All of
the mailbox handling is done in the mailbox-utils library. Longer term,
new CXL devices that are being emulated may want to handle commands
differently, and therefore would need a mechanism to opt in/out of the
specific generic handlers. As such, this is considered sufficient for
now, but may need more depth in the future.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
hw/cxl/cxl-device-utils.c | 38 ++++++++++++++++++++++++++++++++++++-
include/hw/cxl/cxl_device.h | 22 ++++++++++++++++++---
2 files changed, 56 insertions(+), 4 deletions(-)
diff --git a/hw/cxl/cxl-device-utils.c b/hw/cxl/cxl-device-utils.c
index 39011468ef..14336d846d 100644
--- a/hw/cxl/cxl-device-utils.c
+++ b/hw/cxl/cxl-device-utils.c
@@ -137,6 +137,31 @@ static void mailbox_reg_write(void *opaque, hwaddr offset, uint64_t value,
cxl_process_mailbox(cxl_dstate);
}
+static uint64_t mdev_reg_read(void *opaque, hwaddr offset, unsigned size)
+{
+ uint64_t retval = 0;
+
+ retval = FIELD_DP64(retval, CXL_MEM_DEV_STS, MEDIA_STATUS, 1);
+ retval = FIELD_DP64(retval, CXL_MEM_DEV_STS, MBOX_READY, 1);
+
+ return retval;
+}
+
+static const MemoryRegionOps mdev_ops = {
+ .read = mdev_reg_read,
+ .write = NULL, /* memory device register is read only */
+ .endianness = DEVICE_LITTLE_ENDIAN,
+ .valid = {
+ .min_access_size = 1,
+ .max_access_size = 8,
+ .unaligned = false,
+ },
+ .impl = {
+ .min_access_size = 8,
+ .max_access_size = 8,
+ },
+};
+
static const MemoryRegionOps mailbox_ops = {
.read = mailbox_reg_read,
.write = mailbox_reg_write,
@@ -194,6 +219,9 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
"device-status", CXL_DEVICE_REGISTERS_LENGTH);
memory_region_init_io(&cxl_dstate->mailbox, obj, &mailbox_ops, cxl_dstate,
"mailbox", CXL_MAILBOX_REGISTERS_LENGTH);
+ memory_region_init_io(&cxl_dstate->memory_device, obj, &mdev_ops,
+ cxl_dstate, "memory device caps",
+ CXL_MEMORY_DEVICE_REGISTERS_LENGTH);
memory_region_add_subregion(&cxl_dstate->device_registers, 0,
&cxl_dstate->caps);
@@ -203,6 +231,9 @@ void cxl_device_register_block_init(Object *obj, CXLDeviceState *cxl_dstate)
memory_region_add_subregion(&cxl_dstate->device_registers,
CXL_MAILBOX_REGISTERS_OFFSET,
&cxl_dstate->mailbox);
+ memory_region_add_subregion(&cxl_dstate->device_registers,
+ CXL_MEMORY_DEVICE_REGISTERS_OFFSET,
+ &cxl_dstate->memory_device);
}
static void device_reg_init_common(CXLDeviceState *cxl_dstate) { }
@@ -215,10 +246,12 @@ static void mailbox_reg_init_common(CXLDeviceState *cxl_dstate)
cxl_dstate->payload_size = CXL_MAILBOX_MAX_PAYLOAD_SIZE;
}
+static void memdev_reg_init_common(CXLDeviceState *cxl_dstate) { }
+
void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
{
uint64_t *cap_hdrs = cxl_dstate->caps_reg_state64;
- const int cap_count = 2;
+ const int cap_count = 3;
/* CXL Device Capabilities Array Register */
ARRAY_FIELD_DP64(cap_hdrs, CXL_DEV_CAP_ARRAY, CAP_ID, 0);
@@ -231,5 +264,8 @@ void cxl_device_register_init_common(CXLDeviceState *cxl_dstate)
cxl_device_cap_init(cxl_dstate, MAILBOX, 2);
mailbox_reg_init_common(cxl_dstate);
+ cxl_device_cap_init(cxl_dstate, MEMORY_DEVICE, 0x4000);
+ memdev_reg_init_common(cxl_dstate);
+
assert(cxl_initialize_mailbox(cxl_dstate) == 0);
}
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index 49dcca7e44..7fd8d0f616 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -77,15 +77,21 @@
#define CXL_MAILBOX_REGISTERS_LENGTH \
(CXL_MAILBOX_REGISTERS_SIZE + CXL_MAILBOX_MAX_PAYLOAD_SIZE)
-#define CXL_MMIO_SIZE \
- (CXL_DEVICE_CAP_REG_SIZE + CXL_DEVICE_REGISTERS_LENGTH + \
- CXL_MAILBOX_REGISTERS_LENGTH)
+
+#define CXL_MEMORY_DEVICE_REGISTERS_OFFSET \
+ (CXL_MAILBOX_REGISTERS_OFFSET + CXL_MAILBOX_REGISTERS_LENGTH)
+#define CXL_MEMORY_DEVICE_REGISTERS_LENGTH 0x8
+
+#define CXL_MMIO_SIZE \
+ (CXL_DEVICE_CAP_REG_SIZE + CXL_DEVICE_REGISTERS_LENGTH + \
+ CXL_MAILBOX_REGISTERS_LENGTH + CXL_MEMORY_DEVICE_REGISTERS_LENGTH)
typedef struct cxl_device_state {
MemoryRegion device_registers;
/* mmio for device capabilities array - 8.2.8.2 */
MemoryRegion device;
+ MemoryRegion memory_device;
struct {
MemoryRegion caps;
union {
@@ -158,6 +164,9 @@ REG64(CXL_DEV_CAP_ARRAY, 0) /* Documented as 128 bit register but 64 byte access
CXL_DEVICE_CAPABILITY_HEADER_REGISTER(DEVICE, CXL_DEVICE_CAP_HDR1_OFFSET)
CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MAILBOX, CXL_DEVICE_CAP_HDR1_OFFSET + \
CXL_DEVICE_CAP_REG_SIZE)
+CXL_DEVICE_CAPABILITY_HEADER_REGISTER(MEMORY_DEVICE,
+ CXL_DEVICE_CAP_HDR1_OFFSET +
+ CXL_DEVICE_CAP_REG_SIZE * 2)
int cxl_initialize_mailbox(CXLDeviceState *cxl_dstate);
void cxl_process_mailbox(CXLDeviceState *cxl_dstate);
@@ -208,4 +217,11 @@ REG64(CXL_DEV_BG_CMD_STS, 0x18)
REG32(CXL_DEV_CMD_PAYLOAD, 0x20)
+REG64(CXL_MEM_DEV_STS, 0)
+ FIELD(CXL_MEM_DEV_STS, FATAL, 0, 1)
+ FIELD(CXL_MEM_DEV_STS, FW_HALT, 1, 1)
+ FIELD(CXL_MEM_DEV_STS, MEDIA_STATUS, 2, 2)
+ FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1)
+ FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3)
+
#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 08/43] hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (6 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 07/43] hw/cxl/device: Add memory device utilities Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 09/43] hw/cxl/device: Timestamp implementation (8.2.9.3) Jonathan Cameron
` (36 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
Using the previously implemented stubbed helpers, it is now possible to
easily add the missing, required commands to the implementation.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
v5: Follow through on upper casing defines in patch 6.
hw/cxl/cxl-mailbox-utils.c | 27 ++++++++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index d497ec50a6..8aa1b1e525 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -38,6 +38,14 @@
* a register interface that already deals with it.
*/
+enum {
+ EVENTS = 0x01,
+ #define GET_RECORDS 0x0
+ #define CLEAR_RECORDS 0x1
+ #define GET_INTERRUPT_POLICY 0x2
+ #define SET_INTERRUPT_POLICY 0x3
+};
+
/* 8.2.8.4.5.1 Command Return Codes */
typedef enum {
CXL_MBOX_SUCCESS = 0x0,
@@ -93,9 +101,26 @@ struct cxl_cmd {
return CXL_MBOX_SUCCESS; \
}
+DEFINE_MAILBOX_HANDLER_ZEROED(events_get_records, 0x20);
+DEFINE_MAILBOX_HANDLER_NOP(events_clear_records);
+DEFINE_MAILBOX_HANDLER_ZEROED(events_get_interrupt_policy, 4);
+DEFINE_MAILBOX_HANDLER_NOP(events_set_interrupt_policy);
+
static QemuUUID cel_uuid;
-static struct cxl_cmd cxl_cmd_set[256][256] = {};
+#define IMMEDIATE_CONFIG_CHANGE (1 << 1)
+#define IMMEDIATE_LOG_CHANGE (1 << 4)
+
+static struct cxl_cmd cxl_cmd_set[256][256] = {
+ [EVENTS][GET_RECORDS] = { "EVENTS_GET_RECORDS",
+ cmd_events_get_records, 1, 0 },
+ [EVENTS][CLEAR_RECORDS] = { "EVENTS_CLEAR_RECORDS",
+ cmd_events_clear_records, ~0, IMMEDIATE_LOG_CHANGE },
+ [EVENTS][GET_INTERRUPT_POLICY] = { "EVENTS_GET_INTERRUPT_POLICY",
+ cmd_events_get_interrupt_policy, 0, 0 },
+ [EVENTS][SET_INTERRUPT_POLICY] = { "EVENTS_SET_INTERRUPT_POLICY",
+ cmd_events_set_interrupt_policy, 4, IMMEDIATE_CONFIG_CHANGE },
+};
void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
{
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 09/43] hw/cxl/device: Timestamp implementation (8.2.9.3)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (7 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 08/43] hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1) Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 10/43] hw/cxl/device: Add log commands (8.2.9.4) + CEL Jonathan Cameron
` (35 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
Errata F4 to CXL 2.0 clarified the meaning of the timer as the
sum of the value set with the timestamp set command and the number
of nano seconds since it was last set.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
v5: Reponses to Alex's review.
- Change to using the qemu_clock_get_ns()
- Follow through of new approach to mailbox handlers from patch 5.
hw/cxl/cxl-mailbox-utils.c | 44 +++++++++++++++++++++++++++++++++++++
include/hw/cxl/cxl_device.h | 6 +++++
2 files changed, 50 insertions(+)
diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index 8aa1b1e525..258285ab03 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -44,6 +44,9 @@ enum {
#define CLEAR_RECORDS 0x1
#define GET_INTERRUPT_POLICY 0x2
#define SET_INTERRUPT_POLICY 0x3
+ TIMESTAMP = 0x03,
+ #define GET 0x0
+ #define SET 0x1
};
/* 8.2.8.4.5.1 Command Return Codes */
@@ -106,9 +109,48 @@ DEFINE_MAILBOX_HANDLER_NOP(events_clear_records);
DEFINE_MAILBOX_HANDLER_ZEROED(events_get_interrupt_policy, 4);
DEFINE_MAILBOX_HANDLER_NOP(events_set_interrupt_policy);
+/* 8.2.9.3.1 */
+static ret_code cmd_timestamp_get(struct cxl_cmd *cmd,
+ CXLDeviceState *cxl_dstate,
+ uint16_t *len)
+{
+ uint64_t time, delta;
+
+ if (!cxl_dstate->timestamp.set) {
+ *(uint64_t *)cmd->payload = 0;
+ goto done;
+ }
+
+ /* First find the delta from the last time the host set the time. */
+ time = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+ delta = time - cxl_dstate->timestamp.last_set;
+
+ /* Then adjust the actual time */
+ stq_le_p(cmd->payload, cxl_dstate->timestamp.host_set + delta);
+
+done:
+ *len = 8;
+ return CXL_MBOX_SUCCESS;
+}
+
+/* 8.2.9.3.2 */
+static ret_code cmd_timestamp_set(struct cxl_cmd *cmd,
+ CXLDeviceState *cxl_dstate,
+ uint16_t *len)
+{
+ cxl_dstate->timestamp.set = true;
+ cxl_dstate->timestamp.last_set = qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL);
+
+ cxl_dstate->timestamp.host_set = le64_to_cpu(*(uint64_t *)cmd->payload);
+
+ *len = 0;
+ return CXL_MBOX_SUCCESS;
+}
+
static QemuUUID cel_uuid;
#define IMMEDIATE_CONFIG_CHANGE (1 << 1)
+#define IMMEDIATE_POLICY_CHANGE (1 << 3)
#define IMMEDIATE_LOG_CHANGE (1 << 4)
static struct cxl_cmd cxl_cmd_set[256][256] = {
@@ -120,6 +162,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
cmd_events_get_interrupt_policy, 0, 0 },
[EVENTS][SET_INTERRUPT_POLICY] = { "EVENTS_SET_INTERRUPT_POLICY",
cmd_events_set_interrupt_policy, 4, IMMEDIATE_CONFIG_CHANGE },
+ [TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 },
+ [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE },
};
void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index 7fd8d0f616..8102d2a813 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -117,6 +117,12 @@ typedef struct cxl_device_state {
size_t cel_size;
};
+ struct {
+ bool set;
+ uint64_t last_set;
+ uint64_t host_set;
+ } timestamp;
+
/* memory region for persistent memory, HDM */
uint64_t pmem_size;
} CXLDeviceState;
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 10/43] hw/cxl/device: Add log commands (8.2.9.4) + CEL
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (8 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 09/43] hw/cxl/device: Timestamp implementation (8.2.9.3) Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 11/43] hw/pxb: Use a type for realizing expanders Jonathan Cameron
` (34 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
CXL specification provides for the ability to obtain logs from the
device. Logs are either spec defined, like the "Command Effects Log"
(CEL), or vendor specific. UUIDs are defined for all log types.
The CEL is a mechanism to provide information to the host about which
commands are supported. It is useful both to determine which spec'd
optional commands are supported, as well as provide a list of vendor
specified commands that might be used. The CEL is already created as
part of mailbox initialization, but here it is now exported to hosts
that use these log commands.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
v5: Results of Alex's review.
- Follow through on v5 removal of mailbox handler related macros.
It was this patch where Alex highlighted the need to make that
change.
hw/cxl/cxl-mailbox-utils.c | 69 ++++++++++++++++++++++++++++++++++++++
1 file changed, 69 insertions(+)
diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index 258285ab03..16bb998735 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -47,6 +47,9 @@ enum {
TIMESTAMP = 0x03,
#define GET 0x0
#define SET 0x1
+ LOGS = 0x04,
+ #define GET_SUPPORTED 0x0
+ #define GET_LOG 0x1
};
/* 8.2.8.4.5.1 Command Return Codes */
@@ -149,6 +152,70 @@ static ret_code cmd_timestamp_set(struct cxl_cmd *cmd,
static QemuUUID cel_uuid;
+/* 8.2.9.4.1 */
+static ret_code cmd_logs_get_supported(struct cxl_cmd *cmd,
+ CXLDeviceState *cxl_dstate,
+ uint16_t *len)
+{
+ struct {
+ uint16_t entries;
+ uint8_t rsvd[6];
+ struct {
+ QemuUUID uuid;
+ uint32_t size;
+ } log_entries[1];
+ } __attribute__((packed)) *supported_logs = (void *)cmd->payload;
+ _Static_assert(sizeof(*supported_logs) == 0x1c, "Bad supported log size");
+
+ supported_logs->entries = 1;
+ supported_logs->log_entries[0].uuid = cel_uuid;
+ supported_logs->log_entries[0].size = 4 * cxl_dstate->cel_size;
+
+ *len = sizeof(*supported_logs);
+ return CXL_MBOX_SUCCESS;
+}
+
+/* 8.2.9.4.2 */
+static ret_code cmd_logs_get_log(struct cxl_cmd *cmd,
+ CXLDeviceState *cxl_dstate,
+ uint16_t *len)
+{
+ struct {
+ QemuUUID uuid;
+ uint32_t offset;
+ uint32_t length;
+ } __attribute__((packed, __aligned__(16))) *get_log = (void *)cmd->payload;
+
+ /*
+ * 8.2.9.4.2
+ * The device shall return Invalid Parameter if the Offset or Length
+ * fields attempt to access beyond the size of the log as reported by Get
+ * Supported Logs.
+ *
+ * XXX: Spec is wrong, "Invalid Parameter" isn't a thing.
+ * XXX: Spec doesn't address incorrect UUID incorrectness.
+ *
+ * The CEL buffer is large enough to fit all commands in the emulation, so
+ * the only possible failure would be if the mailbox itself isn't big
+ * enough.
+ */
+ if (get_log->offset + get_log->length > cxl_dstate->payload_size) {
+ return CXL_MBOX_INVALID_INPUT;
+ }
+
+ if (!qemu_uuid_is_equal(&get_log->uuid, &cel_uuid)) {
+ return CXL_MBOX_UNSUPPORTED;
+ }
+
+ /* Store off everything to local variables so we can wipe out the payload */
+ *len = get_log->length;
+
+ memmove(cmd->payload, cxl_dstate->cel_log + get_log->offset,
+ get_log->length);
+
+ return CXL_MBOX_SUCCESS;
+}
+
#define IMMEDIATE_CONFIG_CHANGE (1 << 1)
#define IMMEDIATE_POLICY_CHANGE (1 << 3)
#define IMMEDIATE_LOG_CHANGE (1 << 4)
@@ -164,6 +231,8 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
cmd_events_set_interrupt_policy, 4, IMMEDIATE_CONFIG_CHANGE },
[TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 },
[TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE },
+ [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 },
+ [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 },
};
void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 11/43] hw/pxb: Use a type for realizing expanders
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (9 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 10/43] hw/cxl/device: Add log commands (8.2.9.4) + CEL Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 12/43] hw/pci/cxl: Create a CXL bus type Jonathan Cameron
` (33 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
This opens up the possibility for more types of expanders (other than
PCI and PCIe). We'll need this to create a CXL expander.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
hw/pci-bridge/pci_expander_bridge.c | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/hw/pci-bridge/pci_expander_bridge.c b/hw/pci-bridge/pci_expander_bridge.c
index de932286b5..d4514227a8 100644
--- a/hw/pci-bridge/pci_expander_bridge.c
+++ b/hw/pci-bridge/pci_expander_bridge.c
@@ -24,6 +24,8 @@
#include "hw/boards.h"
#include "qom/object.h"
+enum BusType { PCI, PCIE };
+
#define TYPE_PXB_BUS "pxb-bus"
typedef struct PXBBus PXBBus;
DECLARE_INSTANCE_CHECKER(PXBBus, PXB_BUS,
@@ -221,7 +223,8 @@ static gint pxb_compare(gconstpointer a, gconstpointer b)
0;
}
-static void pxb_dev_realize_common(PCIDevice *dev, bool pcie, Error **errp)
+static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,
+ Error **errp)
{
PXBDev *pxb = convert_to_pxb(dev);
DeviceState *ds, *bds = NULL;
@@ -246,7 +249,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, bool pcie, Error **errp)
}
ds = qdev_new(TYPE_PXB_HOST);
- if (pcie) {
+ if (type == PCIE) {
bus = pci_root_bus_new(ds, dev_name, NULL, NULL, 0, TYPE_PXB_PCIE_BUS);
} else {
bus = pci_root_bus_new(ds, "pxb-internal", NULL, NULL, 0, TYPE_PXB_BUS);
@@ -295,7 +298,7 @@ static void pxb_dev_realize(PCIDevice *dev, Error **errp)
return;
}
- pxb_dev_realize_common(dev, false, errp);
+ pxb_dev_realize_common(dev, PCI, errp);
}
static void pxb_dev_exitfn(PCIDevice *pci_dev)
@@ -348,7 +351,7 @@ static void pxb_pcie_dev_realize(PCIDevice *dev, Error **errp)
return;
}
- pxb_dev_realize_common(dev, true, errp);
+ pxb_dev_realize_common(dev, PCIE, errp);
}
static void pxb_pcie_dev_class_init(ObjectClass *klass, void *data)
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 12/43] hw/pci/cxl: Create a CXL bus type
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (10 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 11/43] hw/pxb: Use a type for realizing expanders Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 13/43] hw/pxb: Allow creation of a CXL PXB (host bridge) Jonathan Cameron
` (32 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
The easiest way to differentiate a CXL bus, and a PCIE bus is using a
flag. A CXL bus, in hardware, is backward compatible with PCIE, and
therefore the code tries pretty hard to keep them in sync as much as
possible.
The other way to implement this would be to try to cast the bus to the
correct type. This is less code and useful for debugging via simply
looking at the flags.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
hw/pci-bridge/pci_expander_bridge.c | 9 ++++++++-
include/hw/pci/pci_bus.h | 7 +++++++
2 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/hw/pci-bridge/pci_expander_bridge.c b/hw/pci-bridge/pci_expander_bridge.c
index d4514227a8..a6caa1e7b5 100644
--- a/hw/pci-bridge/pci_expander_bridge.c
+++ b/hw/pci-bridge/pci_expander_bridge.c
@@ -24,7 +24,7 @@
#include "hw/boards.h"
#include "qom/object.h"
-enum BusType { PCI, PCIE };
+enum BusType { PCI, PCIE, CXL };
#define TYPE_PXB_BUS "pxb-bus"
typedef struct PXBBus PXBBus;
@@ -35,6 +35,10 @@ DECLARE_INSTANCE_CHECKER(PXBBus, PXB_BUS,
DECLARE_INSTANCE_CHECKER(PXBBus, PXB_PCIE_BUS,
TYPE_PXB_PCIE_BUS)
+#define TYPE_PXB_CXL_BUS "pxb-cxl-bus"
+DECLARE_INSTANCE_CHECKER(PXBBus, PXB_CXL_BUS,
+ TYPE_PXB_CXL_BUS)
+
struct PXBBus {
/*< private >*/
PCIBus parent_obj;
@@ -251,6 +255,9 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,
ds = qdev_new(TYPE_PXB_HOST);
if (type == PCIE) {
bus = pci_root_bus_new(ds, dev_name, NULL, NULL, 0, TYPE_PXB_PCIE_BUS);
+ } else if (type == CXL) {
+ bus = pci_root_bus_new(ds, dev_name, NULL, NULL, 0, TYPE_PXB_CXL_BUS);
+ bus->flags |= PCI_BUS_CXL;
} else {
bus = pci_root_bus_new(ds, "pxb-internal", NULL, NULL, 0, TYPE_PXB_BUS);
bds = qdev_new("pci-bridge");
diff --git a/include/hw/pci/pci_bus.h b/include/hw/pci/pci_bus.h
index 347440d42c..eb94e7e85c 100644
--- a/include/hw/pci/pci_bus.h
+++ b/include/hw/pci/pci_bus.h
@@ -24,6 +24,8 @@ enum PCIBusFlags {
PCI_BUS_IS_ROOT = 0x0001,
/* PCIe extended configuration space is accessible on this bus */
PCI_BUS_EXTENDED_CONFIG_SPACE = 0x0002,
+ /* This is a CXL Type BUS */
+ PCI_BUS_CXL = 0x0004,
};
struct PCIBus {
@@ -53,6 +55,11 @@ struct PCIBus {
Notifier machine_done;
};
+static inline bool pci_bus_is_cxl(PCIBus *bus)
+{
+ return !!(bus->flags & PCI_BUS_CXL);
+}
+
static inline bool pci_bus_is_root(PCIBus *bus)
{
return !!(bus->flags & PCI_BUS_IS_ROOT);
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 13/43] hw/pxb: Allow creation of a CXL PXB (host bridge)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (11 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 12/43] hw/pci/cxl: Create a CXL bus type Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 14/43] tests/acpi: allow DSDT.viot table changes Jonathan Cameron
` (31 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
This works like adding a typical pxb device, except the name is
'pxb-cxl' instead of 'pxb-pcie'. An example command line would be as
follows:
-device pxb-cxl,id=cxl.0,bus="pcie.0",bus_nr=1
A CXL PXB is backward compatible with PCIe. What this means in practice
is that an operating system that is unaware of CXL should still be able
to enumerate this topology as if it were PCIe.
One can create multiple CXL PXB host bridges, but a host bridge can only
be connected to the main root bus. Host bridges cannot appear elsewhere
in the topology.
Note that as of this patch, the ACPI tables needed for the host bridge
(specifically, an ACPI object in _SB named ACPI0016 and the CEDT) aren't
created. So while this patch internally creates it, it cannot be
properly used by an operating system or other system software.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan.Cameron <Jonathan.Cameron@huawei.com>
---
v5: All in response to Alex's review (thanks!)
- Moved pxb_dev_reset() to cxl realize function instead of doing it
in the common code called from that function.
- Fixed pxb_dev_reset() not being called in other paths due to it
being registered in the wrong class_init. Note it was also broken
so pulled a reference from the PXB_CXL_DEV to the host bridge
back from patch 24 as we now need it here.
hw/pci-bridge/pci_expander_bridge.c | 95 ++++++++++++++++++++++++++++-
hw/pci/pci.c | 7 +++
include/hw/pci/pci.h | 6 ++
3 files changed, 106 insertions(+), 2 deletions(-)
diff --git a/hw/pci-bridge/pci_expander_bridge.c b/hw/pci-bridge/pci_expander_bridge.c
index a6caa1e7b5..c7a28c7b2e 100644
--- a/hw/pci-bridge/pci_expander_bridge.c
+++ b/hw/pci-bridge/pci_expander_bridge.c
@@ -17,6 +17,7 @@
#include "hw/pci/pci_host.h"
#include "hw/qdev-properties.h"
#include "hw/pci/pci_bridge.h"
+#include "hw/cxl/cxl.h"
#include "qemu/range.h"
#include "qemu/error-report.h"
#include "qemu/module.h"
@@ -56,6 +57,17 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,
DECLARE_INSTANCE_CHECKER(PXBDev, PXB_PCIE_DEV,
TYPE_PXB_PCIE_DEVICE)
+#define TYPE_PXB_CXL_DEVICE "pxb-cxl"
+DECLARE_INSTANCE_CHECKER(PXBDev, PXB_CXL_DEV,
+ TYPE_PXB_CXL_DEVICE)
+
+typedef struct CXLHost {
+ PCIHostState parent_obj;
+
+ CXLComponentState cxl_cstate;
+} CXLHost;
+
+
struct PXBDev {
/*< private >*/
PCIDevice parent_obj;
@@ -64,10 +76,18 @@ struct PXBDev {
uint8_t bus_nr;
uint16_t numa_node;
bool bypass_iommu;
+ struct cxl_dev {
+ CXLHost *cxl_host_bridge;
+ } cxl;
};
static PXBDev *convert_to_pxb(PCIDevice *dev)
{
+ /* A CXL PXB's parent bus is PCIe, so the normal check won't work */
+ if (object_dynamic_cast(OBJECT(dev), TYPE_PXB_CXL_DEVICE)) {
+ return PXB_CXL_DEV(dev);
+ }
+
return pci_bus_is_express(pci_get_bus(dev))
? PXB_PCIE_DEV(dev) : PXB_DEV(dev);
}
@@ -76,6 +96,9 @@ static GList *pxb_dev_list;
#define TYPE_PXB_HOST "pxb-host"
+#define TYPE_PXB_CXL_HOST "pxb-cxl-host"
+#define PXB_CXL_HOST(obj) OBJECT_CHECK(CXLHost, (obj), TYPE_PXB_CXL_HOST)
+
static int pxb_bus_num(PCIBus *bus)
{
PXBDev *pxb = convert_to_pxb(bus->parent_dev);
@@ -112,11 +135,20 @@ static const TypeInfo pxb_pcie_bus_info = {
.class_init = pxb_bus_class_init,
};
+static const TypeInfo pxb_cxl_bus_info = {
+ .name = TYPE_PXB_CXL_BUS,
+ .parent = TYPE_CXL_BUS,
+ .instance_size = sizeof(PXBBus),
+ .class_init = pxb_bus_class_init,
+};
+
static const char *pxb_host_root_bus_path(PCIHostState *host_bridge,
PCIBus *rootbus)
{
- PXBBus *bus = pci_bus_is_express(rootbus) ?
- PXB_PCIE_BUS(rootbus) : PXB_BUS(rootbus);
+ PXBBus *bus = pci_bus_is_cxl(rootbus) ?
+ PXB_CXL_BUS(rootbus) :
+ pci_bus_is_express(rootbus) ? PXB_PCIE_BUS(rootbus) :
+ PXB_BUS(rootbus);
snprintf(bus->bus_path, 8, "0000:%02x", pxb_bus_num(rootbus));
return bus->bus_path;
@@ -218,6 +250,16 @@ static int pxb_map_irq_fn(PCIDevice *pci_dev, int pin)
return pin - PCI_SLOT(pxb->devfn);
}
+static void pxb_dev_reset(DeviceState *dev)
+{
+ CXLHost *cxl = PXB_CXL_DEV(dev)->cxl.cxl_host_bridge;
+ CXLComponentState *cxl_cstate = &cxl->cxl_cstate;
+ uint32_t *reg_state = cxl_cstate->crb.cache_mem_registers;
+
+ cxl_component_register_init_common(reg_state, CXL2_ROOT_PORT);
+ ARRAY_FIELD_DP32(reg_state, CXL_HDM_DECODER_CAPABILITY, TARGET_COUNT, 8);
+}
+
static gint pxb_compare(gconstpointer a, gconstpointer b)
{
const PXBDev *pxb_a = a, *pxb_b = b;
@@ -258,6 +300,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,
} else if (type == CXL) {
bus = pci_root_bus_new(ds, dev_name, NULL, NULL, 0, TYPE_PXB_CXL_BUS);
bus->flags |= PCI_BUS_CXL;
+ PXB_CXL_DEV(dev)->cxl.cxl_host_bridge = PXB_CXL_HOST(ds);
} else {
bus = pci_root_bus_new(ds, "pxb-internal", NULL, NULL, 0, TYPE_PXB_BUS);
bds = qdev_new("pci-bridge");
@@ -290,6 +333,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,
pci_config_set_class(dev->config, PCI_CLASS_BRIDGE_HOST);
pxb_dev_list = g_list_insert_sorted(pxb_dev_list, pxb, pxb_compare);
+
return;
err_register_bus:
@@ -389,13 +433,60 @@ static const TypeInfo pxb_pcie_dev_info = {
},
};
+static void pxb_cxl_dev_realize(PCIDevice *dev, Error **errp)
+{
+ /* A CXL PXB's parent bus is still PCIe */
+ if (!pci_bus_is_express(pci_get_bus(dev))) {
+ error_setg(errp, "pxb-cxl devices cannot reside on a PCI bus");
+ return;
+ }
+
+ pxb_dev_realize_common(dev, CXL, errp);
+ pxb_dev_reset(DEVICE(dev));
+}
+
+static void pxb_cxl_dev_class_init(ObjectClass *klass, void *data)
+{
+ DeviceClass *dc = DEVICE_CLASS(klass);
+ PCIDeviceClass *k = PCI_DEVICE_CLASS(klass);
+
+ k->realize = pxb_cxl_dev_realize;
+ k->exit = pxb_dev_exitfn;
+ /*
+ * XXX: These types of bridges don't actually show up in the hierarchy so
+ * vendor, device, class, etc. ids are intentionally left out.
+ */
+
+ dc->desc = "CXL Host Bridge";
+ device_class_set_props(dc, pxb_dev_properties);
+ set_bit(DEVICE_CATEGORY_BRIDGE, dc->categories);
+
+ /* Host bridges aren't hotpluggable. FIXME: spec reference */
+ dc->hotpluggable = false;
+ dc->reset = pxb_dev_reset;
+}
+
+static const TypeInfo pxb_cxl_dev_info = {
+ .name = TYPE_PXB_CXL_DEVICE,
+ .parent = TYPE_PCI_DEVICE,
+ .instance_size = sizeof(PXBDev),
+ .class_init = pxb_cxl_dev_class_init,
+ .interfaces =
+ (InterfaceInfo[]){
+ { INTERFACE_CONVENTIONAL_PCI_DEVICE },
+ {},
+ },
+};
+
static void pxb_register_types(void)
{
type_register_static(&pxb_bus_info);
type_register_static(&pxb_pcie_bus_info);
+ type_register_static(&pxb_cxl_bus_info);
type_register_static(&pxb_host_info);
type_register_static(&pxb_dev_info);
type_register_static(&pxb_pcie_dev_info);
+ type_register_static(&pxb_cxl_dev_info);
}
type_init(pxb_register_types)
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index 474ea98c1d..cafebf6f59 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -229,6 +229,12 @@ static const TypeInfo pcie_bus_info = {
.class_init = pcie_bus_class_init,
};
+static const TypeInfo cxl_bus_info = {
+ .name = TYPE_CXL_BUS,
+ .parent = TYPE_PCIE_BUS,
+ .class_init = pcie_bus_class_init,
+};
+
static PCIBus *pci_find_bus_nr(PCIBus *bus, int bus_num);
static void pci_update_mappings(PCIDevice *d);
static void pci_irq_handler(void *opaque, int irq_num, int level);
@@ -2892,6 +2898,7 @@ static void pci_register_types(void)
{
type_register_static(&pci_bus_info);
type_register_static(&pcie_bus_info);
+ type_register_static(&cxl_bus_info);
type_register_static(&conventional_pci_interface_info);
type_register_static(&cxl_interface_info);
type_register_static(&pcie_interface_info);
diff --git a/include/hw/pci/pci.h b/include/hw/pci/pci.h
index 908896ebe8..97cbbad375 100644
--- a/include/hw/pci/pci.h
+++ b/include/hw/pci/pci.h
@@ -409,6 +409,7 @@ typedef PCIINTxRoute (*pci_route_irq_fn)(void *opaque, int pin);
#define TYPE_PCI_BUS "PCI"
OBJECT_DECLARE_TYPE(PCIBus, PCIBusClass, PCI_BUS)
#define TYPE_PCIE_BUS "PCIE"
+#define TYPE_CXL_BUS "CXL"
typedef void (*pci_bus_dev_fn)(PCIBus *b, PCIDevice *d, void *opaque);
typedef void (*pci_bus_fn)(PCIBus *b, void *opaque);
@@ -768,6 +769,11 @@ static inline void pci_irq_pulse(PCIDevice *pci_dev)
pci_irq_deassert(pci_dev);
}
+static inline int pci_is_cxl(const PCIDevice *d)
+{
+ return d->cap_present & QEMU_PCIE_CAP_CXL;
+}
+
static inline int pci_is_express(const PCIDevice *d)
{
return d->cap_present & QEMU_PCI_CAP_EXPRESS;
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 14/43] tests/acpi: allow DSDT.viot table changes.
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (12 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 13/43] hw/pxb: Allow creation of a CXL PXB (host bridge) Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 15/43] acpi/pci: Consolidate host bridge setup Jonathan Cameron
` (30 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
The next patch unifies some of the PCI host bridge DSDT
generation code and results in some minor changes to this file.
Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
---
v5: No change, but Alex suggested we combine this and next
two patches. I'd like feedback from the bios tables test maintainer
on this question.
tests/qtest/bios-tables-test-allowed-diff.h | 1 +
1 file changed, 1 insertion(+)
diff --git a/tests/qtest/bios-tables-test-allowed-diff.h b/tests/qtest/bios-tables-test-allowed-diff.h
index dfb8523c8b..08a8095432 100644
--- a/tests/qtest/bios-tables-test-allowed-diff.h
+++ b/tests/qtest/bios-tables-test-allowed-diff.h
@@ -1 +1,2 @@
/* List of comma-separated changed AML files to ignore */
+"tests/data/acpi/q35/DSDT.viot",
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 15/43] acpi/pci: Consolidate host bridge setup
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (13 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 14/43] tests/acpi: allow DSDT.viot table changes Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 16/43] tests/acpi: Add update DSDT.viot Jonathan Cameron
` (29 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
This cleanup will make it easier to add support for CXL to the mix.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
v5: Make the PCI bus type a typed enum.
hw/i386/acpi-build.c | 39 ++++++++++++++++++++++-----------------
1 file changed, 22 insertions(+), 17 deletions(-)
diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index ce823e8fcb..09940f6e84 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -1398,6 +1398,24 @@ static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func)
aml_append(table, scope);
}
+typedef enum { PCI, PCIE } PCIBusType;
+static void init_pci_acpi(Aml *dev, int uid, PCIBusType type,
+ bool native_pcie_hp)
+{
+ if (type == PCI) {
+ aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A03")));
+ aml_append(dev, aml_name_decl("_ADR", aml_int(0)));
+ aml_append(dev, aml_name_decl("_UID", aml_int(uid)));
+ } else {
+ aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A08")));
+ aml_append(dev, aml_name_decl("_CID", aml_eisaid("PNP0A03")));
+ aml_append(dev, aml_name_decl("_ADR", aml_int(0)));
+ aml_append(dev, aml_name_decl("_UID", aml_int(uid)));
+ /* Expander bridges do not have ACPI PCI Hot-plug enabled */
+ aml_append(dev, build_q35_osc_method(native_pcie_hp));
+ }
+}
+
static void
build_dsdt(GArray *table_data, BIOSLinker *linker,
AcpiPmInfo *pm, AcpiMiscInfo *misc,
@@ -1429,9 +1447,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
if (misc->is_piix4) {
sb_scope = aml_scope("_SB");
dev = aml_device("PCI0");
- aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A03")));
- aml_append(dev, aml_name_decl("_ADR", aml_int(0)));
- aml_append(dev, aml_name_decl("_UID", aml_int(pcmc->pci_root_uid)));
+ init_pci_acpi(dev, pcmc->pci_root_uid, PCI, false);
aml_append(sb_scope, dev);
aml_append(dsdt, sb_scope);
@@ -1447,11 +1463,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
} else {
sb_scope = aml_scope("_SB");
dev = aml_device("PCI0");
- aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A08")));
- aml_append(dev, aml_name_decl("_CID", aml_eisaid("PNP0A03")));
- aml_append(dev, aml_name_decl("_ADR", aml_int(0)));
- aml_append(dev, aml_name_decl("_UID", aml_int(pcmc->pci_root_uid)));
- aml_append(dev, build_q35_osc_method(!pm->pcihp_bridge_en));
+ init_pci_acpi(dev, pcmc->pci_root_uid, PCIE, !pm->pcihp_bridge_en);
aml_append(sb_scope, dev);
if (mcfg_valid) {
aml_append(sb_scope, build_q35_dram_controller(&mcfg));
@@ -1562,17 +1574,10 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
scope = aml_scope("\\_SB");
dev = aml_device("PC%.02X", bus_num);
- aml_append(dev, aml_name_decl("_UID", aml_int(bus_num)));
aml_append(dev, aml_name_decl("_BBN", aml_int(bus_num)));
- if (pci_bus_is_express(bus)) {
- aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A08")));
- aml_append(dev, aml_name_decl("_CID", aml_eisaid("PNP0A03")));
- /* Expander bridges do not have ACPI PCI Hot-plug enabled */
- aml_append(dev, build_q35_osc_method(true));
- } else {
- aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A03")));
- }
+ init_pci_acpi(dev, bus_num,
+ pci_bus_is_express(bus) ? PCIE : PCI, true);
if (numa_node != NUMA_NODE_UNASSIGNED) {
aml_append(dev, aml_name_decl("_PXM", aml_int(numa_node)));
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 16/43] tests/acpi: Add update DSDT.viot
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (14 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 15/43] acpi/pci: Consolidate host bridge setup Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-04 14:01 ` Michael S. Tsirkin
2022-02-02 14:10 ` [PATCH v5 17/43] cxl: Machine level control on whether CXL support is enabled Jonathan Cameron
` (28 subsequent siblings)
44 siblings, 1 reply; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
The consolidation of DSDT AML generation for PCI host bridges
lead to some minor ordering changes and the addition of _ADR
with a default of 0 for those case that didn't already have it.
Only DSDT.viot test is affected.
Changes all similar to:
Scope (\_SB)
{
Device (PC30)
{
- Name (_UID, 0x30) // _UID: Unique ID
Name (_BBN, 0x30) // _BBN: BIOS Bus Number
Name (_HID, EisaId ("PNP0A08") /* PCI Express Bus */) // _HID: Hardware ID
Name (_CID, EisaId ("PNP0A03") /* PCI Bus */) // _CID: Compatible ID
+ Name (_ADR, Zero) // _ADR: Address
+ Name (_UID, 0x30) // _UID: Unique ID
Method (_OSC, 4, NotSerialized) // _OSC: Operating System Capabilities
Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
---
tests/data/acpi/q35/DSDT.viot | Bin 9398 -> 9416 bytes
tests/qtest/bios-tables-test-allowed-diff.h | 1 -
2 files changed, 1 deletion(-)
diff --git a/tests/data/acpi/q35/DSDT.viot b/tests/data/acpi/q35/DSDT.viot
index 1c3b4da5cbe81ecab5e1ef50d383b561c5e0f55f..207ac5b9ae4c3a4bc0094c2242d1a1b08771b784 100644
GIT binary patch
delta 139
zcmdnydBT&+CD<k8gbD)#<CBeCu5zLdVlnZ-PVv!A?xF$C#s(bmPELMY6KfQhxC}No
z$Z0Y1qbM*kn0!E9nwKNq(Itq1BR<sAg-ZdbOrCM_F9mK?rG^HRr4><?3V@Yv4pmBI
F0sxp4B{u*7
delta 143
zcmX@%xy_TyCD<ion+gL1<MNGMu5zMYqA~HoPVv!Aj-mn1#s(bmp`I>WlVjy%CeC%7
z+^Kj^(SX5#0jQdxl0g7Ptr1kM!sPw((lEse3<_8k8$uNeOjb|?Dc;<vXwM7)8)+to
diff --git a/tests/qtest/bios-tables-test-allowed-diff.h b/tests/qtest/bios-tables-test-allowed-diff.h
index 08a8095432..dfb8523c8b 100644
--- a/tests/qtest/bios-tables-test-allowed-diff.h
+++ b/tests/qtest/bios-tables-test-allowed-diff.h
@@ -1,2 +1 @@
/* List of comma-separated changed AML files to ignore */
-"tests/data/acpi/q35/DSDT.viot",
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* Re: [PATCH v5 16/43] tests/acpi: Add update DSDT.viot
2022-02-02 14:10 ` [PATCH v5 16/43] tests/acpi: Add update DSDT.viot Jonathan Cameron
@ 2022-02-04 14:01 ` Michael S. Tsirkin
2022-02-07 15:10 ` Igor Mammedov
0 siblings, 1 reply; 54+ messages in thread
From: Michael S. Tsirkin @ 2022-02-04 14:01 UTC (permalink / raw)
To: Jonathan Cameron
Cc: qemu-devel, Alex Bennée, Marcel Apfelbaum, Igor Mammedov,
linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
On Wed, Feb 02, 2022 at 02:10:10PM +0000, Jonathan Cameron wrote:
> From: Jonathan Cameron <jonathan.cameron@huawei.com>
>
> The consolidation of DSDT AML generation for PCI host bridges
> lead to some minor ordering changes and the addition of _ADR
> with a default of 0 for those case that didn't already have it.
> Only DSDT.viot test is affected.
>
> Changes all similar to:
>
> Scope (\_SB)
> {
> Device (PC30)
> {
> - Name (_UID, 0x30) // _UID: Unique ID
> Name (_BBN, 0x30) // _BBN: BIOS Bus Number
> Name (_HID, EisaId ("PNP0A08") /* PCI Express Bus */) // _HID: Hardware ID
> Name (_CID, EisaId ("PNP0A03") /* PCI Bus */) // _CID: Compatible ID
> + Name (_ADR, Zero) // _ADR: Address
> + Name (_UID, 0x30) // _UID: Unique ID
> Method (_OSC, 4, NotSerialized) // _OSC: Operating System Capabilities
>
> Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
A bit worried about _ADR here. It's probably fine as it should be
unused but in the past some changes like that confused windows guests
where they would lose e.g. a static ip config since from their
POV device address changed.
Igor, what do you think?
> ---
> tests/data/acpi/q35/DSDT.viot | Bin 9398 -> 9416 bytes
> tests/qtest/bios-tables-test-allowed-diff.h | 1 -
> 2 files changed, 1 deletion(-)
>
> diff --git a/tests/data/acpi/q35/DSDT.viot b/tests/data/acpi/q35/DSDT.viot
> index 1c3b4da5cbe81ecab5e1ef50d383b561c5e0f55f..207ac5b9ae4c3a4bc0094c2242d1a1b08771b784 100644
> GIT binary patch
> delta 139
> zcmdnydBT&+CD<k8gbD)#<CBeCu5zLdVlnZ-PVv!A?xF$C#s(bmPELMY6KfQhxC}No
> z$Z0Y1qbM*kn0!E9nwKNq(Itq1BR<sAg-ZdbOrCM_F9mK?rG^HRr4><?3V@Yv4pmBI
> F0sxp4B{u*7
>
> delta 143
> zcmX@%xy_TyCD<ion+gL1<MNGMu5zMYqA~HoPVv!Aj-mn1#s(bmp`I>WlVjy%CeC%7
> z+^Kj^(SX5#0jQdxl0g7Ptr1kM!sPw((lEse3<_8k8$uNeOjb|?Dc;<vXwM7)8)+to
>
> diff --git a/tests/qtest/bios-tables-test-allowed-diff.h b/tests/qtest/bios-tables-test-allowed-diff.h
> index 08a8095432..dfb8523c8b 100644
> --- a/tests/qtest/bios-tables-test-allowed-diff.h
> +++ b/tests/qtest/bios-tables-test-allowed-diff.h
> @@ -1,2 +1 @@
> /* List of comma-separated changed AML files to ignore */
> -"tests/data/acpi/q35/DSDT.viot",
> --
> 2.32.0
^ permalink raw reply [flat|nested] 54+ messages in thread* Re: [PATCH v5 16/43] tests/acpi: Add update DSDT.viot
2022-02-04 14:01 ` Michael S. Tsirkin
@ 2022-02-07 15:10 ` Igor Mammedov
2022-02-07 18:19 ` Jonathan Cameron
0 siblings, 1 reply; 54+ messages in thread
From: Igor Mammedov @ 2022-02-07 15:10 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Jonathan Cameron, qemu-devel, Alex Bennée, Marcel Apfelbaum,
linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
On Fri, 4 Feb 2022 09:01:31 -0500
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Wed, Feb 02, 2022 at 02:10:10PM +0000, Jonathan Cameron wrote:
> > From: Jonathan Cameron <jonathan.cameron@huawei.com>
> >
> > The consolidation of DSDT AML generation for PCI host bridges
> > lead to some minor ordering changes and the addition of _ADR
> > with a default of 0 for those case that didn't already have it.
> > Only DSDT.viot test is affected.
> >
> > Changes all similar to:
> >
> > Scope (\_SB)
> > {
> > Device (PC30)
> > {
> > - Name (_UID, 0x30) // _UID: Unique ID
> > Name (_BBN, 0x30) // _BBN: BIOS Bus Number
> > Name (_HID, EisaId ("PNP0A08") /* PCI Express Bus */) // _HID: Hardware ID
> > Name (_CID, EisaId ("PNP0A03") /* PCI Bus */) // _CID: Compatible ID
> > + Name (_ADR, Zero) // _ADR: Address
> > + Name (_UID, 0x30) // _UID: Unique ID
> > Method (_OSC, 4, NotSerialized) // _OSC: Operating System Capabilities
> >
> > Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
>
> A bit worried about _ADR here. It's probably fine as it should be
> unused but in the past some changes like that confused windows guests
> where they would lose e.g. a static ip config since from their
> POV device address changed.
Spec[1] doesn't mention _ADR in context of host bridge(s) at all,
for all I know it shouldn't be there. QEMU inherited it from
SeaBIOS where it is dated to 2008 (as part of large blob adding ACPI for PCI).
Instead of spreading undefined field to other places,
I'd prefer removing it from root host bridge.
But as Michael said it should be very well tested with various guest
OSes.
Jonathan,
Can you compare nic naming (as guest sees it) with current master
and without _ADR on root host bridge?
One way to test it could be
1. start QEMU(master) configure static IP addr on an interface,
and shutdown guest
2. start QEMU(-_ARR) with guest image from step 1 and see if
interface is still there with IP address it was configured.
test matrix should be something like that:
PCI(pc machine),PCI-E (q35 machine)/
Windows 2012-whatever latest Windows, some contemporary linux,
ancient linux (pre 'stable' interface naming) (something like
RHEL6 or any other distro from that era)
1) PCI_Firmware_v3.2_01-26-2015_ts_clean_Firmware_Final
> Igor, what do you think?
>
> > ---
> > tests/data/acpi/q35/DSDT.viot | Bin 9398 -> 9416 bytes
> > tests/qtest/bios-tables-test-allowed-diff.h | 1 -
> > 2 files changed, 1 deletion(-)
> >
> > diff --git a/tests/data/acpi/q35/DSDT.viot b/tests/data/acpi/q35/DSDT.viot
> > index 1c3b4da5cbe81ecab5e1ef50d383b561c5e0f55f..207ac5b9ae4c3a4bc0094c2242d1a1b08771b784 100644
> > GIT binary patch
> > delta 139
> > zcmdnydBT&+CD<k8gbD)#<CBeCu5zLdVlnZ-PVv!A?xF$C#s(bmPELMY6KfQhxC}No
> > z$Z0Y1qbM*kn0!E9nwKNq(Itq1BR<sAg-ZdbOrCM_F9mK?rG^HRr4><?3V@Yv4pmBI
> > F0sxp4B{u*7
> >
> > delta 143
> > zcmX@%xy_TyCD<ion+gL1<MNGMu5zMYqA~HoPVv!Aj-mn1#s(bmp`I>WlVjy%CeC%7
> > z+^Kj^(SX5#0jQdxl0g7Ptr1kM!sPw((lEse3<_8k8$uNeOjb|?Dc;<vXwM7)8)+to
> >
> > diff --git a/tests/qtest/bios-tables-test-allowed-diff.h b/tests/qtest/bios-tables-test-allowed-diff.h
> > index 08a8095432..dfb8523c8b 100644
> > --- a/tests/qtest/bios-tables-test-allowed-diff.h
> > +++ b/tests/qtest/bios-tables-test-allowed-diff.h
> > @@ -1,2 +1 @@
> > /* List of comma-separated changed AML files to ignore */
> > -"tests/data/acpi/q35/DSDT.viot",
> > --
> > 2.32.0
>
^ permalink raw reply [flat|nested] 54+ messages in thread* Re: [PATCH v5 16/43] tests/acpi: Add update DSDT.viot
2022-02-07 15:10 ` Igor Mammedov
@ 2022-02-07 18:19 ` Jonathan Cameron
0 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-07 18:19 UTC (permalink / raw)
To: Igor Mammedov
Cc: Michael S. Tsirkin, qemu-devel, Alex Bennée,
Marcel Apfelbaum, linux-cxl, Ben Widawsky, Peter Maydell,
linuxarm, Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
On Mon, 7 Feb 2022 16:10:14 +0100
Igor Mammedov <imammedo@redhat.com> wrote:
> On Fri, 4 Feb 2022 09:01:31 -0500
> "Michael S. Tsirkin" <mst@redhat.com> wrote:
>
> > On Wed, Feb 02, 2022 at 02:10:10PM +0000, Jonathan Cameron wrote:
> > > From: Jonathan Cameron <jonathan.cameron@huawei.com>
> > >
> > > The consolidation of DSDT AML generation for PCI host bridges
> > > lead to some minor ordering changes and the addition of _ADR
> > > with a default of 0 for those case that didn't already have it.
> > > Only DSDT.viot test is affected.
> > >
> > > Changes all similar to:
> > >
> > > Scope (\_SB)
> > > {
> > > Device (PC30)
> > > {
> > > - Name (_UID, 0x30) // _UID: Unique ID
> > > Name (_BBN, 0x30) // _BBN: BIOS Bus Number
> > > Name (_HID, EisaId ("PNP0A08") /* PCI Express Bus */) // _HID: Hardware ID
> > > Name (_CID, EisaId ("PNP0A03") /* PCI Bus */) // _CID: Compatible ID
> > > + Name (_ADR, Zero) // _ADR: Address
> > > + Name (_UID, 0x30) // _UID: Unique ID
> > > Method (_OSC, 4, NotSerialized) // _OSC: Operating System Capabilities
> > >
> > > Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
> >
> > A bit worried about _ADR here. It's probably fine as it should be
> > unused but in the past some changes like that confused windows guests
> > where they would lose e.g. a static ip config since from their
> > POV device address changed.
>
> Spec[1] doesn't mention _ADR in context of host bridge(s) at all,
> for all I know it shouldn't be there. QEMU inherited it from
> SeaBIOS where it is dated to 2008 (as part of large blob adding ACPI for PCI).
>
> Instead of spreading undefined field to other places,
> I'd prefer removing it from root host bridge.
> But as Michael said it should be very well tested with various guest
> OSes.
>
> Jonathan,
> Can you compare nic naming (as guest sees it) with current master
> and without _ADR on root host bridge?
> One way to test it could be
> 1. start QEMU(master) configure static IP addr on an interface,
> and shutdown guest
> 2. start QEMU(-_ARR) with guest image from step 1 and see if
> interface is still there with IP address it was configured.
>
> test matrix should be something like that:
> PCI(pc machine),PCI-E (q35 machine)/
> Windows 2012-whatever latest Windows, some contemporary linux,
> ancient linux (pre 'stable' interface naming) (something like
> RHEL6 or any other distro from that era)
Hi Igor,
Potentially long term I can run those tests, but short term I'd like
to separate this tidy up from introducing the CXL support.
The tidy up / deduplication is rather less useful than when
first introduced now we've decided to only implement CXL support
for PXBs for the short term. Earlier versions included
the main host bridge on x86 which made this change more helpful.
Thanks for the info on what it would require and
I will hopefully get to this once the CXL emulation is in
place (or someone else will beat me to it!) Not going to be
terribly near the top of my todo list though I'm afraid.
Result for v6 will be that patches 14-16 are dropped and a few changes
to later patches as a result.
Thanks,
Jonathan
>
> 1) PCI_Firmware_v3.2_01-26-2015_ts_clean_Firmware_Final
>
> > Igor, what do you think?
> >
> > > ---
> > > tests/data/acpi/q35/DSDT.viot | Bin 9398 -> 9416 bytes
> > > tests/qtest/bios-tables-test-allowed-diff.h | 1 -
> > > 2 files changed, 1 deletion(-)
> > >
> > > diff --git a/tests/data/acpi/q35/DSDT.viot b/tests/data/acpi/q35/DSDT.viot
> > > index 1c3b4da5cbe81ecab5e1ef50d383b561c5e0f55f..207ac5b9ae4c3a4bc0094c2242d1a1b08771b784 100644
> > > GIT binary patch
> > > delta 139
> > > zcmdnydBT&+CD<k8gbD)#<CBeCu5zLdVlnZ-PVv!A?xF$C#s(bmPELMY6KfQhxC}No
> > > z$Z0Y1qbM*kn0!E9nwKNq(Itq1BR<sAg-ZdbOrCM_F9mK?rG^HRr4><?3V@Yv4pmBI
> > > F0sxp4B{u*7
> > >
> > > delta 143
> > > zcmX@%xy_TyCD<ion+gL1<MNGMu5zMYqA~HoPVv!Aj-mn1#s(bmp`I>WlVjy%CeC%7
> > > z+^Kj^(SX5#0jQdxl0g7Ptr1kM!sPw((lEse3<_8k8$uNeOjb|?Dc;<vXwM7)8)+to
> > >
> > > diff --git a/tests/qtest/bios-tables-test-allowed-diff.h b/tests/qtest/bios-tables-test-allowed-diff.h
> > > index 08a8095432..dfb8523c8b 100644
> > > --- a/tests/qtest/bios-tables-test-allowed-diff.h
> > > +++ b/tests/qtest/bios-tables-test-allowed-diff.h
> > > @@ -1,2 +1 @@
> > > /* List of comma-separated changed AML files to ignore */
> > > -"tests/data/acpi/q35/DSDT.viot",
> > > --
> > > 2.32.0
> >
>
^ permalink raw reply [flat|nested] 54+ messages in thread
* [PATCH v5 17/43] cxl: Machine level control on whether CXL support is enabled
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (15 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 16/43] tests/acpi: Add update DSDT.viot Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 18/43] hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) Jonathan Cameron
` (27 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
There are going to be some potential overheads to CXL enablement,
for example the host bridge region reserved in memory maps.
Add a machine level control so that CXL is disabled by default.
Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
---
v5: From Alex review.
* Set default to false in machine_class_init to avoid
having to do it in all the boards.
hw/core/machine.c | 28 ++++++++++++++++++++++++++++
hw/i386/pc.c | 1 +
include/hw/boards.h | 2 ++
include/hw/cxl/cxl.h | 4 ++++
4 files changed, 35 insertions(+)
diff --git a/hw/core/machine.c b/hw/core/machine.c
index d856485cb4..6ff5dba64e 100644
--- a/hw/core/machine.c
+++ b/hw/core/machine.c
@@ -31,6 +31,7 @@
#include "sysemu/qtest.h"
#include "hw/pci/pci.h"
#include "hw/mem/nvdimm.h"
+#include "hw/cxl/cxl.h"
#include "migration/global_state.h"
#include "migration/vmstate.h"
#include "exec/confidential-guest-support.h"
@@ -545,6 +546,20 @@ static void machine_set_nvdimm_persistence(Object *obj, const char *value,
nvdimms_state->persistence_string = g_strdup(value);
}
+static bool machine_get_cxl(Object *obj, Error **errp)
+{
+ MachineState *ms = MACHINE(obj);
+
+ return ms->cxl_devices_state->is_enabled;
+}
+
+static void machine_set_cxl(Object *obj, bool value, Error **errp)
+{
+ MachineState *ms = MACHINE(obj);
+
+ ms->cxl_devices_state->is_enabled = value;
+}
+
void machine_class_allow_dynamic_sysbus_dev(MachineClass *mc, const char *type)
{
QAPI_LIST_PREPEND(mc->allowed_dynamic_sysbus_devices, g_strdup(type));
@@ -777,6 +792,8 @@ static void machine_class_init(ObjectClass *oc, void *data)
mc->default_ram_size = 128 * MiB;
mc->rom_file_has_mr = true;
+ /* Few machines support CXL, so default to off */
+ mc->cxl_supported = false;
/* numa node memory size aligned on 8MB by default.
* On Linux, each node's border has to be 8MB aligned
*/
@@ -922,6 +939,16 @@ static void machine_initfn(Object *obj)
"Valid values are cpu, mem-ctrl");
}
+ if (mc->cxl_supported) {
+ Object *obj = OBJECT(ms);
+
+ ms->cxl_devices_state = g_new0(CXLState, 1);
+ object_property_add_bool(obj, "cxl", machine_get_cxl, machine_set_cxl);
+ object_property_set_description(obj, "cxl",
+ "Set on/off to enable/disable "
+ "CXL instantiation");
+ }
+
if (mc->cpu_index_to_instance_props && mc->get_default_cpu_node_id) {
ms->numa_state = g_new0(NumaState, 1);
object_property_add_bool(obj, "hmat",
@@ -956,6 +983,7 @@ static void machine_finalize(Object *obj)
g_free(ms->device_memory);
g_free(ms->nvdimms_state);
g_free(ms->numa_state);
+ g_free(ms->cxl_devices_state);
}
bool machine_usb(MachineState *machine)
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index c8696ac01e..b6800a511a 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -1739,6 +1739,7 @@ static void pc_machine_class_init(ObjectClass *oc, void *data)
mc->default_cpu_type = TARGET_DEFAULT_CPU_TYPE;
mc->nvdimm_supported = true;
mc->smp_props.dies_supported = true;
+ mc->cxl_supported = true;
mc->default_ram_id = "pc.ram";
object_class_property_add(oc, PC_MACHINE_MAX_RAM_BELOW_4G, "size",
diff --git a/include/hw/boards.h b/include/hw/boards.h
index c92ac8815c..680718dafc 100644
--- a/include/hw/boards.h
+++ b/include/hw/boards.h
@@ -269,6 +269,7 @@ struct MachineClass {
bool ignore_boot_device_suffixes;
bool smbus_no_migration_support;
bool nvdimm_supported;
+ bool cxl_supported;
bool numa_mem_supported;
bool auto_enable_numa;
SMPCompatProps smp_props;
@@ -360,6 +361,7 @@ struct MachineState {
CPUArchIdList *possible_cpus;
CpuTopology smp;
struct NVDIMMState *nvdimms_state;
+ struct CXLState *cxl_devices_state;
struct NumaState *numa_state;
};
diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
index 554ad93b6b..31af92fd5e 100644
--- a/include/hw/cxl/cxl.h
+++ b/include/hw/cxl/cxl.h
@@ -17,4 +17,8 @@
#define CXL_COMPONENT_REG_BAR_IDX 0
#define CXL_DEVICE_REG_BAR_IDX 2
+typedef struct CXLState {
+ bool is_enabled;
+} CXLState;
+
#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 18/43] hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (16 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 17/43] cxl: Machine level control on whether CXL support is enabled Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 19/43] hw/cxl/rp: Add a root port Jonathan Cameron
` (26 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
CXL host bridges themselves may have MMIO. Since host bridges don't have
a BAR they are treated as special for MMIO. This patch includes
i386/pc support.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Co-developed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/i386/acpi-build.c | 26 +++++++++++---
hw/i386/pc.c | 27 ++++++++++++++-
hw/pci-bridge/pci_expander_bridge.c | 53 ++++++++++++++++++++++++++++-
include/hw/cxl/cxl.h | 4 +++
4 files changed, 104 insertions(+), 6 deletions(-)
diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 09940f6e84..1e1e9b9d38 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -28,6 +28,7 @@
#include "qemu/bitmap.h"
#include "qemu/error-report.h"
#include "hw/pci/pci.h"
+#include "hw/cxl/cxl.h"
#include "hw/core/cpu.h"
#include "target/i386/cpu.h"
#include "hw/misc/pvpanic.h"
@@ -1398,7 +1399,7 @@ static void build_smb0(Aml *table, I2CBus *smbus, int devnr, int func)
aml_append(table, scope);
}
-typedef enum { PCI, PCIE } PCIBusType;
+typedef enum { PCI, PCIE, CXL } PCIBusType;
static void init_pci_acpi(Aml *dev, int uid, PCIBusType type,
bool native_pcie_hp)
{
@@ -1562,22 +1563,30 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
QLIST_FOREACH(bus, &bus->child, sibling) {
uint8_t bus_num = pci_bus_num(bus);
uint8_t numa_node = pci_bus_numa_node(bus);
+ int32_t uid = bus_num; /* TODO: Explicit uid */
+ int type;
/* look only for expander root buses */
if (!pci_bus_is_root(bus)) {
continue;
}
+ type = pci_bus_is_cxl(bus) ? CXL :
+ pci_bus_is_express(bus) ? PCIE : PCI;
+
if (bus_num < root_bus_limit) {
root_bus_limit = bus_num - 1;
}
scope = aml_scope("\\_SB");
- dev = aml_device("PC%.02X", bus_num);
+ if (type == CXL) {
+ dev = aml_device("CL%.02X", uid);
+ } else {
+ dev = aml_device("PC%.02X", bus_num);
+ }
aml_append(dev, aml_name_decl("_BBN", aml_int(bus_num)));
- init_pci_acpi(dev, bus_num,
- pci_bus_is_express(bus) ? PCIE : PCI, true);
+ init_pci_acpi(dev, uid, type, true);
if (numa_node != NUMA_NODE_UNASSIGNED) {
aml_append(dev, aml_name_decl("_PXM", aml_int(numa_node)));
@@ -1589,6 +1598,15 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
aml_append(dev, aml_name_decl("_CRS", crs));
aml_append(scope, dev);
aml_append(dsdt, scope);
+
+ /* Handle the ranges for the PXB expanders */
+ if (type == CXL) {
+ MemoryRegion *mr = &machine->cxl_devices_state->host_mr;
+ uint64_t base = mr->addr;
+
+ crs_range_insert(crs_range_set.mem_ranges, base,
+ base + memory_region_size(mr) - 1);
+ }
}
}
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index b6800a511a..7a18dce529 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -75,6 +75,7 @@
#include "acpi-build.h"
#include "hw/mem/pc-dimm.h"
#include "hw/mem/nvdimm.h"
+#include "hw/cxl/cxl.h"
#include "qapi/error.h"
#include "qapi/qapi-visit-common.h"
#include "qapi/qapi-visit-machine.h"
@@ -815,6 +816,7 @@ void pc_memory_init(PCMachineState *pcms,
MachineClass *mc = MACHINE_GET_CLASS(machine);
PCMachineClass *pcmc = PC_MACHINE_GET_CLASS(pcms);
X86MachineState *x86ms = X86_MACHINE(pcms);
+ hwaddr cxl_base;
assert(machine->ram_size == x86ms->below_4g_mem_size +
x86ms->above_4g_mem_size);
@@ -904,6 +906,26 @@ void pc_memory_init(PCMachineState *pcms,
&machine->device_memory->mr);
}
+ if (machine->cxl_devices_state->is_enabled) {
+ MemoryRegion *mr = &machine->cxl_devices_state->host_mr;
+ hwaddr cxl_size = MiB;
+
+ if (pcmc->has_reserved_memory && machine->device_memory->base) {
+ cxl_base = machine->device_memory->base;
+ if (!pcmc->broken_reserved_end) {
+ cxl_base += memory_region_size(&machine->device_memory->mr);
+ }
+ } else if (pcms->sgx_epc.size != 0) {
+ cxl_base = sgx_epc_above_4g_end(&pcms->sgx_epc);
+ } else {
+ cxl_base = 0x100000000ULL + x86ms->above_4g_mem_size;
+ }
+
+ e820_add_entry(cxl_base, cxl_size, E820_RESERVED);
+ memory_region_init(mr, OBJECT(machine), "cxl_host_reg", cxl_size);
+ memory_region_add_subregion(system_memory, cxl_base, mr);
+ }
+
/* Initialize PC system firmware */
pc_system_firmware_init(pcms, rom_memory);
@@ -964,7 +986,10 @@ uint64_t pc_pci_hole64_start(void)
X86MachineState *x86ms = X86_MACHINE(pcms);
uint64_t hole64_start = 0;
- if (pcmc->has_reserved_memory && ms->device_memory->base) {
+ if (ms->cxl_devices_state->host_mr.addr) {
+ hole64_start = ms->cxl_devices_state->host_mr.addr +
+ memory_region_size(&ms->cxl_devices_state->host_mr);
+ } else if (pcmc->has_reserved_memory && ms->device_memory->base) {
hole64_start = ms->device_memory->base;
if (!pcmc->broken_reserved_end) {
hole64_start += memory_region_size(&ms->device_memory->mr);
diff --git a/hw/pci-bridge/pci_expander_bridge.c b/hw/pci-bridge/pci_expander_bridge.c
index c7a28c7b2e..164bdeff9f 100644
--- a/hw/pci-bridge/pci_expander_bridge.c
+++ b/hw/pci-bridge/pci_expander_bridge.c
@@ -204,6 +204,56 @@ static const TypeInfo pxb_host_info = {
.class_init = pxb_host_class_init,
};
+static void pxb_cxl_realize(DeviceState *dev, Error **errp)
+{
+ MachineState *ms = MACHINE(qdev_get_machine());
+ SysBusDevice *sbd = SYS_BUS_DEVICE(dev);
+ CXLHost *cxl = PXB_CXL_HOST(dev);
+ CXLComponentState *cxl_cstate = &cxl->cxl_cstate;
+ struct MemoryRegion *mr = &cxl_cstate->crb.component_registers;
+ hwaddr offset;
+
+ if (!ms->cxl_devices_state->is_enabled) {
+ error_setg(errp, "Machine does not have cxl=on");
+ return;
+ }
+ cxl_component_register_block_init(OBJECT(dev), cxl_cstate,
+ TYPE_PXB_CXL_HOST);
+ sysbus_init_mmio(sbd, mr);
+
+ offset = memory_region_size(mr) * ms->cxl_devices_state->next_mr_idx;
+ if (offset > memory_region_size(&ms->cxl_devices_state->host_mr)) {
+ error_setg(errp, "Insufficient space for pxb cxl host register space");
+ return;
+ }
+
+ memory_region_add_subregion(&ms->cxl_devices_state->host_mr, offset, mr);
+ ms->cxl_devices_state->next_mr_idx++;
+}
+
+static void pxb_cxl_host_class_init(ObjectClass *class, void *data)
+{
+ DeviceClass *dc = DEVICE_CLASS(class);
+ PCIHostBridgeClass *hc = PCI_HOST_BRIDGE_CLASS(class);
+
+ hc->root_bus_path = pxb_host_root_bus_path;
+ dc->fw_name = "cxl";
+ dc->realize = pxb_cxl_realize;
+ /* Reason: Internal part of the pxb/pxb-pcie device, not usable by itself */
+ dc->user_creatable = false;
+}
+
+/*
+ * This is a device to handle the MMIO for a CXL host bridge. It does nothing
+ * else.
+ */
+static const TypeInfo cxl_host_info = {
+ .name = TYPE_PXB_CXL_HOST,
+ .parent = TYPE_PCI_HOST_BRIDGE,
+ .instance_size = sizeof(CXLHost),
+ .class_init = pxb_cxl_host_class_init,
+};
+
/*
* Registers the PXB bus as a child of pci host root bus.
*/
@@ -294,7 +344,7 @@ static void pxb_dev_realize_common(PCIDevice *dev, enum BusType type,
dev_name = dev->qdev.id;
}
- ds = qdev_new(TYPE_PXB_HOST);
+ ds = qdev_new(type == CXL ? TYPE_PXB_CXL_HOST : TYPE_PXB_HOST);
if (type == PCIE) {
bus = pci_root_bus_new(ds, dev_name, NULL, NULL, 0, TYPE_PXB_PCIE_BUS);
} else if (type == CXL) {
@@ -484,6 +534,7 @@ static void pxb_register_types(void)
type_register_static(&pxb_pcie_bus_info);
type_register_static(&pxb_cxl_bus_info);
type_register_static(&pxb_host_info);
+ type_register_static(&cxl_host_info);
type_register_static(&pxb_dev_info);
type_register_static(&pxb_pcie_dev_info);
type_register_static(&pxb_cxl_dev_info);
diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
index 31af92fd5e..75e5bf71e1 100644
--- a/include/hw/cxl/cxl.h
+++ b/include/hw/cxl/cxl.h
@@ -17,8 +17,12 @@
#define CXL_COMPONENT_REG_BAR_IDX 0
#define CXL_DEVICE_REG_BAR_IDX 2
+#define CXL_WINDOW_MAX 10
+
typedef struct CXLState {
bool is_enabled;
+ MemoryRegion host_mr;
+ unsigned int next_mr_idx;
} CXLState;
#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 19/43] hw/cxl/rp: Add a root port
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (17 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 18/43] hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142) Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 20/43] hw/cxl/device: Add a memory device (8.2.8.5) Jonathan Cameron
` (25 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
This adds just enough of a root port implementation to be able to
enumerate root ports (creating the required DVSEC entries). What's not
here yet is the MMIO nor the ability to write some of the DVSEC entries.
This can be added with the qemu commandline by adding a rootport to a
specific CXL host bridge. For example:
-device cxl-rp,id=rp0,bus="cxl.0",addr=0.0,chassis=4
Like the host bridge patch, the ACPI tables aren't generated at this
point and so system software cannot use it.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/pci-bridge/Kconfig | 5 +
hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++++++++++++++++++++++
hw/pci-bridge/meson.build | 1 +
hw/pci-bridge/pcie_root_port.c | 6 +-
hw/pci/pci.c | 4 +-
5 files changed, 245 insertions(+), 2 deletions(-)
create mode 100644 hw/pci-bridge/cxl_root_port.c
diff --git a/hw/pci-bridge/Kconfig b/hw/pci-bridge/Kconfig
index f8df4315ba..02614f49aa 100644
--- a/hw/pci-bridge/Kconfig
+++ b/hw/pci-bridge/Kconfig
@@ -27,3 +27,8 @@ config DEC_PCI
config SIMBA
bool
+
+config CXL
+ bool
+ default y if PCI_EXPRESS && PXB
+ depends on PCI_EXPRESS && MSI_NONBROKEN && PXB
diff --git a/hw/pci-bridge/cxl_root_port.c b/hw/pci-bridge/cxl_root_port.c
new file mode 100644
index 0000000000..dd714db836
--- /dev/null
+++ b/hw/pci-bridge/cxl_root_port.c
@@ -0,0 +1,231 @@
+/*
+ * CXL 2.0 Root Port Implementation
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/log.h"
+#include "qemu/range.h"
+#include "hw/pci/pci_bridge.h"
+#include "hw/pci/pcie_port.h"
+#include "hw/qdev-properties.h"
+#include "hw/sysbus.h"
+#include "qapi/error.h"
+#include "hw/cxl/cxl.h"
+
+#define CXL_ROOT_PORT_DID 0x7075
+
+/* Copied from the gen root port which we derive */
+#define GEN_PCIE_ROOT_PORT_AER_OFFSET 0x100
+#define GEN_PCIE_ROOT_PORT_ACS_OFFSET \
+ (GEN_PCIE_ROOT_PORT_AER_OFFSET + PCI_ERR_SIZEOF)
+#define CXL_ROOT_PORT_DVSEC_OFFSET \
+ (GEN_PCIE_ROOT_PORT_ACS_OFFSET + PCI_ACS_SIZEOF)
+
+typedef struct CXLRootPort {
+ /*< private >*/
+ PCIESlot parent_obj;
+
+ CXLComponentState cxl_cstate;
+ PCIResReserve res_reserve;
+} CXLRootPort;
+
+#define TYPE_CXL_ROOT_PORT "cxl-rp"
+DECLARE_INSTANCE_CHECKER(CXLRootPort, CXL_ROOT_PORT, TYPE_CXL_ROOT_PORT)
+
+static void latch_registers(CXLRootPort *crp)
+{
+ uint32_t *reg_state = crp->cxl_cstate.crb.cache_mem_registers;
+
+ cxl_component_register_init_common(reg_state, CXL2_ROOT_PORT);
+}
+
+static void build_dvsecs(CXLComponentState *cxl)
+{
+ uint8_t *dvsec;
+
+ dvsec = (uint8_t *)&(struct cxl_dvsec_port_extensions){ 0 };
+ cxl_component_create_dvsec(cxl, EXTENSIONS_PORT_DVSEC_LENGTH,
+ EXTENSIONS_PORT_DVSEC,
+ EXTENSIONS_PORT_DVSEC_REVID, dvsec);
+
+ dvsec = (uint8_t *)&(struct cxl_dvsec_port_gpf){
+ .rsvd = 0,
+ .phase1_ctrl = 1, /* 1μs timeout */
+ .phase2_ctrl = 1, /* 1μs timeout */
+ };
+ cxl_component_create_dvsec(cxl, GPF_PORT_DVSEC_LENGTH, GPF_PORT_DVSEC,
+ GPF_PORT_DVSEC_REVID, dvsec);
+
+ dvsec = (uint8_t *)&(struct cxl_dvsec_port_flexbus){
+ .cap = 0x26, /* IO, Mem, non-MLD */
+ .ctrl = 0,
+ .status = 0x26, /* same */
+ .rcvd_mod_ts_data_phase1 = 0xef, /* WTF? */
+ };
+ cxl_component_create_dvsec(cxl, PCIE_FLEXBUS_PORT_DVSEC_LENGTH_2_0,
+ PCIE_FLEXBUS_PORT_DVSEC,
+ PCIE_FLEXBUS_PORT_DVSEC_REVID_2_0, dvsec);
+
+ dvsec = (uint8_t *)&(struct cxl_dvsec_register_locator){
+ .rsvd = 0,
+ .reg0_base_lo = RBI_COMPONENT_REG | CXL_COMPONENT_REG_BAR_IDX,
+ .reg0_base_hi = 0,
+ };
+ cxl_component_create_dvsec(cxl, REG_LOC_DVSEC_LENGTH, REG_LOC_DVSEC,
+ REG_LOC_DVSEC_REVID, dvsec);
+}
+
+static void cxl_rp_realize(DeviceState *dev, Error **errp)
+{
+ PCIDevice *pci_dev = PCI_DEVICE(dev);
+ PCIERootPortClass *rpc = PCIE_ROOT_PORT_GET_CLASS(dev);
+ CXLRootPort *crp = CXL_ROOT_PORT(dev);
+ CXLComponentState *cxl_cstate = &crp->cxl_cstate;
+ ComponentRegisters *cregs = &cxl_cstate->crb;
+ MemoryRegion *component_bar = &cregs->component_registers;
+ Error *local_err = NULL;
+
+ rpc->parent_realize(dev, &local_err);
+ if (local_err) {
+ error_propagate(errp, local_err);
+ return;
+ }
+
+ int rc =
+ pci_bridge_qemu_reserve_cap_init(pci_dev, 0, crp->res_reserve, errp);
+ if (rc < 0) {
+ rpc->parent_class.exit(pci_dev);
+ return;
+ }
+
+ if (!crp->res_reserve.io || crp->res_reserve.io == -1) {
+ pci_word_test_and_clear_mask(pci_dev->wmask + PCI_COMMAND,
+ PCI_COMMAND_IO);
+ pci_dev->wmask[PCI_IO_BASE] = 0;
+ pci_dev->wmask[PCI_IO_LIMIT] = 0;
+ }
+
+ cxl_cstate->dvsec_offset = CXL_ROOT_PORT_DVSEC_OFFSET;
+ cxl_cstate->pdev = pci_dev;
+ build_dvsecs(&crp->cxl_cstate);
+
+ cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
+ TYPE_CXL_ROOT_PORT);
+
+ pci_register_bar(pci_dev, CXL_COMPONENT_REG_BAR_IDX,
+ PCI_BASE_ADDRESS_SPACE_MEMORY |
+ PCI_BASE_ADDRESS_MEM_TYPE_64,
+ component_bar);
+}
+
+static void cxl_rp_reset(DeviceState *dev)
+{
+ PCIERootPortClass *rpc = PCIE_ROOT_PORT_GET_CLASS(dev);
+ CXLRootPort *crp = CXL_ROOT_PORT(dev);
+
+ rpc->parent_reset(dev);
+
+ latch_registers(crp);
+}
+
+static Property gen_rp_props[] = {
+ DEFINE_PROP_UINT32("bus-reserve", CXLRootPort, res_reserve.bus, -1),
+ DEFINE_PROP_SIZE("io-reserve", CXLRootPort, res_reserve.io, -1),
+ DEFINE_PROP_SIZE("mem-reserve", CXLRootPort, res_reserve.mem_non_pref, -1),
+ DEFINE_PROP_SIZE("pref32-reserve", CXLRootPort, res_reserve.mem_pref_32,
+ -1),
+ DEFINE_PROP_SIZE("pref64-reserve", CXLRootPort, res_reserve.mem_pref_64,
+ -1),
+ DEFINE_PROP_END_OF_LIST()
+};
+
+static void cxl_rp_dvsec_write_config(PCIDevice *dev, uint32_t addr,
+ uint32_t val, int len)
+{
+ CXLRootPort *crp = CXL_ROOT_PORT(dev);
+
+ if (range_contains(&crp->cxl_cstate.dvsecs[EXTENSIONS_PORT_DVSEC], addr)) {
+ uint8_t *reg = &dev->config[addr];
+ addr -= crp->cxl_cstate.dvsecs[EXTENSIONS_PORT_DVSEC].lob;
+ if (addr == PORT_CONTROL_OFFSET) {
+ if (pci_get_word(reg) & PORT_CONTROL_UNMASK_SBR) {
+ /* unmask SBR */
+ }
+ if (pci_get_word(reg) & PORT_CONTROL_ALT_MEMID_EN) {
+ /* Alt Memory & ID Space Enable */
+ }
+ }
+ }
+}
+
+static void cxl_rp_write_config(PCIDevice *d, uint32_t address, uint32_t val,
+ int len)
+{
+ uint16_t slt_ctl, slt_sta;
+
+ pcie_cap_slot_get(d, &slt_ctl, &slt_sta);
+ pci_bridge_write_config(d, address, val, len);
+ pcie_cap_flr_write_config(d, address, val, len);
+ pcie_cap_slot_write_config(d, slt_ctl, slt_sta, address, val, len);
+ pcie_aer_write_config(d, address, val, len);
+
+ cxl_rp_dvsec_write_config(d, address, val, len);
+}
+
+static void cxl_root_port_class_init(ObjectClass *oc, void *data)
+{
+ DeviceClass *dc = DEVICE_CLASS(oc);
+ PCIDeviceClass *k = PCI_DEVICE_CLASS(oc);
+ PCIERootPortClass *rpc = PCIE_ROOT_PORT_CLASS(oc);
+
+ k->vendor_id = PCI_VENDOR_ID_INTEL;
+ k->device_id = CXL_ROOT_PORT_DID;
+ dc->desc = "CXL Root Port";
+ k->revision = 0;
+ device_class_set_props(dc, gen_rp_props);
+ k->config_write = cxl_rp_write_config;
+
+ device_class_set_parent_realize(dc, cxl_rp_realize, &rpc->parent_realize);
+ device_class_set_parent_reset(dc, cxl_rp_reset, &rpc->parent_reset);
+
+ rpc->aer_offset = GEN_PCIE_ROOT_PORT_AER_OFFSET;
+ rpc->acs_offset = GEN_PCIE_ROOT_PORT_ACS_OFFSET;
+
+ /*
+ * Explain
+ */
+ dc->hotpluggable = false;
+}
+
+static const TypeInfo cxl_root_port_info = {
+ .name = TYPE_CXL_ROOT_PORT,
+ .parent = TYPE_PCIE_ROOT_PORT,
+ .instance_size = sizeof(CXLRootPort),
+ .class_init = cxl_root_port_class_init,
+ .interfaces = (InterfaceInfo[]) {
+ { INTERFACE_CXL_DEVICE },
+ { }
+ },
+};
+
+static void cxl_register(void)
+{
+ type_register_static(&cxl_root_port_info);
+}
+
+type_init(cxl_register);
diff --git a/hw/pci-bridge/meson.build b/hw/pci-bridge/meson.build
index daab8acf2a..b6d26a03d5 100644
--- a/hw/pci-bridge/meson.build
+++ b/hw/pci-bridge/meson.build
@@ -5,6 +5,7 @@ pci_ss.add(when: 'CONFIG_IOH3420', if_true: files('ioh3420.c'))
pci_ss.add(when: 'CONFIG_PCIE_PORT', if_true: files('pcie_root_port.c', 'gen_pcie_root_port.c', 'pcie_pci_bridge.c'))
pci_ss.add(when: 'CONFIG_PXB', if_true: files('pci_expander_bridge.c'))
pci_ss.add(when: 'CONFIG_XIO3130', if_true: files('xio3130_upstream.c', 'xio3130_downstream.c'))
+pci_ss.add(when: 'CONFIG_CXL', if_true: files('cxl_root_port.c'))
# NewWorld PowerMac
pci_ss.add(when: 'CONFIG_DEC_PCI', if_true: files('dec.c'))
diff --git a/hw/pci-bridge/pcie_root_port.c b/hw/pci-bridge/pcie_root_port.c
index f1cfe9d14a..460e48269d 100644
--- a/hw/pci-bridge/pcie_root_port.c
+++ b/hw/pci-bridge/pcie_root_port.c
@@ -67,7 +67,11 @@ static void rp_realize(PCIDevice *d, Error **errp)
int rc;
pci_config_set_interrupt_pin(d->config, 1);
- pci_bridge_initfn(d, TYPE_PCIE_BUS);
+ if (d->cap_present & QEMU_PCIE_CAP_CXL) {
+ pci_bridge_initfn(d, TYPE_CXL_BUS);
+ } else {
+ pci_bridge_initfn(d, TYPE_PCIE_BUS);
+ }
pcie_port_init_reg(d);
rc = pci_bridge_ssvid_init(d, rpc->ssvid_offset, dc->vendor_id,
diff --git a/hw/pci/pci.c b/hw/pci/pci.c
index cafebf6f59..cc4f06937d 100644
--- a/hw/pci/pci.c
+++ b/hw/pci/pci.c
@@ -2708,7 +2708,9 @@ static void pci_device_class_base_init(ObjectClass *klass, void *data)
object_class_dynamic_cast(klass, INTERFACE_CONVENTIONAL_PCI_DEVICE);
ObjectClass *pcie =
object_class_dynamic_cast(klass, INTERFACE_PCIE_DEVICE);
- assert(conventional || pcie);
+ ObjectClass *cxl =
+ object_class_dynamic_cast(klass, INTERFACE_CXL_DEVICE);
+ assert(conventional || pcie || cxl);
}
}
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 20/43] hw/cxl/device: Add a memory device (8.2.8.5)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (18 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 19/43] hw/cxl/rp: Add a root port Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-11 15:50 ` Ben Widawsky
2022-02-02 14:10 ` [PATCH v5 21/43] hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) Jonathan Cameron
` (24 subsequent siblings)
44 siblings, 1 reply; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
A CXL memory device (AKA Type 3) is a CXL component that contains some
combination of volatile and persistent memory. It also implements the
previously defined mailbox interface as well as the memory device
firmware interface.
Although the memory device is configured like a normal PCIe device, the
memory traffic is on an entirely separate bus conceptually (using the
same physical wires as PCIe, but different protocol).
Once the CXL topology is fully configure and address decoders committed,
the guest physical address for the memory device is part of a larger
window which is owned by the platform. The creation of these windows
is later in this series.
The following example will create a 256M device in a 512M window:
-object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
-device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0"
Note: Dropped PCDIMM info interfaces for now. They can be added if
appropriate at a later date.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/cxl/cxl-mailbox-utils.c | 47 ++++++++++
hw/mem/Kconfig | 5 ++
hw/mem/cxl_type3.c | 170 +++++++++++++++++++++++++++++++++++++
hw/mem/meson.build | 1 +
include/hw/cxl/cxl.h | 1 +
include/hw/cxl/cxl_pci.h | 22 +++++
include/hw/pci/pci_ids.h | 1 +
7 files changed, 247 insertions(+)
create mode 100644 hw/mem/cxl_type3.c
diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index 16bb998735..808faec114 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -50,6 +50,8 @@ enum {
LOGS = 0x04,
#define GET_SUPPORTED 0x0
#define GET_LOG 0x1
+ IDENTIFY = 0x40,
+ #define MEMORY_DEVICE 0x0
};
/* 8.2.8.4.5.1 Command Return Codes */
@@ -216,6 +218,48 @@ static ret_code cmd_logs_get_log(struct cxl_cmd *cmd,
return CXL_MBOX_SUCCESS;
}
+/* 8.2.9.5.1.1 */
+static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,
+ CXLDeviceState *cxl_dstate,
+ uint16_t *len)
+{
+ struct {
+ char fw_revision[0x10];
+ uint64_t total_capacity;
+ uint64_t volatile_capacity;
+ uint64_t persistent_capacity;
+ uint64_t partition_align;
+ uint16_t info_event_log_size;
+ uint16_t warning_event_log_size;
+ uint16_t failure_event_log_size;
+ uint16_t fatal_event_log_size;
+ uint32_t lsa_size;
+ uint8_t poison_list_max_mer[3];
+ uint16_t inject_poison_limit;
+ uint8_t poison_caps;
+ uint8_t qos_telemetry_caps;
+ } __attribute__((packed)) *id;
+ _Static_assert(sizeof(*id) == 0x43, "Bad identify size");
+
+ uint64_t size = cxl_dstate->pmem_size;
+
+ if (!QEMU_IS_ALIGNED(size, 256 << 20)) {
+ return CXL_MBOX_INTERNAL_ERROR;
+ }
+
+ id = (void *)cmd->payload;
+ memset(id, 0, sizeof(*id));
+
+ /* PMEM only */
+ snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0);
+
+ id->total_capacity = size / (256 << 20);
+ id->persistent_capacity = size / (256 << 20);
+
+ *len = sizeof(*id);
+ return CXL_MBOX_SUCCESS;
+}
+
#define IMMEDIATE_CONFIG_CHANGE (1 << 1)
#define IMMEDIATE_POLICY_CHANGE (1 << 3)
#define IMMEDIATE_LOG_CHANGE (1 << 4)
@@ -233,8 +277,11 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
[TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE },
[LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 },
[LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 },
+ [IDENTIFY][MEMORY_DEVICE] = { "IDENTIFY_MEMORY_DEVICE",
+ cmd_identify_memory_device, 0, 0 },
};
+
void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
{
uint16_t ret = CXL_MBOX_SUCCESS;
diff --git a/hw/mem/Kconfig b/hw/mem/Kconfig
index 03dbb3c7df..73c5ae8ad9 100644
--- a/hw/mem/Kconfig
+++ b/hw/mem/Kconfig
@@ -11,3 +11,8 @@ config NVDIMM
config SPARSE_MEM
bool
+
+config CXL_MEM_DEVICE
+ bool
+ default y if CXL
+ select MEM_DEVICE
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
new file mode 100644
index 0000000000..c4021d2434
--- /dev/null
+++ b/hw/mem/cxl_type3.c
@@ -0,0 +1,170 @@
+#include "qemu/osdep.h"
+#include "qemu/units.h"
+#include "qemu/error-report.h"
+#include "hw/mem/memory-device.h"
+#include "hw/mem/pc-dimm.h"
+#include "hw/pci/pci.h"
+#include "hw/qdev-properties.h"
+#include "qapi/error.h"
+#include "qemu/log.h"
+#include "qemu/module.h"
+#include "qemu/range.h"
+#include "qemu/rcu.h"
+#include "sysemu/hostmem.h"
+#include "hw/cxl/cxl.h"
+
+typedef struct cxl_type3_dev {
+ /* Private */
+ PCIDevice parent_obj;
+
+ /* Properties */
+ uint64_t size;
+ HostMemoryBackend *hostmem;
+
+ /* State */
+ CXLComponentState cxl_cstate;
+ CXLDeviceState cxl_dstate;
+} CXLType3Dev;
+
+#define CT3(obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV)
+
+static void build_dvsecs(CXLType3Dev *ct3d)
+{
+ CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+ uint8_t *dvsec;
+
+ dvsec = (uint8_t *)&(struct cxl_dvsec_device){
+ .cap = 0x1e,
+ .ctrl = 0x6,
+ .status2 = 0x2,
+ .range1_size_hi = 0,
+#ifdef SET_PMEM_PADDR
+ .range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | ct3d->size,
+#else
+ .range1_size_lo = 0x3,
+#endif
+ .range1_base_hi = 0,
+ .range1_base_lo = 0,
+ };
+ cxl_component_create_dvsec(cxl_cstate, PCIE_CXL_DEVICE_DVSEC_LENGTH,
+ PCIE_CXL_DEVICE_DVSEC,
+ PCIE_CXL2_DEVICE_DVSEC_REVID, dvsec);
+
+ dvsec = (uint8_t *)&(struct cxl_dvsec_register_locator){
+ .rsvd = 0,
+ .reg0_base_lo = RBI_COMPONENT_REG | CXL_COMPONENT_REG_BAR_IDX,
+ .reg0_base_hi = 0,
+ .reg1_base_lo = RBI_CXL_DEVICE_REG | CXL_DEVICE_REG_BAR_IDX,
+ .reg1_base_hi = 0,
+ };
+ cxl_component_create_dvsec(cxl_cstate, REG_LOC_DVSEC_LENGTH, REG_LOC_DVSEC,
+ REG_LOC_DVSEC_REVID, dvsec);
+}
+
+static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
+{
+ MemoryRegion *mr;
+
+ if (!ct3d->hostmem) {
+ error_setg(errp, "memdev property must be set");
+ return;
+ }
+
+ mr = host_memory_backend_get_memory(ct3d->hostmem);
+ if (!mr) {
+ error_setg(errp, "memdev property must be set");
+ return;
+ }
+ memory_region_set_nonvolatile(mr, true);
+ memory_region_set_enabled(mr, true);
+ host_memory_backend_set_mapped(ct3d->hostmem, true);
+ ct3d->cxl_dstate.pmem_size = ct3d->hostmem->size;
+}
+
+
+static void ct3_realize(PCIDevice *pci_dev, Error **errp)
+{
+ CXLType3Dev *ct3d = CT3(pci_dev);
+ CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+ ComponentRegisters *regs = &cxl_cstate->crb;
+ MemoryRegion *mr = ®s->component_registers;
+ uint8_t *pci_conf = pci_dev->config;
+
+ if (!ct3d->hostmem) {
+ cxl_setup_memory(ct3d, errp);
+ }
+
+ pci_config_set_prog_interface(pci_conf, 0x10);
+ pci_config_set_class(pci_conf, PCI_CLASS_MEMORY_CXL);
+
+ pcie_endpoint_cap_init(pci_dev, 0x80);
+ cxl_cstate->dvsec_offset = 0x100;
+
+ ct3d->cxl_cstate.pdev = pci_dev;
+ build_dvsecs(ct3d);
+
+ cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
+ TYPE_CXL_TYPE3_DEV);
+
+ pci_register_bar(
+ pci_dev, CXL_COMPONENT_REG_BAR_IDX,
+ PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, mr);
+
+ cxl_device_register_block_init(OBJECT(pci_dev), &ct3d->cxl_dstate);
+ pci_register_bar(pci_dev, CXL_DEVICE_REG_BAR_IDX,
+ PCI_BASE_ADDRESS_SPACE_MEMORY |
+ PCI_BASE_ADDRESS_MEM_TYPE_64,
+ &ct3d->cxl_dstate.device_registers);
+}
+
+static void ct3d_reset(DeviceState *dev)
+{
+ CXLType3Dev *ct3d = CT3(dev);
+ uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+
+ cxl_component_register_init_common(reg_state, CXL2_TYPE3_DEVICE);
+ cxl_device_register_init_common(&ct3d->cxl_dstate);
+}
+
+static Property ct3_props[] = {
+ DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
+ DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
+ HostMemoryBackend *),
+ DEFINE_PROP_END_OF_LIST(),
+};
+
+static void ct3_class_init(ObjectClass *oc, void *data)
+{
+ DeviceClass *dc = DEVICE_CLASS(oc);
+ PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
+
+ pc->realize = ct3_realize;
+ pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
+ pc->vendor_id = PCI_VENDOR_ID_INTEL;
+ pc->device_id = 0xd93; /* LVF for now */
+ pc->revision = 1;
+
+ set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
+ dc->desc = "CXL PMEM Device (Type 3)";
+ dc->reset = ct3d_reset;
+ device_class_set_props(dc, ct3_props);
+}
+
+static const TypeInfo ct3d_info = {
+ .name = TYPE_CXL_TYPE3_DEV,
+ .parent = TYPE_PCI_DEVICE,
+ .class_init = ct3_class_init,
+ .instance_size = sizeof(CXLType3Dev),
+ .interfaces = (InterfaceInfo[]) {
+ { INTERFACE_CXL_DEVICE },
+ { INTERFACE_PCIE_DEVICE },
+ {}
+ },
+};
+
+static void ct3d_registers(void)
+{
+ type_register_static(&ct3d_info);
+}
+
+type_init(ct3d_registers);
diff --git a/hw/mem/meson.build b/hw/mem/meson.build
index 82f86d117e..609b2b36fc 100644
--- a/hw/mem/meson.build
+++ b/hw/mem/meson.build
@@ -3,6 +3,7 @@ mem_ss.add(files('memory-device.c'))
mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
+mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
index 75e5bf71e1..6889362230 100644
--- a/include/hw/cxl/cxl.h
+++ b/include/hw/cxl/cxl.h
@@ -17,6 +17,7 @@
#define CXL_COMPONENT_REG_BAR_IDX 0
#define CXL_DEVICE_REG_BAR_IDX 2
+#define TYPE_CXL_TYPE3_DEV "cxl-type3"
#define CXL_WINDOW_MAX 10
typedef struct CXLState {
diff --git a/include/hw/cxl/cxl_pci.h b/include/hw/cxl/cxl_pci.h
index 40c7329afe..e8235b10cc 100644
--- a/include/hw/cxl/cxl_pci.h
+++ b/include/hw/cxl/cxl_pci.h
@@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
* CXL 2.0 Downstream Port: 3, 4, 7, 8
*/
+/* CXL 2.0 - 8.1.3 (ID 0001) */
+struct cxl_dvsec_device {
+ struct dvsec_header hdr;
+ uint16_t cap;
+ uint16_t ctrl;
+ uint16_t status;
+ uint16_t ctrl2;
+ uint16_t status2;
+ uint16_t lock;
+ uint16_t cap2;
+ uint32_t range1_size_hi;
+ uint32_t range1_size_lo;
+ uint32_t range1_base_hi;
+ uint32_t range1_base_lo;
+ uint32_t range2_size_hi;
+ uint32_t range2_size_lo;
+ uint32_t range2_base_hi;
+ uint32_t range2_base_lo;
+};
+_Static_assert(sizeof(struct cxl_dvsec_device) == 0x38,
+ "dvsec device size incorrect");
+
/* CXL 2.0 - 8.1.5 (ID 0003) */
struct cxl_dvsec_port_extensions {
struct dvsec_header hdr;
diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
index 11abe22d46..898083b86f 100644
--- a/include/hw/pci/pci_ids.h
+++ b/include/hw/pci/pci_ids.h
@@ -53,6 +53,7 @@
#define PCI_BASE_CLASS_MEMORY 0x05
#define PCI_CLASS_MEMORY_RAM 0x0500
#define PCI_CLASS_MEMORY_FLASH 0x0501
+#define PCI_CLASS_MEMORY_CXL 0x0502
#define PCI_CLASS_MEMORY_OTHER 0x0580
#define PCI_BASE_CLASS_BRIDGE 0x06
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* Re: [PATCH v5 20/43] hw/cxl/device: Add a memory device (8.2.8.5)
2022-02-02 14:10 ` [PATCH v5 20/43] hw/cxl/device: Add a memory device (8.2.8.5) Jonathan Cameron
@ 2022-02-11 15:50 ` Ben Widawsky
2022-02-11 16:45 ` Jonathan Cameron
0 siblings, 1 reply; 54+ messages in thread
From: Ben Widawsky @ 2022-02-11 15:50 UTC (permalink / raw)
To: Jonathan Cameron
Cc: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov, linux-cxl, Peter Maydell,
linuxarm, Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
On 22-02-02 14:10:14, Jonathan Cameron wrote:
> From: Ben Widawsky <ben.widawsky@intel.com>
>
> A CXL memory device (AKA Type 3) is a CXL component that contains some
> combination of volatile and persistent memory. It also implements the
> previously defined mailbox interface as well as the memory device
> firmware interface.
>
> Although the memory device is configured like a normal PCIe device, the
> memory traffic is on an entirely separate bus conceptually (using the
> same physical wires as PCIe, but different protocol).
>
> Once the CXL topology is fully configure and address decoders committed,
> the guest physical address for the memory device is part of a larger
> window which is owned by the platform. The creation of these windows
> is later in this series.
>
> The following example will create a 256M device in a 512M window:
> -object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
> -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0"
>
> Note: Dropped PCDIMM info interfaces for now. They can be added if
> appropriate at a later date.
>
> Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
> hw/cxl/cxl-mailbox-utils.c | 47 ++++++++++
> hw/mem/Kconfig | 5 ++
> hw/mem/cxl_type3.c | 170 +++++++++++++++++++++++++++++++++++++
> hw/mem/meson.build | 1 +
> include/hw/cxl/cxl.h | 1 +
> include/hw/cxl/cxl_pci.h | 22 +++++
> include/hw/pci/pci_ids.h | 1 +
> 7 files changed, 247 insertions(+)
> create mode 100644 hw/mem/cxl_type3.c
>
> diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
> index 16bb998735..808faec114 100644
> --- a/hw/cxl/cxl-mailbox-utils.c
> +++ b/hw/cxl/cxl-mailbox-utils.c
> @@ -50,6 +50,8 @@ enum {
> LOGS = 0x04,
> #define GET_SUPPORTED 0x0
> #define GET_LOG 0x1
> + IDENTIFY = 0x40,
> + #define MEMORY_DEVICE 0x0
> };
>
> /* 8.2.8.4.5.1 Command Return Codes */
> @@ -216,6 +218,48 @@ static ret_code cmd_logs_get_log(struct cxl_cmd *cmd,
> return CXL_MBOX_SUCCESS;
> }
>
> +/* 8.2.9.5.1.1 */
> +static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,
> + CXLDeviceState *cxl_dstate,
> + uint16_t *len)
> +{
> + struct {
> + char fw_revision[0x10];
> + uint64_t total_capacity;
> + uint64_t volatile_capacity;
> + uint64_t persistent_capacity;
> + uint64_t partition_align;
> + uint16_t info_event_log_size;
> + uint16_t warning_event_log_size;
> + uint16_t failure_event_log_size;
> + uint16_t fatal_event_log_size;
> + uint32_t lsa_size;
> + uint8_t poison_list_max_mer[3];
> + uint16_t inject_poison_limit;
> + uint8_t poison_caps;
> + uint8_t qos_telemetry_caps;
> + } __attribute__((packed)) *id;
> + _Static_assert(sizeof(*id) == 0x43, "Bad identify size");
> +
> + uint64_t size = cxl_dstate->pmem_size;
> +
> + if (!QEMU_IS_ALIGNED(size, 256 << 20)) {
> + return CXL_MBOX_INTERNAL_ERROR;
> + }
> +
> + id = (void *)cmd->payload;
> + memset(id, 0, sizeof(*id));
> +
> + /* PMEM only */
> + snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0);
> +
> + id->total_capacity = size / (256 << 20);
> + id->persistent_capacity = size / (256 << 20);
> +
> + *len = sizeof(*id);
> + return CXL_MBOX_SUCCESS;
> +}
> +
> #define IMMEDIATE_CONFIG_CHANGE (1 << 1)
> #define IMMEDIATE_POLICY_CHANGE (1 << 3)
> #define IMMEDIATE_LOG_CHANGE (1 << 4)
> @@ -233,8 +277,11 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
> [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE },
> [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 },
> [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 },
> + [IDENTIFY][MEMORY_DEVICE] = { "IDENTIFY_MEMORY_DEVICE",
> + cmd_identify_memory_device, 0, 0 },
> };
>
> +
> void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
> {
> uint16_t ret = CXL_MBOX_SUCCESS;
> diff --git a/hw/mem/Kconfig b/hw/mem/Kconfig
> index 03dbb3c7df..73c5ae8ad9 100644
> --- a/hw/mem/Kconfig
> +++ b/hw/mem/Kconfig
> @@ -11,3 +11,8 @@ config NVDIMM
>
> config SPARSE_MEM
> bool
> +
> +config CXL_MEM_DEVICE
> + bool
> + default y if CXL
> + select MEM_DEVICE
> diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
> new file mode 100644
> index 0000000000..c4021d2434
> --- /dev/null
> +++ b/hw/mem/cxl_type3.c
> @@ -0,0 +1,170 @@
> +#include "qemu/osdep.h"
> +#include "qemu/units.h"
> +#include "qemu/error-report.h"
> +#include "hw/mem/memory-device.h"
> +#include "hw/mem/pc-dimm.h"
> +#include "hw/pci/pci.h"
> +#include "hw/qdev-properties.h"
> +#include "qapi/error.h"
> +#include "qemu/log.h"
> +#include "qemu/module.h"
> +#include "qemu/range.h"
> +#include "qemu/rcu.h"
> +#include "sysemu/hostmem.h"
> +#include "hw/cxl/cxl.h"
> +
> +typedef struct cxl_type3_dev {
> + /* Private */
> + PCIDevice parent_obj;
> +
> + /* Properties */
> + uint64_t size;
> + HostMemoryBackend *hostmem;
> +
> + /* State */
> + CXLComponentState cxl_cstate;
> + CXLDeviceState cxl_dstate;
> +} CXLType3Dev;
> +
> +#define CT3(obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV)
> +
> +static void build_dvsecs(CXLType3Dev *ct3d)
> +{
> + CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
> + uint8_t *dvsec;
> +
> + dvsec = (uint8_t *)&(struct cxl_dvsec_device){
> + .cap = 0x1e,
> + .ctrl = 0x6,
> + .status2 = 0x2,
> + .range1_size_hi = 0,
> +#ifdef SET_PMEM_PADDR
> + .range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | ct3d->size,
> +#else
> + .range1_size_lo = 0x3,
> +#endif
> + .range1_base_hi = 0,
> + .range1_base_lo = 0,
> + };
> + cxl_component_create_dvsec(cxl_cstate, PCIE_CXL_DEVICE_DVSEC_LENGTH,
> + PCIE_CXL_DEVICE_DVSEC,
> + PCIE_CXL2_DEVICE_DVSEC_REVID, dvsec);
> +
> + dvsec = (uint8_t *)&(struct cxl_dvsec_register_locator){
> + .rsvd = 0,
> + .reg0_base_lo = RBI_COMPONENT_REG | CXL_COMPONENT_REG_BAR_IDX,
> + .reg0_base_hi = 0,
> + .reg1_base_lo = RBI_CXL_DEVICE_REG | CXL_DEVICE_REG_BAR_IDX,
> + .reg1_base_hi = 0,
> + };
> + cxl_component_create_dvsec(cxl_cstate, REG_LOC_DVSEC_LENGTH, REG_LOC_DVSEC,
> + REG_LOC_DVSEC_REVID, dvsec);
> +}
> +
> +static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
> +{
> + MemoryRegion *mr;
> +
> + if (!ct3d->hostmem) {
> + error_setg(errp, "memdev property must be set");
> + return;
> + }
This check can go away with the below.
> +
> + mr = host_memory_backend_get_memory(ct3d->hostmem);
> + if (!mr) {
> + error_setg(errp, "memdev property must be set");
> + return;
> + }
> + memory_region_set_nonvolatile(mr, true);
> + memory_region_set_enabled(mr, true);
> + host_memory_backend_set_mapped(ct3d->hostmem, true);
> + ct3d->cxl_dstate.pmem_size = ct3d->hostmem->size;
> +}
> +
> +
> +static void ct3_realize(PCIDevice *pci_dev, Error **errp)
> +{
> + CXLType3Dev *ct3d = CT3(pci_dev);
> + CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
> + ComponentRegisters *regs = &cxl_cstate->crb;
> + MemoryRegion *mr = ®s->component_registers;
> + uint8_t *pci_conf = pci_dev->config;
> +
> + if (!ct3d->hostmem) {
> + cxl_setup_memory(ct3d, errp);
> + }
I still haven't had time to review, sorry. We did hit this in our driver
development: I think this should be inverted. The original code worked a bit
differently.
if (ct3d->hostmem) {
cxl_setup_memory(ct3d, errp);
}
> +
> + pci_config_set_prog_interface(pci_conf, 0x10);
> + pci_config_set_class(pci_conf, PCI_CLASS_MEMORY_CXL);
> +
> + pcie_endpoint_cap_init(pci_dev, 0x80);
> + cxl_cstate->dvsec_offset = 0x100;
> +
> + ct3d->cxl_cstate.pdev = pci_dev;
> + build_dvsecs(ct3d);
> +
> + cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
> + TYPE_CXL_TYPE3_DEV);
> +
> + pci_register_bar(
> + pci_dev, CXL_COMPONENT_REG_BAR_IDX,
> + PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, mr);
> +
> + cxl_device_register_block_init(OBJECT(pci_dev), &ct3d->cxl_dstate);
> + pci_register_bar(pci_dev, CXL_DEVICE_REG_BAR_IDX,
> + PCI_BASE_ADDRESS_SPACE_MEMORY |
> + PCI_BASE_ADDRESS_MEM_TYPE_64,
> + &ct3d->cxl_dstate.device_registers);
> +}
> +
> +static void ct3d_reset(DeviceState *dev)
> +{
> + CXLType3Dev *ct3d = CT3(dev);
> + uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
> +
> + cxl_component_register_init_common(reg_state, CXL2_TYPE3_DEVICE);
> + cxl_device_register_init_common(&ct3d->cxl_dstate);
> +}
> +
> +static Property ct3_props[] = {
> + DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
> + DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
> + HostMemoryBackend *),
> + DEFINE_PROP_END_OF_LIST(),
> +};
> +
> +static void ct3_class_init(ObjectClass *oc, void *data)
> +{
> + DeviceClass *dc = DEVICE_CLASS(oc);
> + PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
> +
> + pc->realize = ct3_realize;
> + pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
> + pc->vendor_id = PCI_VENDOR_ID_INTEL;
> + pc->device_id = 0xd93; /* LVF for now */
> + pc->revision = 1;
> +
> + set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
> + dc->desc = "CXL PMEM Device (Type 3)";
> + dc->reset = ct3d_reset;
> + device_class_set_props(dc, ct3_props);
> +}
> +
> +static const TypeInfo ct3d_info = {
> + .name = TYPE_CXL_TYPE3_DEV,
> + .parent = TYPE_PCI_DEVICE,
> + .class_init = ct3_class_init,
> + .instance_size = sizeof(CXLType3Dev),
> + .interfaces = (InterfaceInfo[]) {
> + { INTERFACE_CXL_DEVICE },
> + { INTERFACE_PCIE_DEVICE },
> + {}
> + },
> +};
> +
> +static void ct3d_registers(void)
> +{
> + type_register_static(&ct3d_info);
> +}
> +
> +type_init(ct3d_registers);
> diff --git a/hw/mem/meson.build b/hw/mem/meson.build
> index 82f86d117e..609b2b36fc 100644
> --- a/hw/mem/meson.build
> +++ b/hw/mem/meson.build
> @@ -3,6 +3,7 @@ mem_ss.add(files('memory-device.c'))
> mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
> mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
> mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
> +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
>
> softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
>
> diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
> index 75e5bf71e1..6889362230 100644
> --- a/include/hw/cxl/cxl.h
> +++ b/include/hw/cxl/cxl.h
> @@ -17,6 +17,7 @@
> #define CXL_COMPONENT_REG_BAR_IDX 0
> #define CXL_DEVICE_REG_BAR_IDX 2
>
> +#define TYPE_CXL_TYPE3_DEV "cxl-type3"
> #define CXL_WINDOW_MAX 10
>
> typedef struct CXLState {
> diff --git a/include/hw/cxl/cxl_pci.h b/include/hw/cxl/cxl_pci.h
> index 40c7329afe..e8235b10cc 100644
> --- a/include/hw/cxl/cxl_pci.h
> +++ b/include/hw/cxl/cxl_pci.h
> @@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
> * CXL 2.0 Downstream Port: 3, 4, 7, 8
> */
>
> +/* CXL 2.0 - 8.1.3 (ID 0001) */
> +struct cxl_dvsec_device {
> + struct dvsec_header hdr;
> + uint16_t cap;
> + uint16_t ctrl;
> + uint16_t status;
> + uint16_t ctrl2;
> + uint16_t status2;
> + uint16_t lock;
> + uint16_t cap2;
> + uint32_t range1_size_hi;
> + uint32_t range1_size_lo;
> + uint32_t range1_base_hi;
> + uint32_t range1_base_lo;
> + uint32_t range2_size_hi;
> + uint32_t range2_size_lo;
> + uint32_t range2_base_hi;
> + uint32_t range2_base_lo;
> +};
> +_Static_assert(sizeof(struct cxl_dvsec_device) == 0x38,
> + "dvsec device size incorrect");
> +
> /* CXL 2.0 - 8.1.5 (ID 0003) */
> struct cxl_dvsec_port_extensions {
> struct dvsec_header hdr;
> diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
> index 11abe22d46..898083b86f 100644
> --- a/include/hw/pci/pci_ids.h
> +++ b/include/hw/pci/pci_ids.h
> @@ -53,6 +53,7 @@
> #define PCI_BASE_CLASS_MEMORY 0x05
> #define PCI_CLASS_MEMORY_RAM 0x0500
> #define PCI_CLASS_MEMORY_FLASH 0x0501
> +#define PCI_CLASS_MEMORY_CXL 0x0502
> #define PCI_CLASS_MEMORY_OTHER 0x0580
>
> #define PCI_BASE_CLASS_BRIDGE 0x06
> --
> 2.32.0
>
^ permalink raw reply [flat|nested] 54+ messages in thread* Re: [PATCH v5 20/43] hw/cxl/device: Add a memory device (8.2.8.5)
2022-02-11 15:50 ` Ben Widawsky
@ 2022-02-11 16:45 ` Jonathan Cameron
2022-02-11 16:52 ` Ben Widawsky
0 siblings, 1 reply; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-11 16:45 UTC (permalink / raw)
To: Ben Widawsky
Cc: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov, linux-cxl, Peter Maydell,
linuxarm, Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
On Fri, 11 Feb 2022 07:50:00 -0800
Ben Widawsky <ben.widawsky@intel.com> wrote:
> On 22-02-02 14:10:14, Jonathan Cameron wrote:
> > From: Ben Widawsky <ben.widawsky@intel.com>
> >
> > A CXL memory device (AKA Type 3) is a CXL component that contains some
> > combination of volatile and persistent memory. It also implements the
> > previously defined mailbox interface as well as the memory device
> > firmware interface.
> >
> > Although the memory device is configured like a normal PCIe device, the
> > memory traffic is on an entirely separate bus conceptually (using the
> > same physical wires as PCIe, but different protocol).
> >
> > Once the CXL topology is fully configure and address decoders committed,
> > the guest physical address for the memory device is part of a larger
> > window which is owned by the platform. The creation of these windows
> > is later in this series.
> >
> > The following example will create a 256M device in a 512M window:
> > -object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
> > -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0"
> >
> > Note: Dropped PCDIMM info interfaces for now. They can be added if
> > appropriate at a later date.
> >
> > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > ---
> > hw/cxl/cxl-mailbox-utils.c | 47 ++++++++++
> > hw/mem/Kconfig | 5 ++
> > hw/mem/cxl_type3.c | 170 +++++++++++++++++++++++++++++++++++++
> > hw/mem/meson.build | 1 +
> > include/hw/cxl/cxl.h | 1 +
> > include/hw/cxl/cxl_pci.h | 22 +++++
> > include/hw/pci/pci_ids.h | 1 +
> > 7 files changed, 247 insertions(+)
> > create mode 100644 hw/mem/cxl_type3.c
> >
> > diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
> > index 16bb998735..808faec114 100644
> > --- a/hw/cxl/cxl-mailbox-utils.c
> > +++ b/hw/cxl/cxl-mailbox-utils.c
> > @@ -50,6 +50,8 @@ enum {
> > LOGS = 0x04,
> > #define GET_SUPPORTED 0x0
> > #define GET_LOG 0x1
> > + IDENTIFY = 0x40,
> > + #define MEMORY_DEVICE 0x0
> > };
> >
> > /* 8.2.8.4.5.1 Command Return Codes */
> > @@ -216,6 +218,48 @@ static ret_code cmd_logs_get_log(struct cxl_cmd *cmd,
> > return CXL_MBOX_SUCCESS;
> > }
> >
> > +/* 8.2.9.5.1.1 */
> > +static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,
> > + CXLDeviceState *cxl_dstate,
> > + uint16_t *len)
> > +{
> > + struct {
> > + char fw_revision[0x10];
> > + uint64_t total_capacity;
> > + uint64_t volatile_capacity;
> > + uint64_t persistent_capacity;
> > + uint64_t partition_align;
> > + uint16_t info_event_log_size;
> > + uint16_t warning_event_log_size;
> > + uint16_t failure_event_log_size;
> > + uint16_t fatal_event_log_size;
> > + uint32_t lsa_size;
> > + uint8_t poison_list_max_mer[3];
> > + uint16_t inject_poison_limit;
> > + uint8_t poison_caps;
> > + uint8_t qos_telemetry_caps;
> > + } __attribute__((packed)) *id;
> > + _Static_assert(sizeof(*id) == 0x43, "Bad identify size");
> > +
> > + uint64_t size = cxl_dstate->pmem_size;
> > +
> > + if (!QEMU_IS_ALIGNED(size, 256 << 20)) {
> > + return CXL_MBOX_INTERNAL_ERROR;
> > + }
> > +
> > + id = (void *)cmd->payload;
> > + memset(id, 0, sizeof(*id));
> > +
> > + /* PMEM only */
> > + snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0);
> > +
> > + id->total_capacity = size / (256 << 20);
> > + id->persistent_capacity = size / (256 << 20);
> > +
> > + *len = sizeof(*id);
> > + return CXL_MBOX_SUCCESS;
> > +}
> > +
> > #define IMMEDIATE_CONFIG_CHANGE (1 << 1)
> > #define IMMEDIATE_POLICY_CHANGE (1 << 3)
> > #define IMMEDIATE_LOG_CHANGE (1 << 4)
> > @@ -233,8 +277,11 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
> > [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE },
> > [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 },
> > [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 },
> > + [IDENTIFY][MEMORY_DEVICE] = { "IDENTIFY_MEMORY_DEVICE",
> > + cmd_identify_memory_device, 0, 0 },
> > };
> >
> > +
> > void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
> > {
> > uint16_t ret = CXL_MBOX_SUCCESS;
> > diff --git a/hw/mem/Kconfig b/hw/mem/Kconfig
> > index 03dbb3c7df..73c5ae8ad9 100644
> > --- a/hw/mem/Kconfig
> > +++ b/hw/mem/Kconfig
> > @@ -11,3 +11,8 @@ config NVDIMM
> >
> > config SPARSE_MEM
> > bool
> > +
> > +config CXL_MEM_DEVICE
> > + bool
> > + default y if CXL
> > + select MEM_DEVICE
> > diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
> > new file mode 100644
> > index 0000000000..c4021d2434
> > --- /dev/null
> > +++ b/hw/mem/cxl_type3.c
> > @@ -0,0 +1,170 @@
> > +#include "qemu/osdep.h"
> > +#include "qemu/units.h"
> > +#include "qemu/error-report.h"
> > +#include "hw/mem/memory-device.h"
> > +#include "hw/mem/pc-dimm.h"
> > +#include "hw/pci/pci.h"
> > +#include "hw/qdev-properties.h"
> > +#include "qapi/error.h"
> > +#include "qemu/log.h"
> > +#include "qemu/module.h"
> > +#include "qemu/range.h"
> > +#include "qemu/rcu.h"
> > +#include "sysemu/hostmem.h"
> > +#include "hw/cxl/cxl.h"
> > +
> > +typedef struct cxl_type3_dev {
> > + /* Private */
> > + PCIDevice parent_obj;
> > +
> > + /* Properties */
> > + uint64_t size;
> > + HostMemoryBackend *hostmem;
> > +
> > + /* State */
> > + CXLComponentState cxl_cstate;
> > + CXLDeviceState cxl_dstate;
> > +} CXLType3Dev;
> > +
> > +#define CT3(obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV)
> > +
> > +static void build_dvsecs(CXLType3Dev *ct3d)
> > +{
> > + CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
> > + uint8_t *dvsec;
> > +
> > + dvsec = (uint8_t *)&(struct cxl_dvsec_device){
> > + .cap = 0x1e,
> > + .ctrl = 0x6,
> > + .status2 = 0x2,
> > + .range1_size_hi = 0,
> > +#ifdef SET_PMEM_PADDR
> > + .range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | ct3d->size,
> > +#else
> > + .range1_size_lo = 0x3,
> > +#endif
> > + .range1_base_hi = 0,
> > + .range1_base_lo = 0,
> > + };
> > + cxl_component_create_dvsec(cxl_cstate, PCIE_CXL_DEVICE_DVSEC_LENGTH,
> > + PCIE_CXL_DEVICE_DVSEC,
> > + PCIE_CXL2_DEVICE_DVSEC_REVID, dvsec);
> > +
> > + dvsec = (uint8_t *)&(struct cxl_dvsec_register_locator){
> > + .rsvd = 0,
> > + .reg0_base_lo = RBI_COMPONENT_REG | CXL_COMPONENT_REG_BAR_IDX,
> > + .reg0_base_hi = 0,
> > + .reg1_base_lo = RBI_CXL_DEVICE_REG | CXL_DEVICE_REG_BAR_IDX,
> > + .reg1_base_hi = 0,
> > + };
> > + cxl_component_create_dvsec(cxl_cstate, REG_LOC_DVSEC_LENGTH, REG_LOC_DVSEC,
> > + REG_LOC_DVSEC_REVID, dvsec);
> > +}
> > +
> > +static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
> > +{
> > + MemoryRegion *mr;
> > +
> > + if (!ct3d->hostmem) {
> > + error_setg(errp, "memdev property must be set");
> > + return;
> > + }
>
> This check can go away with the below.
>
> > +
> > + mr = host_memory_backend_get_memory(ct3d->hostmem);
> > + if (!mr) {
> > + error_setg(errp, "memdev property must be set");
> > + return;
> > + }
> > + memory_region_set_nonvolatile(mr, true);
> > + memory_region_set_enabled(mr, true);
> > + host_memory_backend_set_mapped(ct3d->hostmem, true);
> > + ct3d->cxl_dstate.pmem_size = ct3d->hostmem->size;
> > +}
> > +
> > +
> > +static void ct3_realize(PCIDevice *pci_dev, Error **errp)
> > +{
> > + CXLType3Dev *ct3d = CT3(pci_dev);
> > + CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
> > + ComponentRegisters *regs = &cxl_cstate->crb;
> > + MemoryRegion *mr = ®s->component_registers;
> > + uint8_t *pci_conf = pci_dev->config;
> > +
> > + if (!ct3d->hostmem) {
> > + cxl_setup_memory(ct3d, errp);
> > + }
>
> I still haven't had time to review, sorry.
No problem. The first part which is being proposed as possibly getting
picked up shortly is all your code anyway subject to a few trivial
review comment related changes.
Things will get more interesting after that.
> We did hit this in our driver
> development: I think this should be inverted. The original code worked a bit
> differently.
>
> if (ct3d->hostmem) {
> cxl_setup_memory(ct3d, errp);
> }
Good spot. I'll fix this in v7.
I'm curious though, what was result of it not being set? Wrong
pmem_size? Given this wasn't running I'm guessing I don't
need the rest of the stuff in cxl_setup_memory().
>
> > +
> > + pci_config_set_prog_interface(pci_conf, 0x10);
> > + pci_config_set_class(pci_conf, PCI_CLASS_MEMORY_CXL);
> > +
> > + pcie_endpoint_cap_init(pci_dev, 0x80);
> > + cxl_cstate->dvsec_offset = 0x100;
> > +
> > + ct3d->cxl_cstate.pdev = pci_dev;
> > + build_dvsecs(ct3d);
> > +
> > + cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
> > + TYPE_CXL_TYPE3_DEV);
> > +
> > + pci_register_bar(
> > + pci_dev, CXL_COMPONENT_REG_BAR_IDX,
> > + PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, mr);
> > +
> > + cxl_device_register_block_init(OBJECT(pci_dev), &ct3d->cxl_dstate);
> > + pci_register_bar(pci_dev, CXL_DEVICE_REG_BAR_IDX,
> > + PCI_BASE_ADDRESS_SPACE_MEMORY |
> > + PCI_BASE_ADDRESS_MEM_TYPE_64,
> > + &ct3d->cxl_dstate.device_registers);
> > +}
> > +
> > +static void ct3d_reset(DeviceState *dev)
> > +{
> > + CXLType3Dev *ct3d = CT3(dev);
> > + uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
> > +
> > + cxl_component_register_init_common(reg_state, CXL2_TYPE3_DEVICE);
> > + cxl_device_register_init_common(&ct3d->cxl_dstate);
> > +}
> > +
> > +static Property ct3_props[] = {
> > + DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
> > + DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
> > + HostMemoryBackend *),
> > + DEFINE_PROP_END_OF_LIST(),
> > +};
> > +
> > +static void ct3_class_init(ObjectClass *oc, void *data)
> > +{
> > + DeviceClass *dc = DEVICE_CLASS(oc);
> > + PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
> > +
> > + pc->realize = ct3_realize;
> > + pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
> > + pc->vendor_id = PCI_VENDOR_ID_INTEL;
> > + pc->device_id = 0xd93; /* LVF for now */
> > + pc->revision = 1;
> > +
> > + set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
> > + dc->desc = "CXL PMEM Device (Type 3)";
> > + dc->reset = ct3d_reset;
> > + device_class_set_props(dc, ct3_props);
> > +}
> > +
> > +static const TypeInfo ct3d_info = {
> > + .name = TYPE_CXL_TYPE3_DEV,
> > + .parent = TYPE_PCI_DEVICE,
> > + .class_init = ct3_class_init,
> > + .instance_size = sizeof(CXLType3Dev),
> > + .interfaces = (InterfaceInfo[]) {
> > + { INTERFACE_CXL_DEVICE },
> > + { INTERFACE_PCIE_DEVICE },
> > + {}
> > + },
> > +};
> > +
> > +static void ct3d_registers(void)
> > +{
> > + type_register_static(&ct3d_info);
> > +}
> > +
> > +type_init(ct3d_registers);
> > diff --git a/hw/mem/meson.build b/hw/mem/meson.build
> > index 82f86d117e..609b2b36fc 100644
> > --- a/hw/mem/meson.build
> > +++ b/hw/mem/meson.build
> > @@ -3,6 +3,7 @@ mem_ss.add(files('memory-device.c'))
> > mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
> > mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
> > mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
> > +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
> >
> > softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
> >
> > diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
> > index 75e5bf71e1..6889362230 100644
> > --- a/include/hw/cxl/cxl.h
> > +++ b/include/hw/cxl/cxl.h
> > @@ -17,6 +17,7 @@
> > #define CXL_COMPONENT_REG_BAR_IDX 0
> > #define CXL_DEVICE_REG_BAR_IDX 2
> >
> > +#define TYPE_CXL_TYPE3_DEV "cxl-type3"
> > #define CXL_WINDOW_MAX 10
> >
> > typedef struct CXLState {
> > diff --git a/include/hw/cxl/cxl_pci.h b/include/hw/cxl/cxl_pci.h
> > index 40c7329afe..e8235b10cc 100644
> > --- a/include/hw/cxl/cxl_pci.h
> > +++ b/include/hw/cxl/cxl_pci.h
> > @@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
> > * CXL 2.0 Downstream Port: 3, 4, 7, 8
> > */
> >
> > +/* CXL 2.0 - 8.1.3 (ID 0001) */
> > +struct cxl_dvsec_device {
> > + struct dvsec_header hdr;
> > + uint16_t cap;
> > + uint16_t ctrl;
> > + uint16_t status;
> > + uint16_t ctrl2;
> > + uint16_t status2;
> > + uint16_t lock;
> > + uint16_t cap2;
> > + uint32_t range1_size_hi;
> > + uint32_t range1_size_lo;
> > + uint32_t range1_base_hi;
> > + uint32_t range1_base_lo;
> > + uint32_t range2_size_hi;
> > + uint32_t range2_size_lo;
> > + uint32_t range2_base_hi;
> > + uint32_t range2_base_lo;
> > +};
> > +_Static_assert(sizeof(struct cxl_dvsec_device) == 0x38,
> > + "dvsec device size incorrect");
> > +
> > /* CXL 2.0 - 8.1.5 (ID 0003) */
> > struct cxl_dvsec_port_extensions {
> > struct dvsec_header hdr;
> > diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
> > index 11abe22d46..898083b86f 100644
> > --- a/include/hw/pci/pci_ids.h
> > +++ b/include/hw/pci/pci_ids.h
> > @@ -53,6 +53,7 @@
> > #define PCI_BASE_CLASS_MEMORY 0x05
> > #define PCI_CLASS_MEMORY_RAM 0x0500
> > #define PCI_CLASS_MEMORY_FLASH 0x0501
> > +#define PCI_CLASS_MEMORY_CXL 0x0502
> > #define PCI_CLASS_MEMORY_OTHER 0x0580
> >
> > #define PCI_BASE_CLASS_BRIDGE 0x06
> > --
> > 2.32.0
> >
^ permalink raw reply [flat|nested] 54+ messages in thread* Re: [PATCH v5 20/43] hw/cxl/device: Add a memory device (8.2.8.5)
2022-02-11 16:45 ` Jonathan Cameron
@ 2022-02-11 16:52 ` Ben Widawsky
0 siblings, 0 replies; 54+ messages in thread
From: Ben Widawsky @ 2022-02-11 16:52 UTC (permalink / raw)
To: Jonathan Cameron
Cc: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov, linux-cxl, Peter Maydell,
linuxarm, Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
On 22-02-11 16:45:19, Jonathan Cameron wrote:
> On Fri, 11 Feb 2022 07:50:00 -0800
> Ben Widawsky <ben.widawsky@intel.com> wrote:
>
> > On 22-02-02 14:10:14, Jonathan Cameron wrote:
> > > From: Ben Widawsky <ben.widawsky@intel.com>
> > >
> > > A CXL memory device (AKA Type 3) is a CXL component that contains some
> > > combination of volatile and persistent memory. It also implements the
> > > previously defined mailbox interface as well as the memory device
> > > firmware interface.
> > >
> > > Although the memory device is configured like a normal PCIe device, the
> > > memory traffic is on an entirely separate bus conceptually (using the
> > > same physical wires as PCIe, but different protocol).
> > >
> > > Once the CXL topology is fully configure and address decoders committed,
> > > the guest physical address for the memory device is part of a larger
> > > window which is owned by the platform. The creation of these windows
> > > is later in this series.
> > >
> > > The following example will create a 256M device in a 512M window:
> > > -object "memory-backend-file,id=cxl-mem1,share,mem-path=cxl-type3,size=512M"
> > > -device "cxl-type3,bus=rp0,memdev=cxl-mem1,id=cxl-pmem0"
> > >
> > > Note: Dropped PCDIMM info interfaces for now. They can be added if
> > > appropriate at a later date.
> > >
> > > Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
> > > Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > > ---
> > > hw/cxl/cxl-mailbox-utils.c | 47 ++++++++++
> > > hw/mem/Kconfig | 5 ++
> > > hw/mem/cxl_type3.c | 170 +++++++++++++++++++++++++++++++++++++
> > > hw/mem/meson.build | 1 +
> > > include/hw/cxl/cxl.h | 1 +
> > > include/hw/cxl/cxl_pci.h | 22 +++++
> > > include/hw/pci/pci_ids.h | 1 +
> > > 7 files changed, 247 insertions(+)
> > > create mode 100644 hw/mem/cxl_type3.c
> > >
> > > diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
> > > index 16bb998735..808faec114 100644
> > > --- a/hw/cxl/cxl-mailbox-utils.c
> > > +++ b/hw/cxl/cxl-mailbox-utils.c
> > > @@ -50,6 +50,8 @@ enum {
> > > LOGS = 0x04,
> > > #define GET_SUPPORTED 0x0
> > > #define GET_LOG 0x1
> > > + IDENTIFY = 0x40,
> > > + #define MEMORY_DEVICE 0x0
> > > };
> > >
> > > /* 8.2.8.4.5.1 Command Return Codes */
> > > @@ -216,6 +218,48 @@ static ret_code cmd_logs_get_log(struct cxl_cmd *cmd,
> > > return CXL_MBOX_SUCCESS;
> > > }
> > >
> > > +/* 8.2.9.5.1.1 */
> > > +static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,
> > > + CXLDeviceState *cxl_dstate,
> > > + uint16_t *len)
> > > +{
> > > + struct {
> > > + char fw_revision[0x10];
> > > + uint64_t total_capacity;
> > > + uint64_t volatile_capacity;
> > > + uint64_t persistent_capacity;
> > > + uint64_t partition_align;
> > > + uint16_t info_event_log_size;
> > > + uint16_t warning_event_log_size;
> > > + uint16_t failure_event_log_size;
> > > + uint16_t fatal_event_log_size;
> > > + uint32_t lsa_size;
> > > + uint8_t poison_list_max_mer[3];
> > > + uint16_t inject_poison_limit;
> > > + uint8_t poison_caps;
> > > + uint8_t qos_telemetry_caps;
> > > + } __attribute__((packed)) *id;
> > > + _Static_assert(sizeof(*id) == 0x43, "Bad identify size");
> > > +
> > > + uint64_t size = cxl_dstate->pmem_size;
> > > +
> > > + if (!QEMU_IS_ALIGNED(size, 256 << 20)) {
> > > + return CXL_MBOX_INTERNAL_ERROR;
> > > + }
> > > +
> > > + id = (void *)cmd->payload;
> > > + memset(id, 0, sizeof(*id));
> > > +
> > > + /* PMEM only */
> > > + snprintf(id->fw_revision, 0x10, "BWFW VERSION %02d", 0);
> > > +
> > > + id->total_capacity = size / (256 << 20);
> > > + id->persistent_capacity = size / (256 << 20);
> > > +
> > > + *len = sizeof(*id);
> > > + return CXL_MBOX_SUCCESS;
> > > +}
> > > +
> > > #define IMMEDIATE_CONFIG_CHANGE (1 << 1)
> > > #define IMMEDIATE_POLICY_CHANGE (1 << 3)
> > > #define IMMEDIATE_LOG_CHANGE (1 << 4)
> > > @@ -233,8 +277,11 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
> > > [TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE },
> > > [LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 },
> > > [LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 },
> > > + [IDENTIFY][MEMORY_DEVICE] = { "IDENTIFY_MEMORY_DEVICE",
> > > + cmd_identify_memory_device, 0, 0 },
> > > };
> > >
> > > +
> > > void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
> > > {
> > > uint16_t ret = CXL_MBOX_SUCCESS;
> > > diff --git a/hw/mem/Kconfig b/hw/mem/Kconfig
> > > index 03dbb3c7df..73c5ae8ad9 100644
> > > --- a/hw/mem/Kconfig
> > > +++ b/hw/mem/Kconfig
> > > @@ -11,3 +11,8 @@ config NVDIMM
> > >
> > > config SPARSE_MEM
> > > bool
> > > +
> > > +config CXL_MEM_DEVICE
> > > + bool
> > > + default y if CXL
> > > + select MEM_DEVICE
> > > diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
> > > new file mode 100644
> > > index 0000000000..c4021d2434
> > > --- /dev/null
> > > +++ b/hw/mem/cxl_type3.c
> > > @@ -0,0 +1,170 @@
> > > +#include "qemu/osdep.h"
> > > +#include "qemu/units.h"
> > > +#include "qemu/error-report.h"
> > > +#include "hw/mem/memory-device.h"
> > > +#include "hw/mem/pc-dimm.h"
> > > +#include "hw/pci/pci.h"
> > > +#include "hw/qdev-properties.h"
> > > +#include "qapi/error.h"
> > > +#include "qemu/log.h"
> > > +#include "qemu/module.h"
> > > +#include "qemu/range.h"
> > > +#include "qemu/rcu.h"
> > > +#include "sysemu/hostmem.h"
> > > +#include "hw/cxl/cxl.h"
> > > +
> > > +typedef struct cxl_type3_dev {
> > > + /* Private */
> > > + PCIDevice parent_obj;
> > > +
> > > + /* Properties */
> > > + uint64_t size;
> > > + HostMemoryBackend *hostmem;
> > > +
> > > + /* State */
> > > + CXLComponentState cxl_cstate;
> > > + CXLDeviceState cxl_dstate;
> > > +} CXLType3Dev;
> > > +
> > > +#define CT3(obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV)
> > > +
> > > +static void build_dvsecs(CXLType3Dev *ct3d)
> > > +{
> > > + CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
> > > + uint8_t *dvsec;
> > > +
> > > + dvsec = (uint8_t *)&(struct cxl_dvsec_device){
> > > + .cap = 0x1e,
> > > + .ctrl = 0x6,
> > > + .status2 = 0x2,
> > > + .range1_size_hi = 0,
> > > +#ifdef SET_PMEM_PADDR
> > > + .range1_size_lo = (2 << 5) | (2 << 2) | 0x3 | ct3d->size,
> > > +#else
> > > + .range1_size_lo = 0x3,
> > > +#endif
> > > + .range1_base_hi = 0,
> > > + .range1_base_lo = 0,
> > > + };
> > > + cxl_component_create_dvsec(cxl_cstate, PCIE_CXL_DEVICE_DVSEC_LENGTH,
> > > + PCIE_CXL_DEVICE_DVSEC,
> > > + PCIE_CXL2_DEVICE_DVSEC_REVID, dvsec);
> > > +
> > > + dvsec = (uint8_t *)&(struct cxl_dvsec_register_locator){
> > > + .rsvd = 0,
> > > + .reg0_base_lo = RBI_COMPONENT_REG | CXL_COMPONENT_REG_BAR_IDX,
> > > + .reg0_base_hi = 0,
> > > + .reg1_base_lo = RBI_CXL_DEVICE_REG | CXL_DEVICE_REG_BAR_IDX,
> > > + .reg1_base_hi = 0,
> > > + };
> > > + cxl_component_create_dvsec(cxl_cstate, REG_LOC_DVSEC_LENGTH, REG_LOC_DVSEC,
> > > + REG_LOC_DVSEC_REVID, dvsec);
> > > +}
> > > +
> > > +static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
> > > +{
> > > + MemoryRegion *mr;
> > > +
> > > + if (!ct3d->hostmem) {
> > > + error_setg(errp, "memdev property must be set");
> > > + return;
> > > + }
> >
> > This check can go away with the below.
> >
> > > +
> > > + mr = host_memory_backend_get_memory(ct3d->hostmem);
> > > + if (!mr) {
> > > + error_setg(errp, "memdev property must be set");
> > > + return;
> > > + }
> > > + memory_region_set_nonvolatile(mr, true);
> > > + memory_region_set_enabled(mr, true);
> > > + host_memory_backend_set_mapped(ct3d->hostmem, true);
> > > + ct3d->cxl_dstate.pmem_size = ct3d->hostmem->size;
> > > +}
> > > +
> > > +
> > > +static void ct3_realize(PCIDevice *pci_dev, Error **errp)
> > > +{
> > > + CXLType3Dev *ct3d = CT3(pci_dev);
> > > + CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
> > > + ComponentRegisters *regs = &cxl_cstate->crb;
> > > + MemoryRegion *mr = ®s->component_registers;
> > > + uint8_t *pci_conf = pci_dev->config;
> > > +
> > > + if (!ct3d->hostmem) {
> > > + cxl_setup_memory(ct3d, errp);
> > > + }
> >
> > I still haven't had time to review, sorry.
>
> No problem. The first part which is being proposed as possibly getting
> picked up shortly is all your code anyway subject to a few trivial
> review comment related changes.
>
> Things will get more interesting after that.
>
> > We did hit this in our driver
> > development: I think this should be inverted. The original code worked a bit
> > differently.
> >
> > if (ct3d->hostmem) {
> > cxl_setup_memory(ct3d, errp);
> > }
>
> Good spot. I'll fix this in v7.
>
> I'm curious though, what was result of it not being set? Wrong
> pmem_size? Given this wasn't running I'm guessing I don't
> need the rest of the stuff in cxl_setup_memory().
>
We hit it because the ndctl/libcxl work Vishal is doing cares about the pmem
size. The last version I posted of the region driver does not actually care
about how much space the device has available but it would matter for region
creation. There's more work/thought that needs to go in from what I originally
planned based on some feedback from Dan, so I dropped it.
I think the rest is needed IFF you care about the CXL.mem traffic appearing on
the "device" but I don't know for sure.
> >
> > > +
> > > + pci_config_set_prog_interface(pci_conf, 0x10);
> > > + pci_config_set_class(pci_conf, PCI_CLASS_MEMORY_CXL);
> > > +
> > > + pcie_endpoint_cap_init(pci_dev, 0x80);
> > > + cxl_cstate->dvsec_offset = 0x100;
> > > +
> > > + ct3d->cxl_cstate.pdev = pci_dev;
> > > + build_dvsecs(ct3d);
> > > +
> > > + cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
> > > + TYPE_CXL_TYPE3_DEV);
> > > +
> > > + pci_register_bar(
> > > + pci_dev, CXL_COMPONENT_REG_BAR_IDX,
> > > + PCI_BASE_ADDRESS_SPACE_MEMORY | PCI_BASE_ADDRESS_MEM_TYPE_64, mr);
> > > +
> > > + cxl_device_register_block_init(OBJECT(pci_dev), &ct3d->cxl_dstate);
> > > + pci_register_bar(pci_dev, CXL_DEVICE_REG_BAR_IDX,
> > > + PCI_BASE_ADDRESS_SPACE_MEMORY |
> > > + PCI_BASE_ADDRESS_MEM_TYPE_64,
> > > + &ct3d->cxl_dstate.device_registers);
> > > +}
> > > +
> > > +static void ct3d_reset(DeviceState *dev)
> > > +{
> > > + CXLType3Dev *ct3d = CT3(dev);
> > > + uint32_t *reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
> > > +
> > > + cxl_component_register_init_common(reg_state, CXL2_TYPE3_DEVICE);
> > > + cxl_device_register_init_common(&ct3d->cxl_dstate);
> > > +}
> > > +
> > > +static Property ct3_props[] = {
> > > + DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
> > > + DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
> > > + HostMemoryBackend *),
> > > + DEFINE_PROP_END_OF_LIST(),
> > > +};
> > > +
> > > +static void ct3_class_init(ObjectClass *oc, void *data)
> > > +{
> > > + DeviceClass *dc = DEVICE_CLASS(oc);
> > > + PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
> > > +
> > > + pc->realize = ct3_realize;
> > > + pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
> > > + pc->vendor_id = PCI_VENDOR_ID_INTEL;
> > > + pc->device_id = 0xd93; /* LVF for now */
> > > + pc->revision = 1;
> > > +
> > > + set_bit(DEVICE_CATEGORY_STORAGE, dc->categories);
> > > + dc->desc = "CXL PMEM Device (Type 3)";
> > > + dc->reset = ct3d_reset;
> > > + device_class_set_props(dc, ct3_props);
> > > +}
> > > +
> > > +static const TypeInfo ct3d_info = {
> > > + .name = TYPE_CXL_TYPE3_DEV,
> > > + .parent = TYPE_PCI_DEVICE,
> > > + .class_init = ct3_class_init,
> > > + .instance_size = sizeof(CXLType3Dev),
> > > + .interfaces = (InterfaceInfo[]) {
> > > + { INTERFACE_CXL_DEVICE },
> > > + { INTERFACE_PCIE_DEVICE },
> > > + {}
> > > + },
> > > +};
> > > +
> > > +static void ct3d_registers(void)
> > > +{
> > > + type_register_static(&ct3d_info);
> > > +}
> > > +
> > > +type_init(ct3d_registers);
> > > diff --git a/hw/mem/meson.build b/hw/mem/meson.build
> > > index 82f86d117e..609b2b36fc 100644
> > > --- a/hw/mem/meson.build
> > > +++ b/hw/mem/meson.build
> > > @@ -3,6 +3,7 @@ mem_ss.add(files('memory-device.c'))
> > > mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
> > > mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
> > > mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
> > > +mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
> > >
> > > softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
> > >
> > > diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
> > > index 75e5bf71e1..6889362230 100644
> > > --- a/include/hw/cxl/cxl.h
> > > +++ b/include/hw/cxl/cxl.h
> > > @@ -17,6 +17,7 @@
> > > #define CXL_COMPONENT_REG_BAR_IDX 0
> > > #define CXL_DEVICE_REG_BAR_IDX 2
> > >
> > > +#define TYPE_CXL_TYPE3_DEV "cxl-type3"
> > > #define CXL_WINDOW_MAX 10
> > >
> > > typedef struct CXLState {
> > > diff --git a/include/hw/cxl/cxl_pci.h b/include/hw/cxl/cxl_pci.h
> > > index 40c7329afe..e8235b10cc 100644
> > > --- a/include/hw/cxl/cxl_pci.h
> > > +++ b/include/hw/cxl/cxl_pci.h
> > > @@ -64,6 +64,28 @@ _Static_assert(sizeof(struct dvsec_header) == 10,
> > > * CXL 2.0 Downstream Port: 3, 4, 7, 8
> > > */
> > >
> > > +/* CXL 2.0 - 8.1.3 (ID 0001) */
> > > +struct cxl_dvsec_device {
> > > + struct dvsec_header hdr;
> > > + uint16_t cap;
> > > + uint16_t ctrl;
> > > + uint16_t status;
> > > + uint16_t ctrl2;
> > > + uint16_t status2;
> > > + uint16_t lock;
> > > + uint16_t cap2;
> > > + uint32_t range1_size_hi;
> > > + uint32_t range1_size_lo;
> > > + uint32_t range1_base_hi;
> > > + uint32_t range1_base_lo;
> > > + uint32_t range2_size_hi;
> > > + uint32_t range2_size_lo;
> > > + uint32_t range2_base_hi;
> > > + uint32_t range2_base_lo;
> > > +};
> > > +_Static_assert(sizeof(struct cxl_dvsec_device) == 0x38,
> > > + "dvsec device size incorrect");
> > > +
> > > /* CXL 2.0 - 8.1.5 (ID 0003) */
> > > struct cxl_dvsec_port_extensions {
> > > struct dvsec_header hdr;
> > > diff --git a/include/hw/pci/pci_ids.h b/include/hw/pci/pci_ids.h
> > > index 11abe22d46..898083b86f 100644
> > > --- a/include/hw/pci/pci_ids.h
> > > +++ b/include/hw/pci/pci_ids.h
> > > @@ -53,6 +53,7 @@
> > > #define PCI_BASE_CLASS_MEMORY 0x05
> > > #define PCI_CLASS_MEMORY_RAM 0x0500
> > > #define PCI_CLASS_MEMORY_FLASH 0x0501
> > > +#define PCI_CLASS_MEMORY_CXL 0x0502
> > > #define PCI_CLASS_MEMORY_OTHER 0x0580
> > >
> > > #define PCI_BASE_CLASS_BRIDGE 0x06
> > > --
> > > 2.32.0
> > >
>
^ permalink raw reply [flat|nested] 54+ messages in thread
* [PATCH v5 21/43] hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (19 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 20/43] hw/cxl/device: Add a memory device (8.2.8.5) Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 22/43] acpi/cxl: Add _OSC implementation (9.14.2) Jonathan Cameron
` (23 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
A device's volatile and persistent memory are known Host Defined Memory
(HDM) regions. The mechanism by which the device is programmed to claim
the addresses associated with those regions is through dedicated logic
known as the HDM decoder. In order to allow the OS to properly program
the HDMs, the HDM decoders must be modeled.
There are two ways the HDM decoders can be implemented, the legacy
mechanism is through the PCIe DVSEC programming from CXL 1.1 (8.1.3.8),
and MMIO is found in 8.2.5.12 of the spec. For now, 8.1.3.8 is not
implemented.
Much of CXL device logic is implemented in cxl-utils. The HDM decoder
however is implemented directly by the device implementation.
Whilst the implementation currently does no validity checks on the
encoder set up, future work will add sanity checking specific to
the type of cxl component.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Co-developed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/mem/cxl_type3.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 54 insertions(+)
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index c4021d2434..da091157f2 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -61,6 +61,56 @@ static void build_dvsecs(CXLType3Dev *ct3d)
REG_LOC_DVSEC_REVID, dvsec);
}
+static void hdm_decoder_commit(CXLType3Dev *ct3d, int which)
+{
+ ComponentRegisters *cregs = &ct3d->cxl_cstate.crb;
+ uint32_t *cache_mem = cregs->cache_mem_registers;
+
+ assert(which == 0);
+
+ /* TODO: Sanity checks that the decoder is possible */
+ ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, COMMIT, 0);
+ ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, ERR, 0);
+
+ ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, COMMITTED, 1);
+}
+
+static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value,
+ unsigned size)
+{
+ CXLComponentState *cxl_cstate = opaque;
+ ComponentRegisters *cregs = &cxl_cstate->crb;
+ CXLType3Dev *ct3d = container_of(cxl_cstate, CXLType3Dev, cxl_cstate);
+ uint32_t *cache_mem = cregs->cache_mem_registers;
+ bool should_commit = false;
+ int which_hdm = -1;
+
+ assert(size == 4);
+
+ switch (offset) {
+ case A_CXL_HDM_DECODER0_CTRL:
+ should_commit = FIELD_EX32(value, CXL_HDM_DECODER0_CTRL, COMMIT);
+ which_hdm = 0;
+ break;
+ default:
+ break;
+ }
+
+ stl_le_p((uint8_t *)cache_mem + offset, value);
+ if (should_commit) {
+ hdm_decoder_commit(ct3d, which_hdm);
+ }
+}
+
+static void ct3_finalize(Object *obj)
+{
+ CXLType3Dev *ct3d = CT3(obj);
+ CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
+ ComponentRegisters *regs = &cxl_cstate->crb;
+
+ g_free((void *)regs->special_ops);
+}
+
static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
{
MemoryRegion *mr;
@@ -103,6 +153,9 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)
ct3d->cxl_cstate.pdev = pci_dev;
build_dvsecs(ct3d);
+ regs->special_ops = g_new0(MemoryRegionOps, 1);
+ regs->special_ops->write = ct3d_reg_write;
+
cxl_component_register_block_init(OBJECT(pci_dev), cxl_cstate,
TYPE_CXL_TYPE3_DEV);
@@ -155,6 +208,7 @@ static const TypeInfo ct3d_info = {
.parent = TYPE_PCI_DEVICE,
.class_init = ct3_class_init,
.instance_size = sizeof(CXLType3Dev),
+ .instance_finalize = ct3_finalize,
.interfaces = (InterfaceInfo[]) {
{ INTERFACE_CXL_DEVICE },
{ INTERFACE_PCIE_DEVICE },
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 22/43] acpi/cxl: Add _OSC implementation (9.14.2)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (20 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 21/43] hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12) Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 23/43] tests/acpi: allow CEDT table addition Jonathan Cameron
` (22 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
CXL 2.0 specification adds 2 new dwords to the existing _OSC definition
from PCIe. The new dwords are accessed with a new uuid. This
implementation supports what is in the specification.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
v5: Fix for issue seen on in patch 31.
- Introduce stubs as the gpex pxb code is compiled on mips machines.
hw/acpi/Kconfig | 5 ++
hw/acpi/cxl-stub.c | 12 +++++
hw/acpi/cxl.c | 104 ++++++++++++++++++++++++++++++++++++++++++
hw/acpi/meson.build | 4 +-
hw/i386/acpi-build.c | 14 +++++-
include/hw/acpi/cxl.h | 23 ++++++++++
6 files changed, 160 insertions(+), 2 deletions(-)
create mode 100644 hw/acpi/cxl-stub.c
create mode 100644 hw/acpi/cxl.c
create mode 100644 include/hw/acpi/cxl.h
diff --git a/hw/acpi/Kconfig b/hw/acpi/Kconfig
index 622b0b50b7..76cafca652 100644
--- a/hw/acpi/Kconfig
+++ b/hw/acpi/Kconfig
@@ -5,6 +5,7 @@ config ACPI_X86
bool
select ACPI
select ACPI_NVDIMM
+ select ACPI_CXL
select ACPI_CPU_HOTPLUG
select ACPI_MEMORY_HOTPLUG
select ACPI_HMAT
@@ -60,3 +61,7 @@ config ACPI_HW_REDUCED
select ACPI
select ACPI_MEMORY_HOTPLUG
select ACPI_NVDIMM
+
+config ACPI_CXL
+ bool
+ depends on ACPI
diff --git a/hw/acpi/cxl-stub.c b/hw/acpi/cxl-stub.c
new file mode 100644
index 0000000000..15bc21076b
--- /dev/null
+++ b/hw/acpi/cxl-stub.c
@@ -0,0 +1,12 @@
+
+/*
+ * Stubs for ACPI platforms that don't support CXl
+ */
+#include "qemu/osdep.h"
+#include "hw/acpi/aml-build.h"
+#include "hw/acpi/cxl.h"
+
+void build_cxl_osc_method(Aml *dev)
+{
+ g_assert_not_reached();
+}
diff --git a/hw/acpi/cxl.c b/hw/acpi/cxl.c
new file mode 100644
index 0000000000..7124d5a1a3
--- /dev/null
+++ b/hw/acpi/cxl.c
@@ -0,0 +1,104 @@
+/*
+ * CXL ACPI Implementation
+ *
+ * Copyright(C) 2020 Intel Corporation.
+ *
+ * This library is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU Lesser General Public
+ * License as published by the Free Software Foundation; either
+ * version 2 of the License, or (at your option) any later version.
+ *
+ * This library is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * Lesser General Public License for more details.
+ *
+ * You should have received a copy of the GNU Lesser General Public
+ * License along with this library; if not, see <http://www.gnu.org/licenses/>
+ */
+
+#include "qemu/osdep.h"
+#include "hw/cxl/cxl.h"
+#include "hw/acpi/acpi.h"
+#include "hw/acpi/aml-build.h"
+#include "hw/acpi/bios-linker-loader.h"
+#include "hw/acpi/cxl.h"
+#include "qapi/error.h"
+#include "qemu/uuid.h"
+
+static Aml *__build_cxl_osc_method(void)
+{
+ Aml *method, *if_uuid, *else_uuid, *if_arg1_not_1, *if_cxl, *if_caps_masked;
+ Aml *a_ctrl = aml_local(0);
+ Aml *a_cdw1 = aml_name("CDW1");
+
+ method = aml_method("_OSC", 4, AML_NOTSERIALIZED);
+ aml_append(method, aml_create_dword_field(aml_arg(3), aml_int(0), "CDW1"));
+
+ /* 9.14.2.1.4 */
+ if_uuid = aml_if(
+ aml_lor(aml_equal(aml_arg(0),
+ aml_touuid("33DB4D5B-1FF7-401C-9657-7441C03DD766")),
+ aml_equal(aml_arg(0),
+ aml_touuid("68F2D50B-C469-4D8A-BD3D-941A103FD3FC"))));
+ aml_append(if_uuid, aml_create_dword_field(aml_arg(3), aml_int(4), "CDW2"));
+ aml_append(if_uuid, aml_create_dword_field(aml_arg(3), aml_int(8), "CDW3"));
+
+ aml_append(if_uuid, aml_store(aml_name("CDW3"), a_ctrl));
+
+ /* This is all the same as what's used for PCIe */
+ aml_append(if_uuid,
+ aml_and(aml_name("CTRL"), aml_int(0x1F), aml_name("CTRL")));
+
+ if_arg1_not_1 = aml_if(aml_lnot(aml_equal(aml_arg(1), aml_int(0x1))));
+ /* Unknown revision */
+ aml_append(if_arg1_not_1, aml_or(a_cdw1, aml_int(0x08), a_cdw1));
+ aml_append(if_uuid, if_arg1_not_1);
+
+ if_caps_masked = aml_if(aml_lnot(aml_equal(aml_name("CDW3"), a_ctrl)));
+ /* Capability bits were masked */
+ aml_append(if_caps_masked, aml_or(a_cdw1, aml_int(0x10), a_cdw1));
+ aml_append(if_uuid, if_caps_masked);
+
+ aml_append(if_uuid, aml_store(aml_name("CDW2"), aml_name("SUPP")));
+ aml_append(if_uuid, aml_store(aml_name("CDW3"), aml_name("CTRL")));
+
+ if_cxl = aml_if(aml_equal(
+ aml_arg(0), aml_touuid("68F2D50B-C469-4D8A-BD3D-941A103FD3FC")));
+ /* CXL support field */
+ aml_append(if_cxl, aml_create_dword_field(aml_arg(3), aml_int(12), "CDW4"));
+ /* CXL capabilities */
+ aml_append(if_cxl, aml_create_dword_field(aml_arg(3), aml_int(16), "CDW5"));
+ aml_append(if_cxl, aml_store(aml_name("CDW4"), aml_name("SUPC")));
+ aml_append(if_cxl, aml_store(aml_name("CDW5"), aml_name("CTRC")));
+
+ /* CXL 2.0 Port/Device Register access */
+ aml_append(if_cxl,
+ aml_or(aml_name("CDW5"), aml_int(0x1), aml_name("CDW5")));
+ aml_append(if_uuid, if_cxl);
+
+ /* Update DWORD3 (the return value) */
+ aml_append(if_uuid, aml_store(a_ctrl, aml_name("CDW3")));
+
+ aml_append(if_uuid, aml_return(aml_arg(3)));
+ aml_append(method, if_uuid);
+
+ else_uuid = aml_else();
+
+ /* unrecognized uuid */
+ aml_append(else_uuid,
+ aml_or(aml_name("CDW1"), aml_int(0x4), aml_name("CDW1")));
+ aml_append(else_uuid, aml_return(aml_arg(3)));
+ aml_append(method, else_uuid);
+
+ return method;
+}
+
+void build_cxl_osc_method(Aml *dev)
+{
+ aml_append(dev, aml_name_decl("SUPP", aml_int(0)));
+ aml_append(dev, aml_name_decl("CTRL", aml_int(0)));
+ aml_append(dev, aml_name_decl("SUPC", aml_int(0)));
+ aml_append(dev, aml_name_decl("CTRC", aml_int(0)));
+ aml_append(dev, __build_cxl_osc_method());
+}
diff --git a/hw/acpi/meson.build b/hw/acpi/meson.build
index adf6347bc4..b9bc681205 100644
--- a/hw/acpi/meson.build
+++ b/hw/acpi/meson.build
@@ -13,6 +13,7 @@ acpi_ss.add(when: 'CONFIG_ACPI_MEMORY_HOTPLUG', if_false: files('acpi-mem-hotplu
acpi_ss.add(when: 'CONFIG_ACPI_NVDIMM', if_true: files('nvdimm.c'))
acpi_ss.add(when: 'CONFIG_ACPI_NVDIMM', if_false: files('acpi-nvdimm-stub.c'))
acpi_ss.add(when: 'CONFIG_ACPI_PCI', if_true: files('pci.c'))
+acpi_ss.add(when: 'CONFIG_ACPI_CXL', if_true: files('cxl.c'), if_false: files('cxl-stub.c'))
acpi_ss.add(when: 'CONFIG_ACPI_VMGENID', if_true: files('vmgenid.c'))
acpi_ss.add(when: 'CONFIG_ACPI_HW_REDUCED', if_true: files('generic_event_device.c'))
acpi_ss.add(when: 'CONFIG_ACPI_HMAT', if_true: files('hmat.c'))
@@ -30,4 +31,5 @@ softmmu_ss.add_all(when: 'CONFIG_ACPI', if_true: acpi_ss)
softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('acpi-stub.c', 'aml-build-stub.c',
'acpi-x86-stub.c', 'ipmi-stub.c', 'ghes-stub.c',
'acpi-mem-hotplug-stub.c', 'acpi-cpu-hotplug-stub.c',
- 'acpi-pci-hotplug-stub.c', 'acpi-nvdimm-stub.c'))
+ 'acpi-pci-hotplug-stub.c', 'acpi-nvdimm-stub.c',
+ 'cxl-stub.c'))
diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index 1e1e9b9d38..cec7465267 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -65,6 +65,7 @@
#include "hw/acpi/aml-build.h"
#include "hw/acpi/utils.h"
#include "hw/acpi/pci.h"
+#include "hw/acpi/cxl.h"
#include "qom/qom-qobject.h"
#include "hw/i386/amd_iommu.h"
@@ -1407,13 +1408,24 @@ static void init_pci_acpi(Aml *dev, int uid, PCIBusType type,
aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A03")));
aml_append(dev, aml_name_decl("_ADR", aml_int(0)));
aml_append(dev, aml_name_decl("_UID", aml_int(uid)));
- } else {
+ } else if (type == PCIE) {
aml_append(dev, aml_name_decl("_HID", aml_eisaid("PNP0A08")));
aml_append(dev, aml_name_decl("_CID", aml_eisaid("PNP0A03")));
aml_append(dev, aml_name_decl("_ADR", aml_int(0)));
aml_append(dev, aml_name_decl("_UID", aml_int(uid)));
+
/* Expander bridges do not have ACPI PCI Hot-plug enabled */
aml_append(dev, build_q35_osc_method(native_pcie_hp));
+ } else { /* CXL */
+ struct Aml *pkg = aml_package(2);
+
+ aml_append(dev, aml_name_decl("_HID", aml_string("ACPI0016")));
+ aml_append(pkg, aml_eisaid("PNP0A08"));
+ aml_append(pkg, aml_eisaid("PNP0A03"));
+ aml_append(dev, aml_name_decl("_CID", pkg));
+ aml_append(dev, aml_name_decl("_ADR", aml_int(0)));
+ aml_append(dev, aml_name_decl("_UID", aml_int(uid)));
+ build_cxl_osc_method(dev);
}
}
diff --git a/include/hw/acpi/cxl.h b/include/hw/acpi/cxl.h
new file mode 100644
index 0000000000..7b8f3b8a2e
--- /dev/null
+++ b/include/hw/acpi/cxl.h
@@ -0,0 +1,23 @@
+/*
+ * Copyright (C) 2020 Intel Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU General Public License for more details.
+
+ * You should have received a copy of the GNU General Public License along
+ * with this program; if not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef HW_ACPI_CXL_H
+#define HW_ACPI_CXL_H
+
+void build_cxl_osc_method(Aml *dev);
+
+#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 23/43] tests/acpi: allow CEDT table addition
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (21 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 22/43] acpi/cxl: Add _OSC implementation (9.14.2) Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 24/43] acpi/cxl: Create the CEDT (9.14.1) Jonathan Cameron
` (21 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
Following patches will add a new ACPI table, the
CXL Early Discovery Table (CEDT).
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
tests/data/acpi/pc/CEDT | 0
tests/data/acpi/q35/CEDT | 0
tests/qtest/bios-tables-test-allowed-diff.h | 2 ++
3 files changed, 2 insertions(+)
create mode 100644 tests/data/acpi/pc/CEDT
create mode 100644 tests/data/acpi/q35/CEDT
diff --git a/tests/data/acpi/pc/CEDT b/tests/data/acpi/pc/CEDT
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/data/acpi/q35/CEDT b/tests/data/acpi/q35/CEDT
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qtest/bios-tables-test-allowed-diff.h b/tests/qtest/bios-tables-test-allowed-diff.h
index dfb8523c8b..9b07f1e1ff 100644
--- a/tests/qtest/bios-tables-test-allowed-diff.h
+++ b/tests/qtest/bios-tables-test-allowed-diff.h
@@ -1 +1,3 @@
/* List of comma-separated changed AML files to ignore */
+"tests/data/acpi/pc/CEDT",
+"tests/data/acpi/q35/CEDT",
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 24/43] acpi/cxl: Create the CEDT (9.14.1)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (22 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 23/43] tests/acpi: allow CEDT table addition Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 25/43] hw/cxl/device: Add some trivial commands Jonathan Cameron
` (20 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
The CXL Early Discovery Table is defined in the CXL 2.0 specification as
a way for the OS to get CXL specific information from the system
firmware.
CXL 2.0 specification adds an _HID, ACPI0016, for CXL capable host
bridges, with a _CID of PNP0A08 (PCIe host bridge). CXL aware software
is able to use this initiate the proper _OSC method, and get the _UID
which is referenced by the CEDT. Therefore the existence of an ACPI0016
device allows a CXL aware driver perform the necessary actions. For a
CXL capable OS, this works. For a CXL unaware OS, this works.
CEDT awaremess requires more. The motivation for ACPI0017 is to provide
the possibility of having a Linux CXL module that can work on a legacy
Linux kernel. Linux core PCI/ACPI which won't be built as a module,
will see the _CID of PNP0A08 and bind a driver to it. If we later loaded
a driver for ACPI0016, Linux won't be able to bind it to the hardware
because it has already bound the PNP0A08 driver. The ACPI0017 device is
an opportunity to have an object to bind a driver will be used by a
Linux driver to walk the CXL topology and do everything that we would
have preferred to do with ACPI0016.
There is another motivation for an ACPI0017 device which isn't
implemented here. An operating system needs an attach point for a
non-volatile region provider that understands cross-hostbridge
interleaving. Since QEMU emulation doesn't support interleaving yet,
this is more important on the OS side, for now.
As of CXL 2.0 spec, only 1 sub structure is defined, the CXL Host Bridge
Structure (CHBS) which is primarily useful for telling the OS exactly
where the MMIO for the host bridge is.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
v5: Part of this patch moved earlier to fix a reset issue.
hw/acpi/cxl.c | 68 +++++++++++++++++++++++++++++
hw/i386/acpi-build.c | 27 ++++++++++++
hw/pci-bridge/pci_expander_bridge.c | 18 --------
include/hw/acpi/cxl.h | 5 +++
include/hw/pci/pci_bridge.h | 20 +++++++++
5 files changed, 120 insertions(+), 18 deletions(-)
diff --git a/hw/acpi/cxl.c b/hw/acpi/cxl.c
index 7124d5a1a3..442f836a3e 100644
--- a/hw/acpi/cxl.c
+++ b/hw/acpi/cxl.c
@@ -18,7 +18,11 @@
*/
#include "qemu/osdep.h"
+#include "hw/sysbus.h"
+#include "hw/pci/pci_bridge.h"
+#include "hw/pci/pci_host.h"
#include "hw/cxl/cxl.h"
+#include "hw/mem/memory-device.h"
#include "hw/acpi/acpi.h"
#include "hw/acpi/aml-build.h"
#include "hw/acpi/bios-linker-loader.h"
@@ -26,6 +30,70 @@
#include "qapi/error.h"
#include "qemu/uuid.h"
+static void cedt_build_chbs(GArray *table_data, PXBDev *cxl)
+{
+ SysBusDevice *sbd = SYS_BUS_DEVICE(cxl->cxl.cxl_host_bridge);
+ struct MemoryRegion *mr = sbd->mmio[0].memory;
+
+ /* Type */
+ build_append_int_noprefix(table_data, 0, 1);
+
+ /* Reserved */
+ build_append_int_noprefix(table_data, 0, 1);
+
+ /* Record Length */
+ build_append_int_noprefix(table_data, 32, 2);
+
+ /* UID - currently equal to bus number */
+ build_append_int_noprefix(table_data, cxl->bus_nr, 4);
+
+ /* Version */
+ build_append_int_noprefix(table_data, 1, 4);
+
+ /* Reserved */
+ build_append_int_noprefix(table_data, 0, 4);
+
+ /* Base - subregion within a container that is in PA space */
+ build_append_int_noprefix(table_data, mr->container->addr + mr->addr, 8);
+
+ /* Length */
+ build_append_int_noprefix(table_data, memory_region_size(mr), 8);
+}
+
+static int cxl_foreach_pxb_hb(Object *obj, void *opaque)
+{
+ Aml *cedt = opaque;
+
+ if (object_dynamic_cast(obj, TYPE_PXB_CXL_DEVICE)) {
+ cedt_build_chbs(cedt->buf, PXB_CXL_DEV(obj));
+ }
+
+ return 0;
+}
+
+void cxl_build_cedt(MachineState *ms, GArray *table_offsets, GArray *table_data,
+ BIOSLinker *linker, const char *oem_id,
+ const char *oem_table_id)
+{
+ Aml *cedt;
+ AcpiTable table = { .sig = "CEDT", .rev = 1, .oem_id = oem_id,
+ .oem_table_id = oem_table_id };
+
+ acpi_add_table(table_offsets, table_data);
+ acpi_table_begin(&table, table_data);
+ cedt = init_aml_allocator();
+
+ /* reserve space for CEDT header */
+
+ object_child_foreach_recursive(object_get_root(), cxl_foreach_pxb_hb, cedt);
+
+ /* copy AML table into ACPI tables blob and patch header there */
+ g_array_append_vals(table_data, cedt->buf->data, cedt->buf->len);
+ free_aml_allocator();
+
+ acpi_table_end(linker, &table);
+}
+
static Aml *__build_cxl_osc_method(void)
{
Aml *method, *if_uuid, *else_uuid, *if_arg1_not_1, *if_cxl, *if_caps_masked;
diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c
index cec7465267..0479bf4444 100644
--- a/hw/i386/acpi-build.c
+++ b/hw/i386/acpi-build.c
@@ -76,6 +76,8 @@
#include "hw/acpi/hmat.h"
#include "hw/acpi/viot.h"
+#include "hw/acpi/cxl.h"
+
/* These are used to size the ACPI tables for -M pc-i440fx-1.7 and
* -M pc-i440fx-2.0. Even if the actual amount of AML generated grows
* a little bit, there should be plenty of free space since the DSDT
@@ -1429,6 +1431,22 @@ static void init_pci_acpi(Aml *dev, int uid, PCIBusType type,
}
}
+static void build_acpi0017(Aml *table)
+{
+ Aml *dev, *scope, *method;
+
+ scope = aml_scope("_SB");
+ dev = aml_device("CXLM");
+ aml_append(dev, aml_name_decl("_HID", aml_string("ACPI0017")));
+
+ method = aml_method("_STA", 0, AML_NOTSERIALIZED);
+ aml_append(method, aml_return(aml_int(0x01)));
+ aml_append(dev, method);
+
+ aml_append(scope, dev);
+ aml_append(table, scope);
+}
+
static void
build_dsdt(GArray *table_data, BIOSLinker *linker,
AcpiPmInfo *pm, AcpiMiscInfo *misc,
@@ -1448,6 +1466,7 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
#ifdef CONFIG_TPM
TPMIf *tpm = tpm_find();
#endif
+ bool cxl_present = false;
int i;
VMBusBridge *vmbus_bridge = vmbus_bridge_find();
AcpiTable table = { .sig = "DSDT", .rev = 1, .oem_id = x86ms->oem_id,
@@ -1616,12 +1635,17 @@ build_dsdt(GArray *table_data, BIOSLinker *linker,
MemoryRegion *mr = &machine->cxl_devices_state->host_mr;
uint64_t base = mr->addr;
+ cxl_present = true;
crs_range_insert(crs_range_set.mem_ranges, base,
base + memory_region_size(mr) - 1);
}
}
}
+ if (cxl_present) {
+ build_acpi0017(dsdt);
+ }
+
/*
* At this point crs_range_set has all the ranges used by pci
* busses *other* than PCI0. These ranges will be excluded from
@@ -2675,6 +2699,9 @@ void acpi_build(AcpiBuildTables *tables, MachineState *machine)
x86ms->oem_id, x86ms->oem_table_id);
}
+ cxl_build_cedt(machine, table_offsets, tables_blob, tables->linker,
+ x86ms->oem_id, x86ms->oem_table_id);
+
acpi_add_table(table_offsets, tables_blob);
build_waet(tables_blob, tables->linker, x86ms->oem_id, x86ms->oem_table_id);
diff --git a/hw/pci-bridge/pci_expander_bridge.c b/hw/pci-bridge/pci_expander_bridge.c
index 164bdeff9f..9a2710c067 100644
--- a/hw/pci-bridge/pci_expander_bridge.c
+++ b/hw/pci-bridge/pci_expander_bridge.c
@@ -57,30 +57,12 @@ DECLARE_INSTANCE_CHECKER(PXBDev, PXB_DEV,
DECLARE_INSTANCE_CHECKER(PXBDev, PXB_PCIE_DEV,
TYPE_PXB_PCIE_DEVICE)
-#define TYPE_PXB_CXL_DEVICE "pxb-cxl"
-DECLARE_INSTANCE_CHECKER(PXBDev, PXB_CXL_DEV,
- TYPE_PXB_CXL_DEVICE)
-
typedef struct CXLHost {
PCIHostState parent_obj;
CXLComponentState cxl_cstate;
} CXLHost;
-
-struct PXBDev {
- /*< private >*/
- PCIDevice parent_obj;
- /*< public >*/
-
- uint8_t bus_nr;
- uint16_t numa_node;
- bool bypass_iommu;
- struct cxl_dev {
- CXLHost *cxl_host_bridge;
- } cxl;
-};
-
static PXBDev *convert_to_pxb(PCIDevice *dev)
{
/* A CXL PXB's parent bus is PCIe, so the normal check won't work */
diff --git a/include/hw/acpi/cxl.h b/include/hw/acpi/cxl.h
index 7b8f3b8a2e..0c496538c0 100644
--- a/include/hw/acpi/cxl.h
+++ b/include/hw/acpi/cxl.h
@@ -18,6 +18,11 @@
#ifndef HW_ACPI_CXL_H
#define HW_ACPI_CXL_H
+#include "hw/acpi/bios-linker-loader.h"
+
+void cxl_build_cedt(MachineState *ms, GArray *table_offsets, GArray *table_data,
+ BIOSLinker *linker, const char *oem_id,
+ const char *oem_table_id);
void build_cxl_osc_method(Aml *dev);
#endif
diff --git a/include/hw/pci/pci_bridge.h b/include/hw/pci/pci_bridge.h
index 30691a6e57..ba4bafac7c 100644
--- a/include/hw/pci/pci_bridge.h
+++ b/include/hw/pci/pci_bridge.h
@@ -28,6 +28,7 @@
#include "hw/pci/pci.h"
#include "hw/pci/pci_bus.h"
+#include "hw/cxl/cxl.h"
#include "qom/object.h"
typedef struct PCIBridgeWindows PCIBridgeWindows;
@@ -80,6 +81,25 @@ struct PCIBridge {
#define PCI_BRIDGE_DEV_PROP_CHASSIS_NR "chassis_nr"
#define PCI_BRIDGE_DEV_PROP_MSI "msi"
#define PCI_BRIDGE_DEV_PROP_SHPC "shpc"
+typedef struct CXLHost CXLHost;
+
+struct PXBDev {
+ /*< private >*/
+ PCIDevice parent_obj;
+ /*< public >*/
+
+ uint8_t bus_nr;
+ uint16_t numa_node;
+ bool bypass_iommu;
+ struct cxl_dev {
+ CXLHost *cxl_host_bridge; /* Pointer to a CXLHost */
+ } cxl;
+};
+
+typedef struct PXBDev PXBDev;
+#define TYPE_PXB_CXL_DEVICE "pxb-cxl"
+DECLARE_INSTANCE_CHECKER(PXBDev, PXB_CXL_DEV,
+ TYPE_PXB_CXL_DEVICE)
int pci_bridge_ssvid_init(PCIDevice *dev, uint8_t offset,
uint16_t svid, uint16_t ssid,
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 25/43] hw/cxl/device: Add some trivial commands
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (23 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 24/43] acpi/cxl: Create the CEDT (9.14.1) Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 26/43] hw/cxl/device: Plumb real Label Storage Area (LSA) sizing Jonathan Cameron
` (19 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
GET_FW_INFO and GET_PARTITION_INFO, for this emulation, is equivalent to
info already returned in the IDENTIFY command. To have a more robust
implementation, add those.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
v5: Follow through on rework of how mailbox handlers are done.
hw/cxl/cxl-mailbox-utils.c | 69 +++++++++++++++++++++++++++++++++++++-
1 file changed, 68 insertions(+), 1 deletion(-)
diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index 808faec114..d022711b2a 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -44,6 +44,8 @@ enum {
#define CLEAR_RECORDS 0x1
#define GET_INTERRUPT_POLICY 0x2
#define SET_INTERRUPT_POLICY 0x3
+ FIRMWARE_UPDATE = 0x02,
+ #define GET_INFO 0x0
TIMESTAMP = 0x03,
#define GET 0x0
#define SET 0x1
@@ -52,6 +54,8 @@ enum {
#define GET_LOG 0x1
IDENTIFY = 0x40,
#define MEMORY_DEVICE 0x0
+ CCLS = 0x41,
+ #define GET_PARTITION_INFO 0x0
};
/* 8.2.8.4.5.1 Command Return Codes */
@@ -114,6 +118,39 @@ DEFINE_MAILBOX_HANDLER_NOP(events_clear_records);
DEFINE_MAILBOX_HANDLER_ZEROED(events_get_interrupt_policy, 4);
DEFINE_MAILBOX_HANDLER_NOP(events_set_interrupt_policy);
+/* 8.2.9.2.1 */
+static ret_code cmd_firmware_update_get_info(struct cxl_cmd *cmd,
+ CXLDeviceState *cxl_dstate,
+ uint16_t *len)
+{
+ struct {
+ uint8_t slots_supported;
+ uint8_t slot_info;
+ uint8_t caps;
+ uint8_t rsvd[0xd];
+ char fw_rev1[0x10];
+ char fw_rev2[0x10];
+ char fw_rev3[0x10];
+ char fw_rev4[0x10];
+ } __attribute__((packed)) *fw_info;
+ _Static_assert(sizeof(*fw_info) == 0x50, "Bad firmware info size");
+
+ if (cxl_dstate->pmem_size < (256 << 20)) {
+ return CXL_MBOX_INTERNAL_ERROR;
+ }
+
+ fw_info = (void *)cmd->payload;
+ memset(fw_info, 0, sizeof(*fw_info));
+
+ fw_info->slots_supported = 2;
+ fw_info->slot_info = BIT(0) | BIT(3);
+ fw_info->caps = 0;
+ snprintf(fw_info->fw_rev1, 0x10, "BWFW VERSION %02d", 0);
+
+ *len = sizeof(*fw_info);
+ return CXL_MBOX_SUCCESS;
+}
+
/* 8.2.9.3.1 */
static ret_code cmd_timestamp_get(struct cxl_cmd *cmd,
CXLDeviceState *cxl_dstate,
@@ -260,6 +297,33 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,
return CXL_MBOX_SUCCESS;
}
+static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd,
+ CXLDeviceState *cxl_dstate,
+ uint16_t *len)
+{
+ struct {
+ uint64_t active_vmem;
+ uint64_t active_pmem;
+ uint64_t next_vmem;
+ uint64_t next_pmem;
+ } __attribute__((packed)) *part_info = (void *)cmd->payload;
+ _Static_assert(sizeof(*part_info) == 0x20, "Bad get partition info size");
+ uint64_t size = cxl_dstate->pmem_size;
+
+ if (!QEMU_IS_ALIGNED(size, 256 << 20)) {
+ return CXL_MBOX_INTERNAL_ERROR;
+ }
+
+ /* PMEM only */
+ part_info->active_vmem = 0;
+ part_info->next_vmem = 0;
+ part_info->active_pmem = size / (256 << 20);
+ part_info->next_pmem = part_info->active_pmem;
+
+ *len = sizeof(*part_info);
+ return CXL_MBOX_SUCCESS;
+}
+
#define IMMEDIATE_CONFIG_CHANGE (1 << 1)
#define IMMEDIATE_POLICY_CHANGE (1 << 3)
#define IMMEDIATE_LOG_CHANGE (1 << 4)
@@ -273,15 +337,18 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
cmd_events_get_interrupt_policy, 0, 0 },
[EVENTS][SET_INTERRUPT_POLICY] = { "EVENTS_SET_INTERRUPT_POLICY",
cmd_events_set_interrupt_policy, 4, IMMEDIATE_CONFIG_CHANGE },
+ [FIRMWARE_UPDATE][GET_INFO] = { "FIRMWARE_UPDATE_GET_INFO",
+ cmd_firmware_update_get_info, 0, 0 },
[TIMESTAMP][GET] = { "TIMESTAMP_GET", cmd_timestamp_get, 0, 0 },
[TIMESTAMP][SET] = { "TIMESTAMP_SET", cmd_timestamp_set, 8, IMMEDIATE_POLICY_CHANGE },
[LOGS][GET_SUPPORTED] = { "LOGS_GET_SUPPORTED", cmd_logs_get_supported, 0, 0 },
[LOGS][GET_LOG] = { "LOGS_GET_LOG", cmd_logs_get_log, 0x18, 0 },
[IDENTIFY][MEMORY_DEVICE] = { "IDENTIFY_MEMORY_DEVICE",
cmd_identify_memory_device, 0, 0 },
+ [CCLS][GET_PARTITION_INFO] = { "CCLS_GET_PARTITION_INFO",
+ cmd_ccls_get_partition_info, 0, 0 },
};
-
void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
{
uint16_t ret = CXL_MBOX_SUCCESS;
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 26/43] hw/cxl/device: Plumb real Label Storage Area (LSA) sizing
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (24 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 25/43] hw/cxl/device: Add some trivial commands Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 27/43] hw/cxl/device: Implement get/set Label Storage Area (LSA) Jonathan Cameron
` (18 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
This should introduce no change. Subsequent work will make use of this
new class member.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/cxl/cxl-mailbox-utils.c | 3 +++
hw/mem/cxl_type3.c | 24 +++++++++---------------
include/hw/cxl/cxl_device.h | 29 +++++++++++++++++++++++++++++
3 files changed, 41 insertions(+), 15 deletions(-)
diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index d022711b2a..ccf9c3d794 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -278,6 +278,8 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,
} __attribute__((packed)) *id;
_Static_assert(sizeof(*id) == 0x43, "Bad identify size");
+ CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate);
+ CXLType3Class *cvc = CXL_TYPE3_DEV_GET_CLASS(ct3d);
uint64_t size = cxl_dstate->pmem_size;
if (!QEMU_IS_ALIGNED(size, 256 << 20)) {
@@ -292,6 +294,7 @@ static ret_code cmd_identify_memory_device(struct cxl_cmd *cmd,
id->total_capacity = size / (256 << 20);
id->persistent_capacity = size / (256 << 20);
+ id->lsa_size = cvc->get_lsa_size(ct3d);
*len = sizeof(*id);
return CXL_MBOX_SUCCESS;
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index da091157f2..b16262d3cc 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -13,21 +13,6 @@
#include "sysemu/hostmem.h"
#include "hw/cxl/cxl.h"
-typedef struct cxl_type3_dev {
- /* Private */
- PCIDevice parent_obj;
-
- /* Properties */
- uint64_t size;
- HostMemoryBackend *hostmem;
-
- /* State */
- CXLComponentState cxl_cstate;
- CXLDeviceState cxl_dstate;
-} CXLType3Dev;
-
-#define CT3(obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV)
-
static void build_dvsecs(CXLType3Dev *ct3d)
{
CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
@@ -186,10 +171,16 @@ static Property ct3_props[] = {
DEFINE_PROP_END_OF_LIST(),
};
+static uint64_t get_lsa_size(CXLType3Dev *ct3d)
+{
+ return 0;
+}
+
static void ct3_class_init(ObjectClass *oc, void *data)
{
DeviceClass *dc = DEVICE_CLASS(oc);
PCIDeviceClass *pc = PCI_DEVICE_CLASS(oc);
+ CXLType3Class *cvc = CXL_TYPE3_DEV_CLASS(oc);
pc->realize = ct3_realize;
pc->class_id = PCI_CLASS_STORAGE_EXPRESS;
@@ -201,11 +192,14 @@ static void ct3_class_init(ObjectClass *oc, void *data)
dc->desc = "CXL PMEM Device (Type 3)";
dc->reset = ct3d_reset;
device_class_set_props(dc, ct3_props);
+
+ cvc->get_lsa_size = get_lsa_size;
}
static const TypeInfo ct3d_info = {
.name = TYPE_CXL_TYPE3_DEV,
.parent = TYPE_PCI_DEVICE,
+ .class_size = sizeof(struct CXLType3Class),
.class_init = ct3_class_init,
.instance_size = sizeof(CXLType3Dev),
.instance_finalize = ct3_finalize,
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index 8102d2a813..ebb391153a 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -230,4 +230,33 @@ REG64(CXL_MEM_DEV_STS, 0)
FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1)
FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3)
+typedef struct cxl_type3_dev {
+ /* Private */
+ PCIDevice parent_obj;
+
+ /* Properties */
+ uint64_t size;
+ HostMemoryBackend *hostmem;
+ HostMemoryBackend *lsa;
+
+ /* State */
+ CXLComponentState cxl_cstate;
+ CXLDeviceState cxl_dstate;
+} CXLType3Dev;
+
+#ifndef TYPE_CXL_TYPE3_DEV
+#define TYPE_CXL_TYPE3_DEV "cxl-type3"
+#endif
+
+#define CT3(obj) OBJECT_CHECK(CXLType3Dev, (obj), TYPE_CXL_TYPE3_DEV)
+OBJECT_DECLARE_TYPE(CXLType3Device, CXLType3Class, CXL_TYPE3_DEV)
+
+struct CXLType3Class {
+ /* Private */
+ PCIDeviceClass parent_class;
+
+ /* public */
+ uint64_t (*get_lsa_size)(CXLType3Dev *ct3d);
+};
+
#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 27/43] hw/cxl/device: Implement get/set Label Storage Area (LSA)
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (25 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 26/43] hw/cxl/device: Plumb real Label Storage Area (LSA) sizing Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 28/43] hw/cxl/component: Add utils for interleave parameter encoding/decoding Jonathan Cameron
` (17 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
Implement get and set handlers for the Label Storage Area
used to hold data describing persistent memory configuration
so that it can be ensured it is seen in the same configuration
after reboot.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
v5:
Fix wrong bit for IMMEDIATE_DATA_CHANGE
hw/cxl/cxl-mailbox-utils.c | 57 +++++++++++++++++++++++++++++++++++++
hw/mem/cxl_type3.c | 56 +++++++++++++++++++++++++++++++++++-
include/hw/cxl/cxl_device.h | 5 ++++
3 files changed, 117 insertions(+), 1 deletion(-)
diff --git a/hw/cxl/cxl-mailbox-utils.c b/hw/cxl/cxl-mailbox-utils.c
index ccf9c3d794..f4a309ddbf 100644
--- a/hw/cxl/cxl-mailbox-utils.c
+++ b/hw/cxl/cxl-mailbox-utils.c
@@ -56,6 +56,8 @@ enum {
#define MEMORY_DEVICE 0x0
CCLS = 0x41,
#define GET_PARTITION_INFO 0x0
+ #define GET_LSA 0x2
+ #define SET_LSA 0x3
};
/* 8.2.8.4.5.1 Command Return Codes */
@@ -327,7 +329,59 @@ static ret_code cmd_ccls_get_partition_info(struct cxl_cmd *cmd,
return CXL_MBOX_SUCCESS;
}
+static ret_code cmd_ccls_get_lsa(struct cxl_cmd *cmd,
+ CXLDeviceState *cxl_dstate,
+ uint16_t *len)
+{
+ struct {
+ uint32_t offset;
+ uint32_t length;
+ } __attribute__((packed, __aligned__(8))) *get_lsa;
+ CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate);
+ CXLType3Class *cvc = CXL_TYPE3_DEV_GET_CLASS(ct3d);
+ uint32_t offset, length;
+
+ get_lsa = (void *)cmd->payload;
+ offset = get_lsa->offset;
+ length = get_lsa->length;
+
+ *len = 0;
+ if (offset + length > cvc->get_lsa_size(ct3d)) {
+ return CXL_MBOX_INVALID_INPUT;
+ }
+
+ *len = cvc->get_lsa(ct3d, get_lsa, length, offset);
+ return CXL_MBOX_SUCCESS;
+}
+
+static ret_code cmd_ccls_set_lsa(struct cxl_cmd *cmd,
+ CXLDeviceState *cxl_dstate,
+ uint16_t *len)
+{
+ struct {
+ uint32_t offset;
+ uint32_t rsvd;
+ } __attribute__((packed, __aligned__(8))) *set_lsa = (void *)cmd->payload;
+ CXLType3Dev *ct3d = container_of(cxl_dstate, CXLType3Dev, cxl_dstate);
+ CXLType3Class *cvc = CXL_TYPE3_DEV_GET_CLASS(ct3d);
+ uint16_t plen = *len;
+
+ *len = 0;
+ if (!plen) {
+ return CXL_MBOX_SUCCESS;
+ }
+
+ if (set_lsa->offset + plen > cvc->get_lsa_size(ct3d) + sizeof(*set_lsa)) {
+ return CXL_MBOX_INVALID_INPUT;
+ }
+
+ cvc->set_lsa(ct3d, (void *)set_lsa + sizeof(*set_lsa),
+ plen - sizeof(*set_lsa), set_lsa->offset);
+ return CXL_MBOX_SUCCESS;
+}
+
#define IMMEDIATE_CONFIG_CHANGE (1 << 1)
+#define IMMEDIATE_DATA_CHANGE (1 << 2)
#define IMMEDIATE_POLICY_CHANGE (1 << 3)
#define IMMEDIATE_LOG_CHANGE (1 << 4)
@@ -350,6 +404,9 @@ static struct cxl_cmd cxl_cmd_set[256][256] = {
cmd_identify_memory_device, 0, 0 },
[CCLS][GET_PARTITION_INFO] = { "CCLS_GET_PARTITION_INFO",
cmd_ccls_get_partition_info, 0, 0 },
+ [CCLS][GET_LSA] = { "CCLS_GET_LSA", cmd_ccls_get_lsa, 0, 0 },
+ [CCLS][SET_LSA] = { "CCLS_SET_LSA", cmd_ccls_set_lsa,
+ ~0, IMMEDIATE_CONFIG_CHANGE | IMMEDIATE_DATA_CHANGE },
};
void cxl_process_mailbox(CXLDeviceState *cxl_dstate)
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index b16262d3cc..b1ba4bf0de 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -8,6 +8,7 @@
#include "qapi/error.h"
#include "qemu/log.h"
#include "qemu/module.h"
+#include "qemu/pmem.h"
#include "qemu/range.h"
#include "qemu/rcu.h"
#include "sysemu/hostmem.h"
@@ -114,6 +115,11 @@ static void cxl_setup_memory(CXLType3Dev *ct3d, Error **errp)
memory_region_set_enabled(mr, true);
host_memory_backend_set_mapped(ct3d->hostmem, true);
ct3d->cxl_dstate.pmem_size = ct3d->hostmem->size;
+
+ if (!ct3d->lsa) {
+ error_setg(errp, "lsa property must be set");
+ return;
+ }
}
@@ -168,12 +174,58 @@ static Property ct3_props[] = {
DEFINE_PROP_SIZE("size", CXLType3Dev, size, -1),
DEFINE_PROP_LINK("memdev", CXLType3Dev, hostmem, TYPE_MEMORY_BACKEND,
HostMemoryBackend *),
+ DEFINE_PROP_LINK("lsa", CXLType3Dev, lsa, TYPE_MEMORY_BACKEND,
+ HostMemoryBackend *),
DEFINE_PROP_END_OF_LIST(),
};
static uint64_t get_lsa_size(CXLType3Dev *ct3d)
{
- return 0;
+ MemoryRegion *mr;
+
+ mr = host_memory_backend_get_memory(ct3d->lsa);
+ return memory_region_size(mr);
+}
+
+static void validate_lsa_access(MemoryRegion *mr, uint64_t size,
+ uint64_t offset)
+{
+ assert(offset + size <= memory_region_size(mr));
+ assert(offset + size > offset);
+}
+
+static uint64_t get_lsa(CXLType3Dev *ct3d, void *buf, uint64_t size,
+ uint64_t offset)
+{
+ MemoryRegion *mr;
+ void *lsa;
+
+ mr = host_memory_backend_get_memory(ct3d->lsa);
+ validate_lsa_access(mr, size, offset);
+
+ lsa = memory_region_get_ram_ptr(mr) + offset;
+ memcpy(buf, lsa, size);
+
+ return size;
+}
+
+static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size,
+ uint64_t offset)
+{
+ MemoryRegion *mr;
+ void *lsa;
+
+ mr = host_memory_backend_get_memory(ct3d->lsa);
+ validate_lsa_access(mr, size, offset);
+
+ lsa = memory_region_get_ram_ptr(mr) + offset;
+ memcpy(lsa, buf, size);
+ memory_region_set_dirty(mr, offset, size);
+
+ /*
+ * Just like the PMEM, if the guest is not allowed to exit gracefully, label
+ * updates will get lost.
+ */
}
static void ct3_class_init(ObjectClass *oc, void *data)
@@ -194,6 +246,8 @@ static void ct3_class_init(ObjectClass *oc, void *data)
device_class_set_props(dc, ct3_props);
cvc->get_lsa_size = get_lsa_size;
+ cvc->get_lsa = get_lsa;
+ cvc->set_lsa = set_lsa;
}
static const TypeInfo ct3d_info = {
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index ebb391153a..43908f161b 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -257,6 +257,11 @@ struct CXLType3Class {
/* public */
uint64_t (*get_lsa_size)(CXLType3Dev *ct3d);
+
+ uint64_t (*get_lsa)(CXLType3Dev *ct3d, void *buf, uint64_t size,
+ uint64_t offset);
+ void (*set_lsa)(CXLType3Dev *ct3d, const void *buf, uint64_t size,
+ uint64_t offset);
};
#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 28/43] hw/cxl/component: Add utils for interleave parameter encoding/decoding
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (26 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 27/43] hw/cxl/device: Implement get/set Label Storage Area (LSA) Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 29/43] hw/cxl/host: Add support for CXL Fixed Memory Windows Jonathan Cameron
` (16 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
Both registers and the CFMWS entries in CDAT use simple encodings
for the number of interleave ways and the interleave granularity.
Introduce simple conversion functions to/from the unencoded
number / size. So far the iw decode has not been needed so is
it not implemented.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/cxl/cxl-component-utils.c | 34 ++++++++++++++++++++++++++++++++++
include/hw/cxl/cxl_component.h | 8 ++++++++
2 files changed, 42 insertions(+)
diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index 07297b3bbe..795dbc7561 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -9,6 +9,7 @@
#include "qemu/osdep.h"
#include "qemu/log.h"
+#include "qapi/error.h"
#include "hw/pci/pci.h"
#include "hw/cxl/cxl.h"
@@ -217,3 +218,36 @@ void cxl_component_create_dvsec(CXLComponentState *cxl, uint16_t length,
range_init_nofail(&cxl->dvsecs[type], cxl->dvsec_offset, length);
cxl->dvsec_offset += length;
}
+
+uint8_t cxl_interleave_ways_enc(int iw, Error **errp)
+{
+ switch (iw) {
+ case 1: return 0x0;
+ case 2: return 0x1;
+ case 4: return 0x2;
+ case 8: return 0x3;
+ case 16: return 0x4;
+ case 3: return 0x8;
+ case 6: return 0x9;
+ case 12: return 0xa;
+ default:
+ error_setg(errp, "Interleave ways: %d not supported", iw);
+ return 0;
+ }
+}
+
+uint8_t cxl_interleave_granularity_enc(uint64_t gran, Error **errp)
+{
+ switch (gran) {
+ case 256: return 0;
+ case 512: return 1;
+ case 1024: return 2;
+ case 2048: return 3;
+ case 4096: return 4;
+ case 8192: return 5;
+ case 16384: return 6;
+ default:
+ error_setg(errp, "Interleave granularity: %" PRIu64 " invalid", gran);
+ return 0;
+ }
+}
diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
index 33aeab9b99..42cd140f75 100644
--- a/include/hw/cxl/cxl_component.h
+++ b/include/hw/cxl/cxl_component.h
@@ -193,4 +193,12 @@ void cxl_component_register_init_common(uint32_t *reg_state,
void cxl_component_create_dvsec(CXLComponentState *cxl_cstate, uint16_t length,
uint16_t type, uint8_t rev, uint8_t *body);
+uint8_t cxl_interleave_ways_enc(int iw, Error **errp);
+uint8_t cxl_interleave_granularity_enc(uint64_t gran, Error **errp);
+
+static inline hwaddr cxl_decode_ig(int ig)
+{
+ return 1 << (ig + 8);
+}
+
#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 29/43] hw/cxl/host: Add support for CXL Fixed Memory Windows.
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (27 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 28/43] hw/cxl/component: Add utils for interleave parameter encoding/decoding Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 30/43] acpi/cxl: Introduce CFMWS structures in CEDT Jonathan Cameron
` (15 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
The concept of these is introduced in [1] in terms of the
description the CEDT ACPI table. The principal is more general.
Unlike once traffic hits the CXL root bridges, the host system
memory address routing is implementation defined and effectively
static once observable by standard / generic system software.
Each CXL Fixed Memory Windows (CFMW) is a region of PA space
which has fixed system dependent routing configured so that
accesses can be routed to the CXL devices below a set of target
root bridges. The accesses may be interleaved across multiple
root bridges.
For QEMU we could have fully specified these regions in terms
of a base PA + size, but as the absolute address does not matter
it is simpler to let individual platforms place the memory regions.
ExampleS:
-cxl-fixed-memory-window targets=cxl.0,size=128G
-cxl-fixed-memory-window targets=cxl.1,size=128G
-cxl-fixed-memory-window targets=cxl0,targets=cxl.1,size=256G,interleave-granularity=2k
Specifies
* 2x 128G regions not interleaved across root bridges, one for each of
the root bridges with ids cxl.0 and cxl.1
* 256G region interleaved across root bridges with ids cxl.0 and cxl.1
with a 2k interleave granularity.
When system software enumerates the devices below a given root bridge
it can then decide which CFMW to use. If non interleave is desired
(or possible) it can use the appropriate CFMW for the root bridge in
question. If there are suitable devices to interleave across the
two root bridges then it may use the 3rd CFMS.
A number of other designs were considered but the following constraints
made it hard to adapt existing QEMU approaches to this particular problem.
1) The size must be known before a specific architecture / board brings
up it's PA memory map. We need to set up an appropriate region.
2) Using links to the host bridges provides a clean command line interface
but these links cannot be established until command line devices have
been added.
Hence the two step process used here of first establishing the size,
interleave-ways and granularity + caching the ids of the host bridges
and then, once available finding the actual host bridges so they can
be used later to support interleave decoding.
[1] CXL 2.0 ECN: CEDT CFMWS & QTG DSM (computeexpresslink.org / specifications)
Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
---
v5:
Build fix as suggested by Alex to move this from specific_ss to softmmu_ss.
hw/cxl/cxl-host-stubs.c | 22 +++++++
hw/cxl/cxl-host.c | 138 ++++++++++++++++++++++++++++++++++++++++
hw/cxl/meson.build | 6 ++
include/hw/cxl/cxl.h | 20 ++++++
qapi/machine.json | 15 +++++
qemu-options.hx | 37 +++++++++++
softmmu/vl.c | 11 ++++
7 files changed, 249 insertions(+)
create mode 100644 hw/cxl/cxl-host-stubs.c
create mode 100644 hw/cxl/cxl-host.c
diff --git a/hw/cxl/cxl-host-stubs.c b/hw/cxl/cxl-host-stubs.c
new file mode 100644
index 0000000000..f942dda41b
--- /dev/null
+++ b/hw/cxl/cxl-host-stubs.c
@@ -0,0 +1,22 @@
+/*
+ * CXL host parameter parsing routine stubs
+ *
+ * Copyright (c) 2022 Huawei
+ */
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qemu/option.h"
+#include "hw/cxl/cxl.h"
+
+QemuOptsList qemu_cxl_fixed_window_opts = {
+ .name = "cxl-fixed-memory-window",
+ .implied_opt_name = "type",
+ .head = QTAILQ_HEAD_INITIALIZER(qemu_cxl_fixed_window_opts.head),
+ .desc = { { 0 } }
+};
+
+void parse_cxl_fixed_memory_window_opts(MachineState *ms) {};
+
+void cxl_fixed_memory_window_link_targets(Error **errp) {};
+
+const MemoryRegionOps cfmws_ops;
diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c
new file mode 100644
index 0000000000..9f303e6d8e
--- /dev/null
+++ b/hw/cxl/cxl-host.c
@@ -0,0 +1,138 @@
+/*
+ * CXL host parameter parsing routines
+ *
+ * Copyright (c) 2022 Huawei
+ * Modeled loosely on the NUMA options handling in hw/core/numa.c
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/units.h"
+#include "qemu/bitmap.h"
+#include "qemu/error-report.h"
+#include "qapi/error.h"
+#include "sysemu/qtest.h"
+#include "hw/boards.h"
+
+#include "qapi/opts-visitor.h"
+#include "qapi/qapi-visit-machine.h"
+#include "qemu/option.h"
+#include "hw/cxl/cxl.h"
+#include "hw/pci/pci_bus.h"
+#include "hw/pci/pci_bridge.h"
+#include "hw/pci/pci_host.h"
+#include "hw/pci/pcie_port.h"
+
+QemuOptsList qemu_cxl_fixed_window_opts = {
+ .name = "cxl-fixed-memory-window",
+ .implied_opt_name = "type",
+ .head = QTAILQ_HEAD_INITIALIZER(qemu_cxl_fixed_window_opts.head),
+ .desc = { { 0 } }
+};
+
+static void set_cxl_fixed_memory_window_options(MachineState *ms,
+ CXLFixedMemoryWindowOptions *object,
+ Error **errp)
+{
+ CXLFixedWindow *fw = g_malloc0(sizeof(*fw));
+ strList *target;
+ int i;
+
+ for (target = object->targets; target; target = target->next) {
+ fw->num_targets++;
+ }
+
+ fw->enc_int_ways = cxl_interleave_ways_enc(fw->num_targets, errp);
+ if (*errp) {
+ return;
+ }
+
+ fw->targets = g_malloc0_n(fw->num_targets, sizeof(*fw->targets));
+ for (i = 0, target = object->targets; target; i++, target = target->next) {
+ /* This link cannot be resolved yet, so stash the name for now */
+ fw->targets[i] = g_strdup(target->value);
+ }
+
+ if (object->size % (256 * MiB)) {
+ error_setg(errp,
+ "Size of a CXL fixed memory window must my a multiple of 256MiB");
+ return;
+ }
+ fw->size = object->size;
+
+ if (object->has_interleave_granularity) {
+ fw->enc_int_gran =
+ cxl_interleave_granularity_enc(object->interleave_granularity,
+ errp);
+ if (*errp) {
+ return;
+ }
+ } else {
+ /* Default to 256 byte interleave */
+ fw->enc_int_gran = 0;
+ }
+
+ ms->cxl_devices_state->fixed_windows =
+ g_list_append(ms->cxl_devices_state->fixed_windows, fw);
+
+ return;
+}
+
+static int parse_cxl_fixed_memory_window(void *opaque, QemuOpts *opts,
+ Error **errp)
+{
+ CXLFixedMemoryWindowOptions *object = NULL;
+ MachineState *ms = MACHINE(opaque);
+ Error *err = NULL;
+ Visitor *v = opts_visitor_new(opts);
+
+ visit_type_CXLFixedMemoryWindowOptions(v, NULL, &object, errp);
+ visit_free(v);
+ if (!object) {
+ return -1;
+ }
+
+ set_cxl_fixed_memory_window_options(ms, object, &err);
+
+ qapi_free_CXLFixedMemoryWindowOptions(object);
+ if (err) {
+ error_propagate(errp, err);
+ return -1;
+ }
+
+ return 0;
+}
+
+void parse_cxl_fixed_memory_window_opts(MachineState *ms)
+{
+ qemu_opts_foreach(qemu_find_opts("cxl-fixed-memory-window"),
+ parse_cxl_fixed_memory_window, ms, &error_fatal);
+}
+
+void cxl_fixed_memory_window_link_targets(Error **errp)
+{
+ MachineState *ms = MACHINE(qdev_get_machine());
+
+ if (ms->cxl_devices_state && ms->cxl_devices_state->fixed_windows) {
+ GList *it;
+
+ for (it = ms->cxl_devices_state->fixed_windows; it; it = it->next) {
+ CXLFixedWindow *fw = it->data;
+ int i;
+
+ for (i = 0; i < fw->num_targets; i++) {
+ Object *o;
+ bool ambig;
+
+ o = object_resolve_path_type(fw->targets[i],
+ TYPE_PXB_CXL_DEVICE,
+ &ambig);
+ if (!o) {
+ error_setg(errp, "Could not resolve CXLFM target %s",
+ fw->targets[i]);
+ return;
+ }
+ fw->target_hbs[i] = PXB_CXL_DEV(o);
+ }
+ }
+ }
+}
diff --git a/hw/cxl/meson.build b/hw/cxl/meson.build
index e68eea2358..f117b99949 100644
--- a/hw/cxl/meson.build
+++ b/hw/cxl/meson.build
@@ -3,4 +3,10 @@ softmmu_ss.add(when: 'CONFIG_CXL',
'cxl-component-utils.c',
'cxl-device-utils.c',
'cxl-mailbox-utils.c',
+ 'cxl-host.c',
+ ),
+ if_false: files(
+ 'cxl-host-stubs.c',
))
+
+softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('cxl-host-stubs.c'))
diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
index 6889362230..1b72c0b7b7 100644
--- a/include/hw/cxl/cxl.h
+++ b/include/hw/cxl/cxl.h
@@ -10,6 +10,9 @@
#ifndef CXL_H
#define CXL_H
+#include "qapi/qapi-types-machine.h"
+#include "qemu/option.h"
+#include "hw/pci/pci_bridge.h"
#include "cxl_pci.h"
#include "cxl_component.h"
#include "cxl_device.h"
@@ -20,10 +23,27 @@
#define TYPE_CXL_TYPE3_DEV "cxl-type3"
#define CXL_WINDOW_MAX 10
+typedef struct CXLFixedWindow {
+ uint64_t size;
+ char **targets;
+ struct PXBDev *target_hbs[8];
+ uint8_t num_targets;
+ uint8_t enc_int_ways;
+ uint8_t enc_int_gran;
+ /* Todo: XOR based interleaving */
+ MemoryRegion mr;
+ hwaddr base;
+} CXLFixedWindow;
+
typedef struct CXLState {
bool is_enabled;
MemoryRegion host_mr;
unsigned int next_mr_idx;
+ GList *fixed_windows;
} CXLState;
+extern QemuOptsList qemu_cxl_fixed_window_opts;
+void parse_cxl_fixed_memory_window_opts(MachineState *ms);
+void cxl_fixed_memory_window_link_targets(Error **errp);
+
#endif
diff --git a/qapi/machine.json b/qapi/machine.json
index 42fc68403d..0998a9128d 100644
--- a/qapi/machine.json
+++ b/qapi/machine.json
@@ -504,6 +504,21 @@
'dst': 'uint16',
'val': 'uint8' }}
+##
+# @CXLFixedMemoryWindowOptions:
+#
+# Create a CXL Fixed Memory Window (for OptsVisitor)
+#
+# @targets: Target root bridge IDs
+#
+# Since X.X //fixme
+##
+{ 'struct': 'CXLFixedMemoryWindowOptions',
+ 'data': {
+ 'size': 'size',
+ '*interleave-granularity': 'size',
+ 'targets': ['str'] }}
+
##
# @X86CPURegister32:
#
diff --git a/qemu-options.hx b/qemu-options.hx
index ba3ae6a42a..b4d2cc6f48 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -467,6 +467,43 @@ SRST
-numa hmat-cache,node-id=1,size=10K,level=1,associativity=direct,policy=write-back,line=8
ERST
+DEF("cxl-fixed-memory-window", HAS_ARG, QEMU_OPTION_cxl_fixed_memory_window,
+ "-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]\n",
+ QEMU_ARCH_ALL)
+SRST
+``-cxl-fixed-memory-window targets=firsttarget,targets=secondtarget,size=size[,interleave-granularity=granularity]``
+ Define a CXL Fixed Memory Window (CFMW).
+
+ Described in the CXL 2.0 ECN: CEDT CFMWS & QTG _DSM.
+
+ They are regions of Host Physical Addresses (HPA) on a system which
+ may be interleaved across one or more CXL host bridges. The system
+ software will assign particular devices into these windows and
+ configure the downstream Host-managed Device Memory (HDM) decoders
+ in root ports, switch ports and devices appropriately to meet the
+ interleave requirements before enabling the memory devices.
+
+ ``targets=firsttarget`` provides the mapping to CXL host bridges
+ which may be identified by the id provied in the -device entry.
+ Multiple entries are needed to specify all the targets when
+ the fixed memory window represents interleaved memory.
+
+ ``size=size`` sets the size of the CFMW. This must be a multiple of
+ 256MiB. The region will be aligned to 256MiB but the location is
+ platform and configuration dependent.
+
+ ``interleave-granularity=granularity`` sets the granularity of
+ interleave. Default 256KiB. Only 256KiB, 512KiB, 1024KiB, 2048KiB
+ 4096KiB, 8192KiB and 16384KiB granularities supported.
+
+ Example:
+
+ ::
+
+ -cxl-fixed-memory-window -targets=cxl.0,-targets=cxl.1,size=128G,interleave-granularity=512k
+
+ERST
+
DEF("add-fd", HAS_ARG, QEMU_OPTION_add_fd,
"-add-fd fd=fd,set=set[,opaque=opaque]\n"
" Add 'fd' to fd 'set'\n", QEMU_ARCH_ALL)
diff --git a/softmmu/vl.c b/softmmu/vl.c
index 5e1b35ba48..f83f158fff 100644
--- a/softmmu/vl.c
+++ b/softmmu/vl.c
@@ -91,6 +91,7 @@
#include "qemu/config-file.h"
#include "qemu/qemu-options.h"
#include "qemu/main-loop.h"
+#include "hw/cxl/cxl.h"
#ifdef CONFIG_VIRTFS
#include "fsdev/qemu-fsdev.h"
#endif
@@ -2744,6 +2745,7 @@ void qmp_x_exit_preconfig(Error **errp)
qemu_init_board();
qemu_create_cli_devices();
+ cxl_fixed_memory_window_link_targets(errp);
qemu_machine_creation_done();
if (loadvm) {
@@ -2805,6 +2807,7 @@ void qemu_init(int argc, char **argv, char **envp)
qemu_add_opts(&qemu_msg_opts);
qemu_add_opts(&qemu_name_opts);
qemu_add_opts(&qemu_numa_opts);
+ qemu_add_opts(&qemu_cxl_fixed_window_opts);
qemu_add_opts(&qemu_icount_opts);
qemu_add_opts(&qemu_semihosting_config_opts);
qemu_add_opts(&qemu_fw_cfg_opts);
@@ -2927,6 +2930,13 @@ void qemu_init(int argc, char **argv, char **envp)
exit(1);
}
break;
+ case QEMU_OPTION_cxl_fixed_memory_window:
+ opts = qemu_opts_parse_noisily(qemu_find_opts("cxl-fixed-memory-window"),
+ optarg, true);
+ if (!opts) {
+ exit(1);
+ }
+ break;
case QEMU_OPTION_display:
parse_display(optarg);
break;
@@ -3764,6 +3774,7 @@ void qemu_init(int argc, char **argv, char **envp)
qemu_resolve_machine_memdev();
parse_numa_opts(current_machine);
+ parse_cxl_fixed_memory_window_opts(current_machine);
if (vmstate_dump_file) {
/* dump and exit */
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 30/43] acpi/cxl: Introduce CFMWS structures in CEDT
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (28 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 29/43] hw/cxl/host: Add support for CXL Fixed Memory Windows Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 31/43] hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl Jonathan Cameron
` (14 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
The CEDT CXL Fixed Window Memory Window Structures (CFMWs)
define regions of the host phyiscal address map which
(via an impdef means) are configured such that they have
a particular interleave setup across one or more CXL Host Bridges.
Reported-by: Alison Schofield <alison.schofield@intel.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/acpi/cxl.c | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 59 insertions(+)
diff --git a/hw/acpi/cxl.c b/hw/acpi/cxl.c
index 442f836a3e..50efc7f690 100644
--- a/hw/acpi/cxl.c
+++ b/hw/acpi/cxl.c
@@ -60,6 +60,64 @@ static void cedt_build_chbs(GArray *table_data, PXBDev *cxl)
build_append_int_noprefix(table_data, memory_region_size(mr), 8);
}
+/*
+ * CFMWS entries in CXL 2.0 ECN: CEDT CFMWS & QTG _DSM.
+ * Interleave ways encoding in CXL 2.0 ECN: 3, 6, 12 and 16-way memory
+ * interleaving.
+ */
+static void cedt_build_cfmws(GArray *table_data, MachineState *ms)
+{
+ CXLState *cxls = ms->cxl_devices_state;
+ GList *it;
+
+ for (it = cxls->fixed_windows; it; it = it->next) {
+ CXLFixedWindow *fw = it->data;
+ int i;
+
+ /* Type */
+ build_append_int_noprefix(table_data, 1, 1);
+
+ /* Reserved */
+ build_append_int_noprefix(table_data, 0, 1);
+
+ /* Record Length */
+ build_append_int_noprefix(table_data, 36 + 4 * fw->num_targets, 2);
+
+ /* Reserved */
+ build_append_int_noprefix(table_data, 0, 4);
+
+ /* Base HPA */
+ build_append_int_noprefix(table_data, fw->mr.addr, 8);
+
+ /* Window Size */
+ build_append_int_noprefix(table_data, fw->size, 8);
+
+ /* Host Bridge Interleave Ways */
+ build_append_int_noprefix(table_data, fw->enc_int_ways, 1);
+
+ /* Host Bridge Interleave Arithmetic */
+ build_append_int_noprefix(table_data, 0, 1);
+
+ /* Reserved */
+ build_append_int_noprefix(table_data, 0, 2);
+
+ /* Host Bridge Interleave Granularity */
+ build_append_int_noprefix(table_data, fw->enc_int_gran, 4);
+
+ /* Window Restrictions */
+ build_append_int_noprefix(table_data, 0x0f, 2); /* No restrictions */
+
+ /* QTG ID */
+ build_append_int_noprefix(table_data, 0, 2);
+
+ /* Host Bridge List (list of UIDs - currently bus_nr) */
+ for (i = 0; i < fw->num_targets; i++) {
+ g_assert(fw->target_hbs[i]);
+ build_append_int_noprefix(table_data, fw->target_hbs[i]->bus_nr, 4);
+ }
+ }
+}
+
static int cxl_foreach_pxb_hb(Object *obj, void *opaque)
{
Aml *cedt = opaque;
@@ -86,6 +144,7 @@ void cxl_build_cedt(MachineState *ms, GArray *table_offsets, GArray *table_data,
/* reserve space for CEDT header */
object_child_foreach_recursive(object_get_root(), cxl_foreach_pxb_hb, cedt);
+ cedt_build_cfmws(cedt->buf, ms);
/* copy AML table into ACPI tables blob and patch header there */
g_array_append_vals(table_data, cedt->buf->data, cedt->buf->len);
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 31/43] hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (29 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 30/43] acpi/cxl: Introduce CFMWS structures in CEDT Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 32/43] pci/pcie_port: Add pci_find_port_by_pn() Jonathan Cameron
` (13 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
This adds code to instantiate the slightly extended ACPI root port
description in DSDT as per the CXL 2.0 specification.
Basically a cut and paste job from the i386/pc code.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
v5:
No change to this patch, but build issue seen here was fixed at
introduction of build_cxl_osc_method() in patch 22.
hw/arm/Kconfig | 1 +
hw/pci-host/gpex-acpi.c | 22 +++++++++++++++++++---
2 files changed, 20 insertions(+), 3 deletions(-)
diff --git a/hw/arm/Kconfig b/hw/arm/Kconfig
index 2e0049196d..3df419fa6d 100644
--- a/hw/arm/Kconfig
+++ b/hw/arm/Kconfig
@@ -29,6 +29,7 @@ config ARM_VIRT
select ACPI_APEI
select ACPI_VIOT
select VIRTIO_MEM_SUPPORTED
+ select ACPI_CXL
config CHEETAH
bool
diff --git a/hw/pci-host/gpex-acpi.c b/hw/pci-host/gpex-acpi.c
index e7e162a00a..fb60aa517f 100644
--- a/hw/pci-host/gpex-acpi.c
+++ b/hw/pci-host/gpex-acpi.c
@@ -5,6 +5,7 @@
#include "hw/pci/pci_bus.h"
#include "hw/pci/pci_bridge.h"
#include "hw/pci/pcie_host.h"
+#include "hw/acpi/cxl.h"
static void acpi_dsdt_add_pci_route_table(Aml *dev, uint32_t irq)
{
@@ -139,6 +140,7 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)
QLIST_FOREACH(bus, &bus->child, sibling) {
uint8_t bus_num = pci_bus_num(bus);
uint8_t numa_node = pci_bus_numa_node(bus);
+ bool is_cxl;
if (!pci_bus_is_root(bus)) {
continue;
@@ -153,9 +155,19 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)
nr_pcie_buses = bus_num;
}
+ is_cxl = pci_bus_is_cxl(bus);
+
dev = aml_device("PC%.02X", bus_num);
- aml_append(dev, aml_name_decl("_HID", aml_string("PNP0A08")));
- aml_append(dev, aml_name_decl("_CID", aml_string("PNP0A03")));
+ if (is_cxl) {
+ struct Aml *pkg = aml_package(2);
+ aml_append(dev, aml_name_decl("_HID", aml_string("ACPI0016")));
+ aml_append(pkg, aml_eisaid("PNP0A08"));
+ aml_append(pkg, aml_eisaid("PNP0A03"));
+ aml_append(dev, aml_name_decl("_CID", pkg));
+ } else {
+ aml_append(dev, aml_name_decl("_HID", aml_string("PNP0A08")));
+ aml_append(dev, aml_name_decl("_CID", aml_string("PNP0A03")));
+ }
aml_append(dev, aml_name_decl("_BBN", aml_int(bus_num)));
aml_append(dev, aml_name_decl("_UID", aml_int(bus_num)));
aml_append(dev, aml_name_decl("_STR", aml_unicode("pxb Device")));
@@ -175,7 +187,11 @@ void acpi_dsdt_add_gpex(Aml *scope, struct GPEXConfig *cfg)
cfg->pio.base, 0, 0, 0);
aml_append(dev, aml_name_decl("_CRS", crs));
- acpi_dsdt_add_pci_osc(dev);
+ if (is_cxl) {
+ build_cxl_osc_method(dev);
+ } else {
+ acpi_dsdt_add_pci_osc(dev);
+ }
aml_append(scope, dev);
}
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 32/43] pci/pcie_port: Add pci_find_port_by_pn()
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (30 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 31/43] hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 33/43] CXL/cxl_component: Add cxl_get_hb_cstate() Jonathan Cameron
` (12 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
Simple function to search a PCIBus to find a port by
it's port number.
CXL interleave decoding uses the port number as a target
so it is necessary to locate the port when doing interleave
decoding.
Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
---
hw/pci/pcie_port.c | 25 +++++++++++++++++++++++++
include/hw/pci/pcie_port.h | 2 ++
2 files changed, 27 insertions(+)
diff --git a/hw/pci/pcie_port.c b/hw/pci/pcie_port.c
index e95c1e5519..687e4e763a 100644
--- a/hw/pci/pcie_port.c
+++ b/hw/pci/pcie_port.c
@@ -136,6 +136,31 @@ static void pcie_port_class_init(ObjectClass *oc, void *data)
device_class_set_props(dc, pcie_port_props);
}
+PCIDevice *pcie_find_port_by_pn(PCIBus *bus, uint8_t pn)
+{
+ int devfn;
+
+ for (devfn = 0; devfn < ARRAY_SIZE(bus->devices); devfn++) {
+ PCIDevice *d = bus->devices[devfn];
+ PCIEPort *port;
+
+ if (!d || !pci_is_express(d) || !d->exp.exp_cap) {
+ continue;
+ }
+
+ if (!object_dynamic_cast(OBJECT(d), TYPE_PCIE_PORT)) {
+ continue;
+ }
+
+ port = PCIE_PORT(d);
+ if (port->port == pn) {
+ return d;
+ }
+ }
+
+ return NULL;
+}
+
static const TypeInfo pcie_port_type_info = {
.name = TYPE_PCIE_PORT,
.parent = TYPE_PCI_BRIDGE,
diff --git a/include/hw/pci/pcie_port.h b/include/hw/pci/pcie_port.h
index e25b289ce8..7b8193061a 100644
--- a/include/hw/pci/pcie_port.h
+++ b/include/hw/pci/pcie_port.h
@@ -39,6 +39,8 @@ struct PCIEPort {
void pcie_port_init_reg(PCIDevice *d);
+PCIDevice *pcie_find_port_by_pn(PCIBus *bus, uint8_t pn);
+
#define TYPE_PCIE_SLOT "pcie-slot"
OBJECT_DECLARE_SIMPLE_TYPE(PCIESlot, PCIE_SLOT)
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 33/43] CXL/cxl_component: Add cxl_get_hb_cstate()
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (31 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 32/43] pci/pcie_port: Add pci_find_port_by_pn() Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 34/43] mem/cxl_type3: Add read and write functions for associated hostmem Jonathan Cameron
` (11 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
Accessor to get hold of the cxl state for a CXL host bridge
without exposing the internals of the implementation.
Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
---
hw/pci-bridge/pci_expander_bridge.c | 7 +++++++
include/hw/cxl/cxl_component.h | 2 ++
2 files changed, 9 insertions(+)
diff --git a/hw/pci-bridge/pci_expander_bridge.c b/hw/pci-bridge/pci_expander_bridge.c
index 9a2710c067..d53efb09a3 100644
--- a/hw/pci-bridge/pci_expander_bridge.c
+++ b/hw/pci-bridge/pci_expander_bridge.c
@@ -81,6 +81,13 @@ static GList *pxb_dev_list;
#define TYPE_PXB_CXL_HOST "pxb-cxl-host"
#define PXB_CXL_HOST(obj) OBJECT_CHECK(CXLHost, (obj), TYPE_PXB_CXL_HOST)
+CXLComponentState *cxl_get_hb_cstate(PCIHostState *hb)
+{
+ CXLHost *host = PXB_CXL_HOST(hb);
+
+ return &host->cxl_cstate;
+}
+
static int pxb_bus_num(PCIBus *bus)
{
PXBDev *pxb = convert_to_pxb(bus->parent_dev);
diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
index 42cd140f75..29d7268275 100644
--- a/include/hw/cxl/cxl_component.h
+++ b/include/hw/cxl/cxl_component.h
@@ -201,4 +201,6 @@ static inline hwaddr cxl_decode_ig(int ig)
return 1 << (ig + 8);
}
+CXLComponentState *cxl_get_hb_cstate(PCIHostState *hb);
+
#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 34/43] mem/cxl_type3: Add read and write functions for associated hostmem.
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (32 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 33/43] CXL/cxl_component: Add cxl_get_hb_cstate() Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 35/43] cxl/cxl-host: Add memops for CFMWS region Jonathan Cameron
` (10 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
Once a read or write reaches a CXL type 3 device, the HDM decoders
on the device are used to establish the Device Physical Address
which should be accessed. These functions peform the required maths
and then directly access the hostmem->mr to fullfil the actual
operation. Note that failed writes are silent, but failed reads
return poison. Note this is based loosely on:
https://lore.kernel.org/qemu-devel/20200817161853.593247-6-f4bug@amsat.org/
[RFC PATCH 0/9] hw/misc: Add support for interleaved memory accesses
Only lightly tested so far. More complex test cases yet to be written.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/mem/cxl_type3.c | 81 +++++++++++++++++++++++++++++++++++++
include/hw/cxl/cxl_device.h | 5 +++
2 files changed, 86 insertions(+)
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index b1ba4bf0de..064e8c942c 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -161,6 +161,87 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)
&ct3d->cxl_dstate.device_registers);
}
+/* TODO: Support multiple HDM decoders and DPA skip */
+static bool cxl_type3_dpa(CXLType3Dev *ct3d, hwaddr host_addr, uint64_t *dpa)
+{
+ uint32_t *cache_mem = ct3d->cxl_cstate.crb.cache_mem_registers;
+ uint64_t decoder_base, decoder_size, hpa_offset;
+ uint32_t hdm0_ctrl;
+ int ig, iw;
+
+ decoder_base = (((uint64_t)cache_mem[R_CXL_HDM_DECODER0_BASE_HI] << 32) |
+ cache_mem[R_CXL_HDM_DECODER0_BASE_LO]);
+ if ((uint64_t)host_addr < decoder_base) {
+ return false;
+ }
+
+ hpa_offset = (uint64_t)host_addr - decoder_base;
+
+ decoder_size = ((uint64_t)cache_mem[R_CXL_HDM_DECODER0_SIZE_HI] << 32) |
+ cache_mem[R_CXL_HDM_DECODER0_SIZE_LO];
+ if (hpa_offset >= decoder_size) {
+ return false;
+ }
+
+ hdm0_ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL];
+ iw = FIELD_EX32(hdm0_ctrl, CXL_HDM_DECODER0_CTRL, IW);
+ ig = FIELD_EX32(hdm0_ctrl, CXL_HDM_DECODER0_CTRL, IG);
+
+ *dpa = (MAKE_64BIT_MASK(0, 8 + ig) & hpa_offset) |
+ ((MAKE_64BIT_MASK(8 + ig + iw, 64 - 8 - ig - iw) & hpa_offset) >> iw);
+
+ return true;
+}
+
+MemTxResult cxl_type3_read(PCIDevice *d, hwaddr host_addr, uint64_t *data,
+ unsigned size, MemTxAttrs attrs)
+{
+ CXLType3Dev *ct3d = CT3(d);
+ uint64_t dpa_offset;
+ MemoryRegion *mr;
+
+ /* TODO support volatile region */
+ mr = host_memory_backend_get_memory(ct3d->hostmem);
+ if (!mr) {
+ return MEMTX_ERROR;
+ }
+
+ if (!cxl_type3_dpa(ct3d, host_addr, &dpa_offset)) {
+ return MEMTX_ERROR;
+ }
+
+ if (dpa_offset > int128_get64(mr->size)) {
+ return MEMTX_ERROR;
+ }
+
+ return memory_region_dispatch_read(mr, dpa_offset, data,
+ size_memop(size), attrs);
+}
+
+MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host_addr, uint64_t data,
+ unsigned size, MemTxAttrs attrs)
+{
+ CXLType3Dev *ct3d = CT3(d);
+ uint64_t dpa_offset;
+ MemoryRegion *mr;
+
+ mr = host_memory_backend_get_memory(ct3d->hostmem);
+ if (!mr) {
+ return MEMTX_OK;
+ }
+
+ if (!cxl_type3_dpa(ct3d, host_addr, &dpa_offset)) {
+ return MEMTX_OK;
+ }
+
+ if (dpa_offset > int128_get64(mr->size)) {
+ return MEMTX_OK;
+ }
+
+ return memory_region_dispatch_write(mr, dpa_offset, data,
+ size_memop(size), attrs);
+}
+
static void ct3d_reset(DeviceState *dev)
{
CXLType3Dev *ct3d = CT3(dev);
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index 43908f161b..83da5d4e8f 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -264,4 +264,9 @@ struct CXLType3Class {
uint64_t offset);
};
+MemTxResult cxl_type3_read(PCIDevice *d, hwaddr host_addr, uint64_t *data,
+ unsigned size, MemTxAttrs attrs);
+MemTxResult cxl_type3_write(PCIDevice *d, hwaddr host_addr, uint64_t data,
+ unsigned size, MemTxAttrs attrs);
+
#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 35/43] cxl/cxl-host: Add memops for CFMWS region.
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (33 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 34/43] mem/cxl_type3: Add read and write functions for associated hostmem Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 36/43] arm/virt: Allow virt/CEDT creation Jonathan Cameron
` (9 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
These memops perform interleave decoding, walking down the
CXL topology from CFMWS described host interleave
decoder via CXL host bridge HDM decoders, through the CXL
root ports and finally call CXL type 3 specific read and write
functions.
Note that, whilst functional the current implementation does
not support:
* switches
* multiple HDM decoders at a given level.
* unaligned accesses across the interleave boundaries
Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
---
v5: No changes, debugging solution to unaligned access across
interleave boundaries continues.
hw/cxl/cxl-host.c | 125 +++++++++++++++++++++++++++++++++++++++++++
include/hw/cxl/cxl.h | 2 +
2 files changed, 127 insertions(+)
diff --git a/hw/cxl/cxl-host.c b/hw/cxl/cxl-host.c
index 9f303e6d8e..d9cad188a8 100644
--- a/hw/cxl/cxl-host.c
+++ b/hw/cxl/cxl-host.c
@@ -136,3 +136,128 @@ void cxl_fixed_memory_window_link_targets(Error **errp)
}
}
}
+
+/* TODO: support, multiple hdm decoders */
+static bool cxl_hdm_find_target(uint32_t *cache_mem, hwaddr addr,
+ uint8_t *target)
+{
+ uint32_t ctrl;
+ uint32_t ig_enc;
+ uint32_t iw_enc;
+ uint32_t target_reg;
+ uint32_t target_idx;
+
+ ctrl = cache_mem[R_CXL_HDM_DECODER0_CTRL];
+ if (!FIELD_EX32(ctrl, CXL_HDM_DECODER0_CTRL, COMMITTED)) {
+ return false;
+ }
+
+ ig_enc = FIELD_EX32(ctrl, CXL_HDM_DECODER0_CTRL, IG);
+ iw_enc = FIELD_EX32(ctrl, CXL_HDM_DECODER0_CTRL, IW);
+ target_idx = (addr / cxl_decode_ig(ig_enc)) % (1 << iw_enc);
+
+ if (target_idx > 4) {
+ target_reg = cache_mem[R_CXL_HDM_DECODER0_TARGET_LIST_LO];
+ target_reg >>= target_idx * 8;
+ } else {
+ target_reg = cache_mem[R_CXL_HDM_DECODER0_TARGET_LIST_LO];
+ target_reg >>= (target_idx - 4) * 8;
+ }
+ *target = target_reg & 0xff;
+
+ return true;
+}
+
+static PCIDevice *cxl_cfmws_find_device(CXLFixedWindow *fw, hwaddr addr)
+{
+ CXLComponentState *hb_cstate;
+ PCIHostState *hb;
+ int rb_index;
+ uint32_t *cache_mem;
+ uint8_t target;
+ bool target_found;
+ PCIDevice *rp, *d;
+
+ /* Address is relative to memory region. Convert to HPA */
+ addr += fw->base;
+
+ rb_index = (addr / cxl_decode_ig(fw->enc_int_gran)) % fw->num_targets;
+ hb = PCI_HOST_BRIDGE(fw->target_hbs[rb_index]->cxl.cxl_host_bridge);
+ if (!hb || !hb->bus || !pci_bus_is_cxl(hb->bus)) {
+ return NULL;
+ }
+
+ hb_cstate = cxl_get_hb_cstate(hb);
+ if (!hb_cstate) {
+ return NULL;
+ }
+
+ cache_mem = hb_cstate->crb.cache_mem_registers;
+
+ target_found = cxl_hdm_find_target(cache_mem, addr, &target);
+ if (!target_found) {
+ return NULL;
+ }
+
+ rp = pcie_find_port_by_pn(hb->bus, target);
+ if (!rp) {
+ return NULL;
+ }
+
+ d = pci_bridge_get_sec_bus(PCI_BRIDGE(rp))->devices[0];
+
+ if (!d || !object_dynamic_cast(OBJECT(d), TYPE_CXL_TYPE3_DEV)) {
+ return NULL;
+ }
+
+ return d;
+}
+
+static MemTxResult cxl_read_cfmws(void *opaque, hwaddr addr, uint64_t *data,
+ unsigned size, MemTxAttrs attrs)
+{
+ CXLFixedWindow *fw = opaque;
+ PCIDevice *d;
+
+ d = cxl_cfmws_find_device(fw, addr);
+ if (d == NULL) {
+ *data = 0;
+ /* Reads to invalid address return poison */
+ return MEMTX_ERROR;
+ }
+
+ return cxl_type3_read(d, addr + fw->base, data, size, attrs);
+}
+
+static MemTxResult cxl_write_cfmws(void *opaque, hwaddr addr,
+ uint64_t data, unsigned size,
+ MemTxAttrs attrs)
+{
+ CXLFixedWindow *fw = opaque;
+ PCIDevice *d;
+
+ d = cxl_cfmws_find_device(fw, addr);
+ if (d == NULL) {
+ /* Writes to invalid address are silent */
+ return MEMTX_OK;
+ }
+
+ return cxl_type3_write(d, addr + fw->base, data, size, attrs);
+}
+
+const MemoryRegionOps cfmws_ops = {
+ .read_with_attrs = cxl_read_cfmws,
+ .write_with_attrs = cxl_write_cfmws,
+ .endianness = DEVICE_LITTLE_ENDIAN,
+ .valid = {
+ .min_access_size = 1,
+ .max_access_size = 8,
+ .unaligned = true,
+ },
+ .impl = {
+ .min_access_size = 1,
+ .max_access_size = 8,
+ .unaligned = true,
+ },
+};
+
diff --git a/include/hw/cxl/cxl.h b/include/hw/cxl/cxl.h
index 1b72c0b7b7..260d602ec9 100644
--- a/include/hw/cxl/cxl.h
+++ b/include/hw/cxl/cxl.h
@@ -46,4 +46,6 @@ extern QemuOptsList qemu_cxl_fixed_window_opts;
void parse_cxl_fixed_memory_window_opts(MachineState *ms);
void cxl_fixed_memory_window_link_targets(Error **errp);
+extern const MemoryRegionOps cfmws_ops;
+
#endif
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 36/43] arm/virt: Allow virt/CEDT creation
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (34 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 35/43] cxl/cxl-host: Add memops for CFMWS region Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 37/43] hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances pxb-cxl Jonathan Cameron
` (8 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
Allow for the creation of the CEDT ACPI table without
qtest fails due to the unknown ACPI tables.
Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
---
tests/data/acpi/virt/CEDT | 0
tests/qtest/bios-tables-test-allowed-diff.h | 1 +
2 files changed, 1 insertion(+)
create mode 100644 tests/data/acpi/virt/CEDT
diff --git a/tests/data/acpi/virt/CEDT b/tests/data/acpi/virt/CEDT
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/tests/qtest/bios-tables-test-allowed-diff.h b/tests/qtest/bios-tables-test-allowed-diff.h
index 9b07f1e1ff..c7726cad80 100644
--- a/tests/qtest/bios-tables-test-allowed-diff.h
+++ b/tests/qtest/bios-tables-test-allowed-diff.h
@@ -1,3 +1,4 @@
/* List of comma-separated changed AML files to ignore */
"tests/data/acpi/pc/CEDT",
"tests/data/acpi/q35/CEDT",
+"tests/data/acpi/virt/CEDT",
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 37/43] hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances pxb-cxl
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (35 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 36/43] arm/virt: Allow virt/CEDT creation Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 38/43] RFC: softmmu/memory: Add ops to memory_region_ram_init_from_file Jonathan Cameron
` (7 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
Code based on i386/pc enablement.
The memory layout places space for 16 host bridge register regions after
the GIC_REDIST2 in the extended memmap.
The CFMWs are placed above the extended memmap.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
---
hw/arm/virt-acpi-build.c | 30 ++++++++++++++++++++++++++++++
hw/arm/virt.c | 40 +++++++++++++++++++++++++++++++++++++++-
include/hw/arm/virt.h | 1 +
3 files changed, 70 insertions(+), 1 deletion(-)
diff --git a/hw/arm/virt-acpi-build.c b/hw/arm/virt-acpi-build.c
index 449fab0080..865709156a 100644
--- a/hw/arm/virt-acpi-build.c
+++ b/hw/arm/virt-acpi-build.c
@@ -39,6 +39,7 @@
#include "hw/acpi/aml-build.h"
#include "hw/acpi/utils.h"
#include "hw/acpi/pci.h"
+#include "hw/acpi/cxl.h"
#include "hw/acpi/memory_hotplug.h"
#include "hw/acpi/generic_event_device.h"
#include "hw/acpi/tpm.h"
@@ -157,10 +158,29 @@ static void acpi_dsdt_add_virtio(Aml *scope,
}
}
+/* Uses local definition of AcpiBuildState so can't easily be common code */
+static void build_acpi0017(Aml *table)
+{
+ Aml *dev, *scope, *method;
+
+ scope = aml_scope("_SB");
+ dev = aml_device("CXLM");
+ aml_append(dev, aml_name_decl("_HID", aml_string("ACPI0017")));
+
+ method = aml_method("_STA", 0, AML_NOTSERIALIZED);
+ aml_append(method, aml_return(aml_int(0x01)));
+ aml_append(dev, method);
+
+ aml_append(scope, dev);
+ aml_append(table, scope);
+}
+
static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap,
uint32_t irq, VirtMachineState *vms)
{
int ecam_id = VIRT_ECAM_ID(vms->highmem_ecam);
+ bool cxl_present = false;
+ PCIBus *bus = vms->bus;
struct GPEXConfig cfg = {
.mmio32 = memmap[VIRT_PCIE_MMIO],
.pio = memmap[VIRT_PCIE_PIO],
@@ -174,6 +194,14 @@ static void acpi_dsdt_add_pci(Aml *scope, const MemMapEntry *memmap,
}
acpi_dsdt_add_gpex(scope, &cfg);
+ QLIST_FOREACH(bus, &vms->bus->child, sibling) {
+ if (pci_bus_is_cxl(bus)) {
+ cxl_present = true;
+ }
+ }
+ if (cxl_present) {
+ build_acpi0017(scope);
+ }
}
static void acpi_dsdt_add_gpio(Aml *scope, const MemMapEntry *gpio_memmap,
@@ -991,6 +1019,8 @@ void virt_acpi_build(VirtMachineState *vms, AcpiBuildTables *tables)
vms->oem_table_id);
}
}
+ cxl_build_cedt(ms, table_offsets, tables_blob, tables->linker,
+ vms->oem_id, vms->oem_table_id);
if (ms->nvdimms_state->is_enabled) {
nvdimm_build_acpi(table_offsets, tables_blob, tables->linker,
diff --git a/hw/arm/virt.c b/hw/arm/virt.c
index 2b6cc7aa9e..b59e470ae4 100644
--- a/hw/arm/virt.c
+++ b/hw/arm/virt.c
@@ -78,6 +78,7 @@
#include "hw/virtio/virtio-mem-pci.h"
#include "hw/virtio/virtio-iommu.h"
#include "hw/char/pl011.h"
+#include "hw/cxl/cxl.h"
#include "qemu/guest-random.h"
#define DEFINE_VIRT_MACHINE_LATEST(major, minor, latest) \
@@ -178,6 +179,7 @@ static const MemMapEntry base_memmap[] = {
static MemMapEntry extended_memmap[] = {
/* Additional 64 MB redist region (can contain up to 512 redistributors) */
[VIRT_HIGH_GIC_REDIST2] = { 0x0, 64 * MiB },
+ [VIRT_CXL_HOST] = { 0x0, 64 * KiB * 16 }, /* 16 UID */
[VIRT_HIGH_PCIE_ECAM] = { 0x0, 256 * MiB },
/* Second PCIe window */
[VIRT_HIGH_PCIE_MMIO] = { 0x0, 512 * GiB },
@@ -1508,6 +1510,17 @@ static void create_pcie(VirtMachineState *vms)
}
}
+static void create_cxl_host_reg_region(VirtMachineState *vms)
+{
+ MemoryRegion *sysmem = get_system_memory();
+ MachineState *ms = MACHINE(vms);
+ MemoryRegion *mr = &ms->cxl_devices_state->host_mr;
+
+ memory_region_init(mr, OBJECT(ms), "cxl_host_reg",
+ vms->memmap[VIRT_CXL_HOST].size);
+ memory_region_add_subregion(sysmem, vms->memmap[VIRT_CXL_HOST].base, mr);
+}
+
static void create_platform_bus(VirtMachineState *vms)
{
DeviceState *dev;
@@ -1670,7 +1683,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx)
static void virt_set_memmap(VirtMachineState *vms, int pa_bits)
{
MachineState *ms = MACHINE(vms);
- hwaddr base, device_memory_base, device_memory_size, memtop;
+ hwaddr base, device_memory_base, device_memory_size, memtop, cxl_fmw_base;
int i;
vms->memmap = extended_memmap;
@@ -1762,6 +1775,20 @@ static void virt_set_memmap(VirtMachineState *vms, int pa_bits)
memory_region_init(&ms->device_memory->mr, OBJECT(vms),
"device-memory", device_memory_size);
}
+
+ if (ms->cxl_devices_state->fixed_windows) {
+ GList *it;
+
+ cxl_fmw_base = ROUND_UP(base, 256 * MiB);
+ for (it = ms->cxl_devices_state->fixed_windows; it; it = it->next) {
+ CXLFixedWindow *fw = it->data;
+
+ fw->base = cxl_fmw_base;
+ memory_region_init_io(&fw->mr, OBJECT(vms), &cfmws_ops, fw,
+ "cxl-fixed-memory-region", fw->size);
+ cxl_fmw_base += fw->size;
+ }
+ }
}
/*
@@ -2175,6 +2202,15 @@ static void machvirt_init(MachineState *machine)
memory_region_add_subregion(sysmem, machine->device_memory->base,
&machine->device_memory->mr);
}
+ if (machine->cxl_devices_state->fixed_windows) {
+ GList *it;
+ for (it = machine->cxl_devices_state->fixed_windows; it;
+ it = it->next) {
+ CXLFixedWindow *fw = it->data;
+
+ memory_region_add_subregion(sysmem, fw->base, &fw->mr);
+ }
+ }
virt_flash_fdt(vms, sysmem, secure_sysmem ?: sysmem);
@@ -2201,6 +2237,7 @@ static void machvirt_init(MachineState *machine)
create_rtc(vms);
create_pcie(vms);
+ create_cxl_host_reg_region(vms);
if (has_ged && aarch64 && firmware_loaded && virt_is_acpi_enabled(vms)) {
vms->acpi_dev = create_acpi_ged(vms);
@@ -2856,6 +2893,7 @@ static void virt_machine_class_init(ObjectClass *oc, void *data)
hc->unplug = virt_machine_device_unplug_cb;
mc->nvdimm_supported = true;
mc->smp_props.clusters_supported = true;
+ mc->cxl_supported = true;
mc->auto_enable_numa_with_memhp = true;
mc->auto_enable_numa_with_memdev = true;
mc->default_ram_id = "mach-virt.ram";
diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h
index c1ea17d0de..097e1f0c36 100644
--- a/include/hw/arm/virt.h
+++ b/include/hw/arm/virt.h
@@ -92,6 +92,7 @@ enum {
/* indices of IO regions located after the RAM */
enum {
VIRT_HIGH_GIC_REDIST2 = VIRT_LOWMEMMAP_LAST,
+ VIRT_CXL_HOST,
VIRT_HIGH_PCIE_ECAM,
VIRT_HIGH_PCIE_MMIO,
};
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 38/43] RFC: softmmu/memory: Add ops to memory_region_ram_init_from_file
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (36 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 37/43] hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances pxb-cxl Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 39/43] hw/cxl/component Add a dumb HDM decoder handler Jonathan Cameron
` (6 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
Inorder to implement memory interleaving we need a means to proxy
the calls. Adding mem_ops allows such proxying.
Note should have no impact on use cases not using _dispatch_read/write.
For now, only file backed hostmem is considered to seek feedback on
the approach before considering other hostmem backends.
Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
---
softmmu/memory.c | 9 +++++++++
1 file changed, 9 insertions(+)
diff --git a/softmmu/memory.c b/softmmu/memory.c
index 678dc62f06..d537091c63 100644
--- a/softmmu/memory.c
+++ b/softmmu/memory.c
@@ -1606,6 +1606,15 @@ void memory_region_init_ram_from_file(MemoryRegion *mr,
Error *err = NULL;
memory_region_init(mr, owner, name, size);
mr->ram = true;
+
+ /*
+ * ops used only when directly accessing via
+ * - memory_region_dispatch_read()
+ * - memory_region_dispatch_write()
+ */
+ mr->ops = &ram_device_mem_ops;
+ mr->opaque = mr;
+
mr->readonly = readonly;
mr->terminates = true;
mr->destructor = memory_region_destructor_ram;
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 39/43] hw/cxl/component Add a dumb HDM decoder handler
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (37 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 38/43] RFC: softmmu/memory: Add ops to memory_region_ram_init_from_file Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 40/43] i386/pc: Enable CXL fixed memory windows Jonathan Cameron
` (5 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
Add a trivial handler for now to cover the root bridge
where we could do some error checking in future.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/cxl/cxl-component-utils.c | 31 +++++++++++++++++++++++++++++++
1 file changed, 31 insertions(+)
diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index 795dbc7561..c5124708b6 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -32,6 +32,31 @@ static uint64_t cxl_cache_mem_read_reg(void *opaque, hwaddr offset,
}
}
+static void dumb_hdm_handler(CXLComponentState *cxl_cstate, hwaddr offset,
+ uint32_t value)
+{
+ ComponentRegisters *cregs = &cxl_cstate->crb;
+ uint32_t *cache_mem = cregs->cache_mem_registers;
+ bool should_commit = false;
+
+ switch (offset) {
+ case A_CXL_HDM_DECODER0_CTRL:
+ should_commit = FIELD_EX32(value, CXL_HDM_DECODER0_CTRL, COMMIT);
+ break;
+ default:
+ break;
+ }
+
+ memory_region_transaction_begin();
+ stl_le_p((uint8_t *)cache_mem + offset, value);
+ if (should_commit) {
+ ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, COMMIT, 0);
+ ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, ERR, 0);
+ ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, COMMITTED, 1);
+ }
+ memory_region_transaction_commit();
+}
+
static void cxl_cache_mem_write_reg(void *opaque, hwaddr offset, uint64_t value,
unsigned size)
{
@@ -45,6 +70,12 @@ static void cxl_cache_mem_write_reg(void *opaque, hwaddr offset, uint64_t value,
}
if (cregs->special_ops && cregs->special_ops->write) {
cregs->special_ops->write(cxl_cstate, offset, value, size);
+ return;
+ }
+
+ if (offset >= A_CXL_HDM_DECODER_CAPABILITY &&
+ offset <= A_CXL_HDM_DECODER0_TARGET_LIST_HI) {
+ dumb_hdm_handler(cxl_cstate, offset, value);
} else {
cregs->cache_mem_registers[offset / 4] = value;
}
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 40/43] i386/pc: Enable CXL fixed memory windows
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (38 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 39/43] hw/cxl/component Add a dumb HDM decoder handler Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 41/43] qtest/acpi: Add reference CEDT tables Jonathan Cameron
` (4 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
Add the CFMWs memory regions to the memorymap and adjust the
PCI window to avoid hitting the same memory.
Signed-off-by: Jonathan Cameron <jonathan.cameron@huawei.com>
---
hw/i386/pc.c | 31 ++++++++++++++++++++++++++++++-
1 file changed, 30 insertions(+), 1 deletion(-)
diff --git a/hw/i386/pc.c b/hw/i386/pc.c
index 7a18dce529..5ece806d2b 100644
--- a/hw/i386/pc.c
+++ b/hw/i386/pc.c
@@ -816,7 +816,7 @@ void pc_memory_init(PCMachineState *pcms,
MachineClass *mc = MACHINE_GET_CLASS(machine);
PCMachineClass *pcmc = PC_MACHINE_GET_CLASS(pcms);
X86MachineState *x86ms = X86_MACHINE(pcms);
- hwaddr cxl_base;
+ hwaddr cxl_base, cxl_resv_end = 0;
assert(machine->ram_size == x86ms->below_4g_mem_size +
x86ms->above_4g_mem_size);
@@ -924,6 +924,24 @@ void pc_memory_init(PCMachineState *pcms,
e820_add_entry(cxl_base, cxl_size, E820_RESERVED);
memory_region_init(mr, OBJECT(machine), "cxl_host_reg", cxl_size);
memory_region_add_subregion(system_memory, cxl_base, mr);
+ cxl_resv_end = cxl_base + cxl_size;
+ if (machine->cxl_devices_state->fixed_windows) {
+ hwaddr cxl_fmw_base;
+ GList *it;
+
+ cxl_fmw_base = ROUND_UP(cxl_base + cxl_size, 256 * MiB);
+ for (it = machine->cxl_devices_state->fixed_windows; it; it = it->next) {
+ CXLFixedWindow *fw = it->data;
+
+ fw->base = cxl_fmw_base;
+ memory_region_init_io(&fw->mr, OBJECT(machine), &cfmws_ops, fw,
+ "cxl-fixed-memory-region", fw->size);
+ memory_region_add_subregion(system_memory, fw->base, &fw->mr);
+ e820_add_entry(fw->base, fw->size, E820_RESERVED);
+ cxl_fmw_base += fw->size;
+ cxl_resv_end = cxl_fmw_base;
+ }
+ }
}
/* Initialize PC system firmware */
@@ -953,6 +971,10 @@ void pc_memory_init(PCMachineState *pcms,
if (!pcmc->broken_reserved_end) {
res_mem_end += memory_region_size(&machine->device_memory->mr);
}
+
+ if (machine->cxl_devices_state->is_enabled) {
+ res_mem_end = cxl_resv_end;
+ }
*val = cpu_to_le64(ROUND_UP(res_mem_end, 1 * GiB));
fw_cfg_add_file(fw_cfg, "etc/reserved-memory-end", val, sizeof(*val));
}
@@ -989,6 +1011,13 @@ uint64_t pc_pci_hole64_start(void)
if (ms->cxl_devices_state->host_mr.addr) {
hole64_start = ms->cxl_devices_state->host_mr.addr +
memory_region_size(&ms->cxl_devices_state->host_mr);
+ if (ms->cxl_devices_state->fixed_windows) {
+ GList *it;
+ for (it = ms->cxl_devices_state->fixed_windows; it; it = it->next) {
+ CXLFixedWindow *fw = it->data;
+ hole64_start = fw->mr.addr + memory_region_size(&fw->mr);
+ }
+ }
} else if (pcmc->has_reserved_memory && ms->device_memory->base) {
hole64_start = ms->device_memory->base;
if (!pcmc->broken_reserved_end) {
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 41/43] qtest/acpi: Add reference CEDT tables.
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (39 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 40/43] i386/pc: Enable CXL fixed memory windows Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 42/43] qtest/cxl: Add very basic sanity tests Jonathan Cameron
` (3 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Jonathan Cameron <jonathan.cameron@huawei.com>
More sophisticated tests will come later, but for now deal
with the NULL case.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
tests/data/acpi/pc/CEDT | Bin 0 -> 36 bytes
tests/data/acpi/q35/CEDT | Bin 0 -> 36 bytes
tests/data/acpi/virt/CEDT | Bin 0 -> 36 bytes
tests/qtest/bios-tables-test-allowed-diff.h | 3 ---
4 files changed, 3 deletions(-)
diff --git a/tests/data/acpi/pc/CEDT b/tests/data/acpi/pc/CEDT
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..b44db4ce1db980d783ad568a03c17c2915d111b0 100644
GIT binary patch
literal 36
jcmZ>EbqP^nU|?VjaPoKd2v%^42yj*a0!E-1hz+6veU1hJ
literal 0
HcmV?d00001
diff --git a/tests/data/acpi/q35/CEDT b/tests/data/acpi/q35/CEDT
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..b44db4ce1db980d783ad568a03c17c2915d111b0 100644
GIT binary patch
literal 36
jcmZ>EbqP^nU|?VjaPoKd2v%^42yj*a0!E-1hz+6veU1hJ
literal 0
HcmV?d00001
diff --git a/tests/data/acpi/virt/CEDT b/tests/data/acpi/virt/CEDT
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..b44db4ce1db980d783ad568a03c17c2915d111b0 100644
GIT binary patch
literal 36
jcmZ>EbqP^nU|?VjaPoKd2v%^42yj*a0!E-1hz+6veU1hJ
literal 0
HcmV?d00001
diff --git a/tests/qtest/bios-tables-test-allowed-diff.h b/tests/qtest/bios-tables-test-allowed-diff.h
index c7726cad80..dfb8523c8b 100644
--- a/tests/qtest/bios-tables-test-allowed-diff.h
+++ b/tests/qtest/bios-tables-test-allowed-diff.h
@@ -1,4 +1 @@
/* List of comma-separated changed AML files to ignore */
-"tests/data/acpi/pc/CEDT",
-"tests/data/acpi/q35/CEDT",
-"tests/data/acpi/virt/CEDT",
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 42/43] qtest/cxl: Add very basic sanity tests
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (40 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 41/43] qtest/acpi: Add reference CEDT tables Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-02 14:10 ` [PATCH v5 43/43] scripts/device-crash-test: Add exception for pxb-cxl Jonathan Cameron
` (2 subsequent siblings)
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
From: Ben Widawsky <ben.widawsky@intel.com>
Simple 'does it boot tests' with up to
2x PXB host bridge, each with 2x CXL RP and each of those with
a Type 3 memory device. Single CFMWS to interleave across the
two HBs and ultimate the 4 devices.
More complete tests may be possible but CXL interleave setup
is complex so a lot of steps will be needed.
Signed-off-by: Ben Widawsky <ben.widawsky@intel.com>
Co-developed-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
tests/qtest/cxl-test.c | 151 ++++++++++++++++++++++++++++++++++++++++
tests/qtest/meson.build | 4 ++
2 files changed, 155 insertions(+)
create mode 100644 tests/qtest/cxl-test.c
diff --git a/tests/qtest/cxl-test.c b/tests/qtest/cxl-test.c
new file mode 100644
index 0000000000..a50c0c6de4
--- /dev/null
+++ b/tests/qtest/cxl-test.c
@@ -0,0 +1,151 @@
+/*
+ * QTest testcase for CXL
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "libqtest-single.h"
+
+#define QEMU_PXB_CMD "-machine q35,cxl=on " \
+ "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \
+ "-cxl-fixed-memory-window targets=cxl.0,size=4G "
+
+#define QEMU_2PXB_CMD "-machine q35,cxl=on " \
+ "-device pxb-cxl,id=cxl.0,bus=pcie.0,bus_nr=52 " \
+ "-device pxb-cxl,id=cxl.1,bus=pcie.0,bus_nr=53 " \
+ "-cxl-fixed-memory-window targets=cxl.0,targets=cxl.1,size=4G "
+
+#define QEMU_RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 "
+
+/* Dual ports on first pxb */
+#define QEMU_2RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \
+ "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 "
+
+/* Dual ports on each of the pxb instances */
+#define QEMU_4RP "-device cxl-rp,id=rp0,bus=cxl.0,chassis=0,slot=0 " \
+ "-device cxl-rp,id=rp1,bus=cxl.0,chassis=0,slot=1 " \
+ "-device cxl-rp,id=rp2,bus=cxl.1,chassis=0,slot=2 " \
+ "-device cxl-rp,id=rp3,bus=cxl.1,chassis=0,slot=3 "
+
+#define QEMU_T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \
+ "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M "
+
+#define QEMU_2T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \
+ "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " \
+ "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \
+ "-device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M "
+
+#define QEMU_4T3D "-object memory-backend-file,id=cxl-mem0,mem-path=%s,size=256M " \
+ "-device cxl-type3,bus=rp0,memdev=cxl-mem0,id=cxl-pmem0,size=256M " \
+ "-object memory-backend-file,id=cxl-mem1,mem-path=%s,size=256M " \
+ "-device cxl-type3,bus=rp1,memdev=cxl-mem1,id=cxl-pmem1,size=256M " \
+ "-object memory-backend-file,id=cxl-mem2,mem-path=%s,size=256M " \
+ "-device cxl-type3,bus=rp2,memdev=cxl-mem2,id=cxl-pmem2,size=256M " \
+ "-object memory-backend-file,id=cxl-mem3,mem-path=%s,size=256M " \
+ "-device cxl-type3,bus=rp3,memdev=cxl-mem3,id=cxl-pmem3,size=256M "
+
+static void cxl_basic_hb(void)
+{
+ qtest_start("-machine q35,cxl=on");
+ qtest_end();
+}
+
+static void cxl_basic_pxb(void)
+{
+ qtest_start("-machine q35,cxl=on -device pxb-cxl,bus=pcie.0");
+ qtest_end();
+}
+
+static void cxl_pxb_with_window(void)
+{
+ qtest_start(QEMU_PXB_CMD);
+ qtest_end();
+}
+
+static void cxl_2pxb_with_window(void)
+{
+ qtest_start(QEMU_2PXB_CMD);
+ qtest_end();
+}
+
+static void cxl_root_port(void)
+{
+ qtest_start(QEMU_PXB_CMD QEMU_RP);
+ qtest_end();
+}
+
+static void cxl_2root_port(void)
+{
+ qtest_start(QEMU_PXB_CMD QEMU_2RP);
+ qtest_end();
+}
+
+static void cxl_t3d(void)
+{
+ GString *cmdline;
+ char template[] = "/tmp/cxl-test-XXXXXX";
+ const char *tmpfs;
+
+ tmpfs = mkdtemp(template);
+
+ cmdline = g_string_new(NULL);
+ g_string_printf(cmdline, QEMU_PXB_CMD QEMU_RP QEMU_T3D, tmpfs);
+
+ qtest_start(cmdline->str);
+ qtest_end();
+
+ g_string_free(cmdline, TRUE);
+}
+
+static void cxl_1pxb_2rp_2t3d(void)
+{
+ GString *cmdline;
+ char template[] = "/tmp/cxl-test-XXXXXX";
+ const char *tmpfs;
+
+ tmpfs = mkdtemp(template);
+
+ cmdline = g_string_new(NULL);
+ g_string_printf(cmdline, QEMU_PXB_CMD QEMU_2RP QEMU_2T3D, tmpfs, tmpfs);
+
+ qtest_start(cmdline->str);
+ qtest_end();
+
+ g_string_free(cmdline, TRUE);
+}
+
+static void cxl_2pxb_4rp_4t3d(void)
+{
+ GString *cmdline;
+ char template[] = "/tmp/cxl-test-XXXXXX";
+ const char *tmpfs;
+
+ tmpfs = mkdtemp(template);
+
+ cmdline = g_string_new(NULL);
+ g_string_printf(cmdline, QEMU_2PXB_CMD QEMU_4RP QEMU_4T3D,
+ tmpfs, tmpfs, tmpfs, tmpfs);
+
+ qtest_start(cmdline->str);
+ qtest_end();
+
+ g_string_free(cmdline, TRUE);
+}
+
+int main(int argc, char **argv)
+{
+ g_test_init(&argc, &argv, NULL);
+
+ qtest_add_func("/pci/cxl/basic_hostbridge", cxl_basic_hb);
+ qtest_add_func("/pci/cxl/basic_pxb", cxl_basic_pxb);
+ qtest_add_func("/pci/cxl/pxb_with_window", cxl_pxb_with_window);
+ qtest_add_func("/pci/cxl/pxb_x2_with_window", cxl_2pxb_with_window);
+ qtest_add_func("/pci/cxl/rp", cxl_root_port);
+ qtest_add_func("/pci/cxl/rp_x2", cxl_2root_port);
+ qtest_add_func("/pci/cxl/type3_device", cxl_t3d);
+ qtest_add_func("/pci/cxl/rp_x2_type3_x2", cxl_1pxb_2rp_2t3d);
+ qtest_add_func("/pci/cxl/pxb_x2_root_port_x4_type3_x4", cxl_2pxb_4rp_4t3d);
+ return g_test_run();
+}
diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index 842b1df420..52a7bbe177 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -41,6 +41,9 @@ qtests_pci = \
(config_all_devices.has_key('CONFIG_VGA') ? ['display-vga-test'] : []) + \
(config_all_devices.has_key('CONFIG_IVSHMEM_DEVICE') ? ['ivshmem-test'] : [])
+qtests_cxl = \
+ (config_all_devices.has_key('CONFIG_CXL') ? ['cxl-test'] : [])
+
qtests_i386 = \
(slirp.found() ? ['pxe-test', 'test-netfilter'] : []) + \
(config_host.has_key('CONFIG_POSIX') ? ['test-filter-mirror'] : []) + \
@@ -74,6 +77,7 @@ qtests_i386 = \
slirp.found() ? ['virtio-net-failover'] : []) + \
(unpack_edk2_blobs ? ['bios-tables-test'] : []) + \
qtests_pci + \
+ qtests_cxl + \
['fdc-test',
'ide-test',
'hd-geo-test',
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* [PATCH v5 43/43] scripts/device-crash-test: Add exception for pxb-cxl
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (41 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 42/43] qtest/cxl: Add very basic sanity tests Jonathan Cameron
@ 2022-02-02 14:10 ` Jonathan Cameron
2022-02-04 14:03 ` [PATCH v5 00/43] CXl 2.0 emulation Support Michael S. Tsirkin
2022-02-07 14:20 ` Jonathan Cameron
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-02 14:10 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
The CXL expander bridge has several requirements but the one that
is checked first is that it is attached to a PCI Express bus,
not a PCI one so document that.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
v5:
New patch - should probably be pushed down to introduction of pxb-cxl.
Will do that in v6
scripts/device-crash-test | 1 +
1 file changed, 1 insertion(+)
diff --git a/scripts/device-crash-test b/scripts/device-crash-test
index 7fbd99158b..52bd3d8f71 100755
--- a/scripts/device-crash-test
+++ b/scripts/device-crash-test
@@ -93,6 +93,7 @@ ERROR_RULE_LIST = [
{'device':'pci-bridge', 'expected':True}, # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
{'device':'pci-bridge-seat', 'expected':True}, # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
{'device':'pxb', 'expected':True}, # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
+ {'device':'pxb-cxl', 'expected':True}, # pxb-cxl devices cannot reside on a PCI bus.
{'device':'scsi-block', 'expected':True}, # drive property not set
{'device':'scsi-generic', 'expected':True}, # drive property not set
{'device':'scsi-hd', 'expected':True}, # drive property not set
--
2.32.0
^ permalink raw reply related [flat|nested] 54+ messages in thread* Re: [PATCH v5 00/43] CXl 2.0 emulation Support
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (42 preceding siblings ...)
2022-02-02 14:10 ` [PATCH v5 43/43] scripts/device-crash-test: Add exception for pxb-cxl Jonathan Cameron
@ 2022-02-04 14:03 ` Michael S. Tsirkin
2022-02-04 14:27 ` Michael S. Tsirkin
2022-02-07 14:20 ` Jonathan Cameron
44 siblings, 1 reply; 54+ messages in thread
From: Michael S. Tsirkin @ 2022-02-04 14:03 UTC (permalink / raw)
To: Jonathan Cameron
Cc: qemu-devel, Alex Bennée, Marcel Apfelbaum, Igor Mammedov,
linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
On Wed, Feb 02, 2022 at 02:09:54PM +0000, Jonathan Cameron wrote:
> Changes since v4:
> https://lore.kernel.org/linux-cxl/20220124171705.10432-1-Jonathan.Cameron@huawei.com/
>
> Note documentation patch that Alex requested to follow.
> I don't want to delay getting this out as Alex mentioned possibly
> having time to continue reviewing in latter part of this week.
>
> Issues identified by CI / Alex Bennée
> - Stubs added for hw/cxl/cxl-host and hw/acpi/cxl plus related meson
> changes to use them as necessary.
> - Drop uid from cxl-test (result of last minute change in v4 that was not
> carried through to the test)
> - Fix naming clash with field name ERROR which on some arches is defined
> and results in the string being replaced with 0 in some of the
> register field related defines. Call it ERR instead.
> - Fix type issue around mr->size by using 64 bit acessor functions.
> - Add a new patch to exclude pxb-cxl from device-crash-test in similar
> fashion to pxb.
>
> CI tests now passing with exception of checkpatch which has what
> I think is a false positive and build-oss-fuzz which keeps timing out.
> https://gitlab.com/jic23/qemu/-/pipelines/460109208
> There were a few tweaks to patch descriptions after I pushed that
> out (I missed a few RB from Alex).
There's an RFC patch that needs review from core memory maintainers,
so I guess not all of it is for merge just yet?
Is there any way we can start applying this patchset gradually?
> Other changes (mostly from Alex's review)
> - Change component register handling to now report UNIMP and return 0
> for 8 byte registers as we currently don't implement any of them.
> Note that this means we need a kernel fix:
> https://lore.kernel.org/linux-cxl/20220201153437.2873-1-Jonathan.Cameron@huawei.com/
> - Drop majority of the macros used in defining mailbox handlers in
> favour of written out code.
> - Use REG64 where appropriate. This was introduced whilst this set
> has been underdevelopment so I missed it.
> - Clarify some register access options wrt to CXL 2.0 Errata F4.
> - Change timestamp to qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL)
> - Use typed enums to enforce types of function arguements.
> - Default to cxl being off in machine_class_init() removing
> need to set it to off in machines where there is no support as yet.
> - Add Alex's RB where given.
>
> Looking in particular for:
> * Review of the PCI interactions
> * x86 and ARM machine interactions (particularly the memory maps)
> * Review of the interleaving approach - is the basic idea
> acceptable?
> * Review of the command line interface.
> * CXL related review welcome but much of that got reviewed
> in earlier versions and hasn't changed substantially.
>
> Big TODOs:
>
> * Interleave boundary issues. I haven't yet solved this but didn't
> want to futher delay the review of the rest of the series.
>
> * Volatile memory devices (easy but it's more code so left for now).
> * Switch support. Linux kernel support is under review currently,
> so there is now something to test against.
> * Hotplug? May not need much but it's not tested yet!
> * More tests and tighter verification that values written to hardware
> are actually valid - stuff that real hardware would check.
> * Testing, testing and more testing. I have been running a basic
> set of ARM and x86 tests on this, but there is always room for
> more tests and greater automation.
> * CFMWS flags as requested by Ben.
>
> Why do we want QEMU emulation of CXL?
>
> As Ben stated in V3, QEMU support has been critical to getting OS
> software written given lack of availability of hardware supporting the
> latest CXL features (coupled with very high demand for support being
> ready in a timely fashion). What has become clear since Ben's v3
> is that situation is a continuous one. Whilst we can't talk about
> them yet, CXL 3.0 features and OS support have been prototyped on
> top of this support and a lot of the ongoing kernel work is being
> tested against these patches. The kernel CXL mocking code allows
> some forms of testing, but QEMU provides a more versatile and
> exensible platform.
>
> Other features on the qemu-list that build on these include PCI-DOE
> /CDAT support from the Avery Design team further showing how this
> code is useful. Whilst not directly related this is also the test
> platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both
> utilizes and extends those technologies and is likely to be an early
> adopter.
> Refs:
> CMA Kernel: https://lore.kernel.org/all/20210804161839.3492053-1-Jonathan.Cameron@huawei.com/
> CMA Qemu: https://lore.kernel.org/qemu-devel/1624665723-5169-1-git-send-email-cbrowy@avery-design.com/
> DOE Qemu: https://lore.kernel.org/qemu-devel/1623329999-15662-1-git-send-email-cbrowy@avery-design.com/
>
> As can be seen there is non trivial interaction with other areas of
> Qemu, particularly PCI and keeping this set up to date is proving
> a burden we'd rather do without :)
>
> Ben mentioned a few other good reasons in v3:
> https://lore.kernel.org/qemu-devel/20210202005948.241655-1-ben.widawsky@intel.com/
>
> The evolution of this series perhaps leave it in a less than
> entirely obvious order and that may get tidied up in future postings.
> I'm also open to this being considered in bite sized chunks. What
> we have here is about what you need for it to be useful for testing
> currently kernel code. Note the kernel code is moving fast so
> since v4, some features have been introduced we don't yet support in
> QEMU (e.g. use of the PCIe serial number extended capability).
>
> All comments welcome.
>
> qemu-system-aarch64 -M virt,gic-version=3,cxl=on \
> -m 4g,maxmem=8G,slots=8 \
> ...
> -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
> -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M \
> -object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M \
> -object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M \
> -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
> -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M \
> -object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M \
> -object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M \
> -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
> -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \
> -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
> -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,size=256M \
> -device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \
> -device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1,size=256M \
> -device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \
> -device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2,size=256M \
> -device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \
> -device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3,size=256M \
> -cxl-fixed-memory-window targets=cxl.1,size=4G,interleave-granularity=8k \
> -cxl-fixed-memory-window targets=cxl.1,targets=cxl.2,size=4G,interleave-granularity=8k
>
> First CFMWS suitable for up to 2 way interleave, the second for 4 way (2 way
> at host level and 2 way at the host bridge).
> targets=<range of pxb-cxl uids> , multiple entries if range is disjoint.
>
> With the v5.17-rc1 + patch series listed below.
>
> cd /sys/bus/cxl/devices/
> region=$(cat decoder0.1/create_region)
> echo $region > decoder0.1/create_region
> ls -lh
>
> //Note the order of devices and adjust the following to make sure they
> //are in order across the 4 root ports. Easy to do in a tool, but
> //not easy to paste in a cover letter.
>
> cd region0.1\:0
> echo 4 > interleave_ways
> echo mem2 > target0
> echo mem3 > target1
> echo mem0 > target2
> echo mem1 > target3
> echo $((1024<<20)) > size
> echo 4096 > interleave_granularity
> echo region0.1:0 > /sys/bus/cxl/drivers/cxl_region/bind
>
> Tested with devmem2 and files with known content.
> Kernel tree is mainline + (I based on 5.17-rc1)
> [PATCH] cxl/regs: Fix size of CXL Capabilty Header Register
> https://lore.kernel.org/linux-cxl/20220201182934.jjvavjsf4h7oqngv@intel.com/T/#t
>
> [PATCH v3 00/40] CXL.mem Topology Discovery and Hotplug Support
> https://lore.kernel.org/linux-cxl/164298411792.3018233.7493009997525360044.stgit@dwillia2-desk3.amr.corp.intel.com/
> Note that series has a lot of v4/v5 patches are replies but b4 does
> a good job of pulling out the latest.
>
> [PATCH 0/2] cxl/port: Robustness fixes for decoder enumeration
> https://lore.kernel.org/linux-cxl/164317463887.3438644.4087819721493502301.stgit@dwillia2-desk3.amr.corp.intel.com/
>
> [PATCH 0/4] Unify meaning of interleave attributes
> https://lore.kernel.org/linux-cxl/20220127212911.127741-1-ben.widawsky@intel.com/
>
> [PATCH v3 00/14] CXL Region driver
> https://lore.kernel.org/linux-cxl/20220128002707.391076-1-ben.widawsky@intel.com/
>
> What follows is a first attempt at explaining how all these components
> fit together. I'll write up some formal documentation shortly.
>
> Memory Address Map for CXL elements. Note where exactly these regions
> appear is Arch and platform dependent.
>
> Base somewhere far up in the Host PA map.
> _______________________________
> | |
> | CXL Host Bridge 0 Registers |
> | CXL Host Bridge 1 Registers |
> | ... | This bit is normal MMIO register space.
> | CXL Host bridge N registers | including programmable interleave decoders
> |______________________________| for interleave across root ports.
> | |
> ....
> | |
> |______________________________|
> | |
> | CFMW 0, | Note that there can be multiple regions
> | Interleave 2 way, targets | of memory within this 1TB which can be
> | Hostbridge 0, Hostbridge 1 | interleaved differently: in the host bridges
> | Granularity 16KiB, 1TB | across root ports or in switches below the root.
> |______________________________| ports
> | |
> | CFMW 1, |
> | Interleave 1 way, target |
> | Hostbridge 0, 512GiB |
> |______________________________|
> etc for all interleave combinations
> configured, or built in to the
> system before any generic software
> sees it.
>
> System Topology considering CFMW 0 only to keep this simple.
> x marks the match in each decoder level.
> Switches have more interleave decoders and other features
> that we haven't implemented yet in QEMU.
>
> Address Read to CFMW0 base + N
> _________________|________________
> | |
> | Host interconnect |
> | Configured to route CFM |
> | memory access to particular HB |
> |_____x____________________________|
> | |
> Interleave Decoder |
> Matches this HB |
> | |
> _______|__________ _____|____________
> | | | |
> | CXL HB 0 | | CXL HB 1 | Only exist in PCI (mostly)
> | HB IntLv Decoder | | HB IntLv Decoder | via ACPI description
> | PCI Root Bus 0c | | PCI Root Bus 0d |
> |x_________________| |__________________| In CXL have MMIO
> | | | | at location given in CEDT
> | | | | CHBS entry (ACPI)
> ____________|___ __________|__ __|_________ ___|_________
> | Root Port 0 | | Root Port 1 | | Root Port 2| | Root Port 3 |
> | Appears in | | Appears in | | Appears in | | Appear in |
> | PCI topology | | PCI Topology| | PCI Topo | | PCI Topo |
> | As 0c:00.0 | | as 0c:01.0 | | as de:00.0 | | as de:01.0 |
> |_______________| |_____________| |____________| |_____________|
> | | | |
> | | | |
> _____|_________ ______|______ ______|_____ ______|_______
> | x | | | | | | |
> | CXL Type3 0 | | CXL Type3 1 | | CXL type3 2| | CLX Type 3 3 |
> | | | | | | | |
> | PMEM0(Vol LSA)| | PMEM1 (...) | | PMEM2 (...)| | PMEM3 (...) |
> | Decoder to go | | | | | | |
> | from host PA | | PCI 0e:00.0 | | PCI df:00.0| | PCI e0:00.0 |
> | to device PA | | | | | | |
> | PCI as 0d:00.0| | | | | | |
> |_______________| |_____________| |____________| |______________|
>
> Backed by Backed by Backed by Backed by
> file 0 file 1 file 2 file 3
>
> LSA backed by additional files for each device (not yet supported)
>
> So currently we have decoders as follows for each interleaved access.
> 1) CFMW decoder - fixed config so forms part of qemu command line.
> 2) Host bridge decoders - programmable decoders that the system
> software will program either based on user command or based
> on info from the Label Storage Area (not yet emulated)
> 3) Type 3 device decoders. Down to here the address used is the
> Host PA. These decoders convert to the local device PA
> (in simple case - drop some bits in the middle of the address)
>
> Future patches will add decoders in switch upstream ports making
> the above diagram have another layer between root ports and
> the memory devices.
>
> Note, we've focused for now on Persistent Memory devices as they are seen
> as an early and important usecase (and are the most complex one).
> But it should be straight forward to add volatile memory
> support and indeed that would be backed by RAM.
>
> lspci -tv for above shows
>
> -+-[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host Bridge (this is the cxl PXB)f
> | \-OTHER STUFF
> +-[0000:0c]-+-00.0-[0d]----00.0 Intel Corporation Device 0d93
> | \-01.0-[0e]----00.0 Intel Corporation Device 0d93
> \-[0000:de]-+-00.0-[df]----00.0 Intel Corporation Device 0d93
> \-01.0-[e0]----00.0 Intel Corporation Device 0d93
>
> Where those Intel parts are the type 3 devices.
>
> All comments welcome!
>
> Particular thanks to Alex Bennée for his review of v4.
>
> Thanks,
>
> Jonathan
>
> Ben Widawsky (26):
> hw/pci/cxl: Add a CXL component type (interface)
> hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
> hw/cxl/device: Introduce a CXL device (8.2.8)
> hw/cxl/device: Implement the CAP array (8.2.8.1-2)
> hw/cxl/device: Implement basic mailbox (8.2.8.4)
> hw/cxl/device: Add memory device utilities
> hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1)
> hw/cxl/device: Timestamp implementation (8.2.9.3)
> hw/cxl/device: Add log commands (8.2.9.4) + CEL
> hw/pxb: Use a type for realizing expanders
> hw/pci/cxl: Create a CXL bus type
> hw/pxb: Allow creation of a CXL PXB (host bridge)
> acpi/pci: Consolidate host bridge setup
> hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
> hw/cxl/rp: Add a root port
> hw/cxl/device: Add a memory device (8.2.8.5)
> hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
> acpi/cxl: Add _OSC implementation (9.14.2)
> tests/acpi: allow CEDT table addition
> acpi/cxl: Create the CEDT (9.14.1)
> hw/cxl/device: Add some trivial commands
> hw/cxl/device: Plumb real Label Storage Area (LSA) sizing
> hw/cxl/device: Implement get/set Label Storage Area (LSA)
> acpi/cxl: Introduce CFMWS structures in CEDT
> hw/cxl/component Add a dumb HDM decoder handler
> qtest/cxl: Add very basic sanity tests
>
> Jonathan Cameron (17):
> MAINTAINERS: Add entry for Compute Express Link Emulation
> tests/acpi: allow DSDT.viot table changes.
> tests/acpi: Add update DSDT.viot
> cxl: Machine level control on whether CXL support is enabled
> hw/cxl/component: Add utils for interleave parameter encoding/decoding
> hw/cxl/host: Add support for CXL Fixed Memory Windows.
> hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
> pci/pcie_port: Add pci_find_port_by_pn()
> CXL/cxl_component: Add cxl_get_hb_cstate()
> mem/cxl_type3: Add read and write functions for associated hostmem.
> cxl/cxl-host: Add memops for CFMWS region.
> arm/virt: Allow virt/CEDT creation
> hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances
> pxb-cxl
> RFC: softmmu/memory: Add ops to memory_region_ram_init_from_file
> i386/pc: Enable CXL fixed memory windows
> qtest/acpi: Add reference CEDT tables.
> scripts/device-crash-test: Add exception for pxb-cxl
>
> MAINTAINERS | 7 +
> hw/Kconfig | 1 +
> hw/acpi/Kconfig | 5 +
> hw/acpi/cxl-stub.c | 12 +
> hw/acpi/cxl.c | 231 +++++++++++++
> hw/acpi/meson.build | 4 +-
> hw/arm/Kconfig | 1 +
> hw/arm/virt-acpi-build.c | 30 ++
> hw/arm/virt.c | 40 ++-
> hw/core/machine.c | 28 ++
> hw/cxl/Kconfig | 3 +
> hw/cxl/cxl-component-utils.c | 284 ++++++++++++++++
> hw/cxl/cxl-device-utils.c | 271 ++++++++++++++++
> hw/cxl/cxl-host-stubs.c | 22 ++
> hw/cxl/cxl-host.c | 263 +++++++++++++++
> hw/cxl/cxl-mailbox-utils.c | 483 ++++++++++++++++++++++++++++
> hw/cxl/meson.build | 12 +
> hw/i386/acpi-build.c | 98 ++++--
> hw/i386/pc.c | 57 +++-
> hw/mem/Kconfig | 5 +
> hw/mem/cxl_type3.c | 353 ++++++++++++++++++++
> hw/mem/meson.build | 1 +
> hw/meson.build | 1 +
> hw/pci-bridge/Kconfig | 5 +
> hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++
> hw/pci-bridge/meson.build | 1 +
> hw/pci-bridge/pci_expander_bridge.c | 171 +++++++++-
> hw/pci-bridge/pcie_root_port.c | 6 +-
> hw/pci-host/gpex-acpi.c | 22 +-
> hw/pci/pci.c | 21 +-
> hw/pci/pcie_port.c | 25 ++
> include/hw/acpi/cxl.h | 28 ++
> include/hw/arm/virt.h | 1 +
> include/hw/boards.h | 2 +
> include/hw/cxl/cxl.h | 51 +++
> include/hw/cxl/cxl_component.h | 206 ++++++++++++
> include/hw/cxl/cxl_device.h | 272 ++++++++++++++++
> include/hw/cxl/cxl_pci.h | 160 +++++++++
> include/hw/pci/pci.h | 14 +
> include/hw/pci/pci_bridge.h | 20 ++
> include/hw/pci/pci_bus.h | 7 +
> include/hw/pci/pci_ids.h | 1 +
> include/hw/pci/pcie_port.h | 2 +
> qapi/machine.json | 15 +
> qemu-options.hx | 37 +++
> scripts/device-crash-test | 1 +
> softmmu/memory.c | 9 +
> softmmu/vl.c | 11 +
> tests/data/acpi/pc/CEDT | Bin 0 -> 36 bytes
> tests/data/acpi/q35/CEDT | Bin 0 -> 36 bytes
> tests/data/acpi/q35/DSDT.viot | Bin 9398 -> 9416 bytes
> tests/data/acpi/virt/CEDT | Bin 0 -> 36 bytes
> tests/qtest/cxl-test.c | 151 +++++++++
> tests/qtest/meson.build | 4 +
> 54 files changed, 3645 insertions(+), 41 deletions(-)
> create mode 100644 hw/acpi/cxl-stub.c
> create mode 100644 hw/acpi/cxl.c
> create mode 100644 hw/cxl/Kconfig
> create mode 100644 hw/cxl/cxl-component-utils.c
> create mode 100644 hw/cxl/cxl-device-utils.c
> create mode 100644 hw/cxl/cxl-host-stubs.c
> create mode 100644 hw/cxl/cxl-host.c
> create mode 100644 hw/cxl/cxl-mailbox-utils.c
> create mode 100644 hw/cxl/meson.build
> create mode 100644 hw/mem/cxl_type3.c
> create mode 100644 hw/pci-bridge/cxl_root_port.c
> create mode 100644 include/hw/acpi/cxl.h
> create mode 100644 include/hw/cxl/cxl.h
> create mode 100644 include/hw/cxl/cxl_component.h
> create mode 100644 include/hw/cxl/cxl_device.h
> create mode 100644 include/hw/cxl/cxl_pci.h
> create mode 100644 tests/data/acpi/pc/CEDT
> create mode 100644 tests/data/acpi/q35/CEDT
> create mode 100644 tests/data/acpi/virt/CEDT
> create mode 100644 tests/qtest/cxl-test.c
>
> --
> 2.32.0
^ permalink raw reply [flat|nested] 54+ messages in thread* Re: [PATCH v5 00/43] CXl 2.0 emulation Support
2022-02-04 14:03 ` [PATCH v5 00/43] CXl 2.0 emulation Support Michael S. Tsirkin
@ 2022-02-04 14:27 ` Michael S. Tsirkin
2022-02-04 18:23 ` Jonathan Cameron
0 siblings, 1 reply; 54+ messages in thread
From: Michael S. Tsirkin @ 2022-02-04 14:27 UTC (permalink / raw)
To: Jonathan Cameron
Cc: qemu-devel, Alex Bennée, Marcel Apfelbaum, Igor Mammedov,
linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
On Fri, Feb 04, 2022 at 09:03:27AM -0500, Michael S. Tsirkin wrote:
> On Wed, Feb 02, 2022 at 02:09:54PM +0000, Jonathan Cameron wrote:
> > Changes since v4:
> > https://lore.kernel.org/linux-cxl/20220124171705.10432-1-Jonathan.Cameron@huawei.com/
> >
> > Note documentation patch that Alex requested to follow.
> > I don't want to delay getting this out as Alex mentioned possibly
> > having time to continue reviewing in latter part of this week.
> >
> > Issues identified by CI / Alex Bennée
> > - Stubs added for hw/cxl/cxl-host and hw/acpi/cxl plus related meson
> > changes to use them as necessary.
> > - Drop uid from cxl-test (result of last minute change in v4 that was not
> > carried through to the test)
> > - Fix naming clash with field name ERROR which on some arches is defined
> > and results in the string being replaced with 0 in some of the
> > register field related defines. Call it ERR instead.
> > - Fix type issue around mr->size by using 64 bit acessor functions.
> > - Add a new patch to exclude pxb-cxl from device-crash-test in similar
> > fashion to pxb.
> >
> > CI tests now passing with exception of checkpatch which has what
> > I think is a false positive and build-oss-fuzz which keeps timing out.
> > https://gitlab.com/jic23/qemu/-/pipelines/460109208
> > There were a few tweaks to patch descriptions after I pushed that
> > out (I missed a few RB from Alex).
>
> There's an RFC patch that needs review from core memory maintainers,
> so I guess not all of it is for merge just yet?
> Is there any way we can start applying this patchset gradually?
For example, pick up patches 1-13 for now? They seem to be ready ...
>
> > Other changes (mostly from Alex's review)
> > - Change component register handling to now report UNIMP and return 0
> > for 8 byte registers as we currently don't implement any of them.
> > Note that this means we need a kernel fix:
> > https://lore.kernel.org/linux-cxl/20220201153437.2873-1-Jonathan.Cameron@huawei.com/
> > - Drop majority of the macros used in defining mailbox handlers in
> > favour of written out code.
> > - Use REG64 where appropriate. This was introduced whilst this set
> > has been underdevelopment so I missed it.
> > - Clarify some register access options wrt to CXL 2.0 Errata F4.
> > - Change timestamp to qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL)
> > - Use typed enums to enforce types of function arguements.
> > - Default to cxl being off in machine_class_init() removing
> > need to set it to off in machines where there is no support as yet.
> > - Add Alex's RB where given.
> >
> > Looking in particular for:
> > * Review of the PCI interactions
> > * x86 and ARM machine interactions (particularly the memory maps)
> > * Review of the interleaving approach - is the basic idea
> > acceptable?
> > * Review of the command line interface.
> > * CXL related review welcome but much of that got reviewed
> > in earlier versions and hasn't changed substantially.
> >
> > Big TODOs:
> >
> > * Interleave boundary issues. I haven't yet solved this but didn't
> > want to futher delay the review of the rest of the series.
> >
> > * Volatile memory devices (easy but it's more code so left for now).
> > * Switch support. Linux kernel support is under review currently,
> > so there is now something to test against.
> > * Hotplug? May not need much but it's not tested yet!
> > * More tests and tighter verification that values written to hardware
> > are actually valid - stuff that real hardware would check.
> > * Testing, testing and more testing. I have been running a basic
> > set of ARM and x86 tests on this, but there is always room for
> > more tests and greater automation.
> > * CFMWS flags as requested by Ben.
> >
> > Why do we want QEMU emulation of CXL?
> >
> > As Ben stated in V3, QEMU support has been critical to getting OS
> > software written given lack of availability of hardware supporting the
> > latest CXL features (coupled with very high demand for support being
> > ready in a timely fashion). What has become clear since Ben's v3
> > is that situation is a continuous one. Whilst we can't talk about
> > them yet, CXL 3.0 features and OS support have been prototyped on
> > top of this support and a lot of the ongoing kernel work is being
> > tested against these patches. The kernel CXL mocking code allows
> > some forms of testing, but QEMU provides a more versatile and
> > exensible platform.
> >
> > Other features on the qemu-list that build on these include PCI-DOE
> > /CDAT support from the Avery Design team further showing how this
> > code is useful. Whilst not directly related this is also the test
> > platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both
> > utilizes and extends those technologies and is likely to be an early
> > adopter.
> > Refs:
> > CMA Kernel: https://lore.kernel.org/all/20210804161839.3492053-1-Jonathan.Cameron@huawei.com/
> > CMA Qemu: https://lore.kernel.org/qemu-devel/1624665723-5169-1-git-send-email-cbrowy@avery-design.com/
> > DOE Qemu: https://lore.kernel.org/qemu-devel/1623329999-15662-1-git-send-email-cbrowy@avery-design.com/
> >
> > As can be seen there is non trivial interaction with other areas of
> > Qemu, particularly PCI and keeping this set up to date is proving
> > a burden we'd rather do without :)
> >
> > Ben mentioned a few other good reasons in v3:
> > https://lore.kernel.org/qemu-devel/20210202005948.241655-1-ben.widawsky@intel.com/
> >
> > The evolution of this series perhaps leave it in a less than
> > entirely obvious order and that may get tidied up in future postings.
> > I'm also open to this being considered in bite sized chunks. What
> > we have here is about what you need for it to be useful for testing
> > currently kernel code. Note the kernel code is moving fast so
> > since v4, some features have been introduced we don't yet support in
> > QEMU (e.g. use of the PCIe serial number extended capability).
> >
> > All comments welcome.
> >
> > qemu-system-aarch64 -M virt,gic-version=3,cxl=on \
> > -m 4g,maxmem=8G,slots=8 \
> > ...
> > -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
> > -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M \
> > -object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M \
> > -object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M \
> > -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
> > -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M \
> > -object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M \
> > -object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M \
> > -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
> > -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \
> > -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
> > -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,size=256M \
> > -device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \
> > -device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1,size=256M \
> > -device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \
> > -device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2,size=256M \
> > -device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \
> > -device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3,size=256M \
> > -cxl-fixed-memory-window targets=cxl.1,size=4G,interleave-granularity=8k \
> > -cxl-fixed-memory-window targets=cxl.1,targets=cxl.2,size=4G,interleave-granularity=8k
> >
> > First CFMWS suitable for up to 2 way interleave, the second for 4 way (2 way
> > at host level and 2 way at the host bridge).
> > targets=<range of pxb-cxl uids> , multiple entries if range is disjoint.
> >
> > With the v5.17-rc1 + patch series listed below.
> >
> > cd /sys/bus/cxl/devices/
> > region=$(cat decoder0.1/create_region)
> > echo $region > decoder0.1/create_region
> > ls -lh
> >
> > //Note the order of devices and adjust the following to make sure they
> > //are in order across the 4 root ports. Easy to do in a tool, but
> > //not easy to paste in a cover letter.
> >
> > cd region0.1\:0
> > echo 4 > interleave_ways
> > echo mem2 > target0
> > echo mem3 > target1
> > echo mem0 > target2
> > echo mem1 > target3
> > echo $((1024<<20)) > size
> > echo 4096 > interleave_granularity
> > echo region0.1:0 > /sys/bus/cxl/drivers/cxl_region/bind
> >
> > Tested with devmem2 and files with known content.
> > Kernel tree is mainline + (I based on 5.17-rc1)
> > [PATCH] cxl/regs: Fix size of CXL Capabilty Header Register
> > https://lore.kernel.org/linux-cxl/20220201182934.jjvavjsf4h7oqngv@intel.com/T/#t
> >
> > [PATCH v3 00/40] CXL.mem Topology Discovery and Hotplug Support
> > https://lore.kernel.org/linux-cxl/164298411792.3018233.7493009997525360044.stgit@dwillia2-desk3.amr.corp.intel.com/
> > Note that series has a lot of v4/v5 patches are replies but b4 does
> > a good job of pulling out the latest.
> >
> > [PATCH 0/2] cxl/port: Robustness fixes for decoder enumeration
> > https://lore.kernel.org/linux-cxl/164317463887.3438644.4087819721493502301.stgit@dwillia2-desk3.amr.corp.intel.com/
> >
> > [PATCH 0/4] Unify meaning of interleave attributes
> > https://lore.kernel.org/linux-cxl/20220127212911.127741-1-ben.widawsky@intel.com/
> >
> > [PATCH v3 00/14] CXL Region driver
> > https://lore.kernel.org/linux-cxl/20220128002707.391076-1-ben.widawsky@intel.com/
> >
> > What follows is a first attempt at explaining how all these components
> > fit together. I'll write up some formal documentation shortly.
> >
> > Memory Address Map for CXL elements. Note where exactly these regions
> > appear is Arch and platform dependent.
> >
> > Base somewhere far up in the Host PA map.
> > _______________________________
> > | |
> > | CXL Host Bridge 0 Registers |
> > | CXL Host Bridge 1 Registers |
> > | ... | This bit is normal MMIO register space.
> > | CXL Host bridge N registers | including programmable interleave decoders
> > |______________________________| for interleave across root ports.
> > | |
> > ....
> > | |
> > |______________________________|
> > | |
> > | CFMW 0, | Note that there can be multiple regions
> > | Interleave 2 way, targets | of memory within this 1TB which can be
> > | Hostbridge 0, Hostbridge 1 | interleaved differently: in the host bridges
> > | Granularity 16KiB, 1TB | across root ports or in switches below the root.
> > |______________________________| ports
> > | |
> > | CFMW 1, |
> > | Interleave 1 way, target |
> > | Hostbridge 0, 512GiB |
> > |______________________________|
> > etc for all interleave combinations
> > configured, or built in to the
> > system before any generic software
> > sees it.
> >
> > System Topology considering CFMW 0 only to keep this simple.
> > x marks the match in each decoder level.
> > Switches have more interleave decoders and other features
> > that we haven't implemented yet in QEMU.
> >
> > Address Read to CFMW0 base + N
> > _________________|________________
> > | |
> > | Host interconnect |
> > | Configured to route CFM |
> > | memory access to particular HB |
> > |_____x____________________________|
> > | |
> > Interleave Decoder |
> > Matches this HB |
> > | |
> > _______|__________ _____|____________
> > | | | |
> > | CXL HB 0 | | CXL HB 1 | Only exist in PCI (mostly)
> > | HB IntLv Decoder | | HB IntLv Decoder | via ACPI description
> > | PCI Root Bus 0c | | PCI Root Bus 0d |
> > |x_________________| |__________________| In CXL have MMIO
> > | | | | at location given in CEDT
> > | | | | CHBS entry (ACPI)
> > ____________|___ __________|__ __|_________ ___|_________
> > | Root Port 0 | | Root Port 1 | | Root Port 2| | Root Port 3 |
> > | Appears in | | Appears in | | Appears in | | Appear in |
> > | PCI topology | | PCI Topology| | PCI Topo | | PCI Topo |
> > | As 0c:00.0 | | as 0c:01.0 | | as de:00.0 | | as de:01.0 |
> > |_______________| |_____________| |____________| |_____________|
> > | | | |
> > | | | |
> > _____|_________ ______|______ ______|_____ ______|_______
> > | x | | | | | | |
> > | CXL Type3 0 | | CXL Type3 1 | | CXL type3 2| | CLX Type 3 3 |
> > | | | | | | | |
> > | PMEM0(Vol LSA)| | PMEM1 (...) | | PMEM2 (...)| | PMEM3 (...) |
> > | Decoder to go | | | | | | |
> > | from host PA | | PCI 0e:00.0 | | PCI df:00.0| | PCI e0:00.0 |
> > | to device PA | | | | | | |
> > | PCI as 0d:00.0| | | | | | |
> > |_______________| |_____________| |____________| |______________|
> >
> > Backed by Backed by Backed by Backed by
> > file 0 file 1 file 2 file 3
> >
> > LSA backed by additional files for each device (not yet supported)
> >
> > So currently we have decoders as follows for each interleaved access.
> > 1) CFMW decoder - fixed config so forms part of qemu command line.
> > 2) Host bridge decoders - programmable decoders that the system
> > software will program either based on user command or based
> > on info from the Label Storage Area (not yet emulated)
> > 3) Type 3 device decoders. Down to here the address used is the
> > Host PA. These decoders convert to the local device PA
> > (in simple case - drop some bits in the middle of the address)
> >
> > Future patches will add decoders in switch upstream ports making
> > the above diagram have another layer between root ports and
> > the memory devices.
> >
> > Note, we've focused for now on Persistent Memory devices as they are seen
> > as an early and important usecase (and are the most complex one).
> > But it should be straight forward to add volatile memory
> > support and indeed that would be backed by RAM.
> >
> > lspci -tv for above shows
> >
> > -+-[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host Bridge (this is the cxl PXB)f
> > | \-OTHER STUFF
> > +-[0000:0c]-+-00.0-[0d]----00.0 Intel Corporation Device 0d93
> > | \-01.0-[0e]----00.0 Intel Corporation Device 0d93
> > \-[0000:de]-+-00.0-[df]----00.0 Intel Corporation Device 0d93
> > \-01.0-[e0]----00.0 Intel Corporation Device 0d93
> >
> > Where those Intel parts are the type 3 devices.
> >
> > All comments welcome!
> >
> > Particular thanks to Alex Bennée for his review of v4.
> >
> > Thanks,
> >
> > Jonathan
> >
> > Ben Widawsky (26):
> > hw/pci/cxl: Add a CXL component type (interface)
> > hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
> > hw/cxl/device: Introduce a CXL device (8.2.8)
> > hw/cxl/device: Implement the CAP array (8.2.8.1-2)
> > hw/cxl/device: Implement basic mailbox (8.2.8.4)
> > hw/cxl/device: Add memory device utilities
> > hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1)
> > hw/cxl/device: Timestamp implementation (8.2.9.3)
> > hw/cxl/device: Add log commands (8.2.9.4) + CEL
> > hw/pxb: Use a type for realizing expanders
> > hw/pci/cxl: Create a CXL bus type
> > hw/pxb: Allow creation of a CXL PXB (host bridge)
> > acpi/pci: Consolidate host bridge setup
> > hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
> > hw/cxl/rp: Add a root port
> > hw/cxl/device: Add a memory device (8.2.8.5)
> > hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
> > acpi/cxl: Add _OSC implementation (9.14.2)
> > tests/acpi: allow CEDT table addition
> > acpi/cxl: Create the CEDT (9.14.1)
> > hw/cxl/device: Add some trivial commands
> > hw/cxl/device: Plumb real Label Storage Area (LSA) sizing
> > hw/cxl/device: Implement get/set Label Storage Area (LSA)
> > acpi/cxl: Introduce CFMWS structures in CEDT
> > hw/cxl/component Add a dumb HDM decoder handler
> > qtest/cxl: Add very basic sanity tests
> >
> > Jonathan Cameron (17):
> > MAINTAINERS: Add entry for Compute Express Link Emulation
> > tests/acpi: allow DSDT.viot table changes.
> > tests/acpi: Add update DSDT.viot
> > cxl: Machine level control on whether CXL support is enabled
> > hw/cxl/component: Add utils for interleave parameter encoding/decoding
> > hw/cxl/host: Add support for CXL Fixed Memory Windows.
> > hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
> > pci/pcie_port: Add pci_find_port_by_pn()
> > CXL/cxl_component: Add cxl_get_hb_cstate()
> > mem/cxl_type3: Add read and write functions for associated hostmem.
> > cxl/cxl-host: Add memops for CFMWS region.
> > arm/virt: Allow virt/CEDT creation
> > hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances
> > pxb-cxl
> > RFC: softmmu/memory: Add ops to memory_region_ram_init_from_file
> > i386/pc: Enable CXL fixed memory windows
> > qtest/acpi: Add reference CEDT tables.
> > scripts/device-crash-test: Add exception for pxb-cxl
> >
> > MAINTAINERS | 7 +
> > hw/Kconfig | 1 +
> > hw/acpi/Kconfig | 5 +
> > hw/acpi/cxl-stub.c | 12 +
> > hw/acpi/cxl.c | 231 +++++++++++++
> > hw/acpi/meson.build | 4 +-
> > hw/arm/Kconfig | 1 +
> > hw/arm/virt-acpi-build.c | 30 ++
> > hw/arm/virt.c | 40 ++-
> > hw/core/machine.c | 28 ++
> > hw/cxl/Kconfig | 3 +
> > hw/cxl/cxl-component-utils.c | 284 ++++++++++++++++
> > hw/cxl/cxl-device-utils.c | 271 ++++++++++++++++
> > hw/cxl/cxl-host-stubs.c | 22 ++
> > hw/cxl/cxl-host.c | 263 +++++++++++++++
> > hw/cxl/cxl-mailbox-utils.c | 483 ++++++++++++++++++++++++++++
> > hw/cxl/meson.build | 12 +
> > hw/i386/acpi-build.c | 98 ++++--
> > hw/i386/pc.c | 57 +++-
> > hw/mem/Kconfig | 5 +
> > hw/mem/cxl_type3.c | 353 ++++++++++++++++++++
> > hw/mem/meson.build | 1 +
> > hw/meson.build | 1 +
> > hw/pci-bridge/Kconfig | 5 +
> > hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++
> > hw/pci-bridge/meson.build | 1 +
> > hw/pci-bridge/pci_expander_bridge.c | 171 +++++++++-
> > hw/pci-bridge/pcie_root_port.c | 6 +-
> > hw/pci-host/gpex-acpi.c | 22 +-
> > hw/pci/pci.c | 21 +-
> > hw/pci/pcie_port.c | 25 ++
> > include/hw/acpi/cxl.h | 28 ++
> > include/hw/arm/virt.h | 1 +
> > include/hw/boards.h | 2 +
> > include/hw/cxl/cxl.h | 51 +++
> > include/hw/cxl/cxl_component.h | 206 ++++++++++++
> > include/hw/cxl/cxl_device.h | 272 ++++++++++++++++
> > include/hw/cxl/cxl_pci.h | 160 +++++++++
> > include/hw/pci/pci.h | 14 +
> > include/hw/pci/pci_bridge.h | 20 ++
> > include/hw/pci/pci_bus.h | 7 +
> > include/hw/pci/pci_ids.h | 1 +
> > include/hw/pci/pcie_port.h | 2 +
> > qapi/machine.json | 15 +
> > qemu-options.hx | 37 +++
> > scripts/device-crash-test | 1 +
> > softmmu/memory.c | 9 +
> > softmmu/vl.c | 11 +
> > tests/data/acpi/pc/CEDT | Bin 0 -> 36 bytes
> > tests/data/acpi/q35/CEDT | Bin 0 -> 36 bytes
> > tests/data/acpi/q35/DSDT.viot | Bin 9398 -> 9416 bytes
> > tests/data/acpi/virt/CEDT | Bin 0 -> 36 bytes
> > tests/qtest/cxl-test.c | 151 +++++++++
> > tests/qtest/meson.build | 4 +
> > 54 files changed, 3645 insertions(+), 41 deletions(-)
> > create mode 100644 hw/acpi/cxl-stub.c
> > create mode 100644 hw/acpi/cxl.c
> > create mode 100644 hw/cxl/Kconfig
> > create mode 100644 hw/cxl/cxl-component-utils.c
> > create mode 100644 hw/cxl/cxl-device-utils.c
> > create mode 100644 hw/cxl/cxl-host-stubs.c
> > create mode 100644 hw/cxl/cxl-host.c
> > create mode 100644 hw/cxl/cxl-mailbox-utils.c
> > create mode 100644 hw/cxl/meson.build
> > create mode 100644 hw/mem/cxl_type3.c
> > create mode 100644 hw/pci-bridge/cxl_root_port.c
> > create mode 100644 include/hw/acpi/cxl.h
> > create mode 100644 include/hw/cxl/cxl.h
> > create mode 100644 include/hw/cxl/cxl_component.h
> > create mode 100644 include/hw/cxl/cxl_device.h
> > create mode 100644 include/hw/cxl/cxl_pci.h
> > create mode 100644 tests/data/acpi/pc/CEDT
> > create mode 100644 tests/data/acpi/q35/CEDT
> > create mode 100644 tests/data/acpi/virt/CEDT
> > create mode 100644 tests/qtest/cxl-test.c
> >
> > --
> > 2.32.0
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v5 00/43] CXl 2.0 emulation Support
2022-02-04 14:27 ` Michael S. Tsirkin
@ 2022-02-04 18:23 ` Jonathan Cameron
0 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-04 18:23 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: qemu-devel, Alex Bennée, Marcel Apfelbaum, Igor Mammedov,
linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
On Fri, 4 Feb 2022 09:27:08 -0500
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Fri, Feb 04, 2022 at 09:03:27AM -0500, Michael S. Tsirkin wrote:
> > On Wed, Feb 02, 2022 at 02:09:54PM +0000, Jonathan Cameron wrote:
> > > Changes since v4:
> > > https://lore.kernel.org/linux-cxl/20220124171705.10432-1-Jonathan.Cameron@huawei.com/
> > >
> > > Note documentation patch that Alex requested to follow.
> > > I don't want to delay getting this out as Alex mentioned possibly
> > > having time to continue reviewing in latter part of this week.
> > >
> > > Issues identified by CI / Alex Bennée
> > > - Stubs added for hw/cxl/cxl-host and hw/acpi/cxl plus related meson
> > > changes to use them as necessary.
> > > - Drop uid from cxl-test (result of last minute change in v4 that was not
> > > carried through to the test)
> > > - Fix naming clash with field name ERROR which on some arches is defined
> > > and results in the string being replaced with 0 in some of the
> > > register field related defines. Call it ERR instead.
> > > - Fix type issue around mr->size by using 64 bit acessor functions.
> > > - Add a new patch to exclude pxb-cxl from device-crash-test in similar
> > > fashion to pxb.
> > >
> > > CI tests now passing with exception of checkpatch which has what
> > > I think is a false positive and build-oss-fuzz which keeps timing out.
> > > https://gitlab.com/jic23/qemu/-/pipelines/460109208
> > > There were a few tweaks to patch descriptions after I pushed that
> > > out (I missed a few RB from Alex).
> >
> > There's an RFC patch that needs review from core memory maintainers,
> > so I guess not all of it is for merge just yet?
Yes, that patch definitely needs some input. It 'works' but feels
like a bit of a hack and raises questions around that is what else
I might be breaking or whether the approach is maintainable long
term.
> > Is there any way we can start applying this patchset gradually?
>
> For example, pick up patches 1-13 for now? They seem to be ready ...
That would be great! but...
*embarased cough* It doesn't boot at patch 13 (with a pxb-cxl device),
I missed that fixing the reset problem that Alex pointed out in v4
would result in calling into some infrastructure that isn't hooked up
until we implement the host bridge MMIO in patch 18. Fix is to
just move the reset implementation forwards to patch 18. In meantime, up to
patch 12 are fine.
The latter patches (after 13) are ordered in a less than ideal fashion.
To make it easier to take the rest gradually, they could (I think) be
reordered to give us
1) The device enablement (1-13 plus some later patches)
- type 3 device, pxb, root ports. Mostly this is about dragging
feature enablement earlier in the series.
Should be fine to pick this up in several smaller chunks.
2) Host enablement for the root bridges on x86 and ARM.
3) The RFC bit around how to enable memory interleave.
Until we advertise a fixed memory window there will be a missing
component anyway so the OS won't try to enable the interleaving.
Will result in a few additional patches because we'll update
the CEDT ACPI table tests in two steps rather than just once.
I'll have a go at the reorg next week and clearly highlight in the
cover letter which steps make sense in applying gradually.
+ hit those steps with proper testing and at least check
it boots after each patch :(
Thanks for taking a look and your advice on moving this
forwards.
Jonathan
>
> >
> > > Other changes (mostly from Alex's review)
> > > - Change component register handling to now report UNIMP and return 0
> > > for 8 byte registers as we currently don't implement any of them.
> > > Note that this means we need a kernel fix:
> > > https://lore.kernel.org/linux-cxl/20220201153437.2873-1-Jonathan.Cameron@huawei.com/
> > > - Drop majority of the macros used in defining mailbox handlers in
> > > favour of written out code.
> > > - Use REG64 where appropriate. This was introduced whilst this set
> > > has been underdevelopment so I missed it.
> > > - Clarify some register access options wrt to CXL 2.0 Errata F4.
> > > - Change timestamp to qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL)
> > > - Use typed enums to enforce types of function arguements.
> > > - Default to cxl being off in machine_class_init() removing
> > > need to set it to off in machines where there is no support as yet.
> > > - Add Alex's RB where given.
> > >
> > > Looking in particular for:
> > > * Review of the PCI interactions
> > > * x86 and ARM machine interactions (particularly the memory maps)
> > > * Review of the interleaving approach - is the basic idea
> > > acceptable?
> > > * Review of the command line interface.
> > > * CXL related review welcome but much of that got reviewed
> > > in earlier versions and hasn't changed substantially.
> > >
> > > Big TODOs:
> > >
> > > * Interleave boundary issues. I haven't yet solved this but didn't
> > > want to futher delay the review of the rest of the series.
> > >
> > > * Volatile memory devices (easy but it's more code so left for now).
> > > * Switch support. Linux kernel support is under review currently,
> > > so there is now something to test against.
> > > * Hotplug? May not need much but it's not tested yet!
> > > * More tests and tighter verification that values written to hardware
> > > are actually valid - stuff that real hardware would check.
> > > * Testing, testing and more testing. I have been running a basic
> > > set of ARM and x86 tests on this, but there is always room for
> > > more tests and greater automation.
> > > * CFMWS flags as requested by Ben.
> > >
> > > Why do we want QEMU emulation of CXL?
> > >
> > > As Ben stated in V3, QEMU support has been critical to getting OS
> > > software written given lack of availability of hardware supporting the
> > > latest CXL features (coupled with very high demand for support being
> > > ready in a timely fashion). What has become clear since Ben's v3
> > > is that situation is a continuous one. Whilst we can't talk about
> > > them yet, CXL 3.0 features and OS support have been prototyped on
> > > top of this support and a lot of the ongoing kernel work is being
> > > tested against these patches. The kernel CXL mocking code allows
> > > some forms of testing, but QEMU provides a more versatile and
> > > exensible platform.
> > >
> > > Other features on the qemu-list that build on these include PCI-DOE
> > > /CDAT support from the Avery Design team further showing how this
> > > code is useful. Whilst not directly related this is also the test
> > > platform for work on PCI IDE/CMA + related DMTF SPDM as CXL both
> > > utilizes and extends those technologies and is likely to be an early
> > > adopter.
> > > Refs:
> > > CMA Kernel: https://lore.kernel.org/all/20210804161839.3492053-1-Jonathan.Cameron@huawei.com/
> > > CMA Qemu: https://lore.kernel.org/qemu-devel/1624665723-5169-1-git-send-email-cbrowy@avery-design.com/
> > > DOE Qemu: https://lore.kernel.org/qemu-devel/1623329999-15662-1-git-send-email-cbrowy@avery-design.com/
> > >
> > > As can be seen there is non trivial interaction with other areas of
> > > Qemu, particularly PCI and keeping this set up to date is proving
> > > a burden we'd rather do without :)
> > >
> > > Ben mentioned a few other good reasons in v3:
> > > https://lore.kernel.org/qemu-devel/20210202005948.241655-1-ben.widawsky@intel.com/
> > >
> > > The evolution of this series perhaps leave it in a less than
> > > entirely obvious order and that may get tidied up in future postings.
> > > I'm also open to this being considered in bite sized chunks. What
> > > we have here is about what you need for it to be useful for testing
> > > currently kernel code. Note the kernel code is moving fast so
> > > since v4, some features have been introduced we don't yet support in
> > > QEMU (e.g. use of the PCIe serial number extended capability).
> > >
> > > All comments welcome.
> > >
> > > qemu-system-aarch64 -M virt,gic-version=3,cxl=on \
> > > -m 4g,maxmem=8G,slots=8 \
> > > ...
> > > -object memory-backend-file,id=cxl-mem1,share=on,mem-path=/tmp/cxltest.raw,size=256M \
> > > -object memory-backend-file,id=cxl-mem2,share=on,mem-path=/tmp/cxltest2.raw,size=256M \
> > > -object memory-backend-file,id=cxl-mem3,share=on,mem-path=/tmp/cxltest3.raw,size=256M \
> > > -object memory-backend-file,id=cxl-mem4,share=on,mem-path=/tmp/cxltest4.raw,size=256M \
> > > -object memory-backend-file,id=cxl-lsa1,share=on,mem-path=/tmp/lsa.raw,size=256M \
> > > -object memory-backend-file,id=cxl-lsa2,share=on,mem-path=/tmp/lsa2.raw,size=256M \
> > > -object memory-backend-file,id=cxl-lsa3,share=on,mem-path=/tmp/lsa3.raw,size=256M \
> > > -object memory-backend-file,id=cxl-lsa4,share=on,mem-path=/tmp/lsa4.raw,size=256M \
> > > -device pxb-cxl,bus_nr=12,bus=pcie.0,id=cxl.1 \
> > > -device pxb-cxl,bus_nr=222,bus=pcie.0,id=cxl.2 \
> > > -device cxl-rp,port=0,bus=cxl.1,id=root_port13,chassis=0,slot=2 \
> > > -device cxl-type3,bus=root_port13,memdev=cxl-mem1,lsa=cxl-lsa1,id=cxl-pmem0,size=256M \
> > > -device cxl-rp,port=1,bus=cxl.1,id=root_port14,chassis=0,slot=3 \
> > > -device cxl-type3,bus=root_port14,memdev=cxl-mem2,lsa=cxl-lsa2,id=cxl-pmem1,size=256M \
> > > -device cxl-rp,port=0,bus=cxl.2,id=root_port15,chassis=0,slot=5 \
> > > -device cxl-type3,bus=root_port15,memdev=cxl-mem3,lsa=cxl-lsa3,id=cxl-pmem2,size=256M \
> > > -device cxl-rp,port=1,bus=cxl.2,id=root_port16,chassis=0,slot=6 \
> > > -device cxl-type3,bus=root_port16,memdev=cxl-mem4,lsa=cxl-lsa4,id=cxl-pmem3,size=256M \
> > > -cxl-fixed-memory-window targets=cxl.1,size=4G,interleave-granularity=8k \
> > > -cxl-fixed-memory-window targets=cxl.1,targets=cxl.2,size=4G,interleave-granularity=8k
> > >
> > > First CFMWS suitable for up to 2 way interleave, the second for 4 way (2 way
> > > at host level and 2 way at the host bridge).
> > > targets=<range of pxb-cxl uids> , multiple entries if range is disjoint.
> > >
> > > With the v5.17-rc1 + patch series listed below.
> > >
> > > cd /sys/bus/cxl/devices/
> > > region=$(cat decoder0.1/create_region)
> > > echo $region > decoder0.1/create_region
> > > ls -lh
> > >
> > > //Note the order of devices and adjust the following to make sure they
> > > //are in order across the 4 root ports. Easy to do in a tool, but
> > > //not easy to paste in a cover letter.
> > >
> > > cd region0.1\:0
> > > echo 4 > interleave_ways
> > > echo mem2 > target0
> > > echo mem3 > target1
> > > echo mem0 > target2
> > > echo mem1 > target3
> > > echo $((1024<<20)) > size
> > > echo 4096 > interleave_granularity
> > > echo region0.1:0 > /sys/bus/cxl/drivers/cxl_region/bind
> > >
> > > Tested with devmem2 and files with known content.
> > > Kernel tree is mainline + (I based on 5.17-rc1)
> > > [PATCH] cxl/regs: Fix size of CXL Capabilty Header Register
> > > https://lore.kernel.org/linux-cxl/20220201182934.jjvavjsf4h7oqngv@intel.com/T/#t
> > >
> > > [PATCH v3 00/40] CXL.mem Topology Discovery and Hotplug Support
> > > https://lore.kernel.org/linux-cxl/164298411792.3018233.7493009997525360044.stgit@dwillia2-desk3.amr.corp.intel.com/
> > > Note that series has a lot of v4/v5 patches are replies but b4 does
> > > a good job of pulling out the latest.
> > >
> > > [PATCH 0/2] cxl/port: Robustness fixes for decoder enumeration
> > > https://lore.kernel.org/linux-cxl/164317463887.3438644.4087819721493502301.stgit@dwillia2-desk3.amr.corp.intel.com/
> > >
> > > [PATCH 0/4] Unify meaning of interleave attributes
> > > https://lore.kernel.org/linux-cxl/20220127212911.127741-1-ben.widawsky@intel.com/
> > >
> > > [PATCH v3 00/14] CXL Region driver
> > > https://lore.kernel.org/linux-cxl/20220128002707.391076-1-ben.widawsky@intel.com/
> > >
> > > What follows is a first attempt at explaining how all these components
> > > fit together. I'll write up some formal documentation shortly.
> > >
> > > Memory Address Map for CXL elements. Note where exactly these regions
> > > appear is Arch and platform dependent.
> > >
> > > Base somewhere far up in the Host PA map.
> > > _______________________________
> > > | |
> > > | CXL Host Bridge 0 Registers |
> > > | CXL Host Bridge 1 Registers |
> > > | ... | This bit is normal MMIO register space.
> > > | CXL Host bridge N registers | including programmable interleave decoders
> > > |______________________________| for interleave across root ports.
> > > | |
> > > ....
> > > | |
> > > |______________________________|
> > > | |
> > > | CFMW 0, | Note that there can be multiple regions
> > > | Interleave 2 way, targets | of memory within this 1TB which can be
> > > | Hostbridge 0, Hostbridge 1 | interleaved differently: in the host bridges
> > > | Granularity 16KiB, 1TB | across root ports or in switches below the root.
> > > |______________________________| ports
> > > | |
> > > | CFMW 1, |
> > > | Interleave 1 way, target |
> > > | Hostbridge 0, 512GiB |
> > > |______________________________|
> > > etc for all interleave combinations
> > > configured, or built in to the
> > > system before any generic software
> > > sees it.
> > >
> > > System Topology considering CFMW 0 only to keep this simple.
> > > x marks the match in each decoder level.
> > > Switches have more interleave decoders and other features
> > > that we haven't implemented yet in QEMU.
> > >
> > > Address Read to CFMW0 base + N
> > > _________________|________________
> > > | |
> > > | Host interconnect |
> > > | Configured to route CFM |
> > > | memory access to particular HB |
> > > |_____x____________________________|
> > > | |
> > > Interleave Decoder |
> > > Matches this HB |
> > > | |
> > > _______|__________ _____|____________
> > > | | | |
> > > | CXL HB 0 | | CXL HB 1 | Only exist in PCI (mostly)
> > > | HB IntLv Decoder | | HB IntLv Decoder | via ACPI description
> > > | PCI Root Bus 0c | | PCI Root Bus 0d |
> > > |x_________________| |__________________| In CXL have MMIO
> > > | | | | at location given in CEDT
> > > | | | | CHBS entry (ACPI)
> > > ____________|___ __________|__ __|_________ ___|_________
> > > | Root Port 0 | | Root Port 1 | | Root Port 2| | Root Port 3 |
> > > | Appears in | | Appears in | | Appears in | | Appear in |
> > > | PCI topology | | PCI Topology| | PCI Topo | | PCI Topo |
> > > | As 0c:00.0 | | as 0c:01.0 | | as de:00.0 | | as de:01.0 |
> > > |_______________| |_____________| |____________| |_____________|
> > > | | | |
> > > | | | |
> > > _____|_________ ______|______ ______|_____ ______|_______
> > > | x | | | | | | |
> > > | CXL Type3 0 | | CXL Type3 1 | | CXL type3 2| | CLX Type 3 3 |
> > > | | | | | | | |
> > > | PMEM0(Vol LSA)| | PMEM1 (...) | | PMEM2 (...)| | PMEM3 (...) |
> > > | Decoder to go | | | | | | |
> > > | from host PA | | PCI 0e:00.0 | | PCI df:00.0| | PCI e0:00.0 |
> > > | to device PA | | | | | | |
> > > | PCI as 0d:00.0| | | | | | |
> > > |_______________| |_____________| |____________| |______________|
> > >
> > > Backed by Backed by Backed by Backed by
> > > file 0 file 1 file 2 file 3
> > >
> > > LSA backed by additional files for each device (not yet supported)
> > >
> > > So currently we have decoders as follows for each interleaved access.
> > > 1) CFMW decoder - fixed config so forms part of qemu command line.
> > > 2) Host bridge decoders - programmable decoders that the system
> > > software will program either based on user command or based
> > > on info from the Label Storage Area (not yet emulated)
> > > 3) Type 3 device decoders. Down to here the address used is the
> > > Host PA. These decoders convert to the local device PA
> > > (in simple case - drop some bits in the middle of the address)
> > >
> > > Future patches will add decoders in switch upstream ports making
> > > the above diagram have another layer between root ports and
> > > the memory devices.
> > >
> > > Note, we've focused for now on Persistent Memory devices as they are seen
> > > as an early and important usecase (and are the most complex one).
> > > But it should be straight forward to add volatile memory
> > > support and indeed that would be backed by RAM.
> > >
> > > lspci -tv for above shows
> > >
> > > -+-[0000:00]-+-00.0 Red Hat, Inc. QEMU PCIe Host Bridge (this is the cxl PXB)f
> > > | \-OTHER STUFF
> > > +-[0000:0c]-+-00.0-[0d]----00.0 Intel Corporation Device 0d93
> > > | \-01.0-[0e]----00.0 Intel Corporation Device 0d93
> > > \-[0000:de]-+-00.0-[df]----00.0 Intel Corporation Device 0d93
> > > \-01.0-[e0]----00.0 Intel Corporation Device 0d93
> > >
> > > Where those Intel parts are the type 3 devices.
> > >
> > > All comments welcome!
> > >
> > > Particular thanks to Alex Bennée for his review of v4.
> > >
> > > Thanks,
> > >
> > > Jonathan
> > >
> > > Ben Widawsky (26):
> > > hw/pci/cxl: Add a CXL component type (interface)
> > > hw/cxl/component: Introduce CXL components (8.1.x, 8.2.5)
> > > hw/cxl/device: Introduce a CXL device (8.2.8)
> > > hw/cxl/device: Implement the CAP array (8.2.8.1-2)
> > > hw/cxl/device: Implement basic mailbox (8.2.8.4)
> > > hw/cxl/device: Add memory device utilities
> > > hw/cxl/device: Add cheap EVENTS implementation (8.2.9.1)
> > > hw/cxl/device: Timestamp implementation (8.2.9.3)
> > > hw/cxl/device: Add log commands (8.2.9.4) + CEL
> > > hw/pxb: Use a type for realizing expanders
> > > hw/pci/cxl: Create a CXL bus type
> > > hw/pxb: Allow creation of a CXL PXB (host bridge)
> > > acpi/pci: Consolidate host bridge setup
> > > hw/cxl/component: Implement host bridge MMIO (8.2.5, table 142)
> > > hw/cxl/rp: Add a root port
> > > hw/cxl/device: Add a memory device (8.2.8.5)
> > > hw/cxl/device: Implement MMIO HDM decoding (8.2.5.12)
> > > acpi/cxl: Add _OSC implementation (9.14.2)
> > > tests/acpi: allow CEDT table addition
> > > acpi/cxl: Create the CEDT (9.14.1)
> > > hw/cxl/device: Add some trivial commands
> > > hw/cxl/device: Plumb real Label Storage Area (LSA) sizing
> > > hw/cxl/device: Implement get/set Label Storage Area (LSA)
> > > acpi/cxl: Introduce CFMWS structures in CEDT
> > > hw/cxl/component Add a dumb HDM decoder handler
> > > qtest/cxl: Add very basic sanity tests
> > >
> > > Jonathan Cameron (17):
> > > MAINTAINERS: Add entry for Compute Express Link Emulation
> > > tests/acpi: allow DSDT.viot table changes.
> > > tests/acpi: Add update DSDT.viot
> > > cxl: Machine level control on whether CXL support is enabled
> > > hw/cxl/component: Add utils for interleave parameter encoding/decoding
> > > hw/cxl/host: Add support for CXL Fixed Memory Windows.
> > > hw/pci-host/gpex-acpi: Add support for dsdt construction for pxb-cxl
> > > pci/pcie_port: Add pci_find_port_by_pn()
> > > CXL/cxl_component: Add cxl_get_hb_cstate()
> > > mem/cxl_type3: Add read and write functions for associated hostmem.
> > > cxl/cxl-host: Add memops for CFMWS region.
> > > arm/virt: Allow virt/CEDT creation
> > > hw/arm/virt: Basic CXL enablement on pci_expander_bridge instances
> > > pxb-cxl
> > > RFC: softmmu/memory: Add ops to memory_region_ram_init_from_file
> > > i386/pc: Enable CXL fixed memory windows
> > > qtest/acpi: Add reference CEDT tables.
> > > scripts/device-crash-test: Add exception for pxb-cxl
> > >
> > > MAINTAINERS | 7 +
> > > hw/Kconfig | 1 +
> > > hw/acpi/Kconfig | 5 +
> > > hw/acpi/cxl-stub.c | 12 +
> > > hw/acpi/cxl.c | 231 +++++++++++++
> > > hw/acpi/meson.build | 4 +-
> > > hw/arm/Kconfig | 1 +
> > > hw/arm/virt-acpi-build.c | 30 ++
> > > hw/arm/virt.c | 40 ++-
> > > hw/core/machine.c | 28 ++
> > > hw/cxl/Kconfig | 3 +
> > > hw/cxl/cxl-component-utils.c | 284 ++++++++++++++++
> > > hw/cxl/cxl-device-utils.c | 271 ++++++++++++++++
> > > hw/cxl/cxl-host-stubs.c | 22 ++
> > > hw/cxl/cxl-host.c | 263 +++++++++++++++
> > > hw/cxl/cxl-mailbox-utils.c | 483 ++++++++++++++++++++++++++++
> > > hw/cxl/meson.build | 12 +
> > > hw/i386/acpi-build.c | 98 ++++--
> > > hw/i386/pc.c | 57 +++-
> > > hw/mem/Kconfig | 5 +
> > > hw/mem/cxl_type3.c | 353 ++++++++++++++++++++
> > > hw/mem/meson.build | 1 +
> > > hw/meson.build | 1 +
> > > hw/pci-bridge/Kconfig | 5 +
> > > hw/pci-bridge/cxl_root_port.c | 231 +++++++++++++
> > > hw/pci-bridge/meson.build | 1 +
> > > hw/pci-bridge/pci_expander_bridge.c | 171 +++++++++-
> > > hw/pci-bridge/pcie_root_port.c | 6 +-
> > > hw/pci-host/gpex-acpi.c | 22 +-
> > > hw/pci/pci.c | 21 +-
> > > hw/pci/pcie_port.c | 25 ++
> > > include/hw/acpi/cxl.h | 28 ++
> > > include/hw/arm/virt.h | 1 +
> > > include/hw/boards.h | 2 +
> > > include/hw/cxl/cxl.h | 51 +++
> > > include/hw/cxl/cxl_component.h | 206 ++++++++++++
> > > include/hw/cxl/cxl_device.h | 272 ++++++++++++++++
> > > include/hw/cxl/cxl_pci.h | 160 +++++++++
> > > include/hw/pci/pci.h | 14 +
> > > include/hw/pci/pci_bridge.h | 20 ++
> > > include/hw/pci/pci_bus.h | 7 +
> > > include/hw/pci/pci_ids.h | 1 +
> > > include/hw/pci/pcie_port.h | 2 +
> > > qapi/machine.json | 15 +
> > > qemu-options.hx | 37 +++
> > > scripts/device-crash-test | 1 +
> > > softmmu/memory.c | 9 +
> > > softmmu/vl.c | 11 +
> > > tests/data/acpi/pc/CEDT | Bin 0 -> 36 bytes
> > > tests/data/acpi/q35/CEDT | Bin 0 -> 36 bytes
> > > tests/data/acpi/q35/DSDT.viot | Bin 9398 -> 9416 bytes
> > > tests/data/acpi/virt/CEDT | Bin 0 -> 36 bytes
> > > tests/qtest/cxl-test.c | 151 +++++++++
> > > tests/qtest/meson.build | 4 +
> > > 54 files changed, 3645 insertions(+), 41 deletions(-)
> > > create mode 100644 hw/acpi/cxl-stub.c
> > > create mode 100644 hw/acpi/cxl.c
> > > create mode 100644 hw/cxl/Kconfig
> > > create mode 100644 hw/cxl/cxl-component-utils.c
> > > create mode 100644 hw/cxl/cxl-device-utils.c
> > > create mode 100644 hw/cxl/cxl-host-stubs.c
> > > create mode 100644 hw/cxl/cxl-host.c
> > > create mode 100644 hw/cxl/cxl-mailbox-utils.c
> > > create mode 100644 hw/cxl/meson.build
> > > create mode 100644 hw/mem/cxl_type3.c
> > > create mode 100644 hw/pci-bridge/cxl_root_port.c
> > > create mode 100644 include/hw/acpi/cxl.h
> > > create mode 100644 include/hw/cxl/cxl.h
> > > create mode 100644 include/hw/cxl/cxl_component.h
> > > create mode 100644 include/hw/cxl/cxl_device.h
> > > create mode 100644 include/hw/cxl/cxl_pci.h
> > > create mode 100644 tests/data/acpi/pc/CEDT
> > > create mode 100644 tests/data/acpi/q35/CEDT
> > > create mode 100644 tests/data/acpi/virt/CEDT
> > > create mode 100644 tests/qtest/cxl-test.c
> > >
> > > --
> > > 2.32.0
>
^ permalink raw reply [flat|nested] 54+ messages in thread
* Re: [PATCH v5 00/43] CXl 2.0 emulation Support
2022-02-02 14:09 [PATCH v5 00/43] CXl 2.0 emulation Support Jonathan Cameron
` (43 preceding siblings ...)
2022-02-04 14:03 ` [PATCH v5 00/43] CXl 2.0 emulation Support Michael S. Tsirkin
@ 2022-02-07 14:20 ` Jonathan Cameron
44 siblings, 0 replies; 54+ messages in thread
From: Jonathan Cameron @ 2022-02-07 14:20 UTC (permalink / raw)
To: qemu-devel, Alex Bennée, Marcel Apfelbaum,
Michael S . Tsirkin, Igor Mammedov
Cc: linux-cxl, Ben Widawsky, Peter Maydell, linuxarm,
Shameerali Kolothum Thodi, Philippe Mathieu-Daudé,
Saransh Gupta1, Shreyas Shah, Chris Browy, Samarth Saxena,
Dan Williams
On Wed, 2 Feb 2022 14:09:54 +0000
Jonathan Cameron <Jonathan.Cameron@huawei.com> wrote:
> Changes since v4:
> https://lore.kernel.org/linux-cxl/20220124171705.10432-1-Jonathan.Cameron@huawei.com/
>
> Note documentation patch that Alex requested to follow.
> I don't want to delay getting this out as Alex mentioned possibly
> having time to continue reviewing in latter part of this week.
>
> Issues identified by CI / Alex Bennée
> - Stubs added for hw/cxl/cxl-host and hw/acpi/cxl plus related meson
> changes to use them as necessary.
> - Drop uid from cxl-test (result of last minute change in v4 that was not
> carried through to the test)
> - Fix naming clash with field name ERROR which on some arches is defined
> and results in the string being replaced with 0 in some of the
> register field related defines. Call it ERR instead.
> - Fix type issue around mr->size by using 64 bit acessor functions.
> - Add a new patch to exclude pxb-cxl from device-crash-test in similar
> fashion to pxb.
>
> CI tests now passing with exception of checkpatch which has what
> I think is a false positive and build-oss-fuzz which keeps timing out.
> https://gitlab.com/jic23/qemu/-/pipelines/460109208
> There were a few tweaks to patch descriptions after I pushed that
> out (I missed a few RB from Alex).
>
> Other changes (mostly from Alex's review)
> - Change component register handling to now report UNIMP and return 0
> for 8 byte registers as we currently don't implement any of them.
> Note that this means we need a kernel fix:
> https://lore.kernel.org/linux-cxl/20220201153437.2873-1-Jonathan.Cameron@huawei.com/
> - Drop majority of the macros used in defining mailbox handlers in
> favour of written out code.
> - Use REG64 where appropriate. This was introduced whilst this set
> has been underdevelopment so I missed it.
> - Clarify some register access options wrt to CXL 2.0 Errata F4.
> - Change timestamp to qemu_clock_get_ns(QEMU_CLOCK_VIRTUAL)
> - Use typed enums to enforce types of function arguements.
> - Default to cxl being off in machine_class_init() removing
> need to set it to off in machines where there is no support as yet.
> - Add Alex's RB where given.
>
> Looking in particular for:
> * Review of the PCI interactions
> * x86 and ARM machine interactions (particularly the memory maps)
> * Review of the interleaving approach - is the basic idea
> acceptable?
> * Review of the command line interface.
> * CXL related review welcome but much of that got reviewed
> in earlier versions and hasn't changed substantially.
>
> Big TODOs:
>
> * Interleave boundary issues. I haven't yet solved this but didn't
> want to futher delay the review of the rest of the series.
So... After fixing my test, it became clear that Qemu won't issue
unaligned memory accesses to device unless mr->ram == true.
We can't set that for a CXL Fixed Memory Window (CFMW) as we have only an
indirect association with the CXL type3 memory devices and their
backing RAM. The interleave decoding has to sit in between.
So it 'kind of' works without any special handling as QEMU splits the
accesses into two anyway.
I don't yet understand fully the implications of this and whether
it in any real way restricts what can be done with the interleaved
memory under a CXL fixed memory region. Would definitely appreciate
inputs on this aspect.
The really short background story is:
1) Host PA memory region (CFMW) to which expectation is any access that
would be fine to normal DDR/Ram or NVDIMMs should work as long
as appropriate CXL topology and decoder configuration has been done
to get the memory accesses to actual memory.
2) The actual accesses to PAs in that region are interleaved
via several decoders on path to memory - min granularity is 256
bytes so any given access can only end up hitting 1 or 2 devices.
3) Fun corner cases are unaligned access crossing the interleave
boundary.
Jonathan
^ permalink raw reply [flat|nested] 54+ messages in thread