* [PATCH v6 0/8] hw/cxl: RAS error emulation and injection
@ 2023-02-27 11:27 ` Jonathan Cameron via
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron via @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
v6: Thanks to Philippe Mathieu-Daudé
- Added 'Since' entries to qapi docs.
- Added error prints to stubs rather than doing nothing at all.
(these two comments will be applied to the Poison injeciton series as well)
- Picked up tags
Long discussion on whether there was a good way to make the qapi only
exist when CONFIG_CXL* was set. Conclusion (I think) was that
unfortunately there isn't a good way to do this so stubs are currently
the best option. Thanks to Marcus for some great background info on this.
Based on series "[PATCH v4 00/10] hw/cxl: CXL emulation cleanups and minor fixes for upstream"
Based on: Message-Id: 20230206172816.8201-1-Jonathan.Cameron@huawei.com
v3 cover letter.
CXL error reporting is complex. This series only covers the protocol
related errors reported via PCIe AER - Ira Weiny has posted support for
Event log based injection and I will post an update of Poison list injection
shortly. My proposal is to upstream this one first, followed by Ira's Event
Log series, then finally the Poison List handling. That is based on likely
order of Linux kernel support (the support for this type of error reporting
went in during the recent merge window, the others are still under review).
Note we may propose other non error related features in between!
In order to test the kernel support for RAS error handling, I previously
provided this series via gitlab, enabling David Jiang's kernel patches
to be tested.
Now that Linux kernel support is upstream, this series is proposing the
support for upstream inclusion in QEMU. Note that support for Multiple
Header Recording has been added to QEMU the meantime and a kernel
patch to use that feature sent out.
https://lore.kernel.org/linux-cxl/20230113154058.16227-1-Jonathan.Cameron@huawei.com/T/#t
There are two generic PCI AER precursor feature additions.
1) The PCI_ERR_UCOR_MASK register has not been implemented until now
and is necessary for correct emulation.
2) The routing for AER errors, via existing AER error injection, only
covered one of two paths given in the PCIe base specification,
unfortunately not the one used by the Linux kernel CXL support.
The use of MSI for the CXL root ports, both makes sense from the point
of view of how it may well be implemented, and works around the documented
lack of PCI interrupt routing in i386/q35. I have a hack that lets
us correctly route those interrupts but don't currently plan to post it.
The actual CXL error injection uses a new QMP interface as documented
in the final patch description. The existing AER error injection
internals are reused though it's HMP interface is not.
Injection via QMP:
{ "execute": "qmp_capabilities" }
...
{ "execute": "cxl-inject-uncorrectable-errors",
"arguments": {
"path": "/machine/peripheral/cxl-pmem0",
"errors": [
{
"type": "cache-address-parity",
"header": [ 3, 4]
},
{
"type": "cache-data-parity",
"header": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]
},
{
"type": "internal",
"header": [ 1, 2, 4]
}
]
}}
...
{ "execute": "cxl-inject-correctable-error",
"arguments": {
"path": "/machine/peripheral/cxl-pmem0",
"type": "physical"
} }
Jonathan Cameron (8):
hw/pci/aer: Implement PCI_ERR_UNCOR_MASK register
hw/pci/aer: Add missing routing for AER errors
hw/pci-bridge/cxl_root_port: Wire up AER
hw/pci-bridge/cxl_root_port: Wire up MSI
hw/mem/cxl-type3: Add AER extended capability
hw/cxl: Fix endian issues in CXL RAS capability defaults / masks
hw/pci/aer: Make PCIE AER error injection facility available for other
emulation to use.
hw/mem/cxl_type3: Add CXL RAS Error Injection Support.
hw/cxl/cxl-component-utils.c | 20 ++-
hw/mem/cxl_type3.c | 294 +++++++++++++++++++++++++++++++++
hw/mem/cxl_type3_stubs.c | 17 ++
hw/mem/meson.build | 2 +
hw/pci-bridge/cxl_root_port.c | 64 +++++++
hw/pci/pci-internal.h | 1 -
hw/pci/pcie_aer.c | 14 +-
include/hw/cxl/cxl_component.h | 26 +++
include/hw/cxl/cxl_device.h | 11 ++
include/hw/pci/pcie_aer.h | 1 +
include/hw/pci/pcie_regs.h | 3 +
qapi/cxl.json | 128 ++++++++++++++
qapi/meson.build | 1 +
qapi/qapi-schema.json | 1 +
14 files changed, 572 insertions(+), 11 deletions(-)
create mode 100644 hw/mem/cxl_type3_stubs.c
create mode 100644 qapi/cxl.json
--
2.37.2
^ permalink raw reply [flat|nested] 24+ messages in thread* [PATCH v6 1/8] hw/pci/aer: Implement PCI_ERR_UNCOR_MASK register
2023-02-27 11:27 ` Jonathan Cameron via
@ 2023-02-27 11:27 ` Jonathan Cameron via
-1 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
This register in AER should be both writeable and should
have a default value with a couple of the errors masked
including the Uncorrectable Internal Error used by CXL for
it's error reporting.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
hw/pci/pcie_aer.c | 4 ++++
include/hw/pci/pcie_regs.h | 3 +++
2 files changed, 7 insertions(+)
diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
index 9a19be44ae..909e027d99 100644
--- a/hw/pci/pcie_aer.c
+++ b/hw/pci/pcie_aer.c
@@ -112,6 +112,10 @@ int pcie_aer_init(PCIDevice *dev, uint8_t cap_ver, uint16_t offset,
pci_set_long(dev->w1cmask + offset + PCI_ERR_UNCOR_STATUS,
PCI_ERR_UNC_SUPPORTED);
+ pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK,
+ PCI_ERR_UNC_MASK_DEFAULT);
+ pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK,
+ PCI_ERR_UNC_SUPPORTED);
pci_set_long(dev->config + offset + PCI_ERR_UNCOR_SEVER,
PCI_ERR_UNC_SEVERITY_DEFAULT);
diff --git a/include/hw/pci/pcie_regs.h b/include/hw/pci/pcie_regs.h
index 963dc2e170..6ec4785448 100644
--- a/include/hw/pci/pcie_regs.h
+++ b/include/hw/pci/pcie_regs.h
@@ -155,6 +155,9 @@ typedef enum PCIExpLinkWidth {
PCI_ERR_UNC_ATOP_EBLOCKED | \
PCI_ERR_UNC_TLP_PRF_BLOCKED)
+#define PCI_ERR_UNC_MASK_DEFAULT (PCI_ERR_UNC_INTN | \
+ PCI_ERR_UNC_TLP_PRF_BLOCKED)
+
#define PCI_ERR_UNC_SEVERITY_DEFAULT (PCI_ERR_UNC_DLP | \
PCI_ERR_UNC_SDN | \
PCI_ERR_UNC_FCP | \
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v6 1/8] hw/pci/aer: Implement PCI_ERR_UNCOR_MASK register
@ 2023-02-27 11:27 ` Jonathan Cameron via
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron via @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
This register in AER should be both writeable and should
have a default value with a couple of the errors masked
including the Uncorrectable Internal Error used by CXL for
it's error reporting.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
hw/pci/pcie_aer.c | 4 ++++
include/hw/pci/pcie_regs.h | 3 +++
2 files changed, 7 insertions(+)
diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
index 9a19be44ae..909e027d99 100644
--- a/hw/pci/pcie_aer.c
+++ b/hw/pci/pcie_aer.c
@@ -112,6 +112,10 @@ int pcie_aer_init(PCIDevice *dev, uint8_t cap_ver, uint16_t offset,
pci_set_long(dev->w1cmask + offset + PCI_ERR_UNCOR_STATUS,
PCI_ERR_UNC_SUPPORTED);
+ pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK,
+ PCI_ERR_UNC_MASK_DEFAULT);
+ pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK,
+ PCI_ERR_UNC_SUPPORTED);
pci_set_long(dev->config + offset + PCI_ERR_UNCOR_SEVER,
PCI_ERR_UNC_SEVERITY_DEFAULT);
diff --git a/include/hw/pci/pcie_regs.h b/include/hw/pci/pcie_regs.h
index 963dc2e170..6ec4785448 100644
--- a/include/hw/pci/pcie_regs.h
+++ b/include/hw/pci/pcie_regs.h
@@ -155,6 +155,9 @@ typedef enum PCIExpLinkWidth {
PCI_ERR_UNC_ATOP_EBLOCKED | \
PCI_ERR_UNC_TLP_PRF_BLOCKED)
+#define PCI_ERR_UNC_MASK_DEFAULT (PCI_ERR_UNC_INTN | \
+ PCI_ERR_UNC_TLP_PRF_BLOCKED)
+
#define PCI_ERR_UNC_SEVERITY_DEFAULT (PCI_ERR_UNC_DLP | \
PCI_ERR_UNC_SDN | \
PCI_ERR_UNC_FCP | \
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v6 2/8] hw/pci/aer: Add missing routing for AER errors
2023-02-27 11:27 ` Jonathan Cameron via
@ 2023-02-27 11:27 ` Jonathan Cameron via
-1 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
PCIe r6.0 Figure 6-3 "Pseudo Logic Diagram for Selected Error Message Control
and Status Bits" includes a right hand branch under "All PCI Express devices"
that allows for messages to be generated or sent onwards without SERR#
being set as long as the appropriate per error class bit in the PCIe
Device Control Register is set.
Implement that branch thus enabling routing of ERR_COR, ERR_NONFATAL
and ERR_FATAL under OSes that set these bits appropriately (e.g. Linux)
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
hw/pci/pcie_aer.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
index 909e027d99..103667c368 100644
--- a/hw/pci/pcie_aer.c
+++ b/hw/pci/pcie_aer.c
@@ -192,8 +192,16 @@ static void pcie_aer_update_uncor_status(PCIDevice *dev)
static bool
pcie_aer_msg_alldev(PCIDevice *dev, const PCIEAERMsg *msg)
{
+ uint16_t devctl = pci_get_word(dev->config + dev->exp.exp_cap +
+ PCI_EXP_DEVCTL);
if (!(pcie_aer_msg_is_uncor(msg) &&
- (pci_get_word(dev->config + PCI_COMMAND) & PCI_COMMAND_SERR))) {
+ (pci_get_word(dev->config + PCI_COMMAND) & PCI_COMMAND_SERR)) &&
+ !((msg->severity == PCI_ERR_ROOT_CMD_NONFATAL_EN) &&
+ (devctl & PCI_EXP_DEVCTL_NFERE)) &&
+ !((msg->severity == PCI_ERR_ROOT_CMD_COR_EN) &&
+ (devctl & PCI_EXP_DEVCTL_CERE)) &&
+ !((msg->severity == PCI_ERR_ROOT_CMD_FATAL_EN) &&
+ (devctl & PCI_EXP_DEVCTL_FERE))) {
return false;
}
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v6 2/8] hw/pci/aer: Add missing routing for AER errors
@ 2023-02-27 11:27 ` Jonathan Cameron via
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron via @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
PCIe r6.0 Figure 6-3 "Pseudo Logic Diagram for Selected Error Message Control
and Status Bits" includes a right hand branch under "All PCI Express devices"
that allows for messages to be generated or sent onwards without SERR#
being set as long as the appropriate per error class bit in the PCIe
Device Control Register is set.
Implement that branch thus enabling routing of ERR_COR, ERR_NONFATAL
and ERR_FATAL under OSes that set these bits appropriately (e.g. Linux)
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
hw/pci/pcie_aer.c | 10 +++++++++-
1 file changed, 9 insertions(+), 1 deletion(-)
diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c
index 909e027d99..103667c368 100644
--- a/hw/pci/pcie_aer.c
+++ b/hw/pci/pcie_aer.c
@@ -192,8 +192,16 @@ static void pcie_aer_update_uncor_status(PCIDevice *dev)
static bool
pcie_aer_msg_alldev(PCIDevice *dev, const PCIEAERMsg *msg)
{
+ uint16_t devctl = pci_get_word(dev->config + dev->exp.exp_cap +
+ PCI_EXP_DEVCTL);
if (!(pcie_aer_msg_is_uncor(msg) &&
- (pci_get_word(dev->config + PCI_COMMAND) & PCI_COMMAND_SERR))) {
+ (pci_get_word(dev->config + PCI_COMMAND) & PCI_COMMAND_SERR)) &&
+ !((msg->severity == PCI_ERR_ROOT_CMD_NONFATAL_EN) &&
+ (devctl & PCI_EXP_DEVCTL_NFERE)) &&
+ !((msg->severity == PCI_ERR_ROOT_CMD_COR_EN) &&
+ (devctl & PCI_EXP_DEVCTL_CERE)) &&
+ !((msg->severity == PCI_ERR_ROOT_CMD_FATAL_EN) &&
+ (devctl & PCI_EXP_DEVCTL_FERE))) {
return false;
}
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v6 3/8] hw/pci-bridge/cxl_root_port: Wire up AER
2023-02-27 11:27 ` Jonathan Cameron via
@ 2023-02-27 11:27 ` Jonathan Cameron via
-1 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
We are missing necessary config write handling for AER emulation in
the CXL root port. Add it based on pcie_root_port.c
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
hw/pci-bridge/cxl_root_port.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/hw/pci-bridge/cxl_root_port.c b/hw/pci-bridge/cxl_root_port.c
index 6664783974..00195257f7 100644
--- a/hw/pci-bridge/cxl_root_port.c
+++ b/hw/pci-bridge/cxl_root_port.c
@@ -187,12 +187,15 @@ static void cxl_rp_write_config(PCIDevice *d, uint32_t address, uint32_t val,
int len)
{
uint16_t slt_ctl, slt_sta;
+ uint32_t root_cmd =
+ pci_get_long(d->config + d->exp.aer_cap + PCI_ERR_ROOT_COMMAND);
pcie_cap_slot_get(d, &slt_ctl, &slt_sta);
pci_bridge_write_config(d, address, val, len);
pcie_cap_flr_write_config(d, address, val, len);
pcie_cap_slot_write_config(d, slt_ctl, slt_sta, address, val, len);
pcie_aer_write_config(d, address, val, len);
+ pcie_aer_root_write_config(d, address, val, len, root_cmd);
cxl_rp_dvsec_write_config(d, address, val, len);
}
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v6 3/8] hw/pci-bridge/cxl_root_port: Wire up AER
@ 2023-02-27 11:27 ` Jonathan Cameron via
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron via @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
We are missing necessary config write handling for AER emulation in
the CXL root port. Add it based on pcie_root_port.c
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
hw/pci-bridge/cxl_root_port.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/hw/pci-bridge/cxl_root_port.c b/hw/pci-bridge/cxl_root_port.c
index 6664783974..00195257f7 100644
--- a/hw/pci-bridge/cxl_root_port.c
+++ b/hw/pci-bridge/cxl_root_port.c
@@ -187,12 +187,15 @@ static void cxl_rp_write_config(PCIDevice *d, uint32_t address, uint32_t val,
int len)
{
uint16_t slt_ctl, slt_sta;
+ uint32_t root_cmd =
+ pci_get_long(d->config + d->exp.aer_cap + PCI_ERR_ROOT_COMMAND);
pcie_cap_slot_get(d, &slt_ctl, &slt_sta);
pci_bridge_write_config(d, address, val, len);
pcie_cap_flr_write_config(d, address, val, len);
pcie_cap_slot_write_config(d, slt_ctl, slt_sta, address, val, len);
pcie_aer_write_config(d, address, val, len);
+ pcie_aer_root_write_config(d, address, val, len, root_cmd);
cxl_rp_dvsec_write_config(d, address, val, len);
}
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v6 4/8] hw/pci-bridge/cxl_root_port: Wire up MSI
2023-02-27 11:27 ` Jonathan Cameron via
@ 2023-02-27 11:27 ` Jonathan Cameron via
-1 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
Done to avoid fixing ACPI route description of traditional PCI interrupts on q35
and because we should probably move with the times anyway.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
hw/pci-bridge/cxl_root_port.c | 61 +++++++++++++++++++++++++++++++++++
1 file changed, 61 insertions(+)
diff --git a/hw/pci-bridge/cxl_root_port.c b/hw/pci-bridge/cxl_root_port.c
index 00195257f7..7dfd20aa67 100644
--- a/hw/pci-bridge/cxl_root_port.c
+++ b/hw/pci-bridge/cxl_root_port.c
@@ -22,6 +22,7 @@
#include "qemu/range.h"
#include "hw/pci/pci_bridge.h"
#include "hw/pci/pcie_port.h"
+#include "hw/pci/msi.h"
#include "hw/qdev-properties.h"
#include "hw/sysbus.h"
#include "qapi/error.h"
@@ -29,6 +30,10 @@
#define CXL_ROOT_PORT_DID 0x7075
+#define CXL_RP_MSI_OFFSET 0x60
+#define CXL_RP_MSI_SUPPORTED_FLAGS PCI_MSI_FLAGS_MASKBIT
+#define CXL_RP_MSI_NR_VECTOR 2
+
/* Copied from the gen root port which we derive */
#define GEN_PCIE_ROOT_PORT_AER_OFFSET 0x100
#define GEN_PCIE_ROOT_PORT_ACS_OFFSET \
@@ -47,6 +52,49 @@ typedef struct CXLRootPort {
#define TYPE_CXL_ROOT_PORT "cxl-rp"
DECLARE_INSTANCE_CHECKER(CXLRootPort, CXL_ROOT_PORT, TYPE_CXL_ROOT_PORT)
+/*
+ * If two MSI vector are allocated, Advanced Error Interrupt Message Number
+ * is 1. otherwise 0.
+ * 17.12.5.10 RPERRSTS, 32:27 bit Advanced Error Interrupt Message Number.
+ */
+static uint8_t cxl_rp_aer_vector(const PCIDevice *d)
+{
+ switch (msi_nr_vectors_allocated(d)) {
+ case 1:
+ return 0;
+ case 2:
+ return 1;
+ case 4:
+ case 8:
+ case 16:
+ case 32:
+ default:
+ break;
+ }
+ abort();
+ return 0;
+}
+
+static int cxl_rp_interrupts_init(PCIDevice *d, Error **errp)
+{
+ int rc;
+
+ rc = msi_init(d, CXL_RP_MSI_OFFSET, CXL_RP_MSI_NR_VECTOR,
+ CXL_RP_MSI_SUPPORTED_FLAGS & PCI_MSI_FLAGS_64BIT,
+ CXL_RP_MSI_SUPPORTED_FLAGS & PCI_MSI_FLAGS_MASKBIT,
+ errp);
+ if (rc < 0) {
+ assert(rc == -ENOTSUP);
+ }
+
+ return rc;
+}
+
+static void cxl_rp_interrupts_uninit(PCIDevice *d)
+{
+ msi_uninit(d);
+}
+
static void latch_registers(CXLRootPort *crp)
{
uint32_t *reg_state = crp->cxl_cstate.crb.cache_mem_registers;
@@ -183,6 +231,15 @@ static void cxl_rp_dvsec_write_config(PCIDevice *dev, uint32_t addr,
}
}
+static void cxl_rp_aer_vector_update(PCIDevice *d)
+{
+ PCIERootPortClass *rpc = PCIE_ROOT_PORT_GET_CLASS(d);
+
+ if (rpc->aer_vector) {
+ pcie_aer_root_set_vector(d, rpc->aer_vector(d));
+ }
+}
+
static void cxl_rp_write_config(PCIDevice *d, uint32_t address, uint32_t val,
int len)
{
@@ -192,6 +249,7 @@ static void cxl_rp_write_config(PCIDevice *d, uint32_t address, uint32_t val,
pcie_cap_slot_get(d, &slt_ctl, &slt_sta);
pci_bridge_write_config(d, address, val, len);
+ cxl_rp_aer_vector_update(d);
pcie_cap_flr_write_config(d, address, val, len);
pcie_cap_slot_write_config(d, slt_ctl, slt_sta, address, val, len);
pcie_aer_write_config(d, address, val, len);
@@ -220,6 +278,9 @@ static void cxl_root_port_class_init(ObjectClass *oc, void *data)
rpc->aer_offset = GEN_PCIE_ROOT_PORT_AER_OFFSET;
rpc->acs_offset = GEN_PCIE_ROOT_PORT_ACS_OFFSET;
+ rpc->aer_vector = cxl_rp_aer_vector;
+ rpc->interrupts_init = cxl_rp_interrupts_init;
+ rpc->interrupts_uninit = cxl_rp_interrupts_uninit;
dc->hotpluggable = false;
}
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v6 4/8] hw/pci-bridge/cxl_root_port: Wire up MSI
@ 2023-02-27 11:27 ` Jonathan Cameron via
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron via @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
Done to avoid fixing ACPI route description of traditional PCI interrupts on q35
and because we should probably move with the times anyway.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
hw/pci-bridge/cxl_root_port.c | 61 +++++++++++++++++++++++++++++++++++
1 file changed, 61 insertions(+)
diff --git a/hw/pci-bridge/cxl_root_port.c b/hw/pci-bridge/cxl_root_port.c
index 00195257f7..7dfd20aa67 100644
--- a/hw/pci-bridge/cxl_root_port.c
+++ b/hw/pci-bridge/cxl_root_port.c
@@ -22,6 +22,7 @@
#include "qemu/range.h"
#include "hw/pci/pci_bridge.h"
#include "hw/pci/pcie_port.h"
+#include "hw/pci/msi.h"
#include "hw/qdev-properties.h"
#include "hw/sysbus.h"
#include "qapi/error.h"
@@ -29,6 +30,10 @@
#define CXL_ROOT_PORT_DID 0x7075
+#define CXL_RP_MSI_OFFSET 0x60
+#define CXL_RP_MSI_SUPPORTED_FLAGS PCI_MSI_FLAGS_MASKBIT
+#define CXL_RP_MSI_NR_VECTOR 2
+
/* Copied from the gen root port which we derive */
#define GEN_PCIE_ROOT_PORT_AER_OFFSET 0x100
#define GEN_PCIE_ROOT_PORT_ACS_OFFSET \
@@ -47,6 +52,49 @@ typedef struct CXLRootPort {
#define TYPE_CXL_ROOT_PORT "cxl-rp"
DECLARE_INSTANCE_CHECKER(CXLRootPort, CXL_ROOT_PORT, TYPE_CXL_ROOT_PORT)
+/*
+ * If two MSI vector are allocated, Advanced Error Interrupt Message Number
+ * is 1. otherwise 0.
+ * 17.12.5.10 RPERRSTS, 32:27 bit Advanced Error Interrupt Message Number.
+ */
+static uint8_t cxl_rp_aer_vector(const PCIDevice *d)
+{
+ switch (msi_nr_vectors_allocated(d)) {
+ case 1:
+ return 0;
+ case 2:
+ return 1;
+ case 4:
+ case 8:
+ case 16:
+ case 32:
+ default:
+ break;
+ }
+ abort();
+ return 0;
+}
+
+static int cxl_rp_interrupts_init(PCIDevice *d, Error **errp)
+{
+ int rc;
+
+ rc = msi_init(d, CXL_RP_MSI_OFFSET, CXL_RP_MSI_NR_VECTOR,
+ CXL_RP_MSI_SUPPORTED_FLAGS & PCI_MSI_FLAGS_64BIT,
+ CXL_RP_MSI_SUPPORTED_FLAGS & PCI_MSI_FLAGS_MASKBIT,
+ errp);
+ if (rc < 0) {
+ assert(rc == -ENOTSUP);
+ }
+
+ return rc;
+}
+
+static void cxl_rp_interrupts_uninit(PCIDevice *d)
+{
+ msi_uninit(d);
+}
+
static void latch_registers(CXLRootPort *crp)
{
uint32_t *reg_state = crp->cxl_cstate.crb.cache_mem_registers;
@@ -183,6 +231,15 @@ static void cxl_rp_dvsec_write_config(PCIDevice *dev, uint32_t addr,
}
}
+static void cxl_rp_aer_vector_update(PCIDevice *d)
+{
+ PCIERootPortClass *rpc = PCIE_ROOT_PORT_GET_CLASS(d);
+
+ if (rpc->aer_vector) {
+ pcie_aer_root_set_vector(d, rpc->aer_vector(d));
+ }
+}
+
static void cxl_rp_write_config(PCIDevice *d, uint32_t address, uint32_t val,
int len)
{
@@ -192,6 +249,7 @@ static void cxl_rp_write_config(PCIDevice *d, uint32_t address, uint32_t val,
pcie_cap_slot_get(d, &slt_ctl, &slt_sta);
pci_bridge_write_config(d, address, val, len);
+ cxl_rp_aer_vector_update(d);
pcie_cap_flr_write_config(d, address, val, len);
pcie_cap_slot_write_config(d, slt_ctl, slt_sta, address, val, len);
pcie_aer_write_config(d, address, val, len);
@@ -220,6 +278,9 @@ static void cxl_root_port_class_init(ObjectClass *oc, void *data)
rpc->aer_offset = GEN_PCIE_ROOT_PORT_AER_OFFSET;
rpc->acs_offset = GEN_PCIE_ROOT_PORT_ACS_OFFSET;
+ rpc->aer_vector = cxl_rp_aer_vector;
+ rpc->interrupts_init = cxl_rp_interrupts_init;
+ rpc->interrupts_uninit = cxl_rp_interrupts_uninit;
dc->hotpluggable = false;
}
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v6 5/8] hw/mem/cxl-type3: Add AER extended capability
2023-02-27 11:27 ` Jonathan Cameron via
@ 2023-02-27 11:27 ` Jonathan Cameron via
-1 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
This enables AER error injection to function as expected.
It is intended as a building block in enabling CXL RAS error injection
in the following patches.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
hw/mem/cxl_type3.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index 217a5e639b..6cdd988d1d 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -250,6 +250,7 @@ static void ct3d_config_write(PCIDevice *pci_dev, uint32_t addr, uint32_t val,
pcie_doe_write_config(&ct3d->doe_cdat, addr, val, size);
pci_default_write_config(pci_dev, addr, val, size);
+ pcie_aer_write_config(pci_dev, addr, val, size);
}
/*
@@ -452,8 +453,19 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)
cxl_cstate->cdat.free_cdat_table = ct3_free_cdat_table;
cxl_cstate->cdat.private = ct3d;
cxl_doe_cdat_init(cxl_cstate, errp);
+
+ pcie_cap_deverr_init(pci_dev);
+ /* Leave a bit of room for expansion */
+ rc = pcie_aer_init(pci_dev, PCI_ERR_VER, 0x200, PCI_ERR_SIZEOF, NULL);
+ if (rc) {
+ goto err_release_cdat;
+ }
+
return;
+err_release_cdat:
+ cxl_doe_cdat_release(cxl_cstate);
+ g_free(regs->special_ops);
err_address_space_free:
address_space_destroy(&ct3d->hostmem_as);
return;
@@ -465,6 +477,7 @@ static void ct3_exit(PCIDevice *pci_dev)
CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
ComponentRegisters *regs = &cxl_cstate->crb;
+ pcie_aer_exit(pci_dev);
cxl_doe_cdat_release(cxl_cstate);
g_free(regs->special_ops);
address_space_destroy(&ct3d->hostmem_as);
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v6 5/8] hw/mem/cxl-type3: Add AER extended capability
@ 2023-02-27 11:27 ` Jonathan Cameron via
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron via @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
This enables AER error injection to function as expected.
It is intended as a building block in enabling CXL RAS error injection
in the following patches.
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
---
hw/mem/cxl_type3.c | 13 +++++++++++++
1 file changed, 13 insertions(+)
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index 217a5e639b..6cdd988d1d 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -250,6 +250,7 @@ static void ct3d_config_write(PCIDevice *pci_dev, uint32_t addr, uint32_t val,
pcie_doe_write_config(&ct3d->doe_cdat, addr, val, size);
pci_default_write_config(pci_dev, addr, val, size);
+ pcie_aer_write_config(pci_dev, addr, val, size);
}
/*
@@ -452,8 +453,19 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)
cxl_cstate->cdat.free_cdat_table = ct3_free_cdat_table;
cxl_cstate->cdat.private = ct3d;
cxl_doe_cdat_init(cxl_cstate, errp);
+
+ pcie_cap_deverr_init(pci_dev);
+ /* Leave a bit of room for expansion */
+ rc = pcie_aer_init(pci_dev, PCI_ERR_VER, 0x200, PCI_ERR_SIZEOF, NULL);
+ if (rc) {
+ goto err_release_cdat;
+ }
+
return;
+err_release_cdat:
+ cxl_doe_cdat_release(cxl_cstate);
+ g_free(regs->special_ops);
err_address_space_free:
address_space_destroy(&ct3d->hostmem_as);
return;
@@ -465,6 +477,7 @@ static void ct3_exit(PCIDevice *pci_dev)
CXLComponentState *cxl_cstate = &ct3d->cxl_cstate;
ComponentRegisters *regs = &cxl_cstate->crb;
+ pcie_aer_exit(pci_dev);
cxl_doe_cdat_release(cxl_cstate);
g_free(regs->special_ops);
address_space_destroy(&ct3d->hostmem_as);
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v6 6/8] hw/cxl: Fix endian issues in CXL RAS capability defaults / masks
2023-02-27 11:27 ` Jonathan Cameron via
@ 2023-02-27 11:27 ` Jonathan Cameron via
-1 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
As these are about to be modified, fix the endian handle for
this set of registers rather than making it worse.
Note that CXL is currently only supported in QEMU on
x86 (arm64 patches out of tree) so we aren't going to yet hit
an problems with big endian. However it is good to avoid making
things worse for that support in the future.
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/cxl/cxl-component-utils.c | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index 3edd303a33..737b4764b9 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -141,17 +141,17 @@ static void ras_init_common(uint32_t *reg_state, uint32_t *write_msk)
* Error status is RW1C but given bits are not yet set, it can
* be handled as RO.
*/
- reg_state[R_CXL_RAS_UNC_ERR_STATUS] = 0;
+ stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_STATUS, 0);
/* Bits 12-13 and 17-31 reserved in CXL 2.0 */
- reg_state[R_CXL_RAS_UNC_ERR_MASK] = 0x1cfff;
- write_msk[R_CXL_RAS_UNC_ERR_MASK] = 0x1cfff;
- reg_state[R_CXL_RAS_UNC_ERR_SEVERITY] = 0x1cfff;
- write_msk[R_CXL_RAS_UNC_ERR_SEVERITY] = 0x1cfff;
- reg_state[R_CXL_RAS_COR_ERR_STATUS] = 0;
- reg_state[R_CXL_RAS_COR_ERR_MASK] = 0x7f;
- write_msk[R_CXL_RAS_COR_ERR_MASK] = 0x7f;
+ stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
+ stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
+ stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
+ stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
+ stl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS, 0);
+ stl_le_p(reg_state + R_CXL_RAS_COR_ERR_MASK, 0x7f);
+ stl_le_p(write_msk + R_CXL_RAS_COR_ERR_MASK, 0x7f);
/* CXL switches and devices must set */
- reg_state[R_CXL_RAS_ERR_CAP_CTRL] = 0x00;
+ stl_le_p(reg_state + R_CXL_RAS_ERR_CAP_CTRL, 0x00);
}
static void hdm_init_common(uint32_t *reg_state, uint32_t *write_msk,
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v6 6/8] hw/cxl: Fix endian issues in CXL RAS capability defaults / masks
@ 2023-02-27 11:27 ` Jonathan Cameron via
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron via @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
As these are about to be modified, fix the endian handle for
this set of registers rather than making it worse.
Note that CXL is currently only supported in QEMU on
x86 (arm64 patches out of tree) so we aren't going to yet hit
an problems with big endian. However it is good to avoid making
things worse for that support in the future.
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/cxl/cxl-component-utils.c | 18 +++++++++---------
1 file changed, 9 insertions(+), 9 deletions(-)
diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index 3edd303a33..737b4764b9 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -141,17 +141,17 @@ static void ras_init_common(uint32_t *reg_state, uint32_t *write_msk)
* Error status is RW1C but given bits are not yet set, it can
* be handled as RO.
*/
- reg_state[R_CXL_RAS_UNC_ERR_STATUS] = 0;
+ stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_STATUS, 0);
/* Bits 12-13 and 17-31 reserved in CXL 2.0 */
- reg_state[R_CXL_RAS_UNC_ERR_MASK] = 0x1cfff;
- write_msk[R_CXL_RAS_UNC_ERR_MASK] = 0x1cfff;
- reg_state[R_CXL_RAS_UNC_ERR_SEVERITY] = 0x1cfff;
- write_msk[R_CXL_RAS_UNC_ERR_SEVERITY] = 0x1cfff;
- reg_state[R_CXL_RAS_COR_ERR_STATUS] = 0;
- reg_state[R_CXL_RAS_COR_ERR_MASK] = 0x7f;
- write_msk[R_CXL_RAS_COR_ERR_MASK] = 0x7f;
+ stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
+ stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
+ stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
+ stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
+ stl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS, 0);
+ stl_le_p(reg_state + R_CXL_RAS_COR_ERR_MASK, 0x7f);
+ stl_le_p(write_msk + R_CXL_RAS_COR_ERR_MASK, 0x7f);
/* CXL switches and devices must set */
- reg_state[R_CXL_RAS_ERR_CAP_CTRL] = 0x00;
+ stl_le_p(reg_state + R_CXL_RAS_ERR_CAP_CTRL, 0x00);
}
static void hdm_init_common(uint32_t *reg_state, uint32_t *write_msk,
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v6 7/8] hw/pci/aer: Make PCIE AER error injection facility available for other emulation to use.
2023-02-27 11:27 ` Jonathan Cameron via
@ 2023-02-27 11:27 ` Jonathan Cameron via
-1 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
This infrastructure will be reused for CXL RAS error injection
in patches that follow.
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/pci/pci-internal.h | 1 -
include/hw/pci/pcie_aer.h | 1 +
2 files changed, 1 insertion(+), 1 deletion(-)
diff --git a/hw/pci/pci-internal.h b/hw/pci/pci-internal.h
index 2ea356bdf5..a7d6d8a732 100644
--- a/hw/pci/pci-internal.h
+++ b/hw/pci/pci-internal.h
@@ -20,6 +20,5 @@ void pcibus_dev_print(Monitor *mon, DeviceState *dev, int indent);
int pcie_aer_parse_error_string(const char *error_name,
uint32_t *status, bool *correctable);
-int pcie_aer_inject_error(PCIDevice *dev, const PCIEAERErr *err);
#endif
diff --git a/include/hw/pci/pcie_aer.h b/include/hw/pci/pcie_aer.h
index 65e71d98fe..1234fdc4e2 100644
--- a/include/hw/pci/pcie_aer.h
+++ b/include/hw/pci/pcie_aer.h
@@ -100,4 +100,5 @@ void pcie_aer_root_write_config(PCIDevice *dev,
uint32_t addr, uint32_t val, int len,
uint32_t root_cmd_prev);
+int pcie_aer_inject_error(PCIDevice *dev, const PCIEAERErr *err);
#endif /* QEMU_PCIE_AER_H */
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v6 7/8] hw/pci/aer: Make PCIE AER error injection facility available for other emulation to use.
@ 2023-02-27 11:27 ` Jonathan Cameron via
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron via @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
This infrastructure will be reused for CXL RAS error injection
in patches that follow.
Reviewed-by: Dave Jiang <dave.jiang@intel.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
hw/pci/pci-internal.h | 1 -
include/hw/pci/pcie_aer.h | 1 +
2 files changed, 1 insertion(+), 1 deletion(-)
diff --git a/hw/pci/pci-internal.h b/hw/pci/pci-internal.h
index 2ea356bdf5..a7d6d8a732 100644
--- a/hw/pci/pci-internal.h
+++ b/hw/pci/pci-internal.h
@@ -20,6 +20,5 @@ void pcibus_dev_print(Monitor *mon, DeviceState *dev, int indent);
int pcie_aer_parse_error_string(const char *error_name,
uint32_t *status, bool *correctable);
-int pcie_aer_inject_error(PCIDevice *dev, const PCIEAERErr *err);
#endif
diff --git a/include/hw/pci/pcie_aer.h b/include/hw/pci/pcie_aer.h
index 65e71d98fe..1234fdc4e2 100644
--- a/include/hw/pci/pcie_aer.h
+++ b/include/hw/pci/pcie_aer.h
@@ -100,4 +100,5 @@ void pcie_aer_root_write_config(PCIDevice *dev,
uint32_t addr, uint32_t val, int len,
uint32_t root_cmd_prev);
+int pcie_aer_inject_error(PCIDevice *dev, const PCIEAERErr *err);
#endif /* QEMU_PCIE_AER_H */
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread
* [PATCH v6 8/8] hw/mem/cxl_type3: Add CXL RAS Error Injection Support.
2023-02-27 11:27 ` Jonathan Cameron via
@ 2023-02-27 11:27 ` Jonathan Cameron via
-1 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
CXL uses PCI AER Internal errors to signal to the host that an error has
occurred. The host can then read more detailed status from the CXL RAS
capability.
For uncorrectable errors: support multiple injection in one operation
as this is needed to reliably test multiple header logging support in an
OS. The equivalent feature doesn't exist for correctable errors, so only
one error need be injected at a time.
Note:
- Header content needs to be manually specified in a fashion that
matches the specification for what can be in the header for each
error type.
Injection via QMP:
{ "execute": "qmp_capabilities" }
...
{ "execute": "cxl-inject-uncorrectable-errors",
"arguments": {
"path": "/machine/peripheral/cxl-pmem0",
"errors": [
{
"type": "cache-address-parity",
"header": [ 3, 4]
},
{
"type": "cache-data-parity",
"header": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]
},
{
"type": "internal",
"header": [ 1, 2, 4]
}
]
}}
...
{ "execute": "cxl-inject-correctable-error",
"arguments": {
"path": "/machine/peripheral/cxl-pmem0",
"type": "physical"
} }
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
v6: (Thanks to Philippe Mathieu-Daudé)
- Add Since entries in cxl.json
- Add error prints in the stub functions so that if they are called without
CONFIG_CXL_MEM_DEVICE then we get a useful print rather than just silently
eating them.
---
hw/cxl/cxl-component-utils.c | 4 +-
hw/mem/cxl_type3.c | 281 +++++++++++++++++++++++++++++++++
hw/mem/cxl_type3_stubs.c | 17 ++
hw/mem/meson.build | 2 +
include/hw/cxl/cxl_component.h | 26 +++
include/hw/cxl/cxl_device.h | 11 ++
qapi/cxl.json | 128 +++++++++++++++
qapi/meson.build | 1 +
qapi/qapi-schema.json | 1 +
9 files changed, 470 insertions(+), 1 deletion(-)
diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index 737b4764b9..b665d4f565 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -142,16 +142,18 @@ static void ras_init_common(uint32_t *reg_state, uint32_t *write_msk)
* be handled as RO.
*/
stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_STATUS, 0);
+ stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_STATUS, 0x1cfff);
/* Bits 12-13 and 17-31 reserved in CXL 2.0 */
stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
stl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS, 0);
+ stl_le_p(write_msk + R_CXL_RAS_COR_ERR_STATUS, 0x7f);
stl_le_p(reg_state + R_CXL_RAS_COR_ERR_MASK, 0x7f);
stl_le_p(write_msk + R_CXL_RAS_COR_ERR_MASK, 0x7f);
/* CXL switches and devices must set */
- stl_le_p(reg_state + R_CXL_RAS_ERR_CAP_CTRL, 0x00);
+ stl_le_p(reg_state + R_CXL_RAS_ERR_CAP_CTRL, 0x200);
}
static void hdm_init_common(uint32_t *reg_state, uint32_t *write_msk,
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index 6cdd988d1d..abe60b362c 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -1,6 +1,7 @@
#include "qemu/osdep.h"
#include "qemu/units.h"
#include "qemu/error-report.h"
+#include "qapi/qapi-commands-cxl.h"
#include "hw/mem/memory-device.h"
#include "hw/mem/pc-dimm.h"
#include "hw/pci/pci.h"
@@ -323,6 +324,66 @@ static void hdm_decoder_commit(CXLType3Dev *ct3d, int which)
ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, COMMITTED, 1);
}
+static int ct3d_qmp_uncor_err_to_cxl(CxlUncorErrorType qmp_err)
+{
+ switch (qmp_err) {
+ case CXL_UNCOR_ERROR_TYPE_CACHE_DATA_PARITY:
+ return CXL_RAS_UNC_ERR_CACHE_DATA_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_CACHE_ADDRESS_PARITY:
+ return CXL_RAS_UNC_ERR_CACHE_ADDRESS_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_CACHE_BE_PARITY:
+ return CXL_RAS_UNC_ERR_CACHE_BE_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_CACHE_DATA_ECC:
+ return CXL_RAS_UNC_ERR_CACHE_DATA_ECC;
+ case CXL_UNCOR_ERROR_TYPE_MEM_DATA_PARITY:
+ return CXL_RAS_UNC_ERR_MEM_DATA_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_MEM_ADDRESS_PARITY:
+ return CXL_RAS_UNC_ERR_MEM_ADDRESS_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_MEM_BE_PARITY:
+ return CXL_RAS_UNC_ERR_MEM_BE_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_MEM_DATA_ECC:
+ return CXL_RAS_UNC_ERR_MEM_DATA_ECC;
+ case CXL_UNCOR_ERROR_TYPE_REINIT_THRESHOLD:
+ return CXL_RAS_UNC_ERR_REINIT_THRESHOLD;
+ case CXL_UNCOR_ERROR_TYPE_RSVD_ENCODING:
+ return CXL_RAS_UNC_ERR_RSVD_ENCODING;
+ case CXL_UNCOR_ERROR_TYPE_POISON_RECEIVED:
+ return CXL_RAS_UNC_ERR_POISON_RECEIVED;
+ case CXL_UNCOR_ERROR_TYPE_RECEIVER_OVERFLOW:
+ return CXL_RAS_UNC_ERR_RECEIVER_OVERFLOW;
+ case CXL_UNCOR_ERROR_TYPE_INTERNAL:
+ return CXL_RAS_UNC_ERR_INTERNAL;
+ case CXL_UNCOR_ERROR_TYPE_CXL_IDE_TX:
+ return CXL_RAS_UNC_ERR_CXL_IDE_TX;
+ case CXL_UNCOR_ERROR_TYPE_CXL_IDE_RX:
+ return CXL_RAS_UNC_ERR_CXL_IDE_RX;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int ct3d_qmp_cor_err_to_cxl(CxlCorErrorType qmp_err)
+{
+ switch (qmp_err) {
+ case CXL_COR_ERROR_TYPE_CACHE_DATA_ECC:
+ return CXL_RAS_COR_ERR_CACHE_DATA_ECC;
+ case CXL_COR_ERROR_TYPE_MEM_DATA_ECC:
+ return CXL_RAS_COR_ERR_MEM_DATA_ECC;
+ case CXL_COR_ERROR_TYPE_CRC_THRESHOLD:
+ return CXL_RAS_COR_ERR_CRC_THRESHOLD;
+ case CXL_COR_ERROR_TYPE_RETRY_THRESHOLD:
+ return CXL_RAS_COR_ERR_RETRY_THRESHOLD;
+ case CXL_COR_ERROR_TYPE_CACHE_POISON_RECEIVED:
+ return CXL_RAS_COR_ERR_CACHE_POISON_RECEIVED;
+ case CXL_COR_ERROR_TYPE_MEM_POISON_RECEIVED:
+ return CXL_RAS_COR_ERR_MEM_POISON_RECEIVED;
+ case CXL_COR_ERROR_TYPE_PHYSICAL:
+ return CXL_RAS_COR_ERR_PHYSICAL;
+ default:
+ return -EINVAL;
+ }
+}
+
static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value,
unsigned size)
{
@@ -341,6 +402,83 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value,
should_commit = FIELD_EX32(value, CXL_HDM_DECODER0_CTRL, COMMIT);
which_hdm = 0;
break;
+ case A_CXL_RAS_UNC_ERR_STATUS:
+ {
+ uint32_t capctrl = ldl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL);
+ uint32_t fe = FIELD_EX32(capctrl, CXL_RAS_ERR_CAP_CTRL, FIRST_ERROR_POINTER);
+ CXLError *cxl_err;
+ uint32_t unc_err;
+
+ /*
+ * If single bit written that corresponds to the first error
+ * pointer being cleared, update the status and header log.
+ */
+ if (!QTAILQ_EMPTY(&ct3d->error_list)) {
+ if ((1 << fe) ^ value) {
+ CXLError *cxl_next;
+ /*
+ * Software is using wrong flow for multiple header recording
+ * Following behavior in PCIe r6.0 and assuming multiple
+ * header support. Implementation defined choice to clear all
+ * matching records if more than one bit set - which corresponds
+ * closest to behavior of hardware not capable of multiple
+ * header recording.
+ */
+ QTAILQ_FOREACH_SAFE(cxl_err, &ct3d->error_list, node, cxl_next) {
+ if ((1 << cxl_err->type) & value) {
+ QTAILQ_REMOVE(&ct3d->error_list, cxl_err, node);
+ g_free(cxl_err);
+ }
+ }
+ } else {
+ /* Done with previous FE, so drop from list */
+ cxl_err = QTAILQ_FIRST(&ct3d->error_list);
+ QTAILQ_REMOVE(&ct3d->error_list, cxl_err, node);
+ g_free(cxl_err);
+ }
+
+ /*
+ * If there is another FE, then put that in place and update
+ * the header log
+ */
+ if (!QTAILQ_EMPTY(&ct3d->error_list)) {
+ uint32_t *header_log = &cache_mem[R_CXL_RAS_ERR_HEADER0];
+ int i;
+
+ cxl_err = QTAILQ_FIRST(&ct3d->error_list);
+ for (i = 0; i < CXL_RAS_ERR_HEADER_NUM; i++) {
+ stl_le_p(header_log + i, cxl_err->header[i]);
+ }
+ capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
+ FIRST_ERROR_POINTER, cxl_err->type);
+ } else {
+ /*
+ * If no more errors, then follow recomendation of PCI spec
+ * r6.0 6.2.4.2 to set the first error pointer to a status
+ * bit that will never be used.
+ */
+ capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
+ FIRST_ERROR_POINTER,
+ CXL_RAS_UNC_ERR_CXL_UNUSED);
+ }
+ stl_le_p((uint8_t *)cache_mem + A_CXL_RAS_ERR_CAP_CTRL, capctrl);
+ }
+ unc_err = 0;
+ QTAILQ_FOREACH(cxl_err, &ct3d->error_list, node) {
+ unc_err |= 1 << cxl_err->type;
+ }
+ stl_le_p((uint8_t *)cache_mem + offset, unc_err);
+
+ return;
+ }
+ case A_CXL_RAS_COR_ERR_STATUS:
+ {
+ uint32_t rw1c = value;
+ uint32_t temp = ldl_le_p((uint8_t *)cache_mem + offset);
+ temp &= ~rw1c;
+ stl_le_p((uint8_t *)cache_mem + offset, temp);
+ return;
+ }
default:
break;
}
@@ -404,6 +542,8 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)
unsigned short msix_num = 1;
int i, rc;
+ QTAILQ_INIT(&ct3d->error_list);
+
if (!cxl_setup_memory(ct3d, errp)) {
return;
}
@@ -631,6 +771,147 @@ static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size,
*/
}
+/* For uncorrectable errors include support for multiple header recording */
+void qmp_cxl_inject_uncorrectable_errors(const char *path,
+ CXLUncorErrorRecordList *errors,
+ Error **errp)
+{
+ Object *obj = object_resolve_path(path, NULL);
+ static PCIEAERErr err = {};
+ CXLType3Dev *ct3d;
+ CXLError *cxl_err;
+ uint32_t *reg_state;
+ uint32_t unc_err;
+ bool first;
+
+ if (!obj) {
+ error_setg(errp, "Unable to resolve path");
+ return;
+ }
+
+ if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) {
+ error_setg(errp, "Path does not point to a CXL type 3 device");
+ return;
+ }
+
+ err.status = PCI_ERR_UNC_INTN;
+ err.source_id = pci_requester_id(PCI_DEVICE(obj));
+ err.flags = 0;
+
+ ct3d = CXL_TYPE3(obj);
+
+ first = QTAILQ_EMPTY(&ct3d->error_list);
+ reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+ while (errors) {
+ uint32List *header = errors->value->header;
+ uint8_t header_count = 0;
+ int cxl_err_code;
+
+ cxl_err_code = ct3d_qmp_uncor_err_to_cxl(errors->value->type);
+ if (cxl_err_code < 0) {
+ error_setg(errp, "Unknown error code");
+ return;
+ }
+
+ /* If the error is masked, nothing to do here */
+ if (!((1 << cxl_err_code) &
+ ~ldl_le_p(reg_state + R_CXL_RAS_UNC_ERR_MASK))) {
+ errors = errors->next;
+ continue;
+ }
+
+ cxl_err = g_malloc0(sizeof(*cxl_err));
+ if (!cxl_err) {
+ return;
+ }
+
+ cxl_err->type = cxl_err_code;
+ while (header && header_count < 32) {
+ cxl_err->header[header_count++] = header->value;
+ header = header->next;
+ }
+ if (header_count > 32) {
+ error_setg(errp, "Header must be 32 DWORD or less");
+ return;
+ }
+ QTAILQ_INSERT_TAIL(&ct3d->error_list, cxl_err, node);
+
+ errors = errors->next;
+ }
+
+ if (first && !QTAILQ_EMPTY(&ct3d->error_list)) {
+ uint32_t *cache_mem = ct3d->cxl_cstate.crb.cache_mem_registers;
+ uint32_t capctrl = ldl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL);
+ uint32_t *header_log = &cache_mem[R_CXL_RAS_ERR_HEADER0];
+ int i;
+
+ cxl_err = QTAILQ_FIRST(&ct3d->error_list);
+ for (i = 0; i < CXL_RAS_ERR_HEADER_NUM; i++) {
+ stl_le_p(header_log + i, cxl_err->header[i]);
+ }
+
+ capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
+ FIRST_ERROR_POINTER, cxl_err->type);
+ stl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL, capctrl);
+ }
+
+ unc_err = 0;
+ QTAILQ_FOREACH(cxl_err, &ct3d->error_list, node) {
+ unc_err |= (1 << cxl_err->type);
+ }
+ if (!unc_err) {
+ return;
+ }
+
+ stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_STATUS, unc_err);
+ pcie_aer_inject_error(PCI_DEVICE(obj), &err);
+
+ return;
+}
+
+void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type,
+ Error **errp)
+{
+ static PCIEAERErr err = {};
+ Object *obj = object_resolve_path(path, NULL);
+ CXLType3Dev *ct3d;
+ uint32_t *reg_state;
+ uint32_t cor_err;
+ int cxl_err_type;
+
+ if (!obj) {
+ error_setg(errp, "Unable to resolve path");
+ return;
+ }
+ if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) {
+ error_setg(errp, "Path does not point to a CXL type 3 device");
+ return;
+ }
+
+ err.status = PCI_ERR_COR_INTERNAL;
+ err.source_id = pci_requester_id(PCI_DEVICE(obj));
+ err.flags = PCIE_AER_ERR_IS_CORRECTABLE;
+
+ ct3d = CXL_TYPE3(obj);
+ reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+ cor_err = ldl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS);
+
+ cxl_err_type = ct3d_qmp_cor_err_to_cxl(type);
+ if (cxl_err_type < 0) {
+ error_setg(errp, "Invalid COR error");
+ return;
+ }
+ /* If the error is masked, nothting to do here */
+ if (!((1 << cxl_err_type) & ~ldl_le_p(reg_state + R_CXL_RAS_COR_ERR_MASK))) {
+ return;
+ }
+
+ cor_err |= (1 << cxl_err_type);
+ stl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS, cor_err);
+
+ pcie_aer_inject_error(PCI_DEVICE(obj), &err);
+}
+
static void ct3_class_init(ObjectClass *oc, void *data)
{
DeviceClass *dc = DEVICE_CLASS(oc);
diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c
new file mode 100644
index 0000000000..d574c58f9a
--- /dev/null
+++ b/hw/mem/cxl_type3_stubs.c
@@ -0,0 +1,17 @@
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qapi/qapi-commands-cxl.h"
+
+void qmp_cxl_inject_uncorrectable_errors(const char *path,
+ CXLUncorErrorRecordList *errors,
+ Error **errp)
+{
+ error_setg(errp, "CXL Type 3 support is not compiled in");
+}
+
+void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type,
+ Error **errp)
+{
+ error_setg(errp, "CXL Type 3 support is not compiled in");
+}
diff --git a/hw/mem/meson.build b/hw/mem/meson.build
index 609b2b36fc..56c2618b84 100644
--- a/hw/mem/meson.build
+++ b/hw/mem/meson.build
@@ -4,6 +4,8 @@ mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
+softmmu_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_false: files('cxl_type3_stubs.c'))
+softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('cxl_type3_stubs.c'))
softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
index 692d7a5507..ec4203b83f 100644
--- a/include/hw/cxl/cxl_component.h
+++ b/include/hw/cxl/cxl_component.h
@@ -65,11 +65,37 @@ CXLx_CAPABILITY_HEADER(SNOOP, 0x14)
#define CXL_RAS_REGISTERS_OFFSET 0x80
#define CXL_RAS_REGISTERS_SIZE 0x58
REG32(CXL_RAS_UNC_ERR_STATUS, CXL_RAS_REGISTERS_OFFSET)
+#define CXL_RAS_UNC_ERR_CACHE_DATA_PARITY 0
+#define CXL_RAS_UNC_ERR_CACHE_ADDRESS_PARITY 1
+#define CXL_RAS_UNC_ERR_CACHE_BE_PARITY 2
+#define CXL_RAS_UNC_ERR_CACHE_DATA_ECC 3
+#define CXL_RAS_UNC_ERR_MEM_DATA_PARITY 4
+#define CXL_RAS_UNC_ERR_MEM_ADDRESS_PARITY 5
+#define CXL_RAS_UNC_ERR_MEM_BE_PARITY 6
+#define CXL_RAS_UNC_ERR_MEM_DATA_ECC 7
+#define CXL_RAS_UNC_ERR_REINIT_THRESHOLD 8
+#define CXL_RAS_UNC_ERR_RSVD_ENCODING 9
+#define CXL_RAS_UNC_ERR_POISON_RECEIVED 10
+#define CXL_RAS_UNC_ERR_RECEIVER_OVERFLOW 11
+#define CXL_RAS_UNC_ERR_INTERNAL 14
+#define CXL_RAS_UNC_ERR_CXL_IDE_TX 15
+#define CXL_RAS_UNC_ERR_CXL_IDE_RX 16
+#define CXL_RAS_UNC_ERR_CXL_UNUSED 63 /* Magic value */
REG32(CXL_RAS_UNC_ERR_MASK, CXL_RAS_REGISTERS_OFFSET + 0x4)
REG32(CXL_RAS_UNC_ERR_SEVERITY, CXL_RAS_REGISTERS_OFFSET + 0x8)
REG32(CXL_RAS_COR_ERR_STATUS, CXL_RAS_REGISTERS_OFFSET + 0xc)
+#define CXL_RAS_COR_ERR_CACHE_DATA_ECC 0
+#define CXL_RAS_COR_ERR_MEM_DATA_ECC 1
+#define CXL_RAS_COR_ERR_CRC_THRESHOLD 2
+#define CXL_RAS_COR_ERR_RETRY_THRESHOLD 3
+#define CXL_RAS_COR_ERR_CACHE_POISON_RECEIVED 4
+#define CXL_RAS_COR_ERR_MEM_POISON_RECEIVED 5
+#define CXL_RAS_COR_ERR_PHYSICAL 6
REG32(CXL_RAS_COR_ERR_MASK, CXL_RAS_REGISTERS_OFFSET + 0x10)
REG32(CXL_RAS_ERR_CAP_CTRL, CXL_RAS_REGISTERS_OFFSET + 0x14)
+ FIELD(CXL_RAS_ERR_CAP_CTRL, FIRST_ERROR_POINTER, 0, 6)
+REG32(CXL_RAS_ERR_HEADER0, CXL_RAS_REGISTERS_OFFSET + 0x18)
+#define CXL_RAS_ERR_HEADER_NUM 32
/* Offset 0x18 - 0x58 reserved for RAS logs */
/* 8.2.5.10 - CXL Security Capability Structure */
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index 7e5ad65c1d..d589f78202 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -232,6 +232,14 @@ REG64(CXL_MEM_DEV_STS, 0)
FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1)
FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3)
+typedef struct CXLError {
+ QTAILQ_ENTRY(CXLError) node;
+ int type; /* Error code as per FE definition */
+ uint32_t header[32];
+} CXLError;
+
+typedef QTAILQ_HEAD(, CXLError) CXLErrorList;
+
struct CXLType3Dev {
/* Private */
PCIDevice parent_obj;
@@ -248,6 +256,9 @@ struct CXLType3Dev {
/* DOE */
DOECap doe_cdat;
+
+ /* Error injection */
+ CXLErrorList error_list;
};
#define TYPE_CXL_TYPE3 "cxl-type3"
diff --git a/qapi/cxl.json b/qapi/cxl.json
new file mode 100644
index 0000000000..4be7d46041
--- /dev/null
+++ b/qapi/cxl.json
@@ -0,0 +1,128 @@
+# -*- Mode: Python -*-
+# vim: filetype=python
+
+##
+# = CXL devices
+##
+
+##
+# @CxlUncorErrorType:
+#
+# Type of uncorrectable CXL error to inject. These errors are reported via
+# an AER uncorrectable internal error with additional information logged at
+# the CXL device.
+#
+# @cache-data-parity: Data error such as data parity or data ECC error CXL.cache
+# @cache-address-parity: Address parity or other errors associated with the
+# address field on CXL.cache
+# @cache-be-parity: Byte enable parity or other byte enable errors on CXL.cache
+# @cache-data-ecc: ECC error on CXL.cache
+# @mem-data-parity: Data error such as data parity or data ECC error on CXL.mem
+# @mem-address-parity: Address parity or other errors associated with the
+# address field on CXL.mem
+# @mem-be-parity: Byte enable parity or other byte enable errors on CXL.mem.
+# @mem-data-ecc: Data ECC error on CXL.mem.
+# @reinit-threshold: REINIT threshold hit.
+# @rsvd-encoding: Received unrecognized encoding.
+# @poison-received: Received poison from the peer.
+# @receiver-overflow: Buffer overflows (first 3 bits of header log indicate which)
+# @internal: Component specific error
+# @cxl-ide-tx: Integrity and data encryption tx error.
+# @cxl-ide-rx: Integrity and data encryption rx error.
+#
+# Since: 8.0
+##
+
+{ 'enum': 'CxlUncorErrorType',
+ 'data': ['cache-data-parity',
+ 'cache-address-parity',
+ 'cache-be-parity',
+ 'cache-data-ecc',
+ 'mem-data-parity',
+ 'mem-address-parity',
+ 'mem-be-parity',
+ 'mem-data-ecc',
+ 'reinit-threshold',
+ 'rsvd-encoding',
+ 'poison-received',
+ 'receiver-overflow',
+ 'internal',
+ 'cxl-ide-tx',
+ 'cxl-ide-rx'
+ ]
+ }
+
+##
+# @CXLUncorErrorRecord:
+#
+# Record of a single error including header log.
+#
+# @type: Type of error
+# @header: 16 DWORD of header.
+#
+# Since: 8.0
+##
+{ 'struct': 'CXLUncorErrorRecord',
+ 'data': {
+ 'type': 'CxlUncorErrorType',
+ 'header': [ 'uint32' ]
+ }
+}
+
+##
+# @cxl-inject-uncorrectable-errors:
+#
+# Command to allow injection of multiple errors in one go. This allows testing
+# of multiple header log handling in the OS.
+#
+# @path: CXL Type 3 device canonical QOM path
+# @errors: Errors to inject
+#
+# Since: 8.0
+##
+{ 'command': 'cxl-inject-uncorrectable-errors',
+ 'data': { 'path': 'str',
+ 'errors': [ 'CXLUncorErrorRecord' ] }}
+
+##
+# @CxlCorErrorType:
+#
+# Type of CXL correctable error to inject
+#
+# @cache-data-ecc: Data ECC error on CXL.cache
+# @mem-data-ecc: Data ECC error on CXL.mem
+# @crc-threshold: Component specific and applicable to 68 byte Flit mode only.
+# @cache-poison-received: Received poison from a peer on CXL.cache.
+# @mem-poison-received: Received poison from a peer on CXL.mem
+# @physical: Received error indication from the physical layer.
+#
+# Since: 8.0
+##
+{ 'enum': 'CxlCorErrorType',
+ 'data': ['cache-data-ecc',
+ 'mem-data-ecc',
+ 'crc-threshold',
+ 'retry-threshold',
+ 'cache-poison-received',
+ 'mem-poison-received',
+ 'physical']
+}
+
+##
+# @cxl-inject-correctable-error:
+#
+# Command to inject a single correctable error. Multiple error injection
+# of this error type is not interesting as there is no associated header log.
+# These errors are reported via AER as a correctable internal error, with
+# additional detail available from the CXL device.
+#
+# @path: CXL Type 3 device canonical QOM path
+# @type: Type of error.
+#
+# Since: 8.0
+##
+{ 'command': 'cxl-inject-correctable-error',
+ 'data': { 'path': 'str',
+ 'type': 'CxlCorErrorType'
+ }
+}
diff --git a/qapi/meson.build b/qapi/meson.build
index fbdb442fdf..73c3c8c31a 100644
--- a/qapi/meson.build
+++ b/qapi/meson.build
@@ -31,6 +31,7 @@ qapi_all_modules = [
'compat',
'control',
'crypto',
+ 'cxl',
'dump',
'error',
'introspect',
diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json
index f000b90744..079f2a402a 100644
--- a/qapi/qapi-schema.json
+++ b/qapi/qapi-schema.json
@@ -95,3 +95,4 @@
{ 'include': 'pci.json' }
{ 'include': 'stats.json' }
{ 'include': 'virtio.json' }
+{ 'include': 'cxl.json' }
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* [PATCH v6 8/8] hw/mem/cxl_type3: Add CXL RAS Error Injection Support.
@ 2023-02-27 11:27 ` Jonathan Cameron via
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron via @ 2023-02-27 11:27 UTC (permalink / raw)
To: qemu-devel, Michael Tsirkin
Cc: Ben Widawsky, linux-cxl, linuxarm, Ira Weiny, Gregory Price,
Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth
CXL uses PCI AER Internal errors to signal to the host that an error has
occurred. The host can then read more detailed status from the CXL RAS
capability.
For uncorrectable errors: support multiple injection in one operation
as this is needed to reliably test multiple header logging support in an
OS. The equivalent feature doesn't exist for correctable errors, so only
one error need be injected at a time.
Note:
- Header content needs to be manually specified in a fashion that
matches the specification for what can be in the header for each
error type.
Injection via QMP:
{ "execute": "qmp_capabilities" }
...
{ "execute": "cxl-inject-uncorrectable-errors",
"arguments": {
"path": "/machine/peripheral/cxl-pmem0",
"errors": [
{
"type": "cache-address-parity",
"header": [ 3, 4]
},
{
"type": "cache-data-parity",
"header": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]
},
{
"type": "internal",
"header": [ 1, 2, 4]
}
]
}}
...
{ "execute": "cxl-inject-correctable-error",
"arguments": {
"path": "/machine/peripheral/cxl-pmem0",
"type": "physical"
} }
Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
---
v6: (Thanks to Philippe Mathieu-Daudé)
- Add Since entries in cxl.json
- Add error prints in the stub functions so that if they are called without
CONFIG_CXL_MEM_DEVICE then we get a useful print rather than just silently
eating them.
---
hw/cxl/cxl-component-utils.c | 4 +-
hw/mem/cxl_type3.c | 281 +++++++++++++++++++++++++++++++++
hw/mem/cxl_type3_stubs.c | 17 ++
hw/mem/meson.build | 2 +
include/hw/cxl/cxl_component.h | 26 +++
include/hw/cxl/cxl_device.h | 11 ++
qapi/cxl.json | 128 +++++++++++++++
qapi/meson.build | 1 +
qapi/qapi-schema.json | 1 +
9 files changed, 470 insertions(+), 1 deletion(-)
diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
index 737b4764b9..b665d4f565 100644
--- a/hw/cxl/cxl-component-utils.c
+++ b/hw/cxl/cxl-component-utils.c
@@ -142,16 +142,18 @@ static void ras_init_common(uint32_t *reg_state, uint32_t *write_msk)
* be handled as RO.
*/
stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_STATUS, 0);
+ stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_STATUS, 0x1cfff);
/* Bits 12-13 and 17-31 reserved in CXL 2.0 */
stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
stl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS, 0);
+ stl_le_p(write_msk + R_CXL_RAS_COR_ERR_STATUS, 0x7f);
stl_le_p(reg_state + R_CXL_RAS_COR_ERR_MASK, 0x7f);
stl_le_p(write_msk + R_CXL_RAS_COR_ERR_MASK, 0x7f);
/* CXL switches and devices must set */
- stl_le_p(reg_state + R_CXL_RAS_ERR_CAP_CTRL, 0x00);
+ stl_le_p(reg_state + R_CXL_RAS_ERR_CAP_CTRL, 0x200);
}
static void hdm_init_common(uint32_t *reg_state, uint32_t *write_msk,
diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
index 6cdd988d1d..abe60b362c 100644
--- a/hw/mem/cxl_type3.c
+++ b/hw/mem/cxl_type3.c
@@ -1,6 +1,7 @@
#include "qemu/osdep.h"
#include "qemu/units.h"
#include "qemu/error-report.h"
+#include "qapi/qapi-commands-cxl.h"
#include "hw/mem/memory-device.h"
#include "hw/mem/pc-dimm.h"
#include "hw/pci/pci.h"
@@ -323,6 +324,66 @@ static void hdm_decoder_commit(CXLType3Dev *ct3d, int which)
ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, COMMITTED, 1);
}
+static int ct3d_qmp_uncor_err_to_cxl(CxlUncorErrorType qmp_err)
+{
+ switch (qmp_err) {
+ case CXL_UNCOR_ERROR_TYPE_CACHE_DATA_PARITY:
+ return CXL_RAS_UNC_ERR_CACHE_DATA_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_CACHE_ADDRESS_PARITY:
+ return CXL_RAS_UNC_ERR_CACHE_ADDRESS_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_CACHE_BE_PARITY:
+ return CXL_RAS_UNC_ERR_CACHE_BE_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_CACHE_DATA_ECC:
+ return CXL_RAS_UNC_ERR_CACHE_DATA_ECC;
+ case CXL_UNCOR_ERROR_TYPE_MEM_DATA_PARITY:
+ return CXL_RAS_UNC_ERR_MEM_DATA_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_MEM_ADDRESS_PARITY:
+ return CXL_RAS_UNC_ERR_MEM_ADDRESS_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_MEM_BE_PARITY:
+ return CXL_RAS_UNC_ERR_MEM_BE_PARITY;
+ case CXL_UNCOR_ERROR_TYPE_MEM_DATA_ECC:
+ return CXL_RAS_UNC_ERR_MEM_DATA_ECC;
+ case CXL_UNCOR_ERROR_TYPE_REINIT_THRESHOLD:
+ return CXL_RAS_UNC_ERR_REINIT_THRESHOLD;
+ case CXL_UNCOR_ERROR_TYPE_RSVD_ENCODING:
+ return CXL_RAS_UNC_ERR_RSVD_ENCODING;
+ case CXL_UNCOR_ERROR_TYPE_POISON_RECEIVED:
+ return CXL_RAS_UNC_ERR_POISON_RECEIVED;
+ case CXL_UNCOR_ERROR_TYPE_RECEIVER_OVERFLOW:
+ return CXL_RAS_UNC_ERR_RECEIVER_OVERFLOW;
+ case CXL_UNCOR_ERROR_TYPE_INTERNAL:
+ return CXL_RAS_UNC_ERR_INTERNAL;
+ case CXL_UNCOR_ERROR_TYPE_CXL_IDE_TX:
+ return CXL_RAS_UNC_ERR_CXL_IDE_TX;
+ case CXL_UNCOR_ERROR_TYPE_CXL_IDE_RX:
+ return CXL_RAS_UNC_ERR_CXL_IDE_RX;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int ct3d_qmp_cor_err_to_cxl(CxlCorErrorType qmp_err)
+{
+ switch (qmp_err) {
+ case CXL_COR_ERROR_TYPE_CACHE_DATA_ECC:
+ return CXL_RAS_COR_ERR_CACHE_DATA_ECC;
+ case CXL_COR_ERROR_TYPE_MEM_DATA_ECC:
+ return CXL_RAS_COR_ERR_MEM_DATA_ECC;
+ case CXL_COR_ERROR_TYPE_CRC_THRESHOLD:
+ return CXL_RAS_COR_ERR_CRC_THRESHOLD;
+ case CXL_COR_ERROR_TYPE_RETRY_THRESHOLD:
+ return CXL_RAS_COR_ERR_RETRY_THRESHOLD;
+ case CXL_COR_ERROR_TYPE_CACHE_POISON_RECEIVED:
+ return CXL_RAS_COR_ERR_CACHE_POISON_RECEIVED;
+ case CXL_COR_ERROR_TYPE_MEM_POISON_RECEIVED:
+ return CXL_RAS_COR_ERR_MEM_POISON_RECEIVED;
+ case CXL_COR_ERROR_TYPE_PHYSICAL:
+ return CXL_RAS_COR_ERR_PHYSICAL;
+ default:
+ return -EINVAL;
+ }
+}
+
static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value,
unsigned size)
{
@@ -341,6 +402,83 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value,
should_commit = FIELD_EX32(value, CXL_HDM_DECODER0_CTRL, COMMIT);
which_hdm = 0;
break;
+ case A_CXL_RAS_UNC_ERR_STATUS:
+ {
+ uint32_t capctrl = ldl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL);
+ uint32_t fe = FIELD_EX32(capctrl, CXL_RAS_ERR_CAP_CTRL, FIRST_ERROR_POINTER);
+ CXLError *cxl_err;
+ uint32_t unc_err;
+
+ /*
+ * If single bit written that corresponds to the first error
+ * pointer being cleared, update the status and header log.
+ */
+ if (!QTAILQ_EMPTY(&ct3d->error_list)) {
+ if ((1 << fe) ^ value) {
+ CXLError *cxl_next;
+ /*
+ * Software is using wrong flow for multiple header recording
+ * Following behavior in PCIe r6.0 and assuming multiple
+ * header support. Implementation defined choice to clear all
+ * matching records if more than one bit set - which corresponds
+ * closest to behavior of hardware not capable of multiple
+ * header recording.
+ */
+ QTAILQ_FOREACH_SAFE(cxl_err, &ct3d->error_list, node, cxl_next) {
+ if ((1 << cxl_err->type) & value) {
+ QTAILQ_REMOVE(&ct3d->error_list, cxl_err, node);
+ g_free(cxl_err);
+ }
+ }
+ } else {
+ /* Done with previous FE, so drop from list */
+ cxl_err = QTAILQ_FIRST(&ct3d->error_list);
+ QTAILQ_REMOVE(&ct3d->error_list, cxl_err, node);
+ g_free(cxl_err);
+ }
+
+ /*
+ * If there is another FE, then put that in place and update
+ * the header log
+ */
+ if (!QTAILQ_EMPTY(&ct3d->error_list)) {
+ uint32_t *header_log = &cache_mem[R_CXL_RAS_ERR_HEADER0];
+ int i;
+
+ cxl_err = QTAILQ_FIRST(&ct3d->error_list);
+ for (i = 0; i < CXL_RAS_ERR_HEADER_NUM; i++) {
+ stl_le_p(header_log + i, cxl_err->header[i]);
+ }
+ capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
+ FIRST_ERROR_POINTER, cxl_err->type);
+ } else {
+ /*
+ * If no more errors, then follow recomendation of PCI spec
+ * r6.0 6.2.4.2 to set the first error pointer to a status
+ * bit that will never be used.
+ */
+ capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
+ FIRST_ERROR_POINTER,
+ CXL_RAS_UNC_ERR_CXL_UNUSED);
+ }
+ stl_le_p((uint8_t *)cache_mem + A_CXL_RAS_ERR_CAP_CTRL, capctrl);
+ }
+ unc_err = 0;
+ QTAILQ_FOREACH(cxl_err, &ct3d->error_list, node) {
+ unc_err |= 1 << cxl_err->type;
+ }
+ stl_le_p((uint8_t *)cache_mem + offset, unc_err);
+
+ return;
+ }
+ case A_CXL_RAS_COR_ERR_STATUS:
+ {
+ uint32_t rw1c = value;
+ uint32_t temp = ldl_le_p((uint8_t *)cache_mem + offset);
+ temp &= ~rw1c;
+ stl_le_p((uint8_t *)cache_mem + offset, temp);
+ return;
+ }
default:
break;
}
@@ -404,6 +542,8 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)
unsigned short msix_num = 1;
int i, rc;
+ QTAILQ_INIT(&ct3d->error_list);
+
if (!cxl_setup_memory(ct3d, errp)) {
return;
}
@@ -631,6 +771,147 @@ static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size,
*/
}
+/* For uncorrectable errors include support for multiple header recording */
+void qmp_cxl_inject_uncorrectable_errors(const char *path,
+ CXLUncorErrorRecordList *errors,
+ Error **errp)
+{
+ Object *obj = object_resolve_path(path, NULL);
+ static PCIEAERErr err = {};
+ CXLType3Dev *ct3d;
+ CXLError *cxl_err;
+ uint32_t *reg_state;
+ uint32_t unc_err;
+ bool first;
+
+ if (!obj) {
+ error_setg(errp, "Unable to resolve path");
+ return;
+ }
+
+ if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) {
+ error_setg(errp, "Path does not point to a CXL type 3 device");
+ return;
+ }
+
+ err.status = PCI_ERR_UNC_INTN;
+ err.source_id = pci_requester_id(PCI_DEVICE(obj));
+ err.flags = 0;
+
+ ct3d = CXL_TYPE3(obj);
+
+ first = QTAILQ_EMPTY(&ct3d->error_list);
+ reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+ while (errors) {
+ uint32List *header = errors->value->header;
+ uint8_t header_count = 0;
+ int cxl_err_code;
+
+ cxl_err_code = ct3d_qmp_uncor_err_to_cxl(errors->value->type);
+ if (cxl_err_code < 0) {
+ error_setg(errp, "Unknown error code");
+ return;
+ }
+
+ /* If the error is masked, nothing to do here */
+ if (!((1 << cxl_err_code) &
+ ~ldl_le_p(reg_state + R_CXL_RAS_UNC_ERR_MASK))) {
+ errors = errors->next;
+ continue;
+ }
+
+ cxl_err = g_malloc0(sizeof(*cxl_err));
+ if (!cxl_err) {
+ return;
+ }
+
+ cxl_err->type = cxl_err_code;
+ while (header && header_count < 32) {
+ cxl_err->header[header_count++] = header->value;
+ header = header->next;
+ }
+ if (header_count > 32) {
+ error_setg(errp, "Header must be 32 DWORD or less");
+ return;
+ }
+ QTAILQ_INSERT_TAIL(&ct3d->error_list, cxl_err, node);
+
+ errors = errors->next;
+ }
+
+ if (first && !QTAILQ_EMPTY(&ct3d->error_list)) {
+ uint32_t *cache_mem = ct3d->cxl_cstate.crb.cache_mem_registers;
+ uint32_t capctrl = ldl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL);
+ uint32_t *header_log = &cache_mem[R_CXL_RAS_ERR_HEADER0];
+ int i;
+
+ cxl_err = QTAILQ_FIRST(&ct3d->error_list);
+ for (i = 0; i < CXL_RAS_ERR_HEADER_NUM; i++) {
+ stl_le_p(header_log + i, cxl_err->header[i]);
+ }
+
+ capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
+ FIRST_ERROR_POINTER, cxl_err->type);
+ stl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL, capctrl);
+ }
+
+ unc_err = 0;
+ QTAILQ_FOREACH(cxl_err, &ct3d->error_list, node) {
+ unc_err |= (1 << cxl_err->type);
+ }
+ if (!unc_err) {
+ return;
+ }
+
+ stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_STATUS, unc_err);
+ pcie_aer_inject_error(PCI_DEVICE(obj), &err);
+
+ return;
+}
+
+void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type,
+ Error **errp)
+{
+ static PCIEAERErr err = {};
+ Object *obj = object_resolve_path(path, NULL);
+ CXLType3Dev *ct3d;
+ uint32_t *reg_state;
+ uint32_t cor_err;
+ int cxl_err_type;
+
+ if (!obj) {
+ error_setg(errp, "Unable to resolve path");
+ return;
+ }
+ if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) {
+ error_setg(errp, "Path does not point to a CXL type 3 device");
+ return;
+ }
+
+ err.status = PCI_ERR_COR_INTERNAL;
+ err.source_id = pci_requester_id(PCI_DEVICE(obj));
+ err.flags = PCIE_AER_ERR_IS_CORRECTABLE;
+
+ ct3d = CXL_TYPE3(obj);
+ reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
+ cor_err = ldl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS);
+
+ cxl_err_type = ct3d_qmp_cor_err_to_cxl(type);
+ if (cxl_err_type < 0) {
+ error_setg(errp, "Invalid COR error");
+ return;
+ }
+ /* If the error is masked, nothting to do here */
+ if (!((1 << cxl_err_type) & ~ldl_le_p(reg_state + R_CXL_RAS_COR_ERR_MASK))) {
+ return;
+ }
+
+ cor_err |= (1 << cxl_err_type);
+ stl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS, cor_err);
+
+ pcie_aer_inject_error(PCI_DEVICE(obj), &err);
+}
+
static void ct3_class_init(ObjectClass *oc, void *data)
{
DeviceClass *dc = DEVICE_CLASS(oc);
diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c
new file mode 100644
index 0000000000..d574c58f9a
--- /dev/null
+++ b/hw/mem/cxl_type3_stubs.c
@@ -0,0 +1,17 @@
+
+#include "qemu/osdep.h"
+#include "qapi/error.h"
+#include "qapi/qapi-commands-cxl.h"
+
+void qmp_cxl_inject_uncorrectable_errors(const char *path,
+ CXLUncorErrorRecordList *errors,
+ Error **errp)
+{
+ error_setg(errp, "CXL Type 3 support is not compiled in");
+}
+
+void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type,
+ Error **errp)
+{
+ error_setg(errp, "CXL Type 3 support is not compiled in");
+}
diff --git a/hw/mem/meson.build b/hw/mem/meson.build
index 609b2b36fc..56c2618b84 100644
--- a/hw/mem/meson.build
+++ b/hw/mem/meson.build
@@ -4,6 +4,8 @@ mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
+softmmu_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_false: files('cxl_type3_stubs.c'))
+softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('cxl_type3_stubs.c'))
softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
index 692d7a5507..ec4203b83f 100644
--- a/include/hw/cxl/cxl_component.h
+++ b/include/hw/cxl/cxl_component.h
@@ -65,11 +65,37 @@ CXLx_CAPABILITY_HEADER(SNOOP, 0x14)
#define CXL_RAS_REGISTERS_OFFSET 0x80
#define CXL_RAS_REGISTERS_SIZE 0x58
REG32(CXL_RAS_UNC_ERR_STATUS, CXL_RAS_REGISTERS_OFFSET)
+#define CXL_RAS_UNC_ERR_CACHE_DATA_PARITY 0
+#define CXL_RAS_UNC_ERR_CACHE_ADDRESS_PARITY 1
+#define CXL_RAS_UNC_ERR_CACHE_BE_PARITY 2
+#define CXL_RAS_UNC_ERR_CACHE_DATA_ECC 3
+#define CXL_RAS_UNC_ERR_MEM_DATA_PARITY 4
+#define CXL_RAS_UNC_ERR_MEM_ADDRESS_PARITY 5
+#define CXL_RAS_UNC_ERR_MEM_BE_PARITY 6
+#define CXL_RAS_UNC_ERR_MEM_DATA_ECC 7
+#define CXL_RAS_UNC_ERR_REINIT_THRESHOLD 8
+#define CXL_RAS_UNC_ERR_RSVD_ENCODING 9
+#define CXL_RAS_UNC_ERR_POISON_RECEIVED 10
+#define CXL_RAS_UNC_ERR_RECEIVER_OVERFLOW 11
+#define CXL_RAS_UNC_ERR_INTERNAL 14
+#define CXL_RAS_UNC_ERR_CXL_IDE_TX 15
+#define CXL_RAS_UNC_ERR_CXL_IDE_RX 16
+#define CXL_RAS_UNC_ERR_CXL_UNUSED 63 /* Magic value */
REG32(CXL_RAS_UNC_ERR_MASK, CXL_RAS_REGISTERS_OFFSET + 0x4)
REG32(CXL_RAS_UNC_ERR_SEVERITY, CXL_RAS_REGISTERS_OFFSET + 0x8)
REG32(CXL_RAS_COR_ERR_STATUS, CXL_RAS_REGISTERS_OFFSET + 0xc)
+#define CXL_RAS_COR_ERR_CACHE_DATA_ECC 0
+#define CXL_RAS_COR_ERR_MEM_DATA_ECC 1
+#define CXL_RAS_COR_ERR_CRC_THRESHOLD 2
+#define CXL_RAS_COR_ERR_RETRY_THRESHOLD 3
+#define CXL_RAS_COR_ERR_CACHE_POISON_RECEIVED 4
+#define CXL_RAS_COR_ERR_MEM_POISON_RECEIVED 5
+#define CXL_RAS_COR_ERR_PHYSICAL 6
REG32(CXL_RAS_COR_ERR_MASK, CXL_RAS_REGISTERS_OFFSET + 0x10)
REG32(CXL_RAS_ERR_CAP_CTRL, CXL_RAS_REGISTERS_OFFSET + 0x14)
+ FIELD(CXL_RAS_ERR_CAP_CTRL, FIRST_ERROR_POINTER, 0, 6)
+REG32(CXL_RAS_ERR_HEADER0, CXL_RAS_REGISTERS_OFFSET + 0x18)
+#define CXL_RAS_ERR_HEADER_NUM 32
/* Offset 0x18 - 0x58 reserved for RAS logs */
/* 8.2.5.10 - CXL Security Capability Structure */
diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
index 7e5ad65c1d..d589f78202 100644
--- a/include/hw/cxl/cxl_device.h
+++ b/include/hw/cxl/cxl_device.h
@@ -232,6 +232,14 @@ REG64(CXL_MEM_DEV_STS, 0)
FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1)
FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3)
+typedef struct CXLError {
+ QTAILQ_ENTRY(CXLError) node;
+ int type; /* Error code as per FE definition */
+ uint32_t header[32];
+} CXLError;
+
+typedef QTAILQ_HEAD(, CXLError) CXLErrorList;
+
struct CXLType3Dev {
/* Private */
PCIDevice parent_obj;
@@ -248,6 +256,9 @@ struct CXLType3Dev {
/* DOE */
DOECap doe_cdat;
+
+ /* Error injection */
+ CXLErrorList error_list;
};
#define TYPE_CXL_TYPE3 "cxl-type3"
diff --git a/qapi/cxl.json b/qapi/cxl.json
new file mode 100644
index 0000000000..4be7d46041
--- /dev/null
+++ b/qapi/cxl.json
@@ -0,0 +1,128 @@
+# -*- Mode: Python -*-
+# vim: filetype=python
+
+##
+# = CXL devices
+##
+
+##
+# @CxlUncorErrorType:
+#
+# Type of uncorrectable CXL error to inject. These errors are reported via
+# an AER uncorrectable internal error with additional information logged at
+# the CXL device.
+#
+# @cache-data-parity: Data error such as data parity or data ECC error CXL.cache
+# @cache-address-parity: Address parity or other errors associated with the
+# address field on CXL.cache
+# @cache-be-parity: Byte enable parity or other byte enable errors on CXL.cache
+# @cache-data-ecc: ECC error on CXL.cache
+# @mem-data-parity: Data error such as data parity or data ECC error on CXL.mem
+# @mem-address-parity: Address parity or other errors associated with the
+# address field on CXL.mem
+# @mem-be-parity: Byte enable parity or other byte enable errors on CXL.mem.
+# @mem-data-ecc: Data ECC error on CXL.mem.
+# @reinit-threshold: REINIT threshold hit.
+# @rsvd-encoding: Received unrecognized encoding.
+# @poison-received: Received poison from the peer.
+# @receiver-overflow: Buffer overflows (first 3 bits of header log indicate which)
+# @internal: Component specific error
+# @cxl-ide-tx: Integrity and data encryption tx error.
+# @cxl-ide-rx: Integrity and data encryption rx error.
+#
+# Since: 8.0
+##
+
+{ 'enum': 'CxlUncorErrorType',
+ 'data': ['cache-data-parity',
+ 'cache-address-parity',
+ 'cache-be-parity',
+ 'cache-data-ecc',
+ 'mem-data-parity',
+ 'mem-address-parity',
+ 'mem-be-parity',
+ 'mem-data-ecc',
+ 'reinit-threshold',
+ 'rsvd-encoding',
+ 'poison-received',
+ 'receiver-overflow',
+ 'internal',
+ 'cxl-ide-tx',
+ 'cxl-ide-rx'
+ ]
+ }
+
+##
+# @CXLUncorErrorRecord:
+#
+# Record of a single error including header log.
+#
+# @type: Type of error
+# @header: 16 DWORD of header.
+#
+# Since: 8.0
+##
+{ 'struct': 'CXLUncorErrorRecord',
+ 'data': {
+ 'type': 'CxlUncorErrorType',
+ 'header': [ 'uint32' ]
+ }
+}
+
+##
+# @cxl-inject-uncorrectable-errors:
+#
+# Command to allow injection of multiple errors in one go. This allows testing
+# of multiple header log handling in the OS.
+#
+# @path: CXL Type 3 device canonical QOM path
+# @errors: Errors to inject
+#
+# Since: 8.0
+##
+{ 'command': 'cxl-inject-uncorrectable-errors',
+ 'data': { 'path': 'str',
+ 'errors': [ 'CXLUncorErrorRecord' ] }}
+
+##
+# @CxlCorErrorType:
+#
+# Type of CXL correctable error to inject
+#
+# @cache-data-ecc: Data ECC error on CXL.cache
+# @mem-data-ecc: Data ECC error on CXL.mem
+# @crc-threshold: Component specific and applicable to 68 byte Flit mode only.
+# @cache-poison-received: Received poison from a peer on CXL.cache.
+# @mem-poison-received: Received poison from a peer on CXL.mem
+# @physical: Received error indication from the physical layer.
+#
+# Since: 8.0
+##
+{ 'enum': 'CxlCorErrorType',
+ 'data': ['cache-data-ecc',
+ 'mem-data-ecc',
+ 'crc-threshold',
+ 'retry-threshold',
+ 'cache-poison-received',
+ 'mem-poison-received',
+ 'physical']
+}
+
+##
+# @cxl-inject-correctable-error:
+#
+# Command to inject a single correctable error. Multiple error injection
+# of this error type is not interesting as there is no associated header log.
+# These errors are reported via AER as a correctable internal error, with
+# additional detail available from the CXL device.
+#
+# @path: CXL Type 3 device canonical QOM path
+# @type: Type of error.
+#
+# Since: 8.0
+##
+{ 'command': 'cxl-inject-correctable-error',
+ 'data': { 'path': 'str',
+ 'type': 'CxlCorErrorType'
+ }
+}
diff --git a/qapi/meson.build b/qapi/meson.build
index fbdb442fdf..73c3c8c31a 100644
--- a/qapi/meson.build
+++ b/qapi/meson.build
@@ -31,6 +31,7 @@ qapi_all_modules = [
'compat',
'control',
'crypto',
+ 'cxl',
'dump',
'error',
'introspect',
diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json
index f000b90744..079f2a402a 100644
--- a/qapi/qapi-schema.json
+++ b/qapi/qapi-schema.json
@@ -95,3 +95,4 @@
{ 'include': 'pci.json' }
{ 'include': 'stats.json' }
{ 'include': 'virtio.json' }
+{ 'include': 'cxl.json' }
--
2.37.2
^ permalink raw reply related [flat|nested] 24+ messages in thread* Re: [PATCH v6 8/8] hw/mem/cxl_type3: Add CXL RAS Error Injection Support.
2023-02-27 11:27 ` Jonathan Cameron via
@ 2023-03-02 10:06 ` Jonathan Cameron
-1 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron via @ 2023-03-02 10:06 UTC (permalink / raw)
To: Jonathan Cameron via
Cc: Jonathan Cameron, Michael Tsirkin, Ben Widawsky, linux-cxl,
linuxarm, Ira Weiny, Gregory Price, Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth, Michael Roth
+CC Michael Roth.
Michael Tsirkin noted I'd missed CCing QAPI maintaintainers on the
poison injection series so I went back to check the others I had out for review.
For this series I'd managed to pick up one of two as Markus gave comments on
earlier versions.
On Mon, 27 Feb 2023 11:27:51 +0000
Jonathan Cameron via <qemu-devel@nongnu.org> wrote:
> CXL uses PCI AER Internal errors to signal to the host that an error has
> occurred. The host can then read more detailed status from the CXL RAS
> capability.
>
> For uncorrectable errors: support multiple injection in one operation
> as this is needed to reliably test multiple header logging support in an
> OS. The equivalent feature doesn't exist for correctable errors, so only
> one error need be injected at a time.
>
> Note:
> - Header content needs to be manually specified in a fashion that
> matches the specification for what can be in the header for each
> error type.
>
> Injection via QMP:
> { "execute": "qmp_capabilities" }
> ...
> { "execute": "cxl-inject-uncorrectable-errors",
> "arguments": {
> "path": "/machine/peripheral/cxl-pmem0",
> "errors": [
> {
> "type": "cache-address-parity",
> "header": [ 3, 4]
> },
> {
> "type": "cache-data-parity",
> "header": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]
> },
> {
> "type": "internal",
> "header": [ 1, 2, 4]
> }
> ]
> }}
> ...
> { "execute": "cxl-inject-correctable-error",
> "arguments": {
> "path": "/machine/peripheral/cxl-pmem0",
> "type": "physical"
> } }
>
> Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
> v6: (Thanks to Philippe Mathieu-Daudé)
> - Add Since entries in cxl.json
> - Add error prints in the stub functions so that if they are called without
> CONFIG_CXL_MEM_DEVICE then we get a useful print rather than just silently
> eating them.
> ---
> hw/cxl/cxl-component-utils.c | 4 +-
> hw/mem/cxl_type3.c | 281 +++++++++++++++++++++++++++++++++
> hw/mem/cxl_type3_stubs.c | 17 ++
> hw/mem/meson.build | 2 +
> include/hw/cxl/cxl_component.h | 26 +++
> include/hw/cxl/cxl_device.h | 11 ++
> qapi/cxl.json | 128 +++++++++++++++
> qapi/meson.build | 1 +
> qapi/qapi-schema.json | 1 +
> 9 files changed, 470 insertions(+), 1 deletion(-)
>
> diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
> index 737b4764b9..b665d4f565 100644
> --- a/hw/cxl/cxl-component-utils.c
> +++ b/hw/cxl/cxl-component-utils.c
> @@ -142,16 +142,18 @@ static void ras_init_common(uint32_t *reg_state, uint32_t *write_msk)
> * be handled as RO.
> */
> stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_STATUS, 0);
> + stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_STATUS, 0x1cfff);
> /* Bits 12-13 and 17-31 reserved in CXL 2.0 */
> stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
> stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
> stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
> stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
> stl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS, 0);
> + stl_le_p(write_msk + R_CXL_RAS_COR_ERR_STATUS, 0x7f);
> stl_le_p(reg_state + R_CXL_RAS_COR_ERR_MASK, 0x7f);
> stl_le_p(write_msk + R_CXL_RAS_COR_ERR_MASK, 0x7f);
> /* CXL switches and devices must set */
> - stl_le_p(reg_state + R_CXL_RAS_ERR_CAP_CTRL, 0x00);
> + stl_le_p(reg_state + R_CXL_RAS_ERR_CAP_CTRL, 0x200);
> }
>
> static void hdm_init_common(uint32_t *reg_state, uint32_t *write_msk,
> diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
> index 6cdd988d1d..abe60b362c 100644
> --- a/hw/mem/cxl_type3.c
> +++ b/hw/mem/cxl_type3.c
> @@ -1,6 +1,7 @@
> #include "qemu/osdep.h"
> #include "qemu/units.h"
> #include "qemu/error-report.h"
> +#include "qapi/qapi-commands-cxl.h"
> #include "hw/mem/memory-device.h"
> #include "hw/mem/pc-dimm.h"
> #include "hw/pci/pci.h"
> @@ -323,6 +324,66 @@ static void hdm_decoder_commit(CXLType3Dev *ct3d, int which)
> ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, COMMITTED, 1);
> }
>
> +static int ct3d_qmp_uncor_err_to_cxl(CxlUncorErrorType qmp_err)
> +{
> + switch (qmp_err) {
> + case CXL_UNCOR_ERROR_TYPE_CACHE_DATA_PARITY:
> + return CXL_RAS_UNC_ERR_CACHE_DATA_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_CACHE_ADDRESS_PARITY:
> + return CXL_RAS_UNC_ERR_CACHE_ADDRESS_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_CACHE_BE_PARITY:
> + return CXL_RAS_UNC_ERR_CACHE_BE_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_CACHE_DATA_ECC:
> + return CXL_RAS_UNC_ERR_CACHE_DATA_ECC;
> + case CXL_UNCOR_ERROR_TYPE_MEM_DATA_PARITY:
> + return CXL_RAS_UNC_ERR_MEM_DATA_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_MEM_ADDRESS_PARITY:
> + return CXL_RAS_UNC_ERR_MEM_ADDRESS_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_MEM_BE_PARITY:
> + return CXL_RAS_UNC_ERR_MEM_BE_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_MEM_DATA_ECC:
> + return CXL_RAS_UNC_ERR_MEM_DATA_ECC;
> + case CXL_UNCOR_ERROR_TYPE_REINIT_THRESHOLD:
> + return CXL_RAS_UNC_ERR_REINIT_THRESHOLD;
> + case CXL_UNCOR_ERROR_TYPE_RSVD_ENCODING:
> + return CXL_RAS_UNC_ERR_RSVD_ENCODING;
> + case CXL_UNCOR_ERROR_TYPE_POISON_RECEIVED:
> + return CXL_RAS_UNC_ERR_POISON_RECEIVED;
> + case CXL_UNCOR_ERROR_TYPE_RECEIVER_OVERFLOW:
> + return CXL_RAS_UNC_ERR_RECEIVER_OVERFLOW;
> + case CXL_UNCOR_ERROR_TYPE_INTERNAL:
> + return CXL_RAS_UNC_ERR_INTERNAL;
> + case CXL_UNCOR_ERROR_TYPE_CXL_IDE_TX:
> + return CXL_RAS_UNC_ERR_CXL_IDE_TX;
> + case CXL_UNCOR_ERROR_TYPE_CXL_IDE_RX:
> + return CXL_RAS_UNC_ERR_CXL_IDE_RX;
> + default:
> + return -EINVAL;
> + }
> +}
> +
> +static int ct3d_qmp_cor_err_to_cxl(CxlCorErrorType qmp_err)
> +{
> + switch (qmp_err) {
> + case CXL_COR_ERROR_TYPE_CACHE_DATA_ECC:
> + return CXL_RAS_COR_ERR_CACHE_DATA_ECC;
> + case CXL_COR_ERROR_TYPE_MEM_DATA_ECC:
> + return CXL_RAS_COR_ERR_MEM_DATA_ECC;
> + case CXL_COR_ERROR_TYPE_CRC_THRESHOLD:
> + return CXL_RAS_COR_ERR_CRC_THRESHOLD;
> + case CXL_COR_ERROR_TYPE_RETRY_THRESHOLD:
> + return CXL_RAS_COR_ERR_RETRY_THRESHOLD;
> + case CXL_COR_ERROR_TYPE_CACHE_POISON_RECEIVED:
> + return CXL_RAS_COR_ERR_CACHE_POISON_RECEIVED;
> + case CXL_COR_ERROR_TYPE_MEM_POISON_RECEIVED:
> + return CXL_RAS_COR_ERR_MEM_POISON_RECEIVED;
> + case CXL_COR_ERROR_TYPE_PHYSICAL:
> + return CXL_RAS_COR_ERR_PHYSICAL;
> + default:
> + return -EINVAL;
> + }
> +}
> +
> static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value,
> unsigned size)
> {
> @@ -341,6 +402,83 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value,
> should_commit = FIELD_EX32(value, CXL_HDM_DECODER0_CTRL, COMMIT);
> which_hdm = 0;
> break;
> + case A_CXL_RAS_UNC_ERR_STATUS:
> + {
> + uint32_t capctrl = ldl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL);
> + uint32_t fe = FIELD_EX32(capctrl, CXL_RAS_ERR_CAP_CTRL, FIRST_ERROR_POINTER);
> + CXLError *cxl_err;
> + uint32_t unc_err;
> +
> + /*
> + * If single bit written that corresponds to the first error
> + * pointer being cleared, update the status and header log.
> + */
> + if (!QTAILQ_EMPTY(&ct3d->error_list)) {
> + if ((1 << fe) ^ value) {
> + CXLError *cxl_next;
> + /*
> + * Software is using wrong flow for multiple header recording
> + * Following behavior in PCIe r6.0 and assuming multiple
> + * header support. Implementation defined choice to clear all
> + * matching records if more than one bit set - which corresponds
> + * closest to behavior of hardware not capable of multiple
> + * header recording.
> + */
> + QTAILQ_FOREACH_SAFE(cxl_err, &ct3d->error_list, node, cxl_next) {
> + if ((1 << cxl_err->type) & value) {
> + QTAILQ_REMOVE(&ct3d->error_list, cxl_err, node);
> + g_free(cxl_err);
> + }
> + }
> + } else {
> + /* Done with previous FE, so drop from list */
> + cxl_err = QTAILQ_FIRST(&ct3d->error_list);
> + QTAILQ_REMOVE(&ct3d->error_list, cxl_err, node);
> + g_free(cxl_err);
> + }
> +
> + /*
> + * If there is another FE, then put that in place and update
> + * the header log
> + */
> + if (!QTAILQ_EMPTY(&ct3d->error_list)) {
> + uint32_t *header_log = &cache_mem[R_CXL_RAS_ERR_HEADER0];
> + int i;
> +
> + cxl_err = QTAILQ_FIRST(&ct3d->error_list);
> + for (i = 0; i < CXL_RAS_ERR_HEADER_NUM; i++) {
> + stl_le_p(header_log + i, cxl_err->header[i]);
> + }
> + capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
> + FIRST_ERROR_POINTER, cxl_err->type);
> + } else {
> + /*
> + * If no more errors, then follow recomendation of PCI spec
> + * r6.0 6.2.4.2 to set the first error pointer to a status
> + * bit that will never be used.
> + */
> + capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
> + FIRST_ERROR_POINTER,
> + CXL_RAS_UNC_ERR_CXL_UNUSED);
> + }
> + stl_le_p((uint8_t *)cache_mem + A_CXL_RAS_ERR_CAP_CTRL, capctrl);
> + }
> + unc_err = 0;
> + QTAILQ_FOREACH(cxl_err, &ct3d->error_list, node) {
> + unc_err |= 1 << cxl_err->type;
> + }
> + stl_le_p((uint8_t *)cache_mem + offset, unc_err);
> +
> + return;
> + }
> + case A_CXL_RAS_COR_ERR_STATUS:
> + {
> + uint32_t rw1c = value;
> + uint32_t temp = ldl_le_p((uint8_t *)cache_mem + offset);
> + temp &= ~rw1c;
> + stl_le_p((uint8_t *)cache_mem + offset, temp);
> + return;
> + }
> default:
> break;
> }
> @@ -404,6 +542,8 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)
> unsigned short msix_num = 1;
> int i, rc;
>
> + QTAILQ_INIT(&ct3d->error_list);
> +
> if (!cxl_setup_memory(ct3d, errp)) {
> return;
> }
> @@ -631,6 +771,147 @@ static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size,
> */
> }
>
> +/* For uncorrectable errors include support for multiple header recording */
> +void qmp_cxl_inject_uncorrectable_errors(const char *path,
> + CXLUncorErrorRecordList *errors,
> + Error **errp)
> +{
> + Object *obj = object_resolve_path(path, NULL);
> + static PCIEAERErr err = {};
> + CXLType3Dev *ct3d;
> + CXLError *cxl_err;
> + uint32_t *reg_state;
> + uint32_t unc_err;
> + bool first;
> +
> + if (!obj) {
> + error_setg(errp, "Unable to resolve path");
> + return;
> + }
> +
> + if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) {
> + error_setg(errp, "Path does not point to a CXL type 3 device");
> + return;
> + }
> +
> + err.status = PCI_ERR_UNC_INTN;
> + err.source_id = pci_requester_id(PCI_DEVICE(obj));
> + err.flags = 0;
> +
> + ct3d = CXL_TYPE3(obj);
> +
> + first = QTAILQ_EMPTY(&ct3d->error_list);
> + reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
> + while (errors) {
> + uint32List *header = errors->value->header;
> + uint8_t header_count = 0;
> + int cxl_err_code;
> +
> + cxl_err_code = ct3d_qmp_uncor_err_to_cxl(errors->value->type);
> + if (cxl_err_code < 0) {
> + error_setg(errp, "Unknown error code");
> + return;
> + }
> +
> + /* If the error is masked, nothing to do here */
> + if (!((1 << cxl_err_code) &
> + ~ldl_le_p(reg_state + R_CXL_RAS_UNC_ERR_MASK))) {
> + errors = errors->next;
> + continue;
> + }
> +
> + cxl_err = g_malloc0(sizeof(*cxl_err));
> + if (!cxl_err) {
> + return;
> + }
> +
> + cxl_err->type = cxl_err_code;
> + while (header && header_count < 32) {
> + cxl_err->header[header_count++] = header->value;
> + header = header->next;
> + }
> + if (header_count > 32) {
> + error_setg(errp, "Header must be 32 DWORD or less");
> + return;
> + }
> + QTAILQ_INSERT_TAIL(&ct3d->error_list, cxl_err, node);
> +
> + errors = errors->next;
> + }
> +
> + if (first && !QTAILQ_EMPTY(&ct3d->error_list)) {
> + uint32_t *cache_mem = ct3d->cxl_cstate.crb.cache_mem_registers;
> + uint32_t capctrl = ldl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL);
> + uint32_t *header_log = &cache_mem[R_CXL_RAS_ERR_HEADER0];
> + int i;
> +
> + cxl_err = QTAILQ_FIRST(&ct3d->error_list);
> + for (i = 0; i < CXL_RAS_ERR_HEADER_NUM; i++) {
> + stl_le_p(header_log + i, cxl_err->header[i]);
> + }
> +
> + capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
> + FIRST_ERROR_POINTER, cxl_err->type);
> + stl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL, capctrl);
> + }
> +
> + unc_err = 0;
> + QTAILQ_FOREACH(cxl_err, &ct3d->error_list, node) {
> + unc_err |= (1 << cxl_err->type);
> + }
> + if (!unc_err) {
> + return;
> + }
> +
> + stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_STATUS, unc_err);
> + pcie_aer_inject_error(PCI_DEVICE(obj), &err);
> +
> + return;
> +}
> +
> +void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type,
> + Error **errp)
> +{
> + static PCIEAERErr err = {};
> + Object *obj = object_resolve_path(path, NULL);
> + CXLType3Dev *ct3d;
> + uint32_t *reg_state;
> + uint32_t cor_err;
> + int cxl_err_type;
> +
> + if (!obj) {
> + error_setg(errp, "Unable to resolve path");
> + return;
> + }
> + if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) {
> + error_setg(errp, "Path does not point to a CXL type 3 device");
> + return;
> + }
> +
> + err.status = PCI_ERR_COR_INTERNAL;
> + err.source_id = pci_requester_id(PCI_DEVICE(obj));
> + err.flags = PCIE_AER_ERR_IS_CORRECTABLE;
> +
> + ct3d = CXL_TYPE3(obj);
> + reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
> + cor_err = ldl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS);
> +
> + cxl_err_type = ct3d_qmp_cor_err_to_cxl(type);
> + if (cxl_err_type < 0) {
> + error_setg(errp, "Invalid COR error");
> + return;
> + }
> + /* If the error is masked, nothting to do here */
> + if (!((1 << cxl_err_type) & ~ldl_le_p(reg_state + R_CXL_RAS_COR_ERR_MASK))) {
> + return;
> + }
> +
> + cor_err |= (1 << cxl_err_type);
> + stl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS, cor_err);
> +
> + pcie_aer_inject_error(PCI_DEVICE(obj), &err);
> +}
> +
> static void ct3_class_init(ObjectClass *oc, void *data)
> {
> DeviceClass *dc = DEVICE_CLASS(oc);
> diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c
> new file mode 100644
> index 0000000000..d574c58f9a
> --- /dev/null
> +++ b/hw/mem/cxl_type3_stubs.c
> @@ -0,0 +1,17 @@
> +
> +#include "qemu/osdep.h"
> +#include "qapi/error.h"
> +#include "qapi/qapi-commands-cxl.h"
> +
> +void qmp_cxl_inject_uncorrectable_errors(const char *path,
> + CXLUncorErrorRecordList *errors,
> + Error **errp)
> +{
> + error_setg(errp, "CXL Type 3 support is not compiled in");
> +}
> +
> +void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type,
> + Error **errp)
> +{
> + error_setg(errp, "CXL Type 3 support is not compiled in");
> +}
> diff --git a/hw/mem/meson.build b/hw/mem/meson.build
> index 609b2b36fc..56c2618b84 100644
> --- a/hw/mem/meson.build
> +++ b/hw/mem/meson.build
> @@ -4,6 +4,8 @@ mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
> mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
> mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
> mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
> +softmmu_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_false: files('cxl_type3_stubs.c'))
> +softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('cxl_type3_stubs.c'))
>
> softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
>
> diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
> index 692d7a5507..ec4203b83f 100644
> --- a/include/hw/cxl/cxl_component.h
> +++ b/include/hw/cxl/cxl_component.h
> @@ -65,11 +65,37 @@ CXLx_CAPABILITY_HEADER(SNOOP, 0x14)
> #define CXL_RAS_REGISTERS_OFFSET 0x80
> #define CXL_RAS_REGISTERS_SIZE 0x58
> REG32(CXL_RAS_UNC_ERR_STATUS, CXL_RAS_REGISTERS_OFFSET)
> +#define CXL_RAS_UNC_ERR_CACHE_DATA_PARITY 0
> +#define CXL_RAS_UNC_ERR_CACHE_ADDRESS_PARITY 1
> +#define CXL_RAS_UNC_ERR_CACHE_BE_PARITY 2
> +#define CXL_RAS_UNC_ERR_CACHE_DATA_ECC 3
> +#define CXL_RAS_UNC_ERR_MEM_DATA_PARITY 4
> +#define CXL_RAS_UNC_ERR_MEM_ADDRESS_PARITY 5
> +#define CXL_RAS_UNC_ERR_MEM_BE_PARITY 6
> +#define CXL_RAS_UNC_ERR_MEM_DATA_ECC 7
> +#define CXL_RAS_UNC_ERR_REINIT_THRESHOLD 8
> +#define CXL_RAS_UNC_ERR_RSVD_ENCODING 9
> +#define CXL_RAS_UNC_ERR_POISON_RECEIVED 10
> +#define CXL_RAS_UNC_ERR_RECEIVER_OVERFLOW 11
> +#define CXL_RAS_UNC_ERR_INTERNAL 14
> +#define CXL_RAS_UNC_ERR_CXL_IDE_TX 15
> +#define CXL_RAS_UNC_ERR_CXL_IDE_RX 16
> +#define CXL_RAS_UNC_ERR_CXL_UNUSED 63 /* Magic value */
> REG32(CXL_RAS_UNC_ERR_MASK, CXL_RAS_REGISTERS_OFFSET + 0x4)
> REG32(CXL_RAS_UNC_ERR_SEVERITY, CXL_RAS_REGISTERS_OFFSET + 0x8)
> REG32(CXL_RAS_COR_ERR_STATUS, CXL_RAS_REGISTERS_OFFSET + 0xc)
> +#define CXL_RAS_COR_ERR_CACHE_DATA_ECC 0
> +#define CXL_RAS_COR_ERR_MEM_DATA_ECC 1
> +#define CXL_RAS_COR_ERR_CRC_THRESHOLD 2
> +#define CXL_RAS_COR_ERR_RETRY_THRESHOLD 3
> +#define CXL_RAS_COR_ERR_CACHE_POISON_RECEIVED 4
> +#define CXL_RAS_COR_ERR_MEM_POISON_RECEIVED 5
> +#define CXL_RAS_COR_ERR_PHYSICAL 6
> REG32(CXL_RAS_COR_ERR_MASK, CXL_RAS_REGISTERS_OFFSET + 0x10)
> REG32(CXL_RAS_ERR_CAP_CTRL, CXL_RAS_REGISTERS_OFFSET + 0x14)
> + FIELD(CXL_RAS_ERR_CAP_CTRL, FIRST_ERROR_POINTER, 0, 6)
> +REG32(CXL_RAS_ERR_HEADER0, CXL_RAS_REGISTERS_OFFSET + 0x18)
> +#define CXL_RAS_ERR_HEADER_NUM 32
> /* Offset 0x18 - 0x58 reserved for RAS logs */
>
> /* 8.2.5.10 - CXL Security Capability Structure */
> diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
> index 7e5ad65c1d..d589f78202 100644
> --- a/include/hw/cxl/cxl_device.h
> +++ b/include/hw/cxl/cxl_device.h
> @@ -232,6 +232,14 @@ REG64(CXL_MEM_DEV_STS, 0)
> FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1)
> FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3)
>
> +typedef struct CXLError {
> + QTAILQ_ENTRY(CXLError) node;
> + int type; /* Error code as per FE definition */
> + uint32_t header[32];
> +} CXLError;
> +
> +typedef QTAILQ_HEAD(, CXLError) CXLErrorList;
> +
> struct CXLType3Dev {
> /* Private */
> PCIDevice parent_obj;
> @@ -248,6 +256,9 @@ struct CXLType3Dev {
>
> /* DOE */
> DOECap doe_cdat;
> +
> + /* Error injection */
> + CXLErrorList error_list;
> };
>
> #define TYPE_CXL_TYPE3 "cxl-type3"
> diff --git a/qapi/cxl.json b/qapi/cxl.json
> new file mode 100644
> index 0000000000..4be7d46041
> --- /dev/null
> +++ b/qapi/cxl.json
> @@ -0,0 +1,128 @@
> +# -*- Mode: Python -*-
> +# vim: filetype=python
> +
> +##
> +# = CXL devices
> +##
> +
> +##
> +# @CxlUncorErrorType:
> +#
> +# Type of uncorrectable CXL error to inject. These errors are reported via
> +# an AER uncorrectable internal error with additional information logged at
> +# the CXL device.
> +#
> +# @cache-data-parity: Data error such as data parity or data ECC error CXL.cache
> +# @cache-address-parity: Address parity or other errors associated with the
> +# address field on CXL.cache
> +# @cache-be-parity: Byte enable parity or other byte enable errors on CXL.cache
> +# @cache-data-ecc: ECC error on CXL.cache
> +# @mem-data-parity: Data error such as data parity or data ECC error on CXL.mem
> +# @mem-address-parity: Address parity or other errors associated with the
> +# address field on CXL.mem
> +# @mem-be-parity: Byte enable parity or other byte enable errors on CXL.mem.
> +# @mem-data-ecc: Data ECC error on CXL.mem.
> +# @reinit-threshold: REINIT threshold hit.
> +# @rsvd-encoding: Received unrecognized encoding.
> +# @poison-received: Received poison from the peer.
> +# @receiver-overflow: Buffer overflows (first 3 bits of header log indicate which)
> +# @internal: Component specific error
> +# @cxl-ide-tx: Integrity and data encryption tx error.
> +# @cxl-ide-rx: Integrity and data encryption rx error.
> +#
> +# Since: 8.0
> +##
> +
> +{ 'enum': 'CxlUncorErrorType',
> + 'data': ['cache-data-parity',
> + 'cache-address-parity',
> + 'cache-be-parity',
> + 'cache-data-ecc',
> + 'mem-data-parity',
> + 'mem-address-parity',
> + 'mem-be-parity',
> + 'mem-data-ecc',
> + 'reinit-threshold',
> + 'rsvd-encoding',
> + 'poison-received',
> + 'receiver-overflow',
> + 'internal',
> + 'cxl-ide-tx',
> + 'cxl-ide-rx'
> + ]
> + }
> +
> +##
> +# @CXLUncorErrorRecord:
> +#
> +# Record of a single error including header log.
> +#
> +# @type: Type of error
> +# @header: 16 DWORD of header.
> +#
> +# Since: 8.0
> +##
> +{ 'struct': 'CXLUncorErrorRecord',
> + 'data': {
> + 'type': 'CxlUncorErrorType',
> + 'header': [ 'uint32' ]
> + }
> +}
> +
> +##
> +# @cxl-inject-uncorrectable-errors:
> +#
> +# Command to allow injection of multiple errors in one go. This allows testing
> +# of multiple header log handling in the OS.
> +#
> +# @path: CXL Type 3 device canonical QOM path
> +# @errors: Errors to inject
> +#
> +# Since: 8.0
> +##
> +{ 'command': 'cxl-inject-uncorrectable-errors',
> + 'data': { 'path': 'str',
> + 'errors': [ 'CXLUncorErrorRecord' ] }}
> +
> +##
> +# @CxlCorErrorType:
> +#
> +# Type of CXL correctable error to inject
> +#
> +# @cache-data-ecc: Data ECC error on CXL.cache
> +# @mem-data-ecc: Data ECC error on CXL.mem
> +# @crc-threshold: Component specific and applicable to 68 byte Flit mode only.
> +# @cache-poison-received: Received poison from a peer on CXL.cache.
> +# @mem-poison-received: Received poison from a peer on CXL.mem
> +# @physical: Received error indication from the physical layer.
> +#
> +# Since: 8.0
> +##
> +{ 'enum': 'CxlCorErrorType',
> + 'data': ['cache-data-ecc',
> + 'mem-data-ecc',
> + 'crc-threshold',
> + 'retry-threshold',
> + 'cache-poison-received',
> + 'mem-poison-received',
> + 'physical']
> +}
> +
> +##
> +# @cxl-inject-correctable-error:
> +#
> +# Command to inject a single correctable error. Multiple error injection
> +# of this error type is not interesting as there is no associated header log.
> +# These errors are reported via AER as a correctable internal error, with
> +# additional detail available from the CXL device.
> +#
> +# @path: CXL Type 3 device canonical QOM path
> +# @type: Type of error.
> +#
> +# Since: 8.0
> +##
> +{ 'command': 'cxl-inject-correctable-error',
> + 'data': { 'path': 'str',
> + 'type': 'CxlCorErrorType'
> + }
> +}
> diff --git a/qapi/meson.build b/qapi/meson.build
> index fbdb442fdf..73c3c8c31a 100644
> --- a/qapi/meson.build
> +++ b/qapi/meson.build
> @@ -31,6 +31,7 @@ qapi_all_modules = [
> 'compat',
> 'control',
> 'crypto',
> + 'cxl',
> 'dump',
> 'error',
> 'introspect',
> diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json
> index f000b90744..079f2a402a 100644
> --- a/qapi/qapi-schema.json
> +++ b/qapi/qapi-schema.json
> @@ -95,3 +95,4 @@
> { 'include': 'pci.json' }
> { 'include': 'stats.json' }
> { 'include': 'virtio.json' }
> +{ 'include': 'cxl.json' }
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH v6 8/8] hw/mem/cxl_type3: Add CXL RAS Error Injection Support.
@ 2023-03-02 10:06 ` Jonathan Cameron
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron @ 2023-03-02 10:06 UTC (permalink / raw)
To: Jonathan Cameron via
Cc: Jonathan Cameron, Michael Tsirkin, Ben Widawsky, linux-cxl,
linuxarm, Ira Weiny, Gregory Price, Philippe Mathieu-Daudé,
Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth, Michael Roth
+CC Michael Roth.
Michael Tsirkin noted I'd missed CCing QAPI maintaintainers on the
poison injection series so I went back to check the others I had out for review.
For this series I'd managed to pick up one of two as Markus gave comments on
earlier versions.
On Mon, 27 Feb 2023 11:27:51 +0000
Jonathan Cameron via <qemu-devel@nongnu.org> wrote:
> CXL uses PCI AER Internal errors to signal to the host that an error has
> occurred. The host can then read more detailed status from the CXL RAS
> capability.
>
> For uncorrectable errors: support multiple injection in one operation
> as this is needed to reliably test multiple header logging support in an
> OS. The equivalent feature doesn't exist for correctable errors, so only
> one error need be injected at a time.
>
> Note:
> - Header content needs to be manually specified in a fashion that
> matches the specification for what can be in the header for each
> error type.
>
> Injection via QMP:
> { "execute": "qmp_capabilities" }
> ...
> { "execute": "cxl-inject-uncorrectable-errors",
> "arguments": {
> "path": "/machine/peripheral/cxl-pmem0",
> "errors": [
> {
> "type": "cache-address-parity",
> "header": [ 3, 4]
> },
> {
> "type": "cache-data-parity",
> "header": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]
> },
> {
> "type": "internal",
> "header": [ 1, 2, 4]
> }
> ]
> }}
> ...
> { "execute": "cxl-inject-correctable-error",
> "arguments": {
> "path": "/machine/peripheral/cxl-pmem0",
> "type": "physical"
> } }
>
> Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> ---
> v6: (Thanks to Philippe Mathieu-Daudé)
> - Add Since entries in cxl.json
> - Add error prints in the stub functions so that if they are called without
> CONFIG_CXL_MEM_DEVICE then we get a useful print rather than just silently
> eating them.
> ---
> hw/cxl/cxl-component-utils.c | 4 +-
> hw/mem/cxl_type3.c | 281 +++++++++++++++++++++++++++++++++
> hw/mem/cxl_type3_stubs.c | 17 ++
> hw/mem/meson.build | 2 +
> include/hw/cxl/cxl_component.h | 26 +++
> include/hw/cxl/cxl_device.h | 11 ++
> qapi/cxl.json | 128 +++++++++++++++
> qapi/meson.build | 1 +
> qapi/qapi-schema.json | 1 +
> 9 files changed, 470 insertions(+), 1 deletion(-)
>
> diff --git a/hw/cxl/cxl-component-utils.c b/hw/cxl/cxl-component-utils.c
> index 737b4764b9..b665d4f565 100644
> --- a/hw/cxl/cxl-component-utils.c
> +++ b/hw/cxl/cxl-component-utils.c
> @@ -142,16 +142,18 @@ static void ras_init_common(uint32_t *reg_state, uint32_t *write_msk)
> * be handled as RO.
> */
> stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_STATUS, 0);
> + stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_STATUS, 0x1cfff);
> /* Bits 12-13 and 17-31 reserved in CXL 2.0 */
> stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
> stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_MASK, 0x1cfff);
> stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
> stl_le_p(write_msk + R_CXL_RAS_UNC_ERR_SEVERITY, 0x1cfff);
> stl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS, 0);
> + stl_le_p(write_msk + R_CXL_RAS_COR_ERR_STATUS, 0x7f);
> stl_le_p(reg_state + R_CXL_RAS_COR_ERR_MASK, 0x7f);
> stl_le_p(write_msk + R_CXL_RAS_COR_ERR_MASK, 0x7f);
> /* CXL switches and devices must set */
> - stl_le_p(reg_state + R_CXL_RAS_ERR_CAP_CTRL, 0x00);
> + stl_le_p(reg_state + R_CXL_RAS_ERR_CAP_CTRL, 0x200);
> }
>
> static void hdm_init_common(uint32_t *reg_state, uint32_t *write_msk,
> diff --git a/hw/mem/cxl_type3.c b/hw/mem/cxl_type3.c
> index 6cdd988d1d..abe60b362c 100644
> --- a/hw/mem/cxl_type3.c
> +++ b/hw/mem/cxl_type3.c
> @@ -1,6 +1,7 @@
> #include "qemu/osdep.h"
> #include "qemu/units.h"
> #include "qemu/error-report.h"
> +#include "qapi/qapi-commands-cxl.h"
> #include "hw/mem/memory-device.h"
> #include "hw/mem/pc-dimm.h"
> #include "hw/pci/pci.h"
> @@ -323,6 +324,66 @@ static void hdm_decoder_commit(CXLType3Dev *ct3d, int which)
> ARRAY_FIELD_DP32(cache_mem, CXL_HDM_DECODER0_CTRL, COMMITTED, 1);
> }
>
> +static int ct3d_qmp_uncor_err_to_cxl(CxlUncorErrorType qmp_err)
> +{
> + switch (qmp_err) {
> + case CXL_UNCOR_ERROR_TYPE_CACHE_DATA_PARITY:
> + return CXL_RAS_UNC_ERR_CACHE_DATA_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_CACHE_ADDRESS_PARITY:
> + return CXL_RAS_UNC_ERR_CACHE_ADDRESS_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_CACHE_BE_PARITY:
> + return CXL_RAS_UNC_ERR_CACHE_BE_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_CACHE_DATA_ECC:
> + return CXL_RAS_UNC_ERR_CACHE_DATA_ECC;
> + case CXL_UNCOR_ERROR_TYPE_MEM_DATA_PARITY:
> + return CXL_RAS_UNC_ERR_MEM_DATA_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_MEM_ADDRESS_PARITY:
> + return CXL_RAS_UNC_ERR_MEM_ADDRESS_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_MEM_BE_PARITY:
> + return CXL_RAS_UNC_ERR_MEM_BE_PARITY;
> + case CXL_UNCOR_ERROR_TYPE_MEM_DATA_ECC:
> + return CXL_RAS_UNC_ERR_MEM_DATA_ECC;
> + case CXL_UNCOR_ERROR_TYPE_REINIT_THRESHOLD:
> + return CXL_RAS_UNC_ERR_REINIT_THRESHOLD;
> + case CXL_UNCOR_ERROR_TYPE_RSVD_ENCODING:
> + return CXL_RAS_UNC_ERR_RSVD_ENCODING;
> + case CXL_UNCOR_ERROR_TYPE_POISON_RECEIVED:
> + return CXL_RAS_UNC_ERR_POISON_RECEIVED;
> + case CXL_UNCOR_ERROR_TYPE_RECEIVER_OVERFLOW:
> + return CXL_RAS_UNC_ERR_RECEIVER_OVERFLOW;
> + case CXL_UNCOR_ERROR_TYPE_INTERNAL:
> + return CXL_RAS_UNC_ERR_INTERNAL;
> + case CXL_UNCOR_ERROR_TYPE_CXL_IDE_TX:
> + return CXL_RAS_UNC_ERR_CXL_IDE_TX;
> + case CXL_UNCOR_ERROR_TYPE_CXL_IDE_RX:
> + return CXL_RAS_UNC_ERR_CXL_IDE_RX;
> + default:
> + return -EINVAL;
> + }
> +}
> +
> +static int ct3d_qmp_cor_err_to_cxl(CxlCorErrorType qmp_err)
> +{
> + switch (qmp_err) {
> + case CXL_COR_ERROR_TYPE_CACHE_DATA_ECC:
> + return CXL_RAS_COR_ERR_CACHE_DATA_ECC;
> + case CXL_COR_ERROR_TYPE_MEM_DATA_ECC:
> + return CXL_RAS_COR_ERR_MEM_DATA_ECC;
> + case CXL_COR_ERROR_TYPE_CRC_THRESHOLD:
> + return CXL_RAS_COR_ERR_CRC_THRESHOLD;
> + case CXL_COR_ERROR_TYPE_RETRY_THRESHOLD:
> + return CXL_RAS_COR_ERR_RETRY_THRESHOLD;
> + case CXL_COR_ERROR_TYPE_CACHE_POISON_RECEIVED:
> + return CXL_RAS_COR_ERR_CACHE_POISON_RECEIVED;
> + case CXL_COR_ERROR_TYPE_MEM_POISON_RECEIVED:
> + return CXL_RAS_COR_ERR_MEM_POISON_RECEIVED;
> + case CXL_COR_ERROR_TYPE_PHYSICAL:
> + return CXL_RAS_COR_ERR_PHYSICAL;
> + default:
> + return -EINVAL;
> + }
> +}
> +
> static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value,
> unsigned size)
> {
> @@ -341,6 +402,83 @@ static void ct3d_reg_write(void *opaque, hwaddr offset, uint64_t value,
> should_commit = FIELD_EX32(value, CXL_HDM_DECODER0_CTRL, COMMIT);
> which_hdm = 0;
> break;
> + case A_CXL_RAS_UNC_ERR_STATUS:
> + {
> + uint32_t capctrl = ldl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL);
> + uint32_t fe = FIELD_EX32(capctrl, CXL_RAS_ERR_CAP_CTRL, FIRST_ERROR_POINTER);
> + CXLError *cxl_err;
> + uint32_t unc_err;
> +
> + /*
> + * If single bit written that corresponds to the first error
> + * pointer being cleared, update the status and header log.
> + */
> + if (!QTAILQ_EMPTY(&ct3d->error_list)) {
> + if ((1 << fe) ^ value) {
> + CXLError *cxl_next;
> + /*
> + * Software is using wrong flow for multiple header recording
> + * Following behavior in PCIe r6.0 and assuming multiple
> + * header support. Implementation defined choice to clear all
> + * matching records if more than one bit set - which corresponds
> + * closest to behavior of hardware not capable of multiple
> + * header recording.
> + */
> + QTAILQ_FOREACH_SAFE(cxl_err, &ct3d->error_list, node, cxl_next) {
> + if ((1 << cxl_err->type) & value) {
> + QTAILQ_REMOVE(&ct3d->error_list, cxl_err, node);
> + g_free(cxl_err);
> + }
> + }
> + } else {
> + /* Done with previous FE, so drop from list */
> + cxl_err = QTAILQ_FIRST(&ct3d->error_list);
> + QTAILQ_REMOVE(&ct3d->error_list, cxl_err, node);
> + g_free(cxl_err);
> + }
> +
> + /*
> + * If there is another FE, then put that in place and update
> + * the header log
> + */
> + if (!QTAILQ_EMPTY(&ct3d->error_list)) {
> + uint32_t *header_log = &cache_mem[R_CXL_RAS_ERR_HEADER0];
> + int i;
> +
> + cxl_err = QTAILQ_FIRST(&ct3d->error_list);
> + for (i = 0; i < CXL_RAS_ERR_HEADER_NUM; i++) {
> + stl_le_p(header_log + i, cxl_err->header[i]);
> + }
> + capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
> + FIRST_ERROR_POINTER, cxl_err->type);
> + } else {
> + /*
> + * If no more errors, then follow recomendation of PCI spec
> + * r6.0 6.2.4.2 to set the first error pointer to a status
> + * bit that will never be used.
> + */
> + capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
> + FIRST_ERROR_POINTER,
> + CXL_RAS_UNC_ERR_CXL_UNUSED);
> + }
> + stl_le_p((uint8_t *)cache_mem + A_CXL_RAS_ERR_CAP_CTRL, capctrl);
> + }
> + unc_err = 0;
> + QTAILQ_FOREACH(cxl_err, &ct3d->error_list, node) {
> + unc_err |= 1 << cxl_err->type;
> + }
> + stl_le_p((uint8_t *)cache_mem + offset, unc_err);
> +
> + return;
> + }
> + case A_CXL_RAS_COR_ERR_STATUS:
> + {
> + uint32_t rw1c = value;
> + uint32_t temp = ldl_le_p((uint8_t *)cache_mem + offset);
> + temp &= ~rw1c;
> + stl_le_p((uint8_t *)cache_mem + offset, temp);
> + return;
> + }
> default:
> break;
> }
> @@ -404,6 +542,8 @@ static void ct3_realize(PCIDevice *pci_dev, Error **errp)
> unsigned short msix_num = 1;
> int i, rc;
>
> + QTAILQ_INIT(&ct3d->error_list);
> +
> if (!cxl_setup_memory(ct3d, errp)) {
> return;
> }
> @@ -631,6 +771,147 @@ static void set_lsa(CXLType3Dev *ct3d, const void *buf, uint64_t size,
> */
> }
>
> +/* For uncorrectable errors include support for multiple header recording */
> +void qmp_cxl_inject_uncorrectable_errors(const char *path,
> + CXLUncorErrorRecordList *errors,
> + Error **errp)
> +{
> + Object *obj = object_resolve_path(path, NULL);
> + static PCIEAERErr err = {};
> + CXLType3Dev *ct3d;
> + CXLError *cxl_err;
> + uint32_t *reg_state;
> + uint32_t unc_err;
> + bool first;
> +
> + if (!obj) {
> + error_setg(errp, "Unable to resolve path");
> + return;
> + }
> +
> + if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) {
> + error_setg(errp, "Path does not point to a CXL type 3 device");
> + return;
> + }
> +
> + err.status = PCI_ERR_UNC_INTN;
> + err.source_id = pci_requester_id(PCI_DEVICE(obj));
> + err.flags = 0;
> +
> + ct3d = CXL_TYPE3(obj);
> +
> + first = QTAILQ_EMPTY(&ct3d->error_list);
> + reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
> + while (errors) {
> + uint32List *header = errors->value->header;
> + uint8_t header_count = 0;
> + int cxl_err_code;
> +
> + cxl_err_code = ct3d_qmp_uncor_err_to_cxl(errors->value->type);
> + if (cxl_err_code < 0) {
> + error_setg(errp, "Unknown error code");
> + return;
> + }
> +
> + /* If the error is masked, nothing to do here */
> + if (!((1 << cxl_err_code) &
> + ~ldl_le_p(reg_state + R_CXL_RAS_UNC_ERR_MASK))) {
> + errors = errors->next;
> + continue;
> + }
> +
> + cxl_err = g_malloc0(sizeof(*cxl_err));
> + if (!cxl_err) {
> + return;
> + }
> +
> + cxl_err->type = cxl_err_code;
> + while (header && header_count < 32) {
> + cxl_err->header[header_count++] = header->value;
> + header = header->next;
> + }
> + if (header_count > 32) {
> + error_setg(errp, "Header must be 32 DWORD or less");
> + return;
> + }
> + QTAILQ_INSERT_TAIL(&ct3d->error_list, cxl_err, node);
> +
> + errors = errors->next;
> + }
> +
> + if (first && !QTAILQ_EMPTY(&ct3d->error_list)) {
> + uint32_t *cache_mem = ct3d->cxl_cstate.crb.cache_mem_registers;
> + uint32_t capctrl = ldl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL);
> + uint32_t *header_log = &cache_mem[R_CXL_RAS_ERR_HEADER0];
> + int i;
> +
> + cxl_err = QTAILQ_FIRST(&ct3d->error_list);
> + for (i = 0; i < CXL_RAS_ERR_HEADER_NUM; i++) {
> + stl_le_p(header_log + i, cxl_err->header[i]);
> + }
> +
> + capctrl = FIELD_DP32(capctrl, CXL_RAS_ERR_CAP_CTRL,
> + FIRST_ERROR_POINTER, cxl_err->type);
> + stl_le_p(cache_mem + R_CXL_RAS_ERR_CAP_CTRL, capctrl);
> + }
> +
> + unc_err = 0;
> + QTAILQ_FOREACH(cxl_err, &ct3d->error_list, node) {
> + unc_err |= (1 << cxl_err->type);
> + }
> + if (!unc_err) {
> + return;
> + }
> +
> + stl_le_p(reg_state + R_CXL_RAS_UNC_ERR_STATUS, unc_err);
> + pcie_aer_inject_error(PCI_DEVICE(obj), &err);
> +
> + return;
> +}
> +
> +void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type,
> + Error **errp)
> +{
> + static PCIEAERErr err = {};
> + Object *obj = object_resolve_path(path, NULL);
> + CXLType3Dev *ct3d;
> + uint32_t *reg_state;
> + uint32_t cor_err;
> + int cxl_err_type;
> +
> + if (!obj) {
> + error_setg(errp, "Unable to resolve path");
> + return;
> + }
> + if (!object_dynamic_cast(obj, TYPE_CXL_TYPE3)) {
> + error_setg(errp, "Path does not point to a CXL type 3 device");
> + return;
> + }
> +
> + err.status = PCI_ERR_COR_INTERNAL;
> + err.source_id = pci_requester_id(PCI_DEVICE(obj));
> + err.flags = PCIE_AER_ERR_IS_CORRECTABLE;
> +
> + ct3d = CXL_TYPE3(obj);
> + reg_state = ct3d->cxl_cstate.crb.cache_mem_registers;
> + cor_err = ldl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS);
> +
> + cxl_err_type = ct3d_qmp_cor_err_to_cxl(type);
> + if (cxl_err_type < 0) {
> + error_setg(errp, "Invalid COR error");
> + return;
> + }
> + /* If the error is masked, nothting to do here */
> + if (!((1 << cxl_err_type) & ~ldl_le_p(reg_state + R_CXL_RAS_COR_ERR_MASK))) {
> + return;
> + }
> +
> + cor_err |= (1 << cxl_err_type);
> + stl_le_p(reg_state + R_CXL_RAS_COR_ERR_STATUS, cor_err);
> +
> + pcie_aer_inject_error(PCI_DEVICE(obj), &err);
> +}
> +
> static void ct3_class_init(ObjectClass *oc, void *data)
> {
> DeviceClass *dc = DEVICE_CLASS(oc);
> diff --git a/hw/mem/cxl_type3_stubs.c b/hw/mem/cxl_type3_stubs.c
> new file mode 100644
> index 0000000000..d574c58f9a
> --- /dev/null
> +++ b/hw/mem/cxl_type3_stubs.c
> @@ -0,0 +1,17 @@
> +
> +#include "qemu/osdep.h"
> +#include "qapi/error.h"
> +#include "qapi/qapi-commands-cxl.h"
> +
> +void qmp_cxl_inject_uncorrectable_errors(const char *path,
> + CXLUncorErrorRecordList *errors,
> + Error **errp)
> +{
> + error_setg(errp, "CXL Type 3 support is not compiled in");
> +}
> +
> +void qmp_cxl_inject_correctable_error(const char *path, CxlCorErrorType type,
> + Error **errp)
> +{
> + error_setg(errp, "CXL Type 3 support is not compiled in");
> +}
> diff --git a/hw/mem/meson.build b/hw/mem/meson.build
> index 609b2b36fc..56c2618b84 100644
> --- a/hw/mem/meson.build
> +++ b/hw/mem/meson.build
> @@ -4,6 +4,8 @@ mem_ss.add(when: 'CONFIG_DIMM', if_true: files('pc-dimm.c'))
> mem_ss.add(when: 'CONFIG_NPCM7XX', if_true: files('npcm7xx_mc.c'))
> mem_ss.add(when: 'CONFIG_NVDIMM', if_true: files('nvdimm.c'))
> mem_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_true: files('cxl_type3.c'))
> +softmmu_ss.add(when: 'CONFIG_CXL_MEM_DEVICE', if_false: files('cxl_type3_stubs.c'))
> +softmmu_ss.add(when: 'CONFIG_ALL', if_true: files('cxl_type3_stubs.c'))
>
> softmmu_ss.add_all(when: 'CONFIG_MEM_DEVICE', if_true: mem_ss)
>
> diff --git a/include/hw/cxl/cxl_component.h b/include/hw/cxl/cxl_component.h
> index 692d7a5507..ec4203b83f 100644
> --- a/include/hw/cxl/cxl_component.h
> +++ b/include/hw/cxl/cxl_component.h
> @@ -65,11 +65,37 @@ CXLx_CAPABILITY_HEADER(SNOOP, 0x14)
> #define CXL_RAS_REGISTERS_OFFSET 0x80
> #define CXL_RAS_REGISTERS_SIZE 0x58
> REG32(CXL_RAS_UNC_ERR_STATUS, CXL_RAS_REGISTERS_OFFSET)
> +#define CXL_RAS_UNC_ERR_CACHE_DATA_PARITY 0
> +#define CXL_RAS_UNC_ERR_CACHE_ADDRESS_PARITY 1
> +#define CXL_RAS_UNC_ERR_CACHE_BE_PARITY 2
> +#define CXL_RAS_UNC_ERR_CACHE_DATA_ECC 3
> +#define CXL_RAS_UNC_ERR_MEM_DATA_PARITY 4
> +#define CXL_RAS_UNC_ERR_MEM_ADDRESS_PARITY 5
> +#define CXL_RAS_UNC_ERR_MEM_BE_PARITY 6
> +#define CXL_RAS_UNC_ERR_MEM_DATA_ECC 7
> +#define CXL_RAS_UNC_ERR_REINIT_THRESHOLD 8
> +#define CXL_RAS_UNC_ERR_RSVD_ENCODING 9
> +#define CXL_RAS_UNC_ERR_POISON_RECEIVED 10
> +#define CXL_RAS_UNC_ERR_RECEIVER_OVERFLOW 11
> +#define CXL_RAS_UNC_ERR_INTERNAL 14
> +#define CXL_RAS_UNC_ERR_CXL_IDE_TX 15
> +#define CXL_RAS_UNC_ERR_CXL_IDE_RX 16
> +#define CXL_RAS_UNC_ERR_CXL_UNUSED 63 /* Magic value */
> REG32(CXL_RAS_UNC_ERR_MASK, CXL_RAS_REGISTERS_OFFSET + 0x4)
> REG32(CXL_RAS_UNC_ERR_SEVERITY, CXL_RAS_REGISTERS_OFFSET + 0x8)
> REG32(CXL_RAS_COR_ERR_STATUS, CXL_RAS_REGISTERS_OFFSET + 0xc)
> +#define CXL_RAS_COR_ERR_CACHE_DATA_ECC 0
> +#define CXL_RAS_COR_ERR_MEM_DATA_ECC 1
> +#define CXL_RAS_COR_ERR_CRC_THRESHOLD 2
> +#define CXL_RAS_COR_ERR_RETRY_THRESHOLD 3
> +#define CXL_RAS_COR_ERR_CACHE_POISON_RECEIVED 4
> +#define CXL_RAS_COR_ERR_MEM_POISON_RECEIVED 5
> +#define CXL_RAS_COR_ERR_PHYSICAL 6
> REG32(CXL_RAS_COR_ERR_MASK, CXL_RAS_REGISTERS_OFFSET + 0x10)
> REG32(CXL_RAS_ERR_CAP_CTRL, CXL_RAS_REGISTERS_OFFSET + 0x14)
> + FIELD(CXL_RAS_ERR_CAP_CTRL, FIRST_ERROR_POINTER, 0, 6)
> +REG32(CXL_RAS_ERR_HEADER0, CXL_RAS_REGISTERS_OFFSET + 0x18)
> +#define CXL_RAS_ERR_HEADER_NUM 32
> /* Offset 0x18 - 0x58 reserved for RAS logs */
>
> /* 8.2.5.10 - CXL Security Capability Structure */
> diff --git a/include/hw/cxl/cxl_device.h b/include/hw/cxl/cxl_device.h
> index 7e5ad65c1d..d589f78202 100644
> --- a/include/hw/cxl/cxl_device.h
> +++ b/include/hw/cxl/cxl_device.h
> @@ -232,6 +232,14 @@ REG64(CXL_MEM_DEV_STS, 0)
> FIELD(CXL_MEM_DEV_STS, MBOX_READY, 4, 1)
> FIELD(CXL_MEM_DEV_STS, RESET_NEEDED, 5, 3)
>
> +typedef struct CXLError {
> + QTAILQ_ENTRY(CXLError) node;
> + int type; /* Error code as per FE definition */
> + uint32_t header[32];
> +} CXLError;
> +
> +typedef QTAILQ_HEAD(, CXLError) CXLErrorList;
> +
> struct CXLType3Dev {
> /* Private */
> PCIDevice parent_obj;
> @@ -248,6 +256,9 @@ struct CXLType3Dev {
>
> /* DOE */
> DOECap doe_cdat;
> +
> + /* Error injection */
> + CXLErrorList error_list;
> };
>
> #define TYPE_CXL_TYPE3 "cxl-type3"
> diff --git a/qapi/cxl.json b/qapi/cxl.json
> new file mode 100644
> index 0000000000..4be7d46041
> --- /dev/null
> +++ b/qapi/cxl.json
> @@ -0,0 +1,128 @@
> +# -*- Mode: Python -*-
> +# vim: filetype=python
> +
> +##
> +# = CXL devices
> +##
> +
> +##
> +# @CxlUncorErrorType:
> +#
> +# Type of uncorrectable CXL error to inject. These errors are reported via
> +# an AER uncorrectable internal error with additional information logged at
> +# the CXL device.
> +#
> +# @cache-data-parity: Data error such as data parity or data ECC error CXL.cache
> +# @cache-address-parity: Address parity or other errors associated with the
> +# address field on CXL.cache
> +# @cache-be-parity: Byte enable parity or other byte enable errors on CXL.cache
> +# @cache-data-ecc: ECC error on CXL.cache
> +# @mem-data-parity: Data error such as data parity or data ECC error on CXL.mem
> +# @mem-address-parity: Address parity or other errors associated with the
> +# address field on CXL.mem
> +# @mem-be-parity: Byte enable parity or other byte enable errors on CXL.mem.
> +# @mem-data-ecc: Data ECC error on CXL.mem.
> +# @reinit-threshold: REINIT threshold hit.
> +# @rsvd-encoding: Received unrecognized encoding.
> +# @poison-received: Received poison from the peer.
> +# @receiver-overflow: Buffer overflows (first 3 bits of header log indicate which)
> +# @internal: Component specific error
> +# @cxl-ide-tx: Integrity and data encryption tx error.
> +# @cxl-ide-rx: Integrity and data encryption rx error.
> +#
> +# Since: 8.0
> +##
> +
> +{ 'enum': 'CxlUncorErrorType',
> + 'data': ['cache-data-parity',
> + 'cache-address-parity',
> + 'cache-be-parity',
> + 'cache-data-ecc',
> + 'mem-data-parity',
> + 'mem-address-parity',
> + 'mem-be-parity',
> + 'mem-data-ecc',
> + 'reinit-threshold',
> + 'rsvd-encoding',
> + 'poison-received',
> + 'receiver-overflow',
> + 'internal',
> + 'cxl-ide-tx',
> + 'cxl-ide-rx'
> + ]
> + }
> +
> +##
> +# @CXLUncorErrorRecord:
> +#
> +# Record of a single error including header log.
> +#
> +# @type: Type of error
> +# @header: 16 DWORD of header.
> +#
> +# Since: 8.0
> +##
> +{ 'struct': 'CXLUncorErrorRecord',
> + 'data': {
> + 'type': 'CxlUncorErrorType',
> + 'header': [ 'uint32' ]
> + }
> +}
> +
> +##
> +# @cxl-inject-uncorrectable-errors:
> +#
> +# Command to allow injection of multiple errors in one go. This allows testing
> +# of multiple header log handling in the OS.
> +#
> +# @path: CXL Type 3 device canonical QOM path
> +# @errors: Errors to inject
> +#
> +# Since: 8.0
> +##
> +{ 'command': 'cxl-inject-uncorrectable-errors',
> + 'data': { 'path': 'str',
> + 'errors': [ 'CXLUncorErrorRecord' ] }}
> +
> +##
> +# @CxlCorErrorType:
> +#
> +# Type of CXL correctable error to inject
> +#
> +# @cache-data-ecc: Data ECC error on CXL.cache
> +# @mem-data-ecc: Data ECC error on CXL.mem
> +# @crc-threshold: Component specific and applicable to 68 byte Flit mode only.
> +# @cache-poison-received: Received poison from a peer on CXL.cache.
> +# @mem-poison-received: Received poison from a peer on CXL.mem
> +# @physical: Received error indication from the physical layer.
> +#
> +# Since: 8.0
> +##
> +{ 'enum': 'CxlCorErrorType',
> + 'data': ['cache-data-ecc',
> + 'mem-data-ecc',
> + 'crc-threshold',
> + 'retry-threshold',
> + 'cache-poison-received',
> + 'mem-poison-received',
> + 'physical']
> +}
> +
> +##
> +# @cxl-inject-correctable-error:
> +#
> +# Command to inject a single correctable error. Multiple error injection
> +# of this error type is not interesting as there is no associated header log.
> +# These errors are reported via AER as a correctable internal error, with
> +# additional detail available from the CXL device.
> +#
> +# @path: CXL Type 3 device canonical QOM path
> +# @type: Type of error.
> +#
> +# Since: 8.0
> +##
> +{ 'command': 'cxl-inject-correctable-error',
> + 'data': { 'path': 'str',
> + 'type': 'CxlCorErrorType'
> + }
> +}
> diff --git a/qapi/meson.build b/qapi/meson.build
> index fbdb442fdf..73c3c8c31a 100644
> --- a/qapi/meson.build
> +++ b/qapi/meson.build
> @@ -31,6 +31,7 @@ qapi_all_modules = [
> 'compat',
> 'control',
> 'crypto',
> + 'cxl',
> 'dump',
> 'error',
> 'introspect',
> diff --git a/qapi/qapi-schema.json b/qapi/qapi-schema.json
> index f000b90744..079f2a402a 100644
> --- a/qapi/qapi-schema.json
> +++ b/qapi/qapi-schema.json
> @@ -95,3 +95,4 @@
> { 'include': 'pci.json' }
> { 'include': 'stats.json' }
> { 'include': 'virtio.json' }
> +{ 'include': 'cxl.json' }
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH v6 8/8] hw/mem/cxl_type3: Add CXL RAS Error Injection Support.
2023-03-02 10:06 ` Jonathan Cameron
(?)
@ 2023-03-02 10:49 ` Philippe Mathieu-Daudé
2023-03-02 10:55 ` Michael S. Tsirkin
-1 siblings, 1 reply; 24+ messages in thread
From: Philippe Mathieu-Daudé @ 2023-03-02 10:49 UTC (permalink / raw)
To: Jonathan Cameron, Jonathan Cameron via
Cc: Michael Tsirkin, Ben Widawsky, linux-cxl, linuxarm, Ira Weiny,
Gregory Price, Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth, Michael Roth
On 2/3/23 11:06, Jonathan Cameron wrote:
> +CC Michael Roth.
>
> Michael Tsirkin noted I'd missed CCing QAPI maintaintainers on the
> poison injection series so I went back to check the others I had out for review.
> For this series I'd managed to pick up one of two as Markus gave comments on
> earlier versions.
Now LGTM for meson/QAPI (although I'm not a QAPI maintainer).
Hopefully MST can take your series just in time for soft freeze.
> On Mon, 27 Feb 2023 11:27:51 +0000
> Jonathan Cameron via <qemu-devel@nongnu.org> wrote:
>
>> CXL uses PCI AER Internal errors to signal to the host that an error has
>> occurred. The host can then read more detailed status from the CXL RAS
>> capability.
>>
>> For uncorrectable errors: support multiple injection in one operation
>> as this is needed to reliably test multiple header logging support in an
>> OS. The equivalent feature doesn't exist for correctable errors, so only
>> one error need be injected at a time.
>>
>> Note:
>> - Header content needs to be manually specified in a fashion that
>> matches the specification for what can be in the header for each
>> error type.
>>
>> Injection via QMP:
>> { "execute": "qmp_capabilities" }
>> ...
>> { "execute": "cxl-inject-uncorrectable-errors",
>> "arguments": {
>> "path": "/machine/peripheral/cxl-pmem0",
>> "errors": [
>> {
>> "type": "cache-address-parity",
>> "header": [ 3, 4]
>> },
>> {
>> "type": "cache-data-parity",
>> "header": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]
>> },
>> {
>> "type": "internal",
>> "header": [ 1, 2, 4]
>> }
>> ]
>> }}
>> ...
>> { "execute": "cxl-inject-correctable-error",
>> "arguments": {
>> "path": "/machine/peripheral/cxl-pmem0",
>> "type": "physical"
>> } }
>>
>> Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
>> ---
>> v6: (Thanks to Philippe Mathieu-Daudé)
>> - Add Since entries in cxl.json
>> - Add error prints in the stub functions so that if they are called without
>> CONFIG_CXL_MEM_DEVICE then we get a useful print rather than just silently
>> eating them.
>> ---
>> hw/cxl/cxl-component-utils.c | 4 +-
>> hw/mem/cxl_type3.c | 281 +++++++++++++++++++++++++++++++++
>> hw/mem/cxl_type3_stubs.c | 17 ++
>> hw/mem/meson.build | 2 +
>> include/hw/cxl/cxl_component.h | 26 +++
>> include/hw/cxl/cxl_device.h | 11 ++
>> qapi/cxl.json | 128 +++++++++++++++
>> qapi/meson.build | 1 +
>> qapi/qapi-schema.json | 1 +
>> 9 files changed, 470 insertions(+), 1 deletion(-)
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH v6 8/8] hw/mem/cxl_type3: Add CXL RAS Error Injection Support.
2023-03-02 10:49 ` Philippe Mathieu-Daudé
@ 2023-03-02 10:55 ` Michael S. Tsirkin
2023-03-02 17:22 ` Jonathan Cameron via
0 siblings, 1 reply; 24+ messages in thread
From: Michael S. Tsirkin @ 2023-03-02 10:55 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Jonathan Cameron, qemu, Ben Widawsky, linux-cxl, linuxarm,
Ira Weiny, Gregory Price, Mike Maslenkin, Dave Jiang,
Markus Armbruster, Marc-André Lureau, Thomas Huth,
Michael Roth
On Thu, Mar 02, 2023 at 11:49:52AM +0100, Philippe Mathieu-Daudé wrote:
> On 2/3/23 11:06, Jonathan Cameron wrote:
> > +CC Michael Roth.
> >
> > Michael Tsirkin noted I'd missed CCing QAPI maintaintainers on the
> > poison injection series so I went back to check the others I had out for review.
> > For this series I'd managed to pick up one of two as Markus gave comments on
> > earlier versions.
>
> Now LGTM for meson/QAPI (although I'm not a QAPI maintainer).
>
> Hopefully MST can take your series just in time for soft freeze.
Bottleneck is QAPI right now. Consider CC Eric and maybe Daniel
because Daniel is awesome and has been helping out with lots
of stuff among them QAPI.
> > On Mon, 27 Feb 2023 11:27:51 +0000
> > Jonathan Cameron via <qemu-devel@nongnu.org> wrote:
> >
> > > CXL uses PCI AER Internal errors to signal to the host that an error has
> > > occurred. The host can then read more detailed status from the CXL RAS
> > > capability.
> > >
> > > For uncorrectable errors: support multiple injection in one operation
> > > as this is needed to reliably test multiple header logging support in an
> > > OS. The equivalent feature doesn't exist for correctable errors, so only
> > > one error need be injected at a time.
> > >
> > > Note:
> > > - Header content needs to be manually specified in a fashion that
> > > matches the specification for what can be in the header for each
> > > error type.
> > >
> > > Injection via QMP:
> > > { "execute": "qmp_capabilities" }
> > > ...
> > > { "execute": "cxl-inject-uncorrectable-errors",
> > > "arguments": {
> > > "path": "/machine/peripheral/cxl-pmem0",
> > > "errors": [
> > > {
> > > "type": "cache-address-parity",
> > > "header": [ 3, 4]
> > > },
> > > {
> > > "type": "cache-data-parity",
> > > "header": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]
> > > },
> > > {
> > > "type": "internal",
> > > "header": [ 1, 2, 4]
> > > }
> > > ]
> > > }}
> > > ...
> > > { "execute": "cxl-inject-correctable-error",
> > > "arguments": {
> > > "path": "/machine/peripheral/cxl-pmem0",
> > > "type": "physical"
> > > } }
> > >
> > > Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > > ---
> > > v6: (Thanks to Philippe Mathieu-Daudé)
> > > - Add Since entries in cxl.json
> > > - Add error prints in the stub functions so that if they are called without
> > > CONFIG_CXL_MEM_DEVICE then we get a useful print rather than just silently
> > > eating them.
> > > ---
> > > hw/cxl/cxl-component-utils.c | 4 +-
> > > hw/mem/cxl_type3.c | 281 +++++++++++++++++++++++++++++++++
> > > hw/mem/cxl_type3_stubs.c | 17 ++
> > > hw/mem/meson.build | 2 +
> > > include/hw/cxl/cxl_component.h | 26 +++
> > > include/hw/cxl/cxl_device.h | 11 ++
> > > qapi/cxl.json | 128 +++++++++++++++
> > > qapi/meson.build | 1 +
> > > qapi/qapi-schema.json | 1 +
> > > 9 files changed, 470 insertions(+), 1 deletion(-)
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH v6 8/8] hw/mem/cxl_type3: Add CXL RAS Error Injection Support.
2023-03-02 10:55 ` Michael S. Tsirkin
@ 2023-03-02 17:22 ` Jonathan Cameron via
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron @ 2023-03-02 17:22 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Philippe Mathieu-Daudé,
qemu, Ben Widawsky, linux-cxl, linuxarm, Ira Weiny,
Gregory Price, Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth, Michael Roth
On Thu, 2 Mar 2023 05:55:00 -0500
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Mar 02, 2023 at 11:49:52AM +0100, Philippe Mathieu-Daudé wrote:
> > On 2/3/23 11:06, Jonathan Cameron wrote:
> > > +CC Michael Roth.
> > >
> > > Michael Tsirkin noted I'd missed CCing QAPI maintaintainers on the
> > > poison injection series so I went back to check the others I had out for review.
> > > For this series I'd managed to pick up one of two as Markus gave comments on
> > > earlier versions.
> >
> > Now LGTM for meson/QAPI (although I'm not a QAPI maintainer).
> >
> > Hopefully MST can take your series just in time for soft freeze.
>
>
> Bottleneck is QAPI right now. Consider CC Eric and maybe Daniel
> because Daniel is awesome and has been helping out with lots
> of stuff among them QAPI.
Thanks for the tip. Resent with additional CCs (hopefully to right
people!) and a note that we are looking for qapi review.
With the benefit of hindsight I'd have ordered the series differently
if I'd guessed we'd bottleneck here.
Jonathan
>
> > > On Mon, 27 Feb 2023 11:27:51 +0000
> > > Jonathan Cameron via <qemu-devel@nongnu.org> wrote:
> > >
> > > > CXL uses PCI AER Internal errors to signal to the host that an error has
> > > > occurred. The host can then read more detailed status from the CXL RAS
> > > > capability.
> > > >
> > > > For uncorrectable errors: support multiple injection in one operation
> > > > as this is needed to reliably test multiple header logging support in an
> > > > OS. The equivalent feature doesn't exist for correctable errors, so only
> > > > one error need be injected at a time.
> > > >
> > > > Note:
> > > > - Header content needs to be manually specified in a fashion that
> > > > matches the specification for what can be in the header for each
> > > > error type.
> > > >
> > > > Injection via QMP:
> > > > { "execute": "qmp_capabilities" }
> > > > ...
> > > > { "execute": "cxl-inject-uncorrectable-errors",
> > > > "arguments": {
> > > > "path": "/machine/peripheral/cxl-pmem0",
> > > > "errors": [
> > > > {
> > > > "type": "cache-address-parity",
> > > > "header": [ 3, 4]
> > > > },
> > > > {
> > > > "type": "cache-data-parity",
> > > > "header": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]
> > > > },
> > > > {
> > > > "type": "internal",
> > > > "header": [ 1, 2, 4]
> > > > }
> > > > ]
> > > > }}
> > > > ...
> > > > { "execute": "cxl-inject-correctable-error",
> > > > "arguments": {
> > > > "path": "/machine/peripheral/cxl-pmem0",
> > > > "type": "physical"
> > > > } }
> > > >
> > > > Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > > > ---
> > > > v6: (Thanks to Philippe Mathieu-Daudé)
> > > > - Add Since entries in cxl.json
> > > > - Add error prints in the stub functions so that if they are called without
> > > > CONFIG_CXL_MEM_DEVICE then we get a useful print rather than just silently
> > > > eating them.
> > > > ---
> > > > hw/cxl/cxl-component-utils.c | 4 +-
> > > > hw/mem/cxl_type3.c | 281 +++++++++++++++++++++++++++++++++
> > > > hw/mem/cxl_type3_stubs.c | 17 ++
> > > > hw/mem/meson.build | 2 +
> > > > include/hw/cxl/cxl_component.h | 26 +++
> > > > include/hw/cxl/cxl_device.h | 11 ++
> > > > qapi/cxl.json | 128 +++++++++++++++
> > > > qapi/meson.build | 1 +
> > > > qapi/qapi-schema.json | 1 +
> > > > 9 files changed, 470 insertions(+), 1 deletion(-)
>
^ permalink raw reply [flat|nested] 24+ messages in thread* Re: [PATCH v6 8/8] hw/mem/cxl_type3: Add CXL RAS Error Injection Support.
@ 2023-03-02 17:22 ` Jonathan Cameron via
0 siblings, 0 replies; 24+ messages in thread
From: Jonathan Cameron via @ 2023-03-02 17:22 UTC (permalink / raw)
To: Michael S. Tsirkin
Cc: Philippe Mathieu-Daudé,
qemu, Ben Widawsky, linux-cxl, linuxarm, Ira Weiny,
Gregory Price, Mike Maslenkin, Dave Jiang, Markus Armbruster,
Marc-André Lureau, Thomas Huth, Michael Roth
On Thu, 2 Mar 2023 05:55:00 -0500
"Michael S. Tsirkin" <mst@redhat.com> wrote:
> On Thu, Mar 02, 2023 at 11:49:52AM +0100, Philippe Mathieu-Daudé wrote:
> > On 2/3/23 11:06, Jonathan Cameron wrote:
> > > +CC Michael Roth.
> > >
> > > Michael Tsirkin noted I'd missed CCing QAPI maintaintainers on the
> > > poison injection series so I went back to check the others I had out for review.
> > > For this series I'd managed to pick up one of two as Markus gave comments on
> > > earlier versions.
> >
> > Now LGTM for meson/QAPI (although I'm not a QAPI maintainer).
> >
> > Hopefully MST can take your series just in time for soft freeze.
>
>
> Bottleneck is QAPI right now. Consider CC Eric and maybe Daniel
> because Daniel is awesome and has been helping out with lots
> of stuff among them QAPI.
Thanks for the tip. Resent with additional CCs (hopefully to right
people!) and a note that we are looking for qapi review.
With the benefit of hindsight I'd have ordered the series differently
if I'd guessed we'd bottleneck here.
Jonathan
>
> > > On Mon, 27 Feb 2023 11:27:51 +0000
> > > Jonathan Cameron via <qemu-devel@nongnu.org> wrote:
> > >
> > > > CXL uses PCI AER Internal errors to signal to the host that an error has
> > > > occurred. The host can then read more detailed status from the CXL RAS
> > > > capability.
> > > >
> > > > For uncorrectable errors: support multiple injection in one operation
> > > > as this is needed to reliably test multiple header logging support in an
> > > > OS. The equivalent feature doesn't exist for correctable errors, so only
> > > > one error need be injected at a time.
> > > >
> > > > Note:
> > > > - Header content needs to be manually specified in a fashion that
> > > > matches the specification for what can be in the header for each
> > > > error type.
> > > >
> > > > Injection via QMP:
> > > > { "execute": "qmp_capabilities" }
> > > > ...
> > > > { "execute": "cxl-inject-uncorrectable-errors",
> > > > "arguments": {
> > > > "path": "/machine/peripheral/cxl-pmem0",
> > > > "errors": [
> > > > {
> > > > "type": "cache-address-parity",
> > > > "header": [ 3, 4]
> > > > },
> > > > {
> > > > "type": "cache-data-parity",
> > > > "header": [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]
> > > > },
> > > > {
> > > > "type": "internal",
> > > > "header": [ 1, 2, 4]
> > > > }
> > > > ]
> > > > }}
> > > > ...
> > > > { "execute": "cxl-inject-correctable-error",
> > > > "arguments": {
> > > > "path": "/machine/peripheral/cxl-pmem0",
> > > > "type": "physical"
> > > > } }
> > > >
> > > > Signed-off-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
> > > > ---
> > > > v6: (Thanks to Philippe Mathieu-Daudé)
> > > > - Add Since entries in cxl.json
> > > > - Add error prints in the stub functions so that if they are called without
> > > > CONFIG_CXL_MEM_DEVICE then we get a useful print rather than just silently
> > > > eating them.
> > > > ---
> > > > hw/cxl/cxl-component-utils.c | 4 +-
> > > > hw/mem/cxl_type3.c | 281 +++++++++++++++++++++++++++++++++
> > > > hw/mem/cxl_type3_stubs.c | 17 ++
> > > > hw/mem/meson.build | 2 +
> > > > include/hw/cxl/cxl_component.h | 26 +++
> > > > include/hw/cxl/cxl_device.h | 11 ++
> > > > qapi/cxl.json | 128 +++++++++++++++
> > > > qapi/meson.build | 1 +
> > > > qapi/qapi-schema.json | 1 +
> > > > 9 files changed, 470 insertions(+), 1 deletion(-)
>
^ permalink raw reply [flat|nested] 24+ messages in thread