[PATCH v1 0/3] Support PCI Error Injection

[PATCH v1 0/3] Support PCI Error Injection

[Gavin Shan]

The series of patches supports PCI error injection as defined in PAPR spec.
There are 3 RTAS calls related to it:

"ibm,open-errinjct": Apply for token to do error injection
"ibm,close-errinjct": Free the token assigned previously
"ibm,errinjct": Do error injection

For PCI errors, we need manupulate 3 hardware registers by predetermined
OPAL API. In order to export the capability to userland, one OPAL firmware
related sysfs entries are created: /sys/firmware/opal/errinjct. It accepts
strings to call OPAL API to inject specific errors. So it corresponds to
"ibm,errinjct". So "ibm,open-errinjct" and "ibm,close-errinjct" won't be
handled by host kernel and applications have full freedom to implement logic
of themselves for them.

Gavin Shan (3):
  powerpc/powernv: Sync header with firmware
  powerpc/powernv: Support PCI error injection
  powerpc/powernv: Clear PAPR error injection registers

 arch/powerpc/include/asm/opal.h                |  66 +++++++++
 arch/powerpc/platforms/powernv/Makefile        |   2 +-
 arch/powerpc/platforms/powernv/eeh-ioda.c      |  24 ++++
 arch/powerpc/platforms/powernv/opal-errinjct.c | 184 +++++++++++++++++++++++++
 arch/powerpc/platforms/powernv/opal-wrappers.S |   1 +
 arch/powerpc/platforms/powernv/opal.c          |   2 +
 6 files changed, 278 insertions(+), 1 deletion(-)
 create mode 100644 arch/powerpc/platforms/powernv/opal-errinjct.c

-- 
1.8.3.2

[PATCH v1 1/3] powerpc/powernv: Sync header with firmware

[Gavin Shan]

The patch synchronizes firmware header file (opal.h) for PCI error
injection.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/opal.h                | 65 ++++++++++++++++++++++++++
 arch/powerpc/platforms/powernv/opal-wrappers.S |  1 +
 2 files changed, 66 insertions(+)

diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index 66ad7a7..d982bb8 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -175,6 +175,7 @@ extern int opal_enter_rtas(struct rtas_args *args,
 #define OPAL_SET_PARAM				90
 #define OPAL_DUMP_RESEND			91
 #define OPAL_DUMP_INFO2				94
+#define OPAL_ERR_INJECT				96
 
 #ifndef __ASSEMBLY__
 
@@ -219,6 +220,69 @@ enum OpalPciErrorSeverity {
 	OPAL_EEH_SEV_INF	= 5
 };
 
+enum OpalErrinjctType {
+	OpalErrinjctTypeFirst			= 0,
+	OpalErrinjctTypeFatal			= 1,
+	OpalErrinjctTypeRecoverRandomEvent	= 2,
+	OpalErrinjctTypeRecoverSpecialEvent	= 3,
+	OpalErrinjctTypeCorruptedPage		= 4,
+	OpalErrinjctTypeCorruptedSlb		= 5,
+	OpalErrinjctTypeTranslatorFailure	= 6,
+	OpalErrinjctTypeIoaBusError		= 7,
+	OpalErrinjctTypeIoaBusError64		= 8,
+	OpalErrinjctTypePlatformSpecific	= 9,
+	OpalErrinjctTypeDcacheStart		= 10,
+	OpalErrinjctTypeDcacheEnd		= 11,
+	OpalErrinjctTypeIcacheStart		= 12,
+	OpalErrinjctTypeIcacheEnd		= 13,
+	OpalErrinjctTypeTlbStart		= 14,
+	OpalErrinjctTypeTlbEnd			= 15,
+	OpalErrinjctTypeUpstreamIoError		= 16,
+	OpalErrinjctTypeLast			= 17,
+
+	/* IoaBusError & IoaBusError64 */
+	OpalEitIoaLoadMemAddr			= 0,
+	OpalEitIoaLoadMemData			= 1,
+	OpalEitIoaLoadIoAddr			= 2,
+	OpalEitIoaLoadIoData			= 3,
+	OpalEitIoaLoadConfigAddr		= 4,
+	OpalEitIoaLoadConfigData		= 5,
+	OpalEitIoaStoreMemAddr			= 6,
+	OpalEitIoaStoreMemData			= 7,
+	OpalEitIoaStoreIoAddr			= 8,
+	OpalEitIoaStoreIoData			= 9,
+	OpalEitIoaStoreConfigAddr		= 10,
+	OpalEitIoaStoreConfigData		= 11,
+	OpalEitIoaDmaReadMemAddr		= 12,
+	OpalEitIoaDmaReadMemData		= 13,
+	OpalEitIoaDmaReadMemMaster		= 14,
+	OpalEitIoaDmaReadMemTarget		= 15,
+	OpalEitIoaDmaWriteMemAddr		= 16,
+	OpalEitIoaDmaWriteMemData		= 17,
+	OpalEitIoaDmaWriteMemMaster		= 18,
+	OpalEitIoaDmaWriteMemTarget		= 19,
+};
+
+struct OpalErrinjct {
+	int32_t type;
+	union {
+		struct {
+			uint32_t addr;
+			uint32_t mask;
+			uint64_t phb_id;
+			uint32_t pe;
+			uint32_t function;
+		} ioa;
+		struct {
+			uint64_t addr;
+			uint64_t mask;
+			uint64_t phb_id;
+			uint32_t pe;
+			uint32_t function;
+		} ioa64;
+	};
+};
+
 enum OpalShpcAction {
 	OPAL_SHPC_GET_LINK_STATE = 0,
 	OPAL_SHPC_GET_SLOT_STATE = 1
@@ -870,6 +934,7 @@ int64_t opal_update_flash(uint64_t blk_list);
 int64_t opal_dump_init(uint8_t dump_type);
 int64_t opal_dump_info(__be32 *dump_id, __be32 *dump_size);
 int64_t opal_dump_info2(__be32 *dump_id, __be32 *dump_size, __be32 *dump_type);
+int64_t opal_err_injct(struct OpalErrinjct *ei);
 int64_t opal_dump_read(uint32_t dump_id, uint64_t buffer);
 int64_t opal_dump_ack(uint32_t dump_id);
 int64_t opal_dump_resend_notification(void);
diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
index f531ffe..44b3d81 100644
--- a/arch/powerpc/platforms/powernv/opal-wrappers.S
+++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
@@ -136,6 +136,7 @@ OPAL_CALL(opal_resync_timebase,			OPAL_RESYNC_TIMEBASE);
 OPAL_CALL(opal_dump_init,			OPAL_DUMP_INIT);
 OPAL_CALL(opal_dump_info,			OPAL_DUMP_INFO);
 OPAL_CALL(opal_dump_info2,			OPAL_DUMP_INFO2);
+OPAL_CALL(opal_err_injct,			OPAL_ERR_INJECT);
 OPAL_CALL(opal_dump_read,			OPAL_DUMP_READ);
 OPAL_CALL(opal_dump_ack,			OPAL_DUMP_ACK);
 OPAL_CALL(opal_get_msg,				OPAL_GET_MSG);
-- 
1.8.3.2

[PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Gavin Shan]

The patch implements one OPAL firmware sysfs file to support PCI error
injection: "/sys/firmware/opal/errinjct", which will be used like the
way described as follows.

According to PAPR spec, there are 3 RTAS calls related to error injection:
"ibm,open-errinjct": allocate token prior to doing error injection.
"ibm,close-errinjct": release the token allocated from "ibm,open-errinjct".
"ibm,errinjct": do error injection.

Sysfs file /sys/firmware/opal/errinjct accepts strings that have fixed
format "ei_token ...". For now, we only support 32-bits and 64-bits
PCI error injection and they should have following strings written to
/sys/firmware/opal/errinjct as follows. We don't have corresponding
sysfs files for "ibm,open-errinjct" and "ibm,close-errinjct", which
means that we rely on userland to maintain the token by itself.

32-bits PCI error: "7:addr:mask:iommu_group_id:function".
64-bits PCI error: "8:addr:mask:iommu_group_id:function".

The above "7" and "8" represent 32-bits and 64-bits PCI error seperately
and "function" is one of the specific PCI errors (e.g. MMIO access address
parity error), which are defined by PAPR spec.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 arch/powerpc/include/asm/opal.h                |   1 +
 arch/powerpc/platforms/powernv/Makefile        |   2 +-
 arch/powerpc/platforms/powernv/opal-errinjct.c | 184 +++++++++++++++++++++++++
 arch/powerpc/platforms/powernv/opal.c          |   2 +
 4 files changed, 188 insertions(+), 1 deletion(-)
 create mode 100644 arch/powerpc/platforms/powernv/opal-errinjct.c

diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index d982bb8..bf280d9 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -985,6 +985,7 @@ extern int opal_elog_init(void);
 extern void opal_platform_dump_init(void);
 extern void opal_sys_param_init(void);
 extern void opal_msglog_init(void);
+extern void opal_errinjct_init(void);
 
 extern int opal_machine_check(struct pt_regs *regs);
 extern bool opal_mce_check_early_recovery(struct pt_regs *regs);
diff --git a/arch/powerpc/platforms/powernv/Makefile b/arch/powerpc/platforms/powernv/Makefile
index 63cebb9..4711de8 100644
--- a/arch/powerpc/platforms/powernv/Makefile
+++ b/arch/powerpc/platforms/powernv/Makefile
@@ -1,7 +1,7 @@
 obj-y			+= setup.o opal-takeover.o opal-wrappers.o opal.o opal-async.o
 obj-y			+= opal-rtc.o opal-nvram.o opal-lpc.o opal-flash.o
 obj-y			+= rng.o opal-elog.o opal-dump.o opal-sysparam.o opal-sensor.o
-obj-y			+= opal-msglog.o
+obj-y			+= opal-msglog.o opal-errinjct.o
 
 obj-$(CONFIG_SMP)	+= smp.o
 obj-$(CONFIG_PCI)	+= pci.o pci-p5ioc2.o pci-ioda.o
diff --git a/arch/powerpc/platforms/powernv/opal-errinjct.c b/arch/powerpc/platforms/powernv/opal-errinjct.c
new file mode 100644
index 0000000..29c9e83
--- /dev/null
+++ b/arch/powerpc/platforms/powernv/opal-errinjct.c
@@ -0,0 +1,184 @@
+/*
+ * The file supports error injection, which works based on OPAL API.
+ * For now, we only support PCI error injection. We need support
+ * injecting other types of errors in future.
+ *
+ * Copyright Gavin Shan, IBM Corporation 2014.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; either version 2 of the License, or
+ * (at your option) any later version.
+ */
+
+#include <linux/kernel.h>
+#include <linux/init.h>
+#include <linux/msi.h>
+#include <linux/pci.h>
+#include <linux/iommu.h>
+#include <linux/random.h>
+#include <linux/slab.h>
+#include <linux/sysfs.h>
+#include <linux/fs.h>
+#include <linux/fcntl.h>
+#include <linux/kobject.h>
+
+#include <asm/msi_bitmap.h>
+#include <asm/iommu.h>
+#include <asm/pci-bridge.h>
+#include <asm/ppc-pci.h>
+#include <asm/opal.h>
+
+#include "powernv.h"
+#include "pci.h"
+
+static DEFINE_MUTEX(errinjct_mutex);
+
+static int errinjct_iommu_group_to_phb_and_pe(uint32_t iommu_grp_id,
+					      uint64_t *phb_id,
+					      uint32_t *pe_num)
+{
+#ifdef CONFIG_IOMMU_API
+	struct iommu_group *iommu_grp;
+	struct iommu_table *tbl;
+	struct pnv_ioda_pe *pe;
+
+	iommu_grp = iommu_group_get_by_id(iommu_grp_id);
+	if (!iommu_grp)
+		return -ENODEV;
+
+	tbl = iommu_group_get_iommudata(iommu_grp);
+	if (!tbl)
+		return -ENODEV;
+
+	pe = container_of(tbl, struct pnv_ioda_pe, tce32_table);
+	if (!pe->phb)
+		return -ENODEV;
+
+	*phb_id = pe->phb->opal_id;
+	*pe_num = pe->pe_number;
+
+	return 0;
+#endif
+
+	return -ENXIO;
+}
+
+static int errinjct_ioa_bus_error(const char *buf, struct OpalErrinjct *ei)
+{
+	uint32_t iommu_grp_id;
+	int ret;
+
+	/* Extract parameters */
+	ret = sscanf(buf, "%x:%x:%x:%x:%x",
+		     &ei->type, &ei->ioa.addr,
+		     &ei->ioa.mask, &iommu_grp_id, ei->ioa.function);
+	if (ret != 5)
+		return -EINVAL;
+
+	/* Invalid function ? */
+	if (ei->ioa.function < OpalEitIoaLoadMemAddr ||
+	    ei->ioa.function > OpalEitIoaDmaWriteMemTarget)
+		return -ERANGE;
+
+	/* Retrieve PHB ID and PE number */
+	ret = errinjct_iommu_group_to_phb_and_pe(iommu_grp_id,
+						 &ei->ioa.phb_id,
+						 &ei->ioa.pe);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int errinjct_ioa_bus_error64(const char *buf, struct OpalErrinjct *ei)
+{
+	uint32_t iommu_grp_id;
+	int ret;
+
+	/* Extract parameter */
+	ret = sscanf(buf, "%x:%llx:%llx:%x:%x",
+		     &ei->type, &ei->ioa64.addr,
+		     &ei->ioa64.mask, &iommu_grp_id, &ei->ioa64.function);
+	if (ret != 5)
+		return -EINVAL;
+
+	/* Invalid function ? */
+	if (ei->ioa64.function < OpalEitIoaLoadMemAddr ||
+	    ei->ioa64.function > OpalEitIoaDmaWriteMemTarget)
+		return -ERANGE;
+
+	/* Retrieve PHB ID and PE number */
+	ret = errinjct_iommu_group_to_phb_and_pe(iommu_grp_id,
+						 &ei->ioa64.phb_id,
+						 &ei->ioa64.pe);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static ssize_t errinjct_store(struct kobject *kobj,
+			      struct kobj_attribute *attr,
+			      const char *buf, size_t count)
+{
+	struct OpalErrinjct ei;
+	int ret;
+	long rc;
+
+	/* Extract common parameters */
+	ret = sscanf(buf, "%x", &ei.type);
+	if (ret != 1)
+		return -EINVAL;
+
+	/* Error injection might be in progress */
+	if (!mutex_trylock(&errinjct_mutex))
+		return -EAGAIN;
+
+	switch (ei.type) {
+	case OpalErrinjctTypeIoaBusError:
+		ret = errinjct_ioa_bus_error(buf, &ei);
+		break;
+	case OpalErrinjctTypeIoaBusError64:
+		ret = errinjct_ioa_bus_error64(buf, &ei);
+		break;
+	default:
+		ret = -ERANGE;
+	}
+
+	/* Invalid parameters ? */
+	if (ret)
+		goto mutex_unlock_exit;
+
+	/* OPAL call */
+	rc = opal_err_injct(&ei);
+	if (rc == OPAL_SUCCESS)
+		ret = count;
+	else
+		ret = -EIO;
+
+mutex_unlock_exit:
+	mutex_unlock(&errinjct_mutex);
+	return ret;
+}
+
+static struct kobj_attribute errinjct_attr =
+	__ATTR(errinjct, 0600, NULL, errinjct_store);
+
+void __init opal_errinjct_init(void)
+{
+	int ret;
+
+	/* Make sure /sys/firmware/opal directory is created */
+	if (!opal_kobj) {
+		pr_warn("%s: opal kobject is not available\n",
+			__func__);
+		return;
+	}
+
+	/* Create the sysfs files */
+	ret = sysfs_create_file(opal_kobj, &errinjct_attr.attr);
+	if (ret)
+		pr_warn("%s: Cannot create sysfs file (%d)\n",
+			__func__, ret);
+}
diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c
index 360ad80c..cb29bb5 100644
--- a/arch/powerpc/platforms/powernv/opal.c
+++ b/arch/powerpc/platforms/powernv/opal.c
@@ -604,6 +604,8 @@ static int __init opal_init(void)
 		opal_sys_param_init();
 		/* Setup message log interface. */
 		opal_msglog_init();
+		/* Setup error injection interface */
+		opal_errinjct_init();
 	}
 
 	return 0;
-- 
1.8.3.2

[PATCH v1 3/3] powerpc/powernv: Clear PAPR error injection registers

[Gavin Shan]

The frozen state on one specific PE is probably caused by error
injection, which is done with help of PAPR error injection registers.
According to the hardware spec, those registers should be cleared
automatically after one-shot frozen PE. However, that's not always
true, at least on P7IOC of Firebird-L. So we have to clear them
before doing PE reset to avoid recursive EEH errors at recovery
stage.

Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/powernv/eeh-ioda.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/arch/powerpc/platforms/powernv/eeh-ioda.c b/arch/powerpc/platforms/powernv/eeh-ioda.c
index 79193eb..cbc7c98 100644
--- a/arch/powerpc/platforms/powernv/eeh-ioda.c
+++ b/arch/powerpc/platforms/powernv/eeh-ioda.c
@@ -657,6 +657,30 @@ static int ioda_eeh_reset(struct eeh_pe *pe, int option)
 	if (pe->type & EEH_PE_PHB) {
 		ret = ioda_eeh_phb_reset(hose, option);
 	} else {
+		struct pnv_phb *phb;
+		s64 rc;
+
+		/*
+		 * The frozen PE might be caused by PAPR error injection
+		 * registers, which are expected to be cleared after hitting
+		 * frozen PE as stated in the hardware spec. Unfortunately,
+		 * that's not true on P7IOC. So we have to clear it manually
+		 * to avoid recursive EEH errors during recovery.
+		 */
+		phb = hose->private_data;
+		if (phb->model == PNV_PHB_MODEL_P7IOC &&
+		    (option == EEH_RESET_HOT ||
+		    option == EEH_RESET_FUNDAMENTAL)) {
+			rc = opal_pci_reset(phb->opal_id,
+					    OPAL_PHB_ERROR,
+					    OPAL_ASSERT_RESET);
+			if (rc != OPAL_SUCCESS) {
+				pr_warn("%s: Can't clear errinjct reg (%lld)\n",
+					__func__, rc);
+				return -EIO;
+			}
+		}
+
 		bus = eeh_pe_bus_get(pe);
 		if (pci_is_root_bus(bus) ||
 		    pci_is_root_bus(bus->parent))
-- 
1.8.3.2

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Michael Neuling]

On Mon, 2014-06-23 at 12:14 +1000, Gavin Shan wrote:
> The patch implements one OPAL firmware sysfs file to support PCI error
> injection: "/sys/firmware/opal/errinjct", which will be used like the
> way described as follows.
>=20
> According to PAPR spec, there are 3 RTAS calls related to error injection=
:
> "ibm,open-errinjct": allocate token prior to doing error injection.
> "ibm,close-errinjct": release the token allocated from "ibm,open-errinjct=
".
> "ibm,errinjct": do error injection.
>=20
> Sysfs file /sys/firmware/opal/errinjct accepts strings that have fixed
> format "ei_token ...". For now, we only support 32-bits and 64-bits
> PCI error injection and they should have following strings written to
> /sys/firmware/opal/errinjct as follows. We don't have corresponding
> sysfs files for "ibm,open-errinjct" and "ibm,close-errinjct", which
> means that we rely on userland to maintain the token by itself.

This sounds cool. =20

Can you document the sysfs interface in Documentation/powerpc?

Mikey

>=20
> 32-bits PCI error: "7:addr:mask:iommu_group_id:function".
> 64-bits PCI error: "8:addr:mask:iommu_group_id:function".
>=20
> The above "7" and "8" represent 32-bits and 64-bits PCI error seperately
> and "function" is one of the specific PCI errors (e.g. MMIO access addres=
s
> parity error), which are defined by PAPR spec.
>=20
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  arch/powerpc/include/asm/opal.h                |   1 +
>  arch/powerpc/platforms/powernv/Makefile        |   2 +-
>  arch/powerpc/platforms/powernv/opal-errinjct.c | 184 +++++++++++++++++++=
++++++
>  arch/powerpc/platforms/powernv/opal.c          |   2 +
>  4 files changed, 188 insertions(+), 1 deletion(-)
>  create mode 100644 arch/powerpc/platforms/powernv/opal-errinjct.c
>=20
> diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/o=
pal.h
> index d982bb8..bf280d9 100644
> --- a/arch/powerpc/include/asm/opal.h
> +++ b/arch/powerpc/include/asm/opal.h
> @@ -985,6 +985,7 @@ extern int opal_elog_init(void);
>  extern void opal_platform_dump_init(void);
>  extern void opal_sys_param_init(void);
>  extern void opal_msglog_init(void);
> +extern void opal_errinjct_init(void);
> =20
>  extern int opal_machine_check(struct pt_regs *regs);
>  extern bool opal_mce_check_early_recovery(struct pt_regs *regs);
> diff --git a/arch/powerpc/platforms/powernv/Makefile b/arch/powerpc/platf=
orms/powernv/Makefile
> index 63cebb9..4711de8 100644
> --- a/arch/powerpc/platforms/powernv/Makefile
> +++ b/arch/powerpc/platforms/powernv/Makefile
> @@ -1,7 +1,7 @@
>  obj-y			+=3D setup.o opal-takeover.o opal-wrappers.o opal.o opal-async.o
>  obj-y			+=3D opal-rtc.o opal-nvram.o opal-lpc.o opal-flash.o
>  obj-y			+=3D rng.o opal-elog.o opal-dump.o opal-sysparam.o opal-sensor.o
> -obj-y			+=3D opal-msglog.o
> +obj-y			+=3D opal-msglog.o opal-errinjct.o
> =20
>  obj-$(CONFIG_SMP)	+=3D smp.o
>  obj-$(CONFIG_PCI)	+=3D pci.o pci-p5ioc2.o pci-ioda.o
> diff --git a/arch/powerpc/platforms/powernv/opal-errinjct.c b/arch/powerp=
c/platforms/powernv/opal-errinjct.c
> new file mode 100644
> index 0000000..29c9e83
> --- /dev/null
> +++ b/arch/powerpc/platforms/powernv/opal-errinjct.c
> @@ -0,0 +1,184 @@
> +/*
> + * The file supports error injection, which works based on OPAL API.
> + * For now, we only support PCI error injection. We need support
> + * injecting other types of errors in future.
> + *
> + * Copyright Gavin Shan, IBM Corporation 2014.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/init.h>
> +#include <linux/msi.h>
> +#include <linux/pci.h>
> +#include <linux/iommu.h>
> +#include <linux/random.h>
> +#include <linux/slab.h>
> +#include <linux/sysfs.h>
> +#include <linux/fs.h>
> +#include <linux/fcntl.h>
> +#include <linux/kobject.h>
> +
> +#include <asm/msi_bitmap.h>
> +#include <asm/iommu.h>
> +#include <asm/pci-bridge.h>
> +#include <asm/ppc-pci.h>
> +#include <asm/opal.h>
> +
> +#include "powernv.h"
> +#include "pci.h"
> +
> +static DEFINE_MUTEX(errinjct_mutex);
> +
> +static int errinjct_iommu_group_to_phb_and_pe(uint32_t iommu_grp_id,
> +					      uint64_t *phb_id,
> +					      uint32_t *pe_num)
> +{
> +#ifdef CONFIG_IOMMU_API
> +	struct iommu_group *iommu_grp;
> +	struct iommu_table *tbl;
> +	struct pnv_ioda_pe *pe;
> +
> +	iommu_grp =3D iommu_group_get_by_id(iommu_grp_id);
> +	if (!iommu_grp)
> +		return -ENODEV;
> +
> +	tbl =3D iommu_group_get_iommudata(iommu_grp);
> +	if (!tbl)
> +		return -ENODEV;
> +
> +	pe =3D container_of(tbl, struct pnv_ioda_pe, tce32_table);
> +	if (!pe->phb)
> +		return -ENODEV;
> +
> +	*phb_id =3D pe->phb->opal_id;
> +	*pe_num =3D pe->pe_number;
> +
> +	return 0;
> +#endif
> +
> +	return -ENXIO;
> +}
> +
> +static int errinjct_ioa_bus_error(const char *buf, struct OpalErrinjct *=
ei)
> +{
> +	uint32_t iommu_grp_id;
> +	int ret;
> +
> +	/* Extract parameters */
> +	ret =3D sscanf(buf, "%x:%x:%x:%x:%x",
> +		     &ei->type, &ei->ioa.addr,
> +		     &ei->ioa.mask, &iommu_grp_id, ei->ioa.function);
> +	if (ret !=3D 5)
> +		return -EINVAL;
> +
> +	/* Invalid function ? */
> +	if (ei->ioa.function < OpalEitIoaLoadMemAddr ||
> +	    ei->ioa.function > OpalEitIoaDmaWriteMemTarget)
> +		return -ERANGE;
> +
> +	/* Retrieve PHB ID and PE number */
> +	ret =3D errinjct_iommu_group_to_phb_and_pe(iommu_grp_id,
> +						 &ei->ioa.phb_id,
> +						 &ei->ioa.pe);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static int errinjct_ioa_bus_error64(const char *buf, struct OpalErrinjct=
 *ei)
> +{
> +	uint32_t iommu_grp_id;
> +	int ret;
> +
> +	/* Extract parameter */
> +	ret =3D sscanf(buf, "%x:%llx:%llx:%x:%x",
> +		     &ei->type, &ei->ioa64.addr,
> +		     &ei->ioa64.mask, &iommu_grp_id, &ei->ioa64.function);
> +	if (ret !=3D 5)
> +		return -EINVAL;
> +
> +	/* Invalid function ? */
> +	if (ei->ioa64.function < OpalEitIoaLoadMemAddr ||
> +	    ei->ioa64.function > OpalEitIoaDmaWriteMemTarget)
> +		return -ERANGE;
> +
> +	/* Retrieve PHB ID and PE number */
> +	ret =3D errinjct_iommu_group_to_phb_and_pe(iommu_grp_id,
> +						 &ei->ioa64.phb_id,
> +						 &ei->ioa64.pe);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static ssize_t errinjct_store(struct kobject *kobj,
> +			      struct kobj_attribute *attr,
> +			      const char *buf, size_t count)
> +{
> +	struct OpalErrinjct ei;
> +	int ret;
> +	long rc;
> +
> +	/* Extract common parameters */
> +	ret =3D sscanf(buf, "%x", &ei.type);
> +	if (ret !=3D 1)
> +		return -EINVAL;
> +
> +	/* Error injection might be in progress */
> +	if (!mutex_trylock(&errinjct_mutex))
> +		return -EAGAIN;
> +
> +	switch (ei.type) {
> +	case OpalErrinjctTypeIoaBusError:
> +		ret =3D errinjct_ioa_bus_error(buf, &ei);
> +		break;
> +	case OpalErrinjctTypeIoaBusError64:
> +		ret =3D errinjct_ioa_bus_error64(buf, &ei);
> +		break;
> +	default:
> +		ret =3D -ERANGE;
> +	}
> +
> +	/* Invalid parameters ? */
> +	if (ret)
> +		goto mutex_unlock_exit;
> +
> +	/* OPAL call */
> +	rc =3D opal_err_injct(&ei);
> +	if (rc =3D=3D OPAL_SUCCESS)
> +		ret =3D count;
> +	else
> +		ret =3D -EIO;
> +
> +mutex_unlock_exit:
> +	mutex_unlock(&errinjct_mutex);
> +	return ret;
> +}
> +
> +static struct kobj_attribute errinjct_attr =3D
> +	__ATTR(errinjct, 0600, NULL, errinjct_store);
> +
> +void __init opal_errinjct_init(void)
> +{
> +	int ret;
> +
> +	/* Make sure /sys/firmware/opal directory is created */
> +	if (!opal_kobj) {
> +		pr_warn("%s: opal kobject is not available\n",
> +			__func__);
> +		return;
> +	}
> +
> +	/* Create the sysfs files */
> +	ret =3D sysfs_create_file(opal_kobj, &errinjct_attr.attr);
> +	if (ret)
> +		pr_warn("%s: Cannot create sysfs file (%d)\n",
> +			__func__, ret);
> +}
> diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platfor=
ms/powernv/opal.c
> index 360ad80c..cb29bb5 100644
> --- a/arch/powerpc/platforms/powernv/opal.c
> +++ b/arch/powerpc/platforms/powernv/opal.c
> @@ -604,6 +604,8 @@ static int __init opal_init(void)
>  		opal_sys_param_init();
>  		/* Setup message log interface. */
>  		opal_msglog_init();
> +		/* Setup error injection interface */
> +		opal_errinjct_init();
>  	}
> =20
>  	return 0;

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Benjamin Herrenschmidt]

On Mon, 2014-06-23 at 12:14 +1000, Gavin Shan wrote:
> The patch implements one OPAL firmware sysfs file to support PCI error
> injection: "/sys/firmware/opal/errinjct", which will be used like the
> way described as follows.
> 
> According to PAPR spec, there are 3 RTAS calls related to error injection:
> "ibm,open-errinjct": allocate token prior to doing error injection.
> "ibm,close-errinjct": release the token allocated from "ibm,open-errinjct".
> "ibm,errinjct": do error injection.
> 
> Sysfs file /sys/firmware/opal/errinjct accepts strings that have fixed
> format "ei_token ...". For now, we only support 32-bits and 64-bits
> PCI error injection and they should have following strings written to
> /sys/firmware/opal/errinjct as follows. We don't have corresponding
> sysfs files for "ibm,open-errinjct" and "ibm,close-errinjct", which
> means that we rely on userland to maintain the token by itself.

Should we instead look into adding a file in the existing sysfs
directory of the specific PHB ?

Cheers,
Ben.

> 32-bits PCI error: "7:addr:mask:iommu_group_id:function".
> 64-bits PCI error: "8:addr:mask:iommu_group_id:function".
> 
> The above "7" and "8" represent 32-bits and 64-bits PCI error seperately
> and "function" is one of the specific PCI errors (e.g. MMIO access address
> parity error), which are defined by PAPR spec.
> 
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  arch/powerpc/include/asm/opal.h                |   1 +
>  arch/powerpc/platforms/powernv/Makefile        |   2 +-
>  arch/powerpc/platforms/powernv/opal-errinjct.c | 184 +++++++++++++++++++++++++
>  arch/powerpc/platforms/powernv/opal.c          |   2 +
>  4 files changed, 188 insertions(+), 1 deletion(-)
>  create mode 100644 arch/powerpc/platforms/powernv/opal-errinjct.c
> 
> diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
> index d982bb8..bf280d9 100644
> --- a/arch/powerpc/include/asm/opal.h
> +++ b/arch/powerpc/include/asm/opal.h
> @@ -985,6 +985,7 @@ extern int opal_elog_init(void);
>  extern void opal_platform_dump_init(void);
>  extern void opal_sys_param_init(void);
>  extern void opal_msglog_init(void);
> +extern void opal_errinjct_init(void);
>  
>  extern int opal_machine_check(struct pt_regs *regs);
>  extern bool opal_mce_check_early_recovery(struct pt_regs *regs);
> diff --git a/arch/powerpc/platforms/powernv/Makefile b/arch/powerpc/platforms/powernv/Makefile
> index 63cebb9..4711de8 100644
> --- a/arch/powerpc/platforms/powernv/Makefile
> +++ b/arch/powerpc/platforms/powernv/Makefile
> @@ -1,7 +1,7 @@
>  obj-y			+= setup.o opal-takeover.o opal-wrappers.o opal.o opal-async.o
>  obj-y			+= opal-rtc.o opal-nvram.o opal-lpc.o opal-flash.o
>  obj-y			+= rng.o opal-elog.o opal-dump.o opal-sysparam.o opal-sensor.o
> -obj-y			+= opal-msglog.o
> +obj-y			+= opal-msglog.o opal-errinjct.o
>  
>  obj-$(CONFIG_SMP)	+= smp.o
>  obj-$(CONFIG_PCI)	+= pci.o pci-p5ioc2.o pci-ioda.o
> diff --git a/arch/powerpc/platforms/powernv/opal-errinjct.c b/arch/powerpc/platforms/powernv/opal-errinjct.c
> new file mode 100644
> index 0000000..29c9e83
> --- /dev/null
> +++ b/arch/powerpc/platforms/powernv/opal-errinjct.c
> @@ -0,0 +1,184 @@
> +/*
> + * The file supports error injection, which works based on OPAL API.
> + * For now, we only support PCI error injection. We need support
> + * injecting other types of errors in future.
> + *
> + * Copyright Gavin Shan, IBM Corporation 2014.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/init.h>
> +#include <linux/msi.h>
> +#include <linux/pci.h>
> +#include <linux/iommu.h>
> +#include <linux/random.h>
> +#include <linux/slab.h>
> +#include <linux/sysfs.h>
> +#include <linux/fs.h>
> +#include <linux/fcntl.h>
> +#include <linux/kobject.h>
> +
> +#include <asm/msi_bitmap.h>
> +#include <asm/iommu.h>
> +#include <asm/pci-bridge.h>
> +#include <asm/ppc-pci.h>
> +#include <asm/opal.h>
> +
> +#include "powernv.h"
> +#include "pci.h"
> +
> +static DEFINE_MUTEX(errinjct_mutex);
> +
> +static int errinjct_iommu_group_to_phb_and_pe(uint32_t iommu_grp_id,
> +					      uint64_t *phb_id,
> +					      uint32_t *pe_num)
> +{
> +#ifdef CONFIG_IOMMU_API
> +	struct iommu_group *iommu_grp;
> +	struct iommu_table *tbl;
> +	struct pnv_ioda_pe *pe;
> +
> +	iommu_grp = iommu_group_get_by_id(iommu_grp_id);
> +	if (!iommu_grp)
> +		return -ENODEV;
> +
> +	tbl = iommu_group_get_iommudata(iommu_grp);
> +	if (!tbl)
> +		return -ENODEV;
> +
> +	pe = container_of(tbl, struct pnv_ioda_pe, tce32_table);
> +	if (!pe->phb)
> +		return -ENODEV;
> +
> +	*phb_id = pe->phb->opal_id;
> +	*pe_num = pe->pe_number;
> +
> +	return 0;
> +#endif
> +
> +	return -ENXIO;
> +}
> +
> +static int errinjct_ioa_bus_error(const char *buf, struct OpalErrinjct *ei)
> +{
> +	uint32_t iommu_grp_id;
> +	int ret;
> +
> +	/* Extract parameters */
> +	ret = sscanf(buf, "%x:%x:%x:%x:%x",
> +		     &ei->type, &ei->ioa.addr,
> +		     &ei->ioa.mask, &iommu_grp_id, ei->ioa.function);
> +	if (ret != 5)
> +		return -EINVAL;
> +
> +	/* Invalid function ? */
> +	if (ei->ioa.function < OpalEitIoaLoadMemAddr ||
> +	    ei->ioa.function > OpalEitIoaDmaWriteMemTarget)
> +		return -ERANGE;
> +
> +	/* Retrieve PHB ID and PE number */
> +	ret = errinjct_iommu_group_to_phb_and_pe(iommu_grp_id,
> +						 &ei->ioa.phb_id,
> +						 &ei->ioa.pe);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static int errinjct_ioa_bus_error64(const char *buf, struct OpalErrinjct *ei)
> +{
> +	uint32_t iommu_grp_id;
> +	int ret;
> +
> +	/* Extract parameter */
> +	ret = sscanf(buf, "%x:%llx:%llx:%x:%x",
> +		     &ei->type, &ei->ioa64.addr,
> +		     &ei->ioa64.mask, &iommu_grp_id, &ei->ioa64.function);
> +	if (ret != 5)
> +		return -EINVAL;
> +
> +	/* Invalid function ? */
> +	if (ei->ioa64.function < OpalEitIoaLoadMemAddr ||
> +	    ei->ioa64.function > OpalEitIoaDmaWriteMemTarget)
> +		return -ERANGE;
> +
> +	/* Retrieve PHB ID and PE number */
> +	ret = errinjct_iommu_group_to_phb_and_pe(iommu_grp_id,
> +						 &ei->ioa64.phb_id,
> +						 &ei->ioa64.pe);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static ssize_t errinjct_store(struct kobject *kobj,
> +			      struct kobj_attribute *attr,
> +			      const char *buf, size_t count)
> +{
> +	struct OpalErrinjct ei;
> +	int ret;
> +	long rc;
> +
> +	/* Extract common parameters */
> +	ret = sscanf(buf, "%x", &ei.type);
> +	if (ret != 1)
> +		return -EINVAL;
> +
> +	/* Error injection might be in progress */
> +	if (!mutex_trylock(&errinjct_mutex))
> +		return -EAGAIN;
> +
> +	switch (ei.type) {
> +	case OpalErrinjctTypeIoaBusError:
> +		ret = errinjct_ioa_bus_error(buf, &ei);
> +		break;
> +	case OpalErrinjctTypeIoaBusError64:
> +		ret = errinjct_ioa_bus_error64(buf, &ei);
> +		break;
> +	default:
> +		ret = -ERANGE;
> +	}
> +
> +	/* Invalid parameters ? */
> +	if (ret)
> +		goto mutex_unlock_exit;
> +
> +	/* OPAL call */
> +	rc = opal_err_injct(&ei);
> +	if (rc == OPAL_SUCCESS)
> +		ret = count;
> +	else
> +		ret = -EIO;
> +
> +mutex_unlock_exit:
> +	mutex_unlock(&errinjct_mutex);
> +	return ret;
> +}
> +
> +static struct kobj_attribute errinjct_attr =
> +	__ATTR(errinjct, 0600, NULL, errinjct_store);
> +
> +void __init opal_errinjct_init(void)
> +{
> +	int ret;
> +
> +	/* Make sure /sys/firmware/opal directory is created */
> +	if (!opal_kobj) {
> +		pr_warn("%s: opal kobject is not available\n",
> +			__func__);
> +		return;
> +	}
> +
> +	/* Create the sysfs files */
> +	ret = sysfs_create_file(opal_kobj, &errinjct_attr.attr);
> +	if (ret)
> +		pr_warn("%s: Cannot create sysfs file (%d)\n",
> +			__func__, ret);
> +}
> diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c
> index 360ad80c..cb29bb5 100644
> --- a/arch/powerpc/platforms/powernv/opal.c
> +++ b/arch/powerpc/platforms/powernv/opal.c
> @@ -604,6 +604,8 @@ static int __init opal_init(void)
>  		opal_sys_param_init();
>  		/* Setup message log interface. */
>  		opal_msglog_init();
> +		/* Setup error injection interface */
> +		opal_errinjct_init();
>  	}
>  
>  	return 0;

Re: [PATCH v1 1/3] powerpc/powernv: Sync header with firmware

[Benjamin Herrenschmidt]

On Mon, 2014-06-23 at 12:14 +1000, Gavin Shan wrote:
> The patch synchronizes firmware header file (opal.h) for PCI error
> injection

The FW API you expose is not PCI specific. I haven't seen the
corresponding FW patches yet but I'm not fan of that single call
that collates unrelated things.

I much'd prefer see a opal_pci_err_inject that is specific to
IO(D)A errors, which takes a PHB ID and goes via the normal dispatch
to PHB ops inside OPAL. For the rest, especially core specific
injections, we can provide a separate dedicated call.

Cheers, 
Ben.

> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>  arch/powerpc/include/asm/opal.h                | 65 ++++++++++++++++++++++++++
>  arch/powerpc/platforms/powernv/opal-wrappers.S |  1 +
>  2 files changed, 66 insertions(+)
> 
> diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
> index 66ad7a7..d982bb8 100644
> --- a/arch/powerpc/include/asm/opal.h
> +++ b/arch/powerpc/include/asm/opal.h
> @@ -175,6 +175,7 @@ extern int opal_enter_rtas(struct rtas_args *args,
>  #define OPAL_SET_PARAM				90
>  #define OPAL_DUMP_RESEND			91
>  #define OPAL_DUMP_INFO2				94
> +#define OPAL_ERR_INJECT				96
>  
>  #ifndef __ASSEMBLY__
>  
> @@ -219,6 +220,69 @@ enum OpalPciErrorSeverity {
>  	OPAL_EEH_SEV_INF	= 5
>  };
>  
> +enum OpalErrinjctType {
> +	OpalErrinjctTypeFirst			= 0,
> +	OpalErrinjctTypeFatal			= 1,
> +	OpalErrinjctTypeRecoverRandomEvent	= 2,
> +	OpalErrinjctTypeRecoverSpecialEvent	= 3,
> +	OpalErrinjctTypeCorruptedPage		= 4,
> +	OpalErrinjctTypeCorruptedSlb		= 5,
> +	OpalErrinjctTypeTranslatorFailure	= 6,
> +	OpalErrinjctTypeIoaBusError		= 7,
> +	OpalErrinjctTypeIoaBusError64		= 8,
> +	OpalErrinjctTypePlatformSpecific	= 9,
> +	OpalErrinjctTypeDcacheStart		= 10,
> +	OpalErrinjctTypeDcacheEnd		= 11,
> +	OpalErrinjctTypeIcacheStart		= 12,
> +	OpalErrinjctTypeIcacheEnd		= 13,
> +	OpalErrinjctTypeTlbStart		= 14,
> +	OpalErrinjctTypeTlbEnd			= 15,
> +	OpalErrinjctTypeUpstreamIoError		= 16,
> +	OpalErrinjctTypeLast			= 17,
> +
> +	/* IoaBusError & IoaBusError64 */
> +	OpalEitIoaLoadMemAddr			= 0,
> +	OpalEitIoaLoadMemData			= 1,
> +	OpalEitIoaLoadIoAddr			= 2,
> +	OpalEitIoaLoadIoData			= 3,
> +	OpalEitIoaLoadConfigAddr		= 4,
> +	OpalEitIoaLoadConfigData		= 5,
> +	OpalEitIoaStoreMemAddr			= 6,
> +	OpalEitIoaStoreMemData			= 7,
> +	OpalEitIoaStoreIoAddr			= 8,
> +	OpalEitIoaStoreIoData			= 9,
> +	OpalEitIoaStoreConfigAddr		= 10,
> +	OpalEitIoaStoreConfigData		= 11,
> +	OpalEitIoaDmaReadMemAddr		= 12,
> +	OpalEitIoaDmaReadMemData		= 13,
> +	OpalEitIoaDmaReadMemMaster		= 14,
> +	OpalEitIoaDmaReadMemTarget		= 15,
> +	OpalEitIoaDmaWriteMemAddr		= 16,
> +	OpalEitIoaDmaWriteMemData		= 17,
> +	OpalEitIoaDmaWriteMemMaster		= 18,
> +	OpalEitIoaDmaWriteMemTarget		= 19,
> +};
> +
> +struct OpalErrinjct {
> +	int32_t type;
> +	union {
> +		struct {
> +			uint32_t addr;
> +			uint32_t mask;
> +			uint64_t phb_id;
> +			uint32_t pe;
> +			uint32_t function;
> +		} ioa;
> +		struct {
> +			uint64_t addr;
> +			uint64_t mask;
> +			uint64_t phb_id;
> +			uint32_t pe;
> +			uint32_t function;
> +		} ioa64;
> +	};
> +};
> +
>  enum OpalShpcAction {
>  	OPAL_SHPC_GET_LINK_STATE = 0,
>  	OPAL_SHPC_GET_SLOT_STATE = 1
> @@ -870,6 +934,7 @@ int64_t opal_update_flash(uint64_t blk_list);
>  int64_t opal_dump_init(uint8_t dump_type);
>  int64_t opal_dump_info(__be32 *dump_id, __be32 *dump_size);
>  int64_t opal_dump_info2(__be32 *dump_id, __be32 *dump_size, __be32 *dump_type);
> +int64_t opal_err_injct(struct OpalErrinjct *ei);
>  int64_t opal_dump_read(uint32_t dump_id, uint64_t buffer);
>  int64_t opal_dump_ack(uint32_t dump_id);
>  int64_t opal_dump_resend_notification(void);
> diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
> index f531ffe..44b3d81 100644
> --- a/arch/powerpc/platforms/powernv/opal-wrappers.S
> +++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
> @@ -136,6 +136,7 @@ OPAL_CALL(opal_resync_timebase,			OPAL_RESYNC_TIMEBASE);
>  OPAL_CALL(opal_dump_init,			OPAL_DUMP_INIT);
>  OPAL_CALL(opal_dump_info,			OPAL_DUMP_INFO);
>  OPAL_CALL(opal_dump_info2,			OPAL_DUMP_INFO2);
> +OPAL_CALL(opal_err_injct,			OPAL_ERR_INJECT);
>  OPAL_CALL(opal_dump_read,			OPAL_DUMP_READ);
>  OPAL_CALL(opal_dump_ack,			OPAL_DUMP_ACK);
>  OPAL_CALL(opal_get_msg,				OPAL_GET_MSG);

Re: [PATCH v1 1/3] powerpc/powernv: Sync header with firmware

[Gavin Shan]

On Tue, Jun 24, 2014 at 07:10:14AM +1000, Benjamin Herrenschmidt wrote:
>On Mon, 2014-06-23 at 12:14 +1000, Gavin Shan wrote:
>> The patch synchronizes firmware header file (opal.h) for PCI error
>> injection
>
>The FW API you expose is not PCI specific. I haven't seen the
>corresponding FW patches yet but I'm not fan of that single call
>that collates unrelated things.
>
>I much'd prefer see a opal_pci_err_inject that is specific to
>IO(D)A errors, which takes a PHB ID and goes via the normal dispatch
>to PHB ops inside OPAL. For the rest, especially core specific
>injections, we can provide a separate dedicated call.
>

Thanks, Ben. I'll change the firmware API to have a separate
API (opal_pci_err_inject) for PCI errors.

>Cheers, 
>Ben.
>

Thanks,
Gavin

>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> ---
>>  arch/powerpc/include/asm/opal.h                | 65 ++++++++++++++++++++++++++
>>  arch/powerpc/platforms/powernv/opal-wrappers.S |  1 +
>>  2 files changed, 66 insertions(+)
>> 
>> diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
>> index 66ad7a7..d982bb8 100644
>> --- a/arch/powerpc/include/asm/opal.h
>> +++ b/arch/powerpc/include/asm/opal.h
>> @@ -175,6 +175,7 @@ extern int opal_enter_rtas(struct rtas_args *args,
>>  #define OPAL_SET_PARAM				90
>>  #define OPAL_DUMP_RESEND			91
>>  #define OPAL_DUMP_INFO2				94
>> +#define OPAL_ERR_INJECT				96
>>  
>>  #ifndef __ASSEMBLY__
>>  
>> @@ -219,6 +220,69 @@ enum OpalPciErrorSeverity {
>>  	OPAL_EEH_SEV_INF	= 5
>>  };
>>  
>> +enum OpalErrinjctType {
>> +	OpalErrinjctTypeFirst			= 0,
>> +	OpalErrinjctTypeFatal			= 1,
>> +	OpalErrinjctTypeRecoverRandomEvent	= 2,
>> +	OpalErrinjctTypeRecoverSpecialEvent	= 3,
>> +	OpalErrinjctTypeCorruptedPage		= 4,
>> +	OpalErrinjctTypeCorruptedSlb		= 5,
>> +	OpalErrinjctTypeTranslatorFailure	= 6,
>> +	OpalErrinjctTypeIoaBusError		= 7,
>> +	OpalErrinjctTypeIoaBusError64		= 8,
>> +	OpalErrinjctTypePlatformSpecific	= 9,
>> +	OpalErrinjctTypeDcacheStart		= 10,
>> +	OpalErrinjctTypeDcacheEnd		= 11,
>> +	OpalErrinjctTypeIcacheStart		= 12,
>> +	OpalErrinjctTypeIcacheEnd		= 13,
>> +	OpalErrinjctTypeTlbStart		= 14,
>> +	OpalErrinjctTypeTlbEnd			= 15,
>> +	OpalErrinjctTypeUpstreamIoError		= 16,
>> +	OpalErrinjctTypeLast			= 17,
>> +
>> +	/* IoaBusError & IoaBusError64 */
>> +	OpalEitIoaLoadMemAddr			= 0,
>> +	OpalEitIoaLoadMemData			= 1,
>> +	OpalEitIoaLoadIoAddr			= 2,
>> +	OpalEitIoaLoadIoData			= 3,
>> +	OpalEitIoaLoadConfigAddr		= 4,
>> +	OpalEitIoaLoadConfigData		= 5,
>> +	OpalEitIoaStoreMemAddr			= 6,
>> +	OpalEitIoaStoreMemData			= 7,
>> +	OpalEitIoaStoreIoAddr			= 8,
>> +	OpalEitIoaStoreIoData			= 9,
>> +	OpalEitIoaStoreConfigAddr		= 10,
>> +	OpalEitIoaStoreConfigData		= 11,
>> +	OpalEitIoaDmaReadMemAddr		= 12,
>> +	OpalEitIoaDmaReadMemData		= 13,
>> +	OpalEitIoaDmaReadMemMaster		= 14,
>> +	OpalEitIoaDmaReadMemTarget		= 15,
>> +	OpalEitIoaDmaWriteMemAddr		= 16,
>> +	OpalEitIoaDmaWriteMemData		= 17,
>> +	OpalEitIoaDmaWriteMemMaster		= 18,
>> +	OpalEitIoaDmaWriteMemTarget		= 19,
>> +};
>> +
>> +struct OpalErrinjct {
>> +	int32_t type;
>> +	union {
>> +		struct {
>> +			uint32_t addr;
>> +			uint32_t mask;
>> +			uint64_t phb_id;
>> +			uint32_t pe;
>> +			uint32_t function;
>> +		} ioa;
>> +		struct {
>> +			uint64_t addr;
>> +			uint64_t mask;
>> +			uint64_t phb_id;
>> +			uint32_t pe;
>> +			uint32_t function;
>> +		} ioa64;
>> +	};
>> +};
>> +
>>  enum OpalShpcAction {
>>  	OPAL_SHPC_GET_LINK_STATE = 0,
>>  	OPAL_SHPC_GET_SLOT_STATE = 1
>> @@ -870,6 +934,7 @@ int64_t opal_update_flash(uint64_t blk_list);
>>  int64_t opal_dump_init(uint8_t dump_type);
>>  int64_t opal_dump_info(__be32 *dump_id, __be32 *dump_size);
>>  int64_t opal_dump_info2(__be32 *dump_id, __be32 *dump_size, __be32 *dump_type);
>> +int64_t opal_err_injct(struct OpalErrinjct *ei);
>>  int64_t opal_dump_read(uint32_t dump_id, uint64_t buffer);
>>  int64_t opal_dump_ack(uint32_t dump_id);
>>  int64_t opal_dump_resend_notification(void);
>> diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
>> index f531ffe..44b3d81 100644
>> --- a/arch/powerpc/platforms/powernv/opal-wrappers.S
>> +++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
>> @@ -136,6 +136,7 @@ OPAL_CALL(opal_resync_timebase,			OPAL_RESYNC_TIMEBASE);
>>  OPAL_CALL(opal_dump_init,			OPAL_DUMP_INIT);
>>  OPAL_CALL(opal_dump_info,			OPAL_DUMP_INFO);
>>  OPAL_CALL(opal_dump_info2,			OPAL_DUMP_INFO2);
>> +OPAL_CALL(opal_err_injct,			OPAL_ERR_INJECT);
>>  OPAL_CALL(opal_dump_read,			OPAL_DUMP_READ);
>>  OPAL_CALL(opal_dump_ack,			OPAL_DUMP_ACK);
>>  OPAL_CALL(opal_get_msg,				OPAL_GET_MSG);
>
>

Re: [PATCH v1 1/3] powerpc/powernv: Sync header with firmware

[Benjamin Herrenschmidt]

On Tue, 2014-06-24 at 09:44 +1000, Gavin Shan wrote:
> >I much'd prefer see a opal_pci_err_inject that is specific to
> >IO(D)A errors, which takes a PHB ID and goes via the normal dispatch
> >to PHB ops inside OPAL. For the rest, especially core specific
> >injections, we can provide a separate dedicated call.
> >
> 
> Thanks, Ben. I'll change the firmware API to have a separate
> API (opal_pci_err_inject) for PCI errors.

Also, how do we expose to Linux that the new API is supported ?

Linux shouldn't create the additional files if it isn't...

There are two ways, we can have a property in the DT of the PHB
indicating that it supports error injection or we can check for the
existence of the OPAL token at boot (there's an OPAL call to do that).

Cheers,
Ben.

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Mike Qiu]

On 06/23/2014 10:14 AM, Gavin Shan wrote:
> The patch implements one OPAL firmware sysfs file to support PCI error
> injection: "/sys/firmware/opal/errinjct", which will be used like the
> way described as follows.
>
> According to PAPR spec, there are 3 RTAS calls related to error injection:
> "ibm,open-errinjct": allocate token prior to doing error injection.
> "ibm,close-errinjct": release the token allocated from "ibm,open-errinjct".
> "ibm,errinjct": do error injection.
>
> Sysfs file /sys/firmware/opal/errinjct accepts strings that have fixed
> format "ei_token ...". For now, we only support 32-bits and 64-bits
> PCI error injection and they should have following strings written to
> /sys/firmware/opal/errinjct as follows. We don't have corresponding
> sysfs files for "ibm,open-errinjct" and "ibm,close-errinjct", which
> means that we rely on userland to maintain the token by itself.
>
> 32-bits PCI error: "7:addr:mask:iommu_group_id:function".
> 64-bits PCI error: "8:addr:mask:iommu_group_id:function".
>
> The above "7" and "8" represent 32-bits and 64-bits PCI error seperately
> and "function" is one of the specific PCI errors (e.g. MMIO access address
> parity error), which are defined by PAPR spec.
>
> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
> ---
>   arch/powerpc/include/asm/opal.h                |   1 +
>   arch/powerpc/platforms/powernv/Makefile        |   2 +-
>   arch/powerpc/platforms/powernv/opal-errinjct.c | 184 +++++++++++++++++++++++++
>   arch/powerpc/platforms/powernv/opal.c          |   2 +
>   4 files changed, 188 insertions(+), 1 deletion(-)
>   create mode 100644 arch/powerpc/platforms/powernv/opal-errinjct.c
>
> diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
> index d982bb8..bf280d9 100644
> --- a/arch/powerpc/include/asm/opal.h
> +++ b/arch/powerpc/include/asm/opal.h
> @@ -985,6 +985,7 @@ extern int opal_elog_init(void);
>   extern void opal_platform_dump_init(void);
>   extern void opal_sys_param_init(void);
>   extern void opal_msglog_init(void);
> +extern void opal_errinjct_init(void);
>
>   extern int opal_machine_check(struct pt_regs *regs);
>   extern bool opal_mce_check_early_recovery(struct pt_regs *regs);
> diff --git a/arch/powerpc/platforms/powernv/Makefile b/arch/powerpc/platforms/powernv/Makefile
> index 63cebb9..4711de8 100644
> --- a/arch/powerpc/platforms/powernv/Makefile
> +++ b/arch/powerpc/platforms/powernv/Makefile
> @@ -1,7 +1,7 @@
>   obj-y			+= setup.o opal-takeover.o opal-wrappers.o opal.o opal-async.o
>   obj-y			+= opal-rtc.o opal-nvram.o opal-lpc.o opal-flash.o
>   obj-y			+= rng.o opal-elog.o opal-dump.o opal-sysparam.o opal-sensor.o
> -obj-y			+= opal-msglog.o
> +obj-y			+= opal-msglog.o opal-errinjct.o
>
>   obj-$(CONFIG_SMP)	+= smp.o
>   obj-$(CONFIG_PCI)	+= pci.o pci-p5ioc2.o pci-ioda.o
> diff --git a/arch/powerpc/platforms/powernv/opal-errinjct.c b/arch/powerpc/platforms/powernv/opal-errinjct.c
> new file mode 100644
> index 0000000..29c9e83
> --- /dev/null
> +++ b/arch/powerpc/platforms/powernv/opal-errinjct.c
> @@ -0,0 +1,184 @@
> +/*
> + * The file supports error injection, which works based on OPAL API.
> + * For now, we only support PCI error injection. We need support
> + * injecting other types of errors in future.
> + *
> + * Copyright Gavin Shan, IBM Corporation 2014.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; either version 2 of the License, or
> + * (at your option) any later version.
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/init.h>
> +#include <linux/msi.h>
> +#include <linux/pci.h>
> +#include <linux/iommu.h>
> +#include <linux/random.h>
> +#include <linux/slab.h>
> +#include <linux/sysfs.h>
> +#include <linux/fs.h>
> +#include <linux/fcntl.h>
> +#include <linux/kobject.h>
> +
> +#include <asm/msi_bitmap.h>
> +#include <asm/iommu.h>
> +#include <asm/pci-bridge.h>
> +#include <asm/ppc-pci.h>
> +#include <asm/opal.h>
> +
> +#include "powernv.h"
> +#include "pci.h"
> +
> +static DEFINE_MUTEX(errinjct_mutex);
> +
> +static int errinjct_iommu_group_to_phb_and_pe(uint32_t iommu_grp_id,
> +					      uint64_t *phb_id,
> +					      uint32_t *pe_num)
> +{
> +#ifdef CONFIG_IOMMU_API

Is it reasonable to do error injection with "CONFIG_IOMMU_API" ?

That means if use default config(CONFIG_IOMMU_API = n),  we can not do 
error injection to pci devices?

Thanks
Mike
> +	struct iommu_group *iommu_grp;
> +	struct iommu_table *tbl;
> +	struct pnv_ioda_pe *pe;
> +
> +	iommu_grp = iommu_group_get_by_id(iommu_grp_id);
> +	if (!iommu_grp)
> +		return -ENODEV;
> +
> +	tbl = iommu_group_get_iommudata(iommu_grp);
> +	if (!tbl)
> +		return -ENODEV;
> +
> +	pe = container_of(tbl, struct pnv_ioda_pe, tce32_table);
> +	if (!pe->phb)
> +		return -ENODEV;
> +
> +	*phb_id = pe->phb->opal_id;
> +	*pe_num = pe->pe_number;
> +
> +	return 0;
> +#endif
> +
> +	return -ENXIO;
> +}
> +
> +static int errinjct_ioa_bus_error(const char *buf, struct OpalErrinjct *ei)
> +{
> +	uint32_t iommu_grp_id;
> +	int ret;
> +
> +	/* Extract parameters */
> +	ret = sscanf(buf, "%x:%x:%x:%x:%x",
> +		     &ei->type, &ei->ioa.addr,
> +		     &ei->ioa.mask, &iommu_grp_id, ei->ioa.function);
> +	if (ret != 5)
> +		return -EINVAL;
> +
> +	/* Invalid function ? */
> +	if (ei->ioa.function < OpalEitIoaLoadMemAddr ||
> +	    ei->ioa.function > OpalEitIoaDmaWriteMemTarget)
> +		return -ERANGE;
> +
> +	/* Retrieve PHB ID and PE number */
> +	ret = errinjct_iommu_group_to_phb_and_pe(iommu_grp_id,
> +						 &ei->ioa.phb_id,
> +						 &ei->ioa.pe);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static int errinjct_ioa_bus_error64(const char *buf, struct OpalErrinjct *ei)
> +{
> +	uint32_t iommu_grp_id;
> +	int ret;
> +
> +	/* Extract parameter */
> +	ret = sscanf(buf, "%x:%llx:%llx:%x:%x",
> +		     &ei->type, &ei->ioa64.addr,
> +		     &ei->ioa64.mask, &iommu_grp_id, &ei->ioa64.function);
> +	if (ret != 5)
> +		return -EINVAL;
> +
> +	/* Invalid function ? */
> +	if (ei->ioa64.function < OpalEitIoaLoadMemAddr ||
> +	    ei->ioa64.function > OpalEitIoaDmaWriteMemTarget)
> +		return -ERANGE;
> +
> +	/* Retrieve PHB ID and PE number */
> +	ret = errinjct_iommu_group_to_phb_and_pe(iommu_grp_id,
> +						 &ei->ioa64.phb_id,
> +						 &ei->ioa64.pe);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static ssize_t errinjct_store(struct kobject *kobj,
> +			      struct kobj_attribute *attr,
> +			      const char *buf, size_t count)
> +{
> +	struct OpalErrinjct ei;
> +	int ret;
> +	long rc;
> +
> +	/* Extract common parameters */
> +	ret = sscanf(buf, "%x", &ei.type);
> +	if (ret != 1)
> +		return -EINVAL;
> +
> +	/* Error injection might be in progress */
> +	if (!mutex_trylock(&errinjct_mutex))
> +		return -EAGAIN;
> +
> +	switch (ei.type) {
> +	case OpalErrinjctTypeIoaBusError:
> +		ret = errinjct_ioa_bus_error(buf, &ei);
> +		break;
> +	case OpalErrinjctTypeIoaBusError64:
> +		ret = errinjct_ioa_bus_error64(buf, &ei);
> +		break;
> +	default:
> +		ret = -ERANGE;
> +	}
> +
> +	/* Invalid parameters ? */
> +	if (ret)
> +		goto mutex_unlock_exit;
> +
> +	/* OPAL call */
> +	rc = opal_err_injct(&ei);
> +	if (rc == OPAL_SUCCESS)
> +		ret = count;
> +	else
> +		ret = -EIO;
> +
> +mutex_unlock_exit:
> +	mutex_unlock(&errinjct_mutex);
> +	return ret;
> +}
> +
> +static struct kobj_attribute errinjct_attr =
> +	__ATTR(errinjct, 0600, NULL, errinjct_store);
> +
> +void __init opal_errinjct_init(void)
> +{
> +	int ret;
> +
> +	/* Make sure /sys/firmware/opal directory is created */
> +	if (!opal_kobj) {
> +		pr_warn("%s: opal kobject is not available\n",
> +			__func__);
> +		return;
> +	}
> +
> +	/* Create the sysfs files */
> +	ret = sysfs_create_file(opal_kobj, &errinjct_attr.attr);
> +	if (ret)
> +		pr_warn("%s: Cannot create sysfs file (%d)\n",
> +			__func__, ret);
> +}
> diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c
> index 360ad80c..cb29bb5 100644
> --- a/arch/powerpc/platforms/powernv/opal.c
> +++ b/arch/powerpc/platforms/powernv/opal.c
> @@ -604,6 +604,8 @@ static int __init opal_init(void)
>   		opal_sys_param_init();
>   		/* Setup message log interface. */
>   		opal_msglog_init();
> +		/* Setup error injection interface */
> +		opal_errinjct_init();
>   	}
>
>   	return 0;

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Benjamin Herrenschmidt]

> Is it reasonable to do error injection with "CONFIG_IOMMU_API" ?
> 
> That means if use default config(CONFIG_IOMMU_API = n),  we can not do 
> error injection to pci devices?

Well we can't pass them through either so ...

In any case, this is not a priority. First we need to implement a solid
error injection facility for the *host*. The guest one is really really
low on the list.

Cheers,
Ben.

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Mike Qiu]

On 06/24/2014 02:36 PM, Benjamin Herrenschmidt wrote:
>> Is it reasonable to do error injection with "CONFIG_IOMMU_API" ?
>>
>> That means if use default config(CONFIG_IOMMU_API = n),  we can not do
>> error injection to pci devices?
> Well we can't pass them through either so ...
> In any case, this is not a priority. First we need to implement a solid
> error injection facility for the *host*. The guest one is really really

OK.

Is that mean *host* side error injection should base on 
"CONFIG_IOMMU_API" ? If it is just host side(no guest, no pass through), 
can't we do error inject?

Maybe I misunderstand :)

Thanks
Mike
> low on the list.
>
> Cheers,
> Ben.
>
>
>

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Benjamin Herrenschmidt]

On Tue, 2014-06-24 at 14:57 +0800, Mike Qiu wrote:
> Is that mean *host* side error injection should base on 
> "CONFIG_IOMMU_API" ? If it is just host side(no guest, no pass through), 
> can't we do error inject?
> 
> Maybe I misunderstand :)

Ah no, make different patches, we don't want to use IOMMU group ID, just
PE numbers. Maybe we should expose in sysfs the PEs from the platform
code with the error injection files underneath ... 

Cheers,
Ben.

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Gavin Shan]

On Tue, Jun 24, 2014 at 05:00:52PM +1000, Benjamin Herrenschmidt wrote:
>On Tue, 2014-06-24 at 14:57 +0800, Mike Qiu wrote:
>> Is that mean *host* side error injection should base on 
>> "CONFIG_IOMMU_API" ? If it is just host side(no guest, no pass through), 
>> can't we do error inject?
>> 
>> Maybe I misunderstand :)
>
>Ah no, make different patches, we don't want to use IOMMU group ID, just
>PE numbers. Maybe we should expose in sysfs the PEs from the platform
>code with the error injection files underneath ... 
>

Yeah, "errinjct" needs grab PCI_domain_nr+PE number from sysfs. We
already had PE number sysfs file:

[root@ltcfbl8eb 0000:01:00.1]# pwd
/sys/bus/pci/devices/0000:01:00.1
[root@ltcfbl8eb 0000:01:00.1]# cat eeh_pe_config_addr 
0x1

For guest support, we will rely on VFIO group ioctl command, which
naturally depends on pass-through.

---

We probably implement it like this. If there're anything wrong, please
correct me:

- Introduce EEH callback struct eeh_ops::err_inject(), which will be
  implemented for PowerNV (NULL for pSeries) by calling the PCI error
  injection dedicated OPAL API (opal_pci_err_inject()).
- Introduce global function eeh.c::eeh_err_inject(), which calls to
  eeh_ops::err_inject() and newly introduced VFIO EEH operation
  will be implemented based on this function.
- Introduce debugfs /sys/kernel/debug/powerpc/PCIxxxx/errinjct, which
  receives PCI error injection parameters from "errinjct". It could
  have format: "ei_token:addr:mask:PCI_domain_nr:PE_num:function".
  Eventually, eeh_err_inject() is invoked to call the corresponding
  OPAL API.

Thanks,
Gavin

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Gavin Shan]

On Mon, Jun 23, 2014 at 04:36:44PM +1000, Michael Neuling wrote:
>On Mon, 2014-06-23 at 12:14 +1000, Gavin Shan wrote:
>> The patch implements one OPAL firmware sysfs file to support PCI error
>> injection: "/sys/firmware/opal/errinjct", which will be used like the
>> way described as follows.
>> 
>> According to PAPR spec, there are 3 RTAS calls related to error injection:
>> "ibm,open-errinjct": allocate token prior to doing error injection.
>> "ibm,close-errinjct": release the token allocated from "ibm,open-errinjct".
>> "ibm,errinjct": do error injection.
>> 
>> Sysfs file /sys/firmware/opal/errinjct accepts strings that have fixed
>> format "ei_token ...". For now, we only support 32-bits and 64-bits
>> PCI error injection and they should have following strings written to
>> /sys/firmware/opal/errinjct as follows. We don't have corresponding
>> sysfs files for "ibm,open-errinjct" and "ibm,close-errinjct", which
>> means that we rely on userland to maintain the token by itself.
>
>This sounds cool.  
>
>Can you document the sysfs interface in Documentation/powerpc?
>

Yeah, Documentation/powerpc/eeh-pci-error-recovery.txt needs update
as Ben suggested. It's something in my list :-)

Thanks,
Gavin

>> 
>> 32-bits PCI error: "7:addr:mask:iommu_group_id:function".
>> 64-bits PCI error: "8:addr:mask:iommu_group_id:function".
>> 
>> The above "7" and "8" represent 32-bits and 64-bits PCI error seperately
>> and "function" is one of the specific PCI errors (e.g. MMIO access address
>> parity error), which are defined by PAPR spec.
>> 
>> Signed-off-by: Gavin Shan <gwshan@linux.vnet.ibm.com>
>> ---
>>  arch/powerpc/include/asm/opal.h                |   1 +
>>  arch/powerpc/platforms/powernv/Makefile        |   2 +-
>>  arch/powerpc/platforms/powernv/opal-errinjct.c | 184 +++++++++++++++++++++++++
>>  arch/powerpc/platforms/powernv/opal.c          |   2 +
>>  4 files changed, 188 insertions(+), 1 deletion(-)
>>  create mode 100644 arch/powerpc/platforms/powernv/opal-errinjct.c
>> 
>> diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
>> index d982bb8..bf280d9 100644
>> --- a/arch/powerpc/include/asm/opal.h
>> +++ b/arch/powerpc/include/asm/opal.h
>> @@ -985,6 +985,7 @@ extern int opal_elog_init(void);
>>  extern void opal_platform_dump_init(void);
>>  extern void opal_sys_param_init(void);
>>  extern void opal_msglog_init(void);
>> +extern void opal_errinjct_init(void);
>>  
>>  extern int opal_machine_check(struct pt_regs *regs);
>>  extern bool opal_mce_check_early_recovery(struct pt_regs *regs);
>> diff --git a/arch/powerpc/platforms/powernv/Makefile b/arch/powerpc/platforms/powernv/Makefile
>> index 63cebb9..4711de8 100644
>> --- a/arch/powerpc/platforms/powernv/Makefile
>> +++ b/arch/powerpc/platforms/powernv/Makefile
>> @@ -1,7 +1,7 @@
>>  obj-y			+= setup.o opal-takeover.o opal-wrappers.o opal.o opal-async.o
>>  obj-y			+= opal-rtc.o opal-nvram.o opal-lpc.o opal-flash.o
>>  obj-y			+= rng.o opal-elog.o opal-dump.o opal-sysparam.o opal-sensor.o
>> -obj-y			+= opal-msglog.o
>> +obj-y			+= opal-msglog.o opal-errinjct.o
>>  
>>  obj-$(CONFIG_SMP)	+= smp.o
>>  obj-$(CONFIG_PCI)	+= pci.o pci-p5ioc2.o pci-ioda.o
>> diff --git a/arch/powerpc/platforms/powernv/opal-errinjct.c b/arch/powerpc/platforms/powernv/opal-errinjct.c
>> new file mode 100644
>> index 0000000..29c9e83
>> --- /dev/null
>> +++ b/arch/powerpc/platforms/powernv/opal-errinjct.c
>> @@ -0,0 +1,184 @@
>> +/*
>> + * The file supports error injection, which works based on OPAL API.
>> + * For now, we only support PCI error injection. We need support
>> + * injecting other types of errors in future.
>> + *
>> + * Copyright Gavin Shan, IBM Corporation 2014.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; either version 2 of the License, or
>> + * (at your option) any later version.
>> + */
>> +
>> +#include <linux/kernel.h>
>> +#include <linux/init.h>
>> +#include <linux/msi.h>
>> +#include <linux/pci.h>
>> +#include <linux/iommu.h>
>> +#include <linux/random.h>
>> +#include <linux/slab.h>
>> +#include <linux/sysfs.h>
>> +#include <linux/fs.h>
>> +#include <linux/fcntl.h>
>> +#include <linux/kobject.h>
>> +
>> +#include <asm/msi_bitmap.h>
>> +#include <asm/iommu.h>
>> +#include <asm/pci-bridge.h>
>> +#include <asm/ppc-pci.h>
>> +#include <asm/opal.h>
>> +
>> +#include "powernv.h"
>> +#include "pci.h"
>> +
>> +static DEFINE_MUTEX(errinjct_mutex);
>> +
>> +static int errinjct_iommu_group_to_phb_and_pe(uint32_t iommu_grp_id,
>> +					      uint64_t *phb_id,
>> +					      uint32_t *pe_num)
>> +{
>> +#ifdef CONFIG_IOMMU_API
>> +	struct iommu_group *iommu_grp;
>> +	struct iommu_table *tbl;
>> +	struct pnv_ioda_pe *pe;
>> +
>> +	iommu_grp = iommu_group_get_by_id(iommu_grp_id);
>> +	if (!iommu_grp)
>> +		return -ENODEV;
>> +
>> +	tbl = iommu_group_get_iommudata(iommu_grp);
>> +	if (!tbl)
>> +		return -ENODEV;
>> +
>> +	pe = container_of(tbl, struct pnv_ioda_pe, tce32_table);
>> +	if (!pe->phb)
>> +		return -ENODEV;
>> +
>> +	*phb_id = pe->phb->opal_id;
>> +	*pe_num = pe->pe_number;
>> +
>> +	return 0;
>> +#endif
>> +
>> +	return -ENXIO;
>> +}
>> +
>> +static int errinjct_ioa_bus_error(const char *buf, struct OpalErrinjct *ei)
>> +{
>> +	uint32_t iommu_grp_id;
>> +	int ret;
>> +
>> +	/* Extract parameters */
>> +	ret = sscanf(buf, "%x:%x:%x:%x:%x",
>> +		     &ei->type, &ei->ioa.addr,
>> +		     &ei->ioa.mask, &iommu_grp_id, ei->ioa.function);
>> +	if (ret != 5)
>> +		return -EINVAL;
>> +
>> +	/* Invalid function ? */
>> +	if (ei->ioa.function < OpalEitIoaLoadMemAddr ||
>> +	    ei->ioa.function > OpalEitIoaDmaWriteMemTarget)
>> +		return -ERANGE;
>> +
>> +	/* Retrieve PHB ID and PE number */
>> +	ret = errinjct_iommu_group_to_phb_and_pe(iommu_grp_id,
>> +						 &ei->ioa.phb_id,
>> +						 &ei->ioa.pe);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return 0;
>> +}
>> +
>> +static int errinjct_ioa_bus_error64(const char *buf, struct OpalErrinjct *ei)
>> +{
>> +	uint32_t iommu_grp_id;
>> +	int ret;
>> +
>> +	/* Extract parameter */
>> +	ret = sscanf(buf, "%x:%llx:%llx:%x:%x",
>> +		     &ei->type, &ei->ioa64.addr,
>> +		     &ei->ioa64.mask, &iommu_grp_id, &ei->ioa64.function);
>> +	if (ret != 5)
>> +		return -EINVAL;
>> +
>> +	/* Invalid function ? */
>> +	if (ei->ioa64.function < OpalEitIoaLoadMemAddr ||
>> +	    ei->ioa64.function > OpalEitIoaDmaWriteMemTarget)
>> +		return -ERANGE;
>> +
>> +	/* Retrieve PHB ID and PE number */
>> +	ret = errinjct_iommu_group_to_phb_and_pe(iommu_grp_id,
>> +						 &ei->ioa64.phb_id,
>> +						 &ei->ioa64.pe);
>> +	if (ret)
>> +		return ret;
>> +
>> +	return 0;
>> +}
>> +
>> +static ssize_t errinjct_store(struct kobject *kobj,
>> +			      struct kobj_attribute *attr,
>> +			      const char *buf, size_t count)
>> +{
>> +	struct OpalErrinjct ei;
>> +	int ret;
>> +	long rc;
>> +
>> +	/* Extract common parameters */
>> +	ret = sscanf(buf, "%x", &ei.type);
>> +	if (ret != 1)
>> +		return -EINVAL;
>> +
>> +	/* Error injection might be in progress */
>> +	if (!mutex_trylock(&errinjct_mutex))
>> +		return -EAGAIN;
>> +
>> +	switch (ei.type) {
>> +	case OpalErrinjctTypeIoaBusError:
>> +		ret = errinjct_ioa_bus_error(buf, &ei);
>> +		break;
>> +	case OpalErrinjctTypeIoaBusError64:
>> +		ret = errinjct_ioa_bus_error64(buf, &ei);
>> +		break;
>> +	default:
>> +		ret = -ERANGE;
>> +	}
>> +
>> +	/* Invalid parameters ? */
>> +	if (ret)
>> +		goto mutex_unlock_exit;
>> +
>> +	/* OPAL call */
>> +	rc = opal_err_injct(&ei);
>> +	if (rc == OPAL_SUCCESS)
>> +		ret = count;
>> +	else
>> +		ret = -EIO;
>> +
>> +mutex_unlock_exit:
>> +	mutex_unlock(&errinjct_mutex);
>> +	return ret;
>> +}
>> +
>> +static struct kobj_attribute errinjct_attr =
>> +	__ATTR(errinjct, 0600, NULL, errinjct_store);
>> +
>> +void __init opal_errinjct_init(void)
>> +{
>> +	int ret;
>> +
>> +	/* Make sure /sys/firmware/opal directory is created */
>> +	if (!opal_kobj) {
>> +		pr_warn("%s: opal kobject is not available\n",
>> +			__func__);
>> +		return;
>> +	}
>> +
>> +	/* Create the sysfs files */
>> +	ret = sysfs_create_file(opal_kobj, &errinjct_attr.attr);
>> +	if (ret)
>> +		pr_warn("%s: Cannot create sysfs file (%d)\n",
>> +			__func__, ret);
>> +}
>> diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c
>> index 360ad80c..cb29bb5 100644
>> --- a/arch/powerpc/platforms/powernv/opal.c
>> +++ b/arch/powerpc/platforms/powernv/opal.c
>> @@ -604,6 +604,8 @@ static int __init opal_init(void)
>>  		opal_sys_param_init();
>>  		/* Setup message log interface. */
>>  		opal_msglog_init();
>> +		/* Setup error injection interface */
>> +		opal_errinjct_init();
>>  	}
>>  
>>  	return 0;
>
>--
>To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Mike Qiu]

On 06/25/2014 08:03 AM, Gavin Shan wrote:
> On Tue, Jun 24, 2014 at 05:00:52PM +1000, Benjamin Herrenschmidt wrote:
>> On Tue, 2014-06-24 at 14:57 +0800, Mike Qiu wrote:
>>> Is that mean *host* side error injection should base on
>>> "CONFIG_IOMMU_API" ? If it is just host side(no guest, no pass through),
>>> can't we do error inject?
>>>
>>> Maybe I misunderstand :)
>> Ah no, make different patches, we don't want to use IOMMU group ID, just
>> PE numbers. Maybe we should expose in sysfs the PEs from the platform
>> code with the error injection files underneath ...
>>
> Yeah, "errinjct" needs grab PCI_domain_nr+PE number from sysfs. We
> already had PE number sysfs file:
>
> [root@ltcfbl8eb 0000:01:00.1]# pwd
> /sys/bus/pci/devices/0000:01:00.1
> [root@ltcfbl8eb 0000:01:00.1]# cat eeh_pe_config_addr
> 0x1
>
> For guest support, we will rely on VFIO group ioctl command, which
> naturally depends on pass-through.
>
> ---
>
> We probably implement it like this. If there're anything wrong, please
> correct me:
>
> - Introduce EEH callback struct eeh_ops::err_inject(), which will be
>    implemented for PowerNV (NULL for pSeries) by calling the PCI error
>    injection dedicated OPAL API (opal_pci_err_inject()).
> - Introduce global function eeh.c::eeh_err_inject(), which calls to
>    eeh_ops::err_inject() and newly introduced VFIO EEH operation
>    will be implemented based on this function.
> - Introduce debugfs /sys/kernel/debug/powerpc/PCIxxxx/errinjct, which

Here maybe  "/sys/kernel/debug/powerpc/errinjct" is better, because it 
will supply "PCI_domain_nr" in parameters, so no need supply errinjct 
for each PCI domain.

Another reason is error inject not only for PCI(in future), so better 
not in PCI domain entry.

Also it simple for userland tools to has a fixed path.

Thanks
Mike

>    receives PCI error injection parameters from "errinjct". It could
>    have format: "ei_token:addr:mask:PCI_domain_nr:PE_num:function".
>    Eventually, eeh_err_inject() is invoked to call the corresponding
>    OPAL API.
>
> Thanks,
> Gavin
>

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Benjamin Herrenschmidt]

On Wed, 2014-06-25 at 11:05 +0800, Mike Qiu wrote:
> Here maybe  "/sys/kernel/debug/powerpc/errinjct" is better, because
> it 
> will supply "PCI_domain_nr" in parameters, so no need supply errinjct 
> for each PCI domain.
> 
> Another reason is error inject not only for PCI(in future), so better 
> not in PCI domain entry.
> 
> Also it simple for userland tools to has a fixed path.

I don't like this. I much prefer have dedicated error injection files
in their respective locations, something for PCI under the corresponding
PCI bridge etc...

Cheers,
Ben.

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Stewart Smith]

Gavin Shan <gwshan@linux.vnet.ibm.com> writes:
> On Mon, Jun 23, 2014 at 04:36:44PM +1000, Michael Neuling wrote:
>>On Mon, 2014-06-23 at 12:14 +1000, Gavin Shan wrote:
>>> The patch implements one OPAL firmware sysfs file to support PCI error
>>> injection: "/sys/firmware/opal/errinjct", which will be used like the
>>> way described as follows.
>>> 
>>> According to PAPR spec, there are 3 RTAS calls related to error injection:
>>> "ibm,open-errinjct": allocate token prior to doing error injection.
>>> "ibm,close-errinjct": release the token allocated from "ibm,open-errinjct".
>>> "ibm,errinjct": do error injection.
>>> 
>>> Sysfs file /sys/firmware/opal/errinjct accepts strings that have fixed
>>> format "ei_token ...". For now, we only support 32-bits and 64-bits
>>> PCI error injection and they should have following strings written to
>>> /sys/firmware/opal/errinjct as follows. We don't have corresponding
>>> sysfs files for "ibm,open-errinjct" and "ibm,close-errinjct", which
>>> means that we rely on userland to maintain the token by itself.
>>
>>This sounds cool.  
>>
>>Can you document the sysfs interface in Documentation/powerpc?
>>
>
> Yeah, Documentation/powerpc/eeh-pci-error-recovery.txt needs update
> as Ben suggested. It's something in my list :-)

It should probably also/instead be in
Documentation/ABI/(testing|stable)/sysfs-firmware-opal-errinjct  as this
seems to be where sysfs bits get documented.

Also, considering that we're specifically looking at PCI error
injection, should the sysfs name be /sys/firmware/opal/pci-error-inject
instead?

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Stewart Smith]

Gavin Shan <gwshan@linux.vnet.ibm.com> writes:
> +static struct kobj_attribute errinjct_attr =
> +	__ATTR(errinjct, 0600, NULL, errinjct_store);

May also be good to have a read method that either lists current
injected errors? I guess it depends on if they're one time errors or
persistent errors too.

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Mike Qiu]

On 06/25/2014 11:19 AM, Benjamin Herrenschmidt wrote:
> On Wed, 2014-06-25 at 11:05 +0800, Mike Qiu wrote:
>> Here maybe  "/sys/kernel/debug/powerpc/errinjct" is better, because
>> it
>> will supply "PCI_domain_nr" in parameters, so no need supply errinjct
>> for each PCI domain.
>>
>> Another reason is error inject not only for PCI(in future), so better
>> not in PCI domain entry.
>>
>> Also it simple for userland tools to has a fixed path.
> I don't like this. I much prefer have dedicated error injection files
> in their respective locations, something for PCI under the corresponding
> PCI bridge etc...

So PowerNV error injection will be designed rely on debugfs been 
configured, right?

Thanks,
Mike
>
> Cheers,
> Ben.
>
>
>
>

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Benjamin Herrenschmidt]

On Mon, 2014-07-21 at 16:06 +0800, Mike Qiu wrote:
> > I don't like this. I much prefer have dedicated error injection files
> > in their respective locations, something for PCI under the corresponding
> > PCI bridge etc...
> 
> So PowerNV error injection will be designed rely on debugfs been 
> configured, right?

Not necessarily. If we create a better debugfs layout for our PHBs, then
yes. It might be useful to provide more info in there for example access
to some of the counters ...

But on the other hand, for error injection in general, I wonder if we should
be under sysfs instead... something to study a bit.

Cheers,
Ben.

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Mike Qiu]

On 07/22/2014 06:49 AM, Benjamin Herrenschmidt wrote:
> On Mon, 2014-07-21 at 16:06 +0800, Mike Qiu wrote:
>>> I don't like this. I much prefer have dedicated error injection files
>>> in their respective locations, something for PCI under the corresponding
>>> PCI bridge etc...
>> So PowerNV error injection will be designed rely on debugfs been
>> configured, right?
> Not necessarily. If we create a better debugfs layout for our PHBs, then
> yes. It might be useful to provide more info in there for example access
> to some of the counters ...
>
> But on the other hand, for error injection in general, I wonder if we should
> be under sysfs instead... something to study a bit.

In pHyp, general error injection use syscall:

     #define __NR_rtas        255

I don't know if it is a good idea to reuse this syscall for PowerNV.

At least, it is another choice without sysfs rely.

Thanks,
Mike

>
> Cheers,
> Ben.
>
>
>

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Benjamin Herrenschmidt]

On Tue, 2014-07-22 at 11:10 +0800, Mike Qiu wrote:
> On 07/22/2014 06:49 AM, Benjamin Herrenschmidt wrote:
> > On Mon, 2014-07-21 at 16:06 +0800, Mike Qiu wrote:
> >>> I don't like this. I much prefer have dedicated error injection files
> >>> in their respective locations, something for PCI under the corresponding
> >>> PCI bridge etc...
> >> So PowerNV error injection will be designed rely on debugfs been
> >> configured, right?
> > Not necessarily. If we create a better debugfs layout for our PHBs, then
> > yes. It might be useful to provide more info in there for example access
> > to some of the counters ...
> >
> > But on the other hand, for error injection in general, I wonder if we should
> > be under sysfs instead... something to study a bit.
> 
> In pHyp, general error injection use syscall:
> 
>      #define __NR_rtas        255
> 
> I don't know if it is a good idea to reuse this syscall for PowerNV.
> 
> At least, it is another choice without sysfs rely.

No, we certainly don't want that RTAS stuff. I though Linux had some
kind of error injection infrastructure nowadays... somebody needs to
have a look.

Cheers,
Ben.

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Gavin Shan]

On Tue, Jul 22, 2014 at 11:10:42AM +0800, Mike Qiu wrote:
>On 07/22/2014 06:49 AM, Benjamin Herrenschmidt wrote:
>>On Mon, 2014-07-21 at 16:06 +0800, Mike Qiu wrote:
>>>>I don't like this. I much prefer have dedicated error injection files
>>>>in their respective locations, something for PCI under the corresponding
>>>>PCI bridge etc...
>>>So PowerNV error injection will be designed rely on debugfs been
>>>configured, right?
>>Not necessarily. If we create a better debugfs layout for our PHBs, then
>>yes. It might be useful to provide more info in there for example access
>>to some of the counters ...
>>
>>But on the other hand, for error injection in general, I wonder if we should
>>be under sysfs instead... something to study a bit.
>
>In pHyp, general error injection use syscall:
>
>    #define __NR_rtas        255
>
>I don't know if it is a good idea to reuse this syscall for PowerNV.
>
>At least, it is another choice without sysfs rely.
>

We won't use syscall for routing the error injection on PowerNV any more.
Generally speaking, we will use ioctl commands or subcode of EEH ioctl
command, which was invented for EEH support for VFIO devices to suport
QEMU. For the utility (errinjct) running on PowerNV, we will use debugfs
entries. I have premature code for that, but don't have chance to polish
it yet. Let me send you that so that you can start working from there.

Thanks,
Gavin 

Re: [PATCH v1 2/3] powerpc/powernv: Support PCI error injection

[Mike Qiu]

On 07/22/2014 11:26 AM, Gavin Shan wrote:
> On Tue, Jul 22, 2014 at 11:10:42AM +0800, Mike Qiu wrote:
>> On 07/22/2014 06:49 AM, Benjamin Herrenschmidt wrote:
>>> On Mon, 2014-07-21 at 16:06 +0800, Mike Qiu wrote:
>>>>> I don't like this. I much prefer have dedicated error injection files
>>>>> in their respective locations, something for PCI under the corresponding
>>>>> PCI bridge etc...
>>>> So PowerNV error injection will be designed rely on debugfs been
>>>> configured, right?
>>> Not necessarily. If we create a better debugfs layout for our PHBs, then
>>> yes. It might be useful to provide more info in there for example access
>>> to some of the counters ...
>>>
>>> But on the other hand, for error injection in general, I wonder if we should
>>> be under sysfs instead... something to study a bit.
>> In pHyp, general error injection use syscall:
>>
>>     #define __NR_rtas        255
>>
>> I don't know if it is a good idea to reuse this syscall for PowerNV.
>>
>> At least, it is another choice without sysfs rely.
>>
> We won't use syscall for routing the error injection on PowerNV any more.
> Generally speaking, we will use ioctl commands or subcode of EEH ioctl
> command, which was invented for EEH support for VFIO devices to suport
> QEMU. For the utility (errinjct) running on PowerNV, we will use debugfs
> entries. I have premature code for that, but don't have chance to polish
> it yet. Let me send you that so that you can start working from there.

OK, thanks
> Thanks,
> Gavin
>
>
>