[PATCH 0/3] ocxl: Support for an 0penCAPI device in a QEMU guest.

[PATCH 0/3] ocxl: Support for an 0penCAPI device in a QEMU guest.

[christophe lombard]

This series adds support for an 0penCAPI device in a QEMU guest.

It builds on top of the existing ocxl driver +
http://patchwork.ozlabs.org/patch/1177999/

The ocxl module registers either a pci driver or a platform driver, based on
the environment (bare-metal (powernv) or pseries).

Roughly 4/5 of the code is common between the 2 types of driver:
- PCI implementation
- mmio operations
- link management
- sysfs folders
- page fault and context handling

The differences in implementation are essentially based on the interact with
the opal api(s) defined in the host. Several hcalls have been defined
(extension of the PAPR) to:
- configure the Sceduled Process Area
- get specific AFU information
- allocated irq
- handle page fault and process element

When the code needs to call a platform-specific implementation, it does so
through an API. The powervn and pseries implementations each describe
their own definition. See struct ocxl_backend_ops.

It has been tested in a bare-metal and QEMU environment using the memcpy and
the AFP AFUs.

christophe lombard (3):
  ocxl: Introduce implementation-specific API
  ocxl: Add pseries-specific code
  powerpc/pseries: Fixup config space size of OpenCAPI devices

 arch/powerpc/platforms/pseries/pci.c |   9 +
 drivers/misc/ocxl/Makefile           |   3 +
 drivers/misc/ocxl/config.c           |   7 +-
 drivers/misc/ocxl/link.c             |  31 +-
 drivers/misc/ocxl/main.c             |   9 +
 drivers/misc/ocxl/ocxl_internal.h    |  25 ++
 drivers/misc/ocxl/powernv.c          |  88 ++++++
 drivers/misc/ocxl/pseries.c          | 450 +++++++++++++++++++++++++++
 8 files changed, 603 insertions(+), 19 deletions(-)
 create mode 100644 drivers/misc/ocxl/powernv.c
 create mode 100644 drivers/misc/ocxl/pseries.c

-- 
2.21.0

[PATCH 1/3] ocxl: Introduce implementation-specific API

[christophe lombard]

The backend API (in ocxl.h) lists some low-level functions whose
implementation is different on bare-metal and in a guest. Each
environment implements its own functions, and the common code uses
them through function pointers, defined in ocxl_backend_ops

A new powernv.c file is created to call the pnv_ocxl_ API for the
bare-metal environment.

Signed-off-by: Christophe Lombard <clombard@linux.vnet.ibm.com>
---
 drivers/misc/ocxl/Makefile        |  2 +
 drivers/misc/ocxl/config.c        |  7 ++-
 drivers/misc/ocxl/link.c          | 31 +++++------
 drivers/misc/ocxl/main.c          |  5 ++
 drivers/misc/ocxl/ocxl_internal.h | 24 +++++++++
 drivers/misc/ocxl/powernv.c       | 88 +++++++++++++++++++++++++++++++
 6 files changed, 138 insertions(+), 19 deletions(-)
 create mode 100644 drivers/misc/ocxl/powernv.c

diff --git a/drivers/misc/ocxl/Makefile b/drivers/misc/ocxl/Makefile
index d07d1bb8e8d4..bfdaeb232b83 100644
--- a/drivers/misc/ocxl/Makefile
+++ b/drivers/misc/ocxl/Makefile
@@ -4,6 +4,8 @@ ccflags-$(CONFIG_PPC_WERROR)	+= -Werror
 ocxl-y				+= main.o pci.o config.o file.o pasid.o mmio.o
 ocxl-y				+= link.o context.o afu_irq.o sysfs.o trace.o
 ocxl-y				+= core.o
+ocxl-$(CONFIG_PPC_POWERNV)	+= powernv.o
+
 obj-$(CONFIG_OCXL)		+= ocxl.o
 
 # For tracepoints to include our trace.h from tracepoint infrastructure:
diff --git a/drivers/misc/ocxl/config.c b/drivers/misc/ocxl/config.c
index 7ca0f6744125..981a3bcfe742 100644
--- a/drivers/misc/ocxl/config.c
+++ b/drivers/misc/ocxl/config.c
@@ -1,7 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0+
 // Copyright 2017 IBM Corp.
 #include <linux/pci.h>
-#include <asm/pnv-ocxl.h>
 #include <misc/ocxl-config.h>
 #include "ocxl_internal.h"
 
@@ -649,7 +648,7 @@ int ocxl_config_get_actag_info(struct pci_dev *dev, u16 *base, u16 *enabled,
 	 * avoid an external driver using ocxl as a library to call
 	 * platform-dependent code
 	 */
-	rc = pnv_ocxl_get_actag(dev, base, enabled, supported);
+	rc = ocxl_ops->get_actag(dev, base, enabled, supported);
 	if (rc) {
 		dev_err(&dev->dev, "Can't get actag for device: %d\n", rc);
 		return rc;
@@ -673,7 +672,7 @@ EXPORT_SYMBOL_GPL(ocxl_config_set_afu_actag);
 
 int ocxl_config_get_pasid_info(struct pci_dev *dev, int *count)
 {
-	return pnv_ocxl_get_pasid_count(dev, count);
+	return ocxl_ops->get_pasid_count(dev, count);
 }
 
 void ocxl_config_set_afu_pasid(struct pci_dev *dev, int pos, int pasid_base,
@@ -715,7 +714,7 @@ int ocxl_config_set_TL(struct pci_dev *dev, int tl_dvsec)
 	if (PCI_FUNC(dev->devfn) != 0)
 		return 0;
 
-	return pnv_ocxl_set_TL(dev, tl_dvsec);
+	return ocxl_ops->set_tl(dev, tl_dvsec);
 }
 EXPORT_SYMBOL_GPL(ocxl_config_set_TL);
 
diff --git a/drivers/misc/ocxl/link.c b/drivers/misc/ocxl/link.c
index e936a3bd5957..9f4d164180a7 100644
--- a/drivers/misc/ocxl/link.c
+++ b/drivers/misc/ocxl/link.c
@@ -5,7 +5,6 @@
 #include <linux/mm_types.h>
 #include <linux/mmu_context.h>
 #include <asm/copro.h>
-#include <asm/pnv-ocxl.h>
 #include <misc/ocxl.h>
 #include "ocxl_internal.h"
 #include "trace.h"
@@ -83,7 +82,7 @@ static void ack_irq(struct ocxl_link *link, enum xsl_response r)
 				     link->xsl_fault.dsisr,
 				     link->xsl_fault.dar,
 				     reg);
-		pnv_ocxl_handle_fault(link->platform_data, reg);
+		ocxl_ops->ack_irq(link->platform_data, reg);
 	}
 }
 
@@ -146,8 +145,8 @@ static irqreturn_t xsl_fault_handler(int irq, void *data)
 	int pid;
 	bool schedule = false;
 
-	pnv_ocxl_get_fault_state(link->platform_data, &dsisr, &dar,
-				 &pe_handle, &pid);
+	ocxl_ops->get_fault_state(link->platform_data, &dsisr, &dar,
+				  &pe_handle, &pid);
 	trace_ocxl_fault(pe_handle, dsisr, dar, -1);
 
 	/* We could be reading all null values here if the PE is being
@@ -282,8 +281,8 @@ static int alloc_link(struct pci_dev *dev, int PE_mask, struct ocxl_link **out_l
 	INIT_WORK(&link->xsl_fault.fault_work, xsl_fault_handler_bh);
 
 	/* platform specific hook */
-	rc = pnv_ocxl_platform_setup(dev, PE_mask, &xsl_irq,
-				     &link->platform_data);
+	rc = ocxl_ops->platform_setup(dev, PE_mask, &xsl_irq,
+				      &link->platform_data);
 	if (rc)
 		goto err_free;
 
@@ -298,7 +297,7 @@ static int alloc_link(struct pci_dev *dev, int PE_mask, struct ocxl_link **out_l
 	return 0;
 
 err_xsl_irq:
-	pnv_ocxl_platform_release(link->platform_data);
+	ocxl_ops->platform_release(link->platform_data);
 err_free:
 	kfree(link);
 	return rc;
@@ -344,7 +343,7 @@ static void release_xsl(struct kref *ref)
 
 	list_del(&link->list);
 	/* call platform code before releasing data */
-	pnv_ocxl_platform_release(link->platform_data);
+	ocxl_ops->platform_release(link->platform_data);
 	free_link(link);
 }
 
@@ -378,8 +377,8 @@ int ocxl_link_add_pe(void *link_handle, int pasid, u32 pidr, u32 tidr,
 	pe_data->xsl_err_cb = xsl_err_cb;
 	pe_data->xsl_err_data = xsl_err_data;
 
-	rc = pnv_ocxl_set_pe(link->platform_data, mfspr(SPRN_LPID),
-			     pasid, pidr, tidr, amr, &pe_handle);
+	rc = ocxl_ops->set_pe(link->platform_data, pasid, pidr, tidr,
+			      amr, &pe_handle);
 	if (rc) {
 		kfree(pe_data);
 		goto unlock;
@@ -429,7 +428,9 @@ int ocxl_link_update_pe(void *link_handle, int pasid, __u16 tid)
 	int rc;
 
 	mutex_lock(&link->pe_lock);
-	rc = pnv_ocxl_update_pe(link->platform_data, pasid, tid);
+
+	rc = ocxl_ops->update_pe(link->platform_data, pasid, tid);
+
 	mutex_unlock(&link->pe_lock);
 
 	return rc;
@@ -462,8 +463,8 @@ int ocxl_link_remove_pe(void *link_handle, int pasid)
 	 */
 	mutex_lock(&link->pe_lock);
 
-	rc = pnv_ocxl_remove_pe(link->platform_data, pasid, &pid, &tid,
-				&pe_handle);
+	rc = ocxl_ops->remove_pe(link->platform_data, pasid, &pid, &tid,
+				 &pe_handle);
 	if (rc)
 		goto unlock;
 
@@ -494,7 +495,7 @@ int ocxl_link_irq_alloc(void *link_handle, int *hw_irq, u64 *trigger_addr)
 	if (atomic_dec_if_positive(&link->irq_available) < 0)
 		return -ENOSPC;
 
-	rc = pnv_ocxl_alloc_xive_irq(&irq, &addr);
+	rc = ocxl_ops->alloc_xive_irq(link->platform_data, &irq, &addr);
 	if (rc) {
 		atomic_inc(&link->irq_available);
 		return rc;
@@ -510,7 +511,7 @@ void ocxl_link_free_irq(void *link_handle, int hw_irq)
 {
 	struct ocxl_link *link = (struct ocxl_link *) link_handle;
 
-	pnv_ocxl_free_xive_irq(hw_irq);
+	ocxl_ops->free_xive_irq(link->platform_data, hw_irq);
 	atomic_inc(&link->irq_available);
 }
 EXPORT_SYMBOL_GPL(ocxl_link_free_irq);
diff --git a/drivers/misc/ocxl/main.c b/drivers/misc/ocxl/main.c
index 7210d9e059be..95df2ba4d473 100644
--- a/drivers/misc/ocxl/main.c
+++ b/drivers/misc/ocxl/main.c
@@ -4,6 +4,8 @@
 #include <linux/pci.h>
 #include "ocxl_internal.h"
 
+const struct ocxl_backend_ops *ocxl_ops;
+
 static int __init init_ocxl(void)
 {
 	int rc = 0;
@@ -12,6 +14,9 @@ static int __init init_ocxl(void)
 	if (rc)
 		return rc;
 
+	if (cpu_has_feature(CPU_FTR_HVMODE))
+		ocxl_ops = &ocxl_powernv_ops;
+
 	rc = pci_register_driver(&ocxl_pci_driver);
 	if (rc) {
 		ocxl_file_exit();
diff --git a/drivers/misc/ocxl/ocxl_internal.h b/drivers/misc/ocxl/ocxl_internal.h
index 6e1e9cd315c0..2bdea279bdc6 100644
--- a/drivers/misc/ocxl/ocxl_internal.h
+++ b/drivers/misc/ocxl/ocxl_internal.h
@@ -82,6 +82,30 @@ struct ocxl_context {
 	u16 tidr; // Thread ID used for P9 wait implementation
 };
 
+struct ocxl_backend_ops {
+	struct module *module;
+	void (*ack_irq)(void *data, u64 tfc);
+	int (*alloc_xive_irq)(void *data, u32 *irq, u64 *trigger_addr);
+	void (*free_xive_irq)(void *data, u32 irq);
+	int (*get_actag)(struct pci_dev *dev, u16 *base, u16 *enabled,
+			 u16 *supported);
+	void (*get_fault_state)(void *data, u64 *dsisr, u64 *dar,
+				u64 *pe_handle, int *pid);
+	int (*get_pasid_count)(struct pci_dev *dev, int *count);
+	void (*platform_release)(void *data);
+	int (*platform_setup)(struct pci_dev *dev, int PE_mask,
+			      int *hwirq, void **data);
+	int (*remove_pe)(void *data, int pasid, u32 *pid, u32 *tid,
+			 int *pe_handle);
+	int (*set_pe)(void *data, int pasid, u32 pidr, u32 tidr,
+		      u64 amr, int *pe_handle);
+	int (*set_tl)(struct pci_dev *dev, int tl_dvsec);
+	int (*update_pe)(void *data, int pasid, __u16 tid);
+};
+
+extern const struct ocxl_backend_ops ocxl_powernv_ops;
+extern const struct ocxl_backend_ops *ocxl_ops;
+
 int ocxl_create_cdev(struct ocxl_afu *afu);
 void ocxl_destroy_cdev(struct ocxl_afu *afu);
 int ocxl_file_register_afu(struct ocxl_afu *afu);
diff --git a/drivers/misc/ocxl/powernv.c b/drivers/misc/ocxl/powernv.c
new file mode 100644
index 000000000000..13c98fcf7fc7
--- /dev/null
+++ b/drivers/misc/ocxl/powernv.c
@@ -0,0 +1,88 @@
+// SPDX-License-Identifier: GPL-2.0+
+// Copyright 2018 IBM Corp.
+#include <asm/pnv-ocxl.h>
+#include <misc/ocxl-config.h>
+#include "ocxl_internal.h"
+
+static void ack_irq(void *data, u64 tfc)
+{
+	pnv_ocxl_handle_fault(data, tfc);
+}
+
+static int alloc_xive_irq(void *unused, u32 *irq,
+			  u64 *trigger_addr)
+{
+	return pnv_ocxl_alloc_xive_irq(irq, trigger_addr);
+}
+
+static void free_xive_irq(void *unused, u32 irq)
+{
+	pnv_ocxl_free_xive_irq(irq);
+}
+
+static int get_actag(struct pci_dev *dev, u16 *base,
+		     u16 *enabled, u16 *supported)
+{
+	return pnv_ocxl_get_actag(dev, base, enabled, supported);
+}
+
+static void get_fault_state(void *data, u64 *dsisr, u64 *dar,
+			    u64 *pe_handle, int *pid)
+{
+	pnv_ocxl_get_fault_state(data, dsisr, dar, pe_handle, pid);
+}
+
+static int get_pasid_count(struct pci_dev *dev, int *count)
+{
+	return pnv_ocxl_get_pasid_count(dev, count);
+}
+
+static void platform_release(void *data)
+{
+	return pnv_ocxl_platform_release(data);
+}
+
+static int platform_setup(struct pci_dev *dev, int PE_mask, int *hwirq,
+			  void **data)
+{
+	return pnv_ocxl_platform_setup(dev, PE_mask, hwirq, data);
+}
+
+static int remove_pe(void *data, int pasid, u32 *pid,
+		     u32 *tid, int *pe_handle)
+{
+	return pnv_ocxl_remove_pe(data, pasid, pid, tid, pe_handle);
+}
+
+static int set_pe(void *data, int pasid, u32 pidr, u32 tidr,
+		  u64 amr, int *pe_handle)
+{
+	return pnv_ocxl_set_pe(data, mfspr(SPRN_LPID), pasid,
+			       pidr, tidr, amr, pe_handle);
+}
+
+static int set_tl(struct pci_dev *dev, int tl_dvsec)
+{
+	return pnv_ocxl_set_TL(dev, tl_dvsec);
+}
+
+static int update_pe(void *data, int pasid, __u16 tid)
+{
+	return pnv_ocxl_update_pe(data, pasid, tid);
+}
+
+const struct ocxl_backend_ops ocxl_powernv_ops = {
+	.module = THIS_MODULE,
+	.ack_irq = ack_irq,
+	.alloc_xive_irq = alloc_xive_irq,
+	.free_xive_irq = free_xive_irq,
+	.get_actag = get_actag,
+	.get_fault_state = get_fault_state,
+	.get_pasid_count = get_pasid_count,
+	.platform_release = platform_release,
+	.platform_setup = platform_setup,
+	.remove_pe = remove_pe,
+	.set_pe = set_pe,
+	.set_tl = set_tl,
+	.update_pe = update_pe,
+};
-- 
2.21.0

[PATCH 2/3] ocxl: Add pseries-specific code

[christophe lombard]

pseries.c implements the guest-specific callbacks for the backend API.

The hypervisor calls provide an interface to configure and interact with
OpenCAPI devices. It matches the last version of the 'PAPR changes'
document.

The following hcalls are supported:
H_OCXL_CONFIG_ADAPTER   Used to configure OpenCAPI adapter characteristics.

H_OCXL_CONFIG_SPA       Used to configure the schedule process area (SPA)
                        table for an OCAPI device.

H_OCXL_GET_FAULT_STATE  Used to retrieve fault information from an
                        OpenCAPI device.

H_OCXL_HANDLE_FAULT     Used to respond to an OpenCAPI fault.

Each previous hcall supports a config flag parameter, to allows the guest
to manage the CAPI device.

The current values 0xf004 to 0xf007 have been chosen according the
available QEMU hcall values which are specific to qemu / KVM-on-POWER.

Two parameters are common to all hcalls (buid and config_addr) that will
be used to allow QEMU to recover the PCI device.

Signed-off-by: Christophe Lombard <clombard@linux.vnet.ibm.com>
---
 drivers/misc/ocxl/Makefile        |   1 +
 drivers/misc/ocxl/main.c          |   4 +
 drivers/misc/ocxl/ocxl_internal.h |   1 +
 drivers/misc/ocxl/pseries.c       | 450 ++++++++++++++++++++++++++++++
 4 files changed, 456 insertions(+)
 create mode 100644 drivers/misc/ocxl/pseries.c

diff --git a/drivers/misc/ocxl/Makefile b/drivers/misc/ocxl/Makefile
index bfdaeb232b83..3474e912c402 100644
--- a/drivers/misc/ocxl/Makefile
+++ b/drivers/misc/ocxl/Makefile
@@ -5,6 +5,7 @@ ocxl-y				+= main.o pci.o config.o file.o pasid.o mmio.o
 ocxl-y				+= link.o context.o afu_irq.o sysfs.o trace.o
 ocxl-y				+= core.o
 ocxl-$(CONFIG_PPC_POWERNV)	+= powernv.o
+ocxl-$(CONFIG_PPC_PSERIES)	+= pseries.o
 
 obj-$(CONFIG_OCXL)		+= ocxl.o
 
diff --git a/drivers/misc/ocxl/main.c b/drivers/misc/ocxl/main.c
index 95df2ba4d473..bdd9ffa7f769 100644
--- a/drivers/misc/ocxl/main.c
+++ b/drivers/misc/ocxl/main.c
@@ -16,6 +16,10 @@ static int __init init_ocxl(void)
 
 	if (cpu_has_feature(CPU_FTR_HVMODE))
 		ocxl_ops = &ocxl_powernv_ops;
+#ifdef CONFIG_PPC_PSERIES
+	else
+		ocxl_ops = &ocxl_pseries_ops;
+#endif
 
 	rc = pci_register_driver(&ocxl_pci_driver);
 	if (rc) {
diff --git a/drivers/misc/ocxl/ocxl_internal.h b/drivers/misc/ocxl/ocxl_internal.h
index 2bdea279bdc6..c18b32df3fe5 100644
--- a/drivers/misc/ocxl/ocxl_internal.h
+++ b/drivers/misc/ocxl/ocxl_internal.h
@@ -104,6 +104,7 @@ struct ocxl_backend_ops {
 };
 
 extern const struct ocxl_backend_ops ocxl_powernv_ops;
+extern const struct ocxl_backend_ops ocxl_pseries_ops;
 extern const struct ocxl_backend_ops *ocxl_ops;
 
 int ocxl_create_cdev(struct ocxl_afu *afu);
diff --git a/drivers/misc/ocxl/pseries.c b/drivers/misc/ocxl/pseries.c
new file mode 100644
index 000000000000..1d4942d713f7
--- /dev/null
+++ b/drivers/misc/ocxl/pseries.c
@@ -0,0 +1,450 @@
+// SPDX-License-Identifier: GPL-2.0+
+// Copyright 2018 IBM Corp.
+#include <misc/ocxl-config.h>
+#include "ocxl_internal.h"
+#include <asm/xive.h>
+
+#define H_OCXL_CONFIG_ADAPTER	0xf004
+#define H_OCXL_CONFIG_SPA	0xf005
+#define H_OCXL_GET_FAULT_STATE	0xf006
+#define H_OCXL_HANDLE_FAULT	0xf007
+
+#define H_CONFIG_ADAPTER_SETUP		1
+#define H_CONFIG_ADAPTER_RELEASE	2
+#define H_CONFIG_ADAPTER_GET_ACTAG	3
+#define H_CONFIG_ADAPTER_GET_PASID	4
+#define H_CONFIG_ADAPTER_SET_TL		5
+#define H_CONFIG_ADAPTER_ALLOC_IRQ	6
+#define H_CONFIG_ADAPTER_FREE_IRQ	7
+
+#define H_CONFIG_SPA_SET		1
+#define H_CONFIG_SPA_UPDATE		2
+#define H_CONFIG_SPA_REMOVE		3
+
+static char *config_adapter_names[] = {
+	"UNKNOWN_OP",		/* 0 undefined */
+	"SETUP",		/* 1 */
+	"RELEASE",		/* 2 */
+	"GET_ACTAG",		/* 3 */
+	"GET_PASID",		/* 4 */
+	"SET_TL",		/* 5 */
+	"ALLOC_IRQ",		/* 6 */
+	"FREE_IRQ",		/* 7 */
+};
+
+static char *config_spa_names[] = {
+	"UNKNOWN_OP",		/* 0 undefined */
+	"SET",			/* 1 */
+	"UPDATE",		/* 2 */
+	"REMOVE",		/* 3 */
+};
+
+static char *op_str(unsigned int op, char *names[], int len)
+{
+	if (op >= len)
+		return "UNKNOWN_OP";
+	return names[op];
+}
+
+#define OP_STR(op, names)	op_str(op, names, ARRAY_SIZE(names))
+#define OP_STR_CA(op)		OP_STR(op, config_adapter_names)
+#define OP_STR_CS(op)		OP_STR(op, config_spa_names)
+
+#define _PRINT_MSG(rc, format, ...)					\
+	{								\
+		if (rc != H_SUCCESS && rc != H_CONTINUE)		\
+			pr_err(format, __VA_ARGS__);			\
+		else							\
+			pr_debug(format, __VA_ARGS__);			\
+	}								\
+
+struct pseries_data {
+	u64 buid;
+	u32 config_addr;
+	int fault_lisn;
+	int host_irq;
+};
+
+static unsigned int busy_delay(long rc)
+{
+	unsigned int ms = 0;
+
+	if (H_IS_LONG_BUSY(rc))
+		ms = get_longbusy_msecs(rc);
+	else if (rc == H_BUSY)
+		ms = 10;
+
+	if (ms)
+		mdelay(ms);
+
+	return ms;
+}
+
+static long config_adapter(unsigned long *retbuf, u64 *params)
+{
+	long rc;
+
+	do {
+		rc = plpar_hcall9(H_OCXL_CONFIG_ADAPTER, retbuf,
+				  params[0], params[1], params[2],
+				  params[3], params[4]);
+	} while (busy_delay(rc));
+
+	_PRINT_MSG(rc, "%s - buid:%#llx, config_addr: %#llx "
+			"%s(%#llx, %#llx, "
+			"retbuf1: %#lx, retbuf2: %#lx, retbuf3: %#lx): %li\n",
+			__func__, params[0], params[1],
+			OP_STR_CA(params[2]),
+			params[3], params[4],
+			retbuf[0], retbuf[1], retbuf[2], rc);
+
+	switch (rc) {
+	case H_SUCCESS:
+		return 0;
+	case H_PARAMETER:
+	case H_FUNCTION:
+	case H_NOT_FOUND:
+	case H_NOT_AVAILABLE:
+	case H_SG_LIST:
+		return -EINVAL;
+	case H_AUTHORITY:
+	case H_RESOURCE:
+	case H_HARDWARE:
+	case H_STATE:
+	case H_BUSY:
+		return -EBUSY;
+	default:
+		WARN(1, "Unexpected return code: %lx", rc);
+		return -EINVAL;
+	}
+}
+
+static int config_spa(unsigned long *retbuf, u64 *params)
+{
+	long rc;
+
+	do {
+		rc = plpar_hcall9(H_OCXL_CONFIG_SPA, retbuf,
+				  params[0], params[1], params[2],
+				  params[3], params[4], params[5],
+				  params[6]);
+	} while (busy_delay(rc));
+
+	_PRINT_MSG(rc, "%s - buid:%#llx, config_addr: %#llx "
+			"%s(%#llx, %#llx, %#llx, %#llx, "
+			"retbuf1: %#lx, retbuf2: %#lx, retbuf3: %#lx): %li\n",
+			__func__, params[0], params[1],
+			OP_STR_CS(params[2]),
+			params[3], params[4], params[5], params[6],
+			retbuf[0], retbuf[1], retbuf[2], rc);
+
+	switch (rc) {
+	case H_SUCCESS:
+		return 0;
+	case H_PARAMETER:
+	case H_FUNCTION:
+		return -EINVAL;
+	case H_AUTHORITY:
+	case H_RESOURCE:
+	case H_HARDWARE:
+	case H_STATE:
+	case H_BUSY:
+		return -EBUSY;
+	default:
+		WARN(1, "Unexpected return code: %lx", rc);
+		return -EINVAL;
+	}
+}
+
+static u64 get_buid(struct pci_dev *dev)
+{
+	struct device_node *dn;
+	struct pci_dn *pdn;
+
+	dn = pci_device_to_OF_node(dev);
+	pdn = PCI_DN(dn);
+	return pdn->phb->buid;
+}
+
+static u32 get_config_addr(struct pci_dev *dev)
+{
+	int bus, devfn;
+	u32 config_addr;
+
+	bus = dev->bus->number;
+	devfn = dev->devfn;
+	config_addr = ((bus & 0xFF) << 16) + ((devfn & 0xFF) << 8);
+	return config_addr;
+}
+
+static void ack_irq(void *data, u64 tfc)
+{
+	unsigned long retbuf[PLPAR_HCALL_BUFSIZE];
+	struct pseries_data *pdata = data;
+	long rc;
+
+	do {
+		rc = plpar_hcall(H_OCXL_HANDLE_FAULT, retbuf,
+				 pdata->buid, pdata->config_addr, tfc);
+	} while (busy_delay(rc));
+
+	_PRINT_MSG(rc, "%s - buid:%#llx, config_addr: %#x "
+			"(tfc: %#llx): %li\n",
+			__func__, pdata->buid, pdata->config_addr,
+			tfc, rc);
+}
+
+static int alloc_xive_irq(void *data, u32 *irq,
+			  u64 *trigger_addr)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+	struct pseries_data *pdata = data;
+	u64 params[PLPAR_HCALL9_BUFSIZE];
+	struct xive_irq_data *xd;
+	int virq;
+	long rc;
+
+	params[0] = pdata->buid;
+	params[1] = pdata->config_addr;
+	params[2] = H_CONFIG_ADAPTER_ALLOC_IRQ;
+
+	rc = config_adapter(retbuf, params);
+	if (rc)
+		return rc;
+
+	*irq = retbuf[0];
+
+	virq = irq_create_mapping(NULL, *irq);
+	if (!virq) {
+		pr_err("irq_create_mapping failed\n");
+		return -EINVAL;
+	}
+
+	xd = irq_get_handler_data(virq);
+	if (!xd) {
+		pr_err("irq_get_handler_data failed\n");
+		return -EINVAL;
+	}
+	/**trigger_addr = xd->trig_page;*/
+	*trigger_addr = xd->eoi_page;
+
+	pr_debug("%s - buid: %#llx, irq: %d, trigger_addr: %#llx\n",
+		 __func__, pdata->buid, *irq, *trigger_addr);
+	return rc;
+}
+
+static void free_xive_irq(void *data, u32 irq)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+	struct pseries_data *pdata = data;
+	u64 params[PLPAR_HCALL9_BUFSIZE];
+
+	params[0] = pdata->buid;
+	params[1] = pdata->config_addr;
+	params[2] = H_CONFIG_ADAPTER_FREE_IRQ;
+	params[3] = irq;
+
+	config_adapter(retbuf, params);
+}
+
+static int get_actag(struct pci_dev *dev, u16 *base,
+		     u16 *enabled, u16 *supported)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+	u64 params[PLPAR_HCALL9_BUFSIZE];
+	int rc = 0;
+
+	params[0] = get_buid(dev);
+	params[1] = get_config_addr(dev);
+	params[2] = H_CONFIG_ADAPTER_GET_ACTAG;
+
+	rc = config_adapter(retbuf, params);
+	if (!rc) {
+		*base = retbuf[0];
+		*enabled = retbuf[1];
+		*supported = retbuf[2];
+	}
+	return rc;
+}
+
+static void get_fault_state(void *data, u64 *dsisr, u64 *dar,
+			    u64 *pe_handle, int *pid)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+	struct pseries_data *pdata = data;
+	long rc;
+
+	do {
+		rc = plpar_hcall9(H_OCXL_GET_FAULT_STATE, retbuf,
+				  pdata->buid, pdata->config_addr);
+	} while (busy_delay(rc));
+
+	_PRINT_MSG(rc, "%s - buid:%#llx, config_addr: %#x "
+			"(out1: %#lx, out2: %#lx, out3: %#lx, "
+			"out4: %#lx): %li\n",
+			__func__, pdata->buid, pdata->config_addr,
+			retbuf[0], retbuf[1], retbuf[2], retbuf[3], rc);
+
+	*dsisr = retbuf[0];
+	*dar = retbuf[1];
+	*pe_handle = retbuf[2];
+	*pid = retbuf[3];
+}
+
+static int get_pasid_count(struct pci_dev *dev, int *count)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+	u64 params[PLPAR_HCALL9_BUFSIZE];
+	int rc = 0;
+
+	params[0] = get_buid(dev);
+	params[1] = get_config_addr(dev);
+	params[2] = H_CONFIG_ADAPTER_GET_PASID;
+
+	rc = config_adapter(retbuf, params);
+	if (!rc)
+		*count = retbuf[0];
+	return rc;
+}
+
+static void platform_release(void *data)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+	struct pseries_data *pdata = data;
+	u64 params[PLPAR_HCALL9_BUFSIZE];
+
+	params[0] = pdata->buid;
+	params[1] = pdata->config_addr;
+	params[2] = H_CONFIG_ADAPTER_RELEASE;
+	params[3] = pdata->host_irq;
+	params[4] = pdata->fault_lisn;
+
+	config_adapter(retbuf, params);
+
+	kfree(pdata);
+}
+
+static int platform_setup(struct pci_dev *dev, int PE_mask, int *hwirq,
+			  void **data)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+	u64 params[PLPAR_HCALL9_BUFSIZE];
+	struct pseries_data *pdata;
+	int rc = 0;
+
+	rc = of_property_read_u32(dev->dev.of_node, "fault-lisn", hwirq);
+	if (rc) {
+		pr_err("fault-lisn not found (%d)\n", rc);
+		return rc;
+	}
+
+	pdata = kzalloc(sizeof(*pdata), GFP_KERNEL);
+	if (!pdata)
+		return -ENOMEM;
+
+	pdata->buid = get_buid(dev);
+	pdata->config_addr = get_config_addr(dev);
+	*data = pdata;
+
+	params[0] = pdata->buid;
+	params[1] = pdata->config_addr;
+	params[2] = H_CONFIG_ADAPTER_SETUP;
+	params[3] = PE_mask;
+	params[4] = *hwirq;
+
+	rc = config_adapter(retbuf, params);
+	if (!rc) {
+		pdata->fault_lisn = *hwirq;
+		pdata->host_irq = retbuf[0];
+	}
+	return rc;
+}
+
+static int remove_pe(void *data, int pasid, u32 *pid,
+		     u32 *tid, int *pe_handle)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+	struct pseries_data *pdata = data;
+	u64 params[PLPAR_HCALL9_BUFSIZE];
+	int rc;
+
+	params[0] = pdata->buid;
+	params[1] = pdata->config_addr;
+	params[2] = H_CONFIG_SPA_REMOVE;
+	params[3] = pasid;
+
+	rc = config_spa(retbuf, params);
+	if (!rc) {
+		*pid = retbuf[0];
+		*tid = retbuf[1];
+		*pe_handle = retbuf[2];
+	}
+	return rc;
+}
+
+static int set_pe(void *data, int pasid, u32 pidr, u32 tidr,
+		  u64 amr, int *pe_handle)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+	struct pseries_data *pdata = data;
+	u64 params[PLPAR_HCALL9_BUFSIZE];
+	int rc;
+
+	params[0] = pdata->buid;
+	params[1] = pdata->config_addr;
+	params[2] = H_CONFIG_SPA_SET;
+	params[3] = pasid;
+	params[4] = pidr;
+	params[5] = tidr;
+	params[6] = amr;
+
+	rc = config_spa(retbuf, params);
+	if (!rc)
+		*pe_handle = retbuf[0];
+
+	return rc;
+}
+
+static int set_tl(struct pci_dev *dev, int tl_dvsec)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+	u64 params[PLPAR_HCALL9_BUFSIZE];
+
+	params[0] = get_buid(dev);
+	params[1] = get_config_addr(dev);
+	params[2] = H_CONFIG_ADAPTER_SET_TL;
+	params[3] = tl_dvsec;
+
+	return config_adapter(retbuf, params);
+}
+
+static int update_pe(void *data, int pasid, __u16 tid)
+{
+	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
+	struct pseries_data *pdata = data;
+	u64 params[PLPAR_HCALL9_BUFSIZE];
+
+	params[0] = pdata->buid;
+	params[1] = pdata->config_addr;
+	params[2] = H_CONFIG_SPA_UPDATE;
+	params[3] = pasid;
+	params[4] = tid;
+
+	return config_spa(retbuf, params);
+}
+
+const struct ocxl_backend_ops ocxl_pseries_ops = {
+	.module = THIS_MODULE,
+	.ack_irq = ack_irq,
+	.alloc_xive_irq = alloc_xive_irq,
+	.free_xive_irq = free_xive_irq,
+	.get_actag = get_actag,
+	.get_fault_state = get_fault_state,
+	.get_pasid_count = get_pasid_count,
+	.platform_release = platform_release,
+	.platform_setup = platform_setup,
+	.remove_pe = remove_pe,
+	.set_pe = set_pe,
+	.set_tl = set_tl,
+	.update_pe = update_pe,
+};
-- 
2.21.0

Re: [PATCH 2/3] ocxl: Add pseries-specific code

[christophe lombard]

On 22/10/2019 09:52, christophe lombard wrote:
> pseries.c implements the guest-specific callbacks for the backend API.
> 
> The hypervisor calls provide an interface to configure and interact with
> OpenCAPI devices. It matches the last version of the 'PAPR changes'
> document.
> 
> The following hcalls are supported:
> H_OCXL_CONFIG_ADAPTER   Used to configure OpenCAPI adapter characteristics.
> 
> H_OCXL_CONFIG_SPA       Used to configure the schedule process area (SPA)
>                          table for an OCAPI device.
> 
> H_OCXL_GET_FAULT_STATE  Used to retrieve fault information from an
>                          OpenCAPI device.
> 
> H_OCXL_HANDLE_FAULT     Used to respond to an OpenCAPI fault.
> 
> Each previous hcall supports a config flag parameter, to allows the guest
> to manage the CAPI device.
> 
> The current values 0xf004 to 0xf007 have been chosen according the
> available QEMU hcall values which are specific to qemu / KVM-on-POWER.
> 
> Two parameters are common to all hcalls (buid and config_addr) that will
> be used to allow QEMU to recover the PCI device.
> 
> Signed-off-by: Christophe Lombard <clombard@linux.vnet.ibm.com>
> ---
>   drivers/misc/ocxl/Makefile        |   1 +
>   drivers/misc/ocxl/main.c          |   4 +
>   drivers/misc/ocxl/ocxl_internal.h |   1 +
>   drivers/misc/ocxl/pseries.c       | 450 ++++++++++++++++++++++++++++++
>   4 files changed, 456 insertions(+)
>   create mode 100644 drivers/misc/ocxl/pseries.c
> 
> diff --git a/drivers/misc/ocxl/Makefile b/drivers/misc/ocxl/Makefile
> index bfdaeb232b83..3474e912c402 100644
> --- a/drivers/misc/ocxl/Makefile
> +++ b/drivers/misc/ocxl/Makefile
> @@ -5,6 +5,7 @@ ocxl-y				+= main.o pci.o config.o file.o pasid.o mmio.o
>   ocxl-y				+= link.o context.o afu_irq.o sysfs.o trace.o
>   ocxl-y				+= core.o
>   ocxl-$(CONFIG_PPC_POWERNV)	+= powernv.o
> +ocxl-$(CONFIG_PPC_PSERIES)	+= pseries.o
> 
>   obj-$(CONFIG_OCXL)		+= ocxl.o
> 
> diff --git a/drivers/misc/ocxl/main.c b/drivers/misc/ocxl/main.c
> index 95df2ba4d473..bdd9ffa7f769 100644
> --- a/drivers/misc/ocxl/main.c
> +++ b/drivers/misc/ocxl/main.c
> @@ -16,6 +16,10 @@ static int __init init_ocxl(void)
> 
>   	if (cpu_has_feature(CPU_FTR_HVMODE))
>   		ocxl_ops = &ocxl_powernv_ops;
> +#ifdef CONFIG_PPC_PSERIES
> +	else
> +		ocxl_ops = &ocxl_pseries_ops;
> +#endif
> 
>   	rc = pci_register_driver(&ocxl_pci_driver);
>   	if (rc) {
> diff --git a/drivers/misc/ocxl/ocxl_internal.h b/drivers/misc/ocxl/ocxl_internal.h
> index 2bdea279bdc6..c18b32df3fe5 100644
> --- a/drivers/misc/ocxl/ocxl_internal.h
> +++ b/drivers/misc/ocxl/ocxl_internal.h
> @@ -104,6 +104,7 @@ struct ocxl_backend_ops {
>   };
> 
>   extern const struct ocxl_backend_ops ocxl_powernv_ops;
> +extern const struct ocxl_backend_ops ocxl_pseries_ops;
>   extern const struct ocxl_backend_ops *ocxl_ops;
> 
>   int ocxl_create_cdev(struct ocxl_afu *afu);
> diff --git a/drivers/misc/ocxl/pseries.c b/drivers/misc/ocxl/pseries.c
> new file mode 100644
> index 000000000000..1d4942d713f7
> --- /dev/null
> +++ b/drivers/misc/ocxl/pseries.c
> @@ -0,0 +1,450 @@
> +// SPDX-License-Identifier: GPL-2.0+
> +// Copyright 2018 IBM Corp.
> +#include <misc/ocxl-config.h>
> +#include "ocxl_internal.h"
> +#include <asm/xive.h>
> +
> +#define H_OCXL_CONFIG_ADAPTER	0xf004
> +#define H_OCXL_CONFIG_SPA	0xf005
> +#define H_OCXL_GET_FAULT_STATE	0xf006
> +#define H_OCXL_HANDLE_FAULT	0xf007
> +
> +#define H_CONFIG_ADAPTER_SETUP		1
> +#define H_CONFIG_ADAPTER_RELEASE	2
> +#define H_CONFIG_ADAPTER_GET_ACTAG	3
> +#define H_CONFIG_ADAPTER_GET_PASID	4
> +#define H_CONFIG_ADAPTER_SET_TL		5
> +#define H_CONFIG_ADAPTER_ALLOC_IRQ	6
> +#define H_CONFIG_ADAPTER_FREE_IRQ	7
> +
> +#define H_CONFIG_SPA_SET		1
> +#define H_CONFIG_SPA_UPDATE		2
> +#define H_CONFIG_SPA_REMOVE		3
> +
> +static char *config_adapter_names[] = {
> +	"UNKNOWN_OP",		/* 0 undefined */
> +	"SETUP",		/* 1 */
> +	"RELEASE",		/* 2 */
> +	"GET_ACTAG",		/* 3 */
> +	"GET_PASID",		/* 4 */
> +	"SET_TL",		/* 5 */
> +	"ALLOC_IRQ",		/* 6 */
> +	"FREE_IRQ",		/* 7 */
> +};
> +
> +static char *config_spa_names[] = {
> +	"UNKNOWN_OP",		/* 0 undefined */
> +	"SET",			/* 1 */
> +	"UPDATE",		/* 2 */
> +	"REMOVE",		/* 3 */
> +};
> +
> +static char *op_str(unsigned int op, char *names[], int len)
> +{
> +	if (op >= len)
> +		return "UNKNOWN_OP";
> +	return names[op];
> +}
> +
> +#define OP_STR(op, names)	op_str(op, names, ARRAY_SIZE(names))
> +#define OP_STR_CA(op)		OP_STR(op, config_adapter_names)
> +#define OP_STR_CS(op)		OP_STR(op, config_spa_names)
> +
> +#define _PRINT_MSG(rc, format, ...)					\
> +	{								\
> +		if (rc != H_SUCCESS && rc != H_CONTINUE)		\
> +			pr_err(format, __VA_ARGS__);			\
> +		else							\
> +			pr_debug(format, __VA_ARGS__);			\
> +	}								\
> +
> +struct pseries_data {
> +	u64 buid;
> +	u32 config_addr;
> +	int fault_lisn;
> +	int host_irq;
> +};
> +
> +static unsigned int busy_delay(long rc)
> +{
> +	unsigned int ms = 0;
> +
> +	if (H_IS_LONG_BUSY(rc))
> +		ms = get_longbusy_msecs(rc);
> +	else if (rc == H_BUSY)
> +		ms = 10;
> +
> +	if (ms)
> +		mdelay(ms);
> +
> +	return ms;
> +}
> +
> +static long config_adapter(unsigned long *retbuf, u64 *params)
> +{
> +	long rc;
> +
> +	do {
> +		rc = plpar_hcall9(H_OCXL_CONFIG_ADAPTER, retbuf,
> +				  params[0], params[1], params[2],
> +				  params[3], params[4]);
> +	} while (busy_delay(rc));
> +
> +	_PRINT_MSG(rc, "%s - buid:%#llx, config_addr: %#llx "
> +			"%s(%#llx, %#llx, "
> +			"retbuf1: %#lx, retbuf2: %#lx, retbuf3: %#lx): %li\n",
> +			__func__, params[0], params[1],
> +			OP_STR_CA(params[2]),
> +			params[3], params[4],
> +			retbuf[0], retbuf[1], retbuf[2], rc);
> +
> +	switch (rc) {
> +	case H_SUCCESS:
> +		return 0;
> +	case H_PARAMETER:
> +	case H_FUNCTION:
> +	case H_NOT_FOUND:
> +	case H_NOT_AVAILABLE:
> +	case H_SG_LIST:
> +		return -EINVAL;
> +	case H_AUTHORITY:
> +	case H_RESOURCE:
> +	case H_HARDWARE:
> +	case H_STATE:
> +	case H_BUSY:
> +		return -EBUSY;
> +	default:
> +		WARN(1, "Unexpected return code: %lx", rc);
> +		return -EINVAL;
> +	}
> +}
> +
> +static int config_spa(unsigned long *retbuf, u64 *params)
> +{
> +	long rc;
> +
> +	do {
> +		rc = plpar_hcall9(H_OCXL_CONFIG_SPA, retbuf,
> +				  params[0], params[1], params[2],
> +				  params[3], params[4], params[5],
> +				  params[6]);
> +	} while (busy_delay(rc));
> +
> +	_PRINT_MSG(rc, "%s - buid:%#llx, config_addr: %#llx "
> +			"%s(%#llx, %#llx, %#llx, %#llx, "
> +			"retbuf1: %#lx, retbuf2: %#lx, retbuf3: %#lx): %li\n",
> +			__func__, params[0], params[1],
> +			OP_STR_CS(params[2]),
> +			params[3], params[4], params[5], params[6],
> +			retbuf[0], retbuf[1], retbuf[2], rc);
> +
> +	switch (rc) {
> +	case H_SUCCESS:
> +		return 0;
> +	case H_PARAMETER:
> +	case H_FUNCTION:
> +		return -EINVAL;
> +	case H_AUTHORITY:
> +	case H_RESOURCE:
> +	case H_HARDWARE:
> +	case H_STATE:
> +	case H_BUSY:
> +		return -EBUSY;
> +	default:
> +		WARN(1, "Unexpected return code: %lx", rc);
> +		return -EINVAL;
> +	}
> +}
> +
> +static u64 get_buid(struct pci_dev *dev)
> +{
> +	struct device_node *dn;
> +	struct pci_dn *pdn;
> +
> +	dn = pci_device_to_OF_node(dev);
> +	pdn = PCI_DN(dn);
> +	return pdn->phb->buid;
> +}
> +
> +static u32 get_config_addr(struct pci_dev *dev)
> +{
> +	int bus, devfn;
> +	u32 config_addr;
> +
> +	bus = dev->bus->number;
> +	devfn = dev->devfn;
> +	config_addr = ((bus & 0xFF) << 16) + ((devfn & 0xFF) << 8);
> +	return config_addr;
> +}
> +
> +static void ack_irq(void *data, u64 tfc)
> +{
> +	unsigned long retbuf[PLPAR_HCALL_BUFSIZE];
> +	struct pseries_data *pdata = data;
> +	long rc;
> +
> +	do {
> +		rc = plpar_hcall(H_OCXL_HANDLE_FAULT, retbuf,
> +				 pdata->buid, pdata->config_addr, tfc);
> +	} while (busy_delay(rc));
> +
> +	_PRINT_MSG(rc, "%s - buid:%#llx, config_addr: %#x "
> +			"(tfc: %#llx): %li\n",
> +			__func__, pdata->buid, pdata->config_addr,
> +			tfc, rc);
> +}
> +
> +static int alloc_xive_irq(void *data, u32 *irq,
> +			  u64 *trigger_addr)
> +{
> +	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
> +	struct pseries_data *pdata = data;
> +	u64 params[PLPAR_HCALL9_BUFSIZE];
> +	struct xive_irq_data *xd;
> +	int virq;
> +	long rc;
> +
> +	params[0] = pdata->buid;
> +	params[1] = pdata->config_addr;
> +	params[2] = H_CONFIG_ADAPTER_ALLOC_IRQ;
> +
> +	rc = config_adapter(retbuf, params);
> +	if (rc)
> +		return rc;
> +
> +	*irq = retbuf[0];
> +
> +	virq = irq_create_mapping(NULL, *irq);
> +	if (!virq) {
> +		pr_err("irq_create_mapping failed\n");
> +		return -EINVAL;
> +	}
> +
> +	xd = irq_get_handler_data(virq);
> +	if (!xd) {
> +		pr_err("irq_get_handler_data failed\n");

irq_dispose_mapping(virq) has to be called in case of error.

> +		return -EINVAL;
> +	}
> +	/**trigger_addr = xd->trig_page;*/
> +	*trigger_addr = xd->eoi_page;
> +
> +	pr_debug("%s - buid: %#llx, irq: %d, trigger_addr: %#llx\n",
> +		 __func__, pdata->buid, *irq, *trigger_addr);

Same remark, here for irq_dispose_mapping(virq).
irq_create_mapping() was called previously to get trigger_addr,
so no reason to keep virq outside this function.

> +	return rc;
> +}
> +
> +static void free_xive_irq(void *data, u32 irq)
> +{
> +	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
> +	struct pseries_data *pdata = data;
> +	u64 params[PLPAR_HCALL9_BUFSIZE];
> +
> +	params[0] = pdata->buid;
> +	params[1] = pdata->config_addr;
> +	params[2] = H_CONFIG_ADAPTER_FREE_IRQ;
> +	params[3] = irq;
> +
> +	config_adapter(retbuf, params);
> +}
> +
> +static int get_actag(struct pci_dev *dev, u16 *base,
> +		     u16 *enabled, u16 *supported)
> +{
> +	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
> +	u64 params[PLPAR_HCALL9_BUFSIZE];
> +	int rc = 0;
> +
> +	params[0] = get_buid(dev);
> +	params[1] = get_config_addr(dev);
> +	params[2] = H_CONFIG_ADAPTER_GET_ACTAG;
> +
> +	rc = config_adapter(retbuf, params);
> +	if (!rc) {
> +		*base = retbuf[0];
> +		*enabled = retbuf[1];
> +		*supported = retbuf[2];
> +	}
> +	return rc;
> +}
> +
> +static void get_fault_state(void *data, u64 *dsisr, u64 *dar,
> +			    u64 *pe_handle, int *pid)
> +{
> +	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
> +	struct pseries_data *pdata = data;
> +	long rc;
> +
> +	do {
> +		rc = plpar_hcall9(H_OCXL_GET_FAULT_STATE, retbuf,
> +				  pdata->buid, pdata->config_addr);
> +	} while (busy_delay(rc));
> +
> +	_PRINT_MSG(rc, "%s - buid:%#llx, config_addr: %#x "
> +			"(out1: %#lx, out2: %#lx, out3: %#lx, "
> +			"out4: %#lx): %li\n",
> +			__func__, pdata->buid, pdata->config_addr,
> +			retbuf[0], retbuf[1], retbuf[2], retbuf[3], rc);
> +
> +	*dsisr = retbuf[0];
> +	*dar = retbuf[1];
> +	*pe_handle = retbuf[2];
> +	*pid = retbuf[3];
> +}
> +
> +static int get_pasid_count(struct pci_dev *dev, int *count)
> +{
> +	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
> +	u64 params[PLPAR_HCALL9_BUFSIZE];
> +	int rc = 0;
> +
> +	params[0] = get_buid(dev);
> +	params[1] = get_config_addr(dev);
> +	params[2] = H_CONFIG_ADAPTER_GET_PASID;
> +
> +	rc = config_adapter(retbuf, params);
> +	if (!rc)
> +		*count = retbuf[0];
> +	return rc;
> +}
> +
> +static void platform_release(void *data)
> +{
> +	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
> +	struct pseries_data *pdata = data;
> +	u64 params[PLPAR_HCALL9_BUFSIZE];
> +
> +	params[0] = pdata->buid;
> +	params[1] = pdata->config_addr;
> +	params[2] = H_CONFIG_ADAPTER_RELEASE;
> +	params[3] = pdata->host_irq;
> +	params[4] = pdata->fault_lisn;
> +
> +	config_adapter(retbuf, params);
> +
> +	kfree(pdata);
> +}
> +
> +static int platform_setup(struct pci_dev *dev, int PE_mask, int *hwirq,
> +			  void **data)
> +{
> +	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
> +	u64 params[PLPAR_HCALL9_BUFSIZE];
> +	struct pseries_data *pdata;
> +	int rc = 0;
> +
> +	rc = of_property_read_u32(dev->dev.of_node, "fault-lisn", hwirq);
> +	if (rc) {
> +		pr_err("fault-lisn not found (%d)\n", rc);
> +		return rc;
> +	}
> +
> +	pdata = kzalloc(sizeof(*pdata), GFP_KERNEL);
> +	if (!pdata)
> +		return -ENOMEM;
> +
> +	pdata->buid = get_buid(dev);
> +	pdata->config_addr = get_config_addr(dev);
> +	*data = pdata;
> +
> +	params[0] = pdata->buid;
> +	params[1] = pdata->config_addr;
> +	params[2] = H_CONFIG_ADAPTER_SETUP;
> +	params[3] = PE_mask;
> +	params[4] = *hwirq;
> +
> +	rc = config_adapter(retbuf, params);
> +	if (!rc) {
> +		pdata->fault_lisn = *hwirq;
> +		pdata->host_irq = retbuf[0];
> +	}
> +	return rc;
> +}
> +
> +static int remove_pe(void *data, int pasid, u32 *pid,
> +		     u32 *tid, int *pe_handle)
> +{
> +	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
> +	struct pseries_data *pdata = data;
> +	u64 params[PLPAR_HCALL9_BUFSIZE];
> +	int rc;
> +
> +	params[0] = pdata->buid;
> +	params[1] = pdata->config_addr;
> +	params[2] = H_CONFIG_SPA_REMOVE;
> +	params[3] = pasid;
> +
> +	rc = config_spa(retbuf, params);
> +	if (!rc) {
> +		*pid = retbuf[0];
> +		*tid = retbuf[1];
> +		*pe_handle = retbuf[2];
> +	}
> +	return rc;
> +}
> +
> +static int set_pe(void *data, int pasid, u32 pidr, u32 tidr,
> +		  u64 amr, int *pe_handle)
> +{
> +	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
> +	struct pseries_data *pdata = data;
> +	u64 params[PLPAR_HCALL9_BUFSIZE];
> +	int rc;
> +
> +	params[0] = pdata->buid;
> +	params[1] = pdata->config_addr;
> +	params[2] = H_CONFIG_SPA_SET;
> +	params[3] = pasid;
> +	params[4] = pidr;
> +	params[5] = tidr;
> +	params[6] = amr;
> +
> +	rc = config_spa(retbuf, params);
> +	if (!rc)
> +		*pe_handle = retbuf[0];
> +
> +	return rc;
> +}
> +
> +static int set_tl(struct pci_dev *dev, int tl_dvsec)
> +{
> +	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
> +	u64 params[PLPAR_HCALL9_BUFSIZE];
> +
> +	params[0] = get_buid(dev);
> +	params[1] = get_config_addr(dev);
> +	params[2] = H_CONFIG_ADAPTER_SET_TL;
> +	params[3] = tl_dvsec;
> +
> +	return config_adapter(retbuf, params);
> +}
> +
> +static int update_pe(void *data, int pasid, __u16 tid)
> +{
> +	unsigned long retbuf[PLPAR_HCALL9_BUFSIZE];
> +	struct pseries_data *pdata = data;
> +	u64 params[PLPAR_HCALL9_BUFSIZE];
> +
> +	params[0] = pdata->buid;
> +	params[1] = pdata->config_addr;
> +	params[2] = H_CONFIG_SPA_UPDATE;
> +	params[3] = pasid;
> +	params[4] = tid;
> +
> +	return config_spa(retbuf, params);
> +}
> +
> +const struct ocxl_backend_ops ocxl_pseries_ops = {
> +	.module = THIS_MODULE,
> +	.ack_irq = ack_irq,
> +	.alloc_xive_irq = alloc_xive_irq,
> +	.free_xive_irq = free_xive_irq,
> +	.get_actag = get_actag,
> +	.get_fault_state = get_fault_state,
> +	.get_pasid_count = get_pasid_count,
> +	.platform_release = platform_release,
> +	.platform_setup = platform_setup,
> +	.remove_pe = remove_pe,
> +	.set_pe = set_pe,
> +	.set_tl = set_tl,
> +	.update_pe = update_pe,
> +};
> 

[PATCH 3/3] powerpc/pseries: Fixup config space size of OpenCAPI devices

[christophe lombard]

Fix up the pci config size of the OpenCAPI PCIe devices in the pseries
environment.
Most of OpenCAPI PCIe devices have 4096 bytes of configuration space.

Signed-off-by: Christophe Lombard <clombard@linux.vnet.ibm.com>
---
 arch/powerpc/platforms/pseries/pci.c | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/arch/powerpc/platforms/pseries/pci.c b/arch/powerpc/platforms/pseries/pci.c
index 1eae1d09980c..3397784767b0 100644
--- a/arch/powerpc/platforms/pseries/pci.c
+++ b/arch/powerpc/platforms/pseries/pci.c
@@ -291,6 +291,15 @@ static void fixup_winbond_82c105(struct pci_dev* dev)
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_WINBOND, PCI_DEVICE_ID_WINBOND_82C105,
 			 fixup_winbond_82c105);
 
+static void fixup_opencapi_cfg_size(struct pci_dev *pdev)
+{
+	if (!machine_is(pseries))
+		return;
+
+	pdev->cfg_size = PCI_CFG_SPACE_EXP_SIZE;
+}
+DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_IBM, 0x062b, fixup_opencapi_cfg_size);
+
 int pseries_root_bridge_prepare(struct pci_host_bridge *bridge)
 {
 	struct device_node *dn, *pdn;
-- 
2.21.0

Re: [PATCH 3/3] powerpc/pseries: Fixup config space size of OpenCAPI devices

[Andrew Donnellan]

On 22/10/19 6:52 pm, christophe lombard wrote:
> Fix up the pci config size of the OpenCAPI PCIe devices in the pseries
> environment.
> Most of OpenCAPI PCIe devices have 4096 bytes of configuration space.

It's not "most of", it's "all" - the OpenCAPI Discovery and 
Configuration Spec requires the use of extended capabilities that fall 
in the 0x100-0xFFF range.

> 
> Signed-off-by: Christophe Lombard <clombard@linux.vnet.ibm.com>
> ---
>   arch/powerpc/platforms/pseries/pci.c | 9 +++++++++
>   1 file changed, 9 insertions(+)
> 
> diff --git a/arch/powerpc/platforms/pseries/pci.c b/arch/powerpc/platforms/pseries/pci.c
> index 1eae1d09980c..3397784767b0 100644
> --- a/arch/powerpc/platforms/pseries/pci.c
> +++ b/arch/powerpc/platforms/pseries/pci.c
> @@ -291,6 +291,15 @@ static void fixup_winbond_82c105(struct pci_dev* dev)
>   DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_WINBOND, PCI_DEVICE_ID_WINBOND_82C105,
>   			 fixup_winbond_82c105);
>   
> +static void fixup_opencapi_cfg_size(struct pci_dev *pdev)
> +{
> +	if (!machine_is(pseries))
> +		return;
> +
> +	pdev->cfg_size = PCI_CFG_SPACE_EXP_SIZE;
> +}
> +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_IBM, 0x062b, fixup_opencapi_cfg_size);

An OpenCAPI device can have any PCI ID, is there a particular reason 
we're limiting this to 1014:062b? On PowerNV, we check the PHB type to 
determine whether the device is OpenCAPI or not, what's the equivalent 
for pseries?

> +
>   int pseries_root_bridge_prepare(struct pci_host_bridge *bridge)
>   {
>   	struct device_node *dn, *pdn;
> 

-- 
Andrew Donnellan              OzLabs, ADL Canberra
ajd@linux.ibm.com             IBM Australia Limited

Re: [PATCH 3/3] powerpc/pseries: Fixup config space size of OpenCAPI devices

[christophe lombard]

On 05/11/2019 06:01, Andrew Donnellan wrote:
> On 22/10/19 6:52 pm, christophe lombard wrote:
>> Fix up the pci config size of the OpenCAPI PCIe devices in the pseries
>> environment.
>> Most of OpenCAPI PCIe devices have 4096 bytes of configuration space.
> 
> It's not "most of", it's "all" - the OpenCAPI Discovery and 
> Configuration Spec requires the use of extended capabilities that fall 
> in the 0x100-0xFFF range.
> 
>>
>> Signed-off-by: Christophe Lombard <clombard@linux.vnet.ibm.com>
>> ---
>>   arch/powerpc/platforms/pseries/pci.c | 9 +++++++++
>>   1 file changed, 9 insertions(+)
>>
>> diff --git a/arch/powerpc/platforms/pseries/pci.c 
>> b/arch/powerpc/platforms/pseries/pci.c
>> index 1eae1d09980c..3397784767b0 100644
>> --- a/arch/powerpc/platforms/pseries/pci.c
>> +++ b/arch/powerpc/platforms/pseries/pci.c
>> @@ -291,6 +291,15 @@ static void fixup_winbond_82c105(struct pci_dev* 
>> dev)
>>   DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_WINBOND, 
>> PCI_DEVICE_ID_WINBOND_82C105,
>>                fixup_winbond_82c105);
>> +static void fixup_opencapi_cfg_size(struct pci_dev *pdev)
>> +{
>> +    if (!machine_is(pseries))
>> +        return;
>> +
>> +    pdev->cfg_size = PCI_CFG_SPACE_EXP_SIZE;
>> +}
>> +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_IBM, 0x062b, 
>> fixup_opencapi_cfg_size);
> 
> An OpenCAPI device can have any PCI ID, is there a particular reason 
> we're limiting this to 1014:062b? On PowerNV, we check the PHB type to 
> determine whether the device is OpenCAPI or not, what's the equivalent 
> for pseries?
> 

Thanks for the review. For pseries, there is no specific OpenCapi PHB 
type which constraints this kind of request.
We are working to found an other solution.

>> +
>>   int pseries_root_bridge_prepare(struct pci_host_bridge *bridge)
>>   {
>>       struct device_node *dn, *pdn;
>>
> 

Re: [PATCH 3/3] powerpc/pseries: Fixup config space size of OpenCAPI devices

[Greg Kurz]

On Thu, 7 Nov 2019 09:46:25 +0100
christophe lombard <clombard@linux.vnet.ibm.com> wrote:

> On 05/11/2019 06:01, Andrew Donnellan wrote:
> > On 22/10/19 6:52 pm, christophe lombard wrote:
> >> Fix up the pci config size of the OpenCAPI PCIe devices in the pseries
> >> environment.
> >> Most of OpenCAPI PCIe devices have 4096 bytes of configuration space.
> > 
> > It's not "most of", it's "all" - the OpenCAPI Discovery and 
> > Configuration Spec requires the use of extended capabilities that fall 
> > in the 0x100-0xFFF range.
> > 
> >>
> >> Signed-off-by: Christophe Lombard <clombard@linux.vnet.ibm.com>
> >> ---
> >>   arch/powerpc/platforms/pseries/pci.c | 9 +++++++++
> >>   1 file changed, 9 insertions(+)
> >>
> >> diff --git a/arch/powerpc/platforms/pseries/pci.c 
> >> b/arch/powerpc/platforms/pseries/pci.c
> >> index 1eae1d09980c..3397784767b0 100644
> >> --- a/arch/powerpc/platforms/pseries/pci.c
> >> +++ b/arch/powerpc/platforms/pseries/pci.c
> >> @@ -291,6 +291,15 @@ static void fixup_winbond_82c105(struct pci_dev* 
> >> dev)
> >>   DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_WINBOND, 
> >> PCI_DEVICE_ID_WINBOND_82C105,
> >>                fixup_winbond_82c105);
> >> +static void fixup_opencapi_cfg_size(struct pci_dev *pdev)
> >> +{
> >> +    if (!machine_is(pseries))
> >> +        return;
> >> +
> >> +    pdev->cfg_size = PCI_CFG_SPACE_EXP_SIZE;
> >> +}
> >> +DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_IBM, 0x062b, 
> >> fixup_opencapi_cfg_size);
> > 
> > An OpenCAPI device can have any PCI ID, is there a particular reason 
> > we're limiting this to 1014:062b? On PowerNV, we check the PHB type to 
> > determine whether the device is OpenCAPI or not, what's the equivalent 
> > for pseries?
> > 
> 
> Thanks for the review. For pseries, there is no specific OpenCapi PHB 
> type which constraints this kind of request.
> We are working to found an other solution.
> 

Well... we have an old PAPR+ addendum draft that mentions an "open-capi"
PHB type. The specification was never finalized and AFAIK PowerVM doesn't
support the OpenCAPI interface, so we didn't stick to the addendum during
our in-house prototyping. But now that we want to upstream things, I think
we should probably come up with a dedicated PHB type.

> >> +
> >>   int pseries_root_bridge_prepare(struct pci_host_bridge *bridge)
> >>   {
> >>       struct device_node *dn, *pdn;
> >>
> > 
>